mirror of
https://github.com/basicmachines-co/basic-memory
synced 2026-06-21 13:47:35 +00:00
Compare commits
3 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| d175879940 | |||
| de8eac6638 | |||
| 9fac9a9416 |
@@ -6,14 +6,14 @@
|
||||
},
|
||||
"metadata": {
|
||||
"description": "Official Basic Memory plugins from the canonical basic-memory repository",
|
||||
"version": "0.22.1"
|
||||
"version": "0.22.0"
|
||||
},
|
||||
"plugins": [
|
||||
{
|
||||
"name": "basic-memory",
|
||||
"source": "./plugins/claude-code",
|
||||
"description": "The bridge between Claude's working memory and Basic Memory's durable knowledge graph \u2014 session briefings, pre-compaction checkpoints, and capture reflexes",
|
||||
"version": "0.22.1",
|
||||
"version": "0.22.0",
|
||||
"author": {
|
||||
"name": "Basic Machines"
|
||||
},
|
||||
|
||||
@@ -1,49 +1,5 @@
|
||||
# CHANGELOG
|
||||
|
||||
## v0.22.1 (2026-06-12)
|
||||
|
||||
Follow-up patch to v0.22.0. Fixes project and default-project resolution on
|
||||
fresh installs, MCP workspace routing, sync project selection, and CLI startup
|
||||
latency, plus a few MCP parity additions.
|
||||
|
||||
### Features
|
||||
|
||||
- Added a `workspace` parameter to `write_note` for parity with `edit_note`.
|
||||
- **#826**: Added `title` and `tags` annotations to all MCP tool decorators
|
||||
(phase 1).
|
||||
- **#930**: `search_notes` now comma-splits `note_types`, `entity_types`, and
|
||||
`categories`.
|
||||
- **#971**: Added the manual-pages flow — manpage seed schema, flow docs, and
|
||||
verification fixes.
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Fresh installs no longer fail when the projects table is empty: resolve now
|
||||
points them at project setup, the first project is promoted to default when
|
||||
the config default is missing from the database, the promoted default state
|
||||
is returned from the project-create API, and a default can be set when none
|
||||
is currently set. An existing database default is preserved when repairing a
|
||||
missing config default.
|
||||
- **#949**: Sync skips projects without an absolute local path and excludes
|
||||
orphan DB projects that are absent from config.
|
||||
- **#952 / #981**: Resolved workspace display names and tenant ids in qualified
|
||||
project routes, closing out the manual verification findings.
|
||||
- `note_types`/`entity_types`/`categories` are normalized on the direct-call
|
||||
path, with non-string list elements rejected.
|
||||
- Vector-search hydration keys on `(type, id)` to prevent id collisions.
|
||||
- `file_utils` requires line-anchored frontmatter fences.
|
||||
- CLI startup is faster: FastAPI and app imports are deferred out of CLI
|
||||
startup, and rich/typer modules are preloaded before an in-place upgrade.
|
||||
- `config.json` is written atomically.
|
||||
- In-memory SQLite sessions are serialized so concurrent rollbacks cannot
|
||||
destroy writes.
|
||||
|
||||
### Maintenance
|
||||
|
||||
- Release recipes route through PRs and wait for the release PR merge to land
|
||||
before tagging; the release runbook is refreshed and stripped of
|
||||
user-specific absolute paths.
|
||||
|
||||
## v0.22.0 (2026-06-11)
|
||||
|
||||
Team-safe cloud sync. New additive `bm cloud push` and `bm cloud pull`
|
||||
|
||||
@@ -107,6 +107,76 @@ All settings are fields on `BasicMemoryConfig` and can be set via environment va
|
||||
| `semantic_embedding_document_input_type` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_DOCUMENT_INPUT_TYPE` | Auto for known LiteLLM models | Optional LiteLLM `input_type` for indexed document/passages. |
|
||||
| `semantic_embedding_query_input_type` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_QUERY_INPUT_TYPE` | Auto for known LiteLLM models | Optional LiteLLM `input_type` for search queries. |
|
||||
| `semantic_vector_k` | `BASIC_MEMORY_SEMANTIC_VECTOR_K` | `100` | Candidate count for vector nearest-neighbour retrieval. Higher values improve recall at the cost of latency. |
|
||||
| `search_entity_boost_enabled` | `BASIC_MEMORY_SEARCH_ENTITY_BOOST_ENABLED` | `false` | Enable the entity-aware ranking boost in hybrid search (see below). Default off: benchmark-validated as inert on LoCoMo and prone to Title-Case false positives. |
|
||||
| `search_entity_boost_weight` | `BASIC_MEMORY_SEARCH_ENTITY_BOOST_WEIGHT` | `0.15` | Per-matched-term multiplier strength for the entity boost. A candidate matching N query entity terms is scaled by `1 + weight * min(N, max_terms)`. |
|
||||
| `search_entity_boost_max_terms` | `BASIC_MEMORY_SEARCH_ENTITY_BOOST_MAX_TERMS` | `3` | Maximum number of distinct matched entity terms that contribute to the boost, bounding the multiplier. |
|
||||
|
||||
## Entity-Aware Ranking Boost
|
||||
|
||||
Hybrid search fuses keyword (FTS) and vector similarity, but proper nouns in a query
|
||||
carry no special weight against generic semantic similarity. As a result, a document
|
||||
about a *different* entity on the same topic can outrank the document that actually
|
||||
names the queried entity — e.g. "What are Joanna's hobbies?" surfacing a generic
|
||||
hobbies note ahead of Joanna's note (see
|
||||
[#951](https://github.com/basicmachines-co/basic-memory/issues/951)).
|
||||
|
||||
When `search_entity_boost_enabled=true`, hybrid retrieval performs a final,
|
||||
lexical-only re-scoring pass:
|
||||
|
||||
1. It extracts candidate entity terms from the query — capitalized / proper-noun
|
||||
tokens that are not common stopwords (e.g. `Joanna`, `Anthony`, `NASA`).
|
||||
2. For each fused candidate, it counts how many distinct query entity terms appear in
|
||||
the candidate's entity name (its title) or in a relation row's linked entity names.
|
||||
3. Matching candidates have their fused score multiplied by
|
||||
`1 + weight * min(matches, max_terms)`, so an entity-matching document can be
|
||||
promoted above a higher-similarity non-matching one.
|
||||
|
||||
The boost adds **no model inference** — it is pure index/lexical lookup, so per-query
|
||||
latency overhead is trivial. It only affects `hybrid` retrieval; `text` and `vector`
|
||||
modes are unchanged. Non-matching candidates keep their original scores, so ordering
|
||||
among them is preserved.
|
||||
|
||||
```bash
|
||||
export BASIC_MEMORY_SEARCH_ENTITY_BOOST_ENABLED=true
|
||||
# Optional tuning:
|
||||
export BASIC_MEMORY_SEARCH_ENTITY_BOOST_WEIGHT=0.15
|
||||
export BASIC_MEMORY_SEARCH_ENTITY_BOOST_MAX_TERMS=3
|
||||
```
|
||||
|
||||
> **Default off.** This setting is disabled by default. See the benchmark
|
||||
> findings below for why the default stays off and where the boost helps.
|
||||
|
||||
### Benchmark findings
|
||||
|
||||
The boost was benchmarked against LoCoMo (the
|
||||
[basic-memory-benchmarks](https://github.com/basicmachines-co/basic-memory-benchmarks)
|
||||
retrieval suite, hybrid mode) and a hand-built adversarial corpus. Two results
|
||||
drove the decision to keep the default **off** and leave the weight at `0.15`:
|
||||
|
||||
1. **LoCoMo is insensitive to the boost.** Sweeping the weight across
|
||||
`0.15, 0.3, 0.5, 1.0, 2.0` produced *identical* recall@5, recall@10, MRR, and
|
||||
content-hit at every point — no query reordered, no score changed. LoCoMo's
|
||||
documents are titled by conversation/session id and expose speaker names only
|
||||
in body text, never as entity titles or relation names. Because the boost
|
||||
matches query proper nouns against a candidate's **title or linked relation
|
||||
names**, it never fires on this corpus. LoCoMo therefore provides no signal to
|
||||
raise the weight, and the boost neither helps nor harms it.
|
||||
|
||||
2. **A capitalization-only heuristic has false positives.** On a corpus where
|
||||
entity terms appear in titles, the boost correctly promotes the right document
|
||||
for clean proper nouns (e.g. `Katze`) and is correctly inert on
|
||||
lowercase-leading identifiers (e.g. `getUserById`, ignored). But **Title-Case
|
||||
queries can regress**: a query like `What Is The Plan For Q3` extracts `Q3` as
|
||||
an entity term, and even at weight `0.15` it promotes a document that
|
||||
*literally* contains "Q3" above the more relevant document that says "third
|
||||
quarter". Since entity detection is lexical (capitalization, no NER), any
|
||||
capitalized non-entity token in a query is a potential false positive.
|
||||
|
||||
**Guidance.** Enable the boost only on entity-heavy corpora where your queries
|
||||
name entities that are themselves note titles or linked relations (the #951
|
||||
"Joanna" case). Prefer natural-case queries (`What are Joanna's hobbies?`) over
|
||||
Title-Cased phrasing, which can inject spurious entity terms. Leave it off for
|
||||
conversational / body-text-keyed corpora like LoCoMo, where it cannot help.
|
||||
|
||||
## Embedding Providers
|
||||
|
||||
|
||||
@@ -38,7 +38,7 @@ from typing import Any, Callable
|
||||
from agent.memory_provider import MemoryProvider
|
||||
from tools.registry import tool_error
|
||||
|
||||
__version__ = "0.22.1"
|
||||
__version__ = "0.22.0"
|
||||
|
||||
logger = logging.getLogger("hermes.memory.basic-memory")
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
name: basic-memory
|
||||
version: 0.22.1
|
||||
version: 0.22.0
|
||||
description: "Basic Memory — persistent knowledge graph backed by the basic-memory MCP server"
|
||||
pip_dependencies:
|
||||
- mcp
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@basicmemory/openclaw-basic-memory",
|
||||
"version": "0.22.1",
|
||||
"version": "0.22.0",
|
||||
"type": "module",
|
||||
"main": "./dist/index.js",
|
||||
"types": "./dist/index.d.ts",
|
||||
|
||||
@@ -6,14 +6,14 @@
|
||||
},
|
||||
"metadata": {
|
||||
"description": "Official plugins from Basic Machines for knowledge management and AI-assisted development",
|
||||
"version": "0.22.1"
|
||||
"version": "0.22.0"
|
||||
},
|
||||
"plugins": [
|
||||
{
|
||||
"name": "basic-memory",
|
||||
"source": "./",
|
||||
"description": "The bridge between Claude's working memory and Basic Memory's durable knowledge graph \u2014 session briefings, pre-compaction checkpoints, and capture reflexes",
|
||||
"version": "0.22.1",
|
||||
"version": "0.22.0",
|
||||
"author": {
|
||||
"name": "Basic Machines"
|
||||
},
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "basic-memory",
|
||||
"description": "The bridge between Claude's working memory and Basic Memory's durable knowledge graph \u2014 session briefings, pre-compaction checkpoints, and capture reflexes",
|
||||
"version": "0.22.1",
|
||||
"version": "0.22.0",
|
||||
"author": {
|
||||
"name": "Basic Machines"
|
||||
},
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "codex",
|
||||
"version": "0.22.1",
|
||||
"version": "0.22.0",
|
||||
"description": "A Codex-native bridge to Basic Memory for durable engineering context, decisions, and resumable checkpoints.",
|
||||
"author": {
|
||||
"name": "Basic Machines",
|
||||
|
||||
+2
-2
@@ -6,12 +6,12 @@
|
||||
"url": "https://github.com/basicmachines-co/basic-memory.git",
|
||||
"source": "github"
|
||||
},
|
||||
"version": "0.22.1",
|
||||
"version": "0.22.0",
|
||||
"packages": [
|
||||
{
|
||||
"registryType": "pypi",
|
||||
"identifier": "basic-memory",
|
||||
"version": "0.22.1",
|
||||
"version": "0.22.0",
|
||||
"runtimeHint": "uvx",
|
||||
"runtimeArguments": [
|
||||
{
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
"""basic-memory - Local-first knowledge management combining Zettelkasten with knowledge graphs"""
|
||||
|
||||
# Package version - updated by release automation
|
||||
__version__ = "0.22.1"
|
||||
__version__ = "0.22.0"
|
||||
|
||||
# API version for FastAPI - independent of package version
|
||||
__api_version__ = "v0"
|
||||
|
||||
+36
-20
@@ -346,6 +346,42 @@ class BasicMemoryConfig(BaseSettings):
|
||||
"Valid values: text, vector, hybrid. "
|
||||
"When unset, defaults to 'hybrid' if semantic search is enabled, otherwise 'text'.",
|
||||
)
|
||||
# Entity-aware ranking boost (hybrid retrieval).
|
||||
# Trigger: proper nouns in a query (e.g. "Joanna") carry no extra weight against
|
||||
# generic semantic similarity, so documents from the wrong conversation can outrank
|
||||
# the gold document during hybrid fusion (#951).
|
||||
# Why: entities are first-class in Basic Memory, so a candidate whose title or linked
|
||||
# relation names contain a query proper noun is a stronger answer than a same-topic
|
||||
# document about a different entity.
|
||||
# Outcome: when enabled, hybrid fusion multiplies a candidate's fused score by a small
|
||||
# bonus for each distinct query entity term it matches lexically (no model inference).
|
||||
# Default OFF: LoCoMo benchmarking showed the boost is inert there (its docs are keyed
|
||||
# by session id, not entity titles) and an adversarial check found Title-Case queries
|
||||
# can inject spurious entity terms (e.g. "Q3") that regress ranking. See
|
||||
# docs/semantic-search.md "Benchmark findings".
|
||||
search_entity_boost_enabled: bool = Field(
|
||||
default=False,
|
||||
description="Enable entity-aware ranking boost in hybrid search. When enabled, "
|
||||
"hybrid candidates whose title or linked relation names contain a proper-noun "
|
||||
"term from the query are boosted in the final ranking. Lexical-only; adds no "
|
||||
"model inference. Default off: benchmark-validated as inert on LoCoMo and prone "
|
||||
"to Title-Case false positives (see docs/semantic-search.md).",
|
||||
)
|
||||
search_entity_boost_weight: float = Field(
|
||||
default=0.15,
|
||||
description="Per-matched-term multiplier strength for the entity-aware ranking "
|
||||
"boost. A candidate matching N distinct query entity terms has its fused score "
|
||||
"multiplied by (1 + weight * N), capped at search_entity_boost_max_terms terms. "
|
||||
"Only applies when search_entity_boost_enabled is true.",
|
||||
ge=0.0,
|
||||
)
|
||||
search_entity_boost_max_terms: int = Field(
|
||||
default=3,
|
||||
description="Maximum number of distinct matched entity terms that contribute to "
|
||||
"the entity-aware ranking boost, bounding the multiplier so a single candidate "
|
||||
"cannot run away with the ranking.",
|
||||
gt=0,
|
||||
)
|
||||
|
||||
# Database connection pool configuration (Postgres only)
|
||||
db_pool_size: int = Field(
|
||||
@@ -658,26 +694,6 @@ class BasicMemoryConfig(BaseSettings):
|
||||
entry = self.projects.get(project_name)
|
||||
return entry.mode if entry else ProjectMode.CLOUD
|
||||
|
||||
def is_locally_syncable(self, project_name: str, project_path: str) -> bool:
|
||||
"""Whether a project should be synced/watched on the local filesystem.
|
||||
|
||||
Both conditions are required (issue #949):
|
||||
|
||||
* The project is present in config. Config is the source of truth, so a
|
||||
stale database row that was removed from config — but whose deletion
|
||||
has not yet been reconciled, or whose reconciliation failed — must
|
||||
not be synced even though it still has a real directory on disk.
|
||||
* Its path is absolute. An empty or relative path resolves against the
|
||||
process cwd, so syncing it would adopt whatever directory the server
|
||||
was launched from as the project root and mutate unrelated files.
|
||||
|
||||
Cloud-only projects (empty/slug path) and cloud projects with a real
|
||||
local bisync copy (absolute path) are handled correctly by these two
|
||||
conditions, so no separate mode check is needed.
|
||||
"""
|
||||
entry = self.projects.get(project_name)
|
||||
return entry is not None and Path(project_path).is_absolute()
|
||||
|
||||
def set_project_mode(self, project_name: str, mode: ProjectMode) -> None:
|
||||
"""Set the routing mode for a project.
|
||||
|
||||
|
||||
@@ -67,6 +67,9 @@ class PostgresSearchRepository(SearchRepositoryBase):
|
||||
self._semantic_postgres_prepare_concurrency = (
|
||||
self._app_config.semantic_postgres_prepare_concurrency
|
||||
)
|
||||
self._entity_boost_enabled = self._app_config.search_entity_boost_enabled
|
||||
self._entity_boost_weight = self._app_config.search_entity_boost_weight
|
||||
self._entity_boost_max_terms = self._app_config.search_entity_boost_max_terms
|
||||
self._embedding_provider = embedding_provider
|
||||
self._vector_dimensions = 384
|
||||
self._vector_tables_initialized = False
|
||||
|
||||
@@ -45,6 +45,64 @@ _SQLITE_MAX_PREPARE_WINDOW = 8
|
||||
# the vector/hybrid retrieval path must key rows by (type, id) to avoid collisions.
|
||||
type SearchIndexKey = tuple[str, int]
|
||||
|
||||
# --- Entity-aware ranking boost (#951) ---
|
||||
|
||||
# Match word tokens (allowing internal apostrophes/hyphens) so we can inspect
|
||||
# their capitalization to detect proper-noun-like query terms.
|
||||
_ENTITY_TERM_TOKEN_PATTERN = re.compile(r"[A-Za-z][A-Za-z'\-]*")
|
||||
|
||||
# Common capitalized sentence-starters and interrogatives that look like proper
|
||||
# nouns but are not entity references. Kept lowercase for case-insensitive checks.
|
||||
# Intentionally small: a candidate term only boosts a row when it actually matches
|
||||
# that row's title/relation names, so a stray non-entity term simply does nothing.
|
||||
_ENTITY_TERM_STOPWORDS = frozenset(
|
||||
{
|
||||
"a",
|
||||
"an",
|
||||
"and",
|
||||
"are",
|
||||
"as",
|
||||
"at",
|
||||
"be",
|
||||
"but",
|
||||
"by",
|
||||
"do",
|
||||
"does",
|
||||
"for",
|
||||
"from",
|
||||
"has",
|
||||
"have",
|
||||
"how",
|
||||
"i",
|
||||
"in",
|
||||
"is",
|
||||
"it",
|
||||
"of",
|
||||
"on",
|
||||
"or",
|
||||
"the",
|
||||
"their",
|
||||
"they",
|
||||
"this",
|
||||
"to",
|
||||
"was",
|
||||
"we",
|
||||
"were",
|
||||
"what",
|
||||
"when",
|
||||
"where",
|
||||
"which",
|
||||
"who",
|
||||
"whom",
|
||||
"whose",
|
||||
"why",
|
||||
"will",
|
||||
"with",
|
||||
"you",
|
||||
"your",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class VectorSyncBatchResult:
|
||||
@@ -166,6 +224,13 @@ class SearchRepositoryBase(ABC):
|
||||
_vector_dimensions: int
|
||||
_vector_tables_initialized: bool
|
||||
|
||||
# Entity-aware ranking boost (#951). Defaults keep the feature off for any
|
||||
# subclass or test double that does not explicitly configure it. Concrete
|
||||
# backends overwrite these from BasicMemoryConfig in their __init__.
|
||||
_entity_boost_enabled: bool = False
|
||||
_entity_boost_weight: float = 0.0
|
||||
_entity_boost_max_terms: int = 1
|
||||
|
||||
def __init__(self, session_maker: async_sessionmaker[AsyncSession], project_id: int):
|
||||
"""Initialize with session maker and project_id filter.
|
||||
|
||||
@@ -2147,6 +2212,105 @@ class SearchRepositoryBase(ABC):
|
||||
# Shared semantic search: hybrid score-based fusion
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
# --- Entity-aware ranking boost (#951) ---
|
||||
|
||||
@staticmethod
|
||||
def _extract_query_entity_terms(search_text: Optional[str]) -> set[str]:
|
||||
"""Extract candidate entity (proper-noun) terms from a query string.
|
||||
|
||||
Heuristic, lexical only (no model inference): a token is a candidate entity
|
||||
term when it is title-cased or all-caps and is not a common stopword. The
|
||||
result is lowercased so downstream matching is case-insensitive.
|
||||
|
||||
Examples:
|
||||
"What are Joanna's hobbies?" -> {"joanna"}
|
||||
"Who is Anthony?" -> {"anthony"}
|
||||
"Deborah and Jolene" -> {"deborah", "jolene"}
|
||||
"what is the weather" -> set() (no proper nouns)
|
||||
"""
|
||||
if not search_text:
|
||||
return set()
|
||||
|
||||
terms: set[str] = set()
|
||||
for match in _ENTITY_TERM_TOKEN_PATTERN.finditer(search_text):
|
||||
token = match.group(0)
|
||||
# Trigger: token begins with an uppercase letter (Title-Case or ALL-CAPS).
|
||||
# Why: proper nouns and named entities are conventionally capitalized; this
|
||||
# is the cheapest reliable signal without a NER model.
|
||||
# Outcome: lowercase, non-capitalized words are ignored as generic terms.
|
||||
if not token[0].isupper():
|
||||
continue
|
||||
normalized = token.lower()
|
||||
# Strip a trailing possessive so "Joanna's" matches the entity "Joanna".
|
||||
if normalized.endswith("'s"):
|
||||
normalized = normalized[:-2]
|
||||
if normalized in _ENTITY_TERM_STOPWORDS:
|
||||
continue
|
||||
# Single characters (e.g. a stray "I") carry no entity signal.
|
||||
if len(normalized) < 2:
|
||||
continue
|
||||
terms.add(normalized)
|
||||
return terms
|
||||
|
||||
@staticmethod
|
||||
def _row_entity_match_count(row: SearchIndexRow, entity_terms: set[str]) -> int:
|
||||
"""Count distinct query entity terms that a candidate row references.
|
||||
|
||||
Matches against the row's own entity name (title) and the names embedded in
|
||||
a relation row's title (``"From -> To"``). These are the fields where Basic
|
||||
Memory's first-class entity names surface, so a match here is strong evidence
|
||||
the candidate is about the queried entity rather than a same-topic document.
|
||||
"""
|
||||
if not entity_terms:
|
||||
return 0
|
||||
|
||||
haystack_parts = [row.title or ""]
|
||||
# Relation rows encode linked entity names in their title ("From -> To");
|
||||
# the relation_type itself is not an entity name, so it is excluded.
|
||||
haystack = " ".join(part for part in haystack_parts if part)
|
||||
if not haystack:
|
||||
return 0
|
||||
|
||||
haystack_tokens: set[str] = set()
|
||||
for match in _ENTITY_TERM_TOKEN_PATTERN.finditer(haystack):
|
||||
token = match.group(0).lower()
|
||||
# Mirror the query-side possessive stripping so a doc titled
|
||||
# "Joanna's Hobbies" matches the query entity term "joanna".
|
||||
if token.endswith("'s"):
|
||||
token = token[:-2]
|
||||
haystack_tokens.add(token)
|
||||
return len(entity_terms & haystack_tokens)
|
||||
|
||||
def _apply_entity_boost(
|
||||
self,
|
||||
fused_scores: dict[SearchIndexKey, float],
|
||||
rows_by_key: dict[SearchIndexKey, SearchIndexRow],
|
||||
entity_terms: set[str],
|
||||
) -> dict[SearchIndexKey, float]:
|
||||
"""Multiply fused scores by a per-matched-term bonus for entity-matching rows.
|
||||
|
||||
Trigger: entity boosting is enabled and the query contains proper-noun terms.
|
||||
Why: a candidate whose entity/relation names contain a queried proper noun is a
|
||||
stronger answer than a generic same-topic document (#951 cross-conversation
|
||||
confusion).
|
||||
Outcome: ``score * (1 + weight * min(matches, max_terms))``. Rows that match no
|
||||
query entity term are returned unchanged, so relative order among non-matching
|
||||
rows is preserved.
|
||||
"""
|
||||
if not self._entity_boost_enabled or not entity_terms or self._entity_boost_weight <= 0:
|
||||
return fused_scores
|
||||
|
||||
boosted: dict[SearchIndexKey, float] = {}
|
||||
for row_key, score in fused_scores.items():
|
||||
row = rows_by_key.get(row_key)
|
||||
matches = self._row_entity_match_count(row, entity_terms) if row is not None else 0
|
||||
if matches <= 0:
|
||||
boosted[row_key] = score
|
||||
continue
|
||||
capped_matches = min(matches, self._entity_boost_max_terms)
|
||||
boosted[row_key] = score * (1.0 + self._entity_boost_weight * capped_matches)
|
||||
return boosted
|
||||
|
||||
async def _search_hybrid(
|
||||
self,
|
||||
*,
|
||||
@@ -2250,6 +2414,15 @@ class SearchRepositoryBase(ABC):
|
||||
f = fts_scores.get(row_key, 0.0)
|
||||
fused_scores[row_key] = max(v, f) + FUSION_BONUS * min(v, f)
|
||||
|
||||
# Entity-aware ranking boost (#951): runs over the full fused candidate set
|
||||
# before the limit/offset cut, so a boosted entity-matching candidate can be
|
||||
# promoted into the returned window. No-op when the feature is disabled or the
|
||||
# query contains no proper-noun terms, preserving the existing ordering.
|
||||
entity_terms = (
|
||||
self._extract_query_entity_terms(query_text) if self._entity_boost_enabled else set()
|
||||
)
|
||||
fused_scores = self._apply_entity_boost(fused_scores, rows_by_key, entity_terms)
|
||||
|
||||
ranked = sorted(fused_scores.items(), key=lambda item: item[1], reverse=True)
|
||||
output: list[SearchIndexRow] = []
|
||||
for row_key, fused_score in ranked[offset : offset + limit]:
|
||||
|
||||
@@ -55,6 +55,9 @@ class SQLiteSearchRepository(SearchRepositoryBase):
|
||||
self._semantic_embedding_sync_batch_size = (
|
||||
self._app_config.semantic_embedding_sync_batch_size
|
||||
)
|
||||
self._entity_boost_enabled = self._app_config.search_entity_boost_enabled
|
||||
self._entity_boost_weight = self._app_config.search_entity_boost_weight
|
||||
self._entity_boost_max_terms = self._app_config.search_entity_boost_max_terms
|
||||
self._embedding_provider = embedding_provider
|
||||
self._sqlite_vec_load_lock = asyncio.Lock()
|
||||
self._sqlite_prepare_write_lock = asyncio.Lock()
|
||||
|
||||
@@ -13,7 +13,7 @@ from pathlib import Path
|
||||
from loguru import logger
|
||||
|
||||
from basic_memory import db
|
||||
from basic_memory.config import BasicMemoryConfig, DatabaseBackend
|
||||
from basic_memory.config import BasicMemoryConfig, DatabaseBackend, ProjectMode
|
||||
from basic_memory.models import Project
|
||||
from basic_memory.repository import (
|
||||
ProjectRepository,
|
||||
@@ -120,14 +120,18 @@ async def initialize_file_sync(
|
||||
active_projects = [p for p in active_projects if p.name == constrained_project]
|
||||
logger.info(f"Background sync constrained to project: {constrained_project}")
|
||||
|
||||
# Only sync projects that are in config (source of truth) and have an
|
||||
# absolute local path; see BasicMemoryConfig.is_locally_syncable. This keeps
|
||||
# background sync from adopting the process cwd as a project root and
|
||||
# mutating unrelated files (issue #949).
|
||||
skip = [p.name for p in active_projects if not app_config.is_locally_syncable(p.name, p.path)]
|
||||
if skip:
|
||||
active_projects = [p for p in active_projects if p.name not in skip]
|
||||
logger.info(f"Skipping projects that are not locally syncable for sync: {skip}")
|
||||
# Skip cloud-mode projects that have no local directory.
|
||||
# Cloud projects with a local bisync copy (absolute path) are kept for local sync.
|
||||
cloud_skip = []
|
||||
for p in active_projects:
|
||||
if app_config.get_project_mode(p.name) == ProjectMode.CLOUD:
|
||||
entry = app_config.projects.get(p.name)
|
||||
if entry and Path(entry.path).is_absolute():
|
||||
continue # Cloud project with local bisync copy — keep for local sync
|
||||
cloud_skip.append(p.name)
|
||||
if cloud_skip:
|
||||
active_projects = [p for p in active_projects if p.name not in cloud_skip]
|
||||
logger.info(f"Skipping cloud-mode projects for local sync: {cloud_skip}")
|
||||
|
||||
# Start sync for all projects as background tasks (non-blocking)
|
||||
async def sync_project_background(project: Project):
|
||||
|
||||
@@ -10,7 +10,7 @@ from typing import List, Optional, Set, Sequence, Callable, Awaitable, TYPE_CHEC
|
||||
if TYPE_CHECKING:
|
||||
from basic_memory.sync.sync_service import SyncService
|
||||
|
||||
from basic_memory.config import BasicMemoryConfig, WATCH_STATUS_JSON
|
||||
from basic_memory.config import BasicMemoryConfig, ProjectMode, WATCH_STATUS_JSON
|
||||
from basic_memory.ignore_utils import load_gitignore_patterns, should_ignore_path
|
||||
from basic_memory.models import Project
|
||||
from basic_memory.repository import ProjectRepository
|
||||
@@ -184,24 +184,24 @@ class WatchService:
|
||||
``--project``, only that project is watched. This keeps concurrent
|
||||
MCP processes from producing duplicate watchers that race on the
|
||||
same files.
|
||||
2. Projects that are not locally syncable are skipped — those missing
|
||||
from config (config is the source of truth, so stale DB rows must
|
||||
not be watched) or with a non-absolute path (which would resolve
|
||||
against the process cwd and make the watcher observe and mutate the
|
||||
directory the server was launched from). See
|
||||
``BasicMemoryConfig.is_locally_syncable`` (issue #949). Cloud
|
||||
projects with a local bisync copy keep their absolute path and are
|
||||
still watched.
|
||||
2. Cloud-only projects without a local bisync copy are skipped so we
|
||||
don't watch a path that does not exist on disk.
|
||||
"""
|
||||
projects = await self.project_repository.get_active_projects()
|
||||
|
||||
if self.constrained_project:
|
||||
projects = [p for p in projects if p.name == self.constrained_project]
|
||||
|
||||
skip = [p.name for p in projects if not self.app_config.is_locally_syncable(p.name, p.path)]
|
||||
if skip:
|
||||
projects = [p for p in projects if p.name not in skip]
|
||||
logger.debug(f"Skipping projects that are not locally syncable in watch cycle: {skip}")
|
||||
cloud_skip: list[str] = []
|
||||
for p in projects:
|
||||
if self.app_config.get_project_mode(p.name) == ProjectMode.CLOUD:
|
||||
entry = self.app_config.projects.get(p.name)
|
||||
if entry and Path(entry.path).is_absolute():
|
||||
continue # Cloud project with local bisync copy — keep watching
|
||||
cloud_skip.append(p.name)
|
||||
if cloud_skip:
|
||||
projects = [p for p in projects if p.name not in cloud_skip]
|
||||
logger.debug(f"Skipping cloud-mode projects in watch cycle: {cloud_skip}")
|
||||
|
||||
return list(projects)
|
||||
|
||||
|
||||
@@ -0,0 +1,226 @@
|
||||
"""Service-level integration test for the entity-aware ranking boost (#951).
|
||||
|
||||
Drives a fully wired SearchService over a real database with a deterministic stub
|
||||
embedding provider so vector similarity is controlled. Verifies that when the boost
|
||||
is enabled, an entity-matching document outranks a higher-similarity non-matching
|
||||
document, and that ordering is unchanged when the boost is disabled.
|
||||
|
||||
No model inference is involved: the stub provider returns fixed unit vectors, so the
|
||||
test is fast and deterministic on both SQLite and Postgres.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
import pytest_asyncio
|
||||
|
||||
from basic_memory.config import BasicMemoryConfig
|
||||
from basic_memory.repository.entity_repository import EntityRepository
|
||||
from basic_memory.repository.search_repository import SearchRepository
|
||||
from basic_memory.schemas.base import Entity as EntitySchema
|
||||
from basic_memory.schemas.search import SearchQuery, SearchRetrievalMode
|
||||
from basic_memory.services.entity_service import EntityService
|
||||
from basic_memory.services.file_service import FileService
|
||||
from basic_memory.services.search_service import SearchService
|
||||
|
||||
|
||||
# --- Deterministic stub embedding provider ---
|
||||
|
||||
_STUB_DIMENSIONS = 4
|
||||
|
||||
|
||||
def _unit(vector: list[float]) -> list[float]:
|
||||
norm = math.sqrt(sum(component * component for component in vector)) or 1.0
|
||||
return [component / norm for component in vector]
|
||||
|
||||
|
||||
class _StubEmbeddingProvider:
|
||||
"""Maps known text fragments to fixed unit vectors for controlled similarity.
|
||||
|
||||
The query is engineered to sit closer (cosine) to the non-matching "hobbies"
|
||||
document than to the gold "Joanna" document, reproducing the #951 failure where
|
||||
generic semantic similarity outranks the entity-matching gold doc.
|
||||
"""
|
||||
|
||||
model_name = "stub-entity-boost"
|
||||
dimensions = _STUB_DIMENSIONS
|
||||
|
||||
def _vector_for(self, text: str) -> list[float]:
|
||||
lowered = text.lower()
|
||||
if "joanna" in lowered:
|
||||
# Gold doc: shares some direction with the query but less than the decoy.
|
||||
return _unit([0.6, 0.8, 0.0, 0.0])
|
||||
if "hobbies" in lowered or "pastime" in lowered:
|
||||
# Decoy doc: closest to the query direction.
|
||||
return _unit([0.95, 0.31, 0.0, 0.0])
|
||||
return _unit([0.0, 0.0, 1.0, 0.0])
|
||||
|
||||
async def embed_query(self, text: str) -> list[float]:
|
||||
# Query direction is closest to the decoy vector above.
|
||||
return _unit([0.97, 0.24, 0.0, 0.0])
|
||||
|
||||
async def embed_documents(self, texts: list[str]) -> list[list[float]]:
|
||||
return [self._vector_for(text) for text in texts]
|
||||
|
||||
def runtime_log_attrs(self) -> dict[str, Any]:
|
||||
return {}
|
||||
|
||||
|
||||
# --- Fixtures ---
|
||||
|
||||
|
||||
async def _build_search_service(
|
||||
*,
|
||||
session_maker,
|
||||
test_project,
|
||||
base_app_config: BasicMemoryConfig,
|
||||
file_service: FileService,
|
||||
entity_repository: EntityRepository,
|
||||
boost_enabled: bool,
|
||||
) -> SearchService:
|
||||
"""Build a SearchService with semantic search + a deterministic stub provider."""
|
||||
from basic_memory.repository.sqlite_search_repository import SQLiteSearchRepository
|
||||
from basic_memory.repository.postgres_search_repository import PostgresSearchRepository
|
||||
from basic_memory.config import DatabaseBackend
|
||||
|
||||
app_config = base_app_config.model_copy(
|
||||
update={
|
||||
"semantic_search_enabled": True,
|
||||
"semantic_min_similarity": 0.0,
|
||||
"search_entity_boost_enabled": boost_enabled,
|
||||
"search_entity_boost_weight": 0.3,
|
||||
"search_entity_boost_max_terms": 3,
|
||||
}
|
||||
)
|
||||
|
||||
provider = _StubEmbeddingProvider()
|
||||
if app_config.database_backend == DatabaseBackend.POSTGRES: # pragma: no cover
|
||||
search_repo: SearchRepository = PostgresSearchRepository(
|
||||
session_maker,
|
||||
project_id=test_project.id,
|
||||
app_config=app_config,
|
||||
embedding_provider=provider,
|
||||
)
|
||||
else:
|
||||
# Pass the stub provider at construction time so __init__ does not
|
||||
# instantiate the real configured provider when semantic_search_enabled=True.
|
||||
repo = SQLiteSearchRepository(
|
||||
session_maker,
|
||||
project_id=test_project.id,
|
||||
app_config=app_config,
|
||||
embedding_provider=provider,
|
||||
)
|
||||
search_repo = repo
|
||||
|
||||
service = SearchService(search_repo, entity_repository, file_service)
|
||||
await service.init_search_index()
|
||||
return service
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def boost_entities(
|
||||
entity_service: EntityService,
|
||||
):
|
||||
"""Index two entities: a decoy 'hobbies' doc and the gold 'Joanna' doc."""
|
||||
decoy, _ = await entity_service.create_or_update_entity(
|
||||
EntitySchema(
|
||||
title="Common Hobbies and Pastimes",
|
||||
note_type="note",
|
||||
directory="people",
|
||||
content="A general overview of hobbies and pastimes people enjoy.",
|
||||
)
|
||||
)
|
||||
gold, _ = await entity_service.create_or_update_entity(
|
||||
EntitySchema(
|
||||
title="Joanna",
|
||||
note_type="note",
|
||||
directory="people",
|
||||
content="Notes about Joanna and what she likes to do.",
|
||||
)
|
||||
)
|
||||
return decoy, gold
|
||||
|
||||
|
||||
# --- Tests ---
|
||||
|
||||
|
||||
async def _sync_vectors(service: SearchService, entity_ids: list[int]) -> None:
|
||||
"""Embed the indexed entities via the stub provider."""
|
||||
await service.sync_entity_vectors_batch(entity_ids)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_entity_boost_enabled_promotes_gold_doc(
|
||||
session_maker,
|
||||
test_project,
|
||||
app_config,
|
||||
file_service,
|
||||
entity_repository,
|
||||
boost_entities,
|
||||
):
|
||||
decoy, gold = boost_entities
|
||||
service = await _build_search_service(
|
||||
session_maker=session_maker,
|
||||
test_project=test_project,
|
||||
base_app_config=app_config,
|
||||
file_service=file_service,
|
||||
entity_repository=entity_repository,
|
||||
boost_enabled=True,
|
||||
)
|
||||
# Re-index the entities through this service so vector tables exist for it.
|
||||
for entity in (decoy, gold):
|
||||
await service.index_entity(entity)
|
||||
await _sync_vectors(service, [decoy.id, gold.id])
|
||||
|
||||
results = await service.search(
|
||||
SearchQuery(
|
||||
text="What are Joanna's hobbies?",
|
||||
retrieval_mode=SearchRetrievalMode.HYBRID,
|
||||
),
|
||||
limit=10,
|
||||
)
|
||||
|
||||
entity_ids = [r.entity_id for r in results]
|
||||
assert gold.id in entity_ids and decoy.id in entity_ids
|
||||
# With the boost on, the entity-matching gold doc ranks ahead of the
|
||||
# higher-similarity decoy.
|
||||
assert entity_ids.index(gold.id) < entity_ids.index(decoy.id)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_entity_boost_disabled_keeps_similarity_order(
|
||||
session_maker,
|
||||
test_project,
|
||||
app_config,
|
||||
file_service,
|
||||
entity_repository,
|
||||
boost_entities,
|
||||
):
|
||||
decoy, gold = boost_entities
|
||||
service = await _build_search_service(
|
||||
session_maker=session_maker,
|
||||
test_project=test_project,
|
||||
base_app_config=app_config,
|
||||
file_service=file_service,
|
||||
entity_repository=entity_repository,
|
||||
boost_enabled=False,
|
||||
)
|
||||
for entity in (decoy, gold):
|
||||
await service.index_entity(entity)
|
||||
await _sync_vectors(service, [decoy.id, gold.id])
|
||||
|
||||
results = await service.search(
|
||||
SearchQuery(
|
||||
text="What are Joanna's hobbies?",
|
||||
retrieval_mode=SearchRetrievalMode.HYBRID,
|
||||
),
|
||||
limit=10,
|
||||
)
|
||||
|
||||
entity_ids = [r.entity_id for r in results]
|
||||
assert gold.id in entity_ids and decoy.id in entity_ids
|
||||
# With the boost off, the higher-similarity decoy ranks ahead of the gold doc.
|
||||
assert entity_ids.index(decoy.id) < entity_ids.index(gold.id)
|
||||
@@ -134,6 +134,115 @@ HYBRID_KWARGS: dict[str, Any] = dict(
|
||||
)
|
||||
|
||||
|
||||
def _hybrid_kwargs(**overrides: Any) -> dict[str, Any]:
|
||||
"""Return HYBRID_KWARGS with overrides applied, typed as dict[str, Any].
|
||||
|
||||
Keeps the splat into the keyword-only _search_hybrid signature type-clean.
|
||||
"""
|
||||
merged: dict[str, Any] = {**HYBRID_KWARGS, **overrides}
|
||||
return merged
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_entity_boost_promotes_matching_doc_when_enabled():
|
||||
"""With entity boost enabled, an entity-matching doc outranks a higher-similarity
|
||||
non-matching doc.
|
||||
|
||||
Reproduces the #951 cross-conversation confusion: a generic same-topic document
|
||||
(higher raw similarity) initially outranks the gold doc whose title names the
|
||||
queried entity. Enabling the boost flips the order.
|
||||
"""
|
||||
repo = ConcreteSearchRepo()
|
||||
repo._entity_boost_enabled = True
|
||||
repo._entity_boost_weight = 0.15
|
||||
repo._entity_boost_max_terms = 3
|
||||
|
||||
# Row 1: generic hobbies doc from the wrong conversation, higher vector similarity.
|
||||
# Row 2: the gold doc whose title names the queried entity "Joanna".
|
||||
fts_results = []
|
||||
vector_results = [
|
||||
FakeRow(id=1, score=0.80, title="Hobbies and pastimes"),
|
||||
FakeRow(id=2, score=0.72, title="Joanna profile"),
|
||||
]
|
||||
|
||||
with (
|
||||
patch.object(repo, "search", new_callable=AsyncMock, return_value=fts_results),
|
||||
patch.object(
|
||||
repo, "_search_vector_only", new_callable=AsyncMock, return_value=vector_results
|
||||
),
|
||||
):
|
||||
results = await repo._search_hybrid(
|
||||
**_hybrid_kwargs(search_text="What are Joanna's hobbies?")
|
||||
)
|
||||
|
||||
# Boost: row 2 -> 0.72 * 1.15 = 0.828 > row 1's 0.80
|
||||
assert [r.id for r in results] == [2, 1]
|
||||
assert results[0].score == pytest.approx(0.72 * 1.15, rel=1e-6)
|
||||
assert results[1].score == pytest.approx(0.80, rel=1e-6)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_entity_boost_disabled_preserves_ordering():
|
||||
"""With entity boost disabled (default), ordering matches pure similarity."""
|
||||
repo = ConcreteSearchRepo()
|
||||
# Defaults from the base class keep boosting off; assert explicitly.
|
||||
assert repo._entity_boost_enabled is False
|
||||
|
||||
fts_results = []
|
||||
vector_results = [
|
||||
FakeRow(id=1, score=0.80, title="Hobbies and pastimes"),
|
||||
FakeRow(id=2, score=0.72, title="Joanna profile"),
|
||||
]
|
||||
|
||||
with (
|
||||
patch.object(repo, "search", new_callable=AsyncMock, return_value=fts_results),
|
||||
patch.object(
|
||||
repo, "_search_vector_only", new_callable=AsyncMock, return_value=vector_results
|
||||
),
|
||||
):
|
||||
results = await repo._search_hybrid(
|
||||
**_hybrid_kwargs(search_text="What are Joanna's hobbies?")
|
||||
)
|
||||
|
||||
# No boost: original similarity order is preserved, scores unchanged.
|
||||
assert [r.id for r in results] == [1, 2]
|
||||
assert results[0].score == pytest.approx(0.80, rel=1e-6)
|
||||
assert results[1].score == pytest.approx(0.72, rel=1e-6)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_entity_boost_promotes_doc_into_limited_window():
|
||||
"""Boosting runs before the limit cut, so a matching doc ranked below the cutoff
|
||||
can be promoted into the returned window."""
|
||||
repo = ConcreteSearchRepo()
|
||||
repo._entity_boost_enabled = True
|
||||
repo._entity_boost_weight = 0.6
|
||||
repo._entity_boost_max_terms = 3
|
||||
|
||||
fts_results = []
|
||||
# Three non-matching docs above the gold doc, which matches "Anthony".
|
||||
vector_results = [
|
||||
FakeRow(id=1, score=0.90, title="conversation six"),
|
||||
FakeRow(id=2, score=0.85, title="conversation one"),
|
||||
FakeRow(id=3, score=0.60, title="Anthony introduces himself"),
|
||||
]
|
||||
|
||||
with (
|
||||
patch.object(repo, "search", new_callable=AsyncMock, return_value=fts_results),
|
||||
patch.object(
|
||||
repo, "_search_vector_only", new_callable=AsyncMock, return_value=vector_results
|
||||
),
|
||||
):
|
||||
results = await repo._search_hybrid(
|
||||
**_hybrid_kwargs(search_text="Who is Anthony?", limit=1)
|
||||
)
|
||||
|
||||
# Gold doc boost: 0.60 * 1.6 = 0.96 > row 1's 0.90, so it is promoted into the
|
||||
# top-1 window even though it was ranked third before boosting.
|
||||
assert len(results) == 1
|
||||
assert results[0].id == 3
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_high_fts_score_boosts_ranking():
|
||||
"""FTS-only: a high normalized score should outscore a low normalized score."""
|
||||
|
||||
@@ -809,3 +809,182 @@ async def test_sync_entity_vectors_batch_logs_resolved_fastembed_runtime_setting
|
||||
assert runtime_logs[0]["threads"] == 4
|
||||
assert runtime_logs[0]["configured_parallel"] == 2
|
||||
assert runtime_logs[0]["effective_parallel"] == 2
|
||||
|
||||
|
||||
# --- Entity-aware ranking boost (#951) ---
|
||||
|
||||
|
||||
def _make_index_row(
|
||||
*,
|
||||
row_id: int,
|
||||
title: str,
|
||||
row_type: str = SearchItemType.ENTITY.value,
|
||||
) -> SearchIndexRow:
|
||||
"""Build a real SearchIndexRow for entity-boost matching tests."""
|
||||
now = datetime(2026, 1, 1)
|
||||
return SearchIndexRow(
|
||||
project_id=1,
|
||||
id=row_id,
|
||||
type=row_type,
|
||||
file_path=f"notes/{row_id}.md",
|
||||
created_at=now,
|
||||
updated_at=now,
|
||||
permalink=f"notes/{row_id}",
|
||||
title=title,
|
||||
)
|
||||
|
||||
|
||||
class TestExtractQueryEntityTerms:
|
||||
"""Verify proper-noun extraction from query strings."""
|
||||
|
||||
def test_extracts_single_proper_noun(self):
|
||||
terms = SearchRepositoryBase._extract_query_entity_terms("What are Joanna's hobbies?")
|
||||
assert terms == {"joanna"}
|
||||
|
||||
def test_extracts_multiple_proper_nouns(self):
|
||||
terms = SearchRepositoryBase._extract_query_entity_terms(
|
||||
"What symbolic gifts do Deborah and Jolene have from their mothers?"
|
||||
)
|
||||
assert terms == {"deborah", "jolene"}
|
||||
|
||||
def test_who_is_anthony(self):
|
||||
terms = SearchRepositoryBase._extract_query_entity_terms("Who is Anthony?")
|
||||
assert terms == {"anthony"}
|
||||
|
||||
def test_all_lowercase_query_has_no_entity_terms(self):
|
||||
terms = SearchRepositoryBase._extract_query_entity_terms("what is the weather today")
|
||||
assert terms == set()
|
||||
|
||||
def test_capitalized_stopword_at_sentence_start_is_ignored(self):
|
||||
# "What" and "Who" are capitalized interrogatives, not entity names.
|
||||
terms = SearchRepositoryBase._extract_query_entity_terms("What does Sarah like?")
|
||||
assert terms == {"sarah"}
|
||||
|
||||
def test_all_caps_token_is_extracted(self):
|
||||
terms = SearchRepositoryBase._extract_query_entity_terms("who works at NASA")
|
||||
assert terms == {"nasa"}
|
||||
|
||||
def test_possessive_is_stripped(self):
|
||||
terms = SearchRepositoryBase._extract_query_entity_terms("Joanna's mother")
|
||||
assert terms == {"joanna"}
|
||||
|
||||
def test_single_capital_letter_is_ignored(self):
|
||||
# "A" lowercases to a stopword; "X" is a non-stopword single letter that
|
||||
# still carries no entity signal and must be dropped by the length guard.
|
||||
terms = SearchRepositoryBase._extract_query_entity_terms("A is for X and Apple")
|
||||
assert terms == {"apple"}
|
||||
|
||||
def test_empty_and_none_inputs(self):
|
||||
assert SearchRepositoryBase._extract_query_entity_terms("") == set()
|
||||
assert SearchRepositoryBase._extract_query_entity_terms(None) == set()
|
||||
|
||||
|
||||
class TestRowEntityMatchCount:
|
||||
"""Verify lexical match counting between query terms and row entity names."""
|
||||
|
||||
def test_title_match_counts_one(self):
|
||||
row = _make_index_row(row_id=1, title="Joanna's Hobbies")
|
||||
assert SearchRepositoryBase._row_entity_match_count(row, {"joanna"}) == 1
|
||||
|
||||
def test_relation_title_matches_both_endpoints(self):
|
||||
row = _make_index_row(
|
||||
row_id=2,
|
||||
title="Deborah -> Jolene",
|
||||
row_type=SearchItemType.RELATION.value,
|
||||
)
|
||||
count = SearchRepositoryBase._row_entity_match_count(row, {"deborah", "jolene"})
|
||||
assert count == 2
|
||||
|
||||
def test_no_match_returns_zero(self):
|
||||
row = _make_index_row(row_id=3, title="Anthony Profile")
|
||||
assert SearchRepositoryBase._row_entity_match_count(row, {"joanna"}) == 0
|
||||
|
||||
def test_match_is_case_insensitive(self):
|
||||
row = _make_index_row(row_id=4, title="JOANNA notes")
|
||||
assert SearchRepositoryBase._row_entity_match_count(row, {"joanna"}) == 1
|
||||
|
||||
def test_empty_terms_returns_zero(self):
|
||||
row = _make_index_row(row_id=5, title="Joanna")
|
||||
assert SearchRepositoryBase._row_entity_match_count(row, set()) == 0
|
||||
|
||||
def test_missing_title_returns_zero(self):
|
||||
row = _make_index_row(row_id=6, title="")
|
||||
row.title = None
|
||||
assert SearchRepositoryBase._row_entity_match_count(row, {"joanna"}) == 0
|
||||
|
||||
def test_distinct_terms_not_double_counted(self):
|
||||
# A title containing the same term twice still counts as one distinct match.
|
||||
row = _make_index_row(row_id=7, title="Joanna and Joanna")
|
||||
assert SearchRepositoryBase._row_entity_match_count(row, {"joanna"}) == 1
|
||||
|
||||
|
||||
class TestApplyEntityBoost:
|
||||
"""Verify the entity-boost score math and gating."""
|
||||
|
||||
def _repo(self, *, enabled: bool, weight: float = 0.15, max_terms: int = 3) -> _ConcreteRepo:
|
||||
repo = _ConcreteRepo()
|
||||
repo._entity_boost_enabled = enabled
|
||||
repo._entity_boost_weight = weight
|
||||
repo._entity_boost_max_terms = max_terms
|
||||
return repo
|
||||
|
||||
def test_disabled_returns_scores_unchanged(self):
|
||||
repo = self._repo(enabled=False)
|
||||
key = ("entity", 1)
|
||||
rows = {key: _make_index_row(row_id=1, title="Joanna")}
|
||||
scores = {key: 0.5}
|
||||
assert repo._apply_entity_boost(scores, rows, {"joanna"}) == {key: 0.5}
|
||||
|
||||
def test_matching_row_is_boosted(self):
|
||||
repo = self._repo(enabled=True, weight=0.2)
|
||||
key = ("entity", 1)
|
||||
rows = {key: _make_index_row(row_id=1, title="Joanna")}
|
||||
boosted = repo._apply_entity_boost({key: 0.5}, rows, {"joanna"})
|
||||
assert boosted[key] == pytest.approx(0.5 * 1.2)
|
||||
|
||||
def test_non_matching_row_unchanged(self):
|
||||
repo = self._repo(enabled=True, weight=0.2)
|
||||
key = ("entity", 1)
|
||||
rows = {key: _make_index_row(row_id=1, title="Anthony")}
|
||||
boosted = repo._apply_entity_boost({key: 0.5}, rows, {"joanna"})
|
||||
assert boosted[key] == pytest.approx(0.5)
|
||||
|
||||
def test_boost_can_reorder_lower_scored_match_above_higher_non_match(self):
|
||||
repo = self._repo(enabled=True, weight=0.5)
|
||||
generic_key = ("entity", 1)
|
||||
joanna_key = ("entity", 2)
|
||||
rows = {
|
||||
generic_key: _make_index_row(row_id=1, title="Generic topic about hobbies"),
|
||||
joanna_key: _make_index_row(row_id=2, title="Joanna"),
|
||||
}
|
||||
# The generic row starts higher (0.6) but does not match; "Joanna" (0.5) matches.
|
||||
boosted = repo._apply_entity_boost({generic_key: 0.6, joanna_key: 0.5}, rows, {"joanna"})
|
||||
assert boosted[joanna_key] > boosted[generic_key]
|
||||
|
||||
def test_multiple_matches_scale_boost(self):
|
||||
repo = self._repo(enabled=True, weight=0.1, max_terms=3)
|
||||
key = ("relation", 1)
|
||||
rows = {key: _make_index_row(row_id=1, title="Deborah -> Jolene")}
|
||||
boosted = repo._apply_entity_boost({key: 1.0}, rows, {"deborah", "jolene"})
|
||||
# Two matched terms -> 1 + 0.1 * 2 = 1.2
|
||||
assert boosted[key] == pytest.approx(1.2)
|
||||
|
||||
def test_max_terms_caps_the_boost(self):
|
||||
repo = self._repo(enabled=True, weight=0.1, max_terms=1)
|
||||
key = ("relation", 1)
|
||||
rows = {key: _make_index_row(row_id=1, title="Deborah -> Jolene")}
|
||||
boosted = repo._apply_entity_boost({key: 1.0}, rows, {"deborah", "jolene"})
|
||||
# Capped at 1 term -> 1 + 0.1 * 1 = 1.1
|
||||
assert boosted[key] == pytest.approx(1.1)
|
||||
|
||||
def test_zero_weight_is_noop(self):
|
||||
repo = self._repo(enabled=True, weight=0.0)
|
||||
key = ("entity", 1)
|
||||
rows = {key: _make_index_row(row_id=1, title="Joanna")}
|
||||
assert repo._apply_entity_boost({key: 0.5}, rows, {"joanna"}) == {key: 0.5}
|
||||
|
||||
def test_empty_entity_terms_is_noop(self):
|
||||
repo = self._repo(enabled=True, weight=0.2)
|
||||
key = ("entity", 1)
|
||||
rows = {key: _make_index_row(row_id=1, title="Joanna")}
|
||||
assert repo._apply_entity_boost({key: 0.5}, rows, set()) == {key: 0.5}
|
||||
|
||||
@@ -222,57 +222,6 @@ async def test_initialize_file_sync_no_constraint_when_env_unset(
|
||||
assert _FakeWatchService.last_kwargs.get("constrained_project") is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_initialize_file_sync_skips_project_with_non_absolute_path(
|
||||
app_config: BasicMemoryConfig, config_manager, config_home, monkeypatch
|
||||
):
|
||||
"""Projects without an absolute local path are excluded from background sync (issue #949).
|
||||
|
||||
A config entry of ``{"path": ""}`` defaults to LOCAL mode and is not
|
||||
recognized as cloud, yet Path("") resolves to the process cwd. Syncing it
|
||||
would inject frontmatter into unrelated files, so it must be skipped.
|
||||
"""
|
||||
await db.shutdown_db()
|
||||
try:
|
||||
from basic_memory.config import ProjectEntry
|
||||
|
||||
good = config_home / "good"
|
||||
good.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
updated = app_config.model_copy(
|
||||
update={
|
||||
"projects": {
|
||||
"good": ProjectEntry(path=str(good)),
|
||||
# No mode -> defaults to LOCAL, empty (cwd-relative) path.
|
||||
"empty-path": ProjectEntry(path=""),
|
||||
},
|
||||
"default_project": "good",
|
||||
}
|
||||
)
|
||||
config_manager.save_config(updated)
|
||||
|
||||
await initialize_database(updated)
|
||||
await reconcile_projects_with_config(updated)
|
||||
|
||||
_disable_test_env_short_circuit(monkeypatch)
|
||||
monkeypatch.setattr("basic_memory.sync.WatchService", _FakeWatchService)
|
||||
|
||||
infos: list[str] = []
|
||||
monkeypatch.setattr(
|
||||
"basic_memory.services.initialization.logger.info",
|
||||
lambda message, *args, **kwargs: infos.append(message),
|
||||
)
|
||||
|
||||
await initialize_file_sync(updated, quiet=True)
|
||||
|
||||
skip_logs = [m for m in infos if "not locally syncable" in m]
|
||||
assert skip_logs, "expected a skip log for the empty-path project"
|
||||
assert "empty-path" in skip_logs[0]
|
||||
assert "good" not in skip_logs[0]
|
||||
finally:
|
||||
await db.shutdown_db()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_initialize_app_no_precedence_warning_when_not_conflicting(
|
||||
app_config: BasicMemoryConfig, monkeypatch
|
||||
|
||||
@@ -71,18 +71,15 @@ async def _register_local_projects(
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_select_projects_to_watch_returns_all_when_unconstrained(
|
||||
app_config: BasicMemoryConfig, project_repository, tmp_path
|
||||
app_config: BasicMemoryConfig, project_repository
|
||||
):
|
||||
"""Without a --project constraint, every active project is watched."""
|
||||
# Use tmp_path so the project paths are OS-absolute on Windows too — a
|
||||
# POSIX-style "/tmp/alpha" is not absolute on Windows (no drive letter),
|
||||
# and _select_projects_to_watch now skips non-absolute paths (issue #949).
|
||||
await _register_local_projects(
|
||||
app_config,
|
||||
project_repository,
|
||||
[
|
||||
{"name": "project-alpha", "path": str(tmp_path / "alpha")},
|
||||
{"name": "project-beta", "path": str(tmp_path / "beta")},
|
||||
{"name": "project-alpha", "path": "/tmp/alpha"},
|
||||
{"name": "project-beta", "path": "/tmp/beta"},
|
||||
],
|
||||
)
|
||||
|
||||
@@ -97,7 +94,7 @@ async def test_select_projects_to_watch_returns_all_when_unconstrained(
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_select_projects_to_watch_filters_to_constrained_project(
|
||||
app_config: BasicMemoryConfig, project_repository, tmp_path
|
||||
app_config: BasicMemoryConfig, project_repository
|
||||
):
|
||||
"""With ``constrained_project`` set, only that project is returned.
|
||||
|
||||
@@ -109,8 +106,8 @@ async def test_select_projects_to_watch_filters_to_constrained_project(
|
||||
app_config,
|
||||
project_repository,
|
||||
[
|
||||
{"name": "project-alpha", "path": str(tmp_path / "alpha")},
|
||||
{"name": "project-beta", "path": str(tmp_path / "beta")},
|
||||
{"name": "project-alpha", "path": "/tmp/alpha"},
|
||||
{"name": "project-beta", "path": "/tmp/beta"},
|
||||
],
|
||||
)
|
||||
|
||||
@@ -127,13 +124,13 @@ async def test_select_projects_to_watch_filters_to_constrained_project(
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_select_projects_to_watch_empty_when_constrained_project_missing(
|
||||
app_config: BasicMemoryConfig, project_repository, tmp_path
|
||||
app_config: BasicMemoryConfig, project_repository
|
||||
):
|
||||
"""An unknown constraint yields an empty watch set rather than watching everything."""
|
||||
await _register_local_projects(
|
||||
app_config,
|
||||
project_repository,
|
||||
[{"name": "project-alpha", "path": str(tmp_path / "alpha")}],
|
||||
[{"name": "project-alpha", "path": "/tmp/alpha"}],
|
||||
)
|
||||
|
||||
service = WatchService(
|
||||
|
||||
@@ -236,90 +236,6 @@ async def test_run_keeps_cloud_projects_with_local_bisync(monkeypatch, tmp_path)
|
||||
assert seen_project_names == [["local-project", "cloud-bisync"]]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_filters_empty_path_local_mode_project(monkeypatch, tmp_path):
|
||||
"""A project with an empty path is skipped even when mode is LOCAL (issue #949).
|
||||
|
||||
ProjectEntry.mode defaults to LOCAL, so a hand-edited config entry of
|
||||
``{"path": ""}`` is not recognized as cloud. The watch cycle must still skip
|
||||
it: Path("") resolves to the process cwd, and watching that would mutate
|
||||
whatever directory the server was launched from.
|
||||
"""
|
||||
config = BasicMemoryConfig(
|
||||
watch_project_reload_interval=1,
|
||||
projects={
|
||||
"local-project": {"path": str(tmp_path / "local"), "mode": "local"},
|
||||
# No explicit mode -> defaults to LOCAL, with an empty (cwd-relative) path.
|
||||
"empty-path": {"path": ""},
|
||||
},
|
||||
)
|
||||
repo = _Repo(
|
||||
projects_return=[
|
||||
Project(id=1, name="local-project", path=str(tmp_path / "local"), permalink="local"),
|
||||
Project(id=2, name="empty-path", path="", permalink="empty-path"),
|
||||
]
|
||||
)
|
||||
watch_service = _watch_service(config, repo)
|
||||
|
||||
seen_project_names: list[list[str]] = []
|
||||
|
||||
async def watch_cycle_stub(projects, stop_event):
|
||||
seen_project_names.append([p.name for p in projects])
|
||||
watch_service.state.running = False
|
||||
stop_event.set()
|
||||
|
||||
async def fake_write_status():
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(watch_service, "_watch_projects_cycle", watch_cycle_stub)
|
||||
monkeypatch.setattr(watch_service, "write_status", fake_write_status)
|
||||
|
||||
await watch_service.run()
|
||||
|
||||
assert seen_project_names == [["local-project"]]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_filters_orphan_db_project_absent_from_config(monkeypatch, tmp_path):
|
||||
"""A DB row not present in config is skipped even with an absolute path.
|
||||
|
||||
Config is the source of truth. Reconciliation normally deletes orphan rows,
|
||||
but if it is skipped or fails a stale row could remain; watching it would
|
||||
mutate a directory the user already removed from config.
|
||||
"""
|
||||
config = BasicMemoryConfig(
|
||||
watch_project_reload_interval=1,
|
||||
projects={
|
||||
"local-project": {"path": str(tmp_path / "local"), "mode": "local"},
|
||||
},
|
||||
)
|
||||
repo = _Repo(
|
||||
projects_return=[
|
||||
Project(id=1, name="local-project", path=str(tmp_path / "local"), permalink="local"),
|
||||
# Absolute path, but no matching entry in config -> stale/orphan row.
|
||||
Project(id=2, name="orphan", path=str(tmp_path / "orphan"), permalink="orphan"),
|
||||
]
|
||||
)
|
||||
watch_service = _watch_service(config, repo)
|
||||
|
||||
seen_project_names: list[list[str]] = []
|
||||
|
||||
async def watch_cycle_stub(projects, stop_event):
|
||||
seen_project_names.append([p.name for p in projects])
|
||||
watch_service.state.running = False
|
||||
stop_event.set()
|
||||
|
||||
async def fake_write_status():
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(watch_service, "_watch_projects_cycle", watch_cycle_stub)
|
||||
monkeypatch.setattr(watch_service, "write_status", fake_write_status)
|
||||
|
||||
await watch_service.run()
|
||||
|
||||
assert seen_project_names == [["local-project"]]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_run_continues_after_cycle_error(monkeypatch, tmp_path):
|
||||
config = BasicMemoryConfig(
|
||||
|
||||
@@ -1245,30 +1245,6 @@ class TestProjectMode:
|
||||
assert config.projects["research"].mode == ProjectMode.LOCAL
|
||||
assert config.get_project_mode("research") == ProjectMode.LOCAL
|
||||
|
||||
def test_is_locally_syncable_true_for_config_project_with_absolute_path(self, tmp_path):
|
||||
"""A project in config with an absolute path is locally syncable."""
|
||||
abs_path = str(tmp_path / "research")
|
||||
config = BasicMemoryConfig(projects={"research": ProjectEntry(path=abs_path)})
|
||||
assert config.is_locally_syncable("research", abs_path) is True
|
||||
|
||||
def test_is_locally_syncable_false_for_empty_path(self):
|
||||
"""An empty path resolves to cwd, so it is never locally syncable (#949)."""
|
||||
config = BasicMemoryConfig(projects={"empty": ProjectEntry(path="")})
|
||||
assert config.is_locally_syncable("empty", "") is False
|
||||
|
||||
def test_is_locally_syncable_false_for_relative_path(self):
|
||||
"""A relative (slug) path, as used by cloud-only projects, is not syncable."""
|
||||
config = BasicMemoryConfig(projects={"cloud": ProjectEntry(path="cloud-slug")})
|
||||
assert config.is_locally_syncable("cloud", "cloud-slug") is False
|
||||
|
||||
def test_is_locally_syncable_false_for_orphan_not_in_config(self, tmp_path):
|
||||
"""A DB row absent from config is not syncable even with an absolute path.
|
||||
|
||||
Config is the source of truth; stale rows must not be synced (#949).
|
||||
"""
|
||||
config = BasicMemoryConfig(projects={})
|
||||
assert config.is_locally_syncable("orphan", str(tmp_path / "orphan")) is False
|
||||
|
||||
def test_cloud_api_key_defaults_to_none(self):
|
||||
"""Test that cloud_api_key defaults to None."""
|
||||
config = BasicMemoryConfig()
|
||||
|
||||
Reference in New Issue
Block a user