Compare commits

..

2 Commits

Author SHA1 Message Date
phernandez 3480e917e2 chore(sync): remove duplicate resolver comment
Signed-off-by: phernandez <paul@basicmachines.co>
2026-06-06 18:33:34 -05:00
Dennis Hempel 6ac28aeee9 fix(sync): use strict link resolution for deferred forward references
Deferred relation resolution in `sync_service.resolve_forward_references`
and `BatchIndexer._resolve_batch_relations` called
`LinkResolver.resolve_link(to_name)` with the default `strict=False`.
When the link text didn't match any entity's permalink, title, or
file_path, the resolver fell through to BM25/ts_rank fuzzy search and
silently picked `results[0]` — even for ambiguous links that share only
incidental tokens with the wrong entity.

Producers commonly emit disambiguator-style wikilinks like
`[[overview (state-management/session-execution)]]`. basic-memory does
not parse `(qualifier)` as a disambiguator, so no entity matches
exactly. The previous behavior resolved such links to whichever
unrelated note shared the most tokens, polluting the graph with
confidently-wrong edges that `basic-memory status` / `doctor` do not
catch.

Entity-creation already passes `strict=True` (see
`entity_service.update_entity_relations` at line 1030). The deferred
sync and batch-indexer paths must use the same contract so unresolved
relations stay unresolved (`to_id=NULL`) and surface to the producer
instead of getting silently filled with a wrong target.

Adds two regression tests that spy on `resolve_link` and assert
`strict=True` is passed by both deferred paths. Verified failing on
the previous code and passing after.

Reported in basic-memory-bug-report Issue 2.

Signed-off-by: Dennis Hempel <uhg.dennnis@gmail.com>
2026-05-28 09:50:30 +02:00
7 changed files with 219 additions and 97 deletions
+52 -60
View File
@@ -1,6 +1,6 @@
# Basic Memory Cloud CLI Guide
The Basic Memory Cloud CLI provides seamless integration between local and cloud knowledge bases using **project-scoped synchronization**. Personal workspaces can optionally use local rclone mirrors, giving you fine-grained control over what syncs and where.
The Basic Memory Cloud CLI provides seamless integration between local and cloud knowledge bases using **project-scoped synchronization**. Each project can optionally sync with the cloud, giving you fine-grained control over what syncs and where.
## Overview
@@ -8,13 +8,9 @@ The cloud CLI enables you to:
- **Authenticate cloud access** - OAuth/API key credentials are stored locally for cloud operations
- **Project-scoped sync** - Each project independently manages its sync configuration
- **Explicit operations** - Sync only what you want, when you want
- **Bidirectional sync** - Keep Personal workspace local mirrors and cloud in sync with rclone bisync
- **Bidirectional sync** - Keep local and cloud in sync with rclone bisync
- **Offline access** - Work locally, sync when ready
Team workspaces are accessed through the cloud API/MCP and do not support local
multi-user rclone sync/bisync. Use `bm project list --workspace <workspace>` to
inspect Team projects.
## Prerequisites
Before using Basic Memory Cloud, you need:
@@ -43,7 +39,7 @@ If you attempt to log in without an active subscription, you'll receive a "Subsc
**Projects can exist in three states:**
1. **Cloud-only** - Project exists on cloud, no local copy
2. **Cloud + Local (synced)** - Personal workspace project has a local working directory that syncs
2. **Cloud + Local (synced)** - Project has a local working directory that syncs
3. **Local-only** - Project exists locally and is not routed to cloud
**Example:**
@@ -59,8 +55,8 @@ bm project add work --cloud --local-path ~/work-notes
bm project add temp --cloud # No local sync
# Now you can sync individually (after initial --resync):
bm cloud bisync --name research
bm cloud bisync --name work
bm project bisync --name research
bm project bisync --name work
# temp stays cloud-only
```
@@ -91,7 +87,7 @@ Apply OSS discount code `{{OSS_DISCOUNT_CODE}}` during checkout to receive 20% o
### 2. Set Up Sync
Install rclone and configure Personal workspace sync credentials:
Install rclone and configure credentials:
```bash
bm cloud setup
@@ -103,7 +99,7 @@ bm cloud setup
3. Generates scoped S3 credentials for sync
4. Configures single rclone remote: `basic-memory-cloud`
**Result:** You're ready to sync Personal workspace projects. No sync directories created yet - those come with project setup.
**Result:** You're ready to sync projects. No sync directories created yet - those come with project setup.
Rclone setup uses package managers such as Homebrew, MacPorts, apt, dnf, yum, pacman,
zypper, snap, winget, Chocolatey, or Scoop when available. It does not run remote
@@ -112,7 +108,7 @@ manual install instructions.
### 3. Add Projects with Sync
Create Personal workspace projects with optional local sync paths:
Create projects with optional local sync paths:
```bash
# Create cloud project without local sync
@@ -141,10 +137,10 @@ Establish the initial sync baseline. **Best practice:** Always preview with `--d
```bash
# Step 1: Preview the initial sync (recommended)
bm cloud bisync --name research --resync --dry-run
bm project bisync --name research --resync --dry-run
# Step 2: If all looks good, run the actual sync
bm cloud bisync --name research --resync
bm project bisync --name research --resync
```
**What happens under the covers:**
@@ -171,7 +167,7 @@ This will effectively make both Path1 and Path2 filesystems contain a matching s
After the first sync, just run bisync without `--resync`:
```bash
bm cloud bisync --name research
bm project bisync --name research
```
**What happens:**
@@ -239,7 +235,7 @@ bm project add research --cloud --local-path ~/Documents/research
- Stores sync config in `~/.basic-memory/config.json`
- Prepares for bisync (but doesn't sync yet)
**Result:** Project ready to sync. Run `bm cloud bisync --name research --resync` to establish baseline.
**Result:** Project ready to sync. Run `bm project bisync --name research --resync` to establish baseline.
**Use case 3: Add sync to existing cloud project**
@@ -296,24 +292,20 @@ For MCP stdio, routing is always local.
## File Synchronization
Local rclone sync/bisync is supported only for Personal workspaces. Team
workspaces are cloud-only for local CLI usage; access them through cloud API/MCP
routing instead of a local multi-user rclone mirror.
### Understanding the Sync Commands
**There are three sync-related commands:**
1. `bm cloud sync` - One-way: local → cloud (make cloud match local)
2. `bm cloud bisync` - Two-way: local ↔ cloud (recommended)
3. `bm cloud check` - Verify files match (no changes)
1. `bm project sync` - One-way: local → cloud (make cloud match local)
2. `bm project bisync` - Two-way: local ↔ cloud (recommended)
3. `bm project check` - Verify files match (no changes)
### One-Way Sync: Local → Cloud
**Use case:** You made changes locally and want to push to cloud (overwrite cloud).
```bash
bm cloud sync --name research
bm project sync --name research
```
**What happens:**
@@ -335,10 +327,10 @@ bm cloud sync --name research
```bash
# First time - establish baseline
bm cloud bisync --name research --resync
bm project bisync --name research --resync
# Subsequent syncs
bm cloud bisync --name research
bm project bisync --name research
```
**What happens:**
@@ -357,7 +349,7 @@ echo "Local change" > ~/Documents/research/notes.md
# Cloud now has: "Cloud change"
# Run bisync
bm cloud bisync --name research
bm project bisync --name research
# Result: Newer file wins (based on modification time)
# If cloud was more recent, cloud version kept
@@ -374,7 +366,7 @@ bm cloud bisync --name research
**Use case:** Check if local and cloud match without making changes.
```bash
bm cloud check --name research
bm project check --name research
```
**What happens:**
@@ -386,7 +378,7 @@ bm cloud check --name research
```bash
# One-way check (faster)
bm cloud check --name research --one-way
bm project check --name research --one-way
```
### Preview Changes (Dry Run)
@@ -394,7 +386,7 @@ bm cloud check --name research --one-way
**Use case:** See what would change without actually syncing.
```bash
bm cloud bisync --name research --dry-run
bm project bisync --name research --dry-run
```
**What happens:**
@@ -440,20 +432,20 @@ bm project add work --cloud --local-path ~/work-notes
bm project add personal --cloud --local-path ~/personal
# Establish baselines
bm cloud bisync --name research --resync
bm cloud bisync --name work --resync
bm cloud bisync --name personal --resync
bm project bisync --name research --resync
bm project bisync --name work --resync
bm project bisync --name personal --resync
# Daily workflow: sync everything
bm cloud bisync --name research
bm cloud bisync --name work
bm cloud bisync --name personal
bm project bisync --name research
bm project bisync --name work
bm project bisync --name personal
```
**Future:** `--all` flag will sync all configured projects:
```bash
bm cloud bisync --all # Coming soon
bm project bisync --all # Coming soon
```
### Mixed Usage
@@ -470,8 +462,8 @@ bm project add archive --cloud
bm project add temp-notes --cloud
# Sync only the configured ones
bm cloud bisync --name research
bm cloud bisync --name work
bm project bisync --name research
bm project bisync --name work
# Archive and temp-notes stay cloud-only
```
@@ -669,7 +661,7 @@ code ~/.basic-memory/.bmignore
echo "*.tmp" >> ~/.basic-memory/.bmignore
# Next sync uses updated patterns
bm cloud bisync --name research
bm project bisync --name research
```
## Troubleshooting
@@ -732,7 +724,7 @@ bm cloud login
**Solution:**
```bash
bm cloud bisync --name research --resync
bm project bisync --name research --resync
```
**What this does:**
@@ -755,7 +747,7 @@ bm cloud bisync --name research --resync
echo "# Research Notes" > ~/Documents/research/README.md
# Now run bisync
bm cloud bisync --name research --resync
bm project bisync --name research --resync
```
**Why this happens:** Bisync creates listing files that track the state of each side. When both directories are completely empty, these listing files are considered invalid by rclone.
@@ -772,10 +764,10 @@ bm cloud bisync --name research --resync
```bash
# Clear bisync state
bm cloud bisync-reset research
bm project bisync-reset research
# Re-establish baseline
bm cloud bisync --name research --resync
bm project bisync --name research --resync
```
**What this does:**
@@ -795,16 +787,16 @@ bm cloud bisync --name research --resync
```bash
# Check what would be deleted
bm cloud bisync --name research --dry-run
bm project bisync --name research --dry-run
# If correct, establish new baseline
bm cloud bisync --name research --resync
bm project bisync --name research --resync
```
**Solution 2:** Use one-way sync if you know local is correct:
```bash
bm cloud sync --name research
bm project sync --name research
```
### Project Not Configured for Sync
@@ -817,7 +809,7 @@ bm cloud sync --name research
```bash
bm cloud sync-setup research ~/Documents/research
bm cloud bisync --name research --resync
bm project bisync --name research --resync
```
### Connection Issues
@@ -889,19 +881,19 @@ bm project set-local <name> # Revert project to local mode
```bash
# One-way sync (local → cloud)
bm cloud sync --name <project>
bm cloud sync --name <project> --dry-run
bm cloud sync --name <project> --verbose
bm project sync --name <project>
bm project sync --name <project> --dry-run
bm project sync --name <project> --verbose
# Two-way sync (local ↔ cloud) - Recommended
bm cloud bisync --name <project> # After first --resync
bm cloud bisync --name <project> --resync # First time / force baseline
bm cloud bisync --name <project> --dry-run
bm cloud bisync --name <project> --verbose
bm project bisync --name <project> # After first --resync
bm project bisync --name <project> --resync # First time / force baseline
bm project bisync --name <project> --dry-run
bm project bisync --name <project> --verbose
# Integrity check
bm cloud check --name <project>
bm cloud check --name <project> --one-way
bm project check --name <project>
bm project check --name <project> --one-way
# List project files by route
bm project ls --name <project> # Default target: local
@@ -917,9 +909,9 @@ bm project ls --name <project> --cloud --path <subpath>
1. **Authenticate cloud access** - `bm cloud login`
2. **Install rclone** - `bm cloud setup`
3. **Add projects with sync** - `bm project add research --cloud --local-path ~/Documents/research`
4. **Preview first sync** - `bm cloud bisync --name research --resync --dry-run`
5. **Establish baseline** - `bm cloud bisync --name research --resync`
6. **Daily workflow** - `bm cloud bisync --name research`
4. **Preview first sync** - `bm project bisync --name research --resync --dry-run`
5. **Establish baseline** - `bm project bisync --name research --resync`
6. **Daily workflow** - `bm project bisync --name research`
**Key benefits:**
- ✅ Each project independently syncs (or doesn't)
@@ -95,10 +95,7 @@ def sync_project_command(
dry_run: bool = typer.Option(False, "--dry-run", help="Preview changes without syncing"),
verbose: bool = typer.Option(False, "--verbose", "-v", help="Show detailed output"),
) -> None:
"""Personal workspace local mirror only.
One-way sync: local -> cloud (make cloud identical to local).
Not supported for Team workspaces - use cloud API/MCP routing instead.
"""One-way sync: local -> cloud (make cloud identical to local).
Example:
bm cloud sync --name research
@@ -146,10 +143,7 @@ def bisync_project_command(
resync: bool = typer.Option(False, "--resync", help="Force new baseline"),
verbose: bool = typer.Option(False, "--verbose", "-v", help="Show detailed output"),
) -> None:
"""Personal workspace local mirror only.
Two-way sync: local <-> cloud (bidirectional sync).
Not supported for Team workspaces - use cloud API/MCP routing instead.
"""Two-way sync: local <-> cloud (bidirectional sync).
Examples:
bm cloud bisync --name research --resync # First time
@@ -209,10 +203,7 @@ def check_project_command(
name: str = typer.Option(..., "--name", help="Project name to check"),
one_way: bool = typer.Option(False, "--one-way", help="Check one direction only (faster)"),
) -> None:
"""Personal workspace local mirror only.
Verify file integrity between local and cloud.
Not supported for Team workspaces - use cloud API/MCP routing instead.
"""Verify file integrity between local and cloud.
Example:
bm cloud check --name research
@@ -255,10 +246,7 @@ def check_project_command(
def bisync_reset(
name: str = typer.Argument(..., help="Project name to reset bisync state for"),
) -> None:
"""Personal workspace local mirror only.
Clear bisync state for a project.
Not supported for Team workspaces - use cloud API/MCP routing instead.
"""Clear bisync state for a project.
This removes the bisync metadata files, forcing a fresh --resync on next bisync.
Useful when bisync gets into an inconsistent state or when remote path changes.
@@ -289,10 +277,7 @@ def setup_project_sync(
name: str = typer.Argument(..., help="Project name"),
local_path: str = typer.Argument(..., help="Local sync directory"),
) -> None:
"""Personal workspace local mirror only.
Configure local sync for an existing cloud project.
Not supported for Team workspaces - use cloud API/MCP routing instead.
"""Configure local sync for an existing cloud project.
Example:
bm cloud sync-setup research ~/Documents/research
+7 -1
View File
@@ -493,8 +493,14 @@ class BatchIndexer:
async def resolve_relation(relation: Relation) -> int:
async with semaphore:
try:
# strict=True for deferred resolution: only fill in to_id on an
# exact permalink/title/file_path match. Fuzzy fallback would silently
# resolve ambiguous links to whichever entity shares tokens with the
# link text, mismatching this with the sync_service forward-reference
# path and producing confidently-wrong graph edges. See
# sync_service.resolve_forward_references for the same change.
resolved_entity = await self.entity_service.link_resolver.resolve_link(
relation.to_name
relation.to_name, strict=True
)
if resolved_entity is None or resolved_entity.id == relation.from_id:
return 0
+10 -1
View File
@@ -1447,7 +1447,16 @@ class SyncService:
f"to_name={relation.to_name}"
)
resolved_entity = await self.entity_service.link_resolver.resolve_link(relation.to_name)
# Use strict=True: deferred resolution should only fill in to_id when an
# exact permalink/title/file_path match exists. The fuzzy fallback (search-based
# token match) would silently resolve ambiguous links like
# `[[overview (state-management/session-execution)]]` to whichever entity shares
# the most tokens, polluting the graph with confidently-wrong edges that no
# audit catches. Leaving such relations unresolved keeps to_id=NULL so they
# surface as forward references and can be fixed by the producer.
resolved_entity = await self.entity_service.link_resolver.resolve_link(
relation.to_name, strict=True
)
# ignore reference to self
if resolved_entity and resolved_entity.id != relation.from_id:
@@ -12,21 +12,6 @@ from basic_memory.config import ProjectMode
runner = CliRunner()
@pytest.mark.parametrize(
"command",
["sync", "bisync", "check", "bisync-reset", "sync-setup"],
)
def test_cloud_sync_command_help_marks_personal_workspace_only(command):
"""Cloud sync help should explain that local mirrors are Personal-only."""
importlib.import_module("basic_memory.cli.commands.cloud.project_sync")
result = runner.invoke(app, ["cloud", command, "--help"])
assert result.exit_code == 0, result.output
assert "Personal workspace local mirror only" in result.output
assert "Not supported for Team workspaces" in result.output
@pytest.mark.parametrize(
"argv",
[
+72
View File
@@ -688,6 +688,78 @@ async def test_batch_indexer_index_markdown_file_can_defer_relation_resolution(
assert source.outgoing_relations[0].to_name == "Deferred Target"
@pytest.mark.asyncio
async def test_batch_indexer_uses_strict_link_resolution_for_deferred_relations(
app_config,
entity_service,
entity_repository,
relation_repository,
search_service,
file_service,
project_config,
monkeypatch,
):
"""Regression: batch indexer's deferred relation resolution must call
resolve_link with strict=True.
Mirror of sync_service.resolve_forward_references. Fuzzy fallback in the
deferred path silently fills in to_id from BM25/ts_rank results, polluting
the graph with confidently-wrong edges. Entity-creation already uses
strict=True; this is the other deferred path.
"""
path = "notes/source.md"
await _create_file(
project_config.home / path,
dedent(
"""
---
title: Source
type: note
---
# Source
- links_to [[never-resolves-target]]
"""
),
)
batch_indexer = _make_batch_indexer(
app_config,
entity_service,
entity_repository,
relation_repository,
search_service,
file_service,
)
original_resolve_link = entity_service.link_resolver.resolve_link
seen_strict: list[object] = []
async def spy_resolve_link(*args, **kwargs):
seen_strict.append(kwargs.get("strict", False))
return await original_resolve_link(*args, **kwargs)
monkeypatch.setattr(entity_service.link_resolver, "resolve_link", spy_resolve_link)
await batch_indexer.index_files(
{path: await _load_input(file_service, path)},
max_concurrent=1,
)
assert seen_strict, "batch indexer did not invoke link_resolver.resolve_link"
assert all(strict is True for strict in seen_strict), (
f"Deferred resolution must call resolve_link(strict=True). Observed: {seen_strict!r}"
)
# The unresolvable relation stayed unresolved.
source = await entity_repository.get_by_file_path(path)
assert source is not None
assert len(source.outgoing_relations) == 1
assert source.outgoing_relations[0].to_id is None
assert source.outgoing_relations[0].to_name == "never-resolves-target"
@pytest.mark.asyncio
async def test_batch_indexer_strips_frontmatter_from_search_content_when_body_is_empty(
app_config,
+73
View File
@@ -108,6 +108,79 @@ Target content
assert source.relations[0].to_name == target.title
@pytest.mark.asyncio
async def test_resolve_relations_uses_strict_link_resolution(
sync_service: SyncService,
project_config: ProjectConfig,
entity_service: EntityService,
monkeypatch,
):
"""Regression: deferred forward-reference resolution must call resolve_link
with strict=True.
Producers sometimes emit disambiguator-style links like
`[[overview (state-management/session-execution)]]` whose exact text does
not match any entity's permalink, title, or file_path. The previous
behavior fell through to BM25/ts_rank fuzzy search in
LinkResolver._resolve_in_project and silently picked whichever entity
shared the most tokens — polluting the graph with confidently-wrong edges
that no audit catches.
Entity-creation already resolves relations with strict=True (see
entity_service.update_entity_relations). The deferred sync path must use
the same contract; otherwise unresolved relations get silently filled
later by fuzzy search.
"""
# Create a source file with a forward reference. The target doesn't exist,
# so resolution will fail — which is exactly when fuzzy fallback would
# previously silently pick a wrong target.
source_content = dedent("""
---
type: knowledge
---
# Source
## Relations
- part_of [[never-resolves-target]]
""")
await create_test_file(project_config.home / "source.md", source_content)
await sync_service.sync(project_config.home)
project_prefix = generate_permalink(project_config.name)
source = await entity_service.get_by_permalink(f"{project_prefix}/source")
assert len(source.relations) == 1
assert source.relations[0].to_id is None # initial creation already strict
# Spy on resolve_link to capture the strict flag the deferred resolver uses.
original_resolve_link = sync_service.entity_service.link_resolver.resolve_link
seen_strict: list[Any] = []
async def spy_resolve_link(*args, **kwargs):
seen_strict.append(kwargs.get("strict", False))
return await original_resolve_link(*args, **kwargs)
monkeypatch.setattr(
sync_service.entity_service.link_resolver,
"resolve_link",
spy_resolve_link,
)
await sync_service.resolve_relations()
# Deferred resolution invoked resolve_link, and every call passed strict=True.
assert seen_strict, "resolve_relations did not invoke link_resolver.resolve_link"
assert all(strict is True for strict in seen_strict), (
f"Deferred resolution must call resolve_link(strict=True) to avoid silent "
f"fuzzy matching. Observed strict values: {seen_strict!r}"
)
# Sanity check: the unresolvable relation stayed unresolved — no silent
# fuzzy match polluted it.
source = await entity_service.get_by_permalink(f"{project_prefix}/source")
assert source.relations[0].to_id is None
assert source.relations[0].to_name == "never-resolves-target"
@pytest.mark.asyncio
async def test_resolve_relations_deletes_duplicate_unresolved_relation(
sync_service: SyncService,