Commit Graph

843 Commits

Author SHA1 Message Date
jope-bm 4ea5396ddd fix: skip workspace resolution when client factory is active (#630)
Signed-off-by: Joe P <joe@basicmemory.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 14:40:04 -06:00
Paul Hernandez e97eafa55a fix: build_context related_results schema validation failure (#631)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 12:30:44 -06:00
phernandez 496af07ced fix: resolve pyright possibly-unbound errors in edit_note
Initialize entity_id and result before the try/except block and
add assertion before the formatting section to help pyright prove
result is always bound on all code paths.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-26 20:10:35 -06:00
phernandez 54b968b93c fix: reduce excessive log volume by demoting per-request noise to DEBUG (#613)
Demote high-frequency per-request/per-item logs from INFO to DEBUG:
- 🔇 Client routing decisions (async_client.py) — logged every MCP tool call
- 🔇 DB migration checks (db.py) — logged every ASGI client creation
- 🔇 Vector table ensure/ready (sqlite + postgres search repos) — logged every search
- 🔇 Per-entity search index start/complete (search_service.py) — logged every file sync
- 🔇 Incremental scan details + per-file permalink updates (sync_service.py)
- 🔇 MCP search tool params and no-results (search.py)
- 📉 Log retention: "10 days" → 5 files (~50MB cap)

API v2 request/response logs remain at INFO for observability.

Closes #613

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-26 19:48:24 -06:00
phernandez e59b5cb6d9 feat: edit_note append/prepend auto-create note if not found (#614)
When append or prepend targets a non-existent note, the tool now
creates the file automatically instead of returning an error. This
eliminates silent failures for plugins (like openclaw) that use
edit_note(append) to build daily conversation notes — on the first
message of each day, the note didn't exist yet.

- find_replace and replace_section still require an existing note
- JSON output now includes `fileCreated: bool` in all responses
- Path traversal security check applied to auto-created directories
- Updated error messages to suggest append/prepend for missing notes

🧪 25 unit tests, 14 MCP integration tests, 10 CLI integration tests — all passing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-26 19:00:34 -06:00
phernandez f0335b998e fix: handle quoted picoschema enum strings in YAML frontmatter (#612)
The picoschema enum-with-description syntax `[val1, val2], description`
is invalid YAML. Users must quote it so YAML parses it as a string.
This adds `_parse_enum_string()` to extract enum values and description
from the resulting string value (e.g., "[active, blocked], current state").

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-26 18:10:48 -06:00
phernandez 9331126ba1 feat: improve content hit rate in search results (#609)
Three changes to surface more answer text in search results:

- Populate matched_chunk_text for FTS-only hybrid results from content_snippet,
  preventing fallback to truncated content when vector search has no match
- Increase TOP_CHUNKS_PER_RESULT from 3 to 5, catching answers in deeper chunks
  for large notes (~2700 → ~4500 chars of matched context)
- Increase CONTENT_DISPLAY_LIMIT from 2000 to 4000, doubling the safety-net
  content truncation for results without matched_chunk

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-26 11:55:38 -06:00
phernandez 3bbb44af0b feat: add --json output to CLI commands for scripting and CI
Add machine-readable JSON output to five CLI commands:
- `bm status --json` — sync report
- `bm project list --json` — structured project list
- `bm schema validate --json` — validation report
- `bm schema infer --json` — inference report
- `bm schema diff --json` — drift report

Refactored `run_status()` to return data instead of printing directly,
improving testability. Follows the established `bm project info --json`
pattern using `print()` for clean JSON (no Rich markup).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-26 10:33:00 -06:00
phernandez f9b2a075a9 feat: return richer content context in search results (#609)
Two strategies to improve content hit rate when the right document is found:

1. Small notes (<=2000 chars): return full content_snippet as matched_chunk
   so the answer is always present for correctly-retrieved small notes
2. Large notes: return top-3 chunks by similarity joined with \n---\n
   instead of just the single best chunk (~2700 chars vs ~900 chars)

Also raises CONTENT_DISPLAY_LIMIT from 250 to 2000 for richer FTS results.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-26 08:36:54 -06:00
phernandez f5a0e942b0 fix: replace RRF with score-based fusion in hybrid search (#577)
RRF compressed all fused scores to ~0.016, destroying ranking differentiation.
The new formula `max(vec, fts) + FUSION_BONUS * min(vec, fts)` preserves
dominant signals and rewards dual-source agreement.

Changes:
- Remove RRF_K constant; add FUSION_BONUS (0.3) and FTS_GATE_THRESHOLD (0.0)
- Use raw vector similarity scores instead of re-normalizing by vec_max
- Zero-score results now produce zero fused score (no 0.1 weight floor)
- Rename test_hybrid_rrf.py → test_hybrid_fusion.py with updated assertions
- Update docs and docstrings to reflect score-based fusion

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-25 22:53:00 -06:00
phernandez e46555bf2c fix: guard against closed streams in promo and missing vector tables (#579, #607)
- Wrap isatty() in _is_interactive_session() with try/except ValueError
  so MCP stdio transport shutdown no longer produces noisy tracebacks
- Check both search_vector_chunks AND search_vector_embeddings exist
  before running JOIN queries in get_embedding_status(), fixing
  OperationalError when only the chunks table is present
- Add test for closed-stream scenario

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-25 18:01:59 -06:00
phernandez 74e6afdc0f fix: create search_vector_chunks in test fixtures for Postgres compatibility
Embedding status tests were creating search_vector_chunks inline using
SQLite-only DDL (AUTOINCREMENT). Added Postgres DDL constants to
models/search.py and wired them into the test fixture so both backends
create the table at setup time — matching what the Alembic migration
does in production.

Also fixed stub search_vector_embeddings to use chunk_id (Postgres
column name) instead of rowid, and added inter-test cleanup to prevent
ordering-dependent failures.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-25 17:30:35 -06:00
phernandez aa635b8a8b fix: accept null for expected_replacements in edit_note (#606)
MCP clients may send explicit `null` for unused optional fields.
`expected_replacements: int = 1` caused FastMCP's JSON Schema validation
to reject null before the function body ran. Changed to `Optional[int] = None`
with an effective default resolved inside the function body.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-25 14:18:53 -06:00
phernandez 73413486bc feat: merge search_by_metadata into search_notes with optional query
Make `query` optional in `search_notes` so it becomes the single search tool.
Remove `search_by_metadata` entirely — it was unreleased and redundant since
`search_notes` already supports `metadata_filters`, `tags`, and `status`.

- 🔧 `query` param is now `Optional[str] = None`
- 🛡️ Added None guards for project detection and URL resolution
-  Added `no_criteria()` validation with helpful error message
- 🗑️ Deleted `search_by_metadata` tool, imports, tests, and contract entry
- 📝 Updated docs, README, and v0.19.0 release notes

Closes #605

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-25 13:30:42 -06:00
phernandez b09eca1698 feat: add EmbeddingStatus schema and get_embedding_status() service method
Add EmbeddingStatus model to project_info schemas and wire it into
ProjectInfoResponse. ProjectService.get_embedding_status() queries
vector tables for chunk/embedding counts, detects orphaned chunks
and missing embeddings, and recommends reindex when appropriate.
Handles both SQLite and Postgres backends. 🔍

Includes 6 unit tests covering: disabled search, missing vector tables,
entities without chunks, orphaned chunks, healthy state, and integration
with get_project_info().

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-25 11:44:50 -06:00
phernandez 3004d0d1fe feat: replace project info with htop-inspired dashboard
Replace Layout-based display (expanded to terminal width) with a compact
Panel using Table.grid(expand=False). Add horizontal bar charts for note
types (top 5), embedding coverage bar with Unicode blocks, and colored
status dots. Removes verbose sections (most connected, recent activity,
available projects) in favor of a dense, visually engaging dashboard. 📊

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-25 11:11:58 -06:00
phernandez b6369d3d14 fix: cap sqlite-vec knn k parameter at 4096 limit
sqlite-vec enforces k <= 4096 for nearest-neighbor queries. Projects with
>4096 vector chunks would crash all vector/hybrid search because
candidate_limit = max(100, (limit + offset) * 10) exceeded this hard limit.

Clamp the knn k in _run_vector_query while keeping the outer SQL LIMIT
unclamped. Only affects SQLite — pgvector has no such constraint.

Fixes #604

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-24 23:23:15 -06:00
phernandez db6d0dcd9e docs: add tag: shorthand limitation note to search_notes docstring
The tag: query shorthand doesn't work with hybrid search (default when
semantic search is enabled) because it strips the text query. Document
the workaround: use search_type="text" or the tags parameter instead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-24 20:45:42 -06:00
phernandez 79db876bd6 feat: hardcode Umami analytics defaults for FOSS usage tracking
Bake in cloud.umami.is host and basic-memory-foss site ID so all
open-source installs send anonymous CLI events by default. Users
can still opt out with BASIC_MEMORY_NO_PROMOS=1 or override the
endpoint via env vars.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-24 19:50:28 -06:00
phernandez 0130573342 fix: prompts call MCP tools directly, sync handles semantic errors, status uses local routing
- Prompts (search, continue_conversation) now call MCP tools directly
  instead of going through API endpoints, matching the recent_activity
  pattern and fixing #526 where prompts returned empty results
- sync_file catches SemanticDependenciesMissingError separately so
  entities are still returned successfully when vector embedding fails,
  with a clear warning instead of silent failure (#578)
- `bm status` and `bm doctor` default to local routing since they scan
  the local filesystem — cloud routing returned Docker-internal paths

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-24 19:36:11 -06:00
phernandez c372dfb09f fix: use LinkResolver fallback in build_context for flexible identifier matching (#582)
build_context now falls back to LinkResolver when an exact permalink lookup
returns empty results. This reuses the same resolution pipeline as read_note
(permalink candidates, title match, file path, FTS) so callers no longer
get empty results for valid note identifiers.

Also changes ensure_frontmatter_on_sync default to True — frontmatter is
now added during sync by default. Tests updated accordingly.

🔧 ContextService accepts optional LinkResolver, wired via DI in all 3 factory variants
 2027 unit + 278 integration tests passing

Closes #582

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-24 18:58:19 -06:00
phernandez d763d86798 fix: unify project path so path is always the local filesystem path
Cloud projects with bisync had a split-brain problem: `path` held a cloud
slug while the actual local directory lived in `local_sync_path`. This caused
`bm status` and file sync to fail for bisync'd cloud projects.

Changes:
- Config migration promotes `local_sync_path` → `path` for entries where
  `path` is a non-absolute cloud slug
- `ensure_project_paths_exists` skips cloud-only projects with slug paths
- `initialize_file_sync` and watch service now keep cloud projects that have
  an absolute local path (bisync copy) instead of skipping all cloud projects
- `sync-setup` and `project add --cloud --local-path` set both `path` and
  `local_sync_path` to the local directory
- `sync-setup` creates the project in the local DB for immediate MCP use
- `_get_sync_project` falls back from `local_sync_path` to `path`
- Config load errors now show user-friendly messages instead of stack traces

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-24 15:08:09 -06:00
phernandez 538af97cba feat: show spinner while fetching cloud projects in bm project list
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-24 12:05:33 -06:00
phernandez 054178d155 fix: parameterize SQL queries in search repository type filters
Replace f-string interpolation with parameterized queries for note_types,
search_item_types, and metadata filter paths to prevent SQL injection.

Fixes #591

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-23 22:16:46 -06:00
phernandez 18f00f861d fix: remove hardcoded "main" default from default_project (#575)
When a user's config.json had projects with names other than "main" and
no explicit default_project key, the hardcoded field default of "main"
would not match any project. The model_post_init fixup logic existed but
was untested and only handled the stale-name case, not the None case.

Changes:
- Change default_project field default from "main" to None
- Use model_fields_set to distinguish "config omitted the key" (auto-resolve
  to first project) from "user explicitly set None" (preserve for discovery mode)
- Split model_post_init into two branches: auto-resolve when not explicitly
  provided, correct stale names when explicitly set but invalid
- Remove # pragma: no cover from now-tested branches
- Add 10 new tests covering valid defaults, stale defaults, empty string,
  single project, config file round-trips, and discovery mode preservation

Fixes #575

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-23 21:59:15 -06:00
jope-bm da4d369c32 feat: add created_by and last_updated_by user tracking to Entity (#602)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 11:20:59 -07:00
Paul Hernandez 0f3889fdd0 fix: Return matched chunk text in search results (#601)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 21:01:20 -06:00
Paul Hernandez c44291830c chore: rename entity_type to note_type (#600)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 20:28:57 -06:00
phernandez 6ff39076a0 fix: double-default display in project list + stale test updates
Use config.default_project as single source of truth for the Default
column in `bm project list`, removing checks against local DB and cloud
API is_default fields that could independently mark multiple projects.

Also updates tests that were out of date after get_project_mode changed
to default unknown projects to CLOUD:
- test_get_client_local_project_uses_asgi_transport: register "main" as LOCAL
- test_run_filters_cloud_projects_each_cycle: register local project in config
- test_new_project_addition_scenario: register projects as LOCAL in config
- test_get_project_mode_defaults_to_cloud: assert new CLOUD default

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-22 00:33:25 -06:00
phernandez 2e5813d31e feat: CLI refactoring + workspace-aware cloud project listing
Refactor CLI commands to use typed ProjectClient instead of raw HTTP calls,
and add workspace metadata to cloud project listings so users can distinguish
personal vs organization projects.

Key changes:
- 🔧 CLI commands now use ProjectClient typed API clients instead of
  call_get/call_post with manual URL construction
- 🏢 Cloud project listings include workspace_name, workspace_type, and
  workspace_tenant_id for each cloud-sourced project
- Pass config.default_workspace when fetching cloud projects via
  _fetch_cloud_projects() and CLI list_projects
- Add --workspace flag to `bm project list` for explicit workspace override
- Add "Workspace" column to CLI project list table
- Add `bm tool list-projects` and `bm tool list-workspaces` JSON commands
- Comprehensive tests for workspace passthrough, merge behavior, and CLI routing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-21 20:04:03 -06:00
phernandez 2cde8d2659 clean up cli commands
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-21 16:06:43 -06:00
Paul Hernandez 9515130b2a feat: upgrade fastmcp 2.12.3 to 3.0.1 with tool annotations (#598)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-21 12:28:30 -06:00
Paul Hernandez b86dd6fb53 feat: Fix bm CLI runtime defects and audit regressions (#596)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-20 22:59:29 -06:00
Paul Hernandez 8451f2b1d7 feat: add frontmatter validation to schema system (#597)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-20 22:57:19 -06:00
Paul Hernandez ee0397513d fix: recent_activity dedup + pagination across MCP tools (#595)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-20 21:48:58 -06:00
Drew Cain 9c9ff2931d fix: backend-specific distance-to-similarity conversion (#593)
Signed-off-by: Drew Cain <groksrc@gmail.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Co-authored-by: phernandez <paul@basicmachines.co>
2026-02-20 20:15:46 -06:00
Paul Hernandez edb7991ccf fix: strip NUL bytes from content before PostgreSQL search indexing (#592)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-20 18:11:04 -06:00
phernandez d7faeb754e feat: Add destination_folder parameter to move_note tool
Allows callers to move a note into a folder while preserving its original
filename — no need for a separate read_note round-trip to extract the basename.

- destination_folder is mutually exclusive with destination_path
- Rejected for directory moves (is_directory=True)
- Uses Path().name and PureWindowsPath().as_posix() for cross-platform compat
- Validates resolved path against path traversal attacks
- Includes formatting fixes in promo.py and test_analytics.py

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-20 02:06:04 -06:00
jope-bm deef89a724 feat: Add display_name and is_private to ProjectItem (#574)
Signed-off-by: Joe P <joe@basicmemory.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
Co-authored-by: phernandez <paul@basicmachines.co>
Co-authored-by: jope-bm <jope-bm@users.noreply.github.com>
2026-02-20 01:44:07 -06:00
Paul Hernandez 30499a9f61 feat: Let stdio MCP honor per-project cloud routing (#590)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-20 01:43:45 -06:00
bm-clawd 1ac65b944c feat: CLI analytics via Umami event collector (#572)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-20 00:09:42 -06:00
Paul Hernandez 306e562281 chore: Make semantic deps default, auto-backfill embeddings, and default search to semantic (#586)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-19 18:31:15 -06:00
Paul Hernandez 0cb3f95d67 feat: Add JSON output mode for BM MCP tools (default text) (#585)
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
2026-02-18 22:18:29 -06:00
Paul Hernandez f2683291e4 feat: add workspace selection flow for MCP and CLI (#576)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-17 22:48:06 -06:00
phernandez 8c05a9ec80 fix: stabilize semantic search defaults, FTS fallback, and postgres project sync
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-17 15:36:46 -06:00
phernandez a6d8d4c0f6 Add ensure_frontmatter_on_sync with precedence warning
Add a new config option to enforce frontmatter on markdown sync when missing, writing derived title/type/permalink and updating in-memory metadata before upsert. Add startup warning when this option is combined with disable_permalinks to make precedence explicit. Add config/sync/initialization tests for the new behavior, and stabilize project list CLI integration assertions by forcing a wide terminal in tests to avoid Rich truncation.

Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-16 22:08:09 -06:00
phernandez 0239f4abb4 Simplify local/cloud routing and clarify project targeting
Signed-off-by: phernandez <paul@basicmachines.co>
2026-02-16 21:42:25 -06:00
Paul Hernandez 9259a7eb59 feat: min-similarity override, edit-note CLI, and strip-frontmatter (#571)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-16 17:56:00 -06:00
Paul Hernandez 55d675e278 feat: min_similarity override, cloud promo improvements (#570)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-16 16:44:02 -06:00
Drew Cain 6afe4fd0cc feat: expose external_id in EntityResponse and link resolver (#569)
Signed-off-by: Drew Cain <groksrc@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 20:33:37 -06:00