mirror of
https://github.com/basicmachines-co/basic-memory
synced 2026-06-21 13:47:35 +00:00
Compare commits
56 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 440a2aa923 | |||
| a8e452d9b4 | |||
| 22901c366d | |||
| 205196b621 | |||
| 8e7825ba01 | |||
| dc29ba2a00 | |||
| fe9b2e9c95 | |||
| 650f88a2c5 | |||
| 2f7ef136de | |||
| 0a3a6bbd96 | |||
| 7bb7664fae | |||
| db578ccfdb | |||
| df485aa5a4 | |||
| 44ecec2917 | |||
| 49041a5168 | |||
| 8cbe1634b6 | |||
| c4b651f5b0 | |||
| ca9a4d9c12 | |||
| aa9594d82a | |||
| 93494b8c13 | |||
| 93ed34001b | |||
| ec94feb6a4 | |||
| 831b9141a5 | |||
| 2de19713f6 | |||
| 79fcfbce6a | |||
| 3d22ba3004 | |||
| 62229d9d0a | |||
| 03ba268cb1 | |||
| de53e0ecc5 | |||
| 4128cac9ab | |||
| 9b53d7863f | |||
| a8d034b940 | |||
| 7c0937f658 | |||
| 8570d96bad | |||
| 480a2d9468 | |||
| df7452e3ba | |||
| 85a8e59d0f | |||
| 07cb7a606b | |||
| 0ef03edde6 | |||
| efe43a10ea | |||
| 4fe6fe09c8 | |||
| 8acdb49a41 | |||
| f7304bf553 | |||
| 5b034f081d | |||
| 271c883ea8 | |||
| 816ee85fb9 | |||
| 3ba3a9504d | |||
| 1667cdc000 | |||
| f916662ff3 | |||
| 0c9800cd3b | |||
| 476239d878 | |||
| 20bb19f4cd | |||
| f6565b9d23 | |||
| b6e8c636ce | |||
| 442fd1523c | |||
| ce4b5d47a0 |
@@ -0,0 +1,68 @@
|
||||
---
|
||||
name: code-review
|
||||
description: Use when reviewing Basic Machines code for house style, architecture risk, pre-merge hardening, or whether a change fits basic-memory/basic-memory-cloud conventions.
|
||||
license: MIT
|
||||
---
|
||||
|
||||
# Basic Machines Review
|
||||
|
||||
Use this skill for repo-local review passes where ordinary code review needs Basic Machines
|
||||
house style and architecture judgment. Report findings only; do not edit code unless the user
|
||||
asks you to fix specific findings.
|
||||
|
||||
## Scope
|
||||
|
||||
Review the current diff or named files against:
|
||||
|
||||
- The repo's `AGENTS.md` / `CLAUDE.md`
|
||||
- `docs/ENGINEERING_STYLE.md`
|
||||
- The touched code paths and tests
|
||||
|
||||
Apply only the guidance for the active repo. In `basic-memory`, prioritize local-first
|
||||
file/database/MCP boundaries. In `basic-memory-cloud`, prioritize tenant/workspace isolation,
|
||||
cloud worker behavior, and web-v2 state/runtime boundaries.
|
||||
|
||||
## Review Rubric
|
||||
|
||||
Report only concrete, falsifiable risks:
|
||||
|
||||
- **Cognitive load:** Is the change harder to understand than the problem requires?
|
||||
- **Change propagation:** Will one product change force edits across unrelated layers?
|
||||
- **Knowledge duplication:** Is the same rule encoded in multiple places that can drift?
|
||||
- **Accidental complexity:** Did the change add abstractions, fallbacks, or state without need?
|
||||
- **Dependency direction:** Are API/MCP/CLI, services, repositories, and UI stores respecting
|
||||
their intended boundaries?
|
||||
- **Domain model distortion:** Do names and types still match the product concept, or did a
|
||||
transport/storage detail leak into the domain?
|
||||
- **Test oracle quality:** Would the tests fail for the bug or regression the change claims to
|
||||
protect against?
|
||||
|
||||
## House Rules To Check Explicitly
|
||||
|
||||
- No speculative `getattr(obj, "attr", default)` for unknown model shapes.
|
||||
- No broad exception swallowing, warning-only failure paths, or hidden fallback behavior.
|
||||
- No casts or `Any` that hide an unclear type relationship.
|
||||
- Dataclasses for internal value/result objects; Pydantic at validation/serialization
|
||||
boundaries.
|
||||
- Narrow `Protocol`s when only a capability is needed.
|
||||
- Explicit async/resource ownership, cancellation, and cleanup.
|
||||
- Meaningful regression tests or verification for risky changes.
|
||||
- Comments explain why, not what.
|
||||
|
||||
## Reporting Format
|
||||
|
||||
Lead with findings ordered by severity. Each finding should include:
|
||||
|
||||
| Severity | Use for |
|
||||
| -------- | ------- |
|
||||
| `high` | A likely correctness, security, data-loss, or tenant/workspace isolation failure |
|
||||
| `medium` | A concrete maintainability or boundary risk that can cause future defects |
|
||||
| `low` | A minor consistency issue, ambiguous guidance, or review-only cleanup |
|
||||
|
||||
```text
|
||||
severity | file:line | risk category | claim
|
||||
Why: concrete behavior or code path that proves the risk.
|
||||
Fix: smallest practical change, or "none obvious" if the risk needs product input.
|
||||
```
|
||||
|
||||
If there are no findings, say so and note any verification gaps that remain.
|
||||
@@ -0,0 +1,48 @@
|
||||
---
|
||||
name: fix-pr-issues
|
||||
description: Use when addressing Basic Memory pull request feedback, failed checks, or BM Bossbot blockers from Codex.
|
||||
---
|
||||
|
||||
# Fix Basic Memory PR Issues
|
||||
|
||||
Resolve PR feedback and failed checks, then wait for BM Bossbot to approve the
|
||||
new head SHA. This skill never merges a PR.
|
||||
|
||||
## Gather
|
||||
|
||||
1. Identify the PR:
|
||||
- `gh pr view --json number,url,headRefOid,mergeStateStatus,statusCheckRollup`
|
||||
|
||||
2. Collect feedback:
|
||||
- PR comments and review summaries
|
||||
- inline review comments and unresolved review threads
|
||||
- failed GitHub Actions jobs and relevant logs
|
||||
- the managed `BM_BOSSBOT_SUMMARY` block in the PR body
|
||||
|
||||
3. Build a short issue ledger:
|
||||
- source
|
||||
- concrete problem
|
||||
- expected fix
|
||||
- verification needed
|
||||
|
||||
## Fix
|
||||
|
||||
1. Address one ledger item at a time.
|
||||
2. Read each file in full before editing it.
|
||||
3. Keep diffs narrow and preserve unrelated user changes.
|
||||
4. Run the smallest meaningful verification first, then widen as needed.
|
||||
5. Commit with `git commit -s` when code or docs changed.
|
||||
|
||||
## Push And Recheck
|
||||
|
||||
1. Push the branch.
|
||||
2. Watch checks for the new `headRefOid`.
|
||||
3. Wait for the required `BM Bossbot Approval` status to pass on that exact SHA.
|
||||
4. If BM Bossbot reviews an older SHA, treat the approval as stale and keep
|
||||
waiting for the current one.
|
||||
|
||||
## Reply
|
||||
|
||||
For each addressed comment or blocker, reply with the fix commit, verification
|
||||
run, and current BM Bossbot status. Do not resolve or dismiss substantive
|
||||
feedback without evidence.
|
||||
@@ -0,0 +1,7 @@
|
||||
interface:
|
||||
display_name: "Fix PR Issues"
|
||||
short_description: "Address PR feedback and BM Bossbot blockers"
|
||||
icon_small: "./assets/icon.svg"
|
||||
icon_large: "./assets/icon.svg"
|
||||
brand_color: "#2563EB"
|
||||
default_prompt: "Use $fix-pr-issues to address PR feedback and wait for BM Bossbot Approval on the latest head SHA."
|
||||
@@ -0,0 +1,5 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="128" height="128" viewBox="0 0 24 24" fill="none" stroke="#111827" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round">
|
||||
<path d="M3 12h4l2-6 4 12 2-6h6"/>
|
||||
<path d="M4 20h16"/>
|
||||
</svg>
|
||||
|
||||
|
After Width: | Height: | Size: 249 B |
@@ -0,0 +1,247 @@
|
||||
---
|
||||
name: infographics
|
||||
description: Use when generating Basic Memory PR, changelog, release, or weekly images from Codex.
|
||||
---
|
||||
|
||||
# Basic Memory Images
|
||||
|
||||
Generate repository visuals with evidence-grounded content and canonical output
|
||||
paths. The file and marker names still say "infographic" for compatibility, but
|
||||
PR generation is image-first: scene, poster, painting, photograph, cover,
|
||||
tableau, staged artifact, or another editorial visual moment that describes the
|
||||
intent of the PR. PR images are non-gating BM Bossbot artifacts; changelog and
|
||||
release-summary images are manual evidence-pack workflows.
|
||||
|
||||
## Output Contract
|
||||
|
||||
- Base output directory: `docs/assets/infographics/`
|
||||
- PR image: `docs/assets/infographics/pr-<number>.webp`
|
||||
- Changelog image: `docs/assets/infographics/changelog.webp`
|
||||
- Weekly image:
|
||||
- This is always a 2-Week Retro window: previous ISO week through current ISO
|
||||
week (`start-week = current-week - 1`, `end-week = current-week`).
|
||||
- Same year window: `docs/assets/infographics/<year>-w<start-week>-w<end-week>.webp`
|
||||
- Cross-year window:
|
||||
`docs/assets/infographics/<start-year>-w<start-week>-<end-year>-w<end-week>.webp`
|
||||
|
||||
## PR Mode
|
||||
|
||||
PR mode uses the BM Bossbot summary block as source material. Do not hand-write
|
||||
claims that are not present in the PR body.
|
||||
|
||||
1. Fetch the PR body:
|
||||
|
||||
```bash
|
||||
gh pr view <number> --json body --jq '.body // ""' > /tmp/bm-pr-body.md
|
||||
```
|
||||
|
||||
2. Generate the canonical asset:
|
||||
|
||||
```bash
|
||||
uv run --script scripts/generate_pr_infographic.py \
|
||||
--pr-number <number> \
|
||||
--pr-body-file /tmp/bm-pr-body.md \
|
||||
--theme "<optional visual theme>" \
|
||||
--provenance-output /tmp/bm-infographic-provenance.md \
|
||||
--output docs/assets/infographics/pr-<number>.webp
|
||||
```
|
||||
|
||||
If the PR body contains a managed image theme block, the script reads it
|
||||
automatically:
|
||||
|
||||
```markdown
|
||||
<!-- BM_INFOGRAPHIC_THEME:start -->
|
||||
<theme>
|
||||
<!-- BM_INFOGRAPHIC_THEME:end -->
|
||||
```
|
||||
|
||||
Before spending an image call, test the prompt path locally:
|
||||
|
||||
```bash
|
||||
uv run --script scripts/generate_pr_infographic.py \
|
||||
--pr-number <number> \
|
||||
--pr-body-file /tmp/bm-pr-body.md \
|
||||
--theme "<optional visual theme>" \
|
||||
--output docs/assets/infographics/pr-<number>.webp \
|
||||
--print-prompt
|
||||
```
|
||||
|
||||
`--dry-run` is an alias for `--print-prompt`; both print the final prompt and
|
||||
exit without calling OpenAI.
|
||||
|
||||
When no theme is supplied, the script selects a deterministic BM visual
|
||||
direction from the style pool below based on the PR number and Bossbot summary.
|
||||
This keeps repeated PR images from collapsing into the same generic visual.
|
||||
|
||||
When the image is generated, also write provenance with
|
||||
`--provenance-output <path>`. BM Bossbot publishes that managed block into the
|
||||
PR body with these markers:
|
||||
|
||||
```markdown
|
||||
<!-- BM_INFOGRAPHIC_PROVENANCE:start -->
|
||||
...
|
||||
<!-- BM_INFOGRAPHIC_PROVENANCE:end -->
|
||||
```
|
||||
|
||||
The provenance block records the generated asset path, image model, size,
|
||||
quality, image mode, theme source, and selected visual direction. It
|
||||
intentionally does not dump the full generated prompt into the PR body. Treat
|
||||
this block as debugging and creative provenance only; it is not a merge gate.
|
||||
|
||||
The PR image is visual support only. The authoritative merge gate is the
|
||||
GitHub commit status named `BM Bossbot Approval`.
|
||||
|
||||
## Changelog Mode
|
||||
|
||||
Build an evidence pack before writing a prompt:
|
||||
|
||||
- diff truth source: merged PR diffs, merge commits, or local reconstructed diffs
|
||||
- changed-file orientation: `git diff --stat` plus key file reads
|
||||
- impact ledger: before/after outcomes tied to actual changes
|
||||
- discard list: misleading titles, reverted work, rename-only churn, speculative TODOs
|
||||
- chosen image form: poster, scene, tableau, cover, painting, photograph,
|
||||
staged artifact, or another editorial visual moment
|
||||
- chosen BM style category: exactly one category from the selection pool below
|
||||
|
||||
Read these references before drafting the prompt:
|
||||
|
||||
- `references/prompt-blueprint.md`
|
||||
- `references/style-balance.md`
|
||||
|
||||
Read the current `CHANGELOG.md` entries and include the latest meaningful
|
||||
changes.
|
||||
|
||||
## Style And Category Selection
|
||||
|
||||
Select exactly one BM style category per image based on semantic fit. The
|
||||
visual language should be recognizable and tasteful, while staying
|
||||
business-readable.
|
||||
|
||||
Create an image-first visual form that communicates the change: poster, scene,
|
||||
tableau, cover image, painting, photograph, staged artifact, or another
|
||||
editorial visual moment. Maps, diagrams, dossiers, charts, and labels can appear
|
||||
as props inside the scene, but do not make a text-heavy infographic.
|
||||
|
||||
BM category pool:
|
||||
|
||||
- computer science college textbooks: SICP-style diagrams, algorithms lectures,
|
||||
compiler pipelines, automata, database systems, type theory, operating systems
|
||||
- classic literature subjects: sea voyages, gothic manors, Dickensian city maps,
|
||||
Austen social graphs, library marginalia, travel journals
|
||||
- fantasy/D&D-inspired: quest maps, dungeon keys, guild ledgers, spellbooks,
|
||||
bestiaries, tavern notice boards; no copyrighted settings
|
||||
- Music: Metal, Hard Rock, Punk, techno, soul, reggae bands; no pop music, no
|
||||
direct band logos, album covers, or musician likenesses
|
||||
- sci-fi: Star Wars inspired knockoff, Spaceballs-adjacent space opera, fleet
|
||||
routes, mission consoles, contraband manifests; avoid copyrighted characters,
|
||||
logos, or named fictional universes
|
||||
- Conan the barbarian-inspired sword-and-sorcery: ruined temples, desert routes,
|
||||
battle standards, ancient maps; no named character likenesses
|
||||
- Comic books: issue covers, splash pages, action-panel maps, caption boxes,
|
||||
halftone energy, clean sound-effect typography
|
||||
- French new wave movies: poster style, stark typography, city route maps,
|
||||
jump-cut sequencing, high-contrast editorial photography cues
|
||||
- WWII propaganda posters: home-front public-information poster language,
|
||||
logistics arrows, ration charts, mobilization maps, bold simplified figures;
|
||||
no real-world party symbols, hate imagery, dehumanizing slogans, or false
|
||||
historical claims
|
||||
- Italian movie posters: hand-painted drama, bold credits, expressive color,
|
||||
route-map collage, 1960s or 1970s cinema energy; no direct film titles or
|
||||
actor likenesses
|
||||
- Shakespeare: stage maps, acts and scenes, dramatis personae, royal courts,
|
||||
backstage cue sheets
|
||||
- Greek mythology: temple diagrams, constellation routes, hero's journey maps,
|
||||
oracle tablets, labyrinths, ship routes
|
||||
- noir detective boards: case files, red-string maps, typed evidence labels,
|
||||
precinct wall charts
|
||||
- NASA mission-control dashboards: launch timelines, telemetry maps, orbital
|
||||
routes, status boards
|
||||
- space exploration and astronomy: celestial atlases, observatory charts,
|
||||
star-field maps, orbital mechanics diagrams, planetary survey routes,
|
||||
telescope annotations, mission trajectories, deep-space timelines
|
||||
- paintings: abstract painting, classical landscape, Remington-inspired western
|
||||
action painting, Rembrandt-inspired chiaroscuro, historical mural, stormy
|
||||
seascape, allegorical editorial painting
|
||||
- classic black-and-white photography: documentary field report, newsroom
|
||||
archive print, editorial photo essay, street photography, high-contrast
|
||||
darkroom print, contact sheet, civic infrastructure photograph
|
||||
- 80's action movies: practical explosions, smoky backlit warehouses, neon city
|
||||
streets, helicopter searchlights, mission dossiers, heroic silhouettes,
|
||||
high-stakes countdowns, painted ensemble posters; no direct actor likenesses,
|
||||
real film titles, franchise marks, or catchphrases
|
||||
- alchemy manuscripts: transformation diagrams, annotated symbols, recipe-like
|
||||
process maps, illuminated margins
|
||||
- brutalist civic planning: transit maps, concrete signage, zoning blocks,
|
||||
infrastructure diagrams
|
||||
|
||||
Selection rules:
|
||||
|
||||
- Pick one category only; do not create mixed mashups.
|
||||
- Pick the most appropriate image form. Prefer an actual scene, poster,
|
||||
painting, photograph, tableau, or cover over a text-heavy infographic.
|
||||
- Match metaphor to content, but do not overthink it. The category is a creative
|
||||
catalyst, not a semantic constraint.
|
||||
- Use a polished editorial rendering direction: smooth anti-aliased
|
||||
text, high contrast, clean edges, readable labels.
|
||||
- Make the category drive the composition through a readable staged moment,
|
||||
editorial composition, symbolic environment, route, artifact, or visual
|
||||
metaphor.
|
||||
- Keep the structure literal enough to aid understanding, but not so heavy that
|
||||
it obscures engineering meaning.
|
||||
- Give the image generator creative latitude on layout, structure, color palette,
|
||||
and visual metaphors. Be precise about what content to show, loose about how
|
||||
to show it.
|
||||
- Do not use copyrighted characters, logos, or named fictional universes. Use
|
||||
genre cues, knockoffs, and original compositions instead.
|
||||
|
||||
## Content-First Aesthetic Contract
|
||||
|
||||
The meaning must be readable and clearly hierarchical. Everything else is
|
||||
creative territory: image form, layout, visual metaphors, decorative elements,
|
||||
color choices, and category-specific visual language.
|
||||
|
||||
Hierarchy:
|
||||
|
||||
1. Meaning: what shipped, what changed, and why it matters must be clear.
|
||||
2. If the image uses text, labels, sections, or evidence bullets, they must be
|
||||
legible.
|
||||
3. The selected category's visual DNA should drive the composition as a poster,
|
||||
scene, painting, photograph, tableau, cover, or symbolic object arrangement.
|
||||
4. Do not play it safe. A visually striking image that someone wants to look at
|
||||
beats a correct but boring one.
|
||||
|
||||
Hard rules:
|
||||
|
||||
- Content sections and labels must be readable when present. Text cannot be
|
||||
obscured by decorations.
|
||||
- Do not use lore-heavy copy that competes with engineering or business meaning.
|
||||
- Every prompt must include a clear image-first composition cue: a staged scene,
|
||||
poster composition, painting, photograph, symbolic tableau, hero object,
|
||||
mission room, dossier, artifact, route, or visual metaphor.
|
||||
- Do not over-prescribe exact coordinates or panel geometry; give a composition
|
||||
backbone and let the model compose around it.
|
||||
|
||||
## Generation
|
||||
|
||||
1. Write the final prompt to a temporary markdown file.
|
||||
2. Generate with the shared image helper:
|
||||
|
||||
```bash
|
||||
uv run --script scripts/generate_infographic.py \
|
||||
--prompt-file /tmp/bm-infographic-prompt.md \
|
||||
--output docs/assets/infographics/<name>.webp
|
||||
```
|
||||
|
||||
3. Verify the image exists and is readable before reporting success.
|
||||
|
||||
## Quality Bar
|
||||
|
||||
- Tell a concrete before/after value story, not vague improvement claims.
|
||||
- Stay understandable for both engineers and non-technical stakeholders.
|
||||
- Use plain-language section titles and labels when text is present.
|
||||
- Include clear visual hierarchy: title, staged focal point, symbolic scene,
|
||||
evidence props, or hero object.
|
||||
- Avoid invented facts; only use provided source material.
|
||||
- Favor shipped outcomes over intermediate or reverted work.
|
||||
- Preserve readability with high contrast, non-tiny labels, and uncluttered
|
||||
layout.
|
||||
@@ -0,0 +1,7 @@
|
||||
interface:
|
||||
display_name: "Infographics"
|
||||
short_description: "Generate Basic Memory repo infographics"
|
||||
icon_small: "./assets/icon.svg"
|
||||
icon_large: "./assets/icon.svg"
|
||||
brand_color: "#2563EB"
|
||||
default_prompt: "Use $infographics to generate a Basic Memory PR or changelog infographic with canonical output paths."
|
||||
@@ -0,0 +1,5 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="128" height="128" viewBox="0 0 24 24" fill="none" stroke="#111827" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round">
|
||||
<path d="M3 12h4l2-6 4 12 2-6h6"/>
|
||||
<path d="M4 20h16"/>
|
||||
</svg>
|
||||
|
||||
|
After Width: | Height: | Size: 249 B |
@@ -0,0 +1,72 @@
|
||||
# Prompt Blueprint
|
||||
|
||||
Convert an evidence pack into a final visual prompt. Be precise about the
|
||||
content and loose about visual execution.
|
||||
|
||||
## Required Inputs
|
||||
|
||||
- Diff truth source summary
|
||||
- Changed-file orientation summary
|
||||
- Impact ledger with before/after outcomes
|
||||
- Discard list for excluded noise
|
||||
- Chosen image form
|
||||
- Chosen BM style category
|
||||
|
||||
## Prompt Shape
|
||||
|
||||
```text
|
||||
Create a polished Basic Memory editorial image inspired by
|
||||
<BM_STYLE_CATEGORY>. Use a poster, scene, tableau, painting, photograph, cover
|
||||
image, staged artifact, or another image-first form that best communicates the
|
||||
intent. Use HD editorial rendering with smooth anti-aliased text when text is
|
||||
present. Go bold and let the selected category drive the visual language through
|
||||
original, non-infringing cues.
|
||||
|
||||
TITLE:
|
||||
- "<clear title>"
|
||||
- "<scope subtitle>"
|
||||
|
||||
COMPOSITION:
|
||||
- Recreate a clear staged moment or symbolic image that describes the PR
|
||||
intent.
|
||||
- Maps, diagrams, dossiers, route lines, labels, and artifacts can appear as
|
||||
props inside the scene, but the output should read as an image rather than a
|
||||
dense infographic.
|
||||
- Take creative liberty with layout and styling.
|
||||
- The hard rule: the meaning must be readable and clearly hierarchical.
|
||||
- Keep labels plain-language and technical when labels are used.
|
||||
|
||||
CONTENT:
|
||||
1. "<section>"
|
||||
- <evidence-grounded outcome>
|
||||
- <evidence-grounded outcome>
|
||||
|
||||
2. "<section>"
|
||||
- <evidence-grounded outcome>
|
||||
- <evidence-grounded outcome>
|
||||
|
||||
METRICS:
|
||||
- <metric>
|
||||
- <metric>
|
||||
|
||||
STYLE DIRECTION:
|
||||
- Upscaled editorial, high contrast, anti-aliased text, smooth edges.
|
||||
- Let the category's visual DNA drive the composition.
|
||||
- Use genre/category cues only; do not use copyrighted characters, logos, named
|
||||
fictional universes, direct band logos, album art, or celebrity likenesses.
|
||||
|
||||
DO NOT:
|
||||
- Make text unreadable or let decoration obscure content.
|
||||
- Render a text-heavy infographic, dashboard, flowchart, timeline strip,
|
||||
checklist, bullet-list panel, or dense explanatory diagram.
|
||||
- Use crunchy low-resolution pixel art.
|
||||
- Invent facts not present in the evidence pack.
|
||||
```
|
||||
|
||||
## Writing Rules
|
||||
|
||||
- Keep each bullet specific and evidence-grounded.
|
||||
- Prefer outcome language over implementation trivia.
|
||||
- Default to three or four sections; never exceed five.
|
||||
- Give proportionally more space to dominant changes.
|
||||
- Keep the final prompt short, energetic, and readable.
|
||||
@@ -0,0 +1,42 @@
|
||||
# Style Balance Rubric
|
||||
|
||||
## Core Principle
|
||||
|
||||
Be bold, not confusing. The selected BM style category should structure the
|
||||
visual through a readable image-first composition, not decorate a generic grid.
|
||||
Use an editorial scene, poster, painting, photograph, cover, staged artifact, or
|
||||
tableau that turns the PR intent into a visual moment.
|
||||
|
||||
## Required Traits
|
||||
|
||||
- Anti-aliased typography
|
||||
- Smooth edges
|
||||
- High contrast between text and background
|
||||
- Plain-language section labels
|
||||
- Clear composition backbone: staged scene, editorial poster, painting,
|
||||
photograph, symbolic tableau, hero artifact, dossier, mission room, or route
|
||||
embodied as part of the scene
|
||||
- A single coherent BM style category, expressed through original visual cues
|
||||
|
||||
## Reject Or Rewrite If
|
||||
|
||||
- Content text is unreadable.
|
||||
- The prompt lacks a composition backbone.
|
||||
- The prompt over-prescribes exact panel positions or a rigid grid.
|
||||
- The style leans into crunchy low-resolution pixelation.
|
||||
- Copy uses lore-heavy references instead of engineering meaning.
|
||||
- The prompt uses copyrighted characters, logos, named fictional universes,
|
||||
direct band logos, album art, or celebrity likenesses.
|
||||
|
||||
## Creative Integration Patterns
|
||||
|
||||
- Use category-native map details to organize content: textbook diagrams,
|
||||
literary journeys, quest maps, tour posters, mission-control routes, stage
|
||||
blocking, mythic constellations, star charts, mission trajectories, case
|
||||
boards, or civic plans as props inside the image.
|
||||
- Recreate a scene, editorial poster, painting, photograph, cover, artifact, or
|
||||
tableau instead of sectioned bullets.
|
||||
- Map engineering metrics to visual counters, route progress, or status boards
|
||||
only when they naturally belong in the scene.
|
||||
- Let headers and accents borrow from the selected style.
|
||||
- Keep atmospheric details behind or around content, never over it.
|
||||
@@ -0,0 +1,114 @@
|
||||
---
|
||||
name: pr-create
|
||||
description: Use when creating or updating a Basic Memory pull request from Codex with BM Bossbot merge-gate monitoring.
|
||||
---
|
||||
|
||||
# Create A Basic Memory PR
|
||||
|
||||
Create or update a pull request for the current branch, then wait for BM
|
||||
Bossbot to approve the latest head SHA. This skill never merges a PR.
|
||||
|
||||
## Inputs
|
||||
|
||||
- Optional `<theme>`: free-form visual direction for the non-gating PR
|
||||
image. Example: `$pr-create "Italian movie poster"`.
|
||||
- Treat `<theme>` as style guidance only. It must not affect PR readiness,
|
||||
BM Bossbot review, status checks, or merge behavior.
|
||||
|
||||
## How To Use
|
||||
|
||||
Ask Codex to use the skill from a feature branch:
|
||||
|
||||
```text
|
||||
$pr-create
|
||||
$pr-create "Italian movie poster"
|
||||
$pr-create "80's action movies"
|
||||
```
|
||||
|
||||
Use the plain form when you only want the PR workflow. Pass a theme when you
|
||||
want the non-gating image to lean toward a particular visual direction. The
|
||||
theme can be specific ("Rembrandt-inspired approval scene") or broad ("let the
|
||||
model choose from BM categories").
|
||||
|
||||
## What Happens
|
||||
|
||||
1. Codex checks the branch, local verification, GitHub auth, commit sign-offs,
|
||||
and semantic PR title shape.
|
||||
2. Codex pushes the branch, creates or reuses the PR, and adds the optional
|
||||
`BM_INFOGRAPHIC_THEME` block when a theme was supplied.
|
||||
3. BM Bossbot runs from trusted base code, reviews sanitized PR metadata and
|
||||
diff context, and sets the required `BM Bossbot Approval` status for the
|
||||
exact head SHA.
|
||||
4. If approval succeeds, BM Bossbot may publish a non-gating image block and a
|
||||
provenance block:
|
||||
|
||||
```markdown
|
||||
<!-- BM_INFOGRAPHIC_PROVENANCE:start -->
|
||||
...
|
||||
<!-- BM_INFOGRAPHIC_PROVENANCE:end -->
|
||||
```
|
||||
|
||||
The provenance records the image mode, theme source, selected visual
|
||||
direction, and image settings. It is for review/debugging context only.
|
||||
5. Codex reports the PR URL, head SHA, checks watched, verification run, and BM
|
||||
Bossbot verdict.
|
||||
|
||||
The skill never merges, never enables auto-merge, and never treats the image or
|
||||
provenance block as a gate. The only required merge signal is the
|
||||
`BM Bossbot Approval` status on the current PR head SHA.
|
||||
|
||||
## Preflight
|
||||
|
||||
1. Confirm the repo and branch:
|
||||
- `git status --short --branch`
|
||||
- stop if detached or on `main`
|
||||
- keep unrelated user changes intact
|
||||
|
||||
2. Confirm GitHub access:
|
||||
- `gh auth status`
|
||||
- `gh repo view --json nameWithOwner,defaultBranchRef,url`
|
||||
|
||||
3. Check PR readiness:
|
||||
- commits are signed off with `git commit -s`
|
||||
- title uses the repo semantic format
|
||||
- local verification appropriate to the change has run
|
||||
|
||||
## Create Or Reuse
|
||||
|
||||
1. Push the branch:
|
||||
- `git push -u origin HEAD`
|
||||
|
||||
2. Check for an existing PR:
|
||||
- `gh pr view --json number,url,headRefOid,mergeStateStatus,statusCheckRollup`
|
||||
|
||||
3. If no PR exists, create one:
|
||||
- `gh pr create --fill`
|
||||
- adjust the title if it does not satisfy the semantic PR title workflow
|
||||
|
||||
4. If `<theme>` is provided, add or update this managed block in the PR body:
|
||||
|
||||
```markdown
|
||||
<!-- BM_INFOGRAPHIC_THEME:start -->
|
||||
<theme>
|
||||
<!-- BM_INFOGRAPHIC_THEME:end -->
|
||||
```
|
||||
|
||||
Keep the rest of the PR body intact. The theme is non-gating image guidance
|
||||
only.
|
||||
|
||||
5. Do not merge. Do not enable auto-merge.
|
||||
|
||||
## Watch The Gate
|
||||
|
||||
1. Trigger or wait for `.github/workflows/bm-bossbot.yml`.
|
||||
2. Watch the required commit status named `BM Bossbot Approval`.
|
||||
3. Treat approval as valid only when it is green for the current `headRefOid`.
|
||||
4. If the branch changes after approval, wait for BM Bossbot to review the new
|
||||
head SHA.
|
||||
5. If BM Bossbot fails or requests changes, use `$fix-pr-issues`.
|
||||
|
||||
## Report
|
||||
|
||||
Return the PR URL, current head SHA, checks watched, verification run, and the
|
||||
BM Bossbot verdict. Include the image `<theme>` if one was supplied. Be
|
||||
explicit when any check is still pending.
|
||||
@@ -0,0 +1,7 @@
|
||||
interface:
|
||||
display_name: "PR Create"
|
||||
short_description: "Create PRs and wait for BM Bossbot"
|
||||
icon_small: "./assets/icon.svg"
|
||||
icon_large: "./assets/icon.svg"
|
||||
brand_color: "#2563EB"
|
||||
default_prompt: "Use $pr-create to create or update this Basic Memory PR and wait for BM Bossbot Approval."
|
||||
@@ -0,0 +1,5 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="128" height="128" viewBox="0 0 24 24" fill="none" stroke="#111827" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round">
|
||||
<path d="M3 12h4l2-6 4 12 2-6h6"/>
|
||||
<path d="M4 20h16"/>
|
||||
</svg>
|
||||
|
||||
|
After Width: | Height: | Size: 249 B |
@@ -6,14 +6,14 @@
|
||||
},
|
||||
"metadata": {
|
||||
"description": "Official Basic Memory plugins from the canonical basic-memory repository",
|
||||
"version": "0.21.6"
|
||||
"version": "0.22.0"
|
||||
},
|
||||
"plugins": [
|
||||
{
|
||||
"name": "basic-memory",
|
||||
"source": "./plugins/claude-code",
|
||||
"description": "The bridge between Claude's working memory and Basic Memory's durable knowledge graph \u2014 session briefings, pre-compaction checkpoints, and capture reflexes",
|
||||
"version": "0.21.6",
|
||||
"version": "0.22.0",
|
||||
"author": {
|
||||
"name": "Basic Machines"
|
||||
},
|
||||
|
||||
@@ -30,7 +30,8 @@ The justfile target handles:
|
||||
- ✅ Beta version format validation (supports b1, b2, rc1, etc.)
|
||||
- ✅ Git status and branch checks
|
||||
- ✅ Quality checks (`just check` - lint, format, type-check, tests)
|
||||
- ✅ Version update in `src/basic_memory/__init__.py`
|
||||
- ✅ Version update across all consolidated manifests via `just set-version` (Python
|
||||
package + Claude Code plugin/marketplaces + Codex plugin + Hermes + OpenClaw)
|
||||
- ✅ Automatic commit with proper message
|
||||
- ✅ Tag creation and pushing to GitHub
|
||||
- ✅ Beta release workflow trigger
|
||||
@@ -90,6 +91,6 @@ Monitor release: https://github.com/basicmachines-co/basic-memory/actions
|
||||
- Beta releases are pre-releases for testing new features
|
||||
- Automatically published to PyPI with pre-release flag
|
||||
- Uses the automated justfile target for consistency
|
||||
- Version is automatically updated in `__init__.py`
|
||||
- Version is automatically updated across all consolidated manifests via `just set-version`
|
||||
- Ideal for validating changes before stable release
|
||||
- Supports both beta (b1, b2) and release candidate (rc1, rc2) versions
|
||||
- Supports both beta (b1, b2) and release candidate (rc1, rc2) versions
|
||||
|
||||
@@ -42,7 +42,8 @@ The justfile target handles:
|
||||
- ✅ Version format validation
|
||||
- ✅ Git status and branch checks
|
||||
- ✅ Quality checks (`just check` - lint, format, type-check, tests)
|
||||
- ✅ Version update across all consolidated manifests via `just set-version` (Python package + Claude Code plugin/marketplaces + Hermes + OpenClaw)
|
||||
- ✅ Version update across all consolidated manifests via `just set-version` (Python
|
||||
package + Claude Code plugin/marketplaces + Codex plugin + Hermes + OpenClaw)
|
||||
- ✅ Automatic commit with proper message
|
||||
- ✅ Tag creation and pushing to GitHub
|
||||
- ✅ Release workflow trigger (automatic on tag push)
|
||||
@@ -194,12 +195,13 @@ Users can now upgrade:
|
||||
- Version is automatically updated across **all** consolidated manifests via
|
||||
`just set-version <version>` (which calls `scripts/update_versions.py`): the
|
||||
Python package (`__init__.py`, `server.json`) **and** the plugin/agent artifacts
|
||||
(Claude Code `plugin.json` + root/local marketplaces, Hermes `plugin.yaml` +
|
||||
`__init__.py`, OpenClaw `package.json`). To bump only the plugin/agent artifacts
|
||||
(Claude Code `plugin.json` + root/local marketplaces, Codex `plugin.json`,
|
||||
Hermes `plugin.yaml` + `__init__.py`, OpenClaw `package.json`). To bump only
|
||||
the plugin/agent artifacts
|
||||
out of band, use `just set-packages-version <version>` (preview with
|
||||
`just set-packages-version-dry-run <version>`).
|
||||
- Triggers automated GitHub release with changelog
|
||||
- Package is published to PyPI for `pip` and `uv` users
|
||||
- Homebrew formula is automatically updated for stable releases
|
||||
- MCP Registry is updated manually via `mcp-publisher publish`
|
||||
- Supports multiple installation methods (uv, pip, Homebrew)
|
||||
- Supports multiple installation methods (uv, pip, Homebrew)
|
||||
|
||||
+1
@@ -0,0 +1 @@
|
||||
../../.agents/skills/basic-machines-review
|
||||
@@ -1,26 +1,18 @@
|
||||
name: Claude Code Review
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [opened, synchronize]
|
||||
# Optional: Only run on specific file changes
|
||||
# paths:
|
||||
# - "src/**/*.ts"
|
||||
# - "src/**/*.tsx"
|
||||
# - "src/**/*.js"
|
||||
# - "src/**/*.jsx"
|
||||
"on":
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
pr_number:
|
||||
description: Pull request number to review manually
|
||||
required: true
|
||||
|
||||
env:
|
||||
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
|
||||
|
||||
jobs:
|
||||
claude-review:
|
||||
# Only run for organization members and collaborators
|
||||
if: |
|
||||
github.event.pull_request.author_association == 'OWNER' ||
|
||||
github.event.pull_request.author_association == 'MEMBER' ||
|
||||
github.event.pull_request.author_association == 'COLLABORATOR'
|
||||
|
||||
if: inputs.pr_number != ''
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
@@ -43,7 +35,14 @@ jobs:
|
||||
track_progress: true # Enable visual progress tracking
|
||||
allowed_bots: '*'
|
||||
prompt: |
|
||||
Review this Basic Memory PR against our team checklist:
|
||||
Review Basic Memory PR #${{ inputs.pr_number }} as an advisory manual review.
|
||||
|
||||
Use `gh pr view ${{ inputs.pr_number }}` and related `gh pr`/`gh api`
|
||||
commands to inspect the pull request. Do not merge the PR and do not
|
||||
treat this advisory review as the required merge gate. BM Bossbot owns
|
||||
the required `BM Bossbot Approval` status.
|
||||
|
||||
Review the PR against our team checklist:
|
||||
|
||||
## Code Quality & Standards
|
||||
- [ ] Follows Basic Memory's coding conventions in CLAUDE.md
|
||||
|
||||
@@ -13,9 +13,10 @@ env:
|
||||
|
||||
jobs:
|
||||
docker:
|
||||
runs-on: ubuntu-latest
|
||||
runs-on: depot-ubuntu-24.04
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write
|
||||
packages: write
|
||||
|
||||
steps:
|
||||
@@ -24,10 +25,8 @@ jobs:
|
||||
with:
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v4
|
||||
with:
|
||||
platforms: linux/amd64,linux/arm64
|
||||
- name: Set up Depot
|
||||
uses: depot/setup-action@v1
|
||||
|
||||
- name: Log in to GitHub Container Registry
|
||||
uses: docker/login-action@v4
|
||||
@@ -49,13 +48,12 @@ jobs:
|
||||
type=raw,value=latest,enable={{is_default_branch}}
|
||||
|
||||
- name: Build and push Docker image
|
||||
uses: docker/build-push-action@v7
|
||||
uses: depot/build-push-action@v1
|
||||
with:
|
||||
project: ${{ vars.DEPOT_BASIC_MEMORY_PROJECT_ID || vars.DEPOT_PROJECT_ID }}
|
||||
context: .
|
||||
file: ./Dockerfile
|
||||
platforms: linux/amd64,linux/arm64
|
||||
push: true
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
+123
-10
@@ -13,9 +13,43 @@ on:
|
||||
|
||||
env:
|
||||
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
|
||||
# Branch builds (PRs arrive as push events — this workflow has no
|
||||
# pull_request trigger) select only impacted tests from the cached testmon
|
||||
# baseline (branch cache falling back to main's full-run recording). Pushes
|
||||
# to main run the full suite with --testmon-noselect to refresh the baseline.
|
||||
BASIC_MEMORY_TESTMON_FLAGS: ${{ github.ref_name == 'main' && '--testmon-noselect' || '--testmon --testmon-forceselect' }}
|
||||
|
||||
jobs:
|
||||
changes:
|
||||
# Docs/workflow-only changes skip the entire test matrix while the workflow
|
||||
# still concludes successfully, so the BM Bossbot gate (workflow_run on
|
||||
# Tests success) keeps firing and the PR stays mergeable.
|
||||
name: Detect code changes
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
code: ${{ steps.filter.outputs.code }}
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- id: filter
|
||||
uses: dorny/paths-filter@v3
|
||||
with:
|
||||
# Tests only runs on push events; for branch pushes compare against
|
||||
# main (merge-base), for main pushes dorny diffs the push range.
|
||||
base: main
|
||||
filters: |
|
||||
code:
|
||||
- 'src/**'
|
||||
- 'tests/**'
|
||||
- 'test-int/**'
|
||||
- 'alembic/**'
|
||||
- 'pyproject.toml'
|
||||
- 'uv.lock'
|
||||
- 'justfile'
|
||||
- '.github/workflows/test.yml'
|
||||
|
||||
static-checks:
|
||||
needs: changes
|
||||
if: needs.changes.outputs.code == 'true'
|
||||
name: Static Checks (Python 3.12)
|
||||
timeout-minutes: 20
|
||||
runs-on: ubuntu-latest
|
||||
@@ -54,6 +88,8 @@ jobs:
|
||||
just lint
|
||||
|
||||
test-sqlite-unit:
|
||||
needs: changes
|
||||
if: needs.changes.outputs.code == 'true'
|
||||
name: Test SQLite Unit (${{ matrix.os }}, Python ${{ matrix.python-version }})
|
||||
timeout-minutes: 45
|
||||
strategy:
|
||||
@@ -64,6 +100,8 @@ jobs:
|
||||
python-version: "3.12"
|
||||
- os: ubuntu-latest
|
||||
python-version: "3.13"
|
||||
# Python 3.14 unit tests are the longest full-suite slice; keep this
|
||||
# one on GitHub-hosted runners after Depot terminated it mid-suite.
|
||||
- os: ubuntu-latest
|
||||
python-version: "3.14"
|
||||
- os: windows-latest
|
||||
@@ -87,6 +125,19 @@ jobs:
|
||||
|
||||
- uses: extractions/setup-just@v4
|
||||
|
||||
- name: Cache pytest-testmon results
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: |
|
||||
.testmondata
|
||||
.testmondata-shm
|
||||
.testmondata-wal
|
||||
key: ${{ runner.os }}-testmon-sqlite-unit-py${{ matrix.python-version }}-${{ github.ref_name }}-${{ github.run_id }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-testmon-sqlite-unit-py${{ matrix.python-version }}-${{ github.ref_name }}-
|
||||
${{ runner.os }}-testmon-sqlite-unit-py${{ matrix.python-version }}-main-
|
||||
${{ runner.os }}-testmon-sqlite-unit-py${{ matrix.python-version }}-
|
||||
|
||||
- name: Create virtual env
|
||||
run: |
|
||||
uv venv
|
||||
@@ -100,6 +151,8 @@ jobs:
|
||||
just test-unit-sqlite
|
||||
|
||||
test-sqlite-integration:
|
||||
needs: changes
|
||||
if: needs.changes.outputs.code == 'true'
|
||||
name: Test SQLite Integration (${{ matrix.os }}, Python ${{ matrix.python-version }})
|
||||
timeout-minutes: 45
|
||||
strategy:
|
||||
@@ -133,6 +186,19 @@ jobs:
|
||||
|
||||
- uses: extractions/setup-just@v4
|
||||
|
||||
- name: Cache pytest-testmon results
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: |
|
||||
.testmondata
|
||||
.testmondata-shm
|
||||
.testmondata-wal
|
||||
key: ${{ runner.os }}-testmon-sqlite-integration-py${{ matrix.python-version }}-${{ github.ref_name }}-${{ github.run_id }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-testmon-sqlite-integration-py${{ matrix.python-version }}-${{ github.ref_name }}-
|
||||
${{ runner.os }}-testmon-sqlite-integration-py${{ matrix.python-version }}-main-
|
||||
${{ runner.os }}-testmon-sqlite-integration-py${{ matrix.python-version }}-
|
||||
|
||||
- name: Create virtual env
|
||||
run: |
|
||||
uv venv
|
||||
@@ -146,15 +212,19 @@ jobs:
|
||||
just test-int-sqlite
|
||||
|
||||
test-postgres-unit:
|
||||
name: Test Postgres Unit (Python ${{ matrix.python-version }})
|
||||
needs: changes
|
||||
if: needs.changes.outputs.code == 'true'
|
||||
name: Test Postgres Unit (Python ${{ matrix.python-version }}, shard ${{ matrix.group }}/3)
|
||||
timeout-minutes: 60
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- python-version: "3.12"
|
||||
- python-version: "3.13"
|
||||
- python-version: "3.14"
|
||||
# Shard the largest suite across parallel jobs: each shard is a full job
|
||||
# with its own Postgres service running 1/3 of the collection.
|
||||
# Postgres runs on the latest Python only — the SQLite matrix carries
|
||||
# Python-version coverage; Postgres carries backend coverage.
|
||||
group: [1, 2, 3]
|
||||
python-version: ["3.14"]
|
||||
runs-on: ubuntu-latest
|
||||
services:
|
||||
postgres:
|
||||
@@ -190,6 +260,20 @@ jobs:
|
||||
|
||||
- uses: extractions/setup-just@v4
|
||||
|
||||
- name: Cache pytest-testmon results
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: |
|
||||
.testmondata
|
||||
.testmondata-shm
|
||||
.testmondata-wal
|
||||
key: ${{ runner.os }}-testmon-postgres-unit-py${{ matrix.python-version }}-g${{ matrix.group }}-${{ github.ref_name }}-${{ github.run_id }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-testmon-postgres-unit-py${{ matrix.python-version }}-g${{ matrix.group }}-${{ github.ref_name }}-
|
||||
${{ runner.os }}-testmon-postgres-unit-py${{ matrix.python-version }}-g${{ matrix.group }}-main-
|
||||
${{ runner.os }}-testmon-postgres-unit-py${{ matrix.python-version }}-main-
|
||||
${{ runner.os }}-testmon-postgres-unit-py${{ matrix.python-version }}-
|
||||
|
||||
- name: Create virtual env
|
||||
run: |
|
||||
uv venv
|
||||
@@ -200,18 +284,19 @@ jobs:
|
||||
|
||||
- name: Run tests
|
||||
run: |
|
||||
just test-unit-postgres
|
||||
BASIC_MEMORY_PYTEST_SPLIT_FLAGS="--splits 3 --group ${{ matrix.group }}" just test-unit-postgres
|
||||
|
||||
test-postgres-integration:
|
||||
needs: changes
|
||||
if: needs.changes.outputs.code == 'true'
|
||||
name: Test Postgres Integration (Python ${{ matrix.python-version }})
|
||||
timeout-minutes: 45
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- python-version: "3.12"
|
||||
- python-version: "3.13"
|
||||
- python-version: "3.14"
|
||||
# Latest Python only: SQLite carries version coverage, Postgres carries
|
||||
# backend coverage.
|
||||
python-version: ["3.14"]
|
||||
runs-on: ubuntu-latest
|
||||
services:
|
||||
postgres:
|
||||
@@ -247,6 +332,19 @@ jobs:
|
||||
|
||||
- uses: extractions/setup-just@v4
|
||||
|
||||
- name: Cache pytest-testmon results
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: |
|
||||
.testmondata
|
||||
.testmondata-shm
|
||||
.testmondata-wal
|
||||
key: ${{ runner.os }}-testmon-postgres-integration-py${{ matrix.python-version }}-${{ github.ref_name }}-${{ github.run_id }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-testmon-postgres-integration-py${{ matrix.python-version }}-${{ github.ref_name }}-
|
||||
${{ runner.os }}-testmon-postgres-integration-py${{ matrix.python-version }}-main-
|
||||
${{ runner.os }}-testmon-postgres-integration-py${{ matrix.python-version }}-
|
||||
|
||||
- name: Create virtual env
|
||||
run: |
|
||||
uv venv
|
||||
@@ -260,6 +358,8 @@ jobs:
|
||||
just test-int-postgres
|
||||
|
||||
test-semantic:
|
||||
needs: changes
|
||||
if: needs.changes.outputs.code == 'true'
|
||||
name: Test Semantic (Python 3.12)
|
||||
timeout-minutes: 45
|
||||
runs-on: ubuntu-latest
|
||||
@@ -281,6 +381,19 @@ jobs:
|
||||
|
||||
- uses: extractions/setup-just@v4
|
||||
|
||||
- name: Cache pytest-testmon results
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: |
|
||||
.testmondata
|
||||
.testmondata-shm
|
||||
.testmondata-wal
|
||||
key: ${{ runner.os }}-testmon-semantic-py3.12-${{ github.ref_name }}-${{ github.run_id }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-testmon-semantic-py3.12-${{ github.ref_name }}-
|
||||
${{ runner.os }}-testmon-semantic-py3.12-main-
|
||||
${{ runner.os }}-testmon-semantic-py3.12-
|
||||
|
||||
- name: Create virtual env
|
||||
run: |
|
||||
uv venv
|
||||
|
||||
+1
-1
@@ -49,7 +49,7 @@ ENV/
|
||||
/docs/.obsidian/
|
||||
/examples/.obsidian/
|
||||
/examples/.basic-memory/
|
||||
|
||||
/docs/assets
|
||||
|
||||
# claude action
|
||||
claude-output
|
||||
|
||||
@@ -108,13 +108,34 @@ Before opening or updating a PR, run the checks that mirror the common required
|
||||
- Follow the repository pattern for data access
|
||||
- Tools communicate to api routers via the httpx ASGI client (in process)
|
||||
|
||||
### Programming Style
|
||||
|
||||
See [docs/ENGINEERING_STYLE.md](docs/ENGINEERING_STYLE.md) for the fuller house style. The
|
||||
short version for agents:
|
||||
|
||||
- Prefer type-safe, explicit designs over object-heavy indirection. Use Python 3.12 `type`
|
||||
aliases, full annotations, and narrow `Protocol`s when a caller only needs a capability.
|
||||
- Use dataclasses for internal value objects and operation results; use Pydantic v2 at API,
|
||||
CLI, MCP, and persistence boundaries where validation and serialization matter.
|
||||
- Keep async boundaries obvious. Resource-owning code should use context managers, propagate
|
||||
cancellation, and avoid hidden background work unless the lifecycle is explicit.
|
||||
- Fail fast. Do not add silent fallback logic, broad exception swallowing, speculative
|
||||
`getattr`, or casts that hide an unclear model shape.
|
||||
- Keep control flow simple and local. Push branching decisions up, keep leaf helpers focused,
|
||||
and name values after the domain concept they carry.
|
||||
- Use evidence-first testing. Add or update meaningful regression tests for bugs and risky
|
||||
behavior, prefer real code paths over mocks, and run the narrowest command that proves the
|
||||
change before widening verification.
|
||||
- Comments should explain why a branch, invariant, or constraint exists. Avoid comments that
|
||||
merely narrate obvious code.
|
||||
|
||||
### Code Change Guidelines
|
||||
|
||||
- **Full file read before edits**: Before editing any file, read it in full first to ensure complete context; partial reads lead to corrupted edits
|
||||
- **Minimize diffs**: Prefer the smallest change that satisfies the request. Avoid unrelated refactors or style rewrites unless necessary for correctness
|
||||
- **No speculative getattr**: Never use `getattr(obj, "attr", default)` when unsure about attribute names. Check the class definition or source code first
|
||||
- **Fail fast**: Write code with fail-fast logic by default. Do not swallow exceptions with errors or warnings
|
||||
- **No fallback logic**: Do not add fallback logic unless explicitly told to and agreed with the user
|
||||
- **House style is canonical**: Follow the Programming Style section above for type-safe,
|
||||
fail-fast code; do not hide unclear models with speculative attributes, broad exception
|
||||
handling, casts, or unapproved fallback logic
|
||||
- **No guessing**: Do not say "The issue is..." before you actually know what the issue is. Investigate first.
|
||||
|
||||
### Literate Programming Style
|
||||
@@ -348,6 +369,13 @@ See `.claude/commands/release/release.md` (and `beta.md`, `release-check.md`, `c
|
||||
- Manage snapshots: `basic-memory cloud snapshot [create|list|delete|show|browse]`
|
||||
- Restore from snapshot: `basic-memory cloud restore <path> --snapshot <id>`
|
||||
|
||||
**Cloud Sync Commands (Personal and Team workspaces):**
|
||||
- Fetch cloud changes (cloud -> local): `basic-memory cloud pull --name "name"` (Team-safe; additive, never deletes local)
|
||||
- Upload local changes (local -> cloud): `basic-memory cloud push --name "name"` (Team-safe; additive, never deletes cloud)
|
||||
- Resolve conflicts on push/pull: `--on-conflict [fail|keep-local|keep-cloud|keep-both]` (default `fail` lists conflicts and aborts, git-style)
|
||||
- One-way mirror (local -> cloud): `basic-memory cloud sync --name "name"` (Personal workspaces only; deletes cloud files missing locally)
|
||||
- Two-way mirror (local <-> cloud): `basic-memory cloud bisync --name "name"` (Personal workspaces only)
|
||||
|
||||
### MCP Capabilities
|
||||
|
||||
- Basic Memory exposes these MCP tools to LLMs:
|
||||
|
||||
@@ -1,5 +1,70 @@
|
||||
# CHANGELOG
|
||||
|
||||
## v0.22.0 (2026-06-11)
|
||||
|
||||
Team-safe cloud sync. New additive `bm cloud push` and `bm cloud pull`
|
||||
commands work safely on shared Team workspaces, while the destructive mirror
|
||||
commands are gated to Personal workspaces. Also: a large batch of MCP tool
|
||||
fixes, search improvements, and embedding reliability work.
|
||||
|
||||
### Features
|
||||
|
||||
- **#917**: Added Team-safe `bm cloud push` / `bm cloud pull`. Both are
|
||||
additive (they never delete on the destination) and abort on conflicts by
|
||||
default, git-style, with `--on-conflict {fail|keep-local|keep-cloud|keep-both}`.
|
||||
The destructive `bm cloud sync` / `bm cloud bisync` mirrors are now gated
|
||||
to Personal workspaces.
|
||||
- **#920**: Team push/pull uses per-workspace rclone remotes, so remotes and
|
||||
credentials stay scoped to each workspace.
|
||||
- **#908**: Search supports an observation category filter.
|
||||
- **#809**: Added an experimental LiteLLM embedding provider for semantic
|
||||
search (marked experimental, see **#899**).
|
||||
- **#907**: `bm tool write-note` accepts `--type`.
|
||||
- **#906**: `bm status` accepts `--wait` and `--timeout`.
|
||||
- Added the `bm tool delete-note` command.
|
||||
- **#905**: Improved workspace and cloud bisync command discoverability.
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- **#931 / #946**: Truncated observation permalinks are disambiguated to
|
||||
prevent search-index collisions, and `build_context` resolves observations
|
||||
by the same permalink the search index uses (**#909**, **#929**).
|
||||
- **#934**: `edit_note` recovers when the file exists on disk but is not yet
|
||||
indexed (**#581**).
|
||||
- **#911 / #932 / #941**: Comma-separated tags are split consistently in
|
||||
`parse_tags` and the `search_notes` tags parameter, with input normalized
|
||||
for direct callers (**#910**).
|
||||
- **#933**: `read_note` accepts `page`/`page_size` for parity with sibling
|
||||
tools (**#883**).
|
||||
- **#914 / #904 / #916**: `move_note` resolves `memory://` URLs, stops
|
||||
falsely rejecting same-project moves as cross-project, no longer reports
|
||||
false success across project boundaries, and mismatch guidance points at
|
||||
the landing path.
|
||||
- **#915**: Navigation pagination is validated, and `recent_activity` shows
|
||||
the correct project.
|
||||
- **#913**: `bm tool` commands align with MCP behavior (error exit codes,
|
||||
overwrite handling, category support, defaults).
|
||||
- **#923**: `bm cloud setup` no longer overwrites an existing rclone remote
|
||||
(**#922**).
|
||||
- **#912**: The `note_types` search filter is case-insensitive.
|
||||
- `build_context` allows cross-project context traversal, `write_note`
|
||||
resolves overwrite conflicts, and sync uses strict deferred relation
|
||||
resolution.
|
||||
- Embedding reliability: FastEmbed vectors are L2-normalized (**#843**),
|
||||
corrupt FastEmbed model caches self-heal (**#900**), a single embedding
|
||||
provider is reused per process (**#903**), `sqlite-vec` loads for the
|
||||
embedding-status query (**#901**), and engine disposal no longer crashes
|
||||
on the Postgres backend (**#902**).
|
||||
|
||||
### Maintenance
|
||||
|
||||
- Leaner, faster CI: testmon-selected branch builds, sharded Postgres jobs,
|
||||
and faster default test fixtures (**#928**, **#938**, **#945**).
|
||||
- Documented personal-vs-team cloud sync semantics (**#947**, closes
|
||||
**#851**).
|
||||
- Fixed npx skill install docs (**#927**).
|
||||
- Added `glama.json` to claim the Glama MCP directory listing (**#953**).
|
||||
|
||||
## v0.21.6 (2026-06-04)
|
||||
|
||||
Monorepo consolidation plus a redesigned Claude Code plugin. The satellite
|
||||
|
||||
@@ -216,14 +216,14 @@ Source: [`plugins/claude-code`](plugins/claude-code).
|
||||
### Shared skills
|
||||
|
||||
Framework-agnostic `SKILL.md` files live in [`skills/`](skills). If your
|
||||
Skills CLI supports subpath installs:
|
||||
Skills CLI supports repository subdirectory sources:
|
||||
|
||||
```bash
|
||||
npx skills add basicmachines-co/basic-memory --path skills
|
||||
npx skills add basicmachines-co/basic-memory/skills
|
||||
```
|
||||
|
||||
If it does not, copy the `memory-*` directories from `skills/` into your
|
||||
agent's skills directory as a temporary Phase 1 install path.
|
||||
If your installed Skills CLI cannot load that source, update the CLI or copy
|
||||
the `memory-*` directories from `skills/` into your agent's skills directory.
|
||||
|
||||
### Hermes
|
||||
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
# Basic Memory Engineering Style
|
||||
|
||||
Style is how we make code easier to verify. Prefer explicit, typed, local-first code that
|
||||
preserves the file system as the source of truth while keeping the database, API, and MCP
|
||||
surfaces in sync.
|
||||
|
||||
## Design Center
|
||||
|
||||
- Basic Memory is local-first. Markdown files are the durable source; SQLite/Postgres indexes
|
||||
are derived state that should be rebuilt or reconciled from files when needed.
|
||||
- Keep the existing boundary order: CLI/MCP/API entrypoints compose dependencies, services own
|
||||
business behavior, repositories own database access, and file services own filesystem writes.
|
||||
- MCP tools should remain atomic and composable. They should call API routers through typed MCP
|
||||
clients, not reach around into services.
|
||||
- Prefer small, explicit abstractions that match a real domain boundary. Avoid object
|
||||
hierarchies when a function, dataclass, type alias, or protocol describes the concept better.
|
||||
|
||||
## Types And Data
|
||||
|
||||
- Use full type annotations and Python 3.12 syntax. Introduce `type` aliases for repeated
|
||||
structured shapes, callback signatures, or domain concepts that would otherwise become
|
||||
anonymous `dict[str, Any]` values.
|
||||
- Use dataclasses for internal values, operation inputs, and service results. Prefer
|
||||
`frozen=True` when the value should not change and `slots=True` when identity/dynamic
|
||||
attributes are not needed.
|
||||
- Use Pydantic v2 at boundaries that validate, serialize, or deserialize data: API payloads,
|
||||
CLI/MCP schemas, configuration, and persistence-adjacent schemas.
|
||||
- Use narrow `Protocol`s when a caller needs a capability rather than a concrete repository or
|
||||
service. Keep protocols small enough that fake implementations in tests are obvious.
|
||||
- Avoid speculative `getattr`, broad casts, or `Any` as a way to paper over uncertainty. Read
|
||||
the model or schema definition and make the type relationship explicit.
|
||||
|
||||
## Control Flow And Resources
|
||||
|
||||
- Fail fast when an invariant is broken. Do not swallow exceptions, add warning-only error
|
||||
handling, or introduce fallback behavior unless the user explicitly agrees to that behavior.
|
||||
- Keep control flow simple and close to the domain decision. Push `if` statements up into the
|
||||
function that owns orchestration; keep leaf helpers focused on computation or one side effect.
|
||||
- Make async/resource boundaries visible with context managers and explicit lifecycles. Do not
|
||||
start background work without a clear owner, cancellation story, and verification path.
|
||||
- Keep file mutations centralized through the existing file utilities/services so checksum,
|
||||
atomic write, and index synchronization behavior stays coherent.
|
||||
|
||||
## Testing And Verification
|
||||
|
||||
- Use evidence-first testing, not mechanical TDD. For bugs and risky behavior, add or update a
|
||||
regression test that would catch the failure. For small documentation-only edits, use the
|
||||
relevant doc/repo hygiene checks.
|
||||
- Prefer tests that exercise real code paths. Use mocks, doubles, or `monkeypatch` only when
|
||||
the external boundary would be slow, nondeterministic, or impossible to trigger directly.
|
||||
- Keep coverage at 100% for new code. Use `# pragma: no cover` only for code that would require
|
||||
disproportionate mocking and is covered through an integration or runtime path.
|
||||
- Start with targeted commands, then widen as risk grows: focused pytest, `just fast-check`,
|
||||
`just doctor`, package checks for agent packaging changes, and full SQLite/Postgres gates
|
||||
when behavior crosses shared boundaries.
|
||||
|
||||
## Comments And Names
|
||||
|
||||
- Name values after the domain concept they carry: project, entity, permalink, tenant, route,
|
||||
checksum, observation, relation, batch, or index state.
|
||||
- Comments should say why a branch, invariant, retry, lifecycle, or compatibility constraint
|
||||
exists. Section headers are useful when a function or file has clear phases.
|
||||
- Avoid comments that restate the code. If a comment cannot explain a decision, simplify the
|
||||
code or improve the name instead.
|
||||
+175
-55
@@ -8,9 +8,25 @@ The cloud CLI enables you to:
|
||||
- **Authenticate cloud access** - OAuth/API key credentials are stored locally for cloud operations
|
||||
- **Project-scoped sync** - Each project independently manages its sync configuration
|
||||
- **Explicit operations** - Sync only what you want, when you want
|
||||
- **Bidirectional sync** - Keep local and cloud in sync with rclone bisync
|
||||
- **Team-safe push/pull** - Additive, git-style transfers that work on shared Team workspaces
|
||||
- **Bidirectional sync** - Keep local and cloud in sync with rclone bisync (Personal workspaces)
|
||||
- **Offline access** - Work locally, sync when ready
|
||||
|
||||
### Personal vs Team workspaces
|
||||
|
||||
The transfer commands fall into two groups:
|
||||
|
||||
| Command | Direction | Behavior | Personal | Team |
|
||||
|---|---|---|---|---|
|
||||
| `bm cloud pull` | cloud → local | **additive** — never deletes local | ✅ | ✅ |
|
||||
| `bm cloud push` | local → cloud | **additive** — never deletes cloud | ✅ | ✅ |
|
||||
| `bm cloud sync` | local → cloud | **mirror** — deletes cloud files missing locally | ✅ | ❌ |
|
||||
| `bm cloud bisync` | local ↔ cloud | **mirror** — two-way, deletes on both sides | ✅ | ❌ |
|
||||
|
||||
`sync` and `bisync` are mirror operations: one local tree becomes authoritative and files missing on the other side get deleted. That is correct for a Personal workspace (one user, one source of truth) but unsafe on a shared Team bucket, where it could delete a teammate's files. On Team workspaces these commands exit early with a clear error and point you at `push`/`pull`.
|
||||
|
||||
`push` and `pull` are additive (they use `rclone copy`, which never deletes on the destination), so they are safe on both Personal and Team workspaces.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Before using Basic Memory Cloud, you need:
|
||||
@@ -55,8 +71,8 @@ bm project add work --cloud --local-path ~/work-notes
|
||||
bm project add temp --cloud # No local sync
|
||||
|
||||
# Now you can sync individually (after initial --resync):
|
||||
bm project bisync --name research
|
||||
bm project bisync --name work
|
||||
bm cloud bisync --name research
|
||||
bm cloud bisync --name work
|
||||
# temp stays cloud-only
|
||||
```
|
||||
|
||||
@@ -137,10 +153,10 @@ Establish the initial sync baseline. **Best practice:** Always preview with `--d
|
||||
|
||||
```bash
|
||||
# Step 1: Preview the initial sync (recommended)
|
||||
bm project bisync --name research --resync --dry-run
|
||||
bm cloud bisync --name research --resync --dry-run
|
||||
|
||||
# Step 2: If all looks good, run the actual sync
|
||||
bm project bisync --name research --resync
|
||||
bm cloud bisync --name research --resync
|
||||
```
|
||||
|
||||
**What happens under the covers:**
|
||||
@@ -167,7 +183,7 @@ This will effectively make both Path1 and Path2 filesystems contain a matching s
|
||||
After the first sync, just run bisync without `--resync`:
|
||||
|
||||
```bash
|
||||
bm project bisync --name research
|
||||
bm cloud bisync --name research
|
||||
```
|
||||
|
||||
**What happens:**
|
||||
@@ -235,7 +251,7 @@ bm project add research --cloud --local-path ~/Documents/research
|
||||
- Stores sync config in `~/.basic-memory/config.json`
|
||||
- Prepares for bisync (but doesn't sync yet)
|
||||
|
||||
**Result:** Project ready to sync. Run `bm project bisync --name research --resync` to establish baseline.
|
||||
**Result:** Project ready to sync. Run `bm cloud bisync --name research --resync` to establish baseline.
|
||||
|
||||
**Use case 3: Add sync to existing cloud project**
|
||||
|
||||
@@ -294,18 +310,98 @@ For MCP stdio, routing is always local.
|
||||
|
||||
### Understanding the Sync Commands
|
||||
|
||||
**There are three sync-related commands:**
|
||||
**There are five sync-related commands:**
|
||||
|
||||
1. `bm project sync` - One-way: local → cloud (make cloud match local)
|
||||
2. `bm project bisync` - Two-way: local ↔ cloud (recommended)
|
||||
3. `bm project check` - Verify files match (no changes)
|
||||
| Command | Direction | Workspace | Summary |
|
||||
|---|---|---|---|
|
||||
| `bm cloud pull` | cloud → local | Personal + Team | Fetch cloud changes, additively (git-style) |
|
||||
| `bm cloud push` | local → cloud | Personal + Team | Upload local changes, additively (git-style) |
|
||||
| `bm cloud sync` | local → cloud | Personal only | One-way mirror (cloud becomes identical to local) |
|
||||
| `bm cloud bisync` | local ↔ cloud | Personal only | Two-way mirror (recommended for solo use) |
|
||||
| `bm cloud check` | — | Personal only | Verify mirror integrity (no changes) |
|
||||
|
||||
### One-Way Sync: Local → Cloud
|
||||
If you collaborate on a shared Team workspace, use **`push`/`pull`** (see [Team Workspaces](#team-workspaces-push--pull-additive-git-style)). If you are the only writer (a Personal workspace), the mirror commands `sync`/`bisync` give you a single source of truth.
|
||||
|
||||
### Team Workspaces: push / pull (additive, git-style)
|
||||
|
||||
`push` and `pull` are the Team-safe transfer commands. They model `git push` / `git pull`:
|
||||
|
||||
- **`bm cloud pull`** fetches changes from the cloud into your local directory.
|
||||
- **`bm cloud push`** uploads your local changes to the cloud.
|
||||
|
||||
Both use `rclone copy`, so they are **additive — they never delete on the destination**. A conflict (a file that differs on both sides) is never resolved silently: by default the command aborts and lists the conflicting files, exactly like git refusing to clobber your changes.
|
||||
|
||||
#### Pull: fetch cloud changes
|
||||
|
||||
```bash
|
||||
# Preview first (recommended)
|
||||
bm cloud pull --name research --dry-run
|
||||
|
||||
# Fetch new/changed cloud files into local
|
||||
bm cloud pull --name research
|
||||
```
|
||||
|
||||
**What happens:**
|
||||
1. Compares cloud and local with `rclone check`
|
||||
2. Downloads files that are new or changed on the cloud
|
||||
3. Leaves your local-only files untouched (never deletes local)
|
||||
4. If any file differs on both sides, aborts and lists the conflicts (unless you pass `--on-conflict`)
|
||||
|
||||
#### Push: upload local changes
|
||||
|
||||
```bash
|
||||
bm cloud push --name research --dry-run
|
||||
bm cloud push --name research
|
||||
```
|
||||
|
||||
**What happens:**
|
||||
1. Compares local and cloud with `rclone check`
|
||||
2. Uploads files that are new or changed locally
|
||||
3. Leaves cloud-only files untouched (never deletes cloud)
|
||||
4. If any file differs on both sides, aborts and lists the conflicts — pull first, like a rejected `git push`
|
||||
|
||||
#### Resolving conflicts
|
||||
|
||||
When `push`/`pull` reports conflicts, re-run with `--on-conflict` to choose how differing files are handled. The value names exactly what survives, so it reads the same in both directions:
|
||||
|
||||
| `--on-conflict` | Behavior |
|
||||
|---|---|
|
||||
| `fail` *(default)* | List the conflicting files and exit without transferring anything |
|
||||
| `keep-cloud` | Take the cloud version (pull: overwrite local; push: skip those files) |
|
||||
| `keep-local` | Keep the local version (pull: skip those files; push: overwrite cloud) |
|
||||
| `keep-both` | Keep both — write the incoming version beside the existing one as `name.conflict-<date>.md` |
|
||||
|
||||
```bash
|
||||
# A teammate edited notes you also changed locally — pull reports a conflict:
|
||||
bm cloud pull --name research
|
||||
# pull aborted: 1 file(s) differ between local and cloud.
|
||||
# * notes/decisions.md
|
||||
# Re-run with one of: --on-conflict keep-cloud | keep-local | keep-both
|
||||
|
||||
# Take the cloud copy:
|
||||
bm cloud pull --name research --on-conflict keep-cloud
|
||||
|
||||
# Or keep both versions to merge by hand:
|
||||
bm cloud pull --name research --on-conflict keep-both
|
||||
```
|
||||
|
||||
#### Limitations
|
||||
|
||||
`push`/`pull` are deliberately simple, conflict-aware byte transfers — not a full reconciler. Without a sync baseline:
|
||||
|
||||
- **Deletions are not propagated.** A note deleted on one side is not removed from the other (we cannot tell an intentional delete from a file the other side never had). This is surfaced in the command output.
|
||||
- **Every divergence is treated as a conflict.** We cannot tell a teammate's edit from your stale copy, so any differing file prompts a decision rather than auto-resolving.
|
||||
|
||||
For conflict-aware *editing*, write through the MCP/API tools (which merge at the note level). A Team-safe bidirectional reconciler with a real baseline is tracked in [issue #862](https://github.com/basicmachines-co/basic-memory/issues/862).
|
||||
|
||||
### One-Way Sync: Local → Cloud (Personal only)
|
||||
|
||||
**Use case:** You made changes locally and want to push to cloud (overwrite cloud).
|
||||
|
||||
> **Personal workspaces only.** `sync` is a destructive mirror — it deletes cloud files that are not present locally. On a Team workspace it would delete a teammate's files, so it is blocked there. Use `bm cloud push` (additive) on Team workspaces.
|
||||
|
||||
```bash
|
||||
bm project sync --name research
|
||||
bm cloud sync --name research
|
||||
```
|
||||
|
||||
**What happens:**
|
||||
@@ -321,16 +417,18 @@ bm project sync --name research
|
||||
- You want to force cloud to match local
|
||||
- You don't care about cloud changes
|
||||
|
||||
### Two-Way Sync: Local ↔ Cloud (Recommended)
|
||||
### Two-Way Sync: Local ↔ Cloud (Personal only, recommended for solo use)
|
||||
|
||||
**Use case:** You edit files both locally and in cloud UI, want both to stay in sync.
|
||||
|
||||
> **Personal workspaces only.** `bisync` is a two-way mirror that can delete and overwrite on both sides. It is blocked on Team workspaces — use `bm cloud pull` then `bm cloud push` there. A Team-safe bidirectional reconciler is tracked separately ([issue #862](https://github.com/basicmachines-co/basic-memory/issues/862)).
|
||||
|
||||
```bash
|
||||
# First time - establish baseline
|
||||
bm project bisync --name research --resync
|
||||
bm cloud bisync --name research --resync
|
||||
|
||||
# Subsequent syncs
|
||||
bm project bisync --name research
|
||||
bm cloud bisync --name research
|
||||
```
|
||||
|
||||
**What happens:**
|
||||
@@ -349,7 +447,7 @@ echo "Local change" > ~/Documents/research/notes.md
|
||||
# Cloud now has: "Cloud change"
|
||||
|
||||
# Run bisync
|
||||
bm project bisync --name research
|
||||
bm cloud bisync --name research
|
||||
|
||||
# Result: Newer file wins (based on modification time)
|
||||
# If cloud was more recent, cloud version kept
|
||||
@@ -361,12 +459,14 @@ bm project bisync --name research
|
||||
- You edit in multiple places
|
||||
- You want automatic conflict resolution
|
||||
|
||||
### Verify Sync Integrity
|
||||
### Verify Sync Integrity (Personal only)
|
||||
|
||||
**Use case:** Check if local and cloud match without making changes.
|
||||
|
||||
> **Personal workspaces only.** `check` compares against the Personal workspace mirror remote, like `sync`/`bisync`. On Team workspaces use `bm cloud pull --dry-run` / `bm cloud push --dry-run` to preview differences instead.
|
||||
|
||||
```bash
|
||||
bm project check --name research
|
||||
bm cloud check --name research
|
||||
```
|
||||
|
||||
**What happens:**
|
||||
@@ -378,7 +478,7 @@ bm project check --name research
|
||||
|
||||
```bash
|
||||
# One-way check (faster)
|
||||
bm project check --name research --one-way
|
||||
bm cloud check --name research --one-way
|
||||
```
|
||||
|
||||
### Preview Changes (Dry Run)
|
||||
@@ -386,7 +486,7 @@ bm project check --name research --one-way
|
||||
**Use case:** See what would change without actually syncing.
|
||||
|
||||
```bash
|
||||
bm project bisync --name research --dry-run
|
||||
bm cloud bisync --name research --dry-run
|
||||
```
|
||||
|
||||
**What happens:**
|
||||
@@ -432,20 +532,20 @@ bm project add work --cloud --local-path ~/work-notes
|
||||
bm project add personal --cloud --local-path ~/personal
|
||||
|
||||
# Establish baselines
|
||||
bm project bisync --name research --resync
|
||||
bm project bisync --name work --resync
|
||||
bm project bisync --name personal --resync
|
||||
bm cloud bisync --name research --resync
|
||||
bm cloud bisync --name work --resync
|
||||
bm cloud bisync --name personal --resync
|
||||
|
||||
# Daily workflow: sync everything
|
||||
bm project bisync --name research
|
||||
bm project bisync --name work
|
||||
bm project bisync --name personal
|
||||
bm cloud bisync --name research
|
||||
bm cloud bisync --name work
|
||||
bm cloud bisync --name personal
|
||||
```
|
||||
|
||||
**Future:** `--all` flag will sync all configured projects:
|
||||
|
||||
```bash
|
||||
bm project bisync --all # Coming soon
|
||||
bm cloud bisync --all # Coming soon
|
||||
```
|
||||
|
||||
### Mixed Usage
|
||||
@@ -462,8 +562,8 @@ bm project add archive --cloud
|
||||
bm project add temp-notes --cloud
|
||||
|
||||
# Sync only the configured ones
|
||||
bm project bisync --name research
|
||||
bm project bisync --name work
|
||||
bm cloud bisync --name research
|
||||
bm cloud bisync --name work
|
||||
|
||||
# Archive and temp-notes stay cloud-only
|
||||
```
|
||||
@@ -661,7 +761,7 @@ code ~/.basic-memory/.bmignore
|
||||
echo "*.tmp" >> ~/.basic-memory/.bmignore
|
||||
|
||||
# Next sync uses updated patterns
|
||||
bm project bisync --name research
|
||||
bm cloud bisync --name research
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
@@ -724,7 +824,7 @@ bm cloud login
|
||||
**Solution:**
|
||||
|
||||
```bash
|
||||
bm project bisync --name research --resync
|
||||
bm cloud bisync --name research --resync
|
||||
```
|
||||
|
||||
**What this does:**
|
||||
@@ -747,7 +847,7 @@ bm project bisync --name research --resync
|
||||
echo "# Research Notes" > ~/Documents/research/README.md
|
||||
|
||||
# Now run bisync
|
||||
bm project bisync --name research --resync
|
||||
bm cloud bisync --name research --resync
|
||||
```
|
||||
|
||||
**Why this happens:** Bisync creates listing files that track the state of each side. When both directories are completely empty, these listing files are considered invalid by rclone.
|
||||
@@ -764,10 +864,10 @@ bm project bisync --name research --resync
|
||||
|
||||
```bash
|
||||
# Clear bisync state
|
||||
bm project bisync-reset research
|
||||
bm cloud bisync-reset research
|
||||
|
||||
# Re-establish baseline
|
||||
bm project bisync --name research --resync
|
||||
bm cloud bisync --name research --resync
|
||||
```
|
||||
|
||||
**What this does:**
|
||||
@@ -787,16 +887,16 @@ bm project bisync --name research --resync
|
||||
|
||||
```bash
|
||||
# Check what would be deleted
|
||||
bm project bisync --name research --dry-run
|
||||
bm cloud bisync --name research --dry-run
|
||||
|
||||
# If correct, establish new baseline
|
||||
bm project bisync --name research --resync
|
||||
bm cloud bisync --name research --resync
|
||||
```
|
||||
|
||||
**Solution 2:** Use one-way sync if you know local is correct:
|
||||
|
||||
```bash
|
||||
bm project sync --name research
|
||||
bm cloud sync --name research
|
||||
```
|
||||
|
||||
### Project Not Configured for Sync
|
||||
@@ -809,7 +909,7 @@ bm project sync --name research
|
||||
|
||||
```bash
|
||||
bm cloud sync-setup research ~/Documents/research
|
||||
bm project bisync --name research --resync
|
||||
bm cloud bisync --name research --resync
|
||||
```
|
||||
|
||||
### Connection Issues
|
||||
@@ -880,20 +980,30 @@ bm project set-local <name> # Revert project to local mode
|
||||
### File Synchronization
|
||||
|
||||
```bash
|
||||
# One-way sync (local → cloud)
|
||||
bm project sync --name <project>
|
||||
bm project sync --name <project> --dry-run
|
||||
bm project sync --name <project> --verbose
|
||||
# Pull: fetch cloud changes (cloud → local) - Personal + Team, additive
|
||||
bm cloud pull --name <project>
|
||||
bm cloud pull --name <project> --dry-run
|
||||
bm cloud pull --name <project> --on-conflict [fail|keep-local|keep-cloud|keep-both]
|
||||
|
||||
# Two-way sync (local ↔ cloud) - Recommended
|
||||
bm project bisync --name <project> # After first --resync
|
||||
bm project bisync --name <project> --resync # First time / force baseline
|
||||
bm project bisync --name <project> --dry-run
|
||||
bm project bisync --name <project> --verbose
|
||||
# Push: upload local changes (local → cloud) - Personal + Team, additive
|
||||
bm cloud push --name <project>
|
||||
bm cloud push --name <project> --dry-run
|
||||
bm cloud push --name <project> --on-conflict [fail|keep-local|keep-cloud|keep-both]
|
||||
|
||||
# Integrity check
|
||||
bm project check --name <project>
|
||||
bm project check --name <project> --one-way
|
||||
# One-way mirror (local → cloud) - Personal workspaces only
|
||||
bm cloud sync --name <project>
|
||||
bm cloud sync --name <project> --dry-run
|
||||
bm cloud sync --name <project> --verbose
|
||||
|
||||
# Two-way mirror (local ↔ cloud) - Personal workspaces only
|
||||
bm cloud bisync --name <project> # After first --resync
|
||||
bm cloud bisync --name <project> --resync # First time / force baseline
|
||||
bm cloud bisync --name <project> --dry-run
|
||||
bm cloud bisync --name <project> --verbose
|
||||
|
||||
# Integrity check - Personal workspaces only
|
||||
bm cloud check --name <project>
|
||||
bm cloud check --name <project> --one-way
|
||||
|
||||
# List project files by route
|
||||
bm project ls --name <project> # Default target: local
|
||||
@@ -909,15 +1019,25 @@ bm project ls --name <project> --cloud --path <subpath>
|
||||
1. **Authenticate cloud access** - `bm cloud login`
|
||||
2. **Install rclone** - `bm cloud setup`
|
||||
3. **Add projects with sync** - `bm project add research --cloud --local-path ~/Documents/research`
|
||||
4. **Preview first sync** - `bm project bisync --name research --resync --dry-run`
|
||||
5. **Establish baseline** - `bm project bisync --name research --resync`
|
||||
6. **Daily workflow** - `bm project bisync --name research`
|
||||
|
||||
**Personal workspace (solo, mirror) workflow:**
|
||||
|
||||
4. **Preview first sync** - `bm cloud bisync --name research --resync --dry-run`
|
||||
5. **Establish baseline** - `bm cloud bisync --name research --resync`
|
||||
6. **Daily workflow** - `bm cloud bisync --name research`
|
||||
|
||||
**Team workspace (shared, additive) workflow:**
|
||||
|
||||
4. **Fetch teammates' changes** - `bm cloud pull --name research`
|
||||
5. **Upload your changes** - `bm cloud push --name research`
|
||||
6. **Resolve conflicts explicitly** - re-run with `--on-conflict keep-cloud|keep-local|keep-both`
|
||||
|
||||
**Key benefits:**
|
||||
- ✅ Each project independently syncs (or doesn't)
|
||||
- ✅ Projects can live anywhere on disk
|
||||
- ✅ Explicit sync operations (no magic)
|
||||
- ✅ Safe by design (max delete limits, conflict resolution)
|
||||
- ✅ Team-safe push/pull that never delete on the destination
|
||||
- ✅ Safe by design (max delete limits, conflict resolution, git-style conflict aborts)
|
||||
- ✅ Full offline access (work locally, sync when ready)
|
||||
|
||||
**Future enhancements:**
|
||||
|
||||
@@ -0,0 +1,300 @@
|
||||
# LiteLLM Provider
|
||||
|
||||
Basic Memory can use the LiteLLM SDK for semantic search embeddings. This lets you
|
||||
keep Basic Memory's vector indexing and search behavior while routing embedding calls
|
||||
to OpenAI-compatible and provider-specific backends such as OpenAI, Azure OpenAI,
|
||||
Cohere, Bedrock, NVIDIA NIM, and other LiteLLM-supported embedding providers.
|
||||
|
||||
Use this page when you want to try a non-default embedding model, validate a provider,
|
||||
or tune LiteLLM-specific settings.
|
||||
|
||||
> **Experimental — advanced users only.** The LiteLLM provider is experimental and
|
||||
> intended for users who are comfortable operating remote embedding backends. It makes
|
||||
> paid, networked API calls, requires per-model dimension and input-role configuration,
|
||||
> and reindexing a real corpus can be slow and spend provider quota (see
|
||||
> [Reindexing with a remote provider](#reindexing-with-a-remote-provider)). For most
|
||||
> users, the default local **FastEmbed** provider is the recommended choice. Use LiteLLM
|
||||
> only if you know what you're doing.
|
||||
|
||||
## Quick Start
|
||||
|
||||
The default LiteLLM model is OpenAI `text-embedding-3-small` through the LiteLLM
|
||||
model string `openai/text-embedding-3-small`.
|
||||
|
||||
```bash
|
||||
export BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED=true
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER=litellm
|
||||
export OPENAI_API_KEY=sk-...
|
||||
|
||||
bm reindex --embeddings
|
||||
```
|
||||
|
||||
Then use vector or hybrid search:
|
||||
|
||||
```python
|
||||
search_notes("login token flow", search_type="hybrid")
|
||||
```
|
||||
|
||||
## Basic Memory Options
|
||||
|
||||
All options can be set in config or as environment variables.
|
||||
|
||||
| Config Field | Env Var | Default | Notes |
|
||||
|---|---|---|---|
|
||||
| `semantic_search_enabled` | `BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED` | Auto | Set to `true` to force vector/hybrid support on. |
|
||||
| `semantic_embedding_provider` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER` | `fastembed` | Set to `litellm` for the LiteLLM provider. |
|
||||
| `semantic_embedding_model` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_MODEL` | `bge-small-en-v1.5` | With `litellm`, the default is remapped to `openai/text-embedding-3-small`. |
|
||||
| `semantic_embedding_dimensions` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_DIMENSIONS` | Provider default | Required for non-default LiteLLM models because vector tables are dimensioned before the first API call. |
|
||||
| `semantic_embedding_forward_dimensions` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_FORWARD_DIMENSIONS` | Auto | Sends `dimensions` to LiteLLM only when supported. Auto is enabled for `text-embedding-3` model strings. |
|
||||
| `semantic_embedding_document_input_type` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_DOCUMENT_INPUT_TYPE` | Auto | LiteLLM `input_type` for indexed notes/passages. |
|
||||
| `semantic_embedding_query_input_type` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_QUERY_INPUT_TYPE` | Auto | LiteLLM `input_type` for search queries. |
|
||||
| `semantic_embedding_batch_size` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_BATCH_SIZE` | `2` | Number of text chunks per provider request. |
|
||||
| `semantic_embedding_request_concurrency` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_REQUEST_CONCURRENCY` | `4` | Maximum concurrent LiteLLM embedding requests. |
|
||||
| `semantic_embedding_sync_batch_size` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_SYNC_BATCH_SIZE` | `2` | Number of prepared vector jobs flushed through the sync pipeline together. |
|
||||
|
||||
## Dimensions
|
||||
|
||||
Basic Memory needs the vector dimension before it can create SQLite or Postgres
|
||||
vector tables. The OpenAI default is known, so this works without an explicit
|
||||
dimension:
|
||||
|
||||
```bash
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER=litellm
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_MODEL=openai/text-embedding-3-small
|
||||
```
|
||||
|
||||
For every other LiteLLM model, set the dimension explicitly:
|
||||
|
||||
```bash
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER=litellm
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_MODEL=cohere/embed-english-v3.0
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_DIMENSIONS=1024
|
||||
```
|
||||
|
||||
For fixed-size models, `semantic_embedding_dimensions` is Basic Memory's local
|
||||
schema and validation size. For OpenAI/Azure `text-embedding-3` models, LiteLLM
|
||||
can also forward `dimensions` as a provider-side reduced-output request. Basic
|
||||
Memory enables that automatically when the model string contains `text-embedding-3`.
|
||||
|
||||
If you use an Azure deployment alias such as `azure/<deployment-name>`, the model
|
||||
string may not reveal that the underlying model supports reduced output dimensions.
|
||||
Set this only when your deployment supports it:
|
||||
|
||||
```bash
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_FORWARD_DIMENSIONS=true
|
||||
```
|
||||
|
||||
## Asymmetric Models
|
||||
|
||||
Some embedding models use different request roles for indexed documents and
|
||||
search queries. Basic Memory automatically sets these for known LiteLLM families:
|
||||
|
||||
| Model Family | Document `input_type` | Query `input_type` |
|
||||
|---|---|---|
|
||||
| Cohere v3 embeddings | `search_document` | `search_query` |
|
||||
| NVIDIA NIM retrieval embeddings | `passage` | `query` |
|
||||
|
||||
For any other asymmetric model, configure both roles explicitly:
|
||||
|
||||
```bash
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_DOCUMENT_INPUT_TYPE=passage
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_QUERY_INPUT_TYPE=query
|
||||
```
|
||||
|
||||
Changing provider, model, dimensions, dimension-forwarding, or document/query
|
||||
roles changes the meaning of stored vectors. Rebuild embeddings after any of
|
||||
those changes:
|
||||
|
||||
```bash
|
||||
bm reindex --embeddings
|
||||
```
|
||||
|
||||
## Reindexing with a remote provider
|
||||
|
||||
Embedding a real corpus through a network API is far slower than local FastEmbed, and
|
||||
the defaults are tuned for the local case. Two things to know before you run a full
|
||||
reindex.
|
||||
|
||||
**Raise the sync batch size.** `semantic_embedding_sync_batch_size` defaults to `2`, and
|
||||
it — not `semantic_embedding_batch_size` — governs throughput on the sync pipeline. With
|
||||
the default, a full reindex can take tens of seconds *per note* against a remote provider.
|
||||
Raising both to a larger value turns a multi-minute (or longer) reindex into well under a
|
||||
minute for the same corpus:
|
||||
|
||||
```bash
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_SYNC_BATCH_SIZE=32
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_BATCH_SIZE=64
|
||||
```
|
||||
|
||||
Stay within the provider's per-request size and rate limits — Cohere v3, for example,
|
||||
accepts up to 96 inputs per embedding request.
|
||||
|
||||
**Changing dimensions requires recreating the vector table.** Basic Memory dimensions the
|
||||
vector table on first index and refuses to mix sizes. Switching to a model with a
|
||||
different dimension (for example FastEmbed 384 → OpenAI 1536 → Cohere 1024) makes a plain
|
||||
`bm reindex` raise an `Embedding dimension mismatch` error. Recreate the table with a full
|
||||
rebuild — files are the source of truth, so this re-indexes from disk and re-embeds
|
||||
everything:
|
||||
|
||||
```bash
|
||||
bm reset --reindex
|
||||
```
|
||||
|
||||
To trial a provider without disturbing your existing index, point Basic Memory at a
|
||||
throwaway config + database instead:
|
||||
|
||||
```bash
|
||||
export BASIC_MEMORY_CONFIG_DIR=/tmp/bm-litellm-trial
|
||||
```
|
||||
|
||||
## Provider Setup Examples
|
||||
|
||||
LiteLLM reads provider credentials from the environment. These are the examples
|
||||
covered by Basic Memory's live validation harness.
|
||||
|
||||
### OpenAI Through LiteLLM
|
||||
|
||||
```bash
|
||||
export OPENAI_API_KEY=sk-...
|
||||
export BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED=true
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER=litellm
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_MODEL=openai/text-embedding-3-small
|
||||
```
|
||||
|
||||
### Cohere v3
|
||||
|
||||
```bash
|
||||
export COHERE_API_KEY=...
|
||||
export BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED=true
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER=litellm
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_MODEL=cohere/embed-english-v3.0
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_DIMENSIONS=1024
|
||||
```
|
||||
|
||||
The provider auto-selects `search_document` for indexed chunks and `search_query`
|
||||
for search queries.
|
||||
|
||||
### Azure OpenAI
|
||||
|
||||
```bash
|
||||
export AZURE_API_KEY=...
|
||||
export AZURE_API_BASE=https://<resource-name>.openai.azure.com
|
||||
export AZURE_API_VERSION=2024-02-01
|
||||
|
||||
export BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED=true
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER=litellm
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_MODEL=azure/<deployment-name>
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_DIMENSIONS=1536
|
||||
```
|
||||
|
||||
If your Azure deployment is a reduced-dimension `text-embedding-3` deployment,
|
||||
set the dimension you want and enable forwarding:
|
||||
|
||||
```bash
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_DIMENSIONS=512
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_FORWARD_DIMENSIONS=true
|
||||
```
|
||||
|
||||
### NVIDIA NIM
|
||||
|
||||
```bash
|
||||
export NVIDIA_NIM_API_KEY=...
|
||||
# Optional when using a custom or self-hosted NIM endpoint:
|
||||
export NVIDIA_NIM_API_BASE=https://integrate.api.nvidia.com/v1
|
||||
|
||||
export BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED=true
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER=litellm
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_MODEL=nvidia_nim/nvidia/embed-qa-4
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_DIMENSIONS=1024
|
||||
```
|
||||
|
||||
The provider auto-selects `passage` for indexed chunks and `query` for search
|
||||
queries.
|
||||
|
||||
## Testing LiteLLM Providers
|
||||
|
||||
Run the non-live LiteLLM unit and harness tests first:
|
||||
|
||||
```bash
|
||||
uv run pytest tests/repository/test_litellm_provider.py \
|
||||
test-int/semantic/test_litellm_live_harness.py -q
|
||||
```
|
||||
|
||||
Run the SQLite and Postgres vector identity regressions when changing model
|
||||
identity, role, or vector sync behavior:
|
||||
|
||||
```bash
|
||||
uv run pytest \
|
||||
tests/repository/test_sqlite_vector_search_repository.py::test_sqlite_embedding_model_key_includes_litellm_role_settings \
|
||||
-q
|
||||
|
||||
BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest \
|
||||
tests/repository/test_postgres_search_repository.py::test_postgres_litellm_role_change_reembeds_existing_chunks \
|
||||
-q
|
||||
```
|
||||
|
||||
The Postgres command uses testcontainers, so Docker must be running.
|
||||
|
||||
## Live Provider Harness
|
||||
|
||||
The live harness makes real LiteLLM API calls and spends provider quota. It is
|
||||
opt-in by design:
|
||||
|
||||
```bash
|
||||
export OPENAI_API_KEY=sk-...
|
||||
export COHERE_API_KEY=...
|
||||
|
||||
just test-litellm-live
|
||||
```
|
||||
|
||||
Built-in cases run when their API keys are present:
|
||||
|
||||
| Case | Required Env Var | Validates |
|
||||
|---|---|---|
|
||||
| `openai-text-embedding-3-small` | `OPENAI_API_KEY` | OpenAI via LiteLLM, 1536 dimensions, normalized vectors, ranking sanity. |
|
||||
| `cohere-embed-english-v3` | `COHERE_API_KEY` | Cohere v3 role handling, 1024 dimensions, normalized vectors, ranking sanity. |
|
||||
|
||||
Add provider aliases or new backends with a custom cases file:
|
||||
|
||||
```bash
|
||||
cat > /tmp/litellm-cases.json <<'JSON'
|
||||
[
|
||||
{
|
||||
"name": "azure-text-embedding-3-small-512",
|
||||
"model": "azure/<deployment-name>",
|
||||
"dimensions": 512,
|
||||
"api_key_env": "AZURE_API_KEY",
|
||||
"forward_dimensions": true
|
||||
},
|
||||
{
|
||||
"name": "nvidia-embed-qa-4",
|
||||
"model": "nvidia_nim/nvidia/embed-qa-4",
|
||||
"dimensions": 1024,
|
||||
"api_key_env": "NVIDIA_NIM_API_KEY",
|
||||
"document_input_type": "passage",
|
||||
"query_input_type": "query"
|
||||
}
|
||||
]
|
||||
JSON
|
||||
|
||||
just test-litellm-live --cases-file /tmp/litellm-cases.json
|
||||
```
|
||||
|
||||
For CI-style output:
|
||||
|
||||
```bash
|
||||
just test-litellm-live --cases-file /tmp/litellm-cases.json --json
|
||||
```
|
||||
|
||||
The harness embeds two documents and one query, validates dimension and vector
|
||||
normalization, checks that the authentication query ranks the authentication
|
||||
document above a distractor, and reports latency plus role/dimension settings.
|
||||
|
||||
## Provider Reference
|
||||
|
||||
LiteLLM's own provider and embedding docs are the source of truth for current
|
||||
model strings and credential names:
|
||||
|
||||
- [LiteLLM embedding models](https://docs.litellm.ai/docs/embedding/supported_embedding)
|
||||
- [LiteLLM Azure OpenAI provider](https://docs.litellm.ai/docs/providers/azure)
|
||||
- [LiteLLM NVIDIA NIM provider](https://docs.litellm.ai/docs/providers/nvidia_nim)
|
||||
+116
-5
@@ -99,10 +99,13 @@ All settings are fields on `BasicMemoryConfig` and can be set via environment va
|
||||
| Config Field | Env Var | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `semantic_search_enabled` | `BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED` | Auto (`true` when semantic deps are available) | Enable semantic search. Required before vector/hybrid modes work. |
|
||||
| `semantic_embedding_provider` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER` | `"fastembed"` | Embedding provider: `"fastembed"` (local) or `"openai"` (API). |
|
||||
| `semantic_embedding_provider` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER` | `"fastembed"` | Embedding provider: `"fastembed"` (local), `"openai"` (API), or `"litellm"` (multi-provider API, **experimental** — advanced users only). |
|
||||
| `semantic_embedding_model` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_MODEL` | `"bge-small-en-v1.5"` | Model identifier. Auto-adjusted per provider if left at default. |
|
||||
| `semantic_embedding_dimensions` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_DIMENSIONS` | Auto-detected | Vector dimensions. 384 for FastEmbed, 1536 for OpenAI. Override only if using a non-default model. |
|
||||
| `semantic_embedding_batch_size` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_BATCH_SIZE` | `64` | Number of texts to embed per batch. |
|
||||
| `semantic_embedding_dimensions` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_DIMENSIONS` | Provider default | Vector dimensions. 384 for FastEmbed, 1536 for OpenAI/LiteLLM OpenAI. Required when using a non-default LiteLLM model. |
|
||||
| `semantic_embedding_forward_dimensions` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_FORWARD_DIMENSIONS` | Auto | LiteLLM-only override for whether configured dimensions are sent as a provider-side output-size request. |
|
||||
| `semantic_embedding_batch_size` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_BATCH_SIZE` | `2` | Number of texts to embed per batch. |
|
||||
| `semantic_embedding_document_input_type` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_DOCUMENT_INPUT_TYPE` | Auto for known LiteLLM models | Optional LiteLLM `input_type` for indexed document/passages. |
|
||||
| `semantic_embedding_query_input_type` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_QUERY_INPUT_TYPE` | Auto for known LiteLLM models | Optional LiteLLM `input_type` for search queries. |
|
||||
| `semantic_vector_k` | `BASIC_MEMORY_SEMANTIC_VECTOR_K` | `100` | Candidate count for vector nearest-neighbour retrieval. Higher values improve recall at the cost of latency. |
|
||||
|
||||
## Embedding Providers
|
||||
@@ -135,7 +138,114 @@ export BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER=openai
|
||||
export OPENAI_API_KEY=sk-...
|
||||
```
|
||||
|
||||
When switching from FastEmbed to OpenAI (or vice versa), you must rebuild embeddings since the vector dimensions differ:
|
||||
### LiteLLM
|
||||
|
||||
> **Experimental — advanced users only.** The LiteLLM provider is experimental and aimed at users comfortable operating remote embedding backends: paid API calls, per-model dimension and input-role configuration, and slower reindexing of large corpora. For most users, FastEmbed (local, default) is recommended. See [LiteLLM Provider](litellm-provider.md) for the caveats and tuning.
|
||||
|
||||
Uses the LiteLLM SDK to call embedding models from providers such as OpenAI, Cohere, Azure, Bedrock, NVIDIA NIM, and other LiteLLM-supported backends. Requires the provider's API credentials.
|
||||
For the full option reference, provider setup examples, and live validation harness, see [LiteLLM Provider](litellm-provider.md).
|
||||
|
||||
```bash
|
||||
export BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED=true
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER=litellm
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_MODEL=cohere/embed-english-v3.0
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_DIMENSIONS=1024
|
||||
export COHERE_API_KEY=...
|
||||
```
|
||||
|
||||
Basic Memory creates vector tables before the first embedding call, so non-default LiteLLM models must set `BASIC_MEMORY_SEMANTIC_EMBEDDING_DIMENSIONS`. The LiteLLM OpenAI default (`openai/text-embedding-3-small`) uses 1536 dimensions automatically.
|
||||
|
||||
For fixed-size LiteLLM models, dimensions are used as Basic Memory's local vector schema and
|
||||
validation size. Basic Memory automatically sends dimensions as a provider-side output-size
|
||||
request for `text-embedding-3` model strings, where LiteLLM/OpenAI support reduced output
|
||||
dimensions. If an Azure/OpenAI deployment uses an arbitrary LiteLLM model string such as
|
||||
`azure/<deployment-name>` and the underlying model supports reduced dimensions, set
|
||||
`BASIC_MEMORY_SEMANTIC_EMBEDDING_FORWARD_DIMENSIONS=true`.
|
||||
|
||||
Some retrieval models are asymmetric: indexed passages and search queries must be embedded with different provider parameters. Basic Memory automatically sets LiteLLM `input_type` for known asymmetric model families:
|
||||
|
||||
- Cohere v3: documents use `search_document`, queries use `search_query`
|
||||
- NVIDIA NIM retrieval models: documents use `passage`, queries use `query`
|
||||
|
||||
For other asymmetric LiteLLM models, set the input types explicitly:
|
||||
|
||||
```bash
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_DOCUMENT_INPUT_TYPE=passage
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_QUERY_INPUT_TYPE=query
|
||||
```
|
||||
|
||||
#### Live LiteLLM Validation
|
||||
|
||||
Provider APIs differ in subtle ways: some accept `dimensions`, some require separate
|
||||
document/query roles, and some route through deployment aliases that do not reveal the
|
||||
underlying model name. Before adding or changing LiteLLM model support, run the opt-in live
|
||||
evaluation harness:
|
||||
|
||||
```bash
|
||||
export OPENAI_API_KEY=sk-...
|
||||
export COHERE_API_KEY=...
|
||||
just test-litellm-live
|
||||
```
|
||||
|
||||
The built-in live cases cover:
|
||||
|
||||
| Case | Required key | What it validates |
|
||||
|---|---|---|
|
||||
| `openai/text-embedding-3-small` | `OPENAI_API_KEY` | Standard LiteLLM OpenAI embedding calls and normalized 1536-dimensional output. |
|
||||
| `cohere/embed-english-v3.0` | `COHERE_API_KEY` | Cohere v3 asymmetric `search_document` / `search_query` handling and fixed 1024-dimensional output. |
|
||||
|
||||
The harness embeds two documents and one query, checks vector dimensions and normalization,
|
||||
then verifies the authentication query ranks the authentication document above the distractor.
|
||||
It prints a table with per-model scores, norms, latency, role settings, and dimension-forwarding
|
||||
mode.
|
||||
|
||||
To validate provider aliases or additional LiteLLM backends, save custom JSON cases:
|
||||
|
||||
```bash
|
||||
export AZURE_API_KEY=...
|
||||
export AZURE_API_BASE=https://example.openai.azure.com
|
||||
export AZURE_API_VERSION=2024-02-01
|
||||
|
||||
cat > /tmp/litellm-azure-cases.json <<'JSON'
|
||||
[
|
||||
{
|
||||
"name": "azure-text-embedding-3-small-512",
|
||||
"model": "azure/<deployment-name>",
|
||||
"dimensions": 512,
|
||||
"api_key_env": "AZURE_API_KEY",
|
||||
"forward_dimensions": true
|
||||
}
|
||||
]
|
||||
JSON
|
||||
|
||||
just test-litellm-live --cases-file /tmp/litellm-azure-cases.json
|
||||
```
|
||||
|
||||
NVIDIA NIM retrieval models can be checked the same way:
|
||||
|
||||
```bash
|
||||
export NVIDIA_NIM_API_KEY=...
|
||||
|
||||
cat > /tmp/litellm-nvidia-cases.json <<'JSON'
|
||||
[
|
||||
{
|
||||
"name": "nvidia-embed-qa-4",
|
||||
"model": "nvidia_nim/nvidia/embed-qa-4",
|
||||
"dimensions": 1024,
|
||||
"api_key_env": "NVIDIA_NIM_API_KEY",
|
||||
"document_input_type": "passage",
|
||||
"query_input_type": "query"
|
||||
}
|
||||
]
|
||||
JSON
|
||||
|
||||
just test-litellm-live --cases-file /tmp/litellm-nvidia-cases.json
|
||||
```
|
||||
|
||||
For repeatable local runs, put the same JSON array in a file and pass
|
||||
`just test-litellm-live --cases-file path/to/litellm-cases.json`.
|
||||
|
||||
When switching providers, models, dimensions, or LiteLLM document/query input types, rebuild embeddings:
|
||||
|
||||
```bash
|
||||
bm reindex --embeddings
|
||||
@@ -203,9 +313,10 @@ bm reindex -p my-project
|
||||
|
||||
- **Upgrade note**: Migration now performs a one-time automatic embedding backfill on upgrade.
|
||||
- **Manual enable case**: If you explicitly had `semantic_search_enabled=false` and then turn it on
|
||||
- **Provider change**: After switching between `fastembed` and `openai`
|
||||
- **Provider change**: After switching between `fastembed`, `openai`, and `litellm`
|
||||
- **Model change**: After changing `semantic_embedding_model`
|
||||
- **Dimension change**: After changing `semantic_embedding_dimensions`
|
||||
- **LiteLLM role change**: After changing `semantic_embedding_document_input_type` or `semantic_embedding_query_input_type`
|
||||
|
||||
The reindex command shows progress with embedded/skipped/error counts:
|
||||
|
||||
|
||||
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"$schema": "https://glama.ai/mcp/schemas/server.json",
|
||||
"maintainers": [
|
||||
"phernandez",
|
||||
"groksrc"
|
||||
]
|
||||
}
|
||||
@@ -38,7 +38,7 @@ from typing import Any, Callable
|
||||
from agent.memory_provider import MemoryProvider
|
||||
from tools.registry import tool_error
|
||||
|
||||
__version__ = "0.21.6"
|
||||
__version__ = "0.22.0"
|
||||
|
||||
logger = logging.getLogger("hermes.memory.basic-memory")
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
name: basic-memory
|
||||
version: 0.21.6
|
||||
version: 0.22.0
|
||||
description: "Basic Memory — persistent knowledge graph backed by the basic-memory MCP server"
|
||||
pip_dependencies:
|
||||
- mcp
|
||||
|
||||
@@ -150,7 +150,7 @@ This plugin ships with workflow-oriented skills that are automatically loaded wh
|
||||
No manual installation needed. To update skills or install new ones as they become available:
|
||||
|
||||
```bash
|
||||
npx skills add basicmachines-co/basic-memory --path skills --agent openclaw
|
||||
npx skills add basicmachines-co/basic-memory/skills --agent openclaw
|
||||
```
|
||||
|
||||
See the canonical source at [`basic-memory/skills`](../../skills).
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@basicmemory/openclaw-basic-memory",
|
||||
"version": "0.21.6",
|
||||
"version": "0.22.0",
|
||||
"type": "module",
|
||||
"main": "./dist/index.js",
|
||||
"types": "./dist/index.d.ts",
|
||||
|
||||
@@ -1,5 +1,12 @@
|
||||
# Basic Memory - Modern Command Runner
|
||||
|
||||
TESTMON_FLAGS := env_var_or_default("BASIC_MEMORY_TESTMON_FLAGS", "--testmon-noselect")
|
||||
TESTMON_SELECT_FLAGS := env_var_or_default("BASIC_MEMORY_TESTMON_SELECT_FLAGS", "--testmon --testmon-forceselect")
|
||||
TESTMON_REFRESH_FLAGS := env_var_or_default("BASIC_MEMORY_TESTMON_REFRESH_FLAGS", "--testmon-noselect")
|
||||
# CI shards the Postgres unit suite across parallel jobs via pytest-split
|
||||
# (e.g. "--splits 3 --group 2"). Empty locally.
|
||||
PYTEST_SPLIT_FLAGS := env_var_or_default("BASIC_MEMORY_PYTEST_SPLIT_FLAGS", "")
|
||||
|
||||
# Install dependencies
|
||||
install:
|
||||
uv sync
|
||||
@@ -35,40 +42,60 @@ test-sqlite: test-unit-sqlite test-int-sqlite
|
||||
test-postgres: test-unit-postgres test-int-postgres
|
||||
|
||||
# Run unit tests against SQLite
|
||||
test-unit-sqlite:
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov tests
|
||||
test-unit-sqlite: testmon-seed
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov {{TESTMON_FLAGS}} --testmon-env=unit-sqlite tests
|
||||
|
||||
# Run unit tests against Postgres
|
||||
test-unit-postgres:
|
||||
BASIC_MEMORY_ENV=test BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov tests
|
||||
# Exit code 5 (no tests collected) is success: a testmon-selected PR build can
|
||||
# leave a pytest-split shard empty.
|
||||
test-unit-postgres: testmon-seed
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
BASIC_MEMORY_ENV=test BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov {{TESTMON_FLAGS}} {{PYTEST_SPLIT_FLAGS}} --testmon-env=unit-postgres tests || test $? -eq 5
|
||||
|
||||
# Run integration tests against SQLite (excludes semantic benchmarks — use just test-semantic)
|
||||
test-int-sqlite:
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov -m "not semantic" test-int
|
||||
# Run integration tests against SQLite (excludes semantic tests and on-demand benchmarks —
|
||||
# use just test-semantic / run benchmark files explicitly)
|
||||
test-int-sqlite: testmon-seed
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov {{TESTMON_FLAGS}} --testmon-env=int-sqlite -m "not semantic and not benchmark" test-int
|
||||
|
||||
# Run integration tests against Postgres
|
||||
# Note: Uses timeout due to FastMCP Client + asyncpg cleanup hang (tests pass, process hangs on exit)
|
||||
# See: https://github.com/jlowin/fastmcp/issues/1311
|
||||
test-int-postgres:
|
||||
test-int-postgres: testmon-seed
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
# Use gtimeout (macOS/Homebrew) or timeout (Linux)
|
||||
TIMEOUT_CMD=$(command -v gtimeout || command -v timeout || echo "")
|
||||
if [[ -n "$TIMEOUT_CMD" ]]; then
|
||||
$TIMEOUT_CMD --signal=KILL 600 bash -c 'BASIC_MEMORY_ENV=test BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov -m "not semantic" test-int' || test $? -eq 137
|
||||
$TIMEOUT_CMD --signal=KILL 600 bash -c 'BASIC_MEMORY_ENV=test BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov {{TESTMON_FLAGS}} --testmon-env=int-postgres -m "not semantic and not benchmark" test-int' || test $? -eq 137
|
||||
else
|
||||
echo "⚠️ No timeout command found, running without timeout..."
|
||||
BASIC_MEMORY_ENV=test BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov -m "not semantic" test-int
|
||||
BASIC_MEMORY_ENV=test BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov {{TESTMON_FLAGS}} --testmon-env=int-postgres -m "not semantic and not benchmark" test-int
|
||||
fi
|
||||
|
||||
# Run tests impacted by recent changes (requires pytest-testmon)
|
||||
# Pass paths or node ids after `just testmon` to limit the candidate set further.
|
||||
testmon *args:
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov --testmon {{args}}
|
||||
testmon *args: testmon-seed
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov {{TESTMON_SELECT_FLAGS}} --testmon-env=local {{args}}
|
||||
|
||||
# Seed pytest-testmon data into this worktree from the shared Git cache.
|
||||
testmon-seed:
|
||||
uv run python scripts/testmon_cache.py seed
|
||||
|
||||
# Refresh the shared pytest-testmon cache from a full backend test run.
|
||||
testmon-refresh:
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
BASIC_MEMORY_TESTMON_FLAGS="{{TESTMON_REFRESH_FLAGS}}" just test
|
||||
uv run python scripts/testmon_cache.py refresh
|
||||
|
||||
# Show local and shared pytest-testmon cache locations.
|
||||
testmon-status:
|
||||
uv run python scripts/testmon_cache.py status
|
||||
|
||||
# Run MCP smoke test (fast end-to-end loop)
|
||||
test-smoke:
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov -m smoke test-int/mcp/test_smoke_integration.py
|
||||
test-smoke: testmon-seed
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov {{TESTMON_FLAGS}} --testmon-env=smoke -m smoke test-int/mcp/test_smoke_integration.py
|
||||
|
||||
# Fast local loop: lint, format, typecheck, impacted tests via pytest-testmon
|
||||
fast-check:
|
||||
@@ -97,27 +124,31 @@ postgres-migrate:
|
||||
# Run Windows-specific tests only (only works on Windows platform)
|
||||
# These tests verify Windows-specific database optimizations (locking mode, NullPool)
|
||||
# Will be skipped automatically on non-Windows platforms
|
||||
test-windows:
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov -m windows tests test-int
|
||||
test-windows: testmon-seed
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov {{TESTMON_FLAGS}} --testmon-env=windows -m windows tests test-int
|
||||
|
||||
# Run benchmark tests only (performance testing)
|
||||
# These are slow tests that measure sync performance with various file counts
|
||||
# Excluded from default test runs to keep CI fast
|
||||
test-benchmark:
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov -m benchmark tests test-int
|
||||
test-benchmark: testmon-seed
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov {{TESTMON_FLAGS}} --testmon-env=benchmark -m benchmark tests test-int
|
||||
|
||||
# Run semantic search quality benchmarks (all combos)
|
||||
test-semantic:
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov -m semantic test-int/semantic/
|
||||
test-semantic: testmon-seed
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov {{TESTMON_FLAGS}} --testmon-env=semantic -m semantic test-int/semantic/
|
||||
|
||||
# Run semantic benchmarks with JSON artifact output, then show report
|
||||
test-semantic-report:
|
||||
BASIC_MEMORY_ENV=test BASIC_MEMORY_BENCHMARK_OUTPUT=.benchmarks/semantic-quality.jsonl uv run pytest -p pytest_mock -v -s --no-cov -m semantic test-int/semantic/
|
||||
uv run python test-int/semantic/report.py .benchmarks/semantic-quality.jsonl
|
||||
|
||||
# Run opt-in live LiteLLM provider checks against configured external APIs
|
||||
test-litellm-live *args:
|
||||
BASIC_MEMORY_ENV=test BASIC_MEMORY_RUN_LITELLM_INTEGRATION=1 PYTHONPATH=test-int:src uv run python -m semantic.litellm_live_harness {{args}}
|
||||
|
||||
# Run semantic benchmarks (Postgres combos only)
|
||||
test-semantic-postgres:
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov -m semantic -k postgres test-int/semantic/
|
||||
test-semantic-postgres: testmon-seed
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov {{TESTMON_FLAGS}} --testmon-env=semantic-postgres -m semantic -k postgres test-int/semantic/
|
||||
|
||||
# View semantic benchmark results (rich formatted table)
|
||||
# Usage: just semantic-report [--filter-combo sqlite] [--filter-suite paraphrase] [--sort-by avg_latency_ms]
|
||||
@@ -133,8 +164,8 @@ benchmark-compare baseline candidate *args:
|
||||
|
||||
# Run all tests including Windows, Postgres, and Benchmarks (for CI/comprehensive testing)
|
||||
# Use this before releasing to ensure everything works across all backends and platforms
|
||||
test-all:
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov tests test-int
|
||||
test-all: testmon-seed
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov {{TESTMON_FLAGS}} --testmon-env=all tests test-int
|
||||
|
||||
# Generate HTML coverage report
|
||||
coverage:
|
||||
@@ -307,7 +338,7 @@ set-version version scope="all":
|
||||
set-version-dry-run version scope="all":
|
||||
python3 scripts/update_versions.py "{{version}}" --scope "{{scope}}" --dry-run
|
||||
|
||||
# Set the version for just the plugin/agent artifacts (plugin, marketplaces, Hermes, OpenClaw)
|
||||
# Set the version for just the plugin/agent artifacts (plugins, marketplaces, Hermes, OpenClaw)
|
||||
set-packages-version version:
|
||||
just set-version "{{version}}" packages
|
||||
|
||||
@@ -369,6 +400,7 @@ release version:
|
||||
.claude-plugin/marketplace.json \
|
||||
plugins/claude-code/.claude-plugin/plugin.json \
|
||||
plugins/claude-code/.claude-plugin/marketplace.json \
|
||||
plugins/codex/.codex-plugin/plugin.json \
|
||||
integrations/hermes/plugin.yaml \
|
||||
integrations/hermes/__init__.py \
|
||||
integrations/openclaw/package.json
|
||||
@@ -442,6 +474,7 @@ beta version:
|
||||
.claude-plugin/marketplace.json \
|
||||
plugins/claude-code/.claude-plugin/plugin.json \
|
||||
plugins/claude-code/.claude-plugin/marketplace.json \
|
||||
plugins/codex/.codex-plugin/plugin.json \
|
||||
integrations/hermes/plugin.yaml \
|
||||
integrations/hermes/__init__.py \
|
||||
integrations/openclaw/package.json
|
||||
|
||||
@@ -6,14 +6,14 @@
|
||||
},
|
||||
"metadata": {
|
||||
"description": "Official plugins from Basic Machines for knowledge management and AI-assisted development",
|
||||
"version": "0.21.6"
|
||||
"version": "0.22.0"
|
||||
},
|
||||
"plugins": [
|
||||
{
|
||||
"name": "basic-memory",
|
||||
"source": "./",
|
||||
"description": "The bridge between Claude's working memory and Basic Memory's durable knowledge graph \u2014 session briefings, pre-compaction checkpoints, and capture reflexes",
|
||||
"version": "0.21.6",
|
||||
"version": "0.22.0",
|
||||
"author": {
|
||||
"name": "Basic Machines"
|
||||
},
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "basic-memory",
|
||||
"description": "The bridge between Claude's working memory and Basic Memory's durable knowledge graph \u2014 session briefings, pre-compaction checkpoints, and capture reflexes",
|
||||
"version": "0.21.6",
|
||||
"version": "0.22.0",
|
||||
"author": {
|
||||
"name": "Basic Machines"
|
||||
},
|
||||
|
||||
@@ -24,7 +24,7 @@ Memory's durable graph**, rather than a memory layer of its own. See
|
||||
configures the project for the plugin: maps it to a Basic Memory project (picking
|
||||
an existing one or creating a new one), seeds the `session`/`decision`/`task`
|
||||
schemas into the project, installs the shared `memory-*` skills via
|
||||
`npx skills add basicmachines-co/basic-memory --path skills` (the plugin doesn't
|
||||
`npx skills add basicmachines-co/basic-memory/skills` (the plugin doesn't
|
||||
vendor its own copies — `skills/` is the single source of truth, shared with
|
||||
OpenClaw), optionally learns the project's placement conventions, and enables the
|
||||
capture reflexes. Writes the `basicMemory` block to
|
||||
|
||||
@@ -156,7 +156,7 @@ git ls-files skills/ | grep -q memory- && echo "source repo - skip install"
|
||||
Otherwise, run from the project root:
|
||||
|
||||
```
|
||||
npx skills add basicmachines-co/basic-memory --path skills
|
||||
npx skills add basicmachines-co/basic-memory/skills
|
||||
```
|
||||
|
||||
This installs the canonical `memory-*` skills into the user's skills directory — the
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "codex",
|
||||
"version": "0.1.0+codex.20260604201213",
|
||||
"version": "0.22.0",
|
||||
"description": "A Codex-native bridge to Basic Memory for durable engineering context, decisions, and resumable checkpoints.",
|
||||
"author": {
|
||||
"name": "Basic Machines",
|
||||
|
||||
@@ -48,8 +48,13 @@ dependencies = [
|
||||
"fastembed>=0.7.4",
|
||||
"sqlite-vec>=0.1.6",
|
||||
"openai>=1.100.2",
|
||||
"litellm>=1.60.0,<2.0.0",
|
||||
"logfire>=4.19.0",
|
||||
"psutil>=5.9.0",
|
||||
# uvloop's C event loop has no self._ready.popleft() codepath, so the
|
||||
# asyncpg engine-dispose race ("IndexError: pop from an empty deque") that
|
||||
# crashes the Postgres backend cannot fire under it. Not available on Windows.
|
||||
"uvloop>=0.21.0; sys_platform != 'win32'",
|
||||
]
|
||||
|
||||
[project.urls]
|
||||
@@ -71,6 +76,10 @@ addopts = "--cov=basic_memory --cov-report term-missing"
|
||||
testpaths = ["tests", "test-int"]
|
||||
asyncio_mode = "strict"
|
||||
asyncio_default_fixture_loop_scope = "function"
|
||||
# Any test hanging >120s fails with a stack dump instead of stalling the CI job
|
||||
# until the runner times out (the FastMCP/asyncpg cleanup-hang family).
|
||||
timeout = 120
|
||||
timeout_method = "thread"
|
||||
filterwarnings = [
|
||||
"ignore:The @wait_container_is_ready decorator is deprecated.*:DeprecationWarning:testcontainers\\.core\\.waiting_utils",
|
||||
"ignore:The default datetime adapter is deprecated as of Python 3\\.12.*:DeprecationWarning:aiosqlite\\.core",
|
||||
@@ -84,6 +93,7 @@ markers = [
|
||||
"windows: Windows-specific tests (deselect with '-m \"not windows\"')",
|
||||
"smoke: Fast end-to-end smoke tests for MCP flows",
|
||||
"semantic: Tests requiring semantic dependencies (fastembed, sqlite-vec, openai)",
|
||||
"live: Tests that call external provider APIs and require explicit opt-in",
|
||||
]
|
||||
|
||||
[tool.ruff]
|
||||
@@ -109,6 +119,8 @@ dev = [
|
||||
"ty>=0.0.18",
|
||||
"cst-lsp>=0.1.3",
|
||||
"libcst>=1.8.6",
|
||||
"pytest-timeout>=2.4.0",
|
||||
"pytest-split>=0.11.0",
|
||||
]
|
||||
|
||||
[tool.hatch.version]
|
||||
|
||||
@@ -0,0 +1,225 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Seed and refresh shared pytest-testmon data for Git worktrees."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import NamedTuple
|
||||
|
||||
TESTMON_FILENAMES = (".testmondata", ".testmondata-shm", ".testmondata-wal")
|
||||
TESTMON_CACHE_ENV = "BM_TESTMON_CACHE_DIR"
|
||||
|
||||
|
||||
class TestmonCacheResult(NamedTuple):
|
||||
status: str
|
||||
source_dir: Path
|
||||
destination_dir: Path
|
||||
copied: tuple[Path, ...]
|
||||
|
||||
|
||||
def _run_git(args: list[str], cwd: Path) -> str:
|
||||
return subprocess.check_output(["git", *args], cwd=cwd, text=True).strip()
|
||||
|
||||
|
||||
def resolve_repo_root(repo_root: Path | None = None) -> Path:
|
||||
if repo_root is not None:
|
||||
return repo_root.expanduser().resolve()
|
||||
|
||||
return Path(_run_git(["rev-parse", "--show-toplevel"], Path.cwd())).resolve()
|
||||
|
||||
|
||||
def resolve_cache_dir(repo_root: Path, cache_dir: Path | None = None) -> Path:
|
||||
if cache_dir is not None:
|
||||
return cache_dir.expanduser().resolve()
|
||||
|
||||
if env_cache_dir := os.environ.get(TESTMON_CACHE_ENV):
|
||||
return Path(env_cache_dir).expanduser().resolve()
|
||||
|
||||
git_common_dir = Path(_run_git(["rev-parse", "--git-common-dir"], repo_root))
|
||||
if not git_common_dir.is_absolute():
|
||||
git_common_dir = repo_root / git_common_dir
|
||||
|
||||
return git_common_dir.resolve() / "testmon-cache" / "main"
|
||||
|
||||
|
||||
def _testmon_datafile(directory: Path) -> Path:
|
||||
return directory / ".testmondata"
|
||||
|
||||
|
||||
def _testmon_files(directory: Path) -> list[Path]:
|
||||
return [
|
||||
directory / filename for filename in TESTMON_FILENAMES if (directory / filename).is_file()
|
||||
]
|
||||
|
||||
|
||||
def _remove_path(path: Path) -> None:
|
||||
if path.is_dir():
|
||||
shutil.rmtree(path)
|
||||
elif path.exists():
|
||||
path.unlink()
|
||||
|
||||
|
||||
def _copy_testmon_files(source_dir: Path, destination_dir: Path) -> tuple[Path, ...]:
|
||||
destination_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
copied: list[Path] = []
|
||||
for source in _testmon_files(source_dir):
|
||||
destination = destination_dir / source.name
|
||||
shutil.copy2(source, destination)
|
||||
copied.append(destination)
|
||||
|
||||
return tuple(copied)
|
||||
|
||||
|
||||
def seed_testmon_data(repo_root: Path, cache_dir: Path) -> TestmonCacheResult:
|
||||
local_datafile = _testmon_datafile(repo_root)
|
||||
shared_datafile = _testmon_datafile(cache_dir)
|
||||
|
||||
if local_datafile.exists():
|
||||
return TestmonCacheResult(
|
||||
status="exists",
|
||||
source_dir=cache_dir,
|
||||
destination_dir=repo_root,
|
||||
copied=(),
|
||||
)
|
||||
|
||||
if not shared_datafile.exists():
|
||||
return TestmonCacheResult(
|
||||
status="missing",
|
||||
source_dir=cache_dir,
|
||||
destination_dir=repo_root,
|
||||
copied=(),
|
||||
)
|
||||
|
||||
# A worktree with sidecars but no main database is stale; replace the set
|
||||
# together so SQLite never sees a mixed local/cache snapshot.
|
||||
for filename in TESTMON_FILENAMES:
|
||||
_remove_path(repo_root / filename)
|
||||
|
||||
copied = _copy_testmon_files(cache_dir, repo_root)
|
||||
return TestmonCacheResult(
|
||||
status="seeded",
|
||||
source_dir=cache_dir,
|
||||
destination_dir=repo_root,
|
||||
copied=copied,
|
||||
)
|
||||
|
||||
|
||||
def refresh_testmon_data(repo_root: Path, cache_dir: Path) -> TestmonCacheResult:
|
||||
local_datafile = _testmon_datafile(repo_root)
|
||||
|
||||
if not local_datafile.exists():
|
||||
raise FileNotFoundError(
|
||||
f"No local pytest-testmon data at {local_datafile}; run tests first."
|
||||
)
|
||||
|
||||
cache_parent = cache_dir.parent
|
||||
cache_parent.mkdir(parents=True, exist_ok=True)
|
||||
temp_dir = Path(tempfile.mkdtemp(prefix=f".{cache_dir.name}.", dir=cache_parent))
|
||||
backup_dir = cache_parent / f".{cache_dir.name}.previous-{os.getpid()}"
|
||||
copied: tuple[Path, ...] = ()
|
||||
|
||||
try:
|
||||
copied = _copy_testmon_files(repo_root, temp_dir)
|
||||
|
||||
_remove_path(backup_dir)
|
||||
if cache_dir.exists():
|
||||
cache_dir.rename(backup_dir)
|
||||
|
||||
try:
|
||||
temp_dir.rename(cache_dir)
|
||||
except Exception:
|
||||
if backup_dir.exists() and not cache_dir.exists():
|
||||
backup_dir.rename(cache_dir)
|
||||
raise
|
||||
finally:
|
||||
_remove_path(temp_dir)
|
||||
_remove_path(backup_dir)
|
||||
|
||||
return TestmonCacheResult(
|
||||
status="refreshed",
|
||||
source_dir=repo_root,
|
||||
destination_dir=cache_dir,
|
||||
copied=tuple(cache_dir / path.name for path in copied),
|
||||
)
|
||||
|
||||
|
||||
def _print_seed_result(result: TestmonCacheResult) -> None:
|
||||
if result.status == "seeded":
|
||||
print(f"Seeded pytest-testmon data from {result.source_dir} into {result.destination_dir}")
|
||||
elif result.status == "exists":
|
||||
print(f"Local pytest-testmon data already exists at {result.destination_dir}")
|
||||
elif result.status == "missing":
|
||||
print(
|
||||
f"No shared pytest-testmon baseline at {result.source_dir}; "
|
||||
"run `just testmon-refresh` after a full backend test run to create one."
|
||||
)
|
||||
else:
|
||||
raise ValueError(f"Unexpected seed result: {result.status}")
|
||||
|
||||
|
||||
def _print_refresh_result(result: TestmonCacheResult) -> None:
|
||||
print(f"Published pytest-testmon data from {result.source_dir} to {result.destination_dir}")
|
||||
|
||||
|
||||
def _print_status(repo_root: Path, cache_dir: Path) -> None:
|
||||
print(f"Repo root: {repo_root}")
|
||||
print(f"Worktree data: {_testmon_datafile(repo_root)}")
|
||||
print(f"Shared cache: {_testmon_datafile(cache_dir)}")
|
||||
print(f"Worktree ready: {_testmon_datafile(repo_root).exists()}")
|
||||
print(f"Cache ready: {_testmon_datafile(cache_dir).exists()}")
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument(
|
||||
"--repo-root",
|
||||
type=Path,
|
||||
help="Repository root to operate on (default: git rev-parse --show-toplevel)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--cache-dir",
|
||||
type=Path,
|
||||
help=(
|
||||
"Shared testmon cache directory "
|
||||
f"(default: ${TESTMON_CACHE_ENV} or <git-common-dir>/testmon-cache/main)"
|
||||
),
|
||||
)
|
||||
|
||||
subparsers = parser.add_subparsers(dest="command", required=True)
|
||||
subparsers.add_parser("seed", help="Copy shared testmon data into this worktree if missing")
|
||||
subparsers.add_parser("refresh", help="Publish this worktree's testmon data to the cache")
|
||||
subparsers.add_parser("status", help="Show local and shared testmon data paths")
|
||||
return parser
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = build_parser()
|
||||
args = parser.parse_args(argv)
|
||||
repo_root = resolve_repo_root(args.repo_root)
|
||||
cache_dir = resolve_cache_dir(repo_root, args.cache_dir)
|
||||
|
||||
if args.command == "seed":
|
||||
_print_seed_result(seed_testmon_data(repo_root=repo_root, cache_dir=cache_dir))
|
||||
return 0
|
||||
|
||||
if args.command == "refresh":
|
||||
_print_refresh_result(refresh_testmon_data(repo_root=repo_root, cache_dir=cache_dir))
|
||||
return 0
|
||||
|
||||
if args.command == "status":
|
||||
_print_status(repo_root=repo_root, cache_dir=cache_dir)
|
||||
return 0
|
||||
|
||||
parser.error(f"Unknown command: {args.command}")
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -99,7 +99,7 @@ def set_package_version(data: dict[str, Any], version: str) -> None:
|
||||
# Version scopes. The two groups map to the two distribution tracks:
|
||||
# core — the Python package and its MCP registry manifest
|
||||
# packages — the host-native agent artifacts (Claude Code plugin + marketplaces,
|
||||
# Hermes, OpenClaw). These are the "plugin/agent artifacts."
|
||||
# Codex plugin, Hermes, OpenClaw). These are the "plugin/agent artifacts."
|
||||
# `all` writes both. Lockstep releases use `all`; targeted fixes can use one group.
|
||||
SCOPES = ("all", "core", "packages")
|
||||
|
||||
@@ -134,6 +134,11 @@ def _update_packages(version: str, *, dry_run: bool) -> None:
|
||||
lambda data: set_claude_marketplace_version(data, version),
|
||||
dry_run=dry_run,
|
||||
)
|
||||
update_json(
|
||||
"plugins/codex/.codex-plugin/plugin.json",
|
||||
lambda data: set_package_version(data, npm_package_version(version)),
|
||||
dry_run=dry_run,
|
||||
)
|
||||
update_text(
|
||||
"integrations/hermes/plugin.yaml",
|
||||
r"^version:\s*.*$",
|
||||
@@ -174,7 +179,7 @@ def main() -> None:
|
||||
choices=SCOPES,
|
||||
default="all",
|
||||
help="Which artifacts to update: all (default), core (Python + server.json), "
|
||||
"or packages (Claude Code plugin, marketplaces, Hermes, OpenClaw)",
|
||||
"or packages (Claude Code plugin, Codex plugin, marketplaces, Hermes, OpenClaw)",
|
||||
)
|
||||
parser.add_argument("--dry-run", action="store_true", help="Preview changes without writing")
|
||||
args = parser.parse_args()
|
||||
|
||||
@@ -4,9 +4,31 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
PLAIN_SCALAR_MAPPING_VALUE = re.compile(r":(?:\s|$)")
|
||||
|
||||
|
||||
def strip_matching_quotes(value: str) -> str:
|
||||
if len(value) >= 2 and value[0] == value[-1] and value[0] in {'"', "'"}:
|
||||
return value[1:-1]
|
||||
return value
|
||||
|
||||
|
||||
def validate_plain_scalar(path: Path, line_number: int, key: str, value: str) -> None:
|
||||
"""Catch invalid plain-scalar YAML that Codex rejects while loading skills."""
|
||||
stripped = value.strip()
|
||||
if not stripped or stripped[0] in {'"', "'"} or stripped in {"|", ">", "|-", ">-", "|+", ">+"}:
|
||||
return
|
||||
if PLAIN_SCALAR_MAPPING_VALUE.search(stripped):
|
||||
raise SystemExit(
|
||||
f"{path}:{line_number}: invalid YAML frontmatter for {key!r}: "
|
||||
"unquoted ':' followed by whitespace; quote the value"
|
||||
)
|
||||
|
||||
|
||||
def parse_frontmatter(path: Path) -> dict[str, str]:
|
||||
"""Extract top-level frontmatter keys from a Markdown file.
|
||||
|
||||
@@ -15,14 +37,15 @@ def parse_frontmatter(path: Path) -> dict[str, str]:
|
||||
skipped, so nested blocks (a schema note's `schema:`/`settings:` children) can't
|
||||
overwrite a top-level key like `type` or `entity` via last-write-wins. It does
|
||||
not interpret block scalars or multi-line values; callers rely on single-line
|
||||
top-level fields (name, description, type, entity).
|
||||
top-level fields (name, description, type, entity). Keep the Codex-facing YAML
|
||||
guard here dependency-free so package checks work under bare `python3`.
|
||||
"""
|
||||
lines = path.read_text().splitlines()
|
||||
if not lines or lines[0] != "---":
|
||||
raise SystemExit(f"{path}: missing YAML frontmatter")
|
||||
|
||||
frontmatter: dict[str, str] = {}
|
||||
for line in lines[1:]:
|
||||
for line_number, line in enumerate(lines[1:], start=2):
|
||||
if line == "---":
|
||||
break
|
||||
if line[:1] in (" ", "\t"): # nested key — not a top-level field
|
||||
@@ -30,7 +53,10 @@ def parse_frontmatter(path: Path) -> dict[str, str]:
|
||||
if ":" not in line:
|
||||
continue
|
||||
key, value = line.split(":", 1)
|
||||
frontmatter[key.strip()] = value.strip().strip('"')
|
||||
key = key.strip()
|
||||
value = value.strip()
|
||||
validate_plain_scalar(path, line_number, key, value)
|
||||
frontmatter[key] = strip_matching_quotes(value)
|
||||
else:
|
||||
raise SystemExit(f"{path}: unclosed YAML frontmatter")
|
||||
|
||||
|
||||
+2
-2
@@ -6,12 +6,12 @@
|
||||
"url": "https://github.com/basicmachines-co/basic-memory.git",
|
||||
"source": "github"
|
||||
},
|
||||
"version": "0.21.6",
|
||||
"version": "0.22.0",
|
||||
"packages": [
|
||||
{
|
||||
"registryType": "pypi",
|
||||
"identifier": "basic-memory",
|
||||
"version": "0.21.6",
|
||||
"version": "0.22.0",
|
||||
"runtimeHint": "uvx",
|
||||
"runtimeArguments": [
|
||||
{
|
||||
|
||||
@@ -54,13 +54,13 @@ Users can install or update skills with the [Skills CLI](https://github.com/verc
|
||||
|
||||
```bash
|
||||
# Install all skills
|
||||
npx skills add basicmachines-co/basic-memory --path skills
|
||||
npx skills add basicmachines-co/basic-memory/skills
|
||||
|
||||
# Install a specific skill
|
||||
npx skills add basicmachines-co/basic-memory --path skills --skill memory-tasks
|
||||
npx skills add basicmachines-co/basic-memory/skills --skill memory-tasks
|
||||
|
||||
# Install for a specific agent
|
||||
npx skills add basicmachines-co/basic-memory --path skills --agent claude
|
||||
npx skills add basicmachines-co/basic-memory/skills --agent claude
|
||||
```
|
||||
|
||||
## Adding a New Skill
|
||||
|
||||
+5
-5
@@ -47,16 +47,16 @@ Install or update skills using the [Skills CLI](https://github.com/vercel-labs/s
|
||||
|
||||
```bash
|
||||
# Install all skills
|
||||
npx skills add basicmachines-co/basic-memory --path skills
|
||||
npx skills add basicmachines-co/basic-memory/skills
|
||||
|
||||
# Install a specific skill
|
||||
npx skills add basicmachines-co/basic-memory --path skills --skill memory-tasks
|
||||
npx skills add basicmachines-co/basic-memory/skills --skill memory-tasks
|
||||
|
||||
# Install all skills for a specific agent
|
||||
npx skills add basicmachines-co/basic-memory --path skills --agent claude
|
||||
npx skills add basicmachines-co/basic-memory/skills --agent claude
|
||||
|
||||
# List available skills without installing
|
||||
npx skills add basicmachines-co/basic-memory --path skills --list
|
||||
npx skills add basicmachines-co/basic-memory/skills --list
|
||||
|
||||
# Check for updates
|
||||
npx skills check
|
||||
@@ -67,7 +67,7 @@ npx skills update
|
||||
|
||||
Skills are installed to your agent's skills directory (e.g., `~/.claude/skills/` for Claude Code global, or `.claude/skills/` for project-scoped).
|
||||
|
||||
If your installed Skills CLI does not support `--path`, copy the `memory-*` directories manually for now. Phase 2 will add a first-class Codex/package install path.
|
||||
If your installed Skills CLI cannot load `basicmachines-co/basic-memory/skills`, update the CLI or copy the `memory-*` directories manually.
|
||||
|
||||
### Claude Desktop (claude.ai)
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
"""basic-memory - Local-first knowledge management combining Zettelkasten with knowledge graphs"""
|
||||
|
||||
# Package version - updated by release automation
|
||||
__version__ = "0.21.6"
|
||||
__version__ = "0.22.0"
|
||||
|
||||
# API version for FastAPI - independent of package version
|
||||
__api_version__ = "v0"
|
||||
|
||||
@@ -2,8 +2,11 @@
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
from contextlib import suppress
|
||||
from logging.config import fileConfig
|
||||
|
||||
from loguru import logger
|
||||
|
||||
# Allow nested event loops (needed for pytest-asyncio and other async contexts)
|
||||
# Note: nest_asyncio doesn't work with uvloop or Python 3.14+, so we handle those cases separately
|
||||
import sys
|
||||
@@ -13,9 +16,15 @@ if sys.version_info < (3, 14):
|
||||
import nest_asyncio
|
||||
|
||||
nest_asyncio.apply()
|
||||
except (ImportError, ValueError):
|
||||
# nest_asyncio not available or can't patch this loop type (e.g., uvloop)
|
||||
pass
|
||||
except (ImportError, ValueError) as exc:
|
||||
# Trigger: nest_asyncio is absent (ImportError) or refuses to patch the
|
||||
# running loop (ValueError, e.g. the uvloop policy installed for the
|
||||
# Postgres backend - #831/#877).
|
||||
# Why: the uvloop ValueError is now an *expected* path on every Postgres
|
||||
# startup, so swallowing it silently hides a routine branch.
|
||||
# Outcome: log at DEBUG (observable, not noisy) and fall through to the
|
||||
# thread-based migration fallback.
|
||||
logger.debug(f"nest_asyncio not applied ({exc!r}); using thread-based migration fallback")
|
||||
# For Python 3.14+, we rely on the thread-based fallback in run_migrations_online()
|
||||
|
||||
from sqlalchemy import engine_from_config, pool
|
||||
@@ -115,7 +124,15 @@ async def run_async_migrations(connectable):
|
||||
"""Run migrations asynchronously with AsyncEngine."""
|
||||
async with connectable.connect() as connection:
|
||||
await connection.run_sync(do_run_migrations)
|
||||
await connectable.dispose()
|
||||
# Trigger: startup migrations on the asyncpg backend dispose the engine
|
||||
# while the event loop may be tearing down around them.
|
||||
# Why: that race surfaces "IndexError: pop from an empty deque" from
|
||||
# base_events._run_once (#831/#877); shielding lets dispose finish atomically
|
||||
# and suppressing CancelledError keeps a cancelled teardown from re-raising it.
|
||||
# Outcome: the migration engine always disposes cleanly. (uvloop is the
|
||||
# structural fix for the race; this hardens the teardown path.)
|
||||
with suppress(asyncio.CancelledError):
|
||||
await asyncio.shield(connectable.dispose())
|
||||
|
||||
|
||||
def _run_async_migrations_with_asyncio_run(connectable) -> None:
|
||||
|
||||
@@ -10,10 +10,18 @@ Key improvements:
|
||||
- Simplified caching strategies
|
||||
"""
|
||||
|
||||
import os
|
||||
import pathlib
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Response, Path
|
||||
from loguru import logger
|
||||
|
||||
import logfire
|
||||
from basic_memory.ignore_utils import (
|
||||
IGNORED_PATH_REJECTION_DETAIL,
|
||||
load_gitignore_patterns,
|
||||
should_ignore_path,
|
||||
)
|
||||
from basic_memory.deps import (
|
||||
EntityServiceV2ExternalDep,
|
||||
SearchServiceV2ExternalDep,
|
||||
@@ -24,6 +32,7 @@ from basic_memory.deps import (
|
||||
EntityRepositoryV2ExternalDep,
|
||||
RelationRepositoryV2ExternalDep,
|
||||
ProjectExternalIdPathDep,
|
||||
SyncServiceV2ExternalDep,
|
||||
TaskSchedulerDep,
|
||||
)
|
||||
from basic_memory.schemas import DeleteEntitiesResponse
|
||||
@@ -40,8 +49,10 @@ from basic_memory.schemas.v2 import (
|
||||
MoveDirectoryRequestV2,
|
||||
DeleteDirectoryRequestV2,
|
||||
OrphanEntitiesResponse,
|
||||
SyncFileRequest,
|
||||
)
|
||||
from basic_memory.schemas.response import DirectoryMoveResult, DirectoryDeleteResult
|
||||
from basic_memory.utils import validate_project_path
|
||||
|
||||
router = APIRouter(prefix="/knowledge", tags=["knowledge-v2"])
|
||||
|
||||
@@ -236,6 +247,187 @@ async def resolve_identifier(
|
||||
return result
|
||||
|
||||
|
||||
## Single-file sync endpoint
|
||||
|
||||
|
||||
def _canonical_file_path(home: pathlib.Path, segments: list[str]) -> str | None:
|
||||
"""Resolve the actual on-disk casing of a file path under the project home.
|
||||
|
||||
Trigger: case-insensitive filesystems (macOS/Windows) pass existence checks for
|
||||
wrong-cased paths like 'notes/Disk-Note.md' when the file is 'notes/disk-note.md'.
|
||||
Why: indexing the caller-supplied casing misses the existing DB row keyed by the
|
||||
on-disk path and inserts a duplicate entity under the wrong-cased path.
|
||||
Outcome: each segment is matched against real directory entries — exact name first
|
||||
(so distinct case-variant files on case-sensitive filesystems stay distinct),
|
||||
then a unique case-insensitive match. Returns None when any segment cannot be
|
||||
matched to exactly one entry, including missing files. Traversal stops at the
|
||||
project boundary: a directory whose resolved path escapes the project home is
|
||||
never scanned.
|
||||
"""
|
||||
resolved_home = home.resolve()
|
||||
current = home
|
||||
canonical_segments: list[str] = []
|
||||
for segment in segments:
|
||||
# Trigger: a previously matched segment may be a symlink whose target lies
|
||||
# outside the project root (e.g. wrong-cased 'LINK' matched the on-disk
|
||||
# 'link' -> /tmp/outside on a case-sensitive filesystem).
|
||||
# Why: os.scandir follows symlinked directories, so continuing would read
|
||||
# directory contents outside the project boundary even though the
|
||||
# post-canonicalization containment check rejects the request later.
|
||||
# Outcome: bail before scanning the moment resolution escapes the home.
|
||||
if not current.resolve().is_relative_to(resolved_home):
|
||||
return None
|
||||
try:
|
||||
with os.scandir(current) as entries_iter:
|
||||
entries = [entry.name for entry in entries_iter]
|
||||
except OSError:
|
||||
# A parent segment resolved to a non-directory (or vanished): no canonical
|
||||
# path exists for the remaining segments.
|
||||
return None
|
||||
if segment in entries:
|
||||
matched = segment
|
||||
else:
|
||||
matches = [entry for entry in entries if entry.lower() == segment.lower()]
|
||||
if len(matches) != 1:
|
||||
return None
|
||||
matched = matches[0]
|
||||
canonical_segments.append(matched)
|
||||
current = current / matched
|
||||
return "/".join(canonical_segments)
|
||||
|
||||
|
||||
@router.post("/sync-file", response_model=EntityResponseV2)
|
||||
async def sync_file(
|
||||
data: SyncFileRequest,
|
||||
project_id: ProjectExternalIdPathDep,
|
||||
sync_service: SyncServiceV2ExternalDep,
|
||||
project_config: ProjectConfigV2ExternalDep,
|
||||
search_service: SearchServiceV2ExternalDep,
|
||||
app_config: AppConfigDep,
|
||||
) -> EntityResponseV2:
|
||||
"""Index a single markdown file that exists on disk but is not indexed yet.
|
||||
|
||||
Recovery path for files written directly to disk before the watcher indexed
|
||||
them (#581): callers such as edit_note can index the exact file and retry
|
||||
identifier resolution without running a full project sync.
|
||||
|
||||
Args:
|
||||
data: Request containing the markdown file path relative to project root
|
||||
|
||||
Returns:
|
||||
The indexed entity
|
||||
|
||||
Raises:
|
||||
HTTPException: 400 if the path escapes the project root, contains
|
||||
non-normalized segments, matches the project ignore rules, or is
|
||||
not markdown, 404 if the file does not exist on disk
|
||||
"""
|
||||
with logfire.span(
|
||||
"api.request.knowledge.sync_file",
|
||||
entrypoint="api",
|
||||
domain="knowledge",
|
||||
action="sync_file",
|
||||
):
|
||||
logger.info(f"API v2 request: sync_file file_path='{data.file_path}'")
|
||||
|
||||
if not validate_project_path(data.file_path, project_config.home):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"File path '{data.file_path}' is not allowed - "
|
||||
"paths must stay within project boundaries",
|
||||
)
|
||||
|
||||
# Trigger: segments like './' or '//' survive the traversal check above
|
||||
# Why: a non-normalized path would index under a non-canonical DB key
|
||||
# Outcome: reject fail-fast instead of guessing the canonical form
|
||||
segments = data.file_path.replace("\\", "/").split("/")
|
||||
if any(segment in ("", ".") for segment in segments):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"File path '{data.file_path}' is not normalized - "
|
||||
"segments like './' or '//' are not allowed",
|
||||
)
|
||||
|
||||
# Canonicalize to the actual on-disk casing so the DB lookup below hits the
|
||||
# row keyed by the real path instead of inserting a wrong-cased duplicate.
|
||||
file_path = _canonical_file_path(project_config.home, segments)
|
||||
if file_path is None:
|
||||
raise HTTPException(
|
||||
status_code=404, detail=f"File not found on disk: '{data.file_path}'"
|
||||
)
|
||||
# Trigger: canonicalization rewrote a segment to its on-disk form, and that
|
||||
# segment may be a symlink. The pre-check above validated the ORIGINAL
|
||||
# request path — on a case-sensitive filesystem 'LINK/secret.md' does not
|
||||
# exist, so resolve() cannot follow the real 'link' symlink and the check
|
||||
# passes even when 'link' points outside the project root.
|
||||
# Why: indexing through an escaping symlink would read and index content
|
||||
# outside the project boundary — and even an is_file() existence probe on
|
||||
# the joined path would follow the symlink and stat its target, so
|
||||
# containment must hold BEFORE any filesystem probe that follows symlinks.
|
||||
# Path.resolve() only walks symlink names (readlink); it never opens or
|
||||
# stats the final target, so it is safe to run pre-containment.
|
||||
# Outcome: the canonical path is re-validated and the fully-resolved absolute
|
||||
# target must stay inside the resolved project home; escapes get a 400
|
||||
# before the file-existence probe below ever touches the target.
|
||||
resolved_target = (project_config.home / file_path).resolve()
|
||||
if not validate_project_path(file_path, project_config.home) or not (
|
||||
resolved_target.is_relative_to(project_config.home.resolve())
|
||||
):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"File path '{data.file_path}' is not allowed - "
|
||||
"paths must stay within project boundaries",
|
||||
)
|
||||
# Containment holds, so probing the resolved target cannot leave the project.
|
||||
if not resolved_target.is_file():
|
||||
raise HTTPException(
|
||||
status_code=404, detail=f"File not found on disk: '{data.file_path}'"
|
||||
)
|
||||
# Trigger: the canonical path matches the .bmignore / project .gitignore rules
|
||||
# Why: scan and watch flows filter ignored files before they ever reach the
|
||||
# indexer; indexing one here would bypass the ignored-file contract and
|
||||
# make hidden or gitignored content searchable
|
||||
# Outcome: the same should_ignore_path() rules apply to single-file sync
|
||||
ignore_patterns = load_gitignore_patterns(project_config.home)
|
||||
if should_ignore_path(
|
||||
project_config.home / file_path, project_config.home, ignore_patterns
|
||||
):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"File path '{data.file_path}' {IGNORED_PATH_REJECTION_DETAIL} "
|
||||
"and cannot be indexed",
|
||||
)
|
||||
if not sync_service.file_service.is_markdown(file_path):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Only markdown files can be indexed: '{data.file_path}'",
|
||||
)
|
||||
|
||||
# Trigger: the file may already have a DB row (e.g. modified on disk after indexing)
|
||||
# Why: the indexer needs to know whether to insert or update the entity
|
||||
# Outcome: new is computed from the database instead of assumed by the caller
|
||||
existing = await sync_service.entity_repository.get_by_file_path(file_path)
|
||||
synced = await sync_service.sync_one_markdown_file(
|
||||
file_path, new=existing is None, index_search=True
|
||||
)
|
||||
|
||||
# Trigger: semantic search is enabled and the entity index was just refreshed
|
||||
# Why: the project sync flow awaits sync_entity_vectors_batch() inline after
|
||||
# indexing changed files (SyncService.sync); without the single-entity
|
||||
# equivalent, a note recovered via sync-file stays missing or stale in
|
||||
# semantic search until a later edit or full project sync
|
||||
# Outcome: vectors refresh synchronously before the response returns,
|
||||
# mirroring the sync flow instead of the out-of-band scheduler
|
||||
if app_config.semantic_search_enabled:
|
||||
await search_service.sync_entity_vectors_batch([synced.entity.id])
|
||||
|
||||
result = EntityResponseV2.model_validate(synced.entity)
|
||||
logger.info(
|
||||
f"API v2 response: sync_file file_path='{file_path}' external_id={result.external_id}"
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
## Read endpoints
|
||||
|
||||
|
||||
|
||||
@@ -64,7 +64,9 @@ async def search(
|
||||
or query.permalink
|
||||
or query.permalink_match
|
||||
),
|
||||
has_filters=bool(query.note_types or query.entity_types or query.metadata_filters),
|
||||
has_filters=bool(
|
||||
query.note_types or query.entity_types or query.categories or query.metadata_filters
|
||||
),
|
||||
):
|
||||
offset = (page - 1) * page_size
|
||||
exact_count_available = query.retrieval_mode == SearchRetrievalMode.FTS
|
||||
|
||||
@@ -18,7 +18,9 @@ from basic_memory.services.context_service import (
|
||||
|
||||
|
||||
class EntityBatchLookup(Protocol):
|
||||
async def find_by_ids_for_hydration(self, ids: List[int]) -> Sequence[Any]: ...
|
||||
async def find_by_ids_for_hydration(
|
||||
self, ids: List[int], *, include_cross_project: bool = False
|
||||
) -> Sequence[Any]: ...
|
||||
|
||||
|
||||
class EntityServiceBatchLookup(Protocol):
|
||||
@@ -88,7 +90,7 @@ async def to_graph_context(
|
||||
result_count=len(entity_ids_needed),
|
||||
):
|
||||
entities = await entity_repository.find_by_ids_for_hydration(
|
||||
list(entity_ids_needed)
|
||||
list(entity_ids_needed), include_cross_project=True
|
||||
)
|
||||
for e in entities:
|
||||
entity_title_lookup[e.id] = e.title
|
||||
|
||||
@@ -57,6 +57,14 @@ def app_callback(
|
||||
container = CliContainer.create()
|
||||
set_container(container)
|
||||
|
||||
# Trigger: Postgres backend resolved at CLI startup, before any asyncio.run().
|
||||
# Why: uvloop must own the event-loop policy before the loop is created so the
|
||||
# asyncpg engine-dispose race (#831/#877) cannot fire. No-op for SQLite.
|
||||
# Outcome: subsequent asyncio.run() calls in CLI commands use uvloop on Postgres.
|
||||
from basic_memory.db import maybe_install_uvloop
|
||||
|
||||
maybe_install_uvloop(container.config)
|
||||
|
||||
# Trigger: first-run init confirmation before command output.
|
||||
# Why: informational "initialized" message belongs above command results, not in the upsell panel.
|
||||
# Outcome: one-time plain line printed before the subcommand runs.
|
||||
@@ -86,6 +94,7 @@ def app_callback(
|
||||
"reindex",
|
||||
"update",
|
||||
"watch",
|
||||
"workspace",
|
||||
}
|
||||
if (
|
||||
not version
|
||||
|
||||
@@ -9,6 +9,7 @@ from . import (
|
||||
format,
|
||||
schema,
|
||||
update,
|
||||
workspace,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
@@ -27,4 +28,5 @@ __all__ = [
|
||||
"format",
|
||||
"schema",
|
||||
"update",
|
||||
"workspace",
|
||||
]
|
||||
|
||||
@@ -11,9 +11,11 @@ from basic_memory.cli.commands.cloud.project_sync import * # noqa: F401,F403
|
||||
# Register snapshot sub-command group
|
||||
from basic_memory.cli.commands.cloud.snapshot import snapshot_app
|
||||
from basic_memory.cli.commands.cloud.workspace import workspace_app
|
||||
from basic_memory.cli.commands.cloud.shares import share_app
|
||||
|
||||
cloud_app.add_typer(snapshot_app, name="snapshot")
|
||||
cloud_app.add_typer(workspace_app, name="workspace")
|
||||
cloud_app.add_typer(share_app, name="share")
|
||||
|
||||
# Register restore command (directly on cloud_app via decorator)
|
||||
from basic_memory.cli.commands.cloud.restore import restore # noqa: F401, E402
|
||||
|
||||
@@ -32,14 +32,33 @@ def _rclone_exclude_filters(pattern: str) -> list[str]:
|
||||
return [f"- {path_pattern}", f"- {path_pattern}/**"]
|
||||
|
||||
|
||||
async def get_mount_info() -> TenantMountInfo:
|
||||
"""Get current tenant information from cloud API."""
|
||||
def _workspace_id_header(workspace_id: str | None) -> dict[str, str]:
|
||||
"""Header that routes a /tenant/mount/* request to a specific tenant.
|
||||
|
||||
The mount endpoints resolve the workspace from X-Workspace-ID (validating
|
||||
membership + subscription) and fall back to the user's default tenant when
|
||||
it is absent — so omitting it preserves the original default-tenant behavior.
|
||||
"""
|
||||
return {"X-Workspace-ID": workspace_id} if workspace_id else {}
|
||||
|
||||
|
||||
async def get_mount_info(*, workspace_id: str | None = None) -> TenantMountInfo:
|
||||
"""Get tenant mount info (bucket name + tenant id) from the cloud API.
|
||||
|
||||
Args:
|
||||
workspace_id: Tenant id of the target workspace. When omitted, the API
|
||||
uses the authenticated user's default tenant.
|
||||
"""
|
||||
try:
|
||||
config_manager = ConfigManager()
|
||||
config = config_manager.config
|
||||
host_url = config.cloud_host.rstrip("/")
|
||||
|
||||
response = await make_api_request(method="GET", url=f"{host_url}/tenant/mount/info")
|
||||
response = await make_api_request(
|
||||
method="GET",
|
||||
url=f"{host_url}/tenant/mount/info",
|
||||
headers=_workspace_id_header(workspace_id),
|
||||
)
|
||||
|
||||
return TenantMountInfo.model_validate(response.json())
|
||||
except Exception as e:
|
||||
@@ -47,13 +66,24 @@ async def get_mount_info() -> TenantMountInfo:
|
||||
|
||||
|
||||
async def generate_mount_credentials(tenant_id: str) -> MountCredentials:
|
||||
"""Generate scoped credentials for syncing."""
|
||||
"""Generate scoped S3 credentials for syncing a specific tenant's bucket.
|
||||
|
||||
Args:
|
||||
tenant_id: Tenant id whose bucket-scoped credentials to mint. Routed via
|
||||
X-Workspace-ID so team workspaces get their own bucket's credentials.
|
||||
"""
|
||||
try:
|
||||
config_manager = ConfigManager()
|
||||
config = config_manager.config
|
||||
host_url = config.cloud_host.rstrip("/")
|
||||
|
||||
response = await make_api_request(method="POST", url=f"{host_url}/tenant/mount/credentials")
|
||||
# The mount endpoints resolve X-Workspace-ID by matching the workspace's
|
||||
# tenant_id, so passing a tenant_id here is the correct routing key.
|
||||
response = await make_api_request(
|
||||
method="POST",
|
||||
url=f"{host_url}/tenant/mount/credentials",
|
||||
headers=_workspace_id_header(tenant_id),
|
||||
)
|
||||
|
||||
return MountCredentials.model_validate(response.json())
|
||||
except Exception as e:
|
||||
|
||||
@@ -26,15 +26,52 @@ from basic_memory.cli.commands.cloud.bisync_commands import (
|
||||
generate_mount_credentials,
|
||||
get_mount_info,
|
||||
)
|
||||
from basic_memory.cli.commands.cloud.rclone_config import configure_rclone_remote
|
||||
from basic_memory.cli.commands.cloud.rclone_config import (
|
||||
configure_rclone_remote,
|
||||
rclone_remote_exists,
|
||||
remote_name_for_workspace,
|
||||
)
|
||||
from basic_memory.cli.commands.cloud.rclone_installer import (
|
||||
RcloneInstallError,
|
||||
install_rclone,
|
||||
)
|
||||
from basic_memory.mcp.project_context import get_available_workspaces
|
||||
from basic_memory.schemas.cloud import (
|
||||
WorkspaceInfo,
|
||||
format_workspace_choices,
|
||||
format_workspace_selection_choices,
|
||||
workspace_matches_exact_identifier,
|
||||
)
|
||||
|
||||
console = Console()
|
||||
|
||||
|
||||
def _resolve_setup_workspace(identifier: str) -> WorkspaceInfo:
|
||||
"""Resolve a workspace identifier (slug, name, or tenant_id) for setup.
|
||||
|
||||
Errors with copyable choices when the identifier matches zero or multiple
|
||||
workspaces, so the user can disambiguate.
|
||||
"""
|
||||
workspaces = run_with_cleanup(get_available_workspaces())
|
||||
if not workspaces:
|
||||
console.print("[red]No accessible cloud workspaces found for this account[/red]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
matches = [ws for ws in workspaces if workspace_matches_exact_identifier(ws, identifier)]
|
||||
if len(matches) == 1:
|
||||
return matches[0]
|
||||
|
||||
if not matches:
|
||||
console.print(f"[red]No workspace matches '{identifier}'[/red]")
|
||||
console.print("\nAvailable workspaces:")
|
||||
console.print(format_workspace_choices(workspaces))
|
||||
else:
|
||||
console.print(f"[red]'{identifier}' matches multiple workspaces[/red]")
|
||||
console.print("\nDisambiguate with the workspace slug or tenant_id:")
|
||||
console.print(format_workspace_selection_choices(matches))
|
||||
raise typer.Exit(1)
|
||||
|
||||
|
||||
@cloud_app.command()
|
||||
def login():
|
||||
"""Authenticate with WorkOS using OAuth Device Authorization flow."""
|
||||
@@ -164,13 +201,28 @@ def status() -> None:
|
||||
|
||||
|
||||
@cloud_app.command("setup")
|
||||
def setup() -> None:
|
||||
def setup(
|
||||
workspace: str | None = typer.Option(
|
||||
None,
|
||||
"--workspace",
|
||||
help="Set up sync for a specific workspace (slug, name, or tenant_id). "
|
||||
"Omit for your default workspace.",
|
||||
),
|
||||
force: bool = typer.Option(
|
||||
False,
|
||||
"--force",
|
||||
help="Reconfigure an rclone remote that already exists (mints new credentials).",
|
||||
),
|
||||
) -> None:
|
||||
"""Set up cloud sync by installing rclone and configuring credentials.
|
||||
|
||||
After setup, use project commands for syncing:
|
||||
bm project add <name> --cloud --local-path ~/projects/<name>
|
||||
bm project bisync --name <name> --resync # First time
|
||||
bm project bisync --name <name> # Subsequent syncs
|
||||
Run once per workspace you sync. The default workspace uses the
|
||||
'basic-memory-cloud' remote; other (e.g. Team) workspaces each get their own
|
||||
tenant-scoped remote, since Tigris credentials are bucket-scoped.
|
||||
|
||||
After setup, use the cloud sync commands:
|
||||
bm cloud pull --name <name> # fetch cloud changes (Team-safe)
|
||||
bm cloud push --name <name> # upload local changes (Team-safe)
|
||||
"""
|
||||
console.print("[bold blue]Basic Memory Cloud Setup[/bold blue]")
|
||||
console.print("Setting up cloud sync with rclone...\n")
|
||||
@@ -180,35 +232,60 @@ def setup() -> None:
|
||||
console.print("[blue]Step 1: Installing rclone...[/blue]")
|
||||
install_rclone()
|
||||
|
||||
# Step 2: Get tenant info
|
||||
# --- Resolve target workspace ---
|
||||
# Trigger: --workspace given. Why: Tigris keys are tenant-scoped, so a
|
||||
# non-default workspace needs its own bucket + remote. Outcome: scope the
|
||||
# mount-info/credentials calls and name the remote after the workspace.
|
||||
if workspace is not None:
|
||||
target = _resolve_setup_workspace(workspace)
|
||||
workspace_id: str | None = target.tenant_id
|
||||
remote_name = remote_name_for_workspace(target.slug, is_default=target.is_default)
|
||||
console.print(f"[dim]Workspace: {target.name} ({target.slug})[/dim]")
|
||||
else:
|
||||
workspace_id = None # default tenant
|
||||
remote_name = remote_name_for_workspace(None, is_default=True)
|
||||
|
||||
# Trigger: the target rclone remote already exists.
|
||||
# Why: re-running setup mints new credentials and overwrites the remote,
|
||||
# which would silently repoint it — e.g. clobbering the shared
|
||||
# basic-memory-cloud remote that served another tenant. Checked BEFORE
|
||||
# minting so an abort wastes no credentials.
|
||||
# Outcome: stop unless the user explicitly opts in with --force.
|
||||
if rclone_remote_exists(remote_name) and not force:
|
||||
console.print(f"[red]rclone remote '{remote_name}' is already configured.[/red]")
|
||||
console.print(
|
||||
"Re-running setup mints new credentials and overwrites it. "
|
||||
"Pass --force to reconfigure."
|
||||
)
|
||||
raise typer.Exit(1)
|
||||
|
||||
# Step 2: Get tenant info (scoped to the target workspace when given)
|
||||
console.print("\n[blue]Step 2: Getting tenant information...[/blue]")
|
||||
tenant_info = run_with_cleanup(get_mount_info())
|
||||
tenant_info = run_with_cleanup(get_mount_info(workspace_id=workspace_id))
|
||||
console.print(f"[green]Found tenant: {tenant_info.tenant_id}[/green]")
|
||||
|
||||
# Step 3: Generate credentials
|
||||
# Step 3: Generate credentials for that tenant's bucket
|
||||
console.print("\n[blue]Step 3: Generating sync credentials...[/blue]")
|
||||
creds = run_with_cleanup(generate_mount_credentials(tenant_info.tenant_id))
|
||||
console.print("[green]Generated secure credentials[/green]")
|
||||
|
||||
# Step 4: Configure rclone remote
|
||||
# Step 4: Configure the tenant's rclone remote
|
||||
console.print("\n[blue]Step 4: Configuring rclone remote...[/blue]")
|
||||
configure_rclone_remote(
|
||||
access_key=creds.access_key,
|
||||
secret_key=creds.secret_key,
|
||||
remote_name=remote_name,
|
||||
)
|
||||
|
||||
console.print("\n[bold green]Cloud setup completed successfully![/bold green]")
|
||||
console.print("\n[bold]Next steps:[/bold]")
|
||||
console.print("1. Add a project with local sync path:")
|
||||
console.print(" bm project add research --cloud --local-path ~/Documents/research")
|
||||
console.print("\n Or configure sync for an existing project:")
|
||||
console.print("1. Configure sync for a project:")
|
||||
console.print(" bm cloud sync-setup research ~/Documents/research")
|
||||
console.print("\n2. Preview the initial sync (recommended):")
|
||||
console.print(" bm project bisync --name research --resync --dry-run")
|
||||
console.print("\n3. If all looks good, run the actual sync:")
|
||||
console.print(" bm project bisync --name research --resync")
|
||||
console.print("\n4. Subsequent syncs (no --resync needed):")
|
||||
console.print(" bm project bisync --name research")
|
||||
console.print("\n2. Preview a pull (recommended):")
|
||||
console.print(" bm cloud pull --name research --dry-run")
|
||||
console.print("\n3. Fetch cloud changes / upload local changes:")
|
||||
console.print(" bm cloud pull --name research")
|
||||
console.print(" bm cloud push --name research")
|
||||
console.print(
|
||||
"\n[dim]Tip: Always use --dry-run first to preview changes before syncing[/dim]"
|
||||
)
|
||||
@@ -216,6 +293,8 @@ def setup() -> None:
|
||||
except (RcloneInstallError, BisyncError, CloudAPIError) as e:
|
||||
console.print(f"\n[red]Setup failed: {e}[/red]")
|
||||
raise typer.Exit(1)
|
||||
except typer.Exit:
|
||||
raise
|
||||
except Exception as e:
|
||||
console.print(f"\n[red]Unexpected error during setup: {e}[/red]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
@@ -7,6 +7,7 @@ they are cloud-specific operations.
|
||||
|
||||
import os
|
||||
from datetime import datetime
|
||||
from enum import Enum
|
||||
|
||||
import typer
|
||||
from rich.console import Console
|
||||
@@ -16,10 +17,19 @@ from basic_memory.cli.commands.cloud.bisync_commands import get_mount_info
|
||||
from basic_memory.cli.commands.cloud.rclone_commands import (
|
||||
RcloneError,
|
||||
SyncProject,
|
||||
TransferDirection,
|
||||
TransferPlan,
|
||||
get_project_bisync_state,
|
||||
project_bisync,
|
||||
project_check,
|
||||
project_diff,
|
||||
project_sync,
|
||||
project_transfer,
|
||||
)
|
||||
from basic_memory.cli.commands.cloud.rclone_config import (
|
||||
DEFAULT_RCLONE_REMOTE,
|
||||
rclone_remote_exists,
|
||||
remote_name_for_workspace,
|
||||
)
|
||||
from basic_memory.cli.commands.command_utils import run_with_cleanup
|
||||
from basic_memory.cli.commands.routing import force_routing
|
||||
@@ -27,7 +37,12 @@ from basic_memory.config import BasicMemoryConfig, ConfigManager, ProjectEntry
|
||||
from basic_memory.mcp.async_client import get_client
|
||||
from basic_memory.mcp.clients import ProjectClient
|
||||
from basic_memory.mcp.project_context import get_available_workspaces
|
||||
from basic_memory.schemas.cloud import WorkspaceInfo
|
||||
from basic_memory.schemas.cloud import (
|
||||
WorkspaceInfo,
|
||||
format_workspace_choices,
|
||||
format_workspace_selection_choices,
|
||||
workspace_matches_exact_identifier,
|
||||
)
|
||||
from basic_memory.schemas.project_info import ProjectItem
|
||||
from basic_memory.utils import generate_permalink, normalize_project_path
|
||||
|
||||
@@ -35,9 +50,34 @@ console = Console()
|
||||
|
||||
TEAM_WORKSPACE_BISYNC_UNSUPPORTED = (
|
||||
"The bisync operation is only supported on Personal workspaces.\n"
|
||||
"Use `bm cloud sync --name {name}` instead."
|
||||
"Use `bm cloud pull --name {name}` / `bm cloud push --name {name}` instead."
|
||||
)
|
||||
|
||||
TEAM_WORKSPACE_SYNC_UNSUPPORTED = (
|
||||
"The sync operation mirrors local onto the shared bucket and can delete a "
|
||||
"teammate's files, so it is only supported on Personal workspaces.\n"
|
||||
"Use `bm cloud pull --name {name}` (fetch) / `bm cloud push --name {name}` "
|
||||
"(additive upload) instead."
|
||||
)
|
||||
|
||||
|
||||
class ConflictStrategy(str, Enum):
|
||||
"""How push/pull resolves files that differ on both sides.
|
||||
|
||||
Default is ``fail``: surface the conflicts and abort before transferring,
|
||||
leaving the user to re-run with an explicit resolution — like git refusing
|
||||
to clobber local changes.
|
||||
|
||||
This is the Typer-facing enum; the engine in ``rclone_commands`` accepts the
|
||||
same values as a ``ConflictStrategy`` Literal. ``_run_directional_transfer``
|
||||
bridges the two by passing ``on_conflict.value``. Keep the values in sync.
|
||||
"""
|
||||
|
||||
fail = "fail"
|
||||
keep_local = "keep-local"
|
||||
keep_cloud = "keep-cloud"
|
||||
keep_both = "keep-both"
|
||||
|
||||
|
||||
# --- Shared helpers ---
|
||||
|
||||
@@ -59,12 +99,41 @@ def _require_cloud_credentials(config: BasicMemoryConfig) -> None:
|
||||
raise typer.Exit(1)
|
||||
|
||||
|
||||
async def _get_workspace_for_project(name: str, config: BasicMemoryConfig) -> WorkspaceInfo:
|
||||
"""Resolve the cloud workspace targeted by a project-scoped sync command."""
|
||||
async def _get_workspace_for_project(
|
||||
name: str,
|
||||
config: BasicMemoryConfig,
|
||||
*,
|
||||
workspace_override: str | None = None,
|
||||
) -> WorkspaceInfo:
|
||||
"""Resolve the cloud workspace targeted by a project-scoped sync command.
|
||||
|
||||
``workspace_override`` (a slug, name, or tenant_id, e.g. from ``--workspace``)
|
||||
takes precedence over config, letting the user disambiguate a project name
|
||||
that exists in more than one workspace.
|
||||
"""
|
||||
workspaces = await get_available_workspaces()
|
||||
if not workspaces:
|
||||
raise ValueError("No accessible cloud workspaces found for this account")
|
||||
|
||||
# An explicit override wins over config — this is how the user disambiguates.
|
||||
if workspace_override is not None:
|
||||
matches = [
|
||||
item
|
||||
for item in workspaces
|
||||
if workspace_matches_exact_identifier(item, workspace_override)
|
||||
]
|
||||
if len(matches) == 1:
|
||||
return matches[0]
|
||||
if not matches:
|
||||
raise ValueError(
|
||||
f"No accessible workspace matches '{workspace_override}'.\n"
|
||||
f"{format_workspace_choices(workspaces)}"
|
||||
)
|
||||
raise ValueError(
|
||||
f"'{workspace_override}' matches multiple workspaces; use a slug or tenant_id:\n"
|
||||
f"{format_workspace_selection_choices(matches)}"
|
||||
)
|
||||
|
||||
entry = config.projects.get(name)
|
||||
workspace_id = entry.workspace_id if entry and entry.workspace_id else config.default_workspace
|
||||
if workspace_id:
|
||||
@@ -92,8 +161,18 @@ async def _get_workspace_for_project(name: str, config: BasicMemoryConfig) -> Wo
|
||||
)
|
||||
|
||||
|
||||
def _require_personal_workspace(name: str, config: BasicMemoryConfig) -> WorkspaceInfo:
|
||||
"""Exit before bisync work when the target workspace is not personal."""
|
||||
def _require_personal_workspace(
|
||||
name: str,
|
||||
config: BasicMemoryConfig,
|
||||
*,
|
||||
unsupported_message: str = TEAM_WORKSPACE_BISYNC_UNSUPPORTED,
|
||||
) -> WorkspaceInfo:
|
||||
"""Exit before mirror work when the target workspace is not personal.
|
||||
|
||||
Used to gate the destructive mirror operations (`sync`, `bisync`) to
|
||||
Personal workspaces. ``unsupported_message`` lets each command point Team
|
||||
users at the right Team-safe alternative.
|
||||
"""
|
||||
try:
|
||||
workspace = run_with_cleanup(_get_workspace_for_project(name, config))
|
||||
except Exception as exc:
|
||||
@@ -101,15 +180,21 @@ def _require_personal_workspace(name: str, config: BasicMemoryConfig) -> Workspa
|
||||
raise typer.Exit(1)
|
||||
|
||||
if workspace.workspace_type != "personal":
|
||||
console.print(f"[red]{TEAM_WORKSPACE_BISYNC_UNSUPPORTED.format(name=name)}[/red]")
|
||||
console.print(f"[red]{unsupported_message.format(name=name)}[/red]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
return workspace
|
||||
|
||||
|
||||
async def _get_cloud_project(name: str) -> ProjectItem | None:
|
||||
"""Fetch a project by name from the cloud API."""
|
||||
async with get_client(project_name=name) as client:
|
||||
async def _get_cloud_project(name: str, *, workspace_id: str | None = None) -> ProjectItem | None:
|
||||
"""Fetch a project by name from the cloud API.
|
||||
|
||||
``workspace_id`` routes the lookup to a specific tenant so the project
|
||||
metadata comes from the same workspace the transfer targets (otherwise
|
||||
get_client would resolve the workspace from config/default and could read a
|
||||
different tenant — see #920 review).
|
||||
"""
|
||||
async with get_client(project_name=name, workspace=workspace_id) as client:
|
||||
projects_list = await ProjectClient(client).list_projects()
|
||||
for proj in projects_list.projects:
|
||||
if generate_permalink(proj.name) == generate_permalink(name):
|
||||
@@ -118,10 +203,17 @@ async def _get_cloud_project(name: str) -> ProjectItem | None:
|
||||
|
||||
|
||||
def _get_sync_project(
|
||||
name: str, config: BasicMemoryConfig, project_data: ProjectItem
|
||||
name: str,
|
||||
config: BasicMemoryConfig,
|
||||
project_data: ProjectItem,
|
||||
*,
|
||||
remote_name: str = DEFAULT_RCLONE_REMOTE,
|
||||
) -> tuple[SyncProject, str | None]:
|
||||
"""Build a SyncProject and resolve local_sync_path from config.
|
||||
|
||||
``remote_name`` selects which tenant-scoped rclone remote the project routes
|
||||
through (default tenant vs a team workspace remote).
|
||||
|
||||
Returns (sync_project, local_sync_path). Exits if no local_sync_path configured.
|
||||
"""
|
||||
sync_entry = config.projects.get(name)
|
||||
@@ -137,6 +229,7 @@ def _get_sync_project(
|
||||
name=project_data.name,
|
||||
path=normalize_project_path(project_data.path),
|
||||
local_sync_path=local_sync_path,
|
||||
remote_name=remote_name,
|
||||
)
|
||||
return sync_project, local_sync_path
|
||||
|
||||
@@ -146,11 +239,15 @@ def _get_sync_project(
|
||||
|
||||
@cloud_app.command("sync")
|
||||
def sync_project_command(
|
||||
name: str = typer.Option(..., "--name", help="Project name to sync"),
|
||||
name: str = typer.Option(..., "--name", "--project", help="Project name to sync"),
|
||||
dry_run: bool = typer.Option(False, "--dry-run", help="Preview changes without syncing"),
|
||||
verbose: bool = typer.Option(False, "--verbose", "-v", help="Show detailed output"),
|
||||
) -> None:
|
||||
"""One-way sync: local -> cloud (make cloud identical to local).
|
||||
"""One-way mirror: local -> cloud (make cloud identical to local).
|
||||
|
||||
Personal workspaces only. This deletes cloud files not present locally, so
|
||||
on Team workspaces use `bm cloud push` (additive upload) / `bm cloud pull`
|
||||
(fetch) instead.
|
||||
|
||||
Example:
|
||||
bm cloud sync --name research
|
||||
@@ -158,9 +255,13 @@ def sync_project_command(
|
||||
"""
|
||||
config = ConfigManager().config
|
||||
_require_cloud_credentials(config)
|
||||
_require_personal_workspace(name, config, unsupported_message=TEAM_WORKSPACE_SYNC_UNSUPPORTED)
|
||||
|
||||
try:
|
||||
# Get tenant info for bucket name
|
||||
# Get tenant info for bucket name.
|
||||
# TODO(#919): scope to the project's workspace like push/pull. Safe for now
|
||||
# because these mirror commands are gated to the (default-tenant) Personal
|
||||
# workspace, so the default mount info is correct.
|
||||
tenant_info = run_with_cleanup(get_mount_info())
|
||||
bucket_name = tenant_info.bucket_name
|
||||
|
||||
@@ -191,14 +292,231 @@ def sync_project_command(
|
||||
raise typer.Exit(1)
|
||||
|
||||
|
||||
def _print_conflict_abort(name: str, direction: TransferDirection, plan: TransferPlan) -> None:
|
||||
"""Explain a conflict abort and how to resolve it (git-pull style)."""
|
||||
console.print(
|
||||
f"[red]{direction.capitalize()} aborted: {len(plan.conflicts)} file(s) differ between "
|
||||
f"local and cloud.[/red]"
|
||||
)
|
||||
for path in plan.conflicts:
|
||||
console.print(f" [yellow]*[/yellow] {path}")
|
||||
console.print("\nRe-run with one of:")
|
||||
console.print(" [dim]--on-conflict keep-cloud[/dim] take the cloud version")
|
||||
console.print(" [dim]--on-conflict keep-local[/dim] keep your local version")
|
||||
console.print(
|
||||
" [dim]--on-conflict keep-both[/dim] keep both (writes <name>.conflict-<date>)"
|
||||
)
|
||||
|
||||
|
||||
def _run_directional_transfer(
|
||||
name: str,
|
||||
direction: TransferDirection,
|
||||
*,
|
||||
on_conflict: ConflictStrategy,
|
||||
dry_run: bool,
|
||||
verbose: bool,
|
||||
workspace: str | None = None,
|
||||
) -> None:
|
||||
"""Shared orchestration for `bm cloud push` / `bm cloud pull`.
|
||||
|
||||
Detects conflicts first, then aborts (the default) or applies the chosen
|
||||
resolution. Uses additive `rclone copy`, so it never deletes on the
|
||||
destination — safe for Team workspaces and therefore not gated.
|
||||
|
||||
Routes through the resolved workspace's own tenant-scoped rclone remote, so a
|
||||
Team project reads/writes the right bucket (see #919).
|
||||
"""
|
||||
config = ConfigManager().config
|
||||
_require_cloud_credentials(config)
|
||||
|
||||
try:
|
||||
# --- Resolve the target workspace and its tenant-scoped remote ---
|
||||
# Tigris credentials are bucket/tenant-scoped, so each workspace has its
|
||||
# own rclone remote. Resolve which workspace this project belongs to
|
||||
# (config or --workspace override) before touching any bucket.
|
||||
try:
|
||||
target_workspace = run_with_cleanup(
|
||||
_get_workspace_for_project(name, config, workspace_override=workspace)
|
||||
)
|
||||
except Exception as exc:
|
||||
console.print(f"[red]Error resolving workspace for project '{name}': {exc}[/red]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
remote_name = remote_name_for_workspace(
|
||||
target_workspace.slug, is_default=target_workspace.is_default
|
||||
)
|
||||
|
||||
# Trigger: the workspace's remote has not been configured yet.
|
||||
# Why: provisioning mints tenant-scoped credentials and must be explicit
|
||||
# (no surprise key generation); push/pull only transfer.
|
||||
# Outcome: stop with the exact setup command for this workspace.
|
||||
if not rclone_remote_exists(remote_name):
|
||||
setup_target = (
|
||||
"" if target_workspace.is_default else f" --workspace {target_workspace.slug}"
|
||||
)
|
||||
console.print(f"[red]Workspace '{target_workspace.slug}' is not set up for sync.[/red]")
|
||||
console.print(f"\nRun: bm cloud setup{setup_target}")
|
||||
raise typer.Exit(1)
|
||||
|
||||
# Get tenant info for bucket name, scoped to the resolved workspace
|
||||
tenant_info = run_with_cleanup(get_mount_info(workspace_id=target_workspace.tenant_id))
|
||||
bucket_name = tenant_info.bucket_name
|
||||
|
||||
# Get project info from the same workspace we resolved above, so the
|
||||
# project path and the bucket/remote all refer to one tenant.
|
||||
with force_routing(cloud=True):
|
||||
project_data = run_with_cleanup(
|
||||
_get_cloud_project(name, workspace_id=target_workspace.tenant_id)
|
||||
)
|
||||
if not project_data:
|
||||
console.print(f"[red]Error: Project '{name}' not found[/red]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
sync_project, _ = _get_sync_project(name, config, project_data, remote_name=remote_name)
|
||||
|
||||
# --- Detect before transferring ---
|
||||
plan = project_diff(sync_project, bucket_name, direction)
|
||||
|
||||
# Trigger: rclone could not read/hash some files.
|
||||
# Why: comparing is the whole basis for a safe transfer — never guess.
|
||||
# Outcome: abort before moving any bytes.
|
||||
if plan.errors:
|
||||
console.print(
|
||||
f"[red]{direction.capitalize()} aborted: rclone could not compare "
|
||||
f"{len(plan.errors)} file(s)[/red]"
|
||||
)
|
||||
for path in plan.errors:
|
||||
console.print(f" [red]![/red] {path}")
|
||||
raise typer.Exit(1)
|
||||
|
||||
# Trigger: files differ on both sides and the user chose no resolution.
|
||||
# Why: "no surprises" — never silently pick a winner.
|
||||
# Outcome: list the conflicts and exit, like git refusing to clobber.
|
||||
if plan.conflicts and on_conflict is ConflictStrategy.fail:
|
||||
_print_conflict_abort(name, direction, plan)
|
||||
raise typer.Exit(1)
|
||||
|
||||
# --- Transfer ---
|
||||
arrow = "cloud -> local" if direction == "pull" else "local -> cloud"
|
||||
console.print(f"[blue]{direction.capitalize()} {name} ({arrow})...[/blue]")
|
||||
|
||||
conflict_suffix = datetime.now().strftime("%Y%m%d-%H%M%S")
|
||||
success = project_transfer(
|
||||
sync_project,
|
||||
bucket_name,
|
||||
direction,
|
||||
plan,
|
||||
strategy=on_conflict.value,
|
||||
conflict_suffix=conflict_suffix,
|
||||
dry_run=dry_run,
|
||||
verbose=verbose,
|
||||
)
|
||||
|
||||
if not success:
|
||||
console.print(f"[red]{name} {direction} failed[/red]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
console.print(f"[green]{name} {direction} completed successfully[/green]")
|
||||
|
||||
# Without a sync baseline (see #862) we cannot tell an intentional delete
|
||||
# from a file the other side simply never had, so deletions never sync.
|
||||
if plan.dest_only:
|
||||
kept_on = "local" if direction == "pull" else "cloud"
|
||||
console.print(
|
||||
f"[dim]{len(plan.dest_only)} file(s) exist only on {kept_on} and were left "
|
||||
"untouched (deletions are not propagated).[/dim]"
|
||||
)
|
||||
|
||||
except RcloneError as e:
|
||||
console.print(f"[red]{direction.capitalize()} error: {e}[/red]")
|
||||
raise typer.Exit(1)
|
||||
except typer.Exit:
|
||||
# Already-handled exits (not found, conflicts, errors) propagate cleanly.
|
||||
raise
|
||||
except Exception as e:
|
||||
console.print(f"[red]Error: {e}[/red]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
|
||||
@cloud_app.command("pull")
|
||||
def pull_project_command(
|
||||
name: str = typer.Option(..., "--name", "--project", help="Project name to pull"),
|
||||
on_conflict: ConflictStrategy = typer.Option(
|
||||
ConflictStrategy.fail,
|
||||
"--on-conflict",
|
||||
help="Resolve files that differ on both sides (default: fail and list them)",
|
||||
),
|
||||
workspace: str | None = typer.Option(
|
||||
None,
|
||||
"--workspace",
|
||||
help="Workspace (slug, name, or tenant_id) when the project name is ambiguous",
|
||||
),
|
||||
dry_run: bool = typer.Option(False, "--dry-run", help="Preview changes without pulling"),
|
||||
verbose: bool = typer.Option(False, "--verbose", "-v", help="Show detailed output"),
|
||||
) -> None:
|
||||
"""Fetch cloud changes into local (cloud -> local), git-pull style.
|
||||
|
||||
Additive and Team-safe: downloads new/changed cloud files and never deletes
|
||||
local files. A file that differs on both sides is a conflict; by default
|
||||
pull aborts and lists them. Deletions are not propagated (see #862).
|
||||
|
||||
Examples:
|
||||
bm cloud pull --name research
|
||||
bm cloud pull --name research --dry-run
|
||||
bm cloud pull --name research --on-conflict keep-cloud
|
||||
bm cloud pull --name research --workspace acme
|
||||
"""
|
||||
_run_directional_transfer(
|
||||
name, "pull", on_conflict=on_conflict, dry_run=dry_run, verbose=verbose, workspace=workspace
|
||||
)
|
||||
|
||||
|
||||
@cloud_app.command("push")
|
||||
def push_project_command(
|
||||
name: str = typer.Option(..., "--name", "--project", help="Project name to push"),
|
||||
on_conflict: ConflictStrategy = typer.Option(
|
||||
ConflictStrategy.fail,
|
||||
"--on-conflict",
|
||||
help="Resolve files that differ on both sides (default: fail and list them)",
|
||||
),
|
||||
workspace: str | None = typer.Option(
|
||||
None,
|
||||
"--workspace",
|
||||
help="Workspace (slug, name, or tenant_id) when the project name is ambiguous",
|
||||
),
|
||||
dry_run: bool = typer.Option(False, "--dry-run", help="Preview changes without pushing"),
|
||||
verbose: bool = typer.Option(False, "--verbose", "-v", help="Show detailed output"),
|
||||
) -> None:
|
||||
"""Upload local changes to cloud (local -> cloud), additive and Team-safe.
|
||||
|
||||
Uploads new/changed local files and never deletes cloud files. A file that
|
||||
differs on both sides is a conflict; by default push aborts and lists them
|
||||
(like git rejecting a push when the remote is ahead — pull first). Deletions
|
||||
are not propagated (see #862).
|
||||
|
||||
Examples:
|
||||
bm cloud push --name research
|
||||
bm cloud push --name research --dry-run
|
||||
bm cloud push --name research --on-conflict keep-local
|
||||
bm cloud push --name research --workspace acme
|
||||
"""
|
||||
_run_directional_transfer(
|
||||
name, "push", on_conflict=on_conflict, dry_run=dry_run, verbose=verbose, workspace=workspace
|
||||
)
|
||||
|
||||
|
||||
@cloud_app.command("bisync")
|
||||
def bisync_project_command(
|
||||
name: str = typer.Option(..., "--name", help="Project name to bisync"),
|
||||
name: str = typer.Option(..., "--name", "--project", help="Project name to bisync"),
|
||||
dry_run: bool = typer.Option(False, "--dry-run", help="Preview changes without syncing"),
|
||||
resync: bool = typer.Option(False, "--resync", help="Force new baseline"),
|
||||
verbose: bool = typer.Option(False, "--verbose", "-v", help="Show detailed output"),
|
||||
) -> None:
|
||||
"""Two-way sync: local <-> cloud (bidirectional sync).
|
||||
"""Two-way mirror: local <-> cloud (bidirectional sync).
|
||||
|
||||
Personal workspaces only. This mirror can delete and overwrite files on both
|
||||
sides, so on Team workspaces use `bm cloud pull` (fetch) / `bm cloud push`
|
||||
(additive upload) instead.
|
||||
|
||||
Examples:
|
||||
bm cloud bisync --name research --resync # First time
|
||||
@@ -210,7 +528,10 @@ def bisync_project_command(
|
||||
_require_personal_workspace(name, config)
|
||||
|
||||
try:
|
||||
# Get tenant info for bucket name
|
||||
# Get tenant info for bucket name.
|
||||
# TODO(#919): scope to the project's workspace like push/pull. Safe for now
|
||||
# because these mirror commands are gated to the (default-tenant) Personal
|
||||
# workspace, so the default mount info is correct.
|
||||
tenant_info = run_with_cleanup(get_mount_info())
|
||||
bucket_name = tenant_info.bucket_name
|
||||
|
||||
@@ -256,10 +577,14 @@ def bisync_project_command(
|
||||
|
||||
@cloud_app.command("check")
|
||||
def check_project_command(
|
||||
name: str = typer.Option(..., "--name", help="Project name to check"),
|
||||
name: str = typer.Option(..., "--name", "--project", help="Project name to check"),
|
||||
one_way: bool = typer.Option(False, "--one-way", help="Check one direction only (faster)"),
|
||||
) -> None:
|
||||
"""Verify file integrity between local and cloud.
|
||||
"""Verify file integrity between local and cloud (no changes made).
|
||||
|
||||
Personal workspaces only: check compares against the Personal workspace
|
||||
mirror remote. On Team workspaces use `bm cloud pull --dry-run` /
|
||||
`bm cloud push --dry-run` to preview differences instead.
|
||||
|
||||
Example:
|
||||
bm cloud check --name research
|
||||
@@ -268,7 +593,10 @@ def check_project_command(
|
||||
_require_cloud_credentials(config)
|
||||
|
||||
try:
|
||||
# Get tenant info for bucket name
|
||||
# Get tenant info for bucket name.
|
||||
# TODO(#919): scope to the project's workspace like push/pull. Safe for now
|
||||
# because these mirror commands are gated to the (default-tenant) Personal
|
||||
# workspace, so the default mount info is correct.
|
||||
tenant_info = run_with_cleanup(get_mount_info())
|
||||
bucket_name = tenant_info.bucket_name
|
||||
|
||||
@@ -304,6 +632,9 @@ def bisync_reset(
|
||||
) -> None:
|
||||
"""Clear bisync state for a project.
|
||||
|
||||
Personal workspaces only (bisync is a Personal-workspace mirror; on Team
|
||||
workspaces use `bm cloud pull` / `bm cloud push` instead).
|
||||
|
||||
This removes the bisync metadata files, forcing a fresh --resync on next bisync.
|
||||
Useful when bisync gets into an inconsistent state or when remote path changes.
|
||||
"""
|
||||
@@ -395,9 +726,15 @@ def setup_project_sync(
|
||||
|
||||
console.print(f"[green]Sync configured for project '{name}'[/green]")
|
||||
console.print(f"\nLocal sync path: {resolved_path}")
|
||||
# Lead with the Team-safe additive commands (work on any workspace); the
|
||||
# `sync`/`bisync` mirrors are Personal-workspace-only.
|
||||
console.print("\nNext steps:")
|
||||
console.print(f" 1. Preview: bm cloud sync --name {name} --dry-run")
|
||||
console.print(f" 2. Sync: bm cloud sync --name {name}")
|
||||
console.print(f" 1. Preview a pull: bm cloud pull --name {name} --dry-run")
|
||||
console.print(f" 2. Fetch from cloud: bm cloud pull --name {name}")
|
||||
console.print(f" 3. Upload local changes: bm cloud push --name {name}")
|
||||
console.print(
|
||||
f" Personal workspaces can also mirror with: bm cloud bisync --name {name} --resync"
|
||||
)
|
||||
except Exception as e:
|
||||
console.print(f"[red]Error configuring sync: {str(e)}[/red]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
@@ -2,7 +2,8 @@
|
||||
|
||||
This module provides simplified, project-scoped rclone operations:
|
||||
- Each project syncs independently
|
||||
- Uses single "basic-memory-cloud" remote (not tenant-specific)
|
||||
- Routes through the project's tenant-scoped remote (SyncProject.remote_name);
|
||||
the default tenant keeps "basic-memory-cloud", others use their own (see #919)
|
||||
- Balanced defaults from SPEC-8 Phase 4 testing
|
||||
- Per-project bisync state tracking
|
||||
|
||||
@@ -11,10 +12,10 @@ Replaces tenant-wide sync with project-scoped workflows.
|
||||
|
||||
import re
|
||||
import subprocess
|
||||
from dataclasses import dataclass
|
||||
from dataclasses import dataclass, field
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
from typing import Callable, Optional, Protocol
|
||||
from pathlib import Path, PurePosixPath
|
||||
from typing import Callable, Literal, Optional, Protocol
|
||||
|
||||
from loguru import logger
|
||||
from rich.console import Console
|
||||
@@ -42,6 +43,7 @@ TIGRIS_CONSISTENCY_HEADERS = [
|
||||
class RunResult(Protocol):
|
||||
returncode: int
|
||||
stdout: str
|
||||
stderr: str
|
||||
|
||||
|
||||
RunFunc = Callable[..., RunResult]
|
||||
@@ -112,11 +114,15 @@ class SyncProject:
|
||||
name: Project name
|
||||
path: Cloud path (e.g., "app/data/research")
|
||||
local_sync_path: Local directory for syncing (optional)
|
||||
remote_name: rclone remote serving this project's tenant bucket. Defaults
|
||||
to the legacy single remote; team/non-default workspaces use their own
|
||||
(see remote_name_for_workspace).
|
||||
"""
|
||||
|
||||
name: str
|
||||
path: str
|
||||
local_sync_path: Optional[str] = None
|
||||
remote_name: str = "basic-memory-cloud"
|
||||
|
||||
|
||||
def get_bmignore_filter_path() -> Path:
|
||||
@@ -178,10 +184,98 @@ def get_project_remote(project: SyncProject, bucket_name: str) -> str:
|
||||
The API returns paths like "/app/data/basic-memory-llc" because the S3 bucket
|
||||
is mounted at /app/data on the fly machine. We need to strip the /app/data/
|
||||
prefix to get the actual S3 path within the bucket.
|
||||
|
||||
The remote name comes from the project so non-default/team workspaces route
|
||||
through their own tenant-scoped remote (see #919).
|
||||
"""
|
||||
# Normalize path to strip /app/data/ mount point prefix
|
||||
cloud_path = normalize_project_path(project.path).lstrip("/")
|
||||
return f"basic-memory-cloud:{bucket_name}/{cloud_path}"
|
||||
return f"{project.remote_name}:{bucket_name}/{cloud_path}"
|
||||
|
||||
|
||||
# --- Directional transfer primitives (push / pull) ---
|
||||
#
|
||||
# These power the Team-safe `bm cloud push` / `bm cloud pull` commands. Unlike
|
||||
# the mirror operations (`sync`/`bisync`), they use `rclone copy` so they never
|
||||
# delete on the destination, and conflicts are surfaced to the caller rather
|
||||
# than silently resolved. See issue #858 for the full design rationale.
|
||||
|
||||
# push = local -> cloud, pull = cloud -> local.
|
||||
TransferDirection = Literal["push", "pull"]
|
||||
|
||||
# How a directional transfer treats files that differ on both sides. "fail" is
|
||||
# the safe default: the caller is expected to abort before any transfer runs.
|
||||
ConflictStrategy = Literal["fail", "keep-local", "keep-cloud", "keep-both"]
|
||||
|
||||
|
||||
@dataclass
|
||||
class TransferPlan:
|
||||
"""Classification of how local and cloud differ for a directional transfer.
|
||||
|
||||
Built from ``rclone check --combined``. Paths are relative to the project
|
||||
root. ``conflicts`` are files present on both sides with differing content —
|
||||
without a sync baseline (see #862) every divergence is a conflict, because
|
||||
we cannot tell a teammate's edit from a stale local copy.
|
||||
"""
|
||||
|
||||
new: list[str] = field(default_factory=list) # only on source → safe to bring over
|
||||
conflicts: list[str] = field(default_factory=list) # differ on both sides
|
||||
dest_only: list[str] = field(default_factory=list) # only on destination → left untouched
|
||||
errors: list[str] = field(default_factory=list) # rclone could not read/hash
|
||||
|
||||
|
||||
def _transfer_endpoints(project: SyncProject, bucket_name: str) -> tuple[str, str]:
|
||||
"""Return (local_path, remote_path) strings for a project's transfer.
|
||||
|
||||
Raises:
|
||||
RcloneError: If the project has no local_sync_path configured.
|
||||
"""
|
||||
if not project.local_sync_path:
|
||||
raise RcloneError(f"Project {project.name} has no local_sync_path configured")
|
||||
local_path = str(Path(project.local_sync_path).expanduser())
|
||||
remote_path = get_project_remote(project, bucket_name)
|
||||
return local_path, remote_path
|
||||
|
||||
|
||||
def _build_transfer_cmd(
|
||||
operation: str,
|
||||
source: str,
|
||||
dest: str,
|
||||
*,
|
||||
filter_path: Path,
|
||||
dry_run: bool,
|
||||
verbose: bool,
|
||||
extra_flags: tuple[str, ...] = (),
|
||||
) -> list[str]:
|
||||
"""Build an rclone sync/copy command with the shared Basic Memory flags.
|
||||
|
||||
All directional transfers share the same tail: Tigris consistency headers,
|
||||
the .bmignore filter, and --local-no-preallocate (a no-op when local is the
|
||||
source, required when local is the destination on pull — see rclone#6801).
|
||||
"""
|
||||
cmd = [
|
||||
"rclone",
|
||||
operation,
|
||||
source,
|
||||
dest,
|
||||
*TIGRIS_CONSISTENCY_HEADERS,
|
||||
"--filter-from",
|
||||
str(filter_path),
|
||||
# Prevent NUL byte padding on virtual filesystems (e.g. Google Drive File Stream)
|
||||
# See: rclone/rclone#6801
|
||||
"--local-no-preallocate",
|
||||
*extra_flags,
|
||||
]
|
||||
|
||||
if verbose:
|
||||
cmd.append("--verbose")
|
||||
else:
|
||||
cmd.append("--progress")
|
||||
|
||||
if dry_run:
|
||||
cmd.append("--dry-run")
|
||||
|
||||
return cmd
|
||||
|
||||
|
||||
def project_sync(
|
||||
@@ -219,24 +313,199 @@ def project_sync(
|
||||
remote_path = get_project_remote(project, bucket_name)
|
||||
filter_path = filter_path or get_bmignore_filter_path()
|
||||
|
||||
cmd = [
|
||||
"rclone",
|
||||
cmd = _build_transfer_cmd(
|
||||
"sync",
|
||||
str(local_path),
|
||||
remote_path,
|
||||
filter_path=filter_path,
|
||||
dry_run=dry_run,
|
||||
verbose=verbose,
|
||||
)
|
||||
|
||||
result = run(cmd, text=True)
|
||||
return result.returncode == 0
|
||||
|
||||
|
||||
def _parse_check_combined(output: str) -> TransferPlan:
|
||||
"""Parse ``rclone check --combined`` output into a TransferPlan.
|
||||
|
||||
rclone emits one prefixed line per path (src is the transfer source):
|
||||
``=`` identical, ``+`` only on src, ``-`` only on dst, ``*`` differ,
|
||||
``!`` error reading/hashing. We ignore identical files.
|
||||
"""
|
||||
plan = TransferPlan()
|
||||
for line in output.splitlines():
|
||||
symbol, _, path = line.partition(" ")
|
||||
path = path.strip()
|
||||
if not path:
|
||||
continue
|
||||
if symbol == "+":
|
||||
plan.new.append(path)
|
||||
elif symbol == "*":
|
||||
plan.conflicts.append(path)
|
||||
elif symbol == "-":
|
||||
plan.dest_only.append(path)
|
||||
elif symbol == "!":
|
||||
plan.errors.append(path)
|
||||
# "=" (identical) is intentionally dropped.
|
||||
return plan
|
||||
|
||||
|
||||
def project_diff(
|
||||
project: SyncProject,
|
||||
bucket_name: str,
|
||||
direction: TransferDirection,
|
||||
*,
|
||||
run: RunFunc = subprocess.run,
|
||||
is_installed: IsInstalledFunc = is_rclone_installed,
|
||||
filter_path: Path | None = None,
|
||||
) -> TransferPlan:
|
||||
"""Classify how local and cloud differ for a push/pull, without transferring.
|
||||
|
||||
Uses ``rclone check`` (content comparison) so the caller can surface
|
||||
conflicts before any data moves. The source side depends on direction:
|
||||
pull compares cloud→local, push compares local→cloud.
|
||||
|
||||
Raises:
|
||||
RcloneError: If project has no local_sync_path configured or rclone not installed
|
||||
"""
|
||||
check_rclone_installed(is_installed=is_installed)
|
||||
|
||||
local_path, remote_path = _transfer_endpoints(project, bucket_name)
|
||||
filter_path = filter_path or get_bmignore_filter_path()
|
||||
|
||||
# Source/dest order matters: rclone check reports "+" for files only on the
|
||||
# source, which is what we want to bring over.
|
||||
source, dest = (remote_path, local_path) if direction == "pull" else (local_path, remote_path)
|
||||
|
||||
cmd = [
|
||||
"rclone",
|
||||
"check",
|
||||
source,
|
||||
dest,
|
||||
*TIGRIS_CONSISTENCY_HEADERS,
|
||||
"--filter-from",
|
||||
str(filter_path),
|
||||
# Prevent NUL byte padding on virtual filesystems (e.g. Google Drive File Stream)
|
||||
# See: rclone/rclone#6801
|
||||
"--local-no-preallocate",
|
||||
"--combined",
|
||||
"-",
|
||||
]
|
||||
|
||||
# rclone check exits non-zero when files differ — that's expected here, so we
|
||||
# parse the combined listing rather than trusting the return code.
|
||||
result = run(cmd, capture_output=True, text=True)
|
||||
plan = _parse_check_combined(result.stdout)
|
||||
|
||||
# Trigger: non-zero exit AND the combined listing produced no entries at all.
|
||||
# Why: a difference always yields +/-/*/! lines, so an empty listing on a
|
||||
# non-zero exit means the check itself failed (auth, missing remote, network,
|
||||
# bad filter) rather than finding zero differences. Without this guard the
|
||||
# caller would see an empty plan, transfer nothing, and report success.
|
||||
# Outcome: fail fast with rclone's stderr instead of a silent no-op.
|
||||
if result.returncode != 0 and not (plan.new or plan.conflicts or plan.dest_only or plan.errors):
|
||||
detail = result.stderr.strip() or f"rclone check exited with code {result.returncode}"
|
||||
raise RcloneError(f"Failed to compare {project.name} with cloud: {detail}")
|
||||
|
||||
return plan
|
||||
|
||||
|
||||
def project_copy(
|
||||
project: SyncProject,
|
||||
bucket_name: str,
|
||||
direction: TransferDirection,
|
||||
*,
|
||||
overwrite: bool,
|
||||
dry_run: bool = False,
|
||||
verbose: bool = False,
|
||||
run: RunFunc = subprocess.run,
|
||||
is_installed: IsInstalledFunc = is_rclone_installed,
|
||||
filter_path: Path | None = None,
|
||||
) -> bool:
|
||||
"""Additive transfer via ``rclone copy`` — never deletes on the destination.
|
||||
|
||||
Trigger: ``overwrite=False`` adds ``--ignore-existing`` so files already on
|
||||
the destination are left as-is (used when the destination side wins a
|
||||
conflict, and for the no-conflict fast path).
|
||||
Why: keeps the loser's bytes intact unless the caller explicitly chose to
|
||||
overwrite, matching the "no surprises" contract.
|
||||
|
||||
Raises:
|
||||
RcloneError: If project has no local_sync_path configured or rclone not installed
|
||||
"""
|
||||
check_rclone_installed(is_installed=is_installed)
|
||||
|
||||
local_path, remote_path = _transfer_endpoints(project, bucket_name)
|
||||
filter_path = filter_path or get_bmignore_filter_path()
|
||||
|
||||
source, dest = (remote_path, local_path) if direction == "pull" else (local_path, remote_path)
|
||||
# Overwrite mode compares by checksum so the transfer decision matches
|
||||
# project_diff's content-based conflict detection (rclone check). Without
|
||||
# --checksum, copy's default size+modtime comparison could skip a file the
|
||||
# diff flagged as a conflict (same size, destination not older) — silently
|
||||
# ignoring the user's explicit keep-cloud/keep-local choice. New-only mode
|
||||
# uses --ignore-existing, which skips by existence so the comparison basis
|
||||
# does not matter.
|
||||
extra_flags = ("--checksum",) if overwrite else ("--ignore-existing",)
|
||||
|
||||
cmd = _build_transfer_cmd(
|
||||
"copy",
|
||||
source,
|
||||
dest,
|
||||
filter_path=filter_path,
|
||||
dry_run=dry_run,
|
||||
verbose=verbose,
|
||||
extra_flags=extra_flags,
|
||||
)
|
||||
|
||||
result = run(cmd, text=True)
|
||||
return result.returncode == 0
|
||||
|
||||
|
||||
def _conflict_copy_name(rel_path: str, suffix: str) -> str:
|
||||
"""Insert a ``.conflict-<suffix>`` marker before the extension of a rel path."""
|
||||
p = PurePosixPath(rel_path)
|
||||
return str(p.with_name(f"{p.stem}.conflict-{suffix}{p.suffix}"))
|
||||
|
||||
|
||||
def project_copy_file(
|
||||
project: SyncProject,
|
||||
bucket_name: str,
|
||||
direction: TransferDirection,
|
||||
source_rel_path: str,
|
||||
dest_rel_path: str,
|
||||
*,
|
||||
dry_run: bool = False,
|
||||
verbose: bool = False,
|
||||
run: RunFunc = subprocess.run,
|
||||
is_installed: IsInstalledFunc = is_rclone_installed,
|
||||
) -> bool:
|
||||
"""Copy a single file from source to destination under a (possibly renamed) path.
|
||||
|
||||
Used for the ``keep-both`` strategy: the incoming version is written beside
|
||||
the destination's own copy as ``name.conflict-<date>`` so nothing is lost.
|
||||
|
||||
Raises:
|
||||
RcloneError: If project has no local_sync_path configured or rclone not installed
|
||||
"""
|
||||
check_rclone_installed(is_installed=is_installed)
|
||||
|
||||
local_path, remote_path = _transfer_endpoints(project, bucket_name)
|
||||
source_root, dest_root = (
|
||||
(remote_path, local_path) if direction == "pull" else (local_path, remote_path)
|
||||
)
|
||||
|
||||
cmd = [
|
||||
"rclone",
|
||||
"copyto",
|
||||
f"{source_root}/{source_rel_path}",
|
||||
f"{dest_root}/{dest_rel_path}",
|
||||
*TIGRIS_CONSISTENCY_HEADERS,
|
||||
# Matches _build_transfer_cmd: on pull this writes the conflict copy to
|
||||
# the local filesystem, where this prevents NUL byte padding on virtual
|
||||
# filesystems (e.g. Google Drive File Stream). See rclone/rclone#6801.
|
||||
"--local-no-preallocate",
|
||||
]
|
||||
if verbose:
|
||||
cmd.append("--verbose")
|
||||
else:
|
||||
cmd.append("--progress")
|
||||
|
||||
if dry_run:
|
||||
cmd.append("--dry-run")
|
||||
|
||||
@@ -244,6 +513,72 @@ def project_sync(
|
||||
return result.returncode == 0
|
||||
|
||||
|
||||
def _strategy_overwrites_dest(direction: TransferDirection, strategy: ConflictStrategy) -> bool:
|
||||
"""True when the strategy lets the source side overwrite the destination.
|
||||
|
||||
The source side is cloud on pull, local on push. "keep-cloud" wins on pull,
|
||||
"keep-local" wins on push; otherwise the destination is preserved.
|
||||
"""
|
||||
if strategy == "keep-cloud":
|
||||
return direction == "pull"
|
||||
if strategy == "keep-local":
|
||||
return direction == "push"
|
||||
return False # "fail" (no conflicts) and "keep-both" never overwrite existing dest files
|
||||
|
||||
|
||||
def project_transfer(
|
||||
project: SyncProject,
|
||||
bucket_name: str,
|
||||
direction: TransferDirection,
|
||||
plan: TransferPlan,
|
||||
*,
|
||||
strategy: ConflictStrategy = "fail",
|
||||
conflict_suffix: str = "",
|
||||
dry_run: bool = False,
|
||||
verbose: bool = False,
|
||||
run: RunFunc = subprocess.run,
|
||||
is_installed: IsInstalledFunc = is_rclone_installed,
|
||||
filter_path: Path | None = None,
|
||||
) -> bool:
|
||||
"""Execute a directional transfer for the chosen conflict strategy.
|
||||
|
||||
Callers detect conflicts with ``project_diff`` first and abort when
|
||||
``strategy == "fail"`` and conflicts exist; this function assumes that gate
|
||||
has already passed and applies the resolution.
|
||||
"""
|
||||
# keep-both: preserve the destination's version and drop the incoming one
|
||||
# beside it as a conflict copy, then do an additive (new-only) pass.
|
||||
if strategy == "keep-both":
|
||||
for rel_path in plan.conflicts:
|
||||
dest_rel = _conflict_copy_name(rel_path, conflict_suffix)
|
||||
copied = project_copy_file(
|
||||
project,
|
||||
bucket_name,
|
||||
direction,
|
||||
rel_path,
|
||||
dest_rel,
|
||||
dry_run=dry_run,
|
||||
verbose=verbose,
|
||||
run=run,
|
||||
is_installed=is_installed,
|
||||
)
|
||||
if not copied:
|
||||
return False
|
||||
|
||||
overwrite = _strategy_overwrites_dest(direction, strategy)
|
||||
return project_copy(
|
||||
project,
|
||||
bucket_name,
|
||||
direction,
|
||||
overwrite=overwrite,
|
||||
dry_run=dry_run,
|
||||
verbose=verbose,
|
||||
run=run,
|
||||
is_installed=is_installed,
|
||||
filter_path=filter_path,
|
||||
)
|
||||
|
||||
|
||||
def project_bisync(
|
||||
project: SyncProject,
|
||||
bucket_name: str,
|
||||
|
||||
@@ -1,11 +1,14 @@
|
||||
"""rclone configuration management for Basic Memory Cloud.
|
||||
|
||||
This module provides simplified rclone configuration for SPEC-20.
|
||||
Uses a single "basic-memory-cloud" remote for all operations.
|
||||
This module owns rclone remote configuration and naming. The default tenant uses
|
||||
the "basic-memory-cloud" remote (from SPEC-20); non-default/team workspaces each
|
||||
get their own tenant-scoped remote via remote_name_for_workspace (see #919),
|
||||
since Tigris credentials are bucket-scoped.
|
||||
"""
|
||||
|
||||
import configparser
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
@@ -61,25 +64,65 @@ def save_rclone_config(config: configparser.ConfigParser) -> None:
|
||||
console.print(f"[dim]Updated rclone config: {config_path}[/dim]")
|
||||
|
||||
|
||||
# The default remote serves the account's default tenant (back-compat with SPEC-20,
|
||||
# which used a single "basic-memory-cloud" remote). Non-default workspaces each get
|
||||
# their own remote, since Tigris credentials are bucket/tenant-scoped (see #919).
|
||||
DEFAULT_RCLONE_REMOTE = "basic-memory-cloud"
|
||||
|
||||
|
||||
# rclone remote section names allow letters, digits, hyphens, and underscores.
|
||||
# The slug comes from the cloud API (a trust boundary), so validate it before
|
||||
# splicing it into a remote name to avoid a broken/unusable rclone.conf section.
|
||||
_SAFE_SLUG = re.compile(r"^[A-Za-z0-9_-]+$")
|
||||
|
||||
|
||||
def remote_name_for_workspace(slug: str | None, *, is_default: bool) -> str:
|
||||
"""Return the rclone remote name for a workspace.
|
||||
|
||||
The default workspace keeps the legacy ``basic-memory-cloud`` remote so
|
||||
existing setups keep working; other workspaces get ``basic-memory-cloud-<slug>``.
|
||||
|
||||
Raises:
|
||||
RcloneConfigError: If a non-default workspace slug contains characters
|
||||
that are not valid in an rclone remote name.
|
||||
"""
|
||||
if is_default or not slug:
|
||||
return DEFAULT_RCLONE_REMOTE
|
||||
if not _SAFE_SLUG.match(slug):
|
||||
raise RcloneConfigError(
|
||||
f"Workspace slug '{slug}' cannot be used as an rclone remote name "
|
||||
"(allowed: letters, digits, hyphens, underscores)."
|
||||
)
|
||||
return f"{DEFAULT_RCLONE_REMOTE}-{slug}"
|
||||
|
||||
|
||||
def rclone_remote_exists(remote_name: str) -> bool:
|
||||
"""Return whether an rclone remote section is already configured."""
|
||||
return load_rclone_config().has_section(remote_name)
|
||||
|
||||
|
||||
def configure_rclone_remote(
|
||||
access_key: str,
|
||||
secret_key: str,
|
||||
endpoint: str = "https://fly.storage.tigris.dev",
|
||||
region: str = "auto",
|
||||
remote_name: str = DEFAULT_RCLONE_REMOTE,
|
||||
) -> str:
|
||||
"""Configure single rclone remote named 'basic-memory-cloud'.
|
||||
"""Configure an rclone remote for one tenant's bucket.
|
||||
|
||||
This is the simplified approach from SPEC-20 that uses one remote
|
||||
for all Basic Memory cloud operations (not tenant-specific).
|
||||
Each tenant (personal or team) has its own bucket-scoped credentials, so a
|
||||
remote maps 1:1 to a tenant. The default tenant keeps ``basic-memory-cloud``;
|
||||
other workspaces pass ``remote_name`` from :func:`remote_name_for_workspace`.
|
||||
|
||||
Args:
|
||||
access_key: S3 access key ID
|
||||
secret_key: S3 secret access key
|
||||
endpoint: S3-compatible endpoint URL
|
||||
region: S3 region (default: auto)
|
||||
remote_name: rclone remote section name to write
|
||||
|
||||
Returns:
|
||||
The remote name: "basic-memory-cloud"
|
||||
The remote name that was configured
|
||||
"""
|
||||
# Backup existing config
|
||||
backup_rclone_config()
|
||||
@@ -87,24 +130,21 @@ def configure_rclone_remote(
|
||||
# Load existing config
|
||||
config = load_rclone_config()
|
||||
|
||||
# Single remote name (not tenant-specific)
|
||||
REMOTE_NAME = "basic-memory-cloud"
|
||||
|
||||
# Add/update the remote section
|
||||
if not config.has_section(REMOTE_NAME):
|
||||
config.add_section(REMOTE_NAME)
|
||||
if not config.has_section(remote_name):
|
||||
config.add_section(remote_name)
|
||||
|
||||
config.set(REMOTE_NAME, "type", "s3")
|
||||
config.set(REMOTE_NAME, "provider", "Other")
|
||||
config.set(REMOTE_NAME, "access_key_id", access_key)
|
||||
config.set(REMOTE_NAME, "secret_access_key", secret_key)
|
||||
config.set(REMOTE_NAME, "endpoint", endpoint)
|
||||
config.set(REMOTE_NAME, "region", region)
|
||||
config.set(remote_name, "type", "s3")
|
||||
config.set(remote_name, "provider", "Other")
|
||||
config.set(remote_name, "access_key_id", access_key)
|
||||
config.set(remote_name, "secret_access_key", secret_key)
|
||||
config.set(remote_name, "endpoint", endpoint)
|
||||
config.set(remote_name, "region", region)
|
||||
# Prevent unnecessary encoding of filenames (only encode slashes and invalid UTF-8)
|
||||
# This prevents files with spaces like "Hello World.md" from being quoted
|
||||
config.set(REMOTE_NAME, "encoding", "Slash,InvalidUtf8")
|
||||
config.set(remote_name, "encoding", "Slash,InvalidUtf8")
|
||||
# Save updated config
|
||||
save_rclone_config(config)
|
||||
|
||||
console.print(f"[green]Configured rclone remote: {REMOTE_NAME}[/green]")
|
||||
return REMOTE_NAME
|
||||
console.print(f"[green]Configured rclone remote: {remote_name}[/green]")
|
||||
return remote_name
|
||||
|
||||
@@ -0,0 +1,533 @@
|
||||
"""Public share CLI commands for Basic Memory Cloud.
|
||||
|
||||
Surfaces the cloud `/api/shares` endpoints so users can manage public share
|
||||
links for notes without leaving the terminal:
|
||||
|
||||
- POST /api/shares -> create
|
||||
- GET /api/shares -> list
|
||||
- PATCH /api/shares/{token} -> update (enable/disable, set expiration)
|
||||
- DELETE /api/shares/{token} -> revoke
|
||||
|
||||
Auth, config lookup, and error handling reuse the shared `make_api_request()`
|
||||
helper, matching the `snapshot.py` command group.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
from datetime import datetime
|
||||
from typing import Optional
|
||||
from urllib.parse import urlencode
|
||||
from uuid import UUID
|
||||
|
||||
import typer
|
||||
from rich.console import Console
|
||||
from rich.table import Table
|
||||
|
||||
from basic_memory.cli.commands.cloud.api_client import (
|
||||
CloudAPIError,
|
||||
SubscriptionRequiredError,
|
||||
make_api_request,
|
||||
)
|
||||
from basic_memory.config import ConfigManager
|
||||
from basic_memory.mcp.async_client import resolve_configured_workspace
|
||||
from basic_memory.schemas.cloud import WorkspaceInfo
|
||||
|
||||
console = Console()
|
||||
share_app = typer.Typer(help="Manage public share links for notes")
|
||||
|
||||
# Header the cloud uses to route a request to a specific tenant's workspace.
|
||||
# Mirrors basic_memory.cli.commands.cloud.cloud_utils._workspace_headers: the
|
||||
# cloud /api/shares endpoints resolve the workspace from X-Workspace-ID (see
|
||||
# resolve_workspace in basic-memory-cloud deps.py), so without it a team
|
||||
# workspace project would be evaluated against the caller's default tenant.
|
||||
WORKSPACE_ID_HEADER = "X-Workspace-ID"
|
||||
|
||||
|
||||
def _is_uuid(value: str) -> bool:
|
||||
"""Return True when value parses as a UUID in any standard textual form."""
|
||||
try:
|
||||
UUID(value)
|
||||
except ValueError:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _match_workspace_identifier(
|
||||
workspaces: list[WorkspaceInfo], identifier: str
|
||||
) -> Optional[WorkspaceInfo]:
|
||||
"""Match a human workspace identifier with slug > tenant_id > name precedence.
|
||||
|
||||
Mirrors project_context._match_workspace_identifier (PR #979): try the stable
|
||||
slug first (case-insensitive), then the exact tenant_id, then the display name
|
||||
(case-insensitive). The first tier that yields any match wins, so a display
|
||||
name colliding with another workspace's slug never shadows the slug match.
|
||||
"""
|
||||
slug_matches = [ws for ws in workspaces if ws.slug.casefold() == identifier.casefold()]
|
||||
if slug_matches:
|
||||
return slug_matches[0] if len(slug_matches) == 1 else _ambiguous(slug_matches, identifier)
|
||||
|
||||
tenant_matches = [ws for ws in workspaces if ws.tenant_id == identifier]
|
||||
if tenant_matches:
|
||||
return tenant_matches[0]
|
||||
|
||||
name_matches = [ws for ws in workspaces if ws.name.casefold() == identifier.casefold()]
|
||||
if name_matches:
|
||||
return name_matches[0] if len(name_matches) == 1 else _ambiguous(name_matches, identifier)
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def _ambiguous(matches: list[WorkspaceInfo], identifier: str) -> WorkspaceInfo:
|
||||
"""Fail with a clear, copyable error when an identifier matches >1 workspace.
|
||||
|
||||
Trigger: a display name (or slug, defensively) resolves to multiple workspaces.
|
||||
Why: silently picking one would route a share to the wrong tenant.
|
||||
Outcome: list candidate slugs/tenant_ids and exit non-zero so the user re-runs
|
||||
with an unambiguous slug or tenant_id.
|
||||
"""
|
||||
candidates = "\n".join(f" - {ws.slug} (tenant_id: {ws.tenant_id})" for ws in matches)
|
||||
console.print(
|
||||
f"[red]Workspace '{identifier}' is ambiguous; it matches multiple workspaces.[/red]\n"
|
||||
"[yellow]Re-run with a unique workspace slug or tenant_id:[/yellow]\n"
|
||||
f"{candidates}"
|
||||
)
|
||||
raise typer.Exit(1)
|
||||
|
||||
|
||||
async def _resolve_workspace_to_tenant_id(identifier: str) -> str:
|
||||
"""Resolve a human workspace identifier (slug/name/tenant_id) to a tenant UUID.
|
||||
|
||||
Constraint: the cloud's X-Workspace-ID resolver only accepts a workspace/tenant
|
||||
UUID, but users see slugs and display names (in list-workspaces output and
|
||||
memory:// URLs). So when --workspace (or the configured default) is not already
|
||||
a UUID, fetch the caller's workspaces once and map it to the tenant_id here.
|
||||
|
||||
get_available_workspaces is the same workspace-fetch seam project.py uses for
|
||||
CLI workspace resolution; it is awaited directly here because the share
|
||||
commands already run inside their own event loop.
|
||||
"""
|
||||
from basic_memory.mcp.project_context import get_available_workspaces
|
||||
|
||||
workspaces = await get_available_workspaces()
|
||||
match = _match_workspace_identifier(workspaces, identifier)
|
||||
if match is None:
|
||||
available = "\n".join(f" - {ws.slug}" for ws in workspaces)
|
||||
console.print(
|
||||
f"[red]Workspace '{identifier}' was not found.[/red]\n"
|
||||
"[yellow]Use one of these workspace slugs (or a tenant_id):[/yellow]\n"
|
||||
f"{available}"
|
||||
)
|
||||
raise typer.Exit(1)
|
||||
return match.tenant_id
|
||||
|
||||
|
||||
async def _workspace_headers(
|
||||
*,
|
||||
project_name: Optional[str] = None,
|
||||
workspace: Optional[str] = None,
|
||||
) -> dict[str, str]:
|
||||
"""Resolve the target workspace and build the routing header, if any.
|
||||
|
||||
Resolution chain (see resolve_configured_workspace): explicit --workspace,
|
||||
then the project's configured workspace_id, then the global default. Returns
|
||||
an empty dict when nothing resolves so the request falls back to the
|
||||
caller's default tenant exactly as before.
|
||||
|
||||
The cloud's X-Workspace-ID resolver only accepts a workspace/tenant UUID. A
|
||||
UUID is forwarded verbatim (covers per-project config workspace_id values and
|
||||
the default chain, zero extra API calls); any other value is treated as a
|
||||
human identifier and resolved to the tenant UUID via one workspace lookup.
|
||||
"""
|
||||
resolved = resolve_configured_workspace(project_name=project_name, workspace=workspace)
|
||||
if resolved is None:
|
||||
return {}
|
||||
if _is_uuid(resolved):
|
||||
return {WORKSPACE_ID_HEADER: resolved}
|
||||
tenant_id = await _resolve_workspace_to_tenant_id(resolved)
|
||||
return {WORKSPACE_ID_HEADER: tenant_id}
|
||||
|
||||
|
||||
def _format_timestamp(iso_timestamp: Optional[str]) -> str:
|
||||
"""Format an ISO timestamp to a human-readable form, or '-' when absent."""
|
||||
if not iso_timestamp:
|
||||
return "-"
|
||||
try:
|
||||
dt = datetime.fromisoformat(iso_timestamp.replace("Z", "+00:00"))
|
||||
return dt.strftime("%Y-%m-%d %H:%M:%S")
|
||||
except (ValueError, AttributeError):
|
||||
return iso_timestamp
|
||||
|
||||
|
||||
def _parse_expires_at(value: str) -> str:
|
||||
"""Validate an --expires-at value and normalize it to an ISO 8601 string.
|
||||
|
||||
Accepts either a full ISO timestamp ("2025-12-31T23:59:00") or a bare date
|
||||
("2025-12-31"). Exits with a clear error on anything we can't parse so the
|
||||
server never sees a malformed payload.
|
||||
"""
|
||||
try:
|
||||
dt = datetime.fromisoformat(value)
|
||||
except ValueError:
|
||||
console.print(
|
||||
f"[red]Invalid --expires-at value '{value}'. "
|
||||
"Use ISO format, e.g. 2025-12-31 or 2025-12-31T23:59:00.[/red]"
|
||||
)
|
||||
raise typer.Exit(1)
|
||||
return dt.isoformat()
|
||||
|
||||
|
||||
def _print_share_details(data: dict) -> None:
|
||||
"""Print a single share's fields in the snapshot-style detail layout."""
|
||||
console.print(f" Token: {data.get('token', 'unknown')}")
|
||||
console.print(f" URL: [blue underline]{data.get('share_url', '-')}[/blue underline]")
|
||||
console.print(f" Project: {data.get('project_name', '-')}")
|
||||
console.print(f" Note: {data.get('note_permalink', '-')}")
|
||||
console.print(f" Enabled: {'yes' if data.get('enabled', False) else 'no'}")
|
||||
console.print(f" Expires: {_format_timestamp(data.get('expires_at'))}")
|
||||
console.print(f" Views: {data.get('view_count', 0)}")
|
||||
console.print(f" Created: {_format_timestamp(data.get('created_at'))}")
|
||||
|
||||
|
||||
@share_app.command("create")
|
||||
def create(
|
||||
project: str = typer.Argument(
|
||||
...,
|
||||
help="Name of the project the note belongs to",
|
||||
),
|
||||
permalink: str = typer.Argument(
|
||||
...,
|
||||
help="Permalink of the note to share",
|
||||
),
|
||||
expires_at: Optional[str] = typer.Option(
|
||||
None,
|
||||
"--expires-at",
|
||||
"-e",
|
||||
help="Optional expiration date/time (ISO 8601, e.g. 2025-12-31)",
|
||||
),
|
||||
workspace: Optional[str] = typer.Option(
|
||||
None,
|
||||
"--workspace",
|
||||
help="Workspace to route to: a workspace slug, display name, or tenant ID",
|
||||
),
|
||||
) -> None:
|
||||
"""Create a public share link for a note.
|
||||
|
||||
Examples:
|
||||
bm cloud share create my-project notes/my-idea
|
||||
bm cloud share create my-project notes/my-idea --expires-at 2025-12-31
|
||||
bm cloud share create my-project notes/my-idea --workspace acme
|
||||
"""
|
||||
|
||||
# Validate --expires-at before any async/API work so a parse error surfaces
|
||||
# a single clean message and exits, rather than being re-wrapped by the broad
|
||||
# handler below as "Unexpected error: 1" (typer.Exit subclasses Exception).
|
||||
payload: dict = {
|
||||
"project_name": project,
|
||||
"note_permalink": permalink,
|
||||
}
|
||||
if expires_at is not None:
|
||||
payload["expires_at"] = _parse_expires_at(expires_at)
|
||||
|
||||
async def _create():
|
||||
try:
|
||||
config_manager = ConfigManager()
|
||||
config = config_manager.config
|
||||
host_url = config.cloud_host.rstrip("/")
|
||||
|
||||
console.print("[blue]Creating share link...[/blue]")
|
||||
|
||||
response = await make_api_request(
|
||||
method="POST",
|
||||
url=f"{host_url}/api/shares",
|
||||
json_data=payload,
|
||||
headers=await _workspace_headers(project_name=project, workspace=workspace),
|
||||
)
|
||||
|
||||
data = response.json()
|
||||
|
||||
console.print("[green]Share link created successfully[/green]")
|
||||
_print_share_details(data)
|
||||
|
||||
except typer.Exit:
|
||||
raise
|
||||
except SubscriptionRequiredError as e:
|
||||
console.print("\n[red]Subscription Required[/red]\n")
|
||||
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
|
||||
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
|
||||
raise typer.Exit(1)
|
||||
except CloudAPIError as e:
|
||||
if e.status_code == 404:
|
||||
console.print(f"[red]Note not found: {permalink} (project: {project})[/red]")
|
||||
else:
|
||||
console.print(f"[red]Failed to create share link: {e}[/red]")
|
||||
raise typer.Exit(1)
|
||||
except Exception as e:
|
||||
console.print(f"[red]Unexpected error: {e}[/red]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
asyncio.run(_create())
|
||||
|
||||
|
||||
@share_app.command("list")
|
||||
def list_shares(
|
||||
project: Optional[str] = typer.Option(
|
||||
None,
|
||||
"--project",
|
||||
"-p",
|
||||
help="Filter shares by project name",
|
||||
),
|
||||
workspace: Optional[str] = typer.Option(
|
||||
None,
|
||||
"--workspace",
|
||||
help="Workspace to route to: a workspace slug, display name, or tenant ID",
|
||||
),
|
||||
) -> None:
|
||||
"""List public share links.
|
||||
|
||||
Examples:
|
||||
bm cloud share list
|
||||
bm cloud share list --project my-project
|
||||
bm cloud share list --project my-project --workspace acme
|
||||
"""
|
||||
|
||||
async def _list():
|
||||
try:
|
||||
config_manager = ConfigManager()
|
||||
config = config_manager.config
|
||||
host_url = config.cloud_host.rstrip("/")
|
||||
|
||||
url = f"{host_url}/api/shares"
|
||||
if project:
|
||||
# Encode the filter so project names with query-reserved
|
||||
# characters (&, +, #, spaces) reach the server intact rather
|
||||
# than being parsed as extra query parameters.
|
||||
url += f"?{urlencode({'project_name': project})}"
|
||||
|
||||
console.print("[blue]Fetching share links...[/blue]")
|
||||
|
||||
response = await make_api_request(
|
||||
method="GET",
|
||||
url=url,
|
||||
headers=await _workspace_headers(project_name=project, workspace=workspace),
|
||||
)
|
||||
|
||||
data = response.json()
|
||||
shares = data.get("shares", [])
|
||||
total = data.get("total", len(shares))
|
||||
|
||||
if not shares:
|
||||
console.print("[yellow]No share links found[/yellow]")
|
||||
console.print(
|
||||
"\n[dim]Create a share with: bm cloud share create <project> <permalink>[/dim]"
|
||||
)
|
||||
return
|
||||
|
||||
table = Table(title=f"Public Shares ({total} total)")
|
||||
table.add_column("Token", style="cyan", no_wrap=True)
|
||||
table.add_column("Project", style="yellow")
|
||||
table.add_column("Note", style="white")
|
||||
table.add_column("Enabled", style="green")
|
||||
table.add_column("Expires", style="green")
|
||||
table.add_column("Views", style="magenta", justify="right")
|
||||
table.add_column("URL", style="blue", overflow="fold")
|
||||
|
||||
for share in shares:
|
||||
table.add_row(
|
||||
share.get("token", "unknown"),
|
||||
share.get("project_name", "-"),
|
||||
share.get("note_permalink", "-"),
|
||||
"yes" if share.get("enabled", False) else "no",
|
||||
_format_timestamp(share.get("expires_at")),
|
||||
str(share.get("view_count", 0)),
|
||||
share.get("share_url", "-"),
|
||||
)
|
||||
|
||||
console.print(table)
|
||||
|
||||
# Re-raise typer.Exit before the broad handler below: workspace resolution
|
||||
# raises typer.Exit (a subclass of Exception) for ambiguous/unknown
|
||||
# identifiers, and that must not be re-wrapped as "Unexpected error: 1".
|
||||
except typer.Exit:
|
||||
raise
|
||||
except SubscriptionRequiredError as e:
|
||||
console.print("\n[red]Subscription Required[/red]\n")
|
||||
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
|
||||
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
|
||||
raise typer.Exit(1)
|
||||
except CloudAPIError as e:
|
||||
console.print(f"[red]Failed to list share links: {e}[/red]")
|
||||
raise typer.Exit(1)
|
||||
except Exception as e:
|
||||
console.print(f"[red]Unexpected error: {e}[/red]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
asyncio.run(_list())
|
||||
|
||||
|
||||
@share_app.command("update")
|
||||
def update(
|
||||
token: str = typer.Argument(
|
||||
...,
|
||||
help="The token of the share to update",
|
||||
),
|
||||
enable: bool = typer.Option(
|
||||
False,
|
||||
"--enable",
|
||||
help="Enable the share link",
|
||||
),
|
||||
disable: bool = typer.Option(
|
||||
False,
|
||||
"--disable",
|
||||
help="Disable the share link without deleting it",
|
||||
),
|
||||
expires_at: Optional[str] = typer.Option(
|
||||
None,
|
||||
"--expires-at",
|
||||
"-e",
|
||||
help="New expiration date/time (ISO 8601). Use 'none' to clear it.",
|
||||
),
|
||||
workspace: Optional[str] = typer.Option(
|
||||
None,
|
||||
"--workspace",
|
||||
help="Workspace the share belongs to: a workspace slug, display name, or tenant ID",
|
||||
),
|
||||
) -> None:
|
||||
"""Update a share link: enable/disable it or change its expiration.
|
||||
|
||||
Examples:
|
||||
bm cloud share update abc123 --disable
|
||||
bm cloud share update abc123 --enable
|
||||
bm cloud share update abc123 --expires-at 2026-01-01
|
||||
bm cloud share update abc123 --expires-at none
|
||||
bm cloud share update abc123 --disable --workspace acme
|
||||
"""
|
||||
|
||||
async def _update():
|
||||
try:
|
||||
# --- Validate flags ---
|
||||
# Trigger: both toggles passed, or neither toggle and no expiry change.
|
||||
# Why: PATCH needs at least one concrete field, and enable/disable
|
||||
# conflict; reject up front so we don't send an empty/ambiguous body.
|
||||
if enable and disable:
|
||||
console.print("[red]Cannot use --enable and --disable together[/red]")
|
||||
raise typer.Exit(1)
|
||||
if not enable and not disable and expires_at is None:
|
||||
console.print(
|
||||
"[red]Nothing to update. Pass --enable, --disable, or --expires-at.[/red]"
|
||||
)
|
||||
raise typer.Exit(1)
|
||||
|
||||
payload: dict = {}
|
||||
if enable:
|
||||
payload["enabled"] = True
|
||||
if disable:
|
||||
payload["enabled"] = False
|
||||
if expires_at is not None:
|
||||
# "none" clears the expiration; anything else is parsed as a date.
|
||||
payload["expires_at"] = (
|
||||
None if expires_at.lower() == "none" else _parse_expires_at(expires_at)
|
||||
)
|
||||
|
||||
config_manager = ConfigManager()
|
||||
config = config_manager.config
|
||||
host_url = config.cloud_host.rstrip("/")
|
||||
|
||||
console.print("[blue]Updating share link...[/blue]")
|
||||
|
||||
response = await make_api_request(
|
||||
method="PATCH",
|
||||
url=f"{host_url}/api/shares/{token}",
|
||||
json_data=payload,
|
||||
headers=await _workspace_headers(workspace=workspace),
|
||||
)
|
||||
|
||||
data = response.json()
|
||||
|
||||
console.print("[green]Share link updated successfully[/green]")
|
||||
_print_share_details(data)
|
||||
|
||||
except typer.Exit:
|
||||
raise
|
||||
except SubscriptionRequiredError as e:
|
||||
console.print("\n[red]Subscription Required[/red]\n")
|
||||
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
|
||||
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
|
||||
raise typer.Exit(1)
|
||||
except CloudAPIError as e:
|
||||
if e.status_code == 404:
|
||||
console.print(f"[red]Share not found: {token}[/red]")
|
||||
else:
|
||||
console.print(f"[red]Failed to update share link: {e}[/red]")
|
||||
raise typer.Exit(1)
|
||||
except Exception as e:
|
||||
console.print(f"[red]Unexpected error: {e}[/red]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
asyncio.run(_update())
|
||||
|
||||
|
||||
@share_app.command("revoke")
|
||||
def revoke(
|
||||
token: str = typer.Argument(
|
||||
...,
|
||||
help="The token of the share to revoke",
|
||||
),
|
||||
force: bool = typer.Option(
|
||||
False,
|
||||
"--force",
|
||||
"-f",
|
||||
help="Skip confirmation prompt",
|
||||
),
|
||||
workspace: Optional[str] = typer.Option(
|
||||
None,
|
||||
"--workspace",
|
||||
help="Workspace the share belongs to: a workspace slug, display name, or tenant ID",
|
||||
),
|
||||
) -> None:
|
||||
"""Revoke (delete) a public share link.
|
||||
|
||||
Examples:
|
||||
bm cloud share revoke abc123
|
||||
bm cloud share revoke abc123 --force
|
||||
bm cloud share revoke abc123 --force --workspace acme
|
||||
"""
|
||||
|
||||
async def _revoke():
|
||||
try:
|
||||
config_manager = ConfigManager()
|
||||
config = config_manager.config
|
||||
host_url = config.cloud_host.rstrip("/")
|
||||
|
||||
if not force:
|
||||
confirmed = typer.confirm(f"Are you sure you want to revoke share '{token}'?")
|
||||
if not confirmed:
|
||||
console.print("[yellow]Revocation cancelled[/yellow]")
|
||||
raise typer.Exit(0)
|
||||
|
||||
console.print("[blue]Revoking share link...[/blue]")
|
||||
|
||||
await make_api_request(
|
||||
method="DELETE",
|
||||
url=f"{host_url}/api/shares/{token}",
|
||||
headers=await _workspace_headers(workspace=workspace),
|
||||
)
|
||||
|
||||
console.print(f"[green]Share {token} revoked successfully[/green]")
|
||||
|
||||
except typer.Exit:
|
||||
raise
|
||||
except SubscriptionRequiredError as e:
|
||||
console.print("\n[red]Subscription Required[/red]\n")
|
||||
console.print(f"[yellow]{e.args[0]}[/yellow]\n")
|
||||
console.print(f"Subscribe at: [blue underline]{e.subscribe_url}[/blue underline]\n")
|
||||
raise typer.Exit(1)
|
||||
except CloudAPIError as e:
|
||||
if e.status_code == 404:
|
||||
console.print(f"[red]Share not found: {token}[/red]")
|
||||
else:
|
||||
console.print(f"[red]Failed to revoke share link: {e}[/red]")
|
||||
raise typer.Exit(1)
|
||||
except Exception as e:
|
||||
console.print(f"[red]Unexpected error: {e}[/red]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
asyncio.run(_revoke())
|
||||
@@ -98,6 +98,18 @@ def mcp(
|
||||
if transport == "stdio":
|
||||
threading.Thread(target=_run_background_auto_update, daemon=True).start()
|
||||
|
||||
# Trigger: MCP server startup on the Postgres backend, before the transport
|
||||
# creates its event loop.
|
||||
# Why: the watcher/lifespan path runs startup migrations + engine.dispose() on
|
||||
# asyncpg, which races stdlib asyncio teardown and crashes the container loop
|
||||
# (#831/#877). uvloop's C scheduler structurally avoids that race and must own
|
||||
# the loop policy before the loop is created. No-op for SQLite.
|
||||
# Outcome: `basic-memory mcp` on Postgres runs on uvloop. (The CLI callback also
|
||||
# installs it; this keeps the server startup seam explicit and self-contained.)
|
||||
from basic_memory.db import maybe_install_uvloop
|
||||
|
||||
maybe_install_uvloop(ConfigManager().config)
|
||||
|
||||
# Run the MCP server (blocks)
|
||||
# Lifespan handles: initialization, migrations, file sync, cleanup
|
||||
logger.info(f"Starting MCP server with {transport.upper()} transport")
|
||||
|
||||
@@ -1,8 +1,9 @@
|
||||
"""Status command for basic-memory CLI."""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from typing import Set, Dict
|
||||
from typing import Annotated, Optional
|
||||
import time
|
||||
from typing import Annotated, Dict, Optional, Set
|
||||
|
||||
from mcp.server.fastmcp.exceptions import ToolError
|
||||
import typer
|
||||
@@ -142,20 +143,58 @@ def display_changes(
|
||||
console.print(Panel(tree, expand=False))
|
||||
|
||||
|
||||
class StatusTimeout(Exception):
|
||||
"""Raised when --wait does not reach a synced state before the deadline."""
|
||||
|
||||
|
||||
async def run_status(
|
||||
project: Optional[str] = None,
|
||||
wait: bool = False,
|
||||
timeout: float = 30.0,
|
||||
poll_interval: float = 0.5,
|
||||
) -> tuple[str, SyncReportResponse]:
|
||||
"""Fetch sync status of files vs database.
|
||||
|
||||
When ``wait`` is False this performs a single live disk-vs-DB scan and
|
||||
returns immediately. When ``wait`` is True it polls until the project has
|
||||
no pending changes (``sync_report.total == 0``) or the timeout elapses.
|
||||
|
||||
Returns (project_name, sync_report) for the caller to render.
|
||||
|
||||
Raises:
|
||||
StatusTimeout: If ``wait`` is True and the deadline passes before the
|
||||
project reaches a synced state.
|
||||
"""
|
||||
# Resolve default project so get_client() can route per-project
|
||||
project = project or ConfigManager().default_project
|
||||
|
||||
# Reuse a single client/context across polls so we don't reconnect each loop.
|
||||
async with get_client(project_name=project) as client:
|
||||
project_item = await get_active_project(client, project, None)
|
||||
sync_report = await ProjectClient(client).get_status(project_item.external_id)
|
||||
return project_item.name, sync_report
|
||||
project_client = ProjectClient(client)
|
||||
|
||||
# Trigger: caller did not request --wait
|
||||
# Why: preserve the original single-scan behavior for the common case
|
||||
# Outcome: one status scan, returned as-is
|
||||
if not wait:
|
||||
sync_report = await project_client.get_status(project_item.external_id)
|
||||
return project_item.name, sync_report
|
||||
|
||||
# Trigger: --wait requested
|
||||
# Why: callers (bulk imports, benchmarks, tests) need to block until the
|
||||
# index has caught up instead of polling externally
|
||||
# Outcome: poll get_status until total == 0 or the deadline is reached
|
||||
deadline = time.monotonic() + timeout
|
||||
while True:
|
||||
sync_report = await project_client.get_status(project_item.external_id)
|
||||
if sync_report.total == 0:
|
||||
return project_item.name, sync_report
|
||||
if time.monotonic() >= deadline:
|
||||
raise StatusTimeout(
|
||||
f"Timed out after {timeout:g}s waiting for '{project_item.name}' "
|
||||
f"to finish indexing ({sync_report.total} pending change(s) remaining)."
|
||||
)
|
||||
await asyncio.sleep(poll_interval)
|
||||
|
||||
|
||||
@app.command()
|
||||
@@ -166,6 +205,10 @@ def status(
|
||||
] = None,
|
||||
verbose: bool = typer.Option(False, "--verbose", "-v", help="Show detailed file information"),
|
||||
json_output: bool = typer.Option(False, "--json", help="Output in JSON format"),
|
||||
wait: bool = typer.Option(
|
||||
False, "--wait", help="Block until indexing is complete (no pending changes)"
|
||||
),
|
||||
timeout: float = typer.Option(30.0, "--timeout", help="Max seconds to wait when --wait is set"),
|
||||
local: bool = typer.Option(
|
||||
False, "--local", help="Force local API routing (ignore cloud mode)"
|
||||
),
|
||||
@@ -174,11 +217,21 @@ def status(
|
||||
"""Show sync status between files and database.
|
||||
|
||||
Use --json for machine-readable output.
|
||||
Use --wait to block until indexing is complete (e.g. after a bulk import);
|
||||
combine with --timeout to bound the wait. On timeout the command exits 1.
|
||||
Use --local to force local routing when cloud mode is enabled.
|
||||
Use --cloud to force cloud routing when cloud mode is disabled.
|
||||
"""
|
||||
from basic_memory.cli.commands.command_utils import run_with_cleanup
|
||||
|
||||
# Trigger: --wait with a negative --timeout
|
||||
# Why: a negative deadline times out on the very first poll, producing a confusing
|
||||
# "Timed out after -5s" message instead of flagging the bad input. Raised
|
||||
# before the try/except so typer renders a clean usage error (exit 2).
|
||||
# Outcome: reject it up front with a clear parameter error.
|
||||
if wait and timeout < 0:
|
||||
raise typer.BadParameter("--timeout must be >= 0", param_hint="'--timeout'")
|
||||
|
||||
try:
|
||||
validate_routing_flags(local, cloud)
|
||||
# Trigger: no explicit routing flag provided
|
||||
@@ -189,12 +242,24 @@ def status(
|
||||
if not local and not cloud:
|
||||
local = True
|
||||
with force_routing(local=local, cloud=cloud):
|
||||
project_name, sync_report = run_with_cleanup(run_status(project))
|
||||
project_name, sync_report = run_with_cleanup(
|
||||
run_status(project, wait=wait, timeout=timeout)
|
||||
)
|
||||
|
||||
if json_output:
|
||||
print(json.dumps(sync_report.model_dump(mode="json"), indent=2, default=str))
|
||||
else:
|
||||
display_changes(project_name, "Status", sync_report, verbose)
|
||||
except StatusTimeout as e:
|
||||
# Trigger: --wait deadline passed before the project finished indexing
|
||||
# Why: callers depend on exit code 1 to detect that indexing did not
|
||||
# complete in time, while still getting a clear machine/human message
|
||||
# Outcome: emit the timeout message (JSON-shaped under --json) and exit 1
|
||||
if json_output:
|
||||
print(json.dumps({"error": str(e)}, indent=2))
|
||||
else:
|
||||
console.print(f"[red]Error: {e}[/red]")
|
||||
raise typer.Exit(code=1)
|
||||
except (ValueError, ToolError) as e:
|
||||
if json_output:
|
||||
print(json.dumps({"error": str(e)}, indent=2))
|
||||
|
||||
@@ -15,6 +15,7 @@ from basic_memory.cli.app import app
|
||||
from basic_memory.cli.commands.command_utils import run_with_cleanup
|
||||
from basic_memory.cli.commands.routing import force_routing, validate_routing_flags
|
||||
from basic_memory.mcp.tools import build_context as mcp_build_context
|
||||
from basic_memory.mcp.tools import delete_note as mcp_delete_note
|
||||
from basic_memory.mcp.tools import edit_note as mcp_edit_note
|
||||
from basic_memory.mcp.tools import list_memory_projects as mcp_list_projects
|
||||
from basic_memory.mcp.tools import list_workspaces as mcp_list_workspaces
|
||||
@@ -40,6 +41,26 @@ def _print_json(result: Any) -> None:
|
||||
print(json.dumps(result, indent=2, ensure_ascii=True, default=str))
|
||||
|
||||
|
||||
def _delete_note_failure_message(result: dict[str, Any]) -> str | None:
|
||||
"""Return the CLI failure message for delete-note JSON results, if any."""
|
||||
error = result.get("error")
|
||||
if error:
|
||||
return str(error)
|
||||
|
||||
failed_deletes = result.get("failed_deletes")
|
||||
# Trigger: directory deletion can partially fail without raising from the service.
|
||||
# Why: cleanup scripts need a non-zero exit when files remain undeleted.
|
||||
# Outcome: the CLI fails even if older MCP JSON did not include an error field.
|
||||
if (
|
||||
result.get("is_directory") is True
|
||||
and isinstance(failed_deletes, int)
|
||||
and failed_deletes > 0
|
||||
):
|
||||
return f"Directory delete incomplete: {failed_deletes} file(s) failed"
|
||||
|
||||
return None
|
||||
|
||||
|
||||
# --- Commands ---
|
||||
|
||||
|
||||
@@ -56,6 +77,16 @@ def write_note(
|
||||
tags: Annotated[
|
||||
Optional[List[str]], typer.Option(help="A list of tags to apply to the note")
|
||||
] = None,
|
||||
note_type: Annotated[
|
||||
str,
|
||||
typer.Option(
|
||||
"--type",
|
||||
help=(
|
||||
"Note type stored in frontmatter (e.g. 'guide', 'report'). "
|
||||
"A 'type:' in the note's own content frontmatter takes precedence."
|
||||
),
|
||||
),
|
||||
] = "note",
|
||||
project: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(
|
||||
@@ -69,6 +100,11 @@ def write_note(
|
||||
help="Project external_id (UUID). Takes precedence over --project; use to disambiguate same-named projects across cloud workspaces.",
|
||||
),
|
||||
] = None,
|
||||
overwrite: bool = typer.Option(
|
||||
False,
|
||||
"--overwrite",
|
||||
help="Replace an existing note on conflict (matches MCP write_note overwrite=True)",
|
||||
),
|
||||
local: bool = typer.Option(
|
||||
False, "--local", help="Force local API routing (ignore cloud mode)"
|
||||
),
|
||||
@@ -79,7 +115,9 @@ def write_note(
|
||||
Examples:
|
||||
|
||||
bm tool write-note --title "My Note" --folder "notes" --content "Note content"
|
||||
bm tool write-note --title "My Guide" --folder "notes" --content "..." --type guide
|
||||
echo "content" | bm tool write-note --title "My Note" --folder "notes"
|
||||
bm tool write-note --title "My Note" --folder "notes" --overwrite
|
||||
bm tool write-note --title "My Note" --folder "notes" --local
|
||||
"""
|
||||
try:
|
||||
@@ -111,9 +149,23 @@ def write_note(
|
||||
project=project,
|
||||
project_id=project_id,
|
||||
tags=tags,
|
||||
note_type=note_type,
|
||||
overwrite=overwrite,
|
||||
output_format="json",
|
||||
)
|
||||
)
|
||||
|
||||
# MCP tool returns an error field on failure in JSON mode (e.g.
|
||||
# NOTE_ALREADY_EXISTS on a blocked overwrite, SECURITY_VALIDATION_ERROR).
|
||||
# Trigger: result carries a non-empty `error`.
|
||||
# Why: parity with delete-note/edit-note/search-notes so exit-code-driven
|
||||
# scripts detect a failed/blocked write instead of seeing exit 0.
|
||||
# Outcome: print the error to stderr and exit non-zero.
|
||||
if isinstance(result, dict) and result.get("error"):
|
||||
typer.echo(f"Error: {result['error']}", err=True)
|
||||
_print_json(result)
|
||||
raise typer.Exit(1)
|
||||
|
||||
_print_json(result)
|
||||
except ValueError as e:
|
||||
typer.echo(f"Error: {e}", err=True)
|
||||
@@ -167,6 +219,19 @@ def read_note(
|
||||
output_format="json",
|
||||
)
|
||||
)
|
||||
|
||||
# MCP tool returns an error field on failure in JSON mode (e.g.
|
||||
# SECURITY_VALIDATION_ERROR on a path-traversal identifier). A genuine
|
||||
# not-found returns null fields with no `error` key, so it still exits 0.
|
||||
# Trigger: result carries a non-empty `error`.
|
||||
# Why: parity with edit-note/delete-note/search-notes so a blocked read
|
||||
# surfaces a non-zero exit instead of looking like success.
|
||||
# Outcome: print the error to stderr and exit non-zero.
|
||||
if isinstance(result, dict) and result.get("error"):
|
||||
typer.echo(f"Error: {result['error']}", err=True)
|
||||
_print_json(result)
|
||||
raise typer.Exit(1)
|
||||
|
||||
_print_json(result)
|
||||
except ValueError as e:
|
||||
typer.echo(f"Error: {e}", err=True)
|
||||
@@ -178,6 +243,66 @@ def read_note(
|
||||
raise
|
||||
|
||||
|
||||
@tool_app.command("delete-note")
|
||||
def delete_note(
|
||||
identifier: str,
|
||||
is_directory: bool = typer.Option(
|
||||
False, "--is-directory", help="Delete a directory instead of a single note"
|
||||
),
|
||||
project: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(help="The project to use. If not provided, the default project will be used."),
|
||||
] = None,
|
||||
project_id: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(
|
||||
"--project-id",
|
||||
help="Project external_id (UUID). Takes precedence over --project; use to disambiguate same-named projects across cloud workspaces.",
|
||||
),
|
||||
] = None,
|
||||
local: bool = typer.Option(
|
||||
False, "--local", help="Force local API routing (ignore cloud mode)"
|
||||
),
|
||||
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
|
||||
) -> None:
|
||||
"""Delete a note or directory from the knowledge base.
|
||||
|
||||
Examples:
|
||||
|
||||
bm tool delete-note notes/old-draft
|
||||
bm tool delete-note docs/archive --is-directory
|
||||
"""
|
||||
try:
|
||||
validate_routing_flags(local, cloud)
|
||||
|
||||
with force_routing(local=local, cloud=cloud):
|
||||
result = run_with_cleanup(
|
||||
mcp_delete_note(
|
||||
identifier=identifier,
|
||||
is_directory=is_directory,
|
||||
project=project,
|
||||
project_id=project_id,
|
||||
output_format="json",
|
||||
)
|
||||
)
|
||||
|
||||
if isinstance(result, dict):
|
||||
failure_message = _delete_note_failure_message(result)
|
||||
if failure_message:
|
||||
typer.echo(f"Error: {failure_message}", err=True)
|
||||
raise typer.Exit(1)
|
||||
|
||||
_print_json(result)
|
||||
except ValueError as e:
|
||||
typer.echo(f"Error: {e}", err=True)
|
||||
raise typer.Exit(1)
|
||||
except Exception as e: # pragma: no cover
|
||||
if not isinstance(e, typer.Exit):
|
||||
typer.echo(f"Error during delete_note: {e}", err=True)
|
||||
raise typer.Exit(1)
|
||||
raise
|
||||
|
||||
|
||||
@tool_app.command()
|
||||
def edit_note(
|
||||
identifier: str,
|
||||
@@ -324,7 +449,9 @@ def recent_activity(
|
||||
"7d", "--timeframe", help="Timeframe filter (e.g., '7d', '1 week')"
|
||||
),
|
||||
page: int = typer.Option(1, "--page", help="Page number for pagination"),
|
||||
page_size: int = typer.Option(50, "--page-size", help="Number of results per page"),
|
||||
# Match the MCP recent_activity default (page_size=10) so identical default
|
||||
# invocations return the same number of rows from CLI and MCP.
|
||||
page_size: int = typer.Option(10, "--page-size", help="Number of results per page"),
|
||||
project: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(help="The project to use. If not provided, the default project will be used."),
|
||||
@@ -409,6 +536,16 @@ def search_notes(
|
||||
help="Filter by search item type: entity, observation, relation (repeatable)",
|
||||
),
|
||||
] = None,
|
||||
categories: Annotated[
|
||||
Optional[List[str]],
|
||||
typer.Option(
|
||||
"--category",
|
||||
help=(
|
||||
"Filter observation results to exact categories (repeatable); "
|
||||
"pair with --entity-type observation"
|
||||
),
|
||||
),
|
||||
] = None,
|
||||
meta: Annotated[
|
||||
Optional[List[str]],
|
||||
typer.Option("--meta", help="Filter by frontmatter key=value (repeatable)"),
|
||||
@@ -443,6 +580,7 @@ def search_notes(
|
||||
bm tool search-notes --permalink "specs/*"
|
||||
bm tool search-notes --tag python --tag async
|
||||
bm tool search-notes --meta status=draft
|
||||
bm tool search-notes "auth" --entity-type observation --category requirement
|
||||
"""
|
||||
try:
|
||||
validate_routing_flags(local, cloud)
|
||||
@@ -508,6 +646,7 @@ def search_notes(
|
||||
page_size=page_size,
|
||||
note_types=note_types,
|
||||
entity_types=entity_types,
|
||||
categories=categories,
|
||||
metadata_filters=metadata_filters,
|
||||
tags=tags,
|
||||
status=status,
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
"""Top-level `workspace` stub that redirects users to `bm cloud workspace`."""
|
||||
|
||||
import typer
|
||||
|
||||
from basic_memory.cli.app import app
|
||||
|
||||
|
||||
# Trigger: user runs `bm workspace`, `bm workspace list`, etc.
|
||||
# Why: workspace verbs live under `bm cloud workspace`; a bare top-level miss
|
||||
# only emits Typer's terse "No such command 'workspace'." with exit 2.
|
||||
# Outcome: allow_extra_args + ignore_unknown_options absorb any trailing tokens
|
||||
# (e.g. `list`, `set-default foo`) so every invocation reaches this body and
|
||||
# prints actionable guidance instead of being rejected as bad usage.
|
||||
@app.command(
|
||||
"workspace",
|
||||
context_settings={"allow_extra_args": True, "ignore_unknown_options": True},
|
||||
)
|
||||
def workspace_stub(ctx: typer.Context) -> None:
|
||||
"""Point users to the real workspace verbs under `bm cloud workspace`."""
|
||||
typer.echo("'bm workspace' is not a command. Workspace verbs live under 'bm cloud workspace':")
|
||||
typer.echo(" bm cloud workspace list")
|
||||
typer.echo(" bm cloud workspace set-default <name>")
|
||||
raise typer.Exit(1)
|
||||
@@ -31,6 +31,7 @@ if not _version_only_invocation(sys.argv[1:]):
|
||||
status,
|
||||
tool,
|
||||
update,
|
||||
workspace,
|
||||
)
|
||||
|
||||
warnings.filterwarnings("ignore") # pragma: no cover
|
||||
|
||||
@@ -248,7 +248,19 @@ class BasicMemoryConfig(BaseSettings):
|
||||
)
|
||||
semantic_embedding_dimensions: int | None = Field(
|
||||
default=None,
|
||||
description="Embedding vector dimensions. Auto-detected from provider if not set (384 for FastEmbed, 1536 for OpenAI).",
|
||||
description=(
|
||||
"Embedding vector dimensions. Uses provider defaults when unset "
|
||||
"(384 for FastEmbed, 1536 for OpenAI and LiteLLM OpenAI default); "
|
||||
"required for custom LiteLLM models."
|
||||
),
|
||||
)
|
||||
semantic_embedding_forward_dimensions: bool | None = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"LiteLLM-only override for sending semantic_embedding_dimensions as a "
|
||||
"provider-side output-size request parameter. When unset, Basic Memory "
|
||||
"auto-detects model strings such as text-embedding-3."
|
||||
),
|
||||
)
|
||||
# Trigger: full local rebuilds spend most of their time waiting behind shared
|
||||
# embed flushes, not constructing vectors themselves.
|
||||
@@ -266,6 +278,20 @@ class BasicMemoryConfig(BaseSettings):
|
||||
description="Maximum number of concurrent provider requests for batched embedding generation when the active provider supports request-level concurrency.",
|
||||
gt=0,
|
||||
)
|
||||
semantic_embedding_document_input_type: str | None = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"Optional LiteLLM input_type for indexed document/passages. "
|
||||
"Use with asymmetric embedding models such as Cohere or NVIDIA retrieval models."
|
||||
),
|
||||
)
|
||||
semantic_embedding_query_input_type: str | None = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"Optional LiteLLM input_type for search queries. "
|
||||
"Use with asymmetric embedding models such as Cohere or NVIDIA retrieval models."
|
||||
),
|
||||
)
|
||||
semantic_embedding_sync_batch_size: int = Field(
|
||||
default=2,
|
||||
description="Batch size for vector sync orchestration flushes.",
|
||||
|
||||
+63
-6
@@ -1,7 +1,7 @@
|
||||
import asyncio
|
||||
import os
|
||||
import sys
|
||||
from contextlib import asynccontextmanager
|
||||
from contextlib import asynccontextmanager, suppress
|
||||
from enum import Enum, auto
|
||||
from pathlib import Path
|
||||
from typing import AsyncGenerator, Optional
|
||||
@@ -39,6 +39,44 @@ from basic_memory.repository.sqlite_search_repository import SQLiteSearchReposit
|
||||
if sys.platform == "win32": # pragma: no cover
|
||||
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
|
||||
|
||||
|
||||
def maybe_install_uvloop(config: BasicMemoryConfig) -> bool:
|
||||
"""Install the uvloop event-loop policy for the Postgres backend.
|
||||
|
||||
Trigger: process entrypoint starting with database_backend == postgres,
|
||||
uvloop importable, and a non-Windows platform.
|
||||
Why: asyncpg engine teardown (engine.dispose()) races the stdlib asyncio
|
||||
loop shutdown and surfaces "IndexError: pop from an empty deque" from
|
||||
base_events._run_once (see #831/#877). uvloop's C scheduler has no
|
||||
self._ready.popleft() codepath, so that class of crash cannot fire under it.
|
||||
Outcome: Postgres deployments run on uvloop; SQLite users keep the default
|
||||
loop (no behavior change, smaller blast radius). Must run before the event
|
||||
loop is created, i.e. before asyncio.run().
|
||||
|
||||
Returns:
|
||||
True if the uvloop policy was installed, False otherwise.
|
||||
"""
|
||||
# uvloop is not available on Windows; the default loop already differs there.
|
||||
if sys.platform == "win32": # pragma: no cover
|
||||
return False
|
||||
|
||||
# Limit the change to the backend that actually hits the asyncpg dispose race.
|
||||
if config.database_backend != DatabaseBackend.POSTGRES:
|
||||
return False
|
||||
|
||||
# Deferred import: uvloop is an optional, platform-gated dependency and the
|
||||
# default (SQLite) path must not require it to be installed.
|
||||
try:
|
||||
import uvloop
|
||||
except ImportError: # pragma: no cover
|
||||
logger.warning("uvloop not available - using default event loop for Postgres backend")
|
||||
return False
|
||||
|
||||
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
|
||||
logger.info("Installed uvloop event-loop policy for Postgres backend")
|
||||
return True
|
||||
|
||||
|
||||
# Module level state
|
||||
_engine: Optional[AsyncEngine] = None
|
||||
_session_maker: Optional[async_sessionmaker[AsyncSession]] = None
|
||||
@@ -320,7 +358,15 @@ async def shutdown_db() -> None: # pragma: no cover
|
||||
global _engine, _session_maker
|
||||
|
||||
if _engine:
|
||||
await _engine.dispose()
|
||||
# Trigger: teardown can run while the surrounding task is being cancelled
|
||||
# (e.g. lifespan shutdown, unshielded CLI cleanup).
|
||||
# Why: a cancellation landing mid-dispose surfaces the asyncpg
|
||||
# "IndexError: pop from an empty deque" race (#831/#877); shielding lets
|
||||
# dispose finish atomically, and suppressing CancelledError keeps a
|
||||
# cancelled shutdown from re-raising the underlying race.
|
||||
# Outcome: connections always close cleanly even under cancellation.
|
||||
with suppress(asyncio.CancelledError):
|
||||
await asyncio.shield(_engine.dispose())
|
||||
_engine = None
|
||||
_session_maker = None
|
||||
|
||||
@@ -364,7 +410,14 @@ async def engine_session_factory(
|
||||
|
||||
yield created_engine, created_session_maker
|
||||
finally:
|
||||
await created_engine.dispose()
|
||||
# Trigger: context-manager teardown can run while the surrounding task is
|
||||
# being cancelled (e.g. a test aborting mid-fixture).
|
||||
# Why: on the asyncpg backend a cancellation landing mid-dispose surfaces
|
||||
# the "IndexError: pop from an empty deque" race (#831/#877); shield the
|
||||
# dispose and suppress CancelledError to match the other dispose seams.
|
||||
# Outcome: the per-context engine always disposes cleanly under cancellation.
|
||||
with suppress(asyncio.CancelledError):
|
||||
await asyncio.shield(created_engine.dispose())
|
||||
|
||||
# Only clear module-level globals if they still point to this context's
|
||||
# engine/session. This avoids clobbering newer globals from other callers.
|
||||
@@ -433,7 +486,11 @@ async def run_migrations(
|
||||
# Trigger: run_migrations() created a temporary engine while module-level
|
||||
# session maker was not initialized.
|
||||
# Why: temporary aiosqlite worker threads can outlive CLI command execution
|
||||
# and block process shutdown if the engine is not disposed.
|
||||
# Outcome: always dispose temporary engines after migration work completes.
|
||||
# and block process shutdown if the engine is not disposed. On the asyncpg
|
||||
# backend a cancellation landing mid-dispose surfaces the same "IndexError:
|
||||
# pop from an empty deque" race as the other dispose seams (#831/#877), so
|
||||
# shield the dispose and suppress CancelledError to match them.
|
||||
# Outcome: always dispose temporary engines cleanly, even under cancellation.
|
||||
if temp_engine is not None:
|
||||
await temp_engine.dispose()
|
||||
with suppress(asyncio.CancelledError):
|
||||
await asyncio.shield(temp_engine.dispose())
|
||||
|
||||
@@ -7,6 +7,15 @@ from typing import Set
|
||||
from basic_memory.config import resolve_data_dir
|
||||
|
||||
|
||||
# Marker shared by the API ignored-path rejection detail and MCP-side error handling.
|
||||
# The sync-file endpoint embeds it in its 400 detail and edit_note's disk recovery
|
||||
# matches on it, so "exists but ignored" stays distinguishable from generic rejections
|
||||
# without duplicating message text across layers.
|
||||
IGNORED_PATH_REJECTION_DETAIL = (
|
||||
"matches Basic Memory ignore rules (.bmignore or project .gitignore)"
|
||||
)
|
||||
|
||||
|
||||
# Common directories and patterns to ignore by default
|
||||
# These are used as fallback if .bmignore doesn't exist
|
||||
DEFAULT_IGNORE_PATTERNS = {
|
||||
|
||||
@@ -493,8 +493,14 @@ class BatchIndexer:
|
||||
async def resolve_relation(relation: Relation) -> int:
|
||||
async with semaphore:
|
||||
try:
|
||||
# strict=True for deferred resolution: only fill in to_id on an
|
||||
# exact permalink/title/file_path match. Fuzzy fallback would silently
|
||||
# resolve ambiguous links to whichever entity shares tokens with the
|
||||
# link text, mismatching this with the sync_service forward-reference
|
||||
# path and producing confidently-wrong graph edges. See
|
||||
# sync_service.resolve_forward_references for the same change.
|
||||
resolved_entity = await self.entity_service.link_resolver.resolve_link(
|
||||
relation.to_name
|
||||
relation.to_name, strict=True
|
||||
)
|
||||
if resolved_entity is None or resolved_entity.id == relation.from_id:
|
||||
return 0
|
||||
|
||||
@@ -276,6 +276,35 @@ class KnowledgeClient:
|
||||
)
|
||||
return DirectoryDeleteResult.model_validate(response.json())
|
||||
|
||||
# --- Single-file sync ---
|
||||
|
||||
async def sync_file(self, file_path: str) -> EntityResponse:
|
||||
"""Index a markdown file that exists on disk but is not indexed yet.
|
||||
|
||||
Args:
|
||||
file_path: Markdown file path relative to the project root
|
||||
|
||||
Returns:
|
||||
EntityResponse for the indexed entity
|
||||
|
||||
Raises:
|
||||
ToolError: If the file does not exist on disk or indexing fails
|
||||
"""
|
||||
with logfire.span(
|
||||
"mcp.client.knowledge.sync_file",
|
||||
client_name="knowledge",
|
||||
operation="sync_file",
|
||||
):
|
||||
response = await call_post(
|
||||
self.http_client,
|
||||
f"{self._base_path}/sync-file",
|
||||
json={"file_path": file_path},
|
||||
client_name="knowledge",
|
||||
operation="sync_file",
|
||||
path_template="/v2/projects/{project_id}/knowledge/sync-file",
|
||||
)
|
||||
return EntityResponse.model_validate(response.json())
|
||||
|
||||
# --- Orphan detection ---
|
||||
|
||||
async def get_orphans(self) -> list[GraphNode]:
|
||||
|
||||
@@ -209,6 +209,18 @@ async def build_context(
|
||||
Raises:
|
||||
ToolError: If project doesn't exist or depth parameter is invalid
|
||||
"""
|
||||
# Validate pagination arguments before they reach the context service.
|
||||
# Trigger: page < 1 or page_size < 1 (e.g. page_size=0 or negative).
|
||||
# Why: a non-positive page_size flows into context_service as limit, where the
|
||||
# primary slice does primary = primary[:limit] — so limit=0 truncates the
|
||||
# requested entity to [] and the caller's valid memory:// lookup silently
|
||||
# returns primary_count=0. Mirrors recent_activity's guard for consistency.
|
||||
# Outcome: caller gets an explicit ValueError instead of a dropped primary result.
|
||||
if page < 1:
|
||||
raise ValueError(f"page must be >= 1, got {page}")
|
||||
if page_size < 1:
|
||||
raise ValueError(f"page_size must be >= 1, got {page_size}")
|
||||
|
||||
# Detect project from memory URL prefix before routing.
|
||||
# project_id routes by external UUID, so it bypasses URL discovery entirely.
|
||||
if project is None and project_id is None:
|
||||
|
||||
@@ -315,14 +315,22 @@ async def delete_note(
|
||||
)
|
||||
result = await knowledge_client.delete_directory(directory_identifier)
|
||||
if output_format == "json":
|
||||
return {
|
||||
response = {
|
||||
"deleted": result.failed_deletes == 0,
|
||||
"is_directory": True,
|
||||
"identifier": identifier,
|
||||
"total_files": result.total_files,
|
||||
"successful_deletes": result.successful_deletes,
|
||||
"failed_deletes": result.failed_deletes,
|
||||
"deleted_files": result.deleted_files,
|
||||
"errors": [error.model_dump() for error in result.errors],
|
||||
}
|
||||
if result.failed_deletes > 0:
|
||||
response["error"] = (
|
||||
"Directory delete incomplete: "
|
||||
f"{result.failed_deletes} of {result.total_files} file(s) failed"
|
||||
)
|
||||
return response
|
||||
|
||||
# Build success message for directory delete
|
||||
result_lines = [
|
||||
|
||||
@@ -1,13 +1,19 @@
|
||||
"""Edit note tool for Basic Memory MCP server."""
|
||||
|
||||
from typing import Annotated, Optional, Literal
|
||||
from typing import TYPE_CHECKING, Annotated, Optional, Literal
|
||||
|
||||
import logfire
|
||||
from httpx import HTTPStatusError
|
||||
from loguru import logger
|
||||
from fastmcp import Context
|
||||
from mcp.server.fastmcp.exceptions import ToolError
|
||||
from pydantic import AliasChoices, Field
|
||||
|
||||
if TYPE_CHECKING: # pragma: no cover
|
||||
from basic_memory.mcp.clients import KnowledgeClient
|
||||
|
||||
from basic_memory.config import ConfigManager
|
||||
from basic_memory.ignore_utils import IGNORED_PATH_REJECTION_DETAIL
|
||||
from basic_memory.mcp.project_context import (
|
||||
_workspace_identifier_discovery_available,
|
||||
detect_project_from_memory_url_prefix,
|
||||
@@ -16,6 +22,7 @@ from basic_memory.mcp.project_context import (
|
||||
resolve_project_and_path,
|
||||
)
|
||||
from basic_memory.mcp.server import mcp
|
||||
from basic_memory.mcp.tools.utils import _extract_response_data, _response_detail_text
|
||||
from basic_memory.schemas.base import Entity
|
||||
from basic_memory.schemas.response import EntityResponse
|
||||
from basic_memory.services.link_resolver import (
|
||||
@@ -52,6 +59,79 @@ def _parse_identifier_to_title_and_directory(identifier: str) -> tuple[str, str]
|
||||
return title, directory
|
||||
|
||||
|
||||
# Suffixes mimetypes maps to text/markdown (extension matching is case-insensitive),
|
||||
# mirroring FileService.is_markdown which gates the sync-file endpoint server-side.
|
||||
_MARKDOWN_SUFFIXES = (".md", ".markdown")
|
||||
|
||||
|
||||
async def _resolve_after_disk_recovery(
|
||||
knowledge_client: "KnowledgeClient",
|
||||
identifier: str,
|
||||
) -> Optional[str]:
|
||||
"""Recover from a resolution miss when the note exists on disk but is not indexed.
|
||||
|
||||
Trigger: identifier resolution failed with "not found", but the identifier may map
|
||||
to a markdown file written directly to disk before the watcher indexed it (#581).
|
||||
Why: editing an on-disk note should not require a manual full sync or watcher restart.
|
||||
Outcome: the single file is indexed server-side and resolution is retried exactly
|
||||
once. Returns None when the identifier does not map to an indexable file on
|
||||
disk, so the caller keeps its existing not-found handling.
|
||||
"""
|
||||
# Try the identifier as-is first so existing .markdown/.MD files are found; only
|
||||
# fall back to appending markdown suffixes (".md" first, then ".markdown") when
|
||||
# the identifier does not already carry one, so 'notes/foo.markdown' never becomes
|
||||
# 'notes/foo.markdown.md' and a stem identifier still reaches 'notes/foo.markdown'.
|
||||
candidates = [identifier]
|
||||
if not identifier.lower().endswith(_MARKDOWN_SUFFIXES):
|
||||
candidates.extend(f"{identifier}{suffix}" for suffix in _MARKDOWN_SUFFIXES)
|
||||
|
||||
for candidate in candidates:
|
||||
try:
|
||||
synced = await knowledge_client.sync_file(candidate)
|
||||
except ToolError as sync_error:
|
||||
# Trigger: the sync-file request failed
|
||||
# Why: 400/404 are the expected "nothing to recover" rejections (missing
|
||||
# file, traversal, non-markdown) — except the ignored-path 400, which
|
||||
# means the file exists on disk but the ignore rules forbid indexing
|
||||
# it, so falling through to auto-create would silently shadow the
|
||||
# file. Anything else — auth, server, transport-level failures — is a
|
||||
# real error that must not be masked as a not-found miss.
|
||||
# Outcome: ignored-path rejections raise a clear ToolError; other expected
|
||||
# rejections try the next candidate or fall through to the caller's
|
||||
# existing not-found behavior; unexpected failures propagate.
|
||||
cause = sync_error.__cause__
|
||||
candidate_rejected = isinstance(
|
||||
cause, HTTPStatusError
|
||||
) and cause.response.status_code in (400, 404)
|
||||
if not candidate_rejected:
|
||||
raise
|
||||
detail = _response_detail_text(_extract_response_data(cause.response)) or ""
|
||||
if IGNORED_PATH_REJECTION_DETAIL in detail:
|
||||
raise ToolError(
|
||||
f"Note file '{candidate}' exists on disk but {IGNORED_PATH_REJECTION_DETAIL} "
|
||||
"and will not be edited"
|
||||
) from sync_error
|
||||
logger.debug(f"edit_note disk recovery skipped for '{candidate}': {sync_error}")
|
||||
continue
|
||||
|
||||
# Trigger: sync-file succeeded and returned the indexed entity.
|
||||
# Why: the server may have canonicalized the path casing (notes/Disk-Note ->
|
||||
# notes/disk-note.md), so strictly re-resolving the raw identifier can
|
||||
# still miss the entity we just indexed.
|
||||
# Outcome: use the entity identity from the sync-file response directly; only
|
||||
# fall back to a strict re-resolve when an older server omits external_id,
|
||||
# and let that re-resolve fail loudly instead of guessing.
|
||||
if synced.external_id:
|
||||
logger.info(
|
||||
f"edit_note indexed unindexed file '{candidate}' as entity {synced.external_id}"
|
||||
)
|
||||
return synced.external_id
|
||||
logger.info(f"edit_note indexed unindexed file '{candidate}'; retrying resolution")
|
||||
return await knowledge_client.resolve_entity(identifier, strict=True)
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def _compose_workspace_project_route(
|
||||
*,
|
||||
workspace: Optional[str],
|
||||
@@ -126,7 +206,8 @@ The note with identifier '{identifier}' could not be found. The `find_replace` a
|
||||
## Suggestions to try:
|
||||
1. **Use append/prepend instead**: These operations will create the note automatically if it doesn't exist
|
||||
2. **Search for the note first**: Use `search_notes("{project or "project-name"}", "{identifier.split("/")[-1]}")` to find similar notes with exact identifiers
|
||||
3. **Try different exact identifier formats**:
|
||||
3. **File exists on disk but is not indexed yet?**: edit_note indexes the file automatically when the identifier matches its path (e.g. 'folder/note' for 'folder/note.md'). If your identifier is a title or differs from the file path, run a sync (`basic-memory sync`) or wait for the file watcher, then retry
|
||||
4. **Try different exact identifier formats**:
|
||||
- If you used a permalink like "folder/note-title", try the exact title: "{identifier.split("/")[-1].replace("-", " ").title()}"
|
||||
- If you used a title, try the exact permalink format: "{identifier.lower().replace(" ", "-")}"
|
||||
- Use `read_note("{project or "project-name"}", "{identifier}")` first to verify the note exists and get the exact identifier
|
||||
@@ -343,7 +424,9 @@ async def edit_note(
|
||||
|
||||
Note:
|
||||
Edit operations require exact identifier matches. If unsure, use read_note() or
|
||||
search_notes() first to find the correct identifier. The tool provides detailed
|
||||
search_notes() first to find the correct identifier. When the identifier looks
|
||||
like a file path and the file exists on disk but is not indexed yet, edit_note
|
||||
indexes that file automatically and retries the edit. The tool provides detailed
|
||||
error messages with suggestions if operations fail.
|
||||
"""
|
||||
# Resolve effective default: allow MCP clients to send null for optional int field
|
||||
@@ -465,14 +548,27 @@ async def edit_note(
|
||||
strict=True,
|
||||
)
|
||||
except Exception as resolve_error:
|
||||
# Trigger: entity does not exist yet
|
||||
# Why: append/prepend can meaningfully create a new note from the content,
|
||||
# while find_replace/replace_section require existing content to modify
|
||||
# Outcome: note is created via the same path as write_note
|
||||
error_msg = str(resolve_error).lower()
|
||||
is_not_found = "entity not found" in error_msg or "not found" in error_msg
|
||||
|
||||
if is_not_found and operation in ("append", "prepend"):
|
||||
# Trigger: resolution missed but the file may already exist on disk
|
||||
# Why: files written directly to disk are invisible to identifier
|
||||
# resolution until indexed; editing them should just work (#581)
|
||||
# Outcome: the single file is indexed and resolution retried once
|
||||
recovered_entity_id: str | None = None
|
||||
if is_not_found:
|
||||
recovered_entity_id = await _resolve_after_disk_recovery(
|
||||
knowledge_client, entity_identifier
|
||||
)
|
||||
|
||||
if recovered_entity_id is not None:
|
||||
entity_id = recovered_entity_id
|
||||
elif is_not_found and operation in ("append", "prepend"):
|
||||
# Trigger: entity does not exist yet (on disk or in the index)
|
||||
# Why: append/prepend can meaningfully create a new note from the
|
||||
# content, while find_replace/replace_section require existing
|
||||
# content to modify
|
||||
# Outcome: note is created via the same path as write_note
|
||||
title, directory = _parse_identifier_to_title_and_directory(identifier)
|
||||
|
||||
# Validate directory path (same security check as write_note)
|
||||
|
||||
@@ -10,7 +10,7 @@ from mcp.server.fastmcp.exceptions import ToolError
|
||||
from pydantic import AliasChoices, Field
|
||||
|
||||
from basic_memory.mcp.server import mcp
|
||||
from basic_memory.mcp.project_context import get_project_client
|
||||
from basic_memory.mcp.project_context import get_project_client, resolve_project_and_path
|
||||
from basic_memory.utils import validate_project_path
|
||||
|
||||
|
||||
@@ -37,22 +37,35 @@ async def _detect_cross_project_move_attempt(
|
||||
project_list = await project_client.list_projects()
|
||||
project_names = [p.name.lower() for p in project_list.projects]
|
||||
|
||||
# Check if destination path contains any project names
|
||||
dest_lower = destination_path.lower()
|
||||
path_parts = dest_lower.split("/")
|
||||
path_parts = [part for part in dest_lower.split("/") if part]
|
||||
|
||||
# Look for project names in the destination path
|
||||
for part in path_parts:
|
||||
if part in project_names and part != current_project.lower():
|
||||
# Found a different project name in the path
|
||||
# --- Detection 1: leading segment is a known project name ---
|
||||
# Trigger: the first path segment matches a different project's name.
|
||||
# Why: a routing-style destination like "other-project/file.md" expresses an
|
||||
# intent to move into another project, which move_note cannot do — it
|
||||
# would silently create a same-project nested folder instead.
|
||||
# Outcome: reject with cross-project guidance rather than fake success.
|
||||
if path_parts:
|
||||
leading = path_parts[0]
|
||||
if leading in project_names and leading != current_project.lower():
|
||||
matching_project = next(
|
||||
p.name for p in project_list.projects if p.name.lower() == part
|
||||
p.name for p in project_list.projects if p.name.lower() == leading
|
||||
)
|
||||
return _format_cross_project_error_response(
|
||||
identifier, destination_path, current_project, matching_project
|
||||
)
|
||||
|
||||
# No other cross-project patterns detected
|
||||
# NOTE: a "<seg>/projects/<seg>/..." structural heuristic was removed here.
|
||||
# Why: matching any destination whose 2nd segment is literally "projects" is
|
||||
# fundamentally ambiguous — it cannot distinguish the cloud workspace
|
||||
# routing shape from a legitimate same-project nested folder like
|
||||
# "notes/projects/2025/file.md" or "work/projects/q1/report.md". The
|
||||
# heuristic produced false CROSS_PROJECT_MOVE_NOT_SUPPORTED rejections for
|
||||
# those common layouts.
|
||||
# Outcome: cross-workspace routing that does not match a known project name (above)
|
||||
# now falls through to the move and is caught honestly by the MOVE_OUTCOME_MISMATCH
|
||||
# backstop if the result lands somewhere other than the requested path.
|
||||
|
||||
except Exception as e:
|
||||
# If we can't detect, don't interfere with normal error handling
|
||||
@@ -633,12 +646,37 @@ list_directory("{identifier}")
|
||||
move_note("path/to/file.md", "{destination_path}/file.md")
|
||||
```"""
|
||||
|
||||
# Check for potential cross-project move attempts (file moves only)
|
||||
cross_project_error = await _detect_cross_project_move_attempt(
|
||||
client, identifier, destination_path, active_project.name
|
||||
# Import here to avoid circular import
|
||||
from basic_memory.mcp.clients import KnowledgeClient
|
||||
|
||||
# Use typed KnowledgeClient for API calls
|
||||
knowledge_client = KnowledgeClient(client, active_project.external_id)
|
||||
|
||||
# --- Normalize the identifier for memory:// URLs ---
|
||||
# Trigger: caller passed a "memory://..." URL (the docstring advertises this, and
|
||||
# read_note/edit_note/delete_note all accept it).
|
||||
# Why: resolve_entity / the /resolve endpoint expect a bare permalink or title;
|
||||
# the literal "memory://" scheme prefix is not stripped by link resolution and
|
||||
# 404s. resolve_project_and_path strips the prefix and normalizes the path the
|
||||
# same way the sibling tools do.
|
||||
# Outcome: move_note resolves memory:// URLs identically to read/edit/delete.
|
||||
source_project, resolved_identifier, _ = await resolve_project_and_path(
|
||||
client, identifier, active_project.name, context
|
||||
)
|
||||
if cross_project_error:
|
||||
logger.info(f"Detected cross-project move attempt: {identifier} -> {destination_path}")
|
||||
|
||||
# Trigger: a memory:// identifier whose project prefix resolves to a DIFFERENT project
|
||||
# than the one move_note is operating on (e.g. "memory://other-project/...").
|
||||
# Why: get_project_client already bound knowledge_client to the active project, so the
|
||||
# resolve_entity below runs against the active project regardless of the URL's
|
||||
# project. Honoring the cross-project prefix would misroute (path normalized for
|
||||
# the other project, looked up in the active one); move_note cannot move across
|
||||
# projects.
|
||||
# Outcome: reject up front with the cross-project guidance instead of misrouting.
|
||||
if source_project.external_id != active_project.external_id:
|
||||
logger.info(
|
||||
f"Move rejected: source '{identifier}' resolves to project "
|
||||
f"'{source_project.name}', not the active project '{active_project.name}'"
|
||||
)
|
||||
if output_format == "json":
|
||||
return {
|
||||
"moved": False,
|
||||
@@ -649,13 +687,9 @@ move_note("path/to/file.md", "{destination_path}/file.md")
|
||||
"destination": destination_path,
|
||||
"error": "CROSS_PROJECT_MOVE_NOT_SUPPORTED",
|
||||
}
|
||||
return cross_project_error
|
||||
|
||||
# Import here to avoid circular import
|
||||
from basic_memory.mcp.clients import KnowledgeClient
|
||||
|
||||
# Use typed KnowledgeClient for API calls
|
||||
knowledge_client = KnowledgeClient(client, active_project.external_id)
|
||||
return _format_cross_project_error_response(
|
||||
identifier, destination_path, active_project.name, source_project.name
|
||||
)
|
||||
|
||||
# Resolve once and reuse the entity ID across extension validation and move.
|
||||
source_ext = "md" # Default to .md if we can't determine source extension
|
||||
@@ -666,7 +700,9 @@ move_note("path/to/file.md", "{destination_path}/file.md")
|
||||
"""Resolve and cache the source entity ID for the duration of this move."""
|
||||
nonlocal resolved_entity_id
|
||||
if resolved_entity_id is None:
|
||||
resolved_entity_id = await knowledge_client.resolve_entity(identifier, strict=True)
|
||||
resolved_entity_id = await knowledge_client.resolve_entity(
|
||||
resolved_identifier, strict=True
|
||||
)
|
||||
return resolved_entity_id
|
||||
|
||||
try:
|
||||
@@ -756,6 +792,31 @@ The destination folder '{destination_folder}' is not allowed - paths must stay w
|
||||
move_note("{identifier}", destination_folder="notes")
|
||||
```"""
|
||||
|
||||
# --- Cross-boundary intent guard (file moves only) ---
|
||||
# Trigger: destination_path now holds the real combined target, whether it came
|
||||
# from destination_path or was resolved from destination_folder above.
|
||||
# Why: detection must run AFTER folder resolution — running it earlier (when a
|
||||
# caller used destination_folder) saw an empty destination_path and skipped
|
||||
# entirely (#881 Gap 3).
|
||||
# Outcome: a cross-workspace/cross-project routing destination is rejected with
|
||||
# guidance instead of silently degrading to a same-project nested folder.
|
||||
cross_project_error = await _detect_cross_project_move_attempt(
|
||||
client, identifier, destination_path, active_project.name
|
||||
)
|
||||
if cross_project_error:
|
||||
logger.info(f"Detected cross-project move attempt: {identifier} -> {destination_path}")
|
||||
if output_format == "json":
|
||||
return {
|
||||
"moved": False,
|
||||
"title": None,
|
||||
"permalink": None,
|
||||
"file_path": None,
|
||||
"source": identifier,
|
||||
"destination": destination_path,
|
||||
"error": "CROSS_PROJECT_MOVE_NOT_SUPPORTED",
|
||||
}
|
||||
return cross_project_error
|
||||
|
||||
# Validate that destination path includes a file extension
|
||||
if "." not in destination_path or not destination_path.split(".")[-1]:
|
||||
logger.warning(f"Move failed - no file extension provided: {destination_path}")
|
||||
@@ -839,6 +900,53 @@ move_note("{identifier}", destination_folder="notes")
|
||||
|
||||
# Call the move API using KnowledgeClient
|
||||
result = await knowledge_client.move_entity(resolved_entity_id, destination_path)
|
||||
|
||||
# --- Outcome validation (honest success backstop) ---
|
||||
# Trigger: the resulting file_path differs from the destination the caller
|
||||
# requested.
|
||||
# Why: move_entity stores the path relative to the *current* project root, so
|
||||
# a cross-boundary intent silently degrades into a same-project nested
|
||||
# folder. The old code reported "✅ moved successfully" regardless,
|
||||
# misleading the agent (#881 Gap 2). This is the robust backstop behind
|
||||
# the up-front detection: any divergence the caller did not ask for
|
||||
# surfaces as a failure rather than a fake success.
|
||||
# Outcome: report failure with the actual landing path instead of "✅".
|
||||
normalized_requested = PureWindowsPath(destination_path).as_posix().strip("/")
|
||||
normalized_actual = PureWindowsPath(result.file_path).as_posix().strip("/")
|
||||
if normalized_actual != normalized_requested:
|
||||
logger.warning(
|
||||
f"Move outcome diverged from intent: requested={destination_path} "
|
||||
f"actual={result.file_path}"
|
||||
)
|
||||
if output_format == "json":
|
||||
return {
|
||||
"moved": False,
|
||||
"title": result.title,
|
||||
"permalink": result.permalink,
|
||||
"file_path": result.file_path,
|
||||
"source": identifier,
|
||||
"destination": destination_path,
|
||||
"error": "MOVE_OUTCOME_MISMATCH",
|
||||
}
|
||||
return dedent(f"""
|
||||
# Move Failed - Unexpected Result Location
|
||||
|
||||
The move of '{identifier}' did not land at the requested destination.
|
||||
|
||||
**Requested:** `{destination_path}`
|
||||
**Actual:** `{result.file_path}`
|
||||
|
||||
This usually means the destination referenced a different
|
||||
workspace/project, which move_note cannot do — notes can only be
|
||||
moved within the same project. To move content between projects:
|
||||
|
||||
```
|
||||
read_note("{result.file_path}")
|
||||
write_note("Title", "content", "folder", project="target-project")
|
||||
delete_note("{result.file_path}", project="{active_project.name}")
|
||||
```
|
||||
""").strip()
|
||||
|
||||
if output_format == "json":
|
||||
return {
|
||||
"moved": True,
|
||||
|
||||
@@ -1,13 +1,14 @@
|
||||
"""Read note tool for Basic Memory MCP server."""
|
||||
|
||||
from textwrap import dedent
|
||||
from typing import Optional, Literal, cast
|
||||
from typing import Annotated, Optional, Literal, cast
|
||||
|
||||
import logfire
|
||||
import yaml
|
||||
|
||||
from loguru import logger
|
||||
from fastmcp import Context
|
||||
from pydantic import AliasChoices, Field
|
||||
|
||||
from basic_memory.config import ConfigManager
|
||||
from basic_memory.mcp.project_context import (
|
||||
@@ -20,6 +21,17 @@ from basic_memory.mcp.tools.search import search_notes
|
||||
from basic_memory.schemas.memory import memory_url_path
|
||||
from basic_memory.utils import validate_project_path
|
||||
|
||||
# The title-match fallback exists to find THE note by exact title, so it scans
|
||||
# fixed-size pages of title results instead of the caller's page/page_size
|
||||
# (which apply only to the text-search suggestion listing).
|
||||
_TITLE_LOOKUP_PAGE_SIZE = 10
|
||||
|
||||
# Hard safety cap on title-lookup pages. The loop normally stops as soon as an
|
||||
# exact match is found or results run out (has_more=False); the cap only bounds
|
||||
# pathological knowledge bases where hundreds of fuzzy titles contain the
|
||||
# queried phrase. Exhausting the cap falls through to the suggestion behavior.
|
||||
_TITLE_LOOKUP_MAX_PAGES = 10
|
||||
|
||||
|
||||
def _is_exact_title_match(identifier: str, title: str) -> bool:
|
||||
"""Return True when identifier exactly matches a title (case-insensitive)."""
|
||||
@@ -71,6 +83,21 @@ async def read_note(
|
||||
identifier: str,
|
||||
project: Optional[str] = None,
|
||||
project_id: Optional[str] = None,
|
||||
# Accept common pagination aliases models reach for from training data
|
||||
# (page_number/limit/per_page), matching the sibling navigation tools
|
||||
# (search_notes, build_context, recent_activity). The schema advertises
|
||||
# only the canonical names; aliases are silently mapped at validation time.
|
||||
# `offset` is intentionally NOT aliased: offset is item-indexed (skip N
|
||||
# items) while page is a 1-indexed page-number, so direct aliasing would
|
||||
# return the wrong slice.
|
||||
page: Annotated[
|
||||
int,
|
||||
Field(default=1, validation_alias=AliasChoices("page", "page_number")),
|
||||
] = 1,
|
||||
page_size: Annotated[
|
||||
int,
|
||||
Field(default=10, validation_alias=AliasChoices("page_size", "limit", "per_page")),
|
||||
] = 10,
|
||||
output_format: Literal["text", "json"] = "text",
|
||||
include_frontmatter: bool = False,
|
||||
context: Context | None = None,
|
||||
@@ -99,6 +126,14 @@ async def read_note(
|
||||
workspaces. Takes precedence over `project`. Get from list_memory_projects().
|
||||
identifier: The title or permalink of the note to read
|
||||
Can be a full memory:// URL, a permalink, a title, or search text
|
||||
page: Page of fallback-search results to use when the identifier does not
|
||||
resolve to a note directly (default: 1). A direct or exact-title match
|
||||
always returns the full note content — page/page_size never chunk the
|
||||
note itself, and the title-match lookup pages through fixed-size pages
|
||||
of title results until an exact match is found or results are
|
||||
exhausted, regardless of page or page_size.
|
||||
page_size: Number of fallback-search results per page (default: 10). When no
|
||||
match is found, this caps how many related-note suggestions are listed.
|
||||
output_format: "text" returns markdown content or guidance text.
|
||||
"json" returns a structured object with title/permalink/file_path/content/frontmatter.
|
||||
include_frontmatter: When output_format="json", whether content should include the
|
||||
@@ -122,6 +157,9 @@ async def read_note(
|
||||
# Read recent meeting notes
|
||||
read_note("team-docs", "Weekly Standup")
|
||||
|
||||
# Page through fallback-search suggestions when nothing matches directly
|
||||
read_note("unknown topic", page=2, page_size=5)
|
||||
|
||||
Raises:
|
||||
HTTPError: If project doesn't exist or is inaccessible
|
||||
SecurityError: If identifier attempts path traversal
|
||||
@@ -130,6 +168,15 @@ async def read_note(
|
||||
If the exact note isn't found, this tool provides helpful suggestions
|
||||
including related notes, search commands, and note creation templates.
|
||||
"""
|
||||
# Trigger: page < 1 or page_size < 1 (e.g. page_size=0 or negative).
|
||||
# Why: both flow into the fallback search's server-side slicing, where
|
||||
# non-positive values produce empty result pages with unreachable
|
||||
# pagination. Fail fast, matching search_notes/build_context.
|
||||
if page < 1:
|
||||
raise ValueError(f"page must be >= 1, got {page}")
|
||||
if page_size < 1:
|
||||
raise ValueError(f"page_size must be >= 1, got {page_size}")
|
||||
|
||||
# Detect project from a memory URL or permalink prefix before routing.
|
||||
# project_id routes by external UUID, so it bypasses URL discovery entirely.
|
||||
if project is None and project_id is None:
|
||||
@@ -147,6 +194,8 @@ async def read_note(
|
||||
tool_name="read_note",
|
||||
requested_project=project,
|
||||
requested_project_id=project_id,
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
output_format=output_format,
|
||||
include_frontmatter=include_frontmatter,
|
||||
):
|
||||
@@ -245,7 +294,7 @@ async def read_note(
|
||||
]
|
||||
|
||||
async def _search_candidates(
|
||||
identifier_text: str, *, title_only: bool
|
||||
identifier_text: str, *, title_only: bool, lookup_page: int = 1
|
||||
) -> dict[str, object]:
|
||||
# Trigger: direct entity resolution failed for the caller's identifier.
|
||||
# Why: search_notes applies the same memory:// normalization and tool-level
|
||||
@@ -256,11 +305,24 @@ async def read_note(
|
||||
# Without this, project names that collide across workspaces could re-resolve
|
||||
# to a different tenant via the default-workspace fallback (CLI/context=None).
|
||||
search_type = "title" if title_only else "text"
|
||||
# Trigger: title_only — the title search exists to find THE note by
|
||||
# exact title, not to page through suggestions.
|
||||
# Why: paginating it by the caller's page would skip an exact match
|
||||
# sitting on page 1 (read_note("Exact Title", page=2)), and a
|
||||
# small caller page_size could let a higher-ranked fuzzy title
|
||||
# displace the exact match out of the lookup window
|
||||
# (read_note("Foo Bar", page_size=1) when "Foo Bar Foo Bar"
|
||||
# ranks first) — both returning suggestions instead of the note.
|
||||
# Outcome: title lookup uses its own lookup_page with a fixed lookup
|
||||
# size, walked by the caller below; caller page/page_size
|
||||
# apply only to the text-search suggestion listing.
|
||||
response = await search_notes(
|
||||
project=active_project.name,
|
||||
project_id=active_project.external_id,
|
||||
query=identifier_text,
|
||||
search_type=search_type,
|
||||
page=lookup_page if title_only else page,
|
||||
page_size=_TITLE_LOOKUP_PAGE_SIZE if title_only else page_size,
|
||||
output_format="json",
|
||||
context=context,
|
||||
)
|
||||
@@ -297,12 +359,24 @@ async def read_note(
|
||||
logger.info(f"Direct lookup failed for '{entity_path}': {e}")
|
||||
# Continue to fallback methods
|
||||
|
||||
# Fallback 1: Try title search via API
|
||||
# Fallback 1: Try title search via API, walking fixed-size pages of
|
||||
# title results until an exact match is found or results run out.
|
||||
# A single page is not enough: when more than _TITLE_LOOKUP_PAGE_SIZE
|
||||
# higher-ranked fuzzy titles contain the queried phrase, the exact
|
||||
# title lands on a later page and a one-page lookup would miss it.
|
||||
logger.info(f"Search title for: {identifier}")
|
||||
title_results = await _search_candidates(identifier, title_only=True)
|
||||
|
||||
title_candidates = _search_results(title_results)
|
||||
if title_candidates:
|
||||
result: dict[str, object] | None = None
|
||||
for lookup_page in range(1, _TITLE_LOOKUP_MAX_PAGES + 1):
|
||||
title_results = await _search_candidates(
|
||||
identifier, title_only=True, lookup_page=lookup_page
|
||||
)
|
||||
title_candidates = _search_results(title_results)
|
||||
if not title_candidates:
|
||||
logger.info(
|
||||
f"No results in title search for: {identifier} "
|
||||
f"in project {active_project.name}"
|
||||
)
|
||||
break
|
||||
# Trigger: direct resolution failed and title search returned candidates.
|
||||
# Why: avoid returning unrelated notes when search yields only fuzzy matches.
|
||||
# Outcome: fetch content only when a true exact title match exists.
|
||||
@@ -314,33 +388,37 @@ async def read_note(
|
||||
),
|
||||
None,
|
||||
)
|
||||
if not result:
|
||||
if result is not None:
|
||||
break
|
||||
# Trigger: this page held only fuzzy titles and the server reports
|
||||
# no further pages (has_more is False or absent).
|
||||
# Why: continuing past the last page would issue empty lookups.
|
||||
# Outcome: give up on the title fallback and try text search below.
|
||||
if title_results.get("has_more") is not True:
|
||||
logger.info(f"No exact title match found for: {identifier}")
|
||||
elif _result_permalink(result):
|
||||
try:
|
||||
# Resolve the permalink to entity ID
|
||||
entity_id = await knowledge_client.resolve_entity(
|
||||
_result_permalink(result) or "", strict=True
|
||||
)
|
||||
break
|
||||
|
||||
# Fetch content using the entity ID
|
||||
response = await resource_client.read(entity_id)
|
||||
if result is not None and _result_permalink(result):
|
||||
try:
|
||||
# Resolve the permalink to entity ID
|
||||
entity_id = await knowledge_client.resolve_entity(
|
||||
_result_permalink(result) or "", strict=True
|
||||
)
|
||||
|
||||
if response.status_code == 200:
|
||||
logger.info(
|
||||
f"Found note by exact title search: {_result_permalink(result)}"
|
||||
)
|
||||
if output_format == "json":
|
||||
return await _read_json_payload(entity_id)
|
||||
return response.text
|
||||
except Exception as e: # pragma: no cover
|
||||
# Fetch content using the entity ID
|
||||
response = await resource_client.read(entity_id)
|
||||
|
||||
if response.status_code == 200:
|
||||
logger.info(
|
||||
f"Failed to fetch content for found title match {_result_permalink(result)}: {e}"
|
||||
f"Found note by exact title search: {_result_permalink(result)}"
|
||||
)
|
||||
else:
|
||||
logger.info(
|
||||
f"No results in title search for: {identifier} in project {active_project.name}"
|
||||
)
|
||||
if output_format == "json":
|
||||
return await _read_json_payload(entity_id)
|
||||
return response.text
|
||||
except Exception as e: # pragma: no cover
|
||||
logger.info(
|
||||
f"Failed to fetch content for found title match {_result_permalink(result)}: {e}"
|
||||
)
|
||||
|
||||
# Fallback 2: Text search as a last resort
|
||||
logger.info(f"Title search failed, trying text search for: {identifier}")
|
||||
@@ -352,6 +430,9 @@ async def read_note(
|
||||
if output_format == "json":
|
||||
return _empty_json_payload()
|
||||
return format_not_found_message(active_project.name, identifier)
|
||||
# The fallback search is paginated server-side to page_size, so list
|
||||
# the whole returned page instead of a hardcoded cap — otherwise the
|
||||
# caller's page_size would be silently ignored past the cap.
|
||||
if output_format == "json":
|
||||
payload = _empty_json_payload()
|
||||
payload["related_results"] = [
|
||||
@@ -360,10 +441,10 @@ async def read_note(
|
||||
"permalink": _result_permalink(result),
|
||||
"file_path": _result_file_path(result),
|
||||
}
|
||||
for result in text_candidates[:5]
|
||||
for result in text_candidates
|
||||
]
|
||||
return payload
|
||||
return format_related_results(active_project.name, identifier, text_candidates[:5])
|
||||
return format_related_results(active_project.name, identifier, text_candidates)
|
||||
|
||||
|
||||
def format_not_found_message(project: str | None, identifier: str) -> str:
|
||||
|
||||
@@ -300,14 +300,20 @@ async def recent_activity(
|
||||
else:
|
||||
# Project-Specific Mode: Get activity for specific project
|
||||
# Uses get_project_client() for per-project routing (local vs cloud)
|
||||
logger.info(
|
||||
f"Getting recent activity from project {resolved_project}: type={type}, depth={depth}, timeframe={timeframe}"
|
||||
)
|
||||
|
||||
async with get_project_client(resolved_project, context=context, project_id=project_id) as (
|
||||
client,
|
||||
active_project,
|
||||
):
|
||||
# Trigger: caller routed by project_id (a UUID), so resolved_project holds the
|
||||
# raw UUID rather than a human-readable name.
|
||||
# Why: active_project.name is the canonical, display-safe project name regardless
|
||||
# of whether routing was by name or by external_id.
|
||||
# Outcome: logs and the formatted text header always show the project name.
|
||||
logger.info(
|
||||
f"Getting recent activity from project {active_project.name}: "
|
||||
f"type={type}, depth={depth}, timeframe={timeframe}"
|
||||
)
|
||||
|
||||
response = await call_get(
|
||||
client,
|
||||
f"/v2/projects/{active_project.external_id}/memory/recent",
|
||||
@@ -319,7 +325,7 @@ async def recent_activity(
|
||||
return _extract_recent_rows(activity_data)
|
||||
|
||||
# Format project-specific mode output
|
||||
return _format_project_output(resolved_project, activity_data, timeframe, type, page)
|
||||
return _format_project_output(active_project.name, activity_data, timeframe, type, page)
|
||||
|
||||
|
||||
async def _get_project_activity(
|
||||
|
||||
@@ -11,7 +11,13 @@ from fastmcp import Context
|
||||
from pydantic import AliasChoices, BeforeValidator, Field
|
||||
|
||||
from basic_memory.config import ConfigManager, has_cloud_credentials
|
||||
from basic_memory.utils import build_canonical_permalink, coerce_dict, coerce_list
|
||||
from basic_memory.utils import (
|
||||
build_canonical_permalink,
|
||||
coerce_dict,
|
||||
coerce_list,
|
||||
parse_tags,
|
||||
strict_search_tags,
|
||||
)
|
||||
from basic_memory.mcp.async_client import (
|
||||
_explicit_routing,
|
||||
_force_local_mode,
|
||||
@@ -83,6 +89,47 @@ def _format_search_error_response(
|
||||
`search_notes("{project}", "{query}", search_type="{search_type}")`
|
||||
""").strip()
|
||||
|
||||
# Corrupt/missing FastEmbed model cache (interrupted download leaves a partial
|
||||
# snapshot missing model_optimized.onnx; the ONNX runtime then raises NO_SUCHFILE).
|
||||
# Basic Memory self-heals by re-downloading on the next load, but if the user still
|
||||
# hits this, point them at the cache dir to clear manually and offer a text fallback.
|
||||
error_lower = error_message.lower()
|
||||
# "load model from" is the exact ONNX phrasing ("Load model from <path>.onnx failed").
|
||||
# The looser "load model" matched unrelated errors, so we keep only the specific phrase
|
||||
# alongside the onnxruntime / no_suchfile / model_optimized.onnx fingerprints.
|
||||
if (
|
||||
"onnxruntime" in error_lower
|
||||
or "no_suchfile" in error_lower
|
||||
or "model_optimized.onnx" in error_lower
|
||||
or "load model from" in error_lower
|
||||
):
|
||||
# Deferred import: keeps the repository layer out of the tool's import graph
|
||||
# (matches the SearchClient deferral below) and is only needed on this error path.
|
||||
from basic_memory.repository.embedding_provider_factory import _resolve_cache_dir
|
||||
|
||||
try:
|
||||
cache_dir = _resolve_cache_dir(get_container().config)
|
||||
except RuntimeError:
|
||||
cache_dir = _resolve_cache_dir(ConfigManager().config)
|
||||
return dedent(f"""
|
||||
# Search Failed - Embedding Model Missing or Corrupt
|
||||
|
||||
The local FastEmbed model could not be loaded for query '{query}': {error_message}
|
||||
|
||||
This usually means an earlier model download was interrupted and left an
|
||||
incomplete file in the model cache.
|
||||
|
||||
## How to fix
|
||||
1. Delete the FastEmbed model cache so it re-downloads on the next search:
|
||||
`{cache_dir}`
|
||||
2. Run your search again (the model downloads automatically on first use):
|
||||
`search_notes("{project}", "{query}", search_type="{search_type}")`
|
||||
|
||||
## Workaround right now
|
||||
- Use full-text search, which needs no embedding model:
|
||||
`search_notes("{project}", "{query}", search_type="text")`
|
||||
""").strip()
|
||||
|
||||
# FTS5 syntax errors
|
||||
if "syntax error" in error_message.lower() or "fts5" in error_message.lower():
|
||||
clean_query = (
|
||||
@@ -114,7 +161,7 @@ def _format_search_error_response(
|
||||
- Boolean NOT: `project NOT archived`
|
||||
- Grouped: `(project OR planning) AND notes`
|
||||
- Exact phrases: `"weekly standup meeting"`
|
||||
- Content-specific: `tag:example` or `category:observation`
|
||||
- Content-specific: `tag:example`
|
||||
|
||||
## Try again with:
|
||||
```
|
||||
@@ -181,7 +228,7 @@ def _format_search_error_response(
|
||||
|
||||
6. **Try advanced search patterns**:
|
||||
- Tag search: `search_notes("{project}","tag:your-tag")`
|
||||
- Category search: `search_notes("{project}","category:observation")`
|
||||
- Observation category: `search_notes("{project}","{query}", entity_types=["observation"], categories=["requirement"])`
|
||||
- Pattern matching: `search_notes("{project}","*{query}*", search_type="permalink")`
|
||||
|
||||
## Explore what content exists:
|
||||
@@ -258,7 +305,8 @@ Error searching for '{query}': {error_message}
|
||||
- **Boolean**: `term1 AND term2`, `term1 OR term2`, `term1 NOT term2`
|
||||
- **Phrases**: `"exact phrase"`
|
||||
- **Grouping**: `(term1 OR term2) AND term3`
|
||||
- **Patterns**: `tag:example`, `category:observation`"""
|
||||
- **Tags**: `tag:example`
|
||||
- **Observation categories**: `entity_types=["observation"], categories=["requirement"]`"""
|
||||
|
||||
|
||||
def _format_search_markdown(result: SearchResponse, project: str, query: str | None) -> str:
|
||||
@@ -456,6 +504,7 @@ async def _search_all_projects(
|
||||
output_format: Literal["text", "json"],
|
||||
note_types: list[str],
|
||||
entity_types: list[str],
|
||||
categories: list[str],
|
||||
after_date: str | None,
|
||||
metadata_filters: dict[str, Any] | None,
|
||||
tags: list[str] | None,
|
||||
@@ -514,6 +563,7 @@ async def _search_all_projects(
|
||||
output_format="json",
|
||||
note_types=note_types or None,
|
||||
entity_types=entity_types or None,
|
||||
categories=categories or None,
|
||||
after_date=after_date,
|
||||
metadata_filters=metadata_filters,
|
||||
tags=tags,
|
||||
@@ -612,6 +662,14 @@ async def search_notes(
|
||||
"'relation'. Defaults to 'entity'. Do NOT pass schema/frontmatter types like "
|
||||
"'Chapter' here — use note_types instead.",
|
||||
] = None,
|
||||
categories: Annotated[
|
||||
List[str] | None,
|
||||
BeforeValidator(coerce_list),
|
||||
Field(default=None, validation_alias=AliasChoices("categories", "category")),
|
||||
"Filter observation results to these exact categories (e.g. ['requirement']). "
|
||||
"Pair with entity_types=['observation'] to return only observations whose "
|
||||
"category matches exactly — not every row mentioning the word.",
|
||||
] = None,
|
||||
# Time-filter naming varies wildly across APIs.
|
||||
after_date: Annotated[
|
||||
Optional[str],
|
||||
@@ -624,9 +682,15 @@ async def search_notes(
|
||||
Dict[str, Any] | None,
|
||||
BeforeValidator(coerce_dict),
|
||||
] = None,
|
||||
# strict_search_tags, not coerce_list: tags="a,b" must split into ["a", "b"] to
|
||||
# match the tag: query shorthand below and write_note's documented tags convention
|
||||
# (#910). coerce_list would wrap the comma string as the single literal tag
|
||||
# ["a,b"], which matches nothing. Unlike bare parse_tags, the strict wrapper only
|
||||
# splits str/list/None and lets Pydantic reject other types (42, {"a": 1}) with a
|
||||
# clear validation error instead of stringifying them into junk tags.
|
||||
tags: Annotated[
|
||||
List[str] | None,
|
||||
BeforeValidator(coerce_list),
|
||||
BeforeValidator(strict_search_tags),
|
||||
] = None,
|
||||
status: Optional[str] = None,
|
||||
min_similarity: Annotated[
|
||||
@@ -666,7 +730,8 @@ async def search_notes(
|
||||
|
||||
### Content-Specific Searches
|
||||
- `search_notes("research", "tag:example")` - Search within specific tags (if supported by content)
|
||||
- `search_notes("work-project", "category:observation")` - Filter by observation categories
|
||||
- `search_notes("work-project", "req", entity_types=["observation"], categories=["requirement"])`
|
||||
- Return only observations whose category is exactly "requirement"
|
||||
- `search_notes("team-docs", "author:username")` - Find content by author (if metadata available)
|
||||
|
||||
**Note:** `tag:` shorthand is automatically converted to a `tags` filter, so it works
|
||||
@@ -684,6 +749,8 @@ async def search_notes(
|
||||
- `search_notes("my-project", "query", note_types=["note"])` - Search only notes
|
||||
- `search_notes("work-docs", "query", note_types=["note", "person"])` - Multiple note types
|
||||
- `search_notes("research", "query", entity_types=["observation"])` - Filter by entity type
|
||||
- `search_notes("research", "query", entity_types=["observation"], categories=["requirement"])`
|
||||
- Filter observations to an exact category
|
||||
- `search_notes("team-docs", "query", after_date="2024-01-01")` - Recent content only
|
||||
- `search_notes("my-project", "query", after_date="1 week")` - Relative date filtering
|
||||
- `search_notes("my-project", "query", tags=["security"])` - Filter by frontmatter tags
|
||||
@@ -735,9 +802,14 @@ async def search_notes(
|
||||
"json" returns a machine-readable dictionary payload.
|
||||
note_types: Optional list of note types to search (e.g., ["note", "person"])
|
||||
entity_types: Optional list of entity types to filter by (e.g., ["entity", "observation"])
|
||||
categories: Optional list of observation categories for exact matching (e.g.,
|
||||
["requirement"]). Pair with entity_types=["observation"] to return only
|
||||
observations whose category matches exactly.
|
||||
after_date: Optional date filter for recent content (e.g., "1 week", "2d", "2024-01-01")
|
||||
metadata_filters: Optional structured frontmatter filters (e.g., {"status": "in-progress"})
|
||||
tags: Optional tag filter (frontmatter tags); shorthand for metadata_filters["tags"]
|
||||
tags: Optional tag filter (frontmatter tags); shorthand for metadata_filters["tags"].
|
||||
Accepts a list (["a", "b"]) or a comma-separated string ("a,b"), matching the
|
||||
write_note tags convention and the tag: query shorthand.
|
||||
status: Optional status filter (frontmatter status); shorthand for metadata_filters["status"]
|
||||
min_similarity: Optional float to override the global semantic_min_similarity threshold
|
||||
for this query. E.g., 0.0 to see all vector results, or 0.8 for high precision.
|
||||
@@ -808,10 +880,35 @@ async def search_notes(
|
||||
# Explicit project specification
|
||||
results = await search_notes("project planning", project="my-project")
|
||||
"""
|
||||
# Validate pagination arguments before they reach the API/repository layer.
|
||||
# Trigger: page < 1 or page_size < 1 (e.g. page_size=0 or a negative slice).
|
||||
# Why: a non-positive page_size yields zero rows yet the router computes
|
||||
# has_more = offset + len(results) < total, returning a misleading
|
||||
# has_more=True with no reachable page; a negative page_size becomes an
|
||||
# uncapped SQLite LIMIT. Mirrors recent_activity's guard so all navigation
|
||||
# tools reject invalid pagination consistently.
|
||||
# Outcome: caller gets an explicit ValueError instead of a silent bad payload.
|
||||
if page < 1:
|
||||
raise ValueError(f"page must be >= 1, got {page}")
|
||||
if page_size < 1:
|
||||
raise ValueError(f"page_size must be >= 1, got {page_size}")
|
||||
|
||||
# Avoid mutable-default-argument footguns. Treat None as "no filter".
|
||||
# Lowercase note_types so "Chapter" matches the stored "chapter".
|
||||
note_types = [t.lower() for t in note_types] if note_types else []
|
||||
entity_types = entity_types or []
|
||||
# Categories are matched exactly against the indexed observation category,
|
||||
# so preserve their original casing (unlike the lowercased note_types).
|
||||
categories = categories or []
|
||||
|
||||
# Trigger: tags arrived via a direct function call instead of the MCP layer.
|
||||
# Why: the BeforeValidator above only runs through MCP/Pydantic validation; direct
|
||||
# callers (e.g. `bm tool search-notes --tag a,b` in cli/commands/tool.py, which
|
||||
# Typer collects as the one-element list ["a,b"]) would otherwise forward the
|
||||
# comma string as one literal tag that matches nothing (#910).
|
||||
# Outcome: comma-split/list normalization applies on every path; parse_tags is
|
||||
# idempotent, so MCP-validated input passes through unchanged.
|
||||
tags = parse_tags(tags) or None
|
||||
|
||||
# Parse tag:<value> shorthand at tool level so it works with all search modes.
|
||||
# Handles "tag:security", "tag:coffee tag:brewing", "tag:coffee AND tag:brewing".
|
||||
@@ -853,6 +950,7 @@ async def search_notes(
|
||||
output_format=output_format,
|
||||
note_types=note_types,
|
||||
entity_types=entity_types,
|
||||
categories=categories,
|
||||
after_date=after_date,
|
||||
metadata_filters=metadata_filters,
|
||||
tags=tags,
|
||||
@@ -876,8 +974,15 @@ async def search_notes(
|
||||
has_query=bool(query and query.strip()),
|
||||
note_type_filter_count=len(note_types),
|
||||
entity_type_filter_count=len(entity_types),
|
||||
category_filter_count=len(categories),
|
||||
has_filters=bool(
|
||||
metadata_filters or tags or status or note_types or entity_types or after_date
|
||||
metadata_filters
|
||||
or tags
|
||||
or status
|
||||
or note_types
|
||||
or entity_types
|
||||
or categories
|
||||
or after_date
|
||||
),
|
||||
has_tags_filter=bool(tags),
|
||||
has_status_filter=bool(status),
|
||||
@@ -940,6 +1045,8 @@ async def search_notes(
|
||||
# Add optional filters if provided (empty lists are treated as no filter)
|
||||
if entity_types:
|
||||
search_query.entity_types = [SearchItemType(t) for t in entity_types]
|
||||
if categories:
|
||||
search_query.categories = categories
|
||||
if note_types:
|
||||
search_query.note_types = note_types
|
||||
if after_date:
|
||||
@@ -965,7 +1072,8 @@ async def search_notes(
|
||||
return (
|
||||
"# No Search Criteria\n\n"
|
||||
"Please provide at least one of: `query`, `metadata_filters`, "
|
||||
"`tags`, `status`, `note_types`, `entity_types`, or `after_date`."
|
||||
"`tags`, `status`, `note_types`, `entity_types`, `categories`, "
|
||||
"or `after_date`."
|
||||
)
|
||||
|
||||
# Default to entity-level results to avoid returning individual
|
||||
@@ -973,7 +1081,17 @@ async def search_notes(
|
||||
# Applied after no_criteria() so that the implicit default doesn't
|
||||
# mask a truly empty search request.
|
||||
if not search_query.entity_types:
|
||||
search_query.entity_types = [SearchItemType("entity")]
|
||||
# Trigger: a category filter was supplied without an explicit
|
||||
# entity_types.
|
||||
# Why: categories only exist on observations — defaulting to "entity"
|
||||
# (whose rows have NULL category) would AND a category filter against
|
||||
# entity rows and return nothing, defeating a category-only search.
|
||||
# Outcome: scope the implicit default to observations so
|
||||
# search_notes(categories=[...]) returns the matching bullets.
|
||||
if search_query.categories:
|
||||
search_query.entity_types = [SearchItemType("observation")]
|
||||
else:
|
||||
search_query.entity_types = [SearchItemType("entity")]
|
||||
|
||||
logger.debug(
|
||||
f"Search request: project={active_project.name} "
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
"""Write note tool for Basic Memory MCP server."""
|
||||
|
||||
import textwrap
|
||||
from pathlib import Path
|
||||
from typing import Annotated, List, Union, Optional, Literal
|
||||
|
||||
import logfire
|
||||
@@ -86,8 +87,12 @@ async def write_note(
|
||||
Use forward slashes (/) as separators. Use "/" or "" to write to project root.
|
||||
Examples: "notes", "projects/2025", "research/ml", "/" (root)
|
||||
project: Project name to write to. Optional - server will resolve using the
|
||||
hierarchy above. If unknown, use list_memory_projects() to discover
|
||||
available projects.
|
||||
hierarchy above. Use "workspace/project" to route to a project in a
|
||||
specific cloud workspace. A bare name that exists in multiple
|
||||
workspaces resolves to the default workspace, so use the qualified
|
||||
form (or project_id) to disambiguate. If unknown, use
|
||||
list_memory_projects() to discover available projects and their
|
||||
qualified names.
|
||||
project_id: Project external_id (UUID). Prefer this over `project` when known —
|
||||
it routes to the exact project regardless of name collisions across cloud
|
||||
workspaces. Takes precedence over `project`. Get from list_memory_projects().
|
||||
@@ -269,7 +274,19 @@ async def write_note(
|
||||
raise ValueError(
|
||||
"Entity permalink is required for updates"
|
||||
) # pragma: no cover
|
||||
entity_id = await knowledge_client.resolve_entity(entity.permalink)
|
||||
# Resolve the conflicting entity by file_path with strict=True.
|
||||
# The 409 came from a file_service.exists(file_path) check, so this
|
||||
# file_path is the authoritative key for the canonical row. Resolving
|
||||
# by permalink with fuzzy fallback (the previous behavior) could pick
|
||||
# an orphan with a similar permalink — especially in workspace-prefixed
|
||||
# palaces where the client-built permalink omits the workspace slug —
|
||||
# causing the update to write to the wrong row and the next call to
|
||||
# mint a -1/-2 suffix on the canonical entity.
|
||||
# POSIX-normalize so Windows clients send the same form the server stores.
|
||||
file_path_identifier = Path(entity.file_path).as_posix()
|
||||
entity_id = await knowledge_client.resolve_entity(
|
||||
file_path_identifier, strict=True
|
||||
)
|
||||
result = await knowledge_client.update_entity(
|
||||
entity_id, entity.model_dump()
|
||||
)
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
"""Knowledge graph models."""
|
||||
|
||||
import hashlib
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
from basic_memory.utils import ensure_timezone_aware
|
||||
@@ -252,8 +253,18 @@ class Observation(Base):
|
||||
Content is truncated to 200 chars to stay under PostgreSQL's
|
||||
btree index limit of 2704 bytes.
|
||||
"""
|
||||
# Truncate content to avoid exceeding PostgreSQL's btree index limit
|
||||
content_for_permalink = self.content[:200] if len(self.content) > 200 else self.content
|
||||
if len(self.content) > 200:
|
||||
# Trigger: content exceeds the 200-char budget imposed by PostgreSQL's
|
||||
# 2704-byte btree index row limit, so the permalink can only carry a prefix.
|
||||
# Why: two distinct observations with the same category and an identical
|
||||
# 200-char prefix would collide on the same synthetic permalink, and the
|
||||
# search index (permalink-keyed upsert) silently drops the second one.
|
||||
# Outcome: a short stable digest of the FULL content disambiguates
|
||||
# truncated permalinks while staying well under the index limit.
|
||||
digest = hashlib.sha256(self.content.encode("utf-8")).hexdigest()[:12]
|
||||
content_for_permalink = f"{self.content[:200]}-{digest}"
|
||||
else:
|
||||
content_for_permalink = self.content
|
||||
return generate_permalink(
|
||||
f"{self.entity.permalink}/observations/{self.category}/{content_for_permalink}"
|
||||
)
|
||||
|
||||
@@ -3,18 +3,29 @@
|
||||
import os
|
||||
from threading import Lock
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from basic_memory.config import BasicMemoryConfig, default_fastembed_cache_dir
|
||||
from basic_memory.repository.embedding_provider import EmbeddingProvider
|
||||
|
||||
# Cache key fields are limited to values that change the *identity* of the loaded
|
||||
# model (provider, model_name, dimensions, LiteLLM role/input-type/forward-dimension
|
||||
# settings, batch/request knobs that affect the LiteLLM identity, and the resolved
|
||||
# cache dir). Thread/parallel knobs are deliberately excluded — they change ONNX
|
||||
# *execution* only, not the loaded weights. Including them caused #872: in a
|
||||
# container/cgroup the CPU-derived thread count can drift between calls, producing
|
||||
# a fresh cache key and reloading the ~2.3GB model into a CPU arena that never
|
||||
# returns memory to the OS.
|
||||
type ProviderCacheKey = tuple[
|
||||
str,
|
||||
str,
|
||||
int | None,
|
||||
bool | None,
|
||||
int,
|
||||
int,
|
||||
str | None,
|
||||
str | None,
|
||||
str,
|
||||
int | None,
|
||||
int | None,
|
||||
]
|
||||
|
||||
_EMBEDDING_PROVIDER_CACHE: dict[ProviderCacheKey, EmbeddingProvider] = {}
|
||||
@@ -75,22 +86,26 @@ def _resolve_fastembed_runtime_knobs(
|
||||
|
||||
|
||||
def _provider_cache_key(app_config: BasicMemoryConfig) -> ProviderCacheKey:
|
||||
"""Build a stable cache key from provider-relevant semantic embedding config.
|
||||
"""Build a stable cache key from model-identity semantic embedding config.
|
||||
|
||||
Uses the *resolved* cache dir — not the raw config field — so different
|
||||
FASTEMBED_CACHE_PATH values produce distinct cache keys even when the
|
||||
config field itself is unset.
|
||||
|
||||
Deliberately excludes the FastEmbed thread/parallel knobs: they tune ONNX
|
||||
execution, not which model weights are loaded, and resolving them from the
|
||||
runtime CPU budget makes the key drift between calls in a container (#872).
|
||||
"""
|
||||
resolved_threads, resolved_parallel = _resolve_fastembed_runtime_knobs(app_config)
|
||||
return (
|
||||
app_config.semantic_embedding_provider.strip().lower(),
|
||||
app_config.semantic_embedding_model,
|
||||
app_config.semantic_embedding_dimensions,
|
||||
app_config.semantic_embedding_forward_dimensions,
|
||||
app_config.semantic_embedding_batch_size,
|
||||
app_config.semantic_embedding_request_concurrency,
|
||||
app_config.semantic_embedding_document_input_type,
|
||||
app_config.semantic_embedding_query_input_type,
|
||||
_resolve_cache_dir(app_config),
|
||||
resolved_threads,
|
||||
resolved_parallel,
|
||||
)
|
||||
|
||||
|
||||
@@ -103,10 +118,21 @@ def reset_embedding_provider_cache() -> None:
|
||||
def create_embedding_provider(app_config: BasicMemoryConfig) -> EmbeddingProvider:
|
||||
"""Create an embedding provider based on semantic config.
|
||||
|
||||
When semantic_embedding_dimensions is set in config, it overrides
|
||||
the provider's default dimensions (384 for FastEmbed, 1536 for OpenAI).
|
||||
When semantic_embedding_dimensions is set in config, it overrides the
|
||||
provider's default dimensions (384 for FastEmbed, 1536 for OpenAI and
|
||||
the LiteLLM OpenAI default). Custom LiteLLM models require an explicit
|
||||
dimension because the vector table schema is created before the first
|
||||
embedding response is available.
|
||||
"""
|
||||
cache_key = _provider_cache_key(app_config)
|
||||
# Trigger: two threads miss the cache for the same key concurrently.
|
||||
# Why: provider construction loads the ~2.3GB ONNX model and is slow, so we
|
||||
# deliberately build it *outside* the lock to avoid serializing every caller
|
||||
# behind a single cold start. This opens a by-design TOCTOU window where both
|
||||
# threads may construct a provider.
|
||||
# Outcome: the second check-and-set below resolves the race — the first writer
|
||||
# wins and the loser's redundant provider is discarded, so the cache still
|
||||
# yields a single process-wide singleton per key.
|
||||
with _EMBEDDING_PROVIDER_CACHE_LOCK:
|
||||
if cached_provider := _EMBEDDING_PROVIDER_CACHE.get(cache_key):
|
||||
return cached_provider
|
||||
@@ -151,11 +177,49 @@ def create_embedding_provider(app_config: BasicMemoryConfig) -> EmbeddingProvide
|
||||
request_concurrency=app_config.semantic_embedding_request_concurrency,
|
||||
**extra_kwargs,
|
||||
)
|
||||
elif provider_name == "litellm":
|
||||
from basic_memory.repository.litellm_provider import LiteLLMEmbeddingProvider
|
||||
|
||||
model_name = app_config.semantic_embedding_model or "openai/text-embedding-3-small"
|
||||
if model_name == "bge-small-en-v1.5":
|
||||
model_name = "openai/text-embedding-3-small"
|
||||
if (
|
||||
app_config.semantic_embedding_dimensions is None
|
||||
and model_name != "openai/text-embedding-3-small"
|
||||
):
|
||||
raise ValueError(
|
||||
"semantic_embedding_dimensions must be set when "
|
||||
"semantic_embedding_provider='litellm' uses a non-default model. "
|
||||
f"Configured model: {model_name!r}."
|
||||
)
|
||||
provider = LiteLLMEmbeddingProvider(
|
||||
model_name=model_name,
|
||||
batch_size=app_config.semantic_embedding_batch_size,
|
||||
request_concurrency=app_config.semantic_embedding_request_concurrency,
|
||||
document_input_type=app_config.semantic_embedding_document_input_type,
|
||||
query_input_type=app_config.semantic_embedding_query_input_type,
|
||||
forward_dimensions=app_config.semantic_embedding_forward_dimensions,
|
||||
**extra_kwargs,
|
||||
)
|
||||
else:
|
||||
raise ValueError(f"Unsupported semantic embedding provider: {provider_name}")
|
||||
|
||||
with _EMBEDDING_PROVIDER_CACHE_LOCK:
|
||||
if cached_provider := _EMBEDDING_PROVIDER_CACHE.get(cache_key):
|
||||
return cached_provider
|
||||
# Trigger: a distinct cache key is being inserted while the cache already
|
||||
# holds entries for other keys.
|
||||
# Why: the provider is meant to be a process-wide singleton (#872). A second
|
||||
# key means something bypassed reuse — a real config change, or a regression
|
||||
# that reintroduces volatile fields into the key — and each new key reloads
|
||||
# the ~2.3GB ONNX model into a CPU arena that never releases memory.
|
||||
# Outcome: surface the bypass so future leaks are diagnosable from logs.
|
||||
if _EMBEDDING_PROVIDER_CACHE:
|
||||
logger.warning(
|
||||
"Creating a second distinct embedding provider in this process; "
|
||||
"the model will be loaded again. existing_keys={existing} new_key={new}",
|
||||
existing=list(_EMBEDDING_PROVIDER_CACHE.keys()),
|
||||
new=cache_key,
|
||||
)
|
||||
_EMBEDDING_PROVIDER_CACHE[cache_key] = provider
|
||||
return provider
|
||||
|
||||
@@ -178,21 +178,31 @@ class EntityRepository(Repository[Entity]):
|
||||
result = await self.execute_query(query, use_query_options=False)
|
||||
return list(result.scalars().all())
|
||||
|
||||
async def find_by_ids_for_hydration(self, ids: List[int]) -> Sequence[Entity]:
|
||||
async def find_by_ids_for_hydration(
|
||||
self, ids: List[int], *, include_cross_project: bool = False
|
||||
) -> Sequence[Entity]:
|
||||
"""Fetch minimal entity fields needed for context hydration.
|
||||
|
||||
Context hydration only needs an entity's primary key, title, and external
|
||||
UUID. Keeping this separate from find_by_ids avoids the relationship eager
|
||||
loads that are useful for full entity reads but expensive for response shaping.
|
||||
|
||||
Args:
|
||||
ids: Entity IDs to hydrate.
|
||||
include_cross_project: Include IDs outside this repository's project scope.
|
||||
Use only for IDs already reached through validated graph traversal.
|
||||
"""
|
||||
if not ids:
|
||||
return []
|
||||
|
||||
query = (
|
||||
self.select()
|
||||
select(Entity)
|
||||
.where(Entity.id.in_(ids))
|
||||
.options(load_only(Entity.id, Entity.title, Entity.external_id))
|
||||
)
|
||||
if not include_cross_project:
|
||||
query = self._add_project_filter(query)
|
||||
|
||||
result = await self.execute_query(query, use_query_options=False)
|
||||
return list(result.scalars().all())
|
||||
|
||||
|
||||
@@ -3,6 +3,9 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import math
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from loguru import logger
|
||||
@@ -14,6 +17,24 @@ if TYPE_CHECKING:
|
||||
from fastembed import TextEmbedding # pragma: no cover
|
||||
|
||||
|
||||
# Substrings that identify the ONNX "model artifact file is missing" load failure (as
|
||||
# opposed to a config error, a download/network error, or a genuinely offline machine).
|
||||
# An interrupted FastEmbed download can leave the HuggingFace snapshot dir present but
|
||||
# missing ``model_optimized.onnx``; the ONNX runtime then raises ``NO_SUCHFILE`` and every
|
||||
# subsequent load repeats it until the cache is cleared. Matched case-insensitively.
|
||||
#
|
||||
# IMPORTANT: this text match is necessary but NOT sufficient to trigger a purge. The error
|
||||
# text alone cannot distinguish a corrupt cache from a normal cold load (model not yet
|
||||
# downloaded). Purging is gated on a positive filesystem confirmation that the snapshot dir
|
||||
# exists on disk but the model artifact file is missing — see ``_corrupt_model_subdirs``.
|
||||
_MISSING_ARTIFACT_ERROR_MARKERS = (
|
||||
"no_suchfile",
|
||||
"model_optimized.onnx",
|
||||
"file doesn't exist",
|
||||
"no such file",
|
||||
)
|
||||
|
||||
|
||||
class FastEmbedEmbeddingProvider(EmbeddingProvider):
|
||||
"""Local ONNX embedding provider backed by FastEmbed."""
|
||||
|
||||
@@ -52,6 +73,159 @@ class FastEmbedEmbeddingProvider(EmbeddingProvider):
|
||||
self._model: TextEmbedding | None = None
|
||||
self._model_lock = asyncio.Lock()
|
||||
|
||||
def _resolved_model_name(self) -> str:
|
||||
"""Return the FastEmbed model name after applying our local aliases."""
|
||||
return self._MODEL_ALIASES.get(self.model_name, self.model_name)
|
||||
|
||||
def _create_model(self) -> "TextEmbedding":
|
||||
try:
|
||||
from fastembed import TextEmbedding
|
||||
except ImportError as exc: # pragma: no cover - exercised via tests with monkeypatch
|
||||
raise SemanticDependenciesMissingError(
|
||||
"fastembed package is missing. "
|
||||
"Install/update basic-memory to include semantic dependencies: "
|
||||
"pip install -U basic-memory"
|
||||
) from exc
|
||||
resolved_model_name = self._resolved_model_name()
|
||||
# Constraint: onnxruntime's CPU memory arena grows to fit peak usage and never
|
||||
# returns that memory to the OS. If a model is ever loaded more than once in a
|
||||
# long-running process it leaks tens of GB (#872). FastEmbed exposes
|
||||
# enable_cpu_mem_arena via its session-option kwargs, so we disable the arena to
|
||||
# let any transient extra load free memory.
|
||||
model_kwargs: dict = {
|
||||
"model_name": resolved_model_name,
|
||||
"enable_cpu_mem_arena": False,
|
||||
}
|
||||
if self.cache_dir is not None:
|
||||
model_kwargs["cache_dir"] = self.cache_dir
|
||||
if self.threads is not None:
|
||||
model_kwargs["threads"] = self.threads
|
||||
return TextEmbedding(**model_kwargs)
|
||||
|
||||
def _model_cache_candidates(self) -> list[tuple[Path, str]]:
|
||||
"""Resolve ``(snapshot_dir, model_file)`` pairs for this model under ``cache_dir``.
|
||||
|
||||
FastEmbed stores each model under ``<cache_dir>/models--<org>--<repo>`` where the
|
||||
repo is the model's HuggingFace source (e.g. ``BAAI/bge-small-en-v1.5`` resolves to
|
||||
``models--qdrant--bge-small-en-v1.5-onnx-q``). We resolve the source and the expected
|
||||
model artifact filename from FastEmbed's own model description so corruption detection
|
||||
and deletion are scoped to exactly this model's tree — never the whole cache or
|
||||
unrelated models.
|
||||
|
||||
Note: ``TextEmbedding._list_supported_models()`` is an intentional use of an
|
||||
undocumented FastEmbed API. The broad ``except`` below is a known defensive fallback:
|
||||
if the lookup ever changes shape we degrade to "no candidates" (so we never purge)
|
||||
rather than crashing the load path.
|
||||
"""
|
||||
if self.cache_dir is None:
|
||||
return []
|
||||
|
||||
# FastEmbed matches model names case-insensitively (model_management.py:
|
||||
# ``model_name.lower() == model.model.lower()``). Mirror that here so a config like
|
||||
# model="baai/bge-small-en-v1.5" still resolves to the same HF source/cache subdir.
|
||||
resolved_model_name = self._resolved_model_name().lower()
|
||||
candidates: list[tuple[Path, str]] = []
|
||||
seen: set[Path] = set()
|
||||
cache_root = Path(self.cache_dir)
|
||||
try:
|
||||
from fastembed import TextEmbedding
|
||||
|
||||
for description in TextEmbedding._list_supported_models():
|
||||
if description.model.lower() != resolved_model_name:
|
||||
continue
|
||||
hf_source = description.sources.hf
|
||||
model_file = description.model_file
|
||||
if not hf_source or not model_file:
|
||||
continue
|
||||
# HuggingFace hub names cache dirs ``models--<repo with '/' -> '--'>``.
|
||||
snapshot_dir = cache_root / f"models--{hf_source.replace('/', '--')}"
|
||||
if snapshot_dir not in seen:
|
||||
seen.add(snapshot_dir)
|
||||
candidates.append((snapshot_dir, model_file))
|
||||
except Exception as exc: # pragma: no cover - defensive: never block load on lookup
|
||||
logger.warning(
|
||||
"Could not resolve FastEmbed model source for cache cleanup: "
|
||||
"model_name={model_name} error={error}",
|
||||
model_name=resolved_model_name,
|
||||
error=exc,
|
||||
)
|
||||
|
||||
return candidates
|
||||
|
||||
def _corrupt_model_subdirs(self) -> list[Path]:
|
||||
"""Return cache subdirs that are POSITIVELY confirmed corrupt by filesystem state.
|
||||
|
||||
A model is corrupt when its HuggingFace cache dir exists on disk but at least one
|
||||
materialized snapshot revision is missing the expected model artifact file (e.g.
|
||||
``model_optimized.onnx``) — the exact fingerprint of an interrupted download. A normal
|
||||
cold load (no cache dir yet) is NOT corruption and yields no entries here, so it can
|
||||
never trigger a purge.
|
||||
|
||||
Inspection is PER-REVISION on purpose: HuggingFace keeps multiple revisions under one
|
||||
``models--<repo>`` tree, so a corrupt current snapshot can coexist with an older
|
||||
complete one. Checking ``rglob(model_file)`` across the whole tree would let the old
|
||||
artifact mask the broken current revision and leave it self-perpetuating, so we
|
||||
require every revision to carry the artifact.
|
||||
"""
|
||||
corrupt: list[Path] = []
|
||||
for model_dir, model_file in self._model_cache_candidates():
|
||||
# Trigger: the model's cache dir does not exist at all.
|
||||
# Why: this is a normal cold/first load — the model simply hasn't been
|
||||
# downloaded yet. Purging here would be wrong and pointless.
|
||||
# Outcome: skip; not corrupt.
|
||||
if not model_dir.exists():
|
||||
continue
|
||||
snapshots_root = model_dir / "snapshots"
|
||||
revision_dirs = (
|
||||
[d for d in snapshots_root.iterdir() if d.is_dir()]
|
||||
if snapshots_root.is_dir()
|
||||
else []
|
||||
)
|
||||
# Trigger: the cache dir exists but no snapshot revision has materialized.
|
||||
# Why/Outcome: an interrupted download that never wrote a revision — corrupt.
|
||||
if not revision_dirs:
|
||||
corrupt.append(model_dir)
|
||||
continue
|
||||
# Trigger: any individual revision is missing the artifact (rglob covers the
|
||||
# artifact at any depth within that revision, e.g. snapshots/<rev>/onnx/...).
|
||||
# Why: a complete OLD revision must not mask a corrupt CURRENT one.
|
||||
# Outcome: flag the model dir so the whole tree re-downloads cleanly.
|
||||
if any(not any(rev.rglob(model_file)) for rev in revision_dirs):
|
||||
corrupt.append(model_dir)
|
||||
return corrupt
|
||||
|
||||
def _purge_model_subdirs(self, subdirs: list[Path]) -> bool:
|
||||
"""Delete confirmed-corrupt cache subtrees so the next load re-downloads them.
|
||||
|
||||
Returns True when at least one targeted subdir is actually gone afterwards. On
|
||||
Windows a locked file can make ``shutil.rmtree(ignore_errors=True)`` silently no-op;
|
||||
reporting success in that case would let the caller retry against the same broken
|
||||
cache, so each subdir only counts as removed once it has actually disappeared.
|
||||
"""
|
||||
removed_any = False
|
||||
for subdir in subdirs:
|
||||
logger.warning(
|
||||
"Removing corrupt FastEmbed model cache to force re-download: {path}",
|
||||
path=str(subdir),
|
||||
)
|
||||
shutil.rmtree(subdir, ignore_errors=True)
|
||||
# Set removed only when the subdir is truly gone — a silent rmtree no-op
|
||||
# (e.g. a locked file on Windows) must not be reported as a successful purge.
|
||||
if not subdir.exists():
|
||||
removed_any = True
|
||||
return removed_any
|
||||
|
||||
@staticmethod
|
||||
def _is_missing_artifact_error(exc: Exception) -> bool:
|
||||
"""Return True when the load failure text matches the ONNX missing-artifact signature.
|
||||
|
||||
This is only the text-level gate; it is necessary but NOT sufficient to purge. The
|
||||
purge additionally requires filesystem-confirmed corruption (``_corrupt_model_subdirs``)
|
||||
so a transient/offline/"from any source" load error never deletes a valid cache.
|
||||
"""
|
||||
message = str(exc).lower()
|
||||
return any(marker in message for marker in _MISSING_ARTIFACT_ERROR_MARKERS)
|
||||
|
||||
async def _load_model(self) -> "TextEmbedding":
|
||||
if self._model is not None:
|
||||
return self._model
|
||||
@@ -60,36 +234,42 @@ class FastEmbedEmbeddingProvider(EmbeddingProvider):
|
||||
if self._model is not None:
|
||||
return self._model
|
||||
|
||||
def _create_model() -> "TextEmbedding":
|
||||
try:
|
||||
from fastembed import TextEmbedding
|
||||
except (
|
||||
ImportError
|
||||
) as exc: # pragma: no cover - exercised via tests with monkeypatch
|
||||
raise SemanticDependenciesMissingError(
|
||||
"fastembed package is missing. "
|
||||
"Install/update basic-memory to include semantic dependencies: "
|
||||
"pip install -U basic-memory"
|
||||
) from exc
|
||||
resolved_model_name = self._MODEL_ALIASES.get(self.model_name, self.model_name)
|
||||
if self.cache_dir is not None and self.threads is not None:
|
||||
return TextEmbedding(
|
||||
model_name=resolved_model_name,
|
||||
cache_dir=self.cache_dir,
|
||||
threads=self.threads,
|
||||
)
|
||||
if self.cache_dir is not None:
|
||||
return TextEmbedding(model_name=resolved_model_name, cache_dir=self.cache_dir)
|
||||
if self.threads is not None:
|
||||
return TextEmbedding(model_name=resolved_model_name, threads=self.threads)
|
||||
return TextEmbedding(model_name=resolved_model_name)
|
||||
try:
|
||||
self._model = await asyncio.to_thread(self._create_model)
|
||||
except Exception as exc:
|
||||
# Trigger: model construction raised the ONNX missing-artifact error AND a
|
||||
# filesystem check positively confirms a corrupt cache subdir (the
|
||||
# snapshot dir exists but the model artifact file is missing — the
|
||||
# fingerprint of an interrupted download).
|
||||
# Why: the raw ONNXRuntimeError is self-perpetuating — every retry hits the
|
||||
# same broken snapshot until the cache is cleared. We must NOT misread a
|
||||
# normal cold load (no snapshot dir, model simply not downloaded yet) or a
|
||||
# transient/offline "from any source" error as corruption, because purging
|
||||
# then breaks the happy path. Both the error-text gate and the positive
|
||||
# filesystem confirmation are required before we delete anything.
|
||||
# Outcome: confirmed corruption → purge exactly this model's subdir and retry
|
||||
# once so a fresh download can land. Every other failure (including a
|
||||
# retry that still fails) re-raises the ORIGINAL exception so the
|
||||
# message stays actionable and we never loop.
|
||||
if not self._is_missing_artifact_error(exc):
|
||||
raise
|
||||
corrupt_subdirs = self._corrupt_model_subdirs()
|
||||
if not corrupt_subdirs:
|
||||
raise
|
||||
if not self._purge_model_subdirs(corrupt_subdirs):
|
||||
raise
|
||||
logger.info(
|
||||
"Retrying FastEmbed model load after clearing corrupt cache: "
|
||||
"model_name={model_name}",
|
||||
model_name=self._resolved_model_name(),
|
||||
)
|
||||
self._model = await asyncio.to_thread(self._create_model)
|
||||
|
||||
self._model = await asyncio.to_thread(_create_model)
|
||||
logger.info(
|
||||
"FastEmbed model loaded: model_name={model_name} batch_size={batch_size} "
|
||||
"threads={threads} configured_parallel={configured_parallel} "
|
||||
"effective_parallel={effective_parallel}",
|
||||
model_name=self._MODEL_ALIASES.get(self.model_name, self.model_name),
|
||||
model_name=self._resolved_model_name(),
|
||||
batch_size=self.batch_size,
|
||||
threads=self.threads,
|
||||
configured_parallel=self.parallel,
|
||||
@@ -119,10 +299,17 @@ class FastEmbedEmbeddingProvider(EmbeddingProvider):
|
||||
if effective_parallel is not None:
|
||||
embed_kwargs["parallel"] = effective_parallel
|
||||
vectors = list(model.embed(texts, **embed_kwargs))
|
||||
# sqlite_search_repository.py uses a distance-to-similarity formula that assumes
|
||||
# unit-normalized vectors (see the comment on line 65-67 of that file).
|
||||
# Some models (e.g. multilingual ones) return vectors with norm > 1, so we
|
||||
# L2-normalize here to satisfy that contract regardless of the chosen model.
|
||||
normalized: list[list[float]] = []
|
||||
for vector in vectors:
|
||||
values = vector.tolist() if hasattr(vector, "tolist") else vector
|
||||
normalized.append([float(value) for value in values])
|
||||
values = vector.tolist() if hasattr(vector, "tolist") else list(vector)
|
||||
norm = math.sqrt(sum(x * x for x in values))
|
||||
if norm > 0:
|
||||
values = [x / norm for x in values]
|
||||
normalized.append([float(v) for v in values])
|
||||
return normalized
|
||||
|
||||
vectors = await asyncio.to_thread(_embed_batch)
|
||||
|
||||
@@ -0,0 +1,230 @@
|
||||
"""LiteLLM-based embedding provider for semantic indexing.
|
||||
|
||||
Routes embedding requests to 100+ providers (OpenAI, Anthropic, Google, Azure,
|
||||
Bedrock, Cohere, etc.) via the litellm SDK. No proxy server needed.
|
||||
|
||||
Model strings use the ``provider/model`` format, e.g.
|
||||
``openai/text-embedding-3-small``, ``cohere/embed-english-v3.0``,
|
||||
``azure/my-embedding-deployment``.
|
||||
|
||||
See https://docs.litellm.ai/docs/embedding/supported_embedding for all
|
||||
supported embedding models.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import math
|
||||
import os
|
||||
from typing import Any
|
||||
|
||||
from basic_memory.repository.embedding_provider import EmbeddingProvider
|
||||
from basic_memory.repository.semantic_errors import SemanticDependenciesMissingError
|
||||
|
||||
|
||||
def _default_input_types(model_name: str) -> tuple[str | None, str | None]:
|
||||
"""Return role-specific LiteLLM input_type defaults for known asymmetric models."""
|
||||
normalized = model_name.strip().lower()
|
||||
|
||||
# Cohere v3 embeddings require search_document/search_query to distinguish
|
||||
# index-time passages from retrieval-time queries. LiteLLM supports both
|
||||
# direct Cohere model names and provider-prefixed forms.
|
||||
cohere_v3 = (
|
||||
normalized.startswith("cohere/")
|
||||
or normalized.startswith("bedrock/cohere.")
|
||||
or normalized.startswith("cohere.")
|
||||
or normalized.startswith("embed-")
|
||||
) and "-v3" in normalized
|
||||
if cohere_v3:
|
||||
return "search_document", "search_query"
|
||||
|
||||
# NVIDIA retrieval embeddings use passage/query roles. The provider prefix
|
||||
# is part of LiteLLM's model routing, so this stays narrowly scoped.
|
||||
if normalized.startswith("nvidia_nim/"):
|
||||
return "passage", "query"
|
||||
|
||||
return None, None
|
||||
|
||||
|
||||
def _should_forward_dimensions(model_name: str, forward_dimensions: bool | None) -> bool:
|
||||
"""Return whether configured dimensions should be sent to LiteLLM."""
|
||||
if forward_dimensions is not None:
|
||||
return forward_dimensions
|
||||
|
||||
normalized = model_name.strip().lower()
|
||||
|
||||
# Trigger: `dimensions` is both the Basic Memory vector schema size and a
|
||||
# provider-side output-size request parameter in LiteLLM.
|
||||
# Why: fixed-size models such as Cohere v3 still need the schema value for
|
||||
# validation, but LiteLLM maps the request parameter to provider fields they
|
||||
# reject. Only send it for model families that clearly support output-size
|
||||
# control.
|
||||
# Outcome: OpenAI/Azure text-embedding-3 reductions work, while fixed-size
|
||||
# providers keep dimensions local to Basic Memory.
|
||||
return "text-embedding-3" in normalized
|
||||
|
||||
|
||||
def _import_litellm() -> Any:
|
||||
"""Import LiteLLM without letting its import-time dotenv hook read cwd secrets."""
|
||||
# Constraint: LiteLLM 1.85.0 loads .env files at import time when
|
||||
# LITELLM_MODE defaults to DEV. Basic Memory intentionally does not load
|
||||
# arbitrary cwd .env files, so set the production mode before importing
|
||||
# unless the caller already made an explicit LiteLLM choice.
|
||||
os.environ.setdefault("LITELLM_MODE", "PRODUCTION")
|
||||
|
||||
try:
|
||||
import litellm
|
||||
except ImportError as exc:
|
||||
raise SemanticDependenciesMissingError(
|
||||
"litellm dependency is missing. Install with: pip install litellm"
|
||||
) from exc
|
||||
|
||||
return litellm
|
||||
|
||||
|
||||
class LiteLLMEmbeddingProvider(EmbeddingProvider):
|
||||
"""Embedding provider backed by the litellm SDK."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
model_name: str = "openai/text-embedding-3-small",
|
||||
*,
|
||||
batch_size: int = 64,
|
||||
request_concurrency: int = 4,
|
||||
dimensions: int = 1536,
|
||||
api_key: str | None = None,
|
||||
timeout: float = 30.0,
|
||||
document_input_type: str | None = None,
|
||||
query_input_type: str | None = None,
|
||||
forward_dimensions: bool | None = None,
|
||||
) -> None:
|
||||
self.model_name = model_name
|
||||
self.dimensions = dimensions
|
||||
self.batch_size = batch_size
|
||||
self.request_concurrency = request_concurrency
|
||||
self._api_key = api_key
|
||||
self._timeout = timeout
|
||||
default_document_input_type, default_query_input_type = _default_input_types(model_name)
|
||||
self.document_input_type = document_input_type or default_document_input_type
|
||||
self.query_input_type = query_input_type or default_query_input_type
|
||||
self.forward_dimensions = forward_dimensions
|
||||
|
||||
def runtime_log_attrs(self) -> dict[str, Any]:
|
||||
"""Return provider-specific runtime settings suitable for startup logs."""
|
||||
attrs: dict[str, Any] = {
|
||||
"provider_batch_size": self.batch_size,
|
||||
"request_concurrency": self.request_concurrency,
|
||||
}
|
||||
if self.document_input_type:
|
||||
attrs["document_input_type"] = self.document_input_type
|
||||
if self.query_input_type:
|
||||
attrs["query_input_type"] = self.query_input_type
|
||||
if self.forward_dimensions is not None:
|
||||
attrs["forward_dimensions"] = self.forward_dimensions
|
||||
return attrs
|
||||
|
||||
def identity_key(self) -> str:
|
||||
"""Return the embedding semantics that should invalidate stored vectors."""
|
||||
document_input_type = self.document_input_type or "-"
|
||||
query_input_type = self.query_input_type or "-"
|
||||
forward_dimensions = str(
|
||||
_should_forward_dimensions(self.model_name, self.forward_dimensions)
|
||||
).lower()
|
||||
return (
|
||||
f"{self.model_name}:{self.dimensions}:"
|
||||
f"document_input_type={document_input_type}:"
|
||||
f"query_input_type={query_input_type}:"
|
||||
f"forward_dimensions={forward_dimensions}"
|
||||
)
|
||||
|
||||
async def _embed(self, texts: list[str], *, input_type: str | None) -> list[list[float]]:
|
||||
if not texts:
|
||||
return []
|
||||
|
||||
litellm = _import_litellm()
|
||||
|
||||
batches = [
|
||||
texts[start : start + self.batch_size]
|
||||
for start in range(0, len(texts), self.batch_size)
|
||||
]
|
||||
batch_vectors: list[list[list[float]] | None] = [None] * len(batches)
|
||||
semaphore = asyncio.Semaphore(self.request_concurrency)
|
||||
|
||||
async def embed_batch(batch_index: int, batch: list[str]) -> None:
|
||||
async with semaphore:
|
||||
params: dict[str, Any] = {
|
||||
"model": self.model_name,
|
||||
"input": batch,
|
||||
"drop_params": True,
|
||||
"timeout": self._timeout,
|
||||
}
|
||||
if _should_forward_dimensions(self.model_name, self.forward_dimensions):
|
||||
params["dimensions"] = self.dimensions
|
||||
if self._api_key:
|
||||
params["api_key"] = self._api_key
|
||||
if input_type:
|
||||
params["input_type"] = input_type
|
||||
|
||||
response = await litellm.aembedding(**params)
|
||||
|
||||
vectors_by_index: dict[int, list[float]] = {}
|
||||
for item in response.data:
|
||||
if isinstance(item, dict):
|
||||
response_index = int(item["index"])
|
||||
embedding = item["embedding"]
|
||||
else:
|
||||
response_index = int(item.index)
|
||||
embedding = item.embedding
|
||||
if response_index in vectors_by_index:
|
||||
raise RuntimeError(
|
||||
"LiteLLM embedding response returned duplicate vector indexes."
|
||||
)
|
||||
vectors_by_index[response_index] = [float(v) for v in embedding]
|
||||
|
||||
ordered_vectors: list[list[float]] = []
|
||||
for index in range(len(batch)):
|
||||
vector = vectors_by_index.get(index)
|
||||
if vector is None:
|
||||
raise RuntimeError(
|
||||
"LiteLLM embedding response is missing expected vector index."
|
||||
)
|
||||
ordered_vectors.append(vector)
|
||||
|
||||
batch_vectors[batch_index] = ordered_vectors
|
||||
|
||||
await asyncio.gather(
|
||||
*(embed_batch(batch_index, batch) for batch_index, batch in enumerate(batches))
|
||||
)
|
||||
|
||||
all_vectors: list[list[float]] = []
|
||||
for vectors in batch_vectors:
|
||||
if vectors is None:
|
||||
raise RuntimeError("LiteLLM embedding batch did not produce vectors.")
|
||||
all_vectors.extend(vectors)
|
||||
|
||||
# sqlite_search_repository.py maps L2 distance to cosine similarity via
|
||||
# `1 - L²/2`, which is correct only for unit-normalized vectors. LiteLLM
|
||||
# routes to many backends (Cohere, Vertex, Bedrock, etc.); not all of
|
||||
# them return normalized embeddings, so we normalize here to honor the
|
||||
# provider contract regardless of the underlying model.
|
||||
normalized: list[list[float]] = []
|
||||
for vector in all_vectors:
|
||||
norm = math.sqrt(sum(x * x for x in vector))
|
||||
if norm > 0:
|
||||
normalized.append([x / norm for x in vector])
|
||||
else:
|
||||
normalized.append(vector)
|
||||
|
||||
if normalized and len(normalized[0]) != self.dimensions:
|
||||
raise RuntimeError(
|
||||
f"Embedding model returned {len(normalized[0])}-dimensional vectors "
|
||||
f"but provider was configured for {self.dimensions} dimensions."
|
||||
)
|
||||
return normalized
|
||||
|
||||
async def embed_documents(self, texts: list[str]) -> list[list[float]]:
|
||||
return await self._embed(texts, input_type=self.document_input_type)
|
||||
|
||||
async def embed_query(self, text: str) -> list[float]:
|
||||
vectors = await self._embed([text], input_type=self.query_input_type)
|
||||
return vectors[0] if vectors else [0.0] * self.dimensions
|
||||
@@ -705,6 +705,7 @@ class PostgresSearchRepository(SearchRepositoryBase):
|
||||
note_types: Optional[List[str]] = None,
|
||||
after_date: Optional[datetime] = None,
|
||||
search_item_types: Optional[List[SearchItemType]] = None,
|
||||
categories: Optional[List[str]] = None,
|
||||
metadata_filters: Optional[dict] = None,
|
||||
) -> tuple[str, str, dict, str, str]:
|
||||
"""Build Postgres FTS FROM/WHERE params shared by search and count."""
|
||||
@@ -762,14 +763,36 @@ class PostgresSearchRepository(SearchRepositoryBase):
|
||||
type_placeholders.append(f":{param_name}")
|
||||
conditions.append(f"search_index.type IN ({', '.join(type_placeholders)})")
|
||||
|
||||
# Handle note type filter using JSONB containment (parameterized)
|
||||
# Handle observation category filter (parameterized for defense-in-depth).
|
||||
# Trigger: caller passed `categories` to scope observation results.
|
||||
# Why: `entity_types=["observation"]` only narrows to the observation row type;
|
||||
# callers expect exact-category matching, not incidental text matches.
|
||||
# Outcome: only rows whose indexed category exactly equals a requested value
|
||||
# survive (entities/relations have NULL category and are excluded).
|
||||
if categories:
|
||||
category_placeholders = []
|
||||
for idx, category in enumerate(categories):
|
||||
param_name = f"category_{idx}"
|
||||
params[param_name] = category
|
||||
category_placeholders.append(f":{param_name}")
|
||||
conditions.append(f"search_index.category IN ({', '.join(category_placeholders)})")
|
||||
|
||||
# Handle note type filter (frontmatter type field, parameterized).
|
||||
# Trigger: caller passed `note_types` to scope by the frontmatter `type` field.
|
||||
# Why: the stored note_type preserves the frontmatter casing (e.g. `Chapter`),
|
||||
# but the filter is documented case-insensitive. JSONB `@>` containment is
|
||||
# exact-match, so capitalized types were unfindable.
|
||||
# Outcome: compare LOWER(metadata->>'note_type') against lowercased filter
|
||||
# values so `note_types=["Chapter"]` matches a stored `Chapter`.
|
||||
if note_types:
|
||||
type_conditions = []
|
||||
type_placeholders = []
|
||||
for idx, note_type in enumerate(note_types):
|
||||
param_name = f"note_type_{idx}"
|
||||
params[param_name] = json.dumps({"note_type": note_type})
|
||||
type_conditions.append(f"search_index.metadata @> CAST(:{param_name} AS jsonb)")
|
||||
conditions.append(f"({' OR '.join(type_conditions)})")
|
||||
params[param_name] = note_type.lower()
|
||||
type_placeholders.append(f":{param_name}")
|
||||
conditions.append(
|
||||
f"LOWER(search_index.metadata->>'note_type') IN ({', '.join(type_placeholders)})"
|
||||
)
|
||||
|
||||
# Handle date filter
|
||||
if after_date:
|
||||
@@ -879,6 +902,7 @@ class PostgresSearchRepository(SearchRepositoryBase):
|
||||
note_types: Optional[List[str]] = None,
|
||||
after_date: Optional[datetime] = None,
|
||||
search_item_types: Optional[List[SearchItemType]] = None,
|
||||
categories: Optional[List[str]] = None,
|
||||
metadata_filters: Optional[dict] = None,
|
||||
retrieval_mode: SearchRetrievalMode = SearchRetrievalMode.FTS,
|
||||
min_similarity: Optional[float] = None,
|
||||
@@ -895,6 +919,7 @@ class PostgresSearchRepository(SearchRepositoryBase):
|
||||
note_types=note_types,
|
||||
after_date=after_date,
|
||||
search_item_types=search_item_types,
|
||||
categories=categories,
|
||||
metadata_filters=metadata_filters,
|
||||
retrieval_mode=retrieval_mode,
|
||||
min_similarity=min_similarity,
|
||||
@@ -919,6 +944,7 @@ class PostgresSearchRepository(SearchRepositoryBase):
|
||||
note_types=note_types,
|
||||
after_date=after_date,
|
||||
search_item_types=search_item_types,
|
||||
categories=categories,
|
||||
metadata_filters=metadata_filters,
|
||||
)
|
||||
|
||||
@@ -1008,6 +1034,7 @@ class PostgresSearchRepository(SearchRepositoryBase):
|
||||
note_types: Optional[List[str]] = None,
|
||||
after_date: Optional[datetime] = None,
|
||||
search_item_types: Optional[List[SearchItemType]] = None,
|
||||
categories: Optional[List[str]] = None,
|
||||
metadata_filters: Optional[dict] = None,
|
||||
retrieval_mode: SearchRetrievalMode = SearchRetrievalMode.FTS,
|
||||
min_similarity: Optional[float] = None,
|
||||
@@ -1022,6 +1049,7 @@ class PostgresSearchRepository(SearchRepositoryBase):
|
||||
note_types=note_types,
|
||||
after_date=after_date,
|
||||
search_item_types=search_item_types,
|
||||
categories=categories,
|
||||
metadata_filters=metadata_filters,
|
||||
retrieval_mode=retrieval_mode,
|
||||
min_similarity=min_similarity,
|
||||
@@ -1041,6 +1069,7 @@ class PostgresSearchRepository(SearchRepositoryBase):
|
||||
note_types=note_types,
|
||||
after_date=after_date,
|
||||
search_item_types=search_item_types,
|
||||
categories=categories,
|
||||
metadata_filters=metadata_filters,
|
||||
)
|
||||
sql = f"SELECT COUNT(*) FROM {from_clause} WHERE {where_clause}"
|
||||
|
||||
@@ -5,7 +5,7 @@ from typing import Optional, Sequence, Union
|
||||
|
||||
|
||||
from loguru import logger
|
||||
from sqlalchemy import inspect as sa_inspect, select, text
|
||||
from sqlalchemy import Executable, inspect as sa_inspect, select, text
|
||||
from sqlalchemy.exc import NoResultFound, OperationalError
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
|
||||
@@ -258,6 +258,28 @@ class ProjectRepository(Repository[Project]):
|
||||
logger.debug(f"Deleted Project and search rows for project_id: {entity_id}")
|
||||
return True
|
||||
|
||||
async def scalar_vec_query(
|
||||
self, query: Executable, params: Optional[dict] = None
|
||||
) -> Optional[int]:
|
||||
"""Run a scalar COUNT query that reads the sqlite-vec vec0 table.
|
||||
|
||||
Extension loading is per-connection, so the bare pooled session used by
|
||||
`execute_query` cannot read vec0 virtual tables — SQLite raises
|
||||
"no such module: vec0". This helper loads sqlite-vec on the session it
|
||||
opens before running the query, reusing the same loader as project delete.
|
||||
|
||||
Returns None when sqlite-vec cannot be loaded on this Python build, so
|
||||
callers can fall back to the genuinely-missing-dependency path.
|
||||
"""
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
# Trigger: query reads the vec0-backed search_vector_embeddings table.
|
||||
# Why: vec0 modules are only visible on a connection that loaded sqlite-vec.
|
||||
# Outcome: load the extension here, or signal absence so the caller degrades.
|
||||
if not await _load_sqlite_vec_on_session(session):
|
||||
return None
|
||||
result = await session.execute(query, params)
|
||||
return result.scalar()
|
||||
|
||||
async def update_path(self, project_id: int, new_path: str) -> Optional[Project]:
|
||||
"""Update project path.
|
||||
|
||||
|
||||
@@ -13,6 +13,7 @@ from sqlalchemy import Result
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
|
||||
from basic_memory.config import BasicMemoryConfig, ConfigManager, DatabaseBackend
|
||||
from basic_memory.repository.embedding_provider_factory import create_embedding_provider
|
||||
from basic_memory.repository.postgres_search_repository import PostgresSearchRepository
|
||||
from basic_memory.repository.search_index_row import SearchIndexRow
|
||||
from basic_memory.repository.search_repository_base import VectorSyncBatchResult
|
||||
@@ -41,6 +42,7 @@ class SearchRepository(Protocol):
|
||||
note_types: Optional[List[str]] = None,
|
||||
after_date: Optional[datetime] = None,
|
||||
search_item_types: Optional[List[SearchItemType]] = None,
|
||||
categories: Optional[List[str]] = None,
|
||||
metadata_filters: Optional[dict] = None,
|
||||
retrieval_mode: SearchRetrievalMode = SearchRetrievalMode.FTS,
|
||||
min_similarity: Optional[float] = None,
|
||||
@@ -59,6 +61,7 @@ class SearchRepository(Protocol):
|
||||
note_types: Optional[List[str]] = None,
|
||||
after_date: Optional[datetime] = None,
|
||||
search_item_types: Optional[List[SearchItemType]] = None,
|
||||
categories: Optional[List[str]] = None,
|
||||
metadata_filters: Optional[dict] = None,
|
||||
retrieval_mode: SearchRetrievalMode = SearchRetrievalMode.FTS,
|
||||
min_similarity: Optional[float] = None,
|
||||
@@ -120,19 +123,37 @@ def create_search_repository(
|
||||
Returns:
|
||||
SearchRepository: Backend-appropriate search repository instance
|
||||
"""
|
||||
# Prefer explicit parameter; fall back to ConfigManager for backwards compatibility
|
||||
# Resolve config once so backend detection and the shared embedding provider
|
||||
# come from the same source. Prefer the explicit arg; fall back to ConfigManager
|
||||
# for backwards compatibility.
|
||||
config = app_config or ConfigManager().config
|
||||
if database_backend is None:
|
||||
config = app_config or ConfigManager().config
|
||||
database_backend = config.database_backend
|
||||
|
||||
# Trigger: every request, sync batch, and project builds its own search repo.
|
||||
# Why: each repo __init__ would otherwise call create_embedding_provider(), and
|
||||
# the process-wide cache can be bypassed if its key ever drifts (#872), reloading
|
||||
# the ~2.3GB ONNX model and leaking memory in onnxruntime's CPU arena.
|
||||
# Outcome: resolve the cached singleton here once and inject it, so the provider
|
||||
# is the single source of truth across all callers of this factory.
|
||||
embedding_provider = None
|
||||
if config.semantic_search_enabled:
|
||||
embedding_provider = create_embedding_provider(config)
|
||||
|
||||
if database_backend == DatabaseBackend.POSTGRES: # pragma: no cover
|
||||
return PostgresSearchRepository( # pragma: no cover
|
||||
session_maker,
|
||||
project_id=project_id,
|
||||
app_config=app_config,
|
||||
embedding_provider=embedding_provider,
|
||||
)
|
||||
else:
|
||||
return SQLiteSearchRepository(session_maker, project_id=project_id, app_config=app_config)
|
||||
return SQLiteSearchRepository(
|
||||
session_maker,
|
||||
project_id=project_id,
|
||||
app_config=app_config,
|
||||
embedding_provider=embedding_provider,
|
||||
)
|
||||
|
||||
|
||||
__all__ = [
|
||||
|
||||
@@ -218,6 +218,7 @@ class SearchRepositoryBase(ABC):
|
||||
note_types: Optional[List[str]] = None,
|
||||
after_date: Optional[datetime] = None,
|
||||
search_item_types: Optional[List[SearchItemType]] = None,
|
||||
categories: Optional[List[str]] = None,
|
||||
metadata_filters: Optional[Dict[str, Any]] = None,
|
||||
retrieval_mode: SearchRetrievalMode = SearchRetrievalMode.FTS,
|
||||
min_similarity: Optional[float] = None,
|
||||
@@ -234,6 +235,7 @@ class SearchRepositoryBase(ABC):
|
||||
note_types: Filter by note types (from metadata.note_type)
|
||||
after_date: Filter by created_at > after_date
|
||||
search_item_types: Filter by SearchItemType (ENTITY, OBSERVATION, RELATION)
|
||||
categories: Filter observations by exact category (e.g. "requirement")
|
||||
metadata_filters: Structured frontmatter metadata filters
|
||||
limit: Maximum results to return
|
||||
offset: Number of results to skip
|
||||
@@ -256,6 +258,7 @@ class SearchRepositoryBase(ABC):
|
||||
note_types: Optional[List[str]] = None,
|
||||
after_date: Optional[datetime] = None,
|
||||
search_item_types: Optional[List[SearchItemType]] = None,
|
||||
categories: Optional[List[str]] = None,
|
||||
metadata_filters: Optional[Dict[str, Any]] = None,
|
||||
retrieval_mode: SearchRetrievalMode = SearchRetrievalMode.FTS,
|
||||
min_similarity: Optional[float] = None,
|
||||
@@ -586,11 +589,19 @@ class SearchRepositoryBase(ABC):
|
||||
def _embedding_model_key(self) -> str:
|
||||
"""Build a stable model identity for vector invalidation checks."""
|
||||
assert self._embedding_provider is not None
|
||||
return (
|
||||
f"{type(self._embedding_provider).__name__}:"
|
||||
f"{self._embedding_provider.model_name}:"
|
||||
f"{self._embedding_provider.dimensions}"
|
||||
)
|
||||
provider = self._embedding_provider
|
||||
|
||||
provider_identity = f"{provider.model_name}:{provider.dimensions}"
|
||||
from basic_memory.repository.litellm_provider import LiteLLMEmbeddingProvider
|
||||
|
||||
if isinstance(provider, LiteLLMEmbeddingProvider):
|
||||
# Trigger: LiteLLM can change request semantics without changing model/dimensions.
|
||||
# Why: asymmetric providers use role-specific document/query params, and
|
||||
# dimension forwarding changes provider-side output-size behavior.
|
||||
# Outcome: reindex treats those semantic config changes as stale vectors.
|
||||
provider_identity = provider.identity_key()
|
||||
|
||||
return f"{type(provider).__name__}:{provider_identity}"
|
||||
|
||||
def _plan_entity_vector_shard(
|
||||
self,
|
||||
@@ -1777,6 +1788,7 @@ class SearchRepositoryBase(ABC):
|
||||
note_types: Optional[List[str]],
|
||||
after_date: Optional[datetime],
|
||||
search_item_types: Optional[List[SearchItemType]],
|
||||
categories: Optional[List[str]],
|
||||
metadata_filters: Optional[dict],
|
||||
retrieval_mode: SearchRetrievalMode,
|
||||
min_similarity: Optional[float] = None,
|
||||
@@ -1810,6 +1822,7 @@ class SearchRepositoryBase(ABC):
|
||||
note_types=note_types,
|
||||
after_date=after_date,
|
||||
search_item_types=search_item_types,
|
||||
categories=categories,
|
||||
metadata_filters=metadata_filters,
|
||||
min_similarity=min_similarity,
|
||||
limit=limit,
|
||||
@@ -1829,6 +1842,7 @@ class SearchRepositoryBase(ABC):
|
||||
note_types=note_types,
|
||||
after_date=after_date,
|
||||
search_item_types=search_item_types,
|
||||
categories=categories,
|
||||
metadata_filters=metadata_filters,
|
||||
min_similarity=min_similarity,
|
||||
limit=limit,
|
||||
@@ -1858,6 +1872,7 @@ class SearchRepositoryBase(ABC):
|
||||
note_types: Optional[List[str]],
|
||||
after_date: Optional[datetime],
|
||||
search_item_types: Optional[List[SearchItemType]],
|
||||
categories: Optional[List[str]],
|
||||
metadata_filters: Optional[dict],
|
||||
min_similarity: Optional[float] = None,
|
||||
limit: int,
|
||||
@@ -1968,6 +1983,7 @@ class SearchRepositoryBase(ABC):
|
||||
note_types,
|
||||
after_date,
|
||||
search_item_types,
|
||||
categories,
|
||||
metadata_filters,
|
||||
]
|
||||
)
|
||||
@@ -1981,6 +1997,7 @@ class SearchRepositoryBase(ABC):
|
||||
note_types=note_types,
|
||||
after_date=after_date,
|
||||
search_item_types=search_item_types,
|
||||
categories=categories,
|
||||
metadata_filters=metadata_filters,
|
||||
retrieval_mode=SearchRetrievalMode.FTS,
|
||||
limit=VECTOR_FILTER_SCAN_LIMIT,
|
||||
@@ -2131,6 +2148,7 @@ class SearchRepositoryBase(ABC):
|
||||
note_types: Optional[List[str]],
|
||||
after_date: Optional[datetime],
|
||||
search_item_types: Optional[List[SearchItemType]],
|
||||
categories: Optional[List[str]],
|
||||
metadata_filters: Optional[dict],
|
||||
min_similarity: Optional[float] = None,
|
||||
limit: int,
|
||||
@@ -2155,6 +2173,7 @@ class SearchRepositoryBase(ABC):
|
||||
note_types=note_types,
|
||||
after_date=after_date,
|
||||
search_item_types=search_item_types,
|
||||
categories=categories,
|
||||
metadata_filters=metadata_filters,
|
||||
retrieval_mode=SearchRetrievalMode.FTS,
|
||||
limit=candidate_limit,
|
||||
@@ -2170,6 +2189,7 @@ class SearchRepositoryBase(ABC):
|
||||
note_types=note_types,
|
||||
after_date=after_date,
|
||||
search_item_types=search_item_types,
|
||||
categories=categories,
|
||||
metadata_filters=metadata_filters,
|
||||
min_similarity=min_similarity,
|
||||
limit=candidate_limit,
|
||||
|
||||
@@ -733,6 +733,7 @@ class SQLiteSearchRepository(SearchRepositoryBase):
|
||||
note_types: Optional[List[str]] = None,
|
||||
after_date: Optional[datetime] = None,
|
||||
search_item_types: Optional[List[SearchItemType]] = None,
|
||||
categories: Optional[List[str]] = None,
|
||||
metadata_filters: Optional[dict] = None,
|
||||
) -> tuple[str, str, dict, str]:
|
||||
"""Build SQLite FTS FROM/WHERE params shared by search and count."""
|
||||
@@ -794,15 +795,36 @@ class SQLiteSearchRepository(SearchRepositoryBase):
|
||||
type_placeholders.append(f":{param_name}")
|
||||
conditions.append(f"search_index.type IN ({', '.join(type_placeholders)})")
|
||||
|
||||
# Handle note type filter (frontmatter type field, parameterized)
|
||||
# Handle observation category filter (parameterized for defense-in-depth).
|
||||
# Trigger: caller passed `categories` to scope observation results.
|
||||
# Why: `entity_types=["observation"]` only narrows to the observation row type;
|
||||
# callers expect exact-category matching, not incidental text matches.
|
||||
# Outcome: only rows whose indexed category exactly equals a requested value
|
||||
# survive (entities/relations have NULL category and are excluded).
|
||||
if categories:
|
||||
category_placeholders = []
|
||||
for idx, category in enumerate(categories):
|
||||
param_name = f"category_{idx}"
|
||||
params[param_name] = category
|
||||
category_placeholders.append(f":{param_name}")
|
||||
conditions.append(f"search_index.category IN ({', '.join(category_placeholders)})")
|
||||
|
||||
# Handle note type filter (frontmatter type field, parameterized).
|
||||
# Trigger: caller passed `note_types` to scope by the frontmatter `type` field.
|
||||
# Why: the stored note_type preserves the frontmatter casing (e.g. `Chapter`),
|
||||
# but the filter is documented case-insensitive; comparing raw values
|
||||
# would miss capitalized types.
|
||||
# Outcome: fold both sides to lowercase so `note_types=["Chapter"]` matches a
|
||||
# stored `Chapter`, `chapter`, etc.
|
||||
if note_types:
|
||||
type_placeholders = []
|
||||
for idx, t in enumerate(note_types):
|
||||
param_name = f"note_type_{idx}"
|
||||
params[param_name] = t
|
||||
params[param_name] = t.lower()
|
||||
type_placeholders.append(f":{param_name}")
|
||||
conditions.append(
|
||||
f"json_extract(search_index.metadata, '$.note_type') IN ({', '.join(type_placeholders)})"
|
||||
"LOWER(json_extract(search_index.metadata, '$.note_type')) "
|
||||
f"IN ({', '.join(type_placeholders)})"
|
||||
)
|
||||
|
||||
# Handle date filter using datetime() for proper comparison
|
||||
@@ -925,6 +947,7 @@ class SQLiteSearchRepository(SearchRepositoryBase):
|
||||
note_types: Optional[List[str]] = None,
|
||||
after_date: Optional[datetime] = None,
|
||||
search_item_types: Optional[List[SearchItemType]] = None,
|
||||
categories: Optional[List[str]] = None,
|
||||
metadata_filters: Optional[dict] = None,
|
||||
retrieval_mode: SearchRetrievalMode = SearchRetrievalMode.FTS,
|
||||
min_similarity: Optional[float] = None,
|
||||
@@ -941,6 +964,7 @@ class SQLiteSearchRepository(SearchRepositoryBase):
|
||||
note_types=note_types,
|
||||
after_date=after_date,
|
||||
search_item_types=search_item_types,
|
||||
categories=categories,
|
||||
metadata_filters=metadata_filters,
|
||||
retrieval_mode=retrieval_mode,
|
||||
min_similarity=min_similarity,
|
||||
@@ -959,6 +983,7 @@ class SQLiteSearchRepository(SearchRepositoryBase):
|
||||
note_types=note_types,
|
||||
after_date=after_date,
|
||||
search_item_types=search_item_types,
|
||||
categories=categories,
|
||||
metadata_filters=metadata_filters,
|
||||
)
|
||||
|
||||
@@ -1046,6 +1071,7 @@ class SQLiteSearchRepository(SearchRepositoryBase):
|
||||
note_types: Optional[List[str]] = None,
|
||||
after_date: Optional[datetime] = None,
|
||||
search_item_types: Optional[List[SearchItemType]] = None,
|
||||
categories: Optional[List[str]] = None,
|
||||
metadata_filters: Optional[dict] = None,
|
||||
retrieval_mode: SearchRetrievalMode = SearchRetrievalMode.FTS,
|
||||
min_similarity: Optional[float] = None,
|
||||
@@ -1060,6 +1086,7 @@ class SQLiteSearchRepository(SearchRepositoryBase):
|
||||
note_types=note_types,
|
||||
after_date=after_date,
|
||||
search_item_types=search_item_types,
|
||||
categories=categories,
|
||||
metadata_filters=metadata_filters,
|
||||
retrieval_mode=retrieval_mode,
|
||||
min_similarity=min_similarity,
|
||||
@@ -1073,6 +1100,7 @@ class SQLiteSearchRepository(SearchRepositoryBase):
|
||||
note_types=note_types,
|
||||
after_date=after_date,
|
||||
search_item_types=search_item_types,
|
||||
categories=categories,
|
||||
metadata_filters=metadata_filters,
|
||||
)
|
||||
sql = f"SELECT COUNT(*) FROM {from_clause} WHERE {where_clause}"
|
||||
|
||||
@@ -42,6 +42,7 @@ class SearchQuery(BaseModel):
|
||||
Optionally filter results by:
|
||||
- note_types: Limit to specific note types (frontmatter "type")
|
||||
- entity_types: Limit to search item types (entity/observation/relation)
|
||||
- categories: Limit observation results to exact category matches (e.g. "requirement")
|
||||
- after_date: Only items after date
|
||||
- metadata_filters: Structured frontmatter filters (field -> value)
|
||||
- tags: Convenience frontmatter tag filter
|
||||
@@ -63,6 +64,7 @@ class SearchQuery(BaseModel):
|
||||
# Optional filters
|
||||
note_types: Optional[List[str]] = None # Filter by note type (frontmatter "type")
|
||||
entity_types: Optional[List[SearchItemType]] = None # Filter by entity type
|
||||
categories: Optional[List[str]] = None # Filter observations by exact category
|
||||
after_date: Optional[Union[datetime, str]] = None # Time-based filter
|
||||
metadata_filters: Optional[dict[str, Any]] = None # Structured frontmatter filters
|
||||
tags: Optional[List[str]] = None # Convenience tag filter
|
||||
@@ -85,6 +87,7 @@ class SearchQuery(BaseModel):
|
||||
status_is_empty = self.status is None or (isinstance(self.status, str) and not self.status)
|
||||
note_types_is_empty = not self.note_types
|
||||
entity_types_is_empty = not self.entity_types
|
||||
categories_is_empty = not self.categories
|
||||
return (
|
||||
self.permalink is None
|
||||
and self.permalink_match is None
|
||||
@@ -93,6 +96,7 @@ class SearchQuery(BaseModel):
|
||||
and self.after_date is None
|
||||
and note_types_is_empty
|
||||
and entity_types_is_empty
|
||||
and categories_is_empty
|
||||
and metadata_is_empty
|
||||
and tags_is_empty
|
||||
and status_is_empty
|
||||
|
||||
@@ -9,6 +9,7 @@ from basic_memory.schemas.v2.entity import (
|
||||
DeleteDirectoryRequestV2,
|
||||
ProjectResolveRequest,
|
||||
ProjectResolveResponse,
|
||||
SyncFileRequest,
|
||||
)
|
||||
from basic_memory.schemas.v2.graph import (
|
||||
GraphEdge,
|
||||
@@ -31,6 +32,7 @@ __all__ = [
|
||||
"DeleteDirectoryRequestV2",
|
||||
"ProjectResolveRequest",
|
||||
"ProjectResolveResponse",
|
||||
"SyncFileRequest",
|
||||
"GraphEdge",
|
||||
"GraphNode",
|
||||
"GraphResponse",
|
||||
|
||||
@@ -54,6 +54,21 @@ class EntityResolveResponse(BaseModel):
|
||||
)
|
||||
|
||||
|
||||
class SyncFileRequest(BaseModel):
|
||||
"""Request to index a single markdown file that exists on disk.
|
||||
|
||||
Used as a recovery path when an identifier fails resolution but maps to a
|
||||
file written directly to disk that the watcher has not indexed yet (#581).
|
||||
"""
|
||||
|
||||
file_path: str = Field(
|
||||
...,
|
||||
description="Markdown file path to index (relative to project root)",
|
||||
min_length=1,
|
||||
max_length=500,
|
||||
)
|
||||
|
||||
|
||||
class MoveEntityRequestV2(BaseModel):
|
||||
"""V2 request schema for moving an entity to a new file location.
|
||||
|
||||
|
||||
@@ -245,9 +245,12 @@ class ContextService:
|
||||
type="observation",
|
||||
id=obs.id,
|
||||
title=f"{obs.category}: {obs.content[:50]}...",
|
||||
permalink=generate_permalink(
|
||||
f"{primary_item.permalink}/observations/{obs.category}/{obs.content}"
|
||||
),
|
||||
# Observation.permalink is the single definition of the
|
||||
# synthetic permalink format (200-char truncation plus
|
||||
# content digest); rebuilding it inline diverged from the
|
||||
# search index for long observations (#929). The parent
|
||||
# entity is eager-loaded by ObservationRepository.
|
||||
permalink=obs.permalink,
|
||||
file_path=primary_item.file_path,
|
||||
content=obs.content,
|
||||
category=obs.category,
|
||||
@@ -333,9 +336,14 @@ class ContextService:
|
||||
relation_date_filter = ""
|
||||
timeframe_condition = ""
|
||||
|
||||
# Add project filtering for security - ensure all entities and relations belong to the same project
|
||||
project_filter = "AND e.project_id = :project_id"
|
||||
relation_project_filter = "AND e_from.project_id = :project_id"
|
||||
# Trigger: build_context starts from a project-scoped search result.
|
||||
# Why: the seed entity must belong to the requested project, but an
|
||||
# explicit relation edge may point at another project.
|
||||
# Outcome: traversal follows only project-owned edges from reached
|
||||
# entities, instead of forcing every reached entity into the seed project.
|
||||
seed_project_filter = "AND e.project_id = :project_id"
|
||||
connected_entity_project_filter = ""
|
||||
relation_project_filter = "AND e_from.project_id = r.project_id"
|
||||
|
||||
# Use a CTE that operates directly on entity and relation tables
|
||||
# This avoids the overhead of the search_index virtual table
|
||||
@@ -351,7 +359,8 @@ class ContextService:
|
||||
query = self._build_postgres_query(
|
||||
entity_id_values,
|
||||
date_filter,
|
||||
project_filter,
|
||||
seed_project_filter,
|
||||
connected_entity_project_filter,
|
||||
relation_date_filter,
|
||||
relation_project_filter,
|
||||
timeframe_condition,
|
||||
@@ -362,7 +371,8 @@ class ContextService:
|
||||
query = self._build_sqlite_query(
|
||||
entity_id_values,
|
||||
date_filter,
|
||||
project_filter,
|
||||
seed_project_filter,
|
||||
connected_entity_project_filter,
|
||||
relation_date_filter,
|
||||
relation_project_filter,
|
||||
timeframe_condition,
|
||||
@@ -397,7 +407,8 @@ class ContextService:
|
||||
self,
|
||||
entity_id_values: str,
|
||||
date_filter: str,
|
||||
project_filter: str,
|
||||
seed_project_filter: str,
|
||||
connected_entity_project_filter: str,
|
||||
relation_date_filter: str,
|
||||
relation_project_filter: str,
|
||||
timeframe_condition: str,
|
||||
@@ -421,11 +432,13 @@ class ContextService:
|
||||
0 as depth,
|
||||
e.id as root_id,
|
||||
e.created_at,
|
||||
e.created_at as relation_date
|
||||
e.created_at as relation_date,
|
||||
e.project_id as project_id,
|
||||
',' || e.id::text || ',' as entity_path
|
||||
FROM entity e
|
||||
WHERE e.id IN ({entity_id_values})
|
||||
{date_filter}
|
||||
{project_filter}
|
||||
{seed_project_filter}
|
||||
|
||||
UNION ALL
|
||||
|
||||
@@ -477,15 +490,25 @@ class ContextService:
|
||||
CASE
|
||||
WHEN step_type = 1 THEN e_from.created_at
|
||||
ELSE eg.relation_date
|
||||
END as relation_date
|
||||
END as relation_date,
|
||||
CASE
|
||||
WHEN step_type = 1 THEN eg.project_id
|
||||
ELSE e.project_id
|
||||
END as project_id,
|
||||
CASE
|
||||
WHEN step_type = 1 THEN eg.entity_path
|
||||
ELSE eg.entity_path || e.id::text || ','
|
||||
END as entity_path
|
||||
FROM entity_graph eg
|
||||
CROSS JOIN LATERAL (VALUES (1), (2)) AS steps(step_type)
|
||||
JOIN relation r ON (
|
||||
eg.type = 'entity' AND
|
||||
(r.from_id = eg.id OR r.to_id = eg.id)
|
||||
(r.from_id = eg.id OR r.to_id = eg.id) AND
|
||||
r.project_id = eg.project_id
|
||||
)
|
||||
JOIN entity e_from ON (
|
||||
r.from_id = e_from.id
|
||||
{relation_date_filter}
|
||||
{relation_project_filter}
|
||||
)
|
||||
LEFT JOIN entity e ON (
|
||||
@@ -495,10 +518,17 @@ class ContextService:
|
||||
ELSE r.from_id
|
||||
END
|
||||
{date_filter}
|
||||
{project_filter}
|
||||
{connected_entity_project_filter}
|
||||
)
|
||||
WHERE eg.depth < :max_depth
|
||||
AND (step_type = 1 OR (step_type = 2 AND e.id IS NOT NULL AND e.id != eg.id))
|
||||
AND (
|
||||
step_type = 1 OR (
|
||||
step_type = 2
|
||||
AND e.id IS NOT NULL
|
||||
AND e.id != eg.id
|
||||
AND position(',' || e.id::text || ',' in eg.entity_path) = 0
|
||||
)
|
||||
)
|
||||
{timeframe_condition}
|
||||
)
|
||||
-- Materialize and filter
|
||||
@@ -529,7 +559,8 @@ class ContextService:
|
||||
self,
|
||||
entity_id_values: str,
|
||||
date_filter: str,
|
||||
project_filter: str,
|
||||
seed_project_filter: str,
|
||||
connected_entity_project_filter: str,
|
||||
relation_date_filter: str,
|
||||
relation_project_filter: str,
|
||||
timeframe_condition: str,
|
||||
@@ -555,11 +586,13 @@ class ContextService:
|
||||
e.id as root_id,
|
||||
e.created_at,
|
||||
e.created_at as relation_date,
|
||||
0 as is_incoming
|
||||
0 as is_incoming,
|
||||
e.project_id as project_id,
|
||||
',' || e.id || ',' as entity_path
|
||||
FROM entity e
|
||||
WHERE e.id IN ({entity_id_values})
|
||||
{date_filter}
|
||||
{project_filter}
|
||||
{seed_project_filter}
|
||||
|
||||
UNION ALL
|
||||
|
||||
@@ -580,11 +613,14 @@ class ContextService:
|
||||
eg.root_id,
|
||||
e_from.created_at,
|
||||
e_from.created_at as relation_date,
|
||||
CASE WHEN r.from_id = eg.id THEN 0 ELSE 1 END as is_incoming
|
||||
CASE WHEN r.from_id = eg.id THEN 0 ELSE 1 END as is_incoming,
|
||||
eg.project_id as project_id,
|
||||
eg.entity_path as entity_path
|
||||
FROM entity_graph eg
|
||||
JOIN relation r ON (
|
||||
eg.type = 'entity' AND
|
||||
(r.from_id = eg.id OR r.to_id = eg.id)
|
||||
(r.from_id = eg.id OR r.to_id = eg.id) AND
|
||||
r.project_id = eg.project_id
|
||||
)
|
||||
JOIN entity e_from ON (
|
||||
r.from_id = e_from.id
|
||||
@@ -615,7 +651,9 @@ class ContextService:
|
||||
eg.root_id,
|
||||
e.created_at,
|
||||
eg.relation_date,
|
||||
eg.is_incoming
|
||||
eg.is_incoming,
|
||||
e.project_id as project_id,
|
||||
eg.entity_path || e.id || ',' as entity_path
|
||||
FROM entity_graph eg
|
||||
JOIN entity e ON (
|
||||
eg.type = 'relation' AND
|
||||
@@ -624,9 +662,10 @@ class ContextService:
|
||||
ELSE eg.from_id
|
||||
END
|
||||
{date_filter}
|
||||
{project_filter}
|
||||
{connected_entity_project_filter}
|
||||
)
|
||||
WHERE eg.depth < :max_depth
|
||||
AND instr(eg.entity_path, ',' || e.id || ',') = 0
|
||||
{timeframe_condition}
|
||||
)
|
||||
SELECT DISTINCT
|
||||
|
||||
@@ -1047,10 +1047,27 @@ class ProjectService:
|
||||
f"WHERE c.project_id = :project_id {chunk_entity_exists}"
|
||||
)
|
||||
|
||||
embeddings_result = await self.repository.execute_query(
|
||||
embeddings_sql, {"project_id": project_id}
|
||||
)
|
||||
total_embeddings = embeddings_result.scalar() or 0
|
||||
# The embeddings/orphan JOINs read search_vector_embeddings, a vec0
|
||||
# virtual table. On SQLite that table is only visible on a connection
|
||||
# that loaded sqlite-vec, so route these through scalar_vec_query which
|
||||
# loads the extension first. Postgres has no per-connection extension
|
||||
# and uses the bare pooled session.
|
||||
async def _vec_scalar(vec_sql) -> int:
|
||||
if is_postgres:
|
||||
result = await self.repository.execute_query(
|
||||
vec_sql, {"project_id": project_id}
|
||||
)
|
||||
return result.scalar() or 0
|
||||
count = await self.repository.scalar_vec_query(vec_sql, {"project_id": project_id})
|
||||
# Trigger: sqlite-vec genuinely can't load on this Python build.
|
||||
# Why: without the extension the vec0 JOIN can't run at all.
|
||||
# Outcome: raise the canonical error so the except block emits the
|
||||
# true "sqlite-vec unavailable" fallback instead of reporting 0.
|
||||
if count is None:
|
||||
raise SAOperationalError(str(vec_sql), {}, Exception("no such module: vec0"))
|
||||
return count
|
||||
|
||||
total_embeddings = await _vec_scalar(embeddings_sql)
|
||||
|
||||
# Orphaned chunks (chunks without embeddings — indicates interrupted indexing)
|
||||
if is_postgres:
|
||||
@@ -1065,11 +1082,7 @@ class ProjectService:
|
||||
"LEFT JOIN search_vector_embeddings e ON e.rowid = c.id "
|
||||
f"WHERE c.project_id = :project_id AND e.rowid IS NULL {chunk_entity_exists}"
|
||||
)
|
||||
|
||||
orphan_result = await self.repository.execute_query(
|
||||
orphan_sql, {"project_id": project_id}
|
||||
)
|
||||
orphaned_chunks = orphan_result.scalar() or 0
|
||||
orphaned_chunks = await _vec_scalar(orphan_sql)
|
||||
except SAOperationalError as exc:
|
||||
# Trigger: sqlite_master can list vec0 virtual tables even when sqlite-vec
|
||||
# is not loaded in the current Python runtime.
|
||||
|
||||
@@ -75,6 +75,7 @@ class _PreparedSearchQuery:
|
||||
title: str | None
|
||||
note_types: list[str] | None
|
||||
search_item_types: list[SearchItemType] | None
|
||||
categories: list[str] | None
|
||||
after_date: datetime | None
|
||||
metadata_filters: dict[str, Any] | None
|
||||
retrieval_mode: SearchRetrievalMode
|
||||
@@ -194,6 +195,7 @@ class SearchService:
|
||||
title=query.title,
|
||||
note_types=query.note_types,
|
||||
search_item_types=query.entity_types,
|
||||
categories=query.categories,
|
||||
after_date=after_date,
|
||||
metadata_filters=metadata_filters,
|
||||
retrieval_mode=query.retrieval_mode or SearchRetrievalMode.FTS,
|
||||
@@ -207,6 +209,7 @@ class SearchService:
|
||||
or prepared.title
|
||||
or prepared.note_types
|
||||
or prepared.search_item_types
|
||||
or prepared.categories
|
||||
or prepared.after_date
|
||||
or prepared.metadata_filters
|
||||
)
|
||||
@@ -221,6 +224,7 @@ class SearchService:
|
||||
prepared.metadata_filters
|
||||
or prepared.note_types
|
||||
or prepared.search_item_types
|
||||
or prepared.categories
|
||||
or prepared.after_date
|
||||
)
|
||||
|
||||
@@ -239,6 +243,7 @@ class SearchService:
|
||||
title=prepared.title,
|
||||
note_types=prepared.note_types,
|
||||
search_item_types=prepared.search_item_types,
|
||||
categories=prepared.categories,
|
||||
after_date=prepared.after_date,
|
||||
metadata_filters=prepared.metadata_filters,
|
||||
retrieval_mode=prepared.retrieval_mode,
|
||||
@@ -260,6 +265,7 @@ class SearchService:
|
||||
title=prepared.title,
|
||||
note_types=prepared.note_types,
|
||||
search_item_types=prepared.search_item_types,
|
||||
categories=prepared.categories,
|
||||
after_date=prepared.after_date,
|
||||
metadata_filters=prepared.metadata_filters,
|
||||
retrieval_mode=prepared.retrieval_mode,
|
||||
|
||||
@@ -1447,7 +1447,16 @@ class SyncService:
|
||||
f"to_name={relation.to_name}"
|
||||
)
|
||||
|
||||
resolved_entity = await self.entity_service.link_resolver.resolve_link(relation.to_name)
|
||||
# Use strict=True: deferred resolution should only fill in to_id when an
|
||||
# exact permalink/title/file_path match exists. The fuzzy fallback (search-based
|
||||
# token match) would silently resolve ambiguous links like
|
||||
# `[[overview (state-management/session-execution)]]` to whichever entity shares
|
||||
# the most tokens, polluting the graph with confidently-wrong edges that no
|
||||
# audit catches. Leaving such relations unresolved keeps to_id=NULL so they
|
||||
# surface as forward references and can be fixed by the producer.
|
||||
resolved_entity = await self.entity_service.link_resolver.resolve_link(
|
||||
relation.to_name, strict=True
|
||||
)
|
||||
|
||||
# ignore reference to self
|
||||
if resolved_entity and resolved_entity.id != relation.from_id:
|
||||
@@ -1735,7 +1744,9 @@ async def get_sync_service(project: Project) -> SyncService: # pragma: no cover
|
||||
entity_repository = EntityRepository(session_maker, project_id=project.id)
|
||||
observation_repository = ObservationRepository(session_maker, project_id=project.id)
|
||||
relation_repository = RelationRepository(session_maker, project_id=project.id)
|
||||
search_repository = create_search_repository(session_maker, project_id=project.id)
|
||||
search_repository = create_search_repository(
|
||||
session_maker, project_id=project.id, app_config=app_config
|
||||
)
|
||||
project_repository = ProjectRepository(session_maker)
|
||||
|
||||
# Initialize services
|
||||
|
||||
@@ -526,8 +526,20 @@ def parse_tags(tags: Union[List[str], str, None]) -> List[str]:
|
||||
|
||||
# Process list of tags
|
||||
if isinstance(tags, list):
|
||||
# First strip whitespace, then strip leading '#' characters to prevent accumulation
|
||||
return [tag.strip().lstrip("#") for tag in tags if tag and tag.strip()]
|
||||
# Trigger: a list element may itself be a comma-separated string (e.g. typer collects
|
||||
# `--tags "a,b"` into the one-element list `["a,b"]`).
|
||||
# Why: keep the CLI list path and the MCP bare-string path on a single source of truth so
|
||||
# `--tags "a,b"`, `--tags a --tags b`, and `tags="a,b"` all converge to the same tags.
|
||||
# Outcome: flatten by splitting each element on commas before stripping '#' / whitespace.
|
||||
# Skip None entries (e.g. a YAML `tags: [alpha, null]`) so they are not revived as
|
||||
# the literal tag "None" by str(raw); the old list branch ignored such falsy entries.
|
||||
return [
|
||||
tag.strip().lstrip("#")
|
||||
for raw in tags
|
||||
if raw is not None
|
||||
for tag in str(raw).split(",")
|
||||
if tag and tag.strip()
|
||||
]
|
||||
|
||||
# Process string input
|
||||
if isinstance(tags, str):
|
||||
@@ -556,6 +568,38 @@ def parse_tags(tags: Union[List[str], str, None]) -> List[str]:
|
||||
return []
|
||||
|
||||
|
||||
def strict_search_tags(v: Any) -> Any:
|
||||
"""Strictly coerce tag input at the search_notes tool boundary.
|
||||
|
||||
parse_tags stringifies anything (42 -> ["42"], {"a": 1} -> junk tags), which would
|
||||
turn caller type mistakes into silent no-result searches. At the tool boundary only
|
||||
str, all-string lists, and None are valid tag inputs; everything else — including
|
||||
lists with non-string elements like [42] — passes through unchanged so Pydantic
|
||||
rejects it with a clear validation error.
|
||||
|
||||
JSON array strings (the MCP clients-serialize-arrays-as-strings path) get the same
|
||||
all-string check: '[42]' or '["ok", 42]' would otherwise be stringified by
|
||||
parse_tags' recursive JSON handling before Pydantic ever sees the bad elements.
|
||||
"""
|
||||
if isinstance(v, list) and not all(isinstance(item, str) for item in v):
|
||||
return v
|
||||
# Trigger: a str that looks like a JSON array, mirroring parse_tags' detection.
|
||||
# Why: parse_tags recursively parses JSON arrays, stringifying non-string elements
|
||||
# ('[42]' -> ["42"]) and hiding the type error from Pydantic.
|
||||
# Outcome: malformed arrays pass through unchanged so Pydantic rejects them; valid
|
||||
# all-string arrays and plain comma strings still delegate to parse_tags.
|
||||
if isinstance(v, str) and v.strip().startswith("[") and v.strip().endswith("]"):
|
||||
try:
|
||||
parsed = json.loads(v)
|
||||
except json.JSONDecodeError:
|
||||
parsed = None
|
||||
if isinstance(parsed, list) and not all(isinstance(item, str) for item in parsed):
|
||||
return v
|
||||
if v is None or isinstance(v, (str, list)):
|
||||
return parse_tags(v)
|
||||
return v
|
||||
|
||||
|
||||
def coerce_list(v: Any) -> Any:
|
||||
"""Coerce string input to list for MCP clients that serialize lists as strings."""
|
||||
if v is None:
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
"""Bug hunt regression test (#6): `bm tool read-note` exit code on a
|
||||
path-traversal SECURITY_VALIDATION_ERROR.
|
||||
|
||||
The MCP read_note tool detects path-traversal identifiers and returns
|
||||
{"error": "SECURITY_VALIDATION_ERROR", ...}. Every other wrapped tool command
|
||||
exits non-zero on an error payload; read-note used to print the payload and
|
||||
exit 0. These integration tests assert read-note now matches its siblings.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from basic_memory.cli.main import app as cli_app
|
||||
from basic_memory.mcp.tools import read_note as mcp_read_note
|
||||
|
||||
runner = CliRunner()
|
||||
|
||||
TRAVERSAL_IDENTIFIER = "../../../../etc/passwd"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_read_note_security_error_mcp_emits_error_field(
|
||||
app, app_config, test_project, config_manager
|
||||
):
|
||||
"""MCP read_note JSON flags the path traversal with a SECURITY_VALIDATION_ERROR."""
|
||||
result = await mcp_read_note(
|
||||
identifier=TRAVERSAL_IDENTIFIER,
|
||||
project=test_project.name,
|
||||
output_format="json",
|
||||
)
|
||||
assert isinstance(result, dict)
|
||||
assert result.get("error") == "SECURITY_VALIDATION_ERROR"
|
||||
|
||||
|
||||
def test_read_note_security_error_cli_exit_code_matches_other_tools(
|
||||
app, app_config, test_project, config_manager
|
||||
):
|
||||
"""CLI read-note must not exit 0 when the MCP payload carries an error."""
|
||||
result = runner.invoke(
|
||||
cli_app,
|
||||
["tool", "read-note", TRAVERSAL_IDENTIFIER, "--project", test_project.name],
|
||||
)
|
||||
|
||||
combined = result.stdout
|
||||
assert "SECURITY_VALIDATION_ERROR" in combined, combined
|
||||
|
||||
assert result.exit_code != 0, (
|
||||
f"read-note exited {result.exit_code} on a SECURITY_VALIDATION_ERROR; "
|
||||
"other tool commands exit non-zero on error payloads"
|
||||
)
|
||||
@@ -0,0 +1,76 @@
|
||||
"""Bug hunt regression test (#3): `bm tool recent-activity` page_size default.
|
||||
|
||||
The MCP recent_activity tool defaults page_size=10; the CLI wrapper used to
|
||||
default to 50. Because page_size becomes the SQL LIMIT for the query, identical
|
||||
default invocations returned a different number of rows from CLI vs MCP. This
|
||||
integration test proves the CLI default now matches the MCP default of 10.
|
||||
"""
|
||||
|
||||
import json
|
||||
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from basic_memory.cli.main import app as cli_app
|
||||
|
||||
runner = CliRunner()
|
||||
|
||||
MCP_DEFAULT_PAGE_SIZE = 10
|
||||
|
||||
|
||||
def _write_note(title: str, folder: str, content: str) -> None:
|
||||
result = runner.invoke(
|
||||
cli_app,
|
||||
[
|
||||
"tool",
|
||||
"write-note",
|
||||
"--title",
|
||||
title,
|
||||
"--folder",
|
||||
folder,
|
||||
"--content",
|
||||
content,
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
|
||||
|
||||
def test_recent_activity_default_page_size_matches_mcp(
|
||||
app, app_config, test_project, config_manager, monkeypatch
|
||||
):
|
||||
"""CLI recent-activity default page_size must match the MCP tool default (10)."""
|
||||
monkeypatch.setenv("BASIC_MEMORY_MCP_PROJECT", test_project.name)
|
||||
|
||||
for i in range(15):
|
||||
_write_note(
|
||||
f"Parity Note {i:02d}",
|
||||
"parity-recent",
|
||||
f"# Parity Note {i:02d}\n\nUnique body token PARITY{i:02d}.",
|
||||
)
|
||||
|
||||
mcp_default_result = runner.invoke(
|
||||
cli_app,
|
||||
[
|
||||
"tool",
|
||||
"recent-activity",
|
||||
"--project",
|
||||
test_project.name,
|
||||
"--page-size",
|
||||
str(MCP_DEFAULT_PAGE_SIZE),
|
||||
],
|
||||
)
|
||||
assert mcp_default_result.exit_code == 0, mcp_default_result.output
|
||||
mcp_default_rows = json.loads(mcp_default_result.stdout)
|
||||
assert len(mcp_default_rows) == MCP_DEFAULT_PAGE_SIZE
|
||||
|
||||
cli_default_result = runner.invoke(
|
||||
cli_app,
|
||||
["tool", "recent-activity", "--project", test_project.name],
|
||||
)
|
||||
assert cli_default_result.exit_code == 0, cli_default_result.output
|
||||
cli_default_rows = json.loads(cli_default_result.stdout)
|
||||
|
||||
assert len(cli_default_rows) == MCP_DEFAULT_PAGE_SIZE, (
|
||||
f"CLI recent-activity default returned {len(cli_default_rows)} rows but "
|
||||
f"the MCP tool default (page_size={MCP_DEFAULT_PAGE_SIZE}) returns "
|
||||
f"{len(mcp_default_rows)}; the CLI and MCP default page_size must match."
|
||||
)
|
||||
@@ -0,0 +1,73 @@
|
||||
"""Bug hunt regression test (#4): `bm tool search-notes` --category filter.
|
||||
|
||||
The MCP search_notes tool exposes a `categories` parameter for exact-match
|
||||
observation-category filtering. The CLI wrapper had no equivalent flag. This
|
||||
integration test asserts the CLI now exposes `--category` and that it filters
|
||||
observation results to the requested category exactly.
|
||||
"""
|
||||
|
||||
import json
|
||||
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from basic_memory.cli.main import app as cli_app
|
||||
|
||||
runner = CliRunner()
|
||||
|
||||
|
||||
def _write_note(title: str, folder: str, content: str) -> dict:
|
||||
result = runner.invoke(
|
||||
cli_app,
|
||||
[
|
||||
"tool",
|
||||
"write-note",
|
||||
"--title",
|
||||
title,
|
||||
"--folder",
|
||||
folder,
|
||||
"--content",
|
||||
content,
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
return json.loads(result.stdout)
|
||||
|
||||
|
||||
def test_search_notes_exposes_category_filter(app, app_config, test_project, config_manager):
|
||||
"""CLI search-notes should expose --category like the MCP `categories` param."""
|
||||
_write_note(
|
||||
"Category Filter Note",
|
||||
"parity-category",
|
||||
"# Category Filter Note\n\n"
|
||||
"## Observations\n"
|
||||
"- [requirement] system must authenticate users CATTOKEN\n"
|
||||
"- [decision] use OAuth for auth CATTOKEN\n",
|
||||
)
|
||||
|
||||
result = runner.invoke(
|
||||
cli_app,
|
||||
[
|
||||
"tool",
|
||||
"search-notes",
|
||||
"CATTOKEN",
|
||||
"--project",
|
||||
test_project.name,
|
||||
"--entity-type",
|
||||
"observation",
|
||||
"--category",
|
||||
"requirement",
|
||||
],
|
||||
)
|
||||
|
||||
assert result.exit_code == 0, (
|
||||
"`--category` filter is not supported by the CLI search-notes command "
|
||||
"even though the MCP search_notes tool documents a `categories` param. "
|
||||
f"exit_code={result.exit_code} output={result.output}"
|
||||
)
|
||||
|
||||
payload = json.loads(result.stdout)
|
||||
categories = {r.get("category") for r in payload.get("results", []) if r.get("category")}
|
||||
assert categories == {"requirement"}, (
|
||||
"--category requirement should return only requirement observations, "
|
||||
f"got categories={categories}"
|
||||
)
|
||||
@@ -0,0 +1,193 @@
|
||||
"""Bug hunt regression tests: `bm tool write-note` CLI/MCP parity.
|
||||
|
||||
Covers three confirmed bugs found by the integration-test bug hunt:
|
||||
|
||||
- #1 / #5: write-note exits 0 on a conflict/error JSON result (silent failure,
|
||||
inconsistent with delete-note/edit-note/search-notes which exit non-zero).
|
||||
- #2: write-note had no `--overwrite` flag even though the MCP write_note tool
|
||||
supports overwrite=True to replace an existing note.
|
||||
|
||||
These are integration tests: real CliRunner -> CLI command -> MCP tool ->
|
||||
in-process ASGI API -> real SQLite/Postgres DB and filesystem. No mocks.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from basic_memory.cli.main import app as cli_app
|
||||
from basic_memory.mcp.tools import write_note as mcp_write_note
|
||||
|
||||
runner = CliRunner()
|
||||
|
||||
|
||||
# --- #1: write-note exits non-zero on a conflict/error result ---
|
||||
|
||||
|
||||
def _write_conflict(content_token: str):
|
||||
return runner.invoke(
|
||||
cli_app,
|
||||
[
|
||||
"tool",
|
||||
"write-note",
|
||||
"--title",
|
||||
"Conflict Exit Note",
|
||||
"--folder",
|
||||
"parity-conflict",
|
||||
"--content",
|
||||
f"# Conflict Exit Note\n\n{content_token}",
|
||||
"--project",
|
||||
"test-project",
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
def test_write_note_nonzero_exit_on_conflict_error(app, app_config, test_project, config_manager):
|
||||
"""write-note should exit non-zero when the MCP result carries an error."""
|
||||
first = _write_conflict("FIRST")
|
||||
assert first.exit_code == 0, first.output
|
||||
|
||||
second = _write_conflict("SECOND")
|
||||
payload = json.loads(second.stdout)
|
||||
|
||||
# Confirm the MCP layer reported a conflict error in the JSON.
|
||||
assert payload.get("error") == "NOTE_ALREADY_EXISTS", payload
|
||||
assert payload.get("action") == "conflict", payload
|
||||
|
||||
# Parity with delete-note / edit-note / search-notes: an error result
|
||||
# must drive a non-zero exit code so scripts can detect failure.
|
||||
assert second.exit_code != 0, (
|
||||
"write-note returned an error JSON payload "
|
||||
f"({payload.get('error')}) but exited 0. Sibling tool commands "
|
||||
"(delete-note, edit-note, search-notes) exit non-zero on error."
|
||||
)
|
||||
|
||||
|
||||
# --- #5: blocked NOTE_ALREADY_EXISTS write must not report success ---
|
||||
|
||||
|
||||
def _cli_write(project_name: str):
|
||||
return runner.invoke(
|
||||
cli_app,
|
||||
[
|
||||
"tool",
|
||||
"write-note",
|
||||
"--title",
|
||||
"Conflict Note",
|
||||
"--folder",
|
||||
"conflict",
|
||||
"--content",
|
||||
"# Conflict Note\n\nFirst body.\n",
|
||||
"--project",
|
||||
project_name,
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
def test_mcp_write_note_conflict_emits_error(app, app_config, test_project, config_manager):
|
||||
"""Baseline: the MCP tool reports NOTE_ALREADY_EXISTS on a blocked re-write."""
|
||||
|
||||
async def _go():
|
||||
first = await mcp_write_note(
|
||||
title="Conflict Note",
|
||||
content="# Conflict Note\n\nFirst body.\n",
|
||||
directory="conflict",
|
||||
project=test_project.name,
|
||||
output_format="json",
|
||||
)
|
||||
# output_format="json" returns a dict; narrow for the type checker.
|
||||
assert isinstance(first, dict)
|
||||
assert first.get("action") == "created"
|
||||
assert "error" not in first
|
||||
|
||||
second = await mcp_write_note(
|
||||
title="Conflict Note",
|
||||
content="# Conflict Note\n\nSecond body (should be blocked).\n",
|
||||
directory="conflict",
|
||||
project=test_project.name,
|
||||
output_format="json",
|
||||
)
|
||||
return second
|
||||
|
||||
second = asyncio.run(_go())
|
||||
assert isinstance(second, dict)
|
||||
assert second.get("error") == "NOTE_ALREADY_EXISTS"
|
||||
assert second.get("action") == "conflict"
|
||||
assert second.get("file_path") is None
|
||||
|
||||
|
||||
def test_cli_write_note_conflict_should_exit_nonzero(app, app_config, test_project, config_manager):
|
||||
"""CLI write-note must NOT exit 0 when the write was blocked by a conflict."""
|
||||
first = _cli_write(test_project.name)
|
||||
assert first.exit_code == 0, first.output
|
||||
first_payload = json.loads(first.stdout)
|
||||
assert first_payload["action"] == "created"
|
||||
|
||||
second = _cli_write(test_project.name)
|
||||
|
||||
assert "NOTE_ALREADY_EXISTS" in second.stdout, second.output
|
||||
|
||||
assert second.exit_code != 0, (
|
||||
f"write-note exited {second.exit_code} after a blocked NOTE_ALREADY_EXISTS "
|
||||
"write; the note was NOT written but the CLI reported success"
|
||||
)
|
||||
|
||||
|
||||
# --- #2: write-note --overwrite flag (MCP overwrite=True parity) ---
|
||||
|
||||
|
||||
def _write_overwrite(args_extra: list[str]):
|
||||
return runner.invoke(
|
||||
cli_app,
|
||||
[
|
||||
"tool",
|
||||
"write-note",
|
||||
"--title",
|
||||
"Overwrite Parity Note",
|
||||
"--folder",
|
||||
"parity-overwrite",
|
||||
"--content",
|
||||
"# Overwrite Parity Note\n\nVERSION_BODY",
|
||||
*args_extra,
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
def test_write_note_cli_can_overwrite_like_mcp(app, app_config, test_project, config_manager):
|
||||
"""CLI write-note must be able to overwrite an existing note (MCP overwrite=True)."""
|
||||
first = _write_overwrite(["--project", test_project.name])
|
||||
assert first.exit_code == 0, first.output
|
||||
first_data = json.loads(first.stdout)
|
||||
permalink = first_data["permalink"]
|
||||
|
||||
second = runner.invoke(
|
||||
cli_app,
|
||||
[
|
||||
"tool",
|
||||
"write-note",
|
||||
"--title",
|
||||
"Overwrite Parity Note",
|
||||
"--folder",
|
||||
"parity-overwrite",
|
||||
"--content",
|
||||
"# Overwrite Parity Note\n\nNEW_VERSION_BODY",
|
||||
"--project",
|
||||
test_project.name,
|
||||
"--overwrite",
|
||||
],
|
||||
)
|
||||
|
||||
assert second.exit_code == 0, (
|
||||
"CLI write-note has no way to overwrite an existing note even though "
|
||||
"the MCP write_note tool supports overwrite=True. "
|
||||
f"exit_code={second.exit_code} output={second.output}"
|
||||
)
|
||||
|
||||
read = runner.invoke(
|
||||
cli_app,
|
||||
["tool", "read-note", permalink, "--project", test_project.name],
|
||||
)
|
||||
assert read.exit_code == 0, read.output
|
||||
read_data = json.loads(read.stdout)
|
||||
assert "NEW_VERSION_BODY" in (read_data.get("content") or "")
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user