mirror of
https://github.com/basicmachines-co/basic-memory
synced 2026-06-21 13:47:35 +00:00
Compare commits
52 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 513fef79c5 | |||
| afa694ba8d | |||
| 8668118f78 | |||
| e81cdc2980 | |||
| d8c9156ffd | |||
| 291a8085ef | |||
| 49041a5168 | |||
| 8cbe1634b6 | |||
| c4b651f5b0 | |||
| 33fdd3d44e | |||
| ef6c47e674 | |||
| f9c3baf5b9 | |||
| 24abf11dab | |||
| ca9a4d9c12 | |||
| aa9594d82a | |||
| 93494b8c13 | |||
| 93ed34001b | |||
| ec94feb6a4 | |||
| 831b9141a5 | |||
| 2de19713f6 | |||
| 79fcfbce6a | |||
| 3d22ba3004 | |||
| 62229d9d0a | |||
| 03ba268cb1 | |||
| de53e0ecc5 | |||
| 4128cac9ab | |||
| 9b53d7863f | |||
| a8d034b940 | |||
| 7c0937f658 | |||
| 8570d96bad | |||
| 480a2d9468 | |||
| df7452e3ba | |||
| 85a8e59d0f | |||
| 07cb7a606b | |||
| 0ef03edde6 | |||
| efe43a10ea | |||
| 4fe6fe09c8 | |||
| 8acdb49a41 | |||
| f7304bf553 | |||
| 5b034f081d | |||
| 271c883ea8 | |||
| 816ee85fb9 | |||
| 3ba3a9504d | |||
| 1667cdc000 | |||
| f916662ff3 | |||
| 0c9800cd3b | |||
| 476239d878 | |||
| 20bb19f4cd | |||
| f6565b9d23 | |||
| b6e8c636ce | |||
| 442fd1523c | |||
| ce4b5d47a0 |
@@ -0,0 +1,68 @@
|
||||
---
|
||||
name: code-review
|
||||
description: Use when reviewing Basic Machines code for house style, architecture risk, pre-merge hardening, or whether a change fits basic-memory/basic-memory-cloud conventions.
|
||||
license: MIT
|
||||
---
|
||||
|
||||
# Basic Machines Review
|
||||
|
||||
Use this skill for repo-local review passes where ordinary code review needs Basic Machines
|
||||
house style and architecture judgment. Report findings only; do not edit code unless the user
|
||||
asks you to fix specific findings.
|
||||
|
||||
## Scope
|
||||
|
||||
Review the current diff or named files against:
|
||||
|
||||
- The repo's `AGENTS.md` / `CLAUDE.md`
|
||||
- `docs/ENGINEERING_STYLE.md`
|
||||
- The touched code paths and tests
|
||||
|
||||
Apply only the guidance for the active repo. In `basic-memory`, prioritize local-first
|
||||
file/database/MCP boundaries. In `basic-memory-cloud`, prioritize tenant/workspace isolation,
|
||||
cloud worker behavior, and web-v2 state/runtime boundaries.
|
||||
|
||||
## Review Rubric
|
||||
|
||||
Report only concrete, falsifiable risks:
|
||||
|
||||
- **Cognitive load:** Is the change harder to understand than the problem requires?
|
||||
- **Change propagation:** Will one product change force edits across unrelated layers?
|
||||
- **Knowledge duplication:** Is the same rule encoded in multiple places that can drift?
|
||||
- **Accidental complexity:** Did the change add abstractions, fallbacks, or state without need?
|
||||
- **Dependency direction:** Are API/MCP/CLI, services, repositories, and UI stores respecting
|
||||
their intended boundaries?
|
||||
- **Domain model distortion:** Do names and types still match the product concept, or did a
|
||||
transport/storage detail leak into the domain?
|
||||
- **Test oracle quality:** Would the tests fail for the bug or regression the change claims to
|
||||
protect against?
|
||||
|
||||
## House Rules To Check Explicitly
|
||||
|
||||
- No speculative `getattr(obj, "attr", default)` for unknown model shapes.
|
||||
- No broad exception swallowing, warning-only failure paths, or hidden fallback behavior.
|
||||
- No casts or `Any` that hide an unclear type relationship.
|
||||
- Dataclasses for internal value/result objects; Pydantic at validation/serialization
|
||||
boundaries.
|
||||
- Narrow `Protocol`s when only a capability is needed.
|
||||
- Explicit async/resource ownership, cancellation, and cleanup.
|
||||
- Meaningful regression tests or verification for risky changes.
|
||||
- Comments explain why, not what.
|
||||
|
||||
## Reporting Format
|
||||
|
||||
Lead with findings ordered by severity. Each finding should include:
|
||||
|
||||
| Severity | Use for |
|
||||
| -------- | ------- |
|
||||
| `high` | A likely correctness, security, data-loss, or tenant/workspace isolation failure |
|
||||
| `medium` | A concrete maintainability or boundary risk that can cause future defects |
|
||||
| `low` | A minor consistency issue, ambiguous guidance, or review-only cleanup |
|
||||
|
||||
```text
|
||||
severity | file:line | risk category | claim
|
||||
Why: concrete behavior or code path that proves the risk.
|
||||
Fix: smallest practical change, or "none obvious" if the risk needs product input.
|
||||
```
|
||||
|
||||
If there are no findings, say so and note any verification gaps that remain.
|
||||
@@ -0,0 +1,48 @@
|
||||
---
|
||||
name: fix-pr-issues
|
||||
description: Use when addressing Basic Memory pull request feedback, failed checks, or BM Bossbot blockers from Codex.
|
||||
---
|
||||
|
||||
# Fix Basic Memory PR Issues
|
||||
|
||||
Resolve PR feedback and failed checks, then wait for BM Bossbot to approve the
|
||||
new head SHA. This skill never merges a PR.
|
||||
|
||||
## Gather
|
||||
|
||||
1. Identify the PR:
|
||||
- `gh pr view --json number,url,headRefOid,mergeStateStatus,statusCheckRollup`
|
||||
|
||||
2. Collect feedback:
|
||||
- PR comments and review summaries
|
||||
- inline review comments and unresolved review threads
|
||||
- failed GitHub Actions jobs and relevant logs
|
||||
- the managed `BM_BOSSBOT_SUMMARY` block in the PR body
|
||||
|
||||
3. Build a short issue ledger:
|
||||
- source
|
||||
- concrete problem
|
||||
- expected fix
|
||||
- verification needed
|
||||
|
||||
## Fix
|
||||
|
||||
1. Address one ledger item at a time.
|
||||
2. Read each file in full before editing it.
|
||||
3. Keep diffs narrow and preserve unrelated user changes.
|
||||
4. Run the smallest meaningful verification first, then widen as needed.
|
||||
5. Commit with `git commit -s` when code or docs changed.
|
||||
|
||||
## Push And Recheck
|
||||
|
||||
1. Push the branch.
|
||||
2. Watch checks for the new `headRefOid`.
|
||||
3. Wait for the required `BM Bossbot Approval` status to pass on that exact SHA.
|
||||
4. If BM Bossbot reviews an older SHA, treat the approval as stale and keep
|
||||
waiting for the current one.
|
||||
|
||||
## Reply
|
||||
|
||||
For each addressed comment or blocker, reply with the fix commit, verification
|
||||
run, and current BM Bossbot status. Do not resolve or dismiss substantive
|
||||
feedback without evidence.
|
||||
@@ -0,0 +1,7 @@
|
||||
interface:
|
||||
display_name: "Fix PR Issues"
|
||||
short_description: "Address PR feedback and BM Bossbot blockers"
|
||||
icon_small: "./assets/icon.svg"
|
||||
icon_large: "./assets/icon.svg"
|
||||
brand_color: "#2563EB"
|
||||
default_prompt: "Use $fix-pr-issues to address PR feedback and wait for BM Bossbot Approval on the latest head SHA."
|
||||
@@ -0,0 +1,5 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="128" height="128" viewBox="0 0 24 24" fill="none" stroke="#111827" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round">
|
||||
<path d="M3 12h4l2-6 4 12 2-6h6"/>
|
||||
<path d="M4 20h16"/>
|
||||
</svg>
|
||||
|
||||
|
After Width: | Height: | Size: 249 B |
@@ -0,0 +1,247 @@
|
||||
---
|
||||
name: infographics
|
||||
description: Use when generating Basic Memory PR, changelog, release, or weekly images from Codex.
|
||||
---
|
||||
|
||||
# Basic Memory Images
|
||||
|
||||
Generate repository visuals with evidence-grounded content and canonical output
|
||||
paths. The file and marker names still say "infographic" for compatibility, but
|
||||
PR generation is image-first: scene, poster, painting, photograph, cover,
|
||||
tableau, staged artifact, or another editorial visual moment that describes the
|
||||
intent of the PR. PR images are non-gating BM Bossbot artifacts; changelog and
|
||||
release-summary images are manual evidence-pack workflows.
|
||||
|
||||
## Output Contract
|
||||
|
||||
- Base output directory: `docs/assets/infographics/`
|
||||
- PR image: `docs/assets/infographics/pr-<number>.webp`
|
||||
- Changelog image: `docs/assets/infographics/changelog.webp`
|
||||
- Weekly image:
|
||||
- This is always a 2-Week Retro window: previous ISO week through current ISO
|
||||
week (`start-week = current-week - 1`, `end-week = current-week`).
|
||||
- Same year window: `docs/assets/infographics/<year>-w<start-week>-w<end-week>.webp`
|
||||
- Cross-year window:
|
||||
`docs/assets/infographics/<start-year>-w<start-week>-<end-year>-w<end-week>.webp`
|
||||
|
||||
## PR Mode
|
||||
|
||||
PR mode uses the BM Bossbot summary block as source material. Do not hand-write
|
||||
claims that are not present in the PR body.
|
||||
|
||||
1. Fetch the PR body:
|
||||
|
||||
```bash
|
||||
gh pr view <number> --json body --jq '.body // ""' > /tmp/bm-pr-body.md
|
||||
```
|
||||
|
||||
2. Generate the canonical asset:
|
||||
|
||||
```bash
|
||||
uv run --script scripts/generate_pr_infographic.py \
|
||||
--pr-number <number> \
|
||||
--pr-body-file /tmp/bm-pr-body.md \
|
||||
--theme "<optional visual theme>" \
|
||||
--provenance-output /tmp/bm-infographic-provenance.md \
|
||||
--output docs/assets/infographics/pr-<number>.webp
|
||||
```
|
||||
|
||||
If the PR body contains a managed image theme block, the script reads it
|
||||
automatically:
|
||||
|
||||
```markdown
|
||||
<!-- BM_INFOGRAPHIC_THEME:start -->
|
||||
<theme>
|
||||
<!-- BM_INFOGRAPHIC_THEME:end -->
|
||||
```
|
||||
|
||||
Before spending an image call, test the prompt path locally:
|
||||
|
||||
```bash
|
||||
uv run --script scripts/generate_pr_infographic.py \
|
||||
--pr-number <number> \
|
||||
--pr-body-file /tmp/bm-pr-body.md \
|
||||
--theme "<optional visual theme>" \
|
||||
--output docs/assets/infographics/pr-<number>.webp \
|
||||
--print-prompt
|
||||
```
|
||||
|
||||
`--dry-run` is an alias for `--print-prompt`; both print the final prompt and
|
||||
exit without calling OpenAI.
|
||||
|
||||
When no theme is supplied, the script selects a deterministic BM visual
|
||||
direction from the style pool below based on the PR number and Bossbot summary.
|
||||
This keeps repeated PR images from collapsing into the same generic visual.
|
||||
|
||||
When the image is generated, also write provenance with
|
||||
`--provenance-output <path>`. BM Bossbot publishes that managed block into the
|
||||
PR body with these markers:
|
||||
|
||||
```markdown
|
||||
<!-- BM_INFOGRAPHIC_PROVENANCE:start -->
|
||||
...
|
||||
<!-- BM_INFOGRAPHIC_PROVENANCE:end -->
|
||||
```
|
||||
|
||||
The provenance block records the generated asset path, image model, size,
|
||||
quality, image mode, theme source, and selected visual direction. It
|
||||
intentionally does not dump the full generated prompt into the PR body. Treat
|
||||
this block as debugging and creative provenance only; it is not a merge gate.
|
||||
|
||||
The PR image is visual support only. The authoritative merge gate is the
|
||||
GitHub commit status named `BM Bossbot Approval`.
|
||||
|
||||
## Changelog Mode
|
||||
|
||||
Build an evidence pack before writing a prompt:
|
||||
|
||||
- diff truth source: merged PR diffs, merge commits, or local reconstructed diffs
|
||||
- changed-file orientation: `git diff --stat` plus key file reads
|
||||
- impact ledger: before/after outcomes tied to actual changes
|
||||
- discard list: misleading titles, reverted work, rename-only churn, speculative TODOs
|
||||
- chosen image form: poster, scene, tableau, cover, painting, photograph,
|
||||
staged artifact, or another editorial visual moment
|
||||
- chosen BM style category: exactly one category from the selection pool below
|
||||
|
||||
Read these references before drafting the prompt:
|
||||
|
||||
- `references/prompt-blueprint.md`
|
||||
- `references/style-balance.md`
|
||||
|
||||
Read the current `CHANGELOG.md` entries and include the latest meaningful
|
||||
changes.
|
||||
|
||||
## Style And Category Selection
|
||||
|
||||
Select exactly one BM style category per image based on semantic fit. The
|
||||
visual language should be recognizable and tasteful, while staying
|
||||
business-readable.
|
||||
|
||||
Create an image-first visual form that communicates the change: poster, scene,
|
||||
tableau, cover image, painting, photograph, staged artifact, or another
|
||||
editorial visual moment. Maps, diagrams, dossiers, charts, and labels can appear
|
||||
as props inside the scene, but do not make a text-heavy infographic.
|
||||
|
||||
BM category pool:
|
||||
|
||||
- computer science college textbooks: SICP-style diagrams, algorithms lectures,
|
||||
compiler pipelines, automata, database systems, type theory, operating systems
|
||||
- classic literature subjects: sea voyages, gothic manors, Dickensian city maps,
|
||||
Austen social graphs, library marginalia, travel journals
|
||||
- fantasy/D&D-inspired: quest maps, dungeon keys, guild ledgers, spellbooks,
|
||||
bestiaries, tavern notice boards; no copyrighted settings
|
||||
- Music: Metal, Hard Rock, Punk, techno, soul, reggae bands; no pop music, no
|
||||
direct band logos, album covers, or musician likenesses
|
||||
- sci-fi: Star Wars inspired knockoff, Spaceballs-adjacent space opera, fleet
|
||||
routes, mission consoles, contraband manifests; avoid copyrighted characters,
|
||||
logos, or named fictional universes
|
||||
- Conan the barbarian-inspired sword-and-sorcery: ruined temples, desert routes,
|
||||
battle standards, ancient maps; no named character likenesses
|
||||
- Comic books: issue covers, splash pages, action-panel maps, caption boxes,
|
||||
halftone energy, clean sound-effect typography
|
||||
- French new wave movies: poster style, stark typography, city route maps,
|
||||
jump-cut sequencing, high-contrast editorial photography cues
|
||||
- WWII propaganda posters: home-front public-information poster language,
|
||||
logistics arrows, ration charts, mobilization maps, bold simplified figures;
|
||||
no real-world party symbols, hate imagery, dehumanizing slogans, or false
|
||||
historical claims
|
||||
- Italian movie posters: hand-painted drama, bold credits, expressive color,
|
||||
route-map collage, 1960s or 1970s cinema energy; no direct film titles or
|
||||
actor likenesses
|
||||
- Shakespeare: stage maps, acts and scenes, dramatis personae, royal courts,
|
||||
backstage cue sheets
|
||||
- Greek mythology: temple diagrams, constellation routes, hero's journey maps,
|
||||
oracle tablets, labyrinths, ship routes
|
||||
- noir detective boards: case files, red-string maps, typed evidence labels,
|
||||
precinct wall charts
|
||||
- NASA mission-control dashboards: launch timelines, telemetry maps, orbital
|
||||
routes, status boards
|
||||
- space exploration and astronomy: celestial atlases, observatory charts,
|
||||
star-field maps, orbital mechanics diagrams, planetary survey routes,
|
||||
telescope annotations, mission trajectories, deep-space timelines
|
||||
- paintings: abstract painting, classical landscape, Remington-inspired western
|
||||
action painting, Rembrandt-inspired chiaroscuro, historical mural, stormy
|
||||
seascape, allegorical editorial painting
|
||||
- classic black-and-white photography: documentary field report, newsroom
|
||||
archive print, editorial photo essay, street photography, high-contrast
|
||||
darkroom print, contact sheet, civic infrastructure photograph
|
||||
- 80's action movies: practical explosions, smoky backlit warehouses, neon city
|
||||
streets, helicopter searchlights, mission dossiers, heroic silhouettes,
|
||||
high-stakes countdowns, painted ensemble posters; no direct actor likenesses,
|
||||
real film titles, franchise marks, or catchphrases
|
||||
- alchemy manuscripts: transformation diagrams, annotated symbols, recipe-like
|
||||
process maps, illuminated margins
|
||||
- brutalist civic planning: transit maps, concrete signage, zoning blocks,
|
||||
infrastructure diagrams
|
||||
|
||||
Selection rules:
|
||||
|
||||
- Pick one category only; do not create mixed mashups.
|
||||
- Pick the most appropriate image form. Prefer an actual scene, poster,
|
||||
painting, photograph, tableau, or cover over a text-heavy infographic.
|
||||
- Match metaphor to content, but do not overthink it. The category is a creative
|
||||
catalyst, not a semantic constraint.
|
||||
- Use a polished editorial rendering direction: smooth anti-aliased
|
||||
text, high contrast, clean edges, readable labels.
|
||||
- Make the category drive the composition through a readable staged moment,
|
||||
editorial composition, symbolic environment, route, artifact, or visual
|
||||
metaphor.
|
||||
- Keep the structure literal enough to aid understanding, but not so heavy that
|
||||
it obscures engineering meaning.
|
||||
- Give the image generator creative latitude on layout, structure, color palette,
|
||||
and visual metaphors. Be precise about what content to show, loose about how
|
||||
to show it.
|
||||
- Do not use copyrighted characters, logos, or named fictional universes. Use
|
||||
genre cues, knockoffs, and original compositions instead.
|
||||
|
||||
## Content-First Aesthetic Contract
|
||||
|
||||
The meaning must be readable and clearly hierarchical. Everything else is
|
||||
creative territory: image form, layout, visual metaphors, decorative elements,
|
||||
color choices, and category-specific visual language.
|
||||
|
||||
Hierarchy:
|
||||
|
||||
1. Meaning: what shipped, what changed, and why it matters must be clear.
|
||||
2. If the image uses text, labels, sections, or evidence bullets, they must be
|
||||
legible.
|
||||
3. The selected category's visual DNA should drive the composition as a poster,
|
||||
scene, painting, photograph, tableau, cover, or symbolic object arrangement.
|
||||
4. Do not play it safe. A visually striking image that someone wants to look at
|
||||
beats a correct but boring one.
|
||||
|
||||
Hard rules:
|
||||
|
||||
- Content sections and labels must be readable when present. Text cannot be
|
||||
obscured by decorations.
|
||||
- Do not use lore-heavy copy that competes with engineering or business meaning.
|
||||
- Every prompt must include a clear image-first composition cue: a staged scene,
|
||||
poster composition, painting, photograph, symbolic tableau, hero object,
|
||||
mission room, dossier, artifact, route, or visual metaphor.
|
||||
- Do not over-prescribe exact coordinates or panel geometry; give a composition
|
||||
backbone and let the model compose around it.
|
||||
|
||||
## Generation
|
||||
|
||||
1. Write the final prompt to a temporary markdown file.
|
||||
2. Generate with the shared image helper:
|
||||
|
||||
```bash
|
||||
uv run --script scripts/generate_infographic.py \
|
||||
--prompt-file /tmp/bm-infographic-prompt.md \
|
||||
--output docs/assets/infographics/<name>.webp
|
||||
```
|
||||
|
||||
3. Verify the image exists and is readable before reporting success.
|
||||
|
||||
## Quality Bar
|
||||
|
||||
- Tell a concrete before/after value story, not vague improvement claims.
|
||||
- Stay understandable for both engineers and non-technical stakeholders.
|
||||
- Use plain-language section titles and labels when text is present.
|
||||
- Include clear visual hierarchy: title, staged focal point, symbolic scene,
|
||||
evidence props, or hero object.
|
||||
- Avoid invented facts; only use provided source material.
|
||||
- Favor shipped outcomes over intermediate or reverted work.
|
||||
- Preserve readability with high contrast, non-tiny labels, and uncluttered
|
||||
layout.
|
||||
@@ -0,0 +1,7 @@
|
||||
interface:
|
||||
display_name: "Infographics"
|
||||
short_description: "Generate Basic Memory repo infographics"
|
||||
icon_small: "./assets/icon.svg"
|
||||
icon_large: "./assets/icon.svg"
|
||||
brand_color: "#2563EB"
|
||||
default_prompt: "Use $infographics to generate a Basic Memory PR or changelog infographic with canonical output paths."
|
||||
@@ -0,0 +1,5 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="128" height="128" viewBox="0 0 24 24" fill="none" stroke="#111827" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round">
|
||||
<path d="M3 12h4l2-6 4 12 2-6h6"/>
|
||||
<path d="M4 20h16"/>
|
||||
</svg>
|
||||
|
||||
|
After Width: | Height: | Size: 249 B |
@@ -0,0 +1,72 @@
|
||||
# Prompt Blueprint
|
||||
|
||||
Convert an evidence pack into a final visual prompt. Be precise about the
|
||||
content and loose about visual execution.
|
||||
|
||||
## Required Inputs
|
||||
|
||||
- Diff truth source summary
|
||||
- Changed-file orientation summary
|
||||
- Impact ledger with before/after outcomes
|
||||
- Discard list for excluded noise
|
||||
- Chosen image form
|
||||
- Chosen BM style category
|
||||
|
||||
## Prompt Shape
|
||||
|
||||
```text
|
||||
Create a polished Basic Memory editorial image inspired by
|
||||
<BM_STYLE_CATEGORY>. Use a poster, scene, tableau, painting, photograph, cover
|
||||
image, staged artifact, or another image-first form that best communicates the
|
||||
intent. Use HD editorial rendering with smooth anti-aliased text when text is
|
||||
present. Go bold and let the selected category drive the visual language through
|
||||
original, non-infringing cues.
|
||||
|
||||
TITLE:
|
||||
- "<clear title>"
|
||||
- "<scope subtitle>"
|
||||
|
||||
COMPOSITION:
|
||||
- Recreate a clear staged moment or symbolic image that describes the PR
|
||||
intent.
|
||||
- Maps, diagrams, dossiers, route lines, labels, and artifacts can appear as
|
||||
props inside the scene, but the output should read as an image rather than a
|
||||
dense infographic.
|
||||
- Take creative liberty with layout and styling.
|
||||
- The hard rule: the meaning must be readable and clearly hierarchical.
|
||||
- Keep labels plain-language and technical when labels are used.
|
||||
|
||||
CONTENT:
|
||||
1. "<section>"
|
||||
- <evidence-grounded outcome>
|
||||
- <evidence-grounded outcome>
|
||||
|
||||
2. "<section>"
|
||||
- <evidence-grounded outcome>
|
||||
- <evidence-grounded outcome>
|
||||
|
||||
METRICS:
|
||||
- <metric>
|
||||
- <metric>
|
||||
|
||||
STYLE DIRECTION:
|
||||
- Upscaled editorial, high contrast, anti-aliased text, smooth edges.
|
||||
- Let the category's visual DNA drive the composition.
|
||||
- Use genre/category cues only; do not use copyrighted characters, logos, named
|
||||
fictional universes, direct band logos, album art, or celebrity likenesses.
|
||||
|
||||
DO NOT:
|
||||
- Make text unreadable or let decoration obscure content.
|
||||
- Render a text-heavy infographic, dashboard, flowchart, timeline strip,
|
||||
checklist, bullet-list panel, or dense explanatory diagram.
|
||||
- Use crunchy low-resolution pixel art.
|
||||
- Invent facts not present in the evidence pack.
|
||||
```
|
||||
|
||||
## Writing Rules
|
||||
|
||||
- Keep each bullet specific and evidence-grounded.
|
||||
- Prefer outcome language over implementation trivia.
|
||||
- Default to three or four sections; never exceed five.
|
||||
- Give proportionally more space to dominant changes.
|
||||
- Keep the final prompt short, energetic, and readable.
|
||||
@@ -0,0 +1,42 @@
|
||||
# Style Balance Rubric
|
||||
|
||||
## Core Principle
|
||||
|
||||
Be bold, not confusing. The selected BM style category should structure the
|
||||
visual through a readable image-first composition, not decorate a generic grid.
|
||||
Use an editorial scene, poster, painting, photograph, cover, staged artifact, or
|
||||
tableau that turns the PR intent into a visual moment.
|
||||
|
||||
## Required Traits
|
||||
|
||||
- Anti-aliased typography
|
||||
- Smooth edges
|
||||
- High contrast between text and background
|
||||
- Plain-language section labels
|
||||
- Clear composition backbone: staged scene, editorial poster, painting,
|
||||
photograph, symbolic tableau, hero artifact, dossier, mission room, or route
|
||||
embodied as part of the scene
|
||||
- A single coherent BM style category, expressed through original visual cues
|
||||
|
||||
## Reject Or Rewrite If
|
||||
|
||||
- Content text is unreadable.
|
||||
- The prompt lacks a composition backbone.
|
||||
- The prompt over-prescribes exact panel positions or a rigid grid.
|
||||
- The style leans into crunchy low-resolution pixelation.
|
||||
- Copy uses lore-heavy references instead of engineering meaning.
|
||||
- The prompt uses copyrighted characters, logos, named fictional universes,
|
||||
direct band logos, album art, or celebrity likenesses.
|
||||
|
||||
## Creative Integration Patterns
|
||||
|
||||
- Use category-native map details to organize content: textbook diagrams,
|
||||
literary journeys, quest maps, tour posters, mission-control routes, stage
|
||||
blocking, mythic constellations, star charts, mission trajectories, case
|
||||
boards, or civic plans as props inside the image.
|
||||
- Recreate a scene, editorial poster, painting, photograph, cover, artifact, or
|
||||
tableau instead of sectioned bullets.
|
||||
- Map engineering metrics to visual counters, route progress, or status boards
|
||||
only when they naturally belong in the scene.
|
||||
- Let headers and accents borrow from the selected style.
|
||||
- Keep atmospheric details behind or around content, never over it.
|
||||
@@ -0,0 +1,114 @@
|
||||
---
|
||||
name: pr-create
|
||||
description: Use when creating or updating a Basic Memory pull request from Codex with BM Bossbot merge-gate monitoring.
|
||||
---
|
||||
|
||||
# Create A Basic Memory PR
|
||||
|
||||
Create or update a pull request for the current branch, then wait for BM
|
||||
Bossbot to approve the latest head SHA. This skill never merges a PR.
|
||||
|
||||
## Inputs
|
||||
|
||||
- Optional `<theme>`: free-form visual direction for the non-gating PR
|
||||
image. Example: `$pr-create "Italian movie poster"`.
|
||||
- Treat `<theme>` as style guidance only. It must not affect PR readiness,
|
||||
BM Bossbot review, status checks, or merge behavior.
|
||||
|
||||
## How To Use
|
||||
|
||||
Ask Codex to use the skill from a feature branch:
|
||||
|
||||
```text
|
||||
$pr-create
|
||||
$pr-create "Italian movie poster"
|
||||
$pr-create "80's action movies"
|
||||
```
|
||||
|
||||
Use the plain form when you only want the PR workflow. Pass a theme when you
|
||||
want the non-gating image to lean toward a particular visual direction. The
|
||||
theme can be specific ("Rembrandt-inspired approval scene") or broad ("let the
|
||||
model choose from BM categories").
|
||||
|
||||
## What Happens
|
||||
|
||||
1. Codex checks the branch, local verification, GitHub auth, commit sign-offs,
|
||||
and semantic PR title shape.
|
||||
2. Codex pushes the branch, creates or reuses the PR, and adds the optional
|
||||
`BM_INFOGRAPHIC_THEME` block when a theme was supplied.
|
||||
3. BM Bossbot runs from trusted base code, reviews sanitized PR metadata and
|
||||
diff context, and sets the required `BM Bossbot Approval` status for the
|
||||
exact head SHA.
|
||||
4. If approval succeeds, BM Bossbot may publish a non-gating image block and a
|
||||
provenance block:
|
||||
|
||||
```markdown
|
||||
<!-- BM_INFOGRAPHIC_PROVENANCE:start -->
|
||||
...
|
||||
<!-- BM_INFOGRAPHIC_PROVENANCE:end -->
|
||||
```
|
||||
|
||||
The provenance records the image mode, theme source, selected visual
|
||||
direction, and image settings. It is for review/debugging context only.
|
||||
5. Codex reports the PR URL, head SHA, checks watched, verification run, and BM
|
||||
Bossbot verdict.
|
||||
|
||||
The skill never merges, never enables auto-merge, and never treats the image or
|
||||
provenance block as a gate. The only required merge signal is the
|
||||
`BM Bossbot Approval` status on the current PR head SHA.
|
||||
|
||||
## Preflight
|
||||
|
||||
1. Confirm the repo and branch:
|
||||
- `git status --short --branch`
|
||||
- stop if detached or on `main`
|
||||
- keep unrelated user changes intact
|
||||
|
||||
2. Confirm GitHub access:
|
||||
- `gh auth status`
|
||||
- `gh repo view --json nameWithOwner,defaultBranchRef,url`
|
||||
|
||||
3. Check PR readiness:
|
||||
- commits are signed off with `git commit -s`
|
||||
- title uses the repo semantic format
|
||||
- local verification appropriate to the change has run
|
||||
|
||||
## Create Or Reuse
|
||||
|
||||
1. Push the branch:
|
||||
- `git push -u origin HEAD`
|
||||
|
||||
2. Check for an existing PR:
|
||||
- `gh pr view --json number,url,headRefOid,mergeStateStatus,statusCheckRollup`
|
||||
|
||||
3. If no PR exists, create one:
|
||||
- `gh pr create --fill`
|
||||
- adjust the title if it does not satisfy the semantic PR title workflow
|
||||
|
||||
4. If `<theme>` is provided, add or update this managed block in the PR body:
|
||||
|
||||
```markdown
|
||||
<!-- BM_INFOGRAPHIC_THEME:start -->
|
||||
<theme>
|
||||
<!-- BM_INFOGRAPHIC_THEME:end -->
|
||||
```
|
||||
|
||||
Keep the rest of the PR body intact. The theme is non-gating image guidance
|
||||
only.
|
||||
|
||||
5. Do not merge. Do not enable auto-merge.
|
||||
|
||||
## Watch The Gate
|
||||
|
||||
1. Trigger or wait for `.github/workflows/bm-bossbot.yml`.
|
||||
2. Watch the required commit status named `BM Bossbot Approval`.
|
||||
3. Treat approval as valid only when it is green for the current `headRefOid`.
|
||||
4. If the branch changes after approval, wait for BM Bossbot to review the new
|
||||
head SHA.
|
||||
5. If BM Bossbot fails or requests changes, use `$fix-pr-issues`.
|
||||
|
||||
## Report
|
||||
|
||||
Return the PR URL, current head SHA, checks watched, verification run, and the
|
||||
BM Bossbot verdict. Include the image `<theme>` if one was supplied. Be
|
||||
explicit when any check is still pending.
|
||||
@@ -0,0 +1,7 @@
|
||||
interface:
|
||||
display_name: "PR Create"
|
||||
short_description: "Create PRs and wait for BM Bossbot"
|
||||
icon_small: "./assets/icon.svg"
|
||||
icon_large: "./assets/icon.svg"
|
||||
brand_color: "#2563EB"
|
||||
default_prompt: "Use $pr-create to create or update this Basic Memory PR and wait for BM Bossbot Approval."
|
||||
@@ -0,0 +1,5 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="128" height="128" viewBox="0 0 24 24" fill="none" stroke="#111827" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round">
|
||||
<path d="M3 12h4l2-6 4 12 2-6h6"/>
|
||||
<path d="M4 20h16"/>
|
||||
</svg>
|
||||
|
||||
|
After Width: | Height: | Size: 249 B |
@@ -30,7 +30,8 @@ The justfile target handles:
|
||||
- ✅ Beta version format validation (supports b1, b2, rc1, etc.)
|
||||
- ✅ Git status and branch checks
|
||||
- ✅ Quality checks (`just check` - lint, format, type-check, tests)
|
||||
- ✅ Version update in `src/basic_memory/__init__.py`
|
||||
- ✅ Version update across all consolidated manifests via `just set-version` (Python
|
||||
package + Claude Code plugin/marketplaces + Codex plugin + Hermes + OpenClaw)
|
||||
- ✅ Automatic commit with proper message
|
||||
- ✅ Tag creation and pushing to GitHub
|
||||
- ✅ Beta release workflow trigger
|
||||
@@ -90,6 +91,6 @@ Monitor release: https://github.com/basicmachines-co/basic-memory/actions
|
||||
- Beta releases are pre-releases for testing new features
|
||||
- Automatically published to PyPI with pre-release flag
|
||||
- Uses the automated justfile target for consistency
|
||||
- Version is automatically updated in `__init__.py`
|
||||
- Version is automatically updated across all consolidated manifests via `just set-version`
|
||||
- Ideal for validating changes before stable release
|
||||
- Supports both beta (b1, b2) and release candidate (rc1, rc2) versions
|
||||
- Supports both beta (b1, b2) and release candidate (rc1, rc2) versions
|
||||
|
||||
@@ -42,7 +42,8 @@ The justfile target handles:
|
||||
- ✅ Version format validation
|
||||
- ✅ Git status and branch checks
|
||||
- ✅ Quality checks (`just check` - lint, format, type-check, tests)
|
||||
- ✅ Version update across all consolidated manifests via `just set-version` (Python package + Claude Code plugin/marketplaces + Hermes + OpenClaw)
|
||||
- ✅ Version update across all consolidated manifests via `just set-version` (Python
|
||||
package + Claude Code plugin/marketplaces + Codex plugin + Hermes + OpenClaw)
|
||||
- ✅ Automatic commit with proper message
|
||||
- ✅ Tag creation and pushing to GitHub
|
||||
- ✅ Release workflow trigger (automatic on tag push)
|
||||
@@ -194,12 +195,13 @@ Users can now upgrade:
|
||||
- Version is automatically updated across **all** consolidated manifests via
|
||||
`just set-version <version>` (which calls `scripts/update_versions.py`): the
|
||||
Python package (`__init__.py`, `server.json`) **and** the plugin/agent artifacts
|
||||
(Claude Code `plugin.json` + root/local marketplaces, Hermes `plugin.yaml` +
|
||||
`__init__.py`, OpenClaw `package.json`). To bump only the plugin/agent artifacts
|
||||
(Claude Code `plugin.json` + root/local marketplaces, Codex `plugin.json`,
|
||||
Hermes `plugin.yaml` + `__init__.py`, OpenClaw `package.json`). To bump only
|
||||
the plugin/agent artifacts
|
||||
out of band, use `just set-packages-version <version>` (preview with
|
||||
`just set-packages-version-dry-run <version>`).
|
||||
- Triggers automated GitHub release with changelog
|
||||
- Package is published to PyPI for `pip` and `uv` users
|
||||
- Homebrew formula is automatically updated for stable releases
|
||||
- MCP Registry is updated manually via `mcp-publisher publish`
|
||||
- Supports multiple installation methods (uv, pip, Homebrew)
|
||||
- Supports multiple installation methods (uv, pip, Homebrew)
|
||||
|
||||
+1
@@ -0,0 +1 @@
|
||||
../../.agents/skills/basic-machines-review
|
||||
@@ -0,0 +1,193 @@
|
||||
name: BM Bossbot
|
||||
|
||||
"on":
|
||||
workflow_run:
|
||||
workflows:
|
||||
- Tests
|
||||
types:
|
||||
- completed
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
pr_number:
|
||||
description: Pull request number to review
|
||||
required: true
|
||||
# Review-thread activity re-evaluates the approval status without re-running
|
||||
# the full LLM review: new feedback flips the gate to failure, resolving the
|
||||
# last thread restores a previously earned approval for the same head SHA.
|
||||
pull_request_review:
|
||||
types:
|
||||
- submitted
|
||||
pull_request_review_comment:
|
||||
types:
|
||||
- created
|
||||
pull_request_review_thread:
|
||||
types:
|
||||
- resolved
|
||||
- unresolved
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: write
|
||||
statuses: write
|
||||
issues: read
|
||||
|
||||
env:
|
||||
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
|
||||
BM_BOSSBOT_STATUS_CONTEXT: "BM Bossbot Approval"
|
||||
|
||||
jobs:
|
||||
review:
|
||||
name: BM Bossbot Review
|
||||
# Job-level concurrency (not workflow-level): thread-recheck events for the
|
||||
# same PR must never cancel an in-flight review run, and vice versa.
|
||||
concurrency:
|
||||
group: bm-bossbot-review-${{ github.event.workflow_run.pull_requests[0].number || inputs.pr_number }}
|
||||
cancel-in-progress: true
|
||||
if: |
|
||||
github.event_name == 'workflow_dispatch' ||
|
||||
(
|
||||
github.event_name == 'workflow_run' &&
|
||||
github.event.workflow_run.conclusion == 'success' &&
|
||||
github.event.workflow_run.pull_requests[0].number != ''
|
||||
)
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
pr_number: ${{ steps.pr.outputs.pr_number }}
|
||||
should_review: ${{ steps.pr.outputs.should_review }}
|
||||
|
||||
steps:
|
||||
- name: Checkout trusted base ref
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ github.event.repository.default_branch }}
|
||||
fetch-depth: 1
|
||||
|
||||
- name: Set up uv
|
||||
uses: astral-sh/setup-uv@v3
|
||||
|
||||
- name: Normalize PR event
|
||||
id: pr
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
event_file="${RUNNER_TEMP}/bm-bossbot-event.json"
|
||||
should_review=true
|
||||
|
||||
if [ "${GITHUB_EVENT_NAME}" = "workflow_dispatch" ]; then
|
||||
pr_number="${{ inputs.pr_number }}"
|
||||
tested_sha=""
|
||||
else
|
||||
pr_number="$(jq -r '.workflow_run.pull_requests[0].number // ""' "${GITHUB_EVENT_PATH}")"
|
||||
tested_sha="$(jq -r '.workflow_run.head_sha // ""' "${GITHUB_EVENT_PATH}")"
|
||||
fi
|
||||
|
||||
gh api "repos/${GITHUB_REPOSITORY}/pulls/${pr_number}" > "${RUNNER_TEMP}/pull.json"
|
||||
current_head_sha="$(jq -r '.head.sha' "${RUNNER_TEMP}/pull.json")"
|
||||
draft="$(jq -r '.draft' "${RUNNER_TEMP}/pull.json")"
|
||||
|
||||
if [ "${GITHUB_EVENT_NAME}" = "workflow_dispatch" ]; then
|
||||
tests_run_id="$(
|
||||
gh api -X GET "repos/${GITHUB_REPOSITORY}/actions/workflows/test.yml/runs" \
|
||||
-f event=push \
|
||||
-f head_sha="${current_head_sha}" \
|
||||
-f status=completed \
|
||||
--jq '[.workflow_runs[] | select(.conclusion == "success")][0].id // ""'
|
||||
)"
|
||||
if [ -z "${tests_run_id}" ]; then
|
||||
should_review=false
|
||||
echo "BM Bossbot skipped PR ${pr_number}: no successful Tests workflow for ${current_head_sha}."
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ "${draft}" = "true" ]; then
|
||||
should_review=false
|
||||
echo "BM Bossbot skipped PR ${pr_number}: draft pull request."
|
||||
fi
|
||||
|
||||
if [ -n "${tested_sha}" ] && [ "${tested_sha}" != "${current_head_sha}" ]; then
|
||||
should_review=false
|
||||
echo "BM Bossbot skipped PR ${pr_number}: Tests passed for ${tested_sha}, but current head is ${current_head_sha}."
|
||||
fi
|
||||
|
||||
jq --arg repo "${GITHUB_REPOSITORY}" \
|
||||
'{repository:{full_name:$repo}, pull_request:{number:.number,title:.title,body:(.body // ""),html_url:.html_url,head:{sha:.head.sha,ref:.head.ref},base:{ref:.base.ref,sha:.base.sha},author_association:.author_association,draft:.draft}}' \
|
||||
"${RUNNER_TEMP}/pull.json" > "${event_file}"
|
||||
|
||||
echo "event_file=${event_file}" >> "${GITHUB_OUTPUT}"
|
||||
echo "pr_number=$(jq -r '.pull_request.number' "${event_file}")" >> "${GITHUB_OUTPUT}"
|
||||
echo "head_sha=$(jq -r '.pull_request.head.sha' "${event_file}")" >> "${GITHUB_OUTPUT}"
|
||||
echo "head_ref=$(jq -r '.pull_request.head.ref' "${event_file}")" >> "${GITHUB_OUTPUT}"
|
||||
echo "author_association=$(jq -r '.pull_request.author_association // ""' "${event_file}")" >> "${GITHUB_OUTPUT}"
|
||||
echo "tested_sha=${tested_sha}" >> "${GITHUB_OUTPUT}"
|
||||
echo "should_review=${should_review}" >> "${GITHUB_OUTPUT}"
|
||||
|
||||
- name: Classify PR author
|
||||
id: trust
|
||||
if: steps.pr.outputs.should_review == 'true'
|
||||
env:
|
||||
AUTHOR_ASSOCIATION: ${{ steps.pr.outputs.author_association }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
case "${AUTHOR_ASSOCIATION}" in
|
||||
OWNER|MEMBER|COLLABORATOR)
|
||||
trusted_author=true
|
||||
;;
|
||||
*)
|
||||
trusted_author=false
|
||||
;;
|
||||
esac
|
||||
echo "trusted_author=${trusted_author}" >> "${GITHUB_OUTPUT}"
|
||||
echo "author_association=${AUTHOR_ASSOCIATION}" >> "${GITHUB_OUTPUT}"
|
||||
|
||||
- name: Mark BM Bossbot approval pending
|
||||
if: steps.pr.outputs.should_review == 'true'
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
uv run --script scripts/bm_bossbot_status.py pending \
|
||||
--event "${{ steps.pr.outputs.event_file }}" \
|
||||
--repo "${GITHUB_REPOSITORY}" \
|
||||
--run-url "${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
|
||||
|
||||
- name: Finalize BM Bossbot approval
|
||||
if: always() && steps.pr.outputs.should_review == 'true'
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
uv run --script scripts/bm_bossbot_status.py finalize \
|
||||
--event "${{ steps.pr.outputs.event_file }}" \
|
||||
--trusted "${{ steps.trust.outputs.trusted_author }}" \
|
||||
--repo "${GITHUB_REPOSITORY}" \
|
||||
--run-url "${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
|
||||
|
||||
recheck:
|
||||
name: BM Bossbot Thread Recheck
|
||||
if: |
|
||||
github.event_name == 'pull_request_review' ||
|
||||
github.event_name == 'pull_request_review_comment' ||
|
||||
github.event_name == 'pull_request_review_thread'
|
||||
runs-on: ubuntu-latest
|
||||
# Job-level concurrency: collapse bursts of thread events for one PR while
|
||||
# staying isolated from the review job's group so neither cancels the other.
|
||||
concurrency:
|
||||
group: bm-bossbot-recheck-${{ github.event.pull_request.number }}
|
||||
cancel-in-progress: true
|
||||
steps:
|
||||
- name: Checkout trusted base ref
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ github.event.repository.default_branch }}
|
||||
fetch-depth: 1
|
||||
|
||||
- name: Set up uv
|
||||
uses: astral-sh/setup-uv@v3
|
||||
|
||||
- name: Re-evaluate approval from review-thread state
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
uv run --script scripts/bm_bossbot_status.py recheck \
|
||||
--pr-number "${{ github.event.pull_request.number }}" \
|
||||
--repo "${GITHUB_REPOSITORY}" \
|
||||
--run-url "${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
|
||||
@@ -1,26 +1,18 @@
|
||||
name: Claude Code Review
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [opened, synchronize]
|
||||
# Optional: Only run on specific file changes
|
||||
# paths:
|
||||
# - "src/**/*.ts"
|
||||
# - "src/**/*.tsx"
|
||||
# - "src/**/*.js"
|
||||
# - "src/**/*.jsx"
|
||||
"on":
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
pr_number:
|
||||
description: Pull request number to review manually
|
||||
required: true
|
||||
|
||||
env:
|
||||
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
|
||||
|
||||
jobs:
|
||||
claude-review:
|
||||
# Only run for organization members and collaborators
|
||||
if: |
|
||||
github.event.pull_request.author_association == 'OWNER' ||
|
||||
github.event.pull_request.author_association == 'MEMBER' ||
|
||||
github.event.pull_request.author_association == 'COLLABORATOR'
|
||||
|
||||
if: inputs.pr_number != ''
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
@@ -43,7 +35,14 @@ jobs:
|
||||
track_progress: true # Enable visual progress tracking
|
||||
allowed_bots: '*'
|
||||
prompt: |
|
||||
Review this Basic Memory PR against our team checklist:
|
||||
Review Basic Memory PR #${{ inputs.pr_number }} as an advisory manual review.
|
||||
|
||||
Use `gh pr view ${{ inputs.pr_number }}` and related `gh pr`/`gh api`
|
||||
commands to inspect the pull request. Do not merge the PR and do not
|
||||
treat this advisory review as the required merge gate. BM Bossbot owns
|
||||
the required `BM Bossbot Approval` status.
|
||||
|
||||
Review the PR against our team checklist:
|
||||
|
||||
## Code Quality & Standards
|
||||
- [ ] Follows Basic Memory's coding conventions in CLAUDE.md
|
||||
|
||||
@@ -13,9 +13,10 @@ env:
|
||||
|
||||
jobs:
|
||||
docker:
|
||||
runs-on: ubuntu-latest
|
||||
runs-on: depot-ubuntu-24.04
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write
|
||||
packages: write
|
||||
|
||||
steps:
|
||||
@@ -24,10 +25,8 @@ jobs:
|
||||
with:
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v4
|
||||
with:
|
||||
platforms: linux/amd64,linux/arm64
|
||||
- name: Set up Depot
|
||||
uses: depot/setup-action@v1
|
||||
|
||||
- name: Log in to GitHub Container Registry
|
||||
uses: docker/login-action@v4
|
||||
@@ -49,13 +48,12 @@ jobs:
|
||||
type=raw,value=latest,enable={{is_default_branch}}
|
||||
|
||||
- name: Build and push Docker image
|
||||
uses: docker/build-push-action@v7
|
||||
uses: depot/build-push-action@v1
|
||||
with:
|
||||
project: ${{ vars.DEPOT_BASIC_MEMORY_PROJECT_ID || vars.DEPOT_PROJECT_ID }}
|
||||
context: .
|
||||
file: ./Dockerfile
|
||||
platforms: linux/amd64,linux/arm64
|
||||
push: true
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
@@ -13,12 +13,15 @@ on:
|
||||
|
||||
env:
|
||||
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
|
||||
# Required CI records pytest-testmon data but still runs the full selected suite.
|
||||
# The selective mode stays on explicit developer flows such as `just testmon`.
|
||||
BASIC_MEMORY_TESTMON_FLAGS: "--testmon-noselect"
|
||||
|
||||
jobs:
|
||||
static-checks:
|
||||
name: Static Checks (Python 3.12)
|
||||
timeout-minutes: 20
|
||||
runs-on: ubuntu-latest
|
||||
runs-on: depot-ubuntu-24.04
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
@@ -60,10 +63,12 @@ jobs:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- os: ubuntu-latest
|
||||
- os: depot-ubuntu-24.04
|
||||
python-version: "3.12"
|
||||
- os: ubuntu-latest
|
||||
- os: depot-ubuntu-24.04
|
||||
python-version: "3.13"
|
||||
# Python 3.14 unit tests are the longest full-suite slice; keep this
|
||||
# one on GitHub-hosted runners after Depot terminated it mid-suite.
|
||||
- os: ubuntu-latest
|
||||
python-version: "3.14"
|
||||
- os: windows-latest
|
||||
@@ -87,6 +92,19 @@ jobs:
|
||||
|
||||
- uses: extractions/setup-just@v4
|
||||
|
||||
- name: Cache pytest-testmon results
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: |
|
||||
.testmondata
|
||||
.testmondata-shm
|
||||
.testmondata-wal
|
||||
key: ${{ runner.os }}-testmon-sqlite-unit-py${{ matrix.python-version }}-${{ github.ref_name }}-${{ github.run_id }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-testmon-sqlite-unit-py${{ matrix.python-version }}-${{ github.ref_name }}-
|
||||
${{ runner.os }}-testmon-sqlite-unit-py${{ matrix.python-version }}-main-
|
||||
${{ runner.os }}-testmon-sqlite-unit-py${{ matrix.python-version }}-
|
||||
|
||||
- name: Create virtual env
|
||||
run: |
|
||||
uv venv
|
||||
@@ -106,11 +124,11 @@ jobs:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- os: ubuntu-latest
|
||||
- os: depot-ubuntu-24.04
|
||||
python-version: "3.12"
|
||||
- os: ubuntu-latest
|
||||
- os: depot-ubuntu-24.04
|
||||
python-version: "3.13"
|
||||
- os: ubuntu-latest
|
||||
- os: depot-ubuntu-24.04
|
||||
python-version: "3.14"
|
||||
- os: windows-latest
|
||||
python-version: "3.12"
|
||||
@@ -133,6 +151,19 @@ jobs:
|
||||
|
||||
- uses: extractions/setup-just@v4
|
||||
|
||||
- name: Cache pytest-testmon results
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: |
|
||||
.testmondata
|
||||
.testmondata-shm
|
||||
.testmondata-wal
|
||||
key: ${{ runner.os }}-testmon-sqlite-integration-py${{ matrix.python-version }}-${{ github.ref_name }}-${{ github.run_id }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-testmon-sqlite-integration-py${{ matrix.python-version }}-${{ github.ref_name }}-
|
||||
${{ runner.os }}-testmon-sqlite-integration-py${{ matrix.python-version }}-main-
|
||||
${{ runner.os }}-testmon-sqlite-integration-py${{ matrix.python-version }}-
|
||||
|
||||
- name: Create virtual env
|
||||
run: |
|
||||
uv venv
|
||||
@@ -153,9 +184,14 @@ jobs:
|
||||
matrix:
|
||||
include:
|
||||
- python-version: "3.12"
|
||||
os: depot-ubuntu-24.04
|
||||
- python-version: "3.13"
|
||||
os: depot-ubuntu-24.04
|
||||
# Match the SQLite unit slice: this full Python 3.14 path outlived
|
||||
# the Depot runner and was terminated mid-suite.
|
||||
- python-version: "3.14"
|
||||
runs-on: ubuntu-latest
|
||||
os: ubuntu-latest
|
||||
runs-on: ${{ matrix.os }}
|
||||
services:
|
||||
postgres:
|
||||
image: pgvector/pgvector:pg16
|
||||
@@ -190,6 +226,19 @@ jobs:
|
||||
|
||||
- uses: extractions/setup-just@v4
|
||||
|
||||
- name: Cache pytest-testmon results
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: |
|
||||
.testmondata
|
||||
.testmondata-shm
|
||||
.testmondata-wal
|
||||
key: ${{ runner.os }}-testmon-postgres-unit-py${{ matrix.python-version }}-${{ github.ref_name }}-${{ github.run_id }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-testmon-postgres-unit-py${{ matrix.python-version }}-${{ github.ref_name }}-
|
||||
${{ runner.os }}-testmon-postgres-unit-py${{ matrix.python-version }}-main-
|
||||
${{ runner.os }}-testmon-postgres-unit-py${{ matrix.python-version }}-
|
||||
|
||||
- name: Create virtual env
|
||||
run: |
|
||||
uv venv
|
||||
@@ -212,7 +261,7 @@ jobs:
|
||||
- python-version: "3.12"
|
||||
- python-version: "3.13"
|
||||
- python-version: "3.14"
|
||||
runs-on: ubuntu-latest
|
||||
runs-on: depot-ubuntu-24.04
|
||||
services:
|
||||
postgres:
|
||||
image: pgvector/pgvector:pg16
|
||||
@@ -247,6 +296,19 @@ jobs:
|
||||
|
||||
- uses: extractions/setup-just@v4
|
||||
|
||||
- name: Cache pytest-testmon results
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: |
|
||||
.testmondata
|
||||
.testmondata-shm
|
||||
.testmondata-wal
|
||||
key: ${{ runner.os }}-testmon-postgres-integration-py${{ matrix.python-version }}-${{ github.ref_name }}-${{ github.run_id }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-testmon-postgres-integration-py${{ matrix.python-version }}-${{ github.ref_name }}-
|
||||
${{ runner.os }}-testmon-postgres-integration-py${{ matrix.python-version }}-main-
|
||||
${{ runner.os }}-testmon-postgres-integration-py${{ matrix.python-version }}-
|
||||
|
||||
- name: Create virtual env
|
||||
run: |
|
||||
uv venv
|
||||
@@ -262,7 +324,7 @@ jobs:
|
||||
test-semantic:
|
||||
name: Test Semantic (Python 3.12)
|
||||
timeout-minutes: 45
|
||||
runs-on: ubuntu-latest
|
||||
runs-on: depot-ubuntu-24.04
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
@@ -281,6 +343,19 @@ jobs:
|
||||
|
||||
- uses: extractions/setup-just@v4
|
||||
|
||||
- name: Cache pytest-testmon results
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: |
|
||||
.testmondata
|
||||
.testmondata-shm
|
||||
.testmondata-wal
|
||||
key: ${{ runner.os }}-testmon-semantic-py3.12-${{ github.ref_name }}-${{ github.run_id }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-testmon-semantic-py3.12-${{ github.ref_name }}-
|
||||
${{ runner.os }}-testmon-semantic-py3.12-main-
|
||||
${{ runner.os }}-testmon-semantic-py3.12-
|
||||
|
||||
- name: Create virtual env
|
||||
run: |
|
||||
uv venv
|
||||
|
||||
+1
-1
@@ -49,7 +49,7 @@ ENV/
|
||||
/docs/.obsidian/
|
||||
/examples/.obsidian/
|
||||
/examples/.basic-memory/
|
||||
|
||||
/docs/assets
|
||||
|
||||
# claude action
|
||||
claude-output
|
||||
|
||||
@@ -108,13 +108,34 @@ Before opening or updating a PR, run the checks that mirror the common required
|
||||
- Follow the repository pattern for data access
|
||||
- Tools communicate to api routers via the httpx ASGI client (in process)
|
||||
|
||||
### Programming Style
|
||||
|
||||
See [docs/ENGINEERING_STYLE.md](docs/ENGINEERING_STYLE.md) for the fuller house style. The
|
||||
short version for agents:
|
||||
|
||||
- Prefer type-safe, explicit designs over object-heavy indirection. Use Python 3.12 `type`
|
||||
aliases, full annotations, and narrow `Protocol`s when a caller only needs a capability.
|
||||
- Use dataclasses for internal value objects and operation results; use Pydantic v2 at API,
|
||||
CLI, MCP, and persistence boundaries where validation and serialization matter.
|
||||
- Keep async boundaries obvious. Resource-owning code should use context managers, propagate
|
||||
cancellation, and avoid hidden background work unless the lifecycle is explicit.
|
||||
- Fail fast. Do not add silent fallback logic, broad exception swallowing, speculative
|
||||
`getattr`, or casts that hide an unclear model shape.
|
||||
- Keep control flow simple and local. Push branching decisions up, keep leaf helpers focused,
|
||||
and name values after the domain concept they carry.
|
||||
- Use evidence-first testing. Add or update meaningful regression tests for bugs and risky
|
||||
behavior, prefer real code paths over mocks, and run the narrowest command that proves the
|
||||
change before widening verification.
|
||||
- Comments should explain why a branch, invariant, or constraint exists. Avoid comments that
|
||||
merely narrate obvious code.
|
||||
|
||||
### Code Change Guidelines
|
||||
|
||||
- **Full file read before edits**: Before editing any file, read it in full first to ensure complete context; partial reads lead to corrupted edits
|
||||
- **Minimize diffs**: Prefer the smallest change that satisfies the request. Avoid unrelated refactors or style rewrites unless necessary for correctness
|
||||
- **No speculative getattr**: Never use `getattr(obj, "attr", default)` when unsure about attribute names. Check the class definition or source code first
|
||||
- **Fail fast**: Write code with fail-fast logic by default. Do not swallow exceptions with errors or warnings
|
||||
- **No fallback logic**: Do not add fallback logic unless explicitly told to and agreed with the user
|
||||
- **House style is canonical**: Follow the Programming Style section above for type-safe,
|
||||
fail-fast code; do not hide unclear models with speculative attributes, broad exception
|
||||
handling, casts, or unapproved fallback logic
|
||||
- **No guessing**: Do not say "The issue is..." before you actually know what the issue is. Investigate first.
|
||||
|
||||
### Literate Programming Style
|
||||
@@ -348,6 +369,13 @@ See `.claude/commands/release/release.md` (and `beta.md`, `release-check.md`, `c
|
||||
- Manage snapshots: `basic-memory cloud snapshot [create|list|delete|show|browse]`
|
||||
- Restore from snapshot: `basic-memory cloud restore <path> --snapshot <id>`
|
||||
|
||||
**Cloud Sync Commands (Personal and Team workspaces):**
|
||||
- Fetch cloud changes (cloud -> local): `basic-memory cloud pull --name "name"` (Team-safe; additive, never deletes local)
|
||||
- Upload local changes (local -> cloud): `basic-memory cloud push --name "name"` (Team-safe; additive, never deletes cloud)
|
||||
- Resolve conflicts on push/pull: `--on-conflict [fail|keep-local|keep-cloud|keep-both]` (default `fail` lists conflicts and aborts, git-style)
|
||||
- One-way mirror (local -> cloud): `basic-memory cloud sync --name "name"` (Personal workspaces only; deletes cloud files missing locally)
|
||||
- Two-way mirror (local <-> cloud): `basic-memory cloud bisync --name "name"` (Personal workspaces only)
|
||||
|
||||
### MCP Capabilities
|
||||
|
||||
- Basic Memory exposes these MCP tools to LLMs:
|
||||
|
||||
@@ -216,14 +216,14 @@ Source: [`plugins/claude-code`](plugins/claude-code).
|
||||
### Shared skills
|
||||
|
||||
Framework-agnostic `SKILL.md` files live in [`skills/`](skills). If your
|
||||
Skills CLI supports subpath installs:
|
||||
Skills CLI supports repository subdirectory sources:
|
||||
|
||||
```bash
|
||||
npx skills add basicmachines-co/basic-memory --path skills
|
||||
npx skills add basicmachines-co/basic-memory/skills
|
||||
```
|
||||
|
||||
If it does not, copy the `memory-*` directories from `skills/` into your
|
||||
agent's skills directory as a temporary Phase 1 install path.
|
||||
If your installed Skills CLI cannot load that source, update the CLI or copy
|
||||
the `memory-*` directories from `skills/` into your agent's skills directory.
|
||||
|
||||
### Hermes
|
||||
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
# Basic Memory Engineering Style
|
||||
|
||||
Style is how we make code easier to verify. Prefer explicit, typed, local-first code that
|
||||
preserves the file system as the source of truth while keeping the database, API, and MCP
|
||||
surfaces in sync.
|
||||
|
||||
## Design Center
|
||||
|
||||
- Basic Memory is local-first. Markdown files are the durable source; SQLite/Postgres indexes
|
||||
are derived state that should be rebuilt or reconciled from files when needed.
|
||||
- Keep the existing boundary order: CLI/MCP/API entrypoints compose dependencies, services own
|
||||
business behavior, repositories own database access, and file services own filesystem writes.
|
||||
- MCP tools should remain atomic and composable. They should call API routers through typed MCP
|
||||
clients, not reach around into services.
|
||||
- Prefer small, explicit abstractions that match a real domain boundary. Avoid object
|
||||
hierarchies when a function, dataclass, type alias, or protocol describes the concept better.
|
||||
|
||||
## Types And Data
|
||||
|
||||
- Use full type annotations and Python 3.12 syntax. Introduce `type` aliases for repeated
|
||||
structured shapes, callback signatures, or domain concepts that would otherwise become
|
||||
anonymous `dict[str, Any]` values.
|
||||
- Use dataclasses for internal values, operation inputs, and service results. Prefer
|
||||
`frozen=True` when the value should not change and `slots=True` when identity/dynamic
|
||||
attributes are not needed.
|
||||
- Use Pydantic v2 at boundaries that validate, serialize, or deserialize data: API payloads,
|
||||
CLI/MCP schemas, configuration, and persistence-adjacent schemas.
|
||||
- Use narrow `Protocol`s when a caller needs a capability rather than a concrete repository or
|
||||
service. Keep protocols small enough that fake implementations in tests are obvious.
|
||||
- Avoid speculative `getattr`, broad casts, or `Any` as a way to paper over uncertainty. Read
|
||||
the model or schema definition and make the type relationship explicit.
|
||||
|
||||
## Control Flow And Resources
|
||||
|
||||
- Fail fast when an invariant is broken. Do not swallow exceptions, add warning-only error
|
||||
handling, or introduce fallback behavior unless the user explicitly agrees to that behavior.
|
||||
- Keep control flow simple and close to the domain decision. Push `if` statements up into the
|
||||
function that owns orchestration; keep leaf helpers focused on computation or one side effect.
|
||||
- Make async/resource boundaries visible with context managers and explicit lifecycles. Do not
|
||||
start background work without a clear owner, cancellation story, and verification path.
|
||||
- Keep file mutations centralized through the existing file utilities/services so checksum,
|
||||
atomic write, and index synchronization behavior stays coherent.
|
||||
|
||||
## Testing And Verification
|
||||
|
||||
- Use evidence-first testing, not mechanical TDD. For bugs and risky behavior, add or update a
|
||||
regression test that would catch the failure. For small documentation-only edits, use the
|
||||
relevant doc/repo hygiene checks.
|
||||
- Prefer tests that exercise real code paths. Use mocks, doubles, or `monkeypatch` only when
|
||||
the external boundary would be slow, nondeterministic, or impossible to trigger directly.
|
||||
- Keep coverage at 100% for new code. Use `# pragma: no cover` only for code that would require
|
||||
disproportionate mocking and is covered through an integration or runtime path.
|
||||
- Start with targeted commands, then widen as risk grows: focused pytest, `just fast-check`,
|
||||
`just doctor`, package checks for agent packaging changes, and full SQLite/Postgres gates
|
||||
when behavior crosses shared boundaries.
|
||||
|
||||
## Comments And Names
|
||||
|
||||
- Name values after the domain concept they carry: project, entity, permalink, tenant, route,
|
||||
checksum, observation, relation, batch, or index state.
|
||||
- Comments should say why a branch, invariant, retry, lifecycle, or compatibility constraint
|
||||
exists. Section headers are useful when a function or file has clear phases.
|
||||
- Avoid comments that restate the code. If a comment cannot explain a decision, simplify the
|
||||
code or improve the name instead.
|
||||
+171
-53
@@ -8,9 +8,25 @@ The cloud CLI enables you to:
|
||||
- **Authenticate cloud access** - OAuth/API key credentials are stored locally for cloud operations
|
||||
- **Project-scoped sync** - Each project independently manages its sync configuration
|
||||
- **Explicit operations** - Sync only what you want, when you want
|
||||
- **Bidirectional sync** - Keep local and cloud in sync with rclone bisync
|
||||
- **Team-safe push/pull** - Additive, git-style transfers that work on shared Team workspaces
|
||||
- **Bidirectional sync** - Keep local and cloud in sync with rclone bisync (Personal workspaces)
|
||||
- **Offline access** - Work locally, sync when ready
|
||||
|
||||
### Personal vs Team workspaces
|
||||
|
||||
The transfer commands fall into two groups:
|
||||
|
||||
| Command | Direction | Behavior | Personal | Team |
|
||||
|---|---|---|---|---|
|
||||
| `bm cloud pull` | cloud → local | **additive** — never deletes local | ✅ | ✅ |
|
||||
| `bm cloud push` | local → cloud | **additive** — never deletes cloud | ✅ | ✅ |
|
||||
| `bm cloud sync` | local → cloud | **mirror** — deletes cloud files missing locally | ✅ | ❌ |
|
||||
| `bm cloud bisync` | local ↔ cloud | **mirror** — two-way, deletes on both sides | ✅ | ❌ |
|
||||
|
||||
`sync` and `bisync` are mirror operations: one local tree becomes authoritative and files missing on the other side get deleted. That is correct for a Personal workspace (one user, one source of truth) but unsafe on a shared Team bucket, where it could delete a teammate's files. On Team workspaces these commands exit early with a clear error and point you at `push`/`pull`.
|
||||
|
||||
`push` and `pull` are additive (they use `rclone copy`, which never deletes on the destination), so they are safe on both Personal and Team workspaces.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Before using Basic Memory Cloud, you need:
|
||||
@@ -55,8 +71,8 @@ bm project add work --cloud --local-path ~/work-notes
|
||||
bm project add temp --cloud # No local sync
|
||||
|
||||
# Now you can sync individually (after initial --resync):
|
||||
bm project bisync --name research
|
||||
bm project bisync --name work
|
||||
bm cloud bisync --name research
|
||||
bm cloud bisync --name work
|
||||
# temp stays cloud-only
|
||||
```
|
||||
|
||||
@@ -137,10 +153,10 @@ Establish the initial sync baseline. **Best practice:** Always preview with `--d
|
||||
|
||||
```bash
|
||||
# Step 1: Preview the initial sync (recommended)
|
||||
bm project bisync --name research --resync --dry-run
|
||||
bm cloud bisync --name research --resync --dry-run
|
||||
|
||||
# Step 2: If all looks good, run the actual sync
|
||||
bm project bisync --name research --resync
|
||||
bm cloud bisync --name research --resync
|
||||
```
|
||||
|
||||
**What happens under the covers:**
|
||||
@@ -167,7 +183,7 @@ This will effectively make both Path1 and Path2 filesystems contain a matching s
|
||||
After the first sync, just run bisync without `--resync`:
|
||||
|
||||
```bash
|
||||
bm project bisync --name research
|
||||
bm cloud bisync --name research
|
||||
```
|
||||
|
||||
**What happens:**
|
||||
@@ -235,7 +251,7 @@ bm project add research --cloud --local-path ~/Documents/research
|
||||
- Stores sync config in `~/.basic-memory/config.json`
|
||||
- Prepares for bisync (but doesn't sync yet)
|
||||
|
||||
**Result:** Project ready to sync. Run `bm project bisync --name research --resync` to establish baseline.
|
||||
**Result:** Project ready to sync. Run `bm cloud bisync --name research --resync` to establish baseline.
|
||||
|
||||
**Use case 3: Add sync to existing cloud project**
|
||||
|
||||
@@ -294,18 +310,98 @@ For MCP stdio, routing is always local.
|
||||
|
||||
### Understanding the Sync Commands
|
||||
|
||||
**There are three sync-related commands:**
|
||||
**There are five sync-related commands:**
|
||||
|
||||
1. `bm project sync` - One-way: local → cloud (make cloud match local)
|
||||
2. `bm project bisync` - Two-way: local ↔ cloud (recommended)
|
||||
3. `bm project check` - Verify files match (no changes)
|
||||
| Command | Direction | Workspace | Summary |
|
||||
|---|---|---|---|
|
||||
| `bm cloud pull` | cloud → local | Personal + Team | Fetch cloud changes, additively (git-style) |
|
||||
| `bm cloud push` | local → cloud | Personal + Team | Upload local changes, additively (git-style) |
|
||||
| `bm cloud sync` | local → cloud | Personal only | One-way mirror (cloud becomes identical to local) |
|
||||
| `bm cloud bisync` | local ↔ cloud | Personal only | Two-way mirror (recommended for solo use) |
|
||||
| `bm cloud check` | — | Personal + Team | Verify files match (no changes) |
|
||||
|
||||
### One-Way Sync: Local → Cloud
|
||||
If you collaborate on a shared Team workspace, use **`push`/`pull`** (see [Team Workspaces](#team-workspaces-push--pull-additive-git-style)). If you are the only writer (a Personal workspace), the mirror commands `sync`/`bisync` give you a single source of truth.
|
||||
|
||||
### Team Workspaces: push / pull (additive, git-style)
|
||||
|
||||
`push` and `pull` are the Team-safe transfer commands. They model `git push` / `git pull`:
|
||||
|
||||
- **`bm cloud pull`** fetches changes from the cloud into your local directory.
|
||||
- **`bm cloud push`** uploads your local changes to the cloud.
|
||||
|
||||
Both use `rclone copy`, so they are **additive — they never delete on the destination**. A conflict (a file that differs on both sides) is never resolved silently: by default the command aborts and lists the conflicting files, exactly like git refusing to clobber your changes.
|
||||
|
||||
#### Pull: fetch cloud changes
|
||||
|
||||
```bash
|
||||
# Preview first (recommended)
|
||||
bm cloud pull --name research --dry-run
|
||||
|
||||
# Fetch new/changed cloud files into local
|
||||
bm cloud pull --name research
|
||||
```
|
||||
|
||||
**What happens:**
|
||||
1. Compares cloud and local with `rclone check`
|
||||
2. Downloads files that are new or changed on the cloud
|
||||
3. Leaves your local-only files untouched (never deletes local)
|
||||
4. If any file differs on both sides, aborts and lists the conflicts (unless you pass `--on-conflict`)
|
||||
|
||||
#### Push: upload local changes
|
||||
|
||||
```bash
|
||||
bm cloud push --name research --dry-run
|
||||
bm cloud push --name research
|
||||
```
|
||||
|
||||
**What happens:**
|
||||
1. Compares local and cloud with `rclone check`
|
||||
2. Uploads files that are new or changed locally
|
||||
3. Leaves cloud-only files untouched (never deletes cloud)
|
||||
4. If any file differs on both sides, aborts and lists the conflicts — pull first, like a rejected `git push`
|
||||
|
||||
#### Resolving conflicts
|
||||
|
||||
When `push`/`pull` reports conflicts, re-run with `--on-conflict` to choose how differing files are handled. The value names exactly what survives, so it reads the same in both directions:
|
||||
|
||||
| `--on-conflict` | Behavior |
|
||||
|---|---|
|
||||
| `fail` *(default)* | List the conflicting files and exit without transferring anything |
|
||||
| `keep-cloud` | Take the cloud version (pull: overwrite local; push: skip those files) |
|
||||
| `keep-local` | Keep the local version (pull: skip those files; push: overwrite cloud) |
|
||||
| `keep-both` | Keep both — write the incoming version beside the existing one as `name.conflict-<date>.md` |
|
||||
|
||||
```bash
|
||||
# A teammate edited notes you also changed locally — pull reports a conflict:
|
||||
bm cloud pull --name research
|
||||
# pull aborted: 1 file(s) differ between local and cloud.
|
||||
# * notes/decisions.md
|
||||
# Re-run with one of: --on-conflict keep-cloud | keep-local | keep-both
|
||||
|
||||
# Take the cloud copy:
|
||||
bm cloud pull --name research --on-conflict keep-cloud
|
||||
|
||||
# Or keep both versions to merge by hand:
|
||||
bm cloud pull --name research --on-conflict keep-both
|
||||
```
|
||||
|
||||
#### Limitations
|
||||
|
||||
`push`/`pull` are deliberately simple, conflict-aware byte transfers — not a full reconciler. Without a sync baseline:
|
||||
|
||||
- **Deletions are not propagated.** A note deleted on one side is not removed from the other (we cannot tell an intentional delete from a file the other side never had). This is surfaced in the command output.
|
||||
- **Every divergence is treated as a conflict.** We cannot tell a teammate's edit from your stale copy, so any differing file prompts a decision rather than auto-resolving.
|
||||
|
||||
For conflict-aware *editing*, write through the MCP/API tools (which merge at the note level). A Team-safe bidirectional reconciler with a real baseline is tracked in [issue #862](https://github.com/basicmachines-co/basic-memory/issues/862).
|
||||
|
||||
### One-Way Sync: Local → Cloud (Personal only)
|
||||
|
||||
**Use case:** You made changes locally and want to push to cloud (overwrite cloud).
|
||||
|
||||
> **Personal workspaces only.** `sync` is a destructive mirror — it deletes cloud files that are not present locally. On a Team workspace it would delete a teammate's files, so it is blocked there. Use `bm cloud push` (additive) on Team workspaces.
|
||||
|
||||
```bash
|
||||
bm project sync --name research
|
||||
bm cloud sync --name research
|
||||
```
|
||||
|
||||
**What happens:**
|
||||
@@ -321,16 +417,18 @@ bm project sync --name research
|
||||
- You want to force cloud to match local
|
||||
- You don't care about cloud changes
|
||||
|
||||
### Two-Way Sync: Local ↔ Cloud (Recommended)
|
||||
### Two-Way Sync: Local ↔ Cloud (Personal only, recommended for solo use)
|
||||
|
||||
**Use case:** You edit files both locally and in cloud UI, want both to stay in sync.
|
||||
|
||||
> **Personal workspaces only.** `bisync` is a two-way mirror that can delete and overwrite on both sides. It is blocked on Team workspaces — use `bm cloud pull` then `bm cloud push` there. A Team-safe bidirectional reconciler is tracked separately ([issue #862](https://github.com/basicmachines-co/basic-memory/issues/862)).
|
||||
|
||||
```bash
|
||||
# First time - establish baseline
|
||||
bm project bisync --name research --resync
|
||||
bm cloud bisync --name research --resync
|
||||
|
||||
# Subsequent syncs
|
||||
bm project bisync --name research
|
||||
bm cloud bisync --name research
|
||||
```
|
||||
|
||||
**What happens:**
|
||||
@@ -349,7 +447,7 @@ echo "Local change" > ~/Documents/research/notes.md
|
||||
# Cloud now has: "Cloud change"
|
||||
|
||||
# Run bisync
|
||||
bm project bisync --name research
|
||||
bm cloud bisync --name research
|
||||
|
||||
# Result: Newer file wins (based on modification time)
|
||||
# If cloud was more recent, cloud version kept
|
||||
@@ -366,7 +464,7 @@ bm project bisync --name research
|
||||
**Use case:** Check if local and cloud match without making changes.
|
||||
|
||||
```bash
|
||||
bm project check --name research
|
||||
bm cloud check --name research
|
||||
```
|
||||
|
||||
**What happens:**
|
||||
@@ -378,7 +476,7 @@ bm project check --name research
|
||||
|
||||
```bash
|
||||
# One-way check (faster)
|
||||
bm project check --name research --one-way
|
||||
bm cloud check --name research --one-way
|
||||
```
|
||||
|
||||
### Preview Changes (Dry Run)
|
||||
@@ -386,7 +484,7 @@ bm project check --name research --one-way
|
||||
**Use case:** See what would change without actually syncing.
|
||||
|
||||
```bash
|
||||
bm project bisync --name research --dry-run
|
||||
bm cloud bisync --name research --dry-run
|
||||
```
|
||||
|
||||
**What happens:**
|
||||
@@ -432,20 +530,20 @@ bm project add work --cloud --local-path ~/work-notes
|
||||
bm project add personal --cloud --local-path ~/personal
|
||||
|
||||
# Establish baselines
|
||||
bm project bisync --name research --resync
|
||||
bm project bisync --name work --resync
|
||||
bm project bisync --name personal --resync
|
||||
bm cloud bisync --name research --resync
|
||||
bm cloud bisync --name work --resync
|
||||
bm cloud bisync --name personal --resync
|
||||
|
||||
# Daily workflow: sync everything
|
||||
bm project bisync --name research
|
||||
bm project bisync --name work
|
||||
bm project bisync --name personal
|
||||
bm cloud bisync --name research
|
||||
bm cloud bisync --name work
|
||||
bm cloud bisync --name personal
|
||||
```
|
||||
|
||||
**Future:** `--all` flag will sync all configured projects:
|
||||
|
||||
```bash
|
||||
bm project bisync --all # Coming soon
|
||||
bm cloud bisync --all # Coming soon
|
||||
```
|
||||
|
||||
### Mixed Usage
|
||||
@@ -462,8 +560,8 @@ bm project add archive --cloud
|
||||
bm project add temp-notes --cloud
|
||||
|
||||
# Sync only the configured ones
|
||||
bm project bisync --name research
|
||||
bm project bisync --name work
|
||||
bm cloud bisync --name research
|
||||
bm cloud bisync --name work
|
||||
|
||||
# Archive and temp-notes stay cloud-only
|
||||
```
|
||||
@@ -661,7 +759,7 @@ code ~/.basic-memory/.bmignore
|
||||
echo "*.tmp" >> ~/.basic-memory/.bmignore
|
||||
|
||||
# Next sync uses updated patterns
|
||||
bm project bisync --name research
|
||||
bm cloud bisync --name research
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
@@ -724,7 +822,7 @@ bm cloud login
|
||||
**Solution:**
|
||||
|
||||
```bash
|
||||
bm project bisync --name research --resync
|
||||
bm cloud bisync --name research --resync
|
||||
```
|
||||
|
||||
**What this does:**
|
||||
@@ -747,7 +845,7 @@ bm project bisync --name research --resync
|
||||
echo "# Research Notes" > ~/Documents/research/README.md
|
||||
|
||||
# Now run bisync
|
||||
bm project bisync --name research --resync
|
||||
bm cloud bisync --name research --resync
|
||||
```
|
||||
|
||||
**Why this happens:** Bisync creates listing files that track the state of each side. When both directories are completely empty, these listing files are considered invalid by rclone.
|
||||
@@ -764,10 +862,10 @@ bm project bisync --name research --resync
|
||||
|
||||
```bash
|
||||
# Clear bisync state
|
||||
bm project bisync-reset research
|
||||
bm cloud bisync-reset research
|
||||
|
||||
# Re-establish baseline
|
||||
bm project bisync --name research --resync
|
||||
bm cloud bisync --name research --resync
|
||||
```
|
||||
|
||||
**What this does:**
|
||||
@@ -787,16 +885,16 @@ bm project bisync --name research --resync
|
||||
|
||||
```bash
|
||||
# Check what would be deleted
|
||||
bm project bisync --name research --dry-run
|
||||
bm cloud bisync --name research --dry-run
|
||||
|
||||
# If correct, establish new baseline
|
||||
bm project bisync --name research --resync
|
||||
bm cloud bisync --name research --resync
|
||||
```
|
||||
|
||||
**Solution 2:** Use one-way sync if you know local is correct:
|
||||
|
||||
```bash
|
||||
bm project sync --name research
|
||||
bm cloud sync --name research
|
||||
```
|
||||
|
||||
### Project Not Configured for Sync
|
||||
@@ -809,7 +907,7 @@ bm project sync --name research
|
||||
|
||||
```bash
|
||||
bm cloud sync-setup research ~/Documents/research
|
||||
bm project bisync --name research --resync
|
||||
bm cloud bisync --name research --resync
|
||||
```
|
||||
|
||||
### Connection Issues
|
||||
@@ -880,20 +978,30 @@ bm project set-local <name> # Revert project to local mode
|
||||
### File Synchronization
|
||||
|
||||
```bash
|
||||
# One-way sync (local → cloud)
|
||||
bm project sync --name <project>
|
||||
bm project sync --name <project> --dry-run
|
||||
bm project sync --name <project> --verbose
|
||||
# Pull: fetch cloud changes (cloud → local) - Personal + Team, additive
|
||||
bm cloud pull --name <project>
|
||||
bm cloud pull --name <project> --dry-run
|
||||
bm cloud pull --name <project> --on-conflict [fail|keep-local|keep-cloud|keep-both]
|
||||
|
||||
# Two-way sync (local ↔ cloud) - Recommended
|
||||
bm project bisync --name <project> # After first --resync
|
||||
bm project bisync --name <project> --resync # First time / force baseline
|
||||
bm project bisync --name <project> --dry-run
|
||||
bm project bisync --name <project> --verbose
|
||||
# Push: upload local changes (local → cloud) - Personal + Team, additive
|
||||
bm cloud push --name <project>
|
||||
bm cloud push --name <project> --dry-run
|
||||
bm cloud push --name <project> --on-conflict [fail|keep-local|keep-cloud|keep-both]
|
||||
|
||||
# One-way mirror (local → cloud) - Personal workspaces only
|
||||
bm cloud sync --name <project>
|
||||
bm cloud sync --name <project> --dry-run
|
||||
bm cloud sync --name <project> --verbose
|
||||
|
||||
# Two-way mirror (local ↔ cloud) - Personal workspaces only
|
||||
bm cloud bisync --name <project> # After first --resync
|
||||
bm cloud bisync --name <project> --resync # First time / force baseline
|
||||
bm cloud bisync --name <project> --dry-run
|
||||
bm cloud bisync --name <project> --verbose
|
||||
|
||||
# Integrity check
|
||||
bm project check --name <project>
|
||||
bm project check --name <project> --one-way
|
||||
bm cloud check --name <project>
|
||||
bm cloud check --name <project> --one-way
|
||||
|
||||
# List project files by route
|
||||
bm project ls --name <project> # Default target: local
|
||||
@@ -909,15 +1017,25 @@ bm project ls --name <project> --cloud --path <subpath>
|
||||
1. **Authenticate cloud access** - `bm cloud login`
|
||||
2. **Install rclone** - `bm cloud setup`
|
||||
3. **Add projects with sync** - `bm project add research --cloud --local-path ~/Documents/research`
|
||||
4. **Preview first sync** - `bm project bisync --name research --resync --dry-run`
|
||||
5. **Establish baseline** - `bm project bisync --name research --resync`
|
||||
6. **Daily workflow** - `bm project bisync --name research`
|
||||
|
||||
**Personal workspace (solo, mirror) workflow:**
|
||||
|
||||
4. **Preview first sync** - `bm cloud bisync --name research --resync --dry-run`
|
||||
5. **Establish baseline** - `bm cloud bisync --name research --resync`
|
||||
6. **Daily workflow** - `bm cloud bisync --name research`
|
||||
|
||||
**Team workspace (shared, additive) workflow:**
|
||||
|
||||
4. **Fetch teammates' changes** - `bm cloud pull --name research`
|
||||
5. **Upload your changes** - `bm cloud push --name research`
|
||||
6. **Resolve conflicts explicitly** - re-run with `--on-conflict keep-cloud|keep-local|keep-both`
|
||||
|
||||
**Key benefits:**
|
||||
- ✅ Each project independently syncs (or doesn't)
|
||||
- ✅ Projects can live anywhere on disk
|
||||
- ✅ Explicit sync operations (no magic)
|
||||
- ✅ Safe by design (max delete limits, conflict resolution)
|
||||
- ✅ Team-safe push/pull that never delete on the destination
|
||||
- ✅ Safe by design (max delete limits, conflict resolution, git-style conflict aborts)
|
||||
- ✅ Full offline access (work locally, sync when ready)
|
||||
|
||||
**Future enhancements:**
|
||||
|
||||
@@ -0,0 +1,300 @@
|
||||
# LiteLLM Provider
|
||||
|
||||
Basic Memory can use the LiteLLM SDK for semantic search embeddings. This lets you
|
||||
keep Basic Memory's vector indexing and search behavior while routing embedding calls
|
||||
to OpenAI-compatible and provider-specific backends such as OpenAI, Azure OpenAI,
|
||||
Cohere, Bedrock, NVIDIA NIM, and other LiteLLM-supported embedding providers.
|
||||
|
||||
Use this page when you want to try a non-default embedding model, validate a provider,
|
||||
or tune LiteLLM-specific settings.
|
||||
|
||||
> **Experimental — advanced users only.** The LiteLLM provider is experimental and
|
||||
> intended for users who are comfortable operating remote embedding backends. It makes
|
||||
> paid, networked API calls, requires per-model dimension and input-role configuration,
|
||||
> and reindexing a real corpus can be slow and spend provider quota (see
|
||||
> [Reindexing with a remote provider](#reindexing-with-a-remote-provider)). For most
|
||||
> users, the default local **FastEmbed** provider is the recommended choice. Use LiteLLM
|
||||
> only if you know what you're doing.
|
||||
|
||||
## Quick Start
|
||||
|
||||
The default LiteLLM model is OpenAI `text-embedding-3-small` through the LiteLLM
|
||||
model string `openai/text-embedding-3-small`.
|
||||
|
||||
```bash
|
||||
export BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED=true
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER=litellm
|
||||
export OPENAI_API_KEY=sk-...
|
||||
|
||||
bm reindex --embeddings
|
||||
```
|
||||
|
||||
Then use vector or hybrid search:
|
||||
|
||||
```python
|
||||
search_notes("login token flow", search_type="hybrid")
|
||||
```
|
||||
|
||||
## Basic Memory Options
|
||||
|
||||
All options can be set in config or as environment variables.
|
||||
|
||||
| Config Field | Env Var | Default | Notes |
|
||||
|---|---|---|---|
|
||||
| `semantic_search_enabled` | `BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED` | Auto | Set to `true` to force vector/hybrid support on. |
|
||||
| `semantic_embedding_provider` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER` | `fastembed` | Set to `litellm` for the LiteLLM provider. |
|
||||
| `semantic_embedding_model` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_MODEL` | `bge-small-en-v1.5` | With `litellm`, the default is remapped to `openai/text-embedding-3-small`. |
|
||||
| `semantic_embedding_dimensions` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_DIMENSIONS` | Provider default | Required for non-default LiteLLM models because vector tables are dimensioned before the first API call. |
|
||||
| `semantic_embedding_forward_dimensions` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_FORWARD_DIMENSIONS` | Auto | Sends `dimensions` to LiteLLM only when supported. Auto is enabled for `text-embedding-3` model strings. |
|
||||
| `semantic_embedding_document_input_type` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_DOCUMENT_INPUT_TYPE` | Auto | LiteLLM `input_type` for indexed notes/passages. |
|
||||
| `semantic_embedding_query_input_type` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_QUERY_INPUT_TYPE` | Auto | LiteLLM `input_type` for search queries. |
|
||||
| `semantic_embedding_batch_size` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_BATCH_SIZE` | `2` | Number of text chunks per provider request. |
|
||||
| `semantic_embedding_request_concurrency` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_REQUEST_CONCURRENCY` | `4` | Maximum concurrent LiteLLM embedding requests. |
|
||||
| `semantic_embedding_sync_batch_size` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_SYNC_BATCH_SIZE` | `2` | Number of prepared vector jobs flushed through the sync pipeline together. |
|
||||
|
||||
## Dimensions
|
||||
|
||||
Basic Memory needs the vector dimension before it can create SQLite or Postgres
|
||||
vector tables. The OpenAI default is known, so this works without an explicit
|
||||
dimension:
|
||||
|
||||
```bash
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER=litellm
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_MODEL=openai/text-embedding-3-small
|
||||
```
|
||||
|
||||
For every other LiteLLM model, set the dimension explicitly:
|
||||
|
||||
```bash
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER=litellm
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_MODEL=cohere/embed-english-v3.0
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_DIMENSIONS=1024
|
||||
```
|
||||
|
||||
For fixed-size models, `semantic_embedding_dimensions` is Basic Memory's local
|
||||
schema and validation size. For OpenAI/Azure `text-embedding-3` models, LiteLLM
|
||||
can also forward `dimensions` as a provider-side reduced-output request. Basic
|
||||
Memory enables that automatically when the model string contains `text-embedding-3`.
|
||||
|
||||
If you use an Azure deployment alias such as `azure/<deployment-name>`, the model
|
||||
string may not reveal that the underlying model supports reduced output dimensions.
|
||||
Set this only when your deployment supports it:
|
||||
|
||||
```bash
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_FORWARD_DIMENSIONS=true
|
||||
```
|
||||
|
||||
## Asymmetric Models
|
||||
|
||||
Some embedding models use different request roles for indexed documents and
|
||||
search queries. Basic Memory automatically sets these for known LiteLLM families:
|
||||
|
||||
| Model Family | Document `input_type` | Query `input_type` |
|
||||
|---|---|---|
|
||||
| Cohere v3 embeddings | `search_document` | `search_query` |
|
||||
| NVIDIA NIM retrieval embeddings | `passage` | `query` |
|
||||
|
||||
For any other asymmetric model, configure both roles explicitly:
|
||||
|
||||
```bash
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_DOCUMENT_INPUT_TYPE=passage
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_QUERY_INPUT_TYPE=query
|
||||
```
|
||||
|
||||
Changing provider, model, dimensions, dimension-forwarding, or document/query
|
||||
roles changes the meaning of stored vectors. Rebuild embeddings after any of
|
||||
those changes:
|
||||
|
||||
```bash
|
||||
bm reindex --embeddings
|
||||
```
|
||||
|
||||
## Reindexing with a remote provider
|
||||
|
||||
Embedding a real corpus through a network API is far slower than local FastEmbed, and
|
||||
the defaults are tuned for the local case. Two things to know before you run a full
|
||||
reindex.
|
||||
|
||||
**Raise the sync batch size.** `semantic_embedding_sync_batch_size` defaults to `2`, and
|
||||
it — not `semantic_embedding_batch_size` — governs throughput on the sync pipeline. With
|
||||
the default, a full reindex can take tens of seconds *per note* against a remote provider.
|
||||
Raising both to a larger value turns a multi-minute (or longer) reindex into well under a
|
||||
minute for the same corpus:
|
||||
|
||||
```bash
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_SYNC_BATCH_SIZE=32
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_BATCH_SIZE=64
|
||||
```
|
||||
|
||||
Stay within the provider's per-request size and rate limits — Cohere v3, for example,
|
||||
accepts up to 96 inputs per embedding request.
|
||||
|
||||
**Changing dimensions requires recreating the vector table.** Basic Memory dimensions the
|
||||
vector table on first index and refuses to mix sizes. Switching to a model with a
|
||||
different dimension (for example FastEmbed 384 → OpenAI 1536 → Cohere 1024) makes a plain
|
||||
`bm reindex` raise an `Embedding dimension mismatch` error. Recreate the table with a full
|
||||
rebuild — files are the source of truth, so this re-indexes from disk and re-embeds
|
||||
everything:
|
||||
|
||||
```bash
|
||||
bm reset --reindex
|
||||
```
|
||||
|
||||
To trial a provider without disturbing your existing index, point Basic Memory at a
|
||||
throwaway config + database instead:
|
||||
|
||||
```bash
|
||||
export BASIC_MEMORY_CONFIG_DIR=/tmp/bm-litellm-trial
|
||||
```
|
||||
|
||||
## Provider Setup Examples
|
||||
|
||||
LiteLLM reads provider credentials from the environment. These are the examples
|
||||
covered by Basic Memory's live validation harness.
|
||||
|
||||
### OpenAI Through LiteLLM
|
||||
|
||||
```bash
|
||||
export OPENAI_API_KEY=sk-...
|
||||
export BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED=true
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER=litellm
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_MODEL=openai/text-embedding-3-small
|
||||
```
|
||||
|
||||
### Cohere v3
|
||||
|
||||
```bash
|
||||
export COHERE_API_KEY=...
|
||||
export BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED=true
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER=litellm
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_MODEL=cohere/embed-english-v3.0
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_DIMENSIONS=1024
|
||||
```
|
||||
|
||||
The provider auto-selects `search_document` for indexed chunks and `search_query`
|
||||
for search queries.
|
||||
|
||||
### Azure OpenAI
|
||||
|
||||
```bash
|
||||
export AZURE_API_KEY=...
|
||||
export AZURE_API_BASE=https://<resource-name>.openai.azure.com
|
||||
export AZURE_API_VERSION=2024-02-01
|
||||
|
||||
export BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED=true
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER=litellm
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_MODEL=azure/<deployment-name>
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_DIMENSIONS=1536
|
||||
```
|
||||
|
||||
If your Azure deployment is a reduced-dimension `text-embedding-3` deployment,
|
||||
set the dimension you want and enable forwarding:
|
||||
|
||||
```bash
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_DIMENSIONS=512
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_FORWARD_DIMENSIONS=true
|
||||
```
|
||||
|
||||
### NVIDIA NIM
|
||||
|
||||
```bash
|
||||
export NVIDIA_NIM_API_KEY=...
|
||||
# Optional when using a custom or self-hosted NIM endpoint:
|
||||
export NVIDIA_NIM_API_BASE=https://integrate.api.nvidia.com/v1
|
||||
|
||||
export BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED=true
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER=litellm
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_MODEL=nvidia_nim/nvidia/embed-qa-4
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_DIMENSIONS=1024
|
||||
```
|
||||
|
||||
The provider auto-selects `passage` for indexed chunks and `query` for search
|
||||
queries.
|
||||
|
||||
## Testing LiteLLM Providers
|
||||
|
||||
Run the non-live LiteLLM unit and harness tests first:
|
||||
|
||||
```bash
|
||||
uv run pytest tests/repository/test_litellm_provider.py \
|
||||
test-int/semantic/test_litellm_live_harness.py -q
|
||||
```
|
||||
|
||||
Run the SQLite and Postgres vector identity regressions when changing model
|
||||
identity, role, or vector sync behavior:
|
||||
|
||||
```bash
|
||||
uv run pytest \
|
||||
tests/repository/test_sqlite_vector_search_repository.py::test_sqlite_embedding_model_key_includes_litellm_role_settings \
|
||||
-q
|
||||
|
||||
BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest \
|
||||
tests/repository/test_postgres_search_repository.py::test_postgres_litellm_role_change_reembeds_existing_chunks \
|
||||
-q
|
||||
```
|
||||
|
||||
The Postgres command uses testcontainers, so Docker must be running.
|
||||
|
||||
## Live Provider Harness
|
||||
|
||||
The live harness makes real LiteLLM API calls and spends provider quota. It is
|
||||
opt-in by design:
|
||||
|
||||
```bash
|
||||
export OPENAI_API_KEY=sk-...
|
||||
export COHERE_API_KEY=...
|
||||
|
||||
just test-litellm-live
|
||||
```
|
||||
|
||||
Built-in cases run when their API keys are present:
|
||||
|
||||
| Case | Required Env Var | Validates |
|
||||
|---|---|---|
|
||||
| `openai-text-embedding-3-small` | `OPENAI_API_KEY` | OpenAI via LiteLLM, 1536 dimensions, normalized vectors, ranking sanity. |
|
||||
| `cohere-embed-english-v3` | `COHERE_API_KEY` | Cohere v3 role handling, 1024 dimensions, normalized vectors, ranking sanity. |
|
||||
|
||||
Add provider aliases or new backends with a custom cases file:
|
||||
|
||||
```bash
|
||||
cat > /tmp/litellm-cases.json <<'JSON'
|
||||
[
|
||||
{
|
||||
"name": "azure-text-embedding-3-small-512",
|
||||
"model": "azure/<deployment-name>",
|
||||
"dimensions": 512,
|
||||
"api_key_env": "AZURE_API_KEY",
|
||||
"forward_dimensions": true
|
||||
},
|
||||
{
|
||||
"name": "nvidia-embed-qa-4",
|
||||
"model": "nvidia_nim/nvidia/embed-qa-4",
|
||||
"dimensions": 1024,
|
||||
"api_key_env": "NVIDIA_NIM_API_KEY",
|
||||
"document_input_type": "passage",
|
||||
"query_input_type": "query"
|
||||
}
|
||||
]
|
||||
JSON
|
||||
|
||||
just test-litellm-live --cases-file /tmp/litellm-cases.json
|
||||
```
|
||||
|
||||
For CI-style output:
|
||||
|
||||
```bash
|
||||
just test-litellm-live --cases-file /tmp/litellm-cases.json --json
|
||||
```
|
||||
|
||||
The harness embeds two documents and one query, validates dimension and vector
|
||||
normalization, checks that the authentication query ranks the authentication
|
||||
document above a distractor, and reports latency plus role/dimension settings.
|
||||
|
||||
## Provider Reference
|
||||
|
||||
LiteLLM's own provider and embedding docs are the source of truth for current
|
||||
model strings and credential names:
|
||||
|
||||
- [LiteLLM embedding models](https://docs.litellm.ai/docs/embedding/supported_embedding)
|
||||
- [LiteLLM Azure OpenAI provider](https://docs.litellm.ai/docs/providers/azure)
|
||||
- [LiteLLM NVIDIA NIM provider](https://docs.litellm.ai/docs/providers/nvidia_nim)
|
||||
+116
-5
@@ -99,10 +99,13 @@ All settings are fields on `BasicMemoryConfig` and can be set via environment va
|
||||
| Config Field | Env Var | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `semantic_search_enabled` | `BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED` | Auto (`true` when semantic deps are available) | Enable semantic search. Required before vector/hybrid modes work. |
|
||||
| `semantic_embedding_provider` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER` | `"fastembed"` | Embedding provider: `"fastembed"` (local) or `"openai"` (API). |
|
||||
| `semantic_embedding_provider` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER` | `"fastembed"` | Embedding provider: `"fastembed"` (local), `"openai"` (API), or `"litellm"` (multi-provider API, **experimental** — advanced users only). |
|
||||
| `semantic_embedding_model` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_MODEL` | `"bge-small-en-v1.5"` | Model identifier. Auto-adjusted per provider if left at default. |
|
||||
| `semantic_embedding_dimensions` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_DIMENSIONS` | Auto-detected | Vector dimensions. 384 for FastEmbed, 1536 for OpenAI. Override only if using a non-default model. |
|
||||
| `semantic_embedding_batch_size` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_BATCH_SIZE` | `64` | Number of texts to embed per batch. |
|
||||
| `semantic_embedding_dimensions` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_DIMENSIONS` | Provider default | Vector dimensions. 384 for FastEmbed, 1536 for OpenAI/LiteLLM OpenAI. Required when using a non-default LiteLLM model. |
|
||||
| `semantic_embedding_forward_dimensions` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_FORWARD_DIMENSIONS` | Auto | LiteLLM-only override for whether configured dimensions are sent as a provider-side output-size request. |
|
||||
| `semantic_embedding_batch_size` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_BATCH_SIZE` | `2` | Number of texts to embed per batch. |
|
||||
| `semantic_embedding_document_input_type` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_DOCUMENT_INPUT_TYPE` | Auto for known LiteLLM models | Optional LiteLLM `input_type` for indexed document/passages. |
|
||||
| `semantic_embedding_query_input_type` | `BASIC_MEMORY_SEMANTIC_EMBEDDING_QUERY_INPUT_TYPE` | Auto for known LiteLLM models | Optional LiteLLM `input_type` for search queries. |
|
||||
| `semantic_vector_k` | `BASIC_MEMORY_SEMANTIC_VECTOR_K` | `100` | Candidate count for vector nearest-neighbour retrieval. Higher values improve recall at the cost of latency. |
|
||||
|
||||
## Embedding Providers
|
||||
@@ -135,7 +138,114 @@ export BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER=openai
|
||||
export OPENAI_API_KEY=sk-...
|
||||
```
|
||||
|
||||
When switching from FastEmbed to OpenAI (or vice versa), you must rebuild embeddings since the vector dimensions differ:
|
||||
### LiteLLM
|
||||
|
||||
> **Experimental — advanced users only.** The LiteLLM provider is experimental and aimed at users comfortable operating remote embedding backends: paid API calls, per-model dimension and input-role configuration, and slower reindexing of large corpora. For most users, FastEmbed (local, default) is recommended. See [LiteLLM Provider](litellm-provider.md) for the caveats and tuning.
|
||||
|
||||
Uses the LiteLLM SDK to call embedding models from providers such as OpenAI, Cohere, Azure, Bedrock, NVIDIA NIM, and other LiteLLM-supported backends. Requires the provider's API credentials.
|
||||
For the full option reference, provider setup examples, and live validation harness, see [LiteLLM Provider](litellm-provider.md).
|
||||
|
||||
```bash
|
||||
export BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED=true
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_PROVIDER=litellm
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_MODEL=cohere/embed-english-v3.0
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_DIMENSIONS=1024
|
||||
export COHERE_API_KEY=...
|
||||
```
|
||||
|
||||
Basic Memory creates vector tables before the first embedding call, so non-default LiteLLM models must set `BASIC_MEMORY_SEMANTIC_EMBEDDING_DIMENSIONS`. The LiteLLM OpenAI default (`openai/text-embedding-3-small`) uses 1536 dimensions automatically.
|
||||
|
||||
For fixed-size LiteLLM models, dimensions are used as Basic Memory's local vector schema and
|
||||
validation size. Basic Memory automatically sends dimensions as a provider-side output-size
|
||||
request for `text-embedding-3` model strings, where LiteLLM/OpenAI support reduced output
|
||||
dimensions. If an Azure/OpenAI deployment uses an arbitrary LiteLLM model string such as
|
||||
`azure/<deployment-name>` and the underlying model supports reduced dimensions, set
|
||||
`BASIC_MEMORY_SEMANTIC_EMBEDDING_FORWARD_DIMENSIONS=true`.
|
||||
|
||||
Some retrieval models are asymmetric: indexed passages and search queries must be embedded with different provider parameters. Basic Memory automatically sets LiteLLM `input_type` for known asymmetric model families:
|
||||
|
||||
- Cohere v3: documents use `search_document`, queries use `search_query`
|
||||
- NVIDIA NIM retrieval models: documents use `passage`, queries use `query`
|
||||
|
||||
For other asymmetric LiteLLM models, set the input types explicitly:
|
||||
|
||||
```bash
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_DOCUMENT_INPUT_TYPE=passage
|
||||
export BASIC_MEMORY_SEMANTIC_EMBEDDING_QUERY_INPUT_TYPE=query
|
||||
```
|
||||
|
||||
#### Live LiteLLM Validation
|
||||
|
||||
Provider APIs differ in subtle ways: some accept `dimensions`, some require separate
|
||||
document/query roles, and some route through deployment aliases that do not reveal the
|
||||
underlying model name. Before adding or changing LiteLLM model support, run the opt-in live
|
||||
evaluation harness:
|
||||
|
||||
```bash
|
||||
export OPENAI_API_KEY=sk-...
|
||||
export COHERE_API_KEY=...
|
||||
just test-litellm-live
|
||||
```
|
||||
|
||||
The built-in live cases cover:
|
||||
|
||||
| Case | Required key | What it validates |
|
||||
|---|---|---|
|
||||
| `openai/text-embedding-3-small` | `OPENAI_API_KEY` | Standard LiteLLM OpenAI embedding calls and normalized 1536-dimensional output. |
|
||||
| `cohere/embed-english-v3.0` | `COHERE_API_KEY` | Cohere v3 asymmetric `search_document` / `search_query` handling and fixed 1024-dimensional output. |
|
||||
|
||||
The harness embeds two documents and one query, checks vector dimensions and normalization,
|
||||
then verifies the authentication query ranks the authentication document above the distractor.
|
||||
It prints a table with per-model scores, norms, latency, role settings, and dimension-forwarding
|
||||
mode.
|
||||
|
||||
To validate provider aliases or additional LiteLLM backends, save custom JSON cases:
|
||||
|
||||
```bash
|
||||
export AZURE_API_KEY=...
|
||||
export AZURE_API_BASE=https://example.openai.azure.com
|
||||
export AZURE_API_VERSION=2024-02-01
|
||||
|
||||
cat > /tmp/litellm-azure-cases.json <<'JSON'
|
||||
[
|
||||
{
|
||||
"name": "azure-text-embedding-3-small-512",
|
||||
"model": "azure/<deployment-name>",
|
||||
"dimensions": 512,
|
||||
"api_key_env": "AZURE_API_KEY",
|
||||
"forward_dimensions": true
|
||||
}
|
||||
]
|
||||
JSON
|
||||
|
||||
just test-litellm-live --cases-file /tmp/litellm-azure-cases.json
|
||||
```
|
||||
|
||||
NVIDIA NIM retrieval models can be checked the same way:
|
||||
|
||||
```bash
|
||||
export NVIDIA_NIM_API_KEY=...
|
||||
|
||||
cat > /tmp/litellm-nvidia-cases.json <<'JSON'
|
||||
[
|
||||
{
|
||||
"name": "nvidia-embed-qa-4",
|
||||
"model": "nvidia_nim/nvidia/embed-qa-4",
|
||||
"dimensions": 1024,
|
||||
"api_key_env": "NVIDIA_NIM_API_KEY",
|
||||
"document_input_type": "passage",
|
||||
"query_input_type": "query"
|
||||
}
|
||||
]
|
||||
JSON
|
||||
|
||||
just test-litellm-live --cases-file /tmp/litellm-nvidia-cases.json
|
||||
```
|
||||
|
||||
For repeatable local runs, put the same JSON array in a file and pass
|
||||
`just test-litellm-live --cases-file path/to/litellm-cases.json`.
|
||||
|
||||
When switching providers, models, dimensions, or LiteLLM document/query input types, rebuild embeddings:
|
||||
|
||||
```bash
|
||||
bm reindex --embeddings
|
||||
@@ -203,9 +313,10 @@ bm reindex -p my-project
|
||||
|
||||
- **Upgrade note**: Migration now performs a one-time automatic embedding backfill on upgrade.
|
||||
- **Manual enable case**: If you explicitly had `semantic_search_enabled=false` and then turn it on
|
||||
- **Provider change**: After switching between `fastembed` and `openai`
|
||||
- **Provider change**: After switching between `fastembed`, `openai`, and `litellm`
|
||||
- **Model change**: After changing `semantic_embedding_model`
|
||||
- **Dimension change**: After changing `semantic_embedding_dimensions`
|
||||
- **LiteLLM role change**: After changing `semantic_embedding_document_input_type` or `semantic_embedding_query_input_type`
|
||||
|
||||
The reindex command shows progress with embedded/skipped/error counts:
|
||||
|
||||
|
||||
@@ -150,7 +150,7 @@ This plugin ships with workflow-oriented skills that are automatically loaded wh
|
||||
No manual installation needed. To update skills or install new ones as they become available:
|
||||
|
||||
```bash
|
||||
npx skills add basicmachines-co/basic-memory --path skills --agent openclaw
|
||||
npx skills add basicmachines-co/basic-memory/skills --agent openclaw
|
||||
```
|
||||
|
||||
See the canonical source at [`basic-memory/skills`](../../skills).
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
# Basic Memory - Modern Command Runner
|
||||
|
||||
TESTMON_FLAGS := env_var_or_default("BASIC_MEMORY_TESTMON_FLAGS", "--testmon-noselect")
|
||||
TESTMON_SELECT_FLAGS := env_var_or_default("BASIC_MEMORY_TESTMON_SELECT_FLAGS", "--testmon --testmon-forceselect")
|
||||
TESTMON_REFRESH_FLAGS := env_var_or_default("BASIC_MEMORY_TESTMON_REFRESH_FLAGS", "--testmon-noselect")
|
||||
|
||||
# Install dependencies
|
||||
install:
|
||||
uv sync
|
||||
@@ -35,40 +39,56 @@ test-sqlite: test-unit-sqlite test-int-sqlite
|
||||
test-postgres: test-unit-postgres test-int-postgres
|
||||
|
||||
# Run unit tests against SQLite
|
||||
test-unit-sqlite:
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov tests
|
||||
test-unit-sqlite: testmon-seed
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov {{TESTMON_FLAGS}} --testmon-env=unit-sqlite tests
|
||||
|
||||
# Run unit tests against Postgres
|
||||
test-unit-postgres:
|
||||
BASIC_MEMORY_ENV=test BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov tests
|
||||
test-unit-postgres: testmon-seed
|
||||
BASIC_MEMORY_ENV=test BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov {{TESTMON_FLAGS}} --testmon-env=unit-postgres tests
|
||||
|
||||
# Run integration tests against SQLite (excludes semantic benchmarks — use just test-semantic)
|
||||
test-int-sqlite:
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov -m "not semantic" test-int
|
||||
# Run integration tests against SQLite (excludes semantic tests and on-demand benchmarks —
|
||||
# use just test-semantic / run benchmark files explicitly)
|
||||
test-int-sqlite: testmon-seed
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov {{TESTMON_FLAGS}} --testmon-env=int-sqlite -m "not semantic and not benchmark" test-int
|
||||
|
||||
# Run integration tests against Postgres
|
||||
# Note: Uses timeout due to FastMCP Client + asyncpg cleanup hang (tests pass, process hangs on exit)
|
||||
# See: https://github.com/jlowin/fastmcp/issues/1311
|
||||
test-int-postgres:
|
||||
test-int-postgres: testmon-seed
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
# Use gtimeout (macOS/Homebrew) or timeout (Linux)
|
||||
TIMEOUT_CMD=$(command -v gtimeout || command -v timeout || echo "")
|
||||
if [[ -n "$TIMEOUT_CMD" ]]; then
|
||||
$TIMEOUT_CMD --signal=KILL 600 bash -c 'BASIC_MEMORY_ENV=test BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov -m "not semantic" test-int' || test $? -eq 137
|
||||
$TIMEOUT_CMD --signal=KILL 600 bash -c 'BASIC_MEMORY_ENV=test BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov {{TESTMON_FLAGS}} --testmon-env=int-postgres -m "not semantic and not benchmark" test-int' || test $? -eq 137
|
||||
else
|
||||
echo "⚠️ No timeout command found, running without timeout..."
|
||||
BASIC_MEMORY_ENV=test BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov -m "not semantic" test-int
|
||||
BASIC_MEMORY_ENV=test BASIC_MEMORY_TEST_POSTGRES=1 uv run pytest -p pytest_mock -v --no-cov {{TESTMON_FLAGS}} --testmon-env=int-postgres -m "not semantic and not benchmark" test-int
|
||||
fi
|
||||
|
||||
# Run tests impacted by recent changes (requires pytest-testmon)
|
||||
# Pass paths or node ids after `just testmon` to limit the candidate set further.
|
||||
testmon *args:
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov --testmon {{args}}
|
||||
testmon *args: testmon-seed
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov {{TESTMON_SELECT_FLAGS}} --testmon-env=local {{args}}
|
||||
|
||||
# Seed pytest-testmon data into this worktree from the shared Git cache.
|
||||
testmon-seed:
|
||||
uv run python scripts/testmon_cache.py seed
|
||||
|
||||
# Refresh the shared pytest-testmon cache from a full backend test run.
|
||||
testmon-refresh:
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
BASIC_MEMORY_TESTMON_FLAGS="{{TESTMON_REFRESH_FLAGS}}" just test
|
||||
uv run python scripts/testmon_cache.py refresh
|
||||
|
||||
# Show local and shared pytest-testmon cache locations.
|
||||
testmon-status:
|
||||
uv run python scripts/testmon_cache.py status
|
||||
|
||||
# Run MCP smoke test (fast end-to-end loop)
|
||||
test-smoke:
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov -m smoke test-int/mcp/test_smoke_integration.py
|
||||
test-smoke: testmon-seed
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov {{TESTMON_FLAGS}} --testmon-env=smoke -m smoke test-int/mcp/test_smoke_integration.py
|
||||
|
||||
# Fast local loop: lint, format, typecheck, impacted tests via pytest-testmon
|
||||
fast-check:
|
||||
@@ -97,27 +117,31 @@ postgres-migrate:
|
||||
# Run Windows-specific tests only (only works on Windows platform)
|
||||
# These tests verify Windows-specific database optimizations (locking mode, NullPool)
|
||||
# Will be skipped automatically on non-Windows platforms
|
||||
test-windows:
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov -m windows tests test-int
|
||||
test-windows: testmon-seed
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov {{TESTMON_FLAGS}} --testmon-env=windows -m windows tests test-int
|
||||
|
||||
# Run benchmark tests only (performance testing)
|
||||
# These are slow tests that measure sync performance with various file counts
|
||||
# Excluded from default test runs to keep CI fast
|
||||
test-benchmark:
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov -m benchmark tests test-int
|
||||
test-benchmark: testmon-seed
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov {{TESTMON_FLAGS}} --testmon-env=benchmark -m benchmark tests test-int
|
||||
|
||||
# Run semantic search quality benchmarks (all combos)
|
||||
test-semantic:
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov -m semantic test-int/semantic/
|
||||
test-semantic: testmon-seed
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov {{TESTMON_FLAGS}} --testmon-env=semantic -m semantic test-int/semantic/
|
||||
|
||||
# Run semantic benchmarks with JSON artifact output, then show report
|
||||
test-semantic-report:
|
||||
BASIC_MEMORY_ENV=test BASIC_MEMORY_BENCHMARK_OUTPUT=.benchmarks/semantic-quality.jsonl uv run pytest -p pytest_mock -v -s --no-cov -m semantic test-int/semantic/
|
||||
uv run python test-int/semantic/report.py .benchmarks/semantic-quality.jsonl
|
||||
|
||||
# Run opt-in live LiteLLM provider checks against configured external APIs
|
||||
test-litellm-live *args:
|
||||
BASIC_MEMORY_ENV=test BASIC_MEMORY_RUN_LITELLM_INTEGRATION=1 PYTHONPATH=test-int:src uv run python -m semantic.litellm_live_harness {{args}}
|
||||
|
||||
# Run semantic benchmarks (Postgres combos only)
|
||||
test-semantic-postgres:
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov -m semantic -k postgres test-int/semantic/
|
||||
test-semantic-postgres: testmon-seed
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov {{TESTMON_FLAGS}} --testmon-env=semantic-postgres -m semantic -k postgres test-int/semantic/
|
||||
|
||||
# View semantic benchmark results (rich formatted table)
|
||||
# Usage: just semantic-report [--filter-combo sqlite] [--filter-suite paraphrase] [--sort-by avg_latency_ms]
|
||||
@@ -133,8 +157,8 @@ benchmark-compare baseline candidate *args:
|
||||
|
||||
# Run all tests including Windows, Postgres, and Benchmarks (for CI/comprehensive testing)
|
||||
# Use this before releasing to ensure everything works across all backends and platforms
|
||||
test-all:
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov tests test-int
|
||||
test-all: testmon-seed
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov {{TESTMON_FLAGS}} --testmon-env=all tests test-int
|
||||
|
||||
# Generate HTML coverage report
|
||||
coverage:
|
||||
@@ -307,7 +331,7 @@ set-version version scope="all":
|
||||
set-version-dry-run version scope="all":
|
||||
python3 scripts/update_versions.py "{{version}}" --scope "{{scope}}" --dry-run
|
||||
|
||||
# Set the version for just the plugin/agent artifacts (plugin, marketplaces, Hermes, OpenClaw)
|
||||
# Set the version for just the plugin/agent artifacts (plugins, marketplaces, Hermes, OpenClaw)
|
||||
set-packages-version version:
|
||||
just set-version "{{version}}" packages
|
||||
|
||||
@@ -369,6 +393,7 @@ release version:
|
||||
.claude-plugin/marketplace.json \
|
||||
plugins/claude-code/.claude-plugin/plugin.json \
|
||||
plugins/claude-code/.claude-plugin/marketplace.json \
|
||||
plugins/codex/.codex-plugin/plugin.json \
|
||||
integrations/hermes/plugin.yaml \
|
||||
integrations/hermes/__init__.py \
|
||||
integrations/openclaw/package.json
|
||||
@@ -442,6 +467,7 @@ beta version:
|
||||
.claude-plugin/marketplace.json \
|
||||
plugins/claude-code/.claude-plugin/plugin.json \
|
||||
plugins/claude-code/.claude-plugin/marketplace.json \
|
||||
plugins/codex/.codex-plugin/plugin.json \
|
||||
integrations/hermes/plugin.yaml \
|
||||
integrations/hermes/__init__.py \
|
||||
integrations/openclaw/package.json
|
||||
|
||||
@@ -24,7 +24,7 @@ Memory's durable graph**, rather than a memory layer of its own. See
|
||||
configures the project for the plugin: maps it to a Basic Memory project (picking
|
||||
an existing one or creating a new one), seeds the `session`/`decision`/`task`
|
||||
schemas into the project, installs the shared `memory-*` skills via
|
||||
`npx skills add basicmachines-co/basic-memory --path skills` (the plugin doesn't
|
||||
`npx skills add basicmachines-co/basic-memory/skills` (the plugin doesn't
|
||||
vendor its own copies — `skills/` is the single source of truth, shared with
|
||||
OpenClaw), optionally learns the project's placement conventions, and enables the
|
||||
capture reflexes. Writes the `basicMemory` block to
|
||||
|
||||
@@ -156,7 +156,7 @@ git ls-files skills/ | grep -q memory- && echo "source repo - skip install"
|
||||
Otherwise, run from the project root:
|
||||
|
||||
```
|
||||
npx skills add basicmachines-co/basic-memory --path skills
|
||||
npx skills add basicmachines-co/basic-memory/skills
|
||||
```
|
||||
|
||||
This installs the canonical `memory-*` skills into the user's skills directory — the
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "codex",
|
||||
"version": "0.1.0+codex.20260604201213",
|
||||
"version": "0.21.6",
|
||||
"description": "A Codex-native bridge to Basic Memory for durable engineering context, decisions, and resumable checkpoints.",
|
||||
"author": {
|
||||
"name": "Basic Machines",
|
||||
|
||||
@@ -48,8 +48,13 @@ dependencies = [
|
||||
"fastembed>=0.7.4",
|
||||
"sqlite-vec>=0.1.6",
|
||||
"openai>=1.100.2",
|
||||
"litellm>=1.60.0,<2.0.0",
|
||||
"logfire>=4.19.0",
|
||||
"psutil>=5.9.0",
|
||||
# uvloop's C event loop has no self._ready.popleft() codepath, so the
|
||||
# asyncpg engine-dispose race ("IndexError: pop from an empty deque") that
|
||||
# crashes the Postgres backend cannot fire under it. Not available on Windows.
|
||||
"uvloop>=0.21.0; sys_platform != 'win32'",
|
||||
]
|
||||
|
||||
[project.urls]
|
||||
@@ -71,6 +76,10 @@ addopts = "--cov=basic_memory --cov-report term-missing"
|
||||
testpaths = ["tests", "test-int"]
|
||||
asyncio_mode = "strict"
|
||||
asyncio_default_fixture_loop_scope = "function"
|
||||
# Any test hanging >120s fails with a stack dump instead of stalling the CI job
|
||||
# until the runner times out (the FastMCP/asyncpg cleanup-hang family).
|
||||
timeout = 120
|
||||
timeout_method = "thread"
|
||||
filterwarnings = [
|
||||
"ignore:The @wait_container_is_ready decorator is deprecated.*:DeprecationWarning:testcontainers\\.core\\.waiting_utils",
|
||||
"ignore:The default datetime adapter is deprecated as of Python 3\\.12.*:DeprecationWarning:aiosqlite\\.core",
|
||||
@@ -84,6 +93,7 @@ markers = [
|
||||
"windows: Windows-specific tests (deselect with '-m \"not windows\"')",
|
||||
"smoke: Fast end-to-end smoke tests for MCP flows",
|
||||
"semantic: Tests requiring semantic dependencies (fastembed, sqlite-vec, openai)",
|
||||
"live: Tests that call external provider APIs and require explicit opt-in",
|
||||
]
|
||||
|
||||
[tool.ruff]
|
||||
@@ -109,6 +119,7 @@ dev = [
|
||||
"ty>=0.0.18",
|
||||
"cst-lsp>=0.1.3",
|
||||
"libcst>=1.8.6",
|
||||
"pytest-timeout>=2.4.0",
|
||||
]
|
||||
|
||||
[tool.hatch.version]
|
||||
|
||||
Executable
+534
@@ -0,0 +1,534 @@
|
||||
#!/usr/bin/env -S uv run --script
|
||||
# /// script
|
||||
# requires-python = ">=3.12"
|
||||
# dependencies = [
|
||||
# "typer>=0.9.0",
|
||||
# ]
|
||||
# ///
|
||||
"""BM Bossbot status and PR-body helpers.
|
||||
|
||||
BM Bossbot is a deterministic merge gate — no LLM review. It approves a head
|
||||
SHA only when the Tests workflow succeeded for it (enforced by the workflow
|
||||
trigger), the PR is not a draft, the author is trusted, and every review
|
||||
thread is resolved. Code review itself comes from the Codex connector and
|
||||
human reviewers; this gate just refuses to let unaddressed feedback merge.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Annotated, Any, Mapping
|
||||
|
||||
import typer
|
||||
|
||||
|
||||
STATUS_CONTEXT = "BM Bossbot Approval"
|
||||
SUMMARY_START = "<!-- BM_BOSSBOT_SUMMARY:start -->"
|
||||
SUMMARY_END = "<!-- BM_BOSSBOT_SUMMARY:end -->"
|
||||
APPROVED_DESCRIPTION = "BM Bossbot approved this head SHA"
|
||||
PENDING_DESCRIPTION = "BM Bossbot is reviewing this head SHA"
|
||||
app = typer.Typer(
|
||||
add_completion=False,
|
||||
help="Manage deterministic BM Bossbot PR approval statuses.",
|
||||
no_args_is_help=True,
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ApprovalResult:
|
||||
approved: bool
|
||||
state: str
|
||||
description: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PullRequestEvent:
|
||||
repo: str
|
||||
number: int
|
||||
head_sha: str
|
||||
body: str
|
||||
|
||||
|
||||
def read_json(path: Path) -> Any:
|
||||
try:
|
||||
return json.loads(path.read_text(encoding="utf-8"))
|
||||
except FileNotFoundError:
|
||||
raise SystemExit(f"Missing JSON file: {path}") from None
|
||||
except json.JSONDecodeError as exc:
|
||||
raise SystemExit(f"{path}: invalid JSON: {exc}") from None
|
||||
|
||||
|
||||
def pull_request_event(
|
||||
payload: Mapping[str, Any], repo_override: str | None = None
|
||||
) -> PullRequestEvent:
|
||||
pr = payload.get("pull_request")
|
||||
if not isinstance(pr, Mapping):
|
||||
raise SystemExit("GitHub event payload is missing pull_request")
|
||||
|
||||
repo = repo_override
|
||||
if repo is None:
|
||||
repository = payload.get("repository")
|
||||
if isinstance(repository, Mapping):
|
||||
repo = _string(repository.get("full_name"))
|
||||
if not repo:
|
||||
raise SystemExit("Could not determine GitHub repository")
|
||||
|
||||
number = pr.get("number")
|
||||
if not isinstance(number, int):
|
||||
raise SystemExit("GitHub event payload is missing pull_request.number")
|
||||
|
||||
head = pr.get("head")
|
||||
head_sha = (
|
||||
_string(head.get("sha")) if isinstance(head, Mapping) else _string(pr.get("head_sha"))
|
||||
)
|
||||
if not head_sha:
|
||||
raise SystemExit("GitHub event payload is missing pull_request.head.sha")
|
||||
|
||||
return PullRequestEvent(
|
||||
repo=repo,
|
||||
number=number,
|
||||
head_sha=head_sha,
|
||||
body=_string(pr.get("body")),
|
||||
)
|
||||
|
||||
|
||||
def count_unresolved_review_threads(*, token: str, repo: str, number: int) -> int:
|
||||
"""Count unresolved review threads (e.g. open Codex inline comments) on a PR.
|
||||
|
||||
Review threads are the canonical 'outstanding feedback' signal: bot reviewers
|
||||
submit COMMENTED reviews that never flip reviewDecision, so thread resolution
|
||||
is the only deterministic way to know feedback was addressed.
|
||||
"""
|
||||
owner, _, name = repo.partition("/")
|
||||
if not owner or not name:
|
||||
raise SystemExit(f"Invalid repository: {repo}")
|
||||
|
||||
query = """
|
||||
query($owner: String!, $name: String!, $number: Int!, $cursor: String) {
|
||||
repository(owner: $owner, name: $name) {
|
||||
pullRequest(number: $number) {
|
||||
reviewThreads(first: 100, after: $cursor) {
|
||||
pageInfo { hasNextPage endCursor }
|
||||
nodes { isResolved }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
"""
|
||||
|
||||
unresolved = 0
|
||||
cursor: str | None = None
|
||||
while True:
|
||||
response = _github_request(
|
||||
method="POST",
|
||||
path="/graphql",
|
||||
token=token,
|
||||
payload={
|
||||
"query": query,
|
||||
"variables": {"owner": owner, "name": name, "number": number, "cursor": cursor},
|
||||
},
|
||||
)
|
||||
if not isinstance(response, Mapping) or response.get("errors"):
|
||||
raise SystemExit(f"GitHub GraphQL reviewThreads query failed: {response}")
|
||||
try:
|
||||
threads = response["data"]["repository"]["pullRequest"]["reviewThreads"]
|
||||
nodes = threads["nodes"]
|
||||
page_info = threads["pageInfo"]
|
||||
except (KeyError, TypeError):
|
||||
raise SystemExit(
|
||||
"GitHub GraphQL reviewThreads response was missing expected fields"
|
||||
) from None
|
||||
unresolved += sum(1 for node in nodes if not node.get("isResolved"))
|
||||
if not page_info.get("hasNextPage"):
|
||||
return unresolved
|
||||
cursor = page_info.get("endCursor")
|
||||
|
||||
|
||||
def unresolved_threads_result(count: int) -> ApprovalResult:
|
||||
return ApprovalResult(
|
||||
False,
|
||||
"failure",
|
||||
f"BM Bossbot found {count} unresolved review thread(s)",
|
||||
)
|
||||
|
||||
|
||||
def evaluate_gate(
|
||||
*,
|
||||
token: str,
|
||||
repo: str,
|
||||
number: int,
|
||||
trusted: bool,
|
||||
) -> tuple[ApprovalResult, int]:
|
||||
"""Deterministic approval decision: trusted author + zero unresolved threads.
|
||||
|
||||
Tests-passed-for-this-head and non-draft are enforced upstream by the
|
||||
workflow trigger and the normalize step (should_review). Returns the
|
||||
result plus the unresolved-thread count for the PR-body summary.
|
||||
"""
|
||||
if not trusted:
|
||||
return (
|
||||
ApprovalResult(
|
||||
False,
|
||||
"failure",
|
||||
"BM Bossbot only gates owner/member/collaborator PRs",
|
||||
),
|
||||
0,
|
||||
)
|
||||
unresolved = count_unresolved_review_threads(token=token, repo=repo, number=number)
|
||||
if unresolved > 0:
|
||||
return unresolved_threads_result(unresolved), unresolved
|
||||
return ApprovalResult(True, "success", APPROVED_DESCRIPTION), 0
|
||||
|
||||
|
||||
def build_status_payload(*, state: str, description: str, target_url: str) -> dict[str, str]:
|
||||
return {
|
||||
"state": state,
|
||||
"context": STATUS_CONTEXT,
|
||||
"description": description,
|
||||
"target_url": target_url,
|
||||
}
|
||||
|
||||
|
||||
def render_summary(
|
||||
*,
|
||||
head_sha: str,
|
||||
result: ApprovalResult,
|
||||
trusted: bool,
|
||||
unresolved_threads: int,
|
||||
) -> str:
|
||||
return "\n".join(
|
||||
[
|
||||
f"Reviewed SHA: `{head_sha}`",
|
||||
"Gate: deterministic (tests, draft, author trust, review threads)",
|
||||
f"Status: `{result.state}` - {result.description}",
|
||||
"",
|
||||
f"- Trusted author: {'yes' if trusted else 'no'}",
|
||||
f"- Unresolved review threads: {unresolved_threads}",
|
||||
"",
|
||||
"Code review comes from the Codex connector and human reviewers;",
|
||||
"resolve every review thread to (re)gain approval for this head SHA.",
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def upsert_summary_block(body: str, summary: str) -> str:
|
||||
block = f"{SUMMARY_START}\n{summary.rstrip()}\n{SUMMARY_END}"
|
||||
pattern = re.compile(
|
||||
rf"{re.escape(SUMMARY_START)}.*?{re.escape(SUMMARY_END)}",
|
||||
flags=re.DOTALL,
|
||||
)
|
||||
if pattern.search(body):
|
||||
return pattern.sub(block, body, count=1)
|
||||
if body.strip():
|
||||
return f"{body.rstrip()}\n\n{block}\n"
|
||||
return f"{block}\n"
|
||||
|
||||
|
||||
def set_commit_status(*, token: str, repo: str, sha: str, payload: Mapping[str, str]) -> None:
|
||||
_github_request(
|
||||
method="POST",
|
||||
path=f"/repos/{repo}/statuses/{sha}",
|
||||
token=token,
|
||||
payload=payload,
|
||||
)
|
||||
|
||||
|
||||
def update_pull_request_body(*, token: str, repo: str, number: int, body: str) -> None:
|
||||
_github_request(
|
||||
method="PATCH",
|
||||
path=f"/repos/{repo}/pulls/{number}",
|
||||
token=token,
|
||||
payload={"body": body},
|
||||
)
|
||||
|
||||
|
||||
def get_pull_request_body(*, token: str, repo: str, number: int) -> str:
|
||||
response = _github_request(
|
||||
method="GET",
|
||||
path=f"/repos/{repo}/pulls/{number}",
|
||||
token=token,
|
||||
)
|
||||
if not isinstance(response, Mapping):
|
||||
raise SystemExit("GitHub API response for pull request was invalid")
|
||||
return _string(response.get("body"))
|
||||
|
||||
|
||||
def mark_pending(
|
||||
*,
|
||||
event_path: Path,
|
||||
repo: str | None,
|
||||
run_url: str,
|
||||
token_env: str,
|
||||
) -> None:
|
||||
event = pull_request_event(read_json(event_path), repo_override=repo)
|
||||
set_commit_status(
|
||||
token=_token(token_env),
|
||||
repo=event.repo,
|
||||
sha=event.head_sha,
|
||||
payload=build_status_payload(
|
||||
state="pending",
|
||||
description=PENDING_DESCRIPTION,
|
||||
target_url=run_url,
|
||||
),
|
||||
)
|
||||
typer.echo(f"Marked {STATUS_CONTEXT} pending for {event.head_sha}")
|
||||
|
||||
|
||||
def finalize_review(
|
||||
*,
|
||||
event_path: Path,
|
||||
trusted: bool,
|
||||
repo: str | None,
|
||||
run_url: str,
|
||||
token_env: str,
|
||||
) -> ApprovalResult:
|
||||
event = pull_request_event(read_json(event_path), repo_override=repo)
|
||||
token = _token(token_env)
|
||||
|
||||
result, unresolved = evaluate_gate(
|
||||
token=token, repo=event.repo, number=event.number, trusted=trusted
|
||||
)
|
||||
current_body = get_pull_request_body(token=token, repo=event.repo, number=event.number)
|
||||
updated_body = upsert_summary_block(
|
||||
current_body,
|
||||
render_summary(
|
||||
head_sha=event.head_sha,
|
||||
result=result,
|
||||
trusted=trusted,
|
||||
unresolved_threads=unresolved,
|
||||
),
|
||||
)
|
||||
update_pull_request_body(token=token, repo=event.repo, number=event.number, body=updated_body)
|
||||
set_commit_status(
|
||||
token=token,
|
||||
repo=event.repo,
|
||||
sha=event.head_sha,
|
||||
payload=build_status_payload(
|
||||
state=result.state,
|
||||
description=result.description,
|
||||
target_url=run_url,
|
||||
),
|
||||
)
|
||||
typer.echo(f"Marked {STATUS_CONTEXT} {result.state} for {event.head_sha}")
|
||||
return result
|
||||
|
||||
|
||||
def get_pull_request_head_sha(*, token: str, repo: str, number: int) -> str:
|
||||
response = _github_request(
|
||||
method="GET",
|
||||
path=f"/repos/{repo}/pulls/{number}",
|
||||
token=token,
|
||||
)
|
||||
if not isinstance(response, Mapping):
|
||||
raise SystemExit("GitHub API response for pull request was invalid")
|
||||
head = response.get("head")
|
||||
head_sha = _string(head.get("sha")) if isinstance(head, Mapping) else ""
|
||||
if not head_sha:
|
||||
raise SystemExit("GitHub API response was missing pull request head SHA")
|
||||
return head_sha
|
||||
|
||||
|
||||
def head_sha_was_approved(*, token: str, repo: str, sha: str) -> bool:
|
||||
"""Return whether a full BM Bossbot review previously approved this head SHA.
|
||||
|
||||
Commit statuses are append-only history, so the approval record survives a
|
||||
later thread-failure status for the same SHA. The recheck path can post a
|
||||
new status on every review-thread event, so a busy PR can accumulate more
|
||||
than one page of statuses — page through all of them or the approval
|
||||
record falls off page one and a valid approval is never restored.
|
||||
"""
|
||||
page = 1
|
||||
while True:
|
||||
response = _github_request(
|
||||
method="GET",
|
||||
path=f"/repos/{repo}/commits/{sha}/statuses?per_page=100&page={page}",
|
||||
token=token,
|
||||
)
|
||||
if not isinstance(response, list):
|
||||
raise SystemExit("GitHub API response for commit statuses was invalid")
|
||||
if not response:
|
||||
return False
|
||||
if any(
|
||||
isinstance(status, Mapping)
|
||||
and status.get("context") == STATUS_CONTEXT
|
||||
and status.get("state") == "success"
|
||||
and status.get("description") == APPROVED_DESCRIPTION
|
||||
for status in response
|
||||
):
|
||||
return True
|
||||
page += 1
|
||||
|
||||
|
||||
def recheck_threads(
|
||||
*,
|
||||
repo: str,
|
||||
number: int,
|
||||
run_url: str,
|
||||
token_env: str,
|
||||
) -> None:
|
||||
"""Re-evaluate the approval status when review threads change.
|
||||
|
||||
Trigger: pull_request_review / review_comment / review_thread events.
|
||||
Why: the full review runs once per head SHA after Tests; feedback that
|
||||
arrives later (or gets resolved later) must move the gate without
|
||||
re-running the LLM review.
|
||||
Outcome: unresolved threads flip the status to failure; once every thread
|
||||
is resolved, a previously earned approval for the same head SHA is
|
||||
restored. Without a prior approval the status is left untouched so a
|
||||
pending/failed review cannot be upgraded by thread resolution alone.
|
||||
"""
|
||||
token = _token(token_env)
|
||||
head_sha = get_pull_request_head_sha(token=token, repo=repo, number=number)
|
||||
unresolved = count_unresolved_review_threads(token=token, repo=repo, number=number)
|
||||
|
||||
if unresolved > 0:
|
||||
result = unresolved_threads_result(unresolved)
|
||||
elif head_sha_was_approved(token=token, repo=repo, sha=head_sha):
|
||||
result = ApprovalResult(True, "success", APPROVED_DESCRIPTION)
|
||||
else:
|
||||
typer.echo(
|
||||
f"All review threads resolved but no prior approval exists for {head_sha}; "
|
||||
"leaving status unchanged"
|
||||
)
|
||||
return
|
||||
|
||||
set_commit_status(
|
||||
token=token,
|
||||
repo=repo,
|
||||
sha=head_sha,
|
||||
payload=build_status_payload(
|
||||
state=result.state,
|
||||
description=result.description,
|
||||
target_url=run_url,
|
||||
),
|
||||
)
|
||||
typer.echo(f"Marked {STATUS_CONTEXT} {result.state} for {head_sha} ({result.description})")
|
||||
|
||||
|
||||
def _github_request(
|
||||
*,
|
||||
method: str,
|
||||
path: str,
|
||||
token: str,
|
||||
payload: Mapping[str, Any] | None = None,
|
||||
) -> Any:
|
||||
data = None if payload is None else json.dumps(payload).encode("utf-8")
|
||||
request = urllib.request.Request(
|
||||
f"https://api.github.com{path}",
|
||||
data=data,
|
||||
method=method,
|
||||
headers={
|
||||
"Accept": "application/vnd.github+json",
|
||||
"Authorization": f"Bearer {token}",
|
||||
"Content-Type": "application/json",
|
||||
"User-Agent": "basic-memory-bm-bossbot",
|
||||
"X-GitHub-Api-Version": "2022-11-28",
|
||||
},
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=30) as response:
|
||||
response_body = response.read().decode("utf-8")
|
||||
except urllib.error.HTTPError as exc:
|
||||
detail = exc.read().decode("utf-8", errors="replace")
|
||||
raise SystemExit(f"GitHub API request failed: {exc.code} {detail}") from None
|
||||
return json.loads(response_body) if response_body else None
|
||||
|
||||
|
||||
def _string(value: object) -> str:
|
||||
return value if isinstance(value, str) else ""
|
||||
|
||||
|
||||
def _token(env_name: str) -> str:
|
||||
token = os.environ.get(env_name)
|
||||
if not token:
|
||||
raise SystemExit(f"Missing required token environment variable: {env_name}")
|
||||
return token
|
||||
|
||||
|
||||
@app.command("pending")
|
||||
def pending(
|
||||
event: Annotated[
|
||||
Path,
|
||||
typer.Option(
|
||||
"--event",
|
||||
exists=True,
|
||||
dir_okay=False,
|
||||
readable=True,
|
||||
help="GitHub event payload JSON.",
|
||||
),
|
||||
],
|
||||
run_url: Annotated[str, typer.Option("--run-url", help="Workflow run URL.")],
|
||||
repo: Annotated[str | None, typer.Option("--repo", help="owner/name repository.")] = None,
|
||||
token_env: Annotated[
|
||||
str,
|
||||
typer.Option("--token-env", help="Environment variable containing a GitHub token."),
|
||||
] = "GITHUB_TOKEN",
|
||||
) -> None:
|
||||
"""Set BM Bossbot Approval pending on the PR head SHA."""
|
||||
mark_pending(event_path=event, repo=repo, run_url=run_url, token_env=token_env)
|
||||
|
||||
|
||||
@app.command("finalize")
|
||||
def finalize(
|
||||
event: Annotated[
|
||||
Path,
|
||||
typer.Option(
|
||||
"--event",
|
||||
exists=True,
|
||||
dir_okay=False,
|
||||
readable=True,
|
||||
help="GitHub event payload JSON.",
|
||||
),
|
||||
],
|
||||
trusted: Annotated[
|
||||
str,
|
||||
typer.Option(
|
||||
"--trusted",
|
||||
help="Whether the PR author is trusted (true/false from the classify step).",
|
||||
),
|
||||
],
|
||||
run_url: Annotated[str, typer.Option("--run-url", help="Workflow run URL.")],
|
||||
repo: Annotated[str | None, typer.Option("--repo", help="owner/name repository.")] = None,
|
||||
token_env: Annotated[
|
||||
str,
|
||||
typer.Option("--token-env", help="Environment variable containing a GitHub token."),
|
||||
] = "GITHUB_TOKEN",
|
||||
) -> None:
|
||||
"""Finalize BM Bossbot Approval from the deterministic gate."""
|
||||
result = finalize_review(
|
||||
event_path=event,
|
||||
trusted=trusted.strip().lower() == "true",
|
||||
repo=repo,
|
||||
run_url=run_url,
|
||||
token_env=token_env,
|
||||
)
|
||||
if not result.approved:
|
||||
raise typer.Exit(1)
|
||||
|
||||
|
||||
@app.command("recheck")
|
||||
def recheck(
|
||||
pr_number: Annotated[int, typer.Option("--pr-number", min=1, help="Pull request number.")],
|
||||
run_url: Annotated[str, typer.Option("--run-url", help="Workflow run URL.")],
|
||||
repo: Annotated[str, typer.Option("--repo", help="owner/name repository.")],
|
||||
token_env: Annotated[
|
||||
str,
|
||||
typer.Option("--token-env", help="Environment variable containing a GitHub token."),
|
||||
] = "GITHUB_TOKEN",
|
||||
) -> None:
|
||||
"""Re-evaluate BM Bossbot Approval from current review-thread state."""
|
||||
recheck_threads(repo=repo, number=pr_number, run_url=run_url, token_env=token_env)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
app()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+166
@@ -0,0 +1,166 @@
|
||||
#!/usr/bin/env -S uv run --script
|
||||
# /// script
|
||||
# requires-python = ">=3.12"
|
||||
# dependencies = [
|
||||
# "openai>=1.100.2",
|
||||
# "python-dotenv>=1.1.0",
|
||||
# "typer>=0.9.0",
|
||||
# ]
|
||||
# ///
|
||||
"""Generate a BM Bossbot infographic with the OpenAI Images API."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Annotated, Any
|
||||
|
||||
import typer
|
||||
from dotenv import load_dotenv
|
||||
from openai import OpenAI
|
||||
|
||||
|
||||
DEFAULT_MODEL = "gpt-image-2"
|
||||
DEFAULT_SIZE = "1536x1024"
|
||||
DEFAULT_QUALITY = "high"
|
||||
DEFAULT_FORMAT = "webp"
|
||||
DEFAULT_COMPRESSION = 90
|
||||
app = typer.Typer(
|
||||
add_completion=False,
|
||||
help="Generate Basic Memory infographics with the OpenAI Images API.",
|
||||
no_args_is_help=True,
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class GeneratedImage:
|
||||
path: Path
|
||||
revised_prompt: str | None
|
||||
|
||||
|
||||
def validate_output_path(path: Path, *, repo_root: Path | None = None) -> Path:
|
||||
root = (repo_root or Path.cwd()).resolve()
|
||||
output = path.resolve()
|
||||
allowed_root = (root / "docs" / "assets" / "infographics").resolve()
|
||||
if not output.is_relative_to(allowed_root):
|
||||
allowed_path = allowed_root.relative_to(root).as_posix()
|
||||
raise ValueError(f"Output path must be under {allowed_path}")
|
||||
if output.suffix != ".webp":
|
||||
raise ValueError("Output path must end with .webp")
|
||||
return output
|
||||
|
||||
|
||||
def generate_image_result(
|
||||
*,
|
||||
prompt: str,
|
||||
output_path: Path,
|
||||
model: str = DEFAULT_MODEL,
|
||||
size: str = DEFAULT_SIZE,
|
||||
quality: str = DEFAULT_QUALITY,
|
||||
output_format: str = DEFAULT_FORMAT,
|
||||
output_compression: int = DEFAULT_COMPRESSION,
|
||||
client: Any | None = None,
|
||||
retries: int = 2,
|
||||
) -> GeneratedImage:
|
||||
output = validate_output_path(output_path)
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
load_dotenv()
|
||||
openai_client = client or OpenAI()
|
||||
|
||||
for attempt in range(retries + 1):
|
||||
try:
|
||||
response = openai_client.images.generate(
|
||||
model=model,
|
||||
prompt=prompt,
|
||||
size=size,
|
||||
quality=quality,
|
||||
output_format=output_format,
|
||||
output_compression=output_compression,
|
||||
)
|
||||
image = response.data[0]
|
||||
image_b64 = image.b64_json
|
||||
if not image_b64:
|
||||
raise RuntimeError("OpenAI image response did not include b64_json")
|
||||
output.write_bytes(base64.b64decode(image_b64))
|
||||
return GeneratedImage(path=output, revised_prompt=image.revised_prompt)
|
||||
except Exception:
|
||||
if attempt >= retries:
|
||||
raise
|
||||
time.sleep(2**attempt)
|
||||
|
||||
raise RuntimeError("Image generation retry loop exited unexpectedly")
|
||||
|
||||
|
||||
def generate_image(
|
||||
*,
|
||||
prompt: str,
|
||||
output_path: Path,
|
||||
model: str = DEFAULT_MODEL,
|
||||
size: str = DEFAULT_SIZE,
|
||||
quality: str = DEFAULT_QUALITY,
|
||||
output_format: str = DEFAULT_FORMAT,
|
||||
output_compression: int = DEFAULT_COMPRESSION,
|
||||
client: Any | None = None,
|
||||
retries: int = 2,
|
||||
) -> Path:
|
||||
return generate_image_result(
|
||||
prompt=prompt,
|
||||
output_path=output_path,
|
||||
model=model,
|
||||
size=size,
|
||||
quality=quality,
|
||||
output_format=output_format,
|
||||
output_compression=output_compression,
|
||||
client=client,
|
||||
retries=retries,
|
||||
).path
|
||||
|
||||
|
||||
@app.command()
|
||||
def generate(
|
||||
prompt_file: Annotated[
|
||||
Path,
|
||||
typer.Option(
|
||||
"--prompt-file",
|
||||
exists=True,
|
||||
dir_okay=False,
|
||||
readable=True,
|
||||
help="Markdown/text prompt file to send to the image model.",
|
||||
),
|
||||
],
|
||||
output: Annotated[Path, typer.Option("--output", help="Output .webp path.")],
|
||||
model: Annotated[str, typer.Option("--model", help="OpenAI image model.")] = DEFAULT_MODEL,
|
||||
size: Annotated[str, typer.Option("--size", help="Image size.")] = DEFAULT_SIZE,
|
||||
quality: Annotated[str, typer.Option("--quality", help="Image quality.")] = DEFAULT_QUALITY,
|
||||
output_compression: Annotated[
|
||||
int,
|
||||
typer.Option(
|
||||
"--output-compression",
|
||||
min=0,
|
||||
max=100,
|
||||
help="WebP output compression.",
|
||||
),
|
||||
] = DEFAULT_COMPRESSION,
|
||||
retries: Annotated[int, typer.Option("--retries", min=0, help="Retry attempts.")] = 2,
|
||||
) -> None:
|
||||
"""Generate an infographic from a prompt file."""
|
||||
output = generate_image(
|
||||
prompt=prompt_file.read_text(encoding="utf-8"),
|
||||
output_path=output,
|
||||
model=model,
|
||||
size=size,
|
||||
quality=quality,
|
||||
output_compression=output_compression,
|
||||
retries=retries,
|
||||
)
|
||||
typer.echo(output)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
app()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+438
@@ -0,0 +1,438 @@
|
||||
#!/usr/bin/env -S uv run --script
|
||||
# /// script
|
||||
# requires-python = ">=3.12"
|
||||
# dependencies = [
|
||||
# "openai>=1.100.2",
|
||||
# "python-dotenv>=1.1.0",
|
||||
# "typer>=0.9.0",
|
||||
# ]
|
||||
# ///
|
||||
"""Build and generate a non-gating BM Bossbot PR image."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import html
|
||||
import json
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from enum import StrEnum
|
||||
from pathlib import Path
|
||||
from typing import Annotated, Any, Mapping
|
||||
|
||||
import typer
|
||||
|
||||
if __package__:
|
||||
from .generate_infographic import (
|
||||
DEFAULT_MODEL,
|
||||
DEFAULT_QUALITY,
|
||||
DEFAULT_SIZE,
|
||||
generate_image_result,
|
||||
)
|
||||
else:
|
||||
from generate_infographic import (
|
||||
DEFAULT_MODEL,
|
||||
DEFAULT_QUALITY,
|
||||
DEFAULT_SIZE,
|
||||
generate_image_result,
|
||||
)
|
||||
|
||||
|
||||
SUMMARY_START = "<!-- BM_BOSSBOT_SUMMARY:start -->"
|
||||
SUMMARY_END = "<!-- BM_BOSSBOT_SUMMARY:end -->"
|
||||
THEME_START = "<!-- BM_INFOGRAPHIC_THEME:start -->"
|
||||
THEME_END = "<!-- BM_INFOGRAPHIC_THEME:end -->"
|
||||
PROVENANCE_START = "<!-- BM_INFOGRAPHIC_PROVENANCE:start -->"
|
||||
PROVENANCE_END = "<!-- BM_INFOGRAPHIC_PROVENANCE:end -->"
|
||||
IMAGE_START = "<!-- pr-infographic:start -->"
|
||||
IMAGE_END = "<!-- pr-infographic:end -->"
|
||||
# Managed blocks are bot-written artifacts (review verdict, image embed,
|
||||
# provenance). They must never feed the image: sourcing the review summary is
|
||||
# what made every image an "APPROVED" stamp instead of depicting the change.
|
||||
MANAGED_BLOCKS = (
|
||||
(SUMMARY_START, SUMMARY_END),
|
||||
(THEME_START, THEME_END),
|
||||
(PROVENANCE_START, PROVENANCE_END),
|
||||
(IMAGE_START, IMAGE_END),
|
||||
)
|
||||
app = typer.Typer(
|
||||
add_completion=False,
|
||||
help="Generate a non-gating BM Bossbot PR image.",
|
||||
no_args_is_help=True,
|
||||
)
|
||||
|
||||
|
||||
class ThemeSource(StrEnum):
|
||||
AUTO = "auto"
|
||||
CLI = "cli"
|
||||
PR_BODY = "pr-body"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ThemeSelection:
|
||||
theme: str
|
||||
source: ThemeSource
|
||||
|
||||
|
||||
BM_IMAGE_THEME_POOL = (
|
||||
"computer science college textbook: SICP-style diagrams, automata, compiler "
|
||||
"pipelines, type theory, and annotated chalkboard rigor",
|
||||
"classic literature: sea voyages, gothic manors, Dickensian streets, library "
|
||||
"marginalia, and travel-journal artifacts",
|
||||
"fantasy quest ledger: original guild maps, spellbooks, dungeon keys, tavern "
|
||||
"notices, and artifact inventories with no copyrighted settings",
|
||||
"heavy music editorial: metal, hard rock, punk, techno, soul, or reggae "
|
||||
"tour-poster energy with no direct band logos or likenesses",
|
||||
"knockoff space opera: fleet routes, mission consoles, contraband manifests, "
|
||||
"and practical starship drama with no named fictional universes",
|
||||
"sword-and-sorcery: ruined temples, desert roads, battle standards, ancient "
|
||||
"maps, and heroic silhouettes with no named character likenesses",
|
||||
"comic book cover: original splash-page composition, caption boxes, clean "
|
||||
"halftone texture, and bold issue-cover drama",
|
||||
"French new wave movie poster: stark typography, city streets, jump-cut "
|
||||
"composition, and high-contrast editorial photography cues",
|
||||
"WWII public-information poster: home-front logistics, mobilization arrows, "
|
||||
"bold simplified figures, and no real-world party symbols or hate imagery",
|
||||
"Italian movie poster: hand-painted drama, expressive color, credit-block "
|
||||
"energy, and 1960s or 1970s cinema composition with no actor likenesses",
|
||||
"Shakespearean stage: acts and scenes, court intrigue, stage blocking, "
|
||||
"dramatis personae, backstage cue sheets, and theatrical light",
|
||||
"Greek mythology: temple steps, oracle tablets, constellations, labyrinths, "
|
||||
"ship routes, and original heroic allegory",
|
||||
"noir detective photography: case files, typed evidence labels, civic "
|
||||
"infrastructure, streetlight shadows, and newsroom archive grit",
|
||||
"space exploration and astronomy: celestial atlases, observatory charts, "
|
||||
"orbital mechanics, planetary survey routes, and deep-space mission drama",
|
||||
"editorial painting: abstract, classical landscape, western action, "
|
||||
"chiaroscuro, historical mural, stormy seascape, or allegorical canvas",
|
||||
"classic black-and-white photography: documentary field report, contact "
|
||||
"sheet, street photography, civic infrastructure, and darkroom contrast",
|
||||
"80's action movie poster: smoky backlit warehouses, neon streets, practical "
|
||||
"explosions, mission dossiers, countdowns, and no actor likenesses",
|
||||
"alchemy manuscript: transformation diagrams, annotated symbols, recipe-like "
|
||||
"process artifacts, and illuminated margins",
|
||||
"brutalist civic planning: concrete signage, zoning blocks, transit diagrams, "
|
||||
"infrastructure maps, and stern public-service clarity",
|
||||
)
|
||||
|
||||
|
||||
def build_change_shape(
|
||||
context: Mapping[str, Any],
|
||||
*,
|
||||
max_commits: int = 10,
|
||||
max_files: int = 10,
|
||||
) -> str:
|
||||
"""Render a compact factual digest of the PR: labels, linked issues, commits, files.
|
||||
|
||||
Mirrors the delivery context the Basic Memory CI capture flow collects
|
||||
(ProjectUpdateContext): the goal is to ground the image in the theme of the
|
||||
WHOLE change — what it touches and why — not just the title/description.
|
||||
"""
|
||||
lines: list[str] = []
|
||||
|
||||
labels = [str(label.get("name", "")) for label in context.get("labels") or []]
|
||||
labels = [label for label in labels if label]
|
||||
if labels:
|
||||
lines.append(f"Labels: {', '.join(labels)}")
|
||||
|
||||
issues = context.get("closingIssuesReferences") or []
|
||||
if issues:
|
||||
lines.append("Linked issues:")
|
||||
for issue in issues:
|
||||
number = issue.get("number")
|
||||
title = str(issue.get("title") or "").strip()
|
||||
lines.append(f"- #{number}: {title}" if title else f"- #{number}")
|
||||
|
||||
commits = context.get("commits") or []
|
||||
subjects = [str(commit.get("messageHeadline") or "").strip() for commit in commits]
|
||||
# Merge commits carry no thematic signal — they're branch bookkeeping.
|
||||
subjects = [
|
||||
subject
|
||||
for subject in subjects
|
||||
if subject and not subject.startswith(("Merge branch ", "Merge pull request "))
|
||||
]
|
||||
if subjects:
|
||||
lines.append(f"Commit subjects ({len(subjects)} total):")
|
||||
lines.extend(f"- {subject}" for subject in subjects[:max_commits])
|
||||
if len(subjects) > max_commits:
|
||||
lines.append(f"- ... and {len(subjects) - max_commits} more")
|
||||
|
||||
files = context.get("files") or []
|
||||
if files:
|
||||
additions = sum(int(item.get("additions") or 0) for item in files)
|
||||
deletions = sum(int(item.get("deletions") or 0) for item in files)
|
||||
lines.append(f"Files changed ({len(files)} total, +{additions}/-{deletions}):")
|
||||
ranked = sorted(
|
||||
files,
|
||||
key=lambda item: int(item.get("additions") or 0) + int(item.get("deletions") or 0),
|
||||
reverse=True,
|
||||
)
|
||||
for item in ranked[:max_files]:
|
||||
path = str(item.get("path") or "")
|
||||
lines.append(f"- {path} (+{item.get('additions') or 0}/-{item.get('deletions') or 0})")
|
||||
if len(files) > max_files:
|
||||
lines.append(f"- ... and {len(files) - max_files} more files")
|
||||
|
||||
return "\n".join(lines) if lines else "(no additional change context available)"
|
||||
|
||||
|
||||
def extract_pr_content(pr_body: str) -> str:
|
||||
"""Return the author's own PR description with all managed bot blocks removed."""
|
||||
content = pr_body
|
||||
for start, end in MANAGED_BLOCKS:
|
||||
content = re.sub(
|
||||
rf"{re.escape(start)}.*?{re.escape(end)}",
|
||||
"",
|
||||
content,
|
||||
flags=re.DOTALL,
|
||||
)
|
||||
return content.strip()
|
||||
|
||||
|
||||
def extract_infographic_theme(pr_body: str) -> str | None:
|
||||
pattern = re.compile(
|
||||
rf"{re.escape(THEME_START)}\s*(.*?)\s*{re.escape(THEME_END)}",
|
||||
flags=re.DOTALL,
|
||||
)
|
||||
match = pattern.search(pr_body)
|
||||
if not match:
|
||||
return None
|
||||
theme = match.group(1).strip()
|
||||
return theme or None
|
||||
|
||||
|
||||
def select_image_theme(
|
||||
*,
|
||||
pr_number: int,
|
||||
pr_title: str,
|
||||
pr_body: str,
|
||||
theme_override: str | None,
|
||||
) -> ThemeSelection:
|
||||
if theme_override:
|
||||
return ThemeSelection(theme=theme_override, source=ThemeSource.CLI)
|
||||
body_theme = extract_infographic_theme(pr_body)
|
||||
if body_theme:
|
||||
return ThemeSelection(theme=body_theme, source=ThemeSource.PR_BODY)
|
||||
# Seed on author-owned PR identity, not the review summary, so the pick is
|
||||
# stable across re-reviews of the same PR.
|
||||
seed = f"{pr_number}\n{pr_title}".encode("utf-8")
|
||||
index = int.from_bytes(hashlib.sha256(seed).digest()[:2], byteorder="big") % len(
|
||||
BM_IMAGE_THEME_POOL
|
||||
)
|
||||
return ThemeSelection(theme=BM_IMAGE_THEME_POOL[index], source=ThemeSource.AUTO)
|
||||
|
||||
|
||||
def _preformatted(value: str) -> str:
|
||||
return f"<pre><code>{html.escape(value, quote=False)}</code></pre>"
|
||||
|
||||
|
||||
def build_infographic_provenance_block(
|
||||
*,
|
||||
pr_number: int,
|
||||
output_path: Path,
|
||||
model: str,
|
||||
size: str,
|
||||
quality: str,
|
||||
theme: str,
|
||||
theme_source: ThemeSource,
|
||||
) -> str:
|
||||
return f"""
|
||||
{PROVENANCE_START}
|
||||
<details>
|
||||
<summary>BM Bossbot image choices</summary>
|
||||
|
||||
- Pull request: `#{pr_number}`
|
||||
- Generated asset: `{output_path.as_posix()}`
|
||||
- Image model: `{model}`
|
||||
- Size: `{size}`
|
||||
- Quality: `{quality}`
|
||||
- Image mode: `editorial-image`
|
||||
- Theme source: `{theme_source.value}`
|
||||
|
||||
Theme / visual direction:
|
||||
{_preformatted(theme)}
|
||||
|
||||
</details>
|
||||
{PROVENANCE_END}
|
||||
""".strip()
|
||||
|
||||
|
||||
def upsert_managed_block(body: str, *, block: str, start: str, end: str) -> str:
|
||||
pattern = re.compile(rf"{re.escape(start)}.*?{re.escape(end)}", flags=re.DOTALL)
|
||||
if pattern.search(body):
|
||||
return pattern.sub(block, body, count=1)
|
||||
if body.strip():
|
||||
return f"{body.rstrip()}\n\n{block}\n"
|
||||
return f"{block}\n"
|
||||
|
||||
|
||||
def build_infographic_prompt(
|
||||
*,
|
||||
pr_number: int,
|
||||
pr_title: str,
|
||||
pr_content: str,
|
||||
change_shape: str,
|
||||
theme: str,
|
||||
theme_source: ThemeSource,
|
||||
) -> str:
|
||||
theme_label = (
|
||||
"Selected BM visual direction"
|
||||
if theme_source == ThemeSource.AUTO
|
||||
else "User-supplied visual direction"
|
||||
)
|
||||
|
||||
return f"""
|
||||
Create a polished landscape WebP editorial image for Basic Memory PR #{pr_number}.
|
||||
|
||||
Your subject is the CONTENT of the pull request — what the change does and why
|
||||
it matters — described in the title, description, and change shape below.
|
||||
Express the theme of the whole change as a visual story.
|
||||
|
||||
This image is decoration for the PR conversation. It is NOT a review artifact:
|
||||
do not depict review verdicts, approval, or process. Never render approval
|
||||
stamps, "APPROVED"/"SUCCESS"/"VERDICT" wording, rubber stamps, wax seals of
|
||||
approval, badges, checkmarks, checklists, status lines, SHA strings, or
|
||||
BM Bossbot itself. If the composition needs text, draw it from the change's
|
||||
subject matter only.
|
||||
|
||||
Pull request title:
|
||||
{pr_title}
|
||||
|
||||
Pull request description:
|
||||
{pr_content}
|
||||
|
||||
Change shape — factual delivery context (labels, linked issues, commit
|
||||
subjects, changed files). Use it to understand what the whole PR touches and
|
||||
let that steer the imagery. It is context, NOT captions: never render file
|
||||
paths, diff stats, issue numbers, or commit subjects verbatim in the image.
|
||||
{change_shape}
|
||||
|
||||
{theme_label}:
|
||||
{theme}
|
||||
|
||||
Treat the visual direction as style inspiration only. Do not let it override
|
||||
facts, readability, source material, or the prohibition on review imagery.
|
||||
|
||||
Use image-first composition: create a scene, movie poster, editorial painting,
|
||||
classic photograph, cover image, symbolic tableau, staged artifact, or another
|
||||
visual moment that expresses the PR intent.
|
||||
|
||||
Make the selected direction shape the subject, lighting, composition, props,
|
||||
environment, and mood. Use one strong focal point. Prefer visual metaphor over
|
||||
explanatory UI.
|
||||
|
||||
Use at most a short title and zero to three short labels if text helps. Any text
|
||||
that appears must be high-contrast, smooth, anti-aliased, and readable.
|
||||
|
||||
Do not render an infographic, dashboard, flowchart, timeline strip, checklist,
|
||||
bullet-list panel, data panel, or dense explanatory diagram.
|
||||
|
||||
Avoid fake screenshots, code blocks, invented claims, copyrighted characters,
|
||||
logos, named fictional universes, direct band logos, album art, celebrity
|
||||
likenesses, or decorations that obscure content.
|
||||
""".strip()
|
||||
|
||||
|
||||
@app.command()
|
||||
def generate(
|
||||
pr_number: Annotated[
|
||||
int,
|
||||
typer.Option("--pr-number", min=1, help="Pull request number."),
|
||||
],
|
||||
pr_context_file: Annotated[
|
||||
Path,
|
||||
typer.Option(
|
||||
"--pr-context-file",
|
||||
exists=True,
|
||||
dir_okay=False,
|
||||
readable=True,
|
||||
help=(
|
||||
"JSON from `gh pr view --json "
|
||||
"title,body,labels,files,commits,closingIssuesReferences`."
|
||||
),
|
||||
),
|
||||
],
|
||||
output: Annotated[Path, typer.Option("--output", help="Output .webp path.")],
|
||||
model: Annotated[str, typer.Option("--model", help="OpenAI image model.")] = DEFAULT_MODEL,
|
||||
size: Annotated[str, typer.Option("--size", help="Image size.")] = DEFAULT_SIZE,
|
||||
quality: Annotated[str, typer.Option("--quality", help="Image quality.")] = DEFAULT_QUALITY,
|
||||
retries: Annotated[int, typer.Option("--retries", min=0, help="Retry attempts.")] = 2,
|
||||
theme: Annotated[
|
||||
str | None,
|
||||
typer.Option("--theme", help="Optional visual theme preference."),
|
||||
] = None,
|
||||
provenance_output: Annotated[
|
||||
Path | None,
|
||||
typer.Option(
|
||||
"--provenance-output",
|
||||
dir_okay=False,
|
||||
help="Optional file to write the managed PR-body provenance block.",
|
||||
),
|
||||
] = None,
|
||||
print_prompt: Annotated[
|
||||
bool,
|
||||
typer.Option(
|
||||
"--print-prompt",
|
||||
"--dry-run",
|
||||
help="Print the generated prompt and exit without calling OpenAI. Alias: --dry-run.",
|
||||
),
|
||||
] = False,
|
||||
) -> None:
|
||||
"""Generate the canonical PR image from the PR's title, description, and change shape."""
|
||||
context = json.loads(pr_context_file.read_text(encoding="utf-8"))
|
||||
if not isinstance(context, Mapping):
|
||||
raise typer.BadParameter("PR context file must contain a JSON object")
|
||||
pr_title = str(context.get("title") or "")
|
||||
pr_body = str(context.get("body") or "")
|
||||
pr_content = extract_pr_content(pr_body)
|
||||
change_shape = build_change_shape(context)
|
||||
theme_selection = select_image_theme(
|
||||
pr_number=pr_number,
|
||||
pr_title=pr_title,
|
||||
pr_body=pr_body,
|
||||
theme_override=theme,
|
||||
)
|
||||
prompt = build_infographic_prompt(
|
||||
pr_number=pr_number,
|
||||
pr_title=pr_title,
|
||||
pr_content=pr_content,
|
||||
change_shape=change_shape,
|
||||
theme=theme_selection.theme,
|
||||
theme_source=theme_selection.source,
|
||||
)
|
||||
if print_prompt:
|
||||
typer.echo(prompt)
|
||||
raise typer.Exit()
|
||||
|
||||
image_result = generate_image_result(
|
||||
prompt=prompt,
|
||||
output_path=output,
|
||||
model=model,
|
||||
size=size,
|
||||
quality=quality,
|
||||
retries=retries,
|
||||
)
|
||||
output_path = image_result.path
|
||||
if provenance_output:
|
||||
provenance_output.parent.mkdir(parents=True, exist_ok=True)
|
||||
provenance_output.write_text(
|
||||
build_infographic_provenance_block(
|
||||
pr_number=pr_number,
|
||||
output_path=output_path,
|
||||
model=model,
|
||||
size=size,
|
||||
quality=quality,
|
||||
theme=theme_selection.theme,
|
||||
theme_source=theme_selection.source,
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
typer.echo(output_path)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
app()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,225 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Seed and refresh shared pytest-testmon data for Git worktrees."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import NamedTuple
|
||||
|
||||
TESTMON_FILENAMES = (".testmondata", ".testmondata-shm", ".testmondata-wal")
|
||||
TESTMON_CACHE_ENV = "BM_TESTMON_CACHE_DIR"
|
||||
|
||||
|
||||
class TestmonCacheResult(NamedTuple):
|
||||
status: str
|
||||
source_dir: Path
|
||||
destination_dir: Path
|
||||
copied: tuple[Path, ...]
|
||||
|
||||
|
||||
def _run_git(args: list[str], cwd: Path) -> str:
|
||||
return subprocess.check_output(["git", *args], cwd=cwd, text=True).strip()
|
||||
|
||||
|
||||
def resolve_repo_root(repo_root: Path | None = None) -> Path:
|
||||
if repo_root is not None:
|
||||
return repo_root.expanduser().resolve()
|
||||
|
||||
return Path(_run_git(["rev-parse", "--show-toplevel"], Path.cwd())).resolve()
|
||||
|
||||
|
||||
def resolve_cache_dir(repo_root: Path, cache_dir: Path | None = None) -> Path:
|
||||
if cache_dir is not None:
|
||||
return cache_dir.expanduser().resolve()
|
||||
|
||||
if env_cache_dir := os.environ.get(TESTMON_CACHE_ENV):
|
||||
return Path(env_cache_dir).expanduser().resolve()
|
||||
|
||||
git_common_dir = Path(_run_git(["rev-parse", "--git-common-dir"], repo_root))
|
||||
if not git_common_dir.is_absolute():
|
||||
git_common_dir = repo_root / git_common_dir
|
||||
|
||||
return git_common_dir.resolve() / "testmon-cache" / "main"
|
||||
|
||||
|
||||
def _testmon_datafile(directory: Path) -> Path:
|
||||
return directory / ".testmondata"
|
||||
|
||||
|
||||
def _testmon_files(directory: Path) -> list[Path]:
|
||||
return [
|
||||
directory / filename for filename in TESTMON_FILENAMES if (directory / filename).is_file()
|
||||
]
|
||||
|
||||
|
||||
def _remove_path(path: Path) -> None:
|
||||
if path.is_dir():
|
||||
shutil.rmtree(path)
|
||||
elif path.exists():
|
||||
path.unlink()
|
||||
|
||||
|
||||
def _copy_testmon_files(source_dir: Path, destination_dir: Path) -> tuple[Path, ...]:
|
||||
destination_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
copied: list[Path] = []
|
||||
for source in _testmon_files(source_dir):
|
||||
destination = destination_dir / source.name
|
||||
shutil.copy2(source, destination)
|
||||
copied.append(destination)
|
||||
|
||||
return tuple(copied)
|
||||
|
||||
|
||||
def seed_testmon_data(repo_root: Path, cache_dir: Path) -> TestmonCacheResult:
|
||||
local_datafile = _testmon_datafile(repo_root)
|
||||
shared_datafile = _testmon_datafile(cache_dir)
|
||||
|
||||
if local_datafile.exists():
|
||||
return TestmonCacheResult(
|
||||
status="exists",
|
||||
source_dir=cache_dir,
|
||||
destination_dir=repo_root,
|
||||
copied=(),
|
||||
)
|
||||
|
||||
if not shared_datafile.exists():
|
||||
return TestmonCacheResult(
|
||||
status="missing",
|
||||
source_dir=cache_dir,
|
||||
destination_dir=repo_root,
|
||||
copied=(),
|
||||
)
|
||||
|
||||
# A worktree with sidecars but no main database is stale; replace the set
|
||||
# together so SQLite never sees a mixed local/cache snapshot.
|
||||
for filename in TESTMON_FILENAMES:
|
||||
_remove_path(repo_root / filename)
|
||||
|
||||
copied = _copy_testmon_files(cache_dir, repo_root)
|
||||
return TestmonCacheResult(
|
||||
status="seeded",
|
||||
source_dir=cache_dir,
|
||||
destination_dir=repo_root,
|
||||
copied=copied,
|
||||
)
|
||||
|
||||
|
||||
def refresh_testmon_data(repo_root: Path, cache_dir: Path) -> TestmonCacheResult:
|
||||
local_datafile = _testmon_datafile(repo_root)
|
||||
|
||||
if not local_datafile.exists():
|
||||
raise FileNotFoundError(
|
||||
f"No local pytest-testmon data at {local_datafile}; run tests first."
|
||||
)
|
||||
|
||||
cache_parent = cache_dir.parent
|
||||
cache_parent.mkdir(parents=True, exist_ok=True)
|
||||
temp_dir = Path(tempfile.mkdtemp(prefix=f".{cache_dir.name}.", dir=cache_parent))
|
||||
backup_dir = cache_parent / f".{cache_dir.name}.previous-{os.getpid()}"
|
||||
copied: tuple[Path, ...] = ()
|
||||
|
||||
try:
|
||||
copied = _copy_testmon_files(repo_root, temp_dir)
|
||||
|
||||
_remove_path(backup_dir)
|
||||
if cache_dir.exists():
|
||||
cache_dir.rename(backup_dir)
|
||||
|
||||
try:
|
||||
temp_dir.rename(cache_dir)
|
||||
except Exception:
|
||||
if backup_dir.exists() and not cache_dir.exists():
|
||||
backup_dir.rename(cache_dir)
|
||||
raise
|
||||
finally:
|
||||
_remove_path(temp_dir)
|
||||
_remove_path(backup_dir)
|
||||
|
||||
return TestmonCacheResult(
|
||||
status="refreshed",
|
||||
source_dir=repo_root,
|
||||
destination_dir=cache_dir,
|
||||
copied=tuple(cache_dir / path.name for path in copied),
|
||||
)
|
||||
|
||||
|
||||
def _print_seed_result(result: TestmonCacheResult) -> None:
|
||||
if result.status == "seeded":
|
||||
print(f"Seeded pytest-testmon data from {result.source_dir} into {result.destination_dir}")
|
||||
elif result.status == "exists":
|
||||
print(f"Local pytest-testmon data already exists at {result.destination_dir}")
|
||||
elif result.status == "missing":
|
||||
print(
|
||||
f"No shared pytest-testmon baseline at {result.source_dir}; "
|
||||
"run `just testmon-refresh` after a full backend test run to create one."
|
||||
)
|
||||
else:
|
||||
raise ValueError(f"Unexpected seed result: {result.status}")
|
||||
|
||||
|
||||
def _print_refresh_result(result: TestmonCacheResult) -> None:
|
||||
print(f"Published pytest-testmon data from {result.source_dir} to {result.destination_dir}")
|
||||
|
||||
|
||||
def _print_status(repo_root: Path, cache_dir: Path) -> None:
|
||||
print(f"Repo root: {repo_root}")
|
||||
print(f"Worktree data: {_testmon_datafile(repo_root)}")
|
||||
print(f"Shared cache: {_testmon_datafile(cache_dir)}")
|
||||
print(f"Worktree ready: {_testmon_datafile(repo_root).exists()}")
|
||||
print(f"Cache ready: {_testmon_datafile(cache_dir).exists()}")
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument(
|
||||
"--repo-root",
|
||||
type=Path,
|
||||
help="Repository root to operate on (default: git rev-parse --show-toplevel)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--cache-dir",
|
||||
type=Path,
|
||||
help=(
|
||||
"Shared testmon cache directory "
|
||||
f"(default: ${TESTMON_CACHE_ENV} or <git-common-dir>/testmon-cache/main)"
|
||||
),
|
||||
)
|
||||
|
||||
subparsers = parser.add_subparsers(dest="command", required=True)
|
||||
subparsers.add_parser("seed", help="Copy shared testmon data into this worktree if missing")
|
||||
subparsers.add_parser("refresh", help="Publish this worktree's testmon data to the cache")
|
||||
subparsers.add_parser("status", help="Show local and shared testmon data paths")
|
||||
return parser
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = build_parser()
|
||||
args = parser.parse_args(argv)
|
||||
repo_root = resolve_repo_root(args.repo_root)
|
||||
cache_dir = resolve_cache_dir(repo_root, args.cache_dir)
|
||||
|
||||
if args.command == "seed":
|
||||
_print_seed_result(seed_testmon_data(repo_root=repo_root, cache_dir=cache_dir))
|
||||
return 0
|
||||
|
||||
if args.command == "refresh":
|
||||
_print_refresh_result(refresh_testmon_data(repo_root=repo_root, cache_dir=cache_dir))
|
||||
return 0
|
||||
|
||||
if args.command == "status":
|
||||
_print_status(repo_root=repo_root, cache_dir=cache_dir)
|
||||
return 0
|
||||
|
||||
parser.error(f"Unknown command: {args.command}")
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -99,7 +99,7 @@ def set_package_version(data: dict[str, Any], version: str) -> None:
|
||||
# Version scopes. The two groups map to the two distribution tracks:
|
||||
# core — the Python package and its MCP registry manifest
|
||||
# packages — the host-native agent artifacts (Claude Code plugin + marketplaces,
|
||||
# Hermes, OpenClaw). These are the "plugin/agent artifacts."
|
||||
# Codex plugin, Hermes, OpenClaw). These are the "plugin/agent artifacts."
|
||||
# `all` writes both. Lockstep releases use `all`; targeted fixes can use one group.
|
||||
SCOPES = ("all", "core", "packages")
|
||||
|
||||
@@ -134,6 +134,11 @@ def _update_packages(version: str, *, dry_run: bool) -> None:
|
||||
lambda data: set_claude_marketplace_version(data, version),
|
||||
dry_run=dry_run,
|
||||
)
|
||||
update_json(
|
||||
"plugins/codex/.codex-plugin/plugin.json",
|
||||
lambda data: set_package_version(data, npm_package_version(version)),
|
||||
dry_run=dry_run,
|
||||
)
|
||||
update_text(
|
||||
"integrations/hermes/plugin.yaml",
|
||||
r"^version:\s*.*$",
|
||||
@@ -174,7 +179,7 @@ def main() -> None:
|
||||
choices=SCOPES,
|
||||
default="all",
|
||||
help="Which artifacts to update: all (default), core (Python + server.json), "
|
||||
"or packages (Claude Code plugin, marketplaces, Hermes, OpenClaw)",
|
||||
"or packages (Claude Code plugin, Codex plugin, marketplaces, Hermes, OpenClaw)",
|
||||
)
|
||||
parser.add_argument("--dry-run", action="store_true", help="Preview changes without writing")
|
||||
args = parser.parse_args()
|
||||
|
||||
@@ -4,9 +4,31 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
PLAIN_SCALAR_MAPPING_VALUE = re.compile(r":(?:\s|$)")
|
||||
|
||||
|
||||
def strip_matching_quotes(value: str) -> str:
|
||||
if len(value) >= 2 and value[0] == value[-1] and value[0] in {'"', "'"}:
|
||||
return value[1:-1]
|
||||
return value
|
||||
|
||||
|
||||
def validate_plain_scalar(path: Path, line_number: int, key: str, value: str) -> None:
|
||||
"""Catch invalid plain-scalar YAML that Codex rejects while loading skills."""
|
||||
stripped = value.strip()
|
||||
if not stripped or stripped[0] in {'"', "'"} or stripped in {"|", ">", "|-", ">-", "|+", ">+"}:
|
||||
return
|
||||
if PLAIN_SCALAR_MAPPING_VALUE.search(stripped):
|
||||
raise SystemExit(
|
||||
f"{path}:{line_number}: invalid YAML frontmatter for {key!r}: "
|
||||
"unquoted ':' followed by whitespace; quote the value"
|
||||
)
|
||||
|
||||
|
||||
def parse_frontmatter(path: Path) -> dict[str, str]:
|
||||
"""Extract top-level frontmatter keys from a Markdown file.
|
||||
|
||||
@@ -15,14 +37,15 @@ def parse_frontmatter(path: Path) -> dict[str, str]:
|
||||
skipped, so nested blocks (a schema note's `schema:`/`settings:` children) can't
|
||||
overwrite a top-level key like `type` or `entity` via last-write-wins. It does
|
||||
not interpret block scalars or multi-line values; callers rely on single-line
|
||||
top-level fields (name, description, type, entity).
|
||||
top-level fields (name, description, type, entity). Keep the Codex-facing YAML
|
||||
guard here dependency-free so package checks work under bare `python3`.
|
||||
"""
|
||||
lines = path.read_text().splitlines()
|
||||
if not lines or lines[0] != "---":
|
||||
raise SystemExit(f"{path}: missing YAML frontmatter")
|
||||
|
||||
frontmatter: dict[str, str] = {}
|
||||
for line in lines[1:]:
|
||||
for line_number, line in enumerate(lines[1:], start=2):
|
||||
if line == "---":
|
||||
break
|
||||
if line[:1] in (" ", "\t"): # nested key — not a top-level field
|
||||
@@ -30,7 +53,10 @@ def parse_frontmatter(path: Path) -> dict[str, str]:
|
||||
if ":" not in line:
|
||||
continue
|
||||
key, value = line.split(":", 1)
|
||||
frontmatter[key.strip()] = value.strip().strip('"')
|
||||
key = key.strip()
|
||||
value = value.strip()
|
||||
validate_plain_scalar(path, line_number, key, value)
|
||||
frontmatter[key] = strip_matching_quotes(value)
|
||||
else:
|
||||
raise SystemExit(f"{path}: unclosed YAML frontmatter")
|
||||
|
||||
|
||||
@@ -54,13 +54,13 @@ Users can install or update skills with the [Skills CLI](https://github.com/verc
|
||||
|
||||
```bash
|
||||
# Install all skills
|
||||
npx skills add basicmachines-co/basic-memory --path skills
|
||||
npx skills add basicmachines-co/basic-memory/skills
|
||||
|
||||
# Install a specific skill
|
||||
npx skills add basicmachines-co/basic-memory --path skills --skill memory-tasks
|
||||
npx skills add basicmachines-co/basic-memory/skills --skill memory-tasks
|
||||
|
||||
# Install for a specific agent
|
||||
npx skills add basicmachines-co/basic-memory --path skills --agent claude
|
||||
npx skills add basicmachines-co/basic-memory/skills --agent claude
|
||||
```
|
||||
|
||||
## Adding a New Skill
|
||||
|
||||
+5
-5
@@ -47,16 +47,16 @@ Install or update skills using the [Skills CLI](https://github.com/vercel-labs/s
|
||||
|
||||
```bash
|
||||
# Install all skills
|
||||
npx skills add basicmachines-co/basic-memory --path skills
|
||||
npx skills add basicmachines-co/basic-memory/skills
|
||||
|
||||
# Install a specific skill
|
||||
npx skills add basicmachines-co/basic-memory --path skills --skill memory-tasks
|
||||
npx skills add basicmachines-co/basic-memory/skills --skill memory-tasks
|
||||
|
||||
# Install all skills for a specific agent
|
||||
npx skills add basicmachines-co/basic-memory --path skills --agent claude
|
||||
npx skills add basicmachines-co/basic-memory/skills --agent claude
|
||||
|
||||
# List available skills without installing
|
||||
npx skills add basicmachines-co/basic-memory --path skills --list
|
||||
npx skills add basicmachines-co/basic-memory/skills --list
|
||||
|
||||
# Check for updates
|
||||
npx skills check
|
||||
@@ -67,7 +67,7 @@ npx skills update
|
||||
|
||||
Skills are installed to your agent's skills directory (e.g., `~/.claude/skills/` for Claude Code global, or `.claude/skills/` for project-scoped).
|
||||
|
||||
If your installed Skills CLI does not support `--path`, copy the `memory-*` directories manually for now. Phase 2 will add a first-class Codex/package install path.
|
||||
If your installed Skills CLI cannot load `basicmachines-co/basic-memory/skills`, update the CLI or copy the `memory-*` directories manually.
|
||||
|
||||
### Claude Desktop (claude.ai)
|
||||
|
||||
|
||||
@@ -2,8 +2,11 @@
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
from contextlib import suppress
|
||||
from logging.config import fileConfig
|
||||
|
||||
from loguru import logger
|
||||
|
||||
# Allow nested event loops (needed for pytest-asyncio and other async contexts)
|
||||
# Note: nest_asyncio doesn't work with uvloop or Python 3.14+, so we handle those cases separately
|
||||
import sys
|
||||
@@ -13,9 +16,15 @@ if sys.version_info < (3, 14):
|
||||
import nest_asyncio
|
||||
|
||||
nest_asyncio.apply()
|
||||
except (ImportError, ValueError):
|
||||
# nest_asyncio not available or can't patch this loop type (e.g., uvloop)
|
||||
pass
|
||||
except (ImportError, ValueError) as exc:
|
||||
# Trigger: nest_asyncio is absent (ImportError) or refuses to patch the
|
||||
# running loop (ValueError, e.g. the uvloop policy installed for the
|
||||
# Postgres backend - #831/#877).
|
||||
# Why: the uvloop ValueError is now an *expected* path on every Postgres
|
||||
# startup, so swallowing it silently hides a routine branch.
|
||||
# Outcome: log at DEBUG (observable, not noisy) and fall through to the
|
||||
# thread-based migration fallback.
|
||||
logger.debug(f"nest_asyncio not applied ({exc!r}); using thread-based migration fallback")
|
||||
# For Python 3.14+, we rely on the thread-based fallback in run_migrations_online()
|
||||
|
||||
from sqlalchemy import engine_from_config, pool
|
||||
@@ -115,7 +124,15 @@ async def run_async_migrations(connectable):
|
||||
"""Run migrations asynchronously with AsyncEngine."""
|
||||
async with connectable.connect() as connection:
|
||||
await connection.run_sync(do_run_migrations)
|
||||
await connectable.dispose()
|
||||
# Trigger: startup migrations on the asyncpg backend dispose the engine
|
||||
# while the event loop may be tearing down around them.
|
||||
# Why: that race surfaces "IndexError: pop from an empty deque" from
|
||||
# base_events._run_once (#831/#877); shielding lets dispose finish atomically
|
||||
# and suppressing CancelledError keeps a cancelled teardown from re-raising it.
|
||||
# Outcome: the migration engine always disposes cleanly. (uvloop is the
|
||||
# structural fix for the race; this hardens the teardown path.)
|
||||
with suppress(asyncio.CancelledError):
|
||||
await asyncio.shield(connectable.dispose())
|
||||
|
||||
|
||||
def _run_async_migrations_with_asyncio_run(connectable) -> None:
|
||||
|
||||
@@ -64,7 +64,9 @@ async def search(
|
||||
or query.permalink
|
||||
or query.permalink_match
|
||||
),
|
||||
has_filters=bool(query.note_types or query.entity_types or query.metadata_filters),
|
||||
has_filters=bool(
|
||||
query.note_types or query.entity_types or query.categories or query.metadata_filters
|
||||
),
|
||||
):
|
||||
offset = (page - 1) * page_size
|
||||
exact_count_available = query.retrieval_mode == SearchRetrievalMode.FTS
|
||||
|
||||
@@ -18,7 +18,9 @@ from basic_memory.services.context_service import (
|
||||
|
||||
|
||||
class EntityBatchLookup(Protocol):
|
||||
async def find_by_ids_for_hydration(self, ids: List[int]) -> Sequence[Any]: ...
|
||||
async def find_by_ids_for_hydration(
|
||||
self, ids: List[int], *, include_cross_project: bool = False
|
||||
) -> Sequence[Any]: ...
|
||||
|
||||
|
||||
class EntityServiceBatchLookup(Protocol):
|
||||
@@ -88,7 +90,7 @@ async def to_graph_context(
|
||||
result_count=len(entity_ids_needed),
|
||||
):
|
||||
entities = await entity_repository.find_by_ids_for_hydration(
|
||||
list(entity_ids_needed)
|
||||
list(entity_ids_needed), include_cross_project=True
|
||||
)
|
||||
for e in entities:
|
||||
entity_title_lookup[e.id] = e.title
|
||||
|
||||
@@ -57,6 +57,14 @@ def app_callback(
|
||||
container = CliContainer.create()
|
||||
set_container(container)
|
||||
|
||||
# Trigger: Postgres backend resolved at CLI startup, before any asyncio.run().
|
||||
# Why: uvloop must own the event-loop policy before the loop is created so the
|
||||
# asyncpg engine-dispose race (#831/#877) cannot fire. No-op for SQLite.
|
||||
# Outcome: subsequent asyncio.run() calls in CLI commands use uvloop on Postgres.
|
||||
from basic_memory.db import maybe_install_uvloop
|
||||
|
||||
maybe_install_uvloop(container.config)
|
||||
|
||||
# Trigger: first-run init confirmation before command output.
|
||||
# Why: informational "initialized" message belongs above command results, not in the upsell panel.
|
||||
# Outcome: one-time plain line printed before the subcommand runs.
|
||||
@@ -86,6 +94,7 @@ def app_callback(
|
||||
"reindex",
|
||||
"update",
|
||||
"watch",
|
||||
"workspace",
|
||||
}
|
||||
if (
|
||||
not version
|
||||
|
||||
@@ -9,6 +9,7 @@ from . import (
|
||||
format,
|
||||
schema,
|
||||
update,
|
||||
workspace,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
@@ -27,4 +28,5 @@ __all__ = [
|
||||
"format",
|
||||
"schema",
|
||||
"update",
|
||||
"workspace",
|
||||
]
|
||||
|
||||
@@ -32,14 +32,33 @@ def _rclone_exclude_filters(pattern: str) -> list[str]:
|
||||
return [f"- {path_pattern}", f"- {path_pattern}/**"]
|
||||
|
||||
|
||||
async def get_mount_info() -> TenantMountInfo:
|
||||
"""Get current tenant information from cloud API."""
|
||||
def _workspace_id_header(workspace_id: str | None) -> dict[str, str]:
|
||||
"""Header that routes a /tenant/mount/* request to a specific tenant.
|
||||
|
||||
The mount endpoints resolve the workspace from X-Workspace-ID (validating
|
||||
membership + subscription) and fall back to the user's default tenant when
|
||||
it is absent — so omitting it preserves the original default-tenant behavior.
|
||||
"""
|
||||
return {"X-Workspace-ID": workspace_id} if workspace_id else {}
|
||||
|
||||
|
||||
async def get_mount_info(*, workspace_id: str | None = None) -> TenantMountInfo:
|
||||
"""Get tenant mount info (bucket name + tenant id) from the cloud API.
|
||||
|
||||
Args:
|
||||
workspace_id: Tenant id of the target workspace. When omitted, the API
|
||||
uses the authenticated user's default tenant.
|
||||
"""
|
||||
try:
|
||||
config_manager = ConfigManager()
|
||||
config = config_manager.config
|
||||
host_url = config.cloud_host.rstrip("/")
|
||||
|
||||
response = await make_api_request(method="GET", url=f"{host_url}/tenant/mount/info")
|
||||
response = await make_api_request(
|
||||
method="GET",
|
||||
url=f"{host_url}/tenant/mount/info",
|
||||
headers=_workspace_id_header(workspace_id),
|
||||
)
|
||||
|
||||
return TenantMountInfo.model_validate(response.json())
|
||||
except Exception as e:
|
||||
@@ -47,13 +66,24 @@ async def get_mount_info() -> TenantMountInfo:
|
||||
|
||||
|
||||
async def generate_mount_credentials(tenant_id: str) -> MountCredentials:
|
||||
"""Generate scoped credentials for syncing."""
|
||||
"""Generate scoped S3 credentials for syncing a specific tenant's bucket.
|
||||
|
||||
Args:
|
||||
tenant_id: Tenant id whose bucket-scoped credentials to mint. Routed via
|
||||
X-Workspace-ID so team workspaces get their own bucket's credentials.
|
||||
"""
|
||||
try:
|
||||
config_manager = ConfigManager()
|
||||
config = config_manager.config
|
||||
host_url = config.cloud_host.rstrip("/")
|
||||
|
||||
response = await make_api_request(method="POST", url=f"{host_url}/tenant/mount/credentials")
|
||||
# The mount endpoints resolve X-Workspace-ID by matching the workspace's
|
||||
# tenant_id, so passing a tenant_id here is the correct routing key.
|
||||
response = await make_api_request(
|
||||
method="POST",
|
||||
url=f"{host_url}/tenant/mount/credentials",
|
||||
headers=_workspace_id_header(tenant_id),
|
||||
)
|
||||
|
||||
return MountCredentials.model_validate(response.json())
|
||||
except Exception as e:
|
||||
|
||||
@@ -26,15 +26,52 @@ from basic_memory.cli.commands.cloud.bisync_commands import (
|
||||
generate_mount_credentials,
|
||||
get_mount_info,
|
||||
)
|
||||
from basic_memory.cli.commands.cloud.rclone_config import configure_rclone_remote
|
||||
from basic_memory.cli.commands.cloud.rclone_config import (
|
||||
configure_rclone_remote,
|
||||
rclone_remote_exists,
|
||||
remote_name_for_workspace,
|
||||
)
|
||||
from basic_memory.cli.commands.cloud.rclone_installer import (
|
||||
RcloneInstallError,
|
||||
install_rclone,
|
||||
)
|
||||
from basic_memory.mcp.project_context import get_available_workspaces
|
||||
from basic_memory.schemas.cloud import (
|
||||
WorkspaceInfo,
|
||||
format_workspace_choices,
|
||||
format_workspace_selection_choices,
|
||||
workspace_matches_exact_identifier,
|
||||
)
|
||||
|
||||
console = Console()
|
||||
|
||||
|
||||
def _resolve_setup_workspace(identifier: str) -> WorkspaceInfo:
|
||||
"""Resolve a workspace identifier (slug, name, or tenant_id) for setup.
|
||||
|
||||
Errors with copyable choices when the identifier matches zero or multiple
|
||||
workspaces, so the user can disambiguate.
|
||||
"""
|
||||
workspaces = run_with_cleanup(get_available_workspaces())
|
||||
if not workspaces:
|
||||
console.print("[red]No accessible cloud workspaces found for this account[/red]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
matches = [ws for ws in workspaces if workspace_matches_exact_identifier(ws, identifier)]
|
||||
if len(matches) == 1:
|
||||
return matches[0]
|
||||
|
||||
if not matches:
|
||||
console.print(f"[red]No workspace matches '{identifier}'[/red]")
|
||||
console.print("\nAvailable workspaces:")
|
||||
console.print(format_workspace_choices(workspaces))
|
||||
else:
|
||||
console.print(f"[red]'{identifier}' matches multiple workspaces[/red]")
|
||||
console.print("\nDisambiguate with the workspace slug or tenant_id:")
|
||||
console.print(format_workspace_selection_choices(matches))
|
||||
raise typer.Exit(1)
|
||||
|
||||
|
||||
@cloud_app.command()
|
||||
def login():
|
||||
"""Authenticate with WorkOS using OAuth Device Authorization flow."""
|
||||
@@ -164,13 +201,28 @@ def status() -> None:
|
||||
|
||||
|
||||
@cloud_app.command("setup")
|
||||
def setup() -> None:
|
||||
def setup(
|
||||
workspace: str | None = typer.Option(
|
||||
None,
|
||||
"--workspace",
|
||||
help="Set up sync for a specific workspace (slug, name, or tenant_id). "
|
||||
"Omit for your default workspace.",
|
||||
),
|
||||
force: bool = typer.Option(
|
||||
False,
|
||||
"--force",
|
||||
help="Reconfigure an rclone remote that already exists (mints new credentials).",
|
||||
),
|
||||
) -> None:
|
||||
"""Set up cloud sync by installing rclone and configuring credentials.
|
||||
|
||||
After setup, use project commands for syncing:
|
||||
bm project add <name> --cloud --local-path ~/projects/<name>
|
||||
bm project bisync --name <name> --resync # First time
|
||||
bm project bisync --name <name> # Subsequent syncs
|
||||
Run once per workspace you sync. The default workspace uses the
|
||||
'basic-memory-cloud' remote; other (e.g. Team) workspaces each get their own
|
||||
tenant-scoped remote, since Tigris credentials are bucket-scoped.
|
||||
|
||||
After setup, use the cloud sync commands:
|
||||
bm cloud pull --name <name> # fetch cloud changes (Team-safe)
|
||||
bm cloud push --name <name> # upload local changes (Team-safe)
|
||||
"""
|
||||
console.print("[bold blue]Basic Memory Cloud Setup[/bold blue]")
|
||||
console.print("Setting up cloud sync with rclone...\n")
|
||||
@@ -180,35 +232,60 @@ def setup() -> None:
|
||||
console.print("[blue]Step 1: Installing rclone...[/blue]")
|
||||
install_rclone()
|
||||
|
||||
# Step 2: Get tenant info
|
||||
# --- Resolve target workspace ---
|
||||
# Trigger: --workspace given. Why: Tigris keys are tenant-scoped, so a
|
||||
# non-default workspace needs its own bucket + remote. Outcome: scope the
|
||||
# mount-info/credentials calls and name the remote after the workspace.
|
||||
if workspace is not None:
|
||||
target = _resolve_setup_workspace(workspace)
|
||||
workspace_id: str | None = target.tenant_id
|
||||
remote_name = remote_name_for_workspace(target.slug, is_default=target.is_default)
|
||||
console.print(f"[dim]Workspace: {target.name} ({target.slug})[/dim]")
|
||||
else:
|
||||
workspace_id = None # default tenant
|
||||
remote_name = remote_name_for_workspace(None, is_default=True)
|
||||
|
||||
# Trigger: the target rclone remote already exists.
|
||||
# Why: re-running setup mints new credentials and overwrites the remote,
|
||||
# which would silently repoint it — e.g. clobbering the shared
|
||||
# basic-memory-cloud remote that served another tenant. Checked BEFORE
|
||||
# minting so an abort wastes no credentials.
|
||||
# Outcome: stop unless the user explicitly opts in with --force.
|
||||
if rclone_remote_exists(remote_name) and not force:
|
||||
console.print(f"[red]rclone remote '{remote_name}' is already configured.[/red]")
|
||||
console.print(
|
||||
"Re-running setup mints new credentials and overwrites it. "
|
||||
"Pass --force to reconfigure."
|
||||
)
|
||||
raise typer.Exit(1)
|
||||
|
||||
# Step 2: Get tenant info (scoped to the target workspace when given)
|
||||
console.print("\n[blue]Step 2: Getting tenant information...[/blue]")
|
||||
tenant_info = run_with_cleanup(get_mount_info())
|
||||
tenant_info = run_with_cleanup(get_mount_info(workspace_id=workspace_id))
|
||||
console.print(f"[green]Found tenant: {tenant_info.tenant_id}[/green]")
|
||||
|
||||
# Step 3: Generate credentials
|
||||
# Step 3: Generate credentials for that tenant's bucket
|
||||
console.print("\n[blue]Step 3: Generating sync credentials...[/blue]")
|
||||
creds = run_with_cleanup(generate_mount_credentials(tenant_info.tenant_id))
|
||||
console.print("[green]Generated secure credentials[/green]")
|
||||
|
||||
# Step 4: Configure rclone remote
|
||||
# Step 4: Configure the tenant's rclone remote
|
||||
console.print("\n[blue]Step 4: Configuring rclone remote...[/blue]")
|
||||
configure_rclone_remote(
|
||||
access_key=creds.access_key,
|
||||
secret_key=creds.secret_key,
|
||||
remote_name=remote_name,
|
||||
)
|
||||
|
||||
console.print("\n[bold green]Cloud setup completed successfully![/bold green]")
|
||||
console.print("\n[bold]Next steps:[/bold]")
|
||||
console.print("1. Add a project with local sync path:")
|
||||
console.print(" bm project add research --cloud --local-path ~/Documents/research")
|
||||
console.print("\n Or configure sync for an existing project:")
|
||||
console.print("1. Configure sync for a project:")
|
||||
console.print(" bm cloud sync-setup research ~/Documents/research")
|
||||
console.print("\n2. Preview the initial sync (recommended):")
|
||||
console.print(" bm project bisync --name research --resync --dry-run")
|
||||
console.print("\n3. If all looks good, run the actual sync:")
|
||||
console.print(" bm project bisync --name research --resync")
|
||||
console.print("\n4. Subsequent syncs (no --resync needed):")
|
||||
console.print(" bm project bisync --name research")
|
||||
console.print("\n2. Preview a pull (recommended):")
|
||||
console.print(" bm cloud pull --name research --dry-run")
|
||||
console.print("\n3. Fetch cloud changes / upload local changes:")
|
||||
console.print(" bm cloud pull --name research")
|
||||
console.print(" bm cloud push --name research")
|
||||
console.print(
|
||||
"\n[dim]Tip: Always use --dry-run first to preview changes before syncing[/dim]"
|
||||
)
|
||||
@@ -216,6 +293,8 @@ def setup() -> None:
|
||||
except (RcloneInstallError, BisyncError, CloudAPIError) as e:
|
||||
console.print(f"\n[red]Setup failed: {e}[/red]")
|
||||
raise typer.Exit(1)
|
||||
except typer.Exit:
|
||||
raise
|
||||
except Exception as e:
|
||||
console.print(f"\n[red]Unexpected error during setup: {e}[/red]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
@@ -7,6 +7,7 @@ they are cloud-specific operations.
|
||||
|
||||
import os
|
||||
from datetime import datetime
|
||||
from enum import Enum
|
||||
|
||||
import typer
|
||||
from rich.console import Console
|
||||
@@ -16,10 +17,19 @@ from basic_memory.cli.commands.cloud.bisync_commands import get_mount_info
|
||||
from basic_memory.cli.commands.cloud.rclone_commands import (
|
||||
RcloneError,
|
||||
SyncProject,
|
||||
TransferDirection,
|
||||
TransferPlan,
|
||||
get_project_bisync_state,
|
||||
project_bisync,
|
||||
project_check,
|
||||
project_diff,
|
||||
project_sync,
|
||||
project_transfer,
|
||||
)
|
||||
from basic_memory.cli.commands.cloud.rclone_config import (
|
||||
DEFAULT_RCLONE_REMOTE,
|
||||
rclone_remote_exists,
|
||||
remote_name_for_workspace,
|
||||
)
|
||||
from basic_memory.cli.commands.command_utils import run_with_cleanup
|
||||
from basic_memory.cli.commands.routing import force_routing
|
||||
@@ -27,7 +37,12 @@ from basic_memory.config import BasicMemoryConfig, ConfigManager, ProjectEntry
|
||||
from basic_memory.mcp.async_client import get_client
|
||||
from basic_memory.mcp.clients import ProjectClient
|
||||
from basic_memory.mcp.project_context import get_available_workspaces
|
||||
from basic_memory.schemas.cloud import WorkspaceInfo
|
||||
from basic_memory.schemas.cloud import (
|
||||
WorkspaceInfo,
|
||||
format_workspace_choices,
|
||||
format_workspace_selection_choices,
|
||||
workspace_matches_exact_identifier,
|
||||
)
|
||||
from basic_memory.schemas.project_info import ProjectItem
|
||||
from basic_memory.utils import generate_permalink, normalize_project_path
|
||||
|
||||
@@ -35,9 +50,34 @@ console = Console()
|
||||
|
||||
TEAM_WORKSPACE_BISYNC_UNSUPPORTED = (
|
||||
"The bisync operation is only supported on Personal workspaces.\n"
|
||||
"Use `bm cloud sync --name {name}` instead."
|
||||
"Use `bm cloud pull --name {name}` / `bm cloud push --name {name}` instead."
|
||||
)
|
||||
|
||||
TEAM_WORKSPACE_SYNC_UNSUPPORTED = (
|
||||
"The sync operation mirrors local onto the shared bucket and can delete a "
|
||||
"teammate's files, so it is only supported on Personal workspaces.\n"
|
||||
"Use `bm cloud pull --name {name}` (fetch) / `bm cloud push --name {name}` "
|
||||
"(additive upload) instead."
|
||||
)
|
||||
|
||||
|
||||
class ConflictStrategy(str, Enum):
|
||||
"""How push/pull resolves files that differ on both sides.
|
||||
|
||||
Default is ``fail``: surface the conflicts and abort before transferring,
|
||||
leaving the user to re-run with an explicit resolution — like git refusing
|
||||
to clobber local changes.
|
||||
|
||||
This is the Typer-facing enum; the engine in ``rclone_commands`` accepts the
|
||||
same values as a ``ConflictStrategy`` Literal. ``_run_directional_transfer``
|
||||
bridges the two by passing ``on_conflict.value``. Keep the values in sync.
|
||||
"""
|
||||
|
||||
fail = "fail"
|
||||
keep_local = "keep-local"
|
||||
keep_cloud = "keep-cloud"
|
||||
keep_both = "keep-both"
|
||||
|
||||
|
||||
# --- Shared helpers ---
|
||||
|
||||
@@ -59,12 +99,41 @@ def _require_cloud_credentials(config: BasicMemoryConfig) -> None:
|
||||
raise typer.Exit(1)
|
||||
|
||||
|
||||
async def _get_workspace_for_project(name: str, config: BasicMemoryConfig) -> WorkspaceInfo:
|
||||
"""Resolve the cloud workspace targeted by a project-scoped sync command."""
|
||||
async def _get_workspace_for_project(
|
||||
name: str,
|
||||
config: BasicMemoryConfig,
|
||||
*,
|
||||
workspace_override: str | None = None,
|
||||
) -> WorkspaceInfo:
|
||||
"""Resolve the cloud workspace targeted by a project-scoped sync command.
|
||||
|
||||
``workspace_override`` (a slug, name, or tenant_id, e.g. from ``--workspace``)
|
||||
takes precedence over config, letting the user disambiguate a project name
|
||||
that exists in more than one workspace.
|
||||
"""
|
||||
workspaces = await get_available_workspaces()
|
||||
if not workspaces:
|
||||
raise ValueError("No accessible cloud workspaces found for this account")
|
||||
|
||||
# An explicit override wins over config — this is how the user disambiguates.
|
||||
if workspace_override is not None:
|
||||
matches = [
|
||||
item
|
||||
for item in workspaces
|
||||
if workspace_matches_exact_identifier(item, workspace_override)
|
||||
]
|
||||
if len(matches) == 1:
|
||||
return matches[0]
|
||||
if not matches:
|
||||
raise ValueError(
|
||||
f"No accessible workspace matches '{workspace_override}'.\n"
|
||||
f"{format_workspace_choices(workspaces)}"
|
||||
)
|
||||
raise ValueError(
|
||||
f"'{workspace_override}' matches multiple workspaces; use a slug or tenant_id:\n"
|
||||
f"{format_workspace_selection_choices(matches)}"
|
||||
)
|
||||
|
||||
entry = config.projects.get(name)
|
||||
workspace_id = entry.workspace_id if entry and entry.workspace_id else config.default_workspace
|
||||
if workspace_id:
|
||||
@@ -92,8 +161,18 @@ async def _get_workspace_for_project(name: str, config: BasicMemoryConfig) -> Wo
|
||||
)
|
||||
|
||||
|
||||
def _require_personal_workspace(name: str, config: BasicMemoryConfig) -> WorkspaceInfo:
|
||||
"""Exit before bisync work when the target workspace is not personal."""
|
||||
def _require_personal_workspace(
|
||||
name: str,
|
||||
config: BasicMemoryConfig,
|
||||
*,
|
||||
unsupported_message: str = TEAM_WORKSPACE_BISYNC_UNSUPPORTED,
|
||||
) -> WorkspaceInfo:
|
||||
"""Exit before mirror work when the target workspace is not personal.
|
||||
|
||||
Used to gate the destructive mirror operations (`sync`, `bisync`) to
|
||||
Personal workspaces. ``unsupported_message`` lets each command point Team
|
||||
users at the right Team-safe alternative.
|
||||
"""
|
||||
try:
|
||||
workspace = run_with_cleanup(_get_workspace_for_project(name, config))
|
||||
except Exception as exc:
|
||||
@@ -101,15 +180,21 @@ def _require_personal_workspace(name: str, config: BasicMemoryConfig) -> Workspa
|
||||
raise typer.Exit(1)
|
||||
|
||||
if workspace.workspace_type != "personal":
|
||||
console.print(f"[red]{TEAM_WORKSPACE_BISYNC_UNSUPPORTED.format(name=name)}[/red]")
|
||||
console.print(f"[red]{unsupported_message.format(name=name)}[/red]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
return workspace
|
||||
|
||||
|
||||
async def _get_cloud_project(name: str) -> ProjectItem | None:
|
||||
"""Fetch a project by name from the cloud API."""
|
||||
async with get_client(project_name=name) as client:
|
||||
async def _get_cloud_project(name: str, *, workspace_id: str | None = None) -> ProjectItem | None:
|
||||
"""Fetch a project by name from the cloud API.
|
||||
|
||||
``workspace_id`` routes the lookup to a specific tenant so the project
|
||||
metadata comes from the same workspace the transfer targets (otherwise
|
||||
get_client would resolve the workspace from config/default and could read a
|
||||
different tenant — see #920 review).
|
||||
"""
|
||||
async with get_client(project_name=name, workspace=workspace_id) as client:
|
||||
projects_list = await ProjectClient(client).list_projects()
|
||||
for proj in projects_list.projects:
|
||||
if generate_permalink(proj.name) == generate_permalink(name):
|
||||
@@ -118,10 +203,17 @@ async def _get_cloud_project(name: str) -> ProjectItem | None:
|
||||
|
||||
|
||||
def _get_sync_project(
|
||||
name: str, config: BasicMemoryConfig, project_data: ProjectItem
|
||||
name: str,
|
||||
config: BasicMemoryConfig,
|
||||
project_data: ProjectItem,
|
||||
*,
|
||||
remote_name: str = DEFAULT_RCLONE_REMOTE,
|
||||
) -> tuple[SyncProject, str | None]:
|
||||
"""Build a SyncProject and resolve local_sync_path from config.
|
||||
|
||||
``remote_name`` selects which tenant-scoped rclone remote the project routes
|
||||
through (default tenant vs a team workspace remote).
|
||||
|
||||
Returns (sync_project, local_sync_path). Exits if no local_sync_path configured.
|
||||
"""
|
||||
sync_entry = config.projects.get(name)
|
||||
@@ -137,6 +229,7 @@ def _get_sync_project(
|
||||
name=project_data.name,
|
||||
path=normalize_project_path(project_data.path),
|
||||
local_sync_path=local_sync_path,
|
||||
remote_name=remote_name,
|
||||
)
|
||||
return sync_project, local_sync_path
|
||||
|
||||
@@ -146,11 +239,15 @@ def _get_sync_project(
|
||||
|
||||
@cloud_app.command("sync")
|
||||
def sync_project_command(
|
||||
name: str = typer.Option(..., "--name", help="Project name to sync"),
|
||||
name: str = typer.Option(..., "--name", "--project", help="Project name to sync"),
|
||||
dry_run: bool = typer.Option(False, "--dry-run", help="Preview changes without syncing"),
|
||||
verbose: bool = typer.Option(False, "--verbose", "-v", help="Show detailed output"),
|
||||
) -> None:
|
||||
"""One-way sync: local -> cloud (make cloud identical to local).
|
||||
"""One-way mirror: local -> cloud (make cloud identical to local).
|
||||
|
||||
Personal workspaces only. This deletes cloud files not present locally, so
|
||||
on Team workspaces use `bm cloud push` (additive upload) / `bm cloud pull`
|
||||
(fetch) instead.
|
||||
|
||||
Example:
|
||||
bm cloud sync --name research
|
||||
@@ -158,9 +255,13 @@ def sync_project_command(
|
||||
"""
|
||||
config = ConfigManager().config
|
||||
_require_cloud_credentials(config)
|
||||
_require_personal_workspace(name, config, unsupported_message=TEAM_WORKSPACE_SYNC_UNSUPPORTED)
|
||||
|
||||
try:
|
||||
# Get tenant info for bucket name
|
||||
# Get tenant info for bucket name.
|
||||
# TODO(#919): scope to the project's workspace like push/pull. Safe for now
|
||||
# because these mirror commands are gated to the (default-tenant) Personal
|
||||
# workspace, so the default mount info is correct.
|
||||
tenant_info = run_with_cleanup(get_mount_info())
|
||||
bucket_name = tenant_info.bucket_name
|
||||
|
||||
@@ -191,9 +292,222 @@ def sync_project_command(
|
||||
raise typer.Exit(1)
|
||||
|
||||
|
||||
def _print_conflict_abort(name: str, direction: TransferDirection, plan: TransferPlan) -> None:
|
||||
"""Explain a conflict abort and how to resolve it (git-pull style)."""
|
||||
console.print(
|
||||
f"[red]{direction.capitalize()} aborted: {len(plan.conflicts)} file(s) differ between "
|
||||
f"local and cloud.[/red]"
|
||||
)
|
||||
for path in plan.conflicts:
|
||||
console.print(f" [yellow]*[/yellow] {path}")
|
||||
console.print("\nRe-run with one of:")
|
||||
console.print(" [dim]--on-conflict keep-cloud[/dim] take the cloud version")
|
||||
console.print(" [dim]--on-conflict keep-local[/dim] keep your local version")
|
||||
console.print(
|
||||
" [dim]--on-conflict keep-both[/dim] keep both (writes <name>.conflict-<date>)"
|
||||
)
|
||||
|
||||
|
||||
def _run_directional_transfer(
|
||||
name: str,
|
||||
direction: TransferDirection,
|
||||
*,
|
||||
on_conflict: ConflictStrategy,
|
||||
dry_run: bool,
|
||||
verbose: bool,
|
||||
workspace: str | None = None,
|
||||
) -> None:
|
||||
"""Shared orchestration for `bm cloud push` / `bm cloud pull`.
|
||||
|
||||
Detects conflicts first, then aborts (the default) or applies the chosen
|
||||
resolution. Uses additive `rclone copy`, so it never deletes on the
|
||||
destination — safe for Team workspaces and therefore not gated.
|
||||
|
||||
Routes through the resolved workspace's own tenant-scoped rclone remote, so a
|
||||
Team project reads/writes the right bucket (see #919).
|
||||
"""
|
||||
config = ConfigManager().config
|
||||
_require_cloud_credentials(config)
|
||||
|
||||
try:
|
||||
# --- Resolve the target workspace and its tenant-scoped remote ---
|
||||
# Tigris credentials are bucket/tenant-scoped, so each workspace has its
|
||||
# own rclone remote. Resolve which workspace this project belongs to
|
||||
# (config or --workspace override) before touching any bucket.
|
||||
try:
|
||||
target_workspace = run_with_cleanup(
|
||||
_get_workspace_for_project(name, config, workspace_override=workspace)
|
||||
)
|
||||
except Exception as exc:
|
||||
console.print(f"[red]Error resolving workspace for project '{name}': {exc}[/red]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
remote_name = remote_name_for_workspace(
|
||||
target_workspace.slug, is_default=target_workspace.is_default
|
||||
)
|
||||
|
||||
# Trigger: the workspace's remote has not been configured yet.
|
||||
# Why: provisioning mints tenant-scoped credentials and must be explicit
|
||||
# (no surprise key generation); push/pull only transfer.
|
||||
# Outcome: stop with the exact setup command for this workspace.
|
||||
if not rclone_remote_exists(remote_name):
|
||||
setup_target = (
|
||||
"" if target_workspace.is_default else f" --workspace {target_workspace.slug}"
|
||||
)
|
||||
console.print(f"[red]Workspace '{target_workspace.slug}' is not set up for sync.[/red]")
|
||||
console.print(f"\nRun: bm cloud setup{setup_target}")
|
||||
raise typer.Exit(1)
|
||||
|
||||
# Get tenant info for bucket name, scoped to the resolved workspace
|
||||
tenant_info = run_with_cleanup(get_mount_info(workspace_id=target_workspace.tenant_id))
|
||||
bucket_name = tenant_info.bucket_name
|
||||
|
||||
# Get project info from the same workspace we resolved above, so the
|
||||
# project path and the bucket/remote all refer to one tenant.
|
||||
with force_routing(cloud=True):
|
||||
project_data = run_with_cleanup(
|
||||
_get_cloud_project(name, workspace_id=target_workspace.tenant_id)
|
||||
)
|
||||
if not project_data:
|
||||
console.print(f"[red]Error: Project '{name}' not found[/red]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
sync_project, _ = _get_sync_project(name, config, project_data, remote_name=remote_name)
|
||||
|
||||
# --- Detect before transferring ---
|
||||
plan = project_diff(sync_project, bucket_name, direction)
|
||||
|
||||
# Trigger: rclone could not read/hash some files.
|
||||
# Why: comparing is the whole basis for a safe transfer — never guess.
|
||||
# Outcome: abort before moving any bytes.
|
||||
if plan.errors:
|
||||
console.print(
|
||||
f"[red]{direction.capitalize()} aborted: rclone could not compare "
|
||||
f"{len(plan.errors)} file(s)[/red]"
|
||||
)
|
||||
for path in plan.errors:
|
||||
console.print(f" [red]![/red] {path}")
|
||||
raise typer.Exit(1)
|
||||
|
||||
# Trigger: files differ on both sides and the user chose no resolution.
|
||||
# Why: "no surprises" — never silently pick a winner.
|
||||
# Outcome: list the conflicts and exit, like git refusing to clobber.
|
||||
if plan.conflicts and on_conflict is ConflictStrategy.fail:
|
||||
_print_conflict_abort(name, direction, plan)
|
||||
raise typer.Exit(1)
|
||||
|
||||
# --- Transfer ---
|
||||
arrow = "cloud -> local" if direction == "pull" else "local -> cloud"
|
||||
console.print(f"[blue]{direction.capitalize()} {name} ({arrow})...[/blue]")
|
||||
|
||||
conflict_suffix = datetime.now().strftime("%Y%m%d-%H%M%S")
|
||||
success = project_transfer(
|
||||
sync_project,
|
||||
bucket_name,
|
||||
direction,
|
||||
plan,
|
||||
strategy=on_conflict.value,
|
||||
conflict_suffix=conflict_suffix,
|
||||
dry_run=dry_run,
|
||||
verbose=verbose,
|
||||
)
|
||||
|
||||
if not success:
|
||||
console.print(f"[red]{name} {direction} failed[/red]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
console.print(f"[green]{name} {direction} completed successfully[/green]")
|
||||
|
||||
# Without a sync baseline (see #862) we cannot tell an intentional delete
|
||||
# from a file the other side simply never had, so deletions never sync.
|
||||
if plan.dest_only:
|
||||
kept_on = "local" if direction == "pull" else "cloud"
|
||||
console.print(
|
||||
f"[dim]{len(plan.dest_only)} file(s) exist only on {kept_on} and were left "
|
||||
"untouched (deletions are not propagated).[/dim]"
|
||||
)
|
||||
|
||||
except RcloneError as e:
|
||||
console.print(f"[red]{direction.capitalize()} error: {e}[/red]")
|
||||
raise typer.Exit(1)
|
||||
except typer.Exit:
|
||||
# Already-handled exits (not found, conflicts, errors) propagate cleanly.
|
||||
raise
|
||||
except Exception as e:
|
||||
console.print(f"[red]Error: {e}[/red]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
|
||||
@cloud_app.command("pull")
|
||||
def pull_project_command(
|
||||
name: str = typer.Option(..., "--name", "--project", help="Project name to pull"),
|
||||
on_conflict: ConflictStrategy = typer.Option(
|
||||
ConflictStrategy.fail,
|
||||
"--on-conflict",
|
||||
help="Resolve files that differ on both sides (default: fail and list them)",
|
||||
),
|
||||
workspace: str | None = typer.Option(
|
||||
None,
|
||||
"--workspace",
|
||||
help="Workspace (slug, name, or tenant_id) when the project name is ambiguous",
|
||||
),
|
||||
dry_run: bool = typer.Option(False, "--dry-run", help="Preview changes without pulling"),
|
||||
verbose: bool = typer.Option(False, "--verbose", "-v", help="Show detailed output"),
|
||||
) -> None:
|
||||
"""Fetch cloud changes into local (cloud -> local), git-pull style.
|
||||
|
||||
Additive and Team-safe: downloads new/changed cloud files and never deletes
|
||||
local files. A file that differs on both sides is a conflict; by default
|
||||
pull aborts and lists them. Deletions are not propagated (see #862).
|
||||
|
||||
Examples:
|
||||
bm cloud pull --name research
|
||||
bm cloud pull --name research --dry-run
|
||||
bm cloud pull --name research --on-conflict keep-cloud
|
||||
bm cloud pull --name research --workspace acme
|
||||
"""
|
||||
_run_directional_transfer(
|
||||
name, "pull", on_conflict=on_conflict, dry_run=dry_run, verbose=verbose, workspace=workspace
|
||||
)
|
||||
|
||||
|
||||
@cloud_app.command("push")
|
||||
def push_project_command(
|
||||
name: str = typer.Option(..., "--name", "--project", help="Project name to push"),
|
||||
on_conflict: ConflictStrategy = typer.Option(
|
||||
ConflictStrategy.fail,
|
||||
"--on-conflict",
|
||||
help="Resolve files that differ on both sides (default: fail and list them)",
|
||||
),
|
||||
workspace: str | None = typer.Option(
|
||||
None,
|
||||
"--workspace",
|
||||
help="Workspace (slug, name, or tenant_id) when the project name is ambiguous",
|
||||
),
|
||||
dry_run: bool = typer.Option(False, "--dry-run", help="Preview changes without pushing"),
|
||||
verbose: bool = typer.Option(False, "--verbose", "-v", help="Show detailed output"),
|
||||
) -> None:
|
||||
"""Upload local changes to cloud (local -> cloud), additive and Team-safe.
|
||||
|
||||
Uploads new/changed local files and never deletes cloud files. A file that
|
||||
differs on both sides is a conflict; by default push aborts and lists them
|
||||
(like git rejecting a push when the remote is ahead — pull first). Deletions
|
||||
are not propagated (see #862).
|
||||
|
||||
Examples:
|
||||
bm cloud push --name research
|
||||
bm cloud push --name research --dry-run
|
||||
bm cloud push --name research --on-conflict keep-local
|
||||
bm cloud push --name research --workspace acme
|
||||
"""
|
||||
_run_directional_transfer(
|
||||
name, "push", on_conflict=on_conflict, dry_run=dry_run, verbose=verbose, workspace=workspace
|
||||
)
|
||||
|
||||
|
||||
@cloud_app.command("bisync")
|
||||
def bisync_project_command(
|
||||
name: str = typer.Option(..., "--name", help="Project name to bisync"),
|
||||
name: str = typer.Option(..., "--name", "--project", help="Project name to bisync"),
|
||||
dry_run: bool = typer.Option(False, "--dry-run", help="Preview changes without syncing"),
|
||||
resync: bool = typer.Option(False, "--resync", help="Force new baseline"),
|
||||
verbose: bool = typer.Option(False, "--verbose", "-v", help="Show detailed output"),
|
||||
@@ -210,7 +524,10 @@ def bisync_project_command(
|
||||
_require_personal_workspace(name, config)
|
||||
|
||||
try:
|
||||
# Get tenant info for bucket name
|
||||
# Get tenant info for bucket name.
|
||||
# TODO(#919): scope to the project's workspace like push/pull. Safe for now
|
||||
# because these mirror commands are gated to the (default-tenant) Personal
|
||||
# workspace, so the default mount info is correct.
|
||||
tenant_info = run_with_cleanup(get_mount_info())
|
||||
bucket_name = tenant_info.bucket_name
|
||||
|
||||
@@ -256,7 +573,7 @@ def bisync_project_command(
|
||||
|
||||
@cloud_app.command("check")
|
||||
def check_project_command(
|
||||
name: str = typer.Option(..., "--name", help="Project name to check"),
|
||||
name: str = typer.Option(..., "--name", "--project", help="Project name to check"),
|
||||
one_way: bool = typer.Option(False, "--one-way", help="Check one direction only (faster)"),
|
||||
) -> None:
|
||||
"""Verify file integrity between local and cloud.
|
||||
@@ -268,7 +585,10 @@ def check_project_command(
|
||||
_require_cloud_credentials(config)
|
||||
|
||||
try:
|
||||
# Get tenant info for bucket name
|
||||
# Get tenant info for bucket name.
|
||||
# TODO(#919): scope to the project's workspace like push/pull. Safe for now
|
||||
# because these mirror commands are gated to the (default-tenant) Personal
|
||||
# workspace, so the default mount info is correct.
|
||||
tenant_info = run_with_cleanup(get_mount_info())
|
||||
bucket_name = tenant_info.bucket_name
|
||||
|
||||
@@ -395,9 +715,15 @@ def setup_project_sync(
|
||||
|
||||
console.print(f"[green]Sync configured for project '{name}'[/green]")
|
||||
console.print(f"\nLocal sync path: {resolved_path}")
|
||||
# Lead with the Team-safe additive commands (work on any workspace); the
|
||||
# `sync`/`bisync` mirrors are Personal-workspace-only.
|
||||
console.print("\nNext steps:")
|
||||
console.print(f" 1. Preview: bm cloud sync --name {name} --dry-run")
|
||||
console.print(f" 2. Sync: bm cloud sync --name {name}")
|
||||
console.print(f" 1. Preview a pull: bm cloud pull --name {name} --dry-run")
|
||||
console.print(f" 2. Fetch from cloud: bm cloud pull --name {name}")
|
||||
console.print(f" 3. Upload local changes: bm cloud push --name {name}")
|
||||
console.print(
|
||||
f" Personal workspaces can also mirror with: bm cloud bisync --name {name} --resync"
|
||||
)
|
||||
except Exception as e:
|
||||
console.print(f"[red]Error configuring sync: {str(e)}[/red]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
@@ -2,7 +2,8 @@
|
||||
|
||||
This module provides simplified, project-scoped rclone operations:
|
||||
- Each project syncs independently
|
||||
- Uses single "basic-memory-cloud" remote (not tenant-specific)
|
||||
- Routes through the project's tenant-scoped remote (SyncProject.remote_name);
|
||||
the default tenant keeps "basic-memory-cloud", others use their own (see #919)
|
||||
- Balanced defaults from SPEC-8 Phase 4 testing
|
||||
- Per-project bisync state tracking
|
||||
|
||||
@@ -11,10 +12,10 @@ Replaces tenant-wide sync with project-scoped workflows.
|
||||
|
||||
import re
|
||||
import subprocess
|
||||
from dataclasses import dataclass
|
||||
from dataclasses import dataclass, field
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
from typing import Callable, Optional, Protocol
|
||||
from pathlib import Path, PurePosixPath
|
||||
from typing import Callable, Literal, Optional, Protocol
|
||||
|
||||
from loguru import logger
|
||||
from rich.console import Console
|
||||
@@ -42,6 +43,7 @@ TIGRIS_CONSISTENCY_HEADERS = [
|
||||
class RunResult(Protocol):
|
||||
returncode: int
|
||||
stdout: str
|
||||
stderr: str
|
||||
|
||||
|
||||
RunFunc = Callable[..., RunResult]
|
||||
@@ -112,11 +114,15 @@ class SyncProject:
|
||||
name: Project name
|
||||
path: Cloud path (e.g., "app/data/research")
|
||||
local_sync_path: Local directory for syncing (optional)
|
||||
remote_name: rclone remote serving this project's tenant bucket. Defaults
|
||||
to the legacy single remote; team/non-default workspaces use their own
|
||||
(see remote_name_for_workspace).
|
||||
"""
|
||||
|
||||
name: str
|
||||
path: str
|
||||
local_sync_path: Optional[str] = None
|
||||
remote_name: str = "basic-memory-cloud"
|
||||
|
||||
|
||||
def get_bmignore_filter_path() -> Path:
|
||||
@@ -178,10 +184,98 @@ def get_project_remote(project: SyncProject, bucket_name: str) -> str:
|
||||
The API returns paths like "/app/data/basic-memory-llc" because the S3 bucket
|
||||
is mounted at /app/data on the fly machine. We need to strip the /app/data/
|
||||
prefix to get the actual S3 path within the bucket.
|
||||
|
||||
The remote name comes from the project so non-default/team workspaces route
|
||||
through their own tenant-scoped remote (see #919).
|
||||
"""
|
||||
# Normalize path to strip /app/data/ mount point prefix
|
||||
cloud_path = normalize_project_path(project.path).lstrip("/")
|
||||
return f"basic-memory-cloud:{bucket_name}/{cloud_path}"
|
||||
return f"{project.remote_name}:{bucket_name}/{cloud_path}"
|
||||
|
||||
|
||||
# --- Directional transfer primitives (push / pull) ---
|
||||
#
|
||||
# These power the Team-safe `bm cloud push` / `bm cloud pull` commands. Unlike
|
||||
# the mirror operations (`sync`/`bisync`), they use `rclone copy` so they never
|
||||
# delete on the destination, and conflicts are surfaced to the caller rather
|
||||
# than silently resolved. See issue #858 for the full design rationale.
|
||||
|
||||
# push = local -> cloud, pull = cloud -> local.
|
||||
TransferDirection = Literal["push", "pull"]
|
||||
|
||||
# How a directional transfer treats files that differ on both sides. "fail" is
|
||||
# the safe default: the caller is expected to abort before any transfer runs.
|
||||
ConflictStrategy = Literal["fail", "keep-local", "keep-cloud", "keep-both"]
|
||||
|
||||
|
||||
@dataclass
|
||||
class TransferPlan:
|
||||
"""Classification of how local and cloud differ for a directional transfer.
|
||||
|
||||
Built from ``rclone check --combined``. Paths are relative to the project
|
||||
root. ``conflicts`` are files present on both sides with differing content —
|
||||
without a sync baseline (see #862) every divergence is a conflict, because
|
||||
we cannot tell a teammate's edit from a stale local copy.
|
||||
"""
|
||||
|
||||
new: list[str] = field(default_factory=list) # only on source → safe to bring over
|
||||
conflicts: list[str] = field(default_factory=list) # differ on both sides
|
||||
dest_only: list[str] = field(default_factory=list) # only on destination → left untouched
|
||||
errors: list[str] = field(default_factory=list) # rclone could not read/hash
|
||||
|
||||
|
||||
def _transfer_endpoints(project: SyncProject, bucket_name: str) -> tuple[str, str]:
|
||||
"""Return (local_path, remote_path) strings for a project's transfer.
|
||||
|
||||
Raises:
|
||||
RcloneError: If the project has no local_sync_path configured.
|
||||
"""
|
||||
if not project.local_sync_path:
|
||||
raise RcloneError(f"Project {project.name} has no local_sync_path configured")
|
||||
local_path = str(Path(project.local_sync_path).expanduser())
|
||||
remote_path = get_project_remote(project, bucket_name)
|
||||
return local_path, remote_path
|
||||
|
||||
|
||||
def _build_transfer_cmd(
|
||||
operation: str,
|
||||
source: str,
|
||||
dest: str,
|
||||
*,
|
||||
filter_path: Path,
|
||||
dry_run: bool,
|
||||
verbose: bool,
|
||||
extra_flags: tuple[str, ...] = (),
|
||||
) -> list[str]:
|
||||
"""Build an rclone sync/copy command with the shared Basic Memory flags.
|
||||
|
||||
All directional transfers share the same tail: Tigris consistency headers,
|
||||
the .bmignore filter, and --local-no-preallocate (a no-op when local is the
|
||||
source, required when local is the destination on pull — see rclone#6801).
|
||||
"""
|
||||
cmd = [
|
||||
"rclone",
|
||||
operation,
|
||||
source,
|
||||
dest,
|
||||
*TIGRIS_CONSISTENCY_HEADERS,
|
||||
"--filter-from",
|
||||
str(filter_path),
|
||||
# Prevent NUL byte padding on virtual filesystems (e.g. Google Drive File Stream)
|
||||
# See: rclone/rclone#6801
|
||||
"--local-no-preallocate",
|
||||
*extra_flags,
|
||||
]
|
||||
|
||||
if verbose:
|
||||
cmd.append("--verbose")
|
||||
else:
|
||||
cmd.append("--progress")
|
||||
|
||||
if dry_run:
|
||||
cmd.append("--dry-run")
|
||||
|
||||
return cmd
|
||||
|
||||
|
||||
def project_sync(
|
||||
@@ -219,24 +313,199 @@ def project_sync(
|
||||
remote_path = get_project_remote(project, bucket_name)
|
||||
filter_path = filter_path or get_bmignore_filter_path()
|
||||
|
||||
cmd = [
|
||||
"rclone",
|
||||
cmd = _build_transfer_cmd(
|
||||
"sync",
|
||||
str(local_path),
|
||||
remote_path,
|
||||
filter_path=filter_path,
|
||||
dry_run=dry_run,
|
||||
verbose=verbose,
|
||||
)
|
||||
|
||||
result = run(cmd, text=True)
|
||||
return result.returncode == 0
|
||||
|
||||
|
||||
def _parse_check_combined(output: str) -> TransferPlan:
|
||||
"""Parse ``rclone check --combined`` output into a TransferPlan.
|
||||
|
||||
rclone emits one prefixed line per path (src is the transfer source):
|
||||
``=`` identical, ``+`` only on src, ``-`` only on dst, ``*`` differ,
|
||||
``!`` error reading/hashing. We ignore identical files.
|
||||
"""
|
||||
plan = TransferPlan()
|
||||
for line in output.splitlines():
|
||||
symbol, _, path = line.partition(" ")
|
||||
path = path.strip()
|
||||
if not path:
|
||||
continue
|
||||
if symbol == "+":
|
||||
plan.new.append(path)
|
||||
elif symbol == "*":
|
||||
plan.conflicts.append(path)
|
||||
elif symbol == "-":
|
||||
plan.dest_only.append(path)
|
||||
elif symbol == "!":
|
||||
plan.errors.append(path)
|
||||
# "=" (identical) is intentionally dropped.
|
||||
return plan
|
||||
|
||||
|
||||
def project_diff(
|
||||
project: SyncProject,
|
||||
bucket_name: str,
|
||||
direction: TransferDirection,
|
||||
*,
|
||||
run: RunFunc = subprocess.run,
|
||||
is_installed: IsInstalledFunc = is_rclone_installed,
|
||||
filter_path: Path | None = None,
|
||||
) -> TransferPlan:
|
||||
"""Classify how local and cloud differ for a push/pull, without transferring.
|
||||
|
||||
Uses ``rclone check`` (content comparison) so the caller can surface
|
||||
conflicts before any data moves. The source side depends on direction:
|
||||
pull compares cloud→local, push compares local→cloud.
|
||||
|
||||
Raises:
|
||||
RcloneError: If project has no local_sync_path configured or rclone not installed
|
||||
"""
|
||||
check_rclone_installed(is_installed=is_installed)
|
||||
|
||||
local_path, remote_path = _transfer_endpoints(project, bucket_name)
|
||||
filter_path = filter_path or get_bmignore_filter_path()
|
||||
|
||||
# Source/dest order matters: rclone check reports "+" for files only on the
|
||||
# source, which is what we want to bring over.
|
||||
source, dest = (remote_path, local_path) if direction == "pull" else (local_path, remote_path)
|
||||
|
||||
cmd = [
|
||||
"rclone",
|
||||
"check",
|
||||
source,
|
||||
dest,
|
||||
*TIGRIS_CONSISTENCY_HEADERS,
|
||||
"--filter-from",
|
||||
str(filter_path),
|
||||
# Prevent NUL byte padding on virtual filesystems (e.g. Google Drive File Stream)
|
||||
# See: rclone/rclone#6801
|
||||
"--local-no-preallocate",
|
||||
"--combined",
|
||||
"-",
|
||||
]
|
||||
|
||||
# rclone check exits non-zero when files differ — that's expected here, so we
|
||||
# parse the combined listing rather than trusting the return code.
|
||||
result = run(cmd, capture_output=True, text=True)
|
||||
plan = _parse_check_combined(result.stdout)
|
||||
|
||||
# Trigger: non-zero exit AND the combined listing produced no entries at all.
|
||||
# Why: a difference always yields +/-/*/! lines, so an empty listing on a
|
||||
# non-zero exit means the check itself failed (auth, missing remote, network,
|
||||
# bad filter) rather than finding zero differences. Without this guard the
|
||||
# caller would see an empty plan, transfer nothing, and report success.
|
||||
# Outcome: fail fast with rclone's stderr instead of a silent no-op.
|
||||
if result.returncode != 0 and not (plan.new or plan.conflicts or plan.dest_only or plan.errors):
|
||||
detail = result.stderr.strip() or f"rclone check exited with code {result.returncode}"
|
||||
raise RcloneError(f"Failed to compare {project.name} with cloud: {detail}")
|
||||
|
||||
return plan
|
||||
|
||||
|
||||
def project_copy(
|
||||
project: SyncProject,
|
||||
bucket_name: str,
|
||||
direction: TransferDirection,
|
||||
*,
|
||||
overwrite: bool,
|
||||
dry_run: bool = False,
|
||||
verbose: bool = False,
|
||||
run: RunFunc = subprocess.run,
|
||||
is_installed: IsInstalledFunc = is_rclone_installed,
|
||||
filter_path: Path | None = None,
|
||||
) -> bool:
|
||||
"""Additive transfer via ``rclone copy`` — never deletes on the destination.
|
||||
|
||||
Trigger: ``overwrite=False`` adds ``--ignore-existing`` so files already on
|
||||
the destination are left as-is (used when the destination side wins a
|
||||
conflict, and for the no-conflict fast path).
|
||||
Why: keeps the loser's bytes intact unless the caller explicitly chose to
|
||||
overwrite, matching the "no surprises" contract.
|
||||
|
||||
Raises:
|
||||
RcloneError: If project has no local_sync_path configured or rclone not installed
|
||||
"""
|
||||
check_rclone_installed(is_installed=is_installed)
|
||||
|
||||
local_path, remote_path = _transfer_endpoints(project, bucket_name)
|
||||
filter_path = filter_path or get_bmignore_filter_path()
|
||||
|
||||
source, dest = (remote_path, local_path) if direction == "pull" else (local_path, remote_path)
|
||||
# Overwrite mode compares by checksum so the transfer decision matches
|
||||
# project_diff's content-based conflict detection (rclone check). Without
|
||||
# --checksum, copy's default size+modtime comparison could skip a file the
|
||||
# diff flagged as a conflict (same size, destination not older) — silently
|
||||
# ignoring the user's explicit keep-cloud/keep-local choice. New-only mode
|
||||
# uses --ignore-existing, which skips by existence so the comparison basis
|
||||
# does not matter.
|
||||
extra_flags = ("--checksum",) if overwrite else ("--ignore-existing",)
|
||||
|
||||
cmd = _build_transfer_cmd(
|
||||
"copy",
|
||||
source,
|
||||
dest,
|
||||
filter_path=filter_path,
|
||||
dry_run=dry_run,
|
||||
verbose=verbose,
|
||||
extra_flags=extra_flags,
|
||||
)
|
||||
|
||||
result = run(cmd, text=True)
|
||||
return result.returncode == 0
|
||||
|
||||
|
||||
def _conflict_copy_name(rel_path: str, suffix: str) -> str:
|
||||
"""Insert a ``.conflict-<suffix>`` marker before the extension of a rel path."""
|
||||
p = PurePosixPath(rel_path)
|
||||
return str(p.with_name(f"{p.stem}.conflict-{suffix}{p.suffix}"))
|
||||
|
||||
|
||||
def project_copy_file(
|
||||
project: SyncProject,
|
||||
bucket_name: str,
|
||||
direction: TransferDirection,
|
||||
source_rel_path: str,
|
||||
dest_rel_path: str,
|
||||
*,
|
||||
dry_run: bool = False,
|
||||
verbose: bool = False,
|
||||
run: RunFunc = subprocess.run,
|
||||
is_installed: IsInstalledFunc = is_rclone_installed,
|
||||
) -> bool:
|
||||
"""Copy a single file from source to destination under a (possibly renamed) path.
|
||||
|
||||
Used for the ``keep-both`` strategy: the incoming version is written beside
|
||||
the destination's own copy as ``name.conflict-<date>`` so nothing is lost.
|
||||
|
||||
Raises:
|
||||
RcloneError: If project has no local_sync_path configured or rclone not installed
|
||||
"""
|
||||
check_rclone_installed(is_installed=is_installed)
|
||||
|
||||
local_path, remote_path = _transfer_endpoints(project, bucket_name)
|
||||
source_root, dest_root = (
|
||||
(remote_path, local_path) if direction == "pull" else (local_path, remote_path)
|
||||
)
|
||||
|
||||
cmd = [
|
||||
"rclone",
|
||||
"copyto",
|
||||
f"{source_root}/{source_rel_path}",
|
||||
f"{dest_root}/{dest_rel_path}",
|
||||
*TIGRIS_CONSISTENCY_HEADERS,
|
||||
# Matches _build_transfer_cmd: on pull this writes the conflict copy to
|
||||
# the local filesystem, where this prevents NUL byte padding on virtual
|
||||
# filesystems (e.g. Google Drive File Stream). See rclone/rclone#6801.
|
||||
"--local-no-preallocate",
|
||||
]
|
||||
if verbose:
|
||||
cmd.append("--verbose")
|
||||
else:
|
||||
cmd.append("--progress")
|
||||
|
||||
if dry_run:
|
||||
cmd.append("--dry-run")
|
||||
|
||||
@@ -244,6 +513,72 @@ def project_sync(
|
||||
return result.returncode == 0
|
||||
|
||||
|
||||
def _strategy_overwrites_dest(direction: TransferDirection, strategy: ConflictStrategy) -> bool:
|
||||
"""True when the strategy lets the source side overwrite the destination.
|
||||
|
||||
The source side is cloud on pull, local on push. "keep-cloud" wins on pull,
|
||||
"keep-local" wins on push; otherwise the destination is preserved.
|
||||
"""
|
||||
if strategy == "keep-cloud":
|
||||
return direction == "pull"
|
||||
if strategy == "keep-local":
|
||||
return direction == "push"
|
||||
return False # "fail" (no conflicts) and "keep-both" never overwrite existing dest files
|
||||
|
||||
|
||||
def project_transfer(
|
||||
project: SyncProject,
|
||||
bucket_name: str,
|
||||
direction: TransferDirection,
|
||||
plan: TransferPlan,
|
||||
*,
|
||||
strategy: ConflictStrategy = "fail",
|
||||
conflict_suffix: str = "",
|
||||
dry_run: bool = False,
|
||||
verbose: bool = False,
|
||||
run: RunFunc = subprocess.run,
|
||||
is_installed: IsInstalledFunc = is_rclone_installed,
|
||||
filter_path: Path | None = None,
|
||||
) -> bool:
|
||||
"""Execute a directional transfer for the chosen conflict strategy.
|
||||
|
||||
Callers detect conflicts with ``project_diff`` first and abort when
|
||||
``strategy == "fail"`` and conflicts exist; this function assumes that gate
|
||||
has already passed and applies the resolution.
|
||||
"""
|
||||
# keep-both: preserve the destination's version and drop the incoming one
|
||||
# beside it as a conflict copy, then do an additive (new-only) pass.
|
||||
if strategy == "keep-both":
|
||||
for rel_path in plan.conflicts:
|
||||
dest_rel = _conflict_copy_name(rel_path, conflict_suffix)
|
||||
copied = project_copy_file(
|
||||
project,
|
||||
bucket_name,
|
||||
direction,
|
||||
rel_path,
|
||||
dest_rel,
|
||||
dry_run=dry_run,
|
||||
verbose=verbose,
|
||||
run=run,
|
||||
is_installed=is_installed,
|
||||
)
|
||||
if not copied:
|
||||
return False
|
||||
|
||||
overwrite = _strategy_overwrites_dest(direction, strategy)
|
||||
return project_copy(
|
||||
project,
|
||||
bucket_name,
|
||||
direction,
|
||||
overwrite=overwrite,
|
||||
dry_run=dry_run,
|
||||
verbose=verbose,
|
||||
run=run,
|
||||
is_installed=is_installed,
|
||||
filter_path=filter_path,
|
||||
)
|
||||
|
||||
|
||||
def project_bisync(
|
||||
project: SyncProject,
|
||||
bucket_name: str,
|
||||
|
||||
@@ -1,11 +1,14 @@
|
||||
"""rclone configuration management for Basic Memory Cloud.
|
||||
|
||||
This module provides simplified rclone configuration for SPEC-20.
|
||||
Uses a single "basic-memory-cloud" remote for all operations.
|
||||
This module owns rclone remote configuration and naming. The default tenant uses
|
||||
the "basic-memory-cloud" remote (from SPEC-20); non-default/team workspaces each
|
||||
get their own tenant-scoped remote via remote_name_for_workspace (see #919),
|
||||
since Tigris credentials are bucket-scoped.
|
||||
"""
|
||||
|
||||
import configparser
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
@@ -61,25 +64,65 @@ def save_rclone_config(config: configparser.ConfigParser) -> None:
|
||||
console.print(f"[dim]Updated rclone config: {config_path}[/dim]")
|
||||
|
||||
|
||||
# The default remote serves the account's default tenant (back-compat with SPEC-20,
|
||||
# which used a single "basic-memory-cloud" remote). Non-default workspaces each get
|
||||
# their own remote, since Tigris credentials are bucket/tenant-scoped (see #919).
|
||||
DEFAULT_RCLONE_REMOTE = "basic-memory-cloud"
|
||||
|
||||
|
||||
# rclone remote section names allow letters, digits, hyphens, and underscores.
|
||||
# The slug comes from the cloud API (a trust boundary), so validate it before
|
||||
# splicing it into a remote name to avoid a broken/unusable rclone.conf section.
|
||||
_SAFE_SLUG = re.compile(r"^[A-Za-z0-9_-]+$")
|
||||
|
||||
|
||||
def remote_name_for_workspace(slug: str | None, *, is_default: bool) -> str:
|
||||
"""Return the rclone remote name for a workspace.
|
||||
|
||||
The default workspace keeps the legacy ``basic-memory-cloud`` remote so
|
||||
existing setups keep working; other workspaces get ``basic-memory-cloud-<slug>``.
|
||||
|
||||
Raises:
|
||||
RcloneConfigError: If a non-default workspace slug contains characters
|
||||
that are not valid in an rclone remote name.
|
||||
"""
|
||||
if is_default or not slug:
|
||||
return DEFAULT_RCLONE_REMOTE
|
||||
if not _SAFE_SLUG.match(slug):
|
||||
raise RcloneConfigError(
|
||||
f"Workspace slug '{slug}' cannot be used as an rclone remote name "
|
||||
"(allowed: letters, digits, hyphens, underscores)."
|
||||
)
|
||||
return f"{DEFAULT_RCLONE_REMOTE}-{slug}"
|
||||
|
||||
|
||||
def rclone_remote_exists(remote_name: str) -> bool:
|
||||
"""Return whether an rclone remote section is already configured."""
|
||||
return load_rclone_config().has_section(remote_name)
|
||||
|
||||
|
||||
def configure_rclone_remote(
|
||||
access_key: str,
|
||||
secret_key: str,
|
||||
endpoint: str = "https://fly.storage.tigris.dev",
|
||||
region: str = "auto",
|
||||
remote_name: str = DEFAULT_RCLONE_REMOTE,
|
||||
) -> str:
|
||||
"""Configure single rclone remote named 'basic-memory-cloud'.
|
||||
"""Configure an rclone remote for one tenant's bucket.
|
||||
|
||||
This is the simplified approach from SPEC-20 that uses one remote
|
||||
for all Basic Memory cloud operations (not tenant-specific).
|
||||
Each tenant (personal or team) has its own bucket-scoped credentials, so a
|
||||
remote maps 1:1 to a tenant. The default tenant keeps ``basic-memory-cloud``;
|
||||
other workspaces pass ``remote_name`` from :func:`remote_name_for_workspace`.
|
||||
|
||||
Args:
|
||||
access_key: S3 access key ID
|
||||
secret_key: S3 secret access key
|
||||
endpoint: S3-compatible endpoint URL
|
||||
region: S3 region (default: auto)
|
||||
remote_name: rclone remote section name to write
|
||||
|
||||
Returns:
|
||||
The remote name: "basic-memory-cloud"
|
||||
The remote name that was configured
|
||||
"""
|
||||
# Backup existing config
|
||||
backup_rclone_config()
|
||||
@@ -87,24 +130,21 @@ def configure_rclone_remote(
|
||||
# Load existing config
|
||||
config = load_rclone_config()
|
||||
|
||||
# Single remote name (not tenant-specific)
|
||||
REMOTE_NAME = "basic-memory-cloud"
|
||||
|
||||
# Add/update the remote section
|
||||
if not config.has_section(REMOTE_NAME):
|
||||
config.add_section(REMOTE_NAME)
|
||||
if not config.has_section(remote_name):
|
||||
config.add_section(remote_name)
|
||||
|
||||
config.set(REMOTE_NAME, "type", "s3")
|
||||
config.set(REMOTE_NAME, "provider", "Other")
|
||||
config.set(REMOTE_NAME, "access_key_id", access_key)
|
||||
config.set(REMOTE_NAME, "secret_access_key", secret_key)
|
||||
config.set(REMOTE_NAME, "endpoint", endpoint)
|
||||
config.set(REMOTE_NAME, "region", region)
|
||||
config.set(remote_name, "type", "s3")
|
||||
config.set(remote_name, "provider", "Other")
|
||||
config.set(remote_name, "access_key_id", access_key)
|
||||
config.set(remote_name, "secret_access_key", secret_key)
|
||||
config.set(remote_name, "endpoint", endpoint)
|
||||
config.set(remote_name, "region", region)
|
||||
# Prevent unnecessary encoding of filenames (only encode slashes and invalid UTF-8)
|
||||
# This prevents files with spaces like "Hello World.md" from being quoted
|
||||
config.set(REMOTE_NAME, "encoding", "Slash,InvalidUtf8")
|
||||
config.set(remote_name, "encoding", "Slash,InvalidUtf8")
|
||||
# Save updated config
|
||||
save_rclone_config(config)
|
||||
|
||||
console.print(f"[green]Configured rclone remote: {REMOTE_NAME}[/green]")
|
||||
return REMOTE_NAME
|
||||
console.print(f"[green]Configured rclone remote: {remote_name}[/green]")
|
||||
return remote_name
|
||||
|
||||
@@ -98,6 +98,18 @@ def mcp(
|
||||
if transport == "stdio":
|
||||
threading.Thread(target=_run_background_auto_update, daemon=True).start()
|
||||
|
||||
# Trigger: MCP server startup on the Postgres backend, before the transport
|
||||
# creates its event loop.
|
||||
# Why: the watcher/lifespan path runs startup migrations + engine.dispose() on
|
||||
# asyncpg, which races stdlib asyncio teardown and crashes the container loop
|
||||
# (#831/#877). uvloop's C scheduler structurally avoids that race and must own
|
||||
# the loop policy before the loop is created. No-op for SQLite.
|
||||
# Outcome: `basic-memory mcp` on Postgres runs on uvloop. (The CLI callback also
|
||||
# installs it; this keeps the server startup seam explicit and self-contained.)
|
||||
from basic_memory.db import maybe_install_uvloop
|
||||
|
||||
maybe_install_uvloop(ConfigManager().config)
|
||||
|
||||
# Run the MCP server (blocks)
|
||||
# Lifespan handles: initialization, migrations, file sync, cleanup
|
||||
logger.info(f"Starting MCP server with {transport.upper()} transport")
|
||||
|
||||
@@ -1,8 +1,9 @@
|
||||
"""Status command for basic-memory CLI."""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from typing import Set, Dict
|
||||
from typing import Annotated, Optional
|
||||
import time
|
||||
from typing import Annotated, Dict, Optional, Set
|
||||
|
||||
from mcp.server.fastmcp.exceptions import ToolError
|
||||
import typer
|
||||
@@ -142,20 +143,58 @@ def display_changes(
|
||||
console.print(Panel(tree, expand=False))
|
||||
|
||||
|
||||
class StatusTimeout(Exception):
|
||||
"""Raised when --wait does not reach a synced state before the deadline."""
|
||||
|
||||
|
||||
async def run_status(
|
||||
project: Optional[str] = None,
|
||||
wait: bool = False,
|
||||
timeout: float = 30.0,
|
||||
poll_interval: float = 0.5,
|
||||
) -> tuple[str, SyncReportResponse]:
|
||||
"""Fetch sync status of files vs database.
|
||||
|
||||
When ``wait`` is False this performs a single live disk-vs-DB scan and
|
||||
returns immediately. When ``wait`` is True it polls until the project has
|
||||
no pending changes (``sync_report.total == 0``) or the timeout elapses.
|
||||
|
||||
Returns (project_name, sync_report) for the caller to render.
|
||||
|
||||
Raises:
|
||||
StatusTimeout: If ``wait`` is True and the deadline passes before the
|
||||
project reaches a synced state.
|
||||
"""
|
||||
# Resolve default project so get_client() can route per-project
|
||||
project = project or ConfigManager().default_project
|
||||
|
||||
# Reuse a single client/context across polls so we don't reconnect each loop.
|
||||
async with get_client(project_name=project) as client:
|
||||
project_item = await get_active_project(client, project, None)
|
||||
sync_report = await ProjectClient(client).get_status(project_item.external_id)
|
||||
return project_item.name, sync_report
|
||||
project_client = ProjectClient(client)
|
||||
|
||||
# Trigger: caller did not request --wait
|
||||
# Why: preserve the original single-scan behavior for the common case
|
||||
# Outcome: one status scan, returned as-is
|
||||
if not wait:
|
||||
sync_report = await project_client.get_status(project_item.external_id)
|
||||
return project_item.name, sync_report
|
||||
|
||||
# Trigger: --wait requested
|
||||
# Why: callers (bulk imports, benchmarks, tests) need to block until the
|
||||
# index has caught up instead of polling externally
|
||||
# Outcome: poll get_status until total == 0 or the deadline is reached
|
||||
deadline = time.monotonic() + timeout
|
||||
while True:
|
||||
sync_report = await project_client.get_status(project_item.external_id)
|
||||
if sync_report.total == 0:
|
||||
return project_item.name, sync_report
|
||||
if time.monotonic() >= deadline:
|
||||
raise StatusTimeout(
|
||||
f"Timed out after {timeout:g}s waiting for '{project_item.name}' "
|
||||
f"to finish indexing ({sync_report.total} pending change(s) remaining)."
|
||||
)
|
||||
await asyncio.sleep(poll_interval)
|
||||
|
||||
|
||||
@app.command()
|
||||
@@ -166,6 +205,10 @@ def status(
|
||||
] = None,
|
||||
verbose: bool = typer.Option(False, "--verbose", "-v", help="Show detailed file information"),
|
||||
json_output: bool = typer.Option(False, "--json", help="Output in JSON format"),
|
||||
wait: bool = typer.Option(
|
||||
False, "--wait", help="Block until indexing is complete (no pending changes)"
|
||||
),
|
||||
timeout: float = typer.Option(30.0, "--timeout", help="Max seconds to wait when --wait is set"),
|
||||
local: bool = typer.Option(
|
||||
False, "--local", help="Force local API routing (ignore cloud mode)"
|
||||
),
|
||||
@@ -174,11 +217,21 @@ def status(
|
||||
"""Show sync status between files and database.
|
||||
|
||||
Use --json for machine-readable output.
|
||||
Use --wait to block until indexing is complete (e.g. after a bulk import);
|
||||
combine with --timeout to bound the wait. On timeout the command exits 1.
|
||||
Use --local to force local routing when cloud mode is enabled.
|
||||
Use --cloud to force cloud routing when cloud mode is disabled.
|
||||
"""
|
||||
from basic_memory.cli.commands.command_utils import run_with_cleanup
|
||||
|
||||
# Trigger: --wait with a negative --timeout
|
||||
# Why: a negative deadline times out on the very first poll, producing a confusing
|
||||
# "Timed out after -5s" message instead of flagging the bad input. Raised
|
||||
# before the try/except so typer renders a clean usage error (exit 2).
|
||||
# Outcome: reject it up front with a clear parameter error.
|
||||
if wait and timeout < 0:
|
||||
raise typer.BadParameter("--timeout must be >= 0", param_hint="'--timeout'")
|
||||
|
||||
try:
|
||||
validate_routing_flags(local, cloud)
|
||||
# Trigger: no explicit routing flag provided
|
||||
@@ -189,12 +242,24 @@ def status(
|
||||
if not local and not cloud:
|
||||
local = True
|
||||
with force_routing(local=local, cloud=cloud):
|
||||
project_name, sync_report = run_with_cleanup(run_status(project))
|
||||
project_name, sync_report = run_with_cleanup(
|
||||
run_status(project, wait=wait, timeout=timeout)
|
||||
)
|
||||
|
||||
if json_output:
|
||||
print(json.dumps(sync_report.model_dump(mode="json"), indent=2, default=str))
|
||||
else:
|
||||
display_changes(project_name, "Status", sync_report, verbose)
|
||||
except StatusTimeout as e:
|
||||
# Trigger: --wait deadline passed before the project finished indexing
|
||||
# Why: callers depend on exit code 1 to detect that indexing did not
|
||||
# complete in time, while still getting a clear machine/human message
|
||||
# Outcome: emit the timeout message (JSON-shaped under --json) and exit 1
|
||||
if json_output:
|
||||
print(json.dumps({"error": str(e)}, indent=2))
|
||||
else:
|
||||
console.print(f"[red]Error: {e}[/red]")
|
||||
raise typer.Exit(code=1)
|
||||
except (ValueError, ToolError) as e:
|
||||
if json_output:
|
||||
print(json.dumps({"error": str(e)}, indent=2))
|
||||
|
||||
@@ -15,6 +15,7 @@ from basic_memory.cli.app import app
|
||||
from basic_memory.cli.commands.command_utils import run_with_cleanup
|
||||
from basic_memory.cli.commands.routing import force_routing, validate_routing_flags
|
||||
from basic_memory.mcp.tools import build_context as mcp_build_context
|
||||
from basic_memory.mcp.tools import delete_note as mcp_delete_note
|
||||
from basic_memory.mcp.tools import edit_note as mcp_edit_note
|
||||
from basic_memory.mcp.tools import list_memory_projects as mcp_list_projects
|
||||
from basic_memory.mcp.tools import list_workspaces as mcp_list_workspaces
|
||||
@@ -40,6 +41,26 @@ def _print_json(result: Any) -> None:
|
||||
print(json.dumps(result, indent=2, ensure_ascii=True, default=str))
|
||||
|
||||
|
||||
def _delete_note_failure_message(result: dict[str, Any]) -> str | None:
|
||||
"""Return the CLI failure message for delete-note JSON results, if any."""
|
||||
error = result.get("error")
|
||||
if error:
|
||||
return str(error)
|
||||
|
||||
failed_deletes = result.get("failed_deletes")
|
||||
# Trigger: directory deletion can partially fail without raising from the service.
|
||||
# Why: cleanup scripts need a non-zero exit when files remain undeleted.
|
||||
# Outcome: the CLI fails even if older MCP JSON did not include an error field.
|
||||
if (
|
||||
result.get("is_directory") is True
|
||||
and isinstance(failed_deletes, int)
|
||||
and failed_deletes > 0
|
||||
):
|
||||
return f"Directory delete incomplete: {failed_deletes} file(s) failed"
|
||||
|
||||
return None
|
||||
|
||||
|
||||
# --- Commands ---
|
||||
|
||||
|
||||
@@ -56,6 +77,16 @@ def write_note(
|
||||
tags: Annotated[
|
||||
Optional[List[str]], typer.Option(help="A list of tags to apply to the note")
|
||||
] = None,
|
||||
note_type: Annotated[
|
||||
str,
|
||||
typer.Option(
|
||||
"--type",
|
||||
help=(
|
||||
"Note type stored in frontmatter (e.g. 'guide', 'report'). "
|
||||
"A 'type:' in the note's own content frontmatter takes precedence."
|
||||
),
|
||||
),
|
||||
] = "note",
|
||||
project: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(
|
||||
@@ -69,6 +100,11 @@ def write_note(
|
||||
help="Project external_id (UUID). Takes precedence over --project; use to disambiguate same-named projects across cloud workspaces.",
|
||||
),
|
||||
] = None,
|
||||
overwrite: bool = typer.Option(
|
||||
False,
|
||||
"--overwrite",
|
||||
help="Replace an existing note on conflict (matches MCP write_note overwrite=True)",
|
||||
),
|
||||
local: bool = typer.Option(
|
||||
False, "--local", help="Force local API routing (ignore cloud mode)"
|
||||
),
|
||||
@@ -79,7 +115,9 @@ def write_note(
|
||||
Examples:
|
||||
|
||||
bm tool write-note --title "My Note" --folder "notes" --content "Note content"
|
||||
bm tool write-note --title "My Guide" --folder "notes" --content "..." --type guide
|
||||
echo "content" | bm tool write-note --title "My Note" --folder "notes"
|
||||
bm tool write-note --title "My Note" --folder "notes" --overwrite
|
||||
bm tool write-note --title "My Note" --folder "notes" --local
|
||||
"""
|
||||
try:
|
||||
@@ -111,9 +149,23 @@ def write_note(
|
||||
project=project,
|
||||
project_id=project_id,
|
||||
tags=tags,
|
||||
note_type=note_type,
|
||||
overwrite=overwrite,
|
||||
output_format="json",
|
||||
)
|
||||
)
|
||||
|
||||
# MCP tool returns an error field on failure in JSON mode (e.g.
|
||||
# NOTE_ALREADY_EXISTS on a blocked overwrite, SECURITY_VALIDATION_ERROR).
|
||||
# Trigger: result carries a non-empty `error`.
|
||||
# Why: parity with delete-note/edit-note/search-notes so exit-code-driven
|
||||
# scripts detect a failed/blocked write instead of seeing exit 0.
|
||||
# Outcome: print the error to stderr and exit non-zero.
|
||||
if isinstance(result, dict) and result.get("error"):
|
||||
typer.echo(f"Error: {result['error']}", err=True)
|
||||
_print_json(result)
|
||||
raise typer.Exit(1)
|
||||
|
||||
_print_json(result)
|
||||
except ValueError as e:
|
||||
typer.echo(f"Error: {e}", err=True)
|
||||
@@ -167,6 +219,19 @@ def read_note(
|
||||
output_format="json",
|
||||
)
|
||||
)
|
||||
|
||||
# MCP tool returns an error field on failure in JSON mode (e.g.
|
||||
# SECURITY_VALIDATION_ERROR on a path-traversal identifier). A genuine
|
||||
# not-found returns null fields with no `error` key, so it still exits 0.
|
||||
# Trigger: result carries a non-empty `error`.
|
||||
# Why: parity with edit-note/delete-note/search-notes so a blocked read
|
||||
# surfaces a non-zero exit instead of looking like success.
|
||||
# Outcome: print the error to stderr and exit non-zero.
|
||||
if isinstance(result, dict) and result.get("error"):
|
||||
typer.echo(f"Error: {result['error']}", err=True)
|
||||
_print_json(result)
|
||||
raise typer.Exit(1)
|
||||
|
||||
_print_json(result)
|
||||
except ValueError as e:
|
||||
typer.echo(f"Error: {e}", err=True)
|
||||
@@ -178,6 +243,66 @@ def read_note(
|
||||
raise
|
||||
|
||||
|
||||
@tool_app.command("delete-note")
|
||||
def delete_note(
|
||||
identifier: str,
|
||||
is_directory: bool = typer.Option(
|
||||
False, "--is-directory", help="Delete a directory instead of a single note"
|
||||
),
|
||||
project: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(help="The project to use. If not provided, the default project will be used."),
|
||||
] = None,
|
||||
project_id: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(
|
||||
"--project-id",
|
||||
help="Project external_id (UUID). Takes precedence over --project; use to disambiguate same-named projects across cloud workspaces.",
|
||||
),
|
||||
] = None,
|
||||
local: bool = typer.Option(
|
||||
False, "--local", help="Force local API routing (ignore cloud mode)"
|
||||
),
|
||||
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
|
||||
) -> None:
|
||||
"""Delete a note or directory from the knowledge base.
|
||||
|
||||
Examples:
|
||||
|
||||
bm tool delete-note notes/old-draft
|
||||
bm tool delete-note docs/archive --is-directory
|
||||
"""
|
||||
try:
|
||||
validate_routing_flags(local, cloud)
|
||||
|
||||
with force_routing(local=local, cloud=cloud):
|
||||
result = run_with_cleanup(
|
||||
mcp_delete_note(
|
||||
identifier=identifier,
|
||||
is_directory=is_directory,
|
||||
project=project,
|
||||
project_id=project_id,
|
||||
output_format="json",
|
||||
)
|
||||
)
|
||||
|
||||
if isinstance(result, dict):
|
||||
failure_message = _delete_note_failure_message(result)
|
||||
if failure_message:
|
||||
typer.echo(f"Error: {failure_message}", err=True)
|
||||
raise typer.Exit(1)
|
||||
|
||||
_print_json(result)
|
||||
except ValueError as e:
|
||||
typer.echo(f"Error: {e}", err=True)
|
||||
raise typer.Exit(1)
|
||||
except Exception as e: # pragma: no cover
|
||||
if not isinstance(e, typer.Exit):
|
||||
typer.echo(f"Error during delete_note: {e}", err=True)
|
||||
raise typer.Exit(1)
|
||||
raise
|
||||
|
||||
|
||||
@tool_app.command()
|
||||
def edit_note(
|
||||
identifier: str,
|
||||
@@ -324,7 +449,9 @@ def recent_activity(
|
||||
"7d", "--timeframe", help="Timeframe filter (e.g., '7d', '1 week')"
|
||||
),
|
||||
page: int = typer.Option(1, "--page", help="Page number for pagination"),
|
||||
page_size: int = typer.Option(50, "--page-size", help="Number of results per page"),
|
||||
# Match the MCP recent_activity default (page_size=10) so identical default
|
||||
# invocations return the same number of rows from CLI and MCP.
|
||||
page_size: int = typer.Option(10, "--page-size", help="Number of results per page"),
|
||||
project: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(help="The project to use. If not provided, the default project will be used."),
|
||||
@@ -409,6 +536,16 @@ def search_notes(
|
||||
help="Filter by search item type: entity, observation, relation (repeatable)",
|
||||
),
|
||||
] = None,
|
||||
categories: Annotated[
|
||||
Optional[List[str]],
|
||||
typer.Option(
|
||||
"--category",
|
||||
help=(
|
||||
"Filter observation results to exact categories (repeatable); "
|
||||
"pair with --entity-type observation"
|
||||
),
|
||||
),
|
||||
] = None,
|
||||
meta: Annotated[
|
||||
Optional[List[str]],
|
||||
typer.Option("--meta", help="Filter by frontmatter key=value (repeatable)"),
|
||||
@@ -443,6 +580,7 @@ def search_notes(
|
||||
bm tool search-notes --permalink "specs/*"
|
||||
bm tool search-notes --tag python --tag async
|
||||
bm tool search-notes --meta status=draft
|
||||
bm tool search-notes "auth" --entity-type observation --category requirement
|
||||
"""
|
||||
try:
|
||||
validate_routing_flags(local, cloud)
|
||||
@@ -508,6 +646,7 @@ def search_notes(
|
||||
page_size=page_size,
|
||||
note_types=note_types,
|
||||
entity_types=entity_types,
|
||||
categories=categories,
|
||||
metadata_filters=metadata_filters,
|
||||
tags=tags,
|
||||
status=status,
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
"""Top-level `workspace` stub that redirects users to `bm cloud workspace`."""
|
||||
|
||||
import typer
|
||||
|
||||
from basic_memory.cli.app import app
|
||||
|
||||
|
||||
# Trigger: user runs `bm workspace`, `bm workspace list`, etc.
|
||||
# Why: workspace verbs live under `bm cloud workspace`; a bare top-level miss
|
||||
# only emits Typer's terse "No such command 'workspace'." with exit 2.
|
||||
# Outcome: allow_extra_args + ignore_unknown_options absorb any trailing tokens
|
||||
# (e.g. `list`, `set-default foo`) so every invocation reaches this body and
|
||||
# prints actionable guidance instead of being rejected as bad usage.
|
||||
@app.command(
|
||||
"workspace",
|
||||
context_settings={"allow_extra_args": True, "ignore_unknown_options": True},
|
||||
)
|
||||
def workspace_stub(ctx: typer.Context) -> None:
|
||||
"""Point users to the real workspace verbs under `bm cloud workspace`."""
|
||||
typer.echo("'bm workspace' is not a command. Workspace verbs live under 'bm cloud workspace':")
|
||||
typer.echo(" bm cloud workspace list")
|
||||
typer.echo(" bm cloud workspace set-default <name>")
|
||||
raise typer.Exit(1)
|
||||
@@ -31,6 +31,7 @@ if not _version_only_invocation(sys.argv[1:]):
|
||||
status,
|
||||
tool,
|
||||
update,
|
||||
workspace,
|
||||
)
|
||||
|
||||
warnings.filterwarnings("ignore") # pragma: no cover
|
||||
|
||||
@@ -248,7 +248,19 @@ class BasicMemoryConfig(BaseSettings):
|
||||
)
|
||||
semantic_embedding_dimensions: int | None = Field(
|
||||
default=None,
|
||||
description="Embedding vector dimensions. Auto-detected from provider if not set (384 for FastEmbed, 1536 for OpenAI).",
|
||||
description=(
|
||||
"Embedding vector dimensions. Uses provider defaults when unset "
|
||||
"(384 for FastEmbed, 1536 for OpenAI and LiteLLM OpenAI default); "
|
||||
"required for custom LiteLLM models."
|
||||
),
|
||||
)
|
||||
semantic_embedding_forward_dimensions: bool | None = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"LiteLLM-only override for sending semantic_embedding_dimensions as a "
|
||||
"provider-side output-size request parameter. When unset, Basic Memory "
|
||||
"auto-detects model strings such as text-embedding-3."
|
||||
),
|
||||
)
|
||||
# Trigger: full local rebuilds spend most of their time waiting behind shared
|
||||
# embed flushes, not constructing vectors themselves.
|
||||
@@ -266,6 +278,20 @@ class BasicMemoryConfig(BaseSettings):
|
||||
description="Maximum number of concurrent provider requests for batched embedding generation when the active provider supports request-level concurrency.",
|
||||
gt=0,
|
||||
)
|
||||
semantic_embedding_document_input_type: str | None = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"Optional LiteLLM input_type for indexed document/passages. "
|
||||
"Use with asymmetric embedding models such as Cohere or NVIDIA retrieval models."
|
||||
),
|
||||
)
|
||||
semantic_embedding_query_input_type: str | None = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"Optional LiteLLM input_type for search queries. "
|
||||
"Use with asymmetric embedding models such as Cohere or NVIDIA retrieval models."
|
||||
),
|
||||
)
|
||||
semantic_embedding_sync_batch_size: int = Field(
|
||||
default=2,
|
||||
description="Batch size for vector sync orchestration flushes.",
|
||||
|
||||
+63
-6
@@ -1,7 +1,7 @@
|
||||
import asyncio
|
||||
import os
|
||||
import sys
|
||||
from contextlib import asynccontextmanager
|
||||
from contextlib import asynccontextmanager, suppress
|
||||
from enum import Enum, auto
|
||||
from pathlib import Path
|
||||
from typing import AsyncGenerator, Optional
|
||||
@@ -39,6 +39,44 @@ from basic_memory.repository.sqlite_search_repository import SQLiteSearchReposit
|
||||
if sys.platform == "win32": # pragma: no cover
|
||||
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
|
||||
|
||||
|
||||
def maybe_install_uvloop(config: BasicMemoryConfig) -> bool:
|
||||
"""Install the uvloop event-loop policy for the Postgres backend.
|
||||
|
||||
Trigger: process entrypoint starting with database_backend == postgres,
|
||||
uvloop importable, and a non-Windows platform.
|
||||
Why: asyncpg engine teardown (engine.dispose()) races the stdlib asyncio
|
||||
loop shutdown and surfaces "IndexError: pop from an empty deque" from
|
||||
base_events._run_once (see #831/#877). uvloop's C scheduler has no
|
||||
self._ready.popleft() codepath, so that class of crash cannot fire under it.
|
||||
Outcome: Postgres deployments run on uvloop; SQLite users keep the default
|
||||
loop (no behavior change, smaller blast radius). Must run before the event
|
||||
loop is created, i.e. before asyncio.run().
|
||||
|
||||
Returns:
|
||||
True if the uvloop policy was installed, False otherwise.
|
||||
"""
|
||||
# uvloop is not available on Windows; the default loop already differs there.
|
||||
if sys.platform == "win32": # pragma: no cover
|
||||
return False
|
||||
|
||||
# Limit the change to the backend that actually hits the asyncpg dispose race.
|
||||
if config.database_backend != DatabaseBackend.POSTGRES:
|
||||
return False
|
||||
|
||||
# Deferred import: uvloop is an optional, platform-gated dependency and the
|
||||
# default (SQLite) path must not require it to be installed.
|
||||
try:
|
||||
import uvloop
|
||||
except ImportError: # pragma: no cover
|
||||
logger.warning("uvloop not available - using default event loop for Postgres backend")
|
||||
return False
|
||||
|
||||
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
|
||||
logger.info("Installed uvloop event-loop policy for Postgres backend")
|
||||
return True
|
||||
|
||||
|
||||
# Module level state
|
||||
_engine: Optional[AsyncEngine] = None
|
||||
_session_maker: Optional[async_sessionmaker[AsyncSession]] = None
|
||||
@@ -320,7 +358,15 @@ async def shutdown_db() -> None: # pragma: no cover
|
||||
global _engine, _session_maker
|
||||
|
||||
if _engine:
|
||||
await _engine.dispose()
|
||||
# Trigger: teardown can run while the surrounding task is being cancelled
|
||||
# (e.g. lifespan shutdown, unshielded CLI cleanup).
|
||||
# Why: a cancellation landing mid-dispose surfaces the asyncpg
|
||||
# "IndexError: pop from an empty deque" race (#831/#877); shielding lets
|
||||
# dispose finish atomically, and suppressing CancelledError keeps a
|
||||
# cancelled shutdown from re-raising the underlying race.
|
||||
# Outcome: connections always close cleanly even under cancellation.
|
||||
with suppress(asyncio.CancelledError):
|
||||
await asyncio.shield(_engine.dispose())
|
||||
_engine = None
|
||||
_session_maker = None
|
||||
|
||||
@@ -364,7 +410,14 @@ async def engine_session_factory(
|
||||
|
||||
yield created_engine, created_session_maker
|
||||
finally:
|
||||
await created_engine.dispose()
|
||||
# Trigger: context-manager teardown can run while the surrounding task is
|
||||
# being cancelled (e.g. a test aborting mid-fixture).
|
||||
# Why: on the asyncpg backend a cancellation landing mid-dispose surfaces
|
||||
# the "IndexError: pop from an empty deque" race (#831/#877); shield the
|
||||
# dispose and suppress CancelledError to match the other dispose seams.
|
||||
# Outcome: the per-context engine always disposes cleanly under cancellation.
|
||||
with suppress(asyncio.CancelledError):
|
||||
await asyncio.shield(created_engine.dispose())
|
||||
|
||||
# Only clear module-level globals if they still point to this context's
|
||||
# engine/session. This avoids clobbering newer globals from other callers.
|
||||
@@ -433,7 +486,11 @@ async def run_migrations(
|
||||
# Trigger: run_migrations() created a temporary engine while module-level
|
||||
# session maker was not initialized.
|
||||
# Why: temporary aiosqlite worker threads can outlive CLI command execution
|
||||
# and block process shutdown if the engine is not disposed.
|
||||
# Outcome: always dispose temporary engines after migration work completes.
|
||||
# and block process shutdown if the engine is not disposed. On the asyncpg
|
||||
# backend a cancellation landing mid-dispose surfaces the same "IndexError:
|
||||
# pop from an empty deque" race as the other dispose seams (#831/#877), so
|
||||
# shield the dispose and suppress CancelledError to match them.
|
||||
# Outcome: always dispose temporary engines cleanly, even under cancellation.
|
||||
if temp_engine is not None:
|
||||
await temp_engine.dispose()
|
||||
with suppress(asyncio.CancelledError):
|
||||
await asyncio.shield(temp_engine.dispose())
|
||||
|
||||
@@ -493,8 +493,14 @@ class BatchIndexer:
|
||||
async def resolve_relation(relation: Relation) -> int:
|
||||
async with semaphore:
|
||||
try:
|
||||
# strict=True for deferred resolution: only fill in to_id on an
|
||||
# exact permalink/title/file_path match. Fuzzy fallback would silently
|
||||
# resolve ambiguous links to whichever entity shares tokens with the
|
||||
# link text, mismatching this with the sync_service forward-reference
|
||||
# path and producing confidently-wrong graph edges. See
|
||||
# sync_service.resolve_forward_references for the same change.
|
||||
resolved_entity = await self.entity_service.link_resolver.resolve_link(
|
||||
relation.to_name
|
||||
relation.to_name, strict=True
|
||||
)
|
||||
if resolved_entity is None or resolved_entity.id == relation.from_id:
|
||||
return 0
|
||||
|
||||
@@ -209,6 +209,18 @@ async def build_context(
|
||||
Raises:
|
||||
ToolError: If project doesn't exist or depth parameter is invalid
|
||||
"""
|
||||
# Validate pagination arguments before they reach the context service.
|
||||
# Trigger: page < 1 or page_size < 1 (e.g. page_size=0 or negative).
|
||||
# Why: a non-positive page_size flows into context_service as limit, where the
|
||||
# primary slice does primary = primary[:limit] — so limit=0 truncates the
|
||||
# requested entity to [] and the caller's valid memory:// lookup silently
|
||||
# returns primary_count=0. Mirrors recent_activity's guard for consistency.
|
||||
# Outcome: caller gets an explicit ValueError instead of a dropped primary result.
|
||||
if page < 1:
|
||||
raise ValueError(f"page must be >= 1, got {page}")
|
||||
if page_size < 1:
|
||||
raise ValueError(f"page_size must be >= 1, got {page_size}")
|
||||
|
||||
# Detect project from memory URL prefix before routing.
|
||||
# project_id routes by external UUID, so it bypasses URL discovery entirely.
|
||||
if project is None and project_id is None:
|
||||
|
||||
@@ -315,14 +315,22 @@ async def delete_note(
|
||||
)
|
||||
result = await knowledge_client.delete_directory(directory_identifier)
|
||||
if output_format == "json":
|
||||
return {
|
||||
response = {
|
||||
"deleted": result.failed_deletes == 0,
|
||||
"is_directory": True,
|
||||
"identifier": identifier,
|
||||
"total_files": result.total_files,
|
||||
"successful_deletes": result.successful_deletes,
|
||||
"failed_deletes": result.failed_deletes,
|
||||
"deleted_files": result.deleted_files,
|
||||
"errors": [error.model_dump() for error in result.errors],
|
||||
}
|
||||
if result.failed_deletes > 0:
|
||||
response["error"] = (
|
||||
"Directory delete incomplete: "
|
||||
f"{result.failed_deletes} of {result.total_files} file(s) failed"
|
||||
)
|
||||
return response
|
||||
|
||||
# Build success message for directory delete
|
||||
result_lines = [
|
||||
|
||||
@@ -10,7 +10,7 @@ from mcp.server.fastmcp.exceptions import ToolError
|
||||
from pydantic import AliasChoices, Field
|
||||
|
||||
from basic_memory.mcp.server import mcp
|
||||
from basic_memory.mcp.project_context import get_project_client
|
||||
from basic_memory.mcp.project_context import get_project_client, resolve_project_and_path
|
||||
from basic_memory.utils import validate_project_path
|
||||
|
||||
|
||||
@@ -37,22 +37,35 @@ async def _detect_cross_project_move_attempt(
|
||||
project_list = await project_client.list_projects()
|
||||
project_names = [p.name.lower() for p in project_list.projects]
|
||||
|
||||
# Check if destination path contains any project names
|
||||
dest_lower = destination_path.lower()
|
||||
path_parts = dest_lower.split("/")
|
||||
path_parts = [part for part in dest_lower.split("/") if part]
|
||||
|
||||
# Look for project names in the destination path
|
||||
for part in path_parts:
|
||||
if part in project_names and part != current_project.lower():
|
||||
# Found a different project name in the path
|
||||
# --- Detection 1: leading segment is a known project name ---
|
||||
# Trigger: the first path segment matches a different project's name.
|
||||
# Why: a routing-style destination like "other-project/file.md" expresses an
|
||||
# intent to move into another project, which move_note cannot do — it
|
||||
# would silently create a same-project nested folder instead.
|
||||
# Outcome: reject with cross-project guidance rather than fake success.
|
||||
if path_parts:
|
||||
leading = path_parts[0]
|
||||
if leading in project_names and leading != current_project.lower():
|
||||
matching_project = next(
|
||||
p.name for p in project_list.projects if p.name.lower() == part
|
||||
p.name for p in project_list.projects if p.name.lower() == leading
|
||||
)
|
||||
return _format_cross_project_error_response(
|
||||
identifier, destination_path, current_project, matching_project
|
||||
)
|
||||
|
||||
# No other cross-project patterns detected
|
||||
# NOTE: a "<seg>/projects/<seg>/..." structural heuristic was removed here.
|
||||
# Why: matching any destination whose 2nd segment is literally "projects" is
|
||||
# fundamentally ambiguous — it cannot distinguish the cloud workspace
|
||||
# routing shape from a legitimate same-project nested folder like
|
||||
# "notes/projects/2025/file.md" or "work/projects/q1/report.md". The
|
||||
# heuristic produced false CROSS_PROJECT_MOVE_NOT_SUPPORTED rejections for
|
||||
# those common layouts.
|
||||
# Outcome: cross-workspace routing that does not match a known project name (above)
|
||||
# now falls through to the move and is caught honestly by the MOVE_OUTCOME_MISMATCH
|
||||
# backstop if the result lands somewhere other than the requested path.
|
||||
|
||||
except Exception as e:
|
||||
# If we can't detect, don't interfere with normal error handling
|
||||
@@ -633,12 +646,37 @@ list_directory("{identifier}")
|
||||
move_note("path/to/file.md", "{destination_path}/file.md")
|
||||
```"""
|
||||
|
||||
# Check for potential cross-project move attempts (file moves only)
|
||||
cross_project_error = await _detect_cross_project_move_attempt(
|
||||
client, identifier, destination_path, active_project.name
|
||||
# Import here to avoid circular import
|
||||
from basic_memory.mcp.clients import KnowledgeClient
|
||||
|
||||
# Use typed KnowledgeClient for API calls
|
||||
knowledge_client = KnowledgeClient(client, active_project.external_id)
|
||||
|
||||
# --- Normalize the identifier for memory:// URLs ---
|
||||
# Trigger: caller passed a "memory://..." URL (the docstring advertises this, and
|
||||
# read_note/edit_note/delete_note all accept it).
|
||||
# Why: resolve_entity / the /resolve endpoint expect a bare permalink or title;
|
||||
# the literal "memory://" scheme prefix is not stripped by link resolution and
|
||||
# 404s. resolve_project_and_path strips the prefix and normalizes the path the
|
||||
# same way the sibling tools do.
|
||||
# Outcome: move_note resolves memory:// URLs identically to read/edit/delete.
|
||||
source_project, resolved_identifier, _ = await resolve_project_and_path(
|
||||
client, identifier, active_project.name, context
|
||||
)
|
||||
if cross_project_error:
|
||||
logger.info(f"Detected cross-project move attempt: {identifier} -> {destination_path}")
|
||||
|
||||
# Trigger: a memory:// identifier whose project prefix resolves to a DIFFERENT project
|
||||
# than the one move_note is operating on (e.g. "memory://other-project/...").
|
||||
# Why: get_project_client already bound knowledge_client to the active project, so the
|
||||
# resolve_entity below runs against the active project regardless of the URL's
|
||||
# project. Honoring the cross-project prefix would misroute (path normalized for
|
||||
# the other project, looked up in the active one); move_note cannot move across
|
||||
# projects.
|
||||
# Outcome: reject up front with the cross-project guidance instead of misrouting.
|
||||
if source_project.external_id != active_project.external_id:
|
||||
logger.info(
|
||||
f"Move rejected: source '{identifier}' resolves to project "
|
||||
f"'{source_project.name}', not the active project '{active_project.name}'"
|
||||
)
|
||||
if output_format == "json":
|
||||
return {
|
||||
"moved": False,
|
||||
@@ -649,13 +687,9 @@ move_note("path/to/file.md", "{destination_path}/file.md")
|
||||
"destination": destination_path,
|
||||
"error": "CROSS_PROJECT_MOVE_NOT_SUPPORTED",
|
||||
}
|
||||
return cross_project_error
|
||||
|
||||
# Import here to avoid circular import
|
||||
from basic_memory.mcp.clients import KnowledgeClient
|
||||
|
||||
# Use typed KnowledgeClient for API calls
|
||||
knowledge_client = KnowledgeClient(client, active_project.external_id)
|
||||
return _format_cross_project_error_response(
|
||||
identifier, destination_path, active_project.name, source_project.name
|
||||
)
|
||||
|
||||
# Resolve once and reuse the entity ID across extension validation and move.
|
||||
source_ext = "md" # Default to .md if we can't determine source extension
|
||||
@@ -666,7 +700,9 @@ move_note("path/to/file.md", "{destination_path}/file.md")
|
||||
"""Resolve and cache the source entity ID for the duration of this move."""
|
||||
nonlocal resolved_entity_id
|
||||
if resolved_entity_id is None:
|
||||
resolved_entity_id = await knowledge_client.resolve_entity(identifier, strict=True)
|
||||
resolved_entity_id = await knowledge_client.resolve_entity(
|
||||
resolved_identifier, strict=True
|
||||
)
|
||||
return resolved_entity_id
|
||||
|
||||
try:
|
||||
@@ -756,6 +792,31 @@ The destination folder '{destination_folder}' is not allowed - paths must stay w
|
||||
move_note("{identifier}", destination_folder="notes")
|
||||
```"""
|
||||
|
||||
# --- Cross-boundary intent guard (file moves only) ---
|
||||
# Trigger: destination_path now holds the real combined target, whether it came
|
||||
# from destination_path or was resolved from destination_folder above.
|
||||
# Why: detection must run AFTER folder resolution — running it earlier (when a
|
||||
# caller used destination_folder) saw an empty destination_path and skipped
|
||||
# entirely (#881 Gap 3).
|
||||
# Outcome: a cross-workspace/cross-project routing destination is rejected with
|
||||
# guidance instead of silently degrading to a same-project nested folder.
|
||||
cross_project_error = await _detect_cross_project_move_attempt(
|
||||
client, identifier, destination_path, active_project.name
|
||||
)
|
||||
if cross_project_error:
|
||||
logger.info(f"Detected cross-project move attempt: {identifier} -> {destination_path}")
|
||||
if output_format == "json":
|
||||
return {
|
||||
"moved": False,
|
||||
"title": None,
|
||||
"permalink": None,
|
||||
"file_path": None,
|
||||
"source": identifier,
|
||||
"destination": destination_path,
|
||||
"error": "CROSS_PROJECT_MOVE_NOT_SUPPORTED",
|
||||
}
|
||||
return cross_project_error
|
||||
|
||||
# Validate that destination path includes a file extension
|
||||
if "." not in destination_path or not destination_path.split(".")[-1]:
|
||||
logger.warning(f"Move failed - no file extension provided: {destination_path}")
|
||||
@@ -839,6 +900,53 @@ move_note("{identifier}", destination_folder="notes")
|
||||
|
||||
# Call the move API using KnowledgeClient
|
||||
result = await knowledge_client.move_entity(resolved_entity_id, destination_path)
|
||||
|
||||
# --- Outcome validation (honest success backstop) ---
|
||||
# Trigger: the resulting file_path differs from the destination the caller
|
||||
# requested.
|
||||
# Why: move_entity stores the path relative to the *current* project root, so
|
||||
# a cross-boundary intent silently degrades into a same-project nested
|
||||
# folder. The old code reported "✅ moved successfully" regardless,
|
||||
# misleading the agent (#881 Gap 2). This is the robust backstop behind
|
||||
# the up-front detection: any divergence the caller did not ask for
|
||||
# surfaces as a failure rather than a fake success.
|
||||
# Outcome: report failure with the actual landing path instead of "✅".
|
||||
normalized_requested = PureWindowsPath(destination_path).as_posix().strip("/")
|
||||
normalized_actual = PureWindowsPath(result.file_path).as_posix().strip("/")
|
||||
if normalized_actual != normalized_requested:
|
||||
logger.warning(
|
||||
f"Move outcome diverged from intent: requested={destination_path} "
|
||||
f"actual={result.file_path}"
|
||||
)
|
||||
if output_format == "json":
|
||||
return {
|
||||
"moved": False,
|
||||
"title": result.title,
|
||||
"permalink": result.permalink,
|
||||
"file_path": result.file_path,
|
||||
"source": identifier,
|
||||
"destination": destination_path,
|
||||
"error": "MOVE_OUTCOME_MISMATCH",
|
||||
}
|
||||
return dedent(f"""
|
||||
# Move Failed - Unexpected Result Location
|
||||
|
||||
The move of '{identifier}' did not land at the requested destination.
|
||||
|
||||
**Requested:** `{destination_path}`
|
||||
**Actual:** `{result.file_path}`
|
||||
|
||||
This usually means the destination referenced a different
|
||||
workspace/project, which move_note cannot do — notes can only be
|
||||
moved within the same project. To move content between projects:
|
||||
|
||||
```
|
||||
read_note("{result.file_path}")
|
||||
write_note("Title", "content", "folder", project="target-project")
|
||||
delete_note("{result.file_path}", project="{active_project.name}")
|
||||
```
|
||||
""").strip()
|
||||
|
||||
if output_format == "json":
|
||||
return {
|
||||
"moved": True,
|
||||
|
||||
@@ -1,13 +1,14 @@
|
||||
"""Read note tool for Basic Memory MCP server."""
|
||||
|
||||
from textwrap import dedent
|
||||
from typing import Optional, Literal, cast
|
||||
from typing import Annotated, Optional, Literal, cast
|
||||
|
||||
import logfire
|
||||
import yaml
|
||||
|
||||
from loguru import logger
|
||||
from fastmcp import Context
|
||||
from pydantic import AliasChoices, Field
|
||||
|
||||
from basic_memory.config import ConfigManager
|
||||
from basic_memory.mcp.project_context import (
|
||||
@@ -20,6 +21,17 @@ from basic_memory.mcp.tools.search import search_notes
|
||||
from basic_memory.schemas.memory import memory_url_path
|
||||
from basic_memory.utils import validate_project_path
|
||||
|
||||
# The title-match fallback exists to find THE note by exact title, so it scans
|
||||
# fixed-size pages of title results instead of the caller's page/page_size
|
||||
# (which apply only to the text-search suggestion listing).
|
||||
_TITLE_LOOKUP_PAGE_SIZE = 10
|
||||
|
||||
# Hard safety cap on title-lookup pages. The loop normally stops as soon as an
|
||||
# exact match is found or results run out (has_more=False); the cap only bounds
|
||||
# pathological knowledge bases where hundreds of fuzzy titles contain the
|
||||
# queried phrase. Exhausting the cap falls through to the suggestion behavior.
|
||||
_TITLE_LOOKUP_MAX_PAGES = 10
|
||||
|
||||
|
||||
def _is_exact_title_match(identifier: str, title: str) -> bool:
|
||||
"""Return True when identifier exactly matches a title (case-insensitive)."""
|
||||
@@ -71,6 +83,21 @@ async def read_note(
|
||||
identifier: str,
|
||||
project: Optional[str] = None,
|
||||
project_id: Optional[str] = None,
|
||||
# Accept common pagination aliases models reach for from training data
|
||||
# (page_number/limit/per_page), matching the sibling navigation tools
|
||||
# (search_notes, build_context, recent_activity). The schema advertises
|
||||
# only the canonical names; aliases are silently mapped at validation time.
|
||||
# `offset` is intentionally NOT aliased: offset is item-indexed (skip N
|
||||
# items) while page is a 1-indexed page-number, so direct aliasing would
|
||||
# return the wrong slice.
|
||||
page: Annotated[
|
||||
int,
|
||||
Field(default=1, validation_alias=AliasChoices("page", "page_number")),
|
||||
] = 1,
|
||||
page_size: Annotated[
|
||||
int,
|
||||
Field(default=10, validation_alias=AliasChoices("page_size", "limit", "per_page")),
|
||||
] = 10,
|
||||
output_format: Literal["text", "json"] = "text",
|
||||
include_frontmatter: bool = False,
|
||||
context: Context | None = None,
|
||||
@@ -99,6 +126,14 @@ async def read_note(
|
||||
workspaces. Takes precedence over `project`. Get from list_memory_projects().
|
||||
identifier: The title or permalink of the note to read
|
||||
Can be a full memory:// URL, a permalink, a title, or search text
|
||||
page: Page of fallback-search results to use when the identifier does not
|
||||
resolve to a note directly (default: 1). A direct or exact-title match
|
||||
always returns the full note content — page/page_size never chunk the
|
||||
note itself, and the title-match lookup pages through fixed-size pages
|
||||
of title results until an exact match is found or results are
|
||||
exhausted, regardless of page or page_size.
|
||||
page_size: Number of fallback-search results per page (default: 10). When no
|
||||
match is found, this caps how many related-note suggestions are listed.
|
||||
output_format: "text" returns markdown content or guidance text.
|
||||
"json" returns a structured object with title/permalink/file_path/content/frontmatter.
|
||||
include_frontmatter: When output_format="json", whether content should include the
|
||||
@@ -122,6 +157,9 @@ async def read_note(
|
||||
# Read recent meeting notes
|
||||
read_note("team-docs", "Weekly Standup")
|
||||
|
||||
# Page through fallback-search suggestions when nothing matches directly
|
||||
read_note("unknown topic", page=2, page_size=5)
|
||||
|
||||
Raises:
|
||||
HTTPError: If project doesn't exist or is inaccessible
|
||||
SecurityError: If identifier attempts path traversal
|
||||
@@ -130,6 +168,15 @@ async def read_note(
|
||||
If the exact note isn't found, this tool provides helpful suggestions
|
||||
including related notes, search commands, and note creation templates.
|
||||
"""
|
||||
# Trigger: page < 1 or page_size < 1 (e.g. page_size=0 or negative).
|
||||
# Why: both flow into the fallback search's server-side slicing, where
|
||||
# non-positive values produce empty result pages with unreachable
|
||||
# pagination. Fail fast, matching search_notes/build_context.
|
||||
if page < 1:
|
||||
raise ValueError(f"page must be >= 1, got {page}")
|
||||
if page_size < 1:
|
||||
raise ValueError(f"page_size must be >= 1, got {page_size}")
|
||||
|
||||
# Detect project from a memory URL or permalink prefix before routing.
|
||||
# project_id routes by external UUID, so it bypasses URL discovery entirely.
|
||||
if project is None and project_id is None:
|
||||
@@ -147,6 +194,8 @@ async def read_note(
|
||||
tool_name="read_note",
|
||||
requested_project=project,
|
||||
requested_project_id=project_id,
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
output_format=output_format,
|
||||
include_frontmatter=include_frontmatter,
|
||||
):
|
||||
@@ -245,7 +294,7 @@ async def read_note(
|
||||
]
|
||||
|
||||
async def _search_candidates(
|
||||
identifier_text: str, *, title_only: bool
|
||||
identifier_text: str, *, title_only: bool, lookup_page: int = 1
|
||||
) -> dict[str, object]:
|
||||
# Trigger: direct entity resolution failed for the caller's identifier.
|
||||
# Why: search_notes applies the same memory:// normalization and tool-level
|
||||
@@ -256,11 +305,24 @@ async def read_note(
|
||||
# Without this, project names that collide across workspaces could re-resolve
|
||||
# to a different tenant via the default-workspace fallback (CLI/context=None).
|
||||
search_type = "title" if title_only else "text"
|
||||
# Trigger: title_only — the title search exists to find THE note by
|
||||
# exact title, not to page through suggestions.
|
||||
# Why: paginating it by the caller's page would skip an exact match
|
||||
# sitting on page 1 (read_note("Exact Title", page=2)), and a
|
||||
# small caller page_size could let a higher-ranked fuzzy title
|
||||
# displace the exact match out of the lookup window
|
||||
# (read_note("Foo Bar", page_size=1) when "Foo Bar Foo Bar"
|
||||
# ranks first) — both returning suggestions instead of the note.
|
||||
# Outcome: title lookup uses its own lookup_page with a fixed lookup
|
||||
# size, walked by the caller below; caller page/page_size
|
||||
# apply only to the text-search suggestion listing.
|
||||
response = await search_notes(
|
||||
project=active_project.name,
|
||||
project_id=active_project.external_id,
|
||||
query=identifier_text,
|
||||
search_type=search_type,
|
||||
page=lookup_page if title_only else page,
|
||||
page_size=_TITLE_LOOKUP_PAGE_SIZE if title_only else page_size,
|
||||
output_format="json",
|
||||
context=context,
|
||||
)
|
||||
@@ -297,12 +359,24 @@ async def read_note(
|
||||
logger.info(f"Direct lookup failed for '{entity_path}': {e}")
|
||||
# Continue to fallback methods
|
||||
|
||||
# Fallback 1: Try title search via API
|
||||
# Fallback 1: Try title search via API, walking fixed-size pages of
|
||||
# title results until an exact match is found or results run out.
|
||||
# A single page is not enough: when more than _TITLE_LOOKUP_PAGE_SIZE
|
||||
# higher-ranked fuzzy titles contain the queried phrase, the exact
|
||||
# title lands on a later page and a one-page lookup would miss it.
|
||||
logger.info(f"Search title for: {identifier}")
|
||||
title_results = await _search_candidates(identifier, title_only=True)
|
||||
|
||||
title_candidates = _search_results(title_results)
|
||||
if title_candidates:
|
||||
result: dict[str, object] | None = None
|
||||
for lookup_page in range(1, _TITLE_LOOKUP_MAX_PAGES + 1):
|
||||
title_results = await _search_candidates(
|
||||
identifier, title_only=True, lookup_page=lookup_page
|
||||
)
|
||||
title_candidates = _search_results(title_results)
|
||||
if not title_candidates:
|
||||
logger.info(
|
||||
f"No results in title search for: {identifier} "
|
||||
f"in project {active_project.name}"
|
||||
)
|
||||
break
|
||||
# Trigger: direct resolution failed and title search returned candidates.
|
||||
# Why: avoid returning unrelated notes when search yields only fuzzy matches.
|
||||
# Outcome: fetch content only when a true exact title match exists.
|
||||
@@ -314,33 +388,37 @@ async def read_note(
|
||||
),
|
||||
None,
|
||||
)
|
||||
if not result:
|
||||
if result is not None:
|
||||
break
|
||||
# Trigger: this page held only fuzzy titles and the server reports
|
||||
# no further pages (has_more is False or absent).
|
||||
# Why: continuing past the last page would issue empty lookups.
|
||||
# Outcome: give up on the title fallback and try text search below.
|
||||
if title_results.get("has_more") is not True:
|
||||
logger.info(f"No exact title match found for: {identifier}")
|
||||
elif _result_permalink(result):
|
||||
try:
|
||||
# Resolve the permalink to entity ID
|
||||
entity_id = await knowledge_client.resolve_entity(
|
||||
_result_permalink(result) or "", strict=True
|
||||
)
|
||||
break
|
||||
|
||||
# Fetch content using the entity ID
|
||||
response = await resource_client.read(entity_id)
|
||||
if result is not None and _result_permalink(result):
|
||||
try:
|
||||
# Resolve the permalink to entity ID
|
||||
entity_id = await knowledge_client.resolve_entity(
|
||||
_result_permalink(result) or "", strict=True
|
||||
)
|
||||
|
||||
if response.status_code == 200:
|
||||
logger.info(
|
||||
f"Found note by exact title search: {_result_permalink(result)}"
|
||||
)
|
||||
if output_format == "json":
|
||||
return await _read_json_payload(entity_id)
|
||||
return response.text
|
||||
except Exception as e: # pragma: no cover
|
||||
# Fetch content using the entity ID
|
||||
response = await resource_client.read(entity_id)
|
||||
|
||||
if response.status_code == 200:
|
||||
logger.info(
|
||||
f"Failed to fetch content for found title match {_result_permalink(result)}: {e}"
|
||||
f"Found note by exact title search: {_result_permalink(result)}"
|
||||
)
|
||||
else:
|
||||
logger.info(
|
||||
f"No results in title search for: {identifier} in project {active_project.name}"
|
||||
)
|
||||
if output_format == "json":
|
||||
return await _read_json_payload(entity_id)
|
||||
return response.text
|
||||
except Exception as e: # pragma: no cover
|
||||
logger.info(
|
||||
f"Failed to fetch content for found title match {_result_permalink(result)}: {e}"
|
||||
)
|
||||
|
||||
# Fallback 2: Text search as a last resort
|
||||
logger.info(f"Title search failed, trying text search for: {identifier}")
|
||||
@@ -352,6 +430,9 @@ async def read_note(
|
||||
if output_format == "json":
|
||||
return _empty_json_payload()
|
||||
return format_not_found_message(active_project.name, identifier)
|
||||
# The fallback search is paginated server-side to page_size, so list
|
||||
# the whole returned page instead of a hardcoded cap — otherwise the
|
||||
# caller's page_size would be silently ignored past the cap.
|
||||
if output_format == "json":
|
||||
payload = _empty_json_payload()
|
||||
payload["related_results"] = [
|
||||
@@ -360,10 +441,10 @@ async def read_note(
|
||||
"permalink": _result_permalink(result),
|
||||
"file_path": _result_file_path(result),
|
||||
}
|
||||
for result in text_candidates[:5]
|
||||
for result in text_candidates
|
||||
]
|
||||
return payload
|
||||
return format_related_results(active_project.name, identifier, text_candidates[:5])
|
||||
return format_related_results(active_project.name, identifier, text_candidates)
|
||||
|
||||
|
||||
def format_not_found_message(project: str | None, identifier: str) -> str:
|
||||
|
||||
@@ -300,14 +300,20 @@ async def recent_activity(
|
||||
else:
|
||||
# Project-Specific Mode: Get activity for specific project
|
||||
# Uses get_project_client() for per-project routing (local vs cloud)
|
||||
logger.info(
|
||||
f"Getting recent activity from project {resolved_project}: type={type}, depth={depth}, timeframe={timeframe}"
|
||||
)
|
||||
|
||||
async with get_project_client(resolved_project, context=context, project_id=project_id) as (
|
||||
client,
|
||||
active_project,
|
||||
):
|
||||
# Trigger: caller routed by project_id (a UUID), so resolved_project holds the
|
||||
# raw UUID rather than a human-readable name.
|
||||
# Why: active_project.name is the canonical, display-safe project name regardless
|
||||
# of whether routing was by name or by external_id.
|
||||
# Outcome: logs and the formatted text header always show the project name.
|
||||
logger.info(
|
||||
f"Getting recent activity from project {active_project.name}: "
|
||||
f"type={type}, depth={depth}, timeframe={timeframe}"
|
||||
)
|
||||
|
||||
response = await call_get(
|
||||
client,
|
||||
f"/v2/projects/{active_project.external_id}/memory/recent",
|
||||
@@ -319,7 +325,7 @@ async def recent_activity(
|
||||
return _extract_recent_rows(activity_data)
|
||||
|
||||
# Format project-specific mode output
|
||||
return _format_project_output(resolved_project, activity_data, timeframe, type, page)
|
||||
return _format_project_output(active_project.name, activity_data, timeframe, type, page)
|
||||
|
||||
|
||||
async def _get_project_activity(
|
||||
|
||||
@@ -11,7 +11,7 @@ from fastmcp import Context
|
||||
from pydantic import AliasChoices, BeforeValidator, Field
|
||||
|
||||
from basic_memory.config import ConfigManager, has_cloud_credentials
|
||||
from basic_memory.utils import build_canonical_permalink, coerce_dict, coerce_list
|
||||
from basic_memory.utils import build_canonical_permalink, coerce_dict, coerce_list, parse_tags
|
||||
from basic_memory.mcp.async_client import (
|
||||
_explicit_routing,
|
||||
_force_local_mode,
|
||||
@@ -83,6 +83,47 @@ def _format_search_error_response(
|
||||
`search_notes("{project}", "{query}", search_type="{search_type}")`
|
||||
""").strip()
|
||||
|
||||
# Corrupt/missing FastEmbed model cache (interrupted download leaves a partial
|
||||
# snapshot missing model_optimized.onnx; the ONNX runtime then raises NO_SUCHFILE).
|
||||
# Basic Memory self-heals by re-downloading on the next load, but if the user still
|
||||
# hits this, point them at the cache dir to clear manually and offer a text fallback.
|
||||
error_lower = error_message.lower()
|
||||
# "load model from" is the exact ONNX phrasing ("Load model from <path>.onnx failed").
|
||||
# The looser "load model" matched unrelated errors, so we keep only the specific phrase
|
||||
# alongside the onnxruntime / no_suchfile / model_optimized.onnx fingerprints.
|
||||
if (
|
||||
"onnxruntime" in error_lower
|
||||
or "no_suchfile" in error_lower
|
||||
or "model_optimized.onnx" in error_lower
|
||||
or "load model from" in error_lower
|
||||
):
|
||||
# Deferred import: keeps the repository layer out of the tool's import graph
|
||||
# (matches the SearchClient deferral below) and is only needed on this error path.
|
||||
from basic_memory.repository.embedding_provider_factory import _resolve_cache_dir
|
||||
|
||||
try:
|
||||
cache_dir = _resolve_cache_dir(get_container().config)
|
||||
except RuntimeError:
|
||||
cache_dir = _resolve_cache_dir(ConfigManager().config)
|
||||
return dedent(f"""
|
||||
# Search Failed - Embedding Model Missing or Corrupt
|
||||
|
||||
The local FastEmbed model could not be loaded for query '{query}': {error_message}
|
||||
|
||||
This usually means an earlier model download was interrupted and left an
|
||||
incomplete file in the model cache.
|
||||
|
||||
## How to fix
|
||||
1. Delete the FastEmbed model cache so it re-downloads on the next search:
|
||||
`{cache_dir}`
|
||||
2. Run your search again (the model downloads automatically on first use):
|
||||
`search_notes("{project}", "{query}", search_type="{search_type}")`
|
||||
|
||||
## Workaround right now
|
||||
- Use full-text search, which needs no embedding model:
|
||||
`search_notes("{project}", "{query}", search_type="text")`
|
||||
""").strip()
|
||||
|
||||
# FTS5 syntax errors
|
||||
if "syntax error" in error_message.lower() or "fts5" in error_message.lower():
|
||||
clean_query = (
|
||||
@@ -114,7 +155,7 @@ def _format_search_error_response(
|
||||
- Boolean NOT: `project NOT archived`
|
||||
- Grouped: `(project OR planning) AND notes`
|
||||
- Exact phrases: `"weekly standup meeting"`
|
||||
- Content-specific: `tag:example` or `category:observation`
|
||||
- Content-specific: `tag:example`
|
||||
|
||||
## Try again with:
|
||||
```
|
||||
@@ -181,7 +222,7 @@ def _format_search_error_response(
|
||||
|
||||
6. **Try advanced search patterns**:
|
||||
- Tag search: `search_notes("{project}","tag:your-tag")`
|
||||
- Category search: `search_notes("{project}","category:observation")`
|
||||
- Observation category: `search_notes("{project}","{query}", entity_types=["observation"], categories=["requirement"])`
|
||||
- Pattern matching: `search_notes("{project}","*{query}*", search_type="permalink")`
|
||||
|
||||
## Explore what content exists:
|
||||
@@ -258,7 +299,8 @@ Error searching for '{query}': {error_message}
|
||||
- **Boolean**: `term1 AND term2`, `term1 OR term2`, `term1 NOT term2`
|
||||
- **Phrases**: `"exact phrase"`
|
||||
- **Grouping**: `(term1 OR term2) AND term3`
|
||||
- **Patterns**: `tag:example`, `category:observation`"""
|
||||
- **Tags**: `tag:example`
|
||||
- **Observation categories**: `entity_types=["observation"], categories=["requirement"]`"""
|
||||
|
||||
|
||||
def _format_search_markdown(result: SearchResponse, project: str, query: str | None) -> str:
|
||||
@@ -456,6 +498,7 @@ async def _search_all_projects(
|
||||
output_format: Literal["text", "json"],
|
||||
note_types: list[str],
|
||||
entity_types: list[str],
|
||||
categories: list[str],
|
||||
after_date: str | None,
|
||||
metadata_filters: dict[str, Any] | None,
|
||||
tags: list[str] | None,
|
||||
@@ -514,6 +557,7 @@ async def _search_all_projects(
|
||||
output_format="json",
|
||||
note_types=note_types or None,
|
||||
entity_types=entity_types or None,
|
||||
categories=categories or None,
|
||||
after_date=after_date,
|
||||
metadata_filters=metadata_filters,
|
||||
tags=tags,
|
||||
@@ -612,6 +656,14 @@ async def search_notes(
|
||||
"'relation'. Defaults to 'entity'. Do NOT pass schema/frontmatter types like "
|
||||
"'Chapter' here — use note_types instead.",
|
||||
] = None,
|
||||
categories: Annotated[
|
||||
List[str] | None,
|
||||
BeforeValidator(coerce_list),
|
||||
Field(default=None, validation_alias=AliasChoices("categories", "category")),
|
||||
"Filter observation results to these exact categories (e.g. ['requirement']). "
|
||||
"Pair with entity_types=['observation'] to return only observations whose "
|
||||
"category matches exactly — not every row mentioning the word.",
|
||||
] = None,
|
||||
# Time-filter naming varies wildly across APIs.
|
||||
after_date: Annotated[
|
||||
Optional[str],
|
||||
@@ -624,9 +676,13 @@ async def search_notes(
|
||||
Dict[str, Any] | None,
|
||||
BeforeValidator(coerce_dict),
|
||||
] = None,
|
||||
# parse_tags, not coerce_list: tags="a,b" must split into ["a", "b"] to match the
|
||||
# tag: query shorthand below and write_note's documented tags convention (#910).
|
||||
# coerce_list would wrap the comma string as the single literal tag ["a,b"],
|
||||
# which matches nothing.
|
||||
tags: Annotated[
|
||||
List[str] | None,
|
||||
BeforeValidator(coerce_list),
|
||||
BeforeValidator(parse_tags),
|
||||
] = None,
|
||||
status: Optional[str] = None,
|
||||
min_similarity: Annotated[
|
||||
@@ -666,7 +722,8 @@ async def search_notes(
|
||||
|
||||
### Content-Specific Searches
|
||||
- `search_notes("research", "tag:example")` - Search within specific tags (if supported by content)
|
||||
- `search_notes("work-project", "category:observation")` - Filter by observation categories
|
||||
- `search_notes("work-project", "req", entity_types=["observation"], categories=["requirement"])`
|
||||
- Return only observations whose category is exactly "requirement"
|
||||
- `search_notes("team-docs", "author:username")` - Find content by author (if metadata available)
|
||||
|
||||
**Note:** `tag:` shorthand is automatically converted to a `tags` filter, so it works
|
||||
@@ -684,6 +741,8 @@ async def search_notes(
|
||||
- `search_notes("my-project", "query", note_types=["note"])` - Search only notes
|
||||
- `search_notes("work-docs", "query", note_types=["note", "person"])` - Multiple note types
|
||||
- `search_notes("research", "query", entity_types=["observation"])` - Filter by entity type
|
||||
- `search_notes("research", "query", entity_types=["observation"], categories=["requirement"])`
|
||||
- Filter observations to an exact category
|
||||
- `search_notes("team-docs", "query", after_date="2024-01-01")` - Recent content only
|
||||
- `search_notes("my-project", "query", after_date="1 week")` - Relative date filtering
|
||||
- `search_notes("my-project", "query", tags=["security"])` - Filter by frontmatter tags
|
||||
@@ -735,9 +794,14 @@ async def search_notes(
|
||||
"json" returns a machine-readable dictionary payload.
|
||||
note_types: Optional list of note types to search (e.g., ["note", "person"])
|
||||
entity_types: Optional list of entity types to filter by (e.g., ["entity", "observation"])
|
||||
categories: Optional list of observation categories for exact matching (e.g.,
|
||||
["requirement"]). Pair with entity_types=["observation"] to return only
|
||||
observations whose category matches exactly.
|
||||
after_date: Optional date filter for recent content (e.g., "1 week", "2d", "2024-01-01")
|
||||
metadata_filters: Optional structured frontmatter filters (e.g., {"status": "in-progress"})
|
||||
tags: Optional tag filter (frontmatter tags); shorthand for metadata_filters["tags"]
|
||||
tags: Optional tag filter (frontmatter tags); shorthand for metadata_filters["tags"].
|
||||
Accepts a list (["a", "b"]) or a comma-separated string ("a,b"), matching the
|
||||
write_note tags convention and the tag: query shorthand.
|
||||
status: Optional status filter (frontmatter status); shorthand for metadata_filters["status"]
|
||||
min_similarity: Optional float to override the global semantic_min_similarity threshold
|
||||
for this query. E.g., 0.0 to see all vector results, or 0.8 for high precision.
|
||||
@@ -808,10 +872,26 @@ async def search_notes(
|
||||
# Explicit project specification
|
||||
results = await search_notes("project planning", project="my-project")
|
||||
"""
|
||||
# Validate pagination arguments before they reach the API/repository layer.
|
||||
# Trigger: page < 1 or page_size < 1 (e.g. page_size=0 or a negative slice).
|
||||
# Why: a non-positive page_size yields zero rows yet the router computes
|
||||
# has_more = offset + len(results) < total, returning a misleading
|
||||
# has_more=True with no reachable page; a negative page_size becomes an
|
||||
# uncapped SQLite LIMIT. Mirrors recent_activity's guard so all navigation
|
||||
# tools reject invalid pagination consistently.
|
||||
# Outcome: caller gets an explicit ValueError instead of a silent bad payload.
|
||||
if page < 1:
|
||||
raise ValueError(f"page must be >= 1, got {page}")
|
||||
if page_size < 1:
|
||||
raise ValueError(f"page_size must be >= 1, got {page_size}")
|
||||
|
||||
# Avoid mutable-default-argument footguns. Treat None as "no filter".
|
||||
# Lowercase note_types so "Chapter" matches the stored "chapter".
|
||||
note_types = [t.lower() for t in note_types] if note_types else []
|
||||
entity_types = entity_types or []
|
||||
# Categories are matched exactly against the indexed observation category,
|
||||
# so preserve their original casing (unlike the lowercased note_types).
|
||||
categories = categories or []
|
||||
|
||||
# Parse tag:<value> shorthand at tool level so it works with all search modes.
|
||||
# Handles "tag:security", "tag:coffee tag:brewing", "tag:coffee AND tag:brewing".
|
||||
@@ -853,6 +933,7 @@ async def search_notes(
|
||||
output_format=output_format,
|
||||
note_types=note_types,
|
||||
entity_types=entity_types,
|
||||
categories=categories,
|
||||
after_date=after_date,
|
||||
metadata_filters=metadata_filters,
|
||||
tags=tags,
|
||||
@@ -876,8 +957,15 @@ async def search_notes(
|
||||
has_query=bool(query and query.strip()),
|
||||
note_type_filter_count=len(note_types),
|
||||
entity_type_filter_count=len(entity_types),
|
||||
category_filter_count=len(categories),
|
||||
has_filters=bool(
|
||||
metadata_filters or tags or status or note_types or entity_types or after_date
|
||||
metadata_filters
|
||||
or tags
|
||||
or status
|
||||
or note_types
|
||||
or entity_types
|
||||
or categories
|
||||
or after_date
|
||||
),
|
||||
has_tags_filter=bool(tags),
|
||||
has_status_filter=bool(status),
|
||||
@@ -940,6 +1028,8 @@ async def search_notes(
|
||||
# Add optional filters if provided (empty lists are treated as no filter)
|
||||
if entity_types:
|
||||
search_query.entity_types = [SearchItemType(t) for t in entity_types]
|
||||
if categories:
|
||||
search_query.categories = categories
|
||||
if note_types:
|
||||
search_query.note_types = note_types
|
||||
if after_date:
|
||||
@@ -965,7 +1055,8 @@ async def search_notes(
|
||||
return (
|
||||
"# No Search Criteria\n\n"
|
||||
"Please provide at least one of: `query`, `metadata_filters`, "
|
||||
"`tags`, `status`, `note_types`, `entity_types`, or `after_date`."
|
||||
"`tags`, `status`, `note_types`, `entity_types`, `categories`, "
|
||||
"or `after_date`."
|
||||
)
|
||||
|
||||
# Default to entity-level results to avoid returning individual
|
||||
@@ -973,7 +1064,17 @@ async def search_notes(
|
||||
# Applied after no_criteria() so that the implicit default doesn't
|
||||
# mask a truly empty search request.
|
||||
if not search_query.entity_types:
|
||||
search_query.entity_types = [SearchItemType("entity")]
|
||||
# Trigger: a category filter was supplied without an explicit
|
||||
# entity_types.
|
||||
# Why: categories only exist on observations — defaulting to "entity"
|
||||
# (whose rows have NULL category) would AND a category filter against
|
||||
# entity rows and return nothing, defeating a category-only search.
|
||||
# Outcome: scope the implicit default to observations so
|
||||
# search_notes(categories=[...]) returns the matching bullets.
|
||||
if search_query.categories:
|
||||
search_query.entity_types = [SearchItemType("observation")]
|
||||
else:
|
||||
search_query.entity_types = [SearchItemType("entity")]
|
||||
|
||||
logger.debug(
|
||||
f"Search request: project={active_project.name} "
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
"""Write note tool for Basic Memory MCP server."""
|
||||
|
||||
import textwrap
|
||||
from pathlib import Path
|
||||
from typing import Annotated, List, Union, Optional, Literal
|
||||
|
||||
import logfire
|
||||
@@ -86,8 +87,12 @@ async def write_note(
|
||||
Use forward slashes (/) as separators. Use "/" or "" to write to project root.
|
||||
Examples: "notes", "projects/2025", "research/ml", "/" (root)
|
||||
project: Project name to write to. Optional - server will resolve using the
|
||||
hierarchy above. If unknown, use list_memory_projects() to discover
|
||||
available projects.
|
||||
hierarchy above. Use "workspace/project" to route to a project in a
|
||||
specific cloud workspace. A bare name that exists in multiple
|
||||
workspaces resolves to the default workspace, so use the qualified
|
||||
form (or project_id) to disambiguate. If unknown, use
|
||||
list_memory_projects() to discover available projects and their
|
||||
qualified names.
|
||||
project_id: Project external_id (UUID). Prefer this over `project` when known —
|
||||
it routes to the exact project regardless of name collisions across cloud
|
||||
workspaces. Takes precedence over `project`. Get from list_memory_projects().
|
||||
@@ -269,7 +274,19 @@ async def write_note(
|
||||
raise ValueError(
|
||||
"Entity permalink is required for updates"
|
||||
) # pragma: no cover
|
||||
entity_id = await knowledge_client.resolve_entity(entity.permalink)
|
||||
# Resolve the conflicting entity by file_path with strict=True.
|
||||
# The 409 came from a file_service.exists(file_path) check, so this
|
||||
# file_path is the authoritative key for the canonical row. Resolving
|
||||
# by permalink with fuzzy fallback (the previous behavior) could pick
|
||||
# an orphan with a similar permalink — especially in workspace-prefixed
|
||||
# palaces where the client-built permalink omits the workspace slug —
|
||||
# causing the update to write to the wrong row and the next call to
|
||||
# mint a -1/-2 suffix on the canonical entity.
|
||||
# POSIX-normalize so Windows clients send the same form the server stores.
|
||||
file_path_identifier = Path(entity.file_path).as_posix()
|
||||
entity_id = await knowledge_client.resolve_entity(
|
||||
file_path_identifier, strict=True
|
||||
)
|
||||
result = await knowledge_client.update_entity(
|
||||
entity_id, entity.model_dump()
|
||||
)
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
"""Knowledge graph models."""
|
||||
|
||||
import hashlib
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
from basic_memory.utils import ensure_timezone_aware
|
||||
@@ -252,8 +253,18 @@ class Observation(Base):
|
||||
Content is truncated to 200 chars to stay under PostgreSQL's
|
||||
btree index limit of 2704 bytes.
|
||||
"""
|
||||
# Truncate content to avoid exceeding PostgreSQL's btree index limit
|
||||
content_for_permalink = self.content[:200] if len(self.content) > 200 else self.content
|
||||
if len(self.content) > 200:
|
||||
# Trigger: content exceeds the 200-char budget imposed by PostgreSQL's
|
||||
# 2704-byte btree index row limit, so the permalink can only carry a prefix.
|
||||
# Why: two distinct observations with the same category and an identical
|
||||
# 200-char prefix would collide on the same synthetic permalink, and the
|
||||
# search index (permalink-keyed upsert) silently drops the second one.
|
||||
# Outcome: a short stable digest of the FULL content disambiguates
|
||||
# truncated permalinks while staying well under the index limit.
|
||||
digest = hashlib.sha256(self.content.encode("utf-8")).hexdigest()[:12]
|
||||
content_for_permalink = f"{self.content[:200]}-{digest}"
|
||||
else:
|
||||
content_for_permalink = self.content
|
||||
return generate_permalink(
|
||||
f"{self.entity.permalink}/observations/{self.category}/{content_for_permalink}"
|
||||
)
|
||||
|
||||
@@ -3,18 +3,29 @@
|
||||
import os
|
||||
from threading import Lock
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from basic_memory.config import BasicMemoryConfig, default_fastembed_cache_dir
|
||||
from basic_memory.repository.embedding_provider import EmbeddingProvider
|
||||
|
||||
# Cache key fields are limited to values that change the *identity* of the loaded
|
||||
# model (provider, model_name, dimensions, LiteLLM role/input-type/forward-dimension
|
||||
# settings, batch/request knobs that affect the LiteLLM identity, and the resolved
|
||||
# cache dir). Thread/parallel knobs are deliberately excluded — they change ONNX
|
||||
# *execution* only, not the loaded weights. Including them caused #872: in a
|
||||
# container/cgroup the CPU-derived thread count can drift between calls, producing
|
||||
# a fresh cache key and reloading the ~2.3GB model into a CPU arena that never
|
||||
# returns memory to the OS.
|
||||
type ProviderCacheKey = tuple[
|
||||
str,
|
||||
str,
|
||||
int | None,
|
||||
bool | None,
|
||||
int,
|
||||
int,
|
||||
str | None,
|
||||
str | None,
|
||||
str,
|
||||
int | None,
|
||||
int | None,
|
||||
]
|
||||
|
||||
_EMBEDDING_PROVIDER_CACHE: dict[ProviderCacheKey, EmbeddingProvider] = {}
|
||||
@@ -75,22 +86,26 @@ def _resolve_fastembed_runtime_knobs(
|
||||
|
||||
|
||||
def _provider_cache_key(app_config: BasicMemoryConfig) -> ProviderCacheKey:
|
||||
"""Build a stable cache key from provider-relevant semantic embedding config.
|
||||
"""Build a stable cache key from model-identity semantic embedding config.
|
||||
|
||||
Uses the *resolved* cache dir — not the raw config field — so different
|
||||
FASTEMBED_CACHE_PATH values produce distinct cache keys even when the
|
||||
config field itself is unset.
|
||||
|
||||
Deliberately excludes the FastEmbed thread/parallel knobs: they tune ONNX
|
||||
execution, not which model weights are loaded, and resolving them from the
|
||||
runtime CPU budget makes the key drift between calls in a container (#872).
|
||||
"""
|
||||
resolved_threads, resolved_parallel = _resolve_fastembed_runtime_knobs(app_config)
|
||||
return (
|
||||
app_config.semantic_embedding_provider.strip().lower(),
|
||||
app_config.semantic_embedding_model,
|
||||
app_config.semantic_embedding_dimensions,
|
||||
app_config.semantic_embedding_forward_dimensions,
|
||||
app_config.semantic_embedding_batch_size,
|
||||
app_config.semantic_embedding_request_concurrency,
|
||||
app_config.semantic_embedding_document_input_type,
|
||||
app_config.semantic_embedding_query_input_type,
|
||||
_resolve_cache_dir(app_config),
|
||||
resolved_threads,
|
||||
resolved_parallel,
|
||||
)
|
||||
|
||||
|
||||
@@ -103,10 +118,21 @@ def reset_embedding_provider_cache() -> None:
|
||||
def create_embedding_provider(app_config: BasicMemoryConfig) -> EmbeddingProvider:
|
||||
"""Create an embedding provider based on semantic config.
|
||||
|
||||
When semantic_embedding_dimensions is set in config, it overrides
|
||||
the provider's default dimensions (384 for FastEmbed, 1536 for OpenAI).
|
||||
When semantic_embedding_dimensions is set in config, it overrides the
|
||||
provider's default dimensions (384 for FastEmbed, 1536 for OpenAI and
|
||||
the LiteLLM OpenAI default). Custom LiteLLM models require an explicit
|
||||
dimension because the vector table schema is created before the first
|
||||
embedding response is available.
|
||||
"""
|
||||
cache_key = _provider_cache_key(app_config)
|
||||
# Trigger: two threads miss the cache for the same key concurrently.
|
||||
# Why: provider construction loads the ~2.3GB ONNX model and is slow, so we
|
||||
# deliberately build it *outside* the lock to avoid serializing every caller
|
||||
# behind a single cold start. This opens a by-design TOCTOU window where both
|
||||
# threads may construct a provider.
|
||||
# Outcome: the second check-and-set below resolves the race — the first writer
|
||||
# wins and the loser's redundant provider is discarded, so the cache still
|
||||
# yields a single process-wide singleton per key.
|
||||
with _EMBEDDING_PROVIDER_CACHE_LOCK:
|
||||
if cached_provider := _EMBEDDING_PROVIDER_CACHE.get(cache_key):
|
||||
return cached_provider
|
||||
@@ -151,11 +177,49 @@ def create_embedding_provider(app_config: BasicMemoryConfig) -> EmbeddingProvide
|
||||
request_concurrency=app_config.semantic_embedding_request_concurrency,
|
||||
**extra_kwargs,
|
||||
)
|
||||
elif provider_name == "litellm":
|
||||
from basic_memory.repository.litellm_provider import LiteLLMEmbeddingProvider
|
||||
|
||||
model_name = app_config.semantic_embedding_model or "openai/text-embedding-3-small"
|
||||
if model_name == "bge-small-en-v1.5":
|
||||
model_name = "openai/text-embedding-3-small"
|
||||
if (
|
||||
app_config.semantic_embedding_dimensions is None
|
||||
and model_name != "openai/text-embedding-3-small"
|
||||
):
|
||||
raise ValueError(
|
||||
"semantic_embedding_dimensions must be set when "
|
||||
"semantic_embedding_provider='litellm' uses a non-default model. "
|
||||
f"Configured model: {model_name!r}."
|
||||
)
|
||||
provider = LiteLLMEmbeddingProvider(
|
||||
model_name=model_name,
|
||||
batch_size=app_config.semantic_embedding_batch_size,
|
||||
request_concurrency=app_config.semantic_embedding_request_concurrency,
|
||||
document_input_type=app_config.semantic_embedding_document_input_type,
|
||||
query_input_type=app_config.semantic_embedding_query_input_type,
|
||||
forward_dimensions=app_config.semantic_embedding_forward_dimensions,
|
||||
**extra_kwargs,
|
||||
)
|
||||
else:
|
||||
raise ValueError(f"Unsupported semantic embedding provider: {provider_name}")
|
||||
|
||||
with _EMBEDDING_PROVIDER_CACHE_LOCK:
|
||||
if cached_provider := _EMBEDDING_PROVIDER_CACHE.get(cache_key):
|
||||
return cached_provider
|
||||
# Trigger: a distinct cache key is being inserted while the cache already
|
||||
# holds entries for other keys.
|
||||
# Why: the provider is meant to be a process-wide singleton (#872). A second
|
||||
# key means something bypassed reuse — a real config change, or a regression
|
||||
# that reintroduces volatile fields into the key — and each new key reloads
|
||||
# the ~2.3GB ONNX model into a CPU arena that never releases memory.
|
||||
# Outcome: surface the bypass so future leaks are diagnosable from logs.
|
||||
if _EMBEDDING_PROVIDER_CACHE:
|
||||
logger.warning(
|
||||
"Creating a second distinct embedding provider in this process; "
|
||||
"the model will be loaded again. existing_keys={existing} new_key={new}",
|
||||
existing=list(_EMBEDDING_PROVIDER_CACHE.keys()),
|
||||
new=cache_key,
|
||||
)
|
||||
_EMBEDDING_PROVIDER_CACHE[cache_key] = provider
|
||||
return provider
|
||||
|
||||
@@ -178,21 +178,31 @@ class EntityRepository(Repository[Entity]):
|
||||
result = await self.execute_query(query, use_query_options=False)
|
||||
return list(result.scalars().all())
|
||||
|
||||
async def find_by_ids_for_hydration(self, ids: List[int]) -> Sequence[Entity]:
|
||||
async def find_by_ids_for_hydration(
|
||||
self, ids: List[int], *, include_cross_project: bool = False
|
||||
) -> Sequence[Entity]:
|
||||
"""Fetch minimal entity fields needed for context hydration.
|
||||
|
||||
Context hydration only needs an entity's primary key, title, and external
|
||||
UUID. Keeping this separate from find_by_ids avoids the relationship eager
|
||||
loads that are useful for full entity reads but expensive for response shaping.
|
||||
|
||||
Args:
|
||||
ids: Entity IDs to hydrate.
|
||||
include_cross_project: Include IDs outside this repository's project scope.
|
||||
Use only for IDs already reached through validated graph traversal.
|
||||
"""
|
||||
if not ids:
|
||||
return []
|
||||
|
||||
query = (
|
||||
self.select()
|
||||
select(Entity)
|
||||
.where(Entity.id.in_(ids))
|
||||
.options(load_only(Entity.id, Entity.title, Entity.external_id))
|
||||
)
|
||||
if not include_cross_project:
|
||||
query = self._add_project_filter(query)
|
||||
|
||||
result = await self.execute_query(query, use_query_options=False)
|
||||
return list(result.scalars().all())
|
||||
|
||||
|
||||
@@ -3,6 +3,9 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import math
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from loguru import logger
|
||||
@@ -14,6 +17,24 @@ if TYPE_CHECKING:
|
||||
from fastembed import TextEmbedding # pragma: no cover
|
||||
|
||||
|
||||
# Substrings that identify the ONNX "model artifact file is missing" load failure (as
|
||||
# opposed to a config error, a download/network error, or a genuinely offline machine).
|
||||
# An interrupted FastEmbed download can leave the HuggingFace snapshot dir present but
|
||||
# missing ``model_optimized.onnx``; the ONNX runtime then raises ``NO_SUCHFILE`` and every
|
||||
# subsequent load repeats it until the cache is cleared. Matched case-insensitively.
|
||||
#
|
||||
# IMPORTANT: this text match is necessary but NOT sufficient to trigger a purge. The error
|
||||
# text alone cannot distinguish a corrupt cache from a normal cold load (model not yet
|
||||
# downloaded). Purging is gated on a positive filesystem confirmation that the snapshot dir
|
||||
# exists on disk but the model artifact file is missing — see ``_corrupt_model_subdirs``.
|
||||
_MISSING_ARTIFACT_ERROR_MARKERS = (
|
||||
"no_suchfile",
|
||||
"model_optimized.onnx",
|
||||
"file doesn't exist",
|
||||
"no such file",
|
||||
)
|
||||
|
||||
|
||||
class FastEmbedEmbeddingProvider(EmbeddingProvider):
|
||||
"""Local ONNX embedding provider backed by FastEmbed."""
|
||||
|
||||
@@ -52,6 +73,159 @@ class FastEmbedEmbeddingProvider(EmbeddingProvider):
|
||||
self._model: TextEmbedding | None = None
|
||||
self._model_lock = asyncio.Lock()
|
||||
|
||||
def _resolved_model_name(self) -> str:
|
||||
"""Return the FastEmbed model name after applying our local aliases."""
|
||||
return self._MODEL_ALIASES.get(self.model_name, self.model_name)
|
||||
|
||||
def _create_model(self) -> "TextEmbedding":
|
||||
try:
|
||||
from fastembed import TextEmbedding
|
||||
except ImportError as exc: # pragma: no cover - exercised via tests with monkeypatch
|
||||
raise SemanticDependenciesMissingError(
|
||||
"fastembed package is missing. "
|
||||
"Install/update basic-memory to include semantic dependencies: "
|
||||
"pip install -U basic-memory"
|
||||
) from exc
|
||||
resolved_model_name = self._resolved_model_name()
|
||||
# Constraint: onnxruntime's CPU memory arena grows to fit peak usage and never
|
||||
# returns that memory to the OS. If a model is ever loaded more than once in a
|
||||
# long-running process it leaks tens of GB (#872). FastEmbed exposes
|
||||
# enable_cpu_mem_arena via its session-option kwargs, so we disable the arena to
|
||||
# let any transient extra load free memory.
|
||||
model_kwargs: dict = {
|
||||
"model_name": resolved_model_name,
|
||||
"enable_cpu_mem_arena": False,
|
||||
}
|
||||
if self.cache_dir is not None:
|
||||
model_kwargs["cache_dir"] = self.cache_dir
|
||||
if self.threads is not None:
|
||||
model_kwargs["threads"] = self.threads
|
||||
return TextEmbedding(**model_kwargs)
|
||||
|
||||
def _model_cache_candidates(self) -> list[tuple[Path, str]]:
|
||||
"""Resolve ``(snapshot_dir, model_file)`` pairs for this model under ``cache_dir``.
|
||||
|
||||
FastEmbed stores each model under ``<cache_dir>/models--<org>--<repo>`` where the
|
||||
repo is the model's HuggingFace source (e.g. ``BAAI/bge-small-en-v1.5`` resolves to
|
||||
``models--qdrant--bge-small-en-v1.5-onnx-q``). We resolve the source and the expected
|
||||
model artifact filename from FastEmbed's own model description so corruption detection
|
||||
and deletion are scoped to exactly this model's tree — never the whole cache or
|
||||
unrelated models.
|
||||
|
||||
Note: ``TextEmbedding._list_supported_models()`` is an intentional use of an
|
||||
undocumented FastEmbed API. The broad ``except`` below is a known defensive fallback:
|
||||
if the lookup ever changes shape we degrade to "no candidates" (so we never purge)
|
||||
rather than crashing the load path.
|
||||
"""
|
||||
if self.cache_dir is None:
|
||||
return []
|
||||
|
||||
# FastEmbed matches model names case-insensitively (model_management.py:
|
||||
# ``model_name.lower() == model.model.lower()``). Mirror that here so a config like
|
||||
# model="baai/bge-small-en-v1.5" still resolves to the same HF source/cache subdir.
|
||||
resolved_model_name = self._resolved_model_name().lower()
|
||||
candidates: list[tuple[Path, str]] = []
|
||||
seen: set[Path] = set()
|
||||
cache_root = Path(self.cache_dir)
|
||||
try:
|
||||
from fastembed import TextEmbedding
|
||||
|
||||
for description in TextEmbedding._list_supported_models():
|
||||
if description.model.lower() != resolved_model_name:
|
||||
continue
|
||||
hf_source = description.sources.hf
|
||||
model_file = description.model_file
|
||||
if not hf_source or not model_file:
|
||||
continue
|
||||
# HuggingFace hub names cache dirs ``models--<repo with '/' -> '--'>``.
|
||||
snapshot_dir = cache_root / f"models--{hf_source.replace('/', '--')}"
|
||||
if snapshot_dir not in seen:
|
||||
seen.add(snapshot_dir)
|
||||
candidates.append((snapshot_dir, model_file))
|
||||
except Exception as exc: # pragma: no cover - defensive: never block load on lookup
|
||||
logger.warning(
|
||||
"Could not resolve FastEmbed model source for cache cleanup: "
|
||||
"model_name={model_name} error={error}",
|
||||
model_name=resolved_model_name,
|
||||
error=exc,
|
||||
)
|
||||
|
||||
return candidates
|
||||
|
||||
def _corrupt_model_subdirs(self) -> list[Path]:
|
||||
"""Return cache subdirs that are POSITIVELY confirmed corrupt by filesystem state.
|
||||
|
||||
A model is corrupt when its HuggingFace cache dir exists on disk but at least one
|
||||
materialized snapshot revision is missing the expected model artifact file (e.g.
|
||||
``model_optimized.onnx``) — the exact fingerprint of an interrupted download. A normal
|
||||
cold load (no cache dir yet) is NOT corruption and yields no entries here, so it can
|
||||
never trigger a purge.
|
||||
|
||||
Inspection is PER-REVISION on purpose: HuggingFace keeps multiple revisions under one
|
||||
``models--<repo>`` tree, so a corrupt current snapshot can coexist with an older
|
||||
complete one. Checking ``rglob(model_file)`` across the whole tree would let the old
|
||||
artifact mask the broken current revision and leave it self-perpetuating, so we
|
||||
require every revision to carry the artifact.
|
||||
"""
|
||||
corrupt: list[Path] = []
|
||||
for model_dir, model_file in self._model_cache_candidates():
|
||||
# Trigger: the model's cache dir does not exist at all.
|
||||
# Why: this is a normal cold/first load — the model simply hasn't been
|
||||
# downloaded yet. Purging here would be wrong and pointless.
|
||||
# Outcome: skip; not corrupt.
|
||||
if not model_dir.exists():
|
||||
continue
|
||||
snapshots_root = model_dir / "snapshots"
|
||||
revision_dirs = (
|
||||
[d for d in snapshots_root.iterdir() if d.is_dir()]
|
||||
if snapshots_root.is_dir()
|
||||
else []
|
||||
)
|
||||
# Trigger: the cache dir exists but no snapshot revision has materialized.
|
||||
# Why/Outcome: an interrupted download that never wrote a revision — corrupt.
|
||||
if not revision_dirs:
|
||||
corrupt.append(model_dir)
|
||||
continue
|
||||
# Trigger: any individual revision is missing the artifact (rglob covers the
|
||||
# artifact at any depth within that revision, e.g. snapshots/<rev>/onnx/...).
|
||||
# Why: a complete OLD revision must not mask a corrupt CURRENT one.
|
||||
# Outcome: flag the model dir so the whole tree re-downloads cleanly.
|
||||
if any(not any(rev.rglob(model_file)) for rev in revision_dirs):
|
||||
corrupt.append(model_dir)
|
||||
return corrupt
|
||||
|
||||
def _purge_model_subdirs(self, subdirs: list[Path]) -> bool:
|
||||
"""Delete confirmed-corrupt cache subtrees so the next load re-downloads them.
|
||||
|
||||
Returns True when at least one targeted subdir is actually gone afterwards. On
|
||||
Windows a locked file can make ``shutil.rmtree(ignore_errors=True)`` silently no-op;
|
||||
reporting success in that case would let the caller retry against the same broken
|
||||
cache, so each subdir only counts as removed once it has actually disappeared.
|
||||
"""
|
||||
removed_any = False
|
||||
for subdir in subdirs:
|
||||
logger.warning(
|
||||
"Removing corrupt FastEmbed model cache to force re-download: {path}",
|
||||
path=str(subdir),
|
||||
)
|
||||
shutil.rmtree(subdir, ignore_errors=True)
|
||||
# Set removed only when the subdir is truly gone — a silent rmtree no-op
|
||||
# (e.g. a locked file on Windows) must not be reported as a successful purge.
|
||||
if not subdir.exists():
|
||||
removed_any = True
|
||||
return removed_any
|
||||
|
||||
@staticmethod
|
||||
def _is_missing_artifact_error(exc: Exception) -> bool:
|
||||
"""Return True when the load failure text matches the ONNX missing-artifact signature.
|
||||
|
||||
This is only the text-level gate; it is necessary but NOT sufficient to purge. The
|
||||
purge additionally requires filesystem-confirmed corruption (``_corrupt_model_subdirs``)
|
||||
so a transient/offline/"from any source" load error never deletes a valid cache.
|
||||
"""
|
||||
message = str(exc).lower()
|
||||
return any(marker in message for marker in _MISSING_ARTIFACT_ERROR_MARKERS)
|
||||
|
||||
async def _load_model(self) -> "TextEmbedding":
|
||||
if self._model is not None:
|
||||
return self._model
|
||||
@@ -60,36 +234,42 @@ class FastEmbedEmbeddingProvider(EmbeddingProvider):
|
||||
if self._model is not None:
|
||||
return self._model
|
||||
|
||||
def _create_model() -> "TextEmbedding":
|
||||
try:
|
||||
from fastembed import TextEmbedding
|
||||
except (
|
||||
ImportError
|
||||
) as exc: # pragma: no cover - exercised via tests with monkeypatch
|
||||
raise SemanticDependenciesMissingError(
|
||||
"fastembed package is missing. "
|
||||
"Install/update basic-memory to include semantic dependencies: "
|
||||
"pip install -U basic-memory"
|
||||
) from exc
|
||||
resolved_model_name = self._MODEL_ALIASES.get(self.model_name, self.model_name)
|
||||
if self.cache_dir is not None and self.threads is not None:
|
||||
return TextEmbedding(
|
||||
model_name=resolved_model_name,
|
||||
cache_dir=self.cache_dir,
|
||||
threads=self.threads,
|
||||
)
|
||||
if self.cache_dir is not None:
|
||||
return TextEmbedding(model_name=resolved_model_name, cache_dir=self.cache_dir)
|
||||
if self.threads is not None:
|
||||
return TextEmbedding(model_name=resolved_model_name, threads=self.threads)
|
||||
return TextEmbedding(model_name=resolved_model_name)
|
||||
try:
|
||||
self._model = await asyncio.to_thread(self._create_model)
|
||||
except Exception as exc:
|
||||
# Trigger: model construction raised the ONNX missing-artifact error AND a
|
||||
# filesystem check positively confirms a corrupt cache subdir (the
|
||||
# snapshot dir exists but the model artifact file is missing — the
|
||||
# fingerprint of an interrupted download).
|
||||
# Why: the raw ONNXRuntimeError is self-perpetuating — every retry hits the
|
||||
# same broken snapshot until the cache is cleared. We must NOT misread a
|
||||
# normal cold load (no snapshot dir, model simply not downloaded yet) or a
|
||||
# transient/offline "from any source" error as corruption, because purging
|
||||
# then breaks the happy path. Both the error-text gate and the positive
|
||||
# filesystem confirmation are required before we delete anything.
|
||||
# Outcome: confirmed corruption → purge exactly this model's subdir and retry
|
||||
# once so a fresh download can land. Every other failure (including a
|
||||
# retry that still fails) re-raises the ORIGINAL exception so the
|
||||
# message stays actionable and we never loop.
|
||||
if not self._is_missing_artifact_error(exc):
|
||||
raise
|
||||
corrupt_subdirs = self._corrupt_model_subdirs()
|
||||
if not corrupt_subdirs:
|
||||
raise
|
||||
if not self._purge_model_subdirs(corrupt_subdirs):
|
||||
raise
|
||||
logger.info(
|
||||
"Retrying FastEmbed model load after clearing corrupt cache: "
|
||||
"model_name={model_name}",
|
||||
model_name=self._resolved_model_name(),
|
||||
)
|
||||
self._model = await asyncio.to_thread(self._create_model)
|
||||
|
||||
self._model = await asyncio.to_thread(_create_model)
|
||||
logger.info(
|
||||
"FastEmbed model loaded: model_name={model_name} batch_size={batch_size} "
|
||||
"threads={threads} configured_parallel={configured_parallel} "
|
||||
"effective_parallel={effective_parallel}",
|
||||
model_name=self._MODEL_ALIASES.get(self.model_name, self.model_name),
|
||||
model_name=self._resolved_model_name(),
|
||||
batch_size=self.batch_size,
|
||||
threads=self.threads,
|
||||
configured_parallel=self.parallel,
|
||||
@@ -119,10 +299,17 @@ class FastEmbedEmbeddingProvider(EmbeddingProvider):
|
||||
if effective_parallel is not None:
|
||||
embed_kwargs["parallel"] = effective_parallel
|
||||
vectors = list(model.embed(texts, **embed_kwargs))
|
||||
# sqlite_search_repository.py uses a distance-to-similarity formula that assumes
|
||||
# unit-normalized vectors (see the comment on line 65-67 of that file).
|
||||
# Some models (e.g. multilingual ones) return vectors with norm > 1, so we
|
||||
# L2-normalize here to satisfy that contract regardless of the chosen model.
|
||||
normalized: list[list[float]] = []
|
||||
for vector in vectors:
|
||||
values = vector.tolist() if hasattr(vector, "tolist") else vector
|
||||
normalized.append([float(value) for value in values])
|
||||
values = vector.tolist() if hasattr(vector, "tolist") else list(vector)
|
||||
norm = math.sqrt(sum(x * x for x in values))
|
||||
if norm > 0:
|
||||
values = [x / norm for x in values]
|
||||
normalized.append([float(v) for v in values])
|
||||
return normalized
|
||||
|
||||
vectors = await asyncio.to_thread(_embed_batch)
|
||||
|
||||
@@ -0,0 +1,230 @@
|
||||
"""LiteLLM-based embedding provider for semantic indexing.
|
||||
|
||||
Routes embedding requests to 100+ providers (OpenAI, Anthropic, Google, Azure,
|
||||
Bedrock, Cohere, etc.) via the litellm SDK. No proxy server needed.
|
||||
|
||||
Model strings use the ``provider/model`` format, e.g.
|
||||
``openai/text-embedding-3-small``, ``cohere/embed-english-v3.0``,
|
||||
``azure/my-embedding-deployment``.
|
||||
|
||||
See https://docs.litellm.ai/docs/embedding/supported_embedding for all
|
||||
supported embedding models.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import math
|
||||
import os
|
||||
from typing import Any
|
||||
|
||||
from basic_memory.repository.embedding_provider import EmbeddingProvider
|
||||
from basic_memory.repository.semantic_errors import SemanticDependenciesMissingError
|
||||
|
||||
|
||||
def _default_input_types(model_name: str) -> tuple[str | None, str | None]:
|
||||
"""Return role-specific LiteLLM input_type defaults for known asymmetric models."""
|
||||
normalized = model_name.strip().lower()
|
||||
|
||||
# Cohere v3 embeddings require search_document/search_query to distinguish
|
||||
# index-time passages from retrieval-time queries. LiteLLM supports both
|
||||
# direct Cohere model names and provider-prefixed forms.
|
||||
cohere_v3 = (
|
||||
normalized.startswith("cohere/")
|
||||
or normalized.startswith("bedrock/cohere.")
|
||||
or normalized.startswith("cohere.")
|
||||
or normalized.startswith("embed-")
|
||||
) and "-v3" in normalized
|
||||
if cohere_v3:
|
||||
return "search_document", "search_query"
|
||||
|
||||
# NVIDIA retrieval embeddings use passage/query roles. The provider prefix
|
||||
# is part of LiteLLM's model routing, so this stays narrowly scoped.
|
||||
if normalized.startswith("nvidia_nim/"):
|
||||
return "passage", "query"
|
||||
|
||||
return None, None
|
||||
|
||||
|
||||
def _should_forward_dimensions(model_name: str, forward_dimensions: bool | None) -> bool:
|
||||
"""Return whether configured dimensions should be sent to LiteLLM."""
|
||||
if forward_dimensions is not None:
|
||||
return forward_dimensions
|
||||
|
||||
normalized = model_name.strip().lower()
|
||||
|
||||
# Trigger: `dimensions` is both the Basic Memory vector schema size and a
|
||||
# provider-side output-size request parameter in LiteLLM.
|
||||
# Why: fixed-size models such as Cohere v3 still need the schema value for
|
||||
# validation, but LiteLLM maps the request parameter to provider fields they
|
||||
# reject. Only send it for model families that clearly support output-size
|
||||
# control.
|
||||
# Outcome: OpenAI/Azure text-embedding-3 reductions work, while fixed-size
|
||||
# providers keep dimensions local to Basic Memory.
|
||||
return "text-embedding-3" in normalized
|
||||
|
||||
|
||||
def _import_litellm() -> Any:
|
||||
"""Import LiteLLM without letting its import-time dotenv hook read cwd secrets."""
|
||||
# Constraint: LiteLLM 1.85.0 loads .env files at import time when
|
||||
# LITELLM_MODE defaults to DEV. Basic Memory intentionally does not load
|
||||
# arbitrary cwd .env files, so set the production mode before importing
|
||||
# unless the caller already made an explicit LiteLLM choice.
|
||||
os.environ.setdefault("LITELLM_MODE", "PRODUCTION")
|
||||
|
||||
try:
|
||||
import litellm
|
||||
except ImportError as exc:
|
||||
raise SemanticDependenciesMissingError(
|
||||
"litellm dependency is missing. Install with: pip install litellm"
|
||||
) from exc
|
||||
|
||||
return litellm
|
||||
|
||||
|
||||
class LiteLLMEmbeddingProvider(EmbeddingProvider):
|
||||
"""Embedding provider backed by the litellm SDK."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
model_name: str = "openai/text-embedding-3-small",
|
||||
*,
|
||||
batch_size: int = 64,
|
||||
request_concurrency: int = 4,
|
||||
dimensions: int = 1536,
|
||||
api_key: str | None = None,
|
||||
timeout: float = 30.0,
|
||||
document_input_type: str | None = None,
|
||||
query_input_type: str | None = None,
|
||||
forward_dimensions: bool | None = None,
|
||||
) -> None:
|
||||
self.model_name = model_name
|
||||
self.dimensions = dimensions
|
||||
self.batch_size = batch_size
|
||||
self.request_concurrency = request_concurrency
|
||||
self._api_key = api_key
|
||||
self._timeout = timeout
|
||||
default_document_input_type, default_query_input_type = _default_input_types(model_name)
|
||||
self.document_input_type = document_input_type or default_document_input_type
|
||||
self.query_input_type = query_input_type or default_query_input_type
|
||||
self.forward_dimensions = forward_dimensions
|
||||
|
||||
def runtime_log_attrs(self) -> dict[str, Any]:
|
||||
"""Return provider-specific runtime settings suitable for startup logs."""
|
||||
attrs: dict[str, Any] = {
|
||||
"provider_batch_size": self.batch_size,
|
||||
"request_concurrency": self.request_concurrency,
|
||||
}
|
||||
if self.document_input_type:
|
||||
attrs["document_input_type"] = self.document_input_type
|
||||
if self.query_input_type:
|
||||
attrs["query_input_type"] = self.query_input_type
|
||||
if self.forward_dimensions is not None:
|
||||
attrs["forward_dimensions"] = self.forward_dimensions
|
||||
return attrs
|
||||
|
||||
def identity_key(self) -> str:
|
||||
"""Return the embedding semantics that should invalidate stored vectors."""
|
||||
document_input_type = self.document_input_type or "-"
|
||||
query_input_type = self.query_input_type or "-"
|
||||
forward_dimensions = str(
|
||||
_should_forward_dimensions(self.model_name, self.forward_dimensions)
|
||||
).lower()
|
||||
return (
|
||||
f"{self.model_name}:{self.dimensions}:"
|
||||
f"document_input_type={document_input_type}:"
|
||||
f"query_input_type={query_input_type}:"
|
||||
f"forward_dimensions={forward_dimensions}"
|
||||
)
|
||||
|
||||
async def _embed(self, texts: list[str], *, input_type: str | None) -> list[list[float]]:
|
||||
if not texts:
|
||||
return []
|
||||
|
||||
litellm = _import_litellm()
|
||||
|
||||
batches = [
|
||||
texts[start : start + self.batch_size]
|
||||
for start in range(0, len(texts), self.batch_size)
|
||||
]
|
||||
batch_vectors: list[list[list[float]] | None] = [None] * len(batches)
|
||||
semaphore = asyncio.Semaphore(self.request_concurrency)
|
||||
|
||||
async def embed_batch(batch_index: int, batch: list[str]) -> None:
|
||||
async with semaphore:
|
||||
params: dict[str, Any] = {
|
||||
"model": self.model_name,
|
||||
"input": batch,
|
||||
"drop_params": True,
|
||||
"timeout": self._timeout,
|
||||
}
|
||||
if _should_forward_dimensions(self.model_name, self.forward_dimensions):
|
||||
params["dimensions"] = self.dimensions
|
||||
if self._api_key:
|
||||
params["api_key"] = self._api_key
|
||||
if input_type:
|
||||
params["input_type"] = input_type
|
||||
|
||||
response = await litellm.aembedding(**params)
|
||||
|
||||
vectors_by_index: dict[int, list[float]] = {}
|
||||
for item in response.data:
|
||||
if isinstance(item, dict):
|
||||
response_index = int(item["index"])
|
||||
embedding = item["embedding"]
|
||||
else:
|
||||
response_index = int(item.index)
|
||||
embedding = item.embedding
|
||||
if response_index in vectors_by_index:
|
||||
raise RuntimeError(
|
||||
"LiteLLM embedding response returned duplicate vector indexes."
|
||||
)
|
||||
vectors_by_index[response_index] = [float(v) for v in embedding]
|
||||
|
||||
ordered_vectors: list[list[float]] = []
|
||||
for index in range(len(batch)):
|
||||
vector = vectors_by_index.get(index)
|
||||
if vector is None:
|
||||
raise RuntimeError(
|
||||
"LiteLLM embedding response is missing expected vector index."
|
||||
)
|
||||
ordered_vectors.append(vector)
|
||||
|
||||
batch_vectors[batch_index] = ordered_vectors
|
||||
|
||||
await asyncio.gather(
|
||||
*(embed_batch(batch_index, batch) for batch_index, batch in enumerate(batches))
|
||||
)
|
||||
|
||||
all_vectors: list[list[float]] = []
|
||||
for vectors in batch_vectors:
|
||||
if vectors is None:
|
||||
raise RuntimeError("LiteLLM embedding batch did not produce vectors.")
|
||||
all_vectors.extend(vectors)
|
||||
|
||||
# sqlite_search_repository.py maps L2 distance to cosine similarity via
|
||||
# `1 - L²/2`, which is correct only for unit-normalized vectors. LiteLLM
|
||||
# routes to many backends (Cohere, Vertex, Bedrock, etc.); not all of
|
||||
# them return normalized embeddings, so we normalize here to honor the
|
||||
# provider contract regardless of the underlying model.
|
||||
normalized: list[list[float]] = []
|
||||
for vector in all_vectors:
|
||||
norm = math.sqrt(sum(x * x for x in vector))
|
||||
if norm > 0:
|
||||
normalized.append([x / norm for x in vector])
|
||||
else:
|
||||
normalized.append(vector)
|
||||
|
||||
if normalized and len(normalized[0]) != self.dimensions:
|
||||
raise RuntimeError(
|
||||
f"Embedding model returned {len(normalized[0])}-dimensional vectors "
|
||||
f"but provider was configured for {self.dimensions} dimensions."
|
||||
)
|
||||
return normalized
|
||||
|
||||
async def embed_documents(self, texts: list[str]) -> list[list[float]]:
|
||||
return await self._embed(texts, input_type=self.document_input_type)
|
||||
|
||||
async def embed_query(self, text: str) -> list[float]:
|
||||
vectors = await self._embed([text], input_type=self.query_input_type)
|
||||
return vectors[0] if vectors else [0.0] * self.dimensions
|
||||
@@ -705,6 +705,7 @@ class PostgresSearchRepository(SearchRepositoryBase):
|
||||
note_types: Optional[List[str]] = None,
|
||||
after_date: Optional[datetime] = None,
|
||||
search_item_types: Optional[List[SearchItemType]] = None,
|
||||
categories: Optional[List[str]] = None,
|
||||
metadata_filters: Optional[dict] = None,
|
||||
) -> tuple[str, str, dict, str, str]:
|
||||
"""Build Postgres FTS FROM/WHERE params shared by search and count."""
|
||||
@@ -762,14 +763,36 @@ class PostgresSearchRepository(SearchRepositoryBase):
|
||||
type_placeholders.append(f":{param_name}")
|
||||
conditions.append(f"search_index.type IN ({', '.join(type_placeholders)})")
|
||||
|
||||
# Handle note type filter using JSONB containment (parameterized)
|
||||
# Handle observation category filter (parameterized for defense-in-depth).
|
||||
# Trigger: caller passed `categories` to scope observation results.
|
||||
# Why: `entity_types=["observation"]` only narrows to the observation row type;
|
||||
# callers expect exact-category matching, not incidental text matches.
|
||||
# Outcome: only rows whose indexed category exactly equals a requested value
|
||||
# survive (entities/relations have NULL category and are excluded).
|
||||
if categories:
|
||||
category_placeholders = []
|
||||
for idx, category in enumerate(categories):
|
||||
param_name = f"category_{idx}"
|
||||
params[param_name] = category
|
||||
category_placeholders.append(f":{param_name}")
|
||||
conditions.append(f"search_index.category IN ({', '.join(category_placeholders)})")
|
||||
|
||||
# Handle note type filter (frontmatter type field, parameterized).
|
||||
# Trigger: caller passed `note_types` to scope by the frontmatter `type` field.
|
||||
# Why: the stored note_type preserves the frontmatter casing (e.g. `Chapter`),
|
||||
# but the filter is documented case-insensitive. JSONB `@>` containment is
|
||||
# exact-match, so capitalized types were unfindable.
|
||||
# Outcome: compare LOWER(metadata->>'note_type') against lowercased filter
|
||||
# values so `note_types=["Chapter"]` matches a stored `Chapter`.
|
||||
if note_types:
|
||||
type_conditions = []
|
||||
type_placeholders = []
|
||||
for idx, note_type in enumerate(note_types):
|
||||
param_name = f"note_type_{idx}"
|
||||
params[param_name] = json.dumps({"note_type": note_type})
|
||||
type_conditions.append(f"search_index.metadata @> CAST(:{param_name} AS jsonb)")
|
||||
conditions.append(f"({' OR '.join(type_conditions)})")
|
||||
params[param_name] = note_type.lower()
|
||||
type_placeholders.append(f":{param_name}")
|
||||
conditions.append(
|
||||
f"LOWER(search_index.metadata->>'note_type') IN ({', '.join(type_placeholders)})"
|
||||
)
|
||||
|
||||
# Handle date filter
|
||||
if after_date:
|
||||
@@ -879,6 +902,7 @@ class PostgresSearchRepository(SearchRepositoryBase):
|
||||
note_types: Optional[List[str]] = None,
|
||||
after_date: Optional[datetime] = None,
|
||||
search_item_types: Optional[List[SearchItemType]] = None,
|
||||
categories: Optional[List[str]] = None,
|
||||
metadata_filters: Optional[dict] = None,
|
||||
retrieval_mode: SearchRetrievalMode = SearchRetrievalMode.FTS,
|
||||
min_similarity: Optional[float] = None,
|
||||
@@ -895,6 +919,7 @@ class PostgresSearchRepository(SearchRepositoryBase):
|
||||
note_types=note_types,
|
||||
after_date=after_date,
|
||||
search_item_types=search_item_types,
|
||||
categories=categories,
|
||||
metadata_filters=metadata_filters,
|
||||
retrieval_mode=retrieval_mode,
|
||||
min_similarity=min_similarity,
|
||||
@@ -919,6 +944,7 @@ class PostgresSearchRepository(SearchRepositoryBase):
|
||||
note_types=note_types,
|
||||
after_date=after_date,
|
||||
search_item_types=search_item_types,
|
||||
categories=categories,
|
||||
metadata_filters=metadata_filters,
|
||||
)
|
||||
|
||||
@@ -1008,6 +1034,7 @@ class PostgresSearchRepository(SearchRepositoryBase):
|
||||
note_types: Optional[List[str]] = None,
|
||||
after_date: Optional[datetime] = None,
|
||||
search_item_types: Optional[List[SearchItemType]] = None,
|
||||
categories: Optional[List[str]] = None,
|
||||
metadata_filters: Optional[dict] = None,
|
||||
retrieval_mode: SearchRetrievalMode = SearchRetrievalMode.FTS,
|
||||
min_similarity: Optional[float] = None,
|
||||
@@ -1022,6 +1049,7 @@ class PostgresSearchRepository(SearchRepositoryBase):
|
||||
note_types=note_types,
|
||||
after_date=after_date,
|
||||
search_item_types=search_item_types,
|
||||
categories=categories,
|
||||
metadata_filters=metadata_filters,
|
||||
retrieval_mode=retrieval_mode,
|
||||
min_similarity=min_similarity,
|
||||
@@ -1041,6 +1069,7 @@ class PostgresSearchRepository(SearchRepositoryBase):
|
||||
note_types=note_types,
|
||||
after_date=after_date,
|
||||
search_item_types=search_item_types,
|
||||
categories=categories,
|
||||
metadata_filters=metadata_filters,
|
||||
)
|
||||
sql = f"SELECT COUNT(*) FROM {from_clause} WHERE {where_clause}"
|
||||
|
||||
@@ -5,7 +5,7 @@ from typing import Optional, Sequence, Union
|
||||
|
||||
|
||||
from loguru import logger
|
||||
from sqlalchemy import inspect as sa_inspect, select, text
|
||||
from sqlalchemy import Executable, inspect as sa_inspect, select, text
|
||||
from sqlalchemy.exc import NoResultFound, OperationalError
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
|
||||
@@ -258,6 +258,28 @@ class ProjectRepository(Repository[Project]):
|
||||
logger.debug(f"Deleted Project and search rows for project_id: {entity_id}")
|
||||
return True
|
||||
|
||||
async def scalar_vec_query(
|
||||
self, query: Executable, params: Optional[dict] = None
|
||||
) -> Optional[int]:
|
||||
"""Run a scalar COUNT query that reads the sqlite-vec vec0 table.
|
||||
|
||||
Extension loading is per-connection, so the bare pooled session used by
|
||||
`execute_query` cannot read vec0 virtual tables — SQLite raises
|
||||
"no such module: vec0". This helper loads sqlite-vec on the session it
|
||||
opens before running the query, reusing the same loader as project delete.
|
||||
|
||||
Returns None when sqlite-vec cannot be loaded on this Python build, so
|
||||
callers can fall back to the genuinely-missing-dependency path.
|
||||
"""
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
# Trigger: query reads the vec0-backed search_vector_embeddings table.
|
||||
# Why: vec0 modules are only visible on a connection that loaded sqlite-vec.
|
||||
# Outcome: load the extension here, or signal absence so the caller degrades.
|
||||
if not await _load_sqlite_vec_on_session(session):
|
||||
return None
|
||||
result = await session.execute(query, params)
|
||||
return result.scalar()
|
||||
|
||||
async def update_path(self, project_id: int, new_path: str) -> Optional[Project]:
|
||||
"""Update project path.
|
||||
|
||||
|
||||
@@ -13,6 +13,7 @@ from sqlalchemy import Result
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
|
||||
from basic_memory.config import BasicMemoryConfig, ConfigManager, DatabaseBackend
|
||||
from basic_memory.repository.embedding_provider_factory import create_embedding_provider
|
||||
from basic_memory.repository.postgres_search_repository import PostgresSearchRepository
|
||||
from basic_memory.repository.search_index_row import SearchIndexRow
|
||||
from basic_memory.repository.search_repository_base import VectorSyncBatchResult
|
||||
@@ -41,6 +42,7 @@ class SearchRepository(Protocol):
|
||||
note_types: Optional[List[str]] = None,
|
||||
after_date: Optional[datetime] = None,
|
||||
search_item_types: Optional[List[SearchItemType]] = None,
|
||||
categories: Optional[List[str]] = None,
|
||||
metadata_filters: Optional[dict] = None,
|
||||
retrieval_mode: SearchRetrievalMode = SearchRetrievalMode.FTS,
|
||||
min_similarity: Optional[float] = None,
|
||||
@@ -59,6 +61,7 @@ class SearchRepository(Protocol):
|
||||
note_types: Optional[List[str]] = None,
|
||||
after_date: Optional[datetime] = None,
|
||||
search_item_types: Optional[List[SearchItemType]] = None,
|
||||
categories: Optional[List[str]] = None,
|
||||
metadata_filters: Optional[dict] = None,
|
||||
retrieval_mode: SearchRetrievalMode = SearchRetrievalMode.FTS,
|
||||
min_similarity: Optional[float] = None,
|
||||
@@ -120,19 +123,37 @@ def create_search_repository(
|
||||
Returns:
|
||||
SearchRepository: Backend-appropriate search repository instance
|
||||
"""
|
||||
# Prefer explicit parameter; fall back to ConfigManager for backwards compatibility
|
||||
# Resolve config once so backend detection and the shared embedding provider
|
||||
# come from the same source. Prefer the explicit arg; fall back to ConfigManager
|
||||
# for backwards compatibility.
|
||||
config = app_config or ConfigManager().config
|
||||
if database_backend is None:
|
||||
config = app_config or ConfigManager().config
|
||||
database_backend = config.database_backend
|
||||
|
||||
# Trigger: every request, sync batch, and project builds its own search repo.
|
||||
# Why: each repo __init__ would otherwise call create_embedding_provider(), and
|
||||
# the process-wide cache can be bypassed if its key ever drifts (#872), reloading
|
||||
# the ~2.3GB ONNX model and leaking memory in onnxruntime's CPU arena.
|
||||
# Outcome: resolve the cached singleton here once and inject it, so the provider
|
||||
# is the single source of truth across all callers of this factory.
|
||||
embedding_provider = None
|
||||
if config.semantic_search_enabled:
|
||||
embedding_provider = create_embedding_provider(config)
|
||||
|
||||
if database_backend == DatabaseBackend.POSTGRES: # pragma: no cover
|
||||
return PostgresSearchRepository( # pragma: no cover
|
||||
session_maker,
|
||||
project_id=project_id,
|
||||
app_config=app_config,
|
||||
embedding_provider=embedding_provider,
|
||||
)
|
||||
else:
|
||||
return SQLiteSearchRepository(session_maker, project_id=project_id, app_config=app_config)
|
||||
return SQLiteSearchRepository(
|
||||
session_maker,
|
||||
project_id=project_id,
|
||||
app_config=app_config,
|
||||
embedding_provider=embedding_provider,
|
||||
)
|
||||
|
||||
|
||||
__all__ = [
|
||||
|
||||
@@ -218,6 +218,7 @@ class SearchRepositoryBase(ABC):
|
||||
note_types: Optional[List[str]] = None,
|
||||
after_date: Optional[datetime] = None,
|
||||
search_item_types: Optional[List[SearchItemType]] = None,
|
||||
categories: Optional[List[str]] = None,
|
||||
metadata_filters: Optional[Dict[str, Any]] = None,
|
||||
retrieval_mode: SearchRetrievalMode = SearchRetrievalMode.FTS,
|
||||
min_similarity: Optional[float] = None,
|
||||
@@ -234,6 +235,7 @@ class SearchRepositoryBase(ABC):
|
||||
note_types: Filter by note types (from metadata.note_type)
|
||||
after_date: Filter by created_at > after_date
|
||||
search_item_types: Filter by SearchItemType (ENTITY, OBSERVATION, RELATION)
|
||||
categories: Filter observations by exact category (e.g. "requirement")
|
||||
metadata_filters: Structured frontmatter metadata filters
|
||||
limit: Maximum results to return
|
||||
offset: Number of results to skip
|
||||
@@ -256,6 +258,7 @@ class SearchRepositoryBase(ABC):
|
||||
note_types: Optional[List[str]] = None,
|
||||
after_date: Optional[datetime] = None,
|
||||
search_item_types: Optional[List[SearchItemType]] = None,
|
||||
categories: Optional[List[str]] = None,
|
||||
metadata_filters: Optional[Dict[str, Any]] = None,
|
||||
retrieval_mode: SearchRetrievalMode = SearchRetrievalMode.FTS,
|
||||
min_similarity: Optional[float] = None,
|
||||
@@ -586,11 +589,19 @@ class SearchRepositoryBase(ABC):
|
||||
def _embedding_model_key(self) -> str:
|
||||
"""Build a stable model identity for vector invalidation checks."""
|
||||
assert self._embedding_provider is not None
|
||||
return (
|
||||
f"{type(self._embedding_provider).__name__}:"
|
||||
f"{self._embedding_provider.model_name}:"
|
||||
f"{self._embedding_provider.dimensions}"
|
||||
)
|
||||
provider = self._embedding_provider
|
||||
|
||||
provider_identity = f"{provider.model_name}:{provider.dimensions}"
|
||||
from basic_memory.repository.litellm_provider import LiteLLMEmbeddingProvider
|
||||
|
||||
if isinstance(provider, LiteLLMEmbeddingProvider):
|
||||
# Trigger: LiteLLM can change request semantics without changing model/dimensions.
|
||||
# Why: asymmetric providers use role-specific document/query params, and
|
||||
# dimension forwarding changes provider-side output-size behavior.
|
||||
# Outcome: reindex treats those semantic config changes as stale vectors.
|
||||
provider_identity = provider.identity_key()
|
||||
|
||||
return f"{type(provider).__name__}:{provider_identity}"
|
||||
|
||||
def _plan_entity_vector_shard(
|
||||
self,
|
||||
@@ -1777,6 +1788,7 @@ class SearchRepositoryBase(ABC):
|
||||
note_types: Optional[List[str]],
|
||||
after_date: Optional[datetime],
|
||||
search_item_types: Optional[List[SearchItemType]],
|
||||
categories: Optional[List[str]],
|
||||
metadata_filters: Optional[dict],
|
||||
retrieval_mode: SearchRetrievalMode,
|
||||
min_similarity: Optional[float] = None,
|
||||
@@ -1810,6 +1822,7 @@ class SearchRepositoryBase(ABC):
|
||||
note_types=note_types,
|
||||
after_date=after_date,
|
||||
search_item_types=search_item_types,
|
||||
categories=categories,
|
||||
metadata_filters=metadata_filters,
|
||||
min_similarity=min_similarity,
|
||||
limit=limit,
|
||||
@@ -1829,6 +1842,7 @@ class SearchRepositoryBase(ABC):
|
||||
note_types=note_types,
|
||||
after_date=after_date,
|
||||
search_item_types=search_item_types,
|
||||
categories=categories,
|
||||
metadata_filters=metadata_filters,
|
||||
min_similarity=min_similarity,
|
||||
limit=limit,
|
||||
@@ -1858,6 +1872,7 @@ class SearchRepositoryBase(ABC):
|
||||
note_types: Optional[List[str]],
|
||||
after_date: Optional[datetime],
|
||||
search_item_types: Optional[List[SearchItemType]],
|
||||
categories: Optional[List[str]],
|
||||
metadata_filters: Optional[dict],
|
||||
min_similarity: Optional[float] = None,
|
||||
limit: int,
|
||||
@@ -1968,6 +1983,7 @@ class SearchRepositoryBase(ABC):
|
||||
note_types,
|
||||
after_date,
|
||||
search_item_types,
|
||||
categories,
|
||||
metadata_filters,
|
||||
]
|
||||
)
|
||||
@@ -1981,6 +1997,7 @@ class SearchRepositoryBase(ABC):
|
||||
note_types=note_types,
|
||||
after_date=after_date,
|
||||
search_item_types=search_item_types,
|
||||
categories=categories,
|
||||
metadata_filters=metadata_filters,
|
||||
retrieval_mode=SearchRetrievalMode.FTS,
|
||||
limit=VECTOR_FILTER_SCAN_LIMIT,
|
||||
@@ -2131,6 +2148,7 @@ class SearchRepositoryBase(ABC):
|
||||
note_types: Optional[List[str]],
|
||||
after_date: Optional[datetime],
|
||||
search_item_types: Optional[List[SearchItemType]],
|
||||
categories: Optional[List[str]],
|
||||
metadata_filters: Optional[dict],
|
||||
min_similarity: Optional[float] = None,
|
||||
limit: int,
|
||||
@@ -2155,6 +2173,7 @@ class SearchRepositoryBase(ABC):
|
||||
note_types=note_types,
|
||||
after_date=after_date,
|
||||
search_item_types=search_item_types,
|
||||
categories=categories,
|
||||
metadata_filters=metadata_filters,
|
||||
retrieval_mode=SearchRetrievalMode.FTS,
|
||||
limit=candidate_limit,
|
||||
@@ -2170,6 +2189,7 @@ class SearchRepositoryBase(ABC):
|
||||
note_types=note_types,
|
||||
after_date=after_date,
|
||||
search_item_types=search_item_types,
|
||||
categories=categories,
|
||||
metadata_filters=metadata_filters,
|
||||
min_similarity=min_similarity,
|
||||
limit=candidate_limit,
|
||||
|
||||
@@ -733,6 +733,7 @@ class SQLiteSearchRepository(SearchRepositoryBase):
|
||||
note_types: Optional[List[str]] = None,
|
||||
after_date: Optional[datetime] = None,
|
||||
search_item_types: Optional[List[SearchItemType]] = None,
|
||||
categories: Optional[List[str]] = None,
|
||||
metadata_filters: Optional[dict] = None,
|
||||
) -> tuple[str, str, dict, str]:
|
||||
"""Build SQLite FTS FROM/WHERE params shared by search and count."""
|
||||
@@ -794,15 +795,36 @@ class SQLiteSearchRepository(SearchRepositoryBase):
|
||||
type_placeholders.append(f":{param_name}")
|
||||
conditions.append(f"search_index.type IN ({', '.join(type_placeholders)})")
|
||||
|
||||
# Handle note type filter (frontmatter type field, parameterized)
|
||||
# Handle observation category filter (parameterized for defense-in-depth).
|
||||
# Trigger: caller passed `categories` to scope observation results.
|
||||
# Why: `entity_types=["observation"]` only narrows to the observation row type;
|
||||
# callers expect exact-category matching, not incidental text matches.
|
||||
# Outcome: only rows whose indexed category exactly equals a requested value
|
||||
# survive (entities/relations have NULL category and are excluded).
|
||||
if categories:
|
||||
category_placeholders = []
|
||||
for idx, category in enumerate(categories):
|
||||
param_name = f"category_{idx}"
|
||||
params[param_name] = category
|
||||
category_placeholders.append(f":{param_name}")
|
||||
conditions.append(f"search_index.category IN ({', '.join(category_placeholders)})")
|
||||
|
||||
# Handle note type filter (frontmatter type field, parameterized).
|
||||
# Trigger: caller passed `note_types` to scope by the frontmatter `type` field.
|
||||
# Why: the stored note_type preserves the frontmatter casing (e.g. `Chapter`),
|
||||
# but the filter is documented case-insensitive; comparing raw values
|
||||
# would miss capitalized types.
|
||||
# Outcome: fold both sides to lowercase so `note_types=["Chapter"]` matches a
|
||||
# stored `Chapter`, `chapter`, etc.
|
||||
if note_types:
|
||||
type_placeholders = []
|
||||
for idx, t in enumerate(note_types):
|
||||
param_name = f"note_type_{idx}"
|
||||
params[param_name] = t
|
||||
params[param_name] = t.lower()
|
||||
type_placeholders.append(f":{param_name}")
|
||||
conditions.append(
|
||||
f"json_extract(search_index.metadata, '$.note_type') IN ({', '.join(type_placeholders)})"
|
||||
"LOWER(json_extract(search_index.metadata, '$.note_type')) "
|
||||
f"IN ({', '.join(type_placeholders)})"
|
||||
)
|
||||
|
||||
# Handle date filter using datetime() for proper comparison
|
||||
@@ -925,6 +947,7 @@ class SQLiteSearchRepository(SearchRepositoryBase):
|
||||
note_types: Optional[List[str]] = None,
|
||||
after_date: Optional[datetime] = None,
|
||||
search_item_types: Optional[List[SearchItemType]] = None,
|
||||
categories: Optional[List[str]] = None,
|
||||
metadata_filters: Optional[dict] = None,
|
||||
retrieval_mode: SearchRetrievalMode = SearchRetrievalMode.FTS,
|
||||
min_similarity: Optional[float] = None,
|
||||
@@ -941,6 +964,7 @@ class SQLiteSearchRepository(SearchRepositoryBase):
|
||||
note_types=note_types,
|
||||
after_date=after_date,
|
||||
search_item_types=search_item_types,
|
||||
categories=categories,
|
||||
metadata_filters=metadata_filters,
|
||||
retrieval_mode=retrieval_mode,
|
||||
min_similarity=min_similarity,
|
||||
@@ -959,6 +983,7 @@ class SQLiteSearchRepository(SearchRepositoryBase):
|
||||
note_types=note_types,
|
||||
after_date=after_date,
|
||||
search_item_types=search_item_types,
|
||||
categories=categories,
|
||||
metadata_filters=metadata_filters,
|
||||
)
|
||||
|
||||
@@ -1046,6 +1071,7 @@ class SQLiteSearchRepository(SearchRepositoryBase):
|
||||
note_types: Optional[List[str]] = None,
|
||||
after_date: Optional[datetime] = None,
|
||||
search_item_types: Optional[List[SearchItemType]] = None,
|
||||
categories: Optional[List[str]] = None,
|
||||
metadata_filters: Optional[dict] = None,
|
||||
retrieval_mode: SearchRetrievalMode = SearchRetrievalMode.FTS,
|
||||
min_similarity: Optional[float] = None,
|
||||
@@ -1060,6 +1086,7 @@ class SQLiteSearchRepository(SearchRepositoryBase):
|
||||
note_types=note_types,
|
||||
after_date=after_date,
|
||||
search_item_types=search_item_types,
|
||||
categories=categories,
|
||||
metadata_filters=metadata_filters,
|
||||
retrieval_mode=retrieval_mode,
|
||||
min_similarity=min_similarity,
|
||||
@@ -1073,6 +1100,7 @@ class SQLiteSearchRepository(SearchRepositoryBase):
|
||||
note_types=note_types,
|
||||
after_date=after_date,
|
||||
search_item_types=search_item_types,
|
||||
categories=categories,
|
||||
metadata_filters=metadata_filters,
|
||||
)
|
||||
sql = f"SELECT COUNT(*) FROM {from_clause} WHERE {where_clause}"
|
||||
|
||||
@@ -42,6 +42,7 @@ class SearchQuery(BaseModel):
|
||||
Optionally filter results by:
|
||||
- note_types: Limit to specific note types (frontmatter "type")
|
||||
- entity_types: Limit to search item types (entity/observation/relation)
|
||||
- categories: Limit observation results to exact category matches (e.g. "requirement")
|
||||
- after_date: Only items after date
|
||||
- metadata_filters: Structured frontmatter filters (field -> value)
|
||||
- tags: Convenience frontmatter tag filter
|
||||
@@ -63,6 +64,7 @@ class SearchQuery(BaseModel):
|
||||
# Optional filters
|
||||
note_types: Optional[List[str]] = None # Filter by note type (frontmatter "type")
|
||||
entity_types: Optional[List[SearchItemType]] = None # Filter by entity type
|
||||
categories: Optional[List[str]] = None # Filter observations by exact category
|
||||
after_date: Optional[Union[datetime, str]] = None # Time-based filter
|
||||
metadata_filters: Optional[dict[str, Any]] = None # Structured frontmatter filters
|
||||
tags: Optional[List[str]] = None # Convenience tag filter
|
||||
@@ -85,6 +87,7 @@ class SearchQuery(BaseModel):
|
||||
status_is_empty = self.status is None or (isinstance(self.status, str) and not self.status)
|
||||
note_types_is_empty = not self.note_types
|
||||
entity_types_is_empty = not self.entity_types
|
||||
categories_is_empty = not self.categories
|
||||
return (
|
||||
self.permalink is None
|
||||
and self.permalink_match is None
|
||||
@@ -93,6 +96,7 @@ class SearchQuery(BaseModel):
|
||||
and self.after_date is None
|
||||
and note_types_is_empty
|
||||
and entity_types_is_empty
|
||||
and categories_is_empty
|
||||
and metadata_is_empty
|
||||
and tags_is_empty
|
||||
and status_is_empty
|
||||
|
||||
@@ -333,9 +333,14 @@ class ContextService:
|
||||
relation_date_filter = ""
|
||||
timeframe_condition = ""
|
||||
|
||||
# Add project filtering for security - ensure all entities and relations belong to the same project
|
||||
project_filter = "AND e.project_id = :project_id"
|
||||
relation_project_filter = "AND e_from.project_id = :project_id"
|
||||
# Trigger: build_context starts from a project-scoped search result.
|
||||
# Why: the seed entity must belong to the requested project, but an
|
||||
# explicit relation edge may point at another project.
|
||||
# Outcome: traversal follows only project-owned edges from reached
|
||||
# entities, instead of forcing every reached entity into the seed project.
|
||||
seed_project_filter = "AND e.project_id = :project_id"
|
||||
connected_entity_project_filter = ""
|
||||
relation_project_filter = "AND e_from.project_id = r.project_id"
|
||||
|
||||
# Use a CTE that operates directly on entity and relation tables
|
||||
# This avoids the overhead of the search_index virtual table
|
||||
@@ -351,7 +356,8 @@ class ContextService:
|
||||
query = self._build_postgres_query(
|
||||
entity_id_values,
|
||||
date_filter,
|
||||
project_filter,
|
||||
seed_project_filter,
|
||||
connected_entity_project_filter,
|
||||
relation_date_filter,
|
||||
relation_project_filter,
|
||||
timeframe_condition,
|
||||
@@ -362,7 +368,8 @@ class ContextService:
|
||||
query = self._build_sqlite_query(
|
||||
entity_id_values,
|
||||
date_filter,
|
||||
project_filter,
|
||||
seed_project_filter,
|
||||
connected_entity_project_filter,
|
||||
relation_date_filter,
|
||||
relation_project_filter,
|
||||
timeframe_condition,
|
||||
@@ -397,7 +404,8 @@ class ContextService:
|
||||
self,
|
||||
entity_id_values: str,
|
||||
date_filter: str,
|
||||
project_filter: str,
|
||||
seed_project_filter: str,
|
||||
connected_entity_project_filter: str,
|
||||
relation_date_filter: str,
|
||||
relation_project_filter: str,
|
||||
timeframe_condition: str,
|
||||
@@ -421,11 +429,13 @@ class ContextService:
|
||||
0 as depth,
|
||||
e.id as root_id,
|
||||
e.created_at,
|
||||
e.created_at as relation_date
|
||||
e.created_at as relation_date,
|
||||
e.project_id as project_id,
|
||||
',' || e.id::text || ',' as entity_path
|
||||
FROM entity e
|
||||
WHERE e.id IN ({entity_id_values})
|
||||
{date_filter}
|
||||
{project_filter}
|
||||
{seed_project_filter}
|
||||
|
||||
UNION ALL
|
||||
|
||||
@@ -477,15 +487,25 @@ class ContextService:
|
||||
CASE
|
||||
WHEN step_type = 1 THEN e_from.created_at
|
||||
ELSE eg.relation_date
|
||||
END as relation_date
|
||||
END as relation_date,
|
||||
CASE
|
||||
WHEN step_type = 1 THEN eg.project_id
|
||||
ELSE e.project_id
|
||||
END as project_id,
|
||||
CASE
|
||||
WHEN step_type = 1 THEN eg.entity_path
|
||||
ELSE eg.entity_path || e.id::text || ','
|
||||
END as entity_path
|
||||
FROM entity_graph eg
|
||||
CROSS JOIN LATERAL (VALUES (1), (2)) AS steps(step_type)
|
||||
JOIN relation r ON (
|
||||
eg.type = 'entity' AND
|
||||
(r.from_id = eg.id OR r.to_id = eg.id)
|
||||
(r.from_id = eg.id OR r.to_id = eg.id) AND
|
||||
r.project_id = eg.project_id
|
||||
)
|
||||
JOIN entity e_from ON (
|
||||
r.from_id = e_from.id
|
||||
{relation_date_filter}
|
||||
{relation_project_filter}
|
||||
)
|
||||
LEFT JOIN entity e ON (
|
||||
@@ -495,10 +515,17 @@ class ContextService:
|
||||
ELSE r.from_id
|
||||
END
|
||||
{date_filter}
|
||||
{project_filter}
|
||||
{connected_entity_project_filter}
|
||||
)
|
||||
WHERE eg.depth < :max_depth
|
||||
AND (step_type = 1 OR (step_type = 2 AND e.id IS NOT NULL AND e.id != eg.id))
|
||||
AND (
|
||||
step_type = 1 OR (
|
||||
step_type = 2
|
||||
AND e.id IS NOT NULL
|
||||
AND e.id != eg.id
|
||||
AND position(',' || e.id::text || ',' in eg.entity_path) = 0
|
||||
)
|
||||
)
|
||||
{timeframe_condition}
|
||||
)
|
||||
-- Materialize and filter
|
||||
@@ -529,7 +556,8 @@ class ContextService:
|
||||
self,
|
||||
entity_id_values: str,
|
||||
date_filter: str,
|
||||
project_filter: str,
|
||||
seed_project_filter: str,
|
||||
connected_entity_project_filter: str,
|
||||
relation_date_filter: str,
|
||||
relation_project_filter: str,
|
||||
timeframe_condition: str,
|
||||
@@ -555,11 +583,13 @@ class ContextService:
|
||||
e.id as root_id,
|
||||
e.created_at,
|
||||
e.created_at as relation_date,
|
||||
0 as is_incoming
|
||||
0 as is_incoming,
|
||||
e.project_id as project_id,
|
||||
',' || e.id || ',' as entity_path
|
||||
FROM entity e
|
||||
WHERE e.id IN ({entity_id_values})
|
||||
{date_filter}
|
||||
{project_filter}
|
||||
{seed_project_filter}
|
||||
|
||||
UNION ALL
|
||||
|
||||
@@ -580,11 +610,14 @@ class ContextService:
|
||||
eg.root_id,
|
||||
e_from.created_at,
|
||||
e_from.created_at as relation_date,
|
||||
CASE WHEN r.from_id = eg.id THEN 0 ELSE 1 END as is_incoming
|
||||
CASE WHEN r.from_id = eg.id THEN 0 ELSE 1 END as is_incoming,
|
||||
eg.project_id as project_id,
|
||||
eg.entity_path as entity_path
|
||||
FROM entity_graph eg
|
||||
JOIN relation r ON (
|
||||
eg.type = 'entity' AND
|
||||
(r.from_id = eg.id OR r.to_id = eg.id)
|
||||
(r.from_id = eg.id OR r.to_id = eg.id) AND
|
||||
r.project_id = eg.project_id
|
||||
)
|
||||
JOIN entity e_from ON (
|
||||
r.from_id = e_from.id
|
||||
@@ -615,7 +648,9 @@ class ContextService:
|
||||
eg.root_id,
|
||||
e.created_at,
|
||||
eg.relation_date,
|
||||
eg.is_incoming
|
||||
eg.is_incoming,
|
||||
e.project_id as project_id,
|
||||
eg.entity_path || e.id || ',' as entity_path
|
||||
FROM entity_graph eg
|
||||
JOIN entity e ON (
|
||||
eg.type = 'relation' AND
|
||||
@@ -624,9 +659,10 @@ class ContextService:
|
||||
ELSE eg.from_id
|
||||
END
|
||||
{date_filter}
|
||||
{project_filter}
|
||||
{connected_entity_project_filter}
|
||||
)
|
||||
WHERE eg.depth < :max_depth
|
||||
AND instr(eg.entity_path, ',' || e.id || ',') = 0
|
||||
{timeframe_condition}
|
||||
)
|
||||
SELECT DISTINCT
|
||||
|
||||
@@ -1047,10 +1047,27 @@ class ProjectService:
|
||||
f"WHERE c.project_id = :project_id {chunk_entity_exists}"
|
||||
)
|
||||
|
||||
embeddings_result = await self.repository.execute_query(
|
||||
embeddings_sql, {"project_id": project_id}
|
||||
)
|
||||
total_embeddings = embeddings_result.scalar() or 0
|
||||
# The embeddings/orphan JOINs read search_vector_embeddings, a vec0
|
||||
# virtual table. On SQLite that table is only visible on a connection
|
||||
# that loaded sqlite-vec, so route these through scalar_vec_query which
|
||||
# loads the extension first. Postgres has no per-connection extension
|
||||
# and uses the bare pooled session.
|
||||
async def _vec_scalar(vec_sql) -> int:
|
||||
if is_postgres:
|
||||
result = await self.repository.execute_query(
|
||||
vec_sql, {"project_id": project_id}
|
||||
)
|
||||
return result.scalar() or 0
|
||||
count = await self.repository.scalar_vec_query(vec_sql, {"project_id": project_id})
|
||||
# Trigger: sqlite-vec genuinely can't load on this Python build.
|
||||
# Why: without the extension the vec0 JOIN can't run at all.
|
||||
# Outcome: raise the canonical error so the except block emits the
|
||||
# true "sqlite-vec unavailable" fallback instead of reporting 0.
|
||||
if count is None:
|
||||
raise SAOperationalError(str(vec_sql), {}, Exception("no such module: vec0"))
|
||||
return count
|
||||
|
||||
total_embeddings = await _vec_scalar(embeddings_sql)
|
||||
|
||||
# Orphaned chunks (chunks without embeddings — indicates interrupted indexing)
|
||||
if is_postgres:
|
||||
@@ -1065,11 +1082,7 @@ class ProjectService:
|
||||
"LEFT JOIN search_vector_embeddings e ON e.rowid = c.id "
|
||||
f"WHERE c.project_id = :project_id AND e.rowid IS NULL {chunk_entity_exists}"
|
||||
)
|
||||
|
||||
orphan_result = await self.repository.execute_query(
|
||||
orphan_sql, {"project_id": project_id}
|
||||
)
|
||||
orphaned_chunks = orphan_result.scalar() or 0
|
||||
orphaned_chunks = await _vec_scalar(orphan_sql)
|
||||
except SAOperationalError as exc:
|
||||
# Trigger: sqlite_master can list vec0 virtual tables even when sqlite-vec
|
||||
# is not loaded in the current Python runtime.
|
||||
|
||||
@@ -75,6 +75,7 @@ class _PreparedSearchQuery:
|
||||
title: str | None
|
||||
note_types: list[str] | None
|
||||
search_item_types: list[SearchItemType] | None
|
||||
categories: list[str] | None
|
||||
after_date: datetime | None
|
||||
metadata_filters: dict[str, Any] | None
|
||||
retrieval_mode: SearchRetrievalMode
|
||||
@@ -194,6 +195,7 @@ class SearchService:
|
||||
title=query.title,
|
||||
note_types=query.note_types,
|
||||
search_item_types=query.entity_types,
|
||||
categories=query.categories,
|
||||
after_date=after_date,
|
||||
metadata_filters=metadata_filters,
|
||||
retrieval_mode=query.retrieval_mode or SearchRetrievalMode.FTS,
|
||||
@@ -207,6 +209,7 @@ class SearchService:
|
||||
or prepared.title
|
||||
or prepared.note_types
|
||||
or prepared.search_item_types
|
||||
or prepared.categories
|
||||
or prepared.after_date
|
||||
or prepared.metadata_filters
|
||||
)
|
||||
@@ -221,6 +224,7 @@ class SearchService:
|
||||
prepared.metadata_filters
|
||||
or prepared.note_types
|
||||
or prepared.search_item_types
|
||||
or prepared.categories
|
||||
or prepared.after_date
|
||||
)
|
||||
|
||||
@@ -239,6 +243,7 @@ class SearchService:
|
||||
title=prepared.title,
|
||||
note_types=prepared.note_types,
|
||||
search_item_types=prepared.search_item_types,
|
||||
categories=prepared.categories,
|
||||
after_date=prepared.after_date,
|
||||
metadata_filters=prepared.metadata_filters,
|
||||
retrieval_mode=prepared.retrieval_mode,
|
||||
@@ -260,6 +265,7 @@ class SearchService:
|
||||
title=prepared.title,
|
||||
note_types=prepared.note_types,
|
||||
search_item_types=prepared.search_item_types,
|
||||
categories=prepared.categories,
|
||||
after_date=prepared.after_date,
|
||||
metadata_filters=prepared.metadata_filters,
|
||||
retrieval_mode=prepared.retrieval_mode,
|
||||
|
||||
@@ -1447,7 +1447,16 @@ class SyncService:
|
||||
f"to_name={relation.to_name}"
|
||||
)
|
||||
|
||||
resolved_entity = await self.entity_service.link_resolver.resolve_link(relation.to_name)
|
||||
# Use strict=True: deferred resolution should only fill in to_id when an
|
||||
# exact permalink/title/file_path match exists. The fuzzy fallback (search-based
|
||||
# token match) would silently resolve ambiguous links like
|
||||
# `[[overview (state-management/session-execution)]]` to whichever entity shares
|
||||
# the most tokens, polluting the graph with confidently-wrong edges that no
|
||||
# audit catches. Leaving such relations unresolved keeps to_id=NULL so they
|
||||
# surface as forward references and can be fixed by the producer.
|
||||
resolved_entity = await self.entity_service.link_resolver.resolve_link(
|
||||
relation.to_name, strict=True
|
||||
)
|
||||
|
||||
# ignore reference to self
|
||||
if resolved_entity and resolved_entity.id != relation.from_id:
|
||||
@@ -1735,7 +1744,9 @@ async def get_sync_service(project: Project) -> SyncService: # pragma: no cover
|
||||
entity_repository = EntityRepository(session_maker, project_id=project.id)
|
||||
observation_repository = ObservationRepository(session_maker, project_id=project.id)
|
||||
relation_repository = RelationRepository(session_maker, project_id=project.id)
|
||||
search_repository = create_search_repository(session_maker, project_id=project.id)
|
||||
search_repository = create_search_repository(
|
||||
session_maker, project_id=project.id, app_config=app_config
|
||||
)
|
||||
project_repository = ProjectRepository(session_maker)
|
||||
|
||||
# Initialize services
|
||||
|
||||
@@ -526,8 +526,20 @@ def parse_tags(tags: Union[List[str], str, None]) -> List[str]:
|
||||
|
||||
# Process list of tags
|
||||
if isinstance(tags, list):
|
||||
# First strip whitespace, then strip leading '#' characters to prevent accumulation
|
||||
return [tag.strip().lstrip("#") for tag in tags if tag and tag.strip()]
|
||||
# Trigger: a list element may itself be a comma-separated string (e.g. typer collects
|
||||
# `--tags "a,b"` into the one-element list `["a,b"]`).
|
||||
# Why: keep the CLI list path and the MCP bare-string path on a single source of truth so
|
||||
# `--tags "a,b"`, `--tags a --tags b`, and `tags="a,b"` all converge to the same tags.
|
||||
# Outcome: flatten by splitting each element on commas before stripping '#' / whitespace.
|
||||
# Skip None entries (e.g. a YAML `tags: [alpha, null]`) so they are not revived as
|
||||
# the literal tag "None" by str(raw); the old list branch ignored such falsy entries.
|
||||
return [
|
||||
tag.strip().lstrip("#")
|
||||
for raw in tags
|
||||
if raw is not None
|
||||
for tag in str(raw).split(",")
|
||||
if tag and tag.strip()
|
||||
]
|
||||
|
||||
# Process string input
|
||||
if isinstance(tags, str):
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
"""Bug hunt regression test (#6): `bm tool read-note` exit code on a
|
||||
path-traversal SECURITY_VALIDATION_ERROR.
|
||||
|
||||
The MCP read_note tool detects path-traversal identifiers and returns
|
||||
{"error": "SECURITY_VALIDATION_ERROR", ...}. Every other wrapped tool command
|
||||
exits non-zero on an error payload; read-note used to print the payload and
|
||||
exit 0. These integration tests assert read-note now matches its siblings.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from basic_memory.cli.main import app as cli_app
|
||||
from basic_memory.mcp.tools import read_note as mcp_read_note
|
||||
|
||||
runner = CliRunner()
|
||||
|
||||
TRAVERSAL_IDENTIFIER = "../../../../etc/passwd"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_read_note_security_error_mcp_emits_error_field(
|
||||
app, app_config, test_project, config_manager
|
||||
):
|
||||
"""MCP read_note JSON flags the path traversal with a SECURITY_VALIDATION_ERROR."""
|
||||
result = await mcp_read_note(
|
||||
identifier=TRAVERSAL_IDENTIFIER,
|
||||
project=test_project.name,
|
||||
output_format="json",
|
||||
)
|
||||
assert isinstance(result, dict)
|
||||
assert result.get("error") == "SECURITY_VALIDATION_ERROR"
|
||||
|
||||
|
||||
def test_read_note_security_error_cli_exit_code_matches_other_tools(
|
||||
app, app_config, test_project, config_manager
|
||||
):
|
||||
"""CLI read-note must not exit 0 when the MCP payload carries an error."""
|
||||
result = runner.invoke(
|
||||
cli_app,
|
||||
["tool", "read-note", TRAVERSAL_IDENTIFIER, "--project", test_project.name],
|
||||
)
|
||||
|
||||
combined = result.stdout
|
||||
assert "SECURITY_VALIDATION_ERROR" in combined, combined
|
||||
|
||||
assert result.exit_code != 0, (
|
||||
f"read-note exited {result.exit_code} on a SECURITY_VALIDATION_ERROR; "
|
||||
"other tool commands exit non-zero on error payloads"
|
||||
)
|
||||
@@ -0,0 +1,76 @@
|
||||
"""Bug hunt regression test (#3): `bm tool recent-activity` page_size default.
|
||||
|
||||
The MCP recent_activity tool defaults page_size=10; the CLI wrapper used to
|
||||
default to 50. Because page_size becomes the SQL LIMIT for the query, identical
|
||||
default invocations returned a different number of rows from CLI vs MCP. This
|
||||
integration test proves the CLI default now matches the MCP default of 10.
|
||||
"""
|
||||
|
||||
import json
|
||||
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from basic_memory.cli.main import app as cli_app
|
||||
|
||||
runner = CliRunner()
|
||||
|
||||
MCP_DEFAULT_PAGE_SIZE = 10
|
||||
|
||||
|
||||
def _write_note(title: str, folder: str, content: str) -> None:
|
||||
result = runner.invoke(
|
||||
cli_app,
|
||||
[
|
||||
"tool",
|
||||
"write-note",
|
||||
"--title",
|
||||
title,
|
||||
"--folder",
|
||||
folder,
|
||||
"--content",
|
||||
content,
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
|
||||
|
||||
def test_recent_activity_default_page_size_matches_mcp(
|
||||
app, app_config, test_project, config_manager, monkeypatch
|
||||
):
|
||||
"""CLI recent-activity default page_size must match the MCP tool default (10)."""
|
||||
monkeypatch.setenv("BASIC_MEMORY_MCP_PROJECT", test_project.name)
|
||||
|
||||
for i in range(15):
|
||||
_write_note(
|
||||
f"Parity Note {i:02d}",
|
||||
"parity-recent",
|
||||
f"# Parity Note {i:02d}\n\nUnique body token PARITY{i:02d}.",
|
||||
)
|
||||
|
||||
mcp_default_result = runner.invoke(
|
||||
cli_app,
|
||||
[
|
||||
"tool",
|
||||
"recent-activity",
|
||||
"--project",
|
||||
test_project.name,
|
||||
"--page-size",
|
||||
str(MCP_DEFAULT_PAGE_SIZE),
|
||||
],
|
||||
)
|
||||
assert mcp_default_result.exit_code == 0, mcp_default_result.output
|
||||
mcp_default_rows = json.loads(mcp_default_result.stdout)
|
||||
assert len(mcp_default_rows) == MCP_DEFAULT_PAGE_SIZE
|
||||
|
||||
cli_default_result = runner.invoke(
|
||||
cli_app,
|
||||
["tool", "recent-activity", "--project", test_project.name],
|
||||
)
|
||||
assert cli_default_result.exit_code == 0, cli_default_result.output
|
||||
cli_default_rows = json.loads(cli_default_result.stdout)
|
||||
|
||||
assert len(cli_default_rows) == MCP_DEFAULT_PAGE_SIZE, (
|
||||
f"CLI recent-activity default returned {len(cli_default_rows)} rows but "
|
||||
f"the MCP tool default (page_size={MCP_DEFAULT_PAGE_SIZE}) returns "
|
||||
f"{len(mcp_default_rows)}; the CLI and MCP default page_size must match."
|
||||
)
|
||||
@@ -0,0 +1,73 @@
|
||||
"""Bug hunt regression test (#4): `bm tool search-notes` --category filter.
|
||||
|
||||
The MCP search_notes tool exposes a `categories` parameter for exact-match
|
||||
observation-category filtering. The CLI wrapper had no equivalent flag. This
|
||||
integration test asserts the CLI now exposes `--category` and that it filters
|
||||
observation results to the requested category exactly.
|
||||
"""
|
||||
|
||||
import json
|
||||
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from basic_memory.cli.main import app as cli_app
|
||||
|
||||
runner = CliRunner()
|
||||
|
||||
|
||||
def _write_note(title: str, folder: str, content: str) -> dict:
|
||||
result = runner.invoke(
|
||||
cli_app,
|
||||
[
|
||||
"tool",
|
||||
"write-note",
|
||||
"--title",
|
||||
title,
|
||||
"--folder",
|
||||
folder,
|
||||
"--content",
|
||||
content,
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
return json.loads(result.stdout)
|
||||
|
||||
|
||||
def test_search_notes_exposes_category_filter(app, app_config, test_project, config_manager):
|
||||
"""CLI search-notes should expose --category like the MCP `categories` param."""
|
||||
_write_note(
|
||||
"Category Filter Note",
|
||||
"parity-category",
|
||||
"# Category Filter Note\n\n"
|
||||
"## Observations\n"
|
||||
"- [requirement] system must authenticate users CATTOKEN\n"
|
||||
"- [decision] use OAuth for auth CATTOKEN\n",
|
||||
)
|
||||
|
||||
result = runner.invoke(
|
||||
cli_app,
|
||||
[
|
||||
"tool",
|
||||
"search-notes",
|
||||
"CATTOKEN",
|
||||
"--project",
|
||||
test_project.name,
|
||||
"--entity-type",
|
||||
"observation",
|
||||
"--category",
|
||||
"requirement",
|
||||
],
|
||||
)
|
||||
|
||||
assert result.exit_code == 0, (
|
||||
"`--category` filter is not supported by the CLI search-notes command "
|
||||
"even though the MCP search_notes tool documents a `categories` param. "
|
||||
f"exit_code={result.exit_code} output={result.output}"
|
||||
)
|
||||
|
||||
payload = json.loads(result.stdout)
|
||||
categories = {r.get("category") for r in payload.get("results", []) if r.get("category")}
|
||||
assert categories == {"requirement"}, (
|
||||
"--category requirement should return only requirement observations, "
|
||||
f"got categories={categories}"
|
||||
)
|
||||
@@ -0,0 +1,193 @@
|
||||
"""Bug hunt regression tests: `bm tool write-note` CLI/MCP parity.
|
||||
|
||||
Covers three confirmed bugs found by the integration-test bug hunt:
|
||||
|
||||
- #1 / #5: write-note exits 0 on a conflict/error JSON result (silent failure,
|
||||
inconsistent with delete-note/edit-note/search-notes which exit non-zero).
|
||||
- #2: write-note had no `--overwrite` flag even though the MCP write_note tool
|
||||
supports overwrite=True to replace an existing note.
|
||||
|
||||
These are integration tests: real CliRunner -> CLI command -> MCP tool ->
|
||||
in-process ASGI API -> real SQLite/Postgres DB and filesystem. No mocks.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from basic_memory.cli.main import app as cli_app
|
||||
from basic_memory.mcp.tools import write_note as mcp_write_note
|
||||
|
||||
runner = CliRunner()
|
||||
|
||||
|
||||
# --- #1: write-note exits non-zero on a conflict/error result ---
|
||||
|
||||
|
||||
def _write_conflict(content_token: str):
|
||||
return runner.invoke(
|
||||
cli_app,
|
||||
[
|
||||
"tool",
|
||||
"write-note",
|
||||
"--title",
|
||||
"Conflict Exit Note",
|
||||
"--folder",
|
||||
"parity-conflict",
|
||||
"--content",
|
||||
f"# Conflict Exit Note\n\n{content_token}",
|
||||
"--project",
|
||||
"test-project",
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
def test_write_note_nonzero_exit_on_conflict_error(app, app_config, test_project, config_manager):
|
||||
"""write-note should exit non-zero when the MCP result carries an error."""
|
||||
first = _write_conflict("FIRST")
|
||||
assert first.exit_code == 0, first.output
|
||||
|
||||
second = _write_conflict("SECOND")
|
||||
payload = json.loads(second.stdout)
|
||||
|
||||
# Confirm the MCP layer reported a conflict error in the JSON.
|
||||
assert payload.get("error") == "NOTE_ALREADY_EXISTS", payload
|
||||
assert payload.get("action") == "conflict", payload
|
||||
|
||||
# Parity with delete-note / edit-note / search-notes: an error result
|
||||
# must drive a non-zero exit code so scripts can detect failure.
|
||||
assert second.exit_code != 0, (
|
||||
"write-note returned an error JSON payload "
|
||||
f"({payload.get('error')}) but exited 0. Sibling tool commands "
|
||||
"(delete-note, edit-note, search-notes) exit non-zero on error."
|
||||
)
|
||||
|
||||
|
||||
# --- #5: blocked NOTE_ALREADY_EXISTS write must not report success ---
|
||||
|
||||
|
||||
def _cli_write(project_name: str):
|
||||
return runner.invoke(
|
||||
cli_app,
|
||||
[
|
||||
"tool",
|
||||
"write-note",
|
||||
"--title",
|
||||
"Conflict Note",
|
||||
"--folder",
|
||||
"conflict",
|
||||
"--content",
|
||||
"# Conflict Note\n\nFirst body.\n",
|
||||
"--project",
|
||||
project_name,
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
def test_mcp_write_note_conflict_emits_error(app, app_config, test_project, config_manager):
|
||||
"""Baseline: the MCP tool reports NOTE_ALREADY_EXISTS on a blocked re-write."""
|
||||
|
||||
async def _go():
|
||||
first = await mcp_write_note(
|
||||
title="Conflict Note",
|
||||
content="# Conflict Note\n\nFirst body.\n",
|
||||
directory="conflict",
|
||||
project=test_project.name,
|
||||
output_format="json",
|
||||
)
|
||||
# output_format="json" returns a dict; narrow for the type checker.
|
||||
assert isinstance(first, dict)
|
||||
assert first.get("action") == "created"
|
||||
assert "error" not in first
|
||||
|
||||
second = await mcp_write_note(
|
||||
title="Conflict Note",
|
||||
content="# Conflict Note\n\nSecond body (should be blocked).\n",
|
||||
directory="conflict",
|
||||
project=test_project.name,
|
||||
output_format="json",
|
||||
)
|
||||
return second
|
||||
|
||||
second = asyncio.run(_go())
|
||||
assert isinstance(second, dict)
|
||||
assert second.get("error") == "NOTE_ALREADY_EXISTS"
|
||||
assert second.get("action") == "conflict"
|
||||
assert second.get("file_path") is None
|
||||
|
||||
|
||||
def test_cli_write_note_conflict_should_exit_nonzero(app, app_config, test_project, config_manager):
|
||||
"""CLI write-note must NOT exit 0 when the write was blocked by a conflict."""
|
||||
first = _cli_write(test_project.name)
|
||||
assert first.exit_code == 0, first.output
|
||||
first_payload = json.loads(first.stdout)
|
||||
assert first_payload["action"] == "created"
|
||||
|
||||
second = _cli_write(test_project.name)
|
||||
|
||||
assert "NOTE_ALREADY_EXISTS" in second.stdout, second.output
|
||||
|
||||
assert second.exit_code != 0, (
|
||||
f"write-note exited {second.exit_code} after a blocked NOTE_ALREADY_EXISTS "
|
||||
"write; the note was NOT written but the CLI reported success"
|
||||
)
|
||||
|
||||
|
||||
# --- #2: write-note --overwrite flag (MCP overwrite=True parity) ---
|
||||
|
||||
|
||||
def _write_overwrite(args_extra: list[str]):
|
||||
return runner.invoke(
|
||||
cli_app,
|
||||
[
|
||||
"tool",
|
||||
"write-note",
|
||||
"--title",
|
||||
"Overwrite Parity Note",
|
||||
"--folder",
|
||||
"parity-overwrite",
|
||||
"--content",
|
||||
"# Overwrite Parity Note\n\nVERSION_BODY",
|
||||
*args_extra,
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
def test_write_note_cli_can_overwrite_like_mcp(app, app_config, test_project, config_manager):
|
||||
"""CLI write-note must be able to overwrite an existing note (MCP overwrite=True)."""
|
||||
first = _write_overwrite(["--project", test_project.name])
|
||||
assert first.exit_code == 0, first.output
|
||||
first_data = json.loads(first.stdout)
|
||||
permalink = first_data["permalink"]
|
||||
|
||||
second = runner.invoke(
|
||||
cli_app,
|
||||
[
|
||||
"tool",
|
||||
"write-note",
|
||||
"--title",
|
||||
"Overwrite Parity Note",
|
||||
"--folder",
|
||||
"parity-overwrite",
|
||||
"--content",
|
||||
"# Overwrite Parity Note\n\nNEW_VERSION_BODY",
|
||||
"--project",
|
||||
test_project.name,
|
||||
"--overwrite",
|
||||
],
|
||||
)
|
||||
|
||||
assert second.exit_code == 0, (
|
||||
"CLI write-note has no way to overwrite an existing note even though "
|
||||
"the MCP write_note tool supports overwrite=True. "
|
||||
f"exit_code={second.exit_code} output={second.output}"
|
||||
)
|
||||
|
||||
read = runner.invoke(
|
||||
cli_app,
|
||||
["tool", "read-note", permalink, "--project", test_project.name],
|
||||
)
|
||||
assert read.exit_code == 0, read.output
|
||||
read_data = json.loads(read.stdout)
|
||||
assert "NEW_VERSION_BODY" in (read_data.get("content") or "")
|
||||
@@ -0,0 +1,219 @@
|
||||
"""Regression tests for move_note edge cases found by the integration bug hunt.
|
||||
|
||||
Bug #11: move_note did not resolve memory:// URL identifiers, even though its
|
||||
docstring advertises them and read_note/edit_note/delete_note all accept them.
|
||||
|
||||
Bug #12: move_note's structural "<seg>/projects/<seg>/file.md" heuristic wrongly
|
||||
rejected legitimate same-project nested moves (e.g. "notes/projects/2025/file.md")
|
||||
as cross-project moves. The heuristic was removed; cross-project detection now relies
|
||||
on the leading segment matching a known project name plus the post-move outcome
|
||||
backstop.
|
||||
|
||||
These are integration tests: real MCP server -> FastAPI -> database -> filesystem.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from fastmcp import Client
|
||||
|
||||
|
||||
def _result(r):
|
||||
return r.structured_content["result"]
|
||||
|
||||
|
||||
# --- Bug #11: memory:// URL resolution ---
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_move_note_accepts_memory_url(mcp_server, app, test_project):
|
||||
"""move_note should accept a memory:// URL identifier like read/edit/delete do."""
|
||||
async with Client(mcp_server) as client:
|
||||
await client.call_tool(
|
||||
"write_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"title": "Memory Url Move",
|
||||
"directory": "src",
|
||||
"content": "# Memory Url Move\n\nbody",
|
||||
"output_format": "json",
|
||||
},
|
||||
)
|
||||
move = await client.call_tool(
|
||||
"move_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"identifier": "memory://src/memory-url-move",
|
||||
"destination_path": "dst/memory-url-move.md",
|
||||
"output_format": "json",
|
||||
},
|
||||
)
|
||||
result = _result(move)
|
||||
assert result["moved"] is True, (
|
||||
f"move_note should resolve memory:// URLs but failed: {result.get('error')}"
|
||||
)
|
||||
assert result["file_path"] == "dst/memory-url-move.md"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_move_note_bare_permalink_works_control(mcp_server, app, test_project):
|
||||
"""Control: bare permalink (no memory://) DOES work for move_note."""
|
||||
async with Client(mcp_server) as client:
|
||||
await client.call_tool(
|
||||
"write_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"title": "Bare Permalink Move",
|
||||
"directory": "src",
|
||||
"content": "# Bare Permalink Move\n\nbody",
|
||||
"output_format": "json",
|
||||
},
|
||||
)
|
||||
move = await client.call_tool(
|
||||
"move_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"identifier": "src/bare-permalink-move",
|
||||
"destination_path": "dst/bare-permalink-move.md",
|
||||
"output_format": "json",
|
||||
},
|
||||
)
|
||||
result = _result(move)
|
||||
assert result["moved"] is True, result.get("error")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_delete_note_accepts_memory_url_control(mcp_server, app, test_project):
|
||||
"""Control: delete_note DOES resolve memory:// URLs (proves the contract)."""
|
||||
async with Client(mcp_server) as client:
|
||||
await client.call_tool(
|
||||
"write_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"title": "Delete Memory Url",
|
||||
"directory": "src",
|
||||
"content": "# Delete Memory Url\n\nbody",
|
||||
"output_format": "json",
|
||||
},
|
||||
)
|
||||
d = await client.call_tool(
|
||||
"delete_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"identifier": "memory://src/delete-memory-url",
|
||||
"output_format": "json",
|
||||
},
|
||||
)
|
||||
assert _result(d)["deleted"] is True
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_edit_note_accepts_memory_url_control(mcp_server, app, test_project):
|
||||
"""Control: edit_note DOES resolve memory:// URLs (proves the contract)."""
|
||||
async with Client(mcp_server) as client:
|
||||
await client.call_tool(
|
||||
"write_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"title": "Edit Memory Url",
|
||||
"directory": "src",
|
||||
"content": "# Edit Memory Url\n\nbody",
|
||||
"output_format": "json",
|
||||
},
|
||||
)
|
||||
e = await client.call_tool(
|
||||
"edit_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"identifier": "memory://src/edit-memory-url",
|
||||
"operation": "append",
|
||||
"content": "\nEDITED",
|
||||
"output_format": "json",
|
||||
},
|
||||
)
|
||||
ec = _result(e)
|
||||
assert ec.get("error") is None
|
||||
assert ec.get("fileCreated") is False
|
||||
|
||||
|
||||
# --- Bug #12: nested 'projects' folder false positive ---
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_move_into_nested_projects_folder_not_flagged(mcp_server, app, test_project):
|
||||
"""A legit nested folder like notes/projects/2025/note.md must NOT be flagged
|
||||
as a cross-project move (the structural 'projects'-segment heuristic was removed)."""
|
||||
async with Client(mcp_server) as client:
|
||||
await client.call_tool(
|
||||
"write_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"title": "Nested Projects Note",
|
||||
"directory": "inbox",
|
||||
"content": "# Nested\n\nbody",
|
||||
"output_format": "json",
|
||||
},
|
||||
)
|
||||
move = await client.call_tool(
|
||||
"move_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"identifier": "inbox/nested-projects-note",
|
||||
"destination_path": "notes/projects/2025/nested-projects-note.md",
|
||||
"output_format": "json",
|
||||
},
|
||||
)
|
||||
result = _result(move)
|
||||
assert result.get("error") != "CROSS_PROJECT_MOVE_NOT_SUPPORTED", (
|
||||
"Legit nested 'projects' folder wrongly flagged as cross-project move"
|
||||
)
|
||||
assert result["moved"] is True, result.get("error")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_move_outcome_mismatch_guidance_uses_actual_landing_path(
|
||||
mcp_server, app, test_project, monkeypatch
|
||||
):
|
||||
"""The text guidance should point agents at the actual post-move path.
|
||||
|
||||
The real move service stores the requested destination verbatim today, so the
|
||||
mismatch backstop is exercised by returning a divergent file_path after the real
|
||||
move completes.
|
||||
"""
|
||||
async with Client(mcp_server) as client:
|
||||
await client.call_tool(
|
||||
"write_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"title": "Outcome Guidance Note",
|
||||
"directory": "source",
|
||||
"content": "# Outcome Guidance Note\n\nbody",
|
||||
"output_format": "json",
|
||||
},
|
||||
)
|
||||
|
||||
from basic_memory.mcp.clients import KnowledgeClient
|
||||
|
||||
real_move_entity = KnowledgeClient.move_entity
|
||||
actual_landing_path = "unexpected/outcome-guidance-note.md"
|
||||
|
||||
async def diverging_move_entity(self, entity_id, destination_path):
|
||||
result = await real_move_entity(self, entity_id, destination_path)
|
||||
return result.model_copy(update={"file_path": actual_landing_path})
|
||||
|
||||
monkeypatch.setattr(KnowledgeClient, "move_entity", diverging_move_entity)
|
||||
|
||||
move = await client.call_tool(
|
||||
"move_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"identifier": "source/outcome-guidance-note",
|
||||
"destination_path": "target/outcome-guidance-note.md",
|
||||
},
|
||||
)
|
||||
|
||||
guidance = move.content[0].text
|
||||
stale_identifier = "source/outcome-guidance-note"
|
||||
assert "Unexpected Result Location" in guidance
|
||||
assert f'read_note("{actual_landing_path}")' in guidance
|
||||
assert f'delete_note("{actual_landing_path}", project="{test_project.name}")' in guidance
|
||||
assert f'read_note("{stale_identifier}")' not in guidance
|
||||
assert f'delete_note("{stale_identifier}", project="{test_project.name}")' not in guidance
|
||||
@@ -0,0 +1,241 @@
|
||||
"""Bughunt regression tests: pagination + project-name display in navigation tools.
|
||||
|
||||
These integration tests come from the integration-test bug hunt. They exercise the
|
||||
real MCP server (FastMCP Client), real DB, and real ASGI routing — no mocks.
|
||||
|
||||
Covered bugs:
|
||||
- #9 search_notes accepted non-positive page_size and returned a misleading
|
||||
has_more=true with zero rows (inconsistent with recent_activity validation).
|
||||
- #10 build_context with page_size<=0 silently dropped the requested primary entity
|
||||
(primary_count=0 for a valid memory:// URL).
|
||||
- #13 recent_activity text output printed the raw project UUID instead of the
|
||||
project name when routed via project_id.
|
||||
"""
|
||||
|
||||
import json
|
||||
|
||||
import pytest
|
||||
from fastmcp import Client
|
||||
from fastmcp.exceptions import ToolError
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from basic_memory.cli.main import app as cli_app
|
||||
|
||||
runner = CliRunner()
|
||||
|
||||
|
||||
def _parse(result):
|
||||
return json.loads(result.content[0].text)
|
||||
|
||||
|
||||
async def _seed_notes(client, project, n=15):
|
||||
for i in range(n):
|
||||
await client.call_tool(
|
||||
"write_note",
|
||||
{
|
||||
"project": project,
|
||||
"title": f"Pg Note {i + 1:02d}",
|
||||
"directory": "pg",
|
||||
"content": f"# Pg Note {i + 1:02d}\n\npagination probe content.",
|
||||
"tags": "pg,probe",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
# --- Bug #9: search_notes pagination validation ---
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_notes_rejects_nonpositive_page_size(mcp_server, app, test_project):
|
||||
"""search_notes must reject non-positive page_size like recent_activity does,
|
||||
instead of returning a misleading has_more=true with zero rows."""
|
||||
async with Client(mcp_server) as client:
|
||||
await _seed_notes(client, test_project.name, 15)
|
||||
|
||||
# Baseline: recent_activity correctly rejects page_size < 1.
|
||||
with pytest.raises(ToolError, match="page_size"):
|
||||
await client.call_tool(
|
||||
"recent_activity",
|
||||
{"project": test_project.name, "page_size": -3, "output_format": "json"},
|
||||
)
|
||||
|
||||
# search_notes must now reject page_size=0 with the same guard, rather than
|
||||
# returning an empty payload that misleadingly claims more results exist.
|
||||
with pytest.raises(ToolError, match="page_size"):
|
||||
await client.call_tool(
|
||||
"search_notes",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"query": "pagination",
|
||||
"search_type": "text",
|
||||
"page": 1,
|
||||
"page_size": 0,
|
||||
"output_format": "json",
|
||||
},
|
||||
)
|
||||
|
||||
# Negative page_size must not return an arbitrary uncapped slice — also rejected.
|
||||
with pytest.raises(ToolError, match="page_size"):
|
||||
await client.call_tool(
|
||||
"search_notes",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"query": "pagination",
|
||||
"search_type": "text",
|
||||
"page": 1,
|
||||
"page_size": -3,
|
||||
"output_format": "json",
|
||||
},
|
||||
)
|
||||
|
||||
# page < 1 is rejected too.
|
||||
with pytest.raises(ToolError, match="page"):
|
||||
await client.call_tool(
|
||||
"search_notes",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"query": "pagination",
|
||||
"search_type": "text",
|
||||
"page": 0,
|
||||
"page_size": 5,
|
||||
"output_format": "json",
|
||||
},
|
||||
)
|
||||
|
||||
# A valid page_size still works and reports honest pagination.
|
||||
ok = await client.call_tool(
|
||||
"search_notes",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"query": "pagination",
|
||||
"search_type": "text",
|
||||
"page": 1,
|
||||
"page_size": 5,
|
||||
"output_format": "json",
|
||||
},
|
||||
)
|
||||
d_ok = _parse(ok)
|
||||
assert len(d_ok["results"]) == 5
|
||||
assert d_ok["has_more"] is True
|
||||
|
||||
|
||||
# --- Bug #9 (CLI surface): search-notes --page-size 0 ---
|
||||
|
||||
|
||||
def _write_cli(title, folder="pgcli"):
|
||||
return runner.invoke(
|
||||
cli_app,
|
||||
[
|
||||
"tool",
|
||||
"write-note",
|
||||
"--title",
|
||||
title,
|
||||
"--folder",
|
||||
folder,
|
||||
"--content",
|
||||
f"# {title}\n\nclipagination probe body.",
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
def test_cli_search_notes_page_size_zero(app, app_config, test_project, config_manager):
|
||||
"""`bm tool search-notes ... --page-size 0` must fail fast instead of returning a
|
||||
misleading has_more=true with zero rows."""
|
||||
for i in range(5):
|
||||
w = _write_cli(f"CliPg Note {i:02d}")
|
||||
assert w.exit_code == 0, w.output
|
||||
|
||||
res = runner.invoke(
|
||||
cli_app,
|
||||
["tool", "search-notes", "clipagination", "--local", "--page-size", "0"],
|
||||
)
|
||||
# The fix raises ValueError, which the CLI maps to a non-zero exit with a clear
|
||||
# message — the faithful "no misleading pagination signal" outcome at the CLI.
|
||||
assert res.exit_code != 0, res.output
|
||||
assert "page_size" in res.output
|
||||
|
||||
|
||||
# --- Bug #10: build_context must always return the requested primary entity ---
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_build_context_nonpositive_page_size_drops_primary(mcp_server, app, test_project):
|
||||
"""build_context with a valid URL must return its primary entity (or raise) —
|
||||
a non-positive page_size must never silently drop it."""
|
||||
async with Client(mcp_server) as client:
|
||||
await client.call_tool(
|
||||
"write_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"title": "Primary Note",
|
||||
"directory": "ctx",
|
||||
"content": "# Primary Note\n\n## Relations\n- relates_to [[Other Note]]\n",
|
||||
"tags": "ctx",
|
||||
},
|
||||
)
|
||||
await client.call_tool(
|
||||
"write_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"title": "Other Note",
|
||||
"directory": "ctx",
|
||||
"content": "# Other Note\n\nrelated content.\n",
|
||||
"tags": "ctx",
|
||||
},
|
||||
)
|
||||
|
||||
# Sanity: a normal page_size returns the primary note.
|
||||
r_ok = await client.call_tool(
|
||||
"build_context",
|
||||
{"project": test_project.name, "url": "ctx/primary-note", "output_format": "json"},
|
||||
)
|
||||
d_ok = _parse(r_ok)
|
||||
assert d_ok["metadata"]["primary_count"] == 1
|
||||
|
||||
# page_size=0 must NOT silently drop the requested entity. The fix rejects it
|
||||
# with a clear error rather than returning primary_count=0.
|
||||
with pytest.raises(ToolError, match="page_size"):
|
||||
await client.call_tool(
|
||||
"build_context",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"url": "ctx/primary-note",
|
||||
"page_size": 0,
|
||||
"output_format": "json",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
# --- Bug #13: recent_activity header shows project name, not raw UUID ---
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_recent_activity_project_id_header_shows_name_not_uuid(
|
||||
mcp_server, app, test_project, config_manager
|
||||
):
|
||||
"""When routed via project_id, the text header must name the project, not the UUID."""
|
||||
async with Client(mcp_server) as client:
|
||||
await client.call_tool(
|
||||
"write_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"title": "RA Header Note",
|
||||
"directory": "ra",
|
||||
"content": "# RA Header Note\n\nToken",
|
||||
"output_format": "json",
|
||||
},
|
||||
)
|
||||
|
||||
result = await client.call_tool(
|
||||
"recent_activity",
|
||||
{"project_id": test_project.external_id, "output_format": "text"},
|
||||
)
|
||||
text = result.content[0].text
|
||||
|
||||
# The human-readable header should reference the project NAME, not the UUID.
|
||||
assert test_project.external_id not in text, (
|
||||
f"recent_activity header leaked the raw external_id UUID:\n{text[:300]}"
|
||||
)
|
||||
assert test_project.name in text, (
|
||||
f"recent_activity header should contain the project name:\n{text[:300]}"
|
||||
)
|
||||
@@ -0,0 +1,81 @@
|
||||
"""Bug: CLI `write-note --tags "a,b"` does NOT split the comma string, but the
|
||||
MCP write_note(tags="a,b") DOES (parse_tags splits a bare string but treats each
|
||||
list element as a single literal tag).
|
||||
|
||||
Typer collects a single --tags value into a one-element list ['a,b'], and
|
||||
parse_tags(['a,b']) returns ['a,b'] (no per-element comma split). The MCP tool
|
||||
receives the bare string 'a,b' and parse_tags('a,b') returns ['a','b'].
|
||||
|
||||
Result: the SAME comma-string input yields different tags on CLI vs MCP, even
|
||||
though write_note's docstring promises comma-separated-string support.
|
||||
"""
|
||||
|
||||
import json
|
||||
import pytest
|
||||
from fastmcp import Client
|
||||
from typer.testing import CliRunner
|
||||
from basic_memory.cli.main import app as cli_app
|
||||
|
||||
runner = CliRunner()
|
||||
|
||||
|
||||
def test_cli_write_note_comma_tags_split_matches_mcp(app, app_config, test_project, config_manager):
|
||||
# CLI: single --tags value containing a comma
|
||||
write = runner.invoke(
|
||||
cli_app,
|
||||
[
|
||||
"tool",
|
||||
"write-note",
|
||||
"--title",
|
||||
"CLI Comma Split",
|
||||
"--folder",
|
||||
"cli-comma-split",
|
||||
"--content",
|
||||
"# CLI Comma Split\n\nbody",
|
||||
"--tags",
|
||||
"alpha,beta",
|
||||
],
|
||||
)
|
||||
assert write.exit_code == 0, write.output
|
||||
permalink = json.loads(write.stdout)["permalink"]
|
||||
|
||||
read = runner.invoke(
|
||||
cli_app,
|
||||
["tool", "read-note", permalink, "--include-frontmatter", "--local"],
|
||||
)
|
||||
assert read.exit_code == 0, read.output
|
||||
content = json.loads(read.stdout)["content"]
|
||||
|
||||
# Correct behavior: two distinct tags (matching MCP write_note semantics).
|
||||
# splitlines() is line-ending agnostic (Windows CRLF vs POSIX LF).
|
||||
content_lines = content.splitlines()
|
||||
assert "- alpha" in content_lines and "- beta" in content_lines, (
|
||||
"CLI --tags 'alpha,beta' should split into two tags like MCP write_note does; "
|
||||
f"got frontmatter:\n{content}"
|
||||
)
|
||||
assert "alpha,beta" not in content, "comma string must not survive as a single literal tag"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_mcp_write_note_comma_tags_split_baseline(mcp_server, app, test_project):
|
||||
"""Baseline: MCP write_note DOES split comma strings (the behavior CLI should match)."""
|
||||
async with Client(mcp_server) as client:
|
||||
await client.call_tool(
|
||||
"write_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"title": "MCP Comma Split",
|
||||
"directory": "mcp-comma-split",
|
||||
"content": "# MCP Comma Split\n\nbody",
|
||||
"tags": "alpha,beta",
|
||||
},
|
||||
)
|
||||
read = await client.call_tool(
|
||||
"read_note",
|
||||
{"project": test_project.name, "identifier": "MCP Comma Split"},
|
||||
)
|
||||
text = read.content[0].text
|
||||
text_lines = text.splitlines()
|
||||
assert "- alpha" in text_lines and "- beta" in text_lines, (
|
||||
f"MCP write_note should split comma string into two tags; got:\n{text}"
|
||||
)
|
||||
@@ -0,0 +1,277 @@
|
||||
"""Integration tests for `basic-memory tool delete-note`."""
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from basic_memory.cli.main import app as cli_app
|
||||
|
||||
runner = CliRunner()
|
||||
|
||||
|
||||
def _write_note(
|
||||
title: str,
|
||||
folder: str,
|
||||
content: str,
|
||||
*,
|
||||
project: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
args = [
|
||||
"tool",
|
||||
"write-note",
|
||||
"--title",
|
||||
title,
|
||||
"--folder",
|
||||
folder,
|
||||
"--content",
|
||||
content,
|
||||
]
|
||||
if project is not None:
|
||||
args.extend(["--project", project])
|
||||
|
||||
result = runner.invoke(cli_app, args)
|
||||
assert result.exit_code == 0, result.output
|
||||
return json.loads(result.stdout)
|
||||
|
||||
|
||||
def _read_note(identifier: str, *, project: str | None = None) -> dict[str, Any]:
|
||||
args = ["tool", "read-note", identifier]
|
||||
if project is not None:
|
||||
args.extend(["--project", project])
|
||||
|
||||
result = runner.invoke(cli_app, args)
|
||||
assert result.exit_code == 0, result.output
|
||||
return json.loads(result.stdout)
|
||||
|
||||
|
||||
def _delete_note(
|
||||
identifier: str,
|
||||
*,
|
||||
is_directory: bool = False,
|
||||
project: str | None = None,
|
||||
project_id: str | None = None,
|
||||
local: bool = False,
|
||||
) -> tuple[int, dict[str, Any], str]:
|
||||
args = ["tool", "delete-note", identifier]
|
||||
if is_directory:
|
||||
args.append("--is-directory")
|
||||
if project is not None:
|
||||
args.extend(["--project", project])
|
||||
if project_id is not None:
|
||||
args.extend(["--project-id", project_id])
|
||||
if local:
|
||||
args.append("--local")
|
||||
|
||||
result = runner.invoke(cli_app, args)
|
||||
payload = json.loads(result.stdout) if result.stdout else {}
|
||||
return result.exit_code, payload, result.output
|
||||
|
||||
|
||||
def _search_notes(
|
||||
query: str,
|
||||
*,
|
||||
mode_flag: str | None = None,
|
||||
page_size: int = 20,
|
||||
) -> dict[str, Any]:
|
||||
args = ["tool", "search-notes", query, "--page-size", str(page_size)]
|
||||
if mode_flag is not None:
|
||||
args.append(mode_flag)
|
||||
|
||||
result = runner.invoke(
|
||||
cli_app,
|
||||
args,
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
return json.loads(result.stdout)
|
||||
|
||||
|
||||
def _project_file(test_project, file_path: str) -> Path:
|
||||
return Path(test_project.path) / file_path
|
||||
|
||||
|
||||
def test_delete_note_removes_file_database_record_and_search_result(
|
||||
app, app_config, test_project, config_manager
|
||||
) -> None:
|
||||
"""Single-note deletion removes the note from every user-visible surface."""
|
||||
note = _write_note(
|
||||
"CLI Delete Single Note",
|
||||
"delete-cli",
|
||||
"# CLI Delete Single Note\n\nUniqueSingleDeleteToken\n\n- [status] ready to delete",
|
||||
)
|
||||
note_path = _project_file(test_project, note["file_path"])
|
||||
assert note_path.exists()
|
||||
|
||||
exit_code, payload, output = _delete_note(note["permalink"])
|
||||
|
||||
assert exit_code == 0, output
|
||||
assert payload == {
|
||||
"deleted": True,
|
||||
"title": "CLI Delete Single Note",
|
||||
"permalink": note["permalink"],
|
||||
"file_path": note["file_path"],
|
||||
}
|
||||
assert not note_path.exists()
|
||||
|
||||
missing = _read_note(note["permalink"])
|
||||
assert missing["title"] is None
|
||||
assert missing["permalink"] is None
|
||||
assert missing["content"] is None
|
||||
|
||||
search = _search_notes("CLI Delete Single Note", mode_flag="--title")
|
||||
assert search["total"] == 0
|
||||
assert search["results"] == []
|
||||
|
||||
|
||||
def test_delete_note_not_found_returns_json_without_error(
|
||||
app, app_config, test_project, config_manager
|
||||
) -> None:
|
||||
"""A missing note is machine-readable and does not produce a CLI failure."""
|
||||
exit_code, payload, output = _delete_note("delete-cli/missing-note")
|
||||
|
||||
assert exit_code == 0, output
|
||||
assert payload == {
|
||||
"deleted": False,
|
||||
"title": None,
|
||||
"permalink": None,
|
||||
"file_path": None,
|
||||
}
|
||||
|
||||
|
||||
def test_delete_note_case_mismatch_does_not_delete_exact_note(
|
||||
app, app_config, test_project, config_manager
|
||||
) -> None:
|
||||
"""Strict CLI deletes must not fuzzy-match a differently cased title."""
|
||||
note = _write_note(
|
||||
"CLI CamelCase Delete Note",
|
||||
"delete-cli",
|
||||
"# CLI CamelCase Delete Note\n\nCaseSensitiveDeleteToken",
|
||||
)
|
||||
|
||||
exit_code, payload, output = _delete_note("cli camelcase delete note")
|
||||
|
||||
assert exit_code == 0, output
|
||||
assert payload["deleted"] is False
|
||||
still_there = _read_note(note["permalink"])
|
||||
assert still_there["title"] == "CLI CamelCase Delete Note"
|
||||
assert "CaseSensitiveDeleteToken" in still_there["content"]
|
||||
|
||||
|
||||
def test_delete_note_project_id_takes_precedence_over_wrong_project_name(
|
||||
app, app_config, test_project, config_manager
|
||||
) -> None:
|
||||
"""CLI `--project-id` routes destructive operations to the exact project."""
|
||||
note = _write_note(
|
||||
"CLI Delete By Project ID",
|
||||
"delete-cli",
|
||||
"# CLI Delete By Project ID\n\nProjectIdDeleteToken",
|
||||
)
|
||||
|
||||
exit_code, payload, output = _delete_note(
|
||||
note["file_path"],
|
||||
project="not-the-test-project",
|
||||
project_id=test_project.external_id,
|
||||
)
|
||||
|
||||
assert exit_code == 0, output
|
||||
assert payload["deleted"] is True
|
||||
assert payload["title"] == "CLI Delete By Project ID"
|
||||
assert _read_note(note["permalink"])["title"] is None
|
||||
|
||||
|
||||
def test_delete_note_memory_url_detects_project_from_identifier(
|
||||
app, app_config, test_project, config_manager
|
||||
) -> None:
|
||||
"""A memory:// URL can select the project without a separate --project flag."""
|
||||
note = _write_note(
|
||||
"CLI Delete Memory URL",
|
||||
"delete-cli",
|
||||
"# CLI Delete Memory URL\n\nMemoryUrlDeleteToken",
|
||||
project=test_project.name,
|
||||
)
|
||||
memory_url = f"memory://{test_project.name}/{note['permalink']}"
|
||||
|
||||
exit_code, payload, output = _delete_note(memory_url)
|
||||
|
||||
assert exit_code == 0, output
|
||||
assert payload["deleted"] is True
|
||||
assert payload["permalink"] == note["permalink"]
|
||||
assert _read_note(note["permalink"], project=test_project.name)["title"] is None
|
||||
|
||||
|
||||
def test_delete_directory_removes_nested_files_database_records_and_search_results(
|
||||
app, app_config, test_project, config_manager
|
||||
) -> None:
|
||||
"""Directory deletion removes nested notes and reports a complete JSON summary."""
|
||||
notes = [
|
||||
_write_note(
|
||||
"CLI Delete Directory Root",
|
||||
"delete-cli-dir",
|
||||
"# CLI Delete Directory Root\n\nDirectoryDeleteTokenRoot",
|
||||
),
|
||||
_write_note(
|
||||
"CLI Delete Directory Child",
|
||||
"delete-cli-dir/child",
|
||||
"# CLI Delete Directory Child\n\nDirectoryDeleteTokenChild",
|
||||
),
|
||||
_write_note(
|
||||
"CLI Delete Directory Deep Child",
|
||||
"delete-cli-dir/child/deep",
|
||||
"# CLI Delete Directory Deep Child\n\nDirectoryDeleteTokenDeep",
|
||||
),
|
||||
]
|
||||
note_paths = [_project_file(test_project, note["file_path"]) for note in notes]
|
||||
assert all(path.exists() for path in note_paths)
|
||||
|
||||
exit_code, payload, output = _delete_note("delete-cli-dir", is_directory=True, local=True)
|
||||
|
||||
assert exit_code == 0, output
|
||||
assert payload["deleted"] is True
|
||||
assert payload["is_directory"] is True
|
||||
assert payload["identifier"] == "delete-cli-dir"
|
||||
assert payload["total_files"] == 3
|
||||
assert payload["successful_deletes"] == 3
|
||||
assert payload["failed_deletes"] == 0
|
||||
assert payload["errors"] == []
|
||||
assert set(payload["deleted_files"]) == {note["file_path"] for note in notes}
|
||||
assert not any(path.exists() for path in note_paths)
|
||||
|
||||
for note in notes:
|
||||
assert _read_note(note["permalink"])["title"] is None
|
||||
|
||||
search = _search_notes("CLI Delete Directory", mode_flag="--title")
|
||||
assert search["total"] == 0
|
||||
assert search["results"] == []
|
||||
|
||||
|
||||
def test_delete_directory_without_flag_does_not_delete_child_notes(
|
||||
app, app_config, test_project, config_manager
|
||||
) -> None:
|
||||
"""The CLI must not treat a directory path as destructive without --is-directory."""
|
||||
note = _write_note(
|
||||
"CLI Delete Directory Safety",
|
||||
"delete-cli-safety",
|
||||
"# CLI Delete Directory Safety\n\nDirectorySafetyToken",
|
||||
)
|
||||
|
||||
exit_code, payload, output = _delete_note("delete-cli-safety")
|
||||
|
||||
assert exit_code == 0, output
|
||||
assert payload["deleted"] is False
|
||||
still_there = _read_note(note["permalink"])
|
||||
assert still_there["title"] == "CLI Delete Directory Safety"
|
||||
assert _project_file(test_project, note["file_path"]).exists()
|
||||
|
||||
|
||||
def test_delete_note_rejects_conflicting_routing_flags(
|
||||
app, app_config, test_project, config_manager
|
||||
) -> None:
|
||||
"""delete-note validates the same --local/--cloud conflict as other tool commands."""
|
||||
result = runner.invoke(
|
||||
cli_app,
|
||||
["tool", "delete-note", "delete-cli/missing-note", "--local", "--cloud"],
|
||||
)
|
||||
|
||||
assert result.exit_code != 0
|
||||
assert "Cannot specify both --local and --cloud" in result.output
|
||||
@@ -0,0 +1,94 @@
|
||||
"""Integration coverage for `bm tool write-note --type` (Issue #875)."""
|
||||
|
||||
import json
|
||||
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from basic_memory.cli.main import app as cli_app
|
||||
|
||||
runner = CliRunner()
|
||||
|
||||
|
||||
def test_write_note_type_flag_round_trip(app, app_config, test_project, config_manager):
|
||||
"""`--type` sets the persisted note type and is searchable via `--type`."""
|
||||
write_result = runner.invoke(
|
||||
cli_app,
|
||||
[
|
||||
"tool",
|
||||
"write-note",
|
||||
"--title",
|
||||
"CLI Typed Note",
|
||||
"--folder",
|
||||
"typed",
|
||||
"--content",
|
||||
"# CLI Typed Note\n\nCliTypeToken body.",
|
||||
"--type",
|
||||
"guide",
|
||||
],
|
||||
)
|
||||
assert write_result.exit_code == 0, write_result.output
|
||||
write_data = json.loads(write_result.stdout)
|
||||
permalink = write_data["permalink"]
|
||||
|
||||
# Read back the frontmatter to confirm the persisted type.
|
||||
read_result = runner.invoke(
|
||||
cli_app,
|
||||
["tool", "read-note", permalink, "--include-frontmatter"],
|
||||
)
|
||||
assert read_result.exit_code == 0, read_result.output
|
||||
read_data = json.loads(read_result.stdout)
|
||||
assert read_data["frontmatter"]["type"] == "guide"
|
||||
|
||||
# The search note-type filter must return the typed note.
|
||||
search_result = runner.invoke(
|
||||
cli_app,
|
||||
[
|
||||
"tool",
|
||||
"search-notes",
|
||||
"CliTypeToken",
|
||||
"--type",
|
||||
"guide",
|
||||
"--local",
|
||||
"--page-size",
|
||||
"20",
|
||||
],
|
||||
)
|
||||
assert search_result.exit_code == 0, search_result.output
|
||||
search_data = json.loads(search_result.stdout)
|
||||
permalinks = {item["permalink"] for item in search_data["results"]}
|
||||
assert permalink in permalinks
|
||||
|
||||
|
||||
def test_write_note_content_frontmatter_type_wins_over_flag(
|
||||
app, app_config, test_project, config_manager
|
||||
):
|
||||
"""A `type:` in content frontmatter takes precedence over `--type` (documented behavior)."""
|
||||
content = "---\ntype: session\n---\n# Frontmatter Wins\n\nFrontmatterWinsToken body."
|
||||
|
||||
write_result = runner.invoke(
|
||||
cli_app,
|
||||
[
|
||||
"tool",
|
||||
"write-note",
|
||||
"--title",
|
||||
"Frontmatter Wins",
|
||||
"--folder",
|
||||
"typed",
|
||||
"--content",
|
||||
content,
|
||||
"--type",
|
||||
"guide",
|
||||
],
|
||||
)
|
||||
assert write_result.exit_code == 0, write_result.output
|
||||
write_data = json.loads(write_result.stdout)
|
||||
permalink = write_data["permalink"]
|
||||
|
||||
read_result = runner.invoke(
|
||||
cli_app,
|
||||
["tool", "read-note", permalink, "--include-frontmatter"],
|
||||
)
|
||||
assert read_result.exit_code == 0, read_result.output
|
||||
read_data = json.loads(read_result.stdout)
|
||||
# Content frontmatter "session" wins over the --type "guide" flag.
|
||||
assert read_data["frontmatter"]["type"] == "session"
|
||||
@@ -0,0 +1,43 @@
|
||||
"""Integration test for `bm status --wait` against a real local project.
|
||||
|
||||
Unlike the unit tests in tests/cli/test_json_output.py (which mock get_status
|
||||
to drive deterministic poll sequences), this exercises the full stack: the CLI
|
||||
runs a real disk-vs-DB scan via the API/repository layer. After write-note
|
||||
indexes a file, the project is already in sync, so --wait observes total == 0
|
||||
on the first poll and exits 0 immediately.
|
||||
"""
|
||||
|
||||
import json
|
||||
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from basic_memory.cli.main import app as cli_app
|
||||
|
||||
runner = CliRunner()
|
||||
|
||||
|
||||
def test_status_wait_returns_once_indexed(app, app_config, test_project, config_manager):
|
||||
"""status --wait exits 0 with total == 0 when the project is fully indexed."""
|
||||
# Write (and index) a note so the project has real content on disk + in DB.
|
||||
write_result = runner.invoke(
|
||||
cli_app,
|
||||
[
|
||||
"tool",
|
||||
"write-note",
|
||||
"--title",
|
||||
"Wait Test Note",
|
||||
"--folder",
|
||||
"test-notes",
|
||||
"--content",
|
||||
"# Wait Test\n\nContent that should be indexed.",
|
||||
],
|
||||
)
|
||||
assert write_result.exit_code == 0, write_result.output
|
||||
|
||||
# --wait should observe a synced project (total == 0) and exit immediately.
|
||||
result = runner.invoke(cli_app, ["status", "--wait", "--json"])
|
||||
|
||||
assert result.exit_code == 0, result.output
|
||||
start = result.output.index("{")
|
||||
data = json.loads(result.output[start:])
|
||||
assert data["total"] == 0
|
||||
@@ -381,6 +381,13 @@ def app_config(
|
||||
sync_changes=False, # Disable file sync in tests - prevents lifespan from starting blocking task
|
||||
database_backend=database_backend,
|
||||
database_url=database_url,
|
||||
# Trigger: semantic_search_enabled defaults to True whenever fastembed/sqlite-vec
|
||||
# are importable, which they are in dev and CI environments.
|
||||
# Why: with it on, every test that syncs pays the ONNX embedding stack (~5-7s per
|
||||
# sync) — embeddings are covered by test-int/semantic/, which configures
|
||||
# semantic_search_enabled explicitly in its own conftest.
|
||||
# Outcome: non-semantic integration tests skip embedding work entirely.
|
||||
semantic_search_enabled=False,
|
||||
)
|
||||
return app_config
|
||||
|
||||
|
||||
@@ -4,6 +4,8 @@ Integration tests for move_note MCP tool.
|
||||
Tests the complete move note workflow: MCP client -> MCP server -> FastAPI -> database -> file system
|
||||
"""
|
||||
|
||||
import json
|
||||
|
||||
import pytest
|
||||
from fastmcp import Client
|
||||
|
||||
@@ -769,3 +771,224 @@ async def test_move_note_strict_resolution_rejects_fuzzy_match(mcp_server, app,
|
||||
{"project": test_project.name, "identifier": "Move Strict Test B"},
|
||||
)
|
||||
assert "Content B" in read_b.content[0].text
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_move_note_unknown_workspace_shaped_path_allowed(mcp_server, app, test_project):
|
||||
"""A "<seg>/projects/<seg>/..." destination whose leading segment is NOT a known
|
||||
project is a legitimate same-project nested move and must succeed.
|
||||
|
||||
The old "projects"-segment structural heuristic (#904) wrongly rejected this shape
|
||||
even though it is structurally identical to a normal nested folder. Detection now
|
||||
relies only on the leading segment matching a known project name (Detection 1) plus
|
||||
the post-move outcome backstop, so a bare "other-workspace/projects/x/..." that
|
||||
matches no known project is treated as a normal nested move.
|
||||
"""
|
||||
|
||||
async with Client(mcp_server) as client:
|
||||
await client.call_tool(
|
||||
"write_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"title": "Workspace Shape Note",
|
||||
"directory": "source",
|
||||
"content": "# Workspace Shape Note\n\nNested move into a projects folder.",
|
||||
},
|
||||
)
|
||||
|
||||
move_result = await client.call_tool(
|
||||
"move_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"identifier": "Workspace Shape Note",
|
||||
"destination_path": "other-workspace/projects/x/moved-note.md",
|
||||
},
|
||||
)
|
||||
|
||||
move_text = move_result.content[0].text
|
||||
assert "Cross-Project Move Not Supported" not in move_text
|
||||
assert "✅ Note moved successfully" in move_text
|
||||
assert "other-workspace/projects/x/moved-note.md" in move_text
|
||||
|
||||
# The note landed at the requested nested location.
|
||||
read_moved = await client.call_tool(
|
||||
"read_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"identifier": "other-workspace/projects/x/moved-note.md",
|
||||
},
|
||||
)
|
||||
assert "Nested move into a projects folder" in read_moved.content[0].text
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_move_note_destination_folder_boundary_rejected(mcp_server, app, test_project):
|
||||
"""The destination_folder bypass (#881 Gap 3) must now be detected honestly.
|
||||
|
||||
Previously the cross-boundary guard ran before destination_folder was resolved into
|
||||
destination_path, so a boundary-shaped folder slipped through and reported success.
|
||||
"""
|
||||
|
||||
async with Client(mcp_server) as client:
|
||||
await client.call_tool(
|
||||
"create_memory_project",
|
||||
{
|
||||
"project_name": "boundary-target-project",
|
||||
"project_path": "/tmp/boundary-target-project",
|
||||
"set_default": False,
|
||||
},
|
||||
)
|
||||
|
||||
await client.call_tool(
|
||||
"write_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"title": "Folder Boundary Note",
|
||||
"directory": "source",
|
||||
"content": "# Folder Boundary Note\n\nFolder bypass should be caught.",
|
||||
},
|
||||
)
|
||||
|
||||
# destination_folder names another project — resolved path routes cross-project.
|
||||
move_result = await client.call_tool(
|
||||
"move_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"identifier": "Folder Boundary Note",
|
||||
"destination_folder": "boundary-target-project",
|
||||
},
|
||||
)
|
||||
|
||||
error_message = move_result.content[0].text
|
||||
assert "Cross-Project Move Not Supported" in error_message
|
||||
assert "boundary-target-project" in error_message
|
||||
|
||||
# Note stays put.
|
||||
read_original = await client.call_tool(
|
||||
"read_note",
|
||||
{"project": test_project.name, "identifier": "Folder Boundary Note"},
|
||||
)
|
||||
assert "Folder bypass should be caught" in read_original.content[0].text
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_move_note_unknown_workspace_shaped_path_allowed_json(mcp_server, app, test_project):
|
||||
"""JSON output for an unknown-workspace-shaped destination reports moved=True.
|
||||
|
||||
With the ambiguous "projects"-segment heuristic removed (#904), a
|
||||
"<seg>/projects/<seg>/..." path whose leading segment is not a known project is a
|
||||
legitimate same-project nested move, not a cross-project rejection.
|
||||
"""
|
||||
|
||||
async with Client(mcp_server) as client:
|
||||
await client.call_tool(
|
||||
"write_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"title": "Workspace Shape JSON Note",
|
||||
"directory": "source",
|
||||
"content": "# Workspace Shape JSON Note\n\nJSON path.",
|
||||
},
|
||||
)
|
||||
|
||||
move_result = await client.call_tool(
|
||||
"move_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"identifier": "Workspace Shape JSON Note",
|
||||
"destination_path": "team-space/projects/alpha/moved.md",
|
||||
"output_format": "json",
|
||||
},
|
||||
)
|
||||
|
||||
data = json.loads(move_result.content[0].text)
|
||||
assert data["moved"] is True
|
||||
# Success JSON does not include an error key.
|
||||
assert "error" not in data
|
||||
assert data["file_path"] == "team-space/projects/alpha/moved.md"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_move_note_new_nested_folder_still_succeeds(mcp_server, app, test_project):
|
||||
"""A legitimate same-project move into a brand-new nested folder must still succeed.
|
||||
|
||||
Guards against false positives from the broadened cross-boundary detection. Note the
|
||||
top-level "projects/" folder is a valid same-project location and must NOT be flagged.
|
||||
"""
|
||||
|
||||
async with Client(mcp_server) as client:
|
||||
await client.call_tool(
|
||||
"write_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"title": "Legit Nested Note",
|
||||
"directory": "source",
|
||||
"content": "# Legit Nested Note\n\nValid same-project nested move.",
|
||||
},
|
||||
)
|
||||
|
||||
move_result = await client.call_tool(
|
||||
"move_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"identifier": "Legit Nested Note",
|
||||
"destination_path": "projects/2025/q2/legit-nested-note.md",
|
||||
},
|
||||
)
|
||||
|
||||
move_text = move_result.content[0].text
|
||||
assert "✅ Note moved successfully" in move_text
|
||||
assert "projects/2025/q2/legit-nested-note.md" in move_text
|
||||
|
||||
read_result = await client.call_tool(
|
||||
"read_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"identifier": "projects/2025/q2/legit-nested-note.md",
|
||||
},
|
||||
)
|
||||
assert "Valid same-project nested move" in read_result.content[0].text
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_move_note_interior_projects_segment_still_succeeds(mcp_server, app, test_project):
|
||||
"""A path containing an interior 'projects' segment must not trip cross-boundary detection.
|
||||
|
||||
Regression for the false positive where any path containing a 'projects' segment
|
||||
(e.g. "notes/projects/my-project/note.md") was rejected. The ambiguous structural
|
||||
heuristic has been removed (#904): a 'projects' folder anywhere in the path is just a
|
||||
normal nested folder, so this is a valid same-project move and must succeed.
|
||||
"""
|
||||
|
||||
async with Client(mcp_server) as client:
|
||||
await client.call_tool(
|
||||
"write_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"title": "Interior Projects Note",
|
||||
"directory": "source",
|
||||
"content": "# Interior Projects Note\n\nInterior projects segment is fine.",
|
||||
},
|
||||
)
|
||||
|
||||
move_result = await client.call_tool(
|
||||
"move_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"identifier": "Interior Projects Note",
|
||||
"destination_path": "team/2026/projects/alpha/interior-projects-note.md",
|
||||
},
|
||||
)
|
||||
|
||||
move_text = move_result.content[0].text
|
||||
assert "✅ Note moved successfully" in move_text
|
||||
assert "team/2026/projects/alpha/interior-projects-note.md" in move_text
|
||||
|
||||
read_result = await client.call_tool(
|
||||
"read_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"identifier": "team/2026/projects/alpha/interior-projects-note.md",
|
||||
},
|
||||
)
|
||||
assert "Interior projects segment is fine" in read_result.content[0].text
|
||||
|
||||
@@ -13,9 +13,49 @@ import pytest
|
||||
from fastmcp import Client
|
||||
|
||||
|
||||
# --- read_note: pagination params removed in #693 (were no-ops) ---
|
||||
# The `page` / `page_size` parameters were removed because the API endpoint
|
||||
# silently dropped them. Search-fallback pagination is unrelated to read_note.
|
||||
# --- read_note: pagination params restored in #883 ---
|
||||
# `page` / `page_size` were removed in #693 because they were no-ops on the
|
||||
# resource read. #883 restored them for parity with the sibling navigation
|
||||
# tools: they paginate the server-side search fallback (a direct match still
|
||||
# returns the full note content).
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_read_note_accepts_pagination_params_and_aliases(mcp_server, app, test_project):
|
||||
"""Agents passing page/page_size (or their aliases) must not get a validation error."""
|
||||
async with Client(mcp_server) as client:
|
||||
await client.call_tool(
|
||||
"write_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"title": "Paged Read Note",
|
||||
"directory": "test",
|
||||
"content": "# Paged Read Note\n\npaged read body",
|
||||
},
|
||||
)
|
||||
|
||||
result = await client.call_tool(
|
||||
"read_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"identifier": "Paged Read Note",
|
||||
"page": 1,
|
||||
"page_size": 10,
|
||||
},
|
||||
)
|
||||
assert "paged read body" in result.content[0].text
|
||||
|
||||
# Aliases map silently: page_number -> page, limit -> page_size
|
||||
result = await client.call_tool(
|
||||
"read_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"identifier": "Paged Read Note",
|
||||
"page_number": 1,
|
||||
"limit": 10,
|
||||
},
|
||||
)
|
||||
assert "paged read body" in result.content[0].text
|
||||
|
||||
|
||||
# --- edit_note: find_text / content / section aliases ---
|
||||
@@ -485,12 +525,12 @@ async def test_aliases_not_advertised_in_schema(mcp_server, app):
|
||||
|
||||
# tool_name -> (must_have_canonical, must_not_have_aliases)
|
||||
checks = {
|
||||
# read_note has no pagination params (#693 — they were no-ops; removed).
|
||||
# The must_not_have list still includes the rejected aliases so future
|
||||
# contributors don't reintroduce them.
|
||||
# read_note pagination restored in #883 (paginates the search fallback).
|
||||
# Accepted aliases (limit/page_number/per_page) plus the rejected
|
||||
# `offset` must stay out of the advertised schema.
|
||||
"read_note": (
|
||||
[],
|
||||
["page", "page_size", "offset", "limit", "page_number", "per_page"],
|
||||
["page", "page_size"],
|
||||
["offset", "limit", "page_number", "per_page"],
|
||||
),
|
||||
"edit_note": (
|
||||
["find_text", "section", "content"],
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
"""Bughunt: search_notes note_types filter is documented case-insensitive but
|
||||
fails to match capitalized frontmatter `type` values.
|
||||
"""
|
||||
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from fastmcp import Client
|
||||
|
||||
|
||||
def _json(tool_result) -> Any:
|
||||
assert len(tool_result.content) == 1
|
||||
assert tool_result.content[0].type == "text"
|
||||
return json.loads(tool_result.content[0].text)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_note_types_filter_is_case_insensitive(mcp_server, app, test_project):
|
||||
async with Client(mcp_server) as client:
|
||||
content = (
|
||||
"---\n"
|
||||
"title: Capitalized Type Note\n"
|
||||
"type: Chapter\n"
|
||||
"---\n"
|
||||
"# Capitalized Type Note\n\nuniqtoken99 body text\n"
|
||||
)
|
||||
await client.call_tool(
|
||||
"write_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"title": "Capitalized Type Note",
|
||||
"directory": "types",
|
||||
"content": content,
|
||||
},
|
||||
)
|
||||
plain = await client.call_tool(
|
||||
"search_notes",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"query": "uniqtoken99",
|
||||
"search_type": "text",
|
||||
"output_format": "json",
|
||||
},
|
||||
)
|
||||
plain_data = _json(plain)
|
||||
assert plain_data["results"], "note not indexed at all"
|
||||
res = await client.call_tool(
|
||||
"search_notes",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"query": "uniqtoken99",
|
||||
"search_type": "text",
|
||||
"note_types": ["Chapter"],
|
||||
"output_format": "json",
|
||||
},
|
||||
)
|
||||
data = _json(res)
|
||||
titles = [r["title"] for r in data["results"]]
|
||||
assert "Capitalized Type Note" in titles, (
|
||||
"note_types filter is documented case-insensitive but did not match the "
|
||||
f"capitalized frontmatter type 'Chapter'. results={data}"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_note_types_lowercase_control(mcp_server, app, test_project):
|
||||
"""Control: a lowercase frontmatter type DOES match (proves the bug is casing)."""
|
||||
async with Client(mcp_server) as client:
|
||||
content = (
|
||||
"---\n"
|
||||
"title: Lowercase Type Note\n"
|
||||
"type: chapter\n"
|
||||
"---\n"
|
||||
"# Lowercase Type Note\n\nuniqtoken88 body text\n"
|
||||
)
|
||||
await client.call_tool(
|
||||
"write_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"title": "Lowercase Type Note",
|
||||
"directory": "types",
|
||||
"content": content,
|
||||
},
|
||||
)
|
||||
res = await client.call_tool(
|
||||
"search_notes",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"query": "uniqtoken88",
|
||||
"search_type": "text",
|
||||
"note_types": ["Chapter"],
|
||||
"output_format": "json",
|
||||
},
|
||||
)
|
||||
data = _json(res)
|
||||
titles = [r["title"] for r in data["results"]]
|
||||
assert "Lowercase Type Note" in titles, data
|
||||
@@ -5,14 +5,25 @@ Comprehensive tests covering all scenarios including note creation, content form
|
||||
tag handling, error conditions, and edge cases from bug reports.
|
||||
"""
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
from textwrap import dedent
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from fastmcp import Client
|
||||
|
||||
from basic_memory.config import ConfigManager
|
||||
from basic_memory.schemas.project_info import ProjectItem
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def _json_content(tool_result) -> dict[str, Any]:
|
||||
"""Parse a FastMCP tool result content block into a JSON object."""
|
||||
assert len(tool_result.content) == 1
|
||||
assert tool_result.content[0].type == "text"
|
||||
payload = json.loads(tool_result.content[0].text) # pyright: ignore [reportAttributeAccessIssue]
|
||||
assert isinstance(payload, dict)
|
||||
return payload
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -113,6 +124,83 @@ async def test_write_note_update_existing(mcp_server, app, test_project):
|
||||
assert f"[Session: Using project '{test_project.name}']" in response_text
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_write_note_overwrite_resolves_conflict_by_file_path_when_permalink_changes(
|
||||
mcp_server, app, test_project, monkeypatch
|
||||
):
|
||||
"""Overwrite resolves the conflict by strict file path through the MCP client stack."""
|
||||
from basic_memory.mcp.clients import knowledge as knowledge_mod
|
||||
|
||||
original_resolve = knowledge_mod.KnowledgeClient.resolve_entity
|
||||
captured_resolve: dict[str, Any] = {}
|
||||
|
||||
async def spy_resolve(self, identifier: str, *, strict: bool = False) -> str:
|
||||
captured_resolve["identifier"] = identifier
|
||||
captured_resolve["strict"] = strict
|
||||
return await original_resolve(self, identifier, strict=strict)
|
||||
|
||||
monkeypatch.setattr(knowledge_mod.KnowledgeClient, "resolve_entity", spy_resolve)
|
||||
|
||||
async with Client(mcp_server) as client:
|
||||
created = await client.call_tool(
|
||||
"write_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"title": "Overwrite Permalink Change",
|
||||
"directory": "overwrite-conflicts",
|
||||
"content": "# Overwrite Permalink Change\n\nOriginal body.",
|
||||
"output_format": "json",
|
||||
},
|
||||
)
|
||||
created_payload = _json_content(created)
|
||||
assert created_payload["permalink"] == (
|
||||
f"{test_project.name}/overwrite-conflicts/overwrite-permalink-change"
|
||||
)
|
||||
|
||||
replacement = dedent("""
|
||||
---
|
||||
permalink: overwrite-conflicts/custom-overwrite-permalink
|
||||
---
|
||||
|
||||
# Overwrite Permalink Change
|
||||
|
||||
Replacement body.
|
||||
""").strip()
|
||||
|
||||
updated = await client.call_tool(
|
||||
"write_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"title": "Overwrite Permalink Change",
|
||||
"directory": "overwrite-conflicts",
|
||||
"content": replacement,
|
||||
"overwrite": True,
|
||||
"output_format": "json",
|
||||
},
|
||||
)
|
||||
updated_payload = _json_content(updated)
|
||||
assert updated_payload["action"] == "updated"
|
||||
assert updated_payload["permalink"] == "overwrite-conflicts/custom-overwrite-permalink"
|
||||
assert updated_payload["file_path"] == "overwrite-conflicts/Overwrite Permalink Change.md"
|
||||
assert captured_resolve == {
|
||||
"identifier": "overwrite-conflicts/Overwrite Permalink Change.md",
|
||||
"strict": True,
|
||||
}
|
||||
|
||||
read_updated = await client.call_tool(
|
||||
"read_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"identifier": "overwrite-conflicts/custom-overwrite-permalink",
|
||||
"output_format": "json",
|
||||
},
|
||||
)
|
||||
read_payload = _json_content(read_updated)
|
||||
assert read_payload["permalink"] == "overwrite-conflicts/custom-overwrite-permalink"
|
||||
assert "Replacement body." in read_payload["content"]
|
||||
assert "Original body." not in read_payload["content"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_write_note_tag_array(mcp_server, app, test_project):
|
||||
"""Test creating a note with tag array (Issue #38 regression test)."""
|
||||
|
||||
@@ -0,0 +1,154 @@
|
||||
"""Integration tests locking in write_note note_type behavior (Issue #875).
|
||||
|
||||
These exercise the real MCP harness (no mocks) to confirm that:
|
||||
- content frontmatter ``type:`` is persisted as the note type and is searchable
|
||||
via the ``note_types`` filter;
|
||||
- overwriting a note with a different content ``type:`` flips the persisted type.
|
||||
|
||||
The CLI ``--type`` passthrough is covered separately in
|
||||
``test-int/cli/test_cli_tool_write_note_type_integration.py``.
|
||||
"""
|
||||
|
||||
import json
|
||||
from textwrap import dedent
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from fastmcp import Client
|
||||
|
||||
|
||||
def _json_content(tool_result) -> dict[str, Any]:
|
||||
"""Parse a FastMCP tool result content block into a JSON object."""
|
||||
assert len(tool_result.content) == 1
|
||||
assert tool_result.content[0].type == "text"
|
||||
payload = json.loads(tool_result.content[0].text) # pyright: ignore [reportAttributeAccessIssue]
|
||||
assert isinstance(payload, dict)
|
||||
return payload
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_write_note_content_type_is_persisted_and_searchable(mcp_server, app, test_project):
|
||||
"""Content frontmatter ``type:`` persists and is found by the note_types filter."""
|
||||
note = dedent("""
|
||||
---
|
||||
title: Session Log
|
||||
type: session
|
||||
---
|
||||
|
||||
# Session Log
|
||||
|
||||
SessionTypeToken content body.
|
||||
""").strip()
|
||||
|
||||
async with Client(mcp_server) as client:
|
||||
await client.call_tool(
|
||||
"write_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"title": "Session Log",
|
||||
"directory": "logs",
|
||||
"content": note,
|
||||
"output_format": "json",
|
||||
},
|
||||
)
|
||||
|
||||
# The persisted frontmatter should report the content-declared type.
|
||||
read_result = await client.call_tool(
|
||||
"read_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"identifier": "logs/session-log",
|
||||
"include_frontmatter": True,
|
||||
"output_format": "json",
|
||||
},
|
||||
)
|
||||
read_payload = _json_content(read_result)
|
||||
assert read_payload["frontmatter"]["type"] == "session"
|
||||
|
||||
# And the note_types filter must return it (and only it for this token).
|
||||
search_result = await client.call_tool(
|
||||
"search_notes",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"query": "SessionTypeToken",
|
||||
"search_type": "text",
|
||||
"note_types": ["session"],
|
||||
"output_format": "json",
|
||||
},
|
||||
)
|
||||
search_payload = _json_content(search_result)
|
||||
permalinks = {item["permalink"] for item in search_payload["results"]}
|
||||
assert any(p.endswith("logs/session-log") for p in permalinks)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_write_note_overwrite_flips_persisted_type(mcp_server, app, test_project):
|
||||
"""Overwriting with a different content ``type:`` flips the persisted note type."""
|
||||
session_note = dedent("""
|
||||
---
|
||||
title: Type Flip
|
||||
type: session
|
||||
---
|
||||
|
||||
# Type Flip
|
||||
|
||||
Original session body.
|
||||
""").strip()
|
||||
|
||||
schema_note = dedent("""
|
||||
---
|
||||
title: Type Flip
|
||||
type: schema
|
||||
---
|
||||
|
||||
# Type Flip
|
||||
|
||||
Replacement schema body.
|
||||
""").strip()
|
||||
|
||||
async with Client(mcp_server) as client:
|
||||
await client.call_tool(
|
||||
"write_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"title": "Type Flip",
|
||||
"directory": "flip",
|
||||
"content": session_note,
|
||||
"output_format": "json",
|
||||
},
|
||||
)
|
||||
|
||||
before = await client.call_tool(
|
||||
"read_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"identifier": "flip/type-flip",
|
||||
"include_frontmatter": True,
|
||||
"output_format": "json",
|
||||
},
|
||||
)
|
||||
assert _json_content(before)["frontmatter"]["type"] == "session"
|
||||
|
||||
# Overwrite with a different content type.
|
||||
await client.call_tool(
|
||||
"write_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"title": "Type Flip",
|
||||
"directory": "flip",
|
||||
"content": schema_note,
|
||||
"overwrite": True,
|
||||
"output_format": "json",
|
||||
},
|
||||
)
|
||||
|
||||
after = await client.call_tool(
|
||||
"read_note",
|
||||
{
|
||||
"project": test_project.name,
|
||||
"identifier": "flip/type-flip",
|
||||
"include_frontmatter": True,
|
||||
"output_format": "json",
|
||||
},
|
||||
)
|
||||
assert _json_content(after)["frontmatter"]["type"] == "schema"
|
||||
@@ -0,0 +1,401 @@
|
||||
"""Opt-in live LiteLLM embedding evaluation harness."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
from collections.abc import Callable, Mapping, Sequence
|
||||
from dataclasses import asdict, dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from rich.console import Console
|
||||
from rich.table import Table
|
||||
|
||||
from basic_memory.repository.litellm_provider import LiteLLMEmbeddingProvider
|
||||
|
||||
|
||||
RELATED_DOCUMENT = "OAuth login refresh tokens keep an authenticated web session active."
|
||||
DISTRACTOR_DOCUMENT = "A sourdough starter ferments flour and water before bread baking."
|
||||
QUERY_TEXT = "authentication login token flow"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class LiteLLMLiveCase:
|
||||
"""A real LiteLLM embedding model to exercise end-to-end."""
|
||||
|
||||
name: str
|
||||
model: str
|
||||
dimensions: int
|
||||
api_key_env: str | None = None
|
||||
document_input_type: str | None = None
|
||||
query_input_type: str | None = None
|
||||
forward_dimensions: bool | None = None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class LiteLLMLiveResult:
|
||||
"""Measured result for a live LiteLLM embedding case."""
|
||||
|
||||
name: str
|
||||
model: str
|
||||
dimensions: int
|
||||
api_key_env: str | None
|
||||
document_input_type: str | None
|
||||
query_input_type: str | None
|
||||
forward_dimensions: bool | None
|
||||
related_score: float
|
||||
distractor_score: float
|
||||
min_norm: float
|
||||
max_norm: float
|
||||
embed_documents_latency_ms: float
|
||||
embed_query_latency_ms: float
|
||||
total_latency_ms: float
|
||||
|
||||
|
||||
type ProviderFactory = Callable[..., Any]
|
||||
|
||||
|
||||
def _required_string(case_data: Mapping[str, Any], key: str) -> str:
|
||||
value = case_data.get(key)
|
||||
if not isinstance(value, str) or not value:
|
||||
raise ValueError(f"LiteLLM live case must include non-empty string field {key!r}")
|
||||
return value
|
||||
|
||||
|
||||
def _optional_string(case_data: Mapping[str, Any], key: str) -> str | None:
|
||||
value = case_data.get(key)
|
||||
if value is None:
|
||||
return None
|
||||
if not isinstance(value, str) or not value:
|
||||
raise ValueError(f"LiteLLM live case field {key!r} must be a non-empty string")
|
||||
return value
|
||||
|
||||
|
||||
def _optional_bool(case_data: Mapping[str, Any], key: str) -> bool | None:
|
||||
value = case_data.get(key)
|
||||
if value is None:
|
||||
return None
|
||||
if not isinstance(value, bool):
|
||||
raise ValueError(f"LiteLLM live case field {key!r} must be a boolean")
|
||||
return value
|
||||
|
||||
|
||||
def load_custom_cases(raw: str | None) -> list[LiteLLMLiveCase]:
|
||||
"""Load additional live model cases from JSON."""
|
||||
if not raw:
|
||||
return []
|
||||
|
||||
values = json.loads(raw)
|
||||
if not isinstance(values, list):
|
||||
raise ValueError("LiteLLM live cases JSON must be an array")
|
||||
|
||||
cases: list[LiteLLMLiveCase] = []
|
||||
for value in values:
|
||||
if not isinstance(value, dict):
|
||||
raise ValueError("Each LiteLLM live case must be a JSON object")
|
||||
|
||||
case_data: dict[str, Any] = value
|
||||
dimensions = case_data.get("dimensions")
|
||||
if type(dimensions) is not int or dimensions <= 0:
|
||||
raise ValueError("LiteLLM live case must include positive integer field 'dimensions'")
|
||||
|
||||
cases.append(
|
||||
LiteLLMLiveCase(
|
||||
name=_required_string(case_data, "name"),
|
||||
model=_required_string(case_data, "model"),
|
||||
dimensions=dimensions,
|
||||
api_key_env=_optional_string(case_data, "api_key_env"),
|
||||
document_input_type=_optional_string(case_data, "document_input_type"),
|
||||
query_input_type=_optional_string(case_data, "query_input_type"),
|
||||
forward_dimensions=_optional_bool(case_data, "forward_dimensions"),
|
||||
)
|
||||
)
|
||||
return cases
|
||||
|
||||
|
||||
def configured_cases(
|
||||
environ: Mapping[str, str] | None = None,
|
||||
*,
|
||||
custom_cases_raw: str | None = None,
|
||||
) -> list[LiteLLMLiveCase]:
|
||||
"""Return built-in and user-supplied live cases whose credentials are available."""
|
||||
env = os.environ if environ is None else environ
|
||||
cases: list[LiteLLMLiveCase] = []
|
||||
|
||||
if env.get("OPENAI_API_KEY"):
|
||||
cases.append(
|
||||
LiteLLMLiveCase(
|
||||
name="openai-text-embedding-3-small",
|
||||
model="openai/text-embedding-3-small",
|
||||
dimensions=1536,
|
||||
api_key_env="OPENAI_API_KEY",
|
||||
)
|
||||
)
|
||||
|
||||
if env.get("COHERE_API_KEY"):
|
||||
cases.append(
|
||||
LiteLLMLiveCase(
|
||||
name="cohere-embed-english-v3",
|
||||
model="cohere/embed-english-v3.0",
|
||||
dimensions=1024,
|
||||
api_key_env="COHERE_API_KEY",
|
||||
)
|
||||
)
|
||||
|
||||
raw = (
|
||||
custom_cases_raw
|
||||
if custom_cases_raw is not None
|
||||
else env.get("BASIC_MEMORY_TEST_LITELLM_CASES")
|
||||
)
|
||||
cases.extend(load_custom_cases(raw))
|
||||
return cases
|
||||
|
||||
|
||||
def cosine(a: list[float], b: list[float]) -> float:
|
||||
"""Compute cosine similarity for live ranking sanity checks."""
|
||||
dot = sum(x * y for x, y in zip(a, b, strict=True))
|
||||
norm_a = vector_norm(a)
|
||||
norm_b = vector_norm(b)
|
||||
if norm_a == 0 or norm_b == 0:
|
||||
return 0.0
|
||||
return dot / (norm_a * norm_b)
|
||||
|
||||
|
||||
def vector_norm(vector: list[float]) -> float:
|
||||
"""Return the Euclidean norm of a vector."""
|
||||
return math.sqrt(sum(value * value for value in vector))
|
||||
|
||||
|
||||
def assert_valid_vector(vector: list[float], dimensions: int) -> float:
|
||||
"""Assert provider output is a usable normalized vector and return its norm."""
|
||||
assert len(vector) == dimensions
|
||||
assert all(math.isfinite(value) for value in vector)
|
||||
norm = vector_norm(vector)
|
||||
assert math.isclose(norm, 1.0, abs_tol=1e-6)
|
||||
return norm
|
||||
|
||||
|
||||
async def evaluate_case(
|
||||
case: LiteLLMLiveCase,
|
||||
*,
|
||||
environ: Mapping[str, str] | None = None,
|
||||
provider_factory: ProviderFactory = LiteLLMEmbeddingProvider,
|
||||
) -> LiteLLMLiveResult:
|
||||
"""Run one live LiteLLM case and return measured ranking/vector metrics."""
|
||||
env = os.environ if environ is None else environ
|
||||
api_key = env.get(case.api_key_env) if case.api_key_env else None
|
||||
provider = provider_factory(
|
||||
model_name=case.model,
|
||||
dimensions=case.dimensions,
|
||||
batch_size=2,
|
||||
api_key=api_key,
|
||||
timeout=60.0,
|
||||
document_input_type=case.document_input_type,
|
||||
query_input_type=case.query_input_type,
|
||||
forward_dimensions=case.forward_dimensions,
|
||||
)
|
||||
|
||||
start = time.perf_counter()
|
||||
documents_start = time.perf_counter()
|
||||
vectors = await provider.embed_documents([RELATED_DOCUMENT, DISTRACTOR_DOCUMENT])
|
||||
documents_elapsed = time.perf_counter() - documents_start
|
||||
|
||||
query_start = time.perf_counter()
|
||||
query_vector = await provider.embed_query(QUERY_TEXT)
|
||||
query_elapsed = time.perf_counter() - query_start
|
||||
total_elapsed = time.perf_counter() - start
|
||||
|
||||
assert len(vectors) == 2
|
||||
norms = [assert_valid_vector(vector, case.dimensions) for vector in [*vectors, query_vector]]
|
||||
related_score = cosine(query_vector, vectors[0])
|
||||
distractor_score = cosine(query_vector, vectors[1])
|
||||
assert related_score > distractor_score, (
|
||||
f"{case.name} ranked the related document at {related_score:.4f}, "
|
||||
f"not above distractor {distractor_score:.4f}"
|
||||
)
|
||||
|
||||
return LiteLLMLiveResult(
|
||||
name=case.name,
|
||||
model=case.model,
|
||||
dimensions=case.dimensions,
|
||||
api_key_env=case.api_key_env,
|
||||
document_input_type=case.document_input_type,
|
||||
query_input_type=case.query_input_type,
|
||||
forward_dimensions=case.forward_dimensions,
|
||||
related_score=related_score,
|
||||
distractor_score=distractor_score,
|
||||
min_norm=min(norms),
|
||||
max_norm=max(norms),
|
||||
embed_documents_latency_ms=documents_elapsed * 1000,
|
||||
embed_query_latency_ms=query_elapsed * 1000,
|
||||
total_latency_ms=total_elapsed * 1000,
|
||||
)
|
||||
|
||||
|
||||
async def evaluate_cases(
|
||||
cases: Sequence[LiteLLMLiveCase],
|
||||
*,
|
||||
environ: Mapping[str, str] | None = None,
|
||||
provider_factory: ProviderFactory = LiteLLMEmbeddingProvider,
|
||||
) -> tuple[list[LiteLLMLiveResult], list[tuple[LiteLLMLiveCase, str]]]:
|
||||
"""Evaluate cases sequentially and collect all failures instead of failing fast."""
|
||||
env = os.environ if environ is None else environ
|
||||
results: list[LiteLLMLiveResult] = []
|
||||
failures: list[tuple[LiteLLMLiveCase, str]] = []
|
||||
|
||||
for case in cases:
|
||||
if case.api_key_env and not env.get(case.api_key_env):
|
||||
failures.append((case, f"missing required env var {case.api_key_env}"))
|
||||
continue
|
||||
try:
|
||||
results.append(
|
||||
await evaluate_case(case, environ=env, provider_factory=provider_factory)
|
||||
)
|
||||
except Exception as exc: # pragma: no cover - exercised by live provider failures
|
||||
failures.append((case, f"{type(exc).__name__}: {exc}"))
|
||||
|
||||
return results, failures
|
||||
|
||||
|
||||
def build_results_table(
|
||||
results: Sequence[LiteLLMLiveResult],
|
||||
failures: Sequence[tuple[LiteLLMLiveCase, str]],
|
||||
) -> Table:
|
||||
"""Build a rich report table for live LiteLLM results."""
|
||||
table = Table(title="LiteLLM Live Embedding Evaluation", show_lines=False)
|
||||
table.add_column("Status", no_wrap=True)
|
||||
table.add_column("Case", style="cyan", no_wrap=True)
|
||||
table.add_column("Model", style="magenta")
|
||||
table.add_column("Dims", justify="right")
|
||||
table.add_column("Roles", no_wrap=True)
|
||||
table.add_column("Forward dims", no_wrap=True)
|
||||
table.add_column("Related", justify="right")
|
||||
table.add_column("Distractor", justify="right")
|
||||
table.add_column("Norm", justify="right")
|
||||
table.add_column("Total ms", justify="right")
|
||||
|
||||
for result in results:
|
||||
roles = _roles_label(result.document_input_type, result.query_input_type)
|
||||
table.add_row(
|
||||
"[green]PASS[/green]",
|
||||
result.name,
|
||||
result.model,
|
||||
str(result.dimensions),
|
||||
roles,
|
||||
_bool_label(result.forward_dimensions),
|
||||
f"{result.related_score:.4f}",
|
||||
f"{result.distractor_score:.4f}",
|
||||
f"{result.min_norm:.4f}-{result.max_norm:.4f}",
|
||||
f"{result.total_latency_ms:.1f}",
|
||||
)
|
||||
|
||||
for case, reason in failures:
|
||||
roles = _roles_label(case.document_input_type, case.query_input_type)
|
||||
table.add_row(
|
||||
"[red]FAIL[/red]",
|
||||
case.name,
|
||||
case.model,
|
||||
str(case.dimensions),
|
||||
roles,
|
||||
_bool_label(case.forward_dimensions),
|
||||
"-",
|
||||
"-",
|
||||
"-",
|
||||
reason,
|
||||
)
|
||||
|
||||
return table
|
||||
|
||||
|
||||
def _roles_label(document_input_type: str | None, query_input_type: str | None) -> str:
|
||||
if document_input_type or query_input_type:
|
||||
return f"{document_input_type or '-'} / {query_input_type or '-'}"
|
||||
return "auto"
|
||||
|
||||
|
||||
def _bool_label(value: bool | None) -> str:
|
||||
if value is True:
|
||||
return "true"
|
||||
if value is False:
|
||||
return "false"
|
||||
return "auto"
|
||||
|
||||
|
||||
def _load_cases_arg(args: argparse.Namespace) -> str | None:
|
||||
if args.cases_json and args.cases_file:
|
||||
raise ValueError("Use --cases-json or --cases-file, not both")
|
||||
if args.cases_json:
|
||||
return str(args.cases_json)
|
||||
if args.cases_file:
|
||||
return Path(args.cases_file).read_text(encoding="utf-8")
|
||||
return None
|
||||
|
||||
|
||||
async def _async_main(args: argparse.Namespace, environ: Mapping[str, str]) -> int:
|
||||
console = Console()
|
||||
|
||||
# Trigger: this command performs real network calls and can spend API quota.
|
||||
# Why: keeping the same explicit opt-in guard as pytest prevents accidental
|
||||
# live calls when someone discovers the harness through `just --list`.
|
||||
# Outcome: humans get a clear command to run before any provider is called.
|
||||
if environ.get("BASIC_MEMORY_RUN_LITELLM_INTEGRATION") != "1":
|
||||
console.print(
|
||||
"[red]Set BASIC_MEMORY_RUN_LITELLM_INTEGRATION=1 to run live LiteLLM "
|
||||
"provider checks.[/red]"
|
||||
)
|
||||
return 2
|
||||
|
||||
custom_cases_raw = _load_cases_arg(args)
|
||||
cases = configured_cases(environ, custom_cases_raw=custom_cases_raw)
|
||||
if not cases:
|
||||
console.print(
|
||||
"[yellow]No LiteLLM live cases configured.[/yellow]\n"
|
||||
"Set OPENAI_API_KEY, COHERE_API_KEY, or provide custom cases with "
|
||||
"BASIC_MEMORY_TEST_LITELLM_CASES / --cases-file."
|
||||
)
|
||||
return 2
|
||||
|
||||
results, failures = await evaluate_cases(cases, environ=environ)
|
||||
|
||||
if args.json:
|
||||
payload = {
|
||||
"results": [asdict(result) for result in results],
|
||||
"failures": [{"case": asdict(case), "reason": reason} for case, reason in failures],
|
||||
}
|
||||
json.dump(payload, sys.stdout, indent=2)
|
||||
sys.stdout.write("\n")
|
||||
else:
|
||||
console.print(build_results_table(results, failures))
|
||||
|
||||
return 1 if failures else 0
|
||||
|
||||
|
||||
def main(argv: Sequence[str] | None = None) -> int:
|
||||
"""CLI entrypoint for live LiteLLM evaluation."""
|
||||
parser = argparse.ArgumentParser(description="Run opt-in live LiteLLM embedding checks")
|
||||
parser.add_argument(
|
||||
"--cases-json",
|
||||
help="JSON array of custom LiteLLM cases; overrides BASIC_MEMORY_TEST_LITELLM_CASES",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--cases-file",
|
||||
help="Path to a JSON file containing custom LiteLLM cases",
|
||||
)
|
||||
parser.add_argument("--json", action="store_true", help="Emit machine-readable JSON")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
try:
|
||||
return asyncio.run(_async_main(args, os.environ))
|
||||
except ValueError as exc:
|
||||
Console(stderr=True).print(f"[red]{exc}[/red]")
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,123 @@
|
||||
"""Integration tests for process-wide embedding provider reuse (#872).
|
||||
|
||||
A long-running ``basic-memory mcp`` server constructs a new search repository per
|
||||
request, per sync batch, and per project. Each repository used to derive its own
|
||||
embedding provider via ``create_embedding_provider()``. If the provider cache key
|
||||
ever drifted, that reloaded the ~2.3GB FastEmbed/ONNX model and leaked memory in
|
||||
onnxruntime's CPU arena (which never returns memory to the OS).
|
||||
|
||||
These tests use the *real* composition paths — ``create_embedding_provider``, the
|
||||
``create_search_repository`` factory, and the FastAPI deps function
|
||||
``get_search_repository`` — with a real FastEmbed provider. FastEmbed loads the
|
||||
ONNX model lazily on first embed, so constructing providers/repositories here is
|
||||
cheap and never touches the native model.
|
||||
|
||||
They deliberately use the semantic suite's ``sqlite_engine_factory`` (not the
|
||||
parent ``engine_factory``): the parent fixture depends on ``postgres_engine``,
|
||||
which this directory overrides to spin up a Docker testcontainer gated only by a
|
||||
``docker`` binary on PATH. On Windows CI that binary exists without a usable
|
||||
daemon, so requesting the parent factory aborts these SQLite-only tests with a
|
||||
testcontainers Ryuk error (#872 follow-up). Staying on the SQLite factory keeps
|
||||
them backend-correct and Docker-free.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import cast
|
||||
|
||||
import pytest
|
||||
|
||||
from basic_memory.config import BasicMemoryConfig, DatabaseBackend, ProjectEntry
|
||||
from basic_memory.deps.repositories import get_search_repository
|
||||
from basic_memory.repository.embedding_provider_factory import (
|
||||
create_embedding_provider,
|
||||
reset_embedding_provider_cache,
|
||||
)
|
||||
from basic_memory.repository.fastembed_provider import FastEmbedEmbeddingProvider
|
||||
from basic_memory.repository.search_repository import create_search_repository
|
||||
from basic_memory.repository.sqlite_search_repository import SQLiteSearchRepository
|
||||
|
||||
|
||||
def _semantic_config(project_path) -> BasicMemoryConfig:
|
||||
"""Build a semantic-enabled FastEmbed config rooted at the test path."""
|
||||
return BasicMemoryConfig(
|
||||
env="test",
|
||||
projects={"test-project": ProjectEntry(path=str(project_path))},
|
||||
default_project="test-project",
|
||||
database_backend=DatabaseBackend.SQLITE,
|
||||
semantic_search_enabled=True,
|
||||
semantic_embedding_provider="fastembed",
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _reset_provider_cache():
|
||||
reset_embedding_provider_cache()
|
||||
yield
|
||||
reset_embedding_provider_cache()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_factory_resolves_single_provider_across_repositories(
|
||||
tmp_path, sqlite_engine_factory
|
||||
):
|
||||
"""The factory must inject one cached provider into every search repository."""
|
||||
_engine, session_maker = sqlite_engine_factory
|
||||
config = _semantic_config(tmp_path)
|
||||
|
||||
expected_provider = create_embedding_provider(config)
|
||||
assert isinstance(expected_provider, FastEmbedEmbeddingProvider)
|
||||
|
||||
# Two repositories, mimicking per-request / per-sync construction.
|
||||
repo_a = cast(
|
||||
SQLiteSearchRepository,
|
||||
create_search_repository(session_maker, project_id=1, app_config=config),
|
||||
)
|
||||
repo_b = cast(
|
||||
SQLiteSearchRepository,
|
||||
create_search_repository(session_maker, project_id=2, app_config=config),
|
||||
)
|
||||
|
||||
# Both repos reuse the exact same cached provider object — no second model load.
|
||||
assert repo_a._embedding_provider is expected_provider
|
||||
assert repo_b._embedding_provider is expected_provider
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_deps_path_reuses_cached_provider(tmp_path, sqlite_engine_factory):
|
||||
"""The real FastAPI deps function must reuse the cached provider, not rebuild it."""
|
||||
_engine, session_maker = sqlite_engine_factory
|
||||
config = _semantic_config(tmp_path)
|
||||
|
||||
expected_provider = create_embedding_provider(config)
|
||||
|
||||
repo = cast(
|
||||
SQLiteSearchRepository,
|
||||
await get_search_repository(
|
||||
session_maker=session_maker,
|
||||
project_id=1,
|
||||
app_config=config,
|
||||
),
|
||||
)
|
||||
|
||||
assert repo._embedding_provider is expected_provider
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_factory_skips_provider_when_semantic_disabled(tmp_path, sqlite_engine_factory):
|
||||
"""With semantic search off, no provider is created and none is injected."""
|
||||
_engine, session_maker = sqlite_engine_factory
|
||||
config = BasicMemoryConfig(
|
||||
env="test",
|
||||
projects={"test-project": ProjectEntry(path=str(tmp_path))},
|
||||
default_project="test-project",
|
||||
database_backend=DatabaseBackend.SQLITE,
|
||||
semantic_search_enabled=False,
|
||||
)
|
||||
|
||||
repo = cast(
|
||||
SQLiteSearchRepository,
|
||||
create_search_repository(session_maker, project_id=1, app_config=config),
|
||||
)
|
||||
|
||||
assert repo._embedding_provider is None
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user