Compare commits

...

54 Commits

Author SHA1 Message Date
claude[bot] acdab2ecfc fix(mcp): apply workspace-canonical prefix in resolve_project_and_path fallback
When build_context is called with project_id for a project in an organization
workspace (e.g. "big-team/main"), the fallback path in resolve_project_and_path
was only prepending the project permalink ("main/tests/*") and ignoring the
workspace slug. Notes in organization workspaces are stored with the full
workspace-prefixed permalink ("big-team/main/tests/test"), so the GLOB query
never matched.

Fix by calling _canonical_memory_path_for_workspace in the fallback when a
workspace context is active. This produces the correct prefix for both
organization workspaces ("big-team/main/tests/*") and personal workspaces
("main/tests/*"), consistent with how the existing workspace_context and
cached_project paths already handle URLs that begin with the qualified prefix.

Add tests for:
- build_context skips URL detection when project_id is provided
- build_context correctly resolves workspace-prefixed paths in org workspaces

Closes #799

Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
2026-05-08 02:30:13 +00:00
phernandez e871298e95 Merge branch 'main' of github.com:basicmachines-co/basic-memory 2026-05-07 18:50:10 -05:00
phernandez 177ae21ba7 pass context to recent_activity
Signed-off-by: phernandez <paul@basicmachines.co>
2026-05-07 18:50:02 -05:00
Drew Cain 3415fd1014 fix(core): parse picoschema modifier descriptions (#796)
Signed-off-by: Drew Cain <groksrc@gmail.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: phernandez <paul@basicmachines.co>
2026-05-06 12:21:35 -05:00
Paul Hernandez b08659e228 fix(api): accept qualified project resolver hints (#795)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-05-06 10:33:11 -05:00
Paul Hernandez 09a4b09436 feat(api): include search result totals (#791)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-05-04 14:57:04 -05:00
Paul Hernandez a661e924df feat(mcp): create projects by workspace slug (#789)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-05-03 19:31:55 -05:00
Paul Hernandez 05adda1502 fix(mcp): route workspace-qualified memory urls (#790)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-05-03 18:18:10 -05:00
Drew Cain 0a72d81bb3 fix(mcp): cap recent_activity rows with explicit truncation footer (#785)
Signed-off-by: Drew Cain <groksrc@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 17:43:42 -05:00
Drew Cain 5ccf433cad fix(cli): point bm cloud setup hint at bm cloud sync-setup (#780)
Signed-off-by: Drew Cain <groksrc@gmail.com>
2026-05-02 17:35:24 -05:00
Paul Hernandez b4bf14ebf7 fix(mcp): list factory projects across workspaces (#778)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-05-02 09:17:25 -05:00
Paul Hernandez 0b335476d6 fix(mcp): resolve projects by external_id, remove workspace from MCP tools (#777)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-01 05:15:44 -05:00
Paul Hernandez 2fccc74a20 feat(cli): refuse db reset while basic-memory mcp processes run (#776)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-04-29 15:48:19 -05:00
Paul Hernandez c4956cac16 fix(cli): cleanup local DB state on set-cloud/set-local (#775)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-04-29 11:50:14 -05:00
Paul Hernandez 128c2da40c fix(cli): clear default_workspace on cloud logout (#773)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-04-29 09:53:09 -05:00
Paul Hernandez 2bfb9c76df fix(core): degrade gracefully when sqlite-vec cannot load on init (#774)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-04-29 08:57:31 -05:00
Paul Hernandez 3d927b848f fix(installer): mount docker-compose config volume to appuser home (#772)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-04-29 08:57:26 -05:00
Paul Hernandez 953fe20aef test(core): regression guard for vector-row cleanup on entity delete (#764) (#771)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-04-28 23:20:15 -05:00
Paul Hernandez a4282d9f2f fix(core): skip Obsidian callouts in observation parser (#769)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-04-28 23:15:13 -05:00
Paul Hernandez 799dd6c629 fix(mcp): remove no-op pagination params from read_note and view_note (#768)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-04-28 23:05:14 -05:00
Paul Hernandez 26e74ea118 test(core): regression guard for long relation_type values (#721) (#770)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-04-28 22:55:14 -05:00
Paul Hernandez ee1558ea68 feat(mcp): accept training-data-friendly parameter aliases (#766)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-04-28 20:10:36 -05:00
Viktor Szépe 4d62b623db chore(core): fix typos (#761)
Signed-off-by: Viktor Szépe <viktor@szepe.net>
2026-04-23 10:12:46 -05:00
Paul Hernandez 2fe4488eda fix(sync): constrain watch service to --project scope (#759)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-04-20 13:46:13 -05:00
Paul Hernandez f3e46d7984 feat(mcp): discover projects across workspaces (#757)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-04-18 14:04:57 -05:00
jope-bm 56d6f1b4a5 fix(mcp): report cloud projects as source=cloud in factory mode (#752)
Signed-off-by: Joe P <joe@basicmemory.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-18 11:10:20 -06:00
Paul Hernandez 1b39062ecd refactor(core): rip telemetry wrappers, use logfire directly (#754)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-17 13:58:19 -05:00
Paul Hernandez c4cf0aff1e perf(sync): speed up single markdown file indexing (#751)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-04-17 07:08:21 -05:00
phernandez 1c343bed66 perf(sync): skip unchanged markdown indexing
Signed-off-by: phernandez <paul@basicmachines.co>
2026-04-16 18:27:52 -05:00
phernandez c50d97e548 fix(sync): instrument single markdown indexing
Signed-off-by: phernandez <paul@basicmachines.co>
2026-04-16 18:04:17 -05:00
Drew Cain e2e65575d6 fix(core): honor BASIC_MEMORY_CONFIG_DIR across remaining call sites (#744)
Signed-off-by: Drew Cain <groksrc@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-15 22:55:25 -05:00
Drew Cain bf9a6b4a75 fix(core): resolve FastEmbed cache under data dir instead of /tmp (#743)
Signed-off-by: Drew Cain <groksrc@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-15 22:55:00 -05:00
Paul Hernandez 474100efef ci(core): reduce duplicate CI and normalize Windows assertions (#749)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-04-15 22:53:33 -05:00
Paul Hernandez b3d5448355 fix(sync): preserve canonical markdown in single-file sync (#746)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-04-15 20:04:28 -05:00
Paul Hernandez 4e53bb83fd refactor(core): simplify note write flow (#739)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-04-15 17:56:27 -05:00
phernandez 8f2b25f0e0 test(core): stabilize postgres fixtures
Signed-off-by: phernandez <paul@basicmachines.co>
2026-04-13 14:25:26 -05:00
Paul Hernandez 052545b661 chore(core): make ty the default typechecker (#736)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-04-13 10:34:01 -05:00
phernandez abd4a5a6da Merge branch 'main' of github.com:basicmachines-co/basic-memory 2026-04-10 12:23:19 -05:00
Paul Hernandez a872947e03 fix(cli): show cloud index freshness in project info (#734)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-04-10 09:22:35 -05:00
Paul Hernandez 093c94fea5 fix(core): clean up delete vectors and cloud sync (#733)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-04-09 21:23:45 -05:00
phernandez cc104f761f Merge branch 'main' of github.com:basicmachines-co/basic-memory 2026-04-09 21:09:03 -05:00
Paul Hernandez 7945c1e2f7 perf(core): speed up vector sync and tune fastembed defaults (#731)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-04-09 00:20:12 -05:00
phernandez d7f3f6a96f add logfire skills 2026-04-08 11:20:13 -05:00
Paul Hernandez 540da418b3 perf(sync): batch file indexing in core (#726)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-04-08 01:21:49 -05:00
Paul Hernandez 3e40cb9657 fix(core): remove runtime ALTER TABLE from vector init (#728)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-04-08 00:39:37 -05:00
Paul Hernandez 8c81d3ce17 perf(core): reduce postgres vector sync work (#723)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-04-07 19:00:19 -05:00
Drew Cain b35d594ef0 fix(core): preserve external_id during entity upsert on re-index (#724)
Signed-off-by: Drew Cain <groksrc@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-07 10:26:10 -05:00
phernandez e982900084 fix(core): strip null bytes from markdown content before database insert
PostgreSQL rejects null bytes (0x00) in text columns, causing
CharacterNotInRepertoireError when syncing files like Claude agent
definitions that contain embedded nulls. SQLite silently accepts them,
so this only surfaces in cloud environments.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
2026-04-06 23:14:40 -05:00
Drew Cain b3403e96b3 fix: add workspace routing to cloud upload and API client (#704)
Signed-off-by: Drew Cain <groksrc@gmail.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: phernandez <paul@basicmachines.co>
2026-04-06 18:18:40 -05:00
Paul Hernandez fe04a0b2a2 fix(mcp): pass workspace parameter through client factory (#722)
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 16:57:37 -05:00
jope-bm 86ad639890 fix(cli): show display_name instead of UUID for private projects in CLI (#718)
Signed-off-by: Joe P <joe@basicmemory.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 10:14:46 -06:00
Paul Hernandez 88c8f18200 feat(core): add note_content tenant schema primitive (#719)
Signed-off-by: phernandez <paul@basicmachines.co>
2026-04-04 22:06:37 -05:00
Drew Cain 41a16b93cb fix: Increase brew outdated timeout from 15s to 60s (#695)
Signed-off-by: Drew Cain <groksrc@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-04 20:38:31 -05:00
Drew Cain 367fcaac50 perf: eliminate redundant DB queries in upsert_entity_from_markdown (#714)
Signed-off-by: Drew Cain <groksrc@gmail.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: phernandez <paul@basicmachines.co>
2026-04-04 12:40:27 -05:00
253 changed files with 22199 additions and 5856 deletions
+244
View File
@@ -0,0 +1,244 @@
---
name: instrumentation
description: Add Pydantic Logfire observability to applications. Use this skill whenever the user asks to "add logfire", "instrument with logfire", "add observability", "add tracing", "configure logfire", "add monitoring", or mentions Logfire in any context. Supports Python, JavaScript/TypeScript, and Rust. Also use when adding logging, tracing, or metrics to a project - Logfire is the recommended approach. Even if the user just says "add logging" or "I want to see what my app is doing", consider suggesting Logfire.
---
# Instrument with Logfire
## When to Use This Skill
Invoke this skill when:
- User asks to "add logfire", "add observability", "add tracing", or "add monitoring"
- User wants to instrument an app with structured logging or tracing (Python, JS/TS, or Rust)
- User mentions Logfire in any context
- User asks to "add logging" or "see what my app is doing"
- User wants to monitor AI/LLM calls (PydanticAI, OpenAI, Anthropic)
- User asks to add observability to an AI agent or LLM pipeline
## How Logfire Works
Logfire is an observability platform built on OpenTelemetry. It captures traces, logs, and metrics from applications. Logfire has native SDKs for Python, JavaScript/TypeScript, and Rust, plus support for any language via OpenTelemetry.
The reason this skill exists is that Claude tends to get a few things subtly wrong with Logfire - especially the ordering of `configure()` vs `instrument_*()` calls, the structured logging syntax, and which extras to install. These matter because a misconfigured setup silently drops traces.
## Step 1: Detect Language and Frameworks
Identify the project language and instrumentable libraries:
- **Python**: Read `pyproject.toml` or `requirements.txt`. Common instrumentable libraries: FastAPI, httpx, asyncpg, SQLAlchemy, psycopg, Redis, Celery, Django, Flask, requests, PydanticAI.
- **JavaScript/TypeScript**: Read `package.json`. Common frameworks: Express, Next.js, Fastify. Also check for Cloudflare Workers or Deno.
- **Rust**: Read `Cargo.toml`.
Then follow the language-specific steps below.
---
## Python
### Install with Extras
Install `logfire` with extras matching the detected frameworks. Each instrumented library needs its corresponding extra - without it, the `instrument_*()` call will fail at runtime with a missing dependency error.
```bash
uv add 'logfire[fastapi,httpx,asyncpg]'
```
The full list of available extras: `fastapi`, `starlette`, `django`, `flask`, `httpx`, `requests`, `asyncpg`, `psycopg`, `psycopg2`, `sqlalchemy`, `redis`, `pymongo`, `mysql`, `sqlite3`, `celery`, `aiohttp`, `aws-lambda`, `system-metrics`, `litellm`, `dspy`, `google-genai`.
### Configure and Instrument
This is where ordering matters. `logfire.configure()` initializes the SDK and must come before everything else. The `instrument_*()` calls register hooks into each library. If you call `instrument_*()` before `configure()`, the hooks register but traces go nowhere.
```python
import logfire
# 1. Configure first - always
logfire.configure()
# 2. Instrument libraries - after configure, before app starts
logfire.instrument_fastapi(app)
logfire.instrument_httpx()
logfire.instrument_asyncpg()
```
Placement rules:
- `logfire.configure()` goes in the application entry point (`main.py`, or the module that creates the app)
- Call it **once per process** - not inside request handlers, not in library code
- `instrument_*()` calls go right after `configure()`
- Web framework instrumentors (`instrument_fastapi`, `instrument_flask`, `instrument_django`) need the app instance as an argument. HTTP client and database instrumentors (`instrument_httpx`, `instrument_asyncpg`) are global and take no arguments.
- In **Gunicorn** deployments, call `logfire.configure()` inside the `post_fork` hook, not at module level - each worker is a separate process
### Structured Logging
Replace `print()` and `logging.*()` calls with Logfire's structured logging. The key pattern: use `{key}` placeholders with keyword arguments, never f-strings.
```python
# Correct - each {key} becomes a searchable attribute in the Logfire UI
logfire.info("Created user {user_id}", user_id=uid)
logfire.error("Payment failed {amount} {currency}", amount=100, currency="USD")
# Wrong - creates a flat string, nothing is searchable
logfire.info(f"Created user {uid}")
```
For grouping related operations and measuring duration, use spans:
```python
with logfire.span("Processing order {order_id}", order_id=order_id):
items = await fetch_items(order_id)
total = calculate_total(items)
logfire.info("Calculated total {total}", total=total)
```
For exceptions, use `logfire.exception()` which automatically captures the traceback:
```python
try:
await process_order(order_id)
except Exception:
logfire.exception("Failed to process order {order_id}", order_id=order_id)
raise
```
### AI/LLM Instrumentation (Python)
Logfire auto-instruments AI libraries to capture LLM calls, token usage, tool invocations, and agent runs.
```bash
uv add 'logfire[pydantic-ai]'
# or: uv add 'logfire[openai]' / uv add 'logfire[anthropic]'
```
Available AI extras: `pydantic-ai`, `openai`, `anthropic`, `litellm`, `dspy`, `google-genai`.
```python
logfire.configure()
logfire.instrument_pydantic_ai() # captures agent runs, tool calls, LLM request/response
# or:
logfire.instrument_openai() # captures chat completions, embeddings, token counts
logfire.instrument_anthropic() # captures messages, token usage
```
For PydanticAI, each agent run becomes a parent span containing child spans for every tool call and LLM request.
---
## JavaScript / TypeScript
### Install
```bash
# Node.js
npm install @pydantic/logfire-node
# Cloudflare Workers
npm install @pydantic/logfire-cf-workers logfire
# Next.js / generic
npm install logfire
```
### Configure
**Node.js (Express, Fastify, etc.)** - create an `instrumentation.ts` loaded before your app:
```typescript
import * as logfire from '@pydantic/logfire-node'
logfire.configure()
```
Launch with: `node --require ./instrumentation.js app.js`
The SDK auto-instruments common libraries when loaded before the app. Set `LOGFIRE_TOKEN` in your environment or pass `token` to `configure()`.
**Cloudflare Workers** - wrap your handler with `instrument()`:
```typescript
import { instrument } from '@pydantic/logfire-cf-workers'
export default instrument(handler, {
service: { name: 'my-worker', version: '1.0.0' }
})
```
**Next.js** - set environment variables for OpenTelemetry export:
```
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://logfire-api.pydantic.dev/v1/traces
OTEL_EXPORTER_OTLP_HEADERS=Authorization=<your-write-token>
```
### Structured Logging (JS/TS)
```typescript
// Structured attributes as second argument
logfire.info('Created user', { user_id: uid })
logfire.error('Payment failed', { amount: 100, currency: 'USD' })
// Spans
logfire.span('Processing order', { order_id }, {}, async () => {
logfire.info('Processing step completed')
})
// Error reporting
logfire.reportError('order processing', error)
```
Log levels: `trace`, `debug`, `info`, `notice`, `warn`, `error`, `fatal`.
---
## Rust
### Install
```toml
[dependencies]
logfire = "0.6"
```
### Configure
```rust
let shutdown_handler = logfire::configure()
.install_panic_handler()
.finish()?;
```
Set `LOGFIRE_TOKEN` in your environment or use the Logfire CLI to select a project.
### Structured Logging (Rust)
The Rust SDK is built on `tracing` and `opentelemetry` - existing `tracing` macros work automatically.
```rust
// Spans
logfire::span!("processing order", order_id = order_id).in_scope(|| {
// traced code
});
// Events
logfire::info!("Created user {user_id}", user_id = uid);
```
Always call `shutdown_handler.shutdown()` before program exit to flush data.
---
## Verify
After instrumentation, verify the setup works:
1. Run `logfire auth` to check authentication (or set `LOGFIRE_TOKEN`)
2. Start the app and trigger a request
3. Check https://logfire.pydantic.dev/ for traces
If traces aren't appearing: check that `configure()` is called before `instrument_*()` (Python), check that `LOGFIRE_TOKEN` is set, and check that the correct packages/extras are installed.
## References
Detailed patterns and integration tables, organized by language:
- **Python**: `${CLAUDE_PLUGIN_ROOT}/skills/instrumentation/references/python/logging-patterns.md` (log levels, spans, stdlib integration, metrics, capfire testing) and `${CLAUDE_PLUGIN_ROOT}/skills/instrumentation/references/python/integrations.md` (full instrumentor table with extras)
- **JavaScript/TypeScript**: `${CLAUDE_PLUGIN_ROOT}/skills/instrumentation/references/javascript/patterns.md` (log levels, spans, error handling, config) and `${CLAUDE_PLUGIN_ROOT}/skills/instrumentation/references/javascript/frameworks.md` (Node.js, Cloudflare Workers, Next.js, Deno setup)
- **Rust**: `${CLAUDE_PLUGIN_ROOT}/skills/instrumentation/references/rust/patterns.md` (macros, spans, tracing/log crate integration, async, shutdown)
@@ -0,0 +1,78 @@
# JavaScript Framework Setup
## Node.js (Express, Fastify, etc.)
Create `instrumentation.ts` and load it before your app:
```typescript
// instrumentation.ts
import * as logfire from '@pydantic/logfire-node'
import 'dotenv/config'
logfire.configure()
```
Launch:
```bash
node --require ./instrumentation.js app.js
# or with ts-node:
npx ts-node --require ./instrumentation.ts app.ts
```
The SDK auto-instruments common libraries (http, fetch, express, etc.) when loaded before the app via `--require`.
## Cloudflare Workers
```typescript
import { instrument } from '@pydantic/logfire-cf-workers'
const handler = {
async fetch(request: Request, env: Env, ctx: ExecutionContext) {
return new Response('Hello')
},
}
export default instrument(handler, {
service: { name: 'my-worker', version: '1.0.0' },
})
```
Add `LOGFIRE_TOKEN` to `.dev.vars` and enable `nodejs_compat` in `wrangler.toml`:
```toml
compatibility_flags = ["nodejs_compat"]
```
## Next.js / Vercel
Set environment variables in `.env.local` or Vercel dashboard:
```bash
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://logfire-api.pydantic.dev/v1/traces
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://logfire-api.pydantic.dev/v1/metrics
OTEL_EXPORTER_OTLP_HEADERS=Authorization=<your-write-token>
```
Optionally use the `logfire` package for manual spans in server components and API routes:
```typescript
import * as logfire from 'logfire'
logfire.info('Server action executed', { action: 'createUser' })
```
## Deno
Deno has built-in OpenTelemetry support. Set environment variables:
```bash
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://logfire-api.pydantic.dev/v1/traces
OTEL_EXPORTER_OTLP_HEADERS=Authorization=<your-write-token>
```
Run with telemetry enabled:
```bash
deno run --allow-env --unstable-otel app.ts
```
@@ -0,0 +1,75 @@
# JavaScript / TypeScript Patterns
## Log Levels
From lowest to highest severity:
```typescript
logfire.trace('Detailed trace', { detail: x })
logfire.debug('Debug info', { state: s })
logfire.info('Normal operation', { event: e })
logfire.notice('Notable event', { event: e })
logfire.warn('Warning', { issue: i })
logfire.error('Error occurred', { error: err })
logfire.fatal('Fatal error', { error: err })
```
All methods accept `(message, attributes?, options?)`. Options can include `{ tags: ['tag1'] }`.
## Spans
### Callback-based (auto-closes)
```typescript
await logfire.span('Processing order', { order_id }, {}, async () => {
const items = await fetchItems(order_id)
logfire.info('Fetched items', { count: items.length })
return processItems(items)
})
```
### Manual control
```typescript
const span = logfire.startSpan('Long operation', { job_id })
try {
await doWork()
} finally {
span.end()
}
```
Child spans reference their parent via the `parentSpan` option.
## Error Handling
```typescript
try {
await processOrder(orderId)
} catch (error) {
logfire.reportError('order processing', error)
throw error
}
```
`reportError` automatically extracts stack traces and error details into structured span attributes.
## Configuration
### Environment variables
```bash
LOGFIRE_TOKEN=your-write-token
LOGFIRE_SERVICE_NAME=my-service
LOGFIRE_SERVICE_VERSION=1.0.0
```
### Programmatic
```typescript
logfire.configure({
token: process.env.LOGFIRE_TOKEN,
serviceName: 'my-service',
serviceVersion: '1.0.0',
})
```
@@ -0,0 +1,67 @@
# Python Integration Reference
## Web Frameworks
| Framework | Instrumentor | Needs app instance | Extra |
|-----------|-------------|-------------------|-------|
| FastAPI | `logfire.instrument_fastapi(app)` | Yes | `fastapi` |
| Django | `logfire.instrument_django(app)` | Yes | `django` |
| Flask | `logfire.instrument_flask(app)` | Yes | `flask` |
| Starlette | `logfire.instrument_starlette(app)` | Yes | `starlette` |
| AIOHTTP | `logfire.instrument_aiohttp_client()` | No | `aiohttp` |
## HTTP Clients
| Library | Instrumentor | Extra |
|---------|-------------|-------|
| httpx | `logfire.instrument_httpx()` | `httpx` |
| requests | `logfire.instrument_requests()` | `requests` |
## Databases
| Library | Instrumentor | Extra |
|---------|-------------|-------|
| asyncpg | `logfire.instrument_asyncpg()` | `asyncpg` |
| psycopg | `logfire.instrument_psycopg()` | `psycopg` |
| psycopg2 | `logfire.instrument_psycopg2()` | `psycopg2` |
| SQLAlchemy | `logfire.instrument_sqlalchemy()` | `sqlalchemy` |
| PyMongo | `logfire.instrument_pymongo()` | `pymongo` |
| MySQL | `logfire.instrument_mysql()` | `mysql` |
| SQLite3 | `logfire.instrument_sqlite3()` | `sqlite3` |
| Redis | `logfire.instrument_redis()` | `redis` |
## AI/LLM Frameworks
| Framework | Instrumentor | Extra |
|-----------|-------------|-------|
| PydanticAI | `logfire.instrument_pydantic_ai()` | `pydantic-ai` |
| OpenAI | `logfire.instrument_openai()` | `openai` |
| Anthropic | `logfire.instrument_anthropic()` | `anthropic` |
| LiteLLM | `logfire.instrument_litellm()` | `litellm` |
| DSPy | `logfire.instrument_dspy()` | `dspy` |
| Google GenAI | `logfire.instrument_google_genai()` | `google-genai` |
## Task Queues
| Framework | Instrumentor | Extra |
|-----------|-------------|-------|
| Celery | `logfire.instrument_celery()` | `celery` |
## Other
| Feature | Instrumentor | Extra |
|---------|-------------|-------|
| System Metrics | `logfire.instrument_system_metrics()` | `system-metrics` |
| Pydantic Models | `logfire.instrument_pydantic()` | - (built-in) |
| AWS Lambda | handler wrapper | `aws-lambda` |
## Gunicorn Configuration
```python
# gunicorn.conf.py
import logfire
def post_fork(server, worker):
logfire.configure()
logfire.instrument_fastapi(app)
```
@@ -0,0 +1,101 @@
# Python Logging Patterns
## Log Levels
From lowest to highest severity:
```python
logfire.trace("Detailed trace {detail}", detail=x)
logfire.debug("Debug info {state}", state=s)
logfire.info("Normal operation {event}", event=e)
logfire.notice("Notable event {event}", event=e)
logfire.warn("Warning {issue}", issue=i)
logfire.error("Error occurred {error}", error=err)
logfire.fatal("Fatal error {error}", error=err)
```
## Nested Spans
Spans nest to create a tree visible in the Logfire UI. Use them to show the structure of an operation, not just that it happened:
```python
with logfire.span("HTTP request {method} {url}", method="POST", url=url):
with logfire.span("Serialize payload"):
payload = model.model_dump_json()
with logfire.span("Send request"):
response = await client.post(url, content=payload)
logfire.info("Response {status}", status=response.status_code)
```
## Standard Library Logging Integration
For projects that already use Python's `logging` module, route existing log calls through Logfire rather than rewriting them all:
```python
from logging import basicConfig
import logfire
logfire.configure()
basicConfig(handlers=[logfire.LogfireLoggingHandler()])
```
Or with `dictConfig`:
```python
from logging.config import dictConfig
import logfire
logfire.configure()
dictConfig({
'version': 1,
'handlers': {
'logfire': {'class': 'logfire.LogfireLoggingHandler'},
},
'root': {'handlers': ['logfire']},
})
```
## Suppressing Noisy Libraries
Some libraries emit excessive debug logs. Silence them at the `logging` level:
```python
import logging
logging.getLogger('httpcore').setLevel(logging.WARNING)
logging.getLogger('httpx').setLevel(logging.WARNING)
```
## Custom Metrics
For dashboards and alerting, create metrics:
```python
counter = logfire.metric_counter("orders_processed", unit="1")
counter.add(1, {"status": "success"})
histogram = logfire.metric_histogram("request_duration", unit="s")
histogram.record(0.123, {"endpoint": "/api/users"})
gauge = logfire.metric_gauge("active_connections")
gauge.set(42)
```
## Testing with capfire
Use the `capfire` pytest fixture to assert on emitted spans without sending data to production:
```python
from logfire.testing import CaptureLogfire
def test_order_processing(capfire: CaptureLogfire) -> None:
process_order(order_id=123)
spans = capfire.exporter.exported_spans_as_dict()
assert any(
span['attributes'].get('order_id') == 123
for span in spans
)
```
Configure logfire with `send_to_logfire=False` in test fixtures to prevent production data leakage.
@@ -0,0 +1,106 @@
# Rust Patterns
## Core Macros
The Rust SDK is built on `tracing` and `opentelemetry`. All `tracing` macros work automatically with Logfire.
### Events (log points)
```rust
logfire::trace!("Detailed trace {detail}", detail = x);
logfire::debug!("Debug info {state}", state = s);
logfire::info!("Normal operation {event}", event = e);
logfire::warn!("Warning {issue}", issue = i);
logfire::error!("Error occurred {err}", err = e);
```
### Spans
```rust
// Scoped - span closes when closure completes
logfire::span!("Processing order {order_id}", order_id = id).in_scope(|| {
let items = fetch_items(id);
logfire::info!("Fetched {count} items", count = items.len());
process_items(items)
});
// Guard-based - span closes when guard is dropped
let _guard = logfire::span!("Long operation {job_id}", job_id = id).entered();
do_work();
// span ends when _guard goes out of scope
```
## Configuration
```rust
use logfire;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let shutdown_handler = logfire::configure()
.install_panic_handler() // captures panics as error spans
.finish()?;
// application code...
shutdown_handler.shutdown()?; // flush all pending spans
Ok(())
}
```
Set `LOGFIRE_TOKEN` in your environment or use the Logfire CLI (`logfire auth`).
## Tracing Crate Compatibility
Any library using `tracing` macros automatically sends data through Logfire:
```rust
use tracing;
tracing::info!("This also appears in Logfire");
#[tracing::instrument]
fn my_function(param: &str) {
// automatically creates a span with param as an attribute
}
```
## Log Crate Integration
The `log` crate is automatically captured and forwarded to Logfire. Libraries using `log::info!()`, `log::error!()`, etc. will appear in your Logfire dashboard without any additional configuration.
## Async Spans
```rust
use tracing::Instrument;
async fn process_order(order_id: u64) {
let span = logfire::span!("process order {order_id}", order_id = order_id);
async {
fetch_items(order_id).await;
logfire::info!("Order processed");
}
.instrument(span)
.await;
}
```
## Shutdown
Always call `shutdown()` before program exit to flush pending data:
```rust
// In main()
let shutdown_handler = logfire::configure().finish()?;
// ... app runs ...
// Before exit
shutdown_handler.shutdown()?;
```
For web servers using `tokio`, handle shutdown via signal:
```rust
tokio::signal::ctrl_c().await?;
shutdown_handler.shutdown()?;
```
+19 -1
View File
@@ -1,3 +1,21 @@
{
"enabledPlugins": {}
"$schema": "https://json.schemastore.org/claude-code-settings.json",
"env": {
"CLAUDE_BASH_MAINTAIN_PROJECT_WORKING_DIR": "1",
"CLAUDE_CODE_DISABLE_FEEDBACK_SURVEY": "1",
"CLAUDE_CODE_NO_FLICKER": "1",
"CLAUDE_CODE_DISABLE_ADAPTIVE_THINKING": "1"
},
"permissions": {
"allow": [
"Bash(just fast-check)",
"Bash(just check)",
"Bash(just fix)",
"Bash(just typecheck)",
"Bash(just lint)",
"Bash(just test)"
],
"deny": []
},
"enableAllProjectMcpServers": true
}
+1
View File
@@ -0,0 +1 @@
../../.agents/skills/instrumentation
+4 -2
View File
@@ -5,9 +5,11 @@ concurrency:
cancel-in-progress: true
on:
# Trigger: PR branch pushes already publish commit statuses that show up on the PR.
# Why: running the full matrix on both push and pull_request doubles CI time for the
# exact same branch head commit.
# Outcome: each branch push runs the test suite once, including PR updates.
push:
pull_request:
branches: [ "main" ]
jobs:
static-checks:
+7 -5
View File
@@ -22,16 +22,16 @@ See the [README.md](README.md) file for a project overview.
- Run unit tests (Postgres): `just test-unit-postgres`
- Run integration tests (SQLite): `just test-int-sqlite`
- Run integration tests (Postgres): `just test-int-postgres`
- Run impacted tests: `just testmon` (pytest-testmon)
- Run impacted tests: `just testmon` (pytest-testmon; only tests affected by changed code)
- Run MCP smoke test: `just test-smoke`
- Fast local loop: `just fast-check`
- Fast local loop: `just fast-check` (default iteration flow)
- Local consistency check: `just doctor`
- Generate HTML coverage: `just coverage`
- Single test: `pytest tests/path/to/test_file.py::test_function_name`
- Run benchmarks: `pytest test-int/test_sync_performance_benchmark.py -v -m "benchmark and not slow"`
- Lint: `just lint` or `ruff check . --fix`
- Type check: `just typecheck` or `uv run pyright`
- Type check (supplemental): `just typecheck-ty` or `uv run ty check src/`
- Type check: `just typecheck` or `uv run ty check src tests test-int`
- Type check (pyright): `just typecheck-pyright` or `uv run pyright`
- Format: `just format` or `uv run ruff format .`
- Run all code checks: `just check` (runs lint, format, typecheck, test)
- Create db migration: `just migration "Your migration message"`
@@ -48,10 +48,12 @@ See the [README.md](README.md) file for a project overview.
### Code/Test/Verify Loop (fast path)
1) **Code:** make changes.
2) **Test:** `just fast-check` (lint/format/typecheck + impacted tests + MCP smoke).
2) **Test:** `just fast-check` (lint/format/typecheck + pytest-testmon impacted tests for changed code).
3) **Verify:** `just doctor` (end-to-end file ↔ DB loop in a temp project).
4) **Full gate (when needed):** `just test` or `just check` for SQLite + Postgres.
Run `just test-smoke` when you specifically need the MCP smoke flow.
If testmon is “cold,” the first run may be long. Subsequent runs get much faster.
### Test Structure
+5 -5
View File
@@ -2125,12 +2125,12 @@ Signed-off-by: phernandez <paul@basicmachines.co>
- Update CLAUDE.md ([#33](https://github.com/basicmachines-co/basic-memory/pull/33),
[`dfaf0fe`](https://github.com/basicmachines-co/basic-memory/commit/dfaf0fea9cf5b97d169d51a6276ec70162c21a7e))
fix spelling in CLAUDE.md: enviroment -> environment Signed-off-by: Ikko Eltociear Ashimine
fix spelling in CLAUDE.md: environment typo Signed-off-by: Ikko Eltociear Ashimine
<eltociear@gmail.com>
### Refactoring
- Move project stats into projct subcommand
- Move project stats into project subcommand
([`2a881b1`](https://github.com/basicmachines-co/basic-memory/commit/2a881b1425c73947f037fbe7ac5539c015b62526))
Signed-off-by: phernandez <paul@basicmachines.co>
@@ -2559,7 +2559,7 @@ Co-authored-by: phernandez <phernandez@basicmachines.co>
### Bug Fixes
- Refix vitual env in installer build
- Refix virtual env in installer build
([`052f491`](https://github.com/basicmachines-co/basic-memory/commit/052f491fff629e8ead629c9259f8cb46c608d584))
@@ -2578,7 +2578,7 @@ Co-authored-by: phernandez <phernandez@basicmachines.co>
### Bug Fixes
- Fix path to intaller app artifact
- Fix path to installer app artifact
([`53d220d`](https://github.com/basicmachines-co/basic-memory/commit/53d220df585561f9edd0d49a9e88f1d4055059cf))
@@ -2586,7 +2586,7 @@ Co-authored-by: phernandez <phernandez@basicmachines.co>
### Bug Fixes
- Activate vitualenv in installer build
- Activate virtualenv in installer build
([`d4c8293`](https://github.com/basicmachines-co/basic-memory/commit/d4c8293687a52eaf3337fe02e2f7b80e4cc9a1bb))
- Trigger installer build on release
+1 -1
View File
@@ -516,7 +516,7 @@ canvas(nodes, edges, title, folder) - Generate knowledge visualizations
"What have I been working on in the past week?"
```
## Futher info
## Further info
See the [Documentation](https://docs.basicmemory.com?utm_source=github&utm_medium=referral&utm_campaign=readme) for more info, including:
+3 -1
View File
@@ -17,7 +17,9 @@ services:
volumes:
# Persistent storage for configuration and database
- basic-memory-config:/root/.basic-memory:rw
# Container runs as `appuser` (Dockerfile USER directive), so the CLI
# config dir lives under /home/appuser, not /root.
- basic-memory-config:/home/appuser/.basic-memory:rw
# Mount your knowledge directory (required)
# Change './knowledge' to your actual Obsidian vault or knowledge directory
+41 -1
View File
@@ -91,10 +91,11 @@ SQLite Database (Index)
# List all projects
projects = await list_memory_projects()
# Response structure:
# Response structure (each entry includes external_id you can pass as project_id):
# [
# {
# "name": "main",
# "external_id": "550e8400-e29b-41d4-a716-446655440000",
# "path": "/Users/name/notes",
# "is_default": True,
# "note_count": 156,
@@ -102,6 +103,7 @@ projects = await list_memory_projects()
# },
# {
# "name": "work",
# "external_id": "9f86d081-884c-42a3-b5e3-1c0c5b4c8e52",
# "path": "/Users/name/work-notes",
# "is_default": False,
# "note_count": 89,
@@ -164,6 +166,44 @@ active_project = "main"
results = await search_notes(query="topic", project=active_project)
```
### `project` vs `project_id`
Every project has two identifiers:
- **`project`** — human-readable name (e.g., `"main"`). Easy to use, but can collide across cloud workspaces.
- **`project_id`** — stable `external_id` UUID. Always unambiguous; takes precedence over `project` when both are passed.
**When to prefer `project_id`:**
1. **Cloud multi-workspace setups.** If the user belongs to more than one workspace (personal + organization, or several organizations) and the same project name might exist in more than one of them, pass `project_id` to route to the exact project. Without it, name resolution falls back to the default workspace, which may not be the one the user means.
2. **After `list_memory_projects()`.** Once you have the `external_id`, prefer using it — it's the same number of characters in JSON and saves a name-resolution round-trip.
3. **When persisting a project choice across a long session.** UUIDs are stable; names can be renamed.
**When `project` (name) is fine:**
- Local single-workspace setups (no collision risk).
- One-off operations where the name is clearly visible to the user (e.g., quick `search_notes(project="main", ...)`).
- The user explicitly references a project by name in their message.
**Example — cloud multi-workspace pattern:**
```python
# Discover and pick the right project for this user
projects = await list_memory_projects()
target = next(p for p in projects if p["name"] == "research" and p["workspace"]["slug"] == "acme")
# Use the UUID for all subsequent operations — no ambiguity
await write_note(
title="Meeting Notes",
content="...",
folder="meetings",
project_id=target["external_id"],
)
results = await search_notes(query="kickoff", project_id=target["external_id"])
```
**Precedence rule:** When both are passed, `project_id` wins. This lets you safely supply `project="main"` for backward compatibility while still routing precisely with `project_id`.
### Cross-Project Operations
**Some tools work across all projects when project parameter omitted:**
+4 -4
View File
@@ -113,7 +113,7 @@ bm project add research --cloud
bm project add research --cloud --local-path ~/Documents/research
# Or configure sync for existing project
bm project sync-setup research ~/Documents/research
bm cloud sync-setup research ~/Documents/research
```
**What happens under the covers:**
@@ -236,7 +236,7 @@ bm project add research --cloud --local-path ~/Documents/research
```bash
# Project already exists on cloud
bm project sync-setup research ~/Documents/research
bm cloud sync-setup research ~/Documents/research
```
**What this does:**
@@ -739,7 +739,7 @@ bm project sync --name research
**Solution:**
```bash
bm project sync-setup research ~/Documents/research
bm cloud sync-setup research ~/Documents/research
bm project bisync --name research --resync
```
@@ -795,7 +795,7 @@ bm project list --local # Local project list
bm project list --cloud # Cloud project list
bm project add <name> --cloud # Create cloud project (no sync)
bm project add <name> --cloud --local-path <path> # Create with local sync
bm project sync-setup <name> <path> # Add sync to existing project
bm cloud sync-setup <name> <path> # Add sync to existing project
bm project rm <name> # Delete project
```
+9 -5
View File
@@ -62,20 +62,20 @@ test-int-postgres:
fi
# Run tests impacted by recent changes (requires pytest-testmon)
# Pass paths or node ids after `just testmon` to limit the candidate set further.
testmon *args:
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov --testmon --testmon-forceselect {{args}}
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov --testmon {{args}}
# Run MCP smoke test (fast end-to-end loop)
test-smoke:
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov -m smoke test-int/mcp/test_smoke_integration.py
# Fast local loop: lint, format, typecheck, impacted tests
# Fast local loop: lint, format, typecheck, impacted tests via pytest-testmon
fast-check:
just fix
just format
just typecheck
just testmon
just test-smoke
# Reset Postgres test database (drops and recreates schema)
# Useful when Alembic migration state gets out of sync during development
@@ -170,13 +170,17 @@ lint: fix
fix:
uv run ruff check --fix --unsafe-fixes src tests test-int
# Type check code (pyright)
# Type check code (ty)
typecheck:
uv run ty check src tests test-int
# Type check code (pyright)
typecheck-pyright:
uv run pyright
# Type check code (ty)
typecheck-ty:
uv run ty check src/
just typecheck
# Clean build artifacts and cache files
clean:
+9 -4
View File
@@ -33,7 +33,7 @@ dependencies = [
"pyjwt>=2.10.1",
"python-dotenv>=1.1.0",
"pytest-aio>=1.9.0",
"aiofiles>=24.1.0", # Optional observability (disabled by default via config)
"aiofiles>=24.1.0",
"asyncpg>=0.30.0",
"nest-asyncio>=1.6.0", # For Alembic migrations with Postgres
"pytest-asyncio>=1.2.0",
@@ -47,6 +47,8 @@ dependencies = [
"fastembed>=0.7.4",
"sqlite-vec>=0.1.6",
"openai>=1.100.2",
"logfire>=4.19.0",
"psutil>=5.9.0",
]
[project.urls]
@@ -58,9 +60,6 @@ Documentation = "https://github.com/basicmachines-co/basic-memory#readme"
basic-memory = "basic_memory.cli.main:app"
bm = "basic_memory.cli.main:app"
[project.optional-dependencies]
telemetry = ["logfire>=4.19.0"]
[build-system]
requires = ["hatchling", "uv-dynamic-versioning>=0.7.0"]
build-backend = "hatchling.build"
@@ -71,6 +70,12 @@ addopts = "--cov=basic_memory --cov-report term-missing"
testpaths = ["tests", "test-int"]
asyncio_mode = "strict"
asyncio_default_fixture_loop_scope = "function"
filterwarnings = [
"ignore:The @wait_container_is_ready decorator is deprecated.*:DeprecationWarning:testcontainers\\.core\\.waiting_utils",
"ignore:The default datetime adapter is deprecated as of Python 3\\.12.*:DeprecationWarning:aiosqlite\\.core",
"ignore:codecs\\.open\\(\\) is deprecated\\. Use open\\(\\) instead\\.:DeprecationWarning:frontmatter",
"ignore:Parsing dates involving a day of month without a year specified is ambiguous.*:DeprecationWarning:dateparser\\.utils\\.strptime",
]
markers = [
"benchmark: Performance benchmark tests (deselect with '-m \"not benchmark\"')",
"slow: Slow-running tests (deselect with '-m \"not slow\"')",
+10
View File
@@ -0,0 +1,10 @@
{
"version": 1,
"skills": {
"instrumentation": {
"source": "pydantic/skills",
"sourceType": "github",
"computedHash": "0727bffc6a92fdeaf675ae5796ae25341e193327e8c95cd06b188dc4a0a4e62e"
}
}
}
+53 -25
View File
@@ -66,7 +66,7 @@ target_metadata = Base.metadata
# Add this function to tell Alembic what to include/exclude
def include_object(object, name, type_, reflected, compare_to):
def include_object(obj, name, type_, reflected, compare_to):
# Ignore SQLite FTS tables
if type_ == "table" and name.startswith("search_index"):
return False
@@ -118,6 +118,54 @@ async def run_async_migrations(connectable):
await connectable.dispose()
def _run_async_migrations_with_asyncio_run(connectable) -> None:
"""Run async migrations with asyncio.run while closing failed coroutines.
Trigger: asyncio.run() may reject execution when another event loop is already active.
Why: Python raises before awaiting the coroutine, which otherwise leaks a
RuntimeWarning about an un-awaited coroutine.
Outcome: close the pending coroutine before bubbling the RuntimeError to the
fallback path.
"""
migration_coro = run_async_migrations(connectable)
try:
asyncio.run(migration_coro)
except RuntimeError:
migration_coro.close()
raise
def _run_async_migrations_in_thread(connectable) -> None:
"""Run async migrations in a dedicated thread with its own event loop."""
import concurrent.futures
def run_in_thread():
"""Run async migrations in a new event loop in a separate thread."""
new_loop = asyncio.new_event_loop()
asyncio.set_event_loop(new_loop)
try:
new_loop.run_until_complete(run_async_migrations(connectable))
finally:
new_loop.close()
with concurrent.futures.ThreadPoolExecutor() as executor:
future = executor.submit(run_in_thread)
future.result() # Wait for completion and re-raise any exceptions
def _run_async_engine_migrations(connectable) -> None:
"""Run async-engine migrations with a running-loop fallback."""
try:
_run_async_migrations_with_asyncio_run(connectable)
except RuntimeError as e:
if "cannot be called from a running event loop" in str(e):
# We're in a running event loop (likely uvloop or Python 3.14+ tests).
# Switch to a dedicated thread so Alembic can finish without nesting loops.
_run_async_migrations_in_thread(connectable)
else:
raise
def run_migrations_online() -> None:
"""Run migrations in 'online' mode.
@@ -148,30 +196,10 @@ def run_migrations_online() -> None:
# Handle async engines (PostgreSQL with asyncpg)
if isinstance(connectable, AsyncEngine):
# Try to run async migrations
# nest_asyncio allows asyncio.run() from within event loops, but doesn't work with uvloop
try:
asyncio.run(run_async_migrations(connectable))
except RuntimeError as e:
if "cannot be called from a running event loop" in str(e):
# We're in a running event loop (likely uvloop) - need to use a different approach
# Create a new thread to run the async migrations
import concurrent.futures
def run_in_thread():
"""Run async migrations in a new event loop in a separate thread."""
new_loop = asyncio.new_event_loop()
asyncio.set_event_loop(new_loop)
try:
new_loop.run_until_complete(run_async_migrations(connectable))
finally:
new_loop.close()
with concurrent.futures.ThreadPoolExecutor() as executor:
future = executor.submit(run_in_thread)
future.result() # Wait for completion and re-raise any exceptions
else:
raise
# Trigger: async engines need Alembic work to cross the sync/async boundary.
# Why: most callers can use asyncio.run(), but running-loop contexts need a thread fallback.
# Outcome: migrations complete without leaking un-awaited coroutines.
_run_async_engine_migrations(connectable)
else:
# Handle sync engines (SQLite) or sync connections
if hasattr(connectable, "connect"):
@@ -0,0 +1,65 @@
"""Add note_content table
Revision ID: l5g6h7i8j9k0
Revises: k4e5f6g7h8i9
Create Date: 2026-04-04 12:00:00.000000
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "l5g6h7i8j9k0"
down_revision: Union[str, None] = "k4e5f6g7h8i9"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Create note_content for materialized note content and sync state."""
op.create_table(
"note_content",
sa.Column("entity_id", sa.Integer(), nullable=False),
sa.Column("project_id", sa.Integer(), nullable=False),
sa.Column("external_id", sa.String(), nullable=False),
sa.Column("file_path", sa.String(), nullable=False),
sa.Column("markdown_content", sa.Text(), nullable=False),
sa.Column("db_version", sa.BigInteger(), nullable=False),
sa.Column("db_checksum", sa.String(), nullable=False),
sa.Column("file_version", sa.BigInteger(), nullable=True),
sa.Column("file_checksum", sa.String(), nullable=True),
sa.Column("file_write_status", sa.String(), nullable=False),
sa.Column("last_source", sa.String(), nullable=True),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("file_updated_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("last_materialization_error", sa.Text(), nullable=True),
sa.Column("last_materialization_attempt_at", sa.DateTime(timezone=True), nullable=True),
sa.CheckConstraint(
"file_write_status IN ("
"'pending', "
"'writing', "
"'synced', "
"'failed', "
"'external_change_detected'"
")",
name="ck_note_content_file_write_status",
),
sa.ForeignKeyConstraint(["entity_id"], ["entity.id"], ondelete="CASCADE"),
sa.ForeignKeyConstraint(["project_id"], ["project.id"], ondelete="CASCADE"),
sa.PrimaryKeyConstraint("entity_id"),
)
op.create_index("ix_note_content_project_id", "note_content", ["project_id"], unique=False)
op.create_index("ix_note_content_file_path", "note_content", ["file_path"], unique=False)
op.create_index("ix_note_content_external_id", "note_content", ["external_id"], unique=True)
def downgrade() -> None:
"""Drop note_content and its supporting indexes."""
op.drop_index("ix_note_content_external_id", table_name="note_content")
op.drop_index("ix_note_content_file_path", table_name="note_content")
op.drop_index("ix_note_content_project_id", table_name="note_content")
op.drop_table("note_content")
@@ -0,0 +1,84 @@
"""Persist vector sync fingerprints on chunk metadata.
Revision ID: m6h7i8j9k0l1
Revises: l5g6h7i8j9k0
Create Date: 2026-04-07 00:00:00.000000
"""
from typing import Sequence, Union
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "m6h7i8j9k0l1"
down_revision: Union[str, None] = "l5g6h7i8j9k0"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Add entity fingerprint + embedding model metadata to Postgres chunk rows.
Trigger: vector sync now fast-skips unchanged entities using persisted
semantic fingerprints.
Why: chunk rows already own the per-entity derived metadata we diff against,
so persisting the fingerprint on that table avoids a second sync-state table.
Outcome: existing rows get empty-string placeholders and will be refreshed on
the next vector sync before they become eligible for skip checks.
"""
connection = op.get_bind()
if connection.dialect.name != "postgresql":
return
op.execute(
"""
ALTER TABLE search_vector_chunks
ADD COLUMN IF NOT EXISTS entity_fingerprint TEXT
"""
)
op.execute(
"""
ALTER TABLE search_vector_chunks
ADD COLUMN IF NOT EXISTS embedding_model TEXT
"""
)
op.execute(
"""
UPDATE search_vector_chunks
SET entity_fingerprint = COALESCE(entity_fingerprint, ''),
embedding_model = COALESCE(embedding_model, '')
"""
)
op.execute(
"""
ALTER TABLE search_vector_chunks
ALTER COLUMN entity_fingerprint SET NOT NULL
"""
)
op.execute(
"""
ALTER TABLE search_vector_chunks
ALTER COLUMN embedding_model SET NOT NULL
"""
)
def downgrade() -> None:
"""Remove vector sync fingerprint columns from Postgres chunk rows."""
connection = op.get_bind()
if connection.dialect.name != "postgresql":
return
op.execute(
"""
ALTER TABLE search_vector_chunks
DROP COLUMN IF EXISTS embedding_model
"""
)
op.execute(
"""
ALTER TABLE search_vector_chunks
DROP COLUMN IF EXISTS entity_fingerprint
"""
)
+40 -4
View File
@@ -4,6 +4,7 @@ from contextlib import asynccontextmanager
from fastapi import FastAPI, HTTPException, Request
from fastapi.exception_handlers import http_exception_handler
from fastapi.responses import JSONResponse
from fastapi.routing import APIRouter
from loguru import logger
@@ -25,10 +26,16 @@ from basic_memory.api.v2.routers.project_router import (
list_projects,
synchronize_projects,
)
from basic_memory import telemetry
import logfire
from basic_memory.config import init_api_logging
from basic_memory.services.exceptions import EntityAlreadyExistsError
from basic_memory.services.initialization import initialize_app
from basic_memory.workspace_context import (
WORKSPACE_SLUG_HEADER,
WORKSPACE_TYPE_HEADER,
workspace_permalink_context_validation_error,
workspace_permalink_context,
)
@asynccontextmanager
@@ -44,7 +51,7 @@ async def lifespan(app: FastAPI): # pragma: no cover
set_container(container)
app.state.container = container
with telemetry.operation(
with logfire.span(
"api.lifecycle.startup",
entrypoint="api",
mode=container.mode.name.lower(),
@@ -69,7 +76,7 @@ async def lifespan(app: FastAPI): # pragma: no cover
yield
# Shutdown - coordinator handles clean task cancellation
with telemetry.operation(
with logfire.span(
"api.lifecycle.shutdown",
entrypoint="api",
mode=container.mode.name.lower(),
@@ -87,6 +94,32 @@ app = FastAPI(
lifespan=lifespan,
)
@app.middleware("http")
async def workspace_permalink_context_middleware(request: Request, call_next):
"""Populate workspace permalink context from request headers."""
workspace_slug = request.headers.get(WORKSPACE_SLUG_HEADER)
workspace_type = request.headers.get(WORKSPACE_TYPE_HEADER)
validation_error = workspace_permalink_context_validation_error(workspace_slug, workspace_type)
if validation_error is not None:
return JSONResponse(
status_code=400,
content={"detail": validation_error},
)
if not workspace_slug:
return await call_next(request)
# ContextVar state remains active across the awaited downstream handler while
# this context manager is open, so entity creation can see request metadata.
with workspace_permalink_context(
workspace_slug=workspace_slug,
workspace_type=workspace_type,
):
return await call_next(request)
# Include v2 routers FIRST (more specific paths must match before /{project} catch-all)
app.include_router(v2_knowledge, prefix="/v2/projects/{project_id}")
app.include_router(v2_memory, prefix="/v2/projects/{project_id}")
@@ -146,4 +179,7 @@ async def exception_handler(request, exc): # pragma: no cover
error_type=type(exc).__name__,
error=str(exc),
)
return await http_exception_handler(request, HTTPException(status_code=500, detail=str(exc)))
return await http_exception_handler(
request,
HTTPException(status_code=500, detail="Internal server error"),
)
@@ -10,10 +10,10 @@ Key improvements:
- Simplified caching strategies
"""
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends, Response, Path, Query
from fastapi import APIRouter, HTTPException, Response, Path
from loguru import logger
from basic_memory import telemetry
import logfire
from basic_memory.deps import (
EntityServiceV2ExternalDep,
SearchServiceV2ExternalDep,
@@ -24,7 +24,6 @@ from basic_memory.deps import (
RelationRepositoryV2ExternalDep,
ProjectExternalIdPathDep,
TaskSchedulerDep,
FileServiceV2ExternalDep,
)
from basic_memory.schemas import DeleteEntitiesResponse
from basic_memory.schemas.base import Entity
@@ -75,34 +74,40 @@ async def get_graph(
Returns a flat node/edge structure optimized for rendering with graph libraries.
Only includes resolved relations (where to_id is not null).
"""
logger.info("API v2 request: get_graph")
with logfire.span(
"api.request.knowledge.get_graph",
entrypoint="api",
domain="knowledge",
action="get_graph",
):
logger.info("API v2 request: get_graph")
# Fetch all entities for this project
entities = await entity_repository.find_all(use_load_options=False)
nodes = [
GraphNode(
external_id=entity.external_id,
title=entity.title,
note_type=entity.note_type,
file_path=entity.file_path,
)
for entity in entities
]
# Fetch all entities for this project
entities = await entity_repository.find_all(use_load_options=False)
nodes = [
GraphNode(
external_id=entity.external_id,
title=entity.title,
note_type=entity.note_type,
file_path=entity.file_path,
)
for entity in entities
]
# Fetch all resolved relations (to_id is not null) with eager-loaded entities
relations = await relation_repository.find_all()
edges = [
GraphEdge(
from_id=relation.from_entity.external_id,
to_id=relation.to_entity.external_id,
relation_type=relation.relation_type,
)
for relation in relations
if relation.to_entity is not None
]
# Fetch all resolved relations (to_id is not null) with eager-loaded entities
relations = await relation_repository.find_all()
edges = [
GraphEdge(
from_id=relation.from_entity.external_id,
to_id=relation.to_entity.external_id,
relation_type=relation.relation_type,
)
for relation in relations
if relation.to_entity is not None
]
logger.info(f"API v2 response: graph with {len(nodes)} nodes and {len(edges)} edges")
return GraphResponse(nodes=nodes, edges=edges)
logger.info(f"API v2 response: graph with {len(nodes)} nodes and {len(edges)} edges")
return GraphResponse(nodes=nodes, edges=edges)
## Resolution endpoint
@@ -143,7 +148,7 @@ async def resolve_identifier(
"resolution_method": "permalink"
}
"""
with telemetry.operation(
with logfire.span(
"api.request.knowledge.resolve_entity",
entrypoint="api",
domain="knowledge",
@@ -151,25 +156,13 @@ async def resolve_identifier(
):
logger.info(f"API v2 request: resolve_identifier for '{data.identifier}'")
with telemetry.scope(
"api.knowledge.resolve_entity.lookup_entity",
domain="knowledge",
action="resolve_entity",
phase="lookup_entity",
):
entity = await entity_repository.get_by_external_id(data.identifier)
entity = await entity_repository.get_by_external_id(data.identifier)
resolution_method = "external_id" if entity else "search"
if not entity:
with telemetry.scope(
"api.knowledge.resolve_entity.resolve_link",
domain="knowledge",
action="resolve_entity",
phase="resolve_link",
):
entity = await link_resolver.resolve_link(
data.identifier, source_path=data.source_path, strict=data.strict
)
entity = await link_resolver.resolve_link(
data.identifier, source_path=data.source_path, strict=data.strict
)
if entity:
if entity.permalink == data.identifier:
resolution_method = "permalink"
@@ -183,20 +176,14 @@ async def resolve_identifier(
if not entity:
raise HTTPException(status_code=404, detail=f"Entity not found: '{data.identifier}'")
with telemetry.scope(
"api.knowledge.resolve_entity.shape_response",
domain="knowledge",
action="resolve_entity",
phase="shape_response",
):
result = EntityResolveResponse(
external_id=entity.external_id,
entity_id=entity.id,
permalink=entity.permalink,
file_path=entity.file_path,
title=entity.title,
resolution_method=resolution_method,
)
result = EntityResolveResponse(
external_id=entity.external_id,
entity_id=entity.id,
permalink=entity.permalink,
file_path=entity.file_path,
title=entity.title,
resolution_method=resolution_method,
)
logger.debug(
f"API v2 response: resolved '{data.identifier}' to external_id={result.external_id} via {resolution_method}"
@@ -228,7 +215,7 @@ async def get_entity_by_id(
Raises:
HTTPException: 404 if entity not found
"""
with telemetry.operation(
with logfire.span(
"api.request.knowledge.get_entity",
entrypoint="api",
domain="knowledge",
@@ -236,25 +223,13 @@ async def get_entity_by_id(
):
logger.info(f"API v2 request: get_entity_by_id entity_id={entity_id}")
with telemetry.scope(
"api.knowledge.get_entity.load_entity",
domain="knowledge",
action="get_entity",
phase="load_entity",
):
entity = await entity_repository.get_by_external_id(entity_id)
entity = await entity_repository.get_by_external_id(entity_id)
if not entity:
raise HTTPException(
status_code=404, detail=f"Entity with external_id '{entity_id}' not found"
)
with telemetry.scope(
"api.knowledge.get_entity.shape_response",
domain="knowledge",
action="get_entity",
phase="shape_response",
):
result = EntityResponseV2.model_validate(entity)
result = EntityResponseV2.model_validate(entity)
logger.info(f"API v2 response: external_id={entity_id}, title='{result.title}'")
return result
@@ -267,106 +242,44 @@ async def get_entity_by_id(
async def create_entity(
project_id: ProjectExternalIdPathDep,
data: Entity,
background_tasks: BackgroundTasks,
entity_service: EntityServiceV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
task_scheduler: TaskSchedulerDep,
file_service: FileServiceV2ExternalDep,
app_config: AppConfigDep,
fast: bool = Query(
True, description="If true, write quickly and defer indexing to background tasks."
),
) -> EntityResponseV2:
"""Create a new entity.
Args:
data: Entity data to create
fast: If True, defer indexing to background tasks
Returns:
Created entity with generated external_id (UUID) and file content
"""
with telemetry.operation(
with logfire.span(
"api.request.knowledge.create_entity",
entrypoint="api",
domain="knowledge",
action="create_entity",
fast=fast,
):
logger.info(
"API v2 request", endpoint="create_entity", note_type=data.note_type, title=data.title
)
with telemetry.scope(
"api.knowledge.create_entity.write_entity",
domain="knowledge",
action="create_entity",
phase="write_entity",
fast=fast,
):
if fast:
entity = await entity_service.fast_write_entity(data)
written_content = None
search_content = None
else:
write_result = await entity_service.create_entity_with_content(data)
entity = write_result.entity
written_content = write_result.content
search_content = write_result.search_content
if fast:
with telemetry.scope(
"api.knowledge.create_entity.enqueue_reindex",
domain="knowledge",
action="create_entity",
phase="enqueue_reindex",
fast=fast,
):
task_scheduler.schedule(
"reindex_entity",
entity_id=entity.id,
project_id=project_id,
)
else:
with telemetry.scope(
"api.knowledge.create_entity.search_index",
domain="knowledge",
action="create_entity",
phase="search_index",
):
await search_service.index_entity(entity, content=search_content)
with telemetry.scope(
"api.knowledge.create_entity.vector_sync",
domain="knowledge",
action="create_entity",
phase="vector_sync",
):
_schedule_vector_sync_if_enabled(
task_scheduler=task_scheduler,
app_config=app_config,
entity_id=entity.id,
project_id=project_id,
)
# Note writes are now internally consistent before the response returns. We only leave
# truly derived work, like semantic vectors, on the async scheduler.
write_result = await entity_service.create_entity_with_content(data)
entity = write_result.entity
await search_service.index_entity(entity, content=write_result.search_content)
_schedule_vector_sync_if_enabled(
task_scheduler=task_scheduler,
app_config=app_config,
entity_id=entity.id,
project_id=project_id,
)
result = EntityResponseV2.model_validate(entity)
if fast:
result = result.model_copy(update={"observations": [], "relations": []})
with telemetry.scope(
"api.knowledge.create_entity.read_content",
domain="knowledge",
action="create_entity",
phase="read_content",
source="file" if fast else "memory",
):
if fast:
content = await file_service.read_file_content(entity.file_path)
else:
# Non-fast writes already captured the markdown in memory. Reuse it here
# instead of re-reading the file; format_on_save is the one config that can
# still make the persisted file diverge because write_file only returns a checksum.
content = written_content
result = result.model_copy(update={"content": content})
# The write service already returns the canonical markdown accepted for this request.
result = result.model_copy(update={"content": write_result.content})
logger.info(
f"API v2 response: endpoint='create_entity' external_id={entity.external_id}, title={result.title}, permalink={result.permalink}, status_code=201"
@@ -381,18 +294,13 @@ async def create_entity(
async def update_entity_by_id(
data: Entity,
response: Response,
background_tasks: BackgroundTasks,
project_id: ProjectExternalIdPathDep,
entity_service: EntityServiceV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
task_scheduler: TaskSchedulerDep,
file_service: FileServiceV2ExternalDep,
app_config: AppConfigDep,
entity_id: str = Path(..., description="Entity external ID (UUID)"),
fast: bool = Query(
True, description="If true, write quickly and defer indexing to background tasks."
),
) -> EntityResponseV2:
"""Update an entity by external ID.
@@ -401,121 +309,52 @@ async def update_entity_by_id(
Args:
entity_id: External ID (UUID string)
data: Updated entity data
fast: If True, defer indexing to background tasks
Returns:
Updated entity with file content
"""
with telemetry.operation(
with logfire.span(
"api.request.knowledge.update_entity",
entrypoint="api",
domain="knowledge",
action="update_entity",
fast=fast,
):
logger.info(f"API v2 request: update_entity_by_id entity_id={entity_id}")
with telemetry.scope(
"api.knowledge.update_entity.load_entity",
domain="knowledge",
action="update_entity",
phase="load_entity",
):
existing = await entity_repository.get_by_external_id(entity_id)
existing = await entity_repository.get_by_external_id(entity_id)
created = existing is None
with telemetry.scope(
"api.knowledge.update_entity.write_entity",
domain="knowledge",
action="update_entity",
phase="write_entity",
fast=fast,
):
if fast:
entity = await entity_service.fast_write_entity(data, external_id=entity_id)
written_content = None
search_content = None
response.status_code = 200 if existing else 201
else:
if existing:
write_result = await entity_service.update_entity_with_content(existing, data)
entity = write_result.entity
written_content = write_result.content
search_content = write_result.search_content
response.status_code = 200
else:
write_result = await entity_service.create_entity_with_content(data)
entity = write_result.entity
written_content = write_result.content
search_content = write_result.search_content
if entity.external_id != entity_id:
entity = await entity_repository.update(
entity.id,
{"external_id": entity_id},
)
# external_id fixup only changes the DB row. The file content is unchanged,
# so the markdown captured during the write remains valid downstream.
if not entity:
raise HTTPException(
status_code=404,
detail=f"Entity with external_id '{entity_id}' not found",
)
response.status_code = 201
if fast:
with telemetry.scope(
"api.knowledge.update_entity.enqueue_reindex",
domain="knowledge",
action="update_entity",
phase="enqueue_reindex",
fast=fast,
):
task_scheduler.schedule(
"reindex_entity",
entity_id=entity.id,
project_id=project_id,
resolve_relations=created,
)
if existing:
write_result = await entity_service.update_entity_with_content(existing, data)
entity = write_result.entity
response.status_code = 200
else:
with telemetry.scope(
"api.knowledge.update_entity.search_index",
domain="knowledge",
action="update_entity",
phase="search_index",
):
await search_service.index_entity(entity, content=search_content)
with telemetry.scope(
"api.knowledge.update_entity.vector_sync",
domain="knowledge",
action="update_entity",
phase="vector_sync",
):
_schedule_vector_sync_if_enabled(
task_scheduler=task_scheduler,
app_config=app_config,
entity_id=entity.id,
project_id=project_id,
write_result = await entity_service.create_entity_with_content(data)
entity = write_result.entity
if entity.external_id != entity_id:
entity = await entity_repository.update(
entity.id,
{"external_id": entity_id},
)
# external_id fixup only changes the DB row. The file content is unchanged,
# so the markdown captured during the write remains valid downstream.
if not entity:
raise HTTPException(
status_code=404,
detail=f"Entity with external_id '{entity_id}' not found",
)
response.status_code = 201
await search_service.index_entity(entity, content=write_result.search_content)
_schedule_vector_sync_if_enabled(
task_scheduler=task_scheduler,
app_config=app_config,
entity_id=entity.id,
project_id=project_id,
)
result = EntityResponseV2.model_validate(entity)
if fast:
result = result.model_copy(update={"observations": [], "relations": []})
with telemetry.scope(
"api.knowledge.update_entity.read_content",
domain="knowledge",
action="update_entity",
phase="read_content",
source="file" if fast else "memory",
):
if fast:
content = await file_service.read_file_content(entity.file_path)
else:
# Non-fast writes already captured the markdown in memory. Reuse it here
# instead of re-reading the file; format_on_save is the one config that can
# still make the persisted file diverge because write_file only returns a checksum.
content = written_content
result = result.model_copy(update={"content": content})
result = result.model_copy(update={"content": write_result.content})
logger.info(
f"API v2 response: external_id={entity_id}, created={created}, status_code={response.status_code}"
@@ -526,25 +365,19 @@ async def update_entity_by_id(
@router.patch("/entities/{entity_id}", response_model=EntityResponseV2)
async def edit_entity_by_id(
data: EditEntityRequest,
background_tasks: BackgroundTasks,
project_id: ProjectExternalIdPathDep,
entity_service: EntityServiceV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
task_scheduler: TaskSchedulerDep,
file_service: FileServiceV2ExternalDep,
app_config: AppConfigDep,
entity_id: str = Path(..., description="Entity external ID (UUID)"),
fast: bool = Query(
True, description="If true, write quickly and defer indexing to background tasks."
),
) -> EntityResponseV2:
"""Edit an existing entity by external ID using operations like append, prepend, etc.
Args:
entity_id: External ID (UUID string)
data: Edit operation details
fast: If True, defer indexing to background tasks
Returns:
Updated entity with file content
@@ -552,115 +385,43 @@ async def edit_entity_by_id(
Raises:
HTTPException: 404 if entity not found, 400 if edit fails
"""
with telemetry.operation(
with logfire.span(
"api.request.knowledge.edit_entity",
entrypoint="api",
domain="knowledge",
action="edit_entity",
fast=fast,
):
logger.info(
f"API v2 request: edit_entity_by_id entity_id={entity_id}, operation='{data.operation}'"
)
with telemetry.scope(
"api.knowledge.edit_entity.load_entity",
domain="knowledge",
action="edit_entity",
phase="load_entity",
):
entity = await entity_repository.get_by_external_id(entity_id)
entity = await entity_repository.get_by_external_id(entity_id)
if not entity: # pragma: no cover
raise HTTPException(
status_code=404, detail=f"Entity with external_id '{entity_id}' not found"
)
try:
with telemetry.scope(
"api.knowledge.edit_entity.write_entity",
domain="knowledge",
action="edit_entity",
phase="write_entity",
fast=fast,
):
if fast:
updated_entity = await entity_service.fast_edit_entity(
entity=entity,
operation=data.operation,
content=data.content,
section=data.section,
find_text=data.find_text,
expected_replacements=data.expected_replacements,
)
written_content = None
search_content = None
else:
identifier = entity.permalink or entity.file_path
write_result = await entity_service.edit_entity_with_content(
identifier=identifier,
operation=data.operation,
content=data.content,
section=data.section,
find_text=data.find_text,
expected_replacements=data.expected_replacements,
)
updated_entity = write_result.entity
written_content = write_result.content
search_content = write_result.search_content
if fast:
with telemetry.scope(
"api.knowledge.edit_entity.enqueue_reindex",
domain="knowledge",
action="edit_entity",
phase="enqueue_reindex",
fast=fast,
):
task_scheduler.schedule(
"reindex_entity",
entity_id=updated_entity.id,
project_id=project_id,
)
else:
with telemetry.scope(
"api.knowledge.edit_entity.search_index",
domain="knowledge",
action="edit_entity",
phase="search_index",
):
await search_service.index_entity(updated_entity, content=search_content)
with telemetry.scope(
"api.knowledge.edit_entity.vector_sync",
domain="knowledge",
action="edit_entity",
phase="vector_sync",
):
_schedule_vector_sync_if_enabled(
task_scheduler=task_scheduler,
app_config=app_config,
entity_id=updated_entity.id,
project_id=project_id,
)
identifier = entity.permalink or entity.file_path
write_result = await entity_service.edit_entity_with_content(
identifier=identifier,
operation=data.operation,
content=data.content,
section=data.section,
find_text=data.find_text,
expected_replacements=data.expected_replacements,
)
updated_entity = write_result.entity
await search_service.index_entity(updated_entity, content=write_result.search_content)
_schedule_vector_sync_if_enabled(
task_scheduler=task_scheduler,
app_config=app_config,
entity_id=updated_entity.id,
project_id=project_id,
)
result = EntityResponseV2.model_validate(updated_entity)
if fast:
result = result.model_copy(update={"observations": [], "relations": []})
with telemetry.scope(
"api.knowledge.edit_entity.read_content",
domain="knowledge",
action="edit_entity",
phase="read_content",
source="file" if fast else "memory",
):
if fast:
content = await file_service.read_file_content(updated_entity.file_path)
else:
# Non-fast writes already captured the markdown in memory. Reuse it here
# instead of re-reading the file; format_on_save is the one config that can
# still make the persisted file diverge because write_file only returns a checksum.
content = written_content
result = result.model_copy(update={"content": content})
result = result.model_copy(update={"content": write_result.content})
logger.info(
f"API v2 response: external_id={entity_id}, operation='{data.operation}', status_code=200"
@@ -678,12 +439,10 @@ async def edit_entity_by_id(
@router.delete("/entities/{entity_id}", response_model=DeleteEntitiesResponse)
async def delete_entity_by_id(
background_tasks: BackgroundTasks,
project_id: ProjectExternalIdPathDep,
entity_service: EntityServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
entity_id: str = Path(..., description="Entity external ID (UUID)"),
search_service=Depends(lambda: None), # Optional for now
) -> DeleteEntitiesResponse:
"""Delete an entity by external ID.
@@ -695,23 +454,25 @@ async def delete_entity_by_id(
Note: Returns deleted=False if entity doesn't exist (idempotent)
"""
logger.info(f"API v2 request: delete_entity_by_id entity_id={entity_id}")
with logfire.span(
"api.request.knowledge.delete_entity",
entrypoint="api",
domain="knowledge",
action="delete_entity",
):
logger.info(f"API v2 request: delete_entity_by_id entity_id={entity_id}")
entity = await entity_repository.get_by_external_id(entity_id)
if entity is None:
logger.info(f"API v2 response: external_id={entity_id} not found, deleted=False")
return DeleteEntitiesResponse(deleted=False)
entity = await entity_repository.get_by_external_id(entity_id)
if entity is None:
logger.info(f"API v2 response: external_id={entity_id} not found, deleted=False")
return DeleteEntitiesResponse(deleted=False)
# Delete the entity using internal ID
deleted = await entity_service.delete_entity(entity.id)
# Delete the entity using internal ID
deleted = await entity_service.delete_entity(entity.id)
# Remove from search index if search service available
if search_service:
background_tasks.add_task(search_service.handle_delete, entity) # pragma: no cover
logger.info(f"API v2 response: external_id={entity_id}, deleted={deleted}")
logger.info(f"API v2 response: external_id={entity_id}, deleted={deleted}")
return DeleteEntitiesResponse(deleted=deleted)
return DeleteEntitiesResponse(deleted=deleted)
## Move endpoint
@@ -720,7 +481,6 @@ async def delete_entity_by_id(
@router.put("/entities/{entity_id}/move", response_model=EntityResponseV2)
async def move_entity(
data: MoveEntityRequestV2,
background_tasks: BackgroundTasks,
project_id: ProjectExternalIdPathDep,
entity_service: EntityServiceV2ExternalDep,
entity_repository: EntityRepositoryV2ExternalDep,
@@ -743,48 +503,58 @@ async def move_entity(
Returns:
Updated entity with new file path
"""
logger.info(
f"API v2 request: move_entity entity_id={entity_id}, destination='{data.destination_path}'"
)
try:
# First, get the entity by external_id to verify it exists
entity = await entity_repository.get_by_external_id(entity_id)
if not entity: # pragma: no cover
raise HTTPException(
status_code=404, detail=f"Entity with external_id '{entity_id}' not found"
)
# Move the entity using its current file path as identifier
moved_entity = await entity_service.move_entity(
identifier=entity.file_path, # Use file path for resolution
destination_path=data.destination_path,
project_config=project_config,
app_config=app_config,
with logfire.span(
"api.request.knowledge.move_entity",
entrypoint="api",
domain="knowledge",
action="move_entity",
):
logger.info(
f"API v2 request: move_entity entity_id={entity_id}, destination='{data.destination_path}'"
)
# Reindex at new location
reindexed_entity = await entity_service.link_resolver.resolve_link(data.destination_path)
if reindexed_entity:
await search_service.index_entity(reindexed_entity)
_schedule_vector_sync_if_enabled(
task_scheduler=task_scheduler,
try:
# First, get the entity by external_id to verify it exists
entity = await entity_repository.get_by_external_id(entity_id)
if not entity: # pragma: no cover
raise HTTPException(
status_code=404, detail=f"Entity with external_id '{entity_id}' not found"
)
# Move the entity using its current file path as identifier
moved_entity = await entity_service.move_entity(
identifier=entity.file_path, # Use file path for resolution
destination_path=data.destination_path,
project_config=project_config,
app_config=app_config,
entity_id=reindexed_entity.id,
project_id=project_id,
)
result = EntityResponseV2.model_validate(moved_entity)
# Reindex at new location
reindexed_entity = await entity_service.link_resolver.resolve_link(
data.destination_path
)
if reindexed_entity:
await search_service.index_entity(reindexed_entity)
_schedule_vector_sync_if_enabled(
task_scheduler=task_scheduler,
app_config=app_config,
entity_id=reindexed_entity.id,
project_id=project_id,
)
logger.info(f"API v2 response: moved external_id={entity_id} to '{data.destination_path}'")
result = EntityResponseV2.model_validate(moved_entity)
return result
logger.info(
f"API v2 response: moved external_id={entity_id} to '{data.destination_path}'"
)
except HTTPException: # pragma: no cover
raise # pragma: no cover
except Exception as e:
logger.error(f"Error moving entity: {e}")
raise HTTPException(status_code=400, detail=str(e))
return result
except HTTPException: # pragma: no cover
raise # pragma: no cover
except Exception as e:
logger.error(f"Error moving entity: {e}")
raise HTTPException(status_code=400, detail=str(e))
## Move directory endpoint
@@ -793,7 +563,6 @@ async def move_entity(
@router.post("/move-directory", response_model=DirectoryMoveResult)
async def move_directory(
data: MoveDirectoryRequestV2,
background_tasks: BackgroundTasks,
project_id: ProjectExternalIdPathDep,
entity_service: EntityServiceV2ExternalDep,
project_config: ProjectConfigV2ExternalDep,
@@ -814,40 +583,46 @@ async def move_directory(
Returns:
DirectoryMoveResult with counts and details of moved files
"""
logger.info(
f"API v2 request: move_directory source='{data.source_directory}', destination='{data.destination_directory}'"
)
try:
# Move the directory using the service
result = await entity_service.move_directory(
source_directory=data.source_directory,
destination_directory=data.destination_directory,
project_config=project_config,
app_config=app_config,
)
# Reindex moved entities
for file_path in result.moved_files:
entity = await entity_service.link_resolver.resolve_link(file_path)
if entity:
await search_service.index_entity(entity)
_schedule_vector_sync_if_enabled(
task_scheduler=task_scheduler,
app_config=app_config,
entity_id=entity.id,
project_id=project_id,
)
with logfire.span(
"api.request.knowledge.move_directory",
entrypoint="api",
domain="knowledge",
action="move_directory",
):
logger.info(
f"API v2 response: move_directory "
f"total={result.total_files}, success={result.successful_moves}, failed={result.failed_moves}"
f"API v2 request: move_directory source='{data.source_directory}', destination='{data.destination_directory}'"
)
return result
except Exception as e:
logger.error(f"Error moving directory: {e}")
raise HTTPException(status_code=400, detail=str(e))
try:
# Move the directory using the service
result = await entity_service.move_directory(
source_directory=data.source_directory,
destination_directory=data.destination_directory,
project_config=project_config,
app_config=app_config,
)
# Reindex moved entities
for file_path in result.moved_files:
entity = await entity_service.link_resolver.resolve_link(file_path)
if entity:
await search_service.index_entity(entity)
_schedule_vector_sync_if_enabled(
task_scheduler=task_scheduler,
app_config=app_config,
entity_id=entity.id,
project_id=project_id,
)
logger.info(
f"API v2 response: move_directory "
f"total={result.total_files}, success={result.successful_moves}, failed={result.failed_moves}"
)
return result
except Exception as e:
logger.error(f"Error moving directory: {e}")
raise HTTPException(status_code=400, detail=str(e))
## Delete directory endpoint
@@ -872,20 +647,26 @@ async def delete_directory(
Returns:
DirectoryDeleteResult with counts and details of deleted files
"""
logger.info(f"API v2 request: delete_directory directory='{data.directory}'")
with logfire.span(
"api.request.knowledge.delete_directory",
entrypoint="api",
domain="knowledge",
action="delete_directory",
):
logger.info(f"API v2 request: delete_directory directory='{data.directory}'")
try:
# Delete the directory using the service
result = await entity_service.delete_directory(
directory=data.directory,
)
try:
# Delete the directory using the service
result = await entity_service.delete_directory(
directory=data.directory,
)
logger.info(
f"API v2 response: delete_directory "
f"total={result.total_files}, success={result.successful_deletes}, failed={result.failed_deletes}"
)
return result
logger.info(
f"API v2 response: delete_directory "
f"total={result.total_files}, success={result.successful_deletes}, failed={result.failed_deletes}"
)
return result
except Exception as e:
logger.error(f"Error deleting directory: {e}")
raise HTTPException(status_code=400, detail=str(e))
except Exception as e:
logger.error(f"Error deleting directory: {e}")
raise HTTPException(status_code=400, detail=str(e))
@@ -9,7 +9,7 @@ from typing import Annotated, Optional
from fastapi import APIRouter, Query, Path
from loguru import logger
from basic_memory import telemetry
import logfire
from basic_memory.deps import ContextServiceV2ExternalDep, EntityRepositoryV2ExternalDep
from basic_memory.schemas.base import TimeFrame, parse_timeframe
from basic_memory.schemas.memory import (
@@ -51,7 +51,7 @@ async def recent(
Returns:
GraphContext with recent activity and related entities
"""
with telemetry.operation(
with logfire.span(
"api.request.memory.recent_activity",
entrypoint="api",
domain="memory",
@@ -72,7 +72,7 @@ async def recent(
limit = page_size
offset = (page - 1) * page_size
with telemetry.scope(
with logfire.span(
"api.memory.recent_activity.build_context",
domain="memory",
action="recent_activity",
@@ -88,7 +88,7 @@ async def recent(
offset=offset,
max_related=max_related,
)
with telemetry.scope(
with logfire.span(
"api.memory.recent_activity.shape_response",
domain="memory",
action="recent_activity",
@@ -137,7 +137,7 @@ async def get_memory_context(
Returns:
GraphContext with the entity and its related context
"""
with telemetry.operation(
with logfire.span(
"api.request.memory.build_context",
entrypoint="api",
domain="memory",
@@ -154,7 +154,7 @@ async def get_memory_context(
limit = page_size
offset = (page - 1) * page_size
with telemetry.scope(
with logfire.span(
"api.memory.build_context.build_context",
domain="memory",
action="build_context",
@@ -170,7 +170,7 @@ async def get_memory_context(
offset=offset,
max_related=max_related,
)
with telemetry.scope(
with logfire.span(
"api.memory.build_context.shape_response",
domain="memory",
action="build_context",
@@ -11,7 +11,7 @@ Key improvements:
"""
import os
from typing import Optional
from typing import Literal, Optional
from fastapi import APIRouter, HTTPException, Body, Query, Path
from loguru import logger
@@ -25,6 +25,8 @@ from basic_memory.deps import (
ProjectExternalIdPathDep,
)
from basic_memory.schemas import SyncReportResponse
from basic_memory.models import Project
from basic_memory.repository.project_repository import ProjectRepository
from basic_memory.schemas.project_info import (
ProjectItem,
ProjectList,
@@ -36,6 +38,71 @@ from basic_memory.schemas.v2 import ProjectResolveRequest, ProjectResolveRespons
from basic_memory.utils import normalize_project_path, generate_permalink
router = APIRouter(prefix="/projects", tags=["project_management-v2"])
ProjectResolveMethod = Literal["external_id", "name", "permalink"]
def _split_qualified_project_identifier(identifier: str) -> tuple[str | None, str]:
"""Split ``<workspace>/<project>`` identifiers while preserving plain project names."""
cleaned = identifier.strip()
if "/" not in cleaned:
return None, cleaned
workspace_identifier, project_identifier = cleaned.split("/", 1)
if not workspace_identifier or not project_identifier:
return None, cleaned
return workspace_identifier, project_identifier
async def _resolve_project_identifier_candidate(
project_repository: ProjectRepository,
identifier: str,
) -> tuple[Project | None, ProjectResolveMethod]:
"""Resolve one project identifier candidate and report the matching method."""
identifier_permalink = generate_permalink(identifier)
project = await project_repository.get_by_external_id(identifier)
if project:
return project, "external_id"
project = await project_repository.get_by_permalink(identifier_permalink)
if project:
return project, "permalink"
project = await project_repository.get_by_name_case_insensitive(identifier)
if project:
return project, "name" # pragma: no cover
return None, "name"
async def _resolve_project_identifier(
project_repository: ProjectRepository,
identifier: str,
) -> tuple[Project | None, ProjectResolveMethod]:
"""Resolve exact identifiers first, then accepted workspace-qualified forms."""
project, resolution_method = await _resolve_project_identifier_candidate(
project_repository,
identifier,
)
if project:
return project, resolution_method
workspace_identifier, project_identifier = _split_qualified_project_identifier(identifier)
if workspace_identifier is None:
return None, resolution_method
# Trigger: an MCP disambiguation error suggested ``workspace/project``.
# Why: request routing already selected the workspace/tenant; this endpoint
# only needs the project segment to validate the active project.
# Outcome: models can follow the hint verbatim instead of looping on a 404.
project, resolution_method = await _resolve_project_identifier_candidate(
project_repository,
project_identifier,
)
if project:
return project, resolution_method
return None, resolution_method
@router.get("/", response_model=ProjectList)
@@ -247,28 +314,10 @@ async def resolve_project_identifier(
"""
logger.info(f"API v2 request: resolve_project_identifier for '{data.identifier}'")
# Generate permalink for comparison
identifier_permalink = generate_permalink(data.identifier)
resolution_method = "name"
project = None
# Try external_id first (UUID format)
project = await project_repository.get_by_external_id(data.identifier)
if project:
resolution_method = "external_id"
# If not found by external_id, try by permalink (exact match)
if not project:
project = await project_repository.get_by_permalink(identifier_permalink)
if project:
resolution_method = "permalink"
# If not found by permalink, try case-insensitive name search
if not project:
project = await project_repository.get_by_name_case_insensitive(data.identifier)
if project:
resolution_method = "name" # pragma: no cover
project, resolution_method = await _resolve_project_identifier(
project_repository,
data.identifier,
)
if not project:
raise HTTPException(status_code=404, detail=f"Project not found: '{data.identifier}'")
@@ -6,6 +6,7 @@ have entity IDs in URLs - they generate formatted prompts from queries.
"""
from datetime import datetime, timezone
from typing import Any
from fastapi import APIRouter, HTTPException, status, Path
from loguru import logger
@@ -59,6 +60,7 @@ async def continue_conversation(
# Initialize search results
search_results = []
hierarchical_results_for_count = []
# Get data needed for template
if request.topic:
@@ -91,7 +93,8 @@ async def continue_conversation(
# Limit to a reasonable number of total results
all_hierarchical_results = all_hierarchical_results[:10]
template_context = {
hierarchical_results_for_count = all_hierarchical_results
template_context: dict[str, Any] = {
"topic": request.topic,
"timeframe": request.timeframe,
"hierarchical_results": all_hierarchical_results,
@@ -110,6 +113,7 @@ async def continue_conversation(
hierarchical_results = recent_context.results[:5] # Limit to top 5 recent items
hierarchical_results_for_count = hierarchical_results
template_context = {
"topic": f"Recent Activity from ({request.timeframe})",
"timeframe": request.timeframe,
@@ -129,9 +133,6 @@ async def continue_conversation(
relation_count = 0
entity_count = 0
# Get the hierarchical results from the template context
hierarchical_results_for_count = template_context.get("hierarchical_results", [])
# For topic-based search
if request.topic:
for item in hierarchical_results_for_count:
@@ -159,29 +160,24 @@ async def continue_conversation(
elif related.type == "entity": # pragma: no cover
entity_count += 1 # pragma: no cover
# Build metadata
metadata = {
"query": request.topic,
"timeframe": request.timeframe,
"search_count": len(search_results)
if request.topic
else 0, # Original search results count
"context_count": len(hierarchical_results_for_count),
"observation_count": observation_count,
"relation_count": relation_count,
"total_items": (
prompt_metadata = PromptMetadata(
query=request.topic,
timeframe=request.timeframe,
search_count=len(search_results) if request.topic else 0,
context_count=len(hierarchical_results_for_count),
observation_count=observation_count,
relation_count=relation_count,
total_items=(
len(hierarchical_results_for_count)
+ observation_count
+ relation_count
+ entity_count
),
"search_limit": request.search_items_limit,
"context_depth": request.depth,
"related_limit": request.related_items_limit,
"generated_at": datetime.now(timezone.utc).isoformat(),
}
prompt_metadata = PromptMetadata(**metadata)
search_limit=request.search_items_limit,
context_depth=request.depth,
related_limit=request.related_items_limit,
generated_at=datetime.now(timezone.utc).isoformat(),
)
return PromptResponse(
prompt=rendered_prompt, context=template_context, metadata=prompt_metadata
@@ -229,7 +225,7 @@ async def search_prompt(
results = await search_service.search(query, limit=limit, offset=offset)
search_results = await to_search_results(entity_service, results)
template_context = {
template_context: dict[str, Any] = {
"query": request.query,
"timeframe": request.timeframe,
"results": search_results,
@@ -241,22 +237,19 @@ async def search_prompt(
# Render template
rendered_prompt = await template_loader.render("prompts/search.hbs", template_context)
# Build metadata
metadata = {
"query": request.query,
"timeframe": request.timeframe,
"search_count": len(search_results),
"context_count": len(search_results),
"observation_count": 0, # Search results don't include observations
"relation_count": 0, # Search results don't include relations
"total_items": len(search_results),
"search_limit": limit,
"context_depth": 0, # No context depth for basic search
"related_limit": 0, # No related items for basic search
"generated_at": datetime.now(timezone.utc).isoformat(),
}
prompt_metadata = PromptMetadata(**metadata)
prompt_metadata = PromptMetadata(
query=request.query,
timeframe=request.timeframe,
search_count=len(search_results),
context_count=len(search_results),
observation_count=0,
relation_count=0,
total_items=len(search_results),
search_limit=limit,
context_depth=0,
related_limit=0,
generated_at=datetime.now(timezone.utc).isoformat(),
)
return PromptResponse(
prompt=rendered_prompt, context=template_context, metadata=prompt_metadata
@@ -15,7 +15,7 @@ from pathlib import Path as PathLib
from fastapi import APIRouter, HTTPException, Response, Path
from loguru import logger
from basic_memory import telemetry
import logfire
from basic_memory.deps import (
ProjectConfigV2ExternalDep,
FileServiceV2ExternalDep,
@@ -56,7 +56,7 @@ async def get_resource_content(
Raises:
HTTPException: 404 if entity or file not found
"""
with telemetry.operation(
with logfire.span(
"api.request.resource.get_content",
entrypoint="api",
domain="resource",
@@ -64,7 +64,7 @@ async def get_resource_content(
):
logger.debug(f"V2 Getting content for project {project_id}, entity_id: {entity_id}")
with telemetry.scope(
with logfire.span(
"api.resource.get_content.load_entity",
domain="resource",
action="get_content",
@@ -74,7 +74,7 @@ async def get_resource_content(
if not entity:
raise HTTPException(status_code=404, detail=f"Entity {entity_id} not found")
with telemetry.scope(
with logfire.span(
"api.resource.get_content.validate_path",
domain="resource",
action="get_content",
@@ -90,7 +90,7 @@ async def get_resource_content(
detail="Entity contains invalid file path",
)
with telemetry.scope(
with logfire.span(
"api.resource.get_content.ensure_exists",
domain="resource",
action="get_content",
@@ -102,7 +102,7 @@ async def get_resource_content(
detail=f"File not found: {entity.file_path}",
)
with telemetry.scope(
with logfire.span(
"api.resource.get_content.read_content",
domain="resource",
action="get_content",
@@ -139,7 +139,7 @@ async def create_resource(
Raises:
HTTPException: 400 for invalid file paths, 409 if file already exists
"""
with telemetry.operation(
with logfire.span(
"api.request.resource.create",
entrypoint="api",
domain="resource",
@@ -166,7 +166,7 @@ async def create_resource(
f"Use PUT /resource/{existing_entity.external_id} to update it.",
)
with telemetry.scope(
with logfire.span(
"api.resource.create.write_file",
domain="resource",
action="create",
@@ -175,7 +175,7 @@ async def create_resource(
await file_service.ensure_directory(PathLib(data.file_path).parent)
checksum = await file_service.write_file(data.file_path, data.content)
with telemetry.scope(
with logfire.span(
"api.resource.create.read_metadata",
domain="resource",
action="create",
@@ -197,7 +197,7 @@ async def create_resource(
created_at=file_metadata.created_at,
updated_at=file_metadata.modified_at,
)
with telemetry.scope(
with logfire.span(
"api.resource.create.upsert_entity",
domain="resource",
action="create",
@@ -205,13 +205,13 @@ async def create_resource(
):
entity = await entity_repository.add(entity)
with telemetry.scope(
with logfire.span(
"api.resource.create.search_index",
domain="resource",
action="create",
phase="search_index",
):
await search_service.index_entity(entity) # pyright: ignore
await search_service.index_entity(entity)
return ResourceResponse(
entity_id=entity.id,
@@ -258,7 +258,7 @@ async def update_resource(
Raises:
HTTPException: 404 if entity not found, 400 for invalid paths
"""
with telemetry.operation(
with logfire.span(
"api.request.resource.update",
entrypoint="api",
domain="resource",
@@ -282,7 +282,7 @@ async def update_resource(
"Path must be relative and stay within project boundaries.",
)
with telemetry.scope(
with logfire.span(
"api.resource.update.write_file",
domain="resource",
action="update",
@@ -297,7 +297,7 @@ async def update_resource(
checksum = await file_service.write_file(target_file_path, data.content)
with telemetry.scope(
with logfire.span(
"api.resource.update.read_metadata",
domain="resource",
action="update",
@@ -309,7 +309,7 @@ async def update_resource(
content_type = file_service.content_type(target_file_path)
note_type = "canvas" if target_file_path.endswith(".canvas") else "file"
with telemetry.scope(
with logfire.span(
"api.resource.update.update_entity",
domain="resource",
action="update",
@@ -326,14 +326,16 @@ async def update_resource(
"updated_at": file_metadata.modified_at,
},
)
if updated_entity is None:
raise HTTPException(status_code=404, detail=f"Entity {entity_id} not found")
with telemetry.scope(
with logfire.span(
"api.resource.update.search_index",
domain="resource",
action="update",
phase="search_index",
):
await search_service.index_entity(updated_entity) # pyright: ignore
await search_service.index_entity(updated_entity)
return ResourceResponse(
entity_id=entity.id,
@@ -4,15 +4,17 @@ This router uses external_id UUIDs for stable, API-friendly routing.
V1 uses string-based project names which are less efficient and less stable.
"""
import asyncio
from fastapi import APIRouter, HTTPException, Path
from basic_memory import telemetry
import logfire
from basic_memory.api.v2.utils import to_search_results
from basic_memory.repository.semantic_errors import (
SemanticDependenciesMissingError,
SemanticSearchDisabledError,
)
from basic_memory.schemas.search import SearchQuery, SearchResponse
from basic_memory.schemas.search import SearchQuery, SearchResponse, SearchRetrievalMode
from basic_memory.deps import (
SearchServiceV2ExternalDep,
EntityServiceV2ExternalDep,
@@ -48,7 +50,7 @@ async def search(
Returns:
SearchResponse with paginated search results
"""
with telemetry.operation(
with logfire.span(
"api.request.search",
entrypoint="api",
domain="search",
@@ -65,9 +67,9 @@ async def search(
has_filters=bool(query.note_types or query.entity_types or query.metadata_filters),
):
offset = (page - 1) * page_size
fetch_limit = page_size + 1
exact_count_available = query.retrieval_mode == SearchRetrievalMode.FTS
try:
with telemetry.scope(
with logfire.span(
"api.search.search.execute_query",
domain="search",
action="search",
@@ -75,7 +77,14 @@ async def search(
page=page,
page_size=page_size,
):
results = await search_service.search(query, limit=fetch_limit, offset=offset)
if exact_count_available:
results, total = await asyncio.gather(
search_service.search(query, limit=page_size, offset=offset),
search_service.count(query),
)
else:
results = await search_service.search(query, limit=page_size + 1, offset=offset)
total = 0
except SemanticSearchDisabledError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
except SemanticDependenciesMissingError as exc:
@@ -83,18 +92,24 @@ async def search(
except ValueError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
with telemetry.scope(
with logfire.span(
"api.search.search.paginate_results",
domain="search",
action="search",
phase="paginate_results",
result_count=len(results),
):
has_more = len(results) > page_size
if has_more:
results = results[:page_size]
if exact_count_available:
has_more = offset + len(results) < total
else:
# Trigger: semantic modes would need another vector/hybrid retrieval to count.
# Why: search requests should not pay for a second semantic pass.
# Outcome: preserve probe pagination for semantic search and leave total at 0.
has_more = len(results) > page_size
if has_more:
results = results[:page_size]
with telemetry.scope(
with logfire.span(
"api.search.search.hydrate_results",
domain="search",
action="search",
@@ -102,7 +117,7 @@ async def search(
result_count=len(results),
):
search_results = await to_search_results(entity_service, results)
with telemetry.scope(
with logfire.span(
"api.search.search.build_response",
domain="search",
action="search",
@@ -113,6 +128,7 @@ async def search(
results=search_results,
current_page=page,
page_size=page_size,
total=total,
has_more=has_more,
)
+79 -52
View File
@@ -1,8 +1,6 @@
from typing import Optional, List
from typing import Any, Protocol, Optional, List, Sequence
from basic_memory import telemetry
from basic_memory.models import Entity as EntityModel
from basic_memory.repository import EntityRepository
import logfire
from basic_memory.repository.search_repository import SearchIndexRow
from basic_memory.schemas.memory import (
EntitySummary,
@@ -13,20 +11,39 @@ from basic_memory.schemas.memory import (
ContextResult,
)
from basic_memory.schemas.search import SearchItemType, SearchResult
from basic_memory.services import EntityService
from basic_memory.services.context_service import (
ContextResultRow,
ContextResult as ServiceContextResult,
)
class EntityBatchLookup(Protocol):
async def find_by_ids(self, ids: List[int]) -> Sequence[Any]: ...
class EntityServiceBatchLookup(Protocol):
async def get_entities_by_id(self, ids: List[int]) -> Sequence[Any]: ...
def _required_str(value: str | None, field_name: str) -> str:
"""Return a required search field or fail before producing invalid response data."""
if value is None:
raise ValueError(f"Search result is missing required field: {field_name}")
return value
def _search_item_type(value: str | SearchItemType) -> SearchItemType:
"""Normalize repository row type strings into the public search enum."""
return value if isinstance(value, SearchItemType) else SearchItemType(value)
async def to_graph_context(
context_result: ServiceContextResult,
entity_repository: EntityRepository,
entity_repository: EntityBatchLookup,
page: Optional[int] = None,
page_size: Optional[int] = None,
):
with telemetry.scope(
) -> GraphContext:
with logfire.span(
"memory.hydrate_context",
domain="memory",
action="build_context",
@@ -44,17 +61,18 @@ async def to_graph_context(
+ context_item.observations
+ context_item.related_results
):
if item.type == SearchItemType.ENTITY:
item_type = _search_item_type(item.type)
if item_type == SearchItemType.ENTITY:
# Entity's own ID for its external_id
entity_ids_needed.add(item.id)
elif item.type == SearchItemType.OBSERVATION:
elif item_type == SearchItemType.OBSERVATION:
# Parent entity ID for entity_external_id
if item.entity_id: # pyright: ignore
entity_ids_needed.add(item.entity_id) # pyright: ignore
elif item.type == SearchItemType.RELATION:
if item.entity_id:
entity_ids_needed.add(item.entity_id)
elif item_type == SearchItemType.RELATION:
# Source and target entity IDs for external_ids
if item.from_id: # pyright: ignore
entity_ids_needed.add(item.from_id) # pyright: ignore
if item.from_id:
entity_ids_needed.add(item.from_id)
if item.to_id:
entity_ids_needed.add(item.to_id)
@@ -62,7 +80,7 @@ async def to_graph_context(
entity_title_lookup: dict[int, str] = {}
entity_external_id_lookup: dict[int, str] = {}
if entity_ids_needed:
with telemetry.scope(
with logfire.span(
"memory.hydrate_context.lookup_entities",
domain="memory",
action="build_context",
@@ -75,59 +93,62 @@ async def to_graph_context(
entity_external_id_lookup[e.id] = e.external_id
# Helper function to convert items to summaries
def to_summary(item: SearchIndexRow | ContextResultRow):
match item.type:
def to_summary(
item: SearchIndexRow | ContextResultRow,
) -> EntitySummary | ObservationSummary | RelationSummary:
item_type = _search_item_type(item.type)
match item_type:
case SearchItemType.ENTITY:
return EntitySummary(
external_id=entity_external_id_lookup.get(item.id, ""),
entity_id=item.id,
title=item.title, # pyright: ignore
title=_required_str(item.title, "title"),
permalink=item.permalink,
content=item.content,
file_path=item.file_path,
file_path=_required_str(item.file_path, "file_path"),
created_at=item.created_at,
)
case SearchItemType.OBSERVATION:
entity_ext_id = None
if item.entity_id: # pyright: ignore
entity_ext_id = entity_external_id_lookup.get(item.entity_id) # pyright: ignore
entity_title = None
if item.entity_id:
entity_ext_id = entity_external_id_lookup.get(item.entity_id)
entity_title = entity_title_lookup.get(item.entity_id)
return ObservationSummary(
observation_id=item.id,
entity_id=item.entity_id, # pyright: ignore
entity_id=item.entity_id,
entity_external_id=entity_ext_id,
title=entity_title_lookup.get(item.entity_id), # pyright: ignore
file_path=item.file_path,
category=item.category, # pyright: ignore
content=item.content, # pyright: ignore
permalink=item.permalink, # pyright: ignore
title=entity_title,
file_path=_required_str(item.file_path, "file_path"),
category=_required_str(item.category, "category"),
content=_required_str(item.content, "content"),
permalink=_required_str(item.permalink, "permalink"),
created_at=item.created_at,
)
case SearchItemType.RELATION:
from_title = entity_title_lookup.get(item.from_id) if item.from_id else None # pyright: ignore
from_title = entity_title_lookup.get(item.from_id) if item.from_id else None
to_title = entity_title_lookup.get(item.to_id) if item.to_id else None
from_ext_id = (
entity_external_id_lookup.get(item.from_id) if item.from_id else None
) # pyright: ignore
)
to_ext_id = entity_external_id_lookup.get(item.to_id) if item.to_id else None
return RelationSummary(
relation_id=item.id,
entity_id=item.entity_id, # pyright: ignore
title=item.title, # pyright: ignore
file_path=item.file_path,
permalink=item.permalink, # pyright: ignore
relation_type=item.relation_type, # pyright: ignore
entity_id=item.entity_id,
title=_required_str(item.title, "title"),
file_path=_required_str(item.file_path, "file_path"),
permalink=_required_str(item.permalink, "permalink"),
relation_type=_required_str(item.relation_type, "relation_type"),
from_entity=from_title,
from_entity_id=item.from_id, # pyright: ignore
from_entity_id=item.from_id,
from_entity_external_id=from_ext_id,
to_entity=to_title,
to_entity_id=item.to_id,
to_entity_external_id=to_ext_id,
created_at=item.created_at,
)
case _: # pragma: no cover
raise ValueError(f"Unexpected type: {item.type}")
with telemetry.scope(
with logfire.span(
"memory.hydrate_context.shape_results",
domain="memory",
action="build_context",
@@ -137,12 +158,16 @@ async def to_graph_context(
hierarchical_results = []
for context_item in context_result.results:
primary_result = to_summary(context_item.primary_result)
observations = [to_summary(obs) for obs in context_item.observations]
observations = [
summary
for summary in (to_summary(obs) for obs in context_item.observations)
if isinstance(summary, ObservationSummary)
]
related = [to_summary(rel) for rel in context_item.related_results]
hierarchical_results.append(
ContextResult(
primary_result=primary_result,
observations=observations, # pyright: ignore[reportArgumentType]
observations=observations,
related_results=related,
)
)
@@ -170,8 +195,10 @@ async def to_graph_context(
)
async def to_search_results(entity_service: EntityService, results: List[SearchIndexRow]):
with telemetry.scope(
async def to_search_results(
entity_service: EntityServiceBatchLookup, results: List[SearchIndexRow]
) -> list[SearchResult]:
with logfire.span(
"search.hydrate_results",
domain="search",
action="search",
@@ -187,8 +214,8 @@ async def to_search_results(entity_service: EntityService, results: List[SearchI
all_entity_ids.add(eid)
# Single batch fetch for all entities
entities_by_id: dict[int, EntityModel] = {}
with telemetry.scope(
entities_by_id: dict[int, Any] = {}
with logfire.span(
"search.hydrate_results.fetch_entities",
domain="search",
action="search",
@@ -200,7 +227,7 @@ async def to_search_results(entity_service: EntityService, results: List[SearchI
entities_by_id = {e.id: e for e in entities}
search_results = []
with telemetry.scope(
with logfire.span(
"search.hydrate_results.shape_results",
domain="search",
action="search",
@@ -222,20 +249,20 @@ async def to_search_results(entity_service: EntityService, results: List[SearchI
entity_id = result.entity_id
# Look up entities by their specific IDs
parent_entity = entities_by_id.get(result.entity_id) if result.entity_id else None # pyright: ignore
from_entity = entities_by_id.get(result.from_id) if result.from_id else None # pyright: ignore
parent_entity = entities_by_id.get(result.entity_id) if result.entity_id else None
from_entity = entities_by_id.get(result.from_id) if result.from_id else None
to_entity = entities_by_id.get(result.to_id) if result.to_id else None
search_results.append(
SearchResult(
title=result.title, # pyright: ignore
type=result.type, # pyright: ignore
title=_required_str(result.title, "title"),
type=_search_item_type(result.type),
permalink=result.permalink,
score=result.score, # pyright: ignore
score=result.score if result.score is not None else 0.0,
entity=parent_entity.permalink if parent_entity else None,
content=result.content,
matched_chunk=result.matched_chunk_text,
file_path=result.file_path,
file_path=_required_str(result.file_path, "file_path"),
metadata=result.metadata,
entity_id=entity_id,
observation_id=observation_id,
+2 -2
View File
@@ -12,7 +12,7 @@ from basic_memory.cli.auto_update import maybe_run_periodic_auto_update # noqa:
from basic_memory.cli.container import CliContainer, set_container # noqa: E402
from basic_memory.cli.promo import maybe_show_cloud_promo, maybe_show_init_line # noqa: E402
from basic_memory.config import init_cli_logging # noqa: E402
from basic_memory import telemetry # noqa: E402
import logfire # noqa: E402
def version_callback(value: bool) -> None:
@@ -45,7 +45,7 @@ def app_callback(
init_cli_logging()
command_name = ctx.invoked_subcommand or "root"
ctx.with_resource(
telemetry.operation(
logfire.span(
f"cli.command.{command_name}",
entrypoint="cli",
command_name=command_name,
+1 -1
View File
@@ -22,7 +22,7 @@ PACKAGE_NAME = "basic-memory"
PYPI_JSON_URL = "https://pypi.org/pypi/basic-memory/json"
PYPI_TIMEOUT_SECONDS = 5
BREW_OUTDATED_TIMEOUT_SECONDS = 15
BREW_OUTDATED_TIMEOUT_SECONDS = 60
UV_UPGRADE_TIMEOUT_SECONDS = 180
BREW_UPGRADE_TIMEOUT_SECONDS = 600
@@ -99,41 +99,39 @@ async def make_api_request(
response = await client.request(method=method, url=url, headers=headers, json=json_data)
response.raise_for_status()
return response
except httpx.HTTPStatusError as e:
response = e.response
# Try to parse error detail from response
error_detail = None
try:
error_detail = response.json()
except Exception:
# If JSON parsing fails, we'll handle it as a generic error
pass
# Check for subscription_required error (403)
if response.status_code == 403 and isinstance(error_detail, dict):
# Handle both FastAPI HTTPException format (nested under "detail")
# and direct format
detail_obj = error_detail.get("detail", error_detail)
if (
isinstance(detail_obj, dict)
and detail_obj.get("error") == "subscription_required"
):
message = detail_obj.get("message", "Active subscription required")
subscribe_url = detail_obj.get(
"subscribe_url", "https://basicmemory.com/subscribe"
)
raise SubscriptionRequiredError(
message=message, subscribe_url=subscribe_url
) from e
# Raise generic CloudAPIError with status code and detail
raise CloudAPIError(
f"API request failed: {e}",
status_code=response.status_code,
detail=error_detail if isinstance(error_detail, dict) else {},
) from e
except httpx.HTTPError as e:
# Check if this is a response error with response details
if hasattr(e, "response") and e.response is not None: # pyright: ignore [reportAttributeAccessIssue]
response = e.response # type: ignore
# Try to parse error detail from response
error_detail = None
try:
error_detail = response.json()
except Exception:
# If JSON parsing fails, we'll handle it as a generic error
pass
# Check for subscription_required error (403)
if response.status_code == 403 and isinstance(error_detail, dict):
# Handle both FastAPI HTTPException format (nested under "detail")
# and direct format
detail_obj = error_detail.get("detail", error_detail)
if (
isinstance(detail_obj, dict)
and detail_obj.get("error") == "subscription_required"
):
message = detail_obj.get("message", "Active subscription required")
subscribe_url = detail_obj.get(
"subscribe_url", "https://basicmemory.com/subscribe"
)
raise SubscriptionRequiredError(
message=message, subscribe_url=subscribe_url
) from e
# Raise generic CloudAPIError with status code and detail
raise CloudAPIError(
f"API request failed: {e}",
status_code=response.status_code,
detail=error_detail if isinstance(error_detail, dict) else {},
) from e
raise CloudAPIError(f"API request failed: {e}") from e
@@ -41,6 +41,10 @@ async def fetch_cloud_projects(
) -> CloudProjectList:
"""Fetch list of projects from cloud API.
Args:
project_name: Optional project name for workspace resolution
workspace: Cloud workspace tenant_id to list projects from
Returns:
CloudProjectList with projects from cloud
"""
@@ -112,12 +116,12 @@ async def sync_project(project_name: str, force_full: bool = False) -> None:
Args:
project_name: Name of project to sync
force_full: If True, force a full scan bypassing watermark optimization
force_full: ignored, kept for backwards compatibility
"""
try:
from basic_memory.cli.commands.command_utils import run_sync
await run_sync(project=project_name, force_full=force_full)
await run_sync(project=project_name)
except Exception as e:
raise CloudUtilsError(f"Failed to sync project '{project_name}': {e}") from e
@@ -76,10 +76,22 @@ def login():
@cloud_app.command()
def logout():
"""Remove stored OAuth tokens."""
config = ConfigManager().config
"""Remove stored OAuth tokens and clear cached workspace selection."""
config_manager = ConfigManager()
config = config_manager.config
auth = CLIAuth(client_id=config.cloud_client_id, authkit_domain=config.cloud_domain)
auth.logout()
# Trigger: ending a session must invalidate the cached workspace.
# Why: a follow-up `bm cloud login` (often as a different user, or returning
# from an org workspace to personal) inherits the previous selection
# and silently routes everything through the wrong tenant. See #755.
# Outcome: re-login starts from a clean slate; the user picks again via
# `bm cloud workspace set-default` or per-project --workspace.
if config.default_workspace is not None:
config.default_workspace = None
config_manager.save_config(config)
console.print("[dim]API key (if configured) remains available for cloud project routing.[/dim]")
@@ -167,7 +179,7 @@ def setup() -> None:
console.print("1. Add a project with local sync path:")
console.print(" bm project add research --cloud --local-path ~/Documents/research")
console.print("\n Or configure sync for an existing project:")
console.print(" bm project sync-setup research ~/Documents/research")
console.print(" bm cloud sync-setup research ~/Documents/research")
console.print("\n2. Preview the initial sync (recommended):")
console.print(" bm project bisync --name research --resync --dry-run")
console.print("\n3. If all looks good, run the actual sync:")
@@ -124,22 +124,6 @@ def sync_project_command(
if success:
console.print(f"[green]{name} synced successfully[/green]")
# Trigger database sync if not a dry run
if not dry_run:
async def _trigger_db_sync():
async with get_client(project_name=name) as client:
return await ProjectClient(client).sync(
project_data.external_id, force_full=False
)
try:
with force_routing(cloud=True):
result = run_with_cleanup(_trigger_db_sync())
console.print(f"[dim]Database sync initiated: {result.get('message')}[/dim]")
except Exception as e:
console.print(f"[yellow]Warning: Could not trigger database sync: {e}[/yellow]")
else:
console.print(f"[red]{name} sync failed[/red]")
raise typer.Exit(1)
@@ -202,22 +186,6 @@ def bisync_project_command(
sync_entry.last_sync = datetime.now()
sync_entry.bisync_initialized = True
ConfigManager().save_config(config)
# Trigger database sync if not a dry run
if not dry_run:
async def _trigger_db_sync():
async with get_client(project_name=name) as client:
return await ProjectClient(client).sync(
project_data.external_id, force_full=False
)
try:
with force_routing(cloud=True):
result = run_with_cleanup(_trigger_db_sync())
console.print(f"[dim]Database sync initiated: {result.get('message')}[/dim]")
except Exception as e:
console.print(f"[yellow]Warning: Could not trigger database sync: {e}[/yellow]")
else:
console.print(f"[red]{name} bisync failed[/red]")
raise typer.Exit(1)
@@ -20,6 +20,7 @@ from loguru import logger
from rich.console import Console
from basic_memory.cli.commands.cloud.rclone_installer import is_rclone_installed
from basic_memory.config import resolve_data_dir
from basic_memory.utils import normalize_project_path
console = Console()
@@ -138,13 +139,16 @@ def get_bmignore_filter_path() -> Path:
def get_project_bisync_state(project_name: str) -> Path:
"""Get path to project's bisync state directory.
Honors ``BASIC_MEMORY_CONFIG_DIR`` so isolated instances each keep their
own bisync state alongside their config.
Args:
project_name: Name of the project
Returns:
Path to bisync state directory for this project
"""
return Path.home() / ".basic-memory" / "bisync-state" / project_name
return resolve_data_dir() / "bisync-state" / project_name
def bisync_initialized(project_name: str) -> bool:
@@ -106,6 +106,8 @@ def upload(
raise typer.Exit(1)
# Perform upload (or dry run)
if resolved_workspace:
console.print(f"[dim]Using workspace: {resolved_workspace}[/dim]")
if dry_run:
console.print(
f"[yellow]DRY RUN: Showing what would be uploaded to '{project}'[/yellow]"
@@ -140,7 +142,7 @@ def upload(
if sync and not dry_run:
console.print(f"[blue]Syncing project '{project}'...[/blue]")
try:
await sync_project(project, force_full=True)
await sync_project(project)
except Exception as e:
console.print(f"[yellow]Warning: Sync failed: {e}[/yellow]")
console.print("[dim]Files uploaded but may not be indexed yet[/dim]")
+234 -14
View File
@@ -1,7 +1,10 @@
"""Database management commands."""
from pathlib import Path
import os
from dataclasses import dataclass
from pathlib import Path, PurePosixPath, PureWindowsPath
import psutil
import typer
from loguru import logger
from rich.console import Console
@@ -12,6 +15,7 @@ from basic_memory import db
from basic_memory.cli.app import app
from basic_memory.cli.commands.command_utils import run_with_cleanup
from basic_memory.config import ConfigManager, ProjectMode
from basic_memory.indexing import IndexProgress
from basic_memory.repository import ProjectRepository
from basic_memory.services.initialization import reconcile_projects_with_config
from basic_memory.sync.sync_service import get_sync_service
@@ -19,6 +23,136 @@ from basic_memory.sync.sync_service import get_sync_service
console = Console()
def _is_basic_memory_mcp(cmdline: list[str]) -> bool:
"""Heuristic: does this argv represent a `basic-memory mcp` server?
The MCP server can be launched any of:
basic-memory mcp
bm mcp # entrypoint alias from pyproject.toml
python -m basic_memory.cli.main mcp # module form
uv run basic-memory mcp / uv run bm mcp # uv wrappers
/abs/path/to/{bm,basic-memory}[.exe] mcp
A reliable match needs both signals:
1. "mcp" appears as an exact argv token (not "mcp-foo").
2. Some argv token names the basic-memory entrypoint either by
hyphen/underscore form, or as a `bm` script (covers `/usr/local/bin/bm`,
`bm.exe`, etc. via Path.stem).
"""
if "mcp" not in cmdline:
return False
for arg in cmdline:
if "basic-memory" in arg or "basic_memory" in arg:
return True
# Try both POSIX and Windows path interpretations so a test on
# macOS still recognizes `C:\\...\\bm.exe`, and a real Windows
# run still recognizes `/usr/local/bin/bm`. Path() alone uses
# the host OS, which gives wrong stems for foreign separators.
if PurePosixPath(arg).stem == "bm" or PureWindowsPath(arg).stem == "bm":
return True
return False
def _find_live_mcp_processes() -> list[tuple[int, str]]:
"""Return (pid, joined_cmdline) for live `basic-memory mcp` processes.
Why this exists (issue #765):
On POSIX, `Path.unlink()` removes the directory entry but the inode
survives as long as any process holds the file open. A `bm reset`
run while Claude Desktop (or another MCP client) is alive will
therefore "succeed" but the still-running MCP keeps reading the
old, now-invisible memory.db inode and returns phantom rows. On
Windows the OS naturally raises PermissionError on `unlink()`, so
the bug is POSIX-specific. We detect proactively to give the same
error experience on every platform before doing damage.
The current process is excluded so this can be called from inside a
`bm reset` invocation. NoSuchProcess / AccessDenied are swallowed
because process tables race with the scan and we don't want a
transient permission error to mask a real zombie.
"""
me = os.getpid()
matches: list[tuple[int, str]] = []
for proc in psutil.process_iter(["pid", "cmdline"]):
try:
pid = proc.info.get("pid")
if pid is None or pid == me:
continue
cmdline = proc.info.get("cmdline") or []
if not cmdline:
continue
if _is_basic_memory_mcp(cmdline):
matches.append((pid, " ".join(cmdline)))
except (psutil.NoSuchProcess, psutil.AccessDenied):
continue
return matches
def _abort_if_mcp_processes_alive() -> None:
"""Refuse `bm reset` while basic-memory MCP processes are still running.
See _find_live_mcp_processes for the underlying POSIX-vs-Windows
rationale. Prints a per-PID list and platform-appropriate cleanup
instructions, then exits non-zero so destructive work never starts.
"""
zombies = _find_live_mcp_processes()
if not zombies:
return
console.print("[red]Refusing to reset:[/red] basic-memory MCP processes are still running.")
console.print(
"[yellow]On macOS/Linux these would keep reading the deleted memory.db inode "
"and return phantom search results (see #765).[/yellow]"
)
for pid, cmd in zombies:
console.print(f" PID {pid}: {cmd}")
console.print("\n[bold]How to clean up:[/bold]")
console.print(" 1. Quit Claude Desktop and any other MCP clients.")
if os.name == "nt":
console.print(
" 2. Verify nothing remains: "
"[green]Get-CimInstance Win32_Process | "
"Where-Object {$_.CommandLine -like '*basic-memory*mcp*'}[/green]"
)
else:
console.print(" 2. Verify nothing remains: [green]pgrep -fa 'basic-memory mcp'[/green]")
console.print(" 3. Re-run [green]bm reset[/green].")
raise typer.Exit(1)
@dataclass(slots=True)
class EmbeddingProgress:
"""Typed CLI progress payload for embedding backfills."""
entity_id: int
completed: int
total: int
def _format_eta(seconds: float | None) -> str:
"""Render a compact ETA string for CLI progress descriptions."""
if seconds is None:
return "--:--"
whole_seconds = max(int(seconds), 0)
minutes, remaining_seconds = divmod(whole_seconds, 60)
hours, remaining_minutes = divmod(minutes, 60)
if hours:
return f"{hours:d}:{remaining_minutes:02d}:{remaining_seconds:02d}"
return f"{remaining_minutes:02d}:{remaining_seconds:02d}"
def _format_index_progress(progress: IndexProgress) -> str:
"""Render typed index progress as a compact Rich task description."""
files_per_minute = int(progress.files_per_minute) if progress.files_per_minute else 0
return (
" Indexing files... "
f"{progress.files_processed}/{progress.files_total} files | "
f"{progress.batches_completed}/{progress.batches_total} batches | "
f"{files_per_minute}/min | ETA {_format_eta(progress.eta_seconds)}"
)
async def _reindex_projects(app_config):
"""Reindex all projects in a single async context.
@@ -51,6 +185,16 @@ async def _reindex_projects(app_config):
@app.command()
def reset(
reindex: bool = typer.Option(False, "--reindex", help="Rebuild db index from filesystem"),
force: bool = typer.Option(
False,
"--force",
help=(
"Skip the pre-flight check that refuses to reset while "
"basic-memory MCP processes are running. Use only in "
"automated workflows where you've already ensured no MCP "
"clients are attached to the database."
),
),
): # pragma: no cover
"""Reset database (drop all tables and recreate)."""
console.print(
@@ -59,6 +203,14 @@ def reset(
"Use [green]bm reset --reindex[/green] to automatically rebuild the index afterward."
)
if typer.confirm("Reset the database index?"):
# Pre-flight: refuse to proceed if MCP processes still hold the DB
# file open. POSIX would silently let us unlink the inode while
# they keep reading it; Windows would error here anyway. See
# _find_live_mcp_processes for the full story. --force is the
# documented escape hatch for scripted/CI runs.
if not force:
_abort_if_mcp_processes_alive()
logger.info("Resetting database...")
config_manager = ConfigManager()
app_config = config_manager.config
@@ -112,20 +264,30 @@ def reindex(
False, "--embeddings", "-e", help="Rebuild vector embeddings (requires semantic search)"
),
search: bool = typer.Option(False, "--search", "-s", help="Rebuild full-text search index"),
full: bool = typer.Option(
False,
"--full",
help="Force a full filesystem scan and file reindex instead of the default incremental scan",
),
project: str = typer.Option(
None, "--project", "-p", help="Reindex a specific project (default: all)"
),
): # pragma: no cover
"""Rebuild search indexes and/or vector embeddings without dropping the database.
By default rebuilds everything (search + embeddings if semantic is enabled).
Use --search or --embeddings to rebuild only one.
By default runs incremental search + embeddings (if semantic search is enabled).
Use --full to bypass incremental scan optimization, rebuild all file-backed search rows,
and re-embed all eligible notes.
Use --search or --embeddings to rebuild only one side.
Examples:
bm reindex # Rebuild everything
bm reindex # Incremental search + embeddings
bm reindex --full # Full search + full re-embed
bm reindex --embeddings # Only rebuild vector embeddings
bm reindex --search # Only rebuild FTS index
bm reindex -p claw # Reindex only the 'claw' project
bm reindex --full --search # Full search only
bm reindex --full --embeddings # Full re-embed only
bm reindex -p claw --full # Full reindex for only the 'claw' project
"""
# If neither flag is set, do both
if not embeddings and not search:
@@ -144,10 +306,19 @@ def reindex(
if not search:
raise typer.Exit(0)
run_with_cleanup(_reindex(app_config, search=search, embeddings=embeddings, project=project))
run_with_cleanup(
_reindex(app_config, search=search, embeddings=embeddings, full=full, project=project)
)
async def _reindex(app_config, search: bool, embeddings: bool, project: str | None):
async def _reindex(
app_config,
*,
search: bool,
embeddings: bool,
full: bool,
project: str | None,
):
"""Run reindex operations."""
from basic_memory.repository import EntityRepository
from basic_memory.repository.search_repository import create_search_repository
@@ -185,14 +356,47 @@ async def _reindex(app_config, search: bool, embeddings: bool, project: str | No
console.print(f"\n[bold]Project: [cyan]{proj.name}[/cyan][/bold]")
if search:
console.print(" Rebuilding full-text search index...")
search_mode_label = "full scan" if full else "incremental scan"
console.print(
f" Rebuilding full-text search index ([cyan]{search_mode_label}[/cyan])..."
)
sync_service = await get_sync_service(proj)
sync_dir = Path(proj.path)
await sync_service.sync(sync_dir, project_name=proj.name)
console.print(" [green]✓[/green] Full-text search index rebuilt")
with Progress(
SpinnerColumn(),
TextColumn("[progress.description]{task.description}"),
BarColumn(),
TaskProgressColumn(),
console=console,
) as progress:
task = progress.add_task(" Indexing files... scanning changes", total=1)
async def on_index_progress(update: IndexProgress) -> None:
total = update.files_total or 1
completed = update.files_processed if update.files_total else 1
progress.update(
task,
description=_format_index_progress(update),
total=total,
completed=min(completed, total),
)
await sync_service.sync(
sync_dir,
project_name=proj.name,
force_full=full,
sync_embeddings=False,
progress_callback=on_index_progress,
)
progress.update(task, completed=progress.tasks[task].total or 1)
console.print(" [green]done[/green] Full-text search index rebuilt")
if embeddings:
console.print(" Building vector embeddings...")
embedding_mode_label = "full rebuild" if full else "incremental sync"
console.print(
f" Building vector embeddings ([cyan]{embedding_mode_label}[/cyan])..."
)
entity_repository = EntityRepository(session_maker, project_id=proj.id)
search_repository = create_search_repository(
session_maker, project_id=proj.id, app_config=app_config
@@ -213,13 +417,29 @@ async def _reindex(app_config, search: bool, embeddings: bool, project: str | No
task = progress.add_task(" Embedding entities...", total=None)
def on_progress(entity_id, index, total):
progress.update(task, total=total, completed=index)
embedding_progress = EmbeddingProgress(
entity_id=entity_id,
completed=index,
total=total,
)
# Trigger: repository progress now reports terminal entity completion.
# Why: operators need to see finished embedding work rather than
# entities merely entering prepare.
# Outcome: the CLI bar advances steadily with real completed work.
progress.update(
task,
total=embedding_progress.total,
completed=embedding_progress.completed,
)
stats = await search_service.reindex_vectors(progress_callback=on_progress)
stats = await search_service.reindex_vectors(
progress_callback=on_progress,
force_full=full,
)
progress.update(task, completed=stats["total_entities"])
console.print(
f" [green][/green] Embeddings complete: "
f" [green]done[/green] Embeddings complete: "
f"{stats['embedded']} entities embedded, "
f"{stats['skipped']} skipped, "
f"{stats['errors']} errors"
+2 -2
View File
@@ -69,7 +69,7 @@ async def run_doctor() -> None:
content=f"# {api_note_title}\n\n- [note] API to file check",
entity_metadata={"tags": ["doctor"]},
)
api_result = await knowledge_client.create_entity(api_note.model_dump(), fast=False)
api_result = await knowledge_client.create_entity(api_note.model_dump())
api_file = project_path / api_result.file_path
if not api_file.exists():
@@ -101,7 +101,7 @@ async def run_doctor() -> None:
console.print("[green]OK[/green] Manual file written")
sync_data = await project_client.sync(
project_id, force_full=True, run_in_background=False
project_id, force_full=False, run_in_background=False
)
sync_report = SyncReportResponse.model_validate(sync_data)
if sync_report.total == 0:
+366 -9
View File
@@ -14,6 +14,7 @@ from rich.text import Text
from basic_memory.cli.app import app
from basic_memory.cli.auth import CLIAuth
from basic_memory.cli.commands.cloud.api_client import CloudAPIError, make_api_request
from basic_memory.cli.commands.cloud.bisync_commands import get_mount_info
from basic_memory.cli.commands.cloud.project_sync import (
_has_cloud_credentials,
@@ -25,10 +26,14 @@ from basic_memory.cli.commands.cloud.rclone_commands import (
)
from basic_memory.cli.commands.command_utils import get_project_info, run_with_cleanup
from basic_memory.cli.commands.routing import force_routing, validate_routing_flags
from basic_memory.config import ConfigManager, ProjectEntry, ProjectMode
from basic_memory.mcp.async_client import get_client
from basic_memory.config import BasicMemoryConfig, ConfigManager, ProjectEntry, ProjectMode
from basic_memory.mcp.async_client import get_client, resolve_configured_workspace
from basic_memory.mcp.clients import ProjectClient
from basic_memory.schemas.cloud import ProjectVisibility
from basic_memory.schemas.cloud import (
CloudProjectIndexStatus,
CloudTenantIndexStatusResponse,
ProjectVisibility,
)
from basic_memory.schemas.project_info import ProjectItem, ProjectList
from basic_memory.utils import generate_permalink, normalize_project_path
@@ -58,6 +63,211 @@ def make_bar(value: int, max_value: int, width: int = 40) -> Text:
return bar
def _uses_cloud_project_info_route(project_name: str, *, local: bool, cloud: bool) -> bool:
"""Return whether project info should attempt cloud augmentation."""
if local:
return False
if cloud:
return True
config_manager = ConfigManager()
resolved_name, _ = config_manager.get_project(project_name)
effective_name = resolved_name or project_name
return config_manager.config.get_project_mode(effective_name) == ProjectMode.CLOUD
def _resolve_cloud_status_workspace_id(project_name: str) -> str:
"""Resolve the tenant/workspace for cloud index status lookup."""
config_manager = ConfigManager()
config = config_manager.config
if not _has_cloud_credentials(config):
raise RuntimeError(
"Cloud credentials not found. Run `bm cloud api-key save <key>` or `bm cloud login` first."
)
configured_name, _ = config_manager.get_project(project_name)
effective_name = configured_name or project_name
workspace_id = resolve_configured_workspace(config=config, project_name=effective_name)
if workspace_id is not None:
return workspace_id
workspace_id = _resolve_workspace_id(config, None)
if workspace_id is not None:
return workspace_id
raise RuntimeError(
f"Cloud workspace could not be resolved for project '{effective_name}'. "
"Set a project workspace with `bm project set-cloud --workspace ...` or configure a "
"default workspace with `bm cloud workspace set-default ...`."
)
async def _resolve_cloud_status_workspace_id_async(project_name: str) -> str:
"""Resolve the tenant/workspace for cloud index status lookup in async contexts."""
config_manager = ConfigManager()
config = config_manager.config
if not _has_cloud_credentials(config):
raise RuntimeError(
"Cloud credentials not found. Run `bm cloud api-key save <key>` or `bm cloud login` first."
)
configured_name, _ = config_manager.get_project(project_name)
effective_name = configured_name or project_name
workspace_id = resolve_configured_workspace(config=config, project_name=effective_name)
if workspace_id is not None:
return workspace_id
from basic_memory.mcp.project_context import get_available_workspaces
workspaces = await get_available_workspaces()
if len(workspaces) == 1:
return workspaces[0].tenant_id
raise RuntimeError(
f"Cloud workspace could not be resolved for project '{effective_name}'. "
"Set a project workspace with `bm project set-cloud --workspace ...` or configure a "
"default workspace with `bm cloud workspace set-default ...`."
)
def _match_cloud_index_status_project(
project_name: str, projects: list[CloudProjectIndexStatus]
) -> CloudProjectIndexStatus | None:
"""Match the requested project against the tenant index-status payload."""
exact_match = next(
(project for project in projects if project.project_name == project_name), None
)
if exact_match is not None:
return exact_match
project_permalink = generate_permalink(project_name)
permalink_matches = [
project
for project in projects
if generate_permalink(project.project_name) == project_permalink
]
if len(permalink_matches) == 1:
return permalink_matches[0]
return None
def _format_cloud_index_status_error(error: Exception) -> str:
"""Convert cloud lookup failures into concise user-facing text."""
if isinstance(error, CloudAPIError):
detail_message: str | None = None
detail = error.detail.get("detail")
if isinstance(detail, str):
detail_message = detail
elif isinstance(detail, dict):
if isinstance(detail.get("message"), str):
detail_message = detail["message"]
elif isinstance(detail.get("detail"), str):
detail_message = detail["detail"]
if error.status_code and detail_message:
return f"HTTP {error.status_code}: {detail_message}"
if error.status_code:
return f"HTTP {error.status_code}"
return str(error)
async def _fetch_cloud_project_index_status(project_name: str) -> CloudProjectIndexStatus:
"""Fetch cloud index freshness for one project from the admin tenant endpoint."""
workspace_id = await _resolve_cloud_status_workspace_id_async(project_name)
host_url = ConfigManager().config.cloud_host.rstrip("/")
try:
response = await make_api_request(
method="GET",
url=f"{host_url}/admin/tenants/{workspace_id}/index-status",
)
except typer.Exit as exc:
if exc.exit_code not in (None, 0):
raise RuntimeError(
"Cloud credentials not found. Run `bm cloud api-key save <key>` or "
"`bm cloud login` first."
) from exc
raise
tenant_status = CloudTenantIndexStatusResponse.model_validate(response.json())
if tenant_status.error:
raise RuntimeError(tenant_status.error)
project_status = _match_cloud_index_status_project(project_name, tenant_status.projects)
if project_status is None:
raise RuntimeError(
f"Project '{project_name}' was not found in workspace index status "
f"for tenant '{workspace_id}'."
)
return project_status
def _load_cloud_project_index_status(
project_name: str,
) -> tuple[CloudProjectIndexStatus | None, str | None]:
"""Best-effort wrapper around the cloud index freshness lookup."""
try:
return run_with_cleanup(_fetch_cloud_project_index_status(project_name)), None
except Exception as exc:
return None, _format_cloud_index_status_error(exc)
def _build_cloud_index_status_section(
cloud_index_status: CloudProjectIndexStatus | None,
cloud_index_status_error: str | None,
) -> Table | None:
"""Render the optional Cloud Index Status block for rich project info."""
if cloud_index_status is None and cloud_index_status_error is None:
return None
table = Table.grid(padding=(0, 2))
table.add_column("property", style="cyan")
table.add_column("value", style="green")
table.add_row("[bold]Cloud Index Status[/bold]", "")
if cloud_index_status_error is not None:
table.add_row("[yellow]●[/yellow] Warning", f"[yellow]{cloud_index_status_error}[/yellow]")
return table
if cloud_index_status is None:
return table
table.add_row("Files", str(cloud_index_status.current_file_count))
table.add_row(
"Note content",
f"{cloud_index_status.note_content_synced}/{cloud_index_status.current_file_count}",
)
table.add_row(
"Search",
f"{cloud_index_status.total_indexed_entities}/{cloud_index_status.current_file_count}",
)
table.add_row("Embeddable", str(cloud_index_status.embeddable_indexed_entities))
table.add_row(
"Vectorized",
(
f"{cloud_index_status.total_entities_with_chunks}/"
f"{cloud_index_status.embeddable_indexed_entities}"
),
)
if cloud_index_status.reindex_recommended:
table.add_row("[yellow]●[/yellow] Status", "[yellow]Reindex recommended[/yellow]")
if cloud_index_status.reindex_reason:
table.add_row("Reason", f"[yellow]{cloud_index_status.reindex_reason}[/yellow]")
else:
table.add_row("[green]●[/green] Status", "[green]Up to date[/green]")
return table
def _normalize_project_visibility(visibility: str | None) -> ProjectVisibility:
"""Normalize CLI visibility input to the cloud API contract."""
if visibility is None:
@@ -253,6 +463,12 @@ def list_projects(
if cloud_project is not None and cloud_ws_name:
ws_label = f"{cloud_ws_name} ({cloud_ws_type})" if cloud_ws_type else cloud_ws_name
# display_name is a human label for private UUID-named projects (e.g., "My Project").
# Keep "name" as the canonical identifier for scripting/JSON consumers;
# the Rich table uses display_name when available.
display_name = (
cloud_project.display_name if cloud_project and cloud_project.display_name else None
)
row_data = {
"name": project_name,
"permalink": permalink,
@@ -263,6 +479,8 @@ def list_projects(
"sync": has_sync,
"is_default": is_default,
}
if display_name:
row_data["display_name"] = display_name
if ws_label:
row_data["workspace"] = cloud_ws_name or ""
if cloud_ws_type:
@@ -278,7 +496,7 @@ def list_projects(
# --- Rich table output ---
for row_data in project_rows:
table.add_row(
row_data["name"],
row_data.get("display_name") or row_data["name"],
row_data["local_path"],
row_data["cloud_path"],
row_data.get("workspace", "")
@@ -620,6 +838,76 @@ def move_project(
raise typer.Exit(1)
async def _detach_local_project_row(app_config: BasicMemoryConfig, name: str) -> bool:
"""Drop the project's row from the local index DB.
Trigger: `bm project set-cloud` is making a project cloud-only.
Why: the local row is what causes `_merge_projects` to report
`source: "local+cloud"` after the toggle (#680). Removing it
forces the merged listing to honor the user's chosen mode.
Outcome: returns True if a row was deleted, False if there was
nothing to clean up. On-disk note files are not touched.
"""
from basic_memory import db
from basic_memory.repository import ProjectRepository
_, session_maker = await db.get_or_create_db(
db_path=app_config.database_path,
db_type=db.DatabaseType.FILESYSTEM,
)
try:
repo = ProjectRepository(session_maker)
existing = await repo.get_by_name(name)
if existing is None:
return False
await repo.delete(existing.id)
return True
finally:
# CLI-only: safe to tear down the global DB singleton here since
# set-cloud/set-local never run inside a long-lived MCP/API server.
await db.shutdown_db()
async def _attach_local_project_row(app_config: BasicMemoryConfig, name: str, path: str) -> None:
"""Ensure the project has a row in the local index DB at the given path.
Trigger: `bm project set-local` is making a previously cloud-only
project local again.
Why: without a row in the local DB, every local-side tool (`list`,
`info`, sync, indexing) would skip this project.
Outcome: a row is created if missing, or its path is updated to match
the new local home if it already exists. On-disk files are not
touched the caller is responsible for ensuring the directory
exists.
"""
from basic_memory import db
from basic_memory.repository import ProjectRepository
_, session_maker = await db.get_or_create_db(
db_path=app_config.database_path,
db_type=db.DatabaseType.FILESYSTEM,
)
try:
repo = ProjectRepository(session_maker)
existing = await repo.get_by_name(name)
if existing is None:
await repo.create(
{
"name": name,
"path": path,
"permalink": generate_permalink(name),
"is_active": True,
}
)
return
if existing.path != path:
await repo.update_path(existing.id, path)
finally:
# CLI-only: safe to tear down the global DB singleton here since
# set-cloud/set-local never run inside a long-lived MCP/API server.
await db.shutdown_db()
@project_app.command("set-cloud")
def set_cloud(
name: str = typer.Argument(..., help="Name of the project to route through cloud"),
@@ -637,6 +925,13 @@ def set_cloud(
If omitted, uses the default workspace (if set) or auto-selects when
only one workspace is available.
This is a one-way cutover: the project's row in the local index DB is
removed and the local path in config is cleared so the project's
configured state is purely cloud. On-disk note files are preserved
the caller can keep, archive, or delete them as they see fit. To
return to local mode use `bm project set-local <name> --local-path
<path>`.
Examples:
bm project set-cloud research --workspace Personal
bm project set-cloud research --workspace 11111111-...
@@ -665,27 +960,52 @@ def set_cloud(
resolved_workspace_id = _resolve_workspace_id(config, workspace)
# Drop the local DB row first so the user-visible state stays consistent
# even if the config save below raises for some reason. Idempotent: a
# second `set-cloud` simply finds no row and returns False.
previous_path = config.projects[name].path
detached = run_with_cleanup(_detach_local_project_row(config, name))
config.set_project_mode(name, ProjectMode.CLOUD)
if resolved_workspace_id:
config.projects[name].workspace_id = resolved_workspace_id
# Clear local path: source-of-truth for this project is now the cloud
config.projects[name].path = ""
config_manager.save_config(config)
console.print(f"[green]Project '{name}' set to cloud mode[/green]")
if resolved_workspace_id:
console.print(f"[dim]Workspace: {resolved_workspace_id}[/dim]")
if detached and previous_path:
console.print(
f"[dim]Local index entry removed. Files at {previous_path} are preserved on disk.[/dim]"
)
console.print("[dim]MCP tools and CLI commands for this project will route through cloud[/dim]")
@project_app.command("set-local")
def set_local(
name: str = typer.Argument(..., help="Name of the project to revert to local mode"),
local_path: str = typer.Option(
None,
"--local-path",
help=(
"Local filesystem path for this project. Required unless the project "
"was previously local and its prior path is still in config."
),
),
) -> None:
"""Revert a project to local mode (use in-process ASGI transport).
Clears any associated cloud workspace.
Recreates the project's row in the local index DB and clears any
associated cloud workspace. If the project was previously local and
its prior path is still in config (e.g. an older version that didn't
blank `path` on `set-cloud`), `--local-path` may be omitted and that
path will be reused.
Example:
bm project set-local research
Examples:
bm project set-local research --local-path ~/Documents/research
bm project set-local research # reuse prior path
"""
config_manager = ConfigManager()
config = config_manager.config
@@ -695,11 +1015,31 @@ def set_local(
console.print(f"[red]Error: Project '{name}' not found in config[/red]")
raise typer.Exit(1)
entry = config.projects[name]
candidate = local_path or entry.path
if not candidate:
console.print(
f"[red]Error: --local-path is required for '{name}' "
"(no previous local path is recorded)[/red]"
)
raise typer.Exit(1)
resolved_path = Path(os.path.abspath(os.path.expanduser(candidate))).as_posix()
# Recreate the local DB row. Idempotent: if the row exists with the
# same path it's a no-op; if it exists at a stale path the path is
# updated. The directory itself is not auto-created — the user is
# expected to know whether they want to start a fresh project tree
# or point at an existing one.
run_with_cleanup(_attach_local_project_row(config, name, resolved_path))
config.set_project_mode(name, ProjectMode.LOCAL)
config.projects[name].workspace_id = None
config.projects[name].path = resolved_path
config_manager.save_config(config)
console.print(f"[green]Project '{name}' set to local mode[/green]")
console.print(f"[dim]Path: {resolved_path}[/dim]")
console.print("[dim]MCP tools and CLI commands for this project will use local transport[/dim]")
@@ -848,9 +1188,20 @@ def display_project_info(
with force_routing(local=local, cloud=cloud):
info = run_with_cleanup(get_project_info(name))
cloud_index_status: CloudProjectIndexStatus | None = None
cloud_index_status_error: str | None = None
if _uses_cloud_project_info_route(info.project_name, local=local, cloud=cloud):
cloud_index_status, cloud_index_status_error = _load_cloud_project_index_status(
info.project_name
)
if json_output:
# Convert to JSON and print
print(json.dumps(info.model_dump(), indent=2, default=str))
output = info.model_dump()
output["cloud_index_status"] = (
cloud_index_status.model_dump() if cloud_index_status is not None else None
)
output["cloud_index_status_error"] = cloud_index_status_error
print(json.dumps(output, indent=2, default=str))
else:
# --- Left column: Knowledge Graph stats ---
left = Table.grid(padding=(0, 2))
@@ -908,6 +1259,10 @@ def display_project_info(
columns = Table.grid(padding=(0, 4), expand=False)
columns.add_row(left, right)
cloud_section = _build_cloud_index_status_section(
cloud_index_status, cloud_index_status_error
)
# --- Note Types bar chart (top 5 by count) ---
bars_section = None
if info.statistics.note_types:
@@ -946,6 +1301,8 @@ def display_project_info(
# --- Assemble dashboard ---
parts: list = [columns, ""]
if cloud_section is not None:
parts.extend([cloud_section, ""])
if bars_section:
parts.extend([bars_section, ""])
parts.append(footer)
+55 -33
View File
@@ -62,9 +62,12 @@ def write_note(
help="The project to write to. If not provided, the default project will be used."
),
] = None,
workspace: Annotated[
project_id: Annotated[
Optional[str],
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
typer.Option(
"--project-id",
help="Project external_id (UUID). Takes precedence over --project; use to disambiguate same-named projects across cloud workspaces.",
),
] = None,
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
@@ -106,7 +109,7 @@ def write_note(
content=content,
directory=folder,
project=project,
workspace=workspace,
project_id=project_id,
tags=tags,
output_format="json",
)
@@ -128,15 +131,16 @@ def read_note(
include_frontmatter: bool = typer.Option(
False, "--include-frontmatter", help="Include YAML frontmatter in output"
),
page: int = typer.Option(1, "--page", help="Page number for pagination"),
page_size: int = typer.Option(10, "--page-size", help="Number of results per page"),
project: Annotated[
Optional[str],
typer.Option(help="The project to use. If not provided, the default project will be used."),
] = None,
workspace: Annotated[
project_id: Annotated[
Optional[str],
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
typer.Option(
"--project-id",
help="Project external_id (UUID). Takes precedence over --project; use to disambiguate same-named projects across cloud workspaces.",
),
] = None,
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
@@ -149,7 +153,6 @@ def read_note(
bm tool read-note my-note
bm tool read-note my-note --include-frontmatter
bm tool read-note my-note --page 2 --page-size 5
"""
try:
validate_routing_flags(local, cloud)
@@ -159,9 +162,7 @@ def read_note(
mcp_read_note(
identifier=identifier,
project=project,
workspace=workspace,
page=page,
page_size=page_size,
project_id=project_id,
include_frontmatter=include_frontmatter,
output_format="json",
)
@@ -200,9 +201,12 @@ def edit_note(
help="The project to edit. If not provided, the default project will be used."
),
] = None,
workspace: Annotated[
project_id: Annotated[
Optional[str],
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
typer.Option(
"--project-id",
help="Project external_id (UUID). Takes precedence over --project; use to disambiguate same-named projects across cloud workspaces.",
),
] = None,
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
@@ -227,7 +231,7 @@ def edit_note(
operation=operation,
content=content,
project=project,
workspace=workspace,
project_id=project_id,
section=section,
find_text=find_text,
expected_replacements=expected_replacements,
@@ -265,9 +269,12 @@ def build_context(
Optional[str],
typer.Option(help="The project to use. If not provided, the default project will be used."),
] = None,
workspace: Annotated[
project_id: Annotated[
Optional[str],
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
typer.Option(
"--project-id",
help="Project external_id (UUID). Takes precedence over --project; use to disambiguate same-named projects across cloud workspaces.",
),
] = None,
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
@@ -289,7 +296,7 @@ def build_context(
mcp_build_context(
url=url,
project=project,
workspace=workspace,
project_id=project_id,
depth=depth,
timeframe=timeframe,
page=page,
@@ -322,9 +329,12 @@ def recent_activity(
Optional[str],
typer.Option(help="The project to use. If not provided, the default project will be used."),
] = None,
workspace: Annotated[
project_id: Annotated[
Optional[str],
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
typer.Option(
"--project-id",
help="Project external_id (UUID). Takes precedence over --project; use to disambiguate same-named projects across cloud workspaces.",
),
] = None,
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
@@ -345,13 +355,13 @@ def recent_activity(
with force_routing(local=local, cloud=cloud):
result = run_with_cleanup(
mcp_recent_activity(
type=type, # pyright: ignore[reportArgumentType]
type=type or "",
depth=depth if depth is not None else 1,
timeframe=timeframe if timeframe is not None else "7d",
page=page,
page_size=page_size,
project=project,
workspace=workspace,
project_id=project_id,
output_format="json",
)
)
@@ -413,9 +423,12 @@ def search_notes(
Optional[str],
typer.Option(help="The project to use. If not provided, the default project will be used."),
] = None,
workspace: Annotated[
project_id: Annotated[
Optional[str],
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
typer.Option(
"--project-id",
help="Project external_id (UUID). Takes precedence over --project; use to disambiguate same-named projects across cloud workspaces.",
),
] = None,
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
@@ -487,7 +500,7 @@ def search_notes(
mcp_search(
query=query or None,
project=project,
workspace=workspace,
project_id=project_id,
search_type=search_type,
output_format="json",
page=page,
@@ -597,9 +610,12 @@ def schema_validate(
Optional[str],
typer.Option(help="The project to use. If not provided, the default project will be used."),
] = None,
workspace: Annotated[
project_id: Annotated[
Optional[str],
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
typer.Option(
"--project-id",
help="Project external_id (UUID). Takes precedence over --project; use to disambiguate same-named projects across cloud workspaces.",
),
] = None,
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
@@ -634,7 +650,7 @@ def schema_validate(
note_type=note_type,
identifier=identifier,
project=project,
workspace=workspace,
project_id=project_id,
output_format="json",
)
)
@@ -665,9 +681,12 @@ def schema_infer(
Optional[str],
typer.Option(help="The project to use. If not provided, the default project will be used."),
] = None,
workspace: Annotated[
project_id: Annotated[
Optional[str],
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
typer.Option(
"--project-id",
help="Project external_id (UUID). Takes precedence over --project; use to disambiguate same-named projects across cloud workspaces.",
),
] = None,
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
@@ -691,7 +710,7 @@ def schema_infer(
note_type=note_type,
threshold=threshold,
project=project,
workspace=workspace,
project_id=project_id,
output_format="json",
)
)
@@ -719,9 +738,12 @@ def schema_diff(
Optional[str],
typer.Option(help="The project to use. If not provided, the default project will be used."),
] = None,
workspace: Annotated[
project_id: Annotated[
Optional[str],
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
typer.Option(
"--project-id",
help="Project external_id (UUID). Takes precedence over --project; use to disambiguate same-named projects across cloud workspaces.",
),
] = None,
local: bool = typer.Option(
False, "--local", help="Force local API routing (ignore cloud mode)"
@@ -743,7 +765,7 @@ def schema_diff(
mcp_schema_diff(
note_type=note_type,
project=project,
workspace=workspace,
project_id=project_id,
output_format="json",
)
)
+97 -19
View File
@@ -8,7 +8,7 @@ from dataclasses import dataclass
from datetime import datetime
from enum import Enum
from pathlib import Path
from typing import Any, Dict, Literal, Optional, List, Tuple
from typing import TYPE_CHECKING, Any, Dict, Literal, Optional, List, Tuple
from loguru import logger
from pydantic import AliasChoices, BaseModel, Field, model_validator
@@ -50,6 +50,44 @@ def _default_semantic_search_enabled() -> bool:
)
def resolve_data_dir() -> Path:
"""Resolve the Basic Memory data directory.
Single source of truth for the per-user state directory. Honors
``BASIC_MEMORY_CONFIG_DIR`` so each process/worktree can isolate config
and database state; otherwise falls back to ``<user home>/.basic-memory``.
Cross-platform: ``Path.home()`` reads ``$HOME`` on POSIX and
``%USERPROFILE%`` on Windows, so there's no need to check ``$HOME``
explicitly here.
"""
if config_dir := os.getenv("BASIC_MEMORY_CONFIG_DIR"):
return Path(config_dir)
return Path.home() / DATA_DIR_NAME
def default_fastembed_cache_dir() -> str:
"""Return the default cache directory used for FastEmbed model artifacts.
Resolution order:
1. ``FASTEMBED_CACHE_PATH`` env var honors FastEmbed's own convention
so users who already configure it through the environment keep working.
2. ``<basic-memory data dir>/fastembed_cache`` the same stable,
user-writable directory Basic Memory already uses for config and
the default SQLite database. Honors ``BASIC_MEMORY_CONFIG_DIR``.
Why not ``tempfile.gettempdir()``?
FastEmbed's own default is ``<system tmp>/fastembed_cache``, which is
ephemeral in many sandboxed MCP runtimes (e.g. Codex CLI wipes /tmp
between invocations). The model then disappears and every subsequent
ONNX load raises ``NO_SUCHFILE``. Persisting the cache under the
per-user data directory works identically on macOS, Linux, and Windows.
"""
if env_override := os.getenv("FASTEMBED_CACHE_PATH"):
return env_override
return str(resolve_data_dir() / "fastembed_cache")
@dataclass
class ProjectConfig:
"""Configuration for a specific basic-memory project."""
@@ -122,6 +160,11 @@ class ProjectEntry(BaseModel):
class BasicMemoryConfig(BaseSettings):
"""Pydantic model for Basic Memory global configuration."""
if TYPE_CHECKING:
# Pydantic accepts raw constructor data and validates/coerces it at runtime.
# Model attributes remain strongly typed after initialization.
def __init__(self, **data: Any) -> None: ...
env: Environment = Field(default="dev", description="Environment name")
projects: Dict[str, ProjectEntry] = Field(
@@ -188,19 +231,42 @@ class BasicMemoryConfig(BaseSettings):
default=None,
description="Embedding vector dimensions. Auto-detected from provider if not set (384 for FastEmbed, 1536 for OpenAI).",
)
# Trigger: full local rebuilds spend most of their time waiting behind shared
# embed flushes, not constructing vectors themselves.
# Why: smaller FastEmbed batches cut queue wait far more than they increase
# write overhead on real-world projects, which makes full reindex materially faster.
# Outcome: default to the smaller local/cloud-safe batch size we benchmarked as
# the current best end-to-end setting in the shared vector sync pipeline.
semantic_embedding_batch_size: int = Field(
default=64,
default=2,
description="Batch size for embedding generation.",
gt=0,
)
semantic_embedding_request_concurrency: int = Field(
default=4,
description="Maximum number of concurrent provider requests for batched embedding generation when the active provider supports request-level concurrency.",
gt=0,
)
semantic_embedding_sync_batch_size: int = Field(
default=64,
default=2,
description="Batch size for vector sync orchestration flushes.",
gt=0,
)
semantic_postgres_prepare_concurrency: int = Field(
default=4,
description="Number of Postgres entity prepare tasks to run concurrently during vector sync. Postgres only; keep this low to avoid overdriving the database connection pool.",
gt=0,
le=16,
)
semantic_embedding_cache_dir: str | None = Field(
default=None,
description="Optional cache directory for FastEmbed model artifacts.",
description=(
"Optional override for the FastEmbed model cache directory. "
"When unset, Basic Memory resolves this at runtime to "
"<basic-memory data dir>/fastembed_cache (or FASTEMBED_CACHE_PATH "
"when that env var is set) so the model persists across runs "
"without hardcoding a path into config.json."
),
)
semantic_embedding_threads: int | None = Field(
default=None,
@@ -280,6 +346,31 @@ class BasicMemoryConfig(BaseSettings):
description="Maximum number of files to process concurrently during sync. Limits memory usage on large projects (2000+ files). Lower values reduce memory consumption.",
gt=0,
)
index_batch_size: int = Field(
default=32,
description="Maximum number of changed files to load into one indexing batch.",
gt=0,
)
index_batch_max_bytes: int = Field(
default=8 * 1024 * 1024,
description="Maximum total bytes to load into one indexing batch. Large files still run as single-file batches.",
gt=0,
)
index_parse_max_concurrent: int = Field(
default=8,
description="Maximum number of markdown parse tasks to run concurrently inside one indexing batch.",
gt=0,
)
index_entity_max_concurrent: int = Field(
default=4,
description="Maximum number of entity create/update tasks to run concurrently inside one indexing batch.",
gt=0,
)
index_metadata_update_max_concurrent: int = Field(
default=4,
description="Maximum number of metadata/search refresh tasks to run concurrently inside one indexing batch.",
gt=0,
)
kebab_filenames: bool = Field(
default=False,
@@ -662,11 +753,7 @@ class BasicMemoryConfig(BaseSettings):
@property
def data_dir_path(self) -> Path:
"""Get app state directory for config and default SQLite database."""
if config_dir := os.getenv("BASIC_MEMORY_CONFIG_DIR"):
return Path(config_dir)
home = os.getenv("HOME", Path.home())
return Path(home) / DATA_DIR_NAME
return resolve_data_dir()
# Module-level cache for configuration
@@ -684,16 +771,7 @@ class ConfigManager:
def __init__(self) -> None:
"""Initialize the configuration manager."""
home = os.getenv("HOME", Path.home())
if isinstance(home, str):
home = Path(home)
# Allow override via environment variable
if config_dir := os.getenv("BASIC_MEMORY_CONFIG_DIR"):
self.config_dir = Path(config_dir)
else:
self.config_dir = home / DATA_DIR_NAME
self.config_dir = resolve_data_dir()
self.config_file = self.config_dir / CONFIG_FILE_NAME
# Ensure config directory exists
-103
View File
@@ -44,101 +44,6 @@ _engine: Optional[AsyncEngine] = None
_session_maker: Optional[async_sessionmaker[AsyncSession]] = None
async def _needs_semantic_embedding_backfill(
app_config: BasicMemoryConfig,
session_maker: async_sessionmaker[AsyncSession],
) -> bool:
"""Check if entities exist but vector embeddings are empty.
This is the reliable way to detect that embeddings need to be generated,
regardless of how migrations were applied (fresh DB, upgrade, reset, etc.).
"""
if not app_config.semantic_search_enabled:
return False
try:
async with scoped_session(session_maker) as session:
entity_count = (
await session.execute(text("SELECT COUNT(*) FROM entity"))
).scalar() or 0
if entity_count == 0:
return False
# Check if vector chunks table exists and is empty
embedding_count = (
await session.execute(text("SELECT COUNT(*) FROM search_vector_chunks"))
).scalar() or 0
return embedding_count == 0
except Exception as exc:
# Table might not exist yet (pre-migration)
logger.debug(f"Could not check embedding status: {exc}")
return False
async def _run_semantic_embedding_backfill(
app_config: BasicMemoryConfig,
session_maker: async_sessionmaker[AsyncSession],
) -> None:
"""Backfill semantic embeddings for all active projects/entities."""
if not app_config.semantic_search_enabled:
logger.info("Skipping automatic semantic embedding backfill: semantic search is disabled.")
return
async with scoped_session(session_maker) as session:
project_result = await session.execute(
text("SELECT id, name FROM project WHERE is_active = :is_active ORDER BY id"),
{"is_active": True},
)
projects = [(int(row[0]), str(row[1])) for row in project_result.fetchall()]
if not projects:
logger.info("Skipping automatic semantic embedding backfill: no active projects found.")
return
repository_class = (
PostgresSearchRepository
if app_config.database_backend == DatabaseBackend.POSTGRES
else SQLiteSearchRepository
)
total_entities = 0
for project_id, project_name in projects:
async with scoped_session(session_maker) as session:
entity_result = await session.execute(
text("SELECT id FROM entity WHERE project_id = :project_id ORDER BY id"),
{"project_id": project_id},
)
entity_ids = [int(row[0]) for row in entity_result.fetchall()]
if not entity_ids:
continue
total_entities += len(entity_ids)
logger.info(
"Automatic semantic embedding backfill: "
f"project={project_name}, entities={len(entity_ids)}"
)
search_repository = repository_class(
session_maker,
project_id=project_id,
app_config=app_config,
)
batch_result = await search_repository.sync_entity_vectors_batch(entity_ids)
if batch_result.entities_failed > 0:
logger.warning(
"Automatic semantic embedding backfill encountered entity failures: "
f"project={project_name}, failed={batch_result.entities_failed}, "
f"failed_entity_ids={batch_result.failed_entity_ids}"
)
logger.info(
"Automatic semantic embedding backfill complete: "
f"projects={len(projects)}, entities={total_entities}"
)
class DatabaseType(Enum):
"""Types of supported databases."""
@@ -521,14 +426,6 @@ async def run_migrations(
else:
await SQLiteSearchRepository(session_maker, 1).init_search_index()
# Check if backfill is needed — actual backfill runs in background
# from the MCP server lifespan to avoid blocking startup.
if await _needs_semantic_embedding_backfill(app_config, session_maker):
logger.info(
"Semantic embeddings missing — backfill will run in background after startup"
)
else:
logger.info("Semantic embeddings: up to date")
except Exception as e: # pragma: no cover
logger.error(f"Error running migrations: {e}")
raise
-25
View File
@@ -492,7 +492,6 @@ class LocalTaskScheduler:
async def get_task_scheduler(
entity_service: EntityServiceV2ExternalDep,
sync_service: SyncServiceV2ExternalDep,
search_service: SearchServiceV2ExternalDep,
project_config: ProjectConfigV2ExternalDep,
@@ -500,28 +499,6 @@ async def get_task_scheduler(
) -> TaskScheduler:
"""Create a scheduler that maps task specs to coroutines."""
scheduler: LocalTaskScheduler | None = None
async def _reindex_entity(
entity_id: int,
resolve_relations: bool = False,
**_: Any,
) -> None:
await entity_service.reindex_entity(entity_id)
# Trigger: caller requests relation resolution
# Why: resolve forward references created before the entity existed
# Outcome: updates unresolved relations pointing to this entity
if resolve_relations:
await sync_service.resolve_relations(entity_id=entity_id)
# Trigger: semantic search enabled in local config.
# Why: vector chunks are derived and should refresh after canonical reindex completes.
# Outcome: schedules out-of-band vector sync without extending write latency.
if app_config.semantic_search_enabled and scheduler is not None:
scheduler.schedule("sync_entity_vectors", entity_id=entity_id)
async def _resolve_relations(entity_id: int, **_: Any) -> None:
await sync_service.resolve_relations(entity_id=entity_id)
async def _sync_entity_vectors(entity_id: int, **_: Any) -> None:
await search_service.sync_entity_vectors(entity_id)
@@ -537,8 +514,6 @@ async def get_task_scheduler(
scheduler = LocalTaskScheduler(
{
"reindex_entity": _reindex_entity,
"resolve_relations": _resolve_relations,
"sync_entity_vectors": _sync_entity_vectors,
"sync_project": _sync_project,
"reindex_project": _reindex_project,
+14 -1
View File
@@ -114,7 +114,13 @@ async def write_file_atomic(path: FilePath, content: str) -> None:
temp_path = path_obj.with_suffix(".tmp")
try:
# Use aiofiles for non-blocking write
# Trigger: callers hand us normalized Python text, but the final bytes are allowed
# to use the host platform's native newline convention during the write.
# Why: preserving CRLF on Windows keeps local files aligned with editors like
# Obsidian, while FileService now hashes the persisted file bytes instead of
# the pre-write string.
# Outcome: this async write stays editor-friendly across platforms without
# reintroducing checksum drift in sync or move detection.
async with aiofiles.open(temp_path, mode="w", encoding="utf-8") as f:
await f.write(content)
@@ -168,6 +174,13 @@ async def format_markdown_builtin(path: Path) -> Optional[str]:
# Only write if content changed
if formatted_content != content:
# Trigger: mdformat may rewrite markdown content, then the host platform
# decides the newline bytes for the follow-up async text write.
# Why: we want formatter output to preserve native newlines instead of
# forcing LF, and the authoritative checksum comes from rereading the
# stored file bytes later in FileService.
# Outcome: formatting remains compatible with local editors on Windows while
# checksum-based sync logic stays anchored to on-disk bytes.
async with aiofiles.open(path, mode="w", encoding="utf-8") as f:
await f.write(formatted_content)
+8 -3
View File
@@ -4,6 +4,8 @@ import fnmatch
from pathlib import Path
from typing import Set
from basic_memory.config import resolve_data_dir
# Common directories and patterns to ignore by default
# These are used as fallback if .bmignore doesn't exist
@@ -61,9 +63,11 @@ def get_bmignore_path() -> Path:
"""Get path to .bmignore file.
Returns:
Path to ~/.basic-memory/.bmignore
Path to <basic-memory data dir>/.bmignore, honoring
``BASIC_MEMORY_CONFIG_DIR`` so isolated instances each keep their
own ignore file.
"""
return Path.home() / ".basic-memory" / ".bmignore"
return resolve_data_dir() / ".bmignore"
def create_default_bmignore() -> None:
@@ -176,7 +180,8 @@ def load_gitignore_patterns(base_path: Path, use_gitignore: bool = True) -> Set[
"""Load gitignore patterns from .gitignore file and .bmignore.
Combines patterns from:
1. ~/.basic-memory/.bmignore (user's global ignore patterns)
1. <basic-memory data dir>/.bmignore (user's global ignore patterns, honors
BASIC_MEMORY_CONFIG_DIR)
2. {base_path}/.gitignore (project-specific patterns, if use_gitignore=True)
Args:
+7 -6
View File
@@ -39,23 +39,24 @@ def format_timestamp(timestamp: Any) -> str: # pragma: no cover
Returns:
A formatted string representation of the timestamp.
"""
parsed_timestamp = timestamp
if isinstance(timestamp, str):
try:
# Try ISO format
timestamp = datetime.fromisoformat(timestamp.replace("Z", "+00:00"))
parsed_timestamp = datetime.fromisoformat(timestamp.replace("Z", "+00:00"))
except ValueError:
try:
# Try unix timestamp as string
timestamp = datetime.fromtimestamp(float(timestamp)).astimezone()
parsed_timestamp = datetime.fromtimestamp(float(timestamp)).astimezone()
except ValueError:
# Return as is if we can't parse it
return timestamp
elif isinstance(timestamp, (int, float)):
# Unix timestamp
timestamp = datetime.fromtimestamp(timestamp).astimezone()
parsed_timestamp = datetime.fromtimestamp(timestamp).astimezone()
if isinstance(timestamp, datetime):
return timestamp.strftime("%Y-%m-%d %H:%M:%S")
if isinstance(parsed_timestamp, datetime):
return parsed_timestamp.strftime("%Y-%m-%d %H:%M:%S")
# Return as is if we can't format it
return str(timestamp) # pragma: no cover
return str(parsed_timestamp) # pragma: no cover
+31
View File
@@ -0,0 +1,31 @@
"""Reusable indexing primitives shared by local sync and future remote callers."""
from basic_memory.indexing.batch_indexer import BatchIndexer
from basic_memory.indexing.batching import build_index_batches
from basic_memory.indexing.models import (
IndexedEntity,
IndexBatch,
IndexFileMetadata,
IndexFileWriter,
IndexFrontmatterUpdate,
IndexFrontmatterWriteResult,
IndexingBatchResult,
IndexInputFile,
IndexProgress,
SyncedMarkdownFile,
)
__all__ = [
"BatchIndexer",
"IndexedEntity",
"IndexBatch",
"IndexFileMetadata",
"IndexFileWriter",
"IndexFrontmatterUpdate",
"IndexFrontmatterWriteResult",
"IndexingBatchResult",
"IndexInputFile",
"IndexProgress",
"SyncedMarkdownFile",
"build_index_batches",
]
+710
View File
@@ -0,0 +1,710 @@
"""Reusable batch executor for bounded-parallel file indexing."""
from __future__ import annotations
import asyncio
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
from typing import Awaitable, Callable, Mapping, TypeVar
from loguru import logger
from sqlalchemy.exc import IntegrityError
import logfire
from basic_memory.config import BasicMemoryConfig
from basic_memory.file_utils import compute_checksum, has_frontmatter, remove_frontmatter
from basic_memory.markdown.schemas import EntityMarkdown
from basic_memory.indexing.models import (
IndexedEntity,
IndexFileWriter,
IndexFrontmatterUpdate,
IndexingBatchResult,
IndexInputFile,
)
from basic_memory.models import Entity, Relation
from basic_memory.services import EntityService
from basic_memory.services.exceptions import SyncFatalError
from basic_memory.services.search_service import SearchService
from basic_memory.repository import EntityRepository, RelationRepository
T = TypeVar("T")
@dataclass(slots=True)
class _PreparedMarkdownFile:
file: IndexInputFile
content: str
final_checksum: str
markdown: EntityMarkdown
file_contains_frontmatter: bool
@dataclass(slots=True)
class _PreparedEntity:
path: str
entity_id: int
permalink: str | None
checksum: str
content_type: str | None
search_content: str | None
markdown_content: str | None = None
@dataclass(slots=True)
class _PersistedMarkdownFile:
prepared: _PreparedMarkdownFile
entity: Entity
class BatchIndexer:
"""Index already-loaded files without assuming where they came from."""
def __init__(
self,
*,
app_config: BasicMemoryConfig,
entity_service: EntityService,
entity_repository: EntityRepository,
relation_repository: RelationRepository,
search_service: SearchService,
file_writer: IndexFileWriter,
) -> None:
self.app_config = app_config
self.entity_service = entity_service
self.entity_repository = entity_repository
self.relation_repository = relation_repository
self.search_service = search_service
self.file_writer = file_writer
async def index_files(
self,
files: Mapping[str, IndexInputFile],
*,
max_concurrent: int,
parse_max_concurrent: int | None = None,
existing_permalink_by_path: dict[str, str | None] | None = None,
) -> IndexingBatchResult:
"""Index one batch of loaded files with bounded concurrency."""
if max_concurrent <= 0:
raise ValueError("max_concurrent must be greater than zero")
ordered_paths = sorted(files)
if not ordered_paths:
return IndexingBatchResult()
parse_limit = parse_max_concurrent or max_concurrent
error_by_path: dict[str, str] = {}
markdown_paths = [path for path in ordered_paths if self._is_markdown(files[path])]
regular_paths = [path for path in ordered_paths if path not in markdown_paths]
prepared_markdown, parse_errors = await self._run_bounded(
markdown_paths,
limit=parse_limit,
worker=lambda path: self._prepare_markdown_file(files[path]),
)
error_by_path.update(parse_errors)
prepared_markdown, normalization_errors = await self._normalize_markdown_batch(
prepared_markdown,
existing_permalink_by_path=existing_permalink_by_path,
)
error_by_path.update(normalization_errors)
indexed_entities: list[IndexedEntity] = []
resolved_count = 0
unresolved_count = 0
search_indexed = 0
prepared_entities: dict[str, _PreparedEntity] = {}
markdown_upserts, markdown_errors = await self._run_bounded(
[path for path in markdown_paths if path not in error_by_path],
limit=max_concurrent,
worker=lambda path: self._upsert_markdown_file(prepared_markdown[path]),
)
error_by_path.update(markdown_errors)
prepared_entities.update(markdown_upserts)
if existing_permalink_by_path is not None:
for path, prepared_entity in markdown_upserts.items():
existing_permalink_by_path[path] = prepared_entity.permalink
regular_upserts, regular_errors = await self._run_bounded(
regular_paths,
limit=max_concurrent,
worker=lambda path: self._upsert_regular_file(files[path]),
)
error_by_path.update(regular_errors)
prepared_entities.update(regular_upserts)
markdown_entity_ids = [
prepared_entities[path].entity_id
for path in markdown_paths
if path in prepared_entities
]
if markdown_entity_ids:
resolved_count, unresolved_count = await self._resolve_batch_relations(
markdown_entity_ids,
max_concurrent=max_concurrent,
)
refreshed_entities = await self.entity_repository.find_by_ids(
[prepared.entity_id for prepared in prepared_entities.values()]
)
entities_by_id = {entity.id: entity for entity in refreshed_entities}
refreshed, refresh_errors = await self._run_bounded(
[path for path in ordered_paths if path in prepared_entities],
limit=self.app_config.index_metadata_update_max_concurrent,
worker=lambda path: self._refresh_search_index(
prepared_entities[path],
entities_by_id[prepared_entities[path].entity_id],
),
)
error_by_path.update(refresh_errors)
for path in ordered_paths:
indexed = refreshed.get(path)
if indexed is not None:
indexed_entities.append(indexed)
search_indexed = len(indexed_entities)
return IndexingBatchResult(
indexed=indexed_entities,
errors=[(path, error_by_path[path]) for path in ordered_paths if path in error_by_path],
relations_resolved=resolved_count,
relations_unresolved=unresolved_count,
search_indexed=search_indexed,
)
async def index_markdown_file(
self,
file: IndexInputFile,
*,
new: bool | None = None,
existing_permalink_by_path: dict[str, str | None] | None = None,
index_search: bool = True,
resolve_relations: bool = True,
) -> IndexedEntity:
"""Index one markdown file using the same normalization and upsert path as batches."""
if not self._is_markdown(file):
raise ValueError(f"index_markdown_file requires markdown input: {file.path}")
with logfire.span("index.markdown_file.prepare", path=file.path):
prepared = await self._prepare_markdown_file(file)
if existing_permalink_by_path is None:
with logfire.span("index.markdown_file.load_permalink_map", path=file.path):
existing_permalink_by_path = {
path: permalink
for path, permalink in (
await self.entity_repository.get_file_path_to_permalink_map()
).items()
}
reserved_permalinks = {
permalink
for path, permalink in existing_permalink_by_path.items()
if path != file.path and permalink
}
with logfire.span("index.markdown_file.normalize", path=file.path):
prepared = await self._normalize_markdown_file(prepared, reserved_permalinks)
existing_permalink_by_path[file.path] = prepared.markdown.frontmatter.permalink
with logfire.span("index.markdown_file.persist", path=file.path, is_new=new):
persisted = await self._persist_markdown_file(
prepared,
is_new=new,
resolve_relations=resolve_relations,
reload_entity=False,
)
existing_permalink_by_path[file.path] = persisted.entity.permalink
with logfire.span(
"index.markdown_file.reload_entity",
path=file.path,
entity_id=persisted.entity.id,
):
refreshed = await self.entity_repository.find_by_ids([persisted.entity.id])
if len(refreshed) != 1: # pragma: no cover
raise ValueError(f"Failed to reload indexed entity for {file.path}")
entity = refreshed[0]
prepared_entity = self._build_prepared_entity(persisted.prepared, entity)
if index_search:
with logfire.span(
"index.markdown_file.refresh_search_index",
path=file.path,
entity_id=entity.id,
):
return await self._refresh_search_index(prepared_entity, entity)
return IndexedEntity(
path=prepared_entity.path,
entity_id=entity.id,
permalink=entity.permalink,
checksum=prepared_entity.checksum,
content_type=prepared_entity.content_type,
markdown_content=prepared_entity.markdown_content,
)
# --- Preparation ---
async def _prepare_markdown_file(self, file: IndexInputFile) -> _PreparedMarkdownFile:
if file.content is None:
raise ValueError(f"Missing content for markdown file: {file.path}")
content = file.content.decode("utf-8")
file_contains_frontmatter = has_frontmatter(content)
final_checksum = await self._resolve_checksum(file)
entity_markdown = await self.entity_service.entity_parser.parse_markdown_content(
file_path=Path(file.path),
content=content,
mtime=file.last_modified.timestamp() if file.last_modified else None,
ctime=file.created_at.timestamp() if file.created_at else None,
)
return _PreparedMarkdownFile(
file=file,
content=content,
final_checksum=final_checksum,
markdown=entity_markdown,
file_contains_frontmatter=file_contains_frontmatter,
)
async def _normalize_markdown_batch(
self,
prepared_markdown: dict[str, _PreparedMarkdownFile],
*,
existing_permalink_by_path: dict[str, str | None] | None = None,
) -> tuple[dict[str, _PreparedMarkdownFile], dict[str, str]]:
if not prepared_markdown:
return {}, {}
if existing_permalink_by_path is None:
existing_permalink_by_path = {
path: permalink
for path, permalink in (
await self.entity_repository.get_file_path_to_permalink_map()
).items()
}
batch_paths = set(prepared_markdown)
reserved_permalinks = {
permalink
for path, permalink in existing_permalink_by_path.items()
if path not in batch_paths and permalink
}
normalized: dict[str, _PreparedMarkdownFile] = {}
errors: dict[str, str] = {}
for path in sorted(prepared_markdown):
try:
normalized[path] = await self._normalize_markdown_file(
prepared_markdown[path],
reserved_permalinks,
)
existing_permalink_by_path[path] = normalized[path].markdown.frontmatter.permalink
except Exception as exc:
errors[path] = str(exc)
logger.warning("Batch markdown normalization failed", path=path, error=str(exc))
return normalized, errors
async def _normalize_markdown_file(
self,
prepared: _PreparedMarkdownFile,
reserved_permalinks: set[str],
) -> _PreparedMarkdownFile:
final_checksum = prepared.final_checksum
final_content = prepared.content
final_permalink = await self._resolve_batch_permalink(prepared, reserved_permalinks)
# Trigger: markdown file has no frontmatter and sync enforcement is enabled.
# Why: downstream indexing relies on normalized metadata and stable permalinks.
# Outcome: write derived metadata back through the storage-agnostic writer.
if not prepared.file_contains_frontmatter and self.app_config.ensure_frontmatter_on_sync:
frontmatter_updates = {
"title": prepared.markdown.frontmatter.title,
"type": prepared.markdown.frontmatter.type,
"permalink": final_permalink,
}
write_result = await self.file_writer.write_frontmatter(
IndexFrontmatterUpdate(path=prepared.file.path, metadata=frontmatter_updates)
)
final_checksum = write_result.checksum
final_content = write_result.content
prepared.markdown.frontmatter.metadata.update(frontmatter_updates)
# Trigger: existing markdown frontmatter may lack the canonical permalink.
# Why: batch sync keeps permalinks stable without forcing a full rewrite when unchanged.
# Outcome: only the permalink field is updated when it actually differs.
elif (
prepared.file_contains_frontmatter
and not self.app_config.disable_permalinks
and final_permalink != prepared.markdown.frontmatter.permalink
):
prepared.markdown.frontmatter.metadata["permalink"] = final_permalink
write_result = await self.file_writer.write_frontmatter(
IndexFrontmatterUpdate(
path=prepared.file.path,
metadata={"permalink": final_permalink},
)
)
final_checksum = write_result.checksum
final_content = write_result.content
return _PreparedMarkdownFile(
file=prepared.file,
content=final_content,
final_checksum=final_checksum,
markdown=prepared.markdown,
file_contains_frontmatter=prepared.file_contains_frontmatter,
)
async def _resolve_batch_permalink(
self,
prepared: _PreparedMarkdownFile,
reserved_permalinks: set[str],
) -> str | None:
should_resolve_permalink = (
not prepared.file_contains_frontmatter and self.app_config.ensure_frontmatter_on_sync
) or (prepared.file_contains_frontmatter and not self.app_config.disable_permalinks)
if not should_resolve_permalink:
permalink = prepared.markdown.frontmatter.permalink
if permalink:
reserved_permalinks.add(permalink)
return permalink
desired_permalink = await self.entity_service.resolve_permalink(
prepared.file.path,
markdown=prepared.markdown,
skip_conflict_check=True,
)
return self._reserve_batch_permalink(desired_permalink, reserved_permalinks)
def _reserve_batch_permalink(
self,
desired_permalink: str,
reserved_permalinks: set[str],
) -> str:
permalink = desired_permalink
suffix = 1
while permalink in reserved_permalinks:
permalink = f"{desired_permalink}-{suffix}"
suffix += 1
reserved_permalinks.add(permalink)
return permalink
# --- Persistence ---
async def _upsert_markdown_file(self, prepared: _PreparedMarkdownFile) -> _PreparedEntity:
persisted = await self._persist_markdown_file(prepared)
return self._build_prepared_entity(persisted.prepared, persisted.entity)
async def _upsert_regular_file(self, file: IndexInputFile) -> _PreparedEntity:
checksum = await self._resolve_checksum(file)
existing = await self.entity_repository.get_by_file_path(file.path, load_relations=False)
is_new_entity = existing is None
if existing is None:
await self.entity_service.resolve_permalink(file.path, skip_conflict_check=True)
entity = Entity(
note_type="file",
file_path=file.path,
checksum=checksum,
title=Path(file.path).name,
created_at=file.created_at or datetime.now().astimezone(),
updated_at=file.last_modified or datetime.now().astimezone(),
content_type=file.content_type or "text/plain",
mtime=file.last_modified.timestamp() if file.last_modified else None,
size=file.size,
)
try:
created = await self.entity_repository.add(entity)
entity_id = created.id
except IntegrityError as exc:
message = str(exc)
if (
"UNIQUE constraint failed: entity.file_path" in message
or "uix_entity_file_path_project" in message
or (
"duplicate key value violates unique constraint" in message
and "file_path" in message
)
):
existing = await self.entity_repository.get_by_file_path(
file.path,
load_relations=False,
)
if existing is None:
raise ValueError(
f"Entity not found after file_path conflict: {file.path}"
) from exc
entity_id = existing.id
else:
raise
else:
entity_id = existing.id
updated = await self.entity_repository.update(
entity_id,
self._entity_metadata_updates(file, checksum, include_created_at=is_new_entity),
)
if updated is None:
raise ValueError(f"Failed to update file entity metadata for {file.path}")
return _PreparedEntity(
path=file.path,
entity_id=updated.id,
permalink=updated.permalink,
checksum=checksum,
content_type=file.content_type,
search_content=None,
markdown_content=None,
)
# --- Relations ---
async def _resolve_batch_relations(
self,
entity_ids: list[int],
*,
max_concurrent: int,
) -> tuple[int, int]:
unresolved_relation_lists = await asyncio.gather(
*(
self.relation_repository.find_unresolved_relations_for_entity(entity_id)
for entity_id in entity_ids
)
)
unresolved_relations = [
relation for relation_list in unresolved_relation_lists for relation in relation_list
]
if not unresolved_relations:
return 0, 0
semaphore = asyncio.Semaphore(max_concurrent)
async def resolve_relation(relation: Relation) -> int:
async with semaphore:
try:
resolved_entity = await self.entity_service.link_resolver.resolve_link(
relation.to_name
)
if resolved_entity is None or resolved_entity.id == relation.from_id:
return 0
try:
await self.relation_repository.update(
relation.id,
{
"to_id": resolved_entity.id,
"to_name": resolved_entity.title,
},
)
except IntegrityError:
await self.relation_repository.delete(relation.id)
return 1
except Exception as exc: # pragma: no cover - defensive logging
logger.warning(
"Batch relation resolution failed",
relation_id=relation.id,
from_id=relation.from_id,
to_name=relation.to_name,
error=str(exc),
)
return 0
resolved_counts = await asyncio.gather(
*(resolve_relation(relation) for relation in unresolved_relations)
)
remaining_relation_lists = await asyncio.gather(
*(
self.relation_repository.find_unresolved_relations_for_entity(entity_id)
for entity_id in entity_ids
)
)
remaining_unresolved = sum(len(relations) for relations in remaining_relation_lists)
return sum(resolved_counts), remaining_unresolved
# --- Search refresh ---
async def _refresh_search_index(
self, prepared: _PreparedEntity, entity: Entity
) -> IndexedEntity:
await self.search_service.index_entity_data(entity, content=prepared.search_content)
return IndexedEntity(
path=prepared.path,
entity_id=entity.id,
permalink=entity.permalink,
checksum=prepared.checksum,
content_type=prepared.content_type,
markdown_content=prepared.markdown_content,
)
# --- Helpers ---
async def _persist_markdown_file(
self,
prepared: _PreparedMarkdownFile,
*,
is_new: bool | None = None,
resolve_relations: bool = True,
reload_entity: bool = True,
) -> _PersistedMarkdownFile:
existing = await self.entity_repository.get_by_file_path(
prepared.file.path,
load_relations=False,
)
if is_new is None:
is_new = existing is None
entity = await self.entity_service.upsert_entity_from_markdown(
Path(prepared.file.path),
prepared.markdown,
is_new=is_new,
existing_entity=existing,
resolve_relations=resolve_relations,
reload_entity=reload_entity,
)
prepared = await self._reconcile_persisted_permalink(prepared, entity)
metadata_updates = self._entity_metadata_updates(prepared.file, prepared.final_checksum)
updated = await self.entity_repository.update_fields(
entity.id,
metadata_updates,
)
if not updated:
raise ValueError(f"Failed to update markdown entity metadata for {prepared.file.path}")
self._apply_entity_metadata_updates(entity, metadata_updates)
return _PersistedMarkdownFile(prepared=prepared, entity=entity)
async def _reconcile_persisted_permalink(
self,
prepared: _PreparedMarkdownFile,
entity: Entity,
) -> _PreparedMarkdownFile:
# Trigger: the source file started without frontmatter and sync is configured
# to leave frontmatterless files alone.
# Why: upsert may still assign a DB permalink even when disk content should stay untouched.
# Outcome: skip reconciliation writes that would silently inject frontmatter.
if (
self.app_config.disable_permalinks
or (
not prepared.file_contains_frontmatter
and not self.app_config.ensure_frontmatter_on_sync
)
or entity.permalink is None
or entity.permalink == prepared.markdown.frontmatter.permalink
):
return prepared
logger.debug(
"Updating permalink after upsert conflict resolution",
path=prepared.file.path,
old_permalink=prepared.markdown.frontmatter.permalink,
new_permalink=entity.permalink,
)
prepared.markdown.frontmatter.metadata["permalink"] = entity.permalink
write_result = await self.file_writer.write_frontmatter(
IndexFrontmatterUpdate(
path=prepared.file.path,
metadata={"permalink": entity.permalink},
)
)
return _PreparedMarkdownFile(
file=prepared.file,
content=write_result.content,
final_checksum=write_result.checksum,
markdown=prepared.markdown,
file_contains_frontmatter=prepared.file_contains_frontmatter,
)
def _build_prepared_entity(
self,
prepared: _PreparedMarkdownFile,
entity: Entity,
) -> _PreparedEntity:
return _PreparedEntity(
path=prepared.file.path,
entity_id=entity.id,
permalink=entity.permalink,
checksum=prepared.final_checksum,
content_type=prepared.file.content_type,
search_content=(
prepared.markdown.content
if prepared.markdown.content is not None
else remove_frontmatter(prepared.content)
),
markdown_content=prepared.content,
)
async def _resolve_checksum(self, file: IndexInputFile) -> str:
if file.checksum is not None:
return file.checksum
if file.content is None:
raise ValueError(f"Missing checksum and content for file: {file.path}")
return await compute_checksum(file.content)
def _entity_metadata_updates(
self,
file: IndexInputFile,
checksum: str,
*,
include_created_at: bool = True,
) -> dict[str, object]:
updates: dict[str, object] = {
"file_path": file.path,
"checksum": checksum,
"size": file.size,
}
if include_created_at and file.created_at is not None:
updates["created_at"] = file.created_at
if file.last_modified is not None:
updates["updated_at"] = file.last_modified
updates["mtime"] = file.last_modified.timestamp()
if file.content_type is not None:
updates["content_type"] = file.content_type
return updates
def _apply_entity_metadata_updates(self, entity: Entity, updates: dict[str, object]) -> None:
"""Keep the returned entity aligned with metadata written without reload."""
for key, value in updates.items():
setattr(entity, key, value)
def _is_markdown(self, file: IndexInputFile) -> bool:
if file.content_type is not None:
return file.content_type == "text/markdown"
return Path(file.path).suffix.lower() in {".md", ".markdown"}
async def _run_bounded(
self,
paths: list[str],
*,
limit: int,
worker: Callable[[str], Awaitable[T]],
) -> tuple[dict[str, T], dict[str, str]]:
if not paths:
return {}, {}
semaphore = asyncio.Semaphore(limit)
results: dict[str, T] = {}
errors: dict[str, str] = {}
async def run(path: str) -> None:
async with semaphore:
try:
results[path] = await worker(path)
except Exception as exc:
if isinstance(exc, SyncFatalError) or isinstance(exc.__cause__, SyncFatalError):
raise
errors[path] = str(exc)
logger.warning("Batch indexing failed", path=path, error=str(exc))
await asyncio.gather(*(run(path) for path in paths))
return results, errors
+63
View File
@@ -0,0 +1,63 @@
"""Deterministic helpers for planning bounded indexing batches."""
from __future__ import annotations
from collections.abc import Mapping, Sequence
from basic_memory.indexing.models import IndexBatch, IndexFileMetadata
def build_index_batches(
paths: Sequence[str],
metadata_by_path: Mapping[str, IndexFileMetadata],
*,
max_files: int,
max_bytes: int,
) -> list[IndexBatch]:
"""Build deterministic batches bounded by file count and total bytes."""
if max_files <= 0:
raise ValueError("max_files must be greater than zero")
if max_bytes <= 0:
raise ValueError("max_bytes must be greater than zero")
ordered_paths = sorted(paths)
batches: list[IndexBatch] = []
current_paths: list[str] = []
current_bytes = 0
for path in ordered_paths:
metadata = metadata_by_path.get(path)
if metadata is None:
raise KeyError(f"Missing metadata for path: {path}")
file_bytes = max(metadata.size, 0)
# Trigger: the next file would overflow the active batch.
# Why: keep batches memory-bounded and predictable for both local and remote callers.
# Outcome: flush the current batch before placing the next file.
if current_paths and (
len(current_paths) >= max_files or current_bytes + file_bytes > max_bytes
):
batches.append(IndexBatch(paths=current_paths, total_bytes=current_bytes))
current_paths = []
current_bytes = 0
# Trigger: one file is larger than the configured byte budget.
# Why: we still need to index it, but splitting a single file is out of scope.
# Outcome: emit a dedicated single-file batch that may exceed max_bytes.
if file_bytes > max_bytes:
batches.append(IndexBatch(paths=[path], total_bytes=file_bytes))
continue
current_paths.append(path)
current_bytes += file_bytes
if len(current_paths) >= max_files or current_bytes == max_bytes:
batches.append(IndexBatch(paths=current_paths, total_bytes=current_bytes))
current_paths = []
current_bytes = 0
if current_paths:
batches.append(IndexBatch(paths=current_paths, total_bytes=current_bytes))
return batches
+110
View File
@@ -0,0 +1,110 @@
"""Typed models for the reusable indexing execution path."""
from __future__ import annotations
from dataclasses import dataclass, field
from datetime import datetime
from typing import Any, Protocol, TYPE_CHECKING
if TYPE_CHECKING: # pragma: no cover
from basic_memory.models import Entity
@dataclass(slots=True)
class IndexFileMetadata:
"""Storage-agnostic metadata for a file queued for indexing."""
path: str
size: int
checksum: str | None = None
content_type: str | None = None
last_modified: datetime | None = None
created_at: datetime | None = None
@dataclass(slots=True)
class IndexInputFile(IndexFileMetadata):
"""Fully loaded file payload consumed by the batch executor."""
content: bytes | None = None
@dataclass(slots=True)
class IndexBatch:
"""A deterministic batch of files bounded by count and total bytes."""
paths: list[str]
total_bytes: int
@dataclass(slots=True)
class IndexProgress:
"""Batch indexing progress emitted to callers such as the CLI."""
files_total: int
files_processed: int
batches_total: int
batches_completed: int
current_batch_bytes: int = 0
files_per_minute: float = 0.0
eta_seconds: float | None = None
@dataclass(slots=True)
class IndexFrontmatterUpdate:
"""A typed frontmatter write request for a single file."""
path: str
metadata: dict[str, Any]
@dataclass(slots=True)
class IndexFrontmatterWriteResult:
"""Typed result for a frontmatter write performed during indexing."""
checksum: str
content: str
@dataclass(slots=True)
class IndexedEntity:
"""Stable output describing one file that finished indexing successfully."""
path: str
entity_id: int
permalink: str | None
checksum: str
content_type: str | None = None
markdown_content: str | None = None
@dataclass(slots=True)
class SyncedMarkdownFile:
"""Canonical result for syncing one markdown file end-to-end."""
entity: Entity
checksum: str
markdown_content: str
file_path: str
content_type: str
updated_at: datetime
size: int
@dataclass(slots=True)
class IndexingBatchResult:
"""Outcome for one batch execution."""
indexed: list[IndexedEntity] = field(default_factory=list)
errors: list[tuple[str, str]] = field(default_factory=list)
relations_resolved: int = 0
relations_unresolved: int = 0
search_indexed: int = 0
class IndexFileWriter(Protocol):
"""Narrow protocol for frontmatter writes during indexing."""
async def write_frontmatter(
self, update: IndexFrontmatterUpdate
) -> IndexFrontmatterWriteResult: ...
@@ -249,6 +249,10 @@ class EntityParser:
content = strip_bom(content)
# PostgreSQL rejects null bytes (0x00) in text columns.
# Some markdown files (e.g. Claude agent definitions) contain embedded nulls.
content = content.replace("\x00", "")
# Parse frontmatter with proper error handling for malformed YAML.
# We use frontmatter.parse() instead of frontmatter.loads() because
# loads() does Post(content, handler, **metadata), which crashes when
+15
View File
@@ -180,6 +180,9 @@ def observation_plugin(md: MarkdownIt) -> None:
def observation_rule(state: Any) -> None:
"""Process observations in token stream."""
tokens = state.tokens
# Track blockquote nesting so Obsidian callouts (`> [!info] Title`)
# don't get parsed as observations with category `!info`.
blockquote_depth = 0
for idx in range(len(tokens)):
token = tokens[idx]
@@ -187,6 +190,18 @@ def observation_plugin(md: MarkdownIt) -> None:
# Initialize meta for all tokens
token.meta = token.meta or {}
if token.type == "blockquote_open":
blockquote_depth += 1
continue
if token.type == "blockquote_close":
blockquote_depth -= 1
continue
# Skip parsing inside blockquotes — that's Obsidian callout
# territory, not Basic Memory observation syntax.
if blockquote_depth > 0:
continue
# Parse observations in list items
if token.type == "inline" and is_observation(token):
obs = parse_observation(token)
+32 -8
View File
@@ -1,9 +1,9 @@
"""Schema models for entity markdown files."""
from datetime import datetime
from typing import List, Optional
from typing import TYPE_CHECKING, Any, List, Optional
from pydantic import BaseModel
from pydantic import BaseModel, Field, model_validator
class Observation(BaseModel):
@@ -38,23 +38,47 @@ class Relation(BaseModel):
class EntityFrontmatter(BaseModel):
"""Required frontmatter fields for an entity."""
metadata: dict = {}
if TYPE_CHECKING:
# Frontmatter may be built from raw YAML keys. The validator below
# gathers those keys into the metadata mapping used at runtime.
def __init__(self, **data: Any) -> None: ...
metadata: dict[str, Any] = Field(default_factory=dict)
@model_validator(mode="before")
@classmethod
def collect_metadata(cls, data: Any) -> Any:
if not isinstance(data, dict):
return data
if "metadata" not in data:
return {"metadata": data}
metadata = data.get("metadata") or {}
extras = {key: value for key, value in data.items() if key != "metadata"}
if extras:
return {"metadata": {**extras, **metadata}}
return data
@property
def tags(self) -> List[str]:
return self.metadata.get("tags") if self.metadata else None # pyright: ignore
tags = self.metadata.get("tags")
return [str(tag) for tag in tags] if isinstance(tags, list) else []
@property
def title(self) -> str:
return self.metadata.get("title") if self.metadata else None # pyright: ignore
title = self.metadata.get("title")
return title if isinstance(title, str) else ""
@property
def type(self) -> str:
return self.metadata.get("type", "note") if self.metadata else "note" # pyright: ignore
note_type = self.metadata.get("type", "note")
return note_type if isinstance(note_type, str) else "note"
@property
def permalink(self) -> str:
return self.metadata.get("permalink") if self.metadata else None # pyright: ignore
def permalink(self) -> Optional[str]:
permalink = self.metadata.get("permalink")
return permalink if isinstance(permalink, str) else None
class EntityMarkdown(BaseModel):
+12 -6
View File
@@ -5,7 +5,7 @@ from typing import AsyncIterator, Callable, Optional
from httpx import ASGITransport, AsyncClient, Timeout
from loguru import logger
from basic_memory import telemetry
import logfire
from basic_memory.api.app import app as fastapi_app
from basic_memory.config import ConfigManager, ProjectMode
@@ -44,7 +44,7 @@ def _asgi_client(timeout: Timeout) -> AsyncClient:
async def _resolve_cloud_token(config) -> str:
"""Resolve cloud token with API key preferred, OAuth fallback."""
with telemetry.span(
with logfire.span(
"routing.resolve_cloud_credentials",
has_api_key=bool(config.cloud_api_key),
):
@@ -94,9 +94,12 @@ async def _cloud_client(
workspace: Optional[str] = None,
) -> AsyncIterator[AsyncClient]:
"""Create a cloud proxy client with resolved credentials."""
from basic_memory.workspace_context import workspace_permalink_headers
token = await _resolve_cloud_token(config)
proxy_base_url = f"{config.cloud_host}/proxy"
headers = {"Authorization": f"Bearer {token}"}
headers.update(workspace_permalink_headers())
if workspace:
headers["X-Workspace-ID"] = workspace
logger.info(f"Creating HTTP client for cloud proxy at: {proxy_base_url}")
@@ -128,11 +131,14 @@ async def get_cloud_control_plane_client(
yield client
# Optional factory override for dependency injection
_client_factory: Optional[Callable[[], AbstractAsyncContextManager[AsyncClient]]] = None
# Optional factory override for dependency injection.
# The factory accepts an optional workspace keyword argument so that MCP tools
# can route individual requests to a different workspace than the one set at
# connection time. See basic-memory-cloud main.py tenant_asgi_client_factory.
_client_factory: Optional[Callable[..., AbstractAsyncContextManager[AsyncClient]]] = None
def set_client_factory(factory: Callable[[], AbstractAsyncContextManager[AsyncClient]]) -> None:
def set_client_factory(factory: Callable[..., AbstractAsyncContextManager[AsyncClient]]) -> None:
"""Override the default client factory (for cloud app, testing, etc)."""
global _client_factory
_client_factory = factory
@@ -173,7 +179,7 @@ async def get_client(
4. Local ASGI transport by default.
"""
if _client_factory:
async with _client_factory() as client:
async with _client_factory(workspace=workspace) as client:
yield client
return
+11 -26
View File
@@ -7,7 +7,7 @@ from typing import Any
from httpx import AsyncClient
from basic_memory import telemetry
import logfire
from basic_memory.mcp.tools.utils import call_get, call_post, call_put, call_patch, call_delete
from basic_memory.schemas.response import (
EntityResponse,
@@ -44,9 +44,7 @@ class KnowledgeClient:
# --- Entity CRUD Operations ---
async def create_entity(
self, entity_data: dict[str, Any], *, fast: bool | None = None
) -> EntityResponse:
async def create_entity(self, entity_data: dict[str, Any]) -> EntityResponse:
"""Create a new entity.
Args:
@@ -58,18 +56,15 @@ class KnowledgeClient:
Raises:
ToolError: If the request fails
"""
params = {"fast": fast} if fast is not None else None
with telemetry.scope(
with logfire.span(
"mcp.client.knowledge.create_entity",
client_name="knowledge",
operation="create_entity",
fast=fast,
):
response = await call_post(
self.http_client,
f"{self._base_path}/entities",
json=entity_data,
params=params,
client_name="knowledge",
operation="create_entity",
path_template="/v2/projects/{project_id}/knowledge/entities",
@@ -80,8 +75,6 @@ class KnowledgeClient:
self,
entity_id: str,
entity_data: dict[str, Any],
*,
fast: bool | None = None,
) -> EntityResponse:
"""Update an existing entity (full replacement).
@@ -95,18 +88,15 @@ class KnowledgeClient:
Raises:
ToolError: If the request fails
"""
params = {"fast": fast} if fast is not None else None
with telemetry.scope(
with logfire.span(
"mcp.client.knowledge.update_entity",
client_name="knowledge",
operation="update_entity",
fast=fast,
):
response = await call_put(
self.http_client,
f"{self._base_path}/entities/{entity_id}",
json=entity_data,
params=params,
client_name="knowledge",
operation="update_entity",
path_template="/v2/projects/{project_id}/knowledge/entities/{entity_id}",
@@ -125,7 +115,7 @@ class KnowledgeClient:
Raises:
ToolError: If the entity is not found or request fails
"""
with telemetry.scope(
with logfire.span(
"mcp.client.knowledge.get_entity",
client_name="knowledge",
operation="get_entity",
@@ -143,8 +133,6 @@ class KnowledgeClient:
self,
entity_id: str,
patch_data: dict[str, Any],
*,
fast: bool | None = None,
) -> EntityResponse:
"""Partially update an entity.
@@ -158,18 +146,15 @@ class KnowledgeClient:
Raises:
ToolError: If the request fails
"""
params = {"fast": fast} if fast is not None else None
with telemetry.scope(
with logfire.span(
"mcp.client.knowledge.patch_entity",
client_name="knowledge",
operation="patch_entity",
fast=fast,
):
response = await call_patch(
self.http_client,
f"{self._base_path}/entities/{entity_id}",
json=patch_data,
params=params,
client_name="knowledge",
operation="patch_entity",
path_template="/v2/projects/{project_id}/knowledge/entities/{entity_id}",
@@ -188,7 +173,7 @@ class KnowledgeClient:
Raises:
ToolError: If the entity is not found or request fails
"""
with telemetry.scope(
with logfire.span(
"mcp.client.knowledge.delete_entity",
client_name="knowledge",
operation="delete_entity",
@@ -215,7 +200,7 @@ class KnowledgeClient:
Raises:
ToolError: If the request fails
"""
with telemetry.scope(
with logfire.span(
"mcp.client.knowledge.move_entity",
client_name="knowledge",
operation="move_entity",
@@ -245,7 +230,7 @@ class KnowledgeClient:
Raises:
ToolError: If the request fails
"""
with telemetry.scope(
with logfire.span(
"mcp.client.knowledge.move_directory",
client_name="knowledge",
operation="move_directory",
@@ -275,7 +260,7 @@ class KnowledgeClient:
Raises:
ToolError: If the request fails
"""
with telemetry.scope(
with logfire.span(
"mcp.client.knowledge.delete_directory",
client_name="knowledge",
operation="delete_directory",
@@ -305,7 +290,7 @@ class KnowledgeClient:
Raises:
ToolError: If the identifier cannot be resolved
"""
with telemetry.scope(
with logfire.span(
"mcp.client.knowledge.resolve_entity",
client_name="knowledge",
operation="resolve_entity",
+3 -3
View File
@@ -7,7 +7,7 @@ from typing import Optional
from httpx import AsyncClient
from basic_memory import telemetry
import logfire
from basic_memory.mcp.tools.utils import call_get
from basic_memory.schemas.memory import GraphContext
@@ -72,7 +72,7 @@ class MemoryClient:
if timeframe:
params["timeframe"] = timeframe
with telemetry.scope(
with logfire.span(
"mcp.client.memory.build_context",
client_name="memory",
operation="build_context",
@@ -123,7 +123,7 @@ class MemoryClient:
# Join types as comma-separated string if provided
params["type"] = ",".join(types) if isinstance(types, list) else types
with telemetry.scope(
with logfire.span(
"mcp.client.memory.recent_activity",
client_name="memory",
operation="recent_activity",
+3 -22
View File
@@ -3,11 +3,9 @@
Encapsulates all /v2/projects/{project_id}/resource/* endpoints.
"""
from typing import Optional
from httpx import AsyncClient, Response
from basic_memory import telemetry
import logfire
from basic_memory.mcp.tools.utils import call_get
@@ -39,19 +37,11 @@ class ResourceClient:
self.project_id = project_id
self._base_path = f"/v2/projects/{project_id}/resource"
async def read(
self,
entity_id: str,
*,
page: Optional[int] = None,
page_size: Optional[int] = None,
) -> Response:
async def read(self, entity_id: str) -> Response:
"""Read a resource by entity ID.
Args:
entity_id: Entity external_id (UUID)
page: Optional page number for paginated content
page_size: Optional page size for paginated content
Returns:
Raw HTTP Response (caller handles text/binary content)
@@ -59,23 +49,14 @@ class ResourceClient:
Raises:
ToolError: If the resource is not found or request fails
"""
params: dict = {}
if page is not None:
params["page"] = page
if page_size is not None:
params["page_size"] = page_size
with telemetry.scope(
with logfire.span(
"mcp.client.resource.read",
client_name="resource",
operation="read",
page=page,
page_size=page_size,
):
return await call_get(
self.http_client,
f"{self._base_path}/{entity_id}",
params=params if params else None,
client_name="resource",
operation="read",
path_template="/v2/projects/{project_id}/resource/{entity_id}",
+2 -2
View File
@@ -7,7 +7,7 @@ from typing import Any
from httpx import AsyncClient
from basic_memory import telemetry
import logfire
from basic_memory.mcp.tools.utils import call_post
from basic_memory.schemas.search import SearchResponse
@@ -57,7 +57,7 @@ class SearchClient:
Raises:
ToolError: If the request fails
"""
with telemetry.scope(
with logfire.span(
"mcp.client.search.search",
client_name="search",
operation="search",
File diff suppressed because it is too large Load Diff
+7 -7
View File
@@ -95,8 +95,8 @@ def format_prompt_context(context: PromptContext) -> str:
sections = []
# Process each context
for context in context.results: # pyright: ignore
for primary in context.primary_results: # pyright: ignore
for context_item in context.results:
for primary in context_item.primary_results:
if primary.permalink not in added_permalinks:
primary_permalink = primary.permalink
@@ -121,8 +121,8 @@ def format_prompt_context(context: PromptContext) -> str:
section += f"- **Created**: {primary.created_at.strftime('%Y-%m-%d %H:%M')}\n"
# Add content snippet
if hasattr(primary, "content") and primary.content: # pyright: ignore
content = primary.content or "" # pyright: ignore # pragma: no cover
if hasattr(primary, "content") and primary.content:
content = primary.content or "" # pragma: no cover
if content: # pragma: no cover
section += f"\n**Excerpt**:\n{content}\n" # pragma: no cover
@@ -132,14 +132,14 @@ def format_prompt_context(context: PromptContext) -> str:
""")
sections.append(section)
if context.related_results: # pyright: ignore
section += dedent( # pyright: ignore
if context_item.related_results:
section += dedent(
"""
## Related Context
"""
)
for related in context.related_results: # pyright: ignore
for related in context_item.related_results:
section_content = dedent(f"""
- type: **{related.type}**
- title: {related.title}
@@ -18,13 +18,15 @@ Basic Memory creates a semantic knowledge graph from markdown files. Focus on bu
**Resolution priority:**
1. CLI constraint: `BASIC_MEMORY_MCP_PROJECT` env var (highest priority)
2. Explicit parameter: `project="name"` in tool calls
2. Explicit parameter: `project_id="<uuid>"` (preferred when known) or `project="name"` in tool calls
3. Default project: `default_project` in config (fallback)
**`project` vs `project_id`:** Every project has a stable `external_id` (UUID) returned by `list_memory_projects()`. Pass it as `project_id=...` to address a project unambiguously — required when the same project name exists in multiple cloud workspaces. For local single-project setups, the `project` name is fine.
### Quick Setup Check
```python
# Discover projects
# Discover projects (each entry includes external_id you can pass as project_id)
projects = await list_memory_projects()
```
@@ -169,6 +171,14 @@ await write_note(
**Multi-project users:**
- Always specify project explicitly in tool calls
**Cloud multi-workspace users:** project names can collide across workspaces. After calling `list_memory_projects()`, prefer the project's `external_id` via `project_id=...` for any subsequent tool calls — it routes to the exact project regardless of name collisions. The `project` name parameter falls back to the default workspace on ambiguity, which may not be what you want.
```python
# Cloud / multi-workspace: prefer project_id (UUID) once you've discovered it
projects = await list_memory_projects()
results = await search_notes(query="auth", project_id=projects[0]["external_id"])
```
**Discovery:**
```python
# Start with discovery
+5 -39
View File
@@ -2,7 +2,6 @@
Basic Memory FastMCP server.
"""
import asyncio
import time
from contextlib import asynccontextmanager
@@ -13,15 +12,10 @@ from sqlalchemy.ext.asyncio import async_sessionmaker, AsyncSession
from basic_memory import db
from basic_memory.cli.auth import CLIAuth
from basic_memory.config import BasicMemoryConfig
from basic_memory.db import (
scoped_session,
_needs_semantic_embedding_backfill,
_run_semantic_embedding_backfill,
)
from basic_memory.db import scoped_session
from basic_memory.mcp.container import McpContainer, set_container
from basic_memory.services.initialization import initialize_app
from basic_memory import telemetry
import logfire
async def _log_embedding_status(session_maker: async_sessionmaker[AsyncSession]) -> None:
@@ -43,7 +37,7 @@ async def _log_embedding_status(session_maker: async_sessionmaker[AsyncSession])
elif embedding_count == 0:
logger.warning(
f"Semantic embeddings: EMPTY — {entity_count} entities have no embeddings. "
"Backfill running in background..."
"Run 'bm reindex --embeddings' to build them."
)
else:
logger.info(
@@ -54,20 +48,6 @@ async def _log_embedding_status(session_maker: async_sessionmaker[AsyncSession])
logger.debug(f"Could not check embedding status at startup: {exc}")
async def _background_embedding_backfill(
config: BasicMemoryConfig,
session_maker: async_sessionmaker[AsyncSession],
) -> None:
"""Run semantic embedding backfill in the background without blocking startup."""
try:
if await _needs_semantic_embedding_backfill(config, session_maker):
logger.info("Background embedding backfill starting...")
await _run_semantic_embedding_backfill(config, session_maker)
await _log_embedding_status(session_maker)
except Exception as exc:
logger.error(f"Background embedding backfill failed: {exc}")
@asynccontextmanager
async def lifespan(app: FastMCP):
"""Lifecycle manager for the MCP server.
@@ -83,7 +63,7 @@ async def lifespan(app: FastMCP):
set_container(container)
config = container.config
with telemetry.operation(
with logfire.span(
"mcp.lifecycle.startup",
entrypoint="mcp",
mode=container.mode.name.lower(),
@@ -133,14 +113,8 @@ async def lifespan(app: FastMCP):
await initialize_app(container.config)
# Log embedding status so it's easy to spot in the logs
backfill_task: asyncio.Task | None = None # type: ignore[type-arg]
if config.semantic_search_enabled and db._session_maker is not None:
await _log_embedding_status(db._session_maker)
# Launch backfill in background so MCP server is ready immediately
backfill_task = asyncio.create_task(
_background_embedding_backfill(config, db._session_maker),
name="embedding-backfill",
)
# Create and start sync coordinator (lifecycle centralized in coordinator)
sync_coordinator = container.create_sync_coordinator()
@@ -150,21 +124,13 @@ async def lifespan(app: FastMCP):
yield
finally:
# Shutdown - coordinator handles clean task cancellation
with telemetry.operation(
with logfire.span(
"mcp.lifecycle.shutdown",
entrypoint="mcp",
mode=container.mode.name.lower(),
):
logger.debug("Shutting down Basic Memory MCP server")
# Cancel embedding backfill if still running
if backfill_task is not None and not backfill_task.done():
backfill_task.cancel()
try:
await backfill_task
except asyncio.CancelledError:
logger.info("Background embedding backfill cancelled during shutdown")
await sync_coordinator.stop()
# Only shutdown DB if we created it (not if test fixture provided it)
+80 -53
View File
@@ -1,14 +1,15 @@
"""Build context tool for Basic Memory MCP server."""
from typing import Optional, Literal
from typing import Annotated, Optional, Literal
import logfire
from loguru import logger
from fastmcp import Context
from pydantic import AliasChoices, Field
from basic_memory.config import ConfigManager
from basic_memory import telemetry
from basic_memory.mcp.project_context import (
detect_project_from_url_prefix,
detect_project_from_memory_url_prefix,
get_project_client,
resolve_project_and_path,
)
@@ -133,14 +134,34 @@ def _format_context_markdown(graph: GraphContext, project: str) -> str:
annotations={"readOnlyHint": True, "openWorldHint": False},
)
async def build_context(
url: MemoryUrl,
url: Annotated[
MemoryUrl,
Field(validation_alias=AliasChoices("url", "uri", "memory_url")),
],
project: Optional[str] = None,
workspace: Optional[str] = None,
project_id: Optional[str] = None,
depth: str | int | None = 1,
timeframe: Optional[TimeFrame] = "7d",
page: int = 1,
page_size: int = 10,
max_related: int = 10,
timeframe: Annotated[
Optional[TimeFrame],
Field(
default="7d",
validation_alias=AliasChoices("timeframe", "since", "time_range", "lookback"),
),
] = "7d",
# `offset` is intentionally NOT aliased: it has different semantics
# (item-indexed vs. 1-indexed page-number).
page: Annotated[
int,
Field(default=1, validation_alias=AliasChoices("page", "page_number")),
] = 1,
page_size: Annotated[
int,
Field(default=10, validation_alias=AliasChoices("page_size", "limit", "per_page")),
] = 10,
max_related: Annotated[
int,
Field(default=10, validation_alias=AliasChoices("max_related", "max_results")),
] = 10,
output_format: Literal["json", "text"] = "json",
context: Context | None = None,
) -> dict | str:
@@ -158,6 +179,9 @@ async def build_context(
Args:
project: Project name to build context from. Optional - server will resolve using hierarchy.
If unknown, use list_memory_projects() to discover available projects.
project_id: Project external_id (UUID). Prefer this over `project` when known
it routes to the exact project regardless of name collisions across cloud
workspaces. Takes precedence over `project`. Get from list_memory_projects().
url: memory:// URI pointing to discussion content (e.g. memory://specs/search)
depth: How many relation hops to traverse (1-3 recommended for performance)
timeframe: How far back to look. Supports natural language like "2 days ago", "last week"
@@ -185,9 +209,14 @@ async def build_context(
Raises:
ToolError: If project doesn't exist or depth parameter is invalid
"""
# Detect project from memory URL prefix before routing
if project is None:
detected = detect_project_from_url_prefix(url, ConfigManager().config)
# Detect project from memory URL prefix before routing.
# project_id routes by external UUID, so it bypasses URL discovery entirely.
if project is None and project_id is None:
detected = await detect_project_from_memory_url_prefix(
url,
ConfigManager().config,
context=context,
)
if detected:
project = detected
@@ -202,12 +231,12 @@ async def build_context(
# URL is already validated and normalized by MemoryUrl type annotation
with telemetry.operation(
with logfire.span(
"mcp.tool.build_context",
entrypoint="mcp",
tool_name="build_context",
requested_project=project,
workspace_id=workspace,
requested_project_id=project_id,
depth=depth or 1,
timeframe=timeframe,
page=page,
@@ -216,48 +245,46 @@ async def build_context(
output_format=output_format,
is_memory_url=str(url).startswith("memory://"),
):
async with get_project_client(project, workspace, context) as (client, active_project):
with telemetry.contextualize(
project_name=active_project.name,
workspace_id=workspace,
tool_name="build_context",
):
logger.info(
f"MCP tool call tool=build_context project={active_project.name} "
f"url={url} depth={depth} timeframe={timeframe} output_format={output_format}"
)
async with get_project_client(project, context=context, project_id=project_id) as (
client,
active_project,
):
logger.info(
f"MCP tool call tool=build_context project={active_project.name} "
f"url={url} depth={depth} timeframe={timeframe} output_format={output_format}"
)
# Resolve memory:// identifier with project-prefix awareness
_, resolved_path, _ = await resolve_project_and_path(
client,
url,
active_project.name,
context,
)
# Resolve memory:// identifier with project-prefix awareness
_, resolved_path, _ = await resolve_project_and_path(
client,
url,
active_project.name,
context,
)
# Import here to avoid circular import
from basic_memory.mcp.clients import MemoryClient
# Import here to avoid circular import
from basic_memory.mcp.clients import MemoryClient
# Use typed MemoryClient for API calls
memory_client = MemoryClient(client, active_project.external_id)
graph = await memory_client.build_context(
resolved_path,
depth=depth or 1,
timeframe=timeframe,
page=page,
page_size=page_size,
max_related=max_related,
)
# Use typed MemoryClient for API calls
memory_client = MemoryClient(client, active_project.external_id)
graph = await memory_client.build_context(
resolved_path,
depth=depth or 1,
timeframe=timeframe,
page=page,
page_size=page_size,
max_related=max_related,
)
logger.info(
f"MCP tool response: tool=build_context project={active_project.name} "
f"uri={graph.metadata.uri or resolved_path} "
f"primary_count={graph.metadata.primary_count or 0} "
f"related_count={graph.metadata.related_count or 0} "
f"output_format={output_format}"
)
logger.info(
f"MCP tool response: tool=build_context project={active_project.name} "
f"uri={graph.metadata.uri or resolved_path} "
f"primary_count={graph.metadata.primary_count or 0} "
f"related_count={graph.metadata.related_count or 0} "
f"output_format={output_format}"
)
if output_format == "text":
return _format_context_markdown(graph, active_project.name)
if output_format == "text":
return _format_context_markdown(graph, active_project.name)
return graph.model_dump()
return graph.model_dump()
+13 -4
View File
@@ -8,7 +8,7 @@ from typing import Annotated, Dict, List, Any, Optional
from loguru import logger
from fastmcp import Context
from pydantic import BeforeValidator
from pydantic import AliasChoices, BeforeValidator, Field
from basic_memory.mcp.project_context import get_project_client
from basic_memory.utils import coerce_list
@@ -24,9 +24,12 @@ async def canvas(
nodes: Annotated[List[Dict[str, Any]], BeforeValidator(coerce_list)],
edges: Annotated[List[Dict[str, Any]], BeforeValidator(coerce_list)],
title: str,
directory: str,
directory: Annotated[
str,
Field(validation_alias=AliasChoices("directory", "folder", "dir", "path")),
],
project: Optional[str] = None,
workspace: Optional[str] = None,
project_id: Optional[str] = None,
context: Context | None = None,
) -> str:
"""Create an Obsidian canvas file with the provided nodes and edges.
@@ -43,6 +46,9 @@ async def canvas(
Args:
project: Project name to create canvas in. Optional - server will resolve using hierarchy.
If unknown, use list_memory_projects() to discover available projects.
project_id: Project external_id (UUID). Prefer this over `project` when known
it routes to the exact project regardless of name collisions across cloud
workspaces. Takes precedence over `project`. Get from list_memory_projects().
nodes: List of node objects following JSON Canvas 1.0 spec
edges: List of edge objects following JSON Canvas 1.0 spec
title: The title of the canvas (will be saved as title.canvas)
@@ -97,7 +103,10 @@ async def canvas(
Raises:
ToolError: If project doesn't exist or directory path is invalid
"""
async with get_project_client(project, workspace, context) as (client, active_project):
async with get_project_client(project, context=context, project_id=project_id) as (
client,
active_project,
):
# Ensure path has .canvas extension
file_title = title if title.endswith(".canvas") else f"{title}.canvas"
file_path = f"{directory}/{file_title}"
@@ -181,8 +181,6 @@ async def fetch(
content = str(
await read_note(
identifier=id,
page=1,
page_size=10,
context=context,
)
)
+76 -10
View File
@@ -1,13 +1,21 @@
from textwrap import dedent
from typing import Optional, Literal
from typing import Annotated, Optional, Literal
from loguru import logger
from fastmcp import Context
from mcp.server.fastmcp.exceptions import ToolError
from pydantic import AliasChoices, Field
from basic_memory.config import ConfigManager
from basic_memory.mcp.project_context import detect_project_from_url_prefix, get_project_client
from basic_memory.mcp.project_context import (
detect_project_from_memory_url_prefix,
get_project_client,
resolve_project_and_path,
)
from basic_memory.mcp.server import mcp
from basic_memory.schemas.project_info import ProjectItem
from basic_memory.utils import generate_permalink, normalize_project_reference
from basic_memory.workspace_context import current_workspace_permalink_context
def _format_delete_error_response(project: str, error_message: str, identifier: str) -> str:
@@ -147,15 +155,44 @@ delete_note("{project}", "correct-identifier-from-search")
If the note should be deleted but the operation keeps failing, send a message to support@basicmemory.com."""
def _directory_path_for_delete(
target_identifier: str,
active_project: ProjectItem,
*,
include_project_prefix: bool,
) -> str:
"""Return the project-relative directory path expected by the delete API."""
directory = normalize_project_reference(target_identifier).strip("/")
project_permalink = active_project.permalink
route_prefixes: list[str] = []
workspace_context = current_workspace_permalink_context()
if workspace_context and workspace_context.should_prefix_permalinks:
route_prefixes.append(
f"{generate_permalink(workspace_context.workspace_slug)}/{project_permalink}"
)
if include_project_prefix:
route_prefixes.append(project_permalink)
for route_prefix in route_prefixes:
if directory.startswith(f"{route_prefix}/"):
return directory.removeprefix(f"{route_prefix}/")
return directory
@mcp.tool(
description="Delete a note or directory by title, permalink, or path",
annotations={"destructiveHint": True, "openWorldHint": False},
)
async def delete_note(
identifier: str,
is_directory: bool = False,
is_directory: Annotated[
bool,
Field(default=False, validation_alias=AliasChoices("is_directory", "is_dir")),
] = False,
project: Optional[str] = None,
workspace: Optional[str] = None,
project_id: Optional[str] = None,
output_format: Literal["text", "json"] = "text",
context: Context | None = None,
) -> bool | str | dict:
@@ -179,6 +216,9 @@ async def delete_note(
(without file extensions). Defaults to False.
project: Project name to delete from. Optional - server will resolve using hierarchy.
If unknown, use list_memory_projects() to discover available projects.
project_id: Project external_id (UUID). Prefer this over `project` when known
it routes to the exact project regardless of name collisions across cloud
workspaces. Takes precedence over `project`. Get from list_memory_projects().
output_format: "text" preserves existing behavior (bool/string). "json"
returns machine-readable deletion metadata.
context: Optional FastMCP context for performance caching.
@@ -224,16 +264,23 @@ async def delete_note(
commands and alternative formats to try.
"""
# Detect project from memory URL prefix before routing
# Trigger: identifier starts with memory:// and no explicit project was provided
# Trigger: identifier starts with memory:// and no explicit project/project_id was provided
# Why: only gate on memory:// to avoid misrouting plain paths like "research/note"
# where "research" is a directory, not a project name
# Outcome: project is set from the URL prefix, routing goes to the correct project
if project is None and identifier.strip().startswith("memory://"):
detected = detect_project_from_url_prefix(identifier, ConfigManager().config)
if project is None and project_id is None and identifier.strip().startswith("memory://"):
detected = await detect_project_from_memory_url_prefix(
identifier,
ConfigManager().config,
context=context,
)
if detected:
project = detected
async with get_project_client(project, workspace, context) as (client, active_project):
async with get_project_client(project, context=context, project_id=project_id) as (
client,
active_project,
):
logger.debug(
f"Deleting {'directory' if is_directory else 'note'}: {identifier} in project: {active_project.name}"
)
@@ -243,11 +290,30 @@ async def delete_note(
# Use typed KnowledgeClient for API calls
knowledge_client = KnowledgeClient(client, active_project.external_id)
_, target_identifier, is_memory_url = await resolve_project_and_path(
client,
identifier,
active_project.name,
context,
)
# Handle directory deletes
if is_directory:
try:
result = await knowledge_client.delete_directory(identifier)
# Trigger: directory input was routed from a memory:// URL.
# Why: resolve_project_and_path returns canonical permalinks, while
# delete_directory filters by project-relative file_path prefixes.
# Outcome: strip only the route prefix before calling the delete API.
directory_identifier = (
_directory_path_for_delete(
target_identifier,
active_project,
include_project_prefix=ConfigManager().config.permalinks_include_project,
)
if is_memory_url
else target_identifier
)
result = await knowledge_client.delete_directory(directory_identifier)
if output_format == "json":
return {
"deleted": result.failed_deletes == 0,
@@ -329,7 +395,7 @@ delete_note("path/to/file.md")
note_file_path = None
try:
# Resolve identifier to entity ID
entity_id = await knowledge_client.resolve_entity(identifier, strict=True)
entity_id = await knowledge_client.resolve_entity(target_identifier, strict=True)
if output_format == "json":
entity = await knowledge_client.get_entity(entity_id)
note_title = entity.title
+248 -218
View File
@@ -1,16 +1,18 @@
"""Edit note tool for Basic Memory MCP server."""
from typing import Optional, Literal
from typing import Annotated, Optional, Literal
import logfire
from loguru import logger
from fastmcp import Context
from pydantic import AliasChoices, Field
from basic_memory.config import ConfigManager
from basic_memory import telemetry
from basic_memory.mcp.project_context import (
detect_project_from_url_prefix,
detect_project_from_memory_url_prefix,
get_project_client,
add_project_metadata,
resolve_project_and_path,
)
from basic_memory.mcp.server import mcp
from basic_memory.schemas.base import Entity
@@ -170,11 +172,33 @@ Error editing note '{identifier}': {error_message}
async def edit_note(
identifier: str,
operation: str,
content: str,
# Accept common replacement-content aliases. Models trained on diff/patch
# APIs reach for new_content/replacement/replace_with on first try.
content: Annotated[
str,
Field(
validation_alias=AliasChoices("content", "new_content", "replacement", "replace_with")
),
],
project: Optional[str] = None,
workspace: Optional[str] = None,
section: Optional[str] = None,
find_text: Optional[str] = None,
project_id: Optional[str] = None,
# Section/heading naming varies across tools; accept the descriptive forms.
section: Annotated[
Optional[str],
Field(
default=None,
validation_alias=AliasChoices("section", "section_heading", "heading"),
),
] = None,
# find_text is the highest-frequency miss per the issue: models reach for
# find/old_text/old_content/search before find_text every time.
find_text: Annotated[
Optional[str],
Field(
default=None,
validation_alias=AliasChoices("find_text", "find", "old_text", "old_content", "search"),
),
] = None,
expected_replacements: Optional[int] = None,
output_format: Literal["text", "json"] = "text",
context: Context | None = None,
@@ -201,6 +225,9 @@ async def edit_note(
content: The content to add or use for replacement
project: Project name to edit in. Optional - server will resolve using hierarchy.
If unknown, use list_memory_projects() to discover available projects.
project_id: Project external_id (UUID). Prefer this over `project` when known
it routes to the exact project regardless of name collisions across cloud
workspaces. Takes precedence over `project`. Get from list_memory_projects().
section: For replace_section operation - the markdown header to replace content under (e.g., "## Notes", "### Implementation")
find_text: For find_replace operation - the text to find and replace
expected_replacements: For find_replace operation - the expected number of replacements (validation will fail if actual doesn't match)
@@ -262,252 +289,255 @@ async def edit_note(
effective_replacements = expected_replacements if expected_replacements is not None else 1
# Detect project from memory URL prefix before routing
# Trigger: identifier starts with memory:// and no explicit project was provided
# Trigger: identifier starts with memory:// and no explicit project/project_id was provided
# Why: only gate on memory:// to avoid misrouting plain paths like "research/note"
# where "research" is a directory, not a project name
# Outcome: project is set from the URL prefix, routing goes to the correct project
if project is None and identifier.strip().startswith("memory://"):
detected = detect_project_from_url_prefix(identifier, ConfigManager().config)
if project is None and project_id is None and identifier.strip().startswith("memory://"):
detected = await detect_project_from_memory_url_prefix(
identifier,
ConfigManager().config,
context=context,
)
if detected:
project = detected
with telemetry.operation(
with logfire.span(
"mcp.tool.edit_note",
entrypoint="mcp",
tool_name="edit_note",
requested_project=project,
workspace_id=workspace,
requested_project_id=project_id,
edit_operation=operation,
output_format=output_format,
has_section=bool(section),
has_find_text=bool(find_text),
expected_replacements=effective_replacements,
):
async with get_project_client(project, workspace, context) as (client, active_project):
with telemetry.contextualize(
project_name=active_project.name,
workspace_id=workspace,
tool_name="edit_note",
):
logger.info(
f"MCP tool call tool=edit_note project={active_project.name} "
f"identifier={identifier} operation={operation} output_format={output_format}"
async with get_project_client(project, context=context, project_id=project_id) as (
client,
active_project,
):
logger.info(
f"MCP tool call tool=edit_note project={active_project.name} "
f"identifier={identifier} operation={operation} output_format={output_format}"
)
# Validate operation
valid_operations = [
"append",
"prepend",
"find_replace",
"replace_section",
"insert_before_section",
"insert_after_section",
]
if operation not in valid_operations:
raise ValueError(
f"Invalid operation '{operation}'. Must be one of: {', '.join(valid_operations)}"
)
# Validate operation
valid_operations = [
"append",
"prepend",
"find_replace",
"replace_section",
"insert_before_section",
"insert_after_section",
]
if operation not in valid_operations:
raise ValueError(
f"Invalid operation '{operation}'. Must be one of: {', '.join(valid_operations)}"
)
# Validate required parameters for specific operations
if operation == "find_replace" and not find_text:
raise ValueError("find_text parameter is required for find_replace operation")
section_ops = ("replace_section", "insert_before_section", "insert_after_section")
if operation in section_ops and not section:
raise ValueError("section parameter is required for section-based operations")
# Validate required parameters for specific operations
if operation == "find_replace" and not find_text:
raise ValueError("find_text parameter is required for find_replace operation")
section_ops = ("replace_section", "insert_before_section", "insert_after_section")
if operation in section_ops and not section:
raise ValueError("section parameter is required for section-based operations")
# Use the PATCH endpoint to edit the entity
try:
# Import here to avoid circular import
from basic_memory.mcp.clients import KnowledgeClient
# Use the PATCH endpoint to edit the entity
# Use typed KnowledgeClient for API calls
knowledge_client = KnowledgeClient(client, active_project.external_id)
_, entity_identifier, _ = await resolve_project_and_path(
client,
identifier,
active_project.name,
context,
)
file_created = False
entity_id = ""
result: EntityResponse | None = None
# Try to resolve the entity; for append/prepend, create it if not found
try:
# Import here to avoid circular import
from basic_memory.mcp.clients import KnowledgeClient
entity_id = await knowledge_client.resolve_entity(
entity_identifier,
strict=True,
)
except Exception as resolve_error:
# Trigger: entity does not exist yet
# Why: append/prepend can meaningfully create a new note from the content,
# while find_replace/replace_section require existing content to modify
# Outcome: note is created via the same path as write_note
error_msg = str(resolve_error).lower()
is_not_found = "entity not found" in error_msg or "not found" in error_msg
# Use typed KnowledgeClient for API calls
knowledge_client = KnowledgeClient(client, active_project.external_id)
if is_not_found and operation in ("append", "prepend"):
title, directory = _parse_identifier_to_title_and_directory(identifier)
file_created = False
entity_id = ""
result: EntityResponse | None = None
# Try to resolve the entity; for append/prepend, create it if not found
try:
entity_id = await knowledge_client.resolve_entity(identifier, strict=True)
except Exception as resolve_error:
# Trigger: entity does not exist yet
# Why: append/prepend can meaningfully create a new note from the content,
# while find_replace/replace_section require existing content to modify
# Outcome: note is created via the same path as write_note
error_msg = str(resolve_error).lower()
is_not_found = "entity not found" in error_msg or "not found" in error_msg
if is_not_found and operation in ("append", "prepend"):
title, directory = _parse_identifier_to_title_and_directory(identifier)
# Validate directory path (same security check as write_note)
project_path = active_project.home
if directory and not validate_project_path(directory, project_path):
logger.warning(
"Attempted path traversal attack blocked",
directory=directory,
project=active_project.name,
)
if output_format == "json":
return {
"title": title,
"permalink": None,
"file_path": None,
"checksum": None,
"operation": operation,
"fileCreated": False,
"error": "SECURITY_VALIDATION_ERROR",
}
return f"# Error\n\nDirectory path '{directory}' is not allowed - paths must stay within project boundaries"
entity = Entity(
title=title,
# Validate directory path (same security check as write_note)
project_path = active_project.home
if directory and not validate_project_path(directory, project_path):
logger.warning(
"Attempted path traversal attack blocked",
directory=directory,
content_type="text/markdown",
content=content,
project=active_project.name,
)
if output_format == "json":
return {
"title": title,
"permalink": None,
"file_path": None,
"checksum": None,
"operation": operation,
"fileCreated": False,
"error": "SECURITY_VALIDATION_ERROR",
}
return f"# Error\n\nDirectory path '{directory}' is not allowed - paths must stay within project boundaries"
logger.info(
"Creating note via edit_note auto-create",
title=title,
directory=directory,
operation=operation,
)
result = await knowledge_client.create_entity(
entity.model_dump(), fast=False
)
file_created = True
else:
# find_replace/replace_section require existing content — re-raise
raise resolve_error
# --- Standard edit path (entity already existed) ---
if not file_created:
# Prepare the edit request data
edit_data = {
"operation": operation,
"content": content,
}
# Add optional parameters
if section:
edit_data["section"] = section
if find_text:
edit_data["find_text"] = find_text
if effective_replacements != 1: # Only send if different from default
edit_data["expected_replacements"] = str(effective_replacements)
# Call the PATCH endpoint
result = await knowledge_client.patch_entity(
entity_id, edit_data, fast=False
entity = Entity(
title=title,
directory=directory,
content_type="text/markdown",
content=content,
)
# --- Format response ---
# result is always set: either by create_entity (auto-create) or patch_entity (edit)
assert result is not None
if file_created:
summary = [
f"# Created note ({operation})",
f"project: {active_project.name}",
f"file_path: {result.file_path}",
f"permalink: {result.permalink}",
f"checksum: {result.checksum[:8] if result.checksum else 'unknown'}",
"fileCreated: true",
]
lines_added = len(content.split("\n"))
summary.append(f"operation: Created note with {lines_added} lines")
logger.info(
"Creating note via edit_note auto-create",
title=title,
directory=directory,
operation=operation,
)
result = await knowledge_client.create_entity(entity.model_dump())
file_created = True
else:
summary = [
f"# Edited note ({operation})",
f"project: {active_project.name}",
f"file_path: {result.file_path}",
f"permalink: {result.permalink}",
f"checksum: {result.checksum[:8] if result.checksum else 'unknown'}",
]
# find_replace/replace_section require existing content — re-raise
raise resolve_error
# Add operation-specific details
if operation == "append":
lines_added = len(content.split("\n"))
summary.append(f"operation: Added {lines_added} lines to end of note")
elif operation == "prepend":
lines_added = len(content.split("\n"))
summary.append(
f"operation: Added {lines_added} lines to beginning of note"
)
elif operation == "find_replace":
# For find_replace, we can't easily count replacements from here
# since we don't have the original content, but the server handled it
summary.append("operation: Find and replace operation completed")
elif operation == "replace_section":
summary.append(f"operation: Replaced content under section '{section}'")
elif operation == "insert_before_section":
summary.append(
f"operation: Inserted content before section '{section}'"
)
elif operation == "insert_after_section":
summary.append(f"operation: Inserted content after section '{section}'")
# --- Standard edit path (entity already existed) ---
if not file_created:
# Prepare the edit request data
edit_data = {
"operation": operation,
"content": content,
}
# Count observations by category (reuse logic from write_note)
categories = {}
if result.observations:
for obs in result.observations:
categories[obs.category] = categories.get(obs.category, 0) + 1
# Add optional parameters
if section:
edit_data["section"] = section
if find_text:
edit_data["find_text"] = find_text
if effective_replacements != 1: # Only send if different from default
edit_data["expected_replacements"] = str(effective_replacements)
summary.append("\n## Observations")
for category, count in sorted(categories.items()):
summary.append(f"- {category}: {count}")
# Call the PATCH endpoint
result = await knowledge_client.patch_entity(entity_id, edit_data)
# Count resolved/unresolved relations
unresolved = 0
resolved = 0
if result.relations:
unresolved = sum(1 for r in result.relations if not r.to_id)
resolved = len(result.relations) - unresolved
# --- Format response ---
# result is always set: either by create_entity (auto-create) or patch_entity (edit)
assert result is not None
if file_created:
summary = [
f"# Created note ({operation})",
f"project: {active_project.name}",
f"file_path: {result.file_path}",
f"permalink: {result.permalink}",
f"checksum: {result.checksum[:8] if result.checksum else 'unknown'}",
"fileCreated: true",
]
lines_added = len(content.split("\n"))
summary.append(f"operation: Created note with {lines_added} lines")
else:
summary = [
f"# Edited note ({operation})",
f"project: {active_project.name}",
f"file_path: {result.file_path}",
f"permalink: {result.permalink}",
f"checksum: {result.checksum[:8] if result.checksum else 'unknown'}",
]
summary.append("\n## Relations")
summary.append(f"- Resolved: {resolved}")
if unresolved:
summary.append(f"- Unresolved: {unresolved}")
# Add operation-specific details
if operation == "append":
lines_added = len(content.split("\n"))
summary.append(f"operation: Added {lines_added} lines to end of note")
elif operation == "prepend":
lines_added = len(content.split("\n"))
summary.append(f"operation: Added {lines_added} lines to beginning of note")
elif operation == "find_replace":
# For find_replace, we can't easily count replacements from here
# since we don't have the original content, but the server handled it
summary.append("operation: Find and replace operation completed")
elif operation == "replace_section":
summary.append(f"operation: Replaced content under section '{section}'")
elif operation == "insert_before_section":
summary.append(f"operation: Inserted content before section '{section}'")
elif operation == "insert_after_section":
summary.append(f"operation: Inserted content after section '{section}'")
logger.info(
f"MCP tool response: tool=edit_note project={active_project.name} "
f"operation={operation} permalink={result.permalink} "
f"observations_count={len(result.observations)} "
f"relations_count={len(result.relations)} "
f"file_created={str(file_created).lower()}"
)
# Count observations by category (reuse logic from write_note)
categories = {}
if result.observations:
for obs in result.observations:
categories[obs.category] = categories.get(obs.category, 0) + 1
if output_format == "json":
return {
"title": result.title,
"permalink": result.permalink,
"file_path": result.file_path,
"checksum": result.checksum,
"operation": operation,
"fileCreated": file_created,
}
summary.append("\n## Observations")
for category, count in sorted(categories.items()):
summary.append(f"- {category}: {count}")
summary_result = "\n".join(summary)
return add_project_metadata(summary_result, active_project.name)
# Count resolved/unresolved relations
unresolved = 0
resolved = 0
if result.relations:
unresolved = sum(1 for r in result.relations if not r.to_id)
resolved = len(result.relations) - unresolved
except Exception as e:
logger.error(f"Error editing note: {e}")
if output_format == "json":
return {
"title": None,
"permalink": None,
"file_path": None,
"checksum": None,
"operation": operation,
"fileCreated": False,
"error": str(e),
}
return _format_error_response(
str(e),
operation,
identifier,
find_text,
effective_replacements,
active_project.name,
)
summary.append("\n## Relations")
summary.append(f"- Resolved: {resolved}")
if unresolved:
summary.append(f"- Unresolved: {unresolved}")
logger.info(
f"MCP tool response: tool=edit_note project={active_project.name} "
f"operation={operation} permalink={result.permalink} "
f"observations_count={len(result.observations)} "
f"relations_count={len(result.relations)} "
f"file_created={str(file_created).lower()}"
)
if output_format == "json":
return {
"title": result.title,
"permalink": result.permalink,
"file_path": result.file_path,
"checksum": result.checksum,
"operation": operation,
"fileCreated": file_created,
}
summary_result = "\n".join(summary)
return add_project_metadata(summary_result, active_project.name)
except Exception as e:
logger.error(f"Error editing note: {e}")
if output_format == "json":
return {
"title": None,
"permalink": None,
"file_path": None,
"checksum": None,
"operation": operation,
"fileCreated": False,
"error": str(e),
}
return _format_error_response(
str(e),
operation,
identifier,
find_text,
effective_replacements,
active_project.name,
)
+25 -5
View File
@@ -1,9 +1,10 @@
"""List directory tool for Basic Memory MCP server."""
from typing import Optional
from typing import Annotated, Optional
from loguru import logger
from fastmcp import Context
from pydantic import AliasChoices, Field
from basic_memory.mcp.project_context import get_project_client
from basic_memory.mcp.server import mcp
@@ -14,11 +15,24 @@ from basic_memory.mcp.server import mcp
annotations={"readOnlyHint": True, "openWorldHint": False},
)
async def list_directory(
dir_name: str = "/",
# `dir_name` is unusual; models reach for directory/folder/path/dir.
dir_name: Annotated[
str,
Field(
default="/",
validation_alias=AliasChoices("dir_name", "directory", "folder", "path", "dir"),
),
] = "/",
depth: int = 1,
file_name_glob: Optional[str] = None,
file_name_glob: Annotated[
Optional[str],
Field(
default=None,
validation_alias=AliasChoices("file_name_glob", "glob", "pattern", "filter"),
),
] = None,
project: Optional[str] = None,
workspace: Optional[str] = None,
project_id: Optional[str] = None,
context: Context | None = None,
) -> str:
"""List directory contents from the knowledge base with optional filtering.
@@ -36,6 +50,9 @@ async def list_directory(
Examples: "*.md", "*meeting*", "project_*"
project: Project name to list directory from. Optional - server will resolve using hierarchy.
If unknown, use list_memory_projects() to discover available projects.
project_id: Project external_id (UUID). Prefer this over `project` when known
it routes to the exact project regardless of name collisions across cloud
workspaces. Takes precedence over `project`. Get from list_memory_projects().
context: Optional FastMCP context for performance caching.
Returns:
@@ -63,7 +80,10 @@ async def list_directory(
Raises:
ToolError: If project doesn't exist or directory path is invalid
"""
async with get_project_client(project, workspace, context) as (client, active_project):
async with get_project_client(project, context=context, project_id=project_id) as (
client,
active_project,
):
logger.debug(
f"Listing directory '{dir_name}' in project {project} with depth={depth}, glob='{file_name_glob}'"
)
+31 -6
View File
@@ -2,11 +2,12 @@
from pathlib import Path, PureWindowsPath
from textwrap import dedent
from typing import Optional, Literal
from typing import Annotated, Optional, Literal
from loguru import logger
from fastmcp import Context
from mcp.server.fastmcp.exceptions import ToolError
from pydantic import AliasChoices, Field
from basic_memory.mcp.server import mcp
from basic_memory.mcp.project_context import get_project_client
@@ -348,11 +349,29 @@ delete_note("{identifier}")
)
async def move_note(
identifier: str,
destination_path: str = "",
destination_folder: Optional[str] = None,
is_directory: bool = False,
# Move/rename APIs across the ecosystem use `to`/`destination`/`new_path`.
destination_path: Annotated[
str,
Field(
default="",
validation_alias=AliasChoices(
"destination_path", "dest_path", "new_path", "to", "destination"
),
),
] = "",
destination_folder: Annotated[
Optional[str],
Field(
default=None,
validation_alias=AliasChoices("destination_folder", "dest_folder", "to_folder"),
),
] = None,
is_directory: Annotated[
bool,
Field(default=False, validation_alias=AliasChoices("is_directory", "is_dir")),
] = False,
project: Optional[str] = None,
workspace: Optional[str] = None,
project_id: Optional[str] = None,
output_format: Literal["text", "json"] = "text",
context: Context | None = None,
) -> str | dict:
@@ -377,6 +396,9 @@ async def move_note(
(without file extensions). Defaults to False.
project: Project name to move within. Optional - server will resolve using hierarchy.
If unknown, use list_memory_projects() to discover available projects.
project_id: Project external_id (UUID). Prefer this over `project` when known
it routes to the exact project regardless of name collisions across cloud
workspaces. Takes precedence over `project`. Get from list_memory_projects().
output_format: "text" returns existing markdown guidance/success text. "json"
returns machine-readable move metadata.
context: Optional FastMCP context for performance caching.
@@ -476,7 +498,10 @@ async def move_note(
"error": "DESTINATION_FOLDER_NOT_FOR_DIRECTORIES",
}
return f"# Move Failed - Invalid Parameters\n\n{error_msg}"
async with get_project_client(project, workspace, context) as (client, active_project):
async with get_project_client(project, context=context, project_id=project_id) as (
client,
active_project,
):
destination_target = destination_folder or destination_path
logger.info(
f"MCP tool call tool=move_note project={active_project.name} "
+211 -66
View File
@@ -11,7 +11,17 @@ from fastmcp import Context
from loguru import logger
from basic_memory.config import ConfigManager, has_cloud_credentials
from basic_memory.mcp.async_client import get_client, get_cloud_proxy_client, is_factory_mode
from basic_memory.mcp.async_client import (
_explicit_routing,
_force_local_mode,
get_client,
is_factory_mode,
)
from basic_memory.mcp.project_context import (
WorkspaceProjectEntry,
ensure_workspace_project_index,
resolve_workspace_parameter,
)
from basic_memory.mcp.server import mcp
from basic_memory.schemas.project_info import ProjectInfoRequest, ProjectItem, ProjectList
from basic_memory.utils import generate_permalink
@@ -20,30 +30,6 @@ from basic_memory.utils import generate_permalink
# --- Helpers for dual-fetch + merge ---
async def _fetch_cloud_projects(
workspace: str | None = None,
context: Context | None = None,
) -> ProjectList | None:
"""Fetch projects from the cloud API, returning None on failure.
Logs warnings on failure so the caller can fall back to local-only results.
"""
try:
from basic_memory.mcp.clients import ProjectClient
async with get_cloud_proxy_client(workspace=workspace) as cloud_client:
cloud_project_client = ProjectClient(cloud_client)
cloud_list = await cloud_project_client.list_projects()
if context: # pragma: no cover
await context.info(f"Discovered {len(cloud_list.projects)} cloud projects")
return cloud_list
except Exception as exc:
logger.warning(f"Cloud project discovery failed: {exc}")
if context: # pragma: no cover
await context.info("Cloud project discovery failed, showing local projects only")
return None
def _merge_projects(
local_list: ProjectList | None,
cloud_list: ProjectList | None,
@@ -51,6 +37,8 @@ def _merge_projects(
cloud_workspace_name: str | None = None,
cloud_workspace_type: str | None = None,
cloud_workspace_tenant_id: str | None = None,
cloud_workspace_slug: str | None = None,
cloud_workspace_is_default: bool = False,
) -> list[dict]:
"""Merge local and cloud project lists by permalink.
@@ -113,9 +101,13 @@ def _merge_projects(
ws_type = cloud_workspace_type if cloud_proj else None
ws_tenant_id = cloud_workspace_tenant_id if cloud_proj else None
proj = cloud_proj or local_proj
external_id = proj.external_id if proj else None
merged.append(
{
"name": name,
"external_id": external_id,
"path": path,
"local_path": local_path,
"cloud_path": cloud_path,
@@ -126,21 +118,120 @@ def _merge_projects(
"workspace_name": ws_name,
"workspace_type": ws_type,
"workspace_tenant_id": ws_tenant_id,
"workspace_slug": cloud_workspace_slug if cloud_proj else None,
"workspace_is_default": cloud_workspace_is_default if cloud_proj else False,
"qualified_name": (
f"{cloud_workspace_slug}/{permalink}"
if cloud_proj and cloud_workspace_slug
else None
),
}
)
return merged
def _merge_workspace_projects(
local_list: ProjectList | None,
cloud_entries: tuple[WorkspaceProjectEntry, ...],
) -> list[dict]:
"""Merge local projects with cloud projects from every accessible workspace."""
local_by_permalink: dict[str, ProjectItem] = {}
if local_list:
for project in local_list.projects:
local_by_permalink[project.permalink] = project
cloud_permalinks = {entry.project.permalink for entry in cloud_entries}
merged: list[dict] = []
for entry in sorted(
cloud_entries,
key=lambda item: (
not item.workspace.is_default,
item.workspace.workspace_type != "personal",
item.workspace.name.casefold(),
item.project.permalink,
),
):
permalink = entry.project.permalink
local_proj = local_by_permalink.get(permalink)
cloud_proj = entry.project
source = "local+cloud" if local_proj else "cloud"
local_path = local_proj.path if local_proj else None
cloud_path = cloud_proj.path
merged.append(
{
"name": cloud_proj.name,
"external_id": cloud_proj.external_id,
"path": local_path or cloud_path,
"local_path": local_path,
"cloud_path": cloud_path,
"source": source,
"is_default": bool((local_proj and local_proj.is_default) or cloud_proj.is_default),
"is_private": cloud_proj.is_private,
"display_name": cloud_proj.display_name,
"workspace_name": entry.workspace.name,
"workspace_type": entry.workspace.workspace_type,
"workspace_tenant_id": entry.workspace.tenant_id,
"workspace_slug": entry.workspace.slug,
"workspace_is_default": entry.workspace.is_default,
"qualified_name": entry.qualified_name,
}
)
if local_list:
for project in sorted(local_list.projects, key=lambda item: item.permalink):
if project.permalink in cloud_permalinks:
continue
merged.append(
{
"name": project.name,
"external_id": project.external_id,
"path": project.path,
"local_path": project.path,
"cloud_path": None,
"source": "local",
"is_default": project.is_default,
"is_private": project.is_private,
"display_name": project.display_name,
"workspace_name": None,
"workspace_type": None,
"workspace_tenant_id": None,
"workspace_slug": None,
"workspace_is_default": False,
"qualified_name": None,
}
)
return merged
def _format_project_list_text(merged: list[dict]) -> str:
"""Format merged project list as human-readable text."""
result = "Available projects:\n"
current_workspace: tuple[str | None, str | None] | None = None
for project in merged:
workspace_slug = project.get("workspace_slug")
workspace_name = project.get("workspace_name")
if workspace_slug:
workspace_key = (workspace_slug, workspace_name)
if workspace_key != current_workspace:
default_label = " default" if project.get("workspace_is_default") else ""
result += f"\nWorkspace: {workspace_name} ({workspace_slug}{default_label})\n"
current_workspace = workspace_key
elif current_workspace is not None:
result += "\nLocal projects:\n"
current_workspace = None
display_name = project["display_name"]
name = project["name"]
label = f"{display_name} ({name})" if display_name else name
source = project["source"]
result += f"{label} ({source})\n"
external_id = project.get("external_id", "")
id_suffix = f" [{external_id}]" if external_id else ""
result += f"- {label} ({source}){id_suffix}\n"
result += "\n" + "" * 40 + "\n"
result += "Next: Ask which project to use for this session.\n"
@@ -172,7 +263,6 @@ def _format_project_list_json(
)
async def list_memory_projects(
output_format: Literal["text", "json"] = "text",
workspace: str | None = None,
context: Context | None = None,
) -> str | dict:
"""List all available projects with their status.
@@ -180,11 +270,14 @@ async def list_memory_projects(
Shows projects from both local and cloud sources when cloud credentials
are available, merging by permalink to give a unified view.
Each project entry includes an `external_id` (UUID). Pass that value as the
`project_id` parameter on other tools to address a specific project
unambiguously across cloud workspaces useful when the same project name
exists in more than one workspace.
Args:
output_format: "text" returns the existing human-readable project list.
"json" returns structured project metadata.
workspace: Cloud workspace name or tenant_id. Falls back to
config.default_workspace when not specified.
context: Optional FastMCP context for progress/status logging.
"""
if context: # pragma: no cover
@@ -196,18 +289,21 @@ async def list_memory_projects(
# --- Factory mode (cloud app) ---
# Trigger: set_client_factory() was called (e.g., basic-memory-cloud)
# Why: there is no local ASGI server; the factory IS the only source
# Outcome: single fetch, no merge needed
# Why: there is no local ASGI server; the factory IS the cloud source
# Outcome: fetch every accessible workspace so callers can discover cross-workspace IDs
if is_factory_mode():
async with get_client() as client:
project_client = ProjectClient(client)
project_list = await project_client.list_projects()
merged = _merge_projects(project_list, None)
workspace_index = await ensure_workspace_project_index(context=context)
merged = _merge_workspace_projects(None, workspace_index.entries)
default_project = next(
(
entry.project.name
for entry in workspace_index.entries
if entry.workspace.is_default and entry.project.is_default
),
None,
)
if output_format == "json":
return _format_project_list_json(
merged, project_list.default_project, constrained_project
)
return _format_project_list_json(merged, default_project, constrained_project)
if constrained_project:
return _format_constrained_text(constrained_project)
return _format_project_list_text(merged)
@@ -220,39 +316,40 @@ async def list_memory_projects(
# Fetch cloud projects when credentials are available
cloud_list: ProjectList | None = None
cloud_entries: tuple[WorkspaceProjectEntry, ...] = ()
cloud_ws_name: str | None = None
cloud_ws_type: str | None = None
cloud_ws_tenant_id: str | None = None
cloud_ws_slug: str | None = None
cloud_ws_is_default = False
config = ConfigManager().config
if has_cloud_credentials(config):
# Use explicit workspace, fall back to config default
effective_workspace = workspace or config.default_workspace
cloud_list = await _fetch_cloud_projects(effective_workspace, context)
# Resolve workspace metadata so each cloud project carries its workspace info
if cloud_list:
cloud_ws_tenant_id = effective_workspace
try:
from basic_memory.mcp.project_context import get_available_workspaces
workspaces = await get_available_workspaces(context)
matched = next(
(ws for ws in workspaces if ws.tenant_id == effective_workspace),
None,
try:
workspace_index = await ensure_workspace_project_index(context=context)
cloud_entries = workspace_index.entries
except Exception as exc:
logger.warning(
f"Cloud workspace project index discovery failed while listing projects; "
f"showing local-only project list: {exc}"
)
if context: # pragma: no cover
await context.info(
"Cloud workspace project discovery failed while listing projects; "
"showing local projects only"
)
if matched:
cloud_ws_name = matched.name
cloud_ws_type = matched.workspace_type
except Exception:
pass # workspace lookup is best-effort
merged = _merge_projects(
local_list,
cloud_list,
cloud_workspace_name=cloud_ws_name,
cloud_workspace_type=cloud_ws_type,
cloud_workspace_tenant_id=cloud_ws_tenant_id,
)
if cloud_entries:
merged = _merge_workspace_projects(local_list, cloud_entries)
else:
merged = _merge_projects(
local_list,
cloud_list,
cloud_workspace_name=cloud_ws_name,
cloud_workspace_type=cloud_ws_type,
cloud_workspace_tenant_id=cloud_ws_tenant_id,
cloud_workspace_slug=cloud_ws_slug,
cloud_workspace_is_default=cloud_ws_is_default,
)
default_project = local_list.default_project
if output_format == "json":
@@ -272,6 +369,31 @@ def _format_constrained_text(constrained_project: str) -> str:
return result
async def _resolve_create_project_workspace(
workspace: str | None,
context: Context | None,
) -> str | None:
"""Resolve the create-project workspace selector to the routing tenant id."""
if workspace is None:
return None
explicit_cloud_routing = _explicit_routing() and not _force_local_mode()
config = ConfigManager().config
should_resolve_workspace = is_factory_mode() or (
explicit_cloud_routing and has_cloud_credentials(config)
)
if not should_resolve_workspace:
return workspace
# Trigger: cloud routing can use workspace discovery and the caller supplied
# a friendly selector such as a slug, name, or tenant id.
# Why: MCP callers should not need to paste UUIDs, but the transport still
# uses X-Workspace-ID with the tenant id as its routing authority.
# Outcome: resolve once at create time and pass only the tenant id downstream.
resolved_workspace = await resolve_workspace_parameter(workspace=workspace, context=context)
return resolved_workspace.tenant_id
@mcp.tool(
"create_memory_project",
annotations={"destructiveHint": False, "openWorldHint": False},
@@ -280,6 +402,7 @@ async def create_memory_project(
project_name: str,
project_path: str,
set_default: bool = False,
workspace: str | None = None,
output_format: Literal["text", "json"] = "text",
context: Context | None = None,
) -> str | dict:
@@ -292,6 +415,11 @@ async def create_memory_project(
project_name: Name for the new project (must be unique)
project_path: File system path where the project will be stored
set_default: Whether to set this project as the default (optional, defaults to False)
workspace: Optional cloud workspace selector to create the project in. Slug is
preferred for AI callers, but tenant_id and unique name are also accepted.
When omitted, the connection's default workspace is used. Discover values
via `list_workspaces`. Only meaningful in cloud mode; ignored for local
projects.
output_format: "text" returns the existing human-readable result text.
"json" returns structured project creation metadata.
context: Optional FastMCP context for progress/status logging.
@@ -302,8 +430,15 @@ async def create_memory_project(
Example:
create_memory_project("my-research", "~/Documents/research")
create_memory_project("work-notes", "/home/user/work", set_default=True)
create_memory_project("team-notes", "/team/notes", workspace="team-paul")
"""
async with get_client() as client:
workspace_id = await _resolve_create_project_workspace(workspace, context)
# workspace targets a non-default cloud workspace at create time.
# Trigger: caller passed workspace (e.g. a slug discovered via list_workspaces).
# Why: there is no project_id yet for per-project routing — the project doesn't exist.
# Outcome: cloud factory routes the create request to the resolved workspace tenant id.
async with get_client(workspace=workspace_id) as client:
# Check if server is constrained to a specific project
constrained_project = os.environ.get("BASIC_MEMORY_MCP_PROJECT")
if constrained_project:
@@ -347,6 +482,7 @@ async def create_memory_project(
if output_format == "json":
return {
"name": existing_match.name,
"external_id": existing_match.external_id,
"path": existing_match.path,
"is_default": is_default,
"created": False,
@@ -356,17 +492,22 @@ async def create_memory_project(
f"✓ Project already exists: {existing_match.name}\n\n"
f"Project Details:\n"
f"• Name: {existing_match.name}\n"
f"• External ID: {existing_match.external_id}\n"
f"• Path: {existing_match.path}\n"
f"{'• Set as default project\n' if is_default else ''}"
"\nProject is already available for use in tool calls.\n"
)
status_response = await project_client.create_project(project_request.model_dump())
from basic_memory.mcp.project_context import invalidate_workspace_project_index
await invalidate_workspace_project_index(context)
if output_format == "json":
new_project = status_response.new_project
return {
"name": new_project.name if new_project else project_name,
"external_id": new_project.external_id if new_project else None,
"path": new_project.path if new_project else project_path,
"is_default": bool(
(new_project.is_default if new_project else False) or set_default
@@ -380,6 +521,7 @@ async def create_memory_project(
if status_response.new_project:
result += "Project Details:\n"
result += f"• Name: {status_response.new_project.name}\n"
result += f"• External ID: {status_response.new_project.external_id}\n"
result += f"• Path: {status_response.new_project.path}\n"
if set_default:
@@ -451,6 +593,9 @@ async def delete_project(project_name: str, context: Context | None = None) -> s
# Delete project using project external_id
status_response = await project_client.delete_project(target_project.external_id)
from basic_memory.mcp.project_context import invalidate_workspace_project_index
await invalidate_workspace_project_index(context)
result = f"{status_response.message}\n\n"
+27 -10
View File
@@ -8,16 +8,17 @@ Files are read directly without any knowledge graph processing.
import base64
import io
from typing import Optional
from typing import Annotated, Optional
from loguru import logger
from PIL import Image as PILImage
from fastmcp import Context
from pydantic import AliasChoices, Field
from mcp.server.fastmcp.exceptions import ToolError
from basic_memory.config import ConfigManager
from basic_memory.mcp.project_context import (
detect_project_from_url_prefix,
detect_project_from_memory_url_prefix,
get_project_client,
resolve_project_and_path,
)
@@ -158,9 +159,12 @@ def optimize_image(img, content_length, max_output_bytes=350000):
annotations={"readOnlyHint": True, "openWorldHint": False},
)
async def read_content(
path: str,
path: Annotated[
str,
Field(validation_alias=AliasChoices("path", "file_path", "filepath", "file")),
],
project: Optional[str] = None,
workspace: Optional[str] = None,
project_id: Optional[str] = None,
context: Context | None = None,
) -> dict:
"""Read a file's raw content by path or permalink.
@@ -181,6 +185,9 @@ async def read_content(
- A permalink (docs/example)
project: Project name to read from. Optional - server will resolve using hierarchy.
If unknown, use list_memory_projects() to discover available projects.
project_id: Project external_id (UUID). Prefer this over `project` when known
it routes to the exact project regardless of name collisions across cloud
workspaces. Takes precedence over `project`. Get from list_memory_projects().
context: Optional FastMCP context for performance caching.
Returns:
@@ -210,17 +217,27 @@ async def read_content(
HTTPError: If project doesn't exist or is inaccessible
SecurityError: If path attempts path traversal
"""
# Detect project from memory URL prefix before routing
if project is None:
detected = detect_project_from_url_prefix(path, ConfigManager().config)
# Detect project from memory URL prefix before routing.
# project_id routes by external UUID, so it bypasses URL discovery entirely.
if project is None and project_id is None:
detected = await detect_project_from_memory_url_prefix(
path,
ConfigManager().config,
context=context,
)
if detected:
project = detected
logger.info(f"MCP tool call tool=read_content project={project} path={path}")
async with get_project_client(project, workspace, context) as (client, active_project):
# Resolve path with project-prefix awareness for memory:// URLs
_, url, _ = await resolve_project_and_path(client, path, project, context)
async with get_project_client(project, context=context, project_id=project_id) as (
client,
active_project,
):
# Resolve path with project-prefix awareness for memory:// URLs.
# Use active_project.name so resolution stays consistent when project_id
# was used or `project` was wrong/ambiguous (matches the cached resolution).
_, url, _ = await resolve_project_and_path(client, path, active_project.name, context)
# Validate path to prevent path traversal attacks
# For memory:// URLs, validate the extracted path (not the raw URL which
+210 -205
View File
@@ -1,17 +1,17 @@
"""Read note tool for Basic Memory MCP server."""
from textwrap import dedent
from typing import Optional, Literal
from typing import Optional, Literal, cast
import logfire
import yaml
from loguru import logger
from fastmcp import Context
from basic_memory import telemetry
from basic_memory.config import ConfigManager
from basic_memory.mcp.project_context import (
detect_project_from_url_prefix,
detect_project_from_memory_url_prefix,
get_project_client,
resolve_project_and_path,
)
@@ -33,10 +33,10 @@ def _parse_opening_frontmatter(content: str) -> tuple[str, dict | None]:
If parsing fails or frontmatter is not a mapping, returns body unchanged and None.
"""
original_content = content
if not content.startswith("---\n"):
lines = content.splitlines(keepends=True)
if not lines or lines[0].strip() != "---":
return original_content, None
lines = content.splitlines(keepends=True)
closing_index = None
for i in range(1, len(lines)):
if lines[i].strip() == "---":
@@ -70,9 +70,7 @@ def _parse_opening_frontmatter(content: str) -> tuple[str, dict | None]:
async def read_note(
identifier: str,
project: Optional[str] = None,
workspace: Optional[str] = None,
page: int = 1,
page_size: int = 10,
project_id: Optional[str] = None,
output_format: Literal["text", "json"] = "text",
include_frontmatter: bool = False,
context: Context | None = None,
@@ -96,10 +94,11 @@ async def read_note(
project: Project name to read from. Optional - server will resolve using the
hierarchy above. If unknown, use list_memory_projects() to discover
available projects.
project_id: Project external_id (UUID). Prefer this over `project` when known
it routes to the exact project regardless of name collisions across cloud
workspaces. Takes precedence over `project`. Get from list_memory_projects().
identifier: The title or permalink of the note to read
Can be a full memory:// URL, a permalink, a title, or search text
page: Page number for paginated results (default: 1)
page_size: Number of items per page (default: 10)
output_format: "text" returns markdown content or guidance text.
"json" returns a structured object with title/permalink/file_path/content/frontmatter.
include_frontmatter: When output_format="json", whether content should include the
@@ -120,9 +119,6 @@ async def read_note(
# Read with memory URL
read_note("my-research", "memory://specs/search-spec")
# Read with pagination
read_note("work-project", "Project Updates", page=2, page_size=5)
# Read recent meeting notes
read_note("team-docs", "Weekly Standup")
@@ -134,230 +130,239 @@ async def read_note(
If the exact note isn't found, this tool provides helpful suggestions
including related notes, search commands, and note creation templates.
"""
# Detect project from memory URL prefix before routing
if project is None:
detected = detect_project_from_url_prefix(identifier, ConfigManager().config)
# Detect project from memory URL prefix before routing.
# project_id routes by external UUID, so it bypasses URL discovery entirely.
if project is None and project_id is None:
detected = await detect_project_from_memory_url_prefix(
identifier,
ConfigManager().config,
context=context,
)
if detected:
project = detected
with telemetry.operation(
with logfire.span(
"mcp.tool.read_note",
entrypoint="mcp",
tool_name="read_note",
requested_project=project,
workspace_id=workspace,
requested_project_id=project_id,
output_format=output_format,
page=page,
page_size=page_size,
include_frontmatter=include_frontmatter,
):
async with get_project_client(project, workspace, context) as (client, active_project):
with telemetry.contextualize(
project_name=active_project.name,
workspace_id=workspace,
tool_name="read_note",
async with get_project_client(project, context=context, project_id=project_id) as (
client,
active_project,
):
# Resolve identifier with project-prefix awareness for memory:// URLs.
# Pass active_project.name (the canonical resolved name) rather than the
# original `project` arg so the inner get_active_project cache hits even
# when project_id was used or `project` was wrong/ambiguous.
_, entity_path, _ = await resolve_project_and_path(
client, identifier, active_project.name, context
)
# Validate identifier to prevent path traversal attacks
# For memory:// URLs, validate the extracted path (not the raw URL which
# has a scheme prefix that confuses path validation)
raw_path = (
memory_url_path(identifier) if identifier.startswith("memory://") else identifier
)
processed_path = entity_path
project_path = active_project.home
if not validate_project_path(raw_path, project_path) or not validate_project_path(
processed_path, project_path
):
# Resolve identifier with project-prefix awareness for memory:// URLs
_, entity_path, _ = await resolve_project_and_path(
client, identifier, project, context
logger.warning(
"Attempted path traversal attack blocked",
identifier=identifier,
processed_path=processed_path,
project=active_project.name,
)
# Validate identifier to prevent path traversal attacks
# For memory:// URLs, validate the extracted path (not the raw URL which
# has a scheme prefix that confuses path validation)
raw_path = (
memory_url_path(identifier)
if identifier.startswith("memory://")
else identifier
)
processed_path = entity_path
project_path = active_project.home
if not validate_project_path(raw_path, project_path) or not validate_project_path(
processed_path, project_path
):
logger.warning(
"Attempted path traversal attack blocked",
identifier=identifier,
processed_path=processed_path,
project=active_project.name,
)
if output_format == "json":
return {
"title": None,
"permalink": None,
"file_path": None,
"content": None,
"frontmatter": None,
"error": "SECURITY_VALIDATION_ERROR",
}
return f"# Error\n\nIdentifier '{identifier}' is not allowed - paths must stay within project boundaries"
# Get the file via REST API - first try direct identifier resolution
logger.info(
f"Attempting to read note from Project: {active_project.name} identifier: {entity_path}"
)
# Import here to avoid circular import
from basic_memory.mcp.clients import KnowledgeClient, ResourceClient
# Use typed clients for API calls
knowledge_client = KnowledgeClient(client, active_project.external_id)
resource_client = ResourceClient(client, active_project.external_id)
async def _read_json_payload(entity_id: str) -> dict:
with telemetry.scope(
"mcp.read_note.shape_response",
domain="mcp",
action="read_note",
phase="shape_response",
):
entity = await knowledge_client.get_entity(entity_id)
response = await resource_client.read(
entity_id, page=page, page_size=page_size
)
content_text = response.text
body_content, parsed_frontmatter = _parse_opening_frontmatter(content_text)
return {
"title": entity.title,
"permalink": entity.permalink,
"file_path": entity.file_path,
"content": content_text if include_frontmatter else body_content,
"frontmatter": parsed_frontmatter,
}
def _empty_json_payload() -> dict:
if output_format == "json":
return {
"title": None,
"permalink": None,
"file_path": None,
"content": None,
"frontmatter": None,
"error": "SECURITY_VALIDATION_ERROR",
}
return f"# Error\n\nIdentifier '{identifier}' is not allowed - paths must stay within project boundaries"
# Get the file via REST API - first try direct identifier resolution
logger.info(
f"Attempting to read note from Project: {active_project.name} identifier: {entity_path}"
)
# Import here to avoid circular import
from basic_memory.mcp.clients import KnowledgeClient, ResourceClient
# Use typed clients for API calls
knowledge_client = KnowledgeClient(client, active_project.external_id)
resource_client = ResourceClient(client, active_project.external_id)
async def _read_json_payload(entity_id: str) -> dict:
with logfire.span(
"mcp.read_note.shape_response",
domain="mcp",
action="read_note",
phase="shape_response",
):
entity = await knowledge_client.get_entity(entity_id)
response = await resource_client.read(entity_id)
content_text = response.text
body_content, parsed_frontmatter = _parse_opening_frontmatter(content_text)
return {
"title": entity.title,
"permalink": entity.permalink,
"file_path": entity.file_path,
"content": content_text if include_frontmatter else body_content,
"frontmatter": parsed_frontmatter,
}
def _search_results(payload: object) -> list[dict]:
if not isinstance(payload, dict):
return []
results = payload.get("results")
return results if isinstance(results, list) else []
def _empty_json_payload() -> dict:
return {
"title": None,
"permalink": None,
"file_path": None,
"content": None,
"frontmatter": None,
}
async def _search_candidates(identifier_text: str, *, title_only: bool) -> dict:
# Trigger: direct entity resolution failed for the caller's identifier.
# Why: search_notes applies the same memory:// normalization and tool-level
# query handling as the rest of MCP routing, which raw client calls skip.
# Outcome: unresolved memory URLs still fall back through normalized search.
search_type = "title" if title_only else "text"
response = await search_notes(
project=active_project.name,
workspace=workspace,
query=identifier_text,
search_type=search_type,
page=page,
page_size=page_size,
output_format="json",
context=context,
)
return response if isinstance(response, dict) else {}
def _search_results(payload: object) -> list[dict[str, object]]:
if not isinstance(payload, dict):
return []
payload_dict = cast(dict[str, object], payload)
results = payload_dict.get("results")
if not isinstance(results, list):
return []
return [
cast(dict[str, object], result)
for result in results
if isinstance(result, dict)
]
def _result_title(item: dict) -> str:
return str(item.get("title") or "")
async def _search_candidates(
identifier_text: str, *, title_only: bool
) -> dict[str, object]:
# Trigger: direct entity resolution failed for the caller's identifier.
# Why: search_notes applies the same memory:// normalization and tool-level
# query handling as the rest of MCP routing, which raw client calls skip.
# Outcome: unresolved memory URLs still fall back through normalized search.
# Pass project_id (external_id UUID) so the workspace selection from the
# outer get_project_client() is preserved across the inner re-resolution.
# Without this, project names that collide across workspaces could re-resolve
# to a different tenant via the default-workspace fallback (CLI/context=None).
search_type = "title" if title_only else "text"
response = await search_notes(
project=active_project.name,
project_id=active_project.external_id,
query=identifier_text,
search_type=search_type,
output_format="json",
context=context,
)
return cast(dict[str, object], response) if isinstance(response, dict) else {}
def _result_permalink(item: dict) -> Optional[str]:
value = item.get("permalink")
return str(value) if value else None
def _result_title(item: dict[str, object]) -> str:
return str(item.get("title") or "")
def _result_file_path(item: dict) -> Optional[str]:
value = item.get("file_path")
return str(value) if value else None
def _result_permalink(item: dict[str, object]) -> Optional[str]:
value = item.get("permalink")
return str(value) if value else None
try:
# Try to resolve identifier to entity ID
entity_id = await knowledge_client.resolve_entity(entity_path, strict=True)
def _result_file_path(item: dict[str, object]) -> Optional[str]:
value = item.get("file_path")
return str(value) if value else None
# Fetch content using entity ID
response = await resource_client.read(entity_id, page=page, page_size=page_size)
try:
# Try to resolve identifier to entity ID
entity_id = await knowledge_client.resolve_entity(entity_path, strict=True)
# If successful, return the content
if response.status_code == 200:
logger.info(
"Returning read_note result from resource: {path}", path=entity_path
)
if output_format == "json":
return await _read_json_payload(entity_id)
return response.text
except Exception as e: # pragma: no cover
logger.info(f"Direct lookup failed for '{entity_path}': {e}")
# Continue to fallback methods
# Fetch content using entity ID
response = await resource_client.read(entity_id)
# Fallback 1: Try title search via API
logger.info(f"Search title for: {identifier}")
title_results = await _search_candidates(identifier, title_only=True)
title_candidates = _search_results(title_results)
if title_candidates:
# Trigger: direct resolution failed and title search returned candidates.
# Why: avoid returning unrelated notes when search yields only fuzzy matches.
# Outcome: fetch content only when a true exact title match exists.
result = next(
(
candidate
for candidate in title_candidates
if _is_exact_title_match(identifier, _result_title(candidate))
),
None,
)
if not result:
logger.info(f"No exact title match found for: {identifier}")
elif _result_permalink(result):
try:
# Resolve the permalink to entity ID
entity_id = await knowledge_client.resolve_entity(
_result_permalink(result) or "", strict=True
)
# Fetch content using the entity ID
response = await resource_client.read(
entity_id, page=page, page_size=page_size
)
if response.status_code == 200:
logger.info(
f"Found note by exact title search: {_result_permalink(result)}"
)
if output_format == "json":
return await _read_json_payload(entity_id)
return response.text
except Exception as e: # pragma: no cover
logger.info(
f"Failed to fetch content for found title match {_result_permalink(result)}: {e}"
)
else:
# If successful, return the content
if response.status_code == 200:
logger.info(
f"No results in title search for: {identifier} in project {active_project.name}"
"Returning read_note result from resource: {path}", path=entity_path
)
# Fallback 2: Text search as a last resort
logger.info(f"Title search failed, trying text search for: {identifier}")
text_results = await _search_candidates(identifier, title_only=False)
# We didn't find a direct match, construct a helpful error message
text_candidates = _search_results(text_results)
if not text_candidates:
if output_format == "json":
return _empty_json_payload()
return format_not_found_message(active_project.name, identifier)
return await _read_json_payload(entity_id)
return response.text
except Exception as e: # pragma: no cover
logger.info(f"Direct lookup failed for '{entity_path}': {e}")
# Continue to fallback methods
# Fallback 1: Try title search via API
logger.info(f"Search title for: {identifier}")
title_results = await _search_candidates(identifier, title_only=True)
title_candidates = _search_results(title_results)
if title_candidates:
# Trigger: direct resolution failed and title search returned candidates.
# Why: avoid returning unrelated notes when search yields only fuzzy matches.
# Outcome: fetch content only when a true exact title match exists.
result = next(
(
candidate
for candidate in title_candidates
if _is_exact_title_match(identifier, _result_title(candidate))
),
None,
)
if not result:
logger.info(f"No exact title match found for: {identifier}")
elif _result_permalink(result):
try:
# Resolve the permalink to entity ID
entity_id = await knowledge_client.resolve_entity(
_result_permalink(result) or "", strict=True
)
# Fetch content using the entity ID
response = await resource_client.read(entity_id)
if response.status_code == 200:
logger.info(
f"Found note by exact title search: {_result_permalink(result)}"
)
if output_format == "json":
return await _read_json_payload(entity_id)
return response.text
except Exception as e: # pragma: no cover
logger.info(
f"Failed to fetch content for found title match {_result_permalink(result)}: {e}"
)
else:
logger.info(
f"No results in title search for: {identifier} in project {active_project.name}"
)
# Fallback 2: Text search as a last resort
logger.info(f"Title search failed, trying text search for: {identifier}")
text_results = await _search_candidates(identifier, title_only=False)
# We didn't find a direct match, construct a helpful error message
text_candidates = _search_results(text_results)
if not text_candidates:
if output_format == "json":
payload = _empty_json_payload()
payload["related_results"] = [
{
"title": _result_title(result),
"permalink": _result_permalink(result),
"file_path": _result_file_path(result),
}
for result in text_candidates[:5]
]
return payload
return format_related_results(active_project.name, identifier, text_candidates[:5])
return _empty_json_payload()
return format_not_found_message(active_project.name, identifier)
if output_format == "json":
payload = _empty_json_payload()
payload["related_results"] = [
{
"title": _result_title(result),
"permalink": _result_permalink(result),
"file_path": _result_file_path(result),
}
for result in text_candidates[:5]
]
return payload
return format_related_results(active_project.name, identifier, text_candidates[:5])
def format_not_found_message(project: str | None, identifier: str) -> str:
+41 -13
View File
@@ -2,10 +2,11 @@
from datetime import timezone
from pathlib import PurePosixPath
from typing import List, Union, Optional, Literal
from typing import Annotated, List, Union, Optional, Literal
from loguru import logger
from fastmcp import Context
from pydantic import AliasChoices, Field
from basic_memory.mcp.async_client import get_client
from basic_memory.mcp.project_context import (
@@ -38,13 +39,30 @@ from basic_memory.schemas.search import SearchItemType
annotations={"readOnlyHint": True, "openWorldHint": False},
)
async def recent_activity(
type: Union[str, List[str]] = "",
type: Annotated[
Union[str, List[str]],
Field(default="", validation_alias=AliasChoices("type", "types", "kind")),
] = "",
depth: int = 1,
timeframe: TimeFrame = "7d",
page: int = 1,
page_size: int = 10,
timeframe: Annotated[
TimeFrame,
Field(
default="7d",
validation_alias=AliasChoices("timeframe", "since", "time_range", "lookback"),
),
] = "7d",
# `offset` is intentionally NOT aliased: it has different semantics
# (item-indexed vs. 1-indexed page-number).
page: Annotated[
int,
Field(default=1, validation_alias=AliasChoices("page", "page_number")),
] = 1,
page_size: Annotated[
int,
Field(default=10, validation_alias=AliasChoices("page_size", "limit", "per_page")),
] = 10,
project: Optional[str] = None,
workspace: Optional[str] = None,
project_id: Optional[str] = None,
output_format: Literal["text", "json"] = "text",
context: Context | None = None,
) -> str | list[dict]:
@@ -86,6 +104,9 @@ async def recent_activity(
project: Project name to query. Optional - server will resolve using the
hierarchy above. If unknown, use list_memory_projects() to discover
available projects.
project_id: Project external_id (UUID). Prefer this over `project` when known
it routes to the exact project regardless of name collisions across cloud
workspaces. Takes precedence over `project`. Get from list_memory_projects().
output_format: "text" returns human-readable summary text. "json" returns
a flat list of recent items.
context: Optional FastMCP context for performance caching.
@@ -161,9 +182,14 @@ async def recent_activity(
if "type" not in params:
params["type"] = [SearchItemType.ENTITY.value]
# Resolve project parameter using the three-tier hierarchy
# allow_discovery=True enables Discovery Mode, so a project is not required
resolved_project = await resolve_project_parameter(project, allow_discovery=True)
# Resolve project parameter using the three-tier hierarchy.
# allow_discovery=True enables Discovery Mode, so a project is not required.
# project_id (UUID) takes precedence over project name — without this fallback,
# callers passing only project_id would fall into Discovery Mode.
effective_identifier = project_id if project_id else project
resolved_project = await resolve_project_parameter(
effective_identifier, allow_discovery=True, context=context
)
if resolved_project is None:
# Discovery Mode: Get activity across all projects
@@ -278,7 +304,7 @@ async def recent_activity(
f"Getting recent activity from project {resolved_project}: type={type}, depth={depth}, timeframe={timeframe}"
)
async with get_project_client(resolved_project, workspace, context) as (
async with get_project_client(resolved_project, context=context, project_id=project_id) as (
client,
active_project,
):
@@ -464,10 +490,12 @@ def _format_project_output(
elif result.primary_result.type == "observation":
observations.append(result.primary_result)
# Show entities (notes/documents)
# Show entities (notes/documents). Render every row the API returned —
# `page_size` is the single knob for how much comes back, so heading count
# and body row count always agree (regression: #784 silent truncation).
if entities:
lines.append(f"\n**📄 Recent Notes & Documents ({len(entities)}):**")
for entity in entities[:5]: # Show top 5
for entity in entities:
title = entity.title or "Untitled"
# Get folder from file_path
folder = ""
@@ -500,7 +528,7 @@ def _format_project_output(
# Show relations (connections)
if relations:
lines.append(f"\n**🔗 Recent Connections ({len(relations)}):**")
for rel in relations[:5]: # Show top 5
for rel in relations:
rel_type = rel.relation_type
from_entity = rel.from_entity or "Unknown"
to_entity = rel.to_entity
+24 -6
View File
@@ -211,7 +211,7 @@ async def schema_validate(
note_type: Optional[str] = None,
identifier: Optional[str] = None,
project: Optional[str] = None,
workspace: Optional[str] = None,
project_id: Optional[str] = None,
output_format: Literal["text", "json"] = "text",
context: Context | None = None,
) -> ValidationReport | str | dict:
@@ -236,6 +236,9 @@ async def schema_validate(
identifier: Specific note to validate (permalink, title, or path).
If provided, validates only this note.
project: Project name. Optional -- server will resolve.
project_id: Project external_id (UUID). Prefer this over `project` when known
it routes to the exact project regardless of name collisions across cloud
workspaces. Takes precedence over `project`. Get from list_memory_projects().
context: Optional FastMCP context for performance caching.
Returns:
@@ -251,7 +254,10 @@ async def schema_validate(
# Validate in a specific project
schema_validate(note_type="person", project="my-research")
"""
async with get_project_client(project, workspace, context) as (client, active_project):
async with get_project_client(project, context=context, project_id=project_id) as (
client,
active_project,
):
logger.info(
f"MCP tool call tool=schema_validate project={active_project.name} "
f"note_type={note_type} identifier={identifier}"
@@ -318,7 +324,7 @@ async def schema_infer(
note_type: str,
threshold: float = 0.25,
project: Optional[str] = None,
workspace: Optional[str] = None,
project_id: Optional[str] = None,
output_format: Literal["text", "json"] = "text",
context: Context | None = None,
) -> str | dict:
@@ -342,6 +348,9 @@ async def schema_infer(
threshold: Minimum frequency (0-1) for a field to be suggested as optional.
Default 0.25 (25%). Fields above 95% become required.
project: Project name. Optional -- server will resolve.
project_id: Project external_id (UUID). Prefer this over `project` when known
it routes to the exact project regardless of name collisions across cloud
workspaces. Takes precedence over `project`. Get from list_memory_projects().
context: Optional FastMCP context for performance caching.
Returns:
@@ -357,7 +366,10 @@ async def schema_infer(
# Infer in a specific project
schema_infer("person", project="my-research")
"""
async with get_project_client(project, workspace, context) as (client, active_project):
async with get_project_client(project, context=context, project_id=project_id) as (
client,
active_project,
):
logger.info(
f"MCP tool call tool=schema_infer project={active_project.name} "
f"note_type={note_type} threshold={threshold}"
@@ -432,7 +444,7 @@ async def schema_infer(
async def schema_diff(
note_type: str,
project: Optional[str] = None,
workspace: Optional[str] = None,
project_id: Optional[str] = None,
output_format: Literal["text", "json"] = "text",
context: Context | None = None,
) -> str | dict:
@@ -453,6 +465,9 @@ async def schema_diff(
Args:
note_type: The note type to check for drift (e.g., "person").
project: Project name. Optional -- server will resolve.
project_id: Project external_id (UUID). Prefer this over `project` when known
it routes to the exact project regardless of name collisions across cloud
workspaces. Takes precedence over `project`. Get from list_memory_projects().
context: Optional FastMCP context for performance caching.
Returns:
@@ -466,7 +481,10 @@ async def schema_diff(
# Check drift in a specific project
schema_diff("person", project="my-research")
"""
async with get_project_client(project, workspace, context) as (client, active_project):
async with get_project_client(project, context=context, project_id=project_id) as (
client,
active_project,
):
logger.info(
f"MCP tool call tool=schema_diff project={active_project.name} note_type={note_type}"
)
+178 -140
View File
@@ -4,16 +4,16 @@ import re
from textwrap import dedent
from typing import Annotated, List, Optional, Dict, Any, Literal
import logfire
from loguru import logger
from fastmcp import Context
from pydantic import BeforeValidator
from pydantic import AliasChoices, BeforeValidator, Field
from basic_memory import telemetry
from basic_memory.config import ConfigManager
from basic_memory.utils import coerce_dict, coerce_list
from basic_memory.mcp.container import get_container
from basic_memory.mcp.project_context import (
detect_project_from_url_prefix,
detect_project_from_memory_url_prefix,
get_project_client,
resolve_project_and_path,
)
@@ -301,27 +301,51 @@ def _format_search_markdown(result: SearchResponse, project: str, query: str | N
annotations={"readOnlyHint": True, "openWorldHint": False},
)
async def search_notes(
query: Optional[str] = None,
# Accept common search-query aliases models reach for from training data.
# `q` is the universal HTTP convention; `search`/`text` are common in NL APIs.
query: Annotated[
Optional[str],
Field(default=None, validation_alias=AliasChoices("query", "q", "search", "text")),
] = None,
project: Optional[str] = None,
workspace: Optional[str] = None,
page: int = 1,
page_size: int = 10,
project_id: Optional[str] = None,
# `offset` is intentionally NOT aliased to `page`: offset is item-indexed
# (skip N items) while page is 1-indexed page-number. Direct aliasing would
# silently return the wrong slice.
page: Annotated[
int,
Field(default=1, validation_alias=AliasChoices("page", "page_number")),
] = 1,
page_size: Annotated[
int,
Field(default=10, validation_alias=AliasChoices("page_size", "limit", "per_page")),
] = 10,
search_type: str | None = None,
output_format: Literal["text", "json"] = "text",
# Plural-vs-singular trips models constantly. Accept the singular too.
note_types: Annotated[
List[str] | None,
BeforeValidator(coerce_list),
Field(default=None, validation_alias=AliasChoices("note_types", "note_type", "types")),
"Filter by the 'type' field in note frontmatter (e.g. 'note', 'chapter', 'person'). "
"Case-insensitive.",
] = None,
entity_types: Annotated[
List[str] | None,
BeforeValidator(coerce_list),
Field(default=None, validation_alias=AliasChoices("entity_types", "entity_type")),
"Filter by knowledge graph item type: 'entity' (whole notes), 'observation', or "
"'relation'. Defaults to 'entity'. Do NOT pass schema/frontmatter types like "
"'Chapter' here — use note_types instead.",
] = None,
after_date: Optional[str] = None,
# Time-filter naming varies wildly across APIs.
after_date: Annotated[
Optional[str],
Field(
default=None,
validation_alias=AliasChoices("after_date", "since", "after", "from_date"),
),
] = None,
metadata_filters: Annotated[
Dict[str, Any] | None,
BeforeValidator(coerce_dict),
@@ -331,7 +355,13 @@ async def search_notes(
BeforeValidator(coerce_list),
] = None,
status: Optional[str] = None,
min_similarity: Optional[float] = None,
min_similarity: Annotated[
Optional[float],
Field(
default=None,
validation_alias=AliasChoices("min_similarity", "threshold", "similarity_threshold"),
),
] = None,
context: Context | None = None,
) -> dict | str:
"""Search across all content in the knowledge base with comprehensive syntax support.
@@ -415,6 +445,9 @@ async def search_notes(
Omit or pass None for filter-only searches using metadata_filters, tags, or status.
project: Project name to search in. Optional - server will resolve using hierarchy.
If unknown, use list_memory_projects() to discover available projects.
project_id: Project external_id (UUID). Prefer this over `project` when known
it routes to the exact project regardless of name collisions across cloud
workspaces. Takes precedence over `project`. Get from list_memory_projects().
page: The page number of results to return (default 1)
page_size: The number of results to return per page (default 10)
search_type: Type of search to perform, one of:
@@ -518,18 +551,23 @@ async def search_notes(
remainder = re.sub(r"\b(AND|OR|NOT)\b", "", remainder).strip()
query = remainder or None
# Detect project from memory URL prefix before routing
if project is None and query is not None:
detected = detect_project_from_url_prefix(query, ConfigManager().config)
# Detect project from memory URL prefix before routing.
# project_id routes by external UUID, so it bypasses URL discovery entirely.
if project is None and project_id is None and query is not None:
detected = await detect_project_from_memory_url_prefix(
query,
ConfigManager().config,
context=context,
)
if detected:
project = detected
with telemetry.operation(
with logfire.span(
"mcp.tool.search_notes",
entrypoint="mcp",
tool_name="search_notes",
requested_project=project,
workspace_id=workspace,
requested_project_id=project_id,
search_type=search_type or "default",
output_format=output_format,
page=page,
@@ -543,140 +581,140 @@ async def search_notes(
has_tags_filter=bool(tags),
has_status_filter=bool(status),
):
async with get_project_client(project, workspace, context) as (client, active_project):
with telemetry.contextualize(
project_name=active_project.name,
workspace_id=workspace,
tool_name="search_notes",
):
# Handle memory:// URLs by resolving to permalink search
is_memory_url = False
if query is not None:
_, resolved_query, is_memory_url = await resolve_project_and_path(
client, query, project, context
)
if is_memory_url:
query = resolved_query
effective_search_type = search_type or _default_search_type()
async with get_project_client(project, context=context, project_id=project_id) as (
client,
active_project,
):
# Handle memory:// URLs by resolving to permalink search.
# Use active_project.name so resolution hits the cached active project
# when project_id was used or `project` was wrong/ambiguous.
is_memory_url = False
if query is not None:
_, resolved_query, is_memory_url = await resolve_project_and_path(
client, query, active_project.name, context
)
if is_memory_url:
effective_search_type = "permalink"
query = resolved_query
effective_search_type = search_type or _default_search_type()
if is_memory_url:
effective_search_type = "permalink"
try:
# Create a SearchQuery object based on the parameters
search_query = SearchQuery()
try:
# Create a SearchQuery object based on the parameters
search_query = SearchQuery()
# Only map search_type to query fields when there is an actual query string.
# When query is None/empty, skip the search mode block — filters-only path.
effective_query = (query or "").strip()
if effective_query:
valid_search_types = {
"text",
"title",
"permalink",
"vector",
"semantic",
"hybrid",
}
if effective_search_type == "text":
search_query.text = effective_query
search_query.retrieval_mode = SearchRetrievalMode.FTS
elif effective_search_type in ("vector", "semantic"):
search_query.text = effective_query
search_query.retrieval_mode = SearchRetrievalMode.VECTOR
elif effective_search_type == "hybrid":
search_query.text = effective_query
search_query.retrieval_mode = SearchRetrievalMode.HYBRID
elif effective_search_type == "title":
search_query.title = effective_query
elif effective_search_type == "permalink" and "*" in effective_query:
search_query.permalink_match = effective_query
elif effective_search_type == "permalink":
search_query.permalink = effective_query
else:
raise ValueError(
f"Invalid search_type '{effective_search_type}'. "
f"Valid options: {', '.join(sorted(valid_search_types))}"
)
# Add optional filters if provided (empty lists are treated as no filter)
if entity_types:
search_query.entity_types = [SearchItemType(t) for t in entity_types]
if note_types:
search_query.note_types = note_types
if after_date:
search_query.after_date = after_date
if metadata_filters:
# Alias common column/model names to their frontmatter key equivalents.
# Users often pass "note_type" (the entity model column) when the
# frontmatter field is actually "type".
_METADATA_KEY_ALIASES = {"note_type": "type"}
metadata_filters = {
_METADATA_KEY_ALIASES.get(k, k): v for k, v in metadata_filters.items()
}
search_query.metadata_filters = metadata_filters
if tags:
search_query.tags = tags
if status:
search_query.status = status
if min_similarity is not None:
search_query.min_similarity = min_similarity
# Reject searches with no criteria at all
if search_query.no_criteria():
return (
"# No Search Criteria\n\n"
"Please provide at least one of: `query`, `metadata_filters`, "
"`tags`, `status`, `note_types`, `entity_types`, or `after_date`."
# Only map search_type to query fields when there is an actual query string.
# When query is None/empty, skip the search mode block — filters-only path.
effective_query = (query or "").strip()
if effective_query:
valid_search_types = {
"text",
"title",
"permalink",
"vector",
"semantic",
"hybrid",
}
if effective_search_type == "text":
search_query.text = effective_query
search_query.retrieval_mode = SearchRetrievalMode.FTS
elif effective_search_type in ("vector", "semantic"):
search_query.text = effective_query
search_query.retrieval_mode = SearchRetrievalMode.VECTOR
elif effective_search_type == "hybrid":
search_query.text = effective_query
search_query.retrieval_mode = SearchRetrievalMode.HYBRID
elif effective_search_type == "title":
search_query.title = effective_query
elif effective_search_type == "permalink" and "*" in effective_query:
search_query.permalink_match = effective_query
elif effective_search_type == "permalink":
search_query.permalink = effective_query
else:
raise ValueError(
f"Invalid search_type '{effective_search_type}'. "
f"Valid options: {', '.join(sorted(valid_search_types))}"
)
# Default to entity-level results to avoid returning individual
# observations/relations as separate search results (see issue #31).
# Applied after no_criteria() so that the implicit default doesn't
# mask a truly empty search request.
if not search_query.entity_types:
search_query.entity_types = [SearchItemType("entity")]
# Add optional filters if provided (empty lists are treated as no filter)
if entity_types:
search_query.entity_types = [SearchItemType(t) for t in entity_types]
if note_types:
search_query.note_types = note_types
if after_date:
search_query.after_date = after_date
if metadata_filters:
# Alias common column/model names to their frontmatter key equivalents.
# Users often pass "note_type" (the entity model column) when the
# frontmatter field is actually "type".
_METADATA_KEY_ALIASES = {"note_type": "type"}
metadata_filters = {
_METADATA_KEY_ALIASES.get(k, k): v for k, v in metadata_filters.items()
}
search_query.metadata_filters = metadata_filters
if tags:
search_query.tags = tags
if status:
search_query.status = status
if min_similarity is not None:
search_query.min_similarity = min_similarity
# Reject searches with no criteria at all
if search_query.no_criteria():
return (
"# No Search Criteria\n\n"
"Please provide at least one of: `query`, `metadata_filters`, "
"`tags`, `status`, `note_types`, `entity_types`, or `after_date`."
)
# Default to entity-level results to avoid returning individual
# observations/relations as separate search results (see issue #31).
# Applied after no_criteria() so that the implicit default doesn't
# mask a truly empty search request.
if not search_query.entity_types:
search_query.entity_types = [SearchItemType("entity")]
logger.debug(
f"Search request: project={active_project.name} "
f"search_type={effective_search_type} "
f"query={effective_query or '<filters-only>'} "
f"note_types={len(note_types)} entity_types={len(search_query.entity_types or [])} "
f"page={page} page_size={page_size}"
)
# Import here to avoid circular import (tools → clients → utils → tools)
from basic_memory.mcp.clients import SearchClient
# Use typed SearchClient for API calls
search_client = SearchClient(client, active_project.external_id)
result = await search_client.search(
search_query.model_dump(),
page=page,
page_size=page_size,
)
logger.debug(
f"Search response: project={active_project.name} "
f"results={len(result.results)} has_more={str(result.has_more).lower()} "
f"page={result.current_page} page_size={result.page_size}"
)
# Check if we got no results and provide helpful guidance
if not result.results:
logger.debug(
f"Search request: project={active_project.name} "
f"search_type={effective_search_type} "
f"query={effective_query or '<filters-only>'} "
f"note_types={len(note_types)} entity_types={len(search_query.entity_types or [])} "
f"page={page} page_size={page_size}"
f"Search returned no results for query: {query} in project {active_project.name}"
)
# Import here to avoid circular import (tools → clients → utils → tools)
from basic_memory.mcp.clients import SearchClient
# Don't treat this as an error, but the user might want guidance
# We return the empty result as normal - the user can decide if they need help
# Use typed SearchClient for API calls
search_client = SearchClient(client, active_project.external_id)
result = await search_client.search(
search_query.model_dump(),
page=page,
page_size=page_size,
)
logger.debug(
f"Search response: project={active_project.name} "
f"results={len(result.results)} has_more={str(result.has_more).lower()} "
f"page={result.current_page} page_size={result.page_size}"
)
if output_format == "json":
return result.model_dump(mode="json", exclude_none=True)
# Check if we got no results and provide helpful guidance
if not result.results:
logger.debug(
f"Search returned no results for query: {query} in project {active_project.name}"
)
# Don't treat this as an error, but the user might want guidance
# We return the empty result as normal - the user can decide if they need help
return _format_search_markdown(result, active_project.name, query)
if output_format == "json":
return result.model_dump(mode="json", exclude_none=True)
return _format_search_markdown(result, active_project.name, query)
except Exception as e:
logger.error(
f"Search failed for query '{query or ''}': {e}, project: {active_project.name}"
)
# Return formatted error message as string for better user experience
return _format_search_error_response(
active_project.name, str(e), query or "", effective_search_type
)
except Exception as e:
logger.error(
f"Search failed for query '{query or ''}': {e}, project: {active_project.name}"
)
# Return formatted error message as string for better user experience
return _format_search_error_response(
active_project.name, str(e), query or "", effective_search_type
)
+4 -6
View File
@@ -25,6 +25,7 @@ def _text_block(message: str) -> List[ContentBlock]:
async def search_notes_ui(
query: str,
project: Optional[str] = None,
project_id: Optional[str] = None,
page: int = 1,
page_size: int = 10,
search_type: Optional[str] = None,
@@ -49,6 +50,7 @@ async def search_notes_ui(
result = await search_notes(
query=query,
project=project,
project_id=project_id,
page=page,
page_size=page_size,
search_type=search_type,
@@ -97,16 +99,14 @@ async def search_notes_ui(
async def read_note_ui(
identifier: str,
project: Optional[str] = None,
page: int = 1,
page_size: int = 10,
project_id: Optional[str] = None,
context: Context | None = None,
) -> List[ContentBlock]:
"""Return a note preview UI as an embedded MCP-UI resource."""
content = await read_note(
identifier=identifier,
project=project,
page=page,
page_size=page_size,
project_id=project_id,
output_format="text",
context=context,
)
@@ -114,8 +114,6 @@ async def read_note_ui(
render_data = {
"toolInput": {
"identifier": identifier,
"page": page,
"page_size": page_size,
},
"toolOutput": content,
}
+69 -99
View File
@@ -5,9 +5,9 @@ to the Basic Memory API, with improved error handling and logging.
"""
import typing
from contextlib import contextmanager
from typing import Optional
from typing import Any, Optional
import logfire
from httpx import Response, URL, AsyncClient, HTTPStatusError
from httpx._client import UseClientDefault, USE_CLIENT_DEFAULT
from httpx._types import (
@@ -24,7 +24,6 @@ from httpx._types import (
from loguru import logger
from mcp.server.fastmcp.exceptions import ToolError
from basic_memory import telemetry
from basic_memory.config import ConfigManager
@@ -41,43 +40,22 @@ def _classify_http_outcome(status_code: int) -> str:
return "unknown" # pragma: no cover
class _RequestSpan:
"""Small adapter for attaching outcome metadata to a live request span."""
def _response_span_attrs(response: Response) -> dict[str, Any]:
"""Attributes to attach to a request span after a response lands."""
return {
"status_code": response.status_code,
"is_success": response.is_success,
"outcome": _classify_http_outcome(response.status_code),
}
def __init__(self, active_span: typing.Any | None):
self._active_span = active_span
def record_response(self, response: Response) -> None:
self._set_attributes(
{
"status_code": response.status_code,
"is_success": response.is_success,
"outcome": _classify_http_outcome(response.status_code),
}
)
def record_transport_error(self, exc: Exception) -> None:
self._set_attributes(
{
"is_success": False,
"outcome": "transport_error",
"error_type": type(exc).__name__,
}
)
def _set_attributes(self, attrs: dict[str, typing.Any]) -> None:
if self._active_span is None:
return
set_attributes = getattr(self._active_span, "set_attributes", None)
if callable(set_attributes):
set_attributes(attrs)
return
set_attribute = getattr(self._active_span, "set_attribute", None)
if callable(set_attribute):
for key, value in attrs.items():
set_attribute(key, value)
def _transport_error_span_attrs(exc: Exception) -> dict[str, Any]:
"""Attributes to attach when the transport layer fails before any response."""
return {
"is_success": False,
"outcome": "transport_error",
"error_type": type(exc).__name__,
}
def get_error_message(
@@ -130,15 +108,20 @@ def get_error_message(
return f"HTTP error {status_code}: {method} request to '{path}' failed"
def _extract_response_data(response: Response) -> typing.Any:
"""Safely decode response payload for error reporting."""
try:
return response.json()
except Exception:
def _extract_response_data(response: Response) -> Any:
"""Decode the JSON payload of an API response for error reporting.
Upstream gateways (Fly, Cloudflare, load balancers) can return HTML
error pages before the request reaches our FastAPI app; those have no
structured `detail` to surface, so we skip them. A malformed body with
a JSON content-type is a server bug and we let it raise.
"""
if "application/json" not in response.headers.get("content-type", ""):
return None
return response.json()
def _response_detail_text(response_data: typing.Any) -> str | None:
def _response_detail_text(response_data: Any) -> str | None:
"""Extract textual error detail from API payloads."""
if isinstance(response_data, dict):
detail = response_data.get("detail")
@@ -189,31 +172,6 @@ def _resolve_error_message(
return get_error_message(status_code, url, method)
@contextmanager
def _request_scope(
method: str,
*,
client_name: str | None,
operation: str | None,
path_template: str | None,
params: QueryParamTypes | None = None,
has_body: bool = False,
):
"""Create the shared MCP transport span used by all HTTP helpers."""
attrs = {
"method": method,
"client_name": client_name,
"operation": operation,
"path_template": path_template,
"phase": "request",
"has_query": bool(params),
"has_body": has_body,
}
with telemetry.contextualize(**attrs):
with telemetry.started_span("mcp.http.request", **attrs) as active_span:
yield _RequestSpan(active_span)
async def call_get(
client: AsyncClient,
url: URL | str,
@@ -250,15 +208,18 @@ async def call_get(
"""
logger.debug(f"Calling GET '{url}' params: '{params}'")
error_message = None
request_span: _RequestSpan | None = None
request_span: logfire.LogfireSpan | None = None
try:
with _request_scope(
"GET",
with logfire.span(
"mcp.http.request",
method="GET",
client_name=client_name,
operation=operation,
path_template=path_template,
params=params,
phase="request",
has_query=bool(params),
has_body=False,
) as request_span:
response = await client.get(
url,
@@ -270,7 +231,7 @@ async def call_get(
timeout=timeout,
extensions=extensions,
)
request_span.record_response(response)
request_span.set_attributes(_response_span_attrs(response))
if response.is_success:
return response
@@ -299,7 +260,7 @@ async def call_get(
raise ToolError(error_message) from e
except Exception as e:
if request_span is not None:
request_span.record_transport_error(e)
request_span.set_attributes(_transport_error_span_attrs(e))
raise
@@ -347,15 +308,17 @@ async def call_put(
"""
logger.debug(f"Calling PUT '{url}'")
error_message = None
request_span: _RequestSpan | None = None
request_span: logfire.LogfireSpan | None = None
try:
with _request_scope(
"PUT",
with logfire.span(
"mcp.http.request",
method="PUT",
client_name=client_name,
operation=operation,
path_template=path_template,
params=params,
phase="request",
has_query=bool(params),
has_body=any(value is not None for value in (content, data, files, json)),
) as request_span:
response = await client.put(
@@ -372,7 +335,7 @@ async def call_put(
timeout=timeout,
extensions=extensions,
)
request_span.record_response(response)
request_span.set_attributes(_response_span_attrs(response))
if response.is_success:
return response
@@ -402,7 +365,7 @@ async def call_put(
raise ToolError(error_message) from e
except Exception as e:
if request_span is not None:
request_span.record_transport_error(e)
request_span.set_attributes(_transport_error_span_attrs(e))
raise
@@ -449,15 +412,17 @@ async def call_patch(
ToolError: If the request fails with an appropriate error message
"""
logger.debug(f"Calling PATCH '{url}'")
request_span: _RequestSpan | None = None
request_span: logfire.LogfireSpan | None = None
try:
with _request_scope(
"PATCH",
with logfire.span(
"mcp.http.request",
method="PATCH",
client_name=client_name,
operation=operation,
path_template=path_template,
params=params,
phase="request",
has_query=bool(params),
has_body=any(value is not None for value in (content, data, files, json)),
) as request_span:
response = await client.patch(
@@ -474,7 +439,7 @@ async def call_patch(
timeout=timeout,
extensions=extensions,
)
request_span.record_response(response)
request_span.set_attributes(_response_span_attrs(response))
if response.is_success:
return response
@@ -509,7 +474,7 @@ async def call_patch(
raise ToolError(error_message) from e
except Exception as e:
if request_span is not None:
request_span.record_transport_error(e)
request_span.set_attributes(_transport_error_span_attrs(e))
raise
@@ -557,15 +522,17 @@ async def call_post(
"""
logger.debug(f"Calling POST '{url}'")
error_message = None
request_span: _RequestSpan | None = None
request_span: logfire.LogfireSpan | None = None
try:
with _request_scope(
"POST",
with logfire.span(
"mcp.http.request",
method="POST",
client_name=client_name,
operation=operation,
path_template=path_template,
params=params,
phase="request",
has_query=bool(params),
has_body=any(value is not None for value in (content, data, files, json)),
) as request_span:
response = await client.post(
@@ -582,7 +549,7 @@ async def call_post(
timeout=timeout,
extensions=extensions,
)
request_span.record_response(response)
request_span.set_attributes(_response_span_attrs(response))
logger.debug(f"response: {_extract_response_data(response)}")
if response.is_success:
@@ -612,7 +579,7 @@ async def call_post(
raise ToolError(error_message) from e
except Exception as e:
if request_span is not None:
request_span.record_transport_error(e)
request_span.set_attributes(_transport_error_span_attrs(e))
raise
@@ -684,15 +651,18 @@ async def call_delete(
"""
logger.debug(f"Calling DELETE '{url}'")
error_message = None
request_span: _RequestSpan | None = None
request_span: logfire.LogfireSpan | None = None
try:
with _request_scope(
"DELETE",
with logfire.span(
"mcp.http.request",
method="DELETE",
client_name=client_name,
operation=operation,
path_template=path_template,
params=params,
phase="request",
has_query=bool(params),
has_body=False,
) as request_span:
response = await client.delete(
url=url,
@@ -704,7 +674,7 @@ async def call_delete(
timeout=timeout,
extensions=extensions,
)
request_span.record_response(response)
request_span.set_attributes(_response_span_attrs(response))
if response.is_success:
return response
@@ -733,5 +703,5 @@ async def call_delete(
raise ToolError(error_message) from e
except Exception as e:
if request_span is not None:
request_span.record_transport_error(e)
request_span.set_attributes(_transport_error_span_attrs(e))
raise
+5 -11
View File
@@ -17,9 +17,7 @@ from basic_memory.mcp.tools.read_note import read_note
async def view_note(
identifier: str,
project: Optional[str] = None,
workspace: Optional[str] = None,
page: int = 1,
page_size: int = 10,
project_id: Optional[str] = None,
context: Context | None = None,
) -> str:
"""View a markdown note as a formatted artifact.
@@ -32,8 +30,9 @@ async def view_note(
identifier: The title or permalink of the note to view
project: Project name to read from. Optional - server will resolve using hierarchy.
If unknown, use list_memory_projects() to discover available projects.
page: Page number for paginated results (default: 1)
page_size: Number of items per page (default: 10)
project_id: Project external_id (UUID). Prefer this over `project` when known
it routes to the exact project regardless of name collisions across cloud
workspaces. Takes precedence over `project`. Get from list_memory_projects().
context: Optional FastMCP context for performance caching.
Returns:
@@ -46,9 +45,6 @@ async def view_note(
# View a note by permalink
view_note("meetings/weekly-standup")
# View with pagination
view_note("large-document", page=2, page_size=5)
# Explicit project specification
view_note("Meeting Notes", project="my-project")
@@ -63,9 +59,7 @@ async def view_note(
await read_note(
identifier=identifier,
project=project,
workspace=workspace,
page=page,
page_size=page_size,
project_id=project_id,
context=context,
)
)
+43 -29
View File
@@ -6,6 +6,42 @@ from fastmcp import Context
from basic_memory.mcp.project_context import get_available_workspaces
from basic_memory.mcp.server import mcp
from basic_memory.schemas.cloud import WorkspaceInfo, WorkspaceListResponse
def _personal_workspace() -> WorkspaceInfo:
"""Return a display-only personal workspace when discovery has no rows.
This keeps list_workspaces friendly for non-teams or local-only users. It is not
the cloud routing source of truth; project-scoped routing still depends on real
workspace discovery and the workspace project index in project_context.
"""
return WorkspaceInfo(
tenant_id="personal",
workspace_type="personal",
slug="personal",
name="Personal",
role="owner",
is_default=True,
has_active_subscription=True,
)
def _workspace_list_response(workspaces: list[WorkspaceInfo]) -> WorkspaceListResponse:
"""Build the structured MCP response from the shared cloud workspace schema."""
if not workspaces:
workspaces = [_personal_workspace()]
default_workspace_id = next(
(workspace.tenant_id for workspace in workspaces if workspace.is_default),
None,
)
return WorkspaceListResponse(
workspaces=workspaces,
count=len(workspaces),
default_workspace_id=default_workspace_id,
current_workspace_id=None,
)
@mcp.tool(
@@ -24,40 +60,18 @@ async def list_workspaces(
context: Optional FastMCP context for progress/status logging.
"""
workspaces = await get_available_workspaces(context=context)
response = _workspace_list_response(workspaces)
if output_format == "json":
return {
"workspaces": [
{
"tenant_id": ws.tenant_id,
"name": ws.name,
"workspace_type": ws.workspace_type,
"role": ws.role,
"organization_id": ws.organization_id,
"has_active_subscription": ws.has_active_subscription,
}
for ws in workspaces
],
"count": len(workspaces),
}
return response.model_dump(mode="json")
if not workspaces:
return (
"# No Workspaces Available\n\n"
"No accessible workspaces were found for this account. "
"Ensure the account has an active subscription and tenant access."
)
lines = [
f"# Available Workspaces ({len(workspaces)})",
"",
"Use `workspace` as either the `tenant_id` or unique `name` in project-scoped tool calls.",
"",
]
for workspace in workspaces:
lines = [f"# Available Workspaces ({response.count})", ""]
for workspace in response.workspaces:
default_label = ", default" if workspace.is_default else ""
lines.append(
f"- {workspace.name} "
f"(type={workspace.workspace_type}, role={workspace.role}, tenant_id={workspace.tenant_id})"
f"(slug={workspace.slug}, type={workspace.workspace_type}, "
f"role={workspace.role}{default_label}, tenant_id={workspace.tenant_id})"
)
return "\n".join(lines)
+168 -163
View File
@@ -3,10 +3,10 @@
import textwrap
from typing import Annotated, List, Union, Optional, Literal
import logfire
from loguru import logger
from pydantic import BeforeValidator
from pydantic import AliasChoices, BeforeValidator, Field
from basic_memory import telemetry
from basic_memory.config import ConfigManager
from basic_memory.mcp.project_context import get_project_client, add_project_metadata
from basic_memory.mcp.server import mcp
@@ -25,13 +25,21 @@ TagType = Union[List[str], str, None]
async def write_note(
title: str,
content: str,
directory: str,
# Folder/dir/path are interchangeable in models' training data.
directory: Annotated[
str,
Field(validation_alias=AliasChoices("directory", "folder", "dir", "path")),
],
project: Optional[str] = None,
workspace: Optional[str] = None,
project_id: Optional[str] = None,
tags: list[str] | str | None = None,
note_type: str = "note",
metadata: Annotated[dict | None, BeforeValidator(coerce_dict)] = None,
overwrite: bool | None = None,
# Force/replace are the file-write idioms models default to.
overwrite: Annotated[
bool | None,
Field(default=None, validation_alias=AliasChoices("overwrite", "force", "replace")),
] = None,
output_format: Literal["text", "json"] = "text",
context: Context | None = None,
) -> str | dict:
@@ -74,6 +82,9 @@ async def write_note(
project: Project name to write to. Optional - server will resolve using the
hierarchy above. If unknown, use list_memory_projects() to discover
available projects.
project_id: Project external_id (UUID). Prefer this over `project` when known
it routes to the exact project regardless of name collisions across cloud
workspaces. Takes precedence over `project`. Get from list_memory_projects().
tags: Tags to categorize the note. Can be a list of strings, a comma-separated string, or None.
Note: If passing from external MCP clients, use a string format (e.g. "tag1,tag2,tag3")
note_type: Type of note to create (stored in frontmatter). Defaults to "note".
@@ -149,180 +160,174 @@ async def write_note(
overwrite if overwrite is not None else ConfigManager().config.write_note_overwrite_default
)
with telemetry.operation(
with logfire.span(
"mcp.tool.write_note",
entrypoint="mcp",
tool_name="write_note",
requested_project=project,
workspace_id=workspace,
requested_project_id=project_id,
note_type=note_type,
overwrite=effective_overwrite,
output_format=output_format,
):
async with get_project_client(project, workspace, context) as (client, active_project):
with telemetry.contextualize(
project_name=active_project.name,
workspace_id=workspace,
tool_name="write_note",
):
logger.info(
f"MCP tool call tool=write_note project={active_project.name} directory={directory}, title={title}, tags={tags}"
)
async with get_project_client(project, context=context, project_id=project_id) as (
client,
active_project,
):
logger.info(
f"MCP tool call tool=write_note project={active_project.name} directory={directory}, title={title}, tags={tags}"
)
# Normalize "/" to empty string for root directory (must happen before validation)
if directory == "/":
directory = ""
# Normalize "/" to empty string for root directory (must happen before validation)
if directory == "/":
directory = ""
# Validate directory path to prevent path traversal attacks
project_path = active_project.home
if directory and not validate_project_path(directory, project_path):
logger.warning(
"Attempted path traversal attack blocked",
directory=directory,
project=active_project.name,
)
if output_format == "json":
return {
"title": title,
"permalink": None,
"file_path": None,
"checksum": None,
"action": "created",
"error": "SECURITY_VALIDATION_ERROR",
}
return f"# Error\n\nDirectory path '{directory}' is not allowed - paths must stay within project boundaries"
# Process tags using the helper function
tag_list = parse_tags(tags)
# Build entity_metadata from optional metadata, then explicit tags on top
# Order matters: explicit tags parameter takes precedence over metadata["tags"]
entity_metadata = {}
if metadata:
entity_metadata.update(metadata)
if tag_list:
entity_metadata["tags"] = tag_list
entity = Entity(
title=title,
# Validate directory path to prevent path traversal attacks
project_path = active_project.home
if directory and not validate_project_path(directory, project_path):
logger.warning(
"Attempted path traversal attack blocked",
directory=directory,
note_type=note_type,
content_type="text/markdown",
content=content,
entity_metadata=entity_metadata or None,
)
# Import here to avoid circular import
from basic_memory.mcp.clients import KnowledgeClient
# Use typed KnowledgeClient for API calls
knowledge_client = KnowledgeClient(client, active_project.external_id)
# Try to create the entity first (optimistic create)
logger.debug(f"Attempting to create entity permalink={entity.permalink}")
action = "Created" # Default to created
try:
result = await knowledge_client.create_entity(entity.model_dump(), fast=False)
action = "Created"
except Exception as e:
# If creation failed due to conflict (already exists), try to update
if (
"409" in str(e)
or "conflict" in str(e).lower()
or "already exists" in str(e).lower()
):
# Guard: block overwrite unless explicitly enabled
if not effective_overwrite:
logger.warning(
f"write_note blocked: note already exists (overwrite not enabled) "
f"permalink={entity.permalink}"
)
if output_format == "json":
return {
"title": title,
"permalink": entity.permalink,
"file_path": None,
"checksum": None,
"action": "conflict",
"error": "NOTE_ALREADY_EXISTS",
}
return _format_overwrite_error(
title, entity.permalink, active_project.name
)
logger.debug(
f"Entity exists, updating instead permalink={entity.permalink}"
)
try:
if not entity.permalink:
raise ValueError(
"Entity permalink is required for updates"
) # pragma: no cover
entity_id = await knowledge_client.resolve_entity(entity.permalink)
result = await knowledge_client.update_entity(
entity_id, entity.model_dump(), fast=False
)
action = "Updated"
except Exception as update_error: # pragma: no cover
# Re-raise the original error if update also fails
raise e from update_error # pragma: no cover
else:
# Re-raise if it's not a conflict error
raise # pragma: no cover
summary = [
f"# {action} note",
f"project: {active_project.name}",
f"file_path: {result.file_path}",
f"permalink: {result.permalink}",
f"checksum: {result.checksum[:8] if result.checksum else 'unknown'}",
]
# Count observations by category
categories = {}
if result.observations:
for obs in result.observations:
categories[obs.category] = categories.get(obs.category, 0) + 1
summary.append("\n## Observations")
for category, count in sorted(categories.items()):
summary.append(f"- {category}: {count}")
# Count resolved/unresolved relations
unresolved = 0
resolved = 0
if result.relations:
unresolved = sum(1 for r in result.relations if not r.to_id)
resolved = len(result.relations) - unresolved
summary.append("\n## Relations")
summary.append(f"- Resolved: {resolved}")
if unresolved:
summary.append(f"- Unresolved: {unresolved}")
summary.append(
"\nNote: Unresolved relations point to entities that don't exist yet."
)
summary.append(
"They will be automatically resolved when target entities are created or during sync operations."
)
if tag_list:
summary.append(f"\n## Tags\n- {', '.join(tag_list)}")
# Log the response with structured data
logger.info(
f"MCP tool response: tool=write_note project={active_project.name} action={action} permalink={result.permalink} observations_count={len(result.observations)} relations_count={len(result.relations)} resolved_relations={resolved} unresolved_relations={unresolved}"
project=active_project.name,
)
if output_format == "json":
return {
"title": result.title,
"permalink": result.permalink,
"file_path": result.file_path,
"checksum": result.checksum,
"action": action.lower(),
"title": title,
"permalink": None,
"file_path": None,
"checksum": None,
"action": "created",
"error": "SECURITY_VALIDATION_ERROR",
}
return f"# Error\n\nDirectory path '{directory}' is not allowed - paths must stay within project boundaries"
summary_result = "\n".join(summary)
return add_project_metadata(summary_result, active_project.name)
# Process tags using the helper function
tag_list = parse_tags(tags)
# Build entity_metadata from optional metadata, then explicit tags on top
# Order matters: explicit tags parameter takes precedence over metadata["tags"]
entity_metadata = {}
if metadata:
entity_metadata.update(metadata)
if tag_list:
entity_metadata["tags"] = tag_list
entity = Entity(
title=title,
directory=directory,
note_type=note_type,
content_type="text/markdown",
content=content,
entity_metadata=entity_metadata or None,
)
# Import here to avoid circular import
from basic_memory.mcp.clients import KnowledgeClient
# Use typed KnowledgeClient for API calls
knowledge_client = KnowledgeClient(client, active_project.external_id)
# Try to create the entity first (optimistic create)
logger.debug(f"Attempting to create entity permalink={entity.permalink}")
action = "Created" # Default to created
try:
result = await knowledge_client.create_entity(entity.model_dump())
action = "Created"
except Exception as e:
# If creation failed due to conflict (already exists), try to update
if (
"409" in str(e)
or "conflict" in str(e).lower()
or "already exists" in str(e).lower()
):
# Guard: block overwrite unless explicitly enabled
if not effective_overwrite:
logger.warning(
f"write_note blocked: note already exists (overwrite not enabled) "
f"permalink={entity.permalink}"
)
if output_format == "json":
return {
"title": title,
"permalink": entity.permalink,
"file_path": None,
"checksum": None,
"action": "conflict",
"error": "NOTE_ALREADY_EXISTS",
}
return _format_overwrite_error(title, entity.permalink, active_project.name)
logger.debug(f"Entity exists, updating instead permalink={entity.permalink}")
try:
if not entity.permalink:
raise ValueError(
"Entity permalink is required for updates"
) # pragma: no cover
entity_id = await knowledge_client.resolve_entity(entity.permalink)
result = await knowledge_client.update_entity(
entity_id, entity.model_dump()
)
action = "Updated"
except Exception as update_error: # pragma: no cover
# Re-raise the original error if update also fails
raise e from update_error # pragma: no cover
else:
# Re-raise if it's not a conflict error
raise # pragma: no cover
summary = [
f"# {action} note",
f"project: {active_project.name}",
f"file_path: {result.file_path}",
f"permalink: {result.permalink}",
f"checksum: {result.checksum[:8] if result.checksum else 'unknown'}",
]
# Count observations by category
categories = {}
if result.observations:
for obs in result.observations:
categories[obs.category] = categories.get(obs.category, 0) + 1
summary.append("\n## Observations")
for category, count in sorted(categories.items()):
summary.append(f"- {category}: {count}")
# Count resolved/unresolved relations
unresolved = 0
resolved = 0
if result.relations:
unresolved = sum(1 for r in result.relations if not r.to_id)
resolved = len(result.relations) - unresolved
summary.append("\n## Relations")
summary.append(f"- Resolved: {resolved}")
if unresolved:
summary.append(f"- Unresolved: {unresolved}")
summary.append(
"\nNote: Unresolved relations point to entities that don't exist yet."
)
summary.append(
"They will be automatically resolved when target entities are created or during sync operations."
)
if tag_list:
summary.append(f"\n## Tags\n- {', '.join(tag_list)}")
# Log the response with structured data
logger.info(
f"MCP tool response: tool=write_note project={active_project.name} action={action} permalink={result.permalink} observations_count={len(result.observations)} relations_count={len(result.relations)} resolved_relations={resolved} unresolved_relations={unresolved}"
)
if output_format == "json":
return {
"title": result.title,
"permalink": result.permalink,
"file_path": result.file_path,
"checksum": result.checksum,
"action": action.lower(),
}
summary_result = "\n".join(summary)
return add_project_metadata(summary_result, active_project.name)
def _format_overwrite_error(title: str, permalink: str | None, project_name: str) -> str:
+2 -1
View File
@@ -2,12 +2,13 @@
import basic_memory
from basic_memory.models.base import Base
from basic_memory.models.knowledge import Entity, Observation, Relation
from basic_memory.models.knowledge import Entity, NoteContent, Observation, Relation
from basic_memory.models.project import Project
__all__ = [
"Base",
"Entity",
"NoteContent",
"Observation",
"Relation",
"Project",
+4 -1
View File
@@ -1,5 +1,7 @@
"""Base model class for SQLAlchemy models."""
from typing import TYPE_CHECKING
from sqlalchemy.ext.asyncio import AsyncAttrs
from sqlalchemy.orm import DeclarativeBase
@@ -7,4 +9,5 @@ from sqlalchemy.orm import DeclarativeBase
class Base(AsyncAttrs, DeclarativeBase):
"""Base class for all models"""
pass
if TYPE_CHECKING:
id: int
+79 -3
View File
@@ -6,6 +6,8 @@ from basic_memory.utils import ensure_timezone_aware
from typing import Optional
from sqlalchemy import (
BigInteger,
CheckConstraint,
Integer,
String,
Text,
@@ -60,7 +62,7 @@ class Entity(Base):
)
# Core identity
id: Mapped[int] = mapped_column(Integer, primary_key=True)
id: Mapped[int] = mapped_column(Integer, primary_key=True) # pyright: ignore [reportIncompatibleVariableOverride]
# External UUID for API references - stable identifier that won't change
external_id: Mapped[str] = mapped_column(String, unique=True, default=lambda: str(uuid.uuid4()))
title: Mapped[str] = mapped_column(String)
@@ -116,6 +118,12 @@ class Entity(Base):
foreign_keys="[Relation.to_id]",
cascade="all, delete-orphan",
)
note_content = relationship(
"NoteContent",
back_populates="entity",
cascade="all, delete-orphan",
uselist=False,
)
@property
def relations(self):
@@ -141,6 +149,74 @@ class Entity(Base):
return f"Entity(id={self.id}, external_id='{self.external_id}', name='{self.title}', type='{self.note_type}', checksum='{self.checksum}')"
class NoteContent(Base):
"""Materialized markdown content and sync state for a note entity."""
__tablename__ = "note_content"
__table_args__ = (
CheckConstraint(
"file_write_status IN ("
"'pending', "
"'writing', "
"'synced', "
"'failed', "
"'external_change_detected'"
")",
name="ck_note_content_file_write_status",
),
Index("ix_note_content_project_id", "project_id"),
Index("ix_note_content_file_path", "file_path"),
Index("ix_note_content_external_id", "external_id", unique=True),
)
# Core identity mirrored from entity for hot note reads
entity_id: Mapped[int] = mapped_column(
Integer,
ForeignKey("entity.id", ondelete="CASCADE"),
primary_key=True,
)
project_id: Mapped[int] = mapped_column(
Integer,
ForeignKey("project.id", ondelete="CASCADE"),
nullable=False,
)
external_id: Mapped[str] = mapped_column(String, nullable=False)
file_path: Mapped[str] = mapped_column(String, nullable=False)
# Materialized content version tracked in the tenant database
markdown_content: Mapped[str] = mapped_column(Text, nullable=False)
db_version: Mapped[int] = mapped_column(BigInteger, nullable=False)
db_checksum: Mapped[str] = mapped_column(String, nullable=False)
# File materialization state tracked against the latest write attempts
file_version: Mapped[Optional[int]] = mapped_column(BigInteger, nullable=True)
file_checksum: Mapped[Optional[str]] = mapped_column(String, nullable=True)
file_write_status: Mapped[str] = mapped_column(String, nullable=False, default="pending")
last_source: Mapped[Optional[str]] = mapped_column(String, nullable=True)
updated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
default=lambda: datetime.now().astimezone(),
onupdate=lambda: datetime.now().astimezone(),
)
file_updated_at: Mapped[Optional[datetime]] = mapped_column(
DateTime(timezone=True),
nullable=True,
)
last_materialization_error: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
last_materialization_attempt_at: Mapped[Optional[datetime]] = mapped_column(
DateTime(timezone=True),
nullable=True,
)
entity = relationship("Entity", back_populates="note_content")
def __repr__(self) -> str: # pragma: no cover
return (
f"NoteContent(entity_id={self.entity_id}, external_id='{self.external_id}', "
f"file_path='{self.file_path}', file_write_status='{self.file_write_status}')"
)
class Observation(Base):
"""An observation about an entity.
@@ -153,7 +229,7 @@ class Observation(Base):
Index("ix_observation_category", "category"), # Add category index
)
id: Mapped[int] = mapped_column(Integer, primary_key=True)
id: Mapped[int] = mapped_column(Integer, primary_key=True) # pyright: ignore [reportIncompatibleVariableOverride]
project_id: Mapped[int] = mapped_column(Integer, ForeignKey("project.id"), index=True)
entity_id: Mapped[int] = mapped_column(Integer, ForeignKey("entity.id", ondelete="CASCADE"))
content: Mapped[str] = mapped_column(Text)
@@ -200,7 +276,7 @@ class Relation(Base):
Index("ix_relation_to_id", "to_id"),
)
id: Mapped[int] = mapped_column(Integer, primary_key=True)
id: Mapped[int] = mapped_column(Integer, primary_key=True) # pyright: ignore [reportIncompatibleVariableOverride]
project_id: Mapped[int] = mapped_column(Integer, ForeignKey("project.id"), index=True)
from_id: Mapped[int] = mapped_column(Integer, ForeignKey("entity.id", ondelete="CASCADE"))
to_id: Mapped[Optional[int]] = mapped_column(
+4
View File
@@ -104,6 +104,8 @@ CREATE TABLE IF NOT EXISTS search_vector_chunks (
chunk_key TEXT NOT NULL,
chunk_text TEXT NOT NULL,
source_hash TEXT NOT NULL,
entity_fingerprint TEXT NOT NULL,
embedding_model TEXT NOT NULL,
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
UNIQUE (project_id, entity_id, chunk_key)
)
@@ -124,6 +126,8 @@ CREATE TABLE IF NOT EXISTS search_vector_chunks (
chunk_key TEXT NOT NULL,
chunk_text TEXT NOT NULL,
source_hash TEXT NOT NULL,
entity_fingerprint TEXT NOT NULL,
embedding_model TEXT NOT NULL,
updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP
)
""")
+2
View File
@@ -1,10 +1,12 @@
from .entity_repository import EntityRepository
from .note_content_repository import NoteContentRepository
from .observation_repository import ObservationRepository
from .project_repository import ProjectRepository
from .relation_repository import RelationRepository
__all__ = [
"EntityRepository",
"NoteContentRepository",
"ObservationRepository",
"ProjectRepository",
"RelationRepository",
@@ -1,6 +1,6 @@
"""Embedding provider protocol for pluggable semantic backends."""
from typing import Protocol
from typing import Any, Protocol
class EmbeddingProvider(Protocol):
@@ -16,3 +16,7 @@ class EmbeddingProvider(Protocol):
async def embed_documents(self, texts: list[str]) -> list[list[float]]:
"""Embed a list of document chunks."""
...
def runtime_log_attrs(self) -> dict[str, Any]:
"""Return provider-specific runtime settings suitable for startup logs."""
...
@@ -1,26 +1,96 @@
"""Factory for creating configured semantic embedding providers."""
import os
from threading import Lock
from basic_memory.config import BasicMemoryConfig
from basic_memory.config import BasicMemoryConfig, default_fastembed_cache_dir
from basic_memory.repository.embedding_provider import EmbeddingProvider
type ProviderCacheKey = tuple[str, str, int | None, int, str | None, int | None, int | None]
type ProviderCacheKey = tuple[
str,
str,
int | None,
int,
int,
str,
int | None,
int | None,
]
_EMBEDDING_PROVIDER_CACHE: dict[ProviderCacheKey, EmbeddingProvider] = {}
_EMBEDDING_PROVIDER_CACHE_LOCK = Lock()
_FASTEMBED_MAX_THREADS = 8
def _resolve_cache_dir(app_config: BasicMemoryConfig) -> str:
"""Resolve the effective FastEmbed cache dir for this config.
Uses an explicit ``is not None`` check an empty string override from
config or ``BASIC_MEMORY_SEMANTIC_EMBEDDING_CACHE_DIR`` is an invalid
path, not a request to fall back to the default, and FastEmbed's error
message is clearer than silently swapping in a different directory.
"""
configured = app_config.semantic_embedding_cache_dir
if configured is not None:
return configured
return default_fastembed_cache_dir()
def _available_cpu_count() -> int | None:
"""Return the CPU budget available to this process when the runtime exposes it."""
process_cpu_count = getattr(os, "process_cpu_count", None)
if callable(process_cpu_count):
cpu_count = process_cpu_count()
if isinstance(cpu_count, int) and cpu_count > 0:
return cpu_count
cpu_count = os.cpu_count()
return cpu_count if cpu_count is not None and cpu_count > 0 else None
def _resolve_fastembed_runtime_knobs(
app_config: BasicMemoryConfig,
) -> tuple[int | None, int | None]:
"""Resolve FastEmbed threads/parallel from explicit config or CPU-aware defaults."""
configured_threads = app_config.semantic_embedding_threads
configured_parallel = app_config.semantic_embedding_parallel
if configured_threads is not None or configured_parallel is not None:
return configured_threads, configured_parallel
available_cpus = _available_cpu_count()
if available_cpus is None:
return None, None
# Trigger: local laptops and cloud workers expose different CPU budgets.
# Why: full rebuilds got faster when FastEmbed used most, but not all, of
# the available CPUs. Leaving a little headroom avoids starving the rest of
# the pipeline while still giving ONNX enough threads to stay busy.
# Outcome: when config leaves the knobs unset, each process reserves a small
# CPU cushion and keeps FastEmbed on the simpler single-process path.
if available_cpus <= 2:
return available_cpus, 1
threads = min(_FASTEMBED_MAX_THREADS, max(2, available_cpus - 2))
return threads, 1
def _provider_cache_key(app_config: BasicMemoryConfig) -> ProviderCacheKey:
"""Build a stable cache key from provider-relevant semantic embedding config."""
"""Build a stable cache key from provider-relevant semantic embedding config.
Uses the *resolved* cache dir not the raw config field so different
FASTEMBED_CACHE_PATH values produce distinct cache keys even when the
config field itself is unset.
"""
resolved_threads, resolved_parallel = _resolve_fastembed_runtime_knobs(app_config)
return (
app_config.semantic_embedding_provider.strip().lower(),
app_config.semantic_embedding_model,
app_config.semantic_embedding_dimensions,
app_config.semantic_embedding_batch_size,
app_config.semantic_embedding_cache_dir,
app_config.semantic_embedding_threads,
app_config.semantic_embedding_parallel,
app_config.semantic_embedding_request_concurrency,
_resolve_cache_dir(app_config),
resolved_threads,
resolved_parallel,
)
@@ -51,12 +121,17 @@ def create_embedding_provider(app_config: BasicMemoryConfig) -> EmbeddingProvide
# Deferred import: fastembed (and its onnxruntime dep) may not be installed
from basic_memory.repository.fastembed_provider import FastEmbedEmbeddingProvider
if app_config.semantic_embedding_cache_dir is not None:
extra_kwargs["cache_dir"] = app_config.semantic_embedding_cache_dir
if app_config.semantic_embedding_threads is not None:
extra_kwargs["threads"] = app_config.semantic_embedding_threads
if app_config.semantic_embedding_parallel is not None:
extra_kwargs["parallel"] = app_config.semantic_embedding_parallel
resolved_threads, resolved_parallel = _resolve_fastembed_runtime_knobs(app_config)
# Trigger: cache_dir is resolved rather than passed through directly.
# Why: FastEmbed's own default caches to <system tmp>/fastembed_cache,
# which disappears in sandboxed MCP runtimes (e.g. Codex CLI). See #741.
# Outcome: always pass an explicit, user-writable cache dir so the ONNX
# model persists across runs.
extra_kwargs["cache_dir"] = _resolve_cache_dir(app_config)
if resolved_threads is not None:
extra_kwargs["threads"] = resolved_threads
if resolved_parallel is not None:
extra_kwargs["parallel"] = resolved_parallel
provider = FastEmbedEmbeddingProvider(
model_name=app_config.semantic_embedding_model,
@@ -73,6 +148,7 @@ def create_embedding_provider(app_config: BasicMemoryConfig) -> EmbeddingProvide
provider = OpenAIEmbeddingProvider(
model_name=model_name,
batch_size=app_config.semantic_embedding_batch_size,
request_concurrency=app_config.semantic_embedding_request_concurrency,
**extra_kwargs,
)
else:
@@ -33,7 +33,7 @@ class EntityRepository(Repository[Entity]):
"""
super().__init__(session_maker, Entity, project_id=project_id)
async def get_by_id(self, entity_id: int) -> Optional[Entity]: # pragma: no cover
async def get_by_id(self, entity_id: int, *, load_relations: bool = True) -> Optional[Entity]:
"""Get entity by numeric ID.
Args:
@@ -43,9 +43,23 @@ class EntityRepository(Repository[Entity]):
Entity if found, None otherwise
"""
async with db.scoped_session(self.session_maker) as session:
if not load_relations:
result = await session.execute(self.select().where(Entity.id == entity_id))
return result.scalars().one_or_none()
return await self.select_by_id(session, entity_id)
async def get_by_external_id(self, external_id: str) -> Optional[Entity]:
async def _find_one_by_query(self, query, *, load_relations: bool) -> Optional[Entity]:
"""Return one entity row with optional eager loading."""
if load_relations:
return await self.find_one(query)
result = await self.execute_query(query, use_query_options=False)
return result.scalars().one_or_none()
async def get_by_external_id(
self, external_id: str, *, load_relations: bool = True
) -> Optional[Entity]:
"""Get entity by external UUID.
Args:
@@ -54,21 +68,21 @@ class EntityRepository(Repository[Entity]):
Returns:
Entity if found, None otherwise
"""
query = (
self.select().where(Entity.external_id == external_id).options(*self.get_load_options())
)
return await self.find_one(query)
query = self.select().where(Entity.external_id == external_id)
return await self._find_one_by_query(query, load_relations=load_relations)
async def get_by_permalink(self, permalink: str) -> Optional[Entity]:
async def get_by_permalink(
self, permalink: str, *, load_relations: bool = True
) -> Optional[Entity]:
"""Get entity by permalink.
Args:
permalink: Unique identifier for the entity
"""
query = self.select().where(Entity.permalink == permalink).options(*self.get_load_options())
return await self.find_one(query)
query = self.select().where(Entity.permalink == permalink)
return await self._find_one_by_query(query, load_relations=load_relations)
async def get_by_title(self, title: str) -> Sequence[Entity]:
async def get_by_title(self, title: str, *, load_relations: bool = True) -> Sequence[Entity]:
"""Get entities by title, ordered by shortest path first.
When multiple entities share the same title (in different folders),
@@ -82,23 +96,20 @@ class EntityRepository(Repository[Entity]):
self.select()
.where(Entity.title == title)
.order_by(func.length(Entity.file_path), Entity.file_path)
.options(*self.get_load_options())
)
result = await self.execute_query(query)
result = await self.execute_query(query, use_query_options=load_relations)
return list(result.scalars().all())
async def get_by_file_path(self, file_path: Union[Path, str]) -> Optional[Entity]:
async def get_by_file_path(
self, file_path: Union[Path, str], *, load_relations: bool = True
) -> Optional[Entity]:
"""Get entity by file_path.
Args:
file_path: Path to the entity file (will be converted to string internally)
"""
query = (
self.select()
.where(Entity.file_path == Path(file_path).as_posix())
.options(*self.get_load_options())
)
return await self.find_one(query)
query = self.select().where(Entity.file_path == Path(file_path).as_posix())
return await self._find_one_by_query(query, load_relations=load_relations)
# -------------------------------------------------------------------------
# Lightweight methods for permalink resolution (no eager loading)
@@ -381,6 +392,9 @@ class EntityRepository(Repository[Entity]):
# Use merge to avoid session state conflicts
# Set the ID to update existing entity
entity.id = existing_entity.id
# Preserve the stable external_id so that external references
# (e.g. public share links) survive re-indexing
entity.external_id = existing_entity.external_id
# Ensure observations reference the correct entity_id
for obs in entity.observations:
@@ -11,7 +11,7 @@ from basic_memory.repository.embedding_provider import EmbeddingProvider
from basic_memory.repository.semantic_errors import SemanticDependenciesMissingError
if TYPE_CHECKING:
from fastembed import TextEmbedding # type: ignore[import-not-found] # pragma: no cover
from fastembed import TextEmbedding # pragma: no cover
class FastEmbedEmbeddingProvider(EmbeddingProvider):
@@ -24,6 +24,15 @@ class FastEmbedEmbeddingProvider(EmbeddingProvider):
def _effective_parallel(self) -> int | None:
return self.parallel if self.parallel is not None and self.parallel > 1 else None
def runtime_log_attrs(self) -> dict[str, int | str | None]:
"""Return the resolved runtime knobs that shape FastEmbed throughput."""
return {
"provider_batch_size": self.batch_size,
"threads": self.threads,
"configured_parallel": self.parallel,
"effective_parallel": self._effective_parallel(),
}
def __init__(
self,
model_name: str = "bge-small-en-v1.5",
@@ -53,7 +62,7 @@ class FastEmbedEmbeddingProvider(EmbeddingProvider):
def _create_model() -> "TextEmbedding":
try:
from fastembed import TextEmbedding # type: ignore[import-not-found]
from fastembed import TextEmbedding
except (
ImportError
) as exc: # pragma: no cover - exercised via tests with monkeypatch
@@ -0,0 +1,191 @@
"""Repository for managing note materialization state."""
from pathlib import Path
from typing import Any, Mapping, Optional
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from basic_memory import db
from basic_memory.models import Entity, NoteContent
from basic_memory.repository.repository import Repository
NOTE_CONTENT_MUTABLE_FIELDS = frozenset(
{
"markdown_content",
"db_version",
"db_checksum",
"file_version",
"file_checksum",
"file_write_status",
"last_source",
"updated_at",
"file_updated_at",
"last_materialization_error",
"last_materialization_attempt_at",
}
)
class NoteContentRepository(Repository[NoteContent]):
"""Repository for project-scoped note materialization state."""
def __init__(self, session_maker: async_sessionmaker[AsyncSession], project_id: int):
"""Initialize with session maker and project-scoped filtering."""
super().__init__(session_maker, NoteContent, project_id=project_id)
def _coerce_note_content(
self, data: Mapping[str, Any] | NoteContent
) -> tuple[NoteContent, set[str]]:
"""Convert input data to a NoteContent model and track explicit fields."""
if isinstance(data, NoteContent):
model_data = {
key: value for key, value in data.__dict__.items() if key in self.valid_columns
}
else:
model_data = {key: value for key, value in data.items() if key in self.valid_columns}
entity_id = model_data.get("entity_id")
if entity_id is None:
raise ValueError("entity_id is required for note_content writes")
return NoteContent(**model_data), set(model_data)
async def _load_entity_identity(self, session: AsyncSession, entity_id: int) -> Entity:
"""Load the owning entity so duplicated identity fields stay aligned."""
result = await session.execute(select(Entity).where(Entity.id == entity_id))
entity = result.scalar_one_or_none()
if entity is None:
raise ValueError(f"Entity {entity_id} does not exist")
if self.project_id is not None and entity.project_id != self.project_id:
raise ValueError(
f"Entity {entity_id} belongs to project {entity.project_id}, "
f"not repository project {self.project_id}"
)
return entity
async def _align_identity_fields(
self, session: AsyncSession, note_content: NoteContent
) -> None:
"""Mirror project identity from entity before persisting note content."""
entity = await self._load_entity_identity(session, note_content.entity_id)
note_content.project_id = entity.project_id
note_content.external_id = entity.external_id
note_content.file_path = Path(entity.file_path).as_posix()
async def get_by_entity_id(self, entity_id: int) -> Optional[NoteContent]:
"""Get note content by the owning entity identifier."""
return await self.find_by_id(entity_id)
async def get_by_external_id(self, external_id: str) -> Optional[NoteContent]:
"""Get note content by the mirrored entity external identifier."""
query = self.select().where(NoteContent.external_id == external_id)
return await self.find_one(query)
async def get_by_file_path(self, file_path: Path | str) -> Optional[NoteContent]:
"""Get note content by file path, preferring rows whose entity still owns that path."""
normalized_path = Path(file_path).as_posix()
# Trigger: note_content mirrors entity.file_path but does not enforce project-level uniqueness.
# Why: entity renames can leave stale mirrored paths behind until note_content realigns.
# Outcome: prefer the row whose current entity path still matches, then the newest mirror.
query = (
self.select()
.join(Entity, Entity.id == NoteContent.entity_id)
.where(NoteContent.file_path == normalized_path)
.order_by(
(Entity.file_path == normalized_path).desc(),
NoteContent.updated_at.desc(),
NoteContent.entity_id.desc(),
)
.limit(1)
.options(*self.get_load_options())
)
async with db.scoped_session(self.session_maker) as session:
result = await session.execute(query)
return result.scalars().first()
async def create(self, data: Mapping[str, Any] | NoteContent) -> NoteContent:
"""Create a note_content row aligned to its owning entity."""
note_content, _ = self._coerce_note_content(data)
async with db.scoped_session(self.session_maker) as session:
await self._align_identity_fields(session, note_content)
session.add(note_content)
await session.flush()
created = await self.select_by_id(session, note_content.entity_id)
if created is None: # pragma: no cover
raise ValueError(
f"Can't find NoteContent for entity {note_content.entity_id} after add"
)
return created
async def upsert(self, data: Mapping[str, Any] | NoteContent) -> NoteContent:
"""Insert or update note_content while keeping mirrored identity fields in sync."""
note_content, provided_fields = self._coerce_note_content(data)
async with db.scoped_session(self.session_maker) as session:
await self._align_identity_fields(session, note_content)
existing = await self.select_by_id(session, note_content.entity_id)
if existing is None:
session.add(note_content)
await session.flush()
created = await self.select_by_id(session, note_content.entity_id)
if created is None: # pragma: no cover
raise ValueError(
f"Can't find NoteContent for entity {note_content.entity_id} after upsert"
)
return created
fields_to_update = (provided_fields - {"entity_id"}) | {
"project_id",
"external_id",
"file_path",
}
for column_name in fields_to_update:
setattr(existing, column_name, getattr(note_content, column_name))
await session.flush()
updated = await self.select_by_id(session, existing.entity_id)
if updated is None: # pragma: no cover
raise ValueError(
f"Can't find NoteContent for entity {existing.entity_id} after upsert"
)
return updated
async def update_state_fields(self, entity_id: int, **updates: Any) -> Optional[NoteContent]:
"""Update sync fields and re-align project_id, external_id, and file_path from entity."""
invalid_fields = set(updates) - NOTE_CONTENT_MUTABLE_FIELDS
if invalid_fields:
invalid_list = ", ".join(sorted(invalid_fields))
raise ValueError(f"Unsupported note_content update fields: {invalid_list}")
async with db.scoped_session(self.session_maker) as session:
note_content = await self.select_by_id(session, entity_id)
if note_content is None:
return None
await self._align_identity_fields(session, note_content)
for field_name, value in updates.items():
setattr(note_content, field_name, value)
await session.flush()
updated = await self.select_by_id(session, entity_id)
if updated is None: # pragma: no cover
raise ValueError(f"Can't find NoteContent for entity {entity_id} after update")
return updated
async def delete_by_entity_id(self, entity_id: int) -> bool:
"""Delete note_content by entity identifier."""
async with db.scoped_session(self.session_maker) as session:
note_content = await self.select_by_id(session, entity_id)
if note_content is None:
return False
await session.delete(note_content)
return True
+46 -13
View File
@@ -18,6 +18,7 @@ class OpenAIEmbeddingProvider(EmbeddingProvider):
model_name: str = "text-embedding-3-small",
*,
batch_size: int = 64,
request_concurrency: int = 4,
dimensions: int = 1536,
api_key: str | None = None,
base_url: str | None = None,
@@ -26,12 +27,20 @@ class OpenAIEmbeddingProvider(EmbeddingProvider):
self.model_name = model_name
self.dimensions = dimensions
self.batch_size = batch_size
self.request_concurrency = request_concurrency
self._api_key = api_key
self._base_url = base_url
self._timeout = timeout
self._client: Any | None = None
self._client_lock = asyncio.Lock()
def runtime_log_attrs(self) -> dict[str, int]:
"""Return the request fan-out knobs that shape API embedding batches."""
return {
"provider_batch_size": self.batch_size,
"request_concurrency": self.request_concurrency,
}
async def _get_client(self) -> Any:
if self._client is not None:
return self._client
@@ -41,7 +50,7 @@ class OpenAIEmbeddingProvider(EmbeddingProvider):
return self._client
try:
from openai import AsyncOpenAI # type: ignore[import-not-found]
from openai import AsyncOpenAI
except ImportError as exc: # pragma: no cover - covered via monkeypatch tests
raise SemanticDependenciesMissingError(
"OpenAI dependency is missing. "
@@ -67,25 +76,49 @@ class OpenAIEmbeddingProvider(EmbeddingProvider):
return []
client = await self._get_client()
all_vectors: list[list[float]] = []
batches = [
texts[start : start + self.batch_size]
for start in range(0, len(texts), self.batch_size)
]
batch_vectors: list[list[list[float]] | None] = [None] * len(batches)
semaphore = asyncio.Semaphore(self.request_concurrency)
for start in range(0, len(texts), self.batch_size):
batch = texts[start : start + self.batch_size]
response = await client.embeddings.create(
model=self.model_name,
input=batch,
)
vectors_by_index: dict[int, list[float]] = {
int(item.index): [float(value) for value in item.embedding]
for item in response.data
}
async def embed_batch(batch_index: int, batch: list[str]) -> None:
async with semaphore:
response = await client.embeddings.create(
model=self.model_name,
input=batch,
)
vectors_by_index: dict[int, list[float]] = {}
for item in response.data:
response_index = int(item.index)
if response_index in vectors_by_index:
raise RuntimeError(
"OpenAI embedding response returned duplicate vector indexes."
)
vectors_by_index[response_index] = [float(value) for value in item.embedding]
ordered_vectors: list[list[float]] = []
for index in range(len(batch)):
vector = vectors_by_index.get(index)
if vector is None:
raise RuntimeError(
"OpenAI embedding response is missing expected vector index."
)
all_vectors.append(vector)
ordered_vectors.append(vector)
batch_vectors[batch_index] = ordered_vectors
await asyncio.gather(
*(embed_batch(batch_index, batch) for batch_index, batch in enumerate(batches))
)
all_vectors: list[list[float]] = []
for vectors in batch_vectors:
if vectors is None:
raise RuntimeError("OpenAI embedding batch did not produce vectors.")
all_vectors.extend(vectors)
if all_vectors and len(all_vectors[0]) != self.dimensions:
raise RuntimeError(
@@ -15,7 +15,10 @@ from basic_memory.config import BasicMemoryConfig, ConfigManager
from basic_memory.repository.embedding_provider import EmbeddingProvider
from basic_memory.repository.embedding_provider_factory import create_embedding_provider
from basic_memory.repository.search_index_row import SearchIndexRow
from basic_memory.repository.search_repository_base import SearchRepositoryBase
from basic_memory.repository.search_repository_base import (
SearchRepositoryBase,
VectorChunkState,
)
from basic_memory.repository.metadata_filters import parse_metadata_filters
from basic_memory.repository.semantic_errors import SemanticDependenciesMissingError
from basic_memory.schemas.search import SearchItemType, SearchRetrievalMode
@@ -61,6 +64,9 @@ class PostgresSearchRepository(SearchRepositoryBase):
self._semantic_embedding_sync_batch_size = (
self._app_config.semantic_embedding_sync_batch_size
)
self._semantic_postgres_prepare_concurrency = (
self._app_config.semantic_postgres_prepare_concurrency
)
self._embedding_provider = embedding_provider
self._vector_dimensions = 384
self._vector_tables_initialized = False
@@ -285,6 +291,10 @@ class PostgresSearchRepository(SearchRepositoryBase):
) from exc
# --- Chunks table (dimension-independent, may already exist via migration) ---
# Trigger: fresh Postgres projects may not have vector chunk tables yet.
# Why: runtime can bootstrap missing tables, but schema evolution must stay
# in Alembic to avoid concurrent ALTER TABLE deadlocks during indexing.
# Outcome: new installs create the current schema; upgrades rely on migration.
await session.execute(
text(
"""
@@ -295,6 +305,8 @@ class PostgresSearchRepository(SearchRepositoryBase):
chunk_key TEXT NOT NULL,
chunk_text TEXT NOT NULL,
source_hash TEXT NOT NULL,
entity_fingerprint TEXT NOT NULL,
embedding_model TEXT NOT NULL,
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
UNIQUE (project_id, entity_id, chunk_key)
)
@@ -441,35 +453,115 @@ class PostgresSearchRepository(SearchRepositoryBase):
)
return [dict(row) for row in vector_result.mappings().all()]
def _vector_prepare_window_size(self) -> int:
"""Use a bounded config-driven prepare window for Postgres vector sync."""
return self._semantic_postgres_prepare_concurrency
async def _upsert_scheduled_chunk_records(
self,
session: AsyncSession,
*,
entity_id: int,
scheduled_records: list[dict[str, str]],
existing_by_key: dict[str, VectorChunkState],
entity_fingerprint: str,
embedding_model: str,
) -> list[tuple[int, str]]:
"""Use Postgres UPSERT to rewrite only the scheduled chunk rows."""
if not scheduled_records:
return []
upsert_params: dict[str, object] = {
"project_id": self.project_id,
"entity_id": entity_id,
}
upsert_values: list[str] = []
# The SQL template is built from integer enumerate() indices only.
# No user-controlled text is interpolated into the statement.
for index, record in enumerate(scheduled_records):
upsert_params[f"chunk_key_{index}"] = record["chunk_key"]
upsert_params[f"chunk_text_{index}"] = record["chunk_text"]
upsert_params[f"source_hash_{index}"] = record["source_hash"]
upsert_params[f"entity_fingerprint_{index}"] = entity_fingerprint
upsert_params[f"embedding_model_{index}"] = embedding_model
upsert_values.append(
"("
":entity_id, :project_id, "
f":chunk_key_{index}, :chunk_text_{index}, :source_hash_{index}, "
f":entity_fingerprint_{index}, :embedding_model_{index}, NOW()"
")"
)
upsert_result = await session.execute(
text(f"""
INSERT INTO search_vector_chunks (
entity_id,
project_id,
chunk_key,
chunk_text,
source_hash,
entity_fingerprint,
embedding_model,
updated_at
) VALUES {", ".join(upsert_values)}
ON CONFLICT (project_id, entity_id, chunk_key) DO UPDATE SET
chunk_text = EXCLUDED.chunk_text,
source_hash = EXCLUDED.source_hash,
entity_fingerprint = EXCLUDED.entity_fingerprint,
embedding_model = EXCLUDED.embedding_model,
updated_at = NOW()
RETURNING id, chunk_key
"""),
upsert_params,
)
upserted_ids_by_key = {
str(row["chunk_key"]): int(row["id"]) for row in upsert_result.mappings().all()
}
return [
(upserted_ids_by_key[record["chunk_key"]], record["chunk_text"])
for record in scheduled_records
]
async def _write_embeddings(
self,
session: AsyncSession,
jobs: list[tuple[int, str]],
embeddings: list[list[float]],
) -> None:
for (row_id, _), vector in zip(jobs, embeddings, strict=True):
vector_literal = self._format_pgvector_literal(vector)
await session.execute(
text(
"INSERT INTO search_vector_embeddings ("
"chunk_id, project_id, embedding, embedding_dims, updated_at"
") VALUES ("
":chunk_id, :project_id, CAST(:embedding AS vector), :embedding_dims, NOW()"
") "
"ON CONFLICT (chunk_id) DO UPDATE SET "
"project_id = EXCLUDED.project_id, "
"embedding = EXCLUDED.embedding, "
"embedding_dims = EXCLUDED.embedding_dims, "
"updated_at = NOW()"
),
{
"chunk_id": row_id,
"project_id": self.project_id,
"embedding": vector_literal,
"embedding_dims": len(vector),
},
params: dict[str, object] = {"project_id": self.project_id}
value_rows: list[str] = []
# The SQL template is built from integer enumerate() indices only.
# No user-controlled text is interpolated into the statement.
for index, ((row_id, _), vector) in enumerate(zip(jobs, embeddings, strict=True)):
params[f"chunk_id_{index}"] = row_id
params[f"embedding_{index}"] = self._format_pgvector_literal(vector)
params[f"embedding_dims_{index}"] = len(vector)
value_rows.append(
"("
f":chunk_id_{index}, :project_id, CAST(:embedding_{index} AS vector), "
f":embedding_dims_{index}, NOW()"
")"
)
await session.execute(
text(f"""
INSERT INTO search_vector_embeddings (
chunk_id,
project_id,
embedding,
embedding_dims,
updated_at
) VALUES {", ".join(value_rows)}
ON CONFLICT (chunk_id) DO UPDATE SET
project_id = EXCLUDED.project_id,
embedding = EXCLUDED.embedding,
embedding_dims = EXCLUDED.embedding_dims,
updated_at = NOW()
"""),
params,
)
async def _delete_entity_chunks(
self,
session: AsyncSession,
@@ -506,9 +598,6 @@ class PostgresSearchRepository(SearchRepositoryBase):
stale_params,
)
async def _update_timestamp_sql(self) -> str:
return "NOW()" # pragma: no cover
def _distance_to_similarity(self, distance: float) -> float:
"""Convert pgvector cosine distance to cosine similarity.
@@ -597,7 +686,17 @@ class PostgresSearchRepository(SearchRepositoryBase):
# FTS search (Postgres-specific)
# ------------------------------------------------------------------
async def search(
@staticmethod
def _is_tsquery_syntax_error(exc: Exception) -> bool:
msg = str(exc).lower()
return (
"syntax error in tsquery" in msg
or "invalid input syntax for type tsquery" in msg
or "no operand in tsquery" in msg
or "no operator in tsquery" in msg
)
async def _build_fts_query_parts(
self,
search_text: Optional[str] = None,
permalink: Optional[str] = None,
@@ -607,31 +706,8 @@ class PostgresSearchRepository(SearchRepositoryBase):
after_date: Optional[datetime] = None,
search_item_types: Optional[List[SearchItemType]] = None,
metadata_filters: Optional[dict] = None,
retrieval_mode: SearchRetrievalMode = SearchRetrievalMode.FTS,
min_similarity: Optional[float] = None,
limit: int = 10,
offset: int = 0,
) -> List[SearchIndexRow]:
"""Search across all indexed content using PostgreSQL tsvector."""
# --- Dispatch vector / hybrid modes (shared logic) ---
dispatched = await self._dispatch_retrieval_mode(
search_text=search_text,
permalink=permalink,
permalink_match=permalink_match,
title=title,
note_types=note_types,
after_date=after_date,
search_item_types=search_item_types,
metadata_filters=metadata_filters,
retrieval_mode=retrieval_mode,
min_similarity=min_similarity,
limit=limit,
offset=offset,
)
if dispatched is not None:
return dispatched
# --- FTS mode (Postgres-specific) ---
) -> tuple[str, str, dict, str, str]:
"""Build Postgres FTS FROM/WHERE params shared by search and count."""
conditions = []
params = {}
order_by_clause = ""
@@ -779,10 +855,6 @@ class PostgresSearchRepository(SearchRepositoryBase):
params["project_id"] = self.project_id
conditions.append("search_index.project_id = :project_id")
# set limit and offset
params["limit"] = limit
params["offset"] = offset
# Build WHERE clause
where_clause = " AND ".join(conditions) if conditions else "1=1"
@@ -795,6 +867,64 @@ class PostgresSearchRepository(SearchRepositoryBase):
else:
score_expr = "0"
return from_clause, where_clause, params, order_by_clause, score_expr
async def search(
self,
search_text: Optional[str] = None,
permalink: Optional[str] = None,
permalink_match: Optional[str] = None,
title: Optional[str] = None,
note_types: Optional[List[str]] = None,
after_date: Optional[datetime] = None,
search_item_types: Optional[List[SearchItemType]] = None,
metadata_filters: Optional[dict] = None,
retrieval_mode: SearchRetrievalMode = SearchRetrievalMode.FTS,
min_similarity: Optional[float] = None,
limit: int = 10,
offset: int = 0,
) -> List[SearchIndexRow]:
"""Search across all indexed content using PostgreSQL tsvector."""
# --- Dispatch vector / hybrid modes (shared logic) ---
dispatched = await self._dispatch_retrieval_mode(
search_text=search_text,
permalink=permalink,
permalink_match=permalink_match,
title=title,
note_types=note_types,
after_date=after_date,
search_item_types=search_item_types,
metadata_filters=metadata_filters,
retrieval_mode=retrieval_mode,
min_similarity=min_similarity,
limit=limit,
offset=offset,
)
if dispatched is not None:
return dispatched
# --- FTS mode (Postgres-specific) ---
(
from_clause,
where_clause,
params,
order_by_clause,
score_expr,
) = await self._build_fts_query_parts(
search_text=search_text,
permalink=permalink,
permalink_match=permalink_match,
title=title,
note_types=note_types,
after_date=after_date,
search_item_types=search_item_types,
metadata_filters=metadata_filters,
)
# set limit and offset
params["limit"] = limit
params["offset"] = offset
sql = f"""
SELECT
search_index.project_id,
@@ -826,17 +956,7 @@ class PostgresSearchRepository(SearchRepositoryBase):
result = await session.execute(text(sql), params)
rows = result.fetchall()
except Exception as e:
# Handle tsquery syntax errors (and only those).
#
# Important: Postgres errors for other failures (e.g. missing table) will still mention
# `to_tsquery(...)` in the SQL text, so checking for the substring "tsquery" is too broad.
msg = str(e).lower()
if (
"syntax error in tsquery" in msg
or "invalid input syntax for type tsquery" in msg
or "no operand in tsquery" in msg
or "no operator in tsquery" in msg
):
if self._is_tsquery_syntax_error(e):
logger.warning(f"tsquery syntax error for search term: {search_text}, error: {e}")
return []
@@ -877,3 +997,60 @@ class PostgresSearchRepository(SearchRepositoryBase):
)
return results
async def count(
self,
search_text: Optional[str] = None,
permalink: Optional[str] = None,
permalink_match: Optional[str] = None,
title: Optional[str] = None,
note_types: Optional[List[str]] = None,
after_date: Optional[datetime] = None,
search_item_types: Optional[List[SearchItemType]] = None,
metadata_filters: Optional[dict] = None,
retrieval_mode: SearchRetrievalMode = SearchRetrievalMode.FTS,
min_similarity: Optional[float] = None,
) -> int:
"""Count indexed content matching the Postgres FTS query."""
if retrieval_mode != SearchRetrievalMode.FTS:
return await super().count(
search_text=search_text,
permalink=permalink,
permalink_match=permalink_match,
title=title,
note_types=note_types,
after_date=after_date,
search_item_types=search_item_types,
metadata_filters=metadata_filters,
retrieval_mode=retrieval_mode,
min_similarity=min_similarity,
)
(
from_clause,
where_clause,
params,
_order_by_clause,
_score_expr,
) = await self._build_fts_query_parts(
search_text=search_text,
permalink=permalink,
permalink_match=permalink_match,
title=title,
note_types=note_types,
after_date=after_date,
search_item_types=search_item_types,
metadata_filters=metadata_filters,
)
sql = f"SELECT COUNT(*) FROM {from_clause} WHERE {where_clause}"
logger.trace(f"Count {sql} params: {params}")
try:
async with db.scoped_session(self.session_maker) as session:
result = await session.execute(text(sql), params)
return int(result.scalar_one())
except Exception as e:
if self._is_tsquery_syntax_error(e):
logger.warning(f"tsquery syntax error for search term: {search_text}, error: {e}")
return 0
logger.error(f"Database error during search count: {e}")
raise
+40 -5
View File
@@ -13,6 +13,7 @@ from sqlalchemy import (
Result,
and_,
delete,
update as sqlalchemy_update,
)
from sqlalchemy.engine import CursorResult
from sqlalchemy.exc import NoResultFound
@@ -140,6 +141,20 @@ class Repository[T: Base]:
# Query within same session
return await self.select_by_ids(session, [m.id for m in models]) # pyright: ignore [reportAttributeAccessIssue]
async def add_all_no_return(self, models: List[T]) -> int:
"""Insert models without reloading them afterward."""
if not models:
return 0
async with db.scoped_session(self.session_maker) as session:
for model in models:
self._set_project_id_if_needed(model)
session.add_all(models)
await session.flush()
logger.debug(f"Added {len(models)} {self.Model.__name__} records")
return len(models)
def select(self, *entities: Any) -> Select:
"""Create a new SELECT statement.
@@ -268,7 +283,7 @@ class Repository[T: Base]:
return await self.select_by_ids(session, [model.id for model in model_list]) # pyright: ignore [reportAttributeAccessIssue]
async def update(self, entity_id: int, entity_data: dict | T) -> Optional[T]:
async def update(self, entity_id: int, entity_data: dict[str, Any] | T) -> Optional[T]:
"""Update an entity with the given data."""
logger.debug(f"Updating {self.Model.__name__} {entity_id} with data: {entity_data}")
async with db.scoped_session(self.session_maker) as session:
@@ -279,12 +294,13 @@ class Repository[T: Base]:
entity = result.scalars().one()
if isinstance(entity_data, dict):
for key, value in entity_data.items():
if key in self.valid_columns:
setattr(entity, key, value)
update_data = cast(dict[str, Any], entity_data)
for key in self.valid_columns:
if key in update_data:
setattr(entity, key, update_data[key])
elif isinstance(entity_data, self.Model):
for column in self.Model.__table__.columns.keys():
for column in self.valid_columns:
setattr(entity, column, getattr(entity_data, column))
await session.flush() # Make sure changes are flushed
@@ -297,6 +313,25 @@ class Repository[T: Base]:
logger.debug(f"No {self.Model.__name__} found to update: {entity_id}")
return None
async def update_fields(self, entity_id: Any, entity_data: dict[str, Any]) -> bool:
"""Update columns without reloading the model graph afterward."""
update_data = {k: v for k, v in entity_data.items() if k in self.valid_columns}
if not update_data:
return True
async with db.scoped_session(self.session_maker) as session:
conditions = [self.primary_key == entity_id]
if self.has_project_id and self.project_id is not None:
conditions.append(getattr(self.Model, "project_id") == self.project_id)
result = cast(
CursorResult[Any],
await session.execute(
sqlalchemy_update(self.Model).where(and_(*conditions)).values(**update_data)
),
)
return result.rowcount > 0
async def delete(self, entity_id: int) -> bool:
"""Delete an entity from the database."""
logger.debug(f"Deleting {self.Model.__name__}: {entity_id}")
@@ -50,6 +50,22 @@ class SearchRepository(Protocol):
"""Search across indexed content."""
...
async def count(
self,
search_text: Optional[str] = None,
permalink: Optional[str] = None,
permalink_match: Optional[str] = None,
title: Optional[str] = None,
note_types: Optional[List[str]] = None,
after_date: Optional[datetime] = None,
search_item_types: Optional[List[SearchItemType]] = None,
metadata_filters: Optional[dict] = None,
retrieval_mode: SearchRetrievalMode = SearchRetrievalMode.FTS,
min_similarity: Optional[float] = None,
) -> int:
"""Count indexed content matching the same filters as search."""
...
async def index_item(self, search_index_row: SearchIndexRow) -> None:
"""Index a single item."""
...
@@ -70,6 +86,10 @@ class SearchRepository(Protocol):
"""Sync semantic vector chunks for an entity."""
...
async def delete_entity_vector_rows(self, entity_id: int) -> None:
"""Delete semantic vector chunks and embeddings for one entity."""
...
async def sync_entity_vectors_batch(
self,
entity_ids: list[int],
File diff suppressed because it is too large Load Diff
@@ -1,11 +1,11 @@
"""SQLite FTS5-based search repository implementation."""
import asyncio
import json
import re
from contextlib import asynccontextmanager
from datetime import datetime
from typing import List, Optional
import asyncio
from loguru import logger
from sqlalchemy import text
from sqlalchemy.exc import OperationalError as SAOperationalError
@@ -56,7 +56,8 @@ class SQLiteSearchRepository(SearchRepositoryBase):
self._app_config.semantic_embedding_sync_batch_size
)
self._embedding_provider = embedding_provider
self._sqlite_vec_lock = asyncio.Lock()
self._sqlite_vec_load_lock = asyncio.Lock()
self._sqlite_prepare_write_lock = asyncio.Lock()
self._vector_tables_initialized = False
self._vector_dimensions = 384
@@ -93,9 +94,22 @@ class SQLiteSearchRepository(SearchRepositoryBase):
raise e
# Fail fast: create vector tables at startup so missing sqlite-vec
# or embedding provider errors surface immediately
# or embedding provider errors surface immediately.
# Trigger: the runtime semantic stack (sqlite-vec extension or embedding
# provider) is unavailable at startup.
# Why: failing the whole MCP boot for a search-only feature blocks
# Claude Desktop's handshake (#711). Keyword-only search is a
# reasonable fallback while the user resolves the dependency.
# Outcome: log the cause, mark this repository as semantic-disabled so
# downstream calls short-circuit cleanly, and let init complete.
if self._semantic_enabled:
await self._ensure_vector_tables()
try:
await self._ensure_vector_tables()
except SemanticDependenciesMissingError as exc:
logger.warning(
f"Semantic search disabled: {exc}. Falling back to keyword-only search."
)
self._semantic_enabled = False
# ------------------------------------------------------------------
# FTS5 query preparation (backend-specific)
@@ -349,7 +363,7 @@ class SQLiteSearchRepository(SearchRepositoryBase):
pass
try:
import sqlite_vec # type: ignore[import-not-found]
import sqlite_vec
except ImportError as exc:
raise SemanticDependenciesMissingError(
"sqlite-vec package is missing. "
@@ -357,7 +371,13 @@ class SQLiteSearchRepository(SearchRepositoryBase):
"pip install -U basic-memory"
) from exc
async with self._sqlite_vec_lock:
# Trigger: sqlite-vec must be loaded on each SQLite connection before
# vec tables and functions are visible.
# Why: extension loading is connection-local, so we need one narrow
# critical section to avoid racing two coroutines on the same step.
# Outcome: connection setup stays serialized without blocking unrelated
# prepare work behind the write-side lock.
async with self._sqlite_vec_load_lock:
try:
await session.execute(text("SELECT vec_version()"))
return
@@ -367,6 +387,25 @@ class SQLiteSearchRepository(SearchRepositoryBase):
async_connection = await session.connection()
raw_connection = await async_connection.get_raw_connection()
driver_connection = raw_connection.driver_connection
# Trigger: the underlying CPython was built without sqlite extension support.
# Why: python.org's macOS installer ships a stripped sqlite3 module with no
# enable_load_extension; when uvx happens to pick that interpreter (#711),
# the AttributeError surfaces here and previously crashed startup before
# Claude Desktop could complete its MCP handshake.
# Outcome: convert to SemanticDependenciesMissingError so the init-time
# handler can degrade gracefully to keyword search instead of dying.
if not hasattr(driver_connection, "enable_load_extension"):
raise SemanticDependenciesMissingError(
"This Python build does not support SQLite extension loading "
"(no enable_load_extension on sqlite3.Connection). "
"Common cause: python.org Python on macOS. "
"Reinstall basic-memory under a Python that ships extension "
"support (uv-managed CPython, Homebrew Python, or the official "
"Docker image), or set semantic_search_enabled=false in config "
"to silence this and use keyword-only search."
)
await driver_connection.enable_load_extension(True)
await driver_connection.load_extension(sqlite_vec.loadable_path())
await driver_connection.enable_load_extension(False)
@@ -398,10 +437,16 @@ class SQLiteSearchRepository(SearchRepositoryBase):
"chunk_key",
"chunk_text",
"source_hash",
"entity_fingerprint",
"embedding_model",
"updated_at",
}
schema_mismatch = bool(chunks_columns) and set(chunks_columns) != expected_columns
if schema_mismatch:
# Trigger: older SQLite installs are missing newly required chunk metadata columns.
# Why: vector tables store derived data only, so rebuilding them is safer than
# attempting piecemeal ALTER TABLE compatibility across sqlite-vec upgrades.
# Outcome: first startup after the schema change forces a clean re-embed.
logger.warning("search_vector_chunks schema mismatch, recreating vector tables")
await session.execute(text("DROP TABLE IF EXISTS search_vector_embeddings"))
await session.execute(text("DROP TABLE IF EXISTS search_vector_chunks"))
@@ -552,8 +597,60 @@ class SQLiteSearchRepository(SearchRepositoryBase):
stale_params,
)
async def _update_timestamp_sql(self) -> str:
return "CURRENT_TIMESTAMP" # pragma: no cover
async def delete_project_vector_rows(self) -> None:
"""Delete all vector rows for this project on a sqlite-vec-enabled connection."""
await self._ensure_vector_tables()
async with db.scoped_session(self.session_maker) as session:
await self._ensure_sqlite_vec_loaded(session)
# Constraint: sqlite-vec stores embeddings separately with no cascade delete.
# Why: full rebuild must clear embeddings before chunk rows or stale vectors remain.
# Outcome: the next sync recreates the project's derived vectors from scratch.
await session.execute(
text(
"DELETE FROM search_vector_embeddings WHERE rowid IN ("
"SELECT id FROM search_vector_chunks WHERE project_id = :project_id)"
),
{"project_id": self.project_id},
)
await session.execute(
text("DELETE FROM search_vector_chunks WHERE project_id = :project_id"),
{"project_id": self.project_id},
)
await session.commit()
async def delete_stale_vector_rows(self) -> None:
"""Delete vector rows whose source entities no longer exist."""
await self._ensure_vector_tables()
async with db.scoped_session(self.session_maker) as session:
await self._ensure_sqlite_vec_loaded(session)
stale_entity_filter = (
"entity_id NOT IN (SELECT id FROM entity WHERE project_id = :project_id)"
)
params = {"project_id": self.project_id}
# Trigger: deleted entities left behind derived vector rows.
# Why: sqlite-vec does not provide cascade cleanup from our chunk table.
# Outcome: stale vector state disappears before coverage stats or reindex runs.
await session.execute(
text(
"DELETE FROM search_vector_embeddings WHERE rowid IN ("
"SELECT id FROM search_vector_chunks "
f"WHERE project_id = :project_id AND {stale_entity_filter})"
),
params,
)
await session.execute(
text(
"DELETE FROM search_vector_chunks "
f"WHERE project_id = :project_id AND {stale_entity_filter}"
),
params,
)
await session.commit()
def _distance_to_similarity(self, distance: float) -> float:
"""Convert L2 distance to cosine similarity for normalized embeddings.
@@ -563,13 +660,26 @@ class SQLiteSearchRepository(SearchRepositoryBase):
"""
return max(0.0, 1.0 - (distance * distance) / 2.0)
def _orphan_detection_sql(self) -> str:
"""SQLite sqlite-vec uses rowid-based embedding table."""
@asynccontextmanager
async def _prepare_entity_write_scope(self):
"""SQLite keeps the shared read window, but funnels prepare writes through one lock."""
# Trigger: the shared prepare window fans out per entity after batched reads.
# Why: SQLite still benefits from shared reads, but write transactions do
# not get meaningfully faster when we open many at once.
# Outcome: one entity at a time mutates chunk rows, while vec extension
# loading uses its own separate lock and cannot deadlock this path.
async with self._sqlite_prepare_write_lock:
yield
def _prepare_window_existing_rows_sql(self, placeholders: str) -> str:
"""SQLite sqlite-vec stores embeddings by rowid rather than chunk_id."""
return (
"SELECT c.id FROM search_vector_chunks c "
"SELECT c.entity_id, c.id, c.chunk_key, c.source_hash, c.entity_fingerprint, "
"c.embedding_model, (e.rowid IS NOT NULL) AS has_embedding "
"FROM search_vector_chunks c "
"LEFT JOIN search_vector_embeddings e ON e.rowid = c.id "
"WHERE c.project_id = :project_id AND c.entity_id = :entity_id "
"AND e.rowid IS NULL"
f"WHERE c.project_id = :project_id AND c.entity_id IN ({placeholders}) "
"ORDER BY c.entity_id ASC, c.chunk_key ASC"
)
# ------------------------------------------------------------------
@@ -591,7 +701,11 @@ class SQLiteSearchRepository(SearchRepositoryBase):
# FTS search (backend-specific)
# ------------------------------------------------------------------
async def search(
@staticmethod
def _is_fts5_syntax_error(exc: Exception) -> bool:
return "fts5: syntax error" in str(exc).lower()
async def _build_fts_query_parts(
self,
search_text: Optional[str] = None,
permalink: Optional[str] = None,
@@ -601,31 +715,8 @@ class SQLiteSearchRepository(SearchRepositoryBase):
after_date: Optional[datetime] = None,
search_item_types: Optional[List[SearchItemType]] = None,
metadata_filters: Optional[dict] = None,
retrieval_mode: SearchRetrievalMode = SearchRetrievalMode.FTS,
min_similarity: Optional[float] = None,
limit: int = 10,
offset: int = 0,
) -> List[SearchIndexRow]:
"""Search across all indexed content using SQLite FTS5."""
# --- Dispatch vector / hybrid modes (shared logic) ---
dispatched = await self._dispatch_retrieval_mode(
search_text=search_text,
permalink=permalink,
permalink_match=permalink_match,
title=title,
note_types=note_types,
after_date=after_date,
search_item_types=search_item_types,
metadata_filters=metadata_filters,
retrieval_mode=retrieval_mode,
min_similarity=min_similarity,
limit=limit,
offset=offset,
)
if dispatched is not None:
return dispatched
# --- FTS mode (SQLite-specific) ---
) -> tuple[str, str, dict, str]:
"""Build SQLite FTS FROM/WHERE params shared by search and count."""
conditions = []
match_conditions = []
params = {}
@@ -801,13 +892,60 @@ class SQLiteSearchRepository(SearchRepositoryBase):
params["project_id"] = self.project_id
conditions.append("search_index.project_id = :project_id")
# Build WHERE clause
where_clause = " AND ".join(conditions) if conditions else "1=1"
return from_clause, where_clause, params, order_by_clause
async def search(
self,
search_text: Optional[str] = None,
permalink: Optional[str] = None,
permalink_match: Optional[str] = None,
title: Optional[str] = None,
note_types: Optional[List[str]] = None,
after_date: Optional[datetime] = None,
search_item_types: Optional[List[SearchItemType]] = None,
metadata_filters: Optional[dict] = None,
retrieval_mode: SearchRetrievalMode = SearchRetrievalMode.FTS,
min_similarity: Optional[float] = None,
limit: int = 10,
offset: int = 0,
) -> List[SearchIndexRow]:
"""Search across all indexed content using SQLite FTS5."""
# --- Dispatch vector / hybrid modes (shared logic) ---
dispatched = await self._dispatch_retrieval_mode(
search_text=search_text,
permalink=permalink,
permalink_match=permalink_match,
title=title,
note_types=note_types,
after_date=after_date,
search_item_types=search_item_types,
metadata_filters=metadata_filters,
retrieval_mode=retrieval_mode,
min_similarity=min_similarity,
limit=limit,
offset=offset,
)
if dispatched is not None:
return dispatched
# --- FTS mode (SQLite-specific) ---
from_clause, where_clause, params, order_by_clause = await self._build_fts_query_parts(
search_text=search_text,
permalink=permalink,
permalink_match=permalink_match,
title=title,
note_types=note_types,
after_date=after_date,
search_item_types=search_item_types,
metadata_filters=metadata_filters,
)
# set limit on search query
params["limit"] = limit
params["offset"] = offset
# Build WHERE clause
where_clause = " AND ".join(conditions) if conditions else "1=1"
sql = f"""
SELECT
search_index.project_id,
@@ -840,7 +978,7 @@ class SQLiteSearchRepository(SearchRepositoryBase):
rows = result.fetchall()
except Exception as e:
# Handle FTS5 syntax errors and provide user-friendly feedback
if "fts5: syntax error" in str(e).lower(): # pragma: no cover
if self._is_fts5_syntax_error(e): # pragma: no cover
logger.warning(f"FTS5 syntax error for search term: {search_text}, error: {e}")
# Return empty results rather than crashing
return []
@@ -878,3 +1016,54 @@ class SQLiteSearchRepository(SearchRepositoryBase):
)
return results
async def count(
self,
search_text: Optional[str] = None,
permalink: Optional[str] = None,
permalink_match: Optional[str] = None,
title: Optional[str] = None,
note_types: Optional[List[str]] = None,
after_date: Optional[datetime] = None,
search_item_types: Optional[List[SearchItemType]] = None,
metadata_filters: Optional[dict] = None,
retrieval_mode: SearchRetrievalMode = SearchRetrievalMode.FTS,
min_similarity: Optional[float] = None,
) -> int:
"""Count indexed content matching the SQLite FTS query."""
if retrieval_mode != SearchRetrievalMode.FTS:
return await super().count(
search_text=search_text,
permalink=permalink,
permalink_match=permalink_match,
title=title,
note_types=note_types,
after_date=after_date,
search_item_types=search_item_types,
metadata_filters=metadata_filters,
retrieval_mode=retrieval_mode,
min_similarity=min_similarity,
)
from_clause, where_clause, params, _order_by_clause = await self._build_fts_query_parts(
search_text=search_text,
permalink=permalink,
permalink_match=permalink_match,
title=title,
note_types=note_types,
after_date=after_date,
search_item_types=search_item_types,
metadata_filters=metadata_filters,
)
sql = f"SELECT COUNT(*) FROM {from_clause} WHERE {where_clause}"
logger.trace(f"Count {sql} params: {params}")
try:
async with db.scoped_session(self.session_maker) as session:
result = await session.execute(text(sql), params)
return int(result.scalar_one())
except Exception as e:
if self._is_fts5_syntax_error(e): # pragma: no cover
logger.warning(f"FTS5 syntax error for search term: {search_text}, error: {e}")
return 0
logger.error(f"Database error during search count: {e}")
raise
+67 -20
View File
@@ -8,7 +8,9 @@ Syntax reference:
field: type, description # required field
field?: type, description # optional field
field(array): type # array of values
field(array, description): type # array with description
field?(enum): [val1, val2] # enumeration
field?(enum, description): [val1, val2] # enum with description
field?(object): # nested object
sub_field: type
EntityName as type (capitalized) # entity reference
@@ -58,46 +60,88 @@ class SchemaDefinition:
# with an uppercase letter is treated as an entity reference.
SCALAR_TYPES = frozenset({"string", "integer", "number", "boolean", "any"})
MODIFIER_TYPES = frozenset({"array", "enum", "object"})
# --- Field Name Parsing ---
def _parse_field_key(key: str) -> tuple[str, bool, bool, bool, bool]:
def _parse_field_key_parts(key: str) -> tuple[str, bool, bool, bool, bool, str | None]:
"""Parse a Picoschema field key into its components.
Returns (name, required, is_array, is_enum, is_object).
The key format is: name[?][(array|enum|object)]
Returns (name, required, is_array, is_enum, is_object, description).
The key format is: name[?][(array|enum|object[, description])]
Examples:
"name" -> ("name", True, False, False)
"role?" -> ("role", False, False, False)
"tags?(array)" -> ("tags", False, True, False)
"status?(enum)" -> ("status", False, False, True)
"metadata?(object)" -> ("metadata", False, False, False) + children
"name" -> ("name", True, False, False, False, None)
"role?" -> ("role", False, False, False, False, None)
"tags?(array)" -> ("tags", False, True, False, False, None)
"tags?(array, labels)" -> ("tags", False, True, False, False, "labels")
"status?(enum)" -> ("status", False, False, True, False, None)
"metadata?(object)" -> ("metadata", False, False, False, True, None)
"""
required = True
is_array = False
is_enum = False
is_object = False
description = None
# Check for modifier suffix: (array), (enum), (object)
if key.endswith("(array)"):
key, modifier, description = _split_modifier_suffix(key)
if modifier == "array":
is_array = True
key = key[: -len("(array)")]
elif key.endswith("(enum)"):
elif modifier == "enum":
is_enum = True
key = key[: -len("(enum)")]
elif key.endswith("(object)"):
elif modifier == "object":
is_object = True
key = key[: -len("(object)")]
# Check for optional marker
if key.endswith("?"):
required = False
key = key[:-1]
return key, required, is_array, is_enum, is_object
return key.strip(), required, is_array, is_enum, is_object, description
def _parse_field_key(key: str) -> tuple[str, bool, bool, bool, bool]:
"""Parse a Picoschema field key, discarding any modifier description."""
name, required, is_array, is_enum, is_object, _description = _parse_field_key_parts(key)
return name, required, is_array, is_enum, is_object
def _split_modifier_suffix(key: str) -> tuple[str, str | None, str | None]:
"""Split a trailing picoschema modifier from a field key."""
stripped_key = key.rstrip()
if not stripped_key.endswith(")"):
return key, None, None
# Trigger: field names and modifier descriptions may both contain parentheses
# Why: only the parenthesis paired with the final suffix can introduce a modifier
# Outcome: preserves names like "risk(score)" and descriptions like "labels (freeform)"
open_paren_index = -1
depth = 0
for index in range(len(stripped_key) - 1, -1, -1):
char = stripped_key[index]
if char == ")":
depth += 1
elif char == "(":
depth -= 1
if depth == 0:
open_paren_index = index
break
if open_paren_index == -1:
return key, None, None
modifier_text = stripped_key[open_paren_index + 1 : -1].strip()
modifier, separator, description = modifier_text.partition(",")
modifier = modifier.strip()
if modifier not in MODIFIER_TYPES:
return key, None, None
key_without_modifier = stripped_key[:open_paren_index].rstrip()
parsed_description = description.strip() if separator else None
return key_without_modifier, modifier, parsed_description or None
def _parse_type_and_description(value: str) -> tuple[str, str | None]:
@@ -170,7 +214,7 @@ def parse_picoschema(yaml_dict: dict) -> list[SchemaField]:
fields: list[SchemaField] = []
for key, value in yaml_dict.items():
name, required, is_array, is_enum, is_object = _parse_field_key(key)
name, required, is_array, is_enum, is_object, key_description = _parse_field_key_parts(key)
# --- Enum fields ---
# Trigger: value is a list or a string containing bracketed enum values
@@ -179,11 +223,12 @@ def parse_picoschema(yaml_dict: dict) -> list[SchemaField]:
# in YAML to avoid parse errors)
# Outcome: SchemaField with is_enum=True and enum_values populated
if is_enum:
description = None
description = key_description
if isinstance(value, list):
enum_values = [str(v) for v in value]
else:
enum_values, description = _parse_enum_string(str(value))
enum_values, value_description = _parse_enum_string(str(value))
description = description or value_description
fields.append(
SchemaField(
name=name,
@@ -207,13 +252,15 @@ def parse_picoschema(yaml_dict: dict) -> list[SchemaField]:
name=name,
type="object",
required=required,
description=key_description,
children=children,
)
)
continue
# --- Scalar and entity ref fields ---
type_str, description = _parse_type_and_description(str(value))
type_str, value_description = _parse_type_and_description(str(value))
description = key_description or value_description
is_entity_ref = _is_entity_ref_type(type_str)
fields.append(
+55
View File
@@ -63,8 +63,10 @@ class WorkspaceInfo(BaseModel):
tenant_id: str = Field(..., description="Workspace tenant identifier")
workspace_type: str = Field(..., description="Workspace type (personal or organization)")
slug: str = Field(..., description="Stable workspace slug for qualified project routing")
name: str = Field(..., description="Workspace display name")
role: str = Field(..., description="Current user's role in the workspace")
is_default: bool = Field(..., description="Whether this is the default cloud workspace")
organization_id: str | None = Field(None, description="Organization ID for org workspaces")
has_active_subscription: bool = Field(
default=False, description="Whether the workspace has an active subscription"
@@ -78,6 +80,59 @@ class WorkspaceListResponse(BaseModel):
default_factory=list, description="Available workspaces"
)
count: int = Field(default=0, description="Number of available workspaces")
default_workspace_id: str | None = Field(
default=None, description="Default workspace tenant ID when available"
)
current_workspace_id: str | None = Field(
default=None, description="Current workspace tenant ID when available"
)
class CloudProjectIndexStatus(BaseModel):
"""Index freshness summary for one cloud project."""
project_name: str = Field(..., description="Project name")
project_id: int = Field(..., description="Project database identifier")
last_scan_timestamp: float | None = Field(
default=None, description="Last scan timestamp from project metadata"
)
last_file_count: int | None = Field(default=None, description="Last observed file count")
current_file_count: int = Field(..., description="Current markdown file count")
total_entities: int = Field(..., description="Current markdown entity count")
total_note_content_rows: int = Field(..., description="Rows present in note_content")
note_content_synced: int = Field(..., description="Files fully materialized into note_content")
note_content_pending: int = Field(..., description="Pending note_content rows")
note_content_failed: int = Field(..., description="Failed note_content rows")
note_content_external_changes: int = Field(
..., description="Rows flagged with external file changes"
)
total_indexed_entities: int = Field(..., description="Files represented in search_index")
embedding_opt_out_entities: int = Field(..., description="Files opted out of vector embeddings")
embeddable_indexed_entities: int = Field(
..., description="Indexed files eligible for vector embeddings"
)
total_entities_with_chunks: int = Field(..., description="Embeddable files with vector chunks")
total_chunks: int = Field(..., description="Vector chunk row count")
total_embeddings: int = Field(..., description="Vector embedding row count")
orphaned_chunks: int = Field(..., description="Chunks missing embeddings")
vector_tables_exist: bool = Field(..., description="Whether vector tables exist")
materialization_current: bool = Field(
..., description="Whether note content matches the current file set"
)
search_current: bool = Field(..., description="Whether search coverage is current")
embeddings_current: bool = Field(..., description="Whether embedding coverage is current")
project_current: bool = Field(..., description="Whether all freshness checks are current")
reindex_recommended: bool = Field(..., description="Whether a reindex is recommended")
reindex_reason: str | None = Field(default=None, description="Reason a reindex is recommended")
class CloudTenantIndexStatusResponse(BaseModel):
"""Index freshness summary for all projects in one cloud tenant."""
tenant_id: str = Field(..., description="Workspace tenant identifier")
fly_app_name: str = Field(..., description="Cloud tenant application identifier")
email: str | None = Field(default=None, description="Owner email when available")
projects: list[CloudProjectIndexStatus] = Field(
default_factory=list, description="Per-project freshness summaries"
)
error: str | None = Field(default=None, description="Tenant-level lookup error")
+1 -1
View File
@@ -103,7 +103,7 @@ MemoryUrl = Annotated[
memory_url = TypeAdapter(MemoryUrl)
def memory_url_path(url: memory_url) -> str: # pyright: ignore
def memory_url_path(url: str) -> str:
"""
Returns the uri for a url value by removing the prefix "memory://" from a given MemoryUrl.

Some files were not shown because too many files have changed in this diff Show More