mirror of
https://github.com/basicmachines-co/basic-memory
synced 2026-06-21 13:47:35 +00:00
Compare commits
2 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 1c2120963a | |||
| 7c954ae509 |
@@ -1,244 +0,0 @@
|
||||
---
|
||||
name: instrumentation
|
||||
description: Add Pydantic Logfire observability to applications. Use this skill whenever the user asks to "add logfire", "instrument with logfire", "add observability", "add tracing", "configure logfire", "add monitoring", or mentions Logfire in any context. Supports Python, JavaScript/TypeScript, and Rust. Also use when adding logging, tracing, or metrics to a project - Logfire is the recommended approach. Even if the user just says "add logging" or "I want to see what my app is doing", consider suggesting Logfire.
|
||||
---
|
||||
|
||||
# Instrument with Logfire
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
Invoke this skill when:
|
||||
- User asks to "add logfire", "add observability", "add tracing", or "add monitoring"
|
||||
- User wants to instrument an app with structured logging or tracing (Python, JS/TS, or Rust)
|
||||
- User mentions Logfire in any context
|
||||
- User asks to "add logging" or "see what my app is doing"
|
||||
- User wants to monitor AI/LLM calls (PydanticAI, OpenAI, Anthropic)
|
||||
- User asks to add observability to an AI agent or LLM pipeline
|
||||
|
||||
## How Logfire Works
|
||||
|
||||
Logfire is an observability platform built on OpenTelemetry. It captures traces, logs, and metrics from applications. Logfire has native SDKs for Python, JavaScript/TypeScript, and Rust, plus support for any language via OpenTelemetry.
|
||||
|
||||
The reason this skill exists is that Claude tends to get a few things subtly wrong with Logfire - especially the ordering of `configure()` vs `instrument_*()` calls, the structured logging syntax, and which extras to install. These matter because a misconfigured setup silently drops traces.
|
||||
|
||||
## Step 1: Detect Language and Frameworks
|
||||
|
||||
Identify the project language and instrumentable libraries:
|
||||
|
||||
- **Python**: Read `pyproject.toml` or `requirements.txt`. Common instrumentable libraries: FastAPI, httpx, asyncpg, SQLAlchemy, psycopg, Redis, Celery, Django, Flask, requests, PydanticAI.
|
||||
- **JavaScript/TypeScript**: Read `package.json`. Common frameworks: Express, Next.js, Fastify. Also check for Cloudflare Workers or Deno.
|
||||
- **Rust**: Read `Cargo.toml`.
|
||||
|
||||
Then follow the language-specific steps below.
|
||||
|
||||
---
|
||||
|
||||
## Python
|
||||
|
||||
### Install with Extras
|
||||
|
||||
Install `logfire` with extras matching the detected frameworks. Each instrumented library needs its corresponding extra - without it, the `instrument_*()` call will fail at runtime with a missing dependency error.
|
||||
|
||||
```bash
|
||||
uv add 'logfire[fastapi,httpx,asyncpg]'
|
||||
```
|
||||
|
||||
The full list of available extras: `fastapi`, `starlette`, `django`, `flask`, `httpx`, `requests`, `asyncpg`, `psycopg`, `psycopg2`, `sqlalchemy`, `redis`, `pymongo`, `mysql`, `sqlite3`, `celery`, `aiohttp`, `aws-lambda`, `system-metrics`, `litellm`, `dspy`, `google-genai`.
|
||||
|
||||
### Configure and Instrument
|
||||
|
||||
This is where ordering matters. `logfire.configure()` initializes the SDK and must come before everything else. The `instrument_*()` calls register hooks into each library. If you call `instrument_*()` before `configure()`, the hooks register but traces go nowhere.
|
||||
|
||||
```python
|
||||
import logfire
|
||||
|
||||
# 1. Configure first - always
|
||||
logfire.configure()
|
||||
|
||||
# 2. Instrument libraries - after configure, before app starts
|
||||
logfire.instrument_fastapi(app)
|
||||
logfire.instrument_httpx()
|
||||
logfire.instrument_asyncpg()
|
||||
```
|
||||
|
||||
Placement rules:
|
||||
- `logfire.configure()` goes in the application entry point (`main.py`, or the module that creates the app)
|
||||
- Call it **once per process** - not inside request handlers, not in library code
|
||||
- `instrument_*()` calls go right after `configure()`
|
||||
- Web framework instrumentors (`instrument_fastapi`, `instrument_flask`, `instrument_django`) need the app instance as an argument. HTTP client and database instrumentors (`instrument_httpx`, `instrument_asyncpg`) are global and take no arguments.
|
||||
- In **Gunicorn** deployments, call `logfire.configure()` inside the `post_fork` hook, not at module level - each worker is a separate process
|
||||
|
||||
### Structured Logging
|
||||
|
||||
Replace `print()` and `logging.*()` calls with Logfire's structured logging. The key pattern: use `{key}` placeholders with keyword arguments, never f-strings.
|
||||
|
||||
```python
|
||||
# Correct - each {key} becomes a searchable attribute in the Logfire UI
|
||||
logfire.info("Created user {user_id}", user_id=uid)
|
||||
logfire.error("Payment failed {amount} {currency}", amount=100, currency="USD")
|
||||
|
||||
# Wrong - creates a flat string, nothing is searchable
|
||||
logfire.info(f"Created user {uid}")
|
||||
```
|
||||
|
||||
For grouping related operations and measuring duration, use spans:
|
||||
|
||||
```python
|
||||
with logfire.span("Processing order {order_id}", order_id=order_id):
|
||||
items = await fetch_items(order_id)
|
||||
total = calculate_total(items)
|
||||
logfire.info("Calculated total {total}", total=total)
|
||||
```
|
||||
|
||||
For exceptions, use `logfire.exception()` which automatically captures the traceback:
|
||||
|
||||
```python
|
||||
try:
|
||||
await process_order(order_id)
|
||||
except Exception:
|
||||
logfire.exception("Failed to process order {order_id}", order_id=order_id)
|
||||
raise
|
||||
```
|
||||
|
||||
### AI/LLM Instrumentation (Python)
|
||||
|
||||
Logfire auto-instruments AI libraries to capture LLM calls, token usage, tool invocations, and agent runs.
|
||||
|
||||
```bash
|
||||
uv add 'logfire[pydantic-ai]'
|
||||
# or: uv add 'logfire[openai]' / uv add 'logfire[anthropic]'
|
||||
```
|
||||
|
||||
Available AI extras: `pydantic-ai`, `openai`, `anthropic`, `litellm`, `dspy`, `google-genai`.
|
||||
|
||||
```python
|
||||
logfire.configure()
|
||||
logfire.instrument_pydantic_ai() # captures agent runs, tool calls, LLM request/response
|
||||
# or:
|
||||
logfire.instrument_openai() # captures chat completions, embeddings, token counts
|
||||
logfire.instrument_anthropic() # captures messages, token usage
|
||||
```
|
||||
|
||||
For PydanticAI, each agent run becomes a parent span containing child spans for every tool call and LLM request.
|
||||
|
||||
---
|
||||
|
||||
## JavaScript / TypeScript
|
||||
|
||||
### Install
|
||||
|
||||
```bash
|
||||
# Node.js
|
||||
npm install @pydantic/logfire-node
|
||||
|
||||
# Cloudflare Workers
|
||||
npm install @pydantic/logfire-cf-workers logfire
|
||||
|
||||
# Next.js / generic
|
||||
npm install logfire
|
||||
```
|
||||
|
||||
### Configure
|
||||
|
||||
**Node.js (Express, Fastify, etc.)** - create an `instrumentation.ts` loaded before your app:
|
||||
|
||||
```typescript
|
||||
import * as logfire from '@pydantic/logfire-node'
|
||||
logfire.configure()
|
||||
```
|
||||
|
||||
Launch with: `node --require ./instrumentation.js app.js`
|
||||
|
||||
The SDK auto-instruments common libraries when loaded before the app. Set `LOGFIRE_TOKEN` in your environment or pass `token` to `configure()`.
|
||||
|
||||
**Cloudflare Workers** - wrap your handler with `instrument()`:
|
||||
|
||||
```typescript
|
||||
import { instrument } from '@pydantic/logfire-cf-workers'
|
||||
|
||||
export default instrument(handler, {
|
||||
service: { name: 'my-worker', version: '1.0.0' }
|
||||
})
|
||||
```
|
||||
|
||||
**Next.js** - set environment variables for OpenTelemetry export:
|
||||
|
||||
```
|
||||
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://logfire-api.pydantic.dev/v1/traces
|
||||
OTEL_EXPORTER_OTLP_HEADERS=Authorization=<your-write-token>
|
||||
```
|
||||
|
||||
### Structured Logging (JS/TS)
|
||||
|
||||
```typescript
|
||||
// Structured attributes as second argument
|
||||
logfire.info('Created user', { user_id: uid })
|
||||
logfire.error('Payment failed', { amount: 100, currency: 'USD' })
|
||||
|
||||
// Spans
|
||||
logfire.span('Processing order', { order_id }, {}, async () => {
|
||||
logfire.info('Processing step completed')
|
||||
})
|
||||
|
||||
// Error reporting
|
||||
logfire.reportError('order processing', error)
|
||||
```
|
||||
|
||||
Log levels: `trace`, `debug`, `info`, `notice`, `warn`, `error`, `fatal`.
|
||||
|
||||
---
|
||||
|
||||
## Rust
|
||||
|
||||
### Install
|
||||
|
||||
```toml
|
||||
[dependencies]
|
||||
logfire = "0.6"
|
||||
```
|
||||
|
||||
### Configure
|
||||
|
||||
```rust
|
||||
let shutdown_handler = logfire::configure()
|
||||
.install_panic_handler()
|
||||
.finish()?;
|
||||
```
|
||||
|
||||
Set `LOGFIRE_TOKEN` in your environment or use the Logfire CLI to select a project.
|
||||
|
||||
### Structured Logging (Rust)
|
||||
|
||||
The Rust SDK is built on `tracing` and `opentelemetry` - existing `tracing` macros work automatically.
|
||||
|
||||
```rust
|
||||
// Spans
|
||||
logfire::span!("processing order", order_id = order_id).in_scope(|| {
|
||||
// traced code
|
||||
});
|
||||
|
||||
// Events
|
||||
logfire::info!("Created user {user_id}", user_id = uid);
|
||||
```
|
||||
|
||||
Always call `shutdown_handler.shutdown()` before program exit to flush data.
|
||||
|
||||
---
|
||||
|
||||
## Verify
|
||||
|
||||
After instrumentation, verify the setup works:
|
||||
|
||||
1. Run `logfire auth` to check authentication (or set `LOGFIRE_TOKEN`)
|
||||
2. Start the app and trigger a request
|
||||
3. Check https://logfire.pydantic.dev/ for traces
|
||||
|
||||
If traces aren't appearing: check that `configure()` is called before `instrument_*()` (Python), check that `LOGFIRE_TOKEN` is set, and check that the correct packages/extras are installed.
|
||||
|
||||
## References
|
||||
|
||||
Detailed patterns and integration tables, organized by language:
|
||||
|
||||
- **Python**: `${CLAUDE_PLUGIN_ROOT}/skills/instrumentation/references/python/logging-patterns.md` (log levels, spans, stdlib integration, metrics, capfire testing) and `${CLAUDE_PLUGIN_ROOT}/skills/instrumentation/references/python/integrations.md` (full instrumentor table with extras)
|
||||
- **JavaScript/TypeScript**: `${CLAUDE_PLUGIN_ROOT}/skills/instrumentation/references/javascript/patterns.md` (log levels, spans, error handling, config) and `${CLAUDE_PLUGIN_ROOT}/skills/instrumentation/references/javascript/frameworks.md` (Node.js, Cloudflare Workers, Next.js, Deno setup)
|
||||
- **Rust**: `${CLAUDE_PLUGIN_ROOT}/skills/instrumentation/references/rust/patterns.md` (macros, spans, tracing/log crate integration, async, shutdown)
|
||||
@@ -1,78 +0,0 @@
|
||||
# JavaScript Framework Setup
|
||||
|
||||
## Node.js (Express, Fastify, etc.)
|
||||
|
||||
Create `instrumentation.ts` and load it before your app:
|
||||
|
||||
```typescript
|
||||
// instrumentation.ts
|
||||
import * as logfire from '@pydantic/logfire-node'
|
||||
import 'dotenv/config'
|
||||
|
||||
logfire.configure()
|
||||
```
|
||||
|
||||
Launch:
|
||||
|
||||
```bash
|
||||
node --require ./instrumentation.js app.js
|
||||
# or with ts-node:
|
||||
npx ts-node --require ./instrumentation.ts app.ts
|
||||
```
|
||||
|
||||
The SDK auto-instruments common libraries (http, fetch, express, etc.) when loaded before the app via `--require`.
|
||||
|
||||
## Cloudflare Workers
|
||||
|
||||
```typescript
|
||||
import { instrument } from '@pydantic/logfire-cf-workers'
|
||||
|
||||
const handler = {
|
||||
async fetch(request: Request, env: Env, ctx: ExecutionContext) {
|
||||
return new Response('Hello')
|
||||
},
|
||||
}
|
||||
|
||||
export default instrument(handler, {
|
||||
service: { name: 'my-worker', version: '1.0.0' },
|
||||
})
|
||||
```
|
||||
|
||||
Add `LOGFIRE_TOKEN` to `.dev.vars` and enable `nodejs_compat` in `wrangler.toml`:
|
||||
|
||||
```toml
|
||||
compatibility_flags = ["nodejs_compat"]
|
||||
```
|
||||
|
||||
## Next.js / Vercel
|
||||
|
||||
Set environment variables in `.env.local` or Vercel dashboard:
|
||||
|
||||
```bash
|
||||
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://logfire-api.pydantic.dev/v1/traces
|
||||
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://logfire-api.pydantic.dev/v1/metrics
|
||||
OTEL_EXPORTER_OTLP_HEADERS=Authorization=<your-write-token>
|
||||
```
|
||||
|
||||
Optionally use the `logfire` package for manual spans in server components and API routes:
|
||||
|
||||
```typescript
|
||||
import * as logfire from 'logfire'
|
||||
|
||||
logfire.info('Server action executed', { action: 'createUser' })
|
||||
```
|
||||
|
||||
## Deno
|
||||
|
||||
Deno has built-in OpenTelemetry support. Set environment variables:
|
||||
|
||||
```bash
|
||||
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://logfire-api.pydantic.dev/v1/traces
|
||||
OTEL_EXPORTER_OTLP_HEADERS=Authorization=<your-write-token>
|
||||
```
|
||||
|
||||
Run with telemetry enabled:
|
||||
|
||||
```bash
|
||||
deno run --allow-env --unstable-otel app.ts
|
||||
```
|
||||
@@ -1,75 +0,0 @@
|
||||
# JavaScript / TypeScript Patterns
|
||||
|
||||
## Log Levels
|
||||
|
||||
From lowest to highest severity:
|
||||
|
||||
```typescript
|
||||
logfire.trace('Detailed trace', { detail: x })
|
||||
logfire.debug('Debug info', { state: s })
|
||||
logfire.info('Normal operation', { event: e })
|
||||
logfire.notice('Notable event', { event: e })
|
||||
logfire.warn('Warning', { issue: i })
|
||||
logfire.error('Error occurred', { error: err })
|
||||
logfire.fatal('Fatal error', { error: err })
|
||||
```
|
||||
|
||||
All methods accept `(message, attributes?, options?)`. Options can include `{ tags: ['tag1'] }`.
|
||||
|
||||
## Spans
|
||||
|
||||
### Callback-based (auto-closes)
|
||||
|
||||
```typescript
|
||||
await logfire.span('Processing order', { order_id }, {}, async () => {
|
||||
const items = await fetchItems(order_id)
|
||||
logfire.info('Fetched items', { count: items.length })
|
||||
return processItems(items)
|
||||
})
|
||||
```
|
||||
|
||||
### Manual control
|
||||
|
||||
```typescript
|
||||
const span = logfire.startSpan('Long operation', { job_id })
|
||||
try {
|
||||
await doWork()
|
||||
} finally {
|
||||
span.end()
|
||||
}
|
||||
```
|
||||
|
||||
Child spans reference their parent via the `parentSpan` option.
|
||||
|
||||
## Error Handling
|
||||
|
||||
```typescript
|
||||
try {
|
||||
await processOrder(orderId)
|
||||
} catch (error) {
|
||||
logfire.reportError('order processing', error)
|
||||
throw error
|
||||
}
|
||||
```
|
||||
|
||||
`reportError` automatically extracts stack traces and error details into structured span attributes.
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment variables
|
||||
|
||||
```bash
|
||||
LOGFIRE_TOKEN=your-write-token
|
||||
LOGFIRE_SERVICE_NAME=my-service
|
||||
LOGFIRE_SERVICE_VERSION=1.0.0
|
||||
```
|
||||
|
||||
### Programmatic
|
||||
|
||||
```typescript
|
||||
logfire.configure({
|
||||
token: process.env.LOGFIRE_TOKEN,
|
||||
serviceName: 'my-service',
|
||||
serviceVersion: '1.0.0',
|
||||
})
|
||||
```
|
||||
@@ -1,67 +0,0 @@
|
||||
# Python Integration Reference
|
||||
|
||||
## Web Frameworks
|
||||
|
||||
| Framework | Instrumentor | Needs app instance | Extra |
|
||||
|-----------|-------------|-------------------|-------|
|
||||
| FastAPI | `logfire.instrument_fastapi(app)` | Yes | `fastapi` |
|
||||
| Django | `logfire.instrument_django(app)` | Yes | `django` |
|
||||
| Flask | `logfire.instrument_flask(app)` | Yes | `flask` |
|
||||
| Starlette | `logfire.instrument_starlette(app)` | Yes | `starlette` |
|
||||
| AIOHTTP | `logfire.instrument_aiohttp_client()` | No | `aiohttp` |
|
||||
|
||||
## HTTP Clients
|
||||
|
||||
| Library | Instrumentor | Extra |
|
||||
|---------|-------------|-------|
|
||||
| httpx | `logfire.instrument_httpx()` | `httpx` |
|
||||
| requests | `logfire.instrument_requests()` | `requests` |
|
||||
|
||||
## Databases
|
||||
|
||||
| Library | Instrumentor | Extra |
|
||||
|---------|-------------|-------|
|
||||
| asyncpg | `logfire.instrument_asyncpg()` | `asyncpg` |
|
||||
| psycopg | `logfire.instrument_psycopg()` | `psycopg` |
|
||||
| psycopg2 | `logfire.instrument_psycopg2()` | `psycopg2` |
|
||||
| SQLAlchemy | `logfire.instrument_sqlalchemy()` | `sqlalchemy` |
|
||||
| PyMongo | `logfire.instrument_pymongo()` | `pymongo` |
|
||||
| MySQL | `logfire.instrument_mysql()` | `mysql` |
|
||||
| SQLite3 | `logfire.instrument_sqlite3()` | `sqlite3` |
|
||||
| Redis | `logfire.instrument_redis()` | `redis` |
|
||||
|
||||
## AI/LLM Frameworks
|
||||
|
||||
| Framework | Instrumentor | Extra |
|
||||
|-----------|-------------|-------|
|
||||
| PydanticAI | `logfire.instrument_pydantic_ai()` | `pydantic-ai` |
|
||||
| OpenAI | `logfire.instrument_openai()` | `openai` |
|
||||
| Anthropic | `logfire.instrument_anthropic()` | `anthropic` |
|
||||
| LiteLLM | `logfire.instrument_litellm()` | `litellm` |
|
||||
| DSPy | `logfire.instrument_dspy()` | `dspy` |
|
||||
| Google GenAI | `logfire.instrument_google_genai()` | `google-genai` |
|
||||
|
||||
## Task Queues
|
||||
|
||||
| Framework | Instrumentor | Extra |
|
||||
|-----------|-------------|-------|
|
||||
| Celery | `logfire.instrument_celery()` | `celery` |
|
||||
|
||||
## Other
|
||||
|
||||
| Feature | Instrumentor | Extra |
|
||||
|---------|-------------|-------|
|
||||
| System Metrics | `logfire.instrument_system_metrics()` | `system-metrics` |
|
||||
| Pydantic Models | `logfire.instrument_pydantic()` | - (built-in) |
|
||||
| AWS Lambda | handler wrapper | `aws-lambda` |
|
||||
|
||||
## Gunicorn Configuration
|
||||
|
||||
```python
|
||||
# gunicorn.conf.py
|
||||
import logfire
|
||||
|
||||
def post_fork(server, worker):
|
||||
logfire.configure()
|
||||
logfire.instrument_fastapi(app)
|
||||
```
|
||||
@@ -1,101 +0,0 @@
|
||||
# Python Logging Patterns
|
||||
|
||||
## Log Levels
|
||||
|
||||
From lowest to highest severity:
|
||||
|
||||
```python
|
||||
logfire.trace("Detailed trace {detail}", detail=x)
|
||||
logfire.debug("Debug info {state}", state=s)
|
||||
logfire.info("Normal operation {event}", event=e)
|
||||
logfire.notice("Notable event {event}", event=e)
|
||||
logfire.warn("Warning {issue}", issue=i)
|
||||
logfire.error("Error occurred {error}", error=err)
|
||||
logfire.fatal("Fatal error {error}", error=err)
|
||||
```
|
||||
|
||||
## Nested Spans
|
||||
|
||||
Spans nest to create a tree visible in the Logfire UI. Use them to show the structure of an operation, not just that it happened:
|
||||
|
||||
```python
|
||||
with logfire.span("HTTP request {method} {url}", method="POST", url=url):
|
||||
with logfire.span("Serialize payload"):
|
||||
payload = model.model_dump_json()
|
||||
with logfire.span("Send request"):
|
||||
response = await client.post(url, content=payload)
|
||||
logfire.info("Response {status}", status=response.status_code)
|
||||
```
|
||||
|
||||
## Standard Library Logging Integration
|
||||
|
||||
For projects that already use Python's `logging` module, route existing log calls through Logfire rather than rewriting them all:
|
||||
|
||||
```python
|
||||
from logging import basicConfig
|
||||
import logfire
|
||||
|
||||
logfire.configure()
|
||||
basicConfig(handlers=[logfire.LogfireLoggingHandler()])
|
||||
```
|
||||
|
||||
Or with `dictConfig`:
|
||||
|
||||
```python
|
||||
from logging.config import dictConfig
|
||||
import logfire
|
||||
|
||||
logfire.configure()
|
||||
dictConfig({
|
||||
'version': 1,
|
||||
'handlers': {
|
||||
'logfire': {'class': 'logfire.LogfireLoggingHandler'},
|
||||
},
|
||||
'root': {'handlers': ['logfire']},
|
||||
})
|
||||
```
|
||||
|
||||
## Suppressing Noisy Libraries
|
||||
|
||||
Some libraries emit excessive debug logs. Silence them at the `logging` level:
|
||||
|
||||
```python
|
||||
import logging
|
||||
|
||||
logging.getLogger('httpcore').setLevel(logging.WARNING)
|
||||
logging.getLogger('httpx').setLevel(logging.WARNING)
|
||||
```
|
||||
|
||||
## Custom Metrics
|
||||
|
||||
For dashboards and alerting, create metrics:
|
||||
|
||||
```python
|
||||
counter = logfire.metric_counter("orders_processed", unit="1")
|
||||
counter.add(1, {"status": "success"})
|
||||
|
||||
histogram = logfire.metric_histogram("request_duration", unit="s")
|
||||
histogram.record(0.123, {"endpoint": "/api/users"})
|
||||
|
||||
gauge = logfire.metric_gauge("active_connections")
|
||||
gauge.set(42)
|
||||
```
|
||||
|
||||
## Testing with capfire
|
||||
|
||||
Use the `capfire` pytest fixture to assert on emitted spans without sending data to production:
|
||||
|
||||
```python
|
||||
from logfire.testing import CaptureLogfire
|
||||
|
||||
def test_order_processing(capfire: CaptureLogfire) -> None:
|
||||
process_order(order_id=123)
|
||||
|
||||
spans = capfire.exporter.exported_spans_as_dict()
|
||||
assert any(
|
||||
span['attributes'].get('order_id') == 123
|
||||
for span in spans
|
||||
)
|
||||
```
|
||||
|
||||
Configure logfire with `send_to_logfire=False` in test fixtures to prevent production data leakage.
|
||||
@@ -1,106 +0,0 @@
|
||||
# Rust Patterns
|
||||
|
||||
## Core Macros
|
||||
|
||||
The Rust SDK is built on `tracing` and `opentelemetry`. All `tracing` macros work automatically with Logfire.
|
||||
|
||||
### Events (log points)
|
||||
|
||||
```rust
|
||||
logfire::trace!("Detailed trace {detail}", detail = x);
|
||||
logfire::debug!("Debug info {state}", state = s);
|
||||
logfire::info!("Normal operation {event}", event = e);
|
||||
logfire::warn!("Warning {issue}", issue = i);
|
||||
logfire::error!("Error occurred {err}", err = e);
|
||||
```
|
||||
|
||||
### Spans
|
||||
|
||||
```rust
|
||||
// Scoped - span closes when closure completes
|
||||
logfire::span!("Processing order {order_id}", order_id = id).in_scope(|| {
|
||||
let items = fetch_items(id);
|
||||
logfire::info!("Fetched {count} items", count = items.len());
|
||||
process_items(items)
|
||||
});
|
||||
|
||||
// Guard-based - span closes when guard is dropped
|
||||
let _guard = logfire::span!("Long operation {job_id}", job_id = id).entered();
|
||||
do_work();
|
||||
// span ends when _guard goes out of scope
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
```rust
|
||||
use logfire;
|
||||
|
||||
fn main() -> Result<(), Box<dyn std::error::Error>> {
|
||||
let shutdown_handler = logfire::configure()
|
||||
.install_panic_handler() // captures panics as error spans
|
||||
.finish()?;
|
||||
|
||||
// application code...
|
||||
|
||||
shutdown_handler.shutdown()?; // flush all pending spans
|
||||
Ok(())
|
||||
}
|
||||
```
|
||||
|
||||
Set `LOGFIRE_TOKEN` in your environment or use the Logfire CLI (`logfire auth`).
|
||||
|
||||
## Tracing Crate Compatibility
|
||||
|
||||
Any library using `tracing` macros automatically sends data through Logfire:
|
||||
|
||||
```rust
|
||||
use tracing;
|
||||
|
||||
tracing::info!("This also appears in Logfire");
|
||||
|
||||
#[tracing::instrument]
|
||||
fn my_function(param: &str) {
|
||||
// automatically creates a span with param as an attribute
|
||||
}
|
||||
```
|
||||
|
||||
## Log Crate Integration
|
||||
|
||||
The `log` crate is automatically captured and forwarded to Logfire. Libraries using `log::info!()`, `log::error!()`, etc. will appear in your Logfire dashboard without any additional configuration.
|
||||
|
||||
## Async Spans
|
||||
|
||||
```rust
|
||||
use tracing::Instrument;
|
||||
|
||||
async fn process_order(order_id: u64) {
|
||||
let span = logfire::span!("process order {order_id}", order_id = order_id);
|
||||
async {
|
||||
fetch_items(order_id).await;
|
||||
logfire::info!("Order processed");
|
||||
}
|
||||
.instrument(span)
|
||||
.await;
|
||||
}
|
||||
```
|
||||
|
||||
## Shutdown
|
||||
|
||||
Always call `shutdown()` before program exit to flush pending data:
|
||||
|
||||
```rust
|
||||
// In main()
|
||||
let shutdown_handler = logfire::configure().finish()?;
|
||||
|
||||
// ... app runs ...
|
||||
|
||||
// Before exit
|
||||
shutdown_handler.shutdown()?;
|
||||
```
|
||||
|
||||
For web servers using `tokio`, handle shutdown via signal:
|
||||
|
||||
```rust
|
||||
tokio::signal::ctrl_c().await?;
|
||||
shutdown_handler.shutdown()?;
|
||||
```
|
||||
+1
-19
@@ -1,21 +1,3 @@
|
||||
{
|
||||
"$schema": "https://json.schemastore.org/claude-code-settings.json",
|
||||
"env": {
|
||||
"CLAUDE_BASH_MAINTAIN_PROJECT_WORKING_DIR": "1",
|
||||
"CLAUDE_CODE_DISABLE_FEEDBACK_SURVEY": "1",
|
||||
"CLAUDE_CODE_NO_FLICKER": "1",
|
||||
"CLAUDE_CODE_DISABLE_ADAPTIVE_THINKING": "1"
|
||||
},
|
||||
"permissions": {
|
||||
"allow": [
|
||||
"Bash(just fast-check)",
|
||||
"Bash(just check)",
|
||||
"Bash(just fix)",
|
||||
"Bash(just typecheck)",
|
||||
"Bash(just lint)",
|
||||
"Bash(just test)"
|
||||
],
|
||||
"deny": []
|
||||
},
|
||||
"enableAllProjectMcpServers": true
|
||||
"enabledPlugins": {}
|
||||
}
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
../../.agents/skills/instrumentation
|
||||
@@ -10,9 +10,6 @@ on:
|
||||
# - "src/**/*.js"
|
||||
# - "src/**/*.jsx"
|
||||
|
||||
env:
|
||||
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
|
||||
|
||||
jobs:
|
||||
claude-review:
|
||||
# Only run for organization members and collaborators
|
||||
@@ -30,7 +27,7 @@ jobs:
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v6
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 1
|
||||
|
||||
|
||||
@@ -4,9 +4,6 @@ on:
|
||||
issues:
|
||||
types: [opened]
|
||||
|
||||
env:
|
||||
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
|
||||
|
||||
jobs:
|
||||
triage:
|
||||
runs-on: ubuntu-latest
|
||||
@@ -15,7 +12,7 @@ jobs:
|
||||
id-token: write
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v6
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 1
|
||||
|
||||
@@ -71,4 +68,4 @@ jobs:
|
||||
|
||||
Read the issue carefully and provide helpful triage with appropriate labels.
|
||||
|
||||
claude_args: '--allowed-tools "Bash(gh issue:*),Bash(gh search:*),Read"'
|
||||
claude_args: '--allowed-tools "Bash(gh issue:*),Bash(gh search:*),Read"'
|
||||
@@ -12,9 +12,6 @@ on:
|
||||
pull_request_target:
|
||||
types: [opened, synchronize]
|
||||
|
||||
env:
|
||||
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
|
||||
|
||||
jobs:
|
||||
claude:
|
||||
if: |
|
||||
@@ -44,7 +41,7 @@ jobs:
|
||||
actions: read # Required for Claude to read CI results on PRs
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v6
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
# For pull_request_target, checkout the PR head to review the actual changes
|
||||
ref: ${{ github.event_name == 'pull_request_target' && github.event.pull_request.head.sha || github.sha }}
|
||||
@@ -68,3 +65,4 @@ jobs:
|
||||
# See https://github.com/anthropics/claude-code-action/blob/main/docs/usage.md
|
||||
# or https://docs.claude.com/en/docs/claude-code/sdk#command-line for available options
|
||||
# claude_args: '--model claude-opus-4-1-20250805 --allowed-tools Bash(gh pr:*)'
|
||||
|
||||
|
||||
@@ -5,9 +5,6 @@ on:
|
||||
branches: [main]
|
||||
workflow_dispatch: # Allow manual triggering
|
||||
|
||||
env:
|
||||
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
|
||||
|
||||
jobs:
|
||||
dev-release:
|
||||
runs-on: ubuntu-latest
|
||||
@@ -16,12 +13,12 @@ jobs:
|
||||
contents: write
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.12"
|
||||
|
||||
@@ -53,4 +50,4 @@ jobs:
|
||||
uses: pypa/gh-action-pypi-publish@release/v1
|
||||
with:
|
||||
password: ${{ secrets.PYPI_TOKEN }}
|
||||
skip-existing: true # Don't fail if version already exists
|
||||
skip-existing: true # Don't fail if version already exists
|
||||
@@ -9,7 +9,6 @@ on:
|
||||
env:
|
||||
REGISTRY: ghcr.io
|
||||
IMAGE_NAME: basicmachines-co/basic-memory
|
||||
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
|
||||
|
||||
jobs:
|
||||
docker:
|
||||
@@ -20,17 +19,17 @@ jobs:
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v6
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v4
|
||||
uses: docker/setup-buildx-action@v3
|
||||
with:
|
||||
platforms: linux/amd64,linux/arm64
|
||||
|
||||
- name: Log in to GitHub Container Registry
|
||||
uses: docker/login-action@v4
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ${{ env.REGISTRY }}
|
||||
username: ${{ github.actor }}
|
||||
@@ -38,7 +37,7 @@ jobs:
|
||||
|
||||
- name: Extract metadata
|
||||
id: meta
|
||||
uses: docker/metadata-action@v6
|
||||
uses: docker/metadata-action@v5
|
||||
with:
|
||||
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
|
||||
tags: |
|
||||
@@ -49,7 +48,7 @@ jobs:
|
||||
type=raw,value=latest,enable={{is_default_branch}}
|
||||
|
||||
- name: Build and push Docker image
|
||||
uses: docker/build-push-action@v7
|
||||
uses: docker/build-push-action@v5
|
||||
with:
|
||||
context: .
|
||||
file: ./Dockerfile
|
||||
@@ -59,3 +58,4 @@ jobs:
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
|
||||
@@ -7,14 +7,11 @@ on:
|
||||
- edited
|
||||
- synchronize
|
||||
|
||||
env:
|
||||
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
|
||||
|
||||
jobs:
|
||||
main:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: amannn/action-semantic-pull-request@v6
|
||||
- uses: amannn/action-semantic-pull-request@v5
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
with:
|
||||
@@ -41,4 +38,4 @@ jobs:
|
||||
deps
|
||||
installer
|
||||
# Allow breaking changes (needs "!" after type/scope)
|
||||
requireScopeForBreakingChange: true
|
||||
requireScopeForBreakingChange: true
|
||||
@@ -5,9 +5,6 @@ on:
|
||||
tags:
|
||||
- 'v*' # Trigger on version tags like v1.0.0, v0.13.0, etc.
|
||||
|
||||
env:
|
||||
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
|
||||
|
||||
jobs:
|
||||
release:
|
||||
runs-on: ubuntu-latest
|
||||
@@ -16,12 +13,12 @@ jobs:
|
||||
contents: write
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.12"
|
||||
|
||||
@@ -42,7 +39,7 @@ jobs:
|
||||
echo "Build completed successfully"
|
||||
|
||||
- name: Create GitHub Release
|
||||
uses: softprops/action-gh-release@v3
|
||||
uses: softprops/action-gh-release@v2
|
||||
with:
|
||||
files: |
|
||||
dist/*.whl
|
||||
@@ -63,63 +60,26 @@ jobs:
|
||||
# Only run for stable releases (not dev, beta, or rc versions)
|
||||
if: ${{ !contains(github.ref_name, 'dev') && !contains(github.ref_name, 'b') && !contains(github.ref_name, 'rc') }}
|
||||
permissions:
|
||||
contents: read
|
||||
contents: write
|
||||
actions: read
|
||||
steps:
|
||||
# Inline bump replaces mislav/bump-homebrew-formula-action@v4.x.
|
||||
# The action does a HEAD request to api.github.com /repos/.../tarball/<ref>
|
||||
# with the bearer token and expects a 302 redirect. GitHub now returns
|
||||
# 303 on that endpoint when authenticated, which the action treats as a
|
||||
# fatal error. Re-implementing the bump as plain git+sed keeps the same
|
||||
# contract (update url + sha256, commit, push) with no third-party action.
|
||||
- name: Update Homebrew formula
|
||||
uses: mislav/bump-homebrew-formula-action@v3
|
||||
with:
|
||||
# Formula name in homebrew-basic-memory repo
|
||||
formula-name: basic-memory
|
||||
# The tap repository
|
||||
homebrew-tap: basicmachines-co/homebrew-basic-memory
|
||||
# Base branch of the tap repository
|
||||
base-branch: main
|
||||
# Download URL will be automatically constructed from the tag
|
||||
download-url: https://github.com/basicmachines-co/basic-memory/archive/refs/tags/${{ github.ref_name }}.tar.gz
|
||||
# Commit message for the formula update
|
||||
commit-message: |
|
||||
{{formulaName}} {{version}}
|
||||
|
||||
Created by https://github.com/basicmachines-co/basic-memory/actions/runs/${{ github.run_id }}
|
||||
env:
|
||||
HOMEBREW_TOKEN: ${{ secrets.HOMEBREW_TOKEN }}
|
||||
REF: ${{ github.ref_name }}
|
||||
REPO: ${{ github.repository }}
|
||||
RUN_URL: https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# Personal Access Token with repo scope for homebrew-basic-memory repo
|
||||
COMMITTER_TOKEN: ${{ secrets.HOMEBREW_TOKEN }}
|
||||
|
||||
VERSION="${REF#v}"
|
||||
ARCHIVE_URL="https://github.com/${REPO}/archive/refs/tags/${REF}.tar.gz"
|
||||
|
||||
echo "::group::Compute tarball sha256"
|
||||
SHA256="$(curl --fail --silent --location "$ARCHIVE_URL" | sha256sum | awk '{print $1}')"
|
||||
test -n "$SHA256"
|
||||
echo "sha256: $SHA256"
|
||||
echo "::endgroup::"
|
||||
|
||||
echo "::group::Clone tap"
|
||||
git clone \
|
||||
--depth 1 \
|
||||
"https://x-access-token:${HOMEBREW_TOKEN}@github.com/basicmachines-co/homebrew-basic-memory.git" \
|
||||
tap
|
||||
cd tap
|
||||
git config user.name "github-actions[bot]"
|
||||
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
|
||||
echo "::endgroup::"
|
||||
|
||||
echo "::group::Patch Formula/basic-memory.rb"
|
||||
# Pipe-delimited sed because the URL contains slashes. The Formula
|
||||
# only has one `url` and one `sha256` directive, so a first-match
|
||||
# replacement is unambiguous. POSIX character classes ([[:space:]])
|
||||
# keep this portable across BSD and GNU sed.
|
||||
sed -i -E \
|
||||
-e "s|^([[:space:]]*url[[:space:]]+)\"[^\"]+\"|\1\"${ARCHIVE_URL}\"|" \
|
||||
-e "s|^([[:space:]]*sha256[[:space:]]+)\"[^\"]+\"|\1\"${SHA256}\"|" \
|
||||
Formula/basic-memory.rb
|
||||
git --no-pager diff Formula/basic-memory.rb
|
||||
echo "::endgroup::"
|
||||
|
||||
if git diff --quiet Formula/basic-memory.rb; then
|
||||
echo "Formula already at ${REF}; nothing to do."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "::group::Commit & push"
|
||||
git add Formula/basic-memory.rb
|
||||
git commit -m "basic-memory ${VERSION}
|
||||
|
||||
Created by ${RUN_URL}"
|
||||
git push origin HEAD:main
|
||||
echo "::endgroup::"
|
||||
|
||||
+32
-59
@@ -5,14 +5,10 @@ concurrency:
|
||||
cancel-in-progress: true
|
||||
|
||||
on:
|
||||
# Trigger: PR branch pushes already publish commit statuses that show up on the PR.
|
||||
# Why: running the full matrix on both push and pull_request doubles CI time for the
|
||||
# exact same branch head commit.
|
||||
# Outcome: each branch push runs the test suite once, including PR updates.
|
||||
push:
|
||||
|
||||
env:
|
||||
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
|
||||
branches: [ "main" ]
|
||||
pull_request:
|
||||
branches: [ "main" ]
|
||||
|
||||
jobs:
|
||||
static-checks:
|
||||
@@ -21,12 +17,12 @@ jobs:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: true
|
||||
|
||||
- name: Set up Python 3.12
|
||||
uses: actions/setup-python@v6
|
||||
uses: actions/setup-python@v4
|
||||
with:
|
||||
python-version: "3.12"
|
||||
cache: "pip"
|
||||
@@ -35,7 +31,7 @@ jobs:
|
||||
run: |
|
||||
pip install uv
|
||||
|
||||
- uses: extractions/setup-just@v4
|
||||
- uses: extractions/setup-just@v3
|
||||
|
||||
- name: Create virtual env
|
||||
run: |
|
||||
@@ -55,7 +51,8 @@ jobs:
|
||||
|
||||
test-sqlite-unit:
|
||||
name: Test SQLite Unit (${{ matrix.os }}, Python ${{ matrix.python-version }})
|
||||
timeout-minutes: 45
|
||||
timeout-minutes: 30
|
||||
needs: [static-checks]
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
@@ -71,12 +68,12 @@ jobs:
|
||||
runs-on: ${{ matrix.os }}
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: true
|
||||
|
||||
- name: Set up Python ${{ matrix.python-version }}
|
||||
uses: actions/setup-python@v6
|
||||
uses: actions/setup-python@v4
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
cache: 'pip'
|
||||
@@ -85,7 +82,7 @@ jobs:
|
||||
run: |
|
||||
pip install uv
|
||||
|
||||
- uses: extractions/setup-just@v4
|
||||
- uses: extractions/setup-just@v3
|
||||
|
||||
- name: Create virtual env
|
||||
run: |
|
||||
@@ -102,6 +99,7 @@ jobs:
|
||||
test-sqlite-integration:
|
||||
name: Test SQLite Integration (${{ matrix.os }}, Python ${{ matrix.python-version }})
|
||||
timeout-minutes: 45
|
||||
needs: [static-checks]
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
@@ -117,12 +115,12 @@ jobs:
|
||||
runs-on: ${{ matrix.os }}
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: true
|
||||
|
||||
- name: Set up Python ${{ matrix.python-version }}
|
||||
uses: actions/setup-python@v6
|
||||
uses: actions/setup-python@v4
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
cache: 'pip'
|
||||
@@ -131,7 +129,7 @@ jobs:
|
||||
run: |
|
||||
pip install uv
|
||||
|
||||
- uses: extractions/setup-just@v4
|
||||
- uses: extractions/setup-just@v3
|
||||
|
||||
- name: Create virtual env
|
||||
run: |
|
||||
@@ -147,7 +145,8 @@ jobs:
|
||||
|
||||
test-postgres-unit:
|
||||
name: Test Postgres Unit (Python ${{ matrix.python-version }})
|
||||
timeout-minutes: 60
|
||||
timeout-minutes: 30
|
||||
needs: [static-checks]
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
@@ -156,30 +155,16 @@ jobs:
|
||||
- python-version: "3.13"
|
||||
- python-version: "3.14"
|
||||
runs-on: ubuntu-latest
|
||||
services:
|
||||
postgres:
|
||||
image: pgvector/pgvector:pg16
|
||||
env:
|
||||
POSTGRES_USER: basic_memory_user
|
||||
POSTGRES_PASSWORD: dev_password
|
||||
POSTGRES_DB: basic_memory_test
|
||||
ports:
|
||||
- 5432:5432
|
||||
options: >-
|
||||
--health-cmd "pg_isready -U basic_memory_user -d basic_memory_test"
|
||||
--health-interval 10s
|
||||
--health-timeout 5s
|
||||
--health-retries 5
|
||||
env:
|
||||
BASIC_MEMORY_TEST_POSTGRES_URL: postgresql://basic_memory_user:dev_password@127.0.0.1:5432/basic_memory_test
|
||||
|
||||
# Note: No services section needed - testcontainers handles Postgres in Docker
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: true
|
||||
|
||||
- name: Set up Python ${{ matrix.python-version }}
|
||||
uses: actions/setup-python@v6
|
||||
uses: actions/setup-python@v4
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
cache: 'pip'
|
||||
@@ -188,7 +173,7 @@ jobs:
|
||||
run: |
|
||||
pip install uv
|
||||
|
||||
- uses: extractions/setup-just@v4
|
||||
- uses: extractions/setup-just@v3
|
||||
|
||||
- name: Create virtual env
|
||||
run: |
|
||||
@@ -205,6 +190,7 @@ jobs:
|
||||
test-postgres-integration:
|
||||
name: Test Postgres Integration (Python ${{ matrix.python-version }})
|
||||
timeout-minutes: 45
|
||||
needs: [static-checks]
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
@@ -213,30 +199,16 @@ jobs:
|
||||
- python-version: "3.13"
|
||||
- python-version: "3.14"
|
||||
runs-on: ubuntu-latest
|
||||
services:
|
||||
postgres:
|
||||
image: pgvector/pgvector:pg16
|
||||
env:
|
||||
POSTGRES_USER: basic_memory_user
|
||||
POSTGRES_PASSWORD: dev_password
|
||||
POSTGRES_DB: basic_memory_test
|
||||
ports:
|
||||
- 5432:5432
|
||||
options: >-
|
||||
--health-cmd "pg_isready -U basic_memory_user -d basic_memory_test"
|
||||
--health-interval 10s
|
||||
--health-timeout 5s
|
||||
--health-retries 5
|
||||
env:
|
||||
BASIC_MEMORY_TEST_POSTGRES_URL: postgresql://basic_memory_user:dev_password@127.0.0.1:5432/basic_memory_test
|
||||
|
||||
# Note: No services section needed - testcontainers handles Postgres in Docker
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: true
|
||||
|
||||
- name: Set up Python ${{ matrix.python-version }}
|
||||
uses: actions/setup-python@v6
|
||||
uses: actions/setup-python@v4
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
cache: 'pip'
|
||||
@@ -245,7 +217,7 @@ jobs:
|
||||
run: |
|
||||
pip install uv
|
||||
|
||||
- uses: extractions/setup-just@v4
|
||||
- uses: extractions/setup-just@v3
|
||||
|
||||
- name: Create virtual env
|
||||
run: |
|
||||
@@ -262,15 +234,16 @@ jobs:
|
||||
test-semantic:
|
||||
name: Test Semantic (Python 3.12)
|
||||
timeout-minutes: 45
|
||||
needs: [static-checks]
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: true
|
||||
|
||||
- name: Set up Python 3.12
|
||||
uses: actions/setup-python@v6
|
||||
uses: actions/setup-python@v4
|
||||
with:
|
||||
python-version: "3.12"
|
||||
cache: "pip"
|
||||
@@ -279,7 +252,7 @@ jobs:
|
||||
run: |
|
||||
pip install uv
|
||||
|
||||
- uses: extractions/setup-just@v4
|
||||
- uses: extractions/setup-just@v3
|
||||
|
||||
- name: Create virtual env
|
||||
run: |
|
||||
|
||||
@@ -22,16 +22,16 @@ See the [README.md](README.md) file for a project overview.
|
||||
- Run unit tests (Postgres): `just test-unit-postgres`
|
||||
- Run integration tests (SQLite): `just test-int-sqlite`
|
||||
- Run integration tests (Postgres): `just test-int-postgres`
|
||||
- Run impacted tests: `just testmon` (pytest-testmon; only tests affected by changed code)
|
||||
- Run impacted tests: `just testmon` (pytest-testmon)
|
||||
- Run MCP smoke test: `just test-smoke`
|
||||
- Fast local loop: `just fast-check` (default iteration flow)
|
||||
- Fast local loop: `just fast-check`
|
||||
- Local consistency check: `just doctor`
|
||||
- Generate HTML coverage: `just coverage`
|
||||
- Single test: `pytest tests/path/to/test_file.py::test_function_name`
|
||||
- Run benchmarks: `pytest test-int/test_sync_performance_benchmark.py -v -m "benchmark and not slow"`
|
||||
- Lint: `just lint` or `ruff check . --fix`
|
||||
- Type check: `just typecheck` or `uv run ty check src tests test-int`
|
||||
- Type check (pyright): `just typecheck-pyright` or `uv run pyright`
|
||||
- Type check: `just typecheck` or `uv run pyright`
|
||||
- Type check (supplemental): `just typecheck-ty` or `uv run ty check src/`
|
||||
- Format: `just format` or `uv run ruff format .`
|
||||
- Run all code checks: `just check` (runs lint, format, typecheck, test)
|
||||
- Create db migration: `just migration "Your migration message"`
|
||||
@@ -48,23 +48,12 @@ See the [README.md](README.md) file for a project overview.
|
||||
### Code/Test/Verify Loop (fast path)
|
||||
|
||||
1) **Code:** make changes.
|
||||
2) **Test:** `just fast-check` (lint/format/typecheck + pytest-testmon impacted tests for changed code).
|
||||
2) **Test:** `just fast-check` (lint/format/typecheck + impacted tests + MCP smoke).
|
||||
3) **Verify:** `just doctor` (end-to-end file ↔ DB loop in a temp project).
|
||||
4) **Full gate (when needed):** `just test` or `just check` for SQLite + Postgres.
|
||||
|
||||
Run `just test-smoke` when you specifically need the MCP smoke flow.
|
||||
|
||||
If testmon is “cold,” the first run may be long. Subsequent runs get much faster.
|
||||
|
||||
### PR CI Gate
|
||||
|
||||
Before opening or updating a PR, run the checks that mirror the common required CI failures:
|
||||
|
||||
- Run `just typecheck` in addition to targeted `ruff` and `pytest` commands when tests were added or changed.
|
||||
- Sign commits with `git commit -s` so DCO passes. If a PR branch already has unsigned commits, rewrite the branch with signed-off commits before asking for review.
|
||||
- Use a semantic PR title accepted by `.github/workflows/pr-title.yml`: `type(scope): summary`.
|
||||
- Use one of the allowed scopes: `core`, `cli`, `api`, `mcp`, `sync`, `ui`, `deps`, `installer`.
|
||||
|
||||
### Test Structure
|
||||
|
||||
- `tests/` - Unit tests for individual components (mocked, fast)
|
||||
@@ -253,31 +242,6 @@ async_client.set_client_factory(your_custom_factory)
|
||||
|
||||
See SPEC-16 for full context manager refactor details.
|
||||
|
||||
### Release Process
|
||||
|
||||
Releases are driven by `just release` / `just beta` — never by a bare `git tag`. The recipes bump version metadata, run pre-flight checks, commit, tag, and push. GitHub Actions then publishes to PyPI and updates the Homebrew formula.
|
||||
|
||||
**Stable release:**
|
||||
|
||||
```
|
||||
just release v0.21.3
|
||||
```
|
||||
|
||||
The recipe runs `just lint` + `just typecheck`, then updates `__version__` in `src/basic_memory/__init__.py` and `"version"` in `server.json` (MCP registry metadata), commits as `chore: update version to X.Y.Z for vX.Y.Z release`, creates the `vX.Y.Z` tag, and pushes both the commit and the tag to `origin/main`. After the tag lands, the `Release` workflow builds the package, publishes to PyPI, creates the GitHub release with auto-generated notes, and updates the Homebrew formula. The recipe finishes by printing the post-release tasks the workflow doesn't cover.
|
||||
|
||||
**Beta release:** `just beta v0.21.3b1` — same flow with a beta-suffixed tag. PyPI consumers install with `pip install basic-memory --pre`.
|
||||
|
||||
**Development builds:** every commit to `main` publishes a `0.21.3.dev26+468a22f`-style version to PyPI automatically via `.github/workflows/dev-release.yml`. No human action.
|
||||
|
||||
**Do not tag releases by hand.** A bare `git tag vX.Y.Z` skips the in-code version bump. Package metadata is still correct (uv-dynamic-versioning derives it from the git tag) but `basic-memory --version` reports the previous release, which is what happened with v0.21.2 → v0.21.3.
|
||||
|
||||
**Post-release tasks** the recipe surfaces but doesn't run:
|
||||
- `docs.basicmemory.com` — add notes to `src/pages/latest-releases.mdx`
|
||||
- `basicmachines.co` — bump version in `src/components/sections/hero.tsx`
|
||||
- MCP Registry — `mcp-publisher publish` from the repo root
|
||||
|
||||
See `.claude/commands/release/release.md` (and `beta.md`, `release-check.md`, `changelog.md` alongside it) for the full release + post-release runbook, including the slash commands.
|
||||
|
||||
## BASIC MEMORY PRODUCT USAGE
|
||||
|
||||
### Knowledge Structure
|
||||
@@ -478,9 +442,5 @@ With GitHub integration, the development workflow includes:
|
||||
3. **Branch management** - Claude can create feature branches for implementations
|
||||
4. **Documentation maintenance** - Claude can keep documentation updated as the code evolves
|
||||
5. **Code Commits**: ALWAYS sign off commits with `git commit -s`
|
||||
6. **Pull Request Titles**: PR titles must follow the semantic format enforced by `.github/workflows/pr-title.yml`: `type(scope): summary`
|
||||
- Allowed types: `feat`, `fix`, `chore`, `docs`, `style`, `refactor`, `perf`, `test`, `build`, `ci`
|
||||
- Allowed scopes: `core`, `cli`, `api`, `mcp`, `sync`, `ui`, `deps`, `installer`
|
||||
- Example: `fix(cli): propagate cloud workspace routing`
|
||||
|
||||
This level of integration represents a new paradigm in AI-human collaboration, where the AI assistant becomes a full-fledged team member rather than just a tool for generating code snippets.
|
||||
|
||||
+10
-339
@@ -1,343 +1,14 @@
|
||||
# CHANGELOG
|
||||
|
||||
## v0.21.1 (2026-05-16)
|
||||
|
||||
CI-only release. No user-facing changes.
|
||||
|
||||
### Maintenance
|
||||
|
||||
- **#833**: Replace `mislav/bump-homebrew-formula-action` with an inline
|
||||
bash bump step. The action's `resolveRedirect` HEAD-requests
|
||||
`api.github.com /repos/.../tarball/<ref>` expecting HTTP 302; GitHub now
|
||||
returns 303 for authenticated requests on that endpoint, which broke
|
||||
the v0.21.0 Homebrew job and required a manual tap bump. The inline
|
||||
step does the same work (curl + sha256sum, clone tap, sed-bump
|
||||
`url`/`sha256`, commit, push) without any third-party dependency.
|
||||
|
||||
## v0.21.0 (2026-05-16)
|
||||
|
||||
Workspace-aware everywhere: every MCP tool and CLI command now routes through the
|
||||
same workspace/project model, the search and sync paths are noticeably faster,
|
||||
and a handful of long-standing parsing and routing footguns are gone.
|
||||
|
||||
### Breaking Changes
|
||||
|
||||
- Relation parsing no longer treats unquoted multi-word text before a wikilink as a
|
||||
custom relation type. Use single-token relation types like `relates_to [[Target]]`,
|
||||
or quote multi-word relation types like `"relates to" [[Target]]` or
|
||||
`'relates to' [[Target]]`.
|
||||
- Bare list wikilinks like `- [[Target]]` now index as `links_to`.
|
||||
- Prose list items like `- some other thing [[Target]]` now index as `links_to`.
|
||||
- To preserve existing multi-word relation types on re-sync, quote them before upgrading.
|
||||
- See **#824**.
|
||||
## Unreleased
|
||||
|
||||
### Features
|
||||
|
||||
- **#816**: `bm orphan` CLI command surfaces entities whose underlying markdown
|
||||
files are gone, with a flag to clean them out.
|
||||
- **#789**: Create projects directly by cloud workspace slug from MCP
|
||||
(`create_memory_project(workspace=...)`).
|
||||
- **#757**: Discover projects across every accessible cloud workspace in MCP's
|
||||
project list — no more per-workspace blind spots.
|
||||
- **#766**: MCP tools accept training-data-friendly parameter aliases
|
||||
(`q`/`search`/`text` for `query`, etc.) so models reach for them naturally.
|
||||
- **#776**: `bm db reset` refuses to run while a `basic-memory mcp` process is
|
||||
alive, so resets can no longer corrupt an open session.
|
||||
- **#791**: Search responses include result totals so pagers can stop guessing.
|
||||
- **#719**: Cloud `note_content` tenant schema primitive lands on the backend.
|
||||
- **#715**: `bm project add` accepts a `--visibility` flag for cloud projects.
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
#### Search and recent activity
|
||||
- **#832**: SQLite project deletion now sweeps `search_index`,
|
||||
`search_vector_chunks`, and `search_vector_embeddings`, so a project that
|
||||
reuses a recycled auto-increment id can't inherit the previous tenant's content.
|
||||
- **#812**: `recent_activity` orders and filters by `updated_at`, so edits bubble
|
||||
to the top instead of staying pinned to creation time.
|
||||
- **#807**: Multi-project `search_notes` is opt-in (`search_all_projects=True`);
|
||||
default search stays scoped to the resolved project.
|
||||
- **#785**: `recent_activity` caps responses and emits an explicit truncation
|
||||
footer instead of silently dropping rows.
|
||||
- **#713**: Eliminated an N+1 query in search result hydration.
|
||||
|
||||
#### Workspace / project routing
|
||||
- **#822**: `bm project list` now includes projects from every workspace, not
|
||||
just the current one.
|
||||
- **#813**, **#808**, **#806**, **#803**, **#801**, **#795**, **#790**, **#778**,
|
||||
**#777**, **#722**, **#712**, **#704**: Workspace-qualified permalink routing
|
||||
is centralized and applied consistently across `edit_note`, `delete_project`,
|
||||
`build_context`, `memory://` URLs, factory-mode project listing, cloud uploads,
|
||||
and the API client.
|
||||
|
||||
#### Sync
|
||||
- **#827**: `rclone bisync` filters from `.bmignore` are preserved across syncs.
|
||||
- **#815**: Watch service ignores hidden paths relative to the watched project,
|
||||
not just relative to `$HOME`.
|
||||
- **#814**: `scan` subprocesses no longer go through the shell, avoiding quoting
|
||||
issues with paths that contain special characters.
|
||||
- **#759**: Watch service stays inside `--project` scope.
|
||||
- **#746**: Canonical markdown is preserved during single-file sync.
|
||||
|
||||
#### Parsing
|
||||
- **#796**: Picoschema modifier descriptions (`field?(modifier, description)`)
|
||||
parse correctly.
|
||||
- **#769**: Obsidian callout blocks are skipped by the observation parser
|
||||
instead of being mis-extracted as observations.
|
||||
|
||||
#### CLI
|
||||
- **#775**: `bm project set-cloud` / `set-local` cleans up local DB state for
|
||||
the affected project.
|
||||
- **#773**: `bm cloud logout` clears `default_workspace`.
|
||||
- **#780**: `bm cloud setup` hint points at `bm cloud sync-setup`.
|
||||
- **#734**: `bm project info` shows cloud index freshness.
|
||||
- **#718**: Private cloud projects display under their `display_name` instead of
|
||||
raw UUID.
|
||||
- **#768**: `read_note` / `view_note` drop no-op pagination params from their
|
||||
signatures.
|
||||
|
||||
#### Stability
|
||||
- **#774**: `sqlite-vec` failures during init degrade gracefully to keyword-only
|
||||
search instead of crashing startup.
|
||||
- **#733**: Delete-vector and cloud-sync cleanup is now consistent.
|
||||
- **#702**: Race conditions in concurrent `delete_entity` are resolved.
|
||||
- **#724**: `external_id` is preserved when entities are re-upserted during a
|
||||
re-index.
|
||||
- **#728**: Vector init no longer issues runtime `ALTER TABLE`.
|
||||
- **#744**: `BASIC_MEMORY_CONFIG_DIR` is honored across remaining call sites.
|
||||
- **#743**: FastEmbed cache lives under the data dir instead of `/tmp`.
|
||||
- **#752**: Cloud projects report `source=cloud` in factory mode.
|
||||
- Stripped null bytes from markdown content before DB insert.
|
||||
- Allowed long `relation_type` values in API responses.
|
||||
|
||||
#### Installer
|
||||
- **#772**: Docker-compose config volume mounts under the `appuser` home.
|
||||
- **#695**: Bumped `brew outdated` timeout from 15s to 60s.
|
||||
|
||||
### Performance
|
||||
|
||||
- **#828**: CLI startup no longer pulls in the local ASGI FastAPI app when it
|
||||
isn't needed.
|
||||
- **#751**, **#726**, **#717**, **#714**, single-file/batch indexing: marked
|
||||
speedups on the sync hot path; unchanged markdown is skipped entirely.
|
||||
- **#731**, **#723**: Vector sync is faster on both backends, with tuned
|
||||
fastembed defaults.
|
||||
|
||||
### Maintenance
|
||||
|
||||
- **#825**: Dependency refresh + security hardening.
|
||||
- Updated to `fastmcp` 3.3.1.
|
||||
- **#754**: Removed in-house telemetry wrappers in favor of direct `logfire`
|
||||
usage.
|
||||
- **#736**: `ty` is now the default typechecker.
|
||||
- **#771**, **#770**, **#716**: New regression guards for vector-row cleanup,
|
||||
long relation types, and recent activity hydration.
|
||||
|
||||
## v0.20.3 (2026-03-26)
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- **#698**: CLI cloud commands now use API key when configured
|
||||
- `get_authenticated_headers()` only checked OAuth tokens, ignoring `config.cloud_api_key`
|
||||
- All CLI cloud commands (`upload`, `status`, `snapshot`, `restore`, etc.) failed for API-key-only users while MCP tools worked fine
|
||||
- Now mirrors the same credential priority as MCP: API key first, OAuth fallback
|
||||
- Fixes `bm cloud upload --project` returning "project does not exist" when authenticated with `bmc_*` API key
|
||||
|
||||
## v0.20.2 (2026-03-10)
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Fix auto-update Homebrew detection: `brew outdated` exits 1 when a formula is outdated, not on error
|
||||
- Previously treated exit code 1 as a failure, causing "Automatic update check failed" instead of detecting the available update
|
||||
|
||||
## v0.20.1 (2026-03-10)
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- **#661**: Fix `bm project list` MCP column to show transport type (stdio/https) instead of DB presence
|
||||
- Renamed "MCP (stdio)" column to "MCP"
|
||||
- Shows actual routing mode: `stdio` for local, `https` for cloud projects
|
||||
- Clears local path display for cloud-mode projects
|
||||
- **#662**: Invalidate config cache when file is modified by another process
|
||||
- Adds mtime-based cache validation to `ConfigManager.load_config()`
|
||||
- Long-lived processes (MCP stdio server) now detect external config changes
|
||||
- Fixes `bm project set-cloud` having no effect on running MCP server
|
||||
|
||||
## v0.20.0 (2026-03-10)
|
||||
|
||||
### Features
|
||||
|
||||
- **#643**: Default-on auto-update system and `bm update` command
|
||||
- Automatic background update checks for CLI installs (uv tool, Homebrew)
|
||||
- Install-source detection (homebrew, uv_tool, uvx, unknown) with uvx skip behavior
|
||||
- Periodic check gating via `auto_update_last_checked_at` + `update_check_interval` config
|
||||
- Manager-specific update flows: Homebrew (`brew upgrade`) and uv tool (`uv tool upgrade`)
|
||||
- Silent, non-blocking MCP behavior via daemon thread before server run
|
||||
- Manual commands: `bm update` (force check + apply) and `bm update --check` (check only)
|
||||
- New config fields: `auto_update`, `update_check_interval`, `auto_update_last_checked_at`
|
||||
|
||||
## v0.19.2 (2026-03-09)
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- **#657**: Coerce string params to list/dict in MCP tools
|
||||
- MCP clients that serialize `list`/`dict` arguments as JSON strings no longer fail Pydantic validation
|
||||
- Adds `BeforeValidator` coercion to `search_notes` (`entity_types`, `note_types`, `tags`, `metadata_filters`), `write_note` (`metadata`), and `canvas` (`nodes`, `edges`)
|
||||
- **#655**: Handle SQLite and Windows semantic search regressions
|
||||
- Fix embedding status query for non-semantic SQLite databases
|
||||
- Windows-safe log file rotation with per-process log filenames
|
||||
- Robust `setup_logging` that handles all environments cleanly
|
||||
|
||||
## v0.19.1 (2026-03-08)
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- **#649**: Enforce strict entity resolution in destructive MCP tools (`edit_note`, `move_note`, `delete_note`)
|
||||
- Prevents fuzzy-match fallback from silently editing/moving/deleting the wrong note
|
||||
- DST-related timeframe validation fix (round instead of truncate days)
|
||||
|
||||
### Features
|
||||
|
||||
- **#648**: Add `insert_before_section` and `insert_after_section` edit operations
|
||||
- Add `GET /knowledge/graph` endpoint for full graph visualization
|
||||
|
||||
### Dependencies
|
||||
|
||||
- Bump authlib from 1.6.6 to 1.6.7
|
||||
|
||||
## v0.19.0 (2026-03-07)
|
||||
|
||||
### Highlights
|
||||
|
||||
- **Semantic vector search** for SQLite and Postgres with FastEmbed embeddings
|
||||
- **Schema system** for validating and inferring knowledge base structure
|
||||
- **Per-project cloud routing** with API key authentication
|
||||
- **Upgraded to FastMCP 3.0** with tool annotations
|
||||
- **CLI overhaul** with JSON output, workspace awareness, and project dashboard
|
||||
|
||||
### Features
|
||||
|
||||
- **#550**: Add semantic vector search for SQLite and Postgres
|
||||
- FastEmbed-based embeddings with automatic backfill
|
||||
- Hybrid search combining full-text and vector similarity
|
||||
- Score-based fusion replacing RRF for better ranking
|
||||
- `min_similarity` override for tuning search precision
|
||||
- Semantic dependencies are now default, with optional extras fallback
|
||||
|
||||
- **#549**: Schema system for Basic Memory
|
||||
- `schema_infer` — infer schema from existing notes
|
||||
- `schema_validate` — validate notes against a schema definition
|
||||
- `schema_diff` — compare schemas across projects
|
||||
- Frontmatter validation support (#597)
|
||||
- Read schema definitions from file instead of stale DB metadata (#635)
|
||||
|
||||
- **#555**: Per-project local/cloud routing with API key auth
|
||||
- Individual projects route through cloud while others stay local
|
||||
- `basic-memory cloud set-key` and `basic-memory project set-cloud/set-local`
|
||||
- Stdio MCP honors per-project cloud routing (#590)
|
||||
|
||||
- **#598**: Upgrade FastMCP 2.12.3 to 3.0.1 with tool annotations
|
||||
|
||||
- **#585**: Add JSON output mode for MCP tools (default text)
|
||||
- `--json` output for CLI commands for scripting and CI
|
||||
|
||||
- **#576**: Add workspace selection flow for MCP and CLI
|
||||
- Workspace-aware cloud project listing
|
||||
- CLI refactoring for workspace support
|
||||
|
||||
- **#544**: Project-prefixed permalinks and memory URL routing
|
||||
|
||||
- **#632**: Add overwrite guard to `write_note` tool
|
||||
|
||||
- **#614**: `edit_note` append/prepend auto-creates note if not found
|
||||
|
||||
- **#609**: Richer content context in search results
|
||||
- Return matched chunk text in search results (#601)
|
||||
- Improved content hit rate
|
||||
|
||||
- **#602**: Add `created_by` and `last_updated_by` user tracking to Entity
|
||||
|
||||
- **#600**: Rename `entity_type` to `note_type` across codebase
|
||||
|
||||
- **#574**: Add `display_name` and `is_private` to ProjectItem
|
||||
|
||||
- **#569**: Expose `external_id` in EntityResponse and link resolver
|
||||
|
||||
- **#567**: Isolate default SQLite DB by config dir
|
||||
|
||||
- **#560**: Enable `default_project_mode` by default
|
||||
|
||||
- **#559**: Add `basic-memory watch` CLI command
|
||||
|
||||
- **#546**: Add cloud discovery touchpoints to CLI and MCP
|
||||
|
||||
- **#572**: CLI analytics via Umami event collector
|
||||
|
||||
- Replace project info with htop-inspired dashboard
|
||||
|
||||
- Merge `search_by_metadata` into `search_notes` with optional query
|
||||
|
||||
- Add `--strip-frontmatter` to `basic-memory tool read-note`
|
||||
|
||||
- Add `destination_folder` parameter to `move_note` tool
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- **#644**: Fix default project resolution in cloud mode
|
||||
- ChatGPT search/fetch tools broken in cloud mode
|
||||
- `resolve_project_parameter` falls back to projects API
|
||||
|
||||
- **#638**: Restore API backward compatibility for v0.18.x clients
|
||||
|
||||
- **#637**: Create backup before config migration overwrites old format
|
||||
|
||||
- **#636**: `list_workspaces` bypasses factory pattern on cloud MCP server
|
||||
|
||||
- **#631**: `build_context` related_results schema validation failure
|
||||
|
||||
- **#613**: Reduce excessive log volume by demoting per-request noise to DEBUG
|
||||
|
||||
- **#612**: Handle quoted picoschema enum strings in YAML frontmatter
|
||||
|
||||
- **#607**: Guard against closed streams in promo and missing vector tables
|
||||
|
||||
- **#606**: Accept null for `expected_replacements` in `edit_note`
|
||||
|
||||
- **#595**: `recent_activity` dedup and pagination across MCP tools
|
||||
|
||||
- **#593**: Backend-specific distance-to-similarity conversion
|
||||
|
||||
- **#582**: Use LinkResolver fallback in `build_context` for flexible identifier matching
|
||||
|
||||
- **#577**: Replace RRF with score-based fusion in hybrid search
|
||||
|
||||
- **#575**: Remove hardcoded "main" default from `default_project`
|
||||
|
||||
- **#534**: Speed up `bm --version` startup
|
||||
|
||||
- Fix semantic embeddings not generated on fresh DB or upgrade
|
||||
|
||||
- Clarify `search_notes` parameter naming and fix `note_types` case sensitivity
|
||||
|
||||
- Parse `tag:` prefix at MCP tool level to avoid hybrid search failure
|
||||
|
||||
- Cap sqlite-vec knn k parameter at 4096 limit
|
||||
|
||||
- Parameterize SQL queries in search repository type filters
|
||||
|
||||
- Coerce list frontmatter values to strings for title and type fields
|
||||
|
||||
- Avoid `Post(**metadata)` crash when frontmatter contains 'content' or 'handler' keys
|
||||
|
||||
- Upgrade cryptography and python-multipart for security advisories
|
||||
|
||||
### Internal
|
||||
|
||||
- **#594**: Add `ty` as supplemental type checker
|
||||
- Batched vector sync orchestration across repositories
|
||||
- FastEmbed parallel guardrails and provider caching
|
||||
- Improved cloud CLI status and error messages
|
||||
- CI coverage and Postgres test fixes
|
||||
- Default behavior is unchanged: `content` still includes raw markdown with frontmatter.
|
||||
- With `--strip-frontmatter`, both text and JSON modes return body-only markdown content.
|
||||
- JSON output now includes an additive `frontmatter` field with parsed YAML metadata (or `null`
|
||||
when no valid opening frontmatter block exists).
|
||||
|
||||
## v0.18.5 (2026-02-13)
|
||||
|
||||
@@ -2256,12 +1927,12 @@ Signed-off-by: phernandez <paul@basicmachines.co>
|
||||
- Update CLAUDE.md ([#33](https://github.com/basicmachines-co/basic-memory/pull/33),
|
||||
[`dfaf0fe`](https://github.com/basicmachines-co/basic-memory/commit/dfaf0fea9cf5b97d169d51a6276ec70162c21a7e))
|
||||
|
||||
fix spelling in CLAUDE.md: environment typo Signed-off-by: Ikko Eltociear Ashimine
|
||||
fix spelling in CLAUDE.md: enviroment -> environment Signed-off-by: Ikko Eltociear Ashimine
|
||||
<eltociear@gmail.com>
|
||||
|
||||
### Refactoring
|
||||
|
||||
- Move project stats into project subcommand
|
||||
- Move project stats into projct subcommand
|
||||
([`2a881b1`](https://github.com/basicmachines-co/basic-memory/commit/2a881b1425c73947f037fbe7ac5539c015b62526))
|
||||
|
||||
Signed-off-by: phernandez <paul@basicmachines.co>
|
||||
@@ -2690,7 +2361,7 @@ Co-authored-by: phernandez <phernandez@basicmachines.co>
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Refix virtual env in installer build
|
||||
- Refix vitual env in installer build
|
||||
([`052f491`](https://github.com/basicmachines-co/basic-memory/commit/052f491fff629e8ead629c9259f8cb46c608d584))
|
||||
|
||||
|
||||
@@ -2709,7 +2380,7 @@ Co-authored-by: phernandez <phernandez@basicmachines.co>
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Fix path to installer app artifact
|
||||
- Fix path to intaller app artifact
|
||||
([`53d220d`](https://github.com/basicmachines-co/basic-memory/commit/53d220df585561f9edd0d49a9e88f1d4055059cf))
|
||||
|
||||
|
||||
@@ -2717,7 +2388,7 @@ Co-authored-by: phernandez <phernandez@basicmachines.co>
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Activate virtualenv in installer build
|
||||
- Activate vitualenv in installer build
|
||||
([`d4c8293`](https://github.com/basicmachines-co/basic-memory/commit/d4c8293687a52eaf3337fe02e2f7b80e4cc9a1bb))
|
||||
|
||||
- Trigger installer build on release
|
||||
|
||||
@@ -224,6 +224,40 @@ See `test-int/BENCHMARKS.md` for detailed benchmark documentation.
|
||||
- **Fixtures**: Use async pytest fixtures for setup and teardown
|
||||
- **Markers**: Use `@pytest.mark.benchmark` for benchmarks, `@pytest.mark.slow` for slow tests
|
||||
|
||||
## Release Process
|
||||
|
||||
Basic Memory uses automatic versioning based on git tags with `uv-dynamic-versioning`. Here's how releases work:
|
||||
|
||||
### Version Management
|
||||
- **Development versions**: Automatically generated from git commits (e.g., `0.12.4.dev26+468a22f`)
|
||||
- **Beta releases**: Created by tagging with beta suffixes (e.g., `git tag v0.13.0b1`)
|
||||
- **Stable releases**: Created by tagging with version numbers (e.g., `git tag v0.13.0`)
|
||||
|
||||
### Release Workflows
|
||||
|
||||
#### Development Builds
|
||||
- Automatically published to PyPI on every commit to `main`
|
||||
- Version format: `0.12.4.dev26+468a22f` (base version + dev + commit count + hash)
|
||||
- Users install with: `pip install basic-memory --pre --force-reinstall`
|
||||
|
||||
#### Beta Releases
|
||||
1. Create and push a beta tag: `git tag v0.13.0b1 && git push origin v0.13.0b1`
|
||||
2. GitHub Actions automatically builds and publishes to PyPI
|
||||
3. Users install with: `pip install basic-memory --pre`
|
||||
|
||||
#### Stable Releases
|
||||
1. Create and push a version tag: `git tag v0.13.0 && git push origin v0.13.0`
|
||||
2. GitHub Actions automatically:
|
||||
- Builds the package with version `0.13.0`
|
||||
- Creates GitHub release with auto-generated notes
|
||||
- Publishes to PyPI
|
||||
3. Users install with: `pip install basic-memory`
|
||||
|
||||
### For Contributors
|
||||
- No manual version bumping required
|
||||
- Versions are automatically derived from git tags
|
||||
- Focus on code changes, not version management
|
||||
|
||||
## Creating Issues
|
||||
|
||||
If you're planning to work on something, please create an issue first to discuss the approach. Include:
|
||||
|
||||
+2
-67
@@ -8,71 +8,6 @@
|
||||
|
||||
## Reporting a Vulnerability
|
||||
|
||||
If you find a vulnerability, please contact hello@basicmachines.co.
|
||||
Use this section to tell people how to report a vulnerability.
|
||||
|
||||
Please do not open a public GitHub issue for security vulnerabilities. We aim
|
||||
to respond within 72 hours and will coordinate a fix and disclosure timeline
|
||||
with you.
|
||||
|
||||
## Threat Model
|
||||
|
||||
Basic Memory is a local-first MCP server that reads and writes markdown files
|
||||
inside configured project directories. It runs on your machine with your user
|
||||
permissions, so local configuration deserves the same care as any other
|
||||
developer tool that can access your files.
|
||||
|
||||
### What Basic Memory Controls
|
||||
|
||||
- Filesystem-touching tools validate paths against the configured project root
|
||||
with `validate_project_path()`, resolved paths, and `Path.is_relative_to()`.
|
||||
Path traversal attempts such as `../../etc/passwd` are blocked at this layer.
|
||||
- Scan optimizations in `sync_service.py` call `find` through
|
||||
`asyncio.create_subprocess_exec()` with explicit argument lists. Project paths
|
||||
are passed as data, not interpolated into shell strings.
|
||||
- Auto-update code uses hardcoded commands, list-form arguments, and
|
||||
`stdin=DEVNULL`. User-controlled strings do not reach a shell there.
|
||||
|
||||
### MCP Client-Side Risk
|
||||
|
||||
Recent MCP ecosystem research has highlighted a client-side pattern where an
|
||||
MCP host can be configured to run arbitrary commands as "servers." That risk is
|
||||
in the host configuration, not in notes or Basic Memory tool input.
|
||||
|
||||
The recommended Basic Memory MCP configuration uses a known command with
|
||||
explicit arguments:
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"basic-memory": {
|
||||
"command": "uvx",
|
||||
"args": ["basic-memory", "mcp"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Only add MCP server entries from sources you trust. Avoid inline shell scripts
|
||||
or command strings copied from untrusted sources. Treat third-party MCP server
|
||||
configuration with the same scrutiny as any locally executed program.
|
||||
|
||||
Related ecosystem context:
|
||||
|
||||
- OX Security: The Mother of All AI Supply Chains
|
||||
- CSO Online: RCE by design: MCP architectural choice haunts AI agent ecosystem
|
||||
|
||||
### Out Of Scope
|
||||
|
||||
- Basic Memory does not execute note content as code. Notes are returned as
|
||||
data to the LLM.
|
||||
- Basic Memory does not open network ports by default. The MCP server uses
|
||||
stdio; the optional REST API is intended for localhost use.
|
||||
- Basic Memory is designed for single-user local knowledge bases and does not
|
||||
implement access controls between operating-system users.
|
||||
|
||||
## Secure Configuration Checklist
|
||||
|
||||
- MCP config `command` points to `uvx` or a trusted binary, not a shell string.
|
||||
- Project paths in Basic Memory config come from trusted local configuration.
|
||||
- If exposing the REST API, bind it only to localhost.
|
||||
- Review any third-party MCP servers before adding them to your host config.
|
||||
If you find a vulnerability, please contact hello@basicmachines.co
|
||||
|
||||
@@ -1,8 +1,5 @@
|
||||
# Docker Compose configuration for Basic Memory with PostgreSQL
|
||||
# Use this for local development and testing with Postgres backend.
|
||||
#
|
||||
# The Postgres backend requires the pgvector extension (semantic search).
|
||||
# This image bundles pgvector; plain postgres:17 will not work for vector search.
|
||||
# Use this for local development and testing with Postgres backend
|
||||
#
|
||||
# Usage:
|
||||
# docker-compose -f docker-compose-postgres.yml up -d
|
||||
@@ -10,7 +7,7 @@
|
||||
|
||||
services:
|
||||
postgres:
|
||||
image: pgvector/pgvector:pg17
|
||||
image: postgres:17
|
||||
container_name: basic-memory-postgres
|
||||
environment:
|
||||
# Local development/test credentials - NOT for production
|
||||
|
||||
+1
-3
@@ -17,9 +17,7 @@ services:
|
||||
volumes:
|
||||
|
||||
# Persistent storage for configuration and database
|
||||
# Container runs as `appuser` (Dockerfile USER directive), so the CLI
|
||||
# config dir lives under /home/appuser, not /root.
|
||||
- basic-memory-config:/home/appuser/.basic-memory:rw
|
||||
- basic-memory-config:/root/.basic-memory:rw
|
||||
|
||||
# Mount your knowledge directory (required)
|
||||
# Change './knowledge' to your actual Obsidian vault or knowledge directory
|
||||
|
||||
+7
-25
@@ -106,7 +106,7 @@ The parser excludes these list item patterns:
|
||||
|---------|---------|--------|
|
||||
| Checkboxes | `- [ ] Todo item`, `- [x] Done`, `- [-] Cancelled` | Task list syntax |
|
||||
| Markdown links | `- [text](url)` | URL link syntax |
|
||||
| Bare wiki links | `- [[Target]]` | Treated as a `links_to` relation instead |
|
||||
| Bare wiki links | `- [[Target]]` | Treated as a relation instead |
|
||||
|
||||
A list item with `#tags` but no `[category]` is still parsed — the tags are extracted and the category defaults to `Note`.
|
||||
|
||||
@@ -116,65 +116,47 @@ Relations connect documents to form the knowledge graph. There are two kinds.
|
||||
|
||||
### Explicit Relations
|
||||
|
||||
Written as list items with a relation type and a `[[wiki link]]` target. Unquoted
|
||||
relation types are single tokens. Quote relation types that contain spaces.
|
||||
Written as list items with a relation type and a `[[wiki link]]` target.
|
||||
|
||||
**Syntax:**
|
||||
|
||||
```
|
||||
- relation_type [[Target Entity]] (context)
|
||||
- "multi word relation type" [[Target Entity]] (context)
|
||||
- 'multi word relation type' [[Target Entity]] (context)
|
||||
```
|
||||
|
||||
| Part | Required | Description |
|
||||
|------|----------|-------------|
|
||||
| `relation_type` | Yes | Single unquoted token before `[[`, or quoted text for multi-word labels. |
|
||||
| `relation_type` | No | Text before `[[`. Defaults to `relates_to` if omitted. |
|
||||
| `[[Target]]` | Yes | Wiki link to the target entity. Matched by title or permalink. |
|
||||
| `(context)` | No | Parenthesized text after `]]`. Supporting details. |
|
||||
|
||||
### Examples
|
||||
|
||||
Explicit relations:
|
||||
|
||||
```markdown
|
||||
- implements [[Search Design]]
|
||||
- depends_on [[Database Schema]]
|
||||
- works_at [[Y Combinator]] (co-founder)
|
||||
- "based on" [[Customer Interview]]
|
||||
- 'in response to' [[Incident Review]]
|
||||
```
|
||||
|
||||
Bare wiki links and prose list items create implicit `links_to` relations:
|
||||
|
||||
```markdown
|
||||
- [[Some Entity]]
|
||||
- some other thing [[Some Entity]]
|
||||
```
|
||||
|
||||
Both examples above create `links_to [[Some Entity]]`. Use quotes when the words before
|
||||
`[[` are meant to be a multi-word relation type.
|
||||
The last example — a bare `[[wiki link]]` in a list item — gets relation type `relates_to`.
|
||||
|
||||
Common relation types:
|
||||
- `implements`, `depends_on`, `relates_to`, `inspired_by`
|
||||
- `extends`, `part_of`, `contains`, `pairs_with`
|
||||
- `works_at`, `authored`, `collaborated_with`
|
||||
|
||||
Any single-token text or quoted text works as a relation type. These are conventions,
|
||||
not a fixed set.
|
||||
Any text works as a relation type. These are conventions, not a fixed set.
|
||||
|
||||
### Inline References
|
||||
|
||||
Wiki links appearing in regular prose create implicit `links_to` relations. This includes
|
||||
list items that do not match the explicit relation grammar above.
|
||||
Wiki links appearing in regular prose (not as list items) create implicit `links_to` relations.
|
||||
|
||||
```markdown
|
||||
This builds on [[Core Design]] and uses [[Utility Functions]].
|
||||
- We should revisit [[Search Design]] after the API changes.
|
||||
```
|
||||
|
||||
This creates three relations: `links_to [[Core Design]]`, `links_to [[Utility Functions]]`,
|
||||
and `links_to [[Search Design]]`.
|
||||
This creates two relations: `links_to [[Core Design]]` and `links_to [[Utility Functions]]`.
|
||||
|
||||
### Forward References
|
||||
|
||||
|
||||
@@ -91,11 +91,10 @@ SQLite Database (Index)
|
||||
# List all projects
|
||||
projects = await list_memory_projects()
|
||||
|
||||
# Response structure (each entry includes external_id you can pass as project_id):
|
||||
# Response structure:
|
||||
# [
|
||||
# {
|
||||
# "name": "main",
|
||||
# "external_id": "550e8400-e29b-41d4-a716-446655440000",
|
||||
# "path": "/Users/name/notes",
|
||||
# "is_default": True,
|
||||
# "note_count": 156,
|
||||
@@ -103,7 +102,6 @@ projects = await list_memory_projects()
|
||||
# },
|
||||
# {
|
||||
# "name": "work",
|
||||
# "external_id": "9f86d081-884c-42a3-b5e3-1c0c5b4c8e52",
|
||||
# "path": "/Users/name/work-notes",
|
||||
# "is_default": False,
|
||||
# "note_count": 89,
|
||||
@@ -166,44 +164,6 @@ active_project = "main"
|
||||
results = await search_notes(query="topic", project=active_project)
|
||||
```
|
||||
|
||||
### `project` vs `project_id`
|
||||
|
||||
Every project has two identifiers:
|
||||
|
||||
- **`project`** — human-readable name (e.g., `"main"`). Easy to use, but can collide across cloud workspaces.
|
||||
- **`project_id`** — stable `external_id` UUID. Always unambiguous; takes precedence over `project` when both are passed.
|
||||
|
||||
**When to prefer `project_id`:**
|
||||
|
||||
1. **Cloud multi-workspace setups.** If the user belongs to more than one workspace (personal + organization, or several organizations) and the same project name might exist in more than one of them, pass `project_id` to route to the exact project. Without it, name resolution falls back to the default workspace, which may not be the one the user means.
|
||||
2. **After `list_memory_projects()`.** Once you have the `external_id`, prefer using it — it's the same number of characters in JSON and saves a name-resolution round-trip.
|
||||
3. **When persisting a project choice across a long session.** UUIDs are stable; names can be renamed.
|
||||
|
||||
**When `project` (name) is fine:**
|
||||
|
||||
- Local single-workspace setups (no collision risk).
|
||||
- One-off operations where the name is clearly visible to the user (e.g., quick `search_notes(project="main", ...)`).
|
||||
- The user explicitly references a project by name in their message.
|
||||
|
||||
**Example — cloud multi-workspace pattern:**
|
||||
|
||||
```python
|
||||
# Discover and pick the right project for this user
|
||||
projects = await list_memory_projects()
|
||||
target = next(p for p in projects if p["name"] == "research" and p["workspace"]["slug"] == "acme")
|
||||
|
||||
# Use the UUID for all subsequent operations — no ambiguity
|
||||
await write_note(
|
||||
title="Meeting Notes",
|
||||
content="...",
|
||||
folder="meetings",
|
||||
project_id=target["external_id"],
|
||||
)
|
||||
results = await search_notes(query="kickoff", project_id=target["external_id"])
|
||||
```
|
||||
|
||||
**Precedence rule:** When both are passed, `project_id` wins. This lets you safely supply `project="main"` for backward compatibility while still routing precisely with `project_id`.
|
||||
|
||||
### Cross-Project Operations
|
||||
|
||||
**Some tools work across all projects when project parameter omitted:**
|
||||
|
||||
+24
-95
@@ -94,18 +94,13 @@ bm cloud setup
|
||||
```
|
||||
|
||||
**What this does:**
|
||||
1. Installs rclone with a supported package manager (if needed)
|
||||
1. Installs rclone automatically (if needed)
|
||||
2. Fetches your tenant information from cloud
|
||||
3. Generates scoped S3 credentials for sync
|
||||
4. Configures single rclone remote: `basic-memory-cloud`
|
||||
|
||||
**Result:** You're ready to sync projects. No sync directories created yet - those come with project setup.
|
||||
|
||||
Rclone setup uses package managers such as Homebrew, MacPorts, apt, dnf, yum, pacman,
|
||||
zypper, snap, winget, Chocolatey, or Scoop when available. It does not run remote
|
||||
install scripts with `sudo`; if no supported package manager is found, the CLI prints
|
||||
manual install instructions.
|
||||
|
||||
### 3. Add Projects with Sync
|
||||
|
||||
Create projects with optional local sync paths:
|
||||
@@ -118,7 +113,7 @@ bm project add research --cloud
|
||||
bm project add research --cloud --local-path ~/Documents/research
|
||||
|
||||
# Or configure sync for existing project
|
||||
bm cloud sync-setup research ~/Documents/research
|
||||
bm project sync-setup research ~/Documents/research
|
||||
```
|
||||
|
||||
**What happens under the covers:**
|
||||
@@ -241,7 +236,7 @@ bm project add research --cloud --local-path ~/Documents/research
|
||||
|
||||
```bash
|
||||
# Project already exists on cloud
|
||||
bm cloud sync-setup research ~/Documents/research
|
||||
bm project sync-setup research ~/Documents/research
|
||||
```
|
||||
|
||||
**What this does:**
|
||||
@@ -490,9 +485,6 @@ bm cloud create-key "my-laptop" # Creates key and saves it locally
|
||||
```
|
||||
|
||||
The API key is account-level — it grants access to all your cloud projects. It's stored in `~/.basic-memory/config.json` as `cloud_api_key`.
|
||||
On POSIX systems, Basic Memory writes `~/.basic-memory/` as user-private (`0700`) and
|
||||
`config.json` as user-read/write only (`0600`). Treat this config file as a credential
|
||||
file when an API key is saved.
|
||||
|
||||
### Setting Project Modes
|
||||
|
||||
@@ -582,62 +574,33 @@ bm cloud logout
|
||||
**Default patterns:**
|
||||
|
||||
```gitignore
|
||||
# Hidden files and directories
|
||||
.*
|
||||
|
||||
# Basic Memory internals
|
||||
*.db
|
||||
*.db-shm
|
||||
*.db-wal
|
||||
config.json
|
||||
|
||||
# Version control
|
||||
.git
|
||||
.svn
|
||||
.git/**
|
||||
|
||||
# Python
|
||||
__pycache__
|
||||
__pycache__/**
|
||||
*.pyc
|
||||
*.pyo
|
||||
*.pyd
|
||||
.pytest_cache
|
||||
.coverage
|
||||
*.egg-info
|
||||
.tox
|
||||
.mypy_cache
|
||||
.ruff_cache
|
||||
|
||||
# Virtual environments
|
||||
.venv
|
||||
venv
|
||||
env
|
||||
.env
|
||||
.venv/**
|
||||
venv/**
|
||||
|
||||
# Node.js
|
||||
node_modules
|
||||
node_modules/**
|
||||
|
||||
# Build artifacts
|
||||
build
|
||||
dist
|
||||
.cache
|
||||
|
||||
# IDE
|
||||
.idea
|
||||
.vscode
|
||||
# Basic Memory internals
|
||||
memory.db/**
|
||||
memory.db-shm/**
|
||||
memory.db-wal/**
|
||||
config.json/**
|
||||
watch-status.json/**
|
||||
.bmignore.rclone/**
|
||||
|
||||
# OS files
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
desktop.ini
|
||||
.DS_Store/**
|
||||
Thumbs.db/**
|
||||
|
||||
# Obsidian
|
||||
.obsidian
|
||||
|
||||
# Temporary files
|
||||
*.tmp
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
# Environment files
|
||||
.env/**
|
||||
.env.local/**
|
||||
```
|
||||
|
||||
**How it works:**
|
||||
@@ -646,11 +609,6 @@ desktop.ini
|
||||
3. Rclone uses filters during sync
|
||||
4. Same patterns used by all projects
|
||||
|
||||
During conversion, file patterns exclude the direct match and recursive contents.
|
||||
For example, `config.json` becomes both `- config.json` and `- config.json/**`,
|
||||
while `.*` becomes both `- .*` and `- .*/**`. Directory-only patterns keep
|
||||
their trailing slash, so `cache/` becomes `- cache/` and `- cache/**`.
|
||||
|
||||
**Customizing:**
|
||||
|
||||
```bash
|
||||
@@ -658,7 +616,7 @@ their trailing slash, so `cache/` becomes `- cache/` and `- cache/**`.
|
||||
code ~/.basic-memory/.bmignore
|
||||
|
||||
# Add custom patterns
|
||||
echo "*.tmp" >> ~/.basic-memory/.bmignore
|
||||
echo "*.tmp/**" >> ~/.basic-memory/.bmignore
|
||||
|
||||
# Next sync uses updated patterns
|
||||
bm project bisync --name research
|
||||
@@ -666,33 +624,6 @@ bm project bisync --name research
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Rclone Setup Cannot Install Automatically
|
||||
|
||||
**Problem:** `bm cloud setup` cannot find a supported package manager, or package-manager
|
||||
installation fails.
|
||||
|
||||
**Explanation:** The CLI avoids remote privileged install scripts. It only invokes known
|
||||
package managers and otherwise asks you to install rclone manually.
|
||||
|
||||
**Solution:** Install rclone with your OS package manager, then rerun setup:
|
||||
|
||||
```bash
|
||||
# macOS
|
||||
brew install rclone
|
||||
|
||||
# Debian/Ubuntu
|
||||
sudo apt install rclone
|
||||
|
||||
# Fedora
|
||||
sudo dnf install rclone
|
||||
|
||||
# Arch
|
||||
sudo pacman -S rclone
|
||||
|
||||
# After rclone is on PATH
|
||||
bm cloud setup
|
||||
```
|
||||
|
||||
### Authentication Issues
|
||||
|
||||
**Problem:** "Authentication failed" or "Invalid token"
|
||||
@@ -808,7 +739,7 @@ bm project sync --name research
|
||||
**Solution:**
|
||||
|
||||
```bash
|
||||
bm cloud sync-setup research ~/Documents/research
|
||||
bm project sync-setup research ~/Documents/research
|
||||
bm project bisync --name research --resync
|
||||
```
|
||||
|
||||
@@ -828,10 +759,8 @@ If instance is down, wait a few minutes and retry.
|
||||
|
||||
- **Authentication**: OAuth 2.1 with PKCE flow
|
||||
- **Tokens**: Stored securely in `~/.basic-memory/basic-memory-cloud.json`
|
||||
- **API keys**: Stored in `~/.basic-memory/config.json`, which is written with private file permissions on POSIX systems
|
||||
- **Transport**: All data encrypted in transit (HTTPS)
|
||||
- **Credentials**: Scoped S3 credentials (read-write to your tenant only)
|
||||
- **Rclone setup**: Uses package managers or manual instructions; no remote privileged install-script fallback
|
||||
- **Isolation**: Your data isolated from other tenants
|
||||
- **Ignore patterns**: Sensitive files automatically excluded via `.bmignore`
|
||||
|
||||
@@ -856,7 +785,7 @@ bm cloud create-key <name> # Create API key via cloud API (requires OAuth login
|
||||
### Setup
|
||||
|
||||
```bash
|
||||
bm cloud setup # Install rclone via package manager and configure credentials
|
||||
bm cloud setup # Install rclone and configure credentials
|
||||
```
|
||||
|
||||
### Project Management
|
||||
@@ -866,7 +795,7 @@ bm project list --local # Local project list
|
||||
bm project list --cloud # Cloud project list
|
||||
bm project add <name> --cloud # Create cloud project (no sync)
|
||||
bm project add <name> --cloud --local-path <path> # Create with local sync
|
||||
bm cloud sync-setup <name> <path> # Add sync to existing project
|
||||
bm project sync-setup <name> <path> # Add sync to existing project
|
||||
bm project rm <name> # Delete project
|
||||
```
|
||||
|
||||
|
||||
@@ -1,499 +0,0 @@
|
||||
# Logfire Instrumentation Strategy
|
||||
|
||||
## Why
|
||||
|
||||
We want Logfire in Basic Memory for two specific use cases:
|
||||
|
||||
1. Local development and performance investigation
|
||||
2. Cloud deployments where Basic Memory runs inside Basic Memory Cloud
|
||||
|
||||
This instrumentation must be:
|
||||
|
||||
- Disabled by default
|
||||
- Useful when enabled
|
||||
- Safe for local-first users
|
||||
- Searchable in Logfire over time
|
||||
|
||||
The previous integration added telemetry, but it leaned too much on generic framework instrumentation. That created noisy spans with weak names and made the trace view harder to navigate. This strategy favors manual instrumentation around Basic Memory's real units of work.
|
||||
|
||||
## Core Principles
|
||||
|
||||
### 1. Default-off
|
||||
|
||||
Basic Memory should ship with Logfire disabled unless the operator explicitly enables it.
|
||||
|
||||
That means:
|
||||
|
||||
- no required token for normal local usage
|
||||
- no surprise outbound telemetry
|
||||
- no behavior change for existing users
|
||||
|
||||
### 2. Manual spans over automatic framework spans
|
||||
|
||||
We should not rely on broad auto-instrumentation for FastAPI, MCP, SQLAlchemy, or HTTP as the primary experience.
|
||||
|
||||
Why:
|
||||
|
||||
- auto-generated span names are often generic
|
||||
- routes and middleware produce too many low-signal spans
|
||||
- it becomes harder to answer product questions like "why was `write_note` slow?" or "where did sync time go?"
|
||||
|
||||
The preferred model is:
|
||||
|
||||
- one meaningful root span per high-level operation
|
||||
- a small number of child spans for important phases
|
||||
- optional targeted instrumentation only where it adds clear value
|
||||
|
||||
### 3. Logs must live inside traces
|
||||
|
||||
Basic Memory already uses `loguru` pervasively. The Logfire integration should preserve that and make those logs visible inside the active trace/span context.
|
||||
|
||||
If traces exist but the logs are detached from them, the integration is not doing its job.
|
||||
|
||||
### 4. Stable names, selective attributes
|
||||
|
||||
Span names should describe the operation class, not the specific input.
|
||||
|
||||
Good:
|
||||
|
||||
- `mcp.tool.write_note`
|
||||
- `sync.project.scan`
|
||||
- `search.execute`
|
||||
- `routing.resolve_project`
|
||||
|
||||
Bad:
|
||||
|
||||
- `Searching for "foo bar baz"`
|
||||
- `POST /v2/projects/123/search/`
|
||||
- `write note to /specs/api.md`
|
||||
|
||||
Dynamic values belong in attributes, not in the span name.
|
||||
|
||||
## What We Should Not Do
|
||||
|
||||
### Avoid broad FastAPI auto-instrumentation
|
||||
|
||||
We should not turn on `instrument_fastapi()` and treat that as the main telemetry story.
|
||||
|
||||
It may still be useful in narrowly scoped debugging, but it should not define the production trace shape. The meaningful root spans should come from Basic Memory's own entrypoints and service boundaries.
|
||||
|
||||
### Avoid per-file spans by default
|
||||
|
||||
`sync` can process many files. A span per file will explode trace cardinality and make performance views noisy.
|
||||
|
||||
Default behavior should be:
|
||||
|
||||
- one span for the project sync
|
||||
- child spans for scan, move handling, delete handling, markdown sync batch, relation resolution, embedding sync, watermark update
|
||||
- per-file spans only for failures or very slow outliers
|
||||
|
||||
### Avoid high-cardinality attributes on every span
|
||||
|
||||
Do not attach large or highly variable values everywhere:
|
||||
|
||||
- raw note content
|
||||
- file bodies
|
||||
- long search text
|
||||
- arbitrary metadata blobs
|
||||
- unique IDs that make every span shape distinct
|
||||
|
||||
Prefer compact, queryable attributes:
|
||||
|
||||
- `project_name`
|
||||
- `workspace_id`
|
||||
- `route_mode`
|
||||
- `scan_type`
|
||||
- `file_count`
|
||||
- `result_count`
|
||||
- `search_type`
|
||||
- `retrieval_mode`
|
||||
- `duration_ms`
|
||||
|
||||
## Proposed Architecture
|
||||
|
||||
Add a dedicated telemetry module in core Basic Memory, separate from logging setup.
|
||||
|
||||
Suggested shape:
|
||||
|
||||
```python
|
||||
# basic_memory/telemetry.py
|
||||
|
||||
def configure_telemetry(service_name: str, *, enable_logfire: bool) -> None: ...
|
||||
def telemetry_enabled() -> bool: ...
|
||||
def span(name: str, **attrs): ...
|
||||
def bind_telemetry_context(**attrs): ...
|
||||
```
|
||||
|
||||
This module should:
|
||||
|
||||
- configure Logfire only when explicitly enabled
|
||||
- set up the Logfire `loguru` handler
|
||||
- expose lightweight helpers so application code does not import `logfire` directly everywhere
|
||||
- degrade cleanly to no-op behavior when disabled
|
||||
|
||||
This keeps the rest of the codebase readable and makes it easy to reason about what telemetry is doing.
|
||||
|
||||
## Logging Integration Strategy
|
||||
|
||||
### Goal
|
||||
|
||||
When a span is active, logs emitted through `loguru` during that operation should show up in the same trace.
|
||||
|
||||
### Preferred design
|
||||
|
||||
1. Configure Logfire once in the telemetry bootstrap
|
||||
2. Add the Logfire `loguru` handler to the existing `loguru` configuration
|
||||
3. At operation boundaries, bind stable contextual fields with `loguru`
|
||||
4. Let logs emitted inside the span inherit the active trace context
|
||||
|
||||
### Context to bind
|
||||
|
||||
Bind only the fields that help correlate work across the system:
|
||||
|
||||
- `service_name`
|
||||
- `entrypoint`
|
||||
- `project_name`
|
||||
- `workspace_id`
|
||||
- `route_mode`
|
||||
- `tool_name`
|
||||
- `command_name`
|
||||
|
||||
This binding should happen at the root of an operation, not deep in leaf functions.
|
||||
|
||||
### Important nuance
|
||||
|
||||
We should not try to encode the entire trace model into logger extras. The logger context should be a human-meaningful slice of the active operation. Trace linkage comes from the active Logfire/OpenTelemetry context; logger extras are there to improve searchability and readability.
|
||||
|
||||
## Span Model
|
||||
|
||||
### Root spans
|
||||
|
||||
Each user-visible or system-visible operation should get one root span.
|
||||
|
||||
Examples:
|
||||
|
||||
- `cli.command.status`
|
||||
- `cli.command.project_sync`
|
||||
- `api.request.search`
|
||||
- `mcp.tool.write_note`
|
||||
- `mcp.tool.read_note`
|
||||
- `mcp.tool.search_notes`
|
||||
- `sync.project.run`
|
||||
- `db.semantic_backfill`
|
||||
|
||||
### Child spans
|
||||
|
||||
Child spans should represent real phases whose duration we care about.
|
||||
|
||||
Examples:
|
||||
|
||||
- `routing.client_session`
|
||||
- `routing.resolve_project`
|
||||
- `routing.resolve_workspace`
|
||||
- `api.search.execute`
|
||||
- `sync.project.scan`
|
||||
- `sync.project.detect_moves`
|
||||
- `sync.project.apply_changes`
|
||||
- `sync.project.resolve_relations`
|
||||
- `sync.project.sync_embeddings`
|
||||
- `sync.file.markdown`
|
||||
- `sync.file.regular`
|
||||
- `search.execute`
|
||||
- `search.relaxed_fts_retry`
|
||||
- `db.init`
|
||||
- `db.migrate`
|
||||
|
||||
### Span naming rules
|
||||
|
||||
- Use dot-separated names
|
||||
- Start with subsystem
|
||||
- Keep the verb at the end
|
||||
- Keep names stable across runs
|
||||
- Never include request-specific text in the span name
|
||||
|
||||
## Attribute Taxonomy
|
||||
|
||||
### Required attributes on root spans
|
||||
|
||||
Every root span should have a small common set:
|
||||
|
||||
- `service_name`
|
||||
- `entrypoint`
|
||||
- `project_name` when applicable
|
||||
- `workspace_id` when applicable
|
||||
- `route_mode` with values like `local_asgi`, `cloud_proxy`, `factory`
|
||||
|
||||
### Operation-specific attributes
|
||||
|
||||
Examples:
|
||||
|
||||
For search:
|
||||
|
||||
- `search_type`
|
||||
- `retrieval_mode`
|
||||
- `page`
|
||||
- `page_size`
|
||||
- `result_count`
|
||||
- `fallback_used`
|
||||
|
||||
For sync:
|
||||
|
||||
- `scan_type`
|
||||
- `force_full`
|
||||
- `new_count`
|
||||
- `modified_count`
|
||||
- `deleted_count`
|
||||
- `move_count`
|
||||
- `skipped_count`
|
||||
- `embeddings_enabled`
|
||||
|
||||
For note operations:
|
||||
|
||||
- `tool_name`
|
||||
- `note_type`
|
||||
- `directory`
|
||||
- `overwrite`
|
||||
- `output_format`
|
||||
|
||||
### Attributes to avoid by default
|
||||
|
||||
- full `query.text`
|
||||
- full note titles if they create privacy or cardinality issues
|
||||
- file content
|
||||
- raw frontmatter
|
||||
- raw HTTP bodies
|
||||
|
||||
If we need richer payloads for a local debugging session, that should be an explicit temporary mode, not the default telemetry shape.
|
||||
|
||||
## Instrumentation Plan By Layer
|
||||
|
||||
### 1. Entrypoints
|
||||
|
||||
Instrument these first:
|
||||
|
||||
- `cli.app` callback and major commands
|
||||
- API lifespan and selected routers
|
||||
- MCP server lifespan
|
||||
- MCP tool entrypoints
|
||||
|
||||
Why:
|
||||
|
||||
- this establishes clean root spans
|
||||
- it gives us trace boundaries that match how users think about the product
|
||||
|
||||
### 2. Routing and context resolution
|
||||
|
||||
Instrument:
|
||||
|
||||
- client routing decisions
|
||||
- workspace resolution
|
||||
- project resolution
|
||||
- default-project fallback
|
||||
|
||||
Why:
|
||||
|
||||
- Basic Memory has local/cloud/per-project routing logic
|
||||
- when something is slow or surprising, we need to know which path was taken
|
||||
|
||||
### 3. Sync and indexing
|
||||
|
||||
This is the highest-value area to instrument deeply.
|
||||
|
||||
Instrument:
|
||||
|
||||
- sync root
|
||||
- scan strategy decision
|
||||
- filesystem scan
|
||||
- move detection
|
||||
- delete handling
|
||||
- markdown sync phase
|
||||
- relation resolution
|
||||
- vector embedding sync
|
||||
- scan watermark update
|
||||
|
||||
Why:
|
||||
|
||||
- this is where performance work will happen
|
||||
- cloud and local both benefit from this visibility
|
||||
|
||||
### 4. Search
|
||||
|
||||
Instrument:
|
||||
|
||||
- search execution
|
||||
- retrieval mode
|
||||
- relaxed FTS fallback
|
||||
- result shaping
|
||||
|
||||
Why:
|
||||
|
||||
- search is user-facing and latency-sensitive
|
||||
- hybrid/vector/FTS paths need to be distinguishable
|
||||
|
||||
### 5. Database and initialization
|
||||
|
||||
Instrument selectively:
|
||||
|
||||
- DB init
|
||||
- migrations
|
||||
- semantic backfill
|
||||
- connection mode selection
|
||||
|
||||
Avoid full automatic SQL span firehose by default.
|
||||
|
||||
## Recommended Rollout Phases
|
||||
|
||||
## Task List
|
||||
|
||||
- [x] Phase 1: Bootstrap and config gating
|
||||
- [x] Phase 2: Root spans for entrypoints and primary operations
|
||||
- [x] Phase 3: Child spans for sync, search, and routing
|
||||
- [x] Phase 4: Failure-focused detail and final verification
|
||||
- [x] Phase 5: Loguru context binding and scoped context inheritance
|
||||
|
||||
## Recommended Rollout Phases
|
||||
|
||||
### Phase 1: Bootstrap and config gating
|
||||
|
||||
Add:
|
||||
|
||||
- telemetry bootstrap module
|
||||
- config/env gating
|
||||
- `loguru` + Logfire handler integration
|
||||
|
||||
This gives immediate value with low noise.
|
||||
|
||||
### Phase 2: Root spans for entrypoints and primary operations
|
||||
|
||||
Add:
|
||||
|
||||
- root spans for CLI, API, MCP, and main MCP tools
|
||||
- stable root attributes for project, workspace, route mode, and operation type
|
||||
|
||||
This gives us clean top-level traces that match how users think about the product.
|
||||
|
||||
### Phase 3: Child spans for sync, search, and routing
|
||||
|
||||
Add child spans to:
|
||||
|
||||
- sync
|
||||
- search
|
||||
- routing
|
||||
|
||||
This is the main performance-investigation layer.
|
||||
|
||||
### Phase 4: Failure-focused detail
|
||||
|
||||
Add selective deeper spans/log enrichment for:
|
||||
|
||||
- sync failures
|
||||
- relation resolution failures
|
||||
- slow file operations
|
||||
- cloud routing/auth failures
|
||||
|
||||
This keeps normal traces clean while improving debuggability.
|
||||
|
||||
### Phase 5: Loguru context binding and scoped context inheritance
|
||||
|
||||
Add:
|
||||
|
||||
- context-local telemetry state in `basic_memory.telemetry`
|
||||
- a shared `scope(...)` helper that opens a span and binds stable logger context together
|
||||
- context inheritance for routing, sync, and search so downstream `loguru` logs carry the active operation fields
|
||||
|
||||
This makes the trace view and the log stream tell the same story without forcing logger rewrites across the codebase.
|
||||
|
||||
## Local Dev Playbook
|
||||
|
||||
The fastest way to sanity-check the current trace shape is:
|
||||
|
||||
```bash
|
||||
LOGFIRE_TOKEN=lf_... just telemetry-smoke
|
||||
```
|
||||
|
||||
What this does:
|
||||
|
||||
- creates an isolated temp home, config dir, and project path
|
||||
- enables Logfire for the run
|
||||
- automatically exports to Logfire when `LOGFIRE_TOKEN` is present
|
||||
- defaults `BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED=false` so the smoke run stays fast and trace-friendly
|
||||
- disables promo telemetry so the trace is about Basic Memory work, not analytics noise
|
||||
- runs a small CLI workflow:
|
||||
- `project add`
|
||||
- `tool write-note`
|
||||
- `tool read-note`
|
||||
- `tool edit-note`
|
||||
- `tool build-context`
|
||||
- `tool search-notes`
|
||||
- `doctor`
|
||||
|
||||
If you want to exercise the instrumentation without exporting anything upstream:
|
||||
|
||||
```bash
|
||||
BASIC_MEMORY_LOGFIRE_SEND_TO_LOGFIRE=false just telemetry-smoke
|
||||
```
|
||||
|
||||
If you want the smoke run to include vector or hybrid retrieval spans too:
|
||||
|
||||
```bash
|
||||
LOGFIRE_TOKEN=lf_... BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED=true just telemetry-smoke
|
||||
```
|
||||
|
||||
The recipe sets `BASIC_MEMORY_LOGFIRE_ENVIRONMENT=telemetry-smoke` by default so these traces are easy to isolate in Logfire. Override it if you want the smoke traces grouped under a different environment name.
|
||||
|
||||
### What to look for
|
||||
|
||||
You should see a small set of comparable root spans rather than a framework-generated span forest:
|
||||
|
||||
- `cli.command.project`
|
||||
- `cli.command.tool`
|
||||
- `mcp.tool.write_note`
|
||||
- `mcp.tool.read_note`
|
||||
- `mcp.tool.edit_note`
|
||||
- `mcp.tool.build_context`
|
||||
- `mcp.tool.search_notes`
|
||||
- `sync.project.run`
|
||||
|
||||
You should also see correlated logs under those traces with stable fields like:
|
||||
|
||||
- `project_name`
|
||||
- `route_mode`
|
||||
- `tool_name`
|
||||
- `entrypoint`
|
||||
|
||||
### Expected nuance
|
||||
|
||||
`doctor` creates its own temporary project on purpose. That means the sync trace will usually show a different project name than the `telemetry-smoke` write/search traces. That is fine for smoke testing because the goal is to confirm:
|
||||
|
||||
- root span names are meaningful
|
||||
- scoped logs stay attached to the active trace
|
||||
- routing, tool, search, and sync phases are easy to distinguish
|
||||
|
||||
## Validation Checklist
|
||||
|
||||
We should consider the integration successful when the following are true:
|
||||
|
||||
1. With telemetry disabled, Basic Memory behaves exactly as it does today.
|
||||
2. With telemetry enabled, one user action produces one obvious root span.
|
||||
3. Logs emitted during that action are visible inside the same trace.
|
||||
4. A search in Logfire for `mcp.tool.write_note` or `sync.project.run` returns comparable spans across runs.
|
||||
5. Trace views show phase timing clearly without drowning in framework noise.
|
||||
6. Sensitive payloads are not captured by default.
|
||||
|
||||
## Immediate Implementation Direction
|
||||
|
||||
When we start coding, the first pass should be:
|
||||
|
||||
1. Add `basic_memory.telemetry`
|
||||
2. Add config/env switches for `enabled`, `send_to_logfire`, and service name
|
||||
3. Wire telemetry bootstrap into CLI, API, and MCP entrypoints
|
||||
4. Configure `loguru` to emit to both existing sinks and the Logfire handler when enabled
|
||||
5. Add manual root spans around:
|
||||
- CLI commands
|
||||
- API request handlers we care about
|
||||
- MCP tool entrypoints
|
||||
- sync root
|
||||
- search root
|
||||
6. Add child spans to the sync and routing phases only after the root span model feels clean
|
||||
|
||||
That gives us a strong foundation without repeating the earlier "turn on instrumentation everywhere" approach.
|
||||
@@ -263,7 +263,6 @@ The sqlite-vec extension is loaded per-connection. Vector tables are created laz
|
||||
### Postgres (cloud)
|
||||
|
||||
- **Vector storage**: [pgvector](https://github.com/pgvector/pgvector) with HNSW indexing
|
||||
- **Local Docker**: use `docker-compose-postgres.yml` (`pgvector/pgvector:pg17`). Plain `postgres:17` lacks the extension; run `CREATE EXTENSION IF NOT EXISTS vector;` on any external instance before first migration.
|
||||
- **Chunk metadata table**: Created via Alembic migration (`search_vector_chunks` with `BIGSERIAL` primary key)
|
||||
- **Embedding table**: `search_vector_embeddings` created at runtime (dimension-dependent, same pattern as SQLite)
|
||||
- **Index**: HNSW index on the embedding column for fast approximate nearest-neighbour queries
|
||||
|
||||
@@ -62,20 +62,38 @@ test-int-postgres:
|
||||
fi
|
||||
|
||||
# Run tests impacted by recent changes (requires pytest-testmon)
|
||||
# Pass paths or node ids after `just testmon` to limit the candidate set further.
|
||||
testmon *args:
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov --testmon {{args}}
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov --testmon --testmon-forceselect {{args}}
|
||||
|
||||
# Run MCP smoke test (fast end-to-end loop)
|
||||
test-smoke:
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov -m smoke test-int/mcp/test_smoke_integration.py
|
||||
|
||||
# Fast local loop: lint, format, typecheck, impacted tests via pytest-testmon
|
||||
# Run graph intelligence API contract tests only
|
||||
test-graph-intel-api:
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov tests/api/v2/test_graph_intelligence_router.py
|
||||
|
||||
# Run graph intelligence MCP tests only
|
||||
test-graph-intel-mcp:
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov tests/mcp/clients/test_graph_clients.py tests/mcp/test_tool_graph_intelligence.py tests/mcp/test_tool_contracts.py
|
||||
|
||||
# Run graph intelligence CLI passthrough tests only
|
||||
test-graph-intel-cli:
|
||||
BASIC_MEMORY_ENV=test uv run pytest -p pytest_mock -v --no-cov tests/cli/test_cli_tool_graph_intelligence_json_output.py
|
||||
|
||||
# Run the full graph intelligence fast iteration slice
|
||||
test-graph-intel:
|
||||
just test-graph-intel-api
|
||||
just test-graph-intel-mcp
|
||||
just test-graph-intel-cli
|
||||
|
||||
# Fast local loop: lint, format, typecheck, impacted tests
|
||||
fast-check:
|
||||
just fix
|
||||
just format
|
||||
just typecheck
|
||||
just testmon
|
||||
just test-smoke
|
||||
|
||||
# Reset Postgres test database (drops and recreates schema)
|
||||
# Useful when Alembic migration state gets out of sync during development
|
||||
@@ -170,17 +188,13 @@ lint: fix
|
||||
fix:
|
||||
uv run ruff check --fix --unsafe-fixes src tests test-int
|
||||
|
||||
# Type check code (ty)
|
||||
typecheck:
|
||||
uv run ty check src tests test-int
|
||||
|
||||
# Type check code (pyright)
|
||||
typecheck-pyright:
|
||||
typecheck:
|
||||
uv run pyright
|
||||
|
||||
# Type check code (ty)
|
||||
typecheck-ty:
|
||||
just typecheck
|
||||
uv run ty check src/
|
||||
|
||||
# Clean build artifacts and cache files
|
||||
clean:
|
||||
@@ -209,51 +223,6 @@ doctor:
|
||||
BASIC_MEMORY_CONFIG_DIR="$TMP_CONFIG" \
|
||||
./.venv/bin/python -m basic_memory.cli.main doctor --local
|
||||
|
||||
# Run an isolated Logfire smoke workflow for local trace inspection
|
||||
telemetry-smoke:
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
TMP_HOME=$(mktemp -d)
|
||||
TMP_CONFIG=$(mktemp -d)
|
||||
TMP_PROJECT=$(mktemp -d)
|
||||
export HOME="$TMP_HOME"
|
||||
export BASIC_MEMORY_ENV="${BASIC_MEMORY_ENV:-dev}"
|
||||
export BASIC_MEMORY_HOME="$TMP_PROJECT/home-root"
|
||||
export BASIC_MEMORY_CONFIG_DIR="$TMP_CONFIG"
|
||||
export BASIC_MEMORY_NO_PROMOS=1
|
||||
export BASIC_MEMORY_LOG_LEVEL="${BASIC_MEMORY_LOG_LEVEL:-INFO}"
|
||||
export BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED="${BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED:-false}"
|
||||
export BASIC_MEMORY_LOGFIRE_ENABLED="${BASIC_MEMORY_LOGFIRE_ENABLED:-true}"
|
||||
export BASIC_MEMORY_LOGFIRE_ENVIRONMENT="${BASIC_MEMORY_LOGFIRE_ENVIRONMENT:-telemetry-smoke}"
|
||||
if [[ -z "${BASIC_MEMORY_LOGFIRE_SEND_TO_LOGFIRE:-}" ]]; then
|
||||
if [[ -n "${LOGFIRE_TOKEN:-}" ]]; then
|
||||
export BASIC_MEMORY_LOGFIRE_SEND_TO_LOGFIRE=true
|
||||
else
|
||||
export BASIC_MEMORY_LOGFIRE_SEND_TO_LOGFIRE=false
|
||||
fi
|
||||
fi
|
||||
mkdir -p "$BASIC_MEMORY_HOME"
|
||||
echo "Telemetry smoke setup:"
|
||||
echo " logfire_enabled=$BASIC_MEMORY_LOGFIRE_ENABLED"
|
||||
echo " send_to_logfire=$BASIC_MEMORY_LOGFIRE_SEND_TO_LOGFIRE"
|
||||
echo " log_level=$BASIC_MEMORY_LOG_LEVEL"
|
||||
echo " semantic_search_enabled=$BASIC_MEMORY_SEMANTIC_SEARCH_ENABLED"
|
||||
echo " logfire_environment=$BASIC_MEMORY_LOGFIRE_ENVIRONMENT"
|
||||
echo " project_path=$TMP_PROJECT"
|
||||
./.venv/bin/python -m basic_memory.cli.main project add telemetry-smoke "$TMP_PROJECT" --default --local
|
||||
./.venv/bin/python -m basic_memory.cli.main tool write-note --title "Telemetry Smoke" --folder notes --content "hello from smoke" --project telemetry-smoke --local
|
||||
./.venv/bin/python -m basic_memory.cli.main tool read-note notes/telemetry-smoke --project telemetry-smoke --local
|
||||
./.venv/bin/python -m basic_memory.cli.main tool edit-note notes/telemetry-smoke --operation append --content $'\n\nsmoke edit line' --project telemetry-smoke --local
|
||||
./.venv/bin/python -m basic_memory.cli.main tool build-context notes/telemetry-smoke --project telemetry-smoke --local --page-size 5 --max-related 5
|
||||
./.venv/bin/python -m basic_memory.cli.main tool search-notes telemetry --project telemetry-smoke --local
|
||||
./.venv/bin/python -m basic_memory.cli.main doctor --local
|
||||
echo ""
|
||||
echo "Telemetry smoke complete."
|
||||
echo "Search Logfire for:"
|
||||
echo " service_name: basic-memory-cli"
|
||||
echo " environment: $BASIC_MEMORY_LOGFIRE_ENVIRONMENT"
|
||||
echo " span names: mcp.tool.write_note, mcp.tool.read_note, mcp.tool.edit_note, mcp.tool.build_context, mcp.tool.search_notes, sync.project.run"
|
||||
|
||||
|
||||
# Update all dependencies to latest versions
|
||||
update-deps:
|
||||
|
||||
+1
-17
@@ -54,22 +54,6 @@ Or for a one-time sync:
|
||||
basic-memory sync
|
||||
```
|
||||
|
||||
### 4. Updating Basic Memory
|
||||
|
||||
Basic Memory supports automatic updates by default for `uv tool` and Homebrew installs.
|
||||
|
||||
For manual checks and upgrades:
|
||||
|
||||
```bash
|
||||
# Check now and install if supported
|
||||
bm update
|
||||
|
||||
# Check only, do not install
|
||||
bm update --check
|
||||
```
|
||||
|
||||
To disable automatic updates, set `"auto_update": false` in `~/.basic-memory/config.json`.
|
||||
|
||||
## Configuration Options
|
||||
|
||||
### Custom Directory
|
||||
@@ -141,4 +125,4 @@ If you encounter issues:
|
||||
cat ~/.basic-memory/basic-memory.log
|
||||
```
|
||||
|
||||
For more detailed information, refer to the [full documentation](https://docs.basicmemory.com/).
|
||||
For more detailed information, refer to the [full documentation](https://memory.basicmachines.co/).
|
||||
+3
-13
@@ -25,16 +25,15 @@ dependencies = [
|
||||
"unidecode>=1.3.8",
|
||||
"dateparser>=1.2.0",
|
||||
"watchfiles>=1.0.4",
|
||||
"fastapi[standard]>=0.136.1",
|
||||
"fastapi[standard]>=0.115.8",
|
||||
"alembic>=1.14.1",
|
||||
"pillow>=11.1.0",
|
||||
"pybars3>=0.9.7",
|
||||
# Keep FastMCP pinned until each minor upgrade passes the MCP transport matrix.
|
||||
"fastmcp==3.3.1",
|
||||
"fastmcp>=3.0.1,<4",
|
||||
"pyjwt>=2.10.1",
|
||||
"python-dotenv>=1.1.0",
|
||||
"pytest-aio>=1.9.0",
|
||||
"aiofiles>=24.1.0",
|
||||
"aiofiles>=24.1.0", # Optional observability (disabled by default via config)
|
||||
"asyncpg>=0.30.0",
|
||||
"nest-asyncio>=1.6.0", # For Alembic migrations with Postgres
|
||||
"pytest-asyncio>=1.2.0",
|
||||
@@ -48,8 +47,6 @@ dependencies = [
|
||||
"fastembed>=0.7.4",
|
||||
"sqlite-vec>=0.1.6",
|
||||
"openai>=1.100.2",
|
||||
"logfire>=4.19.0",
|
||||
"psutil>=5.9.0",
|
||||
]
|
||||
|
||||
[project.urls]
|
||||
@@ -71,12 +68,6 @@ addopts = "--cov=basic_memory --cov-report term-missing"
|
||||
testpaths = ["tests", "test-int"]
|
||||
asyncio_mode = "strict"
|
||||
asyncio_default_fixture_loop_scope = "function"
|
||||
filterwarnings = [
|
||||
"ignore:The @wait_container_is_ready decorator is deprecated.*:DeprecationWarning:testcontainers\\.core\\.waiting_utils",
|
||||
"ignore:The default datetime adapter is deprecated as of Python 3\\.12.*:DeprecationWarning:aiosqlite\\.core",
|
||||
"ignore:codecs\\.open\\(\\) is deprecated\\. Use open\\(\\) instead\\.:DeprecationWarning:frontmatter",
|
||||
"ignore:Parsing dates involving a day of month without a year specified is ambiguous.*:DeprecationWarning:dateparser\\.utils\\.strptime",
|
||||
]
|
||||
markers = [
|
||||
"benchmark: Performance benchmark tests (deselect with '-m \"not benchmark\"')",
|
||||
"slow: Slow-running tests (deselect with '-m \"not slow\"')",
|
||||
@@ -92,7 +83,6 @@ target-version = "py312"
|
||||
|
||||
[dependency-groups]
|
||||
dev = [
|
||||
"logfire>=4.19.0",
|
||||
"gevent>=24.11.1",
|
||||
"icecream>=2.1.3",
|
||||
"pytest>=8.3.4",
|
||||
|
||||
+2
-2
@@ -6,12 +6,12 @@
|
||||
"url": "https://github.com/basicmachines-co/basic-memory.git",
|
||||
"source": "github"
|
||||
},
|
||||
"version": "0.21.4",
|
||||
"version": "0.18.5",
|
||||
"packages": [
|
||||
{
|
||||
"registryType": "pypi",
|
||||
"identifier": "basic-memory",
|
||||
"version": "0.21.4",
|
||||
"version": "0.18.5",
|
||||
"runtimeHint": "uvx",
|
||||
"runtimeArguments": [
|
||||
{"type": "positional", "value": "basic-memory"},
|
||||
|
||||
@@ -1,10 +0,0 @@
|
||||
{
|
||||
"version": 1,
|
||||
"skills": {
|
||||
"instrumentation": {
|
||||
"source": "pydantic/skills",
|
||||
"sourceType": "github",
|
||||
"computedHash": "0727bffc6a92fdeaf675ae5796ae25341e193327e8c95cd06b188dc4a0a4e62e"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,7 +1,7 @@
|
||||
"""basic-memory - Local-first knowledge management combining Zettelkasten with knowledge graphs"""
|
||||
|
||||
# Package version - updated by release automation
|
||||
__version__ = "0.21.4"
|
||||
__version__ = "0.18.5"
|
||||
|
||||
# API version for FastAPI - independent of package version
|
||||
__api_version__ = "v0"
|
||||
|
||||
@@ -66,7 +66,7 @@ target_metadata = Base.metadata
|
||||
|
||||
|
||||
# Add this function to tell Alembic what to include/exclude
|
||||
def include_object(obj, name, type_, reflected, compare_to):
|
||||
def include_object(object, name, type_, reflected, compare_to):
|
||||
# Ignore SQLite FTS tables
|
||||
if type_ == "table" and name.startswith("search_index"):
|
||||
return False
|
||||
@@ -118,54 +118,6 @@ async def run_async_migrations(connectable):
|
||||
await connectable.dispose()
|
||||
|
||||
|
||||
def _run_async_migrations_with_asyncio_run(connectable) -> None:
|
||||
"""Run async migrations with asyncio.run while closing failed coroutines.
|
||||
|
||||
Trigger: asyncio.run() may reject execution when another event loop is already active.
|
||||
Why: Python raises before awaiting the coroutine, which otherwise leaks a
|
||||
RuntimeWarning about an un-awaited coroutine.
|
||||
Outcome: close the pending coroutine before bubbling the RuntimeError to the
|
||||
fallback path.
|
||||
"""
|
||||
migration_coro = run_async_migrations(connectable)
|
||||
try:
|
||||
asyncio.run(migration_coro)
|
||||
except RuntimeError:
|
||||
migration_coro.close()
|
||||
raise
|
||||
|
||||
|
||||
def _run_async_migrations_in_thread(connectable) -> None:
|
||||
"""Run async migrations in a dedicated thread with its own event loop."""
|
||||
import concurrent.futures
|
||||
|
||||
def run_in_thread():
|
||||
"""Run async migrations in a new event loop in a separate thread."""
|
||||
new_loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(new_loop)
|
||||
try:
|
||||
new_loop.run_until_complete(run_async_migrations(connectable))
|
||||
finally:
|
||||
new_loop.close()
|
||||
|
||||
with concurrent.futures.ThreadPoolExecutor() as executor:
|
||||
future = executor.submit(run_in_thread)
|
||||
future.result() # Wait for completion and re-raise any exceptions
|
||||
|
||||
|
||||
def _run_async_engine_migrations(connectable) -> None:
|
||||
"""Run async-engine migrations with a running-loop fallback."""
|
||||
try:
|
||||
_run_async_migrations_with_asyncio_run(connectable)
|
||||
except RuntimeError as e:
|
||||
if "cannot be called from a running event loop" in str(e):
|
||||
# We're in a running event loop (likely uvloop or Python 3.14+ tests).
|
||||
# Switch to a dedicated thread so Alembic can finish without nesting loops.
|
||||
_run_async_migrations_in_thread(connectable)
|
||||
else:
|
||||
raise
|
||||
|
||||
|
||||
def run_migrations_online() -> None:
|
||||
"""Run migrations in 'online' mode.
|
||||
|
||||
@@ -196,10 +148,30 @@ def run_migrations_online() -> None:
|
||||
|
||||
# Handle async engines (PostgreSQL with asyncpg)
|
||||
if isinstance(connectable, AsyncEngine):
|
||||
# Trigger: async engines need Alembic work to cross the sync/async boundary.
|
||||
# Why: most callers can use asyncio.run(), but running-loop contexts need a thread fallback.
|
||||
# Outcome: migrations complete without leaking un-awaited coroutines.
|
||||
_run_async_engine_migrations(connectable)
|
||||
# Try to run async migrations
|
||||
# nest_asyncio allows asyncio.run() from within event loops, but doesn't work with uvloop
|
||||
try:
|
||||
asyncio.run(run_async_migrations(connectable))
|
||||
except RuntimeError as e:
|
||||
if "cannot be called from a running event loop" in str(e):
|
||||
# We're in a running event loop (likely uvloop) - need to use a different approach
|
||||
# Create a new thread to run the async migrations
|
||||
import concurrent.futures
|
||||
|
||||
def run_in_thread():
|
||||
"""Run async migrations in a new event loop in a separate thread."""
|
||||
new_loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(new_loop)
|
||||
try:
|
||||
new_loop.run_until_complete(run_async_migrations(connectable))
|
||||
finally:
|
||||
new_loop.close()
|
||||
|
||||
with concurrent.futures.ThreadPoolExecutor() as executor:
|
||||
future = executor.submit(run_in_thread)
|
||||
future.result() # Wait for completion and re-raise any exceptions
|
||||
else:
|
||||
raise
|
||||
else:
|
||||
# Handle sync engines (SQLite) or sync connections
|
||||
if hasattr(connectable, "connect"):
|
||||
|
||||
@@ -1,65 +0,0 @@
|
||||
"""Add note_content table
|
||||
|
||||
Revision ID: l5g6h7i8j9k0
|
||||
Revises: k4e5f6g7h8i9
|
||||
Create Date: 2026-04-04 12:00:00.000000
|
||||
|
||||
"""
|
||||
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "l5g6h7i8j9k0"
|
||||
down_revision: Union[str, None] = "k4e5f6g7h8i9"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
"""Create note_content for materialized note content and sync state."""
|
||||
op.create_table(
|
||||
"note_content",
|
||||
sa.Column("entity_id", sa.Integer(), nullable=False),
|
||||
sa.Column("project_id", sa.Integer(), nullable=False),
|
||||
sa.Column("external_id", sa.String(), nullable=False),
|
||||
sa.Column("file_path", sa.String(), nullable=False),
|
||||
sa.Column("markdown_content", sa.Text(), nullable=False),
|
||||
sa.Column("db_version", sa.BigInteger(), nullable=False),
|
||||
sa.Column("db_checksum", sa.String(), nullable=False),
|
||||
sa.Column("file_version", sa.BigInteger(), nullable=True),
|
||||
sa.Column("file_checksum", sa.String(), nullable=True),
|
||||
sa.Column("file_write_status", sa.String(), nullable=False),
|
||||
sa.Column("last_source", sa.String(), nullable=True),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("file_updated_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("last_materialization_error", sa.Text(), nullable=True),
|
||||
sa.Column("last_materialization_attempt_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.CheckConstraint(
|
||||
"file_write_status IN ("
|
||||
"'pending', "
|
||||
"'writing', "
|
||||
"'synced', "
|
||||
"'failed', "
|
||||
"'external_change_detected'"
|
||||
")",
|
||||
name="ck_note_content_file_write_status",
|
||||
),
|
||||
sa.ForeignKeyConstraint(["entity_id"], ["entity.id"], ondelete="CASCADE"),
|
||||
sa.ForeignKeyConstraint(["project_id"], ["project.id"], ondelete="CASCADE"),
|
||||
sa.PrimaryKeyConstraint("entity_id"),
|
||||
)
|
||||
op.create_index("ix_note_content_project_id", "note_content", ["project_id"], unique=False)
|
||||
op.create_index("ix_note_content_file_path", "note_content", ["file_path"], unique=False)
|
||||
op.create_index("ix_note_content_external_id", "note_content", ["external_id"], unique=True)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
"""Drop note_content and its supporting indexes."""
|
||||
op.drop_index("ix_note_content_external_id", table_name="note_content")
|
||||
op.drop_index("ix_note_content_file_path", table_name="note_content")
|
||||
op.drop_index("ix_note_content_project_id", table_name="note_content")
|
||||
op.drop_table("note_content")
|
||||
@@ -1,84 +0,0 @@
|
||||
"""Persist vector sync fingerprints on chunk metadata.
|
||||
|
||||
Revision ID: m6h7i8j9k0l1
|
||||
Revises: l5g6h7i8j9k0
|
||||
Create Date: 2026-04-07 00:00:00.000000
|
||||
|
||||
"""
|
||||
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "m6h7i8j9k0l1"
|
||||
down_revision: Union[str, None] = "l5g6h7i8j9k0"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
"""Add entity fingerprint + embedding model metadata to Postgres chunk rows.
|
||||
|
||||
Trigger: vector sync now fast-skips unchanged entities using persisted
|
||||
semantic fingerprints.
|
||||
Why: chunk rows already own the per-entity derived metadata we diff against,
|
||||
so persisting the fingerprint on that table avoids a second sync-state table.
|
||||
Outcome: existing rows get empty-string placeholders and will be refreshed on
|
||||
the next vector sync before they become eligible for skip checks.
|
||||
"""
|
||||
connection = op.get_bind()
|
||||
if connection.dialect.name != "postgresql":
|
||||
return
|
||||
|
||||
op.execute(
|
||||
"""
|
||||
ALTER TABLE search_vector_chunks
|
||||
ADD COLUMN IF NOT EXISTS entity_fingerprint TEXT
|
||||
"""
|
||||
)
|
||||
op.execute(
|
||||
"""
|
||||
ALTER TABLE search_vector_chunks
|
||||
ADD COLUMN IF NOT EXISTS embedding_model TEXT
|
||||
"""
|
||||
)
|
||||
op.execute(
|
||||
"""
|
||||
UPDATE search_vector_chunks
|
||||
SET entity_fingerprint = COALESCE(entity_fingerprint, ''),
|
||||
embedding_model = COALESCE(embedding_model, '')
|
||||
"""
|
||||
)
|
||||
op.execute(
|
||||
"""
|
||||
ALTER TABLE search_vector_chunks
|
||||
ALTER COLUMN entity_fingerprint SET NOT NULL
|
||||
"""
|
||||
)
|
||||
op.execute(
|
||||
"""
|
||||
ALTER TABLE search_vector_chunks
|
||||
ALTER COLUMN embedding_model SET NOT NULL
|
||||
"""
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
"""Remove vector sync fingerprint columns from Postgres chunk rows."""
|
||||
connection = op.get_bind()
|
||||
if connection.dialect.name != "postgresql":
|
||||
return
|
||||
|
||||
op.execute(
|
||||
"""
|
||||
ALTER TABLE search_vector_chunks
|
||||
DROP COLUMN IF EXISTS embedding_model
|
||||
"""
|
||||
)
|
||||
op.execute(
|
||||
"""
|
||||
ALTER TABLE search_vector_chunks
|
||||
DROP COLUMN IF EXISTS entity_fingerprint
|
||||
"""
|
||||
)
|
||||
@@ -1,86 +0,0 @@
|
||||
"""Remove orphaned search rows whose project was already deleted.
|
||||
|
||||
Revision ID: n7i8j9k0l1m2
|
||||
Revises: m6h7i8j9k0l1
|
||||
Create Date: 2026-05-15 18:30:00.000000
|
||||
|
||||
"""
|
||||
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy import inspect
|
||||
|
||||
|
||||
revision: str = "n7i8j9k0l1m2"
|
||||
down_revision: Union[str, None] = "m6h7i8j9k0l1"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def _table_exists(connection, table_name: str) -> bool:
|
||||
"""Inspector-based table check, dialect agnostic.
|
||||
|
||||
Trigger: SQLite creates search_index as an FTS5 virtual table at runtime
|
||||
via SearchRepository.init_search_index, not through Alembic, so fresh
|
||||
installs hit this migration before the table exists.
|
||||
Why: a blind DELETE against a missing table fails the whole upgrade.
|
||||
Outcome: callers skip the sweep when the table isn't present yet — the
|
||||
runtime-created table on a fresh DB has no orphans to clean.
|
||||
"""
|
||||
return table_name in inspect(connection).get_table_names()
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
"""Purge orphaned search rows left over from prior project deletions.
|
||||
|
||||
Trigger: project deletion on SQLite never removed the derived FTS rows,
|
||||
because the FTS5 virtual table can't carry a foreign key. The leak shows
|
||||
up in two shapes:
|
||||
1. project_id no longer exists in `project` (deleted project, id never
|
||||
reused).
|
||||
2. project_id still exists but `entity_id` no longer exists in `entity`
|
||||
— auto-increment handed the id to a brand-new project and the FTS
|
||||
rows from the deleted predecessor masquerade as the new tenant's data.
|
||||
Why: search_index.project_id is the only scope predicate the search
|
||||
repository applies, so leftover rows surface under the wrong project on
|
||||
every search.
|
||||
Outcome: a one-time sweep deletes both shapes, from the FTS index and
|
||||
from search_vector_chunks. Postgres already cascaded on FK delete, so
|
||||
these statements are no-ops there.
|
||||
"""
|
||||
connection = op.get_bind()
|
||||
|
||||
if _table_exists(connection, "search_index"):
|
||||
op.execute(
|
||||
"""
|
||||
DELETE FROM search_index
|
||||
WHERE project_id NOT IN (SELECT id FROM project)
|
||||
"""
|
||||
)
|
||||
op.execute(
|
||||
"""
|
||||
DELETE FROM search_index
|
||||
WHERE entity_id IS NOT NULL
|
||||
AND entity_id NOT IN (SELECT id FROM entity)
|
||||
"""
|
||||
)
|
||||
|
||||
if _table_exists(connection, "search_vector_chunks"):
|
||||
op.execute(
|
||||
"""
|
||||
DELETE FROM search_vector_chunks
|
||||
WHERE project_id NOT IN (SELECT id FROM project)
|
||||
"""
|
||||
)
|
||||
op.execute(
|
||||
"""
|
||||
DELETE FROM search_vector_chunks
|
||||
WHERE entity_id NOT IN (SELECT id FROM entity)
|
||||
"""
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
"""No-op: orphan rows cannot be reconstructed."""
|
||||
pass
|
||||
+21
-63
@@ -4,7 +4,6 @@ from contextlib import asynccontextmanager
|
||||
|
||||
from fastapi import FastAPI, HTTPException, Request
|
||||
from fastapi.exception_handlers import http_exception_handler
|
||||
from fastapi.responses import JSONResponse
|
||||
from fastapi.routing import APIRouter
|
||||
from loguru import logger
|
||||
|
||||
@@ -20,22 +19,17 @@ from basic_memory.api.v2.routers import (
|
||||
prompt_router as v2_prompt,
|
||||
importer_router as v2_importer,
|
||||
schema_router as v2_schema,
|
||||
graph_router as v2_graph,
|
||||
fcm_router as v2_fcm,
|
||||
)
|
||||
from basic_memory.api.v2.routers.project_router import (
|
||||
add_project,
|
||||
list_projects,
|
||||
synchronize_projects,
|
||||
)
|
||||
import logfire
|
||||
from basic_memory.config import init_api_logging
|
||||
from basic_memory.services.exceptions import EntityAlreadyExistsError
|
||||
from basic_memory.services.initialization import initialize_app
|
||||
from basic_memory.workspace_context import (
|
||||
WORKSPACE_SLUG_HEADER,
|
||||
WORKSPACE_TYPE_HEADER,
|
||||
workspace_permalink_context_validation_error,
|
||||
workspace_permalink_context,
|
||||
)
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
@@ -51,39 +45,30 @@ async def lifespan(app: FastAPI): # pragma: no cover
|
||||
set_container(container)
|
||||
app.state.container = container
|
||||
|
||||
with logfire.span(
|
||||
"api.lifecycle.startup",
|
||||
entrypoint="api",
|
||||
mode=container.mode.name.lower(),
|
||||
):
|
||||
logger.info(f"Starting Basic Memory API (mode={container.mode.name})")
|
||||
logger.info(f"Starting Basic Memory API (mode={container.mode.name})")
|
||||
|
||||
await initialize_app(container.config)
|
||||
await initialize_app(container.config)
|
||||
|
||||
# Cache database connections in app state for performance
|
||||
logger.info("Initializing database and caching connections...")
|
||||
engine, session_maker = await container.init_database()
|
||||
app.state.engine = engine
|
||||
app.state.session_maker = session_maker
|
||||
logger.info("Database connections cached in app state")
|
||||
# Cache database connections in app state for performance
|
||||
logger.info("Initializing database and caching connections...")
|
||||
engine, session_maker = await container.init_database()
|
||||
app.state.engine = engine
|
||||
app.state.session_maker = session_maker
|
||||
logger.info("Database connections cached in app state")
|
||||
|
||||
# Create and start sync coordinator (lifecycle centralized in coordinator)
|
||||
sync_coordinator = container.create_sync_coordinator()
|
||||
await sync_coordinator.start()
|
||||
app.state.sync_coordinator = sync_coordinator
|
||||
# Create and start sync coordinator (lifecycle centralized in coordinator)
|
||||
sync_coordinator = container.create_sync_coordinator()
|
||||
await sync_coordinator.start()
|
||||
app.state.sync_coordinator = sync_coordinator
|
||||
|
||||
# Proceed with startup
|
||||
yield
|
||||
|
||||
# Shutdown - coordinator handles clean task cancellation
|
||||
with logfire.span(
|
||||
"api.lifecycle.shutdown",
|
||||
entrypoint="api",
|
||||
mode=container.mode.name.lower(),
|
||||
):
|
||||
logger.info("Shutting down Basic Memory API")
|
||||
await sync_coordinator.stop()
|
||||
await container.shutdown_database()
|
||||
logger.info("Shutting down Basic Memory API")
|
||||
await sync_coordinator.stop()
|
||||
|
||||
await container.shutdown_database()
|
||||
|
||||
|
||||
# Initialize FastAPI app
|
||||
@@ -94,32 +79,6 @@ app = FastAPI(
|
||||
lifespan=lifespan,
|
||||
)
|
||||
|
||||
|
||||
@app.middleware("http")
|
||||
async def workspace_permalink_context_middleware(request: Request, call_next):
|
||||
"""Populate workspace permalink context from request headers."""
|
||||
workspace_slug = request.headers.get(WORKSPACE_SLUG_HEADER)
|
||||
workspace_type = request.headers.get(WORKSPACE_TYPE_HEADER)
|
||||
|
||||
validation_error = workspace_permalink_context_validation_error(workspace_slug, workspace_type)
|
||||
if validation_error is not None:
|
||||
return JSONResponse(
|
||||
status_code=400,
|
||||
content={"detail": validation_error},
|
||||
)
|
||||
|
||||
if not workspace_slug:
|
||||
return await call_next(request)
|
||||
|
||||
# ContextVar state remains active across the awaited downstream handler while
|
||||
# this context manager is open, so entity creation can see request metadata.
|
||||
with workspace_permalink_context(
|
||||
workspace_slug=workspace_slug,
|
||||
workspace_type=workspace_type,
|
||||
):
|
||||
return await call_next(request)
|
||||
|
||||
|
||||
# Include v2 routers FIRST (more specific paths must match before /{project} catch-all)
|
||||
app.include_router(v2_knowledge, prefix="/v2/projects/{project_id}")
|
||||
app.include_router(v2_memory, prefix="/v2/projects/{project_id}")
|
||||
@@ -129,6 +88,8 @@ app.include_router(v2_directory, prefix="/v2/projects/{project_id}")
|
||||
app.include_router(v2_prompt, prefix="/v2/projects/{project_id}")
|
||||
app.include_router(v2_importer, prefix="/v2/projects/{project_id}")
|
||||
app.include_router(v2_schema, prefix="/v2/projects/{project_id}")
|
||||
app.include_router(v2_graph, prefix="/v2/projects/{project_id}")
|
||||
app.include_router(v2_fcm, prefix="/v2/projects/{project_id}")
|
||||
app.include_router(v2_project, prefix="/v2")
|
||||
|
||||
# Legacy web app proxy paths (compat with /proxy/projects/projects)
|
||||
@@ -179,7 +140,4 @@ async def exception_handler(request, exc): # pragma: no cover
|
||||
error_type=type(exc).__name__,
|
||||
error=str(exc),
|
||||
)
|
||||
return await http_exception_handler(
|
||||
request,
|
||||
HTTPException(status_code=500, detail="Internal server error"),
|
||||
)
|
||||
return await http_exception_handler(request, HTTPException(status_code=500, detail=str(exc)))
|
||||
|
||||
@@ -21,6 +21,8 @@ from basic_memory.api.v2.routers import (
|
||||
directory_router,
|
||||
prompt_router,
|
||||
importer_router,
|
||||
graph_router,
|
||||
fcm_router,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
@@ -32,4 +34,6 @@ __all__ = [
|
||||
"directory_router",
|
||||
"prompt_router",
|
||||
"importer_router",
|
||||
"graph_router",
|
||||
"fcm_router",
|
||||
]
|
||||
|
||||
@@ -9,6 +9,8 @@ from basic_memory.api.v2.routers.directory_router import router as directory_rou
|
||||
from basic_memory.api.v2.routers.prompt_router import router as prompt_router
|
||||
from basic_memory.api.v2.routers.importer_router import router as importer_router
|
||||
from basic_memory.api.v2.routers.schema_router import router as schema_router
|
||||
from basic_memory.api.v2.routers.graph_router import router as graph_router
|
||||
from basic_memory.api.v2.routers.fcm_router import router as fcm_router
|
||||
|
||||
__all__ = [
|
||||
"knowledge_router",
|
||||
@@ -20,4 +22,6 @@ __all__ = [
|
||||
"prompt_router",
|
||||
"importer_router",
|
||||
"schema_router",
|
||||
"graph_router",
|
||||
"fcm_router",
|
||||
]
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
"""V2 router for FCM simulation and interop endpoints."""
|
||||
|
||||
from fastapi import APIRouter
|
||||
|
||||
from basic_memory.deps import FCMServiceV2ExternalDep, ProjectExternalIdPathDep
|
||||
from basic_memory.schemas.graph_intelligence import (
|
||||
FCMExportRequest,
|
||||
FCMExportResponse,
|
||||
FCMImportRequest,
|
||||
FCMImportResponse,
|
||||
FCMRankActionsRequest,
|
||||
FCMRankActionsResponse,
|
||||
FCMSimulateRequest,
|
||||
FCMSimulateResponse,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/fcm", tags=["fcm-v2"])
|
||||
|
||||
|
||||
@router.post("/simulate", response_model=FCMSimulateResponse)
|
||||
async def fcm_simulate(
|
||||
request: FCMSimulateRequest,
|
||||
fcm_service: FCMServiceV2ExternalDep,
|
||||
project_id: ProjectExternalIdPathDep,
|
||||
) -> FCMSimulateResponse:
|
||||
"""Run an FCM scenario simulation."""
|
||||
_ = project_id
|
||||
return await fcm_service.simulate(request)
|
||||
|
||||
|
||||
@router.post("/rank-actions", response_model=FCMRankActionsResponse)
|
||||
async def fcm_rank_actions(
|
||||
request: FCMRankActionsRequest,
|
||||
fcm_service: FCMServiceV2ExternalDep,
|
||||
project_id: ProjectExternalIdPathDep,
|
||||
) -> FCMRankActionsResponse:
|
||||
"""Rank action candidates toward a goal."""
|
||||
_ = project_id
|
||||
return await fcm_service.rank_actions(request)
|
||||
|
||||
|
||||
@router.post("/import", response_model=FCMImportResponse)
|
||||
async def fcm_import(
|
||||
request: FCMImportRequest,
|
||||
fcm_service: FCMServiceV2ExternalDep,
|
||||
project_id: ProjectExternalIdPathDep,
|
||||
) -> FCMImportResponse:
|
||||
"""Import an FCM model using a supported interchange format."""
|
||||
_ = project_id
|
||||
return await fcm_service.import_model(request)
|
||||
|
||||
|
||||
@router.post("/export", response_model=FCMExportResponse)
|
||||
async def fcm_export(
|
||||
request: FCMExportRequest,
|
||||
fcm_service: FCMServiceV2ExternalDep,
|
||||
project_id: ProjectExternalIdPathDep,
|
||||
) -> FCMExportResponse:
|
||||
"""Export an FCM model using a supported interchange format."""
|
||||
_ = project_id
|
||||
return await fcm_service.export_model(request)
|
||||
@@ -0,0 +1,71 @@
|
||||
"""V2 router for graph intelligence endpoints."""
|
||||
|
||||
from fastapi import APIRouter, Query
|
||||
|
||||
from basic_memory.deps import (
|
||||
GraphIntelligenceServiceV2ExternalDep,
|
||||
ProjectExternalIdPathDep,
|
||||
TaskSchedulerDep,
|
||||
)
|
||||
from basic_memory.schemas.graph_intelligence import (
|
||||
GraphHealthResponse,
|
||||
GraphImpactRequest,
|
||||
GraphImpactResponse,
|
||||
GraphLineageRequest,
|
||||
GraphLineageResponse,
|
||||
GraphReindexRequest,
|
||||
GraphReindexResponse,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/graph", tags=["graph-v2"])
|
||||
|
||||
|
||||
@router.post("/lineage", response_model=GraphLineageResponse)
|
||||
async def graph_lineage(
|
||||
request: GraphLineageRequest,
|
||||
graph_service: GraphIntelligenceServiceV2ExternalDep,
|
||||
project_id: ProjectExternalIdPathDep,
|
||||
) -> GraphLineageResponse:
|
||||
"""Build lineage paths from a start node toward an optional goal."""
|
||||
_ = project_id
|
||||
return await graph_service.lineage(request)
|
||||
|
||||
|
||||
@router.post("/impact", response_model=GraphImpactResponse)
|
||||
async def graph_impact(
|
||||
request: GraphImpactRequest,
|
||||
graph_service: GraphIntelligenceServiceV2ExternalDep,
|
||||
project_id: ProjectExternalIdPathDep,
|
||||
) -> GraphImpactResponse:
|
||||
"""Compute impact radius from a target node."""
|
||||
_ = project_id
|
||||
return await graph_service.impact(request)
|
||||
|
||||
|
||||
@router.get("/health", response_model=GraphHealthResponse)
|
||||
async def graph_health(
|
||||
graph_service: GraphIntelligenceServiceV2ExternalDep,
|
||||
project_id: ProjectExternalIdPathDep,
|
||||
scope: str | None = Query(default=None),
|
||||
timeframe: str | None = Query(default=None),
|
||||
) -> GraphHealthResponse:
|
||||
"""Report graph quality metrics and issue candidates."""
|
||||
_ = project_id
|
||||
return await graph_service.health(scope=scope, timeframe=timeframe)
|
||||
|
||||
|
||||
@router.post("/reindex", response_model=GraphReindexResponse)
|
||||
async def graph_reindex(
|
||||
request: GraphReindexRequest,
|
||||
graph_service: GraphIntelligenceServiceV2ExternalDep,
|
||||
task_scheduler: TaskSchedulerDep,
|
||||
project_id: ProjectExternalIdPathDep,
|
||||
) -> GraphReindexResponse:
|
||||
"""Queue a graph reindex operation for the current project."""
|
||||
task_scheduler.schedule(
|
||||
"reindex_graph_project",
|
||||
project_id=project_id,
|
||||
mode=request.mode,
|
||||
reason=request.reason,
|
||||
)
|
||||
return await graph_service.start_reindex_job()
|
||||
@@ -10,7 +10,6 @@ import logging
|
||||
from fastapi import APIRouter, Form, HTTPException, UploadFile, status, Path
|
||||
|
||||
from basic_memory.deps import (
|
||||
AppConfigDep,
|
||||
ChatGPTImporterV2ExternalDep,
|
||||
ClaudeConversationsImporterV2ExternalDep,
|
||||
ClaudeProjectsImporterV2ExternalDep,
|
||||
@@ -28,21 +27,9 @@ logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/import", tags=["import-v2"])
|
||||
|
||||
|
||||
async def read_import_upload(file: UploadFile, max_bytes: int) -> bytes:
|
||||
"""Read an import upload with a hard cap before JSON parsing."""
|
||||
content = await file.read(max_bytes + 1)
|
||||
if len(content) > max_bytes:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_413_CONTENT_TOO_LARGE,
|
||||
detail=f"Import file exceeds maximum size of {max_bytes} bytes.",
|
||||
)
|
||||
return content
|
||||
|
||||
|
||||
@router.post("/chatgpt", response_model=ChatImportResult)
|
||||
async def import_chatgpt(
|
||||
importer: ChatGPTImporterV2ExternalDep,
|
||||
config: AppConfigDep,
|
||||
file: UploadFile,
|
||||
project_id: str = Path(..., description="Project external UUID"),
|
||||
directory: str = Form("conversations"),
|
||||
@@ -62,13 +49,12 @@ async def import_chatgpt(
|
||||
HTTPException: If import fails.
|
||||
"""
|
||||
logger.info(f"V2 Importing ChatGPT conversations for project {project_id}")
|
||||
return await import_file(importer, file, directory, config.import_upload_max_bytes)
|
||||
return await import_file(importer, file, directory)
|
||||
|
||||
|
||||
@router.post("/claude/conversations", response_model=ChatImportResult)
|
||||
async def import_claude_conversations(
|
||||
importer: ClaudeConversationsImporterV2ExternalDep,
|
||||
config: AppConfigDep,
|
||||
file: UploadFile,
|
||||
project_id: str = Path(..., description="Project external UUID"),
|
||||
directory: str = Form("conversations"),
|
||||
@@ -88,13 +74,12 @@ async def import_claude_conversations(
|
||||
HTTPException: If import fails.
|
||||
"""
|
||||
logger.info(f"V2 Importing Claude conversations for project {project_id}")
|
||||
return await import_file(importer, file, directory, config.import_upload_max_bytes)
|
||||
return await import_file(importer, file, directory)
|
||||
|
||||
|
||||
@router.post("/claude/projects", response_model=ProjectImportResult)
|
||||
async def import_claude_projects(
|
||||
importer: ClaudeProjectsImporterV2ExternalDep,
|
||||
config: AppConfigDep,
|
||||
file: UploadFile,
|
||||
project_id: str = Path(..., description="Project external UUID"),
|
||||
directory: str = Form("projects"),
|
||||
@@ -114,13 +99,12 @@ async def import_claude_projects(
|
||||
HTTPException: If import fails.
|
||||
"""
|
||||
logger.info(f"V2 Importing Claude projects for project {project_id}")
|
||||
return await import_file(importer, file, directory, config.import_upload_max_bytes)
|
||||
return await import_file(importer, file, directory)
|
||||
|
||||
|
||||
@router.post("/memory-json", response_model=EntityImportResult)
|
||||
async def import_memory_json(
|
||||
importer: MemoryJsonImporterV2ExternalDep,
|
||||
config: AppConfigDep,
|
||||
file: UploadFile,
|
||||
project_id: str = Path(..., description="Project external UUID"),
|
||||
directory: str = Form("conversations"),
|
||||
@@ -142,7 +126,7 @@ async def import_memory_json(
|
||||
logger.info(f"V2 Importing memory.json for project {project_id}")
|
||||
try:
|
||||
file_data = []
|
||||
file_bytes = await read_import_upload(file, config.import_upload_max_bytes)
|
||||
file_bytes = await file.read()
|
||||
file_str = file_bytes.decode("utf-8")
|
||||
for line in file_str.splitlines():
|
||||
json_data = json.loads(line)
|
||||
@@ -154,8 +138,6 @@ async def import_memory_json(
|
||||
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
|
||||
detail=result.error_message or "Import failed",
|
||||
)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.exception("V2 Import failed")
|
||||
raise HTTPException(
|
||||
@@ -165,16 +147,13 @@ async def import_memory_json(
|
||||
return result
|
||||
|
||||
|
||||
async def import_file(
|
||||
importer: Importer, file: UploadFile, destination_directory: str, max_bytes: int
|
||||
):
|
||||
async def import_file(importer: Importer, file: UploadFile, destination_directory: str):
|
||||
"""Helper function to import a file using an importer instance.
|
||||
|
||||
Args:
|
||||
importer: The importer instance to use
|
||||
file: The file to import
|
||||
destination_directory: Destination directory for imported content
|
||||
max_bytes: Maximum upload size in bytes; raises HTTP 413 if exceeded
|
||||
|
||||
Returns:
|
||||
Import result from the importer
|
||||
@@ -184,8 +163,7 @@ async def import_file(
|
||||
"""
|
||||
try:
|
||||
# Process file
|
||||
upload_bytes = await read_import_upload(file, max_bytes)
|
||||
json_data = json.loads(upload_bytes)
|
||||
json_data = json.load(file.file)
|
||||
result = await importer.import_data(json_data, destination_directory)
|
||||
if not result.success: # pragma: no cover
|
||||
raise HTTPException(
|
||||
@@ -195,8 +173,6 @@ async def import_file(
|
||||
|
||||
return result
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.exception("V2 Import failed")
|
||||
raise HTTPException(
|
||||
|
||||
@@ -10,10 +10,9 @@ Key improvements:
|
||||
- Simplified caching strategies
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Response, Path
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends, Response, Path, Query
|
||||
from loguru import logger
|
||||
|
||||
import logfire
|
||||
from basic_memory.deps import (
|
||||
EntityServiceV2ExternalDep,
|
||||
SearchServiceV2ExternalDep,
|
||||
@@ -21,9 +20,9 @@ from basic_memory.deps import (
|
||||
ProjectConfigV2ExternalDep,
|
||||
AppConfigDep,
|
||||
EntityRepositoryV2ExternalDep,
|
||||
RelationRepositoryV2ExternalDep,
|
||||
ProjectExternalIdPathDep,
|
||||
TaskSchedulerDep,
|
||||
FileServiceV2ExternalDep,
|
||||
)
|
||||
from basic_memory.schemas import DeleteEntitiesResponse
|
||||
from basic_memory.schemas.base import Entity
|
||||
@@ -32,13 +31,9 @@ from basic_memory.schemas.v2 import (
|
||||
EntityResolveRequest,
|
||||
EntityResolveResponse,
|
||||
EntityResponseV2,
|
||||
GraphEdge,
|
||||
GraphNode,
|
||||
GraphResponse,
|
||||
MoveEntityRequestV2,
|
||||
MoveDirectoryRequestV2,
|
||||
DeleteDirectoryRequestV2,
|
||||
OrphanEntitiesResponse,
|
||||
)
|
||||
from basic_memory.schemas.response import DirectoryMoveResult, DirectoryDeleteResult
|
||||
|
||||
@@ -61,88 +56,6 @@ def _schedule_vector_sync_if_enabled(
|
||||
)
|
||||
|
||||
|
||||
## Graph endpoint
|
||||
|
||||
|
||||
@router.get("/graph", response_model=GraphResponse)
|
||||
async def get_graph(
|
||||
project_id: ProjectExternalIdPathDep,
|
||||
entity_repository: EntityRepositoryV2ExternalDep,
|
||||
relation_repository: RelationRepositoryV2ExternalDep,
|
||||
) -> GraphResponse:
|
||||
"""Return all entities and resolved relations for knowledge graph visualization.
|
||||
|
||||
Returns a flat node/edge structure optimized for rendering with graph libraries.
|
||||
Only includes resolved relations (where to_id is not null).
|
||||
"""
|
||||
with logfire.span(
|
||||
"api.request.knowledge.get_graph",
|
||||
entrypoint="api",
|
||||
domain="knowledge",
|
||||
action="get_graph",
|
||||
):
|
||||
logger.info("API v2 request: get_graph")
|
||||
|
||||
# Fetch all entities for this project
|
||||
entities = await entity_repository.find_all(use_load_options=False)
|
||||
nodes = [
|
||||
GraphNode(
|
||||
external_id=entity.external_id,
|
||||
title=entity.title,
|
||||
note_type=entity.note_type,
|
||||
file_path=entity.file_path,
|
||||
)
|
||||
for entity in entities
|
||||
]
|
||||
|
||||
# Fetch all resolved relations (to_id is not null) with eager-loaded entities
|
||||
relations = await relation_repository.find_all()
|
||||
edges = [
|
||||
GraphEdge(
|
||||
from_id=relation.from_entity.external_id,
|
||||
to_id=relation.to_entity.external_id,
|
||||
relation_type=relation.relation_type,
|
||||
)
|
||||
for relation in relations
|
||||
if relation.to_entity is not None
|
||||
]
|
||||
|
||||
logger.info(f"API v2 response: graph with {len(nodes)} nodes and {len(edges)} edges")
|
||||
return GraphResponse(nodes=nodes, edges=edges)
|
||||
|
||||
|
||||
## Orphan entities endpoint
|
||||
|
||||
|
||||
@router.get("/orphans", response_model=OrphanEntitiesResponse)
|
||||
async def get_orphan_entities(
|
||||
project_id: ProjectExternalIdPathDep,
|
||||
entity_repository: EntityRepositoryV2ExternalDep,
|
||||
) -> OrphanEntitiesResponse:
|
||||
"""Return entities that have no incoming or outgoing relations."""
|
||||
with logfire.span(
|
||||
"api.request.knowledge.get_orphans",
|
||||
entrypoint="api",
|
||||
domain="knowledge",
|
||||
action="get_orphans",
|
||||
):
|
||||
logger.info("API v2 request: get_orphan_entities")
|
||||
|
||||
entities = await entity_repository.find_without_relations()
|
||||
nodes = [
|
||||
GraphNode(
|
||||
external_id=entity.external_id,
|
||||
title=entity.title,
|
||||
note_type=entity.note_type,
|
||||
file_path=entity.file_path,
|
||||
)
|
||||
for entity in entities
|
||||
]
|
||||
|
||||
logger.info(f"API v2 response: {len(nodes)} orphan entities")
|
||||
return OrphanEntitiesResponse(entities=nodes, total=len(nodes))
|
||||
|
||||
|
||||
## Resolution endpoint
|
||||
|
||||
|
||||
@@ -181,48 +94,47 @@ async def resolve_identifier(
|
||||
"resolution_method": "permalink"
|
||||
}
|
||||
"""
|
||||
with logfire.span(
|
||||
"api.request.knowledge.resolve_entity",
|
||||
entrypoint="api",
|
||||
domain="knowledge",
|
||||
action="resolve_entity",
|
||||
):
|
||||
logger.info(f"API v2 request: resolve_identifier for '{data.identifier}'")
|
||||
logger.info(f"API v2 request: resolve_identifier for '{data.identifier}'")
|
||||
|
||||
entity = await entity_repository.get_by_external_id(data.identifier)
|
||||
resolution_method = "external_id" if entity else "search"
|
||||
# Try to resolve by external_id first
|
||||
entity = await entity_repository.get_by_external_id(data.identifier)
|
||||
resolution_method = "external_id" if entity else "search"
|
||||
|
||||
if not entity:
|
||||
entity = await link_resolver.resolve_link(
|
||||
data.identifier, source_path=data.source_path, strict=data.strict
|
||||
)
|
||||
if entity:
|
||||
if entity.permalink == data.identifier:
|
||||
resolution_method = "permalink"
|
||||
elif entity.title == data.identifier:
|
||||
resolution_method = "title"
|
||||
elif entity.file_path == data.identifier:
|
||||
resolution_method = "path"
|
||||
else:
|
||||
resolution_method = "search"
|
||||
|
||||
if not entity:
|
||||
raise HTTPException(status_code=404, detail=f"Entity not found: '{data.identifier}'")
|
||||
|
||||
result = EntityResolveResponse(
|
||||
external_id=entity.external_id,
|
||||
entity_id=entity.id,
|
||||
permalink=entity.permalink,
|
||||
file_path=entity.file_path,
|
||||
title=entity.title,
|
||||
resolution_method=resolution_method,
|
||||
# If not found by external_id, try other resolution methods
|
||||
# Pass source_path for context-aware resolution (prefers notes closer to source)
|
||||
# Pass strict to control fuzzy search fallback (default False allows fuzzy matching)
|
||||
if not entity:
|
||||
entity = await link_resolver.resolve_link(
|
||||
data.identifier, source_path=data.source_path, strict=data.strict
|
||||
)
|
||||
if entity:
|
||||
# Determine resolution method
|
||||
if entity.permalink == data.identifier:
|
||||
resolution_method = "permalink"
|
||||
elif entity.title == data.identifier:
|
||||
resolution_method = "title"
|
||||
elif entity.file_path == data.identifier:
|
||||
resolution_method = "path"
|
||||
else:
|
||||
resolution_method = "search"
|
||||
|
||||
logger.debug(
|
||||
f"API v2 response: resolved '{data.identifier}' to external_id={result.external_id} via {resolution_method}"
|
||||
)
|
||||
if not entity:
|
||||
raise HTTPException(status_code=404, detail=f"Entity not found: '{data.identifier}'")
|
||||
|
||||
return result
|
||||
result = EntityResolveResponse(
|
||||
external_id=entity.external_id,
|
||||
entity_id=entity.id,
|
||||
permalink=entity.permalink,
|
||||
file_path=entity.file_path,
|
||||
title=entity.title,
|
||||
resolution_method=resolution_method,
|
||||
)
|
||||
|
||||
logger.debug(
|
||||
f"API v2 response: resolved '{data.identifier}' to external_id={result.external_id} via {resolution_method}"
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
## Read endpoints
|
||||
@@ -248,24 +160,18 @@ async def get_entity_by_id(
|
||||
Raises:
|
||||
HTTPException: 404 if entity not found
|
||||
"""
|
||||
with logfire.span(
|
||||
"api.request.knowledge.get_entity",
|
||||
entrypoint="api",
|
||||
domain="knowledge",
|
||||
action="get_entity",
|
||||
):
|
||||
logger.info(f"API v2 request: get_entity_by_id entity_id={entity_id}")
|
||||
logger.info(f"API v2 request: get_entity_by_id entity_id={entity_id}")
|
||||
|
||||
entity = await entity_repository.get_by_external_id(entity_id)
|
||||
if not entity:
|
||||
raise HTTPException(
|
||||
status_code=404, detail=f"Entity with external_id '{entity_id}' not found"
|
||||
)
|
||||
entity = await entity_repository.get_by_external_id(entity_id)
|
||||
if not entity:
|
||||
raise HTTPException(
|
||||
status_code=404, detail=f"Entity with external_id '{entity_id}' not found"
|
||||
)
|
||||
|
||||
result = EntityResponseV2.model_validate(entity)
|
||||
logger.info(f"API v2 response: external_id={entity_id}, title='{result.title}'")
|
||||
result = EntityResponseV2.model_validate(entity)
|
||||
logger.info(f"API v2 response: external_id={entity_id}, title='{result.title}'")
|
||||
|
||||
return result
|
||||
return result
|
||||
|
||||
|
||||
## Create endpoints
|
||||
@@ -275,34 +181,39 @@ async def get_entity_by_id(
|
||||
async def create_entity(
|
||||
project_id: ProjectExternalIdPathDep,
|
||||
data: Entity,
|
||||
background_tasks: BackgroundTasks,
|
||||
entity_service: EntityServiceV2ExternalDep,
|
||||
search_service: SearchServiceV2ExternalDep,
|
||||
task_scheduler: TaskSchedulerDep,
|
||||
file_service: FileServiceV2ExternalDep,
|
||||
app_config: AppConfigDep,
|
||||
fast: bool = Query(
|
||||
True, description="If true, write quickly and defer indexing to background tasks."
|
||||
),
|
||||
) -> EntityResponseV2:
|
||||
"""Create a new entity.
|
||||
|
||||
Args:
|
||||
data: Entity data to create
|
||||
fast: If True, defer indexing to background tasks
|
||||
|
||||
Returns:
|
||||
Created entity with generated external_id (UUID) and file content
|
||||
"""
|
||||
with logfire.span(
|
||||
"api.request.knowledge.create_entity",
|
||||
entrypoint="api",
|
||||
domain="knowledge",
|
||||
action="create_entity",
|
||||
):
|
||||
logger.info(
|
||||
"API v2 request", endpoint="create_entity", note_type=data.note_type, title=data.title
|
||||
)
|
||||
logger.info(
|
||||
"API v2 request", endpoint="create_entity", note_type=data.note_type, title=data.title
|
||||
)
|
||||
|
||||
# Note writes are now internally consistent before the response returns. We only leave
|
||||
# truly derived work, like semantic vectors, on the async scheduler.
|
||||
write_result = await entity_service.create_entity_with_content(data)
|
||||
entity = write_result.entity
|
||||
await search_service.index_entity(entity, content=write_result.search_content)
|
||||
if fast:
|
||||
entity = await entity_service.fast_write_entity(data)
|
||||
task_scheduler.schedule(
|
||||
"reindex_entity",
|
||||
entity_id=entity.id,
|
||||
project_id=project_id,
|
||||
)
|
||||
else:
|
||||
entity = await entity_service.create_entity(data)
|
||||
await search_service.index_entity(entity)
|
||||
_schedule_vector_sync_if_enabled(
|
||||
task_scheduler=task_scheduler,
|
||||
app_config=app_config,
|
||||
@@ -310,14 +221,18 @@ async def create_entity(
|
||||
project_id=project_id,
|
||||
)
|
||||
|
||||
result = EntityResponseV2.model_validate(entity)
|
||||
# The write service already returns the canonical markdown accepted for this request.
|
||||
result = result.model_copy(update={"content": write_result.content})
|
||||
result = EntityResponseV2.model_validate(entity)
|
||||
if fast:
|
||||
result = result.model_copy(update={"observations": [], "relations": []})
|
||||
|
||||
logger.info(
|
||||
f"API v2 response: endpoint='create_entity' external_id={entity.external_id}, title={result.title}, permalink={result.permalink}, status_code=201"
|
||||
)
|
||||
return result
|
||||
# Always read and return file content
|
||||
content = await file_service.read_file_content(entity.file_path)
|
||||
result = result.model_copy(update={"content": content})
|
||||
|
||||
logger.info(
|
||||
f"API v2 response: endpoint='create_entity' external_id={entity.external_id}, title={result.title}, permalink={result.permalink}, status_code=201"
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
## Update endpoints
|
||||
@@ -327,13 +242,18 @@ async def create_entity(
|
||||
async def update_entity_by_id(
|
||||
data: Entity,
|
||||
response: Response,
|
||||
background_tasks: BackgroundTasks,
|
||||
project_id: ProjectExternalIdPathDep,
|
||||
entity_service: EntityServiceV2ExternalDep,
|
||||
search_service: SearchServiceV2ExternalDep,
|
||||
entity_repository: EntityRepositoryV2ExternalDep,
|
||||
task_scheduler: TaskSchedulerDep,
|
||||
file_service: FileServiceV2ExternalDep,
|
||||
app_config: AppConfigDep,
|
||||
entity_id: str = Path(..., description="Entity external ID (UUID)"),
|
||||
fast: bool = Query(
|
||||
True, description="If true, write quickly and defer indexing to background tasks."
|
||||
),
|
||||
) -> EntityResponseV2:
|
||||
"""Update an entity by external ID.
|
||||
|
||||
@@ -342,35 +262,39 @@ async def update_entity_by_id(
|
||||
Args:
|
||||
entity_id: External ID (UUID string)
|
||||
data: Updated entity data
|
||||
fast: If True, defer indexing to background tasks
|
||||
|
||||
Returns:
|
||||
Updated entity with file content
|
||||
"""
|
||||
with logfire.span(
|
||||
"api.request.knowledge.update_entity",
|
||||
entrypoint="api",
|
||||
domain="knowledge",
|
||||
action="update_entity",
|
||||
):
|
||||
logger.info(f"API v2 request: update_entity_by_id entity_id={entity_id}")
|
||||
logger.info(f"API v2 request: update_entity_by_id entity_id={entity_id}")
|
||||
|
||||
existing = await entity_repository.get_by_external_id(entity_id)
|
||||
created = existing is None
|
||||
# Check if entity exists (external_id is the source of truth for v2)
|
||||
existing = await entity_repository.get_by_external_id(entity_id)
|
||||
created = existing is None
|
||||
|
||||
if fast:
|
||||
entity = await entity_service.fast_write_entity(data, external_id=entity_id)
|
||||
response.status_code = 200 if existing else 201
|
||||
task_scheduler.schedule(
|
||||
"reindex_entity",
|
||||
entity_id=entity.id,
|
||||
project_id=project_id,
|
||||
resolve_relations=created,
|
||||
)
|
||||
else:
|
||||
if existing:
|
||||
write_result = await entity_service.update_entity_with_content(existing, data)
|
||||
entity = write_result.entity
|
||||
# Update the existing entity in-place to avoid path-based duplication
|
||||
entity = await entity_service.update_entity(existing, data)
|
||||
response.status_code = 200
|
||||
else:
|
||||
write_result = await entity_service.create_entity_with_content(data)
|
||||
entity = write_result.entity
|
||||
# Create new entity, then bind external_id to the requested UUID
|
||||
entity = await entity_service.create_entity(data)
|
||||
if entity.external_id != entity_id:
|
||||
entity = await entity_repository.update(
|
||||
entity.id,
|
||||
{"external_id": entity_id},
|
||||
)
|
||||
# external_id fixup only changes the DB row. The file content is unchanged,
|
||||
# so the markdown captured during the write remains valid downstream.
|
||||
if not entity:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
@@ -378,7 +302,7 @@ async def update_entity_by_id(
|
||||
)
|
||||
response.status_code = 201
|
||||
|
||||
await search_service.index_entity(entity, content=write_result.search_content)
|
||||
await search_service.index_entity(entity)
|
||||
_schedule_vector_sync_if_enabled(
|
||||
task_scheduler=task_scheduler,
|
||||
app_config=app_config,
|
||||
@@ -386,31 +310,42 @@ async def update_entity_by_id(
|
||||
project_id=project_id,
|
||||
)
|
||||
|
||||
result = EntityResponseV2.model_validate(entity)
|
||||
result = result.model_copy(update={"content": write_result.content})
|
||||
result = EntityResponseV2.model_validate(entity)
|
||||
if fast:
|
||||
result = result.model_copy(update={"observations": [], "relations": []})
|
||||
|
||||
logger.info(
|
||||
f"API v2 response: external_id={entity_id}, created={created}, status_code={response.status_code}"
|
||||
)
|
||||
return result
|
||||
# Always read and return file content
|
||||
content = await file_service.read_file_content(entity.file_path)
|
||||
result = result.model_copy(update={"content": content})
|
||||
|
||||
logger.info(
|
||||
f"API v2 response: external_id={entity_id}, created={created}, status_code={response.status_code}"
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
@router.patch("/entities/{entity_id}", response_model=EntityResponseV2)
|
||||
async def edit_entity_by_id(
|
||||
data: EditEntityRequest,
|
||||
background_tasks: BackgroundTasks,
|
||||
project_id: ProjectExternalIdPathDep,
|
||||
entity_service: EntityServiceV2ExternalDep,
|
||||
search_service: SearchServiceV2ExternalDep,
|
||||
entity_repository: EntityRepositoryV2ExternalDep,
|
||||
task_scheduler: TaskSchedulerDep,
|
||||
file_service: FileServiceV2ExternalDep,
|
||||
app_config: AppConfigDep,
|
||||
entity_id: str = Path(..., description="Entity external ID (UUID)"),
|
||||
fast: bool = Query(
|
||||
True, description="If true, write quickly and defer indexing to background tasks."
|
||||
),
|
||||
) -> EntityResponseV2:
|
||||
"""Edit an existing entity by external ID using operations like append, prepend, etc.
|
||||
|
||||
Args:
|
||||
entity_id: External ID (UUID string)
|
||||
data: Edit operation details
|
||||
fast: If True, defer indexing to background tasks
|
||||
|
||||
Returns:
|
||||
Updated entity with file content
|
||||
@@ -418,25 +353,36 @@ async def edit_entity_by_id(
|
||||
Raises:
|
||||
HTTPException: 404 if entity not found, 400 if edit fails
|
||||
"""
|
||||
with logfire.span(
|
||||
"api.request.knowledge.edit_entity",
|
||||
entrypoint="api",
|
||||
domain="knowledge",
|
||||
action="edit_entity",
|
||||
):
|
||||
logger.info(
|
||||
f"API v2 request: edit_entity_by_id entity_id={entity_id}, operation='{data.operation}'"
|
||||
logger.info(
|
||||
f"API v2 request: edit_entity_by_id entity_id={entity_id}, operation='{data.operation}'"
|
||||
)
|
||||
|
||||
# Verify entity exists
|
||||
entity = await entity_repository.get_by_external_id(entity_id)
|
||||
if not entity: # pragma: no cover
|
||||
raise HTTPException(
|
||||
status_code=404, detail=f"Entity with external_id '{entity_id}' not found"
|
||||
)
|
||||
|
||||
entity = await entity_repository.get_by_external_id(entity_id)
|
||||
if not entity: # pragma: no cover
|
||||
raise HTTPException(
|
||||
status_code=404, detail=f"Entity with external_id '{entity_id}' not found"
|
||||
try:
|
||||
if fast:
|
||||
updated_entity = await entity_service.fast_edit_entity(
|
||||
entity=entity,
|
||||
operation=data.operation,
|
||||
content=data.content,
|
||||
section=data.section,
|
||||
find_text=data.find_text,
|
||||
expected_replacements=data.expected_replacements,
|
||||
)
|
||||
|
||||
try:
|
||||
task_scheduler.schedule(
|
||||
"reindex_entity",
|
||||
entity_id=updated_entity.id,
|
||||
project_id=project_id,
|
||||
)
|
||||
else:
|
||||
# Edit using the entity's permalink or path
|
||||
identifier = entity.permalink or entity.file_path
|
||||
write_result = await entity_service.edit_entity_with_content(
|
||||
updated_entity = await entity_service.edit_entity(
|
||||
identifier=identifier,
|
||||
operation=data.operation,
|
||||
content=data.content,
|
||||
@@ -444,8 +390,8 @@ async def edit_entity_by_id(
|
||||
find_text=data.find_text,
|
||||
expected_replacements=data.expected_replacements,
|
||||
)
|
||||
updated_entity = write_result.entity
|
||||
await search_service.index_entity(updated_entity, content=write_result.search_content)
|
||||
|
||||
await search_service.index_entity(updated_entity)
|
||||
_schedule_vector_sync_if_enabled(
|
||||
task_scheduler=task_scheduler,
|
||||
app_config=app_config,
|
||||
@@ -453,18 +399,23 @@ async def edit_entity_by_id(
|
||||
project_id=project_id,
|
||||
)
|
||||
|
||||
result = EntityResponseV2.model_validate(updated_entity)
|
||||
result = result.model_copy(update={"content": write_result.content})
|
||||
result = EntityResponseV2.model_validate(updated_entity)
|
||||
if fast:
|
||||
result = result.model_copy(update={"observations": [], "relations": []})
|
||||
|
||||
logger.info(
|
||||
f"API v2 response: external_id={entity_id}, operation='{data.operation}', status_code=200"
|
||||
)
|
||||
# Always read and return file content
|
||||
content = await file_service.read_file_content(updated_entity.file_path)
|
||||
result = result.model_copy(update={"content": content})
|
||||
|
||||
return result
|
||||
logger.info(
|
||||
f"API v2 response: external_id={entity_id}, operation='{data.operation}', status_code=200"
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error editing entity {entity_id}: {e}")
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
return result
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error editing entity {entity_id}: {e}")
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
|
||||
|
||||
## Delete endpoints
|
||||
@@ -472,10 +423,12 @@ async def edit_entity_by_id(
|
||||
|
||||
@router.delete("/entities/{entity_id}", response_model=DeleteEntitiesResponse)
|
||||
async def delete_entity_by_id(
|
||||
background_tasks: BackgroundTasks,
|
||||
project_id: ProjectExternalIdPathDep,
|
||||
entity_service: EntityServiceV2ExternalDep,
|
||||
entity_repository: EntityRepositoryV2ExternalDep,
|
||||
entity_id: str = Path(..., description="Entity external ID (UUID)"),
|
||||
search_service=Depends(lambda: None), # Optional for now
|
||||
) -> DeleteEntitiesResponse:
|
||||
"""Delete an entity by external ID.
|
||||
|
||||
@@ -487,25 +440,23 @@ async def delete_entity_by_id(
|
||||
|
||||
Note: Returns deleted=False if entity doesn't exist (idempotent)
|
||||
"""
|
||||
with logfire.span(
|
||||
"api.request.knowledge.delete_entity",
|
||||
entrypoint="api",
|
||||
domain="knowledge",
|
||||
action="delete_entity",
|
||||
):
|
||||
logger.info(f"API v2 request: delete_entity_by_id entity_id={entity_id}")
|
||||
logger.info(f"API v2 request: delete_entity_by_id entity_id={entity_id}")
|
||||
|
||||
entity = await entity_repository.get_by_external_id(entity_id)
|
||||
if entity is None:
|
||||
logger.info(f"API v2 response: external_id={entity_id} not found, deleted=False")
|
||||
return DeleteEntitiesResponse(deleted=False)
|
||||
entity = await entity_repository.get_by_external_id(entity_id)
|
||||
if entity is None:
|
||||
logger.info(f"API v2 response: external_id={entity_id} not found, deleted=False")
|
||||
return DeleteEntitiesResponse(deleted=False)
|
||||
|
||||
# Delete the entity using internal ID
|
||||
deleted = await entity_service.delete_entity(entity.id)
|
||||
# Delete the entity using internal ID
|
||||
deleted = await entity_service.delete_entity(entity.id)
|
||||
|
||||
logger.info(f"API v2 response: external_id={entity_id}, deleted={deleted}")
|
||||
# Remove from search index if search service available
|
||||
if search_service:
|
||||
background_tasks.add_task(search_service.handle_delete, entity) # pragma: no cover
|
||||
|
||||
return DeleteEntitiesResponse(deleted=deleted)
|
||||
logger.info(f"API v2 response: external_id={entity_id}, deleted={deleted}")
|
||||
|
||||
return DeleteEntitiesResponse(deleted=deleted)
|
||||
|
||||
|
||||
## Move endpoint
|
||||
@@ -514,6 +465,7 @@ async def delete_entity_by_id(
|
||||
@router.put("/entities/{entity_id}/move", response_model=EntityResponseV2)
|
||||
async def move_entity(
|
||||
data: MoveEntityRequestV2,
|
||||
background_tasks: BackgroundTasks,
|
||||
project_id: ProjectExternalIdPathDep,
|
||||
entity_service: EntityServiceV2ExternalDep,
|
||||
entity_repository: EntityRepositoryV2ExternalDep,
|
||||
@@ -536,58 +488,48 @@ async def move_entity(
|
||||
Returns:
|
||||
Updated entity with new file path
|
||||
"""
|
||||
with logfire.span(
|
||||
"api.request.knowledge.move_entity",
|
||||
entrypoint="api",
|
||||
domain="knowledge",
|
||||
action="move_entity",
|
||||
):
|
||||
logger.info(
|
||||
f"API v2 request: move_entity entity_id={entity_id}, destination='{data.destination_path}'"
|
||||
logger.info(
|
||||
f"API v2 request: move_entity entity_id={entity_id}, destination='{data.destination_path}'"
|
||||
)
|
||||
|
||||
try:
|
||||
# First, get the entity by external_id to verify it exists
|
||||
entity = await entity_repository.get_by_external_id(entity_id)
|
||||
if not entity: # pragma: no cover
|
||||
raise HTTPException(
|
||||
status_code=404, detail=f"Entity with external_id '{entity_id}' not found"
|
||||
)
|
||||
|
||||
# Move the entity using its current file path as identifier
|
||||
moved_entity = await entity_service.move_entity(
|
||||
identifier=entity.file_path, # Use file path for resolution
|
||||
destination_path=data.destination_path,
|
||||
project_config=project_config,
|
||||
app_config=app_config,
|
||||
)
|
||||
|
||||
try:
|
||||
# First, get the entity by external_id to verify it exists
|
||||
entity = await entity_repository.get_by_external_id(entity_id)
|
||||
if not entity: # pragma: no cover
|
||||
raise HTTPException(
|
||||
status_code=404, detail=f"Entity with external_id '{entity_id}' not found"
|
||||
)
|
||||
|
||||
# Move the entity using its current file path as identifier
|
||||
moved_entity = await entity_service.move_entity(
|
||||
identifier=entity.file_path, # Use file path for resolution
|
||||
destination_path=data.destination_path,
|
||||
project_config=project_config,
|
||||
# Reindex at new location
|
||||
reindexed_entity = await entity_service.link_resolver.resolve_link(data.destination_path)
|
||||
if reindexed_entity:
|
||||
await search_service.index_entity(reindexed_entity)
|
||||
_schedule_vector_sync_if_enabled(
|
||||
task_scheduler=task_scheduler,
|
||||
app_config=app_config,
|
||||
entity_id=reindexed_entity.id,
|
||||
project_id=project_id,
|
||||
)
|
||||
|
||||
# Reindex at new location
|
||||
reindexed_entity = await entity_service.link_resolver.resolve_link(
|
||||
data.destination_path
|
||||
)
|
||||
if reindexed_entity:
|
||||
await search_service.index_entity(reindexed_entity)
|
||||
_schedule_vector_sync_if_enabled(
|
||||
task_scheduler=task_scheduler,
|
||||
app_config=app_config,
|
||||
entity_id=reindexed_entity.id,
|
||||
project_id=project_id,
|
||||
)
|
||||
result = EntityResponseV2.model_validate(moved_entity)
|
||||
|
||||
result = EntityResponseV2.model_validate(moved_entity)
|
||||
logger.info(f"API v2 response: moved external_id={entity_id} to '{data.destination_path}'")
|
||||
|
||||
logger.info(
|
||||
f"API v2 response: moved external_id={entity_id} to '{data.destination_path}'"
|
||||
)
|
||||
return result
|
||||
|
||||
return result
|
||||
|
||||
except HTTPException: # pragma: no cover
|
||||
raise # pragma: no cover
|
||||
except Exception as e:
|
||||
logger.error(f"Error moving entity: {e}")
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
except HTTPException: # pragma: no cover
|
||||
raise # pragma: no cover
|
||||
except Exception as e:
|
||||
logger.error(f"Error moving entity: {e}")
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
|
||||
|
||||
## Move directory endpoint
|
||||
@@ -596,6 +538,7 @@ async def move_entity(
|
||||
@router.post("/move-directory", response_model=DirectoryMoveResult)
|
||||
async def move_directory(
|
||||
data: MoveDirectoryRequestV2,
|
||||
background_tasks: BackgroundTasks,
|
||||
project_id: ProjectExternalIdPathDep,
|
||||
entity_service: EntityServiceV2ExternalDep,
|
||||
project_config: ProjectConfigV2ExternalDep,
|
||||
@@ -616,46 +559,40 @@ async def move_directory(
|
||||
Returns:
|
||||
DirectoryMoveResult with counts and details of moved files
|
||||
"""
|
||||
with logfire.span(
|
||||
"api.request.knowledge.move_directory",
|
||||
entrypoint="api",
|
||||
domain="knowledge",
|
||||
action="move_directory",
|
||||
):
|
||||
logger.info(
|
||||
f"API v2 request: move_directory source='{data.source_directory}', destination='{data.destination_directory}'"
|
||||
logger.info(
|
||||
f"API v2 request: move_directory source='{data.source_directory}', destination='{data.destination_directory}'"
|
||||
)
|
||||
|
||||
try:
|
||||
# Move the directory using the service
|
||||
result = await entity_service.move_directory(
|
||||
source_directory=data.source_directory,
|
||||
destination_directory=data.destination_directory,
|
||||
project_config=project_config,
|
||||
app_config=app_config,
|
||||
)
|
||||
|
||||
try:
|
||||
# Move the directory using the service
|
||||
result = await entity_service.move_directory(
|
||||
source_directory=data.source_directory,
|
||||
destination_directory=data.destination_directory,
|
||||
project_config=project_config,
|
||||
app_config=app_config,
|
||||
)
|
||||
# Reindex moved entities
|
||||
for file_path in result.moved_files:
|
||||
entity = await entity_service.link_resolver.resolve_link(file_path)
|
||||
if entity:
|
||||
await search_service.index_entity(entity)
|
||||
_schedule_vector_sync_if_enabled(
|
||||
task_scheduler=task_scheduler,
|
||||
app_config=app_config,
|
||||
entity_id=entity.id,
|
||||
project_id=project_id,
|
||||
)
|
||||
|
||||
# Reindex moved entities
|
||||
for file_path in result.moved_files:
|
||||
entity = await entity_service.link_resolver.resolve_link(file_path)
|
||||
if entity:
|
||||
await search_service.index_entity(entity)
|
||||
_schedule_vector_sync_if_enabled(
|
||||
task_scheduler=task_scheduler,
|
||||
app_config=app_config,
|
||||
entity_id=entity.id,
|
||||
project_id=project_id,
|
||||
)
|
||||
logger.info(
|
||||
f"API v2 response: move_directory "
|
||||
f"total={result.total_files}, success={result.successful_moves}, failed={result.failed_moves}"
|
||||
)
|
||||
return result
|
||||
|
||||
logger.info(
|
||||
f"API v2 response: move_directory "
|
||||
f"total={result.total_files}, success={result.successful_moves}, failed={result.failed_moves}"
|
||||
)
|
||||
return result
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error moving directory: {e}")
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
except Exception as e:
|
||||
logger.error(f"Error moving directory: {e}")
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
|
||||
|
||||
## Delete directory endpoint
|
||||
@@ -680,26 +617,20 @@ async def delete_directory(
|
||||
Returns:
|
||||
DirectoryDeleteResult with counts and details of deleted files
|
||||
"""
|
||||
with logfire.span(
|
||||
"api.request.knowledge.delete_directory",
|
||||
entrypoint="api",
|
||||
domain="knowledge",
|
||||
action="delete_directory",
|
||||
):
|
||||
logger.info(f"API v2 request: delete_directory directory='{data.directory}'")
|
||||
logger.info(f"API v2 request: delete_directory directory='{data.directory}'")
|
||||
|
||||
try:
|
||||
# Delete the directory using the service
|
||||
result = await entity_service.delete_directory(
|
||||
directory=data.directory,
|
||||
)
|
||||
try:
|
||||
# Delete the directory using the service
|
||||
result = await entity_service.delete_directory(
|
||||
directory=data.directory,
|
||||
)
|
||||
|
||||
logger.info(
|
||||
f"API v2 response: delete_directory "
|
||||
f"total={result.total_files}, success={result.successful_deletes}, failed={result.failed_deletes}"
|
||||
)
|
||||
return result
|
||||
logger.info(
|
||||
f"API v2 response: delete_directory "
|
||||
f"total={result.total_files}, success={result.successful_deletes}, failed={result.failed_deletes}"
|
||||
)
|
||||
return result
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting directory: {e}")
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting directory: {e}")
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
|
||||
@@ -9,7 +9,6 @@ from typing import Annotated, Optional
|
||||
from fastapi import APIRouter, Query, Path
|
||||
from loguru import logger
|
||||
|
||||
import logfire
|
||||
from basic_memory.deps import ContextServiceV2ExternalDep, EntityRepositoryV2ExternalDep
|
||||
from basic_memory.schemas.base import TimeFrame, parse_timeframe
|
||||
from basic_memory.schemas.memory import (
|
||||
@@ -51,55 +50,30 @@ async def recent(
|
||||
Returns:
|
||||
GraphContext with recent activity and related entities
|
||||
"""
|
||||
with logfire.span(
|
||||
"api.request.memory.recent_activity",
|
||||
entrypoint="api",
|
||||
domain="memory",
|
||||
action="recent_activity",
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
):
|
||||
types = (
|
||||
[SearchItemType.ENTITY, SearchItemType.RELATION, SearchItemType.OBSERVATION]
|
||||
if not type
|
||||
else type
|
||||
)
|
||||
# return all types by default
|
||||
types = (
|
||||
[SearchItemType.ENTITY, SearchItemType.RELATION, SearchItemType.OBSERVATION]
|
||||
if not type
|
||||
else type
|
||||
)
|
||||
|
||||
logger.debug(
|
||||
f"V2 Getting recent context for project {project_id}: `{types}` depth: `{depth}` timeframe: `{timeframe}` page: `{page}` page_size: `{page_size}` max_related: `{max_related}`"
|
||||
)
|
||||
since = parse_timeframe(timeframe)
|
||||
limit = page_size
|
||||
offset = (page - 1) * page_size
|
||||
logger.debug(
|
||||
f"V2 Getting recent context for project {project_id}: `{types}` depth: `{depth}` timeframe: `{timeframe}` page: `{page}` page_size: `{page_size}` max_related: `{max_related}`"
|
||||
)
|
||||
# Parse timeframe
|
||||
since = parse_timeframe(timeframe)
|
||||
limit = page_size
|
||||
offset = (page - 1) * page_size
|
||||
|
||||
with logfire.span(
|
||||
"api.memory.recent_activity.build_context",
|
||||
domain="memory",
|
||||
action="recent_activity",
|
||||
phase="build_context",
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
):
|
||||
context = await context_service.build_context(
|
||||
types=types,
|
||||
depth=depth,
|
||||
since=since,
|
||||
limit=limit,
|
||||
offset=offset,
|
||||
max_related=max_related,
|
||||
)
|
||||
with logfire.span(
|
||||
"api.memory.recent_activity.shape_response",
|
||||
domain="memory",
|
||||
action="recent_activity",
|
||||
phase="shape_response",
|
||||
result_count=len(context.results),
|
||||
):
|
||||
recent_context = await to_graph_context(
|
||||
context, entity_repository=entity_repository, page=page, page_size=page_size
|
||||
)
|
||||
logger.debug(f"V2 Recent context: {recent_context.model_dump_json()}")
|
||||
return recent_context
|
||||
# Build context
|
||||
context = await context_service.build_context(
|
||||
types=types, depth=depth, since=since, limit=limit, offset=offset, max_related=max_related
|
||||
)
|
||||
recent_context = await to_graph_context(
|
||||
context, entity_repository=entity_repository, page=page, page_size=page_size
|
||||
)
|
||||
logger.debug(f"V2 Recent context: {recent_context.model_dump_json()}")
|
||||
return recent_context
|
||||
|
||||
|
||||
# get_memory_context needs to be declared last so other paths can match
|
||||
@@ -137,46 +111,20 @@ async def get_memory_context(
|
||||
Returns:
|
||||
GraphContext with the entity and its related context
|
||||
"""
|
||||
with logfire.span(
|
||||
"api.request.memory.build_context",
|
||||
entrypoint="api",
|
||||
domain="memory",
|
||||
action="build_context",
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
):
|
||||
logger.debug(
|
||||
f"V2 Getting context for project {project_id}, URI: `{uri}` depth: `{depth}` timeframe: `{timeframe}` page: `{page}` page_size: `{page_size}` max_related: `{max_related}`"
|
||||
)
|
||||
memory_url = normalize_memory_url(uri)
|
||||
logger.debug(
|
||||
f"V2 Getting context for project {project_id}, URI: `{uri}` depth: `{depth}` timeframe: `{timeframe}` page: `{page}` page_size: `{page_size}` max_related: `{max_related}`"
|
||||
)
|
||||
memory_url = normalize_memory_url(uri)
|
||||
|
||||
since = parse_timeframe(timeframe) if timeframe else None
|
||||
limit = page_size
|
||||
offset = (page - 1) * page_size
|
||||
# Parse timeframe
|
||||
since = parse_timeframe(timeframe) if timeframe else None
|
||||
limit = page_size
|
||||
offset = (page - 1) * page_size
|
||||
|
||||
with logfire.span(
|
||||
"api.memory.build_context.build_context",
|
||||
domain="memory",
|
||||
action="build_context",
|
||||
phase="build_context",
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
):
|
||||
context = await context_service.build_context(
|
||||
memory_url,
|
||||
depth=depth,
|
||||
since=since,
|
||||
limit=limit,
|
||||
offset=offset,
|
||||
max_related=max_related,
|
||||
)
|
||||
with logfire.span(
|
||||
"api.memory.build_context.shape_response",
|
||||
domain="memory",
|
||||
action="build_context",
|
||||
phase="shape_response",
|
||||
result_count=len(context.results),
|
||||
):
|
||||
return await to_graph_context(
|
||||
context, entity_repository=entity_repository, page=page, page_size=page_size
|
||||
)
|
||||
# Build context
|
||||
context = await context_service.build_context(
|
||||
memory_url, depth=depth, since=since, limit=limit, offset=offset, max_related=max_related
|
||||
)
|
||||
return await to_graph_context(
|
||||
context, entity_repository=entity_repository, page=page, page_size=page_size
|
||||
)
|
||||
|
||||
@@ -11,7 +11,7 @@ Key improvements:
|
||||
"""
|
||||
|
||||
import os
|
||||
from typing import Literal, Optional
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Body, Query, Path
|
||||
from loguru import logger
|
||||
@@ -25,8 +25,6 @@ from basic_memory.deps import (
|
||||
ProjectExternalIdPathDep,
|
||||
)
|
||||
from basic_memory.schemas import SyncReportResponse
|
||||
from basic_memory.models import Project
|
||||
from basic_memory.repository.project_repository import ProjectRepository
|
||||
from basic_memory.schemas.project_info import (
|
||||
ProjectItem,
|
||||
ProjectList,
|
||||
@@ -38,71 +36,6 @@ from basic_memory.schemas.v2 import ProjectResolveRequest, ProjectResolveRespons
|
||||
from basic_memory.utils import normalize_project_path, generate_permalink
|
||||
|
||||
router = APIRouter(prefix="/projects", tags=["project_management-v2"])
|
||||
ProjectResolveMethod = Literal["external_id", "name", "permalink"]
|
||||
|
||||
|
||||
def _split_qualified_project_identifier(identifier: str) -> tuple[str | None, str]:
|
||||
"""Split ``<workspace>/<project>`` identifiers while preserving plain project names."""
|
||||
cleaned = identifier.strip()
|
||||
if "/" not in cleaned:
|
||||
return None, cleaned
|
||||
|
||||
workspace_identifier, project_identifier = cleaned.split("/", 1)
|
||||
if not workspace_identifier or not project_identifier:
|
||||
return None, cleaned
|
||||
return workspace_identifier, project_identifier
|
||||
|
||||
|
||||
async def _resolve_project_identifier_candidate(
|
||||
project_repository: ProjectRepository,
|
||||
identifier: str,
|
||||
) -> tuple[Project | None, ProjectResolveMethod]:
|
||||
"""Resolve one project identifier candidate and report the matching method."""
|
||||
identifier_permalink = generate_permalink(identifier)
|
||||
|
||||
project = await project_repository.get_by_external_id(identifier)
|
||||
if project:
|
||||
return project, "external_id"
|
||||
|
||||
project = await project_repository.get_by_permalink(identifier_permalink)
|
||||
if project:
|
||||
return project, "permalink"
|
||||
|
||||
project = await project_repository.get_by_name_case_insensitive(identifier)
|
||||
if project:
|
||||
return project, "name" # pragma: no cover
|
||||
|
||||
return None, "name"
|
||||
|
||||
|
||||
async def _resolve_project_identifier(
|
||||
project_repository: ProjectRepository,
|
||||
identifier: str,
|
||||
) -> tuple[Project | None, ProjectResolveMethod]:
|
||||
"""Resolve exact identifiers first, then accepted workspace-qualified forms."""
|
||||
project, resolution_method = await _resolve_project_identifier_candidate(
|
||||
project_repository,
|
||||
identifier,
|
||||
)
|
||||
if project:
|
||||
return project, resolution_method
|
||||
|
||||
workspace_identifier, project_identifier = _split_qualified_project_identifier(identifier)
|
||||
if workspace_identifier is None:
|
||||
return None, resolution_method
|
||||
|
||||
# Trigger: an MCP disambiguation error suggested ``workspace/project``.
|
||||
# Why: request routing already selected the workspace/tenant; this endpoint
|
||||
# only needs the project segment to validate the active project.
|
||||
# Outcome: models can follow the hint verbatim instead of looping on a 404.
|
||||
project, resolution_method = await _resolve_project_identifier_candidate(
|
||||
project_repository,
|
||||
project_identifier,
|
||||
)
|
||||
if project:
|
||||
return project, resolution_method
|
||||
|
||||
return None, resolution_method
|
||||
|
||||
|
||||
@router.get("/", response_model=ProjectList)
|
||||
@@ -314,10 +247,28 @@ async def resolve_project_identifier(
|
||||
"""
|
||||
logger.info(f"API v2 request: resolve_project_identifier for '{data.identifier}'")
|
||||
|
||||
project, resolution_method = await _resolve_project_identifier(
|
||||
project_repository,
|
||||
data.identifier,
|
||||
)
|
||||
# Generate permalink for comparison
|
||||
identifier_permalink = generate_permalink(data.identifier)
|
||||
|
||||
resolution_method = "name"
|
||||
project = None
|
||||
|
||||
# Try external_id first (UUID format)
|
||||
project = await project_repository.get_by_external_id(data.identifier)
|
||||
if project:
|
||||
resolution_method = "external_id"
|
||||
|
||||
# If not found by external_id, try by permalink (exact match)
|
||||
if not project:
|
||||
project = await project_repository.get_by_permalink(identifier_permalink)
|
||||
if project:
|
||||
resolution_method = "permalink"
|
||||
|
||||
# If not found by permalink, try case-insensitive name search
|
||||
if not project:
|
||||
project = await project_repository.get_by_name_case_insensitive(data.identifier)
|
||||
if project:
|
||||
resolution_method = "name" # pragma: no cover
|
||||
|
||||
if not project:
|
||||
raise HTTPException(status_code=404, detail=f"Project not found: '{data.identifier}'")
|
||||
|
||||
@@ -6,7 +6,6 @@ have entity IDs in URLs - they generate formatted prompts from queries.
|
||||
"""
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
from fastapi import APIRouter, HTTPException, status, Path
|
||||
from loguru import logger
|
||||
|
||||
@@ -60,7 +59,6 @@ async def continue_conversation(
|
||||
|
||||
# Initialize search results
|
||||
search_results = []
|
||||
hierarchical_results_for_count = []
|
||||
|
||||
# Get data needed for template
|
||||
if request.topic:
|
||||
@@ -93,8 +91,7 @@ async def continue_conversation(
|
||||
# Limit to a reasonable number of total results
|
||||
all_hierarchical_results = all_hierarchical_results[:10]
|
||||
|
||||
hierarchical_results_for_count = all_hierarchical_results
|
||||
template_context: dict[str, Any] = {
|
||||
template_context = {
|
||||
"topic": request.topic,
|
||||
"timeframe": request.timeframe,
|
||||
"hierarchical_results": all_hierarchical_results,
|
||||
@@ -113,7 +110,6 @@ async def continue_conversation(
|
||||
|
||||
hierarchical_results = recent_context.results[:5] # Limit to top 5 recent items
|
||||
|
||||
hierarchical_results_for_count = hierarchical_results
|
||||
template_context = {
|
||||
"topic": f"Recent Activity from ({request.timeframe})",
|
||||
"timeframe": request.timeframe,
|
||||
@@ -133,6 +129,9 @@ async def continue_conversation(
|
||||
relation_count = 0
|
||||
entity_count = 0
|
||||
|
||||
# Get the hierarchical results from the template context
|
||||
hierarchical_results_for_count = template_context.get("hierarchical_results", [])
|
||||
|
||||
# For topic-based search
|
||||
if request.topic:
|
||||
for item in hierarchical_results_for_count:
|
||||
@@ -160,24 +159,29 @@ async def continue_conversation(
|
||||
elif related.type == "entity": # pragma: no cover
|
||||
entity_count += 1 # pragma: no cover
|
||||
|
||||
prompt_metadata = PromptMetadata(
|
||||
query=request.topic,
|
||||
timeframe=request.timeframe,
|
||||
search_count=len(search_results) if request.topic else 0,
|
||||
context_count=len(hierarchical_results_for_count),
|
||||
observation_count=observation_count,
|
||||
relation_count=relation_count,
|
||||
total_items=(
|
||||
# Build metadata
|
||||
metadata = {
|
||||
"query": request.topic,
|
||||
"timeframe": request.timeframe,
|
||||
"search_count": len(search_results)
|
||||
if request.topic
|
||||
else 0, # Original search results count
|
||||
"context_count": len(hierarchical_results_for_count),
|
||||
"observation_count": observation_count,
|
||||
"relation_count": relation_count,
|
||||
"total_items": (
|
||||
len(hierarchical_results_for_count)
|
||||
+ observation_count
|
||||
+ relation_count
|
||||
+ entity_count
|
||||
),
|
||||
search_limit=request.search_items_limit,
|
||||
context_depth=request.depth,
|
||||
related_limit=request.related_items_limit,
|
||||
generated_at=datetime.now(timezone.utc).isoformat(),
|
||||
)
|
||||
"search_limit": request.search_items_limit,
|
||||
"context_depth": request.depth,
|
||||
"related_limit": request.related_items_limit,
|
||||
"generated_at": datetime.now(timezone.utc).isoformat(),
|
||||
}
|
||||
|
||||
prompt_metadata = PromptMetadata(**metadata)
|
||||
|
||||
return PromptResponse(
|
||||
prompt=rendered_prompt, context=template_context, metadata=prompt_metadata
|
||||
@@ -225,7 +229,7 @@ async def search_prompt(
|
||||
results = await search_service.search(query, limit=limit, offset=offset)
|
||||
search_results = await to_search_results(entity_service, results)
|
||||
|
||||
template_context: dict[str, Any] = {
|
||||
template_context = {
|
||||
"query": request.query,
|
||||
"timeframe": request.timeframe,
|
||||
"results": search_results,
|
||||
@@ -237,19 +241,22 @@ async def search_prompt(
|
||||
# Render template
|
||||
rendered_prompt = await template_loader.render("prompts/search.hbs", template_context)
|
||||
|
||||
prompt_metadata = PromptMetadata(
|
||||
query=request.query,
|
||||
timeframe=request.timeframe,
|
||||
search_count=len(search_results),
|
||||
context_count=len(search_results),
|
||||
observation_count=0,
|
||||
relation_count=0,
|
||||
total_items=len(search_results),
|
||||
search_limit=limit,
|
||||
context_depth=0,
|
||||
related_limit=0,
|
||||
generated_at=datetime.now(timezone.utc).isoformat(),
|
||||
)
|
||||
# Build metadata
|
||||
metadata = {
|
||||
"query": request.query,
|
||||
"timeframe": request.timeframe,
|
||||
"search_count": len(search_results),
|
||||
"context_count": len(search_results),
|
||||
"observation_count": 0, # Search results don't include observations
|
||||
"relation_count": 0, # Search results don't include relations
|
||||
"total_items": len(search_results),
|
||||
"search_limit": limit,
|
||||
"context_depth": 0, # No context depth for basic search
|
||||
"related_limit": 0, # No related items for basic search
|
||||
"generated_at": datetime.now(timezone.utc).isoformat(),
|
||||
}
|
||||
|
||||
prompt_metadata = PromptMetadata(**metadata)
|
||||
|
||||
return PromptResponse(
|
||||
prompt=rendered_prompt, context=template_context, metadata=prompt_metadata
|
||||
|
||||
@@ -15,7 +15,6 @@ from pathlib import Path as PathLib
|
||||
from fastapi import APIRouter, HTTPException, Response, Path
|
||||
from loguru import logger
|
||||
|
||||
import logfire
|
||||
from basic_memory.deps import (
|
||||
ProjectConfigV2ExternalDep,
|
||||
FileServiceV2ExternalDep,
|
||||
@@ -56,62 +55,36 @@ async def get_resource_content(
|
||||
Raises:
|
||||
HTTPException: 404 if entity or file not found
|
||||
"""
|
||||
with logfire.span(
|
||||
"api.request.resource.get_content",
|
||||
entrypoint="api",
|
||||
domain="resource",
|
||||
action="get_content",
|
||||
):
|
||||
logger.debug(f"V2 Getting content for project {project_id}, entity_id: {entity_id}")
|
||||
logger.debug(f"V2 Getting content for project {project_id}, entity_id: {entity_id}")
|
||||
|
||||
with logfire.span(
|
||||
"api.resource.get_content.load_entity",
|
||||
domain="resource",
|
||||
action="get_content",
|
||||
phase="load_entity",
|
||||
):
|
||||
entity = await entity_repository.get_by_external_id(entity_id)
|
||||
if not entity:
|
||||
raise HTTPException(status_code=404, detail=f"Entity {entity_id} not found")
|
||||
# Get entity by external_id
|
||||
entity = await entity_repository.get_by_external_id(entity_id)
|
||||
if not entity:
|
||||
raise HTTPException(status_code=404, detail=f"Entity {entity_id} not found")
|
||||
|
||||
with logfire.span(
|
||||
"api.resource.get_content.validate_path",
|
||||
domain="resource",
|
||||
action="get_content",
|
||||
phase="validate_path",
|
||||
):
|
||||
project_path = PathLib(config.home)
|
||||
if not validate_project_path(entity.file_path, project_path):
|
||||
logger.error( # pragma: no cover
|
||||
f"Invalid file path in entity {entity.id}: {entity.file_path}"
|
||||
)
|
||||
raise HTTPException( # pragma: no cover
|
||||
status_code=500,
|
||||
detail="Entity contains invalid file path",
|
||||
)
|
||||
# Validate entity file path to prevent path traversal
|
||||
project_path = PathLib(config.home)
|
||||
if not validate_project_path(entity.file_path, project_path):
|
||||
logger.error( # pragma: no cover
|
||||
f"Invalid file path in entity {entity.id}: {entity.file_path}"
|
||||
)
|
||||
raise HTTPException( # pragma: no cover
|
||||
status_code=500,
|
||||
detail="Entity contains invalid file path",
|
||||
)
|
||||
|
||||
with logfire.span(
|
||||
"api.resource.get_content.ensure_exists",
|
||||
domain="resource",
|
||||
action="get_content",
|
||||
phase="ensure_exists",
|
||||
):
|
||||
if not await file_service.exists(entity.file_path):
|
||||
raise HTTPException( # pragma: no cover
|
||||
status_code=404,
|
||||
detail=f"File not found: {entity.file_path}",
|
||||
)
|
||||
# Check file exists via file_service (for cloud compatibility)
|
||||
if not await file_service.exists(entity.file_path):
|
||||
raise HTTPException( # pragma: no cover
|
||||
status_code=404,
|
||||
detail=f"File not found: {entity.file_path}",
|
||||
)
|
||||
|
||||
with logfire.span(
|
||||
"api.resource.get_content.read_content",
|
||||
domain="resource",
|
||||
action="get_content",
|
||||
phase="read_content",
|
||||
):
|
||||
content = await file_service.read_file_bytes(entity.file_path)
|
||||
content_type = file_service.content_type(entity.file_path)
|
||||
# Read content via file_service as bytes (works with both local and S3)
|
||||
content = await file_service.read_file_bytes(entity.file_path)
|
||||
content_type = file_service.content_type(entity.file_path)
|
||||
|
||||
return Response(content=content, media_type=content_type)
|
||||
return Response(content=content, media_type=content_type)
|
||||
|
||||
|
||||
@router.post("", response_model=ResourceResponse)
|
||||
@@ -139,94 +112,74 @@ async def create_resource(
|
||||
Raises:
|
||||
HTTPException: 400 for invalid file paths, 409 if file already exists
|
||||
"""
|
||||
with logfire.span(
|
||||
"api.request.resource.create",
|
||||
entrypoint="api",
|
||||
domain="resource",
|
||||
action="create",
|
||||
):
|
||||
try:
|
||||
# Validate path to prevent path traversal attacks
|
||||
project_path = PathLib(config.home)
|
||||
if not validate_project_path(data.file_path, project_path):
|
||||
logger.warning(
|
||||
f"Invalid file path attempted: {data.file_path} in project {config.name}"
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Invalid file path: {data.file_path}. "
|
||||
"Path must be relative and stay within project boundaries.",
|
||||
)
|
||||
|
||||
existing_entity = await entity_repository.get_by_file_path(data.file_path)
|
||||
if existing_entity:
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail=f"Resource already exists at {data.file_path} with entity_id {existing_entity.external_id}. "
|
||||
f"Use PUT /resource/{existing_entity.external_id} to update it.",
|
||||
)
|
||||
|
||||
with logfire.span(
|
||||
"api.resource.create.write_file",
|
||||
domain="resource",
|
||||
action="create",
|
||||
phase="write_file",
|
||||
):
|
||||
await file_service.ensure_directory(PathLib(data.file_path).parent)
|
||||
checksum = await file_service.write_file(data.file_path, data.content)
|
||||
|
||||
with logfire.span(
|
||||
"api.resource.create.read_metadata",
|
||||
domain="resource",
|
||||
action="create",
|
||||
phase="read_metadata",
|
||||
):
|
||||
file_metadata = await file_service.get_file_metadata(data.file_path)
|
||||
|
||||
file_name = PathLib(data.file_path).name
|
||||
content_type = file_service.content_type(data.file_path)
|
||||
note_type = "canvas" if data.file_path.endswith(".canvas") else "file"
|
||||
|
||||
entity = EntityModel(
|
||||
external_id=str(uuid.uuid4()),
|
||||
title=file_name,
|
||||
note_type=note_type,
|
||||
content_type=content_type,
|
||||
file_path=data.file_path,
|
||||
checksum=checksum,
|
||||
created_at=file_metadata.created_at,
|
||||
updated_at=file_metadata.modified_at,
|
||||
try:
|
||||
# Validate path to prevent path traversal attacks
|
||||
project_path = PathLib(config.home)
|
||||
if not validate_project_path(data.file_path, project_path):
|
||||
logger.warning(
|
||||
f"Invalid file path attempted: {data.file_path} in project {config.name}"
|
||||
)
|
||||
with logfire.span(
|
||||
"api.resource.create.upsert_entity",
|
||||
domain="resource",
|
||||
action="create",
|
||||
phase="upsert_entity",
|
||||
):
|
||||
entity = await entity_repository.add(entity)
|
||||
|
||||
with logfire.span(
|
||||
"api.resource.create.search_index",
|
||||
domain="resource",
|
||||
action="create",
|
||||
phase="search_index",
|
||||
):
|
||||
await search_service.index_entity(entity)
|
||||
|
||||
return ResourceResponse(
|
||||
entity_id=entity.id,
|
||||
external_id=entity.external_id,
|
||||
file_path=data.file_path,
|
||||
checksum=checksum,
|
||||
size=file_metadata.size,
|
||||
created_at=file_metadata.created_at.timestamp(),
|
||||
modified_at=file_metadata.modified_at.timestamp(),
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Invalid file path: {data.file_path}. "
|
||||
"Path must be relative and stay within project boundaries.",
|
||||
)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e: # pragma: no cover
|
||||
logger.error(f"Error creating resource {data.file_path}: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Failed to create resource: {str(e)}")
|
||||
|
||||
# Check if entity already exists
|
||||
existing_entity = await entity_repository.get_by_file_path(data.file_path)
|
||||
if existing_entity:
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail=f"Resource already exists at {data.file_path} with entity_id {existing_entity.external_id}. "
|
||||
f"Use PUT /resource/{existing_entity.external_id} to update it.",
|
||||
)
|
||||
|
||||
# Cloud compatibility: avoid assuming a local filesystem path.
|
||||
# Delegate directory creation + writes to FileService (local or S3).
|
||||
await file_service.ensure_directory(PathLib(data.file_path).parent)
|
||||
checksum = await file_service.write_file(data.file_path, data.content)
|
||||
|
||||
# Get file info
|
||||
file_metadata = await file_service.get_file_metadata(data.file_path)
|
||||
|
||||
# Determine file details
|
||||
file_name = PathLib(data.file_path).name
|
||||
content_type = file_service.content_type(data.file_path)
|
||||
note_type = "canvas" if data.file_path.endswith(".canvas") else "file"
|
||||
|
||||
# Create a new entity model
|
||||
# Explicitly set external_id to ensure NOT NULL constraint is satisfied (fixes #512)
|
||||
entity = EntityModel(
|
||||
external_id=str(uuid.uuid4()),
|
||||
title=file_name,
|
||||
note_type=note_type,
|
||||
content_type=content_type,
|
||||
file_path=data.file_path,
|
||||
checksum=checksum,
|
||||
created_at=file_metadata.created_at,
|
||||
updated_at=file_metadata.modified_at,
|
||||
)
|
||||
entity = await entity_repository.add(entity)
|
||||
|
||||
# Index the file for search
|
||||
await search_service.index_entity(entity) # pyright: ignore
|
||||
|
||||
# Return success response
|
||||
return ResourceResponse(
|
||||
entity_id=entity.id,
|
||||
external_id=entity.external_id,
|
||||
file_path=data.file_path,
|
||||
checksum=checksum,
|
||||
size=file_metadata.size,
|
||||
created_at=file_metadata.created_at.timestamp(),
|
||||
modified_at=file_metadata.modified_at.timestamp(),
|
||||
)
|
||||
except HTTPException:
|
||||
# Re-raise HTTP exceptions without wrapping
|
||||
raise
|
||||
except Exception as e: # pragma: no cover
|
||||
logger.error(f"Error creating resource {data.file_path}: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Failed to create resource: {str(e)}")
|
||||
|
||||
|
||||
@router.put("/{entity_id}", response_model=ResourceResponse)
|
||||
@@ -258,96 +211,79 @@ async def update_resource(
|
||||
Raises:
|
||||
HTTPException: 404 if entity not found, 400 for invalid paths
|
||||
"""
|
||||
with logfire.span(
|
||||
"api.request.resource.update",
|
||||
entrypoint="api",
|
||||
domain="resource",
|
||||
action="update",
|
||||
):
|
||||
try:
|
||||
entity = await entity_repository.get_by_external_id(entity_id)
|
||||
if not entity:
|
||||
raise HTTPException(status_code=404, detail=f"Entity {entity_id} not found")
|
||||
try:
|
||||
# Get existing entity by external_id
|
||||
entity = await entity_repository.get_by_external_id(entity_id)
|
||||
if not entity:
|
||||
raise HTTPException(status_code=404, detail=f"Entity {entity_id} not found")
|
||||
|
||||
target_file_path = data.file_path if data.file_path else entity.file_path
|
||||
# Determine target file path
|
||||
target_file_path = data.file_path if data.file_path else entity.file_path
|
||||
|
||||
project_path = PathLib(config.home)
|
||||
if not validate_project_path(target_file_path, project_path):
|
||||
logger.warning(
|
||||
f"Invalid file path attempted: {target_file_path} in project {config.name}"
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Invalid file path: {target_file_path}. "
|
||||
"Path must be relative and stay within project boundaries.",
|
||||
)
|
||||
|
||||
with logfire.span(
|
||||
"api.resource.update.write_file",
|
||||
domain="resource",
|
||||
action="update",
|
||||
phase="write_file",
|
||||
):
|
||||
if data.file_path and data.file_path != entity.file_path:
|
||||
await file_service.ensure_directory(PathLib(target_file_path).parent)
|
||||
if await file_service.exists(entity.file_path):
|
||||
await file_service.delete_file(entity.file_path)
|
||||
else:
|
||||
await file_service.ensure_directory(PathLib(target_file_path).parent)
|
||||
|
||||
checksum = await file_service.write_file(target_file_path, data.content)
|
||||
|
||||
with logfire.span(
|
||||
"api.resource.update.read_metadata",
|
||||
domain="resource",
|
||||
action="update",
|
||||
phase="read_metadata",
|
||||
):
|
||||
file_metadata = await file_service.get_file_metadata(target_file_path)
|
||||
|
||||
file_name = PathLib(target_file_path).name
|
||||
content_type = file_service.content_type(target_file_path)
|
||||
note_type = "canvas" if target_file_path.endswith(".canvas") else "file"
|
||||
|
||||
with logfire.span(
|
||||
"api.resource.update.update_entity",
|
||||
domain="resource",
|
||||
action="update",
|
||||
phase="update_entity",
|
||||
):
|
||||
updated_entity = await entity_repository.update(
|
||||
entity.id,
|
||||
{
|
||||
"title": file_name,
|
||||
"note_type": note_type,
|
||||
"content_type": content_type,
|
||||
"file_path": target_file_path,
|
||||
"checksum": checksum,
|
||||
"updated_at": file_metadata.modified_at,
|
||||
},
|
||||
)
|
||||
if updated_entity is None:
|
||||
raise HTTPException(status_code=404, detail=f"Entity {entity_id} not found")
|
||||
|
||||
with logfire.span(
|
||||
"api.resource.update.search_index",
|
||||
domain="resource",
|
||||
action="update",
|
||||
phase="search_index",
|
||||
):
|
||||
await search_service.index_entity(updated_entity)
|
||||
|
||||
return ResourceResponse(
|
||||
entity_id=entity.id,
|
||||
external_id=entity.external_id,
|
||||
file_path=target_file_path,
|
||||
checksum=checksum,
|
||||
size=file_metadata.size,
|
||||
created_at=file_metadata.created_at.timestamp(),
|
||||
modified_at=file_metadata.modified_at.timestamp(),
|
||||
# Validate path to prevent path traversal attacks
|
||||
project_path = PathLib(config.home)
|
||||
if not validate_project_path(target_file_path, project_path):
|
||||
logger.warning(
|
||||
f"Invalid file path attempted: {target_file_path} in project {config.name}"
|
||||
)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e: # pragma: no cover
|
||||
logger.error(f"Error updating resource {entity_id}: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Failed to update resource: {str(e)}")
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Invalid file path: {target_file_path}. "
|
||||
"Path must be relative and stay within project boundaries.",
|
||||
)
|
||||
|
||||
# If moving file, handle the move
|
||||
if data.file_path and data.file_path != entity.file_path:
|
||||
# Ensure new parent directory exists (no-op for S3)
|
||||
await file_service.ensure_directory(PathLib(target_file_path).parent)
|
||||
|
||||
# If old file exists, remove it via file_service (for cloud compatibility)
|
||||
if await file_service.exists(entity.file_path):
|
||||
await file_service.delete_file(entity.file_path)
|
||||
else:
|
||||
# Ensure directory exists for in-place update
|
||||
await file_service.ensure_directory(PathLib(target_file_path).parent)
|
||||
|
||||
# Write content to target file
|
||||
checksum = await file_service.write_file(target_file_path, data.content)
|
||||
|
||||
# Get file info
|
||||
file_metadata = await file_service.get_file_metadata(target_file_path)
|
||||
|
||||
# Determine file details
|
||||
file_name = PathLib(target_file_path).name
|
||||
content_type = file_service.content_type(target_file_path)
|
||||
note_type = "canvas" if target_file_path.endswith(".canvas") else "file"
|
||||
|
||||
# Update entity using internal ID
|
||||
updated_entity = await entity_repository.update(
|
||||
entity.id,
|
||||
{
|
||||
"title": file_name,
|
||||
"note_type": note_type,
|
||||
"content_type": content_type,
|
||||
"file_path": target_file_path,
|
||||
"checksum": checksum,
|
||||
"updated_at": file_metadata.modified_at,
|
||||
},
|
||||
)
|
||||
|
||||
# Index the updated file for search
|
||||
await search_service.index_entity(updated_entity) # pyright: ignore
|
||||
|
||||
# Return success response
|
||||
return ResourceResponse(
|
||||
entity_id=entity.id,
|
||||
external_id=entity.external_id,
|
||||
file_path=target_file_path,
|
||||
checksum=checksum,
|
||||
size=file_metadata.size,
|
||||
created_at=file_metadata.created_at.timestamp(),
|
||||
modified_at=file_metadata.modified_at.timestamp(),
|
||||
)
|
||||
except HTTPException:
|
||||
# Re-raise HTTP exceptions without wrapping
|
||||
raise
|
||||
except Exception as e: # pragma: no cover
|
||||
logger.error(f"Error updating resource {entity_id}: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Failed to update resource: {str(e)}")
|
||||
|
||||
@@ -4,17 +4,14 @@ This router uses external_id UUIDs for stable, API-friendly routing.
|
||||
V1 uses string-based project names which are less efficient and less stable.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Path
|
||||
|
||||
import logfire
|
||||
from basic_memory.api.v2.utils import to_search_results
|
||||
from basic_memory.repository.semantic_errors import (
|
||||
SemanticDependenciesMissingError,
|
||||
SemanticSearchDisabledError,
|
||||
)
|
||||
from basic_memory.schemas.search import SearchQuery, SearchResponse, SearchRetrievalMode
|
||||
from basic_memory.schemas.search import SearchQuery, SearchResponse
|
||||
from basic_memory.deps import (
|
||||
SearchServiceV2ExternalDep,
|
||||
EntityServiceV2ExternalDep,
|
||||
@@ -50,87 +47,29 @@ async def search(
|
||||
Returns:
|
||||
SearchResponse with paginated search results
|
||||
"""
|
||||
with logfire.span(
|
||||
"api.request.search",
|
||||
entrypoint="api",
|
||||
domain="search",
|
||||
action="search",
|
||||
page=page,
|
||||
offset = (page - 1) * page_size
|
||||
# Fetch one extra item to detect whether more pages exist (N+1 trick)
|
||||
fetch_limit = page_size + 1
|
||||
try:
|
||||
results = await search_service.search(query, limit=fetch_limit, offset=offset)
|
||||
except SemanticSearchDisabledError as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
except SemanticDependenciesMissingError as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
|
||||
has_more = len(results) > page_size
|
||||
if has_more:
|
||||
results = results[:page_size]
|
||||
|
||||
search_results = await to_search_results(entity_service, results)
|
||||
return SearchResponse(
|
||||
results=search_results,
|
||||
current_page=page,
|
||||
page_size=page_size,
|
||||
retrieval_mode=query.retrieval_mode.value,
|
||||
has_query=bool(
|
||||
(query.text and query.text.strip())
|
||||
or query.title
|
||||
or query.permalink
|
||||
or query.permalink_match
|
||||
),
|
||||
has_filters=bool(query.note_types or query.entity_types or query.metadata_filters),
|
||||
):
|
||||
offset = (page - 1) * page_size
|
||||
exact_count_available = query.retrieval_mode == SearchRetrievalMode.FTS
|
||||
try:
|
||||
with logfire.span(
|
||||
"api.search.search.execute_query",
|
||||
domain="search",
|
||||
action="search",
|
||||
phase="execute_query",
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
):
|
||||
if exact_count_available:
|
||||
results, total = await asyncio.gather(
|
||||
search_service.search(query, limit=page_size, offset=offset),
|
||||
search_service.count(query),
|
||||
)
|
||||
else:
|
||||
results = await search_service.search(query, limit=page_size + 1, offset=offset)
|
||||
total = 0
|
||||
except SemanticSearchDisabledError as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
except SemanticDependenciesMissingError as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
|
||||
with logfire.span(
|
||||
"api.search.search.paginate_results",
|
||||
domain="search",
|
||||
action="search",
|
||||
phase="paginate_results",
|
||||
result_count=len(results),
|
||||
):
|
||||
if exact_count_available:
|
||||
has_more = offset + len(results) < total
|
||||
else:
|
||||
# Trigger: semantic modes would need another vector/hybrid retrieval to count.
|
||||
# Why: search requests should not pay for a second semantic pass.
|
||||
# Outcome: preserve probe pagination for semantic search and leave total at 0.
|
||||
has_more = len(results) > page_size
|
||||
if has_more:
|
||||
results = results[:page_size]
|
||||
|
||||
with logfire.span(
|
||||
"api.search.search.hydrate_results",
|
||||
domain="search",
|
||||
action="search",
|
||||
phase="hydrate_results",
|
||||
result_count=len(results),
|
||||
):
|
||||
search_results = await to_search_results(entity_service, results)
|
||||
with logfire.span(
|
||||
"api.search.search.build_response",
|
||||
domain="search",
|
||||
action="search",
|
||||
phase="build_response",
|
||||
result_count=len(search_results),
|
||||
):
|
||||
return SearchResponse(
|
||||
results=search_results,
|
||||
current_page=page,
|
||||
page_size=page_size,
|
||||
total=total,
|
||||
has_more=has_more,
|
||||
)
|
||||
has_more=has_more,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/search/reindex")
|
||||
|
||||
+164
-250
@@ -1,6 +1,6 @@
|
||||
from typing import Any, Protocol, Optional, List, Sequence
|
||||
from typing import Optional, List
|
||||
|
||||
import logfire
|
||||
from basic_memory.repository import EntityRepository
|
||||
from basic_memory.repository.search_repository import SearchIndexRow
|
||||
from basic_memory.schemas.memory import (
|
||||
EntitySummary,
|
||||
@@ -11,268 +11,182 @@ from basic_memory.schemas.memory import (
|
||||
ContextResult,
|
||||
)
|
||||
from basic_memory.schemas.search import SearchItemType, SearchResult
|
||||
from basic_memory.services import EntityService
|
||||
from basic_memory.services.context_service import (
|
||||
ContextResultRow,
|
||||
ContextResult as ServiceContextResult,
|
||||
)
|
||||
|
||||
|
||||
class EntityBatchLookup(Protocol):
|
||||
async def find_by_ids_for_hydration(self, ids: List[int]) -> Sequence[Any]: ...
|
||||
|
||||
|
||||
class EntityServiceBatchLookup(Protocol):
|
||||
async def get_entities_by_id(self, ids: List[int]) -> Sequence[Any]: ...
|
||||
|
||||
|
||||
def _required_str(value: str | None, field_name: str) -> str:
|
||||
"""Return a required search field or fail before producing invalid response data."""
|
||||
if value is None:
|
||||
raise ValueError(f"Search result is missing required field: {field_name}")
|
||||
return value
|
||||
|
||||
|
||||
def _search_item_type(value: str | SearchItemType) -> SearchItemType:
|
||||
"""Normalize repository row type strings into the public search enum."""
|
||||
return value if isinstance(value, SearchItemType) else SearchItemType(value)
|
||||
|
||||
|
||||
async def to_graph_context(
|
||||
context_result: ServiceContextResult,
|
||||
entity_repository: EntityBatchLookup,
|
||||
entity_repository: EntityRepository,
|
||||
page: Optional[int] = None,
|
||||
page_size: Optional[int] = None,
|
||||
) -> GraphContext:
|
||||
with logfire.span(
|
||||
"memory.hydrate_context",
|
||||
domain="memory",
|
||||
action="build_context",
|
||||
phase="hydrate_context",
|
||||
):
|
||||
# First pass: collect all entity IDs needed for external_id lookup
|
||||
# This includes: entity primary results, observation parent entities, relation from/to entities
|
||||
entity_ids_needed: set[int] = set()
|
||||
for context_item in context_result.results:
|
||||
for item in (
|
||||
[context_item.primary_result] + context_item.observations + context_item.related_results
|
||||
):
|
||||
if item.type == SearchItemType.ENTITY:
|
||||
# Entity's own ID for its external_id
|
||||
entity_ids_needed.add(item.id)
|
||||
elif item.type == SearchItemType.OBSERVATION:
|
||||
# Parent entity ID for entity_external_id
|
||||
if item.entity_id: # pyright: ignore
|
||||
entity_ids_needed.add(item.entity_id) # pyright: ignore
|
||||
elif item.type == SearchItemType.RELATION:
|
||||
# Source and target entity IDs for external_ids
|
||||
if item.from_id: # pyright: ignore
|
||||
entity_ids_needed.add(item.from_id) # pyright: ignore
|
||||
if item.to_id:
|
||||
entity_ids_needed.add(item.to_id)
|
||||
|
||||
# Batch fetch all entities at once - get both title and external_id
|
||||
entity_title_lookup: dict[int, str] = {}
|
||||
entity_external_id_lookup: dict[int, str] = {}
|
||||
if entity_ids_needed:
|
||||
entities = await entity_repository.find_by_ids(list(entity_ids_needed))
|
||||
for e in entities:
|
||||
entity_title_lookup[e.id] = e.title
|
||||
entity_external_id_lookup[e.id] = e.external_id
|
||||
|
||||
# Helper function to convert items to summaries
|
||||
def to_summary(item: SearchIndexRow | ContextResultRow):
|
||||
match item.type:
|
||||
case SearchItemType.ENTITY:
|
||||
return EntitySummary(
|
||||
external_id=entity_external_id_lookup.get(item.id, ""),
|
||||
entity_id=item.id,
|
||||
title=item.title, # pyright: ignore
|
||||
permalink=item.permalink,
|
||||
content=item.content,
|
||||
file_path=item.file_path,
|
||||
created_at=item.created_at,
|
||||
)
|
||||
case SearchItemType.OBSERVATION:
|
||||
entity_ext_id = None
|
||||
if item.entity_id: # pyright: ignore
|
||||
entity_ext_id = entity_external_id_lookup.get(item.entity_id) # pyright: ignore
|
||||
return ObservationSummary(
|
||||
observation_id=item.id,
|
||||
entity_id=item.entity_id, # pyright: ignore
|
||||
entity_external_id=entity_ext_id,
|
||||
title=entity_title_lookup.get(item.entity_id), # pyright: ignore
|
||||
file_path=item.file_path,
|
||||
category=item.category, # pyright: ignore
|
||||
content=item.content, # pyright: ignore
|
||||
permalink=item.permalink, # pyright: ignore
|
||||
created_at=item.created_at,
|
||||
)
|
||||
case SearchItemType.RELATION:
|
||||
from_title = entity_title_lookup.get(item.from_id) if item.from_id else None # pyright: ignore
|
||||
to_title = entity_title_lookup.get(item.to_id) if item.to_id else None
|
||||
from_ext_id = entity_external_id_lookup.get(item.from_id) if item.from_id else None # pyright: ignore
|
||||
to_ext_id = entity_external_id_lookup.get(item.to_id) if item.to_id else None
|
||||
return RelationSummary(
|
||||
relation_id=item.id,
|
||||
entity_id=item.entity_id, # pyright: ignore
|
||||
title=item.title, # pyright: ignore
|
||||
file_path=item.file_path,
|
||||
permalink=item.permalink, # pyright: ignore
|
||||
relation_type=item.relation_type, # pyright: ignore
|
||||
from_entity=from_title,
|
||||
from_entity_id=item.from_id, # pyright: ignore
|
||||
from_entity_external_id=from_ext_id,
|
||||
to_entity=to_title,
|
||||
to_entity_id=item.to_id,
|
||||
to_entity_external_id=to_ext_id,
|
||||
created_at=item.created_at,
|
||||
)
|
||||
case _: # pragma: no cover
|
||||
raise ValueError(f"Unexpected type: {item.type}")
|
||||
|
||||
# Process the hierarchical results
|
||||
hierarchical_results = []
|
||||
for context_item in context_result.results:
|
||||
# Process primary result
|
||||
primary_result = to_summary(context_item.primary_result)
|
||||
|
||||
# Process observations (always ObservationSummary, validated by context_service)
|
||||
observations = [to_summary(obs) for obs in context_item.observations]
|
||||
|
||||
# Process related results
|
||||
related = [to_summary(rel) for rel in context_item.related_results]
|
||||
|
||||
# Add to hierarchical results
|
||||
hierarchical_results.append(
|
||||
ContextResult(
|
||||
primary_result=primary_result,
|
||||
observations=observations, # pyright: ignore[reportArgumentType]
|
||||
related_results=related,
|
||||
)
|
||||
)
|
||||
|
||||
# Create schema metadata from service metadata
|
||||
metadata = MemoryMetadata(
|
||||
uri=context_result.metadata.uri,
|
||||
types=context_result.metadata.types,
|
||||
depth=context_result.metadata.depth,
|
||||
timeframe=context_result.metadata.timeframe,
|
||||
generated_at=context_result.metadata.generated_at,
|
||||
primary_count=context_result.metadata.primary_count,
|
||||
related_count=context_result.metadata.related_count,
|
||||
total_results=context_result.metadata.primary_count + context_result.metadata.related_count,
|
||||
total_relations=context_result.metadata.total_relations,
|
||||
total_observations=context_result.metadata.total_observations,
|
||||
)
|
||||
|
||||
# Return new GraphContext with just hierarchical results
|
||||
return GraphContext(
|
||||
results=hierarchical_results,
|
||||
metadata=metadata,
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
result_count=len(context_result.results),
|
||||
):
|
||||
# First pass: collect all entity IDs needed for external_id lookup
|
||||
# This includes: entity primary results, observation parent entities, relation from/to entities
|
||||
entity_ids_needed: set[int] = set()
|
||||
for context_item in context_result.results:
|
||||
for item in (
|
||||
[context_item.primary_result]
|
||||
+ context_item.observations
|
||||
+ context_item.related_results
|
||||
):
|
||||
item_type = _search_item_type(item.type)
|
||||
if item_type == SearchItemType.ENTITY:
|
||||
# Entity's own ID for its external_id
|
||||
entity_ids_needed.add(item.id)
|
||||
elif item_type == SearchItemType.OBSERVATION:
|
||||
# Parent entity ID for entity_external_id
|
||||
if item.entity_id:
|
||||
entity_ids_needed.add(item.entity_id)
|
||||
elif item_type == SearchItemType.RELATION:
|
||||
# Source and target entity IDs for external_ids
|
||||
if item.from_id:
|
||||
entity_ids_needed.add(item.from_id)
|
||||
if item.to_id:
|
||||
entity_ids_needed.add(item.to_id)
|
||||
has_more=context_result.metadata.has_more,
|
||||
)
|
||||
|
||||
# Batch fetch just the entity fields needed to shape the response.
|
||||
entity_title_lookup: dict[int, str] = {}
|
||||
entity_external_id_lookup: dict[int, str] = {}
|
||||
if entity_ids_needed:
|
||||
with logfire.span(
|
||||
"memory.hydrate_context.lookup_entities",
|
||||
domain="memory",
|
||||
action="build_context",
|
||||
phase="lookup_entities",
|
||||
result_count=len(entity_ids_needed),
|
||||
):
|
||||
entities = await entity_repository.find_by_ids_for_hydration(
|
||||
list(entity_ids_needed)
|
||||
)
|
||||
for e in entities:
|
||||
entity_title_lookup[e.id] = e.title
|
||||
entity_external_id_lookup[e.id] = e.external_id
|
||||
|
||||
# Helper function to convert items to summaries
|
||||
def to_summary(
|
||||
item: SearchIndexRow | ContextResultRow,
|
||||
) -> EntitySummary | ObservationSummary | RelationSummary:
|
||||
item_type = _search_item_type(item.type)
|
||||
match item_type:
|
||||
case SearchItemType.ENTITY:
|
||||
return EntitySummary(
|
||||
external_id=entity_external_id_lookup.get(item.id, ""),
|
||||
entity_id=item.id,
|
||||
title=_required_str(item.title, "title"),
|
||||
permalink=item.permalink,
|
||||
content=item.content,
|
||||
file_path=_required_str(item.file_path, "file_path"),
|
||||
created_at=item.created_at,
|
||||
)
|
||||
case SearchItemType.OBSERVATION:
|
||||
entity_ext_id = None
|
||||
entity_title = None
|
||||
if item.entity_id:
|
||||
entity_ext_id = entity_external_id_lookup.get(item.entity_id)
|
||||
entity_title = entity_title_lookup.get(item.entity_id)
|
||||
return ObservationSummary(
|
||||
observation_id=item.id,
|
||||
entity_id=item.entity_id,
|
||||
entity_external_id=entity_ext_id,
|
||||
title=entity_title,
|
||||
file_path=_required_str(item.file_path, "file_path"),
|
||||
category=_required_str(item.category, "category"),
|
||||
content=_required_str(item.content, "content"),
|
||||
permalink=_required_str(item.permalink, "permalink"),
|
||||
created_at=item.created_at,
|
||||
)
|
||||
case SearchItemType.RELATION:
|
||||
from_title = entity_title_lookup.get(item.from_id) if item.from_id else None
|
||||
to_title = entity_title_lookup.get(item.to_id) if item.to_id else None
|
||||
from_ext_id = (
|
||||
entity_external_id_lookup.get(item.from_id) if item.from_id else None
|
||||
)
|
||||
to_ext_id = entity_external_id_lookup.get(item.to_id) if item.to_id else None
|
||||
return RelationSummary(
|
||||
relation_id=item.id,
|
||||
entity_id=item.entity_id,
|
||||
title=_required_str(item.title, "title"),
|
||||
file_path=_required_str(item.file_path, "file_path"),
|
||||
permalink=_required_str(item.permalink, "permalink"),
|
||||
relation_type=_required_str(item.relation_type, "relation_type"),
|
||||
from_entity=from_title,
|
||||
from_entity_id=item.from_id,
|
||||
from_entity_external_id=from_ext_id,
|
||||
to_entity=to_title,
|
||||
to_entity_id=item.to_id,
|
||||
to_entity_external_id=to_ext_id,
|
||||
created_at=item.created_at,
|
||||
)
|
||||
async def to_search_results(entity_service: EntityService, results: List[SearchIndexRow]):
|
||||
search_results = []
|
||||
for r in results:
|
||||
entities = await entity_service.get_entities_by_id([r.entity_id, r.from_id, r.to_id]) # pyright: ignore
|
||||
|
||||
with logfire.span(
|
||||
"memory.hydrate_context.shape_results",
|
||||
domain="memory",
|
||||
action="build_context",
|
||||
phase="shape_results",
|
||||
result_count=len(context_result.results),
|
||||
):
|
||||
hierarchical_results = []
|
||||
for context_item in context_result.results:
|
||||
primary_result = to_summary(context_item.primary_result)
|
||||
observations = [
|
||||
summary
|
||||
for summary in (to_summary(obs) for obs in context_item.observations)
|
||||
if isinstance(summary, ObservationSummary)
|
||||
]
|
||||
related = [to_summary(rel) for rel in context_item.related_results]
|
||||
hierarchical_results.append(
|
||||
ContextResult(
|
||||
primary_result=primary_result,
|
||||
observations=observations,
|
||||
related_results=related,
|
||||
)
|
||||
)
|
||||
# Determine which IDs to set based on type
|
||||
entity_id = None
|
||||
observation_id = None
|
||||
relation_id = None
|
||||
|
||||
metadata = MemoryMetadata(
|
||||
uri=context_result.metadata.uri,
|
||||
types=context_result.metadata.types,
|
||||
depth=context_result.metadata.depth,
|
||||
timeframe=context_result.metadata.timeframe,
|
||||
generated_at=context_result.metadata.generated_at,
|
||||
primary_count=context_result.metadata.primary_count,
|
||||
related_count=context_result.metadata.related_count,
|
||||
total_results=context_result.metadata.primary_count
|
||||
+ context_result.metadata.related_count,
|
||||
total_relations=context_result.metadata.total_relations,
|
||||
total_observations=context_result.metadata.total_observations,
|
||||
if r.type == SearchItemType.ENTITY:
|
||||
entity_id = r.id
|
||||
elif r.type == SearchItemType.OBSERVATION:
|
||||
observation_id = r.id
|
||||
entity_id = r.entity_id # Parent entity
|
||||
elif r.type == SearchItemType.RELATION:
|
||||
relation_id = r.id
|
||||
entity_id = r.entity_id # Parent entity
|
||||
|
||||
search_results.append(
|
||||
SearchResult(
|
||||
title=r.title, # pyright: ignore
|
||||
type=r.type, # pyright: ignore
|
||||
permalink=r.permalink,
|
||||
score=r.score, # pyright: ignore
|
||||
entity=entities[0].permalink if entities else None,
|
||||
content=r.content,
|
||||
matched_chunk=r.matched_chunk_text,
|
||||
file_path=r.file_path,
|
||||
metadata=r.metadata,
|
||||
entity_id=entity_id,
|
||||
observation_id=observation_id,
|
||||
relation_id=relation_id,
|
||||
category=r.category,
|
||||
from_entity=entities[0].permalink if entities else None,
|
||||
to_entity=entities[1].permalink if len(entities) > 1 else None,
|
||||
relation_type=r.relation_type,
|
||||
)
|
||||
)
|
||||
|
||||
return GraphContext(
|
||||
results=hierarchical_results,
|
||||
metadata=metadata,
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
has_more=context_result.metadata.has_more,
|
||||
)
|
||||
|
||||
|
||||
async def to_search_results(
|
||||
entity_service: EntityServiceBatchLookup, results: List[SearchIndexRow]
|
||||
) -> list[SearchResult]:
|
||||
with logfire.span(
|
||||
"search.hydrate_results",
|
||||
domain="search",
|
||||
action="search",
|
||||
phase="hydrate_results",
|
||||
result_count=len(results),
|
||||
):
|
||||
# Collect all unique entity IDs across all results in a single pass
|
||||
# This avoids N+1 queries — one batch fetch instead of one per result
|
||||
all_entity_ids: set[int] = set()
|
||||
for result in results:
|
||||
for eid in (result.entity_id, result.from_id, result.to_id):
|
||||
if eid is not None:
|
||||
all_entity_ids.add(eid)
|
||||
|
||||
# Single batch fetch for all entities
|
||||
entities_by_id: dict[int, Any] = {}
|
||||
with logfire.span(
|
||||
"search.hydrate_results.fetch_entities",
|
||||
domain="search",
|
||||
action="search",
|
||||
phase="fetch_entities",
|
||||
result_count=len(all_entity_ids),
|
||||
):
|
||||
if all_entity_ids:
|
||||
entities = await entity_service.get_entities_by_id(list(all_entity_ids))
|
||||
entities_by_id = {e.id: e for e in entities}
|
||||
|
||||
search_results = []
|
||||
with logfire.span(
|
||||
"search.hydrate_results.shape_results",
|
||||
domain="search",
|
||||
action="search",
|
||||
phase="shape_results",
|
||||
result_count=len(results),
|
||||
):
|
||||
for result in results:
|
||||
entity_id = None
|
||||
observation_id = None
|
||||
relation_id = None
|
||||
|
||||
if result.type == SearchItemType.ENTITY:
|
||||
entity_id = result.id
|
||||
elif result.type == SearchItemType.OBSERVATION:
|
||||
observation_id = result.id
|
||||
entity_id = result.entity_id
|
||||
elif result.type == SearchItemType.RELATION:
|
||||
relation_id = result.id
|
||||
entity_id = result.entity_id
|
||||
|
||||
# Look up entities by their specific IDs
|
||||
parent_entity = entities_by_id.get(result.entity_id) if result.entity_id else None
|
||||
from_entity = entities_by_id.get(result.from_id) if result.from_id else None
|
||||
to_entity = entities_by_id.get(result.to_id) if result.to_id else None
|
||||
|
||||
search_results.append(
|
||||
SearchResult(
|
||||
title=_required_str(result.title, "title"),
|
||||
type=_search_item_type(result.type),
|
||||
permalink=result.permalink,
|
||||
score=result.score if result.score is not None else 0.0,
|
||||
entity=parent_entity.permalink if parent_entity else None,
|
||||
content=result.content,
|
||||
matched_chunk=result.matched_chunk_text,
|
||||
file_path=_required_str(result.file_path, "file_path"),
|
||||
metadata=result.metadata,
|
||||
entity_id=entity_id,
|
||||
observation_id=observation_id,
|
||||
relation_id=relation_id,
|
||||
category=result.category,
|
||||
from_entity=from_entity.permalink if from_entity else None,
|
||||
to_entity=to_entity.permalink if to_entity else None,
|
||||
relation_type=result.relation_type,
|
||||
)
|
||||
)
|
||||
return search_results
|
||||
return search_results
|
||||
|
||||
@@ -8,11 +8,9 @@ from typing import Optional # noqa: E402
|
||||
|
||||
import typer # noqa: E402
|
||||
|
||||
from basic_memory.cli.auto_update import maybe_run_periodic_auto_update # noqa: E402
|
||||
from basic_memory.cli.container import CliContainer, set_container # noqa: E402
|
||||
from basic_memory.cli.promo import maybe_show_cloud_promo, maybe_show_init_line # noqa: E402
|
||||
from basic_memory.config import init_cli_logging # noqa: E402
|
||||
import logfire # noqa: E402
|
||||
|
||||
|
||||
def version_callback(value: bool) -> None:
|
||||
@@ -43,14 +41,6 @@ def app_callback(
|
||||
|
||||
# Initialize logging for CLI (file only, no stdout)
|
||||
init_cli_logging()
|
||||
command_name = ctx.invoked_subcommand or "root"
|
||||
ctx.with_resource(
|
||||
logfire.span(
|
||||
f"cli.command.{command_name}",
|
||||
entrypoint="cli",
|
||||
command_name=command_name,
|
||||
)
|
||||
)
|
||||
|
||||
# --- Composition Root ---
|
||||
# Create container and read config (single point of config access)
|
||||
@@ -62,14 +52,10 @@ def app_callback(
|
||||
# Outcome: one-time plain line printed before the subcommand runs.
|
||||
maybe_show_init_line(ctx.invoked_subcommand)
|
||||
|
||||
# Trigger: register post-command messaging callbacks.
|
||||
# Why: informational/promo/update output belongs below command results.
|
||||
# Outcome: command output remains primary, with optional follow-up notices afterwards.
|
||||
def _post_command_messages() -> None:
|
||||
maybe_show_cloud_promo(ctx.invoked_subcommand)
|
||||
maybe_run_periodic_auto_update(ctx.invoked_subcommand)
|
||||
|
||||
ctx.call_on_close(_post_command_messages)
|
||||
# Trigger: register promo as a post-command callback.
|
||||
# Why: promo output should appear after the command's own output, not before.
|
||||
# Outcome: promo panel renders below the command results (status tree, table, etc.).
|
||||
ctx.call_on_close(lambda: maybe_show_cloud_promo(ctx.invoked_subcommand))
|
||||
|
||||
# Run initialization for commands that don't use the API
|
||||
# Skip for 'mcp' command - it has its own lifespan that handles initialization
|
||||
@@ -84,7 +70,6 @@ def app_callback(
|
||||
"tool",
|
||||
"reset",
|
||||
"reindex",
|
||||
"update",
|
||||
"watch",
|
||||
}
|
||||
if (
|
||||
|
||||
@@ -1,389 +0,0 @@
|
||||
"""Automatic update checks and upgrades for the Basic Memory CLI."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timedelta
|
||||
from enum import Enum
|
||||
|
||||
from loguru import logger
|
||||
from packaging.version import InvalidVersion, Version
|
||||
from rich.console import Console
|
||||
|
||||
import basic_memory
|
||||
from basic_memory.config import ConfigManager
|
||||
|
||||
PACKAGE_NAME = "basic-memory"
|
||||
PYPI_JSON_URL = "https://pypi.org/pypi/basic-memory/json"
|
||||
|
||||
PYPI_TIMEOUT_SECONDS = 5
|
||||
BREW_OUTDATED_TIMEOUT_SECONDS = 60
|
||||
UV_UPGRADE_TIMEOUT_SECONDS = 180
|
||||
BREW_UPGRADE_TIMEOUT_SECONDS = 600
|
||||
|
||||
|
||||
class InstallSource(str, Enum):
|
||||
"""How the running CLI appears to have been installed."""
|
||||
|
||||
HOMEBREW = "homebrew"
|
||||
UV_TOOL = "uv_tool"
|
||||
UVX = "uvx"
|
||||
UNKNOWN = "unknown"
|
||||
|
||||
|
||||
class AutoUpdateStatus(str, Enum):
|
||||
"""Result classification for update checks and installs."""
|
||||
|
||||
SKIPPED = "skipped"
|
||||
UP_TO_DATE = "up_to_date"
|
||||
UPDATE_AVAILABLE = "update_available"
|
||||
UPDATED = "updated"
|
||||
FAILED = "failed"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AutoUpdateResult:
|
||||
"""Structured result for update checks/install attempts."""
|
||||
|
||||
status: AutoUpdateStatus
|
||||
source: InstallSource
|
||||
checked: bool
|
||||
update_available: bool
|
||||
updated: bool
|
||||
latest_version: str | None = None
|
||||
message: str | None = None
|
||||
error: str | None = None
|
||||
restart_recommended: bool = False
|
||||
|
||||
|
||||
def detect_install_source(executable: str | None = None) -> InstallSource:
|
||||
"""Infer installation source from the active interpreter path."""
|
||||
active_executable = executable or sys.executable
|
||||
normalized = active_executable.lower().replace("\\", "/")
|
||||
|
||||
if "cellar/basic-memory" in normalized:
|
||||
return InstallSource.HOMEBREW
|
||||
if "uv/tools/basic-memory" in normalized:
|
||||
return InstallSource.UV_TOOL
|
||||
if "/uv/archive-" in normalized:
|
||||
return InstallSource.UVX
|
||||
return InstallSource.UNKNOWN
|
||||
|
||||
|
||||
def _is_interactive_session() -> bool:
|
||||
"""Return whether stdin/stdout are interactive terminals."""
|
||||
try:
|
||||
return sys.stdin.isatty() and sys.stdout.isatty()
|
||||
except ValueError:
|
||||
# Trigger: stdin/stdout may be closed during transport teardown.
|
||||
# Why: isatty() raises ValueError on closed descriptors.
|
||||
# Outcome: treat as non-interactive and suppress periodic output.
|
||||
return False
|
||||
|
||||
|
||||
def _run_subprocess(
|
||||
command: list[str],
|
||||
*,
|
||||
timeout_seconds: int,
|
||||
silent: bool,
|
||||
capture_output: bool,
|
||||
) -> subprocess.CompletedProcess[str]:
|
||||
"""Run a subprocess with explicit stdio behavior for protocol safety."""
|
||||
# Trigger: silent operation (MCP/background) with no need for subprocess output.
|
||||
# Why: prevent protocol/terminal pollution from child process output.
|
||||
# Outcome: stdout/stderr are discarded unless explicit capture is requested.
|
||||
use_devnull = silent and not capture_output
|
||||
stdout_target = subprocess.DEVNULL if use_devnull else subprocess.PIPE
|
||||
stderr_target = subprocess.DEVNULL if use_devnull else subprocess.PIPE
|
||||
|
||||
return subprocess.run(
|
||||
command,
|
||||
stdin=subprocess.DEVNULL,
|
||||
stdout=stdout_target,
|
||||
stderr=stderr_target,
|
||||
text=True,
|
||||
timeout=timeout_seconds,
|
||||
check=False,
|
||||
)
|
||||
|
||||
|
||||
def _version_from_pypi() -> str:
|
||||
"""Fetch the latest published package version from PyPI."""
|
||||
request = urllib.request.Request(
|
||||
PYPI_JSON_URL,
|
||||
headers={"User-Agent": f"basic-memory-cli/{basic_memory.__version__}"},
|
||||
)
|
||||
with urllib.request.urlopen(request, timeout=PYPI_TIMEOUT_SECONDS) as response:
|
||||
payload = json.loads(response.read().decode("utf-8"))
|
||||
latest = payload.get("info", {}).get("version")
|
||||
if not latest:
|
||||
raise RuntimeError("PyPI JSON response did not include info.version")
|
||||
return str(latest)
|
||||
|
||||
|
||||
def _check_homebrew_update_available(silent: bool) -> tuple[bool, str | None]:
|
||||
"""Check whether Homebrew reports an outdated basic-memory formula."""
|
||||
result = _run_subprocess(
|
||||
["brew", "outdated", "--quiet", PACKAGE_NAME],
|
||||
timeout_seconds=BREW_OUTDATED_TIMEOUT_SECONDS,
|
||||
silent=silent,
|
||||
capture_output=True,
|
||||
)
|
||||
# Trigger: brew outdated exits 1 when the formula IS outdated (with name on stdout).
|
||||
# Why: non-zero exit here means "outdated", not "error".
|
||||
# Outcome: check stdout for the package name to determine outdated status.
|
||||
stdout = (result.stdout or "").strip()
|
||||
is_outdated = PACKAGE_NAME in stdout
|
||||
return is_outdated, None
|
||||
|
||||
|
||||
def _check_pypi_update_available() -> tuple[bool, str]:
|
||||
"""Compare installed package version with PyPI latest version."""
|
||||
latest = _version_from_pypi()
|
||||
try:
|
||||
current_version = Version(basic_memory.__version__)
|
||||
latest_version = Version(latest)
|
||||
except InvalidVersion as exc:
|
||||
raise RuntimeError(
|
||||
f"Could not compare versions (current={basic_memory.__version__}, latest={latest})"
|
||||
) from exc
|
||||
|
||||
return latest_version > current_version, latest
|
||||
|
||||
|
||||
def _manual_update_hint(source: InstallSource) -> str:
|
||||
"""Return manager-appropriate manual update instructions."""
|
||||
if source == InstallSource.UV_TOOL:
|
||||
return "Run `uv tool upgrade basic-memory`."
|
||||
if source == InstallSource.HOMEBREW:
|
||||
return "Run `brew upgrade basic-memory`."
|
||||
return (
|
||||
"Automatic install is not supported for this environment. "
|
||||
"Update with your package manager (for pip: `python3 -m pip install -U basic-memory`)."
|
||||
)
|
||||
|
||||
|
||||
def _save_last_checked_timestamp(config_manager: ConfigManager, checked_at: datetime) -> None:
|
||||
"""Persist the timestamp for the most recent attempted update check."""
|
||||
config = config_manager.load_config()
|
||||
config.auto_update_last_checked_at = checked_at
|
||||
config_manager.save_config(config)
|
||||
|
||||
|
||||
def run_auto_update(
|
||||
*,
|
||||
force: bool = False,
|
||||
check_only: bool = False,
|
||||
silent: bool = False,
|
||||
config_manager: ConfigManager | None = None,
|
||||
now: datetime | None = None,
|
||||
executable: str | None = None,
|
||||
) -> AutoUpdateResult:
|
||||
"""Run update check/install flow and return a structured result."""
|
||||
manager = config_manager or ConfigManager()
|
||||
config = manager.load_config()
|
||||
source = detect_install_source(executable)
|
||||
checked_at = now or datetime.now()
|
||||
|
||||
if source == InstallSource.UVX:
|
||||
return AutoUpdateResult(
|
||||
status=AutoUpdateStatus.SKIPPED,
|
||||
source=source,
|
||||
checked=False,
|
||||
update_available=False,
|
||||
updated=False,
|
||||
message="uvx runtime detected; updates are managed by uvx cache resolution.",
|
||||
)
|
||||
|
||||
if not force and not config.auto_update:
|
||||
return AutoUpdateResult(
|
||||
status=AutoUpdateStatus.SKIPPED,
|
||||
source=source,
|
||||
checked=False,
|
||||
update_available=False,
|
||||
updated=False,
|
||||
message="Auto-update is disabled in config.",
|
||||
)
|
||||
|
||||
if not force and config.auto_update_last_checked_at is not None:
|
||||
try:
|
||||
elapsed = checked_at - config.auto_update_last_checked_at
|
||||
except TypeError:
|
||||
# Trigger: mixed naive/aware datetimes from manual config edits.
|
||||
# Why: datetime subtraction fails for mixed tz-awareness.
|
||||
# Outcome: ignore the gate once and continue with a forced check path.
|
||||
logger.warning("Auto-update interval gate skipped due to incompatible timestamp format")
|
||||
else:
|
||||
if elapsed < timedelta(seconds=config.update_check_interval):
|
||||
return AutoUpdateResult(
|
||||
status=AutoUpdateStatus.SKIPPED,
|
||||
source=source,
|
||||
checked=False,
|
||||
update_available=False,
|
||||
updated=False,
|
||||
message="Update check interval has not elapsed.",
|
||||
)
|
||||
|
||||
try:
|
||||
# --- Availability check ---
|
||||
latest_version: str | None = None
|
||||
if source == InstallSource.HOMEBREW:
|
||||
update_available, latest_version = _check_homebrew_update_available(silent=silent)
|
||||
else:
|
||||
update_available, latest_version = _check_pypi_update_available()
|
||||
|
||||
if not update_available:
|
||||
return AutoUpdateResult(
|
||||
status=AutoUpdateStatus.UP_TO_DATE,
|
||||
source=source,
|
||||
checked=True,
|
||||
update_available=False,
|
||||
updated=False,
|
||||
latest_version=latest_version,
|
||||
message=f"Basic Memory is up to date ({basic_memory.__version__}).",
|
||||
)
|
||||
|
||||
if check_only:
|
||||
return AutoUpdateResult(
|
||||
status=AutoUpdateStatus.UPDATE_AVAILABLE,
|
||||
source=source,
|
||||
checked=True,
|
||||
update_available=True,
|
||||
updated=False,
|
||||
latest_version=latest_version,
|
||||
message=(
|
||||
f"Update available (latest: {latest_version or 'unknown'}). "
|
||||
f"{_manual_update_hint(source)}"
|
||||
),
|
||||
)
|
||||
|
||||
if source == InstallSource.UNKNOWN:
|
||||
return AutoUpdateResult(
|
||||
status=AutoUpdateStatus.UPDATE_AVAILABLE,
|
||||
source=source,
|
||||
checked=True,
|
||||
update_available=True,
|
||||
updated=False,
|
||||
latest_version=latest_version,
|
||||
message=(
|
||||
f"Update available (latest: {latest_version or 'unknown'}). "
|
||||
f"{_manual_update_hint(source)}"
|
||||
),
|
||||
)
|
||||
|
||||
# --- Automatic install ---
|
||||
command = (
|
||||
["uv", "tool", "upgrade", PACKAGE_NAME]
|
||||
if source == InstallSource.UV_TOOL
|
||||
else ["brew", "upgrade", PACKAGE_NAME]
|
||||
)
|
||||
timeout = (
|
||||
UV_UPGRADE_TIMEOUT_SECONDS
|
||||
if source == InstallSource.UV_TOOL
|
||||
else BREW_UPGRADE_TIMEOUT_SECONDS
|
||||
)
|
||||
|
||||
install_result = _run_subprocess(
|
||||
command,
|
||||
timeout_seconds=timeout,
|
||||
silent=silent,
|
||||
capture_output=not silent,
|
||||
)
|
||||
if install_result.returncode != 0:
|
||||
stderr = (install_result.stderr or "").strip() if install_result.stderr else ""
|
||||
stdout = (install_result.stdout or "").strip() if install_result.stdout else ""
|
||||
detail = stderr or stdout or "update command failed"
|
||||
return AutoUpdateResult(
|
||||
status=AutoUpdateStatus.FAILED,
|
||||
source=source,
|
||||
checked=True,
|
||||
update_available=True,
|
||||
updated=False,
|
||||
latest_version=latest_version,
|
||||
message="Automatic update failed.",
|
||||
error=detail,
|
||||
)
|
||||
|
||||
return AutoUpdateResult(
|
||||
status=AutoUpdateStatus.UPDATED,
|
||||
source=source,
|
||||
checked=True,
|
||||
update_available=True,
|
||||
updated=True,
|
||||
latest_version=latest_version,
|
||||
message=(
|
||||
"Basic Memory was updated successfully. "
|
||||
"Restart running sessions to use the new version."
|
||||
),
|
||||
restart_recommended=True,
|
||||
)
|
||||
|
||||
except (
|
||||
RuntimeError,
|
||||
urllib.error.URLError,
|
||||
ValueError,
|
||||
TimeoutError,
|
||||
subprocess.SubprocessError,
|
||||
OSError,
|
||||
) as exc:
|
||||
logger.warning(f"Auto-update check failed: {exc}")
|
||||
return AutoUpdateResult(
|
||||
status=AutoUpdateStatus.FAILED,
|
||||
source=source,
|
||||
checked=True,
|
||||
update_available=False,
|
||||
updated=False,
|
||||
message="Automatic update check failed.",
|
||||
error=str(exc),
|
||||
)
|
||||
finally:
|
||||
# Trigger: we attempted a check path (including failures).
|
||||
# Why: repeated failing checks on every command create noise and unnecessary network load.
|
||||
# Outcome: next periodic check is gated by update_check_interval.
|
||||
try:
|
||||
_save_last_checked_timestamp(manager, checked_at)
|
||||
except Exception as exc: # pragma: no cover
|
||||
logger.warning(f"Failed to persist auto-update timestamp: {exc}")
|
||||
|
||||
|
||||
def maybe_run_periodic_auto_update(
|
||||
invoked_subcommand: str | None,
|
||||
*,
|
||||
config_manager: ConfigManager | None = None,
|
||||
is_interactive: bool | None = None,
|
||||
console: Console | None = None,
|
||||
) -> AutoUpdateResult | None:
|
||||
"""Run a periodic auto-update check for interactive CLI sessions."""
|
||||
interactive = _is_interactive_session() if is_interactive is None else is_interactive
|
||||
if not interactive:
|
||||
return None
|
||||
if invoked_subcommand in {None, "mcp", "update"}:
|
||||
return None
|
||||
|
||||
result = run_auto_update(
|
||||
force=False,
|
||||
check_only=False,
|
||||
silent=False,
|
||||
config_manager=config_manager,
|
||||
)
|
||||
|
||||
if result.status in {
|
||||
AutoUpdateStatus.UPDATE_AVAILABLE,
|
||||
AutoUpdateStatus.UPDATED,
|
||||
AutoUpdateStatus.FAILED,
|
||||
}:
|
||||
out = console or Console()
|
||||
if result.status == AutoUpdateStatus.UPDATED:
|
||||
out.print(f"[green]{result.message}[/green]")
|
||||
elif result.status == AutoUpdateStatus.FAILED:
|
||||
error_detail = f" {result.error}" if result.error else ""
|
||||
out.print(f"[yellow]{result.message}{error_detail}[/yellow]")
|
||||
elif result.message:
|
||||
out.print(f"[cyan]{result.message}[/cyan]")
|
||||
|
||||
return result
|
||||
@@ -1,6 +1,6 @@
|
||||
"""CLI commands for basic-memory."""
|
||||
|
||||
from . import status, db, doctor, import_memory_json, mcp, import_claude_conversations, orphans
|
||||
from . import status, db, doctor, import_memory_json, mcp, import_claude_conversations
|
||||
from . import (
|
||||
import_claude_projects,
|
||||
import_chatgpt,
|
||||
@@ -8,7 +8,6 @@ from . import (
|
||||
project,
|
||||
format,
|
||||
schema,
|
||||
update,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
@@ -18,12 +17,10 @@ __all__ = [
|
||||
"import_memory_json",
|
||||
"mcp",
|
||||
"import_claude_conversations",
|
||||
"orphans",
|
||||
"import_claude_projects",
|
||||
"import_chatgpt",
|
||||
"tool",
|
||||
"project",
|
||||
"format",
|
||||
"schema",
|
||||
"update",
|
||||
]
|
||||
|
||||
@@ -45,26 +45,14 @@ def get_cloud_config() -> tuple[str, str, str]:
|
||||
|
||||
async def get_authenticated_headers(auth: CLIAuth | None = None) -> dict[str, str]:
|
||||
"""
|
||||
Get authentication headers for cloud API requests.
|
||||
|
||||
Credential priority mirrors async_client._resolve_cloud_token():
|
||||
1. API key (config.cloud_api_key) — fast, no refresh needed
|
||||
2. OAuth token via CLIAuth — handles JWT refresh automatically
|
||||
Get authentication headers with JWT token.
|
||||
handles jwt refresh if needed.
|
||||
"""
|
||||
# --- API key (preferred) ---
|
||||
config_manager = ConfigManager()
|
||||
api_key = config_manager.config.cloud_api_key
|
||||
if api_key:
|
||||
return {"Authorization": f"Bearer {api_key}"}
|
||||
|
||||
# --- OAuth fallback ---
|
||||
client_id, domain, _ = get_cloud_config()
|
||||
auth_obj = auth or CLIAuth(client_id=client_id, authkit_domain=domain)
|
||||
token = await auth_obj.get_valid_token()
|
||||
if not token:
|
||||
console.print(
|
||||
"[red]Not authenticated. Run 'bm cloud set-key <key>' or 'bm cloud login' first.[/red]"
|
||||
)
|
||||
console.print("[red]Not authenticated. Please run 'bm cloud login' first.[/red]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
return {"Authorization": f"Bearer {token}"}
|
||||
@@ -99,39 +87,41 @@ async def make_api_request(
|
||||
response = await client.request(method=method, url=url, headers=headers, json=json_data)
|
||||
response.raise_for_status()
|
||||
return response
|
||||
except httpx.HTTPStatusError as e:
|
||||
response = e.response
|
||||
|
||||
# Try to parse error detail from response
|
||||
error_detail = None
|
||||
try:
|
||||
error_detail = response.json()
|
||||
except Exception:
|
||||
# If JSON parsing fails, we'll handle it as a generic error
|
||||
pass
|
||||
|
||||
# Check for subscription_required error (403)
|
||||
if response.status_code == 403 and isinstance(error_detail, dict):
|
||||
# Handle both FastAPI HTTPException format (nested under "detail")
|
||||
# and direct format
|
||||
detail_obj = error_detail.get("detail", error_detail)
|
||||
if (
|
||||
isinstance(detail_obj, dict)
|
||||
and detail_obj.get("error") == "subscription_required"
|
||||
):
|
||||
message = detail_obj.get("message", "Active subscription required")
|
||||
subscribe_url = detail_obj.get(
|
||||
"subscribe_url", "https://basicmemory.com/subscribe"
|
||||
)
|
||||
raise SubscriptionRequiredError(
|
||||
message=message, subscribe_url=subscribe_url
|
||||
) from e
|
||||
|
||||
# Raise generic CloudAPIError with status code and detail
|
||||
raise CloudAPIError(
|
||||
f"API request failed: {e}",
|
||||
status_code=response.status_code,
|
||||
detail=error_detail if isinstance(error_detail, dict) else {},
|
||||
) from e
|
||||
except httpx.HTTPError as e:
|
||||
# Check if this is a response error with response details
|
||||
if hasattr(e, "response") and e.response is not None: # pyright: ignore [reportAttributeAccessIssue]
|
||||
response = e.response # type: ignore
|
||||
|
||||
# Try to parse error detail from response
|
||||
error_detail = None
|
||||
try:
|
||||
error_detail = response.json()
|
||||
except Exception:
|
||||
# If JSON parsing fails, we'll handle it as a generic error
|
||||
pass
|
||||
|
||||
# Check for subscription_required error (403)
|
||||
if response.status_code == 403 and isinstance(error_detail, dict):
|
||||
# Handle both FastAPI HTTPException format (nested under "detail")
|
||||
# and direct format
|
||||
detail_obj = error_detail.get("detail", error_detail)
|
||||
if (
|
||||
isinstance(detail_obj, dict)
|
||||
and detail_obj.get("error") == "subscription_required"
|
||||
):
|
||||
message = detail_obj.get("message", "Active subscription required")
|
||||
subscribe_url = detail_obj.get(
|
||||
"subscribe_url", "https://basicmemory.com/subscribe"
|
||||
)
|
||||
raise SubscriptionRequiredError(
|
||||
message=message, subscribe_url=subscribe_url
|
||||
) from e
|
||||
|
||||
# Raise generic CloudAPIError with status code and detail
|
||||
raise CloudAPIError(
|
||||
f"API request failed: {e}",
|
||||
status_code=response.status_code,
|
||||
detail=error_detail if isinstance(error_detail, dict) else {},
|
||||
) from e
|
||||
|
||||
raise CloudAPIError(f"API request failed: {e}") from e
|
||||
|
||||
@@ -14,24 +14,6 @@ class BisyncError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def _rclone_exclude_filters(pattern: str) -> list[str]:
|
||||
"""Return rclone exclude filters for a gitignore-style pattern."""
|
||||
if pattern.endswith("/"):
|
||||
# Trigger: gitignore-style patterns ending in / are directory-only rules.
|
||||
# Why: stripping the slash would also exclude a same-named file.
|
||||
# Outcome: rclone keeps the directory rule and excludes recursive contents.
|
||||
return [f"- {pattern}", f"- {pattern}**"]
|
||||
|
||||
path_pattern = pattern.removesuffix("/**")
|
||||
|
||||
# Trigger: rclone treats a directory contents filter separately from the
|
||||
# directory/file path itself.
|
||||
# Why: files like config.json and directory markers like .obsidian must both
|
||||
# be excluded, along with anything below matching directories.
|
||||
# Outcome: every ignore pattern excludes the direct match and recursive children.
|
||||
return [f"- {path_pattern}", f"- {path_pattern}/**"]
|
||||
|
||||
|
||||
async def get_mount_info() -> TenantMountInfo:
|
||||
"""Get current tenant information from cloud API."""
|
||||
try:
|
||||
@@ -96,11 +78,19 @@ def convert_bmignore_to_rclone_filters() -> Path:
|
||||
patterns.append(line)
|
||||
continue
|
||||
|
||||
patterns.extend(_rclone_exclude_filters(line))
|
||||
# Convert gitignore pattern to rclone filter syntax
|
||||
# gitignore: node_modules → rclone: - node_modules/**
|
||||
# gitignore: *.pyc → rclone: - *.pyc
|
||||
if "*" in line:
|
||||
# Pattern already has wildcard, just add exclude prefix
|
||||
patterns.append(f"- {line}")
|
||||
else:
|
||||
# Directory pattern - add /** for recursive exclude
|
||||
patterns.append(f"- {line}/**")
|
||||
|
||||
except Exception:
|
||||
# If we can't read the file, create a minimal filter
|
||||
patterns = ["# Error reading .bmignore, using minimal filters", "- .git", "- .git/**"]
|
||||
patterns = ["# Error reading .bmignore, using minimal filters", "- .git/**"]
|
||||
|
||||
# Write rclone filter file
|
||||
rclone_filter_path.write_text("\n".join(patterns) + "\n")
|
||||
|
||||
@@ -2,12 +2,10 @@
|
||||
|
||||
from basic_memory.cli.commands.cloud.api_client import make_api_request
|
||||
from basic_memory.config import ConfigManager
|
||||
from basic_memory.mcp.async_client import resolve_configured_workspace
|
||||
from basic_memory.schemas.cloud import (
|
||||
CloudProjectList,
|
||||
CloudProjectCreateRequest,
|
||||
CloudProjectCreateResponse,
|
||||
ProjectVisibility,
|
||||
)
|
||||
from basic_memory.utils import generate_permalink
|
||||
|
||||
@@ -18,33 +16,12 @@ class CloudUtilsError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def _workspace_headers(
|
||||
*,
|
||||
project_name: str | None = None,
|
||||
workspace: str | None = None,
|
||||
) -> dict[str, str]:
|
||||
"""Build optional workspace headers using the CLI config resolution chain."""
|
||||
resolved_workspace = resolve_configured_workspace(
|
||||
project_name=project_name,
|
||||
workspace=workspace,
|
||||
)
|
||||
if resolved_workspace is None:
|
||||
return {}
|
||||
return {"X-Workspace-ID": resolved_workspace}
|
||||
|
||||
|
||||
async def fetch_cloud_projects(
|
||||
*,
|
||||
project_name: str | None = None,
|
||||
workspace: str | None = None,
|
||||
api_request=make_api_request,
|
||||
) -> CloudProjectList:
|
||||
"""Fetch list of projects from cloud API.
|
||||
|
||||
Args:
|
||||
project_name: Optional project name for workspace resolution
|
||||
workspace: Cloud workspace tenant_id to list projects from
|
||||
|
||||
Returns:
|
||||
CloudProjectList with projects from cloud
|
||||
"""
|
||||
@@ -53,11 +30,7 @@ async def fetch_cloud_projects(
|
||||
config = config_manager.config
|
||||
host_url = config.cloud_host.rstrip("/")
|
||||
|
||||
response = await api_request(
|
||||
method="GET",
|
||||
url=f"{host_url}/proxy/v2/projects/",
|
||||
headers=_workspace_headers(project_name=project_name, workspace=workspace),
|
||||
)
|
||||
response = await api_request(method="GET", url=f"{host_url}/proxy/v2/projects/")
|
||||
|
||||
return CloudProjectList.model_validate(response.json())
|
||||
except Exception as e:
|
||||
@@ -67,16 +40,12 @@ async def fetch_cloud_projects(
|
||||
async def create_cloud_project(
|
||||
project_name: str,
|
||||
*,
|
||||
workspace: str | None = None,
|
||||
visibility: ProjectVisibility = "workspace",
|
||||
api_request=make_api_request,
|
||||
) -> CloudProjectCreateResponse:
|
||||
"""Create a new project on cloud.
|
||||
|
||||
Args:
|
||||
project_name: Name of project to create
|
||||
workspace: Optional workspace override for tenant-scoped project creation
|
||||
visibility: Visibility for the created cloud project
|
||||
|
||||
Returns:
|
||||
CloudProjectCreateResponse with project details from API
|
||||
@@ -93,16 +62,12 @@ async def create_cloud_project(
|
||||
name=project_name,
|
||||
path=project_path,
|
||||
set_default=False,
|
||||
visibility=visibility,
|
||||
)
|
||||
|
||||
response = await api_request(
|
||||
method="POST",
|
||||
url=f"{host_url}/proxy/v2/projects/",
|
||||
headers={
|
||||
"Content-Type": "application/json",
|
||||
**_workspace_headers(project_name=project_name, workspace=workspace),
|
||||
},
|
||||
headers={"Content-Type": "application/json"},
|
||||
json_data=project_data.model_dump(),
|
||||
)
|
||||
|
||||
@@ -116,38 +81,28 @@ async def sync_project(project_name: str, force_full: bool = False) -> None:
|
||||
|
||||
Args:
|
||||
project_name: Name of project to sync
|
||||
force_full: ignored, kept for backwards compatibility
|
||||
force_full: If True, force a full scan bypassing watermark optimization
|
||||
"""
|
||||
try:
|
||||
from basic_memory.cli.commands.command_utils import run_sync
|
||||
|
||||
await run_sync(project=project_name)
|
||||
await run_sync(project=project_name, force_full=force_full)
|
||||
except Exception as e:
|
||||
raise CloudUtilsError(f"Failed to sync project '{project_name}': {e}") from e
|
||||
|
||||
|
||||
async def project_exists(
|
||||
project_name: str,
|
||||
*,
|
||||
workspace: str | None = None,
|
||||
api_request=make_api_request,
|
||||
) -> bool:
|
||||
async def project_exists(project_name: str, *, api_request=make_api_request) -> bool:
|
||||
"""Check if a project exists on cloud.
|
||||
|
||||
Args:
|
||||
project_name: Name of project to check
|
||||
workspace: Optional workspace override for tenant-scoped project lookup
|
||||
|
||||
Returns:
|
||||
True if project exists, False otherwise
|
||||
|
||||
Raises:
|
||||
CloudUtilsError: If the project list cannot be fetched from cloud
|
||||
"""
|
||||
projects = await fetch_cloud_projects(
|
||||
project_name=project_name,
|
||||
workspace=workspace,
|
||||
api_request=api_request,
|
||||
)
|
||||
project_names = {p.name for p in projects.projects}
|
||||
return project_name in project_names
|
||||
try:
|
||||
projects = await fetch_cloud_projects(api_request=api_request)
|
||||
project_names = {p.name for p in projects.projects}
|
||||
return project_name in project_names
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
@@ -76,22 +76,10 @@ def login():
|
||||
|
||||
@cloud_app.command()
|
||||
def logout():
|
||||
"""Remove stored OAuth tokens and clear cached workspace selection."""
|
||||
config_manager = ConfigManager()
|
||||
config = config_manager.config
|
||||
"""Remove stored OAuth tokens."""
|
||||
config = ConfigManager().config
|
||||
auth = CLIAuth(client_id=config.cloud_client_id, authkit_domain=config.cloud_domain)
|
||||
auth.logout()
|
||||
|
||||
# Trigger: ending a session must invalidate the cached workspace.
|
||||
# Why: a follow-up `bm cloud login` (often as a different user, or returning
|
||||
# from an org workspace to personal) inherits the previous selection
|
||||
# and silently routes everything through the wrong tenant. See #755.
|
||||
# Outcome: re-login starts from a clean slate; the user picks again via
|
||||
# `bm cloud workspace set-default` or per-project --workspace.
|
||||
if config.default_workspace is not None:
|
||||
config.default_workspace = None
|
||||
config_manager.save_config(config)
|
||||
|
||||
console.print("[dim]API key (if configured) remains available for cloud project routing.[/dim]")
|
||||
|
||||
|
||||
@@ -179,7 +167,7 @@ def setup() -> None:
|
||||
console.print("1. Add a project with local sync path:")
|
||||
console.print(" bm project add research --cloud --local-path ~/Documents/research")
|
||||
console.print("\n Or configure sync for an existing project:")
|
||||
console.print(" bm cloud sync-setup research ~/Documents/research")
|
||||
console.print(" bm project sync-setup research ~/Documents/research")
|
||||
console.print("\n2. Preview the initial sync (recommended):")
|
||||
console.print(" bm project bisync --name research --resync --dry-run")
|
||||
console.print("\n3. If all looks good, run the actual sync:")
|
||||
|
||||
@@ -54,7 +54,7 @@ def _require_cloud_credentials(config) -> None:
|
||||
|
||||
async def _get_cloud_project(name: str) -> ProjectItem | None:
|
||||
"""Fetch a project by name from the cloud API."""
|
||||
async with get_client(project_name=name) as client:
|
||||
async with get_client() as client:
|
||||
projects_list = await ProjectClient(client).list_projects()
|
||||
for proj in projects_list.projects:
|
||||
if generate_permalink(proj.name) == generate_permalink(name):
|
||||
@@ -124,6 +124,22 @@ def sync_project_command(
|
||||
|
||||
if success:
|
||||
console.print(f"[green]{name} synced successfully[/green]")
|
||||
|
||||
# Trigger database sync if not a dry run
|
||||
if not dry_run:
|
||||
|
||||
async def _trigger_db_sync():
|
||||
async with get_client() as client:
|
||||
return await ProjectClient(client).sync(
|
||||
project_data.external_id, force_full=True
|
||||
)
|
||||
|
||||
try:
|
||||
with force_routing(cloud=True):
|
||||
result = run_with_cleanup(_trigger_db_sync())
|
||||
console.print(f"[dim]Database sync initiated: {result.get('message')}[/dim]")
|
||||
except Exception as e:
|
||||
console.print(f"[yellow]Warning: Could not trigger database sync: {e}[/yellow]")
|
||||
else:
|
||||
console.print(f"[red]{name} sync failed[/red]")
|
||||
raise typer.Exit(1)
|
||||
@@ -179,13 +195,26 @@ def bisync_project_command(
|
||||
# Update config — sync_entry is guaranteed non-None because
|
||||
# _get_sync_project validated local_sync_path (which comes from sync_entry)
|
||||
sync_entry = config.projects.get(name)
|
||||
if sync_entry is None:
|
||||
raise RuntimeError(
|
||||
f"Sync entry for project '{name}' unexpectedly missing after validation"
|
||||
)
|
||||
assert sync_entry is not None
|
||||
sync_entry.last_sync = datetime.now()
|
||||
sync_entry.bisync_initialized = True
|
||||
ConfigManager().save_config(config)
|
||||
|
||||
# Trigger database sync if not a dry run
|
||||
if not dry_run:
|
||||
|
||||
async def _trigger_db_sync():
|
||||
async with get_client() as client:
|
||||
return await ProjectClient(client).sync(
|
||||
project_data.external_id, force_full=True
|
||||
)
|
||||
|
||||
try:
|
||||
with force_routing(cloud=True):
|
||||
result = run_with_cleanup(_trigger_db_sync())
|
||||
console.print(f"[dim]Database sync initiated: {result.get('message')}[/dim]")
|
||||
except Exception as e:
|
||||
console.print(f"[yellow]Warning: Could not trigger database sync: {e}[/yellow]")
|
||||
else:
|
||||
console.print(f"[red]{name} bisync failed[/red]")
|
||||
raise typer.Exit(1)
|
||||
@@ -291,7 +320,7 @@ def setup_project_sync(
|
||||
|
||||
async def _verify_project_exists():
|
||||
"""Verify the project exists on cloud by listing all projects."""
|
||||
async with get_client(project_name=name) as client:
|
||||
async with get_client() as client:
|
||||
projects_list = await ProjectClient(client).list_projects()
|
||||
project_names = [p.name for p in projects_list.projects]
|
||||
if name not in project_names:
|
||||
|
||||
@@ -20,7 +20,6 @@ from loguru import logger
|
||||
from rich.console import Console
|
||||
|
||||
from basic_memory.cli.commands.cloud.rclone_installer import is_rclone_installed
|
||||
from basic_memory.config import resolve_data_dir
|
||||
from basic_memory.utils import normalize_project_path
|
||||
|
||||
console = Console()
|
||||
@@ -139,16 +138,13 @@ def get_bmignore_filter_path() -> Path:
|
||||
def get_project_bisync_state(project_name: str) -> Path:
|
||||
"""Get path to project's bisync state directory.
|
||||
|
||||
Honors ``BASIC_MEMORY_CONFIG_DIR`` so isolated instances each keep their
|
||||
own bisync state alongside their config.
|
||||
|
||||
Args:
|
||||
project_name: Name of the project
|
||||
|
||||
Returns:
|
||||
Path to bisync state directory for this project
|
||||
"""
|
||||
return resolve_data_dir() / "bisync-state" / project_name
|
||||
return Path.home() / ".basic-memory" / "bisync-state" / project_name
|
||||
|
||||
|
||||
def bisync_initialized(project_name: str) -> bool:
|
||||
|
||||
@@ -4,7 +4,7 @@ import os
|
||||
import platform
|
||||
import shutil
|
||||
import subprocess
|
||||
from typing import Any, Optional, cast
|
||||
from typing import Optional
|
||||
|
||||
from rich.console import Console
|
||||
|
||||
@@ -53,9 +53,7 @@ def run_command(command: list[str], check: bool = True) -> subprocess.CompletedP
|
||||
|
||||
|
||||
def install_rclone_macos() -> None:
|
||||
"""Install rclone on macOS using package managers."""
|
||||
install_errors: list[str] = []
|
||||
|
||||
"""Install rclone on macOS using Homebrew or official script."""
|
||||
# Try Homebrew first
|
||||
if shutil.which("brew"):
|
||||
try:
|
||||
@@ -63,37 +61,35 @@ def install_rclone_macos() -> None:
|
||||
run_command(["brew", "install", "rclone"])
|
||||
console.print("[green]rclone installed via Homebrew[/green]")
|
||||
return
|
||||
except RcloneInstallError as exc:
|
||||
install_errors.append(f"Homebrew failed: {exc}")
|
||||
console.print("[yellow]Homebrew installation failed, trying MacPorts...[/yellow]")
|
||||
except RcloneInstallError:
|
||||
console.print(
|
||||
"[yellow]Homebrew installation failed, trying official script...[/yellow]"
|
||||
)
|
||||
|
||||
if shutil.which("port"):
|
||||
try:
|
||||
console.print("[blue]Installing rclone via MacPorts...[/blue]")
|
||||
run_command(["sudo", "port", "install", "rclone"])
|
||||
console.print("[green]rclone installed via MacPorts[/green]")
|
||||
return
|
||||
except RcloneInstallError as exc:
|
||||
install_errors.append(f"MacPorts failed: {exc}")
|
||||
console.print("[yellow]MacPorts installation failed[/yellow]")
|
||||
|
||||
details = "\n".join(f"- {error}" for error in install_errors)
|
||||
if details:
|
||||
details = f"\n\nAttempts:\n{details}"
|
||||
raise RcloneInstallError(
|
||||
"Could not install rclone automatically with an available package manager.\n\n"
|
||||
"Install rclone manually with one of:\n"
|
||||
" brew install rclone\n"
|
||||
" sudo port install rclone\n"
|
||||
" Download from https://rclone.org/downloads/ and add rclone to PATH"
|
||||
f"{details}"
|
||||
)
|
||||
# Fallback to official script
|
||||
console.print("[blue]Installing rclone via official script...[/blue]")
|
||||
try:
|
||||
run_command(["sh", "-c", "curl https://rclone.org/install.sh | sudo bash"])
|
||||
console.print("[green]rclone installed via official script[/green]")
|
||||
except RcloneInstallError:
|
||||
raise RcloneInstallError(
|
||||
"Failed to install rclone. Please install manually: brew install rclone"
|
||||
)
|
||||
|
||||
|
||||
def install_rclone_linux() -> None:
|
||||
"""Install rclone on Linux using package managers."""
|
||||
install_errors: list[str] = []
|
||||
"""Install rclone on Linux using package managers or official script."""
|
||||
# Try snap first (most universal)
|
||||
if shutil.which("snap"):
|
||||
try:
|
||||
console.print("[blue]Installing rclone via snap...[/blue]")
|
||||
run_command(["sudo", "snap", "install", "rclone"])
|
||||
console.print("[green]rclone installed via snap[/green]")
|
||||
return
|
||||
except RcloneInstallError:
|
||||
console.print("[yellow]Snap installation failed, trying apt...[/yellow]")
|
||||
|
||||
# Try apt (Debian/Ubuntu)
|
||||
if shutil.which("apt"):
|
||||
try:
|
||||
console.print("[blue]Installing rclone via apt...[/blue]")
|
||||
@@ -101,75 +97,18 @@ def install_rclone_linux() -> None:
|
||||
run_command(["sudo", "apt", "install", "-y", "rclone"])
|
||||
console.print("[green]rclone installed via apt[/green]")
|
||||
return
|
||||
except RcloneInstallError as exc:
|
||||
install_errors.append(f"apt failed: {exc}")
|
||||
console.print("[yellow]apt installation failed, trying dnf...[/yellow]")
|
||||
except RcloneInstallError:
|
||||
console.print("[yellow]apt installation failed, trying official script...[/yellow]")
|
||||
|
||||
if shutil.which("dnf"):
|
||||
try:
|
||||
console.print("[blue]Installing rclone via dnf...[/blue]")
|
||||
run_command(["sudo", "dnf", "install", "-y", "rclone"])
|
||||
console.print("[green]rclone installed via dnf[/green]")
|
||||
return
|
||||
except RcloneInstallError as exc:
|
||||
install_errors.append(f"dnf failed: {exc}")
|
||||
console.print("[yellow]dnf installation failed, trying yum...[/yellow]")
|
||||
|
||||
if shutil.which("yum"):
|
||||
try:
|
||||
console.print("[blue]Installing rclone via yum...[/blue]")
|
||||
run_command(["sudo", "yum", "install", "-y", "rclone"])
|
||||
console.print("[green]rclone installed via yum[/green]")
|
||||
return
|
||||
except RcloneInstallError as exc:
|
||||
install_errors.append(f"yum failed: {exc}")
|
||||
console.print("[yellow]yum installation failed, trying pacman...[/yellow]")
|
||||
|
||||
if shutil.which("pacman"):
|
||||
try:
|
||||
console.print("[blue]Installing rclone via pacman...[/blue]")
|
||||
run_command(["sudo", "pacman", "-S", "--noconfirm", "rclone"])
|
||||
console.print("[green]rclone installed via pacman[/green]")
|
||||
return
|
||||
except RcloneInstallError as exc:
|
||||
install_errors.append(f"pacman failed: {exc}")
|
||||
console.print("[yellow]pacman installation failed, trying zypper...[/yellow]")
|
||||
|
||||
if shutil.which("zypper"):
|
||||
try:
|
||||
console.print("[blue]Installing rclone via zypper...[/blue]")
|
||||
run_command(["sudo", "zypper", "--non-interactive", "install", "rclone"])
|
||||
console.print("[green]rclone installed via zypper[/green]")
|
||||
return
|
||||
except RcloneInstallError as exc:
|
||||
install_errors.append(f"zypper failed: {exc}")
|
||||
console.print("[yellow]zypper installation failed, trying snap...[/yellow]")
|
||||
|
||||
if shutil.which("snap"):
|
||||
try:
|
||||
console.print("[blue]Installing rclone via snap...[/blue]")
|
||||
run_command(["sudo", "snap", "install", "rclone"])
|
||||
console.print("[green]rclone installed via snap[/green]")
|
||||
return
|
||||
except RcloneInstallError as exc:
|
||||
install_errors.append(f"snap failed: {exc}")
|
||||
console.print("[yellow]snap installation failed[/yellow]")
|
||||
|
||||
details = "\n".join(f"- {error}" for error in install_errors)
|
||||
if details:
|
||||
details = f"\n\nAttempts:\n{details}"
|
||||
raise RcloneInstallError(
|
||||
"Could not install rclone automatically with an available package manager.\n\n"
|
||||
"Install rclone manually with one of your OS package managers, for example:\n"
|
||||
" sudo apt install rclone\n"
|
||||
" sudo dnf install rclone\n"
|
||||
" sudo yum install rclone\n"
|
||||
" sudo pacman -S rclone\n"
|
||||
" sudo zypper install rclone\n"
|
||||
" sudo snap install rclone\n"
|
||||
"Or download from https://rclone.org/downloads/ and add rclone to PATH"
|
||||
f"{details}"
|
||||
)
|
||||
# Fallback to official script
|
||||
console.print("[blue]Installing rclone via official script...[/blue]")
|
||||
try:
|
||||
run_command(["sh", "-c", "curl https://rclone.org/install.sh | sudo bash"])
|
||||
console.print("[green]rclone installed via official script[/green]")
|
||||
except RcloneInstallError:
|
||||
raise RcloneInstallError(
|
||||
"Failed to install rclone. Please install manually: sudo snap install rclone"
|
||||
)
|
||||
|
||||
|
||||
def install_rclone_windows() -> None:
|
||||
@@ -270,38 +209,27 @@ def refresh_windows_path() -> None:
|
||||
if platform.system().lower() != "windows":
|
||||
return
|
||||
|
||||
# Importing here after performing platform detection. Non-Windows type checkers may still
|
||||
# resolve a stub without registry members, so keep this platform-only module dynamic here.
|
||||
# Importing here after performing platform detection. Also note that we have to ignore pylance/pyright
|
||||
# warnings about winreg attributes so that "errors" don't appear on non-Windows platforms.
|
||||
import winreg
|
||||
|
||||
winreg_module = cast(Any, winreg)
|
||||
user_key_path = r"Environment"
|
||||
system_key_path = r"System\CurrentControlSet\Control\Session Manager\Environment"
|
||||
new_path = ""
|
||||
|
||||
# Read user PATH
|
||||
try:
|
||||
reg_key = winreg_module.OpenKey(
|
||||
winreg_module.HKEY_CURRENT_USER,
|
||||
user_key_path,
|
||||
0,
|
||||
winreg_module.KEY_READ,
|
||||
)
|
||||
user_path, _ = winreg_module.QueryValueEx(reg_key, "PATH")
|
||||
winreg_module.CloseKey(reg_key)
|
||||
reg_key = winreg.OpenKey(winreg.HKEY_CURRENT_USER, user_key_path, 0, winreg.KEY_READ) # type: ignore[reportAttributeAccessIssue]
|
||||
user_path, _ = winreg.QueryValueEx(reg_key, "PATH") # type: ignore[reportAttributeAccessIssue]
|
||||
winreg.CloseKey(reg_key) # type: ignore[reportAttributeAccessIssue]
|
||||
except Exception:
|
||||
user_path = ""
|
||||
|
||||
# Read system PATH
|
||||
try:
|
||||
reg_key = winreg_module.OpenKey(
|
||||
winreg_module.HKEY_LOCAL_MACHINE,
|
||||
system_key_path,
|
||||
0,
|
||||
winreg_module.KEY_READ,
|
||||
)
|
||||
system_path, _ = winreg_module.QueryValueEx(reg_key, "PATH")
|
||||
winreg_module.CloseKey(reg_key)
|
||||
reg_key = winreg.OpenKey(winreg.HKEY_LOCAL_MACHINE, system_key_path, 0, winreg.KEY_READ) # type: ignore[reportAttributeAccessIssue]
|
||||
system_path, _ = winreg.QueryValueEx(reg_key, "PATH") # type: ignore[reportAttributeAccessIssue]
|
||||
winreg.CloseKey(reg_key) # type: ignore[reportAttributeAccessIssue]
|
||||
except Exception:
|
||||
system_path = ""
|
||||
|
||||
|
||||
@@ -1,6 +1,5 @@
|
||||
"""Upload CLI commands for basic-memory projects."""
|
||||
|
||||
from functools import partial
|
||||
from pathlib import Path
|
||||
|
||||
import typer
|
||||
@@ -9,16 +8,12 @@ from rich.console import Console
|
||||
from basic_memory.cli.app import cloud_app
|
||||
from basic_memory.cli.commands.command_utils import run_with_cleanup
|
||||
from basic_memory.cli.commands.cloud.cloud_utils import (
|
||||
CloudUtilsError,
|
||||
create_cloud_project,
|
||||
project_exists,
|
||||
sync_project,
|
||||
)
|
||||
from basic_memory.cli.commands.cloud.upload import upload_path
|
||||
from basic_memory.mcp.async_client import (
|
||||
get_cloud_control_plane_client,
|
||||
resolve_configured_workspace,
|
||||
)
|
||||
from basic_memory.mcp.async_client import get_cloud_control_plane_client
|
||||
|
||||
console = Console()
|
||||
|
||||
@@ -78,20 +73,12 @@ def upload(
|
||||
"""
|
||||
|
||||
async def _upload():
|
||||
resolved_workspace = resolve_configured_workspace(project_name=project)
|
||||
|
||||
try:
|
||||
project_already_exists = await project_exists(project, workspace=resolved_workspace)
|
||||
except CloudUtilsError as e:
|
||||
console.print(f"[red]Failed to check cloud project '{project}': {e}[/red]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
# Check if project exists
|
||||
if not project_already_exists:
|
||||
if not await project_exists(project):
|
||||
if create_project:
|
||||
console.print(f"[blue]Creating cloud project '{project}'...[/blue]")
|
||||
try:
|
||||
await create_cloud_project(project, workspace=resolved_workspace)
|
||||
await create_cloud_project(project)
|
||||
console.print(f"[green]Created project '{project}'[/green]")
|
||||
except Exception as e:
|
||||
console.print(f"[red]Failed to create project: {e}[/red]")
|
||||
@@ -106,8 +93,6 @@ def upload(
|
||||
raise typer.Exit(1)
|
||||
|
||||
# Perform upload (or dry run)
|
||||
if resolved_workspace:
|
||||
console.print(f"[dim]Using workspace: {resolved_workspace}[/dim]")
|
||||
if dry_run:
|
||||
console.print(
|
||||
f"[yellow]DRY RUN: Showing what would be uploaded to '{project}'[/yellow]"
|
||||
@@ -121,10 +106,7 @@ def upload(
|
||||
verbose=verbose,
|
||||
use_gitignore=not no_gitignore,
|
||||
dry_run=dry_run,
|
||||
client_cm_factory=partial(
|
||||
get_cloud_control_plane_client,
|
||||
workspace=resolved_workspace,
|
||||
),
|
||||
client_cm_factory=get_cloud_control_plane_client,
|
||||
)
|
||||
if not success:
|
||||
console.print("[red]Upload failed[/red]")
|
||||
@@ -135,14 +117,12 @@ def upload(
|
||||
else:
|
||||
console.print(f"[green]Successfully uploaded to '{project}'[/green]")
|
||||
|
||||
# Sync project if requested (skip on dry run).
|
||||
# Trigger: upload adds new files the watcher has not observed locally.
|
||||
# Why: force_full ensures those freshly uploaded files are indexed immediately.
|
||||
# Outcome: upload keeps its eager reindex while sync/bisync stay incremental.
|
||||
# Sync project if requested (skip on dry run)
|
||||
# Force full scan after bisync to ensure database is up-to-date with synced files
|
||||
if sync and not dry_run:
|
||||
console.print(f"[blue]Syncing project '{project}'...[/blue]")
|
||||
try:
|
||||
await sync_project(project)
|
||||
await sync_project(project, force_full=True)
|
||||
except Exception as e:
|
||||
console.print(f"[yellow]Warning: Sync failed: {e}[/yellow]")
|
||||
console.print("[dim]Files uploaded but may not be indexed yet[/dim]")
|
||||
|
||||
@@ -6,11 +6,10 @@ from rich.table import Table
|
||||
|
||||
from basic_memory.cli.commands.command_utils import run_with_cleanup
|
||||
from basic_memory.config import ConfigManager
|
||||
from basic_memory.mcp.project_context import get_available_workspaces
|
||||
from basic_memory.schemas.cloud import (
|
||||
format_workspace_choices,
|
||||
format_workspace_selection_choices,
|
||||
workspace_matches_identifier,
|
||||
from basic_memory.mcp.project_context import (
|
||||
_workspace_choices,
|
||||
_workspace_matches_identifier,
|
||||
get_available_workspaces,
|
||||
)
|
||||
|
||||
console = Console()
|
||||
@@ -63,10 +62,7 @@ def list_workspaces() -> None:
|
||||
|
||||
@workspace_app.command("set-default")
|
||||
def set_default_workspace(
|
||||
identifier: str = typer.Argument(
|
||||
...,
|
||||
help="Workspace name, slug, type, or tenant_id to set as default",
|
||||
),
|
||||
identifier: str = typer.Argument(..., help="Workspace name or tenant_id to set as default"),
|
||||
) -> None:
|
||||
"""Set the default cloud workspace.
|
||||
|
||||
@@ -75,7 +71,6 @@ def set_default_workspace(
|
||||
|
||||
Examples:
|
||||
bm cloud workspace set-default Personal
|
||||
bm cloud workspace set-default organization
|
||||
bm cloud workspace set-default 11111111-1111-1111-1111-111111111111
|
||||
"""
|
||||
|
||||
@@ -92,19 +87,19 @@ def set_default_workspace(
|
||||
console.print("[yellow]No accessible workspaces found.[/yellow]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
matches = [ws for ws in workspaces if workspace_matches_identifier(ws, identifier)]
|
||||
matches = [ws for ws in workspaces if _workspace_matches_identifier(ws, identifier)]
|
||||
|
||||
if not matches:
|
||||
console.print(f"[red]Error: Workspace '{identifier}' not found[/red]")
|
||||
console.print(f"[dim]Available:\n{format_workspace_choices(workspaces)}[/dim]")
|
||||
console.print(f"[dim]Available:\n{_workspace_choices(workspaces)}[/dim]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
if len(matches) > 1:
|
||||
console.print(f"[red]Error: Workspace '{identifier}' matches multiple workspaces.[/red]")
|
||||
console.print(
|
||||
"[dim]Choose one of these matching workspaces by slug:\n"
|
||||
f"{format_workspace_selection_choices(matches)}[/dim]"
|
||||
f"[red]Error: Workspace name '{identifier}' matches multiple workspaces. "
|
||||
f"Use tenant_id instead.[/red]"
|
||||
)
|
||||
console.print(f"[dim]Available:\n{_workspace_choices(workspaces)}[/dim]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
selected = matches[0]
|
||||
|
||||
@@ -1,10 +1,7 @@
|
||||
"""Database management commands."""
|
||||
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path, PurePosixPath, PureWindowsPath
|
||||
from pathlib import Path
|
||||
|
||||
import psutil
|
||||
import typer
|
||||
from loguru import logger
|
||||
from rich.console import Console
|
||||
@@ -15,7 +12,6 @@ from basic_memory import db
|
||||
from basic_memory.cli.app import app
|
||||
from basic_memory.cli.commands.command_utils import run_with_cleanup
|
||||
from basic_memory.config import ConfigManager, ProjectMode
|
||||
from basic_memory.indexing import IndexProgress
|
||||
from basic_memory.repository import ProjectRepository
|
||||
from basic_memory.services.initialization import reconcile_projects_with_config
|
||||
from basic_memory.sync.sync_service import get_sync_service
|
||||
@@ -23,136 +19,6 @@ from basic_memory.sync.sync_service import get_sync_service
|
||||
console = Console()
|
||||
|
||||
|
||||
def _is_basic_memory_mcp(cmdline: list[str]) -> bool:
|
||||
"""Heuristic: does this argv represent a `basic-memory mcp` server?
|
||||
|
||||
The MCP server can be launched any of:
|
||||
basic-memory mcp
|
||||
bm mcp # entrypoint alias from pyproject.toml
|
||||
python -m basic_memory.cli.main mcp # module form
|
||||
uv run basic-memory mcp / uv run bm mcp # uv wrappers
|
||||
/abs/path/to/{bm,basic-memory}[.exe] mcp
|
||||
|
||||
A reliable match needs both signals:
|
||||
1. "mcp" appears as an exact argv token (not "mcp-foo").
|
||||
2. Some argv token names the basic-memory entrypoint — either by
|
||||
hyphen/underscore form, or as a `bm` script (covers `/usr/local/bin/bm`,
|
||||
`bm.exe`, etc. via Path.stem).
|
||||
"""
|
||||
if "mcp" not in cmdline:
|
||||
return False
|
||||
for arg in cmdline:
|
||||
if "basic-memory" in arg or "basic_memory" in arg:
|
||||
return True
|
||||
# Try both POSIX and Windows path interpretations so a test on
|
||||
# macOS still recognizes `C:\\...\\bm.exe`, and a real Windows
|
||||
# run still recognizes `/usr/local/bin/bm`. Path() alone uses
|
||||
# the host OS, which gives wrong stems for foreign separators.
|
||||
if PurePosixPath(arg).stem == "bm" or PureWindowsPath(arg).stem == "bm":
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _find_live_mcp_processes() -> list[tuple[int, str]]:
|
||||
"""Return (pid, joined_cmdline) for live `basic-memory mcp` processes.
|
||||
|
||||
Why this exists (issue #765):
|
||||
On POSIX, `Path.unlink()` removes the directory entry but the inode
|
||||
survives as long as any process holds the file open. A `bm reset`
|
||||
run while Claude Desktop (or another MCP client) is alive will
|
||||
therefore "succeed" — but the still-running MCP keeps reading the
|
||||
old, now-invisible memory.db inode and returns phantom rows. On
|
||||
Windows the OS naturally raises PermissionError on `unlink()`, so
|
||||
the bug is POSIX-specific. We detect proactively to give the same
|
||||
error experience on every platform before doing damage.
|
||||
|
||||
The current process is excluded so this can be called from inside a
|
||||
`bm reset` invocation. NoSuchProcess / AccessDenied are swallowed
|
||||
because process tables race with the scan and we don't want a
|
||||
transient permission error to mask a real zombie.
|
||||
"""
|
||||
me = os.getpid()
|
||||
matches: list[tuple[int, str]] = []
|
||||
for proc in psutil.process_iter(["pid", "cmdline"]):
|
||||
try:
|
||||
pid = proc.info.get("pid")
|
||||
if pid is None or pid == me:
|
||||
continue
|
||||
cmdline = proc.info.get("cmdline") or []
|
||||
if not cmdline:
|
||||
continue
|
||||
if _is_basic_memory_mcp(cmdline):
|
||||
matches.append((pid, " ".join(cmdline)))
|
||||
except (psutil.NoSuchProcess, psutil.AccessDenied):
|
||||
continue
|
||||
return matches
|
||||
|
||||
|
||||
def _abort_if_mcp_processes_alive() -> None:
|
||||
"""Refuse `bm reset` while basic-memory MCP processes are still running.
|
||||
|
||||
See _find_live_mcp_processes for the underlying POSIX-vs-Windows
|
||||
rationale. Prints a per-PID list and platform-appropriate cleanup
|
||||
instructions, then exits non-zero so destructive work never starts.
|
||||
"""
|
||||
zombies = _find_live_mcp_processes()
|
||||
if not zombies:
|
||||
return
|
||||
|
||||
console.print("[red]Refusing to reset:[/red] basic-memory MCP processes are still running.")
|
||||
console.print(
|
||||
"[yellow]On macOS/Linux these would keep reading the deleted memory.db inode "
|
||||
"and return phantom search results (see #765).[/yellow]"
|
||||
)
|
||||
for pid, cmd in zombies:
|
||||
console.print(f" PID {pid}: {cmd}")
|
||||
console.print("\n[bold]How to clean up:[/bold]")
|
||||
console.print(" 1. Quit Claude Desktop and any other MCP clients.")
|
||||
if os.name == "nt":
|
||||
console.print(
|
||||
" 2. Verify nothing remains: "
|
||||
"[green]Get-CimInstance Win32_Process | "
|
||||
"Where-Object {$_.CommandLine -like '*basic-memory*mcp*'}[/green]"
|
||||
)
|
||||
else:
|
||||
console.print(" 2. Verify nothing remains: [green]pgrep -fa 'basic-memory mcp'[/green]")
|
||||
console.print(" 3. Re-run [green]bm reset[/green].")
|
||||
raise typer.Exit(1)
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class EmbeddingProgress:
|
||||
"""Typed CLI progress payload for embedding backfills."""
|
||||
|
||||
entity_id: int
|
||||
completed: int
|
||||
total: int
|
||||
|
||||
|
||||
def _format_eta(seconds: float | None) -> str:
|
||||
"""Render a compact ETA string for CLI progress descriptions."""
|
||||
if seconds is None:
|
||||
return "--:--"
|
||||
|
||||
whole_seconds = max(int(seconds), 0)
|
||||
minutes, remaining_seconds = divmod(whole_seconds, 60)
|
||||
hours, remaining_minutes = divmod(minutes, 60)
|
||||
if hours:
|
||||
return f"{hours:d}:{remaining_minutes:02d}:{remaining_seconds:02d}"
|
||||
return f"{remaining_minutes:02d}:{remaining_seconds:02d}"
|
||||
|
||||
|
||||
def _format_index_progress(progress: IndexProgress) -> str:
|
||||
"""Render typed index progress as a compact Rich task description."""
|
||||
files_per_minute = int(progress.files_per_minute) if progress.files_per_minute else 0
|
||||
return (
|
||||
" Indexing files... "
|
||||
f"{progress.files_processed}/{progress.files_total} files | "
|
||||
f"{progress.batches_completed}/{progress.batches_total} batches | "
|
||||
f"{files_per_minute}/min | ETA {_format_eta(progress.eta_seconds)}"
|
||||
)
|
||||
|
||||
|
||||
async def _reindex_projects(app_config):
|
||||
"""Reindex all projects in a single async context.
|
||||
|
||||
@@ -185,16 +51,6 @@ async def _reindex_projects(app_config):
|
||||
@app.command()
|
||||
def reset(
|
||||
reindex: bool = typer.Option(False, "--reindex", help="Rebuild db index from filesystem"),
|
||||
force: bool = typer.Option(
|
||||
False,
|
||||
"--force",
|
||||
help=(
|
||||
"Skip the pre-flight check that refuses to reset while "
|
||||
"basic-memory MCP processes are running. Use only in "
|
||||
"automated workflows where you've already ensured no MCP "
|
||||
"clients are attached to the database."
|
||||
),
|
||||
),
|
||||
): # pragma: no cover
|
||||
"""Reset database (drop all tables and recreate)."""
|
||||
console.print(
|
||||
@@ -203,14 +59,6 @@ def reset(
|
||||
"Use [green]bm reset --reindex[/green] to automatically rebuild the index afterward."
|
||||
)
|
||||
if typer.confirm("Reset the database index?"):
|
||||
# Pre-flight: refuse to proceed if MCP processes still hold the DB
|
||||
# file open. POSIX would silently let us unlink the inode while
|
||||
# they keep reading it; Windows would error here anyway. See
|
||||
# _find_live_mcp_processes for the full story. --force is the
|
||||
# documented escape hatch for scripted/CI runs.
|
||||
if not force:
|
||||
_abort_if_mcp_processes_alive()
|
||||
|
||||
logger.info("Resetting database...")
|
||||
config_manager = ConfigManager()
|
||||
app_config = config_manager.config
|
||||
@@ -264,30 +112,20 @@ def reindex(
|
||||
False, "--embeddings", "-e", help="Rebuild vector embeddings (requires semantic search)"
|
||||
),
|
||||
search: bool = typer.Option(False, "--search", "-s", help="Rebuild full-text search index"),
|
||||
full: bool = typer.Option(
|
||||
False,
|
||||
"--full",
|
||||
help="Force a full filesystem scan and file reindex instead of the default incremental scan",
|
||||
),
|
||||
project: str = typer.Option(
|
||||
None, "--project", "-p", help="Reindex a specific project (default: all)"
|
||||
),
|
||||
): # pragma: no cover
|
||||
"""Rebuild search indexes and/or vector embeddings without dropping the database.
|
||||
|
||||
By default runs incremental search + embeddings (if semantic search is enabled).
|
||||
Use --full to bypass incremental scan optimization, rebuild all file-backed search rows,
|
||||
and re-embed all eligible notes.
|
||||
Use --search or --embeddings to rebuild only one side.
|
||||
By default rebuilds everything (search + embeddings if semantic is enabled).
|
||||
Use --search or --embeddings to rebuild only one.
|
||||
|
||||
Examples:
|
||||
bm reindex # Incremental search + embeddings
|
||||
bm reindex --full # Full search + full re-embed
|
||||
bm reindex # Rebuild everything
|
||||
bm reindex --embeddings # Only rebuild vector embeddings
|
||||
bm reindex --search # Only rebuild FTS index
|
||||
bm reindex --full --search # Full search only
|
||||
bm reindex --full --embeddings # Full re-embed only
|
||||
bm reindex -p claw --full # Full reindex for only the 'claw' project
|
||||
bm reindex -p claw # Reindex only the 'claw' project
|
||||
"""
|
||||
# If neither flag is set, do both
|
||||
if not embeddings and not search:
|
||||
@@ -306,19 +144,10 @@ def reindex(
|
||||
if not search:
|
||||
raise typer.Exit(0)
|
||||
|
||||
run_with_cleanup(
|
||||
_reindex(app_config, search=search, embeddings=embeddings, full=full, project=project)
|
||||
)
|
||||
run_with_cleanup(_reindex(app_config, search=search, embeddings=embeddings, project=project))
|
||||
|
||||
|
||||
async def _reindex(
|
||||
app_config,
|
||||
*,
|
||||
search: bool,
|
||||
embeddings: bool,
|
||||
full: bool,
|
||||
project: str | None,
|
||||
):
|
||||
async def _reindex(app_config, search: bool, embeddings: bool, project: str | None):
|
||||
"""Run reindex operations."""
|
||||
from basic_memory.repository import EntityRepository
|
||||
from basic_memory.repository.search_repository import create_search_repository
|
||||
@@ -356,47 +185,14 @@ async def _reindex(
|
||||
console.print(f"\n[bold]Project: [cyan]{proj.name}[/cyan][/bold]")
|
||||
|
||||
if search:
|
||||
search_mode_label = "full scan" if full else "incremental scan"
|
||||
console.print(
|
||||
f" Rebuilding full-text search index ([cyan]{search_mode_label}[/cyan])..."
|
||||
)
|
||||
console.print(" Rebuilding full-text search index...")
|
||||
sync_service = await get_sync_service(proj)
|
||||
sync_dir = Path(proj.path)
|
||||
with Progress(
|
||||
SpinnerColumn(),
|
||||
TextColumn("[progress.description]{task.description}"),
|
||||
BarColumn(),
|
||||
TaskProgressColumn(),
|
||||
console=console,
|
||||
) as progress:
|
||||
task = progress.add_task(" Indexing files... scanning changes", total=1)
|
||||
|
||||
async def on_index_progress(update: IndexProgress) -> None:
|
||||
total = update.files_total or 1
|
||||
completed = update.files_processed if update.files_total else 1
|
||||
progress.update(
|
||||
task,
|
||||
description=_format_index_progress(update),
|
||||
total=total,
|
||||
completed=min(completed, total),
|
||||
)
|
||||
|
||||
await sync_service.sync(
|
||||
sync_dir,
|
||||
project_name=proj.name,
|
||||
force_full=full,
|
||||
sync_embeddings=False,
|
||||
progress_callback=on_index_progress,
|
||||
)
|
||||
progress.update(task, completed=progress.tasks[task].total or 1)
|
||||
|
||||
console.print(" [green]done[/green] Full-text search index rebuilt")
|
||||
await sync_service.sync(sync_dir, project_name=proj.name)
|
||||
console.print(" [green]✓[/green] Full-text search index rebuilt")
|
||||
|
||||
if embeddings:
|
||||
embedding_mode_label = "full rebuild" if full else "incremental sync"
|
||||
console.print(
|
||||
f" Building vector embeddings ([cyan]{embedding_mode_label}[/cyan])..."
|
||||
)
|
||||
console.print(" Building vector embeddings...")
|
||||
entity_repository = EntityRepository(session_maker, project_id=proj.id)
|
||||
search_repository = create_search_repository(
|
||||
session_maker, project_id=proj.id, app_config=app_config
|
||||
@@ -417,29 +213,13 @@ async def _reindex(
|
||||
task = progress.add_task(" Embedding entities...", total=None)
|
||||
|
||||
def on_progress(entity_id, index, total):
|
||||
embedding_progress = EmbeddingProgress(
|
||||
entity_id=entity_id,
|
||||
completed=index,
|
||||
total=total,
|
||||
)
|
||||
# Trigger: repository progress now reports terminal entity completion.
|
||||
# Why: operators need to see finished embedding work rather than
|
||||
# entities merely entering prepare.
|
||||
# Outcome: the CLI bar advances steadily with real completed work.
|
||||
progress.update(
|
||||
task,
|
||||
total=embedding_progress.total,
|
||||
completed=embedding_progress.completed,
|
||||
)
|
||||
progress.update(task, total=total, completed=index)
|
||||
|
||||
stats = await search_service.reindex_vectors(
|
||||
progress_callback=on_progress,
|
||||
force_full=full,
|
||||
)
|
||||
stats = await search_service.reindex_vectors(progress_callback=on_progress)
|
||||
progress.update(task, completed=stats["total_entities"])
|
||||
|
||||
console.print(
|
||||
f" [green]done[/green] Embeddings complete: "
|
||||
f" [green]✓[/green] Embeddings complete: "
|
||||
f"{stats['embedded']} entities embedded, "
|
||||
f"{stats['skipped']} skipped, "
|
||||
f"{stats['errors']} errors"
|
||||
|
||||
@@ -54,9 +54,6 @@ async def run_doctor() -> None:
|
||||
if not status.new_project:
|
||||
raise ValueError("Failed to create doctor project")
|
||||
project_id = status.new_project.external_id
|
||||
# Use the resolved path from the server — when project_root is configured,
|
||||
# the actual project directory differs from the requested temp_path
|
||||
project_path = Path(status.new_project.path)
|
||||
console.print(f"[green]OK[/green] Created doctor project: {project_name}")
|
||||
|
||||
# --- DB -> File: create an entity via API ---
|
||||
@@ -69,9 +66,9 @@ async def run_doctor() -> None:
|
||||
content=f"# {api_note_title}\n\n- [note] API to file check",
|
||||
entity_metadata={"tags": ["doctor"]},
|
||||
)
|
||||
api_result = await knowledge_client.create_entity(api_note.model_dump())
|
||||
api_result = await knowledge_client.create_entity(api_note.model_dump(), fast=False)
|
||||
|
||||
api_file = project_path / api_result.file_path
|
||||
api_file = temp_path / api_result.file_path
|
||||
if not api_file.exists():
|
||||
raise ValueError(f"API note file missing: {api_result.file_path}")
|
||||
|
||||
@@ -82,7 +79,7 @@ async def run_doctor() -> None:
|
||||
console.print("[green]OK[/green] API write created file")
|
||||
|
||||
# --- File -> DB: write markdown file directly, then sync ---
|
||||
parser = EntityParser(project_path)
|
||||
parser = EntityParser(temp_path)
|
||||
processor = MarkdownProcessor(parser)
|
||||
manual_markdown = EntityMarkdown(
|
||||
frontmatter=EntityFrontmatter(
|
||||
@@ -96,12 +93,12 @@ async def run_doctor() -> None:
|
||||
content=f"# {manual_note_title}\n\n- [note] File to DB check",
|
||||
)
|
||||
|
||||
manual_path = project_path / "doctor" / "manual-note.md"
|
||||
manual_path = temp_path / "doctor" / "manual-note.md"
|
||||
await processor.write_file(manual_path, manual_markdown)
|
||||
console.print("[green]OK[/green] Manual file written")
|
||||
|
||||
sync_data = await project_client.sync(
|
||||
project_id, force_full=False, run_in_background=False
|
||||
project_id, force_full=True, run_in_background=False
|
||||
)
|
||||
sync_report = SyncReportResponse.model_validate(sync_data)
|
||||
if sync_report.total == 0:
|
||||
|
||||
@@ -1,14 +1,12 @@
|
||||
"""MCP server command with streamable HTTP transport."""
|
||||
|
||||
import os
|
||||
import threading
|
||||
from typing import Any, Optional
|
||||
|
||||
import typer
|
||||
from loguru import logger
|
||||
|
||||
from basic_memory.cli.app import app
|
||||
from basic_memory.cli.auto_update import AutoUpdateStatus, run_auto_update
|
||||
from basic_memory.config import ConfigManager, init_mcp_logging
|
||||
|
||||
|
||||
@@ -82,22 +80,6 @@ def mcp(
|
||||
os.environ["BASIC_MEMORY_MCP_PROJECT"] = project_name
|
||||
logger.info(f"MCP server constrained to project: {project_name}")
|
||||
|
||||
def _run_background_auto_update() -> None:
|
||||
result = run_auto_update(force=False, check_only=False, silent=True)
|
||||
if result.restart_recommended:
|
||||
logger.info(
|
||||
"A newer Basic Memory version was installed and will apply on next restart."
|
||||
)
|
||||
elif result.status == AutoUpdateStatus.FAILED and result.error:
|
||||
logger.warning(f"MCP background auto-update failed: {result.error}")
|
||||
|
||||
# Trigger: stdio transport corresponds to local user installs.
|
||||
# Why: server transports (HTTP/SSE) run in managed environments where
|
||||
# package-manager self-upgrades are inappropriate.
|
||||
# Outcome: background auto-update runs only for local stdio MCP sessions.
|
||||
if transport == "stdio":
|
||||
threading.Thread(target=_run_background_auto_update, daemon=True).start()
|
||||
|
||||
# Run the MCP server (blocks)
|
||||
# Lifespan handles: initialization, migrations, file sync, cleanup
|
||||
logger.info(f"Starting MCP server with {transport.upper()} transport")
|
||||
|
||||
@@ -1,93 +0,0 @@
|
||||
"""Orphans command - show entities with no relations in the knowledge graph."""
|
||||
|
||||
import json
|
||||
from typing import Annotated, Optional
|
||||
|
||||
import typer
|
||||
from loguru import logger
|
||||
from mcp.server.fastmcp.exceptions import ToolError
|
||||
from rich.console import Console
|
||||
from rich.table import Table
|
||||
|
||||
from basic_memory.cli.app import app
|
||||
from basic_memory.cli.commands.routing import force_routing, validate_routing_flags
|
||||
from basic_memory.config import ConfigManager
|
||||
from basic_memory.mcp.async_client import get_client
|
||||
from basic_memory.mcp.clients.knowledge import KnowledgeClient
|
||||
from basic_memory.mcp.project_context import get_active_project
|
||||
from basic_memory.schemas.v2.graph import GraphNode
|
||||
|
||||
console = Console()
|
||||
|
||||
|
||||
async def run_orphans(project: Optional[str] = None) -> tuple[str, list[GraphNode]]:
|
||||
"""Fetch entities that have no relations in the knowledge graph."""
|
||||
project = project or ConfigManager().default_project
|
||||
|
||||
async with get_client(project_name=project) as client:
|
||||
project_item = await get_active_project(client, project, None)
|
||||
entities = await KnowledgeClient(client, project_item.external_id).get_orphans()
|
||||
return project_item.name, entities
|
||||
|
||||
|
||||
@app.command()
|
||||
def orphans(
|
||||
project: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(help="The project name."),
|
||||
] = None,
|
||||
json_output: bool = typer.Option(False, "--json", help="Output in JSON format"),
|
||||
local: bool = typer.Option(
|
||||
False, "--local", help="Force local API routing (ignore cloud mode)"
|
||||
),
|
||||
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
|
||||
):
|
||||
"""Show entities that have no relations in the knowledge graph.
|
||||
|
||||
Orphan entities have no incoming or outgoing connections. These may indicate
|
||||
newly created notes not yet linked to other entities, or notes that have had
|
||||
their relations removed.
|
||||
"""
|
||||
from basic_memory.cli.commands.command_utils import run_with_cleanup
|
||||
|
||||
try:
|
||||
validate_routing_flags(local, cloud)
|
||||
with force_routing(local=local, cloud=cloud):
|
||||
project_name, entities = run_with_cleanup(run_orphans(project))
|
||||
|
||||
if json_output:
|
||||
print(json.dumps([entity.model_dump(mode="json") for entity in entities], indent=2))
|
||||
return
|
||||
|
||||
if not entities:
|
||||
console.print(f"[green]No orphan entities in project '{project_name}'[/green]")
|
||||
return
|
||||
|
||||
table = Table(title=f"{project_name}: Entities Without Relations ({len(entities)} total)")
|
||||
table.add_column("Title", style="cyan")
|
||||
table.add_column("File Path", style="yellow")
|
||||
table.add_column("Type", style="green")
|
||||
|
||||
for entity in entities:
|
||||
table.add_row(
|
||||
entity.title,
|
||||
entity.file_path,
|
||||
entity.note_type or "",
|
||||
)
|
||||
|
||||
console.print(table)
|
||||
except (ValueError, ToolError) as exc:
|
||||
if json_output:
|
||||
print(json.dumps({"error": str(exc)}, indent=2))
|
||||
else:
|
||||
console.print(f"[red]Error: {exc}[/red]")
|
||||
raise typer.Exit(code=1)
|
||||
except typer.Exit:
|
||||
raise
|
||||
except Exception as exc:
|
||||
logger.error(f"Error fetching orphan entities: {exc}")
|
||||
if json_output:
|
||||
print(json.dumps({"error": str(exc)}, indent=2))
|
||||
else:
|
||||
console.print(f"[red]Error: {exc}[/red]")
|
||||
raise typer.Exit(code=1) # pragma: no cover
|
||||
File diff suppressed because it is too large
Load Diff
@@ -16,6 +16,13 @@ from basic_memory.cli.commands.command_utils import run_with_cleanup
|
||||
from basic_memory.cli.commands.routing import force_routing, validate_routing_flags
|
||||
from basic_memory.mcp.tools import build_context as mcp_build_context
|
||||
from basic_memory.mcp.tools import edit_note as mcp_edit_note
|
||||
from basic_memory.mcp.tools import fcm_export_model as mcp_fcm_export_model
|
||||
from basic_memory.mcp.tools import fcm_import_model as mcp_fcm_import_model
|
||||
from basic_memory.mcp.tools import fcm_rank_actions as mcp_fcm_rank_actions
|
||||
from basic_memory.mcp.tools import fcm_simulate as mcp_fcm_simulate
|
||||
from basic_memory.mcp.tools import graph_health as mcp_graph_health
|
||||
from basic_memory.mcp.tools import graph_impact as mcp_graph_impact
|
||||
from basic_memory.mcp.tools import graph_lineage as mcp_graph_lineage
|
||||
from basic_memory.mcp.tools import list_memory_projects as mcp_list_projects
|
||||
from basic_memory.mcp.tools import list_workspaces as mcp_list_workspaces
|
||||
from basic_memory.mcp.tools import read_note as mcp_read_note
|
||||
@@ -40,6 +47,17 @@ def _print_json(result: Any) -> None:
|
||||
print(json.dumps(result, indent=2, ensure_ascii=True, default=str))
|
||||
|
||||
|
||||
def _parse_json_option(raw_value: Optional[str], option_name: str) -> Any:
|
||||
"""Parse a JSON CLI option with deterministic error handling."""
|
||||
if raw_value is None:
|
||||
return None
|
||||
try:
|
||||
return json.loads(raw_value)
|
||||
except json.JSONDecodeError as exc:
|
||||
typer.echo(f"Invalid JSON for {option_name}: {exc}", err=True)
|
||||
raise typer.Exit(1)
|
||||
|
||||
|
||||
# --- Commands ---
|
||||
|
||||
|
||||
@@ -62,12 +80,9 @@ def write_note(
|
||||
help="The project to write to. If not provided, the default project will be used."
|
||||
),
|
||||
] = None,
|
||||
project_id: Annotated[
|
||||
workspace: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(
|
||||
"--project-id",
|
||||
help="Project external_id (UUID). Takes precedence over --project; use to disambiguate same-named projects across cloud workspaces.",
|
||||
),
|
||||
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
|
||||
] = None,
|
||||
local: bool = typer.Option(
|
||||
False, "--local", help="Force local API routing (ignore cloud mode)"
|
||||
@@ -109,7 +124,7 @@ def write_note(
|
||||
content=content,
|
||||
directory=folder,
|
||||
project=project,
|
||||
project_id=project_id,
|
||||
workspace=workspace,
|
||||
tags=tags,
|
||||
output_format="json",
|
||||
)
|
||||
@@ -131,16 +146,15 @@ def read_note(
|
||||
include_frontmatter: bool = typer.Option(
|
||||
False, "--include-frontmatter", help="Include YAML frontmatter in output"
|
||||
),
|
||||
page: int = typer.Option(1, "--page", help="Page number for pagination"),
|
||||
page_size: int = typer.Option(10, "--page-size", help="Number of results per page"),
|
||||
project: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(help="The project to use. If not provided, the default project will be used."),
|
||||
] = None,
|
||||
project_id: Annotated[
|
||||
workspace: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(
|
||||
"--project-id",
|
||||
help="Project external_id (UUID). Takes precedence over --project; use to disambiguate same-named projects across cloud workspaces.",
|
||||
),
|
||||
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
|
||||
] = None,
|
||||
local: bool = typer.Option(
|
||||
False, "--local", help="Force local API routing (ignore cloud mode)"
|
||||
@@ -153,6 +167,7 @@ def read_note(
|
||||
|
||||
bm tool read-note my-note
|
||||
bm tool read-note my-note --include-frontmatter
|
||||
bm tool read-note my-note --page 2 --page-size 5
|
||||
"""
|
||||
try:
|
||||
validate_routing_flags(local, cloud)
|
||||
@@ -162,7 +177,9 @@ def read_note(
|
||||
mcp_read_note(
|
||||
identifier=identifier,
|
||||
project=project,
|
||||
project_id=project_id,
|
||||
workspace=workspace,
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
include_frontmatter=include_frontmatter,
|
||||
output_format="json",
|
||||
)
|
||||
@@ -201,12 +218,9 @@ def edit_note(
|
||||
help="The project to edit. If not provided, the default project will be used."
|
||||
),
|
||||
] = None,
|
||||
project_id: Annotated[
|
||||
workspace: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(
|
||||
"--project-id",
|
||||
help="Project external_id (UUID). Takes precedence over --project; use to disambiguate same-named projects across cloud workspaces.",
|
||||
),
|
||||
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
|
||||
] = None,
|
||||
local: bool = typer.Option(
|
||||
False, "--local", help="Force local API routing (ignore cloud mode)"
|
||||
@@ -231,7 +245,7 @@ def edit_note(
|
||||
operation=operation,
|
||||
content=content,
|
||||
project=project,
|
||||
project_id=project_id,
|
||||
workspace=workspace,
|
||||
section=section,
|
||||
find_text=find_text,
|
||||
expected_replacements=expected_replacements,
|
||||
@@ -269,12 +283,9 @@ def build_context(
|
||||
Optional[str],
|
||||
typer.Option(help="The project to use. If not provided, the default project will be used."),
|
||||
] = None,
|
||||
project_id: Annotated[
|
||||
workspace: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(
|
||||
"--project-id",
|
||||
help="Project external_id (UUID). Takes precedence over --project; use to disambiguate same-named projects across cloud workspaces.",
|
||||
),
|
||||
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
|
||||
] = None,
|
||||
local: bool = typer.Option(
|
||||
False, "--local", help="Force local API routing (ignore cloud mode)"
|
||||
@@ -296,7 +307,7 @@ def build_context(
|
||||
mcp_build_context(
|
||||
url=url,
|
||||
project=project,
|
||||
project_id=project_id,
|
||||
workspace=workspace,
|
||||
depth=depth,
|
||||
timeframe=timeframe,
|
||||
page=page,
|
||||
@@ -329,12 +340,9 @@ def recent_activity(
|
||||
Optional[str],
|
||||
typer.Option(help="The project to use. If not provided, the default project will be used."),
|
||||
] = None,
|
||||
project_id: Annotated[
|
||||
workspace: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(
|
||||
"--project-id",
|
||||
help="Project external_id (UUID). Takes precedence over --project; use to disambiguate same-named projects across cloud workspaces.",
|
||||
),
|
||||
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
|
||||
] = None,
|
||||
local: bool = typer.Option(
|
||||
False, "--local", help="Force local API routing (ignore cloud mode)"
|
||||
@@ -355,13 +363,13 @@ def recent_activity(
|
||||
with force_routing(local=local, cloud=cloud):
|
||||
result = run_with_cleanup(
|
||||
mcp_recent_activity(
|
||||
type=type or "",
|
||||
type=type, # pyright: ignore[reportArgumentType]
|
||||
depth=depth if depth is not None else 1,
|
||||
timeframe=timeframe if timeframe is not None else "7d",
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
project=project,
|
||||
project_id=project_id,
|
||||
workspace=workspace,
|
||||
output_format="json",
|
||||
)
|
||||
)
|
||||
@@ -376,6 +384,372 @@ def recent_activity(
|
||||
raise
|
||||
|
||||
|
||||
@tool_app.command("graph-lineage")
|
||||
def graph_lineage(
|
||||
start: Annotated[str, typer.Argument(help="Start node identifier or memory:// reference")],
|
||||
goal: Annotated[
|
||||
Optional[str],
|
||||
typer.Option("--goal", help="Optional goal node identifier for targeted lineage"),
|
||||
] = None,
|
||||
max_hops: int = typer.Option(4, "--max-hops", help="Maximum traversal hops (1-6)"),
|
||||
relation_filters: Annotated[
|
||||
Optional[List[str]],
|
||||
typer.Option("--relation-filter", help="Relation filters (repeatable)"),
|
||||
] = None,
|
||||
project: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(help="The project to use. If not provided, the default project will be used."),
|
||||
] = None,
|
||||
workspace: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
|
||||
] = None,
|
||||
local: bool = typer.Option(
|
||||
False, "--local", help="Force local API routing (ignore cloud mode)"
|
||||
),
|
||||
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
|
||||
):
|
||||
"""Get graph lineage paths from a start node."""
|
||||
try:
|
||||
validate_routing_flags(local, cloud)
|
||||
with force_routing(local=local, cloud=cloud):
|
||||
result = run_with_cleanup(
|
||||
mcp_graph_lineage(
|
||||
start=start,
|
||||
goal=goal,
|
||||
max_hops=max_hops,
|
||||
relation_filters=relation_filters or [],
|
||||
project=project,
|
||||
workspace=workspace,
|
||||
output_format="json",
|
||||
)
|
||||
)
|
||||
_print_json(result)
|
||||
except ValueError as e:
|
||||
typer.echo(f"Error: {e}", err=True)
|
||||
raise typer.Exit(1)
|
||||
except Exception as e: # pragma: no cover
|
||||
if not isinstance(e, typer.Exit):
|
||||
typer.echo(f"Error during graph_lineage: {e}", err=True)
|
||||
raise typer.Exit(1)
|
||||
raise
|
||||
|
||||
|
||||
@tool_app.command("graph-impact")
|
||||
def graph_impact(
|
||||
target: Annotated[str, typer.Argument(help="Target node identifier or memory:// reference")],
|
||||
horizon: int = typer.Option(2, "--horizon", help="Impact horizon in hops (1-4)"),
|
||||
relation_filters: Annotated[
|
||||
Optional[List[str]],
|
||||
typer.Option("--relation-filter", help="Relation filters (repeatable)"),
|
||||
] = None,
|
||||
include_reasons: bool = typer.Option(
|
||||
True,
|
||||
"--include-reasons/--no-include-reasons",
|
||||
help="Include reason strings in impact output",
|
||||
),
|
||||
project: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(help="The project to use. If not provided, the default project will be used."),
|
||||
] = None,
|
||||
workspace: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
|
||||
] = None,
|
||||
local: bool = typer.Option(
|
||||
False, "--local", help="Force local API routing (ignore cloud mode)"
|
||||
),
|
||||
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
|
||||
):
|
||||
"""Get impact radius for a target node."""
|
||||
try:
|
||||
validate_routing_flags(local, cloud)
|
||||
with force_routing(local=local, cloud=cloud):
|
||||
result = run_with_cleanup(
|
||||
mcp_graph_impact(
|
||||
target=target,
|
||||
horizon=horizon,
|
||||
relation_filters=relation_filters or [],
|
||||
include_reasons=include_reasons,
|
||||
project=project,
|
||||
workspace=workspace,
|
||||
output_format="json",
|
||||
)
|
||||
)
|
||||
_print_json(result)
|
||||
except ValueError as e:
|
||||
typer.echo(f"Error: {e}", err=True)
|
||||
raise typer.Exit(1)
|
||||
except Exception as e: # pragma: no cover
|
||||
if not isinstance(e, typer.Exit):
|
||||
typer.echo(f"Error during graph_impact: {e}", err=True)
|
||||
raise typer.Exit(1)
|
||||
raise
|
||||
|
||||
|
||||
@tool_app.command("graph-health")
|
||||
def graph_health(
|
||||
scope: Annotated[Optional[str], typer.Option("--scope", help="Optional scope prefix")] = None,
|
||||
timeframe: Annotated[
|
||||
Optional[str], typer.Option("--timeframe", help="Optional timeframe filter")
|
||||
] = None,
|
||||
project: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(help="The project to use. If not provided, the default project will be used."),
|
||||
] = None,
|
||||
workspace: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
|
||||
] = None,
|
||||
local: bool = typer.Option(
|
||||
False, "--local", help="Force local API routing (ignore cloud mode)"
|
||||
),
|
||||
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
|
||||
):
|
||||
"""Get graph health metrics and issue candidates."""
|
||||
try:
|
||||
validate_routing_flags(local, cloud)
|
||||
with force_routing(local=local, cloud=cloud):
|
||||
result = run_with_cleanup(
|
||||
mcp_graph_health(
|
||||
scope=scope,
|
||||
timeframe=timeframe,
|
||||
project=project,
|
||||
workspace=workspace,
|
||||
output_format="json",
|
||||
)
|
||||
)
|
||||
_print_json(result)
|
||||
except ValueError as e:
|
||||
typer.echo(f"Error: {e}", err=True)
|
||||
raise typer.Exit(1)
|
||||
except Exception as e: # pragma: no cover
|
||||
if not isinstance(e, typer.Exit):
|
||||
typer.echo(f"Error during graph_health: {e}", err=True)
|
||||
raise typer.Exit(1)
|
||||
raise
|
||||
|
||||
|
||||
@tool_app.command("fcm-simulate")
|
||||
def fcm_simulate(
|
||||
actions_json: Annotated[
|
||||
str,
|
||||
typer.Option(
|
||||
"--actions-json",
|
||||
help='JSON array of actions, e.g. [{"node_id":"n1","delta":0.2}]',
|
||||
),
|
||||
],
|
||||
scenario_json: Annotated[
|
||||
Optional[str],
|
||||
typer.Option("--scenario-json", help="Optional JSON scenario object"),
|
||||
] = None,
|
||||
clamp_rules_json: Annotated[
|
||||
Optional[str],
|
||||
typer.Option("--clamp-rules-json", help="Optional JSON array of clamp rules"),
|
||||
] = None,
|
||||
project: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(help="The project to use. If not provided, the default project will be used."),
|
||||
] = None,
|
||||
workspace: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
|
||||
] = None,
|
||||
local: bool = typer.Option(
|
||||
False, "--local", help="Force local API routing (ignore cloud mode)"
|
||||
),
|
||||
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
|
||||
):
|
||||
"""Run an FCM simulation."""
|
||||
actions = _parse_json_option(actions_json, "--actions-json")
|
||||
scenario = _parse_json_option(scenario_json, "--scenario-json")
|
||||
clamp_rules = _parse_json_option(clamp_rules_json, "--clamp-rules-json")
|
||||
if not isinstance(actions, list):
|
||||
typer.echo("Invalid JSON for --actions-json: expected a JSON array", err=True)
|
||||
raise typer.Exit(1)
|
||||
if scenario is not None and not isinstance(scenario, dict):
|
||||
typer.echo("Invalid JSON for --scenario-json: expected a JSON object", err=True)
|
||||
raise typer.Exit(1)
|
||||
if clamp_rules is not None and not isinstance(clamp_rules, list):
|
||||
typer.echo("Invalid JSON for --clamp-rules-json: expected a JSON array", err=True)
|
||||
raise typer.Exit(1)
|
||||
|
||||
try:
|
||||
validate_routing_flags(local, cloud)
|
||||
with force_routing(local=local, cloud=cloud):
|
||||
result = run_with_cleanup(
|
||||
mcp_fcm_simulate(
|
||||
actions=actions,
|
||||
scenario=scenario,
|
||||
clamp_rules=clamp_rules,
|
||||
project=project,
|
||||
workspace=workspace,
|
||||
output_format="json",
|
||||
)
|
||||
)
|
||||
_print_json(result)
|
||||
except ValueError as e:
|
||||
typer.echo(f"Error: {e}", err=True)
|
||||
raise typer.Exit(1)
|
||||
except Exception as e: # pragma: no cover
|
||||
if not isinstance(e, typer.Exit):
|
||||
typer.echo(f"Error during fcm_simulate: {e}", err=True)
|
||||
raise typer.Exit(1)
|
||||
raise
|
||||
|
||||
|
||||
@tool_app.command("fcm-rank-actions")
|
||||
def fcm_rank_actions(
|
||||
goal: Annotated[str, typer.Argument(help="Goal node identifier")],
|
||||
constraints_json: Annotated[
|
||||
Optional[str],
|
||||
typer.Option("--constraints-json", help="Optional JSON object of ranking constraints"),
|
||||
] = None,
|
||||
top_k: int = typer.Option(10, "--top-k", help="Number of recommendations to return"),
|
||||
project: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(help="The project to use. If not provided, the default project will be used."),
|
||||
] = None,
|
||||
workspace: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
|
||||
] = None,
|
||||
local: bool = typer.Option(
|
||||
False, "--local", help="Force local API routing (ignore cloud mode)"
|
||||
),
|
||||
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
|
||||
):
|
||||
"""Rank intervention actions for an FCM goal."""
|
||||
constraints = _parse_json_option(constraints_json, "--constraints-json")
|
||||
if constraints is not None and not isinstance(constraints, dict):
|
||||
typer.echo("Invalid JSON for --constraints-json: expected a JSON object", err=True)
|
||||
raise typer.Exit(1)
|
||||
|
||||
try:
|
||||
validate_routing_flags(local, cloud)
|
||||
with force_routing(local=local, cloud=cloud):
|
||||
result = run_with_cleanup(
|
||||
mcp_fcm_rank_actions(
|
||||
goal=goal,
|
||||
constraints=constraints,
|
||||
top_k=top_k,
|
||||
project=project,
|
||||
workspace=workspace,
|
||||
output_format="json",
|
||||
)
|
||||
)
|
||||
_print_json(result)
|
||||
except ValueError as e:
|
||||
typer.echo(f"Error: {e}", err=True)
|
||||
raise typer.Exit(1)
|
||||
except Exception as e: # pragma: no cover
|
||||
if not isinstance(e, typer.Exit):
|
||||
typer.echo(f"Error during fcm_rank_actions: {e}", err=True)
|
||||
raise typer.Exit(1)
|
||||
raise
|
||||
|
||||
|
||||
@tool_app.command("fcm-import-model")
|
||||
def fcm_import_model(
|
||||
source: Annotated[str, typer.Argument(help="Source path or URI for import payload")],
|
||||
format: Annotated[
|
||||
str,
|
||||
typer.Option("--format", help="Import format (currently csv_bundle_v1)"),
|
||||
] = "csv_bundle_v1",
|
||||
merge_mode: Annotated[
|
||||
str,
|
||||
typer.Option("--merge-mode", help="Merge strategy: replace or upsert"),
|
||||
] = "upsert",
|
||||
project: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(help="The project to use. If not provided, the default project will be used."),
|
||||
] = None,
|
||||
workspace: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
|
||||
] = None,
|
||||
local: bool = typer.Option(
|
||||
False, "--local", help="Force local API routing (ignore cloud mode)"
|
||||
),
|
||||
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
|
||||
):
|
||||
"""Import an FCM model."""
|
||||
try:
|
||||
validate_routing_flags(local, cloud)
|
||||
with force_routing(local=local, cloud=cloud):
|
||||
result = run_with_cleanup(
|
||||
mcp_fcm_import_model(
|
||||
source=source,
|
||||
format=format, # pyright: ignore[reportArgumentType]
|
||||
merge_mode=merge_mode, # pyright: ignore[reportArgumentType]
|
||||
project=project,
|
||||
workspace=workspace,
|
||||
output_format="json",
|
||||
)
|
||||
)
|
||||
_print_json(result)
|
||||
except ValueError as e:
|
||||
typer.echo(f"Error: {e}", err=True)
|
||||
raise typer.Exit(1)
|
||||
except Exception as e: # pragma: no cover
|
||||
if not isinstance(e, typer.Exit):
|
||||
typer.echo(f"Error during fcm_import_model: {e}", err=True)
|
||||
raise typer.Exit(1)
|
||||
raise
|
||||
|
||||
|
||||
@tool_app.command("fcm-export-model")
|
||||
def fcm_export_model(
|
||||
format: Annotated[
|
||||
str,
|
||||
typer.Option("--format", help="Export format (currently csv_bundle_v1)"),
|
||||
] = "csv_bundle_v1",
|
||||
selection_json: Annotated[
|
||||
Optional[str],
|
||||
typer.Option("--selection-json", help="Optional JSON object selection payload"),
|
||||
] = None,
|
||||
project: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(help="The project to use. If not provided, the default project will be used."),
|
||||
] = None,
|
||||
workspace: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
|
||||
] = None,
|
||||
local: bool = typer.Option(
|
||||
False, "--local", help="Force local API routing (ignore cloud mode)"
|
||||
),
|
||||
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
|
||||
):
|
||||
"""Export an FCM model."""
|
||||
selection = _parse_json_option(selection_json, "--selection-json")
|
||||
if selection is not None and not isinstance(selection, dict):
|
||||
typer.echo("Invalid JSON for --selection-json: expected a JSON object", err=True)
|
||||
raise typer.Exit(1)
|
||||
|
||||
try:
|
||||
validate_routing_flags(local, cloud)
|
||||
with force_routing(local=local, cloud=cloud):
|
||||
result = run_with_cleanup(
|
||||
mcp_fcm_export_model(
|
||||
format=format, # pyright: ignore[reportArgumentType]
|
||||
selection=selection,
|
||||
project=project,
|
||||
workspace=workspace,
|
||||
output_format="json",
|
||||
)
|
||||
)
|
||||
_print_json(result)
|
||||
except ValueError as e:
|
||||
typer.echo(f"Error: {e}", err=True)
|
||||
raise typer.Exit(1)
|
||||
except Exception as e: # pragma: no cover
|
||||
if not isinstance(e, typer.Exit):
|
||||
typer.echo(f"Error during fcm_export_model: {e}", err=True)
|
||||
raise typer.Exit(1)
|
||||
raise
|
||||
|
||||
|
||||
@tool_app.command("search-notes")
|
||||
def search_notes(
|
||||
query: Annotated[
|
||||
@@ -423,12 +797,9 @@ def search_notes(
|
||||
Optional[str],
|
||||
typer.Option(help="The project to use. If not provided, the default project will be used."),
|
||||
] = None,
|
||||
project_id: Annotated[
|
||||
workspace: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(
|
||||
"--project-id",
|
||||
help="Project external_id (UUID). Takes precedence over --project; use to disambiguate same-named projects across cloud workspaces.",
|
||||
),
|
||||
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
|
||||
] = None,
|
||||
local: bool = typer.Option(
|
||||
False, "--local", help="Force local API routing (ignore cloud mode)"
|
||||
@@ -500,7 +871,7 @@ def search_notes(
|
||||
mcp_search(
|
||||
query=query or None,
|
||||
project=project,
|
||||
project_id=project_id,
|
||||
workspace=workspace,
|
||||
search_type=search_type,
|
||||
output_format="json",
|
||||
page=page,
|
||||
@@ -610,12 +981,9 @@ def schema_validate(
|
||||
Optional[str],
|
||||
typer.Option(help="The project to use. If not provided, the default project will be used."),
|
||||
] = None,
|
||||
project_id: Annotated[
|
||||
workspace: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(
|
||||
"--project-id",
|
||||
help="Project external_id (UUID). Takes precedence over --project; use to disambiguate same-named projects across cloud workspaces.",
|
||||
),
|
||||
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
|
||||
] = None,
|
||||
local: bool = typer.Option(
|
||||
False, "--local", help="Force local API routing (ignore cloud mode)"
|
||||
@@ -650,7 +1018,7 @@ def schema_validate(
|
||||
note_type=note_type,
|
||||
identifier=identifier,
|
||||
project=project,
|
||||
project_id=project_id,
|
||||
workspace=workspace,
|
||||
output_format="json",
|
||||
)
|
||||
)
|
||||
@@ -681,12 +1049,9 @@ def schema_infer(
|
||||
Optional[str],
|
||||
typer.Option(help="The project to use. If not provided, the default project will be used."),
|
||||
] = None,
|
||||
project_id: Annotated[
|
||||
workspace: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(
|
||||
"--project-id",
|
||||
help="Project external_id (UUID). Takes precedence over --project; use to disambiguate same-named projects across cloud workspaces.",
|
||||
),
|
||||
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
|
||||
] = None,
|
||||
local: bool = typer.Option(
|
||||
False, "--local", help="Force local API routing (ignore cloud mode)"
|
||||
@@ -710,7 +1075,7 @@ def schema_infer(
|
||||
note_type=note_type,
|
||||
threshold=threshold,
|
||||
project=project,
|
||||
project_id=project_id,
|
||||
workspace=workspace,
|
||||
output_format="json",
|
||||
)
|
||||
)
|
||||
@@ -738,12 +1103,9 @@ def schema_diff(
|
||||
Optional[str],
|
||||
typer.Option(help="The project to use. If not provided, the default project will be used."),
|
||||
] = None,
|
||||
project_id: Annotated[
|
||||
workspace: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(
|
||||
"--project-id",
|
||||
help="Project external_id (UUID). Takes precedence over --project; use to disambiguate same-named projects across cloud workspaces.",
|
||||
),
|
||||
typer.Option(help="Cloud workspace tenant ID or unique name to route this request."),
|
||||
] = None,
|
||||
local: bool = typer.Option(
|
||||
False, "--local", help="Force local API routing (ignore cloud mode)"
|
||||
@@ -765,7 +1127,7 @@ def schema_diff(
|
||||
mcp_schema_diff(
|
||||
note_type=note_type,
|
||||
project=project,
|
||||
project_id=project_id,
|
||||
workspace=workspace,
|
||||
output_format="json",
|
||||
)
|
||||
)
|
||||
|
||||
@@ -1,40 +0,0 @@
|
||||
"""Manual update command for Basic Memory CLI."""
|
||||
|
||||
import typer
|
||||
from rich.console import Console
|
||||
|
||||
from basic_memory.cli.app import app
|
||||
from basic_memory.cli.auto_update import AutoUpdateStatus, run_auto_update
|
||||
|
||||
console = Console()
|
||||
|
||||
|
||||
@app.command("update")
|
||||
def update(
|
||||
check: bool = typer.Option(
|
||||
False,
|
||||
"--check",
|
||||
help="Check for updates only (do not install).",
|
||||
),
|
||||
) -> None:
|
||||
"""Check for updates and install when supported."""
|
||||
result = run_auto_update(force=True, check_only=check, silent=False)
|
||||
|
||||
if result.status == AutoUpdateStatus.FAILED:
|
||||
detail = f" {result.error}" if result.error else ""
|
||||
console.print(f"[red]{result.message or 'Update failed.'}{detail}[/red]")
|
||||
raise typer.Exit(1)
|
||||
|
||||
if result.status == AutoUpdateStatus.UPDATED:
|
||||
console.print(f"[green]{result.message or 'Basic Memory updated successfully.'}[/green]")
|
||||
return
|
||||
|
||||
if result.status == AutoUpdateStatus.UP_TO_DATE:
|
||||
console.print(f"[green]{result.message or 'Basic Memory is up to date.'}[/green]")
|
||||
return
|
||||
|
||||
if result.status == AutoUpdateStatus.UPDATE_AVAILABLE:
|
||||
console.print(f"[cyan]{result.message or 'Update available.'}[/cyan]")
|
||||
return
|
||||
|
||||
console.print(f"[dim]{result.message or 'No update action was performed.'}[/dim]")
|
||||
@@ -24,12 +24,10 @@ if not _version_only_invocation(sys.argv[1:]):
|
||||
import_claude_projects,
|
||||
import_memory_json,
|
||||
mcp,
|
||||
orphans,
|
||||
project,
|
||||
schema,
|
||||
status,
|
||||
tool,
|
||||
update,
|
||||
)
|
||||
|
||||
warnings.filterwarnings("ignore") # pragma: no cover
|
||||
|
||||
@@ -12,7 +12,7 @@ from basic_memory.config import ConfigManager
|
||||
|
||||
OSS_DISCOUNT_CODE = "BMFOSS"
|
||||
CLOUD_LEARN_MORE_URL = (
|
||||
"https://basicmemory.com?utm_source=bm-foss&utm_medium=promo&utm_campaign=cloud-upsell"
|
||||
"https://basicmemory.com?utm_source=bm-cli&utm_medium=promo&utm_campaign=cloud-upsell"
|
||||
)
|
||||
|
||||
|
||||
|
||||
+36
-244
@@ -6,42 +6,26 @@ import os
|
||||
import shutil
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from enum import Enum
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any, Dict, Literal, Optional, List, Tuple
|
||||
from typing import Any, Dict, Literal, Optional, List, Tuple
|
||||
from enum import Enum
|
||||
|
||||
from loguru import logger
|
||||
from pydantic import AliasChoices, BaseModel, Field, model_validator
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
||||
from basic_memory import __version__
|
||||
from basic_memory.telemetry import configure_telemetry
|
||||
from basic_memory.utils import setup_logging, generate_permalink
|
||||
|
||||
|
||||
DATABASE_NAME = "memory.db"
|
||||
APP_DATABASE_NAME = "memory.db" # Using the same name but in the app directory
|
||||
DATA_DIR_NAME = "basic-memory"
|
||||
DATA_DIR_NAME = ".basic-memory"
|
||||
CONFIG_FILE_NAME = "config.json"
|
||||
WATCH_STATUS_JSON = "watch-status.json"
|
||||
CONFIG_DIR_MODE = 0o700
|
||||
CONFIG_FILE_MODE = 0o600
|
||||
|
||||
Environment = Literal["test", "dev", "user"]
|
||||
|
||||
|
||||
def _secure_config_dir(path: Path) -> None:
|
||||
"""Restrict config directory permissions on platforms with POSIX modes."""
|
||||
if os.name != "nt":
|
||||
path.chmod(CONFIG_DIR_MODE)
|
||||
|
||||
|
||||
def _secure_config_file(path: Path) -> None:
|
||||
"""Restrict config file permissions because config can contain cloud credentials."""
|
||||
if os.name != "nt":
|
||||
path.chmod(CONFIG_FILE_MODE)
|
||||
|
||||
|
||||
class ProjectMode(str, Enum):
|
||||
"""Per-project routing mode."""
|
||||
|
||||
@@ -64,47 +48,6 @@ def _default_semantic_search_enabled() -> bool:
|
||||
)
|
||||
|
||||
|
||||
def resolve_data_dir() -> Path:
|
||||
"""Resolve the Basic Memory data directory.
|
||||
|
||||
Single source of truth for the per-user state directory. Honors
|
||||
``BASIC_MEMORY_CONFIG_DIR`` so each process/worktree can isolate config
|
||||
and database state; otherwise falls back to ``<user home>/.basic-memory``,
|
||||
and then to ``XDG_CONFIG_HOME``.
|
||||
|
||||
Cross-platform: ``Path.home()`` reads ``$HOME`` on POSIX and
|
||||
``%USERPROFILE%`` on Windows, so there's no need to check ``$HOME``
|
||||
explicitly here.
|
||||
"""
|
||||
if basic_memory_dir := os.getenv("BASIC_MEMORY_CONFIG_DIR"):
|
||||
return Path(basic_memory_dir)
|
||||
if xdg_config := os.getenv("XDG_CONFIG_HOME"):
|
||||
return Path(xdg_config) / DATA_DIR_NAME
|
||||
return Path.home() / ("." + DATA_DIR_NAME)
|
||||
|
||||
|
||||
def default_fastembed_cache_dir() -> str:
|
||||
"""Return the default cache directory used for FastEmbed model artifacts.
|
||||
|
||||
Resolution order:
|
||||
1. ``FASTEMBED_CACHE_PATH`` env var — honors FastEmbed's own convention
|
||||
so users who already configure it through the environment keep working.
|
||||
2. ``<basic-memory data dir>/fastembed_cache`` — the same stable,
|
||||
user-writable directory Basic Memory already uses for config and
|
||||
the default SQLite database. Honors ``BASIC_MEMORY_CONFIG_DIR``.
|
||||
|
||||
Why not ``tempfile.gettempdir()``?
|
||||
FastEmbed's own default is ``<system tmp>/fastembed_cache``, which is
|
||||
ephemeral in many sandboxed MCP runtimes (e.g. Codex CLI wipes /tmp
|
||||
between invocations). The model then disappears and every subsequent
|
||||
ONNX load raises ``NO_SUCHFILE``. Persisting the cache under the
|
||||
per-user data directory works identically on macOS, Linux, and Windows.
|
||||
"""
|
||||
if env_override := os.getenv("FASTEMBED_CACHE_PATH"):
|
||||
return env_override
|
||||
return str(resolve_data_dir() / "fastembed_cache")
|
||||
|
||||
|
||||
@dataclass
|
||||
class ProjectConfig:
|
||||
"""Configuration for a specific basic-memory project."""
|
||||
@@ -177,23 +120,16 @@ class ProjectEntry(BaseModel):
|
||||
class BasicMemoryConfig(BaseSettings):
|
||||
"""Pydantic model for Basic Memory global configuration."""
|
||||
|
||||
if TYPE_CHECKING:
|
||||
# Pydantic accepts raw constructor data and validates/coerces it at runtime.
|
||||
# Model attributes remain strongly typed after initialization.
|
||||
def __init__(self, **data: Any) -> None: ...
|
||||
|
||||
env: Environment = Field(default="dev", description="Environment name")
|
||||
|
||||
projects: Dict[str, ProjectEntry] = Field(
|
||||
default_factory=lambda: (
|
||||
{
|
||||
"main": ProjectEntry(
|
||||
path=str(Path(os.getenv("BASIC_MEMORY_HOME", Path.home() / "basic-memory")))
|
||||
)
|
||||
}
|
||||
if os.getenv("BASIC_MEMORY_HOME")
|
||||
else {}
|
||||
),
|
||||
default_factory=lambda: {
|
||||
"main": ProjectEntry(
|
||||
path=str(Path(os.getenv("BASIC_MEMORY_HOME", Path.home() / "basic-memory")))
|
||||
)
|
||||
}
|
||||
if os.getenv("BASIC_MEMORY_HOME")
|
||||
else {},
|
||||
description="Mapping of project names to their ProjectEntry configuration",
|
||||
)
|
||||
default_project: Optional[str] = Field(
|
||||
@@ -204,24 +140,6 @@ class BasicMemoryConfig(BaseSettings):
|
||||
# overridden by ~/.basic-memory/config.json
|
||||
log_level: str = "INFO"
|
||||
|
||||
# Optional Logfire telemetry (disabled by default)
|
||||
logfire_enabled: bool = Field(
|
||||
default=False,
|
||||
description="Enable Logfire instrumentation for local development or managed deployments.",
|
||||
)
|
||||
logfire_send_to_logfire: bool = Field(
|
||||
default=False,
|
||||
description="When true, allow Logfire to export telemetry to the configured backend.",
|
||||
)
|
||||
logfire_service_name: str = Field(
|
||||
default="basic-memory",
|
||||
description="Base service name used when constructing entrypoint-specific Logfire service names.",
|
||||
)
|
||||
logfire_environment: str | None = Field(
|
||||
default=None,
|
||||
description="Optional override for Logfire environment. Defaults to env when unset.",
|
||||
)
|
||||
|
||||
# Database configuration
|
||||
database_backend: DatabaseBackend = Field(
|
||||
default=DatabaseBackend.SQLITE,
|
||||
@@ -250,42 +168,19 @@ class BasicMemoryConfig(BaseSettings):
|
||||
default=None,
|
||||
description="Embedding vector dimensions. Auto-detected from provider if not set (384 for FastEmbed, 1536 for OpenAI).",
|
||||
)
|
||||
# Trigger: full local rebuilds spend most of their time waiting behind shared
|
||||
# embed flushes, not constructing vectors themselves.
|
||||
# Why: smaller FastEmbed batches cut queue wait far more than they increase
|
||||
# write overhead on real-world projects, which makes full reindex materially faster.
|
||||
# Outcome: default to the smaller local/cloud-safe batch size we benchmarked as
|
||||
# the current best end-to-end setting in the shared vector sync pipeline.
|
||||
semantic_embedding_batch_size: int = Field(
|
||||
default=2,
|
||||
default=64,
|
||||
description="Batch size for embedding generation.",
|
||||
gt=0,
|
||||
)
|
||||
semantic_embedding_request_concurrency: int = Field(
|
||||
default=4,
|
||||
description="Maximum number of concurrent provider requests for batched embedding generation when the active provider supports request-level concurrency.",
|
||||
gt=0,
|
||||
)
|
||||
semantic_embedding_sync_batch_size: int = Field(
|
||||
default=2,
|
||||
default=64,
|
||||
description="Batch size for vector sync orchestration flushes.",
|
||||
gt=0,
|
||||
)
|
||||
semantic_postgres_prepare_concurrency: int = Field(
|
||||
default=4,
|
||||
description="Number of Postgres entity prepare tasks to run concurrently during vector sync. Postgres only; keep this low to avoid overdriving the database connection pool.",
|
||||
gt=0,
|
||||
le=16,
|
||||
)
|
||||
semantic_embedding_cache_dir: str | None = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"Optional override for the FastEmbed model cache directory. "
|
||||
"When unset, Basic Memory resolves this at runtime to "
|
||||
"<basic-memory data dir>/fastembed_cache (or FASTEMBED_CACHE_PATH "
|
||||
"when that env var is set) so the model persists across runs "
|
||||
"without hardcoding a path into config.json."
|
||||
),
|
||||
description="Optional cache directory for FastEmbed model artifacts.",
|
||||
)
|
||||
semantic_embedding_threads: int | None = Field(
|
||||
default=None,
|
||||
@@ -297,11 +192,6 @@ class BasicMemoryConfig(BaseSettings):
|
||||
description="Optional FastEmbed embed() parallelism override.",
|
||||
gt=0,
|
||||
)
|
||||
import_upload_max_bytes: int = Field(
|
||||
default=100 * 1024 * 1024,
|
||||
description="Maximum uploaded JSON export size accepted by API import endpoints.",
|
||||
gt=0,
|
||||
)
|
||||
semantic_vector_k: int = Field(
|
||||
default=100,
|
||||
description="Vector candidate count for vector and hybrid retrieval.",
|
||||
@@ -313,12 +203,6 @@ class BasicMemoryConfig(BaseSettings):
|
||||
ge=0.0,
|
||||
le=1.0,
|
||||
)
|
||||
default_search_type: Literal["text", "vector", "hybrid"] | None = Field(
|
||||
default=None,
|
||||
description="Default search type for search_notes when not specified per-query. "
|
||||
"Valid values: text, vector, hybrid. "
|
||||
"When unset, defaults to 'hybrid' if semantic search is enabled, otherwise 'text'.",
|
||||
)
|
||||
|
||||
# Database connection pool configuration (Postgres only)
|
||||
db_pool_size: int = Field(
|
||||
@@ -370,31 +254,6 @@ class BasicMemoryConfig(BaseSettings):
|
||||
description="Maximum number of files to process concurrently during sync. Limits memory usage on large projects (2000+ files). Lower values reduce memory consumption.",
|
||||
gt=0,
|
||||
)
|
||||
index_batch_size: int = Field(
|
||||
default=32,
|
||||
description="Maximum number of changed files to load into one indexing batch.",
|
||||
gt=0,
|
||||
)
|
||||
index_batch_max_bytes: int = Field(
|
||||
default=8 * 1024 * 1024,
|
||||
description="Maximum total bytes to load into one indexing batch. Large files still run as single-file batches.",
|
||||
gt=0,
|
||||
)
|
||||
index_parse_max_concurrent: int = Field(
|
||||
default=8,
|
||||
description="Maximum number of markdown parse tasks to run concurrently inside one indexing batch.",
|
||||
gt=0,
|
||||
)
|
||||
index_entity_max_concurrent: int = Field(
|
||||
default=4,
|
||||
description="Maximum number of entity create/update tasks to run concurrently inside one indexing batch.",
|
||||
gt=0,
|
||||
)
|
||||
index_metadata_update_max_concurrent: int = Field(
|
||||
default=4,
|
||||
description="Maximum number of metadata/search refresh tasks to run concurrently inside one indexing batch.",
|
||||
gt=0,
|
||||
)
|
||||
|
||||
kebab_filenames: bool = Field(
|
||||
default=False,
|
||||
@@ -492,22 +351,6 @@ class BasicMemoryConfig(BaseSettings):
|
||||
description="Most recent cloud promo version shown in CLI.",
|
||||
)
|
||||
|
||||
auto_update: bool = Field(
|
||||
default=True,
|
||||
description="Enable automatic CLI update checks and installs when supported.",
|
||||
)
|
||||
|
||||
update_check_interval: int = Field(
|
||||
default=86400,
|
||||
description="Seconds between automatic update checks.",
|
||||
gt=0,
|
||||
)
|
||||
|
||||
auto_update_last_checked_at: Optional[datetime] = Field(
|
||||
default=None,
|
||||
description="Timestamp of the last attempted automatic update check.",
|
||||
)
|
||||
|
||||
cloud_api_key: Optional[str] = Field(
|
||||
default=None,
|
||||
description="API key for cloud access (bmc_ prefixed). Account-level, not per-project.",
|
||||
@@ -777,17 +620,15 @@ class BasicMemoryConfig(BaseSettings):
|
||||
@property
|
||||
def data_dir_path(self) -> Path:
|
||||
"""Get app state directory for config and default SQLite database."""
|
||||
return resolve_data_dir()
|
||||
if config_dir := os.getenv("BASIC_MEMORY_CONFIG_DIR"):
|
||||
return Path(config_dir)
|
||||
|
||||
home = os.getenv("HOME", Path.home())
|
||||
return Path(home) / DATA_DIR_NAME
|
||||
|
||||
|
||||
# Module-level cache for configuration
|
||||
_CONFIG_CACHE: Optional[BasicMemoryConfig] = None
|
||||
# Track config file mtime+size so cross-process changes (e.g. `bm project set-cloud`
|
||||
# in a separate terminal) invalidate the cache in long-lived processes like the
|
||||
# MCP stdio server. Using both mtime and size guards against coarse-granularity
|
||||
# filesystems where two writes within the same second share the same mtime.
|
||||
_CONFIG_MTIME: Optional[float] = None
|
||||
_CONFIG_SIZE: Optional[int] = None
|
||||
|
||||
|
||||
class ConfigManager:
|
||||
@@ -795,12 +636,20 @@ class ConfigManager:
|
||||
|
||||
def __init__(self) -> None:
|
||||
"""Initialize the configuration manager."""
|
||||
self.config_dir = resolve_data_dir()
|
||||
home = os.getenv("HOME", Path.home())
|
||||
if isinstance(home, str):
|
||||
home = Path(home)
|
||||
|
||||
# Allow override via environment variable
|
||||
if config_dir := os.getenv("BASIC_MEMORY_CONFIG_DIR"):
|
||||
self.config_dir = Path(config_dir)
|
||||
else:
|
||||
self.config_dir = home / DATA_DIR_NAME
|
||||
|
||||
self.config_file = self.config_dir / CONFIG_FILE_NAME
|
||||
|
||||
# Ensure config directory exists
|
||||
self.config_dir.mkdir(parents=True, exist_ok=True)
|
||||
_secure_config_dir(self.config_dir)
|
||||
|
||||
@property
|
||||
def config(self) -> BasicMemoryConfig:
|
||||
@@ -813,38 +662,13 @@ class ConfigManager:
|
||||
Environment variables take precedence over file config values,
|
||||
following Pydantic Settings best practices.
|
||||
|
||||
Uses module-level cache with file mtime validation so that
|
||||
cross-process config changes (e.g. `bm project set-cloud` in a
|
||||
separate terminal) are picked up by long-lived processes like
|
||||
the MCP stdio server.
|
||||
Uses module-level cache for performance across ConfigManager instances.
|
||||
"""
|
||||
global _CONFIG_CACHE, _CONFIG_MTIME, _CONFIG_SIZE
|
||||
global _CONFIG_CACHE
|
||||
|
||||
# Trigger: cached config exists but the on-disk file may have been
|
||||
# modified by another process (CLI command in a different terminal).
|
||||
# Why: the MCP server is long-lived; without this check it would
|
||||
# serve stale project routing forever.
|
||||
# Outcome: cheap os.stat() per access; re-read only when mtime or size differs.
|
||||
# Return cached config if available
|
||||
if _CONFIG_CACHE is not None:
|
||||
try:
|
||||
st = self.config_file.stat()
|
||||
current_mtime = st.st_mtime
|
||||
current_size = st.st_size
|
||||
except OSError:
|
||||
current_mtime = None
|
||||
current_size = None
|
||||
|
||||
if (
|
||||
current_mtime is not None
|
||||
and current_mtime == _CONFIG_MTIME
|
||||
and current_size == _CONFIG_SIZE
|
||||
):
|
||||
return _CONFIG_CACHE
|
||||
|
||||
# mtime/size changed or file gone — invalidate and fall through to re-read
|
||||
_CONFIG_CACHE = None
|
||||
_CONFIG_MTIME = None
|
||||
_CONFIG_SIZE = None
|
||||
return _CONFIG_CACHE
|
||||
|
||||
if self.config_file.exists():
|
||||
try:
|
||||
@@ -899,21 +723,11 @@ class ConfigManager:
|
||||
|
||||
_CONFIG_CACHE = BasicMemoryConfig(**merged_data)
|
||||
|
||||
# Record mtime+size so subsequent calls detect cross-process changes
|
||||
try:
|
||||
st = self.config_file.stat()
|
||||
_CONFIG_MTIME = st.st_mtime
|
||||
_CONFIG_SIZE = st.st_size
|
||||
except OSError:
|
||||
_CONFIG_MTIME = None
|
||||
_CONFIG_SIZE = None
|
||||
|
||||
# Re-save to normalize legacy config into current format
|
||||
if needs_resave:
|
||||
# Create backup before overwriting so users can revert if needed
|
||||
backup_path = self.config_file.with_suffix(".json.bak")
|
||||
shutil.copy2(self.config_file, backup_path)
|
||||
_secure_config_file(backup_path)
|
||||
logger.info(f"Migrating config to current format (backup: {backup_path})")
|
||||
save_basic_memory_config(self.config_file, _CONFIG_CACHE)
|
||||
|
||||
@@ -939,12 +753,10 @@ class ConfigManager:
|
||||
|
||||
def save_config(self, config: BasicMemoryConfig) -> None:
|
||||
"""Save configuration to file and invalidate cache."""
|
||||
global _CONFIG_CACHE, _CONFIG_MTIME, _CONFIG_SIZE
|
||||
global _CONFIG_CACHE
|
||||
save_basic_memory_config(self.config_file, config)
|
||||
# Invalidate cache so next load_config() reads fresh data
|
||||
_CONFIG_CACHE = None
|
||||
_CONFIG_MTIME = None
|
||||
_CONFIG_SIZE = None
|
||||
|
||||
@property
|
||||
def projects(self) -> Dict[str, str]:
|
||||
@@ -1069,12 +881,9 @@ def has_cloud_credentials(config: BasicMemoryConfig) -> bool:
|
||||
def save_basic_memory_config(file_path: Path, config: BasicMemoryConfig) -> None:
|
||||
"""Save configuration to file."""
|
||||
try:
|
||||
file_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
_secure_config_dir(file_path.parent)
|
||||
# Use model_dump with mode='json' to serialize datetime objects properly
|
||||
config_dict = config.model_dump(mode="json")
|
||||
file_path.write_text(json.dumps(config_dict, indent=2))
|
||||
_secure_config_file(file_path)
|
||||
except Exception as e: # pragma: no cover
|
||||
logger.error(f"Failed to save config: {e}")
|
||||
|
||||
@@ -1082,50 +891,33 @@ def save_basic_memory_config(file_path: Path, config: BasicMemoryConfig) -> None
|
||||
# Logging initialization functions for different entry points
|
||||
|
||||
|
||||
def _configure_logfire_for_entrypoint(entrypoint: str) -> None:
|
||||
"""Configure optional Logfire telemetry for a specific entrypoint."""
|
||||
config = ConfigManager().config
|
||||
service_name = f"{config.logfire_service_name}-{entrypoint}"
|
||||
environment = config.logfire_environment or config.env
|
||||
configure_telemetry(
|
||||
service_name=service_name,
|
||||
environment=environment,
|
||||
service_version=__version__,
|
||||
enable_logfire=config.logfire_enabled,
|
||||
send_to_logfire=config.logfire_send_to_logfire,
|
||||
)
|
||||
|
||||
|
||||
def init_cli_logging() -> None:
|
||||
def init_cli_logging() -> None: # pragma: no cover
|
||||
"""Initialize logging for CLI commands - file only.
|
||||
|
||||
CLI commands should not log to stdout to avoid interfering with
|
||||
command output and shell integration.
|
||||
"""
|
||||
log_level = os.getenv("BASIC_MEMORY_LOG_LEVEL", "INFO")
|
||||
_configure_logfire_for_entrypoint("cli")
|
||||
setup_logging(log_level=log_level, log_to_file=True)
|
||||
|
||||
|
||||
def init_mcp_logging() -> None:
|
||||
def init_mcp_logging() -> None: # pragma: no cover
|
||||
"""Initialize logging for MCP server - file only.
|
||||
|
||||
MCP server must not log to stdout as it would corrupt the
|
||||
JSON-RPC protocol communication.
|
||||
"""
|
||||
log_level = os.getenv("BASIC_MEMORY_LOG_LEVEL", "INFO")
|
||||
_configure_logfire_for_entrypoint("mcp")
|
||||
setup_logging(log_level=log_level, log_to_file=True)
|
||||
|
||||
|
||||
def init_api_logging() -> None:
|
||||
def init_api_logging() -> None: # pragma: no cover
|
||||
"""Initialize logging for API server.
|
||||
|
||||
Cloud mode (BASIC_MEMORY_CLOUD_MODE=1): stdout with structured context
|
||||
Local mode: file only
|
||||
"""
|
||||
log_level = os.getenv("BASIC_MEMORY_LOG_LEVEL", "INFO")
|
||||
_configure_logfire_for_entrypoint("api")
|
||||
cloud_mode = os.getenv("BASIC_MEMORY_CLOUD_MODE", "").lower() in ("1", "true")
|
||||
if cloud_mode:
|
||||
setup_logging(log_level=log_level, log_to_stdout=True, structured_context=True)
|
||||
|
||||
+123
-2
@@ -43,6 +43,104 @@ if sys.platform == "win32": # pragma: no cover
|
||||
_engine: Optional[AsyncEngine] = None
|
||||
_session_maker: Optional[async_sessionmaker[AsyncSession]] = None
|
||||
|
||||
# Alembic revision that enables one-time automatic embedding backfill.
|
||||
SEMANTIC_EMBEDDING_BACKFILL_REVISION = "i2c3d4e5f6g7"
|
||||
|
||||
|
||||
async def _load_applied_alembic_revisions(
|
||||
session_maker: async_sessionmaker[AsyncSession],
|
||||
) -> set[str]:
|
||||
"""Load applied Alembic revisions from alembic_version.
|
||||
|
||||
Returns an empty set when the version table does not exist yet
|
||||
(fresh database before first migration).
|
||||
"""
|
||||
try:
|
||||
async with scoped_session(session_maker) as session:
|
||||
result = await session.execute(text("SELECT version_num FROM alembic_version"))
|
||||
return {str(row[0]) for row in result.fetchall() if row[0]}
|
||||
except Exception as exc:
|
||||
error_message = str(exc).lower()
|
||||
if "alembic_version" in error_message and (
|
||||
"no such table" in error_message or "does not exist" in error_message
|
||||
):
|
||||
return set()
|
||||
raise
|
||||
|
||||
|
||||
def _should_run_semantic_embedding_backfill(
|
||||
revisions_before_upgrade: set[str],
|
||||
revisions_after_upgrade: set[str],
|
||||
) -> bool:
|
||||
"""Check if this migration run newly applied the backfill-trigger revision."""
|
||||
return (
|
||||
SEMANTIC_EMBEDDING_BACKFILL_REVISION in revisions_after_upgrade
|
||||
and SEMANTIC_EMBEDDING_BACKFILL_REVISION not in revisions_before_upgrade
|
||||
)
|
||||
|
||||
|
||||
async def _run_semantic_embedding_backfill(
|
||||
app_config: BasicMemoryConfig,
|
||||
session_maker: async_sessionmaker[AsyncSession],
|
||||
) -> None:
|
||||
"""Backfill semantic embeddings for all active projects/entities."""
|
||||
if not app_config.semantic_search_enabled:
|
||||
logger.info("Skipping automatic semantic embedding backfill: semantic search is disabled.")
|
||||
return
|
||||
|
||||
async with scoped_session(session_maker) as session:
|
||||
project_result = await session.execute(
|
||||
text("SELECT id, name FROM project WHERE is_active = :is_active ORDER BY id"),
|
||||
{"is_active": True},
|
||||
)
|
||||
projects = [(int(row[0]), str(row[1])) for row in project_result.fetchall()]
|
||||
|
||||
if not projects:
|
||||
logger.info("Skipping automatic semantic embedding backfill: no active projects found.")
|
||||
return
|
||||
|
||||
repository_class = (
|
||||
PostgresSearchRepository
|
||||
if app_config.database_backend == DatabaseBackend.POSTGRES
|
||||
else SQLiteSearchRepository
|
||||
)
|
||||
|
||||
total_entities = 0
|
||||
for project_id, project_name in projects:
|
||||
async with scoped_session(session_maker) as session:
|
||||
entity_result = await session.execute(
|
||||
text("SELECT id FROM entity WHERE project_id = :project_id ORDER BY id"),
|
||||
{"project_id": project_id},
|
||||
)
|
||||
entity_ids = [int(row[0]) for row in entity_result.fetchall()]
|
||||
|
||||
if not entity_ids:
|
||||
continue
|
||||
|
||||
total_entities += len(entity_ids)
|
||||
logger.info(
|
||||
"Automatic semantic embedding backfill: "
|
||||
f"project={project_name}, entities={len(entity_ids)}"
|
||||
)
|
||||
|
||||
search_repository = repository_class(
|
||||
session_maker,
|
||||
project_id=project_id,
|
||||
app_config=app_config,
|
||||
)
|
||||
batch_result = await search_repository.sync_entity_vectors_batch(entity_ids)
|
||||
if batch_result.entities_failed > 0:
|
||||
logger.warning(
|
||||
"Automatic semantic embedding backfill encountered entity failures: "
|
||||
f"project={project_name}, failed={batch_result.entities_failed}, "
|
||||
f"failed_entity_ids={batch_result.failed_entity_ids}"
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"Automatic semantic embedding backfill complete: "
|
||||
f"projects={len(projects)}, entities={total_entities}"
|
||||
)
|
||||
|
||||
|
||||
class DatabaseType(Enum):
|
||||
"""Types of supported databases."""
|
||||
@@ -382,9 +480,26 @@ async def run_migrations(
|
||||
Note: Alembic tracks which migrations have been applied via the alembic_version table,
|
||||
so it's safe to call this multiple times - it will only run pending migrations.
|
||||
"""
|
||||
logger.info("Running database migrations...")
|
||||
logger.debug("Running database migrations...")
|
||||
temp_engine: AsyncEngine | None = None
|
||||
try:
|
||||
revisions_before_upgrade: set[str] = set()
|
||||
# Trigger: run_migrations() can be invoked before module-level session maker is set.
|
||||
# Why: we still need reliable before/after revision detection for one-time backfill.
|
||||
# Outcome: create a short-lived session maker when needed, then dispose it immediately.
|
||||
if _session_maker is None:
|
||||
precheck_engine, temp_session_maker = _create_engine_and_session(
|
||||
app_config.database_path,
|
||||
database_type,
|
||||
app_config,
|
||||
)
|
||||
try:
|
||||
revisions_before_upgrade = await _load_applied_alembic_revisions(temp_session_maker)
|
||||
finally:
|
||||
await precheck_engine.dispose()
|
||||
else:
|
||||
revisions_before_upgrade = await _load_applied_alembic_revisions(_session_maker)
|
||||
|
||||
# Get the absolute path to the alembic directory relative to this file
|
||||
alembic_dir = Path(__file__).parent / "alembic"
|
||||
config = Config()
|
||||
@@ -404,7 +519,7 @@ async def run_migrations(
|
||||
config.set_main_option("sqlalchemy.url", db_url)
|
||||
|
||||
command.upgrade(config, "head")
|
||||
logger.info("Migrations completed successfully")
|
||||
logger.debug("Migrations completed successfully")
|
||||
|
||||
# Get session maker - ensure we don't trigger recursive migration calls
|
||||
if _session_maker is None:
|
||||
@@ -426,6 +541,12 @@ async def run_migrations(
|
||||
else:
|
||||
await SQLiteSearchRepository(session_maker, 1).init_search_index()
|
||||
|
||||
revisions_after_upgrade = await _load_applied_alembic_revisions(session_maker)
|
||||
if _should_run_semantic_embedding_backfill(
|
||||
revisions_before_upgrade,
|
||||
revisions_after_upgrade,
|
||||
):
|
||||
await _run_semantic_embedding_backfill(app_config, session_maker)
|
||||
except Exception as e: # pragma: no cover
|
||||
logger.error(f"Error running migrations: {e}")
|
||||
raise
|
||||
|
||||
@@ -131,6 +131,10 @@ from basic_memory.deps.services import (
|
||||
DirectoryServiceV2Dep,
|
||||
get_directory_service_v2_external,
|
||||
DirectoryServiceV2ExternalDep,
|
||||
get_graph_intelligence_service_v2_external,
|
||||
GraphIntelligenceServiceV2ExternalDep,
|
||||
get_fcm_service_v2_external,
|
||||
FCMServiceV2ExternalDep,
|
||||
)
|
||||
|
||||
from basic_memory.deps.importers import (
|
||||
@@ -269,6 +273,10 @@ __all__ = [
|
||||
"DirectoryServiceV2Dep",
|
||||
"get_directory_service_v2_external",
|
||||
"DirectoryServiceV2ExternalDep",
|
||||
"get_graph_intelligence_service_v2_external",
|
||||
"GraphIntelligenceServiceV2ExternalDep",
|
||||
"get_fcm_service_v2_external",
|
||||
"FCMServiceV2ExternalDep",
|
||||
# Importers
|
||||
"get_chatgpt_importer",
|
||||
"ChatGPTImporterDep",
|
||||
|
||||
@@ -39,6 +39,8 @@ from basic_memory.deps.repositories import (
|
||||
from basic_memory.markdown import EntityParser
|
||||
from basic_memory.markdown.markdown_processor import MarkdownProcessor
|
||||
from basic_memory.services import EntityService, ProjectService
|
||||
from basic_memory.services.fcm_service import FCMService
|
||||
from basic_memory.services.graph_intelligence_service import GraphIntelligenceService
|
||||
from basic_memory.services.context_service import ContextService
|
||||
from basic_memory.services.directory_service import DirectoryService
|
||||
from basic_memory.services.file_service import FileService
|
||||
@@ -358,6 +360,30 @@ async def get_context_service_v2_external(
|
||||
ContextServiceV2ExternalDep = Annotated[ContextService, Depends(get_context_service_v2_external)]
|
||||
|
||||
|
||||
# --- Graph Intelligence Service ---
|
||||
|
||||
|
||||
async def get_graph_intelligence_service_v2_external() -> GraphIntelligenceService:
|
||||
"""Create GraphIntelligenceService for v2 API (uses external_id routing)."""
|
||||
return GraphIntelligenceService()
|
||||
|
||||
|
||||
GraphIntelligenceServiceV2ExternalDep = Annotated[
|
||||
GraphIntelligenceService, Depends(get_graph_intelligence_service_v2_external)
|
||||
]
|
||||
|
||||
|
||||
# --- FCM Service ---
|
||||
|
||||
|
||||
async def get_fcm_service_v2_external() -> FCMService:
|
||||
"""Create FCMService for v2 API (uses external_id routing)."""
|
||||
return FCMService()
|
||||
|
||||
|
||||
FCMServiceV2ExternalDep = Annotated[FCMService, Depends(get_fcm_service_v2_external)]
|
||||
|
||||
|
||||
# --- Sync Service ---
|
||||
|
||||
|
||||
@@ -447,12 +473,8 @@ class TaskScheduler(Protocol):
|
||||
|
||||
|
||||
def _log_task_failure(completed: asyncio.Task) -> None:
|
||||
if completed.cancelled():
|
||||
return
|
||||
try:
|
||||
completed.result()
|
||||
except asyncio.CancelledError:
|
||||
return
|
||||
except Exception as exc: # pragma: no cover
|
||||
logger.exception("Background task failed", error=str(exc))
|
||||
|
||||
@@ -496,6 +518,7 @@ class LocalTaskScheduler:
|
||||
|
||||
|
||||
async def get_task_scheduler(
|
||||
entity_service: EntityServiceV2ExternalDep,
|
||||
sync_service: SyncServiceV2ExternalDep,
|
||||
search_service: SearchServiceV2ExternalDep,
|
||||
project_config: ProjectConfigV2ExternalDep,
|
||||
@@ -503,6 +526,28 @@ async def get_task_scheduler(
|
||||
) -> TaskScheduler:
|
||||
"""Create a scheduler that maps task specs to coroutines."""
|
||||
|
||||
scheduler: LocalTaskScheduler | None = None
|
||||
|
||||
async def _reindex_entity(
|
||||
entity_id: int,
|
||||
resolve_relations: bool = False,
|
||||
**_: Any,
|
||||
) -> None:
|
||||
await entity_service.reindex_entity(entity_id)
|
||||
# Trigger: caller requests relation resolution
|
||||
# Why: resolve forward references created before the entity existed
|
||||
# Outcome: updates unresolved relations pointing to this entity
|
||||
if resolve_relations:
|
||||
await sync_service.resolve_relations(entity_id=entity_id)
|
||||
# Trigger: semantic search enabled in local config.
|
||||
# Why: vector chunks are derived and should refresh after canonical reindex completes.
|
||||
# Outcome: schedules out-of-band vector sync without extending write latency.
|
||||
if app_config.semantic_search_enabled and scheduler is not None:
|
||||
scheduler.schedule("sync_entity_vectors", entity_id=entity_id)
|
||||
|
||||
async def _resolve_relations(entity_id: int, **_: Any) -> None:
|
||||
await sync_service.resolve_relations(entity_id=entity_id)
|
||||
|
||||
async def _sync_entity_vectors(entity_id: int, **_: Any) -> None:
|
||||
await search_service.sync_entity_vectors(entity_id)
|
||||
|
||||
@@ -516,11 +561,31 @@ async def get_task_scheduler(
|
||||
async def _reindex_project(**_: Any) -> None:
|
||||
await search_service.reindex_all()
|
||||
|
||||
async def _sync_graph_entity(entity_id: int, **extra_payload: Any) -> None:
|
||||
# Trigger: graph-entity sync task is scheduled from graph lifecycle hooks.
|
||||
# Why: keep scheduler contract stable while graph index provider work lands in later phases.
|
||||
# Outcome: no-op in phase 1; task name remains valid for API and tool contracts.
|
||||
del entity_id, extra_payload
|
||||
|
||||
async def _sync_graph_project(force_full: bool = False, **_: Any) -> None:
|
||||
await _sync_project(force_full=force_full)
|
||||
|
||||
async def _reindex_graph_project(**_: Any) -> None:
|
||||
# Trigger: graph reindex requested.
|
||||
# Why: phase 1 has no dedicated graph index worker yet.
|
||||
# Outcome: run project sync path so writes stay coherent while graph provider ships.
|
||||
await _sync_project(force_full=True)
|
||||
|
||||
scheduler = LocalTaskScheduler(
|
||||
{
|
||||
"reindex_entity": _reindex_entity,
|
||||
"resolve_relations": _resolve_relations,
|
||||
"sync_entity_vectors": _sync_entity_vectors,
|
||||
"sync_project": _sync_project,
|
||||
"reindex_project": _reindex_project,
|
||||
"sync_graph_entity": _sync_graph_entity,
|
||||
"sync_graph_project": _sync_graph_project,
|
||||
"reindex_graph_project": _reindex_graph_project,
|
||||
},
|
||||
test_mode=app_config.is_test_env,
|
||||
)
|
||||
|
||||
@@ -114,13 +114,7 @@ async def write_file_atomic(path: FilePath, content: str) -> None:
|
||||
temp_path = path_obj.with_suffix(".tmp")
|
||||
|
||||
try:
|
||||
# Trigger: callers hand us normalized Python text, but the final bytes are allowed
|
||||
# to use the host platform's native newline convention during the write.
|
||||
# Why: preserving CRLF on Windows keeps local files aligned with editors like
|
||||
# Obsidian, while FileService now hashes the persisted file bytes instead of
|
||||
# the pre-write string.
|
||||
# Outcome: this async write stays editor-friendly across platforms without
|
||||
# reintroducing checksum drift in sync or move detection.
|
||||
# Use aiofiles for non-blocking write
|
||||
async with aiofiles.open(temp_path, mode="w", encoding="utf-8") as f:
|
||||
await f.write(content)
|
||||
|
||||
@@ -174,13 +168,6 @@ async def format_markdown_builtin(path: Path) -> Optional[str]:
|
||||
|
||||
# Only write if content changed
|
||||
if formatted_content != content:
|
||||
# Trigger: mdformat may rewrite markdown content, then the host platform
|
||||
# decides the newline bytes for the follow-up async text write.
|
||||
# Why: we want formatter output to preserve native newlines instead of
|
||||
# forcing LF, and the authoritative checksum comes from rereading the
|
||||
# stored file bytes later in FileService.
|
||||
# Outcome: formatting remains compatible with local editors on Windows while
|
||||
# checksum-based sync logic stays anchored to on-disk bytes.
|
||||
async with aiofiles.open(path, mode="w", encoding="utf-8") as f:
|
||||
await f.write(formatted_content)
|
||||
|
||||
@@ -460,11 +447,6 @@ def sanitize_for_filename(text: str, replacement: str = "-") -> str:
|
||||
# compress multiple, repeated replacements
|
||||
text = re.sub(f"{re.escape(replacement)}+", replacement, text)
|
||||
|
||||
# Strip trailing periods — they cause "hi-everyone..md" double-dot filenames
|
||||
# when ".md" is appended, which triggers path traversal false positives.
|
||||
# Trailing periods are also invalid on Windows filesystems.
|
||||
text = text.strip(".")
|
||||
|
||||
return text.strip(replacement)
|
||||
|
||||
|
||||
|
||||
@@ -4,8 +4,6 @@ import fnmatch
|
||||
from pathlib import Path
|
||||
from typing import Set
|
||||
|
||||
from basic_memory.config import resolve_data_dir
|
||||
|
||||
|
||||
# Common directories and patterns to ignore by default
|
||||
# These are used as fallback if .bmignore doesn't exist
|
||||
@@ -63,11 +61,9 @@ def get_bmignore_path() -> Path:
|
||||
"""Get path to .bmignore file.
|
||||
|
||||
Returns:
|
||||
Path to <basic-memory data dir>/.bmignore, honoring
|
||||
``BASIC_MEMORY_CONFIG_DIR`` so isolated instances each keep their
|
||||
own ignore file.
|
||||
Path to ~/.basic-memory/.bmignore
|
||||
"""
|
||||
return resolve_data_dir() / ".bmignore"
|
||||
return Path.home() / ".basic-memory" / ".bmignore"
|
||||
|
||||
|
||||
def create_default_bmignore() -> None:
|
||||
@@ -180,8 +176,7 @@ def load_gitignore_patterns(base_path: Path, use_gitignore: bool = True) -> Set[
|
||||
"""Load gitignore patterns from .gitignore file and .bmignore.
|
||||
|
||||
Combines patterns from:
|
||||
1. <basic-memory data dir>/.bmignore (user's global ignore patterns, honors
|
||||
BASIC_MEMORY_CONFIG_DIR)
|
||||
1. ~/.basic-memory/.bmignore (user's global ignore patterns)
|
||||
2. {base_path}/.gitignore (project-specific patterns, if use_gitignore=True)
|
||||
|
||||
Args:
|
||||
|
||||
@@ -39,24 +39,23 @@ def format_timestamp(timestamp: Any) -> str: # pragma: no cover
|
||||
Returns:
|
||||
A formatted string representation of the timestamp.
|
||||
"""
|
||||
parsed_timestamp = timestamp
|
||||
if isinstance(timestamp, str):
|
||||
try:
|
||||
# Try ISO format
|
||||
parsed_timestamp = datetime.fromisoformat(timestamp.replace("Z", "+00:00"))
|
||||
timestamp = datetime.fromisoformat(timestamp.replace("Z", "+00:00"))
|
||||
except ValueError:
|
||||
try:
|
||||
# Try unix timestamp as string
|
||||
parsed_timestamp = datetime.fromtimestamp(float(timestamp)).astimezone()
|
||||
timestamp = datetime.fromtimestamp(float(timestamp)).astimezone()
|
||||
except ValueError:
|
||||
# Return as is if we can't parse it
|
||||
return timestamp
|
||||
elif isinstance(timestamp, (int, float)):
|
||||
# Unix timestamp
|
||||
parsed_timestamp = datetime.fromtimestamp(timestamp).astimezone()
|
||||
timestamp = datetime.fromtimestamp(timestamp).astimezone()
|
||||
|
||||
if isinstance(parsed_timestamp, datetime):
|
||||
return parsed_timestamp.strftime("%Y-%m-%d %H:%M:%S")
|
||||
if isinstance(timestamp, datetime):
|
||||
return timestamp.strftime("%Y-%m-%d %H:%M:%S")
|
||||
|
||||
# Return as is if we can't format it
|
||||
return str(parsed_timestamp) # pragma: no cover
|
||||
return str(timestamp) # pragma: no cover
|
||||
|
||||
@@ -1,31 +0,0 @@
|
||||
"""Reusable indexing primitives shared by local sync and future remote callers."""
|
||||
|
||||
from basic_memory.indexing.batch_indexer import BatchIndexer
|
||||
from basic_memory.indexing.batching import build_index_batches
|
||||
from basic_memory.indexing.models import (
|
||||
IndexedEntity,
|
||||
IndexBatch,
|
||||
IndexFileMetadata,
|
||||
IndexFileWriter,
|
||||
IndexFrontmatterUpdate,
|
||||
IndexFrontmatterWriteResult,
|
||||
IndexingBatchResult,
|
||||
IndexInputFile,
|
||||
IndexProgress,
|
||||
SyncedMarkdownFile,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"BatchIndexer",
|
||||
"IndexedEntity",
|
||||
"IndexBatch",
|
||||
"IndexFileMetadata",
|
||||
"IndexFileWriter",
|
||||
"IndexFrontmatterUpdate",
|
||||
"IndexFrontmatterWriteResult",
|
||||
"IndexingBatchResult",
|
||||
"IndexInputFile",
|
||||
"IndexProgress",
|
||||
"SyncedMarkdownFile",
|
||||
"build_index_batches",
|
||||
]
|
||||
@@ -1,710 +0,0 @@
|
||||
"""Reusable batch executor for bounded-parallel file indexing."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Awaitable, Callable, Mapping, TypeVar
|
||||
|
||||
from loguru import logger
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
|
||||
import logfire
|
||||
from basic_memory.config import BasicMemoryConfig
|
||||
from basic_memory.file_utils import compute_checksum, has_frontmatter, remove_frontmatter
|
||||
from basic_memory.markdown.schemas import EntityMarkdown
|
||||
from basic_memory.indexing.models import (
|
||||
IndexedEntity,
|
||||
IndexFileWriter,
|
||||
IndexFrontmatterUpdate,
|
||||
IndexingBatchResult,
|
||||
IndexInputFile,
|
||||
)
|
||||
from basic_memory.models import Entity, Relation
|
||||
from basic_memory.services import EntityService
|
||||
from basic_memory.services.exceptions import SyncFatalError
|
||||
from basic_memory.services.search_service import SearchService
|
||||
from basic_memory.repository import EntityRepository, RelationRepository
|
||||
|
||||
T = TypeVar("T")
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class _PreparedMarkdownFile:
|
||||
file: IndexInputFile
|
||||
content: str
|
||||
final_checksum: str
|
||||
markdown: EntityMarkdown
|
||||
file_contains_frontmatter: bool
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class _PreparedEntity:
|
||||
path: str
|
||||
entity_id: int
|
||||
permalink: str | None
|
||||
checksum: str
|
||||
content_type: str | None
|
||||
search_content: str | None
|
||||
markdown_content: str | None = None
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class _PersistedMarkdownFile:
|
||||
prepared: _PreparedMarkdownFile
|
||||
entity: Entity
|
||||
|
||||
|
||||
class BatchIndexer:
|
||||
"""Index already-loaded files without assuming where they came from."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
app_config: BasicMemoryConfig,
|
||||
entity_service: EntityService,
|
||||
entity_repository: EntityRepository,
|
||||
relation_repository: RelationRepository,
|
||||
search_service: SearchService,
|
||||
file_writer: IndexFileWriter,
|
||||
) -> None:
|
||||
self.app_config = app_config
|
||||
self.entity_service = entity_service
|
||||
self.entity_repository = entity_repository
|
||||
self.relation_repository = relation_repository
|
||||
self.search_service = search_service
|
||||
self.file_writer = file_writer
|
||||
|
||||
async def index_files(
|
||||
self,
|
||||
files: Mapping[str, IndexInputFile],
|
||||
*,
|
||||
max_concurrent: int,
|
||||
parse_max_concurrent: int | None = None,
|
||||
existing_permalink_by_path: dict[str, str | None] | None = None,
|
||||
) -> IndexingBatchResult:
|
||||
"""Index one batch of loaded files with bounded concurrency."""
|
||||
if max_concurrent <= 0:
|
||||
raise ValueError("max_concurrent must be greater than zero")
|
||||
|
||||
ordered_paths = sorted(files)
|
||||
if not ordered_paths:
|
||||
return IndexingBatchResult()
|
||||
|
||||
parse_limit = parse_max_concurrent or max_concurrent
|
||||
error_by_path: dict[str, str] = {}
|
||||
|
||||
markdown_paths = [path for path in ordered_paths if self._is_markdown(files[path])]
|
||||
regular_paths = [path for path in ordered_paths if path not in markdown_paths]
|
||||
|
||||
prepared_markdown, parse_errors = await self._run_bounded(
|
||||
markdown_paths,
|
||||
limit=parse_limit,
|
||||
worker=lambda path: self._prepare_markdown_file(files[path]),
|
||||
)
|
||||
error_by_path.update(parse_errors)
|
||||
|
||||
prepared_markdown, normalization_errors = await self._normalize_markdown_batch(
|
||||
prepared_markdown,
|
||||
existing_permalink_by_path=existing_permalink_by_path,
|
||||
)
|
||||
error_by_path.update(normalization_errors)
|
||||
|
||||
indexed_entities: list[IndexedEntity] = []
|
||||
resolved_count = 0
|
||||
unresolved_count = 0
|
||||
search_indexed = 0
|
||||
|
||||
prepared_entities: dict[str, _PreparedEntity] = {}
|
||||
|
||||
markdown_upserts, markdown_errors = await self._run_bounded(
|
||||
[path for path in markdown_paths if path not in error_by_path],
|
||||
limit=max_concurrent,
|
||||
worker=lambda path: self._upsert_markdown_file(prepared_markdown[path]),
|
||||
)
|
||||
error_by_path.update(markdown_errors)
|
||||
prepared_entities.update(markdown_upserts)
|
||||
if existing_permalink_by_path is not None:
|
||||
for path, prepared_entity in markdown_upserts.items():
|
||||
existing_permalink_by_path[path] = prepared_entity.permalink
|
||||
|
||||
regular_upserts, regular_errors = await self._run_bounded(
|
||||
regular_paths,
|
||||
limit=max_concurrent,
|
||||
worker=lambda path: self._upsert_regular_file(files[path]),
|
||||
)
|
||||
error_by_path.update(regular_errors)
|
||||
prepared_entities.update(regular_upserts)
|
||||
|
||||
markdown_entity_ids = [
|
||||
prepared_entities[path].entity_id
|
||||
for path in markdown_paths
|
||||
if path in prepared_entities
|
||||
]
|
||||
if markdown_entity_ids:
|
||||
resolved_count, unresolved_count = await self._resolve_batch_relations(
|
||||
markdown_entity_ids,
|
||||
max_concurrent=max_concurrent,
|
||||
)
|
||||
|
||||
refreshed_entities = await self.entity_repository.find_by_ids(
|
||||
[prepared.entity_id for prepared in prepared_entities.values()]
|
||||
)
|
||||
entities_by_id = {entity.id: entity for entity in refreshed_entities}
|
||||
|
||||
refreshed, refresh_errors = await self._run_bounded(
|
||||
[path for path in ordered_paths if path in prepared_entities],
|
||||
limit=self.app_config.index_metadata_update_max_concurrent,
|
||||
worker=lambda path: self._refresh_search_index(
|
||||
prepared_entities[path],
|
||||
entities_by_id[prepared_entities[path].entity_id],
|
||||
),
|
||||
)
|
||||
error_by_path.update(refresh_errors)
|
||||
|
||||
for path in ordered_paths:
|
||||
indexed = refreshed.get(path)
|
||||
if indexed is not None:
|
||||
indexed_entities.append(indexed)
|
||||
|
||||
search_indexed = len(indexed_entities)
|
||||
|
||||
return IndexingBatchResult(
|
||||
indexed=indexed_entities,
|
||||
errors=[(path, error_by_path[path]) for path in ordered_paths if path in error_by_path],
|
||||
relations_resolved=resolved_count,
|
||||
relations_unresolved=unresolved_count,
|
||||
search_indexed=search_indexed,
|
||||
)
|
||||
|
||||
async def index_markdown_file(
|
||||
self,
|
||||
file: IndexInputFile,
|
||||
*,
|
||||
new: bool | None = None,
|
||||
existing_permalink_by_path: dict[str, str | None] | None = None,
|
||||
index_search: bool = True,
|
||||
resolve_relations: bool = True,
|
||||
) -> IndexedEntity:
|
||||
"""Index one markdown file using the same normalization and upsert path as batches."""
|
||||
if not self._is_markdown(file):
|
||||
raise ValueError(f"index_markdown_file requires markdown input: {file.path}")
|
||||
|
||||
with logfire.span("index.markdown_file.prepare", path=file.path):
|
||||
prepared = await self._prepare_markdown_file(file)
|
||||
if existing_permalink_by_path is None:
|
||||
with logfire.span("index.markdown_file.load_permalink_map", path=file.path):
|
||||
existing_permalink_by_path = {
|
||||
path: permalink
|
||||
for path, permalink in (
|
||||
await self.entity_repository.get_file_path_to_permalink_map()
|
||||
).items()
|
||||
}
|
||||
|
||||
reserved_permalinks = {
|
||||
permalink
|
||||
for path, permalink in existing_permalink_by_path.items()
|
||||
if path != file.path and permalink
|
||||
}
|
||||
with logfire.span("index.markdown_file.normalize", path=file.path):
|
||||
prepared = await self._normalize_markdown_file(prepared, reserved_permalinks)
|
||||
existing_permalink_by_path[file.path] = prepared.markdown.frontmatter.permalink
|
||||
|
||||
with logfire.span("index.markdown_file.persist", path=file.path, is_new=new):
|
||||
persisted = await self._persist_markdown_file(
|
||||
prepared,
|
||||
is_new=new,
|
||||
resolve_relations=resolve_relations,
|
||||
reload_entity=False,
|
||||
)
|
||||
existing_permalink_by_path[file.path] = persisted.entity.permalink
|
||||
|
||||
with logfire.span(
|
||||
"index.markdown_file.reload_entity",
|
||||
path=file.path,
|
||||
entity_id=persisted.entity.id,
|
||||
):
|
||||
refreshed = await self.entity_repository.find_by_ids([persisted.entity.id])
|
||||
if len(refreshed) != 1: # pragma: no cover
|
||||
raise ValueError(f"Failed to reload indexed entity for {file.path}")
|
||||
entity = refreshed[0]
|
||||
prepared_entity = self._build_prepared_entity(persisted.prepared, entity)
|
||||
|
||||
if index_search:
|
||||
with logfire.span(
|
||||
"index.markdown_file.refresh_search_index",
|
||||
path=file.path,
|
||||
entity_id=entity.id,
|
||||
):
|
||||
return await self._refresh_search_index(prepared_entity, entity)
|
||||
|
||||
return IndexedEntity(
|
||||
path=prepared_entity.path,
|
||||
entity_id=entity.id,
|
||||
permalink=entity.permalink,
|
||||
checksum=prepared_entity.checksum,
|
||||
content_type=prepared_entity.content_type,
|
||||
markdown_content=prepared_entity.markdown_content,
|
||||
)
|
||||
|
||||
# --- Preparation ---
|
||||
|
||||
async def _prepare_markdown_file(self, file: IndexInputFile) -> _PreparedMarkdownFile:
|
||||
if file.content is None:
|
||||
raise ValueError(f"Missing content for markdown file: {file.path}")
|
||||
|
||||
content = file.content.decode("utf-8")
|
||||
file_contains_frontmatter = has_frontmatter(content)
|
||||
final_checksum = await self._resolve_checksum(file)
|
||||
entity_markdown = await self.entity_service.entity_parser.parse_markdown_content(
|
||||
file_path=Path(file.path),
|
||||
content=content,
|
||||
mtime=file.last_modified.timestamp() if file.last_modified else None,
|
||||
ctime=file.created_at.timestamp() if file.created_at else None,
|
||||
)
|
||||
|
||||
return _PreparedMarkdownFile(
|
||||
file=file,
|
||||
content=content,
|
||||
final_checksum=final_checksum,
|
||||
markdown=entity_markdown,
|
||||
file_contains_frontmatter=file_contains_frontmatter,
|
||||
)
|
||||
|
||||
async def _normalize_markdown_batch(
|
||||
self,
|
||||
prepared_markdown: dict[str, _PreparedMarkdownFile],
|
||||
*,
|
||||
existing_permalink_by_path: dict[str, str | None] | None = None,
|
||||
) -> tuple[dict[str, _PreparedMarkdownFile], dict[str, str]]:
|
||||
if not prepared_markdown:
|
||||
return {}, {}
|
||||
|
||||
if existing_permalink_by_path is None:
|
||||
existing_permalink_by_path = {
|
||||
path: permalink
|
||||
for path, permalink in (
|
||||
await self.entity_repository.get_file_path_to_permalink_map()
|
||||
).items()
|
||||
}
|
||||
|
||||
batch_paths = set(prepared_markdown)
|
||||
reserved_permalinks = {
|
||||
permalink
|
||||
for path, permalink in existing_permalink_by_path.items()
|
||||
if path not in batch_paths and permalink
|
||||
}
|
||||
|
||||
normalized: dict[str, _PreparedMarkdownFile] = {}
|
||||
errors: dict[str, str] = {}
|
||||
|
||||
for path in sorted(prepared_markdown):
|
||||
try:
|
||||
normalized[path] = await self._normalize_markdown_file(
|
||||
prepared_markdown[path],
|
||||
reserved_permalinks,
|
||||
)
|
||||
existing_permalink_by_path[path] = normalized[path].markdown.frontmatter.permalink
|
||||
except Exception as exc:
|
||||
errors[path] = str(exc)
|
||||
logger.warning("Batch markdown normalization failed", path=path, error=str(exc))
|
||||
|
||||
return normalized, errors
|
||||
|
||||
async def _normalize_markdown_file(
|
||||
self,
|
||||
prepared: _PreparedMarkdownFile,
|
||||
reserved_permalinks: set[str],
|
||||
) -> _PreparedMarkdownFile:
|
||||
final_checksum = prepared.final_checksum
|
||||
final_content = prepared.content
|
||||
final_permalink = await self._resolve_batch_permalink(prepared, reserved_permalinks)
|
||||
|
||||
# Trigger: markdown file has no frontmatter and sync enforcement is enabled.
|
||||
# Why: downstream indexing relies on normalized metadata and stable permalinks.
|
||||
# Outcome: write derived metadata back through the storage-agnostic writer.
|
||||
if not prepared.file_contains_frontmatter and self.app_config.ensure_frontmatter_on_sync:
|
||||
frontmatter_updates = {
|
||||
"title": prepared.markdown.frontmatter.title,
|
||||
"type": prepared.markdown.frontmatter.type,
|
||||
"permalink": final_permalink,
|
||||
}
|
||||
write_result = await self.file_writer.write_frontmatter(
|
||||
IndexFrontmatterUpdate(path=prepared.file.path, metadata=frontmatter_updates)
|
||||
)
|
||||
final_checksum = write_result.checksum
|
||||
final_content = write_result.content
|
||||
prepared.markdown.frontmatter.metadata.update(frontmatter_updates)
|
||||
|
||||
# Trigger: existing markdown frontmatter may lack the canonical permalink.
|
||||
# Why: batch sync keeps permalinks stable without forcing a full rewrite when unchanged.
|
||||
# Outcome: only the permalink field is updated when it actually differs.
|
||||
elif (
|
||||
prepared.file_contains_frontmatter
|
||||
and not self.app_config.disable_permalinks
|
||||
and final_permalink != prepared.markdown.frontmatter.permalink
|
||||
):
|
||||
prepared.markdown.frontmatter.metadata["permalink"] = final_permalink
|
||||
write_result = await self.file_writer.write_frontmatter(
|
||||
IndexFrontmatterUpdate(
|
||||
path=prepared.file.path,
|
||||
metadata={"permalink": final_permalink},
|
||||
)
|
||||
)
|
||||
final_checksum = write_result.checksum
|
||||
final_content = write_result.content
|
||||
|
||||
return _PreparedMarkdownFile(
|
||||
file=prepared.file,
|
||||
content=final_content,
|
||||
final_checksum=final_checksum,
|
||||
markdown=prepared.markdown,
|
||||
file_contains_frontmatter=prepared.file_contains_frontmatter,
|
||||
)
|
||||
|
||||
async def _resolve_batch_permalink(
|
||||
self,
|
||||
prepared: _PreparedMarkdownFile,
|
||||
reserved_permalinks: set[str],
|
||||
) -> str | None:
|
||||
should_resolve_permalink = (
|
||||
not prepared.file_contains_frontmatter and self.app_config.ensure_frontmatter_on_sync
|
||||
) or (prepared.file_contains_frontmatter and not self.app_config.disable_permalinks)
|
||||
if not should_resolve_permalink:
|
||||
permalink = prepared.markdown.frontmatter.permalink
|
||||
if permalink:
|
||||
reserved_permalinks.add(permalink)
|
||||
return permalink
|
||||
|
||||
desired_permalink = await self.entity_service.resolve_permalink(
|
||||
prepared.file.path,
|
||||
markdown=prepared.markdown,
|
||||
skip_conflict_check=True,
|
||||
)
|
||||
return self._reserve_batch_permalink(desired_permalink, reserved_permalinks)
|
||||
|
||||
def _reserve_batch_permalink(
|
||||
self,
|
||||
desired_permalink: str,
|
||||
reserved_permalinks: set[str],
|
||||
) -> str:
|
||||
permalink = desired_permalink
|
||||
suffix = 1
|
||||
while permalink in reserved_permalinks:
|
||||
permalink = f"{desired_permalink}-{suffix}"
|
||||
suffix += 1
|
||||
reserved_permalinks.add(permalink)
|
||||
return permalink
|
||||
|
||||
# --- Persistence ---
|
||||
|
||||
async def _upsert_markdown_file(self, prepared: _PreparedMarkdownFile) -> _PreparedEntity:
|
||||
persisted = await self._persist_markdown_file(prepared)
|
||||
return self._build_prepared_entity(persisted.prepared, persisted.entity)
|
||||
|
||||
async def _upsert_regular_file(self, file: IndexInputFile) -> _PreparedEntity:
|
||||
checksum = await self._resolve_checksum(file)
|
||||
existing = await self.entity_repository.get_by_file_path(file.path, load_relations=False)
|
||||
is_new_entity = existing is None
|
||||
|
||||
if existing is None:
|
||||
await self.entity_service.resolve_permalink(file.path, skip_conflict_check=True)
|
||||
entity = Entity(
|
||||
note_type="file",
|
||||
file_path=file.path,
|
||||
checksum=checksum,
|
||||
title=Path(file.path).name,
|
||||
created_at=file.created_at or datetime.now().astimezone(),
|
||||
updated_at=file.last_modified or datetime.now().astimezone(),
|
||||
content_type=file.content_type or "text/plain",
|
||||
mtime=file.last_modified.timestamp() if file.last_modified else None,
|
||||
size=file.size,
|
||||
)
|
||||
|
||||
try:
|
||||
created = await self.entity_repository.add(entity)
|
||||
entity_id = created.id
|
||||
except IntegrityError as exc:
|
||||
message = str(exc)
|
||||
if (
|
||||
"UNIQUE constraint failed: entity.file_path" in message
|
||||
or "uix_entity_file_path_project" in message
|
||||
or (
|
||||
"duplicate key value violates unique constraint" in message
|
||||
and "file_path" in message
|
||||
)
|
||||
):
|
||||
existing = await self.entity_repository.get_by_file_path(
|
||||
file.path,
|
||||
load_relations=False,
|
||||
)
|
||||
if existing is None:
|
||||
raise ValueError(
|
||||
f"Entity not found after file_path conflict: {file.path}"
|
||||
) from exc
|
||||
entity_id = existing.id
|
||||
else:
|
||||
raise
|
||||
else:
|
||||
entity_id = existing.id
|
||||
|
||||
updated = await self.entity_repository.update(
|
||||
entity_id,
|
||||
self._entity_metadata_updates(file, checksum, include_created_at=is_new_entity),
|
||||
)
|
||||
if updated is None:
|
||||
raise ValueError(f"Failed to update file entity metadata for {file.path}")
|
||||
|
||||
return _PreparedEntity(
|
||||
path=file.path,
|
||||
entity_id=updated.id,
|
||||
permalink=updated.permalink,
|
||||
checksum=checksum,
|
||||
content_type=file.content_type,
|
||||
search_content=None,
|
||||
markdown_content=None,
|
||||
)
|
||||
|
||||
# --- Relations ---
|
||||
|
||||
async def _resolve_batch_relations(
|
||||
self,
|
||||
entity_ids: list[int],
|
||||
*,
|
||||
max_concurrent: int,
|
||||
) -> tuple[int, int]:
|
||||
unresolved_relation_lists = await asyncio.gather(
|
||||
*(
|
||||
self.relation_repository.find_unresolved_relations_for_entity(entity_id)
|
||||
for entity_id in entity_ids
|
||||
)
|
||||
)
|
||||
unresolved_relations = [
|
||||
relation for relation_list in unresolved_relation_lists for relation in relation_list
|
||||
]
|
||||
|
||||
if not unresolved_relations:
|
||||
return 0, 0
|
||||
|
||||
semaphore = asyncio.Semaphore(max_concurrent)
|
||||
|
||||
async def resolve_relation(relation: Relation) -> int:
|
||||
async with semaphore:
|
||||
try:
|
||||
resolved_entity = await self.entity_service.link_resolver.resolve_link(
|
||||
relation.to_name
|
||||
)
|
||||
if resolved_entity is None or resolved_entity.id == relation.from_id:
|
||||
return 0
|
||||
|
||||
try:
|
||||
await self.relation_repository.update(
|
||||
relation.id,
|
||||
{
|
||||
"to_id": resolved_entity.id,
|
||||
"to_name": resolved_entity.title,
|
||||
},
|
||||
)
|
||||
except IntegrityError:
|
||||
await self.relation_repository.delete(relation.id)
|
||||
return 1
|
||||
except Exception as exc: # pragma: no cover - defensive logging
|
||||
logger.warning(
|
||||
"Batch relation resolution failed",
|
||||
relation_id=relation.id,
|
||||
from_id=relation.from_id,
|
||||
to_name=relation.to_name,
|
||||
error=str(exc),
|
||||
)
|
||||
return 0
|
||||
|
||||
resolved_counts = await asyncio.gather(
|
||||
*(resolve_relation(relation) for relation in unresolved_relations)
|
||||
)
|
||||
|
||||
remaining_relation_lists = await asyncio.gather(
|
||||
*(
|
||||
self.relation_repository.find_unresolved_relations_for_entity(entity_id)
|
||||
for entity_id in entity_ids
|
||||
)
|
||||
)
|
||||
remaining_unresolved = sum(len(relations) for relations in remaining_relation_lists)
|
||||
|
||||
return sum(resolved_counts), remaining_unresolved
|
||||
|
||||
# --- Search refresh ---
|
||||
|
||||
async def _refresh_search_index(
|
||||
self, prepared: _PreparedEntity, entity: Entity
|
||||
) -> IndexedEntity:
|
||||
await self.search_service.index_entity_data(entity, content=prepared.search_content)
|
||||
return IndexedEntity(
|
||||
path=prepared.path,
|
||||
entity_id=entity.id,
|
||||
permalink=entity.permalink,
|
||||
checksum=prepared.checksum,
|
||||
content_type=prepared.content_type,
|
||||
markdown_content=prepared.markdown_content,
|
||||
)
|
||||
|
||||
# --- Helpers ---
|
||||
|
||||
async def _persist_markdown_file(
|
||||
self,
|
||||
prepared: _PreparedMarkdownFile,
|
||||
*,
|
||||
is_new: bool | None = None,
|
||||
resolve_relations: bool = True,
|
||||
reload_entity: bool = True,
|
||||
) -> _PersistedMarkdownFile:
|
||||
existing = await self.entity_repository.get_by_file_path(
|
||||
prepared.file.path,
|
||||
load_relations=False,
|
||||
)
|
||||
if is_new is None:
|
||||
is_new = existing is None
|
||||
entity = await self.entity_service.upsert_entity_from_markdown(
|
||||
Path(prepared.file.path),
|
||||
prepared.markdown,
|
||||
is_new=is_new,
|
||||
existing_entity=existing,
|
||||
resolve_relations=resolve_relations,
|
||||
reload_entity=reload_entity,
|
||||
)
|
||||
prepared = await self._reconcile_persisted_permalink(prepared, entity)
|
||||
metadata_updates = self._entity_metadata_updates(prepared.file, prepared.final_checksum)
|
||||
updated = await self.entity_repository.update_fields(
|
||||
entity.id,
|
||||
metadata_updates,
|
||||
)
|
||||
if not updated:
|
||||
raise ValueError(f"Failed to update markdown entity metadata for {prepared.file.path}")
|
||||
self._apply_entity_metadata_updates(entity, metadata_updates)
|
||||
return _PersistedMarkdownFile(prepared=prepared, entity=entity)
|
||||
|
||||
async def _reconcile_persisted_permalink(
|
||||
self,
|
||||
prepared: _PreparedMarkdownFile,
|
||||
entity: Entity,
|
||||
) -> _PreparedMarkdownFile:
|
||||
# Trigger: the source file started without frontmatter and sync is configured
|
||||
# to leave frontmatterless files alone.
|
||||
# Why: upsert may still assign a DB permalink even when disk content should stay untouched.
|
||||
# Outcome: skip reconciliation writes that would silently inject frontmatter.
|
||||
if (
|
||||
self.app_config.disable_permalinks
|
||||
or (
|
||||
not prepared.file_contains_frontmatter
|
||||
and not self.app_config.ensure_frontmatter_on_sync
|
||||
)
|
||||
or entity.permalink is None
|
||||
or entity.permalink == prepared.markdown.frontmatter.permalink
|
||||
):
|
||||
return prepared
|
||||
|
||||
logger.debug(
|
||||
"Updating permalink after upsert conflict resolution",
|
||||
path=prepared.file.path,
|
||||
old_permalink=prepared.markdown.frontmatter.permalink,
|
||||
new_permalink=entity.permalink,
|
||||
)
|
||||
prepared.markdown.frontmatter.metadata["permalink"] = entity.permalink
|
||||
write_result = await self.file_writer.write_frontmatter(
|
||||
IndexFrontmatterUpdate(
|
||||
path=prepared.file.path,
|
||||
metadata={"permalink": entity.permalink},
|
||||
)
|
||||
)
|
||||
return _PreparedMarkdownFile(
|
||||
file=prepared.file,
|
||||
content=write_result.content,
|
||||
final_checksum=write_result.checksum,
|
||||
markdown=prepared.markdown,
|
||||
file_contains_frontmatter=prepared.file_contains_frontmatter,
|
||||
)
|
||||
|
||||
def _build_prepared_entity(
|
||||
self,
|
||||
prepared: _PreparedMarkdownFile,
|
||||
entity: Entity,
|
||||
) -> _PreparedEntity:
|
||||
return _PreparedEntity(
|
||||
path=prepared.file.path,
|
||||
entity_id=entity.id,
|
||||
permalink=entity.permalink,
|
||||
checksum=prepared.final_checksum,
|
||||
content_type=prepared.file.content_type,
|
||||
search_content=(
|
||||
prepared.markdown.content
|
||||
if prepared.markdown.content is not None
|
||||
else remove_frontmatter(prepared.content)
|
||||
),
|
||||
markdown_content=prepared.content,
|
||||
)
|
||||
|
||||
async def _resolve_checksum(self, file: IndexInputFile) -> str:
|
||||
if file.checksum is not None:
|
||||
return file.checksum
|
||||
if file.content is None:
|
||||
raise ValueError(f"Missing checksum and content for file: {file.path}")
|
||||
return await compute_checksum(file.content)
|
||||
|
||||
def _entity_metadata_updates(
|
||||
self,
|
||||
file: IndexInputFile,
|
||||
checksum: str,
|
||||
*,
|
||||
include_created_at: bool = True,
|
||||
) -> dict[str, object]:
|
||||
updates: dict[str, object] = {
|
||||
"file_path": file.path,
|
||||
"checksum": checksum,
|
||||
"size": file.size,
|
||||
}
|
||||
if include_created_at and file.created_at is not None:
|
||||
updates["created_at"] = file.created_at
|
||||
if file.last_modified is not None:
|
||||
updates["updated_at"] = file.last_modified
|
||||
updates["mtime"] = file.last_modified.timestamp()
|
||||
if file.content_type is not None:
|
||||
updates["content_type"] = file.content_type
|
||||
return updates
|
||||
|
||||
def _apply_entity_metadata_updates(self, entity: Entity, updates: dict[str, object]) -> None:
|
||||
"""Keep the returned entity aligned with metadata written without reload."""
|
||||
for key, value in updates.items():
|
||||
setattr(entity, key, value)
|
||||
|
||||
def _is_markdown(self, file: IndexInputFile) -> bool:
|
||||
if file.content_type is not None:
|
||||
return file.content_type == "text/markdown"
|
||||
return Path(file.path).suffix.lower() in {".md", ".markdown"}
|
||||
|
||||
async def _run_bounded(
|
||||
self,
|
||||
paths: list[str],
|
||||
*,
|
||||
limit: int,
|
||||
worker: Callable[[str], Awaitable[T]],
|
||||
) -> tuple[dict[str, T], dict[str, str]]:
|
||||
if not paths:
|
||||
return {}, {}
|
||||
|
||||
semaphore = asyncio.Semaphore(limit)
|
||||
results: dict[str, T] = {}
|
||||
errors: dict[str, str] = {}
|
||||
|
||||
async def run(path: str) -> None:
|
||||
async with semaphore:
|
||||
try:
|
||||
results[path] = await worker(path)
|
||||
except Exception as exc:
|
||||
if isinstance(exc, SyncFatalError) or isinstance(exc.__cause__, SyncFatalError):
|
||||
raise
|
||||
errors[path] = str(exc)
|
||||
logger.warning("Batch indexing failed", path=path, error=str(exc))
|
||||
|
||||
await asyncio.gather(*(run(path) for path in paths))
|
||||
return results, errors
|
||||
@@ -1,63 +0,0 @@
|
||||
"""Deterministic helpers for planning bounded indexing batches."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping, Sequence
|
||||
|
||||
from basic_memory.indexing.models import IndexBatch, IndexFileMetadata
|
||||
|
||||
|
||||
def build_index_batches(
|
||||
paths: Sequence[str],
|
||||
metadata_by_path: Mapping[str, IndexFileMetadata],
|
||||
*,
|
||||
max_files: int,
|
||||
max_bytes: int,
|
||||
) -> list[IndexBatch]:
|
||||
"""Build deterministic batches bounded by file count and total bytes."""
|
||||
if max_files <= 0:
|
||||
raise ValueError("max_files must be greater than zero")
|
||||
if max_bytes <= 0:
|
||||
raise ValueError("max_bytes must be greater than zero")
|
||||
|
||||
ordered_paths = sorted(paths)
|
||||
batches: list[IndexBatch] = []
|
||||
current_paths: list[str] = []
|
||||
current_bytes = 0
|
||||
|
||||
for path in ordered_paths:
|
||||
metadata = metadata_by_path.get(path)
|
||||
if metadata is None:
|
||||
raise KeyError(f"Missing metadata for path: {path}")
|
||||
|
||||
file_bytes = max(metadata.size, 0)
|
||||
|
||||
# Trigger: the next file would overflow the active batch.
|
||||
# Why: keep batches memory-bounded and predictable for both local and remote callers.
|
||||
# Outcome: flush the current batch before placing the next file.
|
||||
if current_paths and (
|
||||
len(current_paths) >= max_files or current_bytes + file_bytes > max_bytes
|
||||
):
|
||||
batches.append(IndexBatch(paths=current_paths, total_bytes=current_bytes))
|
||||
current_paths = []
|
||||
current_bytes = 0
|
||||
|
||||
# Trigger: one file is larger than the configured byte budget.
|
||||
# Why: we still need to index it, but splitting a single file is out of scope.
|
||||
# Outcome: emit a dedicated single-file batch that may exceed max_bytes.
|
||||
if file_bytes > max_bytes:
|
||||
batches.append(IndexBatch(paths=[path], total_bytes=file_bytes))
|
||||
continue
|
||||
|
||||
current_paths.append(path)
|
||||
current_bytes += file_bytes
|
||||
|
||||
if len(current_paths) >= max_files or current_bytes == max_bytes:
|
||||
batches.append(IndexBatch(paths=current_paths, total_bytes=current_bytes))
|
||||
current_paths = []
|
||||
current_bytes = 0
|
||||
|
||||
if current_paths:
|
||||
batches.append(IndexBatch(paths=current_paths, total_bytes=current_bytes))
|
||||
|
||||
return batches
|
||||
@@ -1,110 +0,0 @@
|
||||
"""Typed models for the reusable indexing execution path."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
from typing import Any, Protocol, TYPE_CHECKING
|
||||
|
||||
if TYPE_CHECKING: # pragma: no cover
|
||||
from basic_memory.models import Entity
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class IndexFileMetadata:
|
||||
"""Storage-agnostic metadata for a file queued for indexing."""
|
||||
|
||||
path: str
|
||||
size: int
|
||||
checksum: str | None = None
|
||||
content_type: str | None = None
|
||||
last_modified: datetime | None = None
|
||||
created_at: datetime | None = None
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class IndexInputFile(IndexFileMetadata):
|
||||
"""Fully loaded file payload consumed by the batch executor."""
|
||||
|
||||
content: bytes | None = None
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class IndexBatch:
|
||||
"""A deterministic batch of files bounded by count and total bytes."""
|
||||
|
||||
paths: list[str]
|
||||
total_bytes: int
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class IndexProgress:
|
||||
"""Batch indexing progress emitted to callers such as the CLI."""
|
||||
|
||||
files_total: int
|
||||
files_processed: int
|
||||
batches_total: int
|
||||
batches_completed: int
|
||||
current_batch_bytes: int = 0
|
||||
files_per_minute: float = 0.0
|
||||
eta_seconds: float | None = None
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class IndexFrontmatterUpdate:
|
||||
"""A typed frontmatter write request for a single file."""
|
||||
|
||||
path: str
|
||||
metadata: dict[str, Any]
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class IndexFrontmatterWriteResult:
|
||||
"""Typed result for a frontmatter write performed during indexing."""
|
||||
|
||||
checksum: str
|
||||
content: str
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class IndexedEntity:
|
||||
"""Stable output describing one file that finished indexing successfully."""
|
||||
|
||||
path: str
|
||||
entity_id: int
|
||||
permalink: str | None
|
||||
checksum: str
|
||||
content_type: str | None = None
|
||||
markdown_content: str | None = None
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class SyncedMarkdownFile:
|
||||
"""Canonical result for syncing one markdown file end-to-end."""
|
||||
|
||||
entity: Entity
|
||||
checksum: str
|
||||
markdown_content: str
|
||||
file_path: str
|
||||
content_type: str
|
||||
updated_at: datetime
|
||||
size: int
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class IndexingBatchResult:
|
||||
"""Outcome for one batch execution."""
|
||||
|
||||
indexed: list[IndexedEntity] = field(default_factory=list)
|
||||
errors: list[tuple[str, str]] = field(default_factory=list)
|
||||
relations_resolved: int = 0
|
||||
relations_unresolved: int = 0
|
||||
search_indexed: int = 0
|
||||
|
||||
|
||||
class IndexFileWriter(Protocol):
|
||||
"""Narrow protocol for frontmatter writes during indexing."""
|
||||
|
||||
async def write_frontmatter(
|
||||
self, update: IndexFrontmatterUpdate
|
||||
) -> IndexFrontmatterWriteResult: ...
|
||||
@@ -249,10 +249,6 @@ class EntityParser:
|
||||
|
||||
content = strip_bom(content)
|
||||
|
||||
# PostgreSQL rejects null bytes (0x00) in text columns.
|
||||
# Some markdown files (e.g. Claude agent definitions) contain embedded nulls.
|
||||
content = content.replace("\x00", "")
|
||||
|
||||
# Parse frontmatter with proper error handling for malformed YAML.
|
||||
# We use frontmatter.parse() instead of frontmatter.loads() because
|
||||
# loads() does Post(content, handler, **metadata), which crashes when
|
||||
|
||||
@@ -85,28 +85,6 @@ def parse_observation(token: Token) -> Dict[str, Any]:
|
||||
|
||||
|
||||
# Relation handling functions
|
||||
def parse_relation_type(content: str) -> str | None:
|
||||
"""Return the explicit relation label before the first wikilink, if any."""
|
||||
before_link = content.partition("[[")[0].strip()
|
||||
if not before_link:
|
||||
return None
|
||||
|
||||
# Trigger: relation labels that need spaces must be quoted.
|
||||
# Why: unquoted multi-word prefixes are indistinguishable from prose
|
||||
# containing a wikilink.
|
||||
# Outcome: `some_type [[Target]]`, `"some type" [[Target]]`, and
|
||||
# `'some type' [[Target]]` are explicit; `some other thing [[Target]]`
|
||||
# falls back to inline `links_to` handling.
|
||||
quote = before_link[0]
|
||||
if quote in {"'", '"'} and before_link.endswith(quote):
|
||||
quoted_label = before_link[1:-1].strip()
|
||||
return quoted_label or None
|
||||
|
||||
if any(char.isspace() for char in before_link):
|
||||
return None
|
||||
return before_link
|
||||
|
||||
|
||||
def is_explicit_relation(token: Token) -> bool:
|
||||
"""Check if token looks like our relation format."""
|
||||
if token.type != "inline": # pragma: no cover
|
||||
@@ -114,7 +92,7 @@ def is_explicit_relation(token: Token) -> bool:
|
||||
|
||||
# Use token.tag which contains the actual content for test tokens, fallback to content
|
||||
content = (token.tag or token.content).strip()
|
||||
return "[[" in content and "]]" in content and parse_relation_type(content) is not None
|
||||
return "[[" in content and "]]" in content
|
||||
|
||||
|
||||
def parse_relation(token: Token) -> Dict[str, Any] | None:
|
||||
@@ -123,18 +101,20 @@ def parse_relation(token: Token) -> Dict[str, Any] | None:
|
||||
# Use token.tag which contains the actual content for test tokens, fallback to content
|
||||
content = (token.tag or token.content).strip()
|
||||
|
||||
rel_type = parse_relation_type(content)
|
||||
if rel_type is None:
|
||||
return None
|
||||
|
||||
# Extract [[target]]
|
||||
target = None
|
||||
rel_type = "relates_to" # default
|
||||
context = None
|
||||
|
||||
start = content.find("[[")
|
||||
end = content.find("]]", start + 2)
|
||||
end = content.find("]]")
|
||||
|
||||
if start != -1 and end != -1:
|
||||
# Get text before link as relation type
|
||||
before = content[:start].strip()
|
||||
if before:
|
||||
rel_type = before
|
||||
|
||||
# Get target
|
||||
target = normalize_project_reference(content[start + 2 : end].strip())
|
||||
|
||||
@@ -200,9 +180,6 @@ def observation_plugin(md: MarkdownIt) -> None:
|
||||
def observation_rule(state: Any) -> None:
|
||||
"""Process observations in token stream."""
|
||||
tokens = state.tokens
|
||||
# Track blockquote nesting so Obsidian callouts (`> [!info] Title`)
|
||||
# don't get parsed as observations with category `!info`.
|
||||
blockquote_depth = 0
|
||||
|
||||
for idx in range(len(tokens)):
|
||||
token = tokens[idx]
|
||||
@@ -210,18 +187,6 @@ def observation_plugin(md: MarkdownIt) -> None:
|
||||
# Initialize meta for all tokens
|
||||
token.meta = token.meta or {}
|
||||
|
||||
if token.type == "blockquote_open":
|
||||
blockquote_depth += 1
|
||||
continue
|
||||
if token.type == "blockquote_close":
|
||||
blockquote_depth -= 1
|
||||
continue
|
||||
|
||||
# Skip parsing inside blockquotes — that's Obsidian callout
|
||||
# territory, not Basic Memory observation syntax.
|
||||
if blockquote_depth > 0:
|
||||
continue
|
||||
|
||||
# Parse observations in list items
|
||||
if token.type == "inline" and is_observation(token):
|
||||
obs = parse_observation(token)
|
||||
@@ -237,8 +202,6 @@ def relation_plugin(md: MarkdownIt) -> None:
|
||||
|
||||
Explicit relations:
|
||||
- relation_type [[target]] (context)
|
||||
- "multi word relation type" [[target]] (context)
|
||||
- 'multi word relation type' [[target]] (context)
|
||||
|
||||
Implicit relations (links in content):
|
||||
Some text with [[target]] reference
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
"""Schema models for entity markdown files."""
|
||||
|
||||
from datetime import datetime
|
||||
from typing import TYPE_CHECKING, Any, List, Optional
|
||||
from typing import List, Optional
|
||||
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
from pydantic import BaseModel
|
||||
|
||||
|
||||
class Observation(BaseModel):
|
||||
@@ -38,47 +38,23 @@ class Relation(BaseModel):
|
||||
class EntityFrontmatter(BaseModel):
|
||||
"""Required frontmatter fields for an entity."""
|
||||
|
||||
if TYPE_CHECKING:
|
||||
# Frontmatter may be built from raw YAML keys. The validator below
|
||||
# gathers those keys into the metadata mapping used at runtime.
|
||||
def __init__(self, **data: Any) -> None: ...
|
||||
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
@model_validator(mode="before")
|
||||
@classmethod
|
||||
def collect_metadata(cls, data: Any) -> Any:
|
||||
if not isinstance(data, dict):
|
||||
return data
|
||||
|
||||
if "metadata" not in data:
|
||||
return {"metadata": data}
|
||||
|
||||
metadata = data.get("metadata") or {}
|
||||
extras = {key: value for key, value in data.items() if key != "metadata"}
|
||||
if extras:
|
||||
return {"metadata": {**extras, **metadata}}
|
||||
return data
|
||||
metadata: dict = {}
|
||||
|
||||
@property
|
||||
def tags(self) -> List[str]:
|
||||
tags = self.metadata.get("tags")
|
||||
return [str(tag) for tag in tags] if isinstance(tags, list) else []
|
||||
return self.metadata.get("tags") if self.metadata else None # pyright: ignore
|
||||
|
||||
@property
|
||||
def title(self) -> str:
|
||||
title = self.metadata.get("title")
|
||||
return title if isinstance(title, str) else ""
|
||||
return self.metadata.get("title") if self.metadata else None # pyright: ignore
|
||||
|
||||
@property
|
||||
def type(self) -> str:
|
||||
note_type = self.metadata.get("type", "note")
|
||||
return note_type if isinstance(note_type, str) else "note"
|
||||
return self.metadata.get("type", "note") if self.metadata else "note" # pyright: ignore
|
||||
|
||||
@property
|
||||
def permalink(self) -> Optional[str]:
|
||||
permalink = self.metadata.get("permalink")
|
||||
return permalink if isinstance(permalink, str) else None
|
||||
def permalink(self) -> str:
|
||||
return self.metadata.get("permalink") if self.metadata else None # pyright: ignore
|
||||
|
||||
|
||||
class EntityMarkdown(BaseModel):
|
||||
|
||||
@@ -5,7 +5,7 @@ from typing import AsyncIterator, Callable, Optional
|
||||
from httpx import ASGITransport, AsyncClient, Timeout
|
||||
from loguru import logger
|
||||
|
||||
import logfire
|
||||
from basic_memory.api.app import app as fastapi_app
|
||||
from basic_memory.config import ConfigManager, ProjectMode
|
||||
|
||||
|
||||
@@ -36,64 +36,28 @@ def _build_timeout() -> Timeout:
|
||||
|
||||
def _asgi_client(timeout: Timeout) -> AsyncClient:
|
||||
"""Create a local ASGI client."""
|
||||
# Import on first local-client use so CLI help/version paths can import
|
||||
# routing helpers without constructing the full FastAPI router graph.
|
||||
from basic_memory.api.app import app as fastapi_app
|
||||
from basic_memory.workspace_context import workspace_permalink_headers
|
||||
|
||||
return AsyncClient(
|
||||
transport=ASGITransport(app=fastapi_app),
|
||||
base_url="http://test",
|
||||
timeout=timeout,
|
||||
# Local ASGI calls still cross the HTTP boundary, so request handlers need
|
||||
# the same workspace permalink metadata that cloud proxy calls receive.
|
||||
headers=workspace_permalink_headers(),
|
||||
transport=ASGITransport(app=fastapi_app), base_url="http://test", timeout=timeout
|
||||
)
|
||||
|
||||
|
||||
async def _resolve_cloud_token(config) -> str:
|
||||
"""Resolve cloud token with API key preferred, OAuth fallback."""
|
||||
with logfire.span(
|
||||
"routing.resolve_cloud_credentials",
|
||||
has_api_key=bool(config.cloud_api_key),
|
||||
):
|
||||
token = config.cloud_api_key
|
||||
if token:
|
||||
return token
|
||||
token = config.cloud_api_key
|
||||
if token:
|
||||
return token
|
||||
|
||||
from basic_memory.cli.auth import CLIAuth
|
||||
from basic_memory.cli.auth import CLIAuth
|
||||
|
||||
auth = CLIAuth(client_id=config.cloud_client_id, authkit_domain=config.cloud_domain)
|
||||
token = await auth.get_valid_token()
|
||||
if token:
|
||||
return token
|
||||
auth = CLIAuth(client_id=config.cloud_client_id, authkit_domain=config.cloud_domain)
|
||||
token = await auth.get_valid_token()
|
||||
if token:
|
||||
return token
|
||||
|
||||
logger.error("Cloud routing requested but no credentials were available")
|
||||
raise RuntimeError(
|
||||
"Cloud routing requested but no credentials found. "
|
||||
"Run 'bm cloud api-key save <key>' or 'bm cloud login' first."
|
||||
)
|
||||
|
||||
|
||||
def resolve_configured_workspace(
|
||||
*,
|
||||
config=None,
|
||||
project_name: Optional[str] = None,
|
||||
workspace: Optional[str] = None,
|
||||
) -> Optional[str]:
|
||||
"""Resolve workspace from explicit input, per-project config, then global default."""
|
||||
if workspace is not None:
|
||||
return workspace
|
||||
|
||||
if config is None:
|
||||
config = ConfigManager().config
|
||||
|
||||
if project_name is not None:
|
||||
project_entry = config.projects.get(project_name)
|
||||
if project_entry and project_entry.workspace_id:
|
||||
return project_entry.workspace_id
|
||||
|
||||
return config.default_workspace
|
||||
raise RuntimeError(
|
||||
"Cloud routing requested but no credentials found. "
|
||||
"Run 'bm cloud api-key save <key>' or 'bm cloud login' first."
|
||||
)
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
@@ -103,12 +67,9 @@ async def _cloud_client(
|
||||
workspace: Optional[str] = None,
|
||||
) -> AsyncIterator[AsyncClient]:
|
||||
"""Create a cloud proxy client with resolved credentials."""
|
||||
from basic_memory.workspace_context import workspace_permalink_headers
|
||||
|
||||
token = await _resolve_cloud_token(config)
|
||||
proxy_base_url = f"{config.cloud_host}/proxy"
|
||||
headers = {"Authorization": f"Bearer {token}"}
|
||||
headers.update(workspace_permalink_headers())
|
||||
if workspace:
|
||||
headers["X-Workspace-ID"] = workspace
|
||||
logger.info(f"Creating HTTP client for cloud proxy at: {proxy_base_url}")
|
||||
@@ -121,33 +82,25 @@ async def _cloud_client(
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def get_cloud_control_plane_client(
|
||||
workspace: Optional[str] = None,
|
||||
) -> AsyncIterator[AsyncClient]:
|
||||
async def get_cloud_control_plane_client() -> AsyncIterator[AsyncClient]:
|
||||
"""Create a control-plane cloud client for endpoints outside /proxy."""
|
||||
config = ConfigManager().config
|
||||
timeout = _build_timeout()
|
||||
token = await _resolve_cloud_token(config)
|
||||
headers = {"Authorization": f"Bearer {token}"}
|
||||
if workspace:
|
||||
headers["X-Workspace-ID"] = workspace
|
||||
logger.info(f"Creating HTTP client for cloud control plane at: {config.cloud_host}")
|
||||
async with AsyncClient(
|
||||
base_url=config.cloud_host,
|
||||
headers=headers,
|
||||
headers={"Authorization": f"Bearer {token}"},
|
||||
timeout=timeout,
|
||||
) as client:
|
||||
yield client
|
||||
|
||||
|
||||
# Optional factory override for dependency injection.
|
||||
# The factory accepts an optional workspace keyword argument so that MCP tools
|
||||
# can route individual requests to a different workspace than the one set at
|
||||
# connection time. See basic-memory-cloud main.py tenant_asgi_client_factory.
|
||||
_client_factory: Optional[Callable[..., AbstractAsyncContextManager[AsyncClient]]] = None
|
||||
# Optional factory override for dependency injection
|
||||
_client_factory: Optional[Callable[[], AbstractAsyncContextManager[AsyncClient]]] = None
|
||||
|
||||
|
||||
def set_client_factory(factory: Callable[..., AbstractAsyncContextManager[AsyncClient]]) -> None:
|
||||
def set_client_factory(factory: Callable[[], AbstractAsyncContextManager[AsyncClient]]) -> None:
|
||||
"""Override the default client factory (for cloud app, testing, etc)."""
|
||||
global _client_factory
|
||||
_client_factory = factory
|
||||
@@ -188,7 +141,7 @@ async def get_client(
|
||||
4. Local ASGI transport by default.
|
||||
"""
|
||||
if _client_factory:
|
||||
async with _client_factory(workspace=workspace) as client:
|
||||
async with _client_factory() as client:
|
||||
yield client
|
||||
return
|
||||
|
||||
@@ -208,12 +161,7 @@ async def get_client(
|
||||
|
||||
if _force_cloud_mode():
|
||||
logger.debug("Explicit cloud routing enabled - using cloud proxy client")
|
||||
effective_workspace = resolve_configured_workspace(
|
||||
config=config,
|
||||
project_name=project_name,
|
||||
workspace=workspace,
|
||||
)
|
||||
async with _cloud_client(config, timeout, workspace=effective_workspace) as client:
|
||||
async with _cloud_client(config, timeout, workspace=workspace) as client:
|
||||
yield client
|
||||
return
|
||||
|
||||
@@ -225,13 +173,8 @@ async def get_client(
|
||||
project_mode = config.get_project_mode(project_name)
|
||||
if project_mode == ProjectMode.CLOUD:
|
||||
logger.debug(f"Project '{project_name}' is cloud mode - using cloud proxy client")
|
||||
effective_workspace = resolve_configured_workspace(
|
||||
config=config,
|
||||
project_name=project_name,
|
||||
workspace=workspace,
|
||||
)
|
||||
try:
|
||||
async with _cloud_client(config, timeout, workspace=effective_workspace) as client:
|
||||
async with _cloud_client(config, timeout, workspace=workspace) as client:
|
||||
yield client
|
||||
except RuntimeError as exc:
|
||||
raise RuntimeError(
|
||||
|
||||
@@ -18,6 +18,8 @@ from basic_memory.mcp.clients.directory import DirectoryClient
|
||||
from basic_memory.mcp.clients.resource import ResourceClient
|
||||
from basic_memory.mcp.clients.project import ProjectClient
|
||||
from basic_memory.mcp.clients.schema import SchemaClient
|
||||
from basic_memory.mcp.clients.graph import GraphClient
|
||||
from basic_memory.mcp.clients.fcm import FCMClient
|
||||
|
||||
__all__ = [
|
||||
"KnowledgeClient",
|
||||
@@ -27,4 +29,6 @@ __all__ = [
|
||||
"ResourceClient",
|
||||
"ProjectClient",
|
||||
"SchemaClient",
|
||||
"GraphClient",
|
||||
"FCMClient",
|
||||
]
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
"""Typed client for FCM API operations."""
|
||||
|
||||
from httpx import AsyncClient
|
||||
|
||||
from basic_memory.mcp.tools.utils import call_post
|
||||
from basic_memory.schemas.graph_intelligence import (
|
||||
FCMExportRequest,
|
||||
FCMExportResponse,
|
||||
FCMImportRequest,
|
||||
FCMImportResponse,
|
||||
FCMRankActionsRequest,
|
||||
FCMRankActionsResponse,
|
||||
FCMSimulateRequest,
|
||||
FCMSimulateResponse,
|
||||
)
|
||||
|
||||
|
||||
class FCMClient:
|
||||
"""Typed client for FCM operations."""
|
||||
|
||||
def __init__(self, http_client: AsyncClient, project_id: str):
|
||||
self.http_client = http_client
|
||||
self.project_id = project_id
|
||||
self._base_path = f"/v2/projects/{project_id}/fcm"
|
||||
|
||||
async def simulate(self, request: FCMSimulateRequest) -> FCMSimulateResponse:
|
||||
response = await call_post(
|
||||
self.http_client,
|
||||
f"{self._base_path}/simulate",
|
||||
json=request.model_dump(mode="json"),
|
||||
)
|
||||
return FCMSimulateResponse.model_validate(response.json())
|
||||
|
||||
async def rank_actions(self, request: FCMRankActionsRequest) -> FCMRankActionsResponse:
|
||||
response = await call_post(
|
||||
self.http_client,
|
||||
f"{self._base_path}/rank-actions",
|
||||
json=request.model_dump(mode="json"),
|
||||
)
|
||||
return FCMRankActionsResponse.model_validate(response.json())
|
||||
|
||||
async def import_model(self, request: FCMImportRequest) -> FCMImportResponse:
|
||||
response = await call_post(
|
||||
self.http_client,
|
||||
f"{self._base_path}/import",
|
||||
json=request.model_dump(mode="json"),
|
||||
)
|
||||
return FCMImportResponse.model_validate(response.json())
|
||||
|
||||
async def export_model(self, request: FCMExportRequest) -> FCMExportResponse:
|
||||
response = await call_post(
|
||||
self.http_client,
|
||||
f"{self._base_path}/export",
|
||||
json=request.model_dump(mode="json"),
|
||||
)
|
||||
return FCMExportResponse.model_validate(response.json())
|
||||
@@ -0,0 +1,62 @@
|
||||
"""Typed client for graph intelligence API operations."""
|
||||
|
||||
from httpx import AsyncClient
|
||||
|
||||
from basic_memory.mcp.tools.utils import call_get, call_post
|
||||
from basic_memory.schemas.graph_intelligence import (
|
||||
GraphHealthResponse,
|
||||
GraphImpactRequest,
|
||||
GraphImpactResponse,
|
||||
GraphLineageRequest,
|
||||
GraphLineageResponse,
|
||||
GraphReindexRequest,
|
||||
GraphReindexResponse,
|
||||
)
|
||||
|
||||
|
||||
class GraphClient:
|
||||
"""Typed client for graph intelligence operations."""
|
||||
|
||||
def __init__(self, http_client: AsyncClient, project_id: str):
|
||||
self.http_client = http_client
|
||||
self.project_id = project_id
|
||||
self._base_path = f"/v2/projects/{project_id}/graph"
|
||||
|
||||
async def lineage(self, request: GraphLineageRequest) -> GraphLineageResponse:
|
||||
response = await call_post(
|
||||
self.http_client,
|
||||
f"{self._base_path}/lineage",
|
||||
json=request.model_dump(mode="json"),
|
||||
)
|
||||
return GraphLineageResponse.model_validate(response.json())
|
||||
|
||||
async def impact(self, request: GraphImpactRequest) -> GraphImpactResponse:
|
||||
response = await call_post(
|
||||
self.http_client,
|
||||
f"{self._base_path}/impact",
|
||||
json=request.model_dump(mode="json"),
|
||||
)
|
||||
return GraphImpactResponse.model_validate(response.json())
|
||||
|
||||
async def health(
|
||||
self, scope: str | None = None, timeframe: str | None = None
|
||||
) -> GraphHealthResponse:
|
||||
params: dict[str, str] = {}
|
||||
if scope is not None:
|
||||
params["scope"] = scope
|
||||
if timeframe is not None:
|
||||
params["timeframe"] = timeframe
|
||||
response = await call_get(
|
||||
self.http_client,
|
||||
f"{self._base_path}/health",
|
||||
params=params,
|
||||
)
|
||||
return GraphHealthResponse.model_validate(response.json())
|
||||
|
||||
async def reindex(self, request: GraphReindexRequest) -> GraphReindexResponse:
|
||||
response = await call_post(
|
||||
self.http_client,
|
||||
f"{self._base_path}/reindex",
|
||||
json=request.model_dump(mode="json"),
|
||||
)
|
||||
return GraphReindexResponse.model_validate(response.json())
|
||||
@@ -7,7 +7,6 @@ from typing import Any
|
||||
|
||||
from httpx import AsyncClient
|
||||
|
||||
import logfire
|
||||
from basic_memory.mcp.tools.utils import call_get, call_post, call_put, call_patch, call_delete
|
||||
from basic_memory.schemas.response import (
|
||||
EntityResponse,
|
||||
@@ -15,7 +14,6 @@ from basic_memory.schemas.response import (
|
||||
DirectoryMoveResult,
|
||||
DirectoryDeleteResult,
|
||||
)
|
||||
from basic_memory.schemas.v2.graph import GraphNode, OrphanEntitiesResponse
|
||||
|
||||
|
||||
class KnowledgeClient:
|
||||
@@ -45,7 +43,9 @@ class KnowledgeClient:
|
||||
|
||||
# --- Entity CRUD Operations ---
|
||||
|
||||
async def create_entity(self, entity_data: dict[str, Any]) -> EntityResponse:
|
||||
async def create_entity(
|
||||
self, entity_data: dict[str, Any], *, fast: bool | None = None
|
||||
) -> EntityResponse:
|
||||
"""Create a new entity.
|
||||
|
||||
Args:
|
||||
@@ -57,25 +57,21 @@ class KnowledgeClient:
|
||||
Raises:
|
||||
ToolError: If the request fails
|
||||
"""
|
||||
with logfire.span(
|
||||
"mcp.client.knowledge.create_entity",
|
||||
client_name="knowledge",
|
||||
operation="create_entity",
|
||||
):
|
||||
response = await call_post(
|
||||
self.http_client,
|
||||
f"{self._base_path}/entities",
|
||||
json=entity_data,
|
||||
client_name="knowledge",
|
||||
operation="create_entity",
|
||||
path_template="/v2/projects/{project_id}/knowledge/entities",
|
||||
)
|
||||
params = {"fast": fast} if fast is not None else None
|
||||
response = await call_post(
|
||||
self.http_client,
|
||||
f"{self._base_path}/entities",
|
||||
json=entity_data,
|
||||
params=params,
|
||||
)
|
||||
return EntityResponse.model_validate(response.json())
|
||||
|
||||
async def update_entity(
|
||||
self,
|
||||
entity_id: str,
|
||||
entity_data: dict[str, Any],
|
||||
*,
|
||||
fast: bool | None = None,
|
||||
) -> EntityResponse:
|
||||
"""Update an existing entity (full replacement).
|
||||
|
||||
@@ -89,19 +85,13 @@ class KnowledgeClient:
|
||||
Raises:
|
||||
ToolError: If the request fails
|
||||
"""
|
||||
with logfire.span(
|
||||
"mcp.client.knowledge.update_entity",
|
||||
client_name="knowledge",
|
||||
operation="update_entity",
|
||||
):
|
||||
response = await call_put(
|
||||
self.http_client,
|
||||
f"{self._base_path}/entities/{entity_id}",
|
||||
json=entity_data,
|
||||
client_name="knowledge",
|
||||
operation="update_entity",
|
||||
path_template="/v2/projects/{project_id}/knowledge/entities/{entity_id}",
|
||||
)
|
||||
params = {"fast": fast} if fast is not None else None
|
||||
response = await call_put(
|
||||
self.http_client,
|
||||
f"{self._base_path}/entities/{entity_id}",
|
||||
json=entity_data,
|
||||
params=params,
|
||||
)
|
||||
return EntityResponse.model_validate(response.json())
|
||||
|
||||
async def get_entity(self, entity_id: str) -> EntityResponse:
|
||||
@@ -116,24 +106,18 @@ class KnowledgeClient:
|
||||
Raises:
|
||||
ToolError: If the entity is not found or request fails
|
||||
"""
|
||||
with logfire.span(
|
||||
"mcp.client.knowledge.get_entity",
|
||||
client_name="knowledge",
|
||||
operation="get_entity",
|
||||
):
|
||||
response = await call_get(
|
||||
self.http_client,
|
||||
f"{self._base_path}/entities/{entity_id}",
|
||||
client_name="knowledge",
|
||||
operation="get_entity",
|
||||
path_template="/v2/projects/{project_id}/knowledge/entities/{entity_id}",
|
||||
)
|
||||
response = await call_get(
|
||||
self.http_client,
|
||||
f"{self._base_path}/entities/{entity_id}",
|
||||
)
|
||||
return EntityResponse.model_validate(response.json())
|
||||
|
||||
async def patch_entity(
|
||||
self,
|
||||
entity_id: str,
|
||||
patch_data: dict[str, Any],
|
||||
*,
|
||||
fast: bool | None = None,
|
||||
) -> EntityResponse:
|
||||
"""Partially update an entity.
|
||||
|
||||
@@ -147,19 +131,13 @@ class KnowledgeClient:
|
||||
Raises:
|
||||
ToolError: If the request fails
|
||||
"""
|
||||
with logfire.span(
|
||||
"mcp.client.knowledge.patch_entity",
|
||||
client_name="knowledge",
|
||||
operation="patch_entity",
|
||||
):
|
||||
response = await call_patch(
|
||||
self.http_client,
|
||||
f"{self._base_path}/entities/{entity_id}",
|
||||
json=patch_data,
|
||||
client_name="knowledge",
|
||||
operation="patch_entity",
|
||||
path_template="/v2/projects/{project_id}/knowledge/entities/{entity_id}",
|
||||
)
|
||||
params = {"fast": fast} if fast is not None else None
|
||||
response = await call_patch(
|
||||
self.http_client,
|
||||
f"{self._base_path}/entities/{entity_id}",
|
||||
json=patch_data,
|
||||
params=params,
|
||||
)
|
||||
return EntityResponse.model_validate(response.json())
|
||||
|
||||
async def delete_entity(self, entity_id: str) -> DeleteEntitiesResponse:
|
||||
@@ -174,18 +152,10 @@ class KnowledgeClient:
|
||||
Raises:
|
||||
ToolError: If the entity is not found or request fails
|
||||
"""
|
||||
with logfire.span(
|
||||
"mcp.client.knowledge.delete_entity",
|
||||
client_name="knowledge",
|
||||
operation="delete_entity",
|
||||
):
|
||||
response = await call_delete(
|
||||
self.http_client,
|
||||
f"{self._base_path}/entities/{entity_id}",
|
||||
client_name="knowledge",
|
||||
operation="delete_entity",
|
||||
path_template="/v2/projects/{project_id}/knowledge/entities/{entity_id}",
|
||||
)
|
||||
response = await call_delete(
|
||||
self.http_client,
|
||||
f"{self._base_path}/entities/{entity_id}",
|
||||
)
|
||||
return DeleteEntitiesResponse.model_validate(response.json())
|
||||
|
||||
async def move_entity(self, entity_id: str, destination_path: str) -> EntityResponse:
|
||||
@@ -201,19 +171,11 @@ class KnowledgeClient:
|
||||
Raises:
|
||||
ToolError: If the request fails
|
||||
"""
|
||||
with logfire.span(
|
||||
"mcp.client.knowledge.move_entity",
|
||||
client_name="knowledge",
|
||||
operation="move_entity",
|
||||
):
|
||||
response = await call_put(
|
||||
self.http_client,
|
||||
f"{self._base_path}/entities/{entity_id}/move",
|
||||
json={"destination_path": destination_path},
|
||||
client_name="knowledge",
|
||||
operation="move_entity",
|
||||
path_template="/v2/projects/{project_id}/knowledge/entities/{entity_id}/move",
|
||||
)
|
||||
response = await call_put(
|
||||
self.http_client,
|
||||
f"{self._base_path}/entities/{entity_id}/move",
|
||||
json={"destination_path": destination_path},
|
||||
)
|
||||
return EntityResponse.model_validate(response.json())
|
||||
|
||||
async def move_directory(
|
||||
@@ -231,22 +193,14 @@ class KnowledgeClient:
|
||||
Raises:
|
||||
ToolError: If the request fails
|
||||
"""
|
||||
with logfire.span(
|
||||
"mcp.client.knowledge.move_directory",
|
||||
client_name="knowledge",
|
||||
operation="move_directory",
|
||||
):
|
||||
response = await call_post(
|
||||
self.http_client,
|
||||
f"{self._base_path}/move-directory",
|
||||
json={
|
||||
"source_directory": source_directory,
|
||||
"destination_directory": destination_directory,
|
||||
},
|
||||
client_name="knowledge",
|
||||
operation="move_directory",
|
||||
path_template="/v2/projects/{project_id}/knowledge/move-directory",
|
||||
)
|
||||
response = await call_post(
|
||||
self.http_client,
|
||||
f"{self._base_path}/move-directory",
|
||||
json={
|
||||
"source_directory": source_directory,
|
||||
"destination_directory": destination_directory,
|
||||
},
|
||||
)
|
||||
return DirectoryMoveResult.model_validate(response.json())
|
||||
|
||||
async def delete_directory(self, directory: str) -> DirectoryDeleteResult:
|
||||
@@ -261,39 +215,13 @@ class KnowledgeClient:
|
||||
Raises:
|
||||
ToolError: If the request fails
|
||||
"""
|
||||
with logfire.span(
|
||||
"mcp.client.knowledge.delete_directory",
|
||||
client_name="knowledge",
|
||||
operation="delete_directory",
|
||||
):
|
||||
response = await call_post(
|
||||
self.http_client,
|
||||
f"{self._base_path}/delete-directory",
|
||||
json={"directory": directory},
|
||||
client_name="knowledge",
|
||||
operation="delete_directory",
|
||||
path_template="/v2/projects/{project_id}/knowledge/delete-directory",
|
||||
)
|
||||
response = await call_post(
|
||||
self.http_client,
|
||||
f"{self._base_path}/delete-directory",
|
||||
json={"directory": directory},
|
||||
)
|
||||
return DirectoryDeleteResult.model_validate(response.json())
|
||||
|
||||
# --- Orphan detection ---
|
||||
|
||||
async def get_orphans(self) -> list[GraphNode]:
|
||||
"""Get entities that have no incoming or outgoing relations."""
|
||||
with logfire.span(
|
||||
"mcp.client.knowledge.get_orphans",
|
||||
client_name="knowledge",
|
||||
operation="get_orphans",
|
||||
):
|
||||
response = await call_get(
|
||||
self.http_client,
|
||||
f"{self._base_path}/orphans",
|
||||
client_name="knowledge",
|
||||
operation="get_orphans",
|
||||
path_template="/v2/projects/{project_id}/knowledge/orphans",
|
||||
)
|
||||
return OrphanEntitiesResponse.model_validate(response.json()).entities
|
||||
|
||||
# --- Resolution ---
|
||||
|
||||
async def resolve_entity(self, identifier: str, *, strict: bool = False) -> str:
|
||||
@@ -309,18 +237,10 @@ class KnowledgeClient:
|
||||
Raises:
|
||||
ToolError: If the identifier cannot be resolved
|
||||
"""
|
||||
with logfire.span(
|
||||
"mcp.client.knowledge.resolve_entity",
|
||||
client_name="knowledge",
|
||||
operation="resolve_entity",
|
||||
):
|
||||
response = await call_post(
|
||||
self.http_client,
|
||||
f"{self._base_path}/resolve",
|
||||
json={"identifier": identifier, "strict": strict},
|
||||
client_name="knowledge",
|
||||
operation="resolve_entity",
|
||||
path_template="/v2/projects/{project_id}/knowledge/resolve",
|
||||
)
|
||||
response = await call_post(
|
||||
self.http_client,
|
||||
f"{self._base_path}/resolve",
|
||||
json={"identifier": identifier, "strict": strict},
|
||||
)
|
||||
data = response.json()
|
||||
return data["external_id"]
|
||||
|
||||
@@ -7,7 +7,6 @@ from typing import Optional
|
||||
|
||||
from httpx import AsyncClient
|
||||
|
||||
import logfire
|
||||
from basic_memory.mcp.tools.utils import call_get
|
||||
from basic_memory.schemas.memory import GraphContext
|
||||
|
||||
@@ -72,21 +71,11 @@ class MemoryClient:
|
||||
if timeframe:
|
||||
params["timeframe"] = timeframe
|
||||
|
||||
with logfire.span(
|
||||
"mcp.client.memory.build_context",
|
||||
client_name="memory",
|
||||
operation="build_context",
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
):
|
||||
response = await call_get(
|
||||
self.http_client,
|
||||
f"{self._base_path}/{path}",
|
||||
params=params,
|
||||
client_name="memory",
|
||||
operation="build_context",
|
||||
path_template="/v2/projects/{project_id}/memory/{path}",
|
||||
)
|
||||
response = await call_get(
|
||||
self.http_client,
|
||||
f"{self._base_path}/{path}",
|
||||
params=params,
|
||||
)
|
||||
return GraphContext.model_validate(response.json())
|
||||
|
||||
async def recent(
|
||||
@@ -123,19 +112,9 @@ class MemoryClient:
|
||||
# Join types as comma-separated string if provided
|
||||
params["type"] = ",".join(types) if isinstance(types, list) else types
|
||||
|
||||
with logfire.span(
|
||||
"mcp.client.memory.recent_activity",
|
||||
client_name="memory",
|
||||
operation="recent_activity",
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
):
|
||||
response = await call_get(
|
||||
self.http_client,
|
||||
f"{self._base_path}/recent",
|
||||
params=params,
|
||||
client_name="memory",
|
||||
operation="recent_activity",
|
||||
path_template="/v2/projects/{project_id}/memory/recent",
|
||||
)
|
||||
response = await call_get(
|
||||
self.http_client,
|
||||
f"{self._base_path}/recent",
|
||||
params=params,
|
||||
)
|
||||
return GraphContext.model_validate(response.json())
|
||||
|
||||
@@ -3,9 +3,10 @@
|
||||
Encapsulates all /v2/projects/{project_id}/resource/* endpoints.
|
||||
"""
|
||||
|
||||
from typing import Optional
|
||||
|
||||
from httpx import AsyncClient, Response
|
||||
|
||||
import logfire
|
||||
from basic_memory.mcp.tools.utils import call_get
|
||||
|
||||
|
||||
@@ -37,11 +38,19 @@ class ResourceClient:
|
||||
self.project_id = project_id
|
||||
self._base_path = f"/v2/projects/{project_id}/resource"
|
||||
|
||||
async def read(self, entity_id: str) -> Response:
|
||||
async def read(
|
||||
self,
|
||||
entity_id: str,
|
||||
*,
|
||||
page: Optional[int] = None,
|
||||
page_size: Optional[int] = None,
|
||||
) -> Response:
|
||||
"""Read a resource by entity ID.
|
||||
|
||||
Args:
|
||||
entity_id: Entity external_id (UUID)
|
||||
page: Optional page number for paginated content
|
||||
page_size: Optional page size for paginated content
|
||||
|
||||
Returns:
|
||||
Raw HTTP Response (caller handles text/binary content)
|
||||
@@ -49,15 +58,14 @@ class ResourceClient:
|
||||
Raises:
|
||||
ToolError: If the resource is not found or request fails
|
||||
"""
|
||||
with logfire.span(
|
||||
"mcp.client.resource.read",
|
||||
client_name="resource",
|
||||
operation="read",
|
||||
):
|
||||
return await call_get(
|
||||
self.http_client,
|
||||
f"{self._base_path}/{entity_id}",
|
||||
client_name="resource",
|
||||
operation="read",
|
||||
path_template="/v2/projects/{project_id}/resource/{entity_id}",
|
||||
)
|
||||
params: dict = {}
|
||||
if page is not None:
|
||||
params["page"] = page
|
||||
if page_size is not None:
|
||||
params["page_size"] = page_size
|
||||
|
||||
return await call_get(
|
||||
self.http_client,
|
||||
f"{self._base_path}/{entity_id}",
|
||||
params=params if params else None,
|
||||
)
|
||||
|
||||
@@ -7,7 +7,6 @@ from typing import Any
|
||||
|
||||
from httpx import AsyncClient
|
||||
|
||||
import logfire
|
||||
from basic_memory.mcp.tools.utils import call_post
|
||||
from basic_memory.schemas.search import SearchResponse
|
||||
|
||||
@@ -57,20 +56,10 @@ class SearchClient:
|
||||
Raises:
|
||||
ToolError: If the request fails
|
||||
"""
|
||||
with logfire.span(
|
||||
"mcp.client.search.search",
|
||||
client_name="search",
|
||||
operation="search",
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
):
|
||||
response = await call_post(
|
||||
self.http_client,
|
||||
f"{self._base_path}/",
|
||||
json=query,
|
||||
params={"page": page, "page_size": page_size},
|
||||
client_name="search",
|
||||
operation="search",
|
||||
path_template="/v2/projects/{project_id}/search/",
|
||||
)
|
||||
response = await call_post(
|
||||
self.http_client,
|
||||
f"{self._base_path}/",
|
||||
json=query,
|
||||
params={"page": page, "page_size": page_size},
|
||||
)
|
||||
return SearchResponse.model_validate(response.json())
|
||||
|
||||
+241
-1301
File diff suppressed because it is too large
Load Diff
@@ -95,8 +95,8 @@ def format_prompt_context(context: PromptContext) -> str:
|
||||
sections = []
|
||||
|
||||
# Process each context
|
||||
for context_item in context.results:
|
||||
for primary in context_item.primary_results:
|
||||
for context in context.results: # pyright: ignore
|
||||
for primary in context.primary_results: # pyright: ignore
|
||||
if primary.permalink not in added_permalinks:
|
||||
primary_permalink = primary.permalink
|
||||
|
||||
@@ -121,8 +121,8 @@ def format_prompt_context(context: PromptContext) -> str:
|
||||
section += f"- **Created**: {primary.created_at.strftime('%Y-%m-%d %H:%M')}\n"
|
||||
|
||||
# Add content snippet
|
||||
if hasattr(primary, "content") and primary.content:
|
||||
content = primary.content or "" # pragma: no cover
|
||||
if hasattr(primary, "content") and primary.content: # pyright: ignore
|
||||
content = primary.content or "" # pyright: ignore # pragma: no cover
|
||||
if content: # pragma: no cover
|
||||
section += f"\n**Excerpt**:\n{content}\n" # pragma: no cover
|
||||
|
||||
@@ -132,14 +132,14 @@ def format_prompt_context(context: PromptContext) -> str:
|
||||
""")
|
||||
sections.append(section)
|
||||
|
||||
if context_item.related_results:
|
||||
section += dedent(
|
||||
if context.related_results: # pyright: ignore
|
||||
section += dedent( # pyright: ignore
|
||||
"""
|
||||
## Related Context
|
||||
"""
|
||||
)
|
||||
|
||||
for related in context_item.related_results:
|
||||
for related in context.related_results: # pyright: ignore
|
||||
section_content = dedent(f"""
|
||||
- type: **{related.type}**
|
||||
- title: {related.title}
|
||||
|
||||
@@ -18,15 +18,13 @@ Basic Memory creates a semantic knowledge graph from markdown files. Focus on bu
|
||||
|
||||
**Resolution priority:**
|
||||
1. CLI constraint: `BASIC_MEMORY_MCP_PROJECT` env var (highest priority)
|
||||
2. Explicit parameter: `project_id="<uuid>"` (preferred when known) or `project="name"` in tool calls
|
||||
2. Explicit parameter: `project="name"` in tool calls
|
||||
3. Default project: `default_project` in config (fallback)
|
||||
|
||||
**`project` vs `project_id`:** Every project has a stable `external_id` (UUID) returned by `list_memory_projects()`. Pass it as `project_id=...` to address a project unambiguously — required when the same project name exists in multiple cloud workspaces. For local single-project setups, the `project` name is fine.
|
||||
|
||||
### Quick Setup Check
|
||||
|
||||
```python
|
||||
# Discover projects (each entry includes external_id you can pass as project_id)
|
||||
# Discover projects
|
||||
projects = await list_memory_projects()
|
||||
```
|
||||
|
||||
@@ -171,14 +169,6 @@ await write_note(
|
||||
**Multi-project users:**
|
||||
- Always specify project explicitly in tool calls
|
||||
|
||||
**Cloud multi-workspace users:** project names can collide across workspaces. After calling `list_memory_projects()`, prefer the project's `external_id` via `project_id=...` for any subsequent tool calls — it routes to the exact project regardless of name collisions. The `project` name parameter falls back to the default workspace on ambiguity, which may not be what you want.
|
||||
|
||||
```python
|
||||
# Cloud / multi-workspace: prefer project_id (UUID) once you've discovered it
|
||||
projects = await list_memory_projects()
|
||||
results = await search_notes(query="auth", project_id=projects[0]["external_id"])
|
||||
```
|
||||
|
||||
**Discovery:**
|
||||
```python
|
||||
# Start with discovery
|
||||
|
||||
@@ -7,45 +7,11 @@ from contextlib import asynccontextmanager
|
||||
|
||||
from fastmcp import FastMCP
|
||||
from loguru import logger
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.ext.asyncio import async_sessionmaker, AsyncSession
|
||||
|
||||
from basic_memory import db
|
||||
from basic_memory.cli.auth import CLIAuth
|
||||
from basic_memory.db import scoped_session
|
||||
from basic_memory.mcp.container import McpContainer, set_container
|
||||
from basic_memory.services.initialization import initialize_app
|
||||
import logfire
|
||||
|
||||
|
||||
async def _log_embedding_status(session_maker: async_sessionmaker[AsyncSession]) -> None:
|
||||
"""Log a clear summary of semantic embedding status at startup."""
|
||||
try:
|
||||
async with scoped_session(session_maker) as session:
|
||||
entity_count = (
|
||||
await session.execute(text("SELECT COUNT(*) FROM entity"))
|
||||
).scalar() or 0
|
||||
chunk_count = (
|
||||
await session.execute(text("SELECT COUNT(*) FROM search_vector_chunks"))
|
||||
).scalar() or 0
|
||||
embedding_count = (
|
||||
await session.execute(text("SELECT COUNT(*) FROM search_vector_embeddings_rowids"))
|
||||
).scalar() or 0
|
||||
|
||||
if entity_count == 0:
|
||||
logger.info("Semantic embeddings: no entities yet")
|
||||
elif embedding_count == 0:
|
||||
logger.warning(
|
||||
f"Semantic embeddings: EMPTY — {entity_count} entities have no embeddings. "
|
||||
"Run 'bm reindex --embeddings' to build them."
|
||||
)
|
||||
else:
|
||||
logger.info(
|
||||
f"Semantic embeddings: {embedding_count} embeddings "
|
||||
f"across {chunk_count} chunks for {entity_count} entities"
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.debug(f"Could not check embedding status at startup: {exc}")
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
@@ -63,82 +29,64 @@ async def lifespan(app: FastMCP):
|
||||
set_container(container)
|
||||
|
||||
config = container.config
|
||||
with logfire.span(
|
||||
"mcp.lifecycle.startup",
|
||||
entrypoint="mcp",
|
||||
mode=container.mode.name.lower(),
|
||||
default_project=config.default_project,
|
||||
):
|
||||
logger.info(f"Starting Basic Memory MCP server (mode={container.mode.name})")
|
||||
logger.info(f"Starting Basic Memory MCP server (mode={container.mode.name})")
|
||||
logger.info(
|
||||
f"Config: database_backend={config.database_backend.value}, "
|
||||
f"semantic_search_enabled={config.semantic_search_enabled}, "
|
||||
f"default_project={config.default_project}"
|
||||
)
|
||||
if config.semantic_search_enabled:
|
||||
logger.info(
|
||||
f"Config: database_backend={config.database_backend.value}, "
|
||||
f"semantic_search_enabled={config.semantic_search_enabled}, "
|
||||
f"default_project={config.default_project}"
|
||||
f"Semantic search: provider={config.semantic_embedding_provider}, "
|
||||
f"model={config.semantic_embedding_model}, "
|
||||
f"dimensions={config.semantic_embedding_dimensions or 'auto'}, "
|
||||
f"batch_size={config.semantic_embedding_batch_size}"
|
||||
)
|
||||
if config.semantic_search_enabled:
|
||||
logger.info(
|
||||
f"Semantic search: provider={config.semantic_embedding_provider}, "
|
||||
f"model={config.semantic_embedding_model}, "
|
||||
f"dimensions={config.semantic_embedding_dimensions or 'auto'}, "
|
||||
f"batch_size={config.semantic_embedding_batch_size}"
|
||||
)
|
||||
|
||||
# Log configured projects with their routing mode
|
||||
for name, entry in config.projects.items():
|
||||
default = " (default)" if name == config.default_project else ""
|
||||
logger.info(f"Project: {name} -> {entry.path} [mode={entry.mode.value}]{default}")
|
||||
# Log configured projects with their routing mode
|
||||
for name, entry in config.projects.items():
|
||||
default = " (default)" if name == config.default_project else ""
|
||||
logger.info(f"Project: {name} -> {entry.path} [mode={entry.mode.value}]{default}")
|
||||
|
||||
# Check cloud auth status (local file check, no network call)
|
||||
auth = CLIAuth(client_id=config.cloud_client_id, authkit_domain=config.cloud_domain)
|
||||
tokens = auth.load_tokens()
|
||||
if tokens is not None:
|
||||
if not auth.is_token_valid(tokens):
|
||||
expires_at = tokens.get("expires_at", 0)
|
||||
expired_ago = int(time.time() - expires_at)
|
||||
logger.warning(
|
||||
f"Cloud token expired {expired_ago}s ago - may need 'bm cloud login'"
|
||||
)
|
||||
else:
|
||||
logger.info("Cloud: authenticated (OAuth token valid)")
|
||||
# Check cloud auth status (local file check, no network call)
|
||||
auth = CLIAuth(client_id=config.cloud_client_id, authkit_domain=config.cloud_domain)
|
||||
tokens = auth.load_tokens()
|
||||
if tokens is not None:
|
||||
if not auth.is_token_valid(tokens):
|
||||
expires_at = tokens.get("expires_at", 0)
|
||||
expired_ago = int(time.time() - expires_at)
|
||||
logger.warning(f"Cloud token expired {expired_ago}s ago - may need 'bm cloud login'")
|
||||
else:
|
||||
logger.info("Cloud: authenticated (OAuth token valid)")
|
||||
|
||||
if config.cloud_api_key:
|
||||
logger.info("Cloud: API key configured")
|
||||
if config.cloud_api_key:
|
||||
logger.info("Cloud: API key configured")
|
||||
|
||||
# Track if we created the engine (vs test fixtures providing it)
|
||||
# This prevents disposing an engine provided by test fixtures when
|
||||
# multiple Client connections are made in the same test
|
||||
engine_was_none = db._engine is None
|
||||
# Track if we created the engine (vs test fixtures providing it)
|
||||
# This prevents disposing an engine provided by test fixtures when
|
||||
# multiple Client connections are made in the same test
|
||||
engine_was_none = db._engine is None
|
||||
|
||||
# Initialize app (runs migrations, reconciles projects)
|
||||
await initialize_app(container.config)
|
||||
# Initialize app (runs migrations, reconciles projects)
|
||||
await initialize_app(container.config)
|
||||
|
||||
# Log embedding status so it's easy to spot in the logs
|
||||
if config.semantic_search_enabled and db._session_maker is not None:
|
||||
await _log_embedding_status(db._session_maker)
|
||||
|
||||
# Create and start sync coordinator (lifecycle centralized in coordinator)
|
||||
sync_coordinator = container.create_sync_coordinator()
|
||||
await sync_coordinator.start()
|
||||
# Create and start sync coordinator (lifecycle centralized in coordinator)
|
||||
sync_coordinator = container.create_sync_coordinator()
|
||||
await sync_coordinator.start()
|
||||
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
# Shutdown - coordinator handles clean task cancellation
|
||||
with logfire.span(
|
||||
"mcp.lifecycle.shutdown",
|
||||
entrypoint="mcp",
|
||||
mode=container.mode.name.lower(),
|
||||
):
|
||||
logger.debug("Shutting down Basic Memory MCP server")
|
||||
logger.debug("Shutting down Basic Memory MCP server")
|
||||
await sync_coordinator.stop()
|
||||
|
||||
await sync_coordinator.stop()
|
||||
|
||||
# Only shutdown DB if we created it (not if test fixture provided it)
|
||||
if engine_was_none:
|
||||
await db.shutdown_db()
|
||||
logger.debug("Database connections closed")
|
||||
else: # pragma: no cover
|
||||
logger.debug("Skipping DB shutdown - engine provided externally")
|
||||
# Only shutdown DB if we created it (not if test fixture provided it)
|
||||
if engine_was_none:
|
||||
await db.shutdown_db()
|
||||
logger.debug("Database connections closed")
|
||||
else: # pragma: no cover
|
||||
logger.debug("Skipping DB shutdown - engine provided externally")
|
||||
|
||||
|
||||
mcp = FastMCP(
|
||||
|
||||
@@ -24,6 +24,16 @@ from basic_memory.mcp.tools.list_directory import list_directory
|
||||
from basic_memory.mcp.tools.edit_note import edit_note
|
||||
from basic_memory.mcp.tools.move_note import move_note
|
||||
from basic_memory.mcp.tools.workspaces import list_workspaces
|
||||
from basic_memory.mcp.tools.graph_intelligence import (
|
||||
graph_lineage,
|
||||
graph_impact,
|
||||
graph_health,
|
||||
graph_reindex,
|
||||
fcm_simulate,
|
||||
fcm_rank_actions,
|
||||
fcm_import_model,
|
||||
fcm_export_model,
|
||||
)
|
||||
from basic_memory.mcp.tools.project_management import (
|
||||
list_memory_projects,
|
||||
create_memory_project,
|
||||
@@ -44,7 +54,15 @@ __all__ = [
|
||||
"delete_note",
|
||||
"delete_project",
|
||||
"edit_note",
|
||||
"fcm_export_model",
|
||||
"fcm_import_model",
|
||||
"fcm_rank_actions",
|
||||
"fcm_simulate",
|
||||
"fetch",
|
||||
"graph_health",
|
||||
"graph_impact",
|
||||
"graph_lineage",
|
||||
"graph_reindex",
|
||||
"list_directory",
|
||||
"list_memory_projects",
|
||||
"list_workspaces",
|
||||
|
||||
@@ -1,15 +1,13 @@
|
||||
"""Build context tool for Basic Memory MCP server."""
|
||||
|
||||
from typing import Annotated, Optional, Literal
|
||||
from typing import Optional, Literal
|
||||
|
||||
import logfire
|
||||
from loguru import logger
|
||||
from fastmcp import Context
|
||||
from pydantic import AliasChoices, Field
|
||||
|
||||
from basic_memory.config import ConfigManager
|
||||
from basic_memory.mcp.project_context import (
|
||||
detect_project_from_memory_url_prefix,
|
||||
detect_project_from_url_prefix,
|
||||
get_project_client,
|
||||
resolve_project_and_path,
|
||||
)
|
||||
@@ -134,34 +132,14 @@ def _format_context_markdown(graph: GraphContext, project: str) -> str:
|
||||
annotations={"readOnlyHint": True, "openWorldHint": False},
|
||||
)
|
||||
async def build_context(
|
||||
url: Annotated[
|
||||
MemoryUrl,
|
||||
Field(validation_alias=AliasChoices("url", "uri", "memory_url")),
|
||||
],
|
||||
url: MemoryUrl,
|
||||
project: Optional[str] = None,
|
||||
project_id: Optional[str] = None,
|
||||
workspace: Optional[str] = None,
|
||||
depth: str | int | None = 1,
|
||||
timeframe: Annotated[
|
||||
Optional[TimeFrame],
|
||||
Field(
|
||||
default="7d",
|
||||
validation_alias=AliasChoices("timeframe", "since", "time_range", "lookback"),
|
||||
),
|
||||
] = "7d",
|
||||
# `offset` is intentionally NOT aliased: it has different semantics
|
||||
# (item-indexed vs. 1-indexed page-number).
|
||||
page: Annotated[
|
||||
int,
|
||||
Field(default=1, validation_alias=AliasChoices("page", "page_number")),
|
||||
] = 1,
|
||||
page_size: Annotated[
|
||||
int,
|
||||
Field(default=10, validation_alias=AliasChoices("page_size", "limit", "per_page")),
|
||||
] = 10,
|
||||
max_related: Annotated[
|
||||
int,
|
||||
Field(default=10, validation_alias=AliasChoices("max_related", "max_results")),
|
||||
] = 10,
|
||||
timeframe: Optional[TimeFrame] = "7d",
|
||||
page: int = 1,
|
||||
page_size: int = 10,
|
||||
max_related: int = 10,
|
||||
output_format: Literal["json", "text"] = "json",
|
||||
context: Context | None = None,
|
||||
) -> dict | str:
|
||||
@@ -179,9 +157,6 @@ async def build_context(
|
||||
Args:
|
||||
project: Project name to build context from. Optional - server will resolve using hierarchy.
|
||||
If unknown, use list_memory_projects() to discover available projects.
|
||||
project_id: Project external_id (UUID). Prefer this over `project` when known —
|
||||
it routes to the exact project regardless of name collisions across cloud
|
||||
workspaces. Takes precedence over `project`. Get from list_memory_projects().
|
||||
url: memory:// URI pointing to discussion content (e.g. memory://specs/search)
|
||||
depth: How many relation hops to traverse (1-3 recommended for performance)
|
||||
timeframe: How far back to look. Supports natural language like "2 days ago", "last week"
|
||||
@@ -209,17 +184,14 @@ async def build_context(
|
||||
Raises:
|
||||
ToolError: If project doesn't exist or depth parameter is invalid
|
||||
"""
|
||||
# Detect project from memory URL prefix before routing.
|
||||
# project_id routes by external UUID, so it bypasses URL discovery entirely.
|
||||
if project is None and project_id is None:
|
||||
detected = await detect_project_from_memory_url_prefix(
|
||||
url,
|
||||
ConfigManager().config,
|
||||
context=context,
|
||||
)
|
||||
# Detect project from memory URL prefix before routing
|
||||
if project is None:
|
||||
detected = detect_project_from_url_prefix(url, ConfigManager().config)
|
||||
if detected:
|
||||
project = detected
|
||||
|
||||
logger.info(f"Building context from {url} in project {project}")
|
||||
|
||||
# Convert string depth to integer if needed
|
||||
if isinstance(depth, str):
|
||||
try:
|
||||
@@ -231,60 +203,25 @@ async def build_context(
|
||||
|
||||
# URL is already validated and normalized by MemoryUrl type annotation
|
||||
|
||||
with logfire.span(
|
||||
"mcp.tool.build_context",
|
||||
entrypoint="mcp",
|
||||
tool_name="build_context",
|
||||
requested_project=project,
|
||||
requested_project_id=project_id,
|
||||
depth=depth or 1,
|
||||
timeframe=timeframe,
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
max_related=max_related,
|
||||
output_format=output_format,
|
||||
is_memory_url=str(url).startswith("memory://"),
|
||||
):
|
||||
async with get_project_client(project, context=context, project_id=project_id) as (
|
||||
client,
|
||||
active_project,
|
||||
):
|
||||
logger.info(
|
||||
f"MCP tool call tool=build_context project={active_project.name} "
|
||||
f"url={url} depth={depth} timeframe={timeframe} output_format={output_format}"
|
||||
)
|
||||
async with get_project_client(project, workspace, context) as (client, active_project):
|
||||
# Resolve memory:// identifier with project-prefix awareness
|
||||
_, resolved_path, _ = await resolve_project_and_path(client, url, project, context)
|
||||
|
||||
# Resolve memory:// identifier with project-prefix awareness
|
||||
_, resolved_path, _ = await resolve_project_and_path(
|
||||
client,
|
||||
url,
|
||||
active_project.name,
|
||||
context,
|
||||
)
|
||||
# Import here to avoid circular import
|
||||
from basic_memory.mcp.clients import MemoryClient
|
||||
|
||||
# Import here to avoid circular import
|
||||
from basic_memory.mcp.clients import MemoryClient
|
||||
# Use typed MemoryClient for API calls
|
||||
memory_client = MemoryClient(client, active_project.external_id)
|
||||
graph = await memory_client.build_context(
|
||||
resolved_path,
|
||||
depth=depth or 1,
|
||||
timeframe=timeframe,
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
max_related=max_related,
|
||||
)
|
||||
|
||||
# Use typed MemoryClient for API calls
|
||||
memory_client = MemoryClient(client, active_project.external_id)
|
||||
graph = await memory_client.build_context(
|
||||
resolved_path,
|
||||
depth=depth or 1,
|
||||
timeframe=timeframe,
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
max_related=max_related,
|
||||
)
|
||||
if output_format == "text":
|
||||
return _format_context_markdown(graph, active_project.name)
|
||||
|
||||
logger.info(
|
||||
f"MCP tool response: tool=build_context project={active_project.name} "
|
||||
f"uri={graph.metadata.uri or resolved_path} "
|
||||
f"primary_count={graph.metadata.primary_count or 0} "
|
||||
f"related_count={graph.metadata.related_count or 0} "
|
||||
f"output_format={output_format}"
|
||||
)
|
||||
|
||||
if output_format == "text":
|
||||
return _format_context_markdown(graph, active_project.name)
|
||||
|
||||
return graph.model_dump()
|
||||
return graph.model_dump()
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user