From a026412a13511f2d5788ffaec9889d3181b2acf0 Mon Sep 17 00:00:00 2001 From: phernandez Date: Thu, 5 Mar 2026 10:00:49 -0600 Subject: [PATCH] docs: capture graph intelligence phase status and plan Signed-off-by: phernandez --- ...-GRAPH-INTELLIGENCE-IMPLEMENTATION-PLAN.md | 594 +++++++++++++ .../SPEC-LOCAL-GRAPH-INTELLIGENCE-MASTER.md | 782 ++++++++++++++++++ ...L-GRAPH-INTELLIGENCE-TECHNICAL-ADDENDUM.md | 299 +++++++ docs/specs/SPEC-LOCAL-GRAPH-INTELLIGENCE.md | 262 ++++++ 4 files changed, 1937 insertions(+) create mode 100644 docs/specs/SPEC-LOCAL-GRAPH-INTELLIGENCE-IMPLEMENTATION-PLAN.md create mode 100644 docs/specs/SPEC-LOCAL-GRAPH-INTELLIGENCE-MASTER.md create mode 100644 docs/specs/SPEC-LOCAL-GRAPH-INTELLIGENCE-TECHNICAL-ADDENDUM.md create mode 100644 docs/specs/SPEC-LOCAL-GRAPH-INTELLIGENCE.md diff --git a/docs/specs/SPEC-LOCAL-GRAPH-INTELLIGENCE-IMPLEMENTATION-PLAN.md b/docs/specs/SPEC-LOCAL-GRAPH-INTELLIGENCE-IMPLEMENTATION-PLAN.md new file mode 100644 index 00000000..ff4a3c95 --- /dev/null +++ b/docs/specs/SPEC-LOCAL-GRAPH-INTELLIGENCE-IMPLEMENTATION-PLAN.md @@ -0,0 +1,594 @@ +# SPEC-LOCAL-GRAPH-INTELLIGENCE-IMPLEMENTATION-PLAN + +**Status:** Draft (Decision-Complete) +**Date:** 2026-03-05 +**Owner:** Basic Memory Engineering +**Implementation Status (2026-03-05):** Phase 1 contract skeleton implemented in `basic-memory` branch `codex/graph-intelligence-phase1`. +**Related Specs:** +1. `/docs/specs/SPEC-LOCAL-GRAPH-INTELLIGENCE-MASTER.md` +2. `/docs/specs/SPEC-LOCAL-GRAPH-INTELLIGENCE-TECHNICAL-ADDENDUM.md` +3. `/docs/specs/SPEC-LOCAL-GRAPH-INTELLIGENCE.md` + +## Scope and Intent + +This document is the execution handoff for Local+ Graph Intelligence. + +It defines exactly how we will deliver graph and FCM capabilities inside the existing Basic Memory architecture: +1. FastAPI-first business logic. +2. MCP and CLI as thin facades. +3. Local/cloud contract parity. +4. Tight build-test-iterate loop for fast delivery. + +This file is intentionally implementation-oriented and does not duplicate pricing narrative from the master spec. + +## Architecture Alignment (FastAPI-first, MCP/CLI facade) + +Locked architecture alignment for implementation: +1. MCP tools remain thin proxy facades. +2. CLI `bm tool` commands call MCP tools in JSON mode. +3. Core logic lives in FastAPI routers and services. +4. Cloud and local share the same REST contracts. +5. Per-project routing continues through existing project client patterns. + +Execution mapping: +1. API routers define public contracts in `/graph` and `/fcm` domains. +2. Services own traversal, scoring, simulation, and fallback logic. +3. Repositories and index providers own data access and graph index operations. +4. MCP typed clients call REST endpoints and return JSON-first tool output. +5. CLI passthrough executes tool calls and prints machine-friendly JSON. + +## Locked Decisions + +1. SQLite remains operational source for entities, relations, embeddings, and project state. +2. Markdown remains source of truth. +3. Oxigraph/pyoxigraph is the derived graph index for deep traversal. +4. FCM simulation runs in Python service layer; it is not delegated to graph DB query engines. +5. Graph index is rebuildable and disposable; stale index never blocks user workflows. +6. FCM model and scenario artifacts persist in app database. +7. Local+ features are gated by config flags first; entitlement wiring follows later. +8. Graph-first vertical slices ship before deep FCM expansion. +9. Atomic tools ship first; orchestration workflows are deferred. +10. Existing `build_context` and `search_notes` remain backward compatible with no breaking change. + +## Progress Snapshot (as of 2026-03-05) + +Completed in Phase 1: +1. Added `/graph` and `/fcm` v2 routers with all required contract endpoints. +2. Added graph/FCM request and response schemas for all public API contracts. +3. Added service-layer implementations for graph and FCM contract endpoints. +4. Added typed MCP clients for graph and FCM API calls. +5. Added MCP tools: `graph_lineage`, `graph_impact`, `graph_health`, `graph_reindex`, `fcm_simulate`, `fcm_rank_actions`, `fcm_import_model`, `fcm_export_model`. +6. Added CLI passthrough commands under `bm tool ...` for all planned graph/FCM operations. +7. Added scheduler task names for graph lifecycle: `sync_graph_entity`, `sync_graph_project`, `reindex_graph_project`. +8. Added focused tests for API, MCP clients/tools, and CLI graph/FCM passthrough. +9. Added fast-loop `just` targets: `test-graph-intel-api`, `test-graph-intel-mcp`, `test-graph-intel-cli`, `test-graph-intel`. + +Validation completed: +1. `just test-graph-intel` passes. +2. `ruff check` passes on changed files. +3. `pyright` passes on changed files. + +Still pending after Phase 1: +1. SQL-backed traversal/scoring for graph `lineage`, `impact`, and `health`. +2. Oxigraph provider integration and stale-index catch-up flow. +3. Persistent FCM model/scenario state and interop round-trip guarantees. +4. Config-flag and entitlement gating at API/tool boundaries. +5. Performance instrumentation and p95 envelope enforcement. + +## Delivery Phases + +### Phase 1: Contract skeleton + +Status: Completed (2026-03-05) + +Deliverables: +1. Add `/graph` and `/fcm` API routers with request/response schemas. +2. Add typed MCP clients for graph and FCM endpoints. +3. Add MCP tool passthrough commands for all new operations. +4. Add CLI `bm tool` passthrough commands mirroring MCP surface. +5. Add minimal smoke tests for route reachability and schema validation. + +Exit criteria: +1. All endpoints return structured success and error envelopes. +2. MCP/CLI paths execute end-to-end with stubbed service responses. + +### Phase 2: Graph capabilities on SQL-backed logic + +Status: Next active phase + +Deliverables: +1. Implement `lineage`, `impact`, and `health` in service layer using SQL-backed traversal and scoring. +2. Add provenance/evidence linking in graph outputs. +3. Add deterministic graph-health calculations for fixed snapshots. + +Exit criteria: +1. `graph_lineage`, `graph_impact`, and `graph_health` pass contract tests. +2. SQL fallback behavior is explicit and covered by tests. + +### Phase 3: Oxigraph derived index provider + +Status: Planned + +Deliverables: +1. Introduce Oxigraph provider behind graph-query interface. +2. Add lazy catch-up jobs and project-wide reindex operation. +3. Preserve SQL fallback when index is missing or stale. + +Exit criteria: +1. Stale index path serves results via SQL and schedules catch-up. +2. Index rebuild can be triggered and completed without data loss. + +### Phase 4: FCM import/simulate/rank/export + +Status: Planned (contract endpoints complete, full behavior pending) + +Deliverables: +1. Implement CSV-first import/export contracts. +2. Implement deterministic simulation core with convergence metadata. +3. Implement action ranking with evidence references and confidence output. +4. Persist scenario inputs and result artifacts. + +Exit criteria: +1. Research flow scenario passes: import -> simulate -> rank -> export. +2. Interop round-trip preserves node/edge counts and signed weights. + +### Phase 5: Hardening + +Status: Planned + +Deliverables: +1. Performance tuning against published latency envelopes. +2. Local/cloud parity tests for semantics and error behavior. +3. MCP prompt/docs updates for new graph and FCM tools. +4. Operational docs for reindex, fallback, and troubleshooting. + +Exit criteria: +1. `just check` passes before merge. +2. Acceptance criteria in this document are fully met. + +## API and Interface Additions + +### Shared API conventions + +1. All endpoints are project-scoped under `/v2/projects/{project_id}`. +2. Request and response bodies are JSON-first and agent-friendly. +3. Success envelope is endpoint-specific payload with deterministic fields and optional probabilistic fields. +4. Error envelope: +```json +{ + "error": { + "code": "INVALID_ARGUMENT|NOT_FOUND|INDEX_NOT_READY|MODEL_INVALID|RESOURCE_LIMIT_EXCEEDED|INTERNAL_ERROR", + "message": "string", + "details": {} + } +} +``` +5. Latency and scale targets are p95 targets for local default hardware profile. + +### 1) `POST /v2/projects/{project_id}/graph/lineage` + +Purpose: explain decision lineage and supporting evidence paths. + +Request schema: +```json +{ + "start": "string", + "goal": "string|null", + "max_hops": 4, + "relation_filters": ["string"] +} +``` + +Response schema: +```json +{ + "root": {"id": "string", "title": "string", "permalink": "string"}, + "paths": [ + { + "path_id": "string", + "nodes": [{"id": "string", "title": "string"}], + "edges": [{"relation": "string", "direction": "outgoing|incoming"}], + "deterministic_path_score": 0.0, + "confidence": 0.0, + "evidence_refs": ["memory://..."] + } + ], + "generated_at": "RFC3339" +} +``` + +Deterministic fields: `root`, `paths.nodes`, `paths.edges`, `deterministic_path_score`, `generated_at`. +Probabilistic fields: `confidence`. +Latency target: p95 <= 450ms with `max_hops<=4`. +Scale envelope: up to 50k nodes and 300k edges. + +### 2) `POST /v2/projects/{project_id}/graph/impact` + +Purpose: preview impact radius before edits or decisions. + +Request schema: +```json +{ + "target": "string", + "horizon": 2, + "relation_filters": ["string"], + "include_reasons": true +} +``` + +Response schema: +```json +{ + "target": {"id": "string", "title": "string"}, + "affected": [ + { + "id": "string", + "title": "string", + "distance": 1, + "impact_score": 0.0, + "confidence": 0.0, + "reasons": ["string"], + "evidence_refs": ["memory://..."] + } + ], + "summary": {"total_considered": 0, "total_returned": 0} +} +``` + +Deterministic fields: membership, distance, summary counts. +Probabilistic fields: `impact_score`, `confidence`. +Latency target: p95 <= 650ms for `horizon<=3`. +Scale envelope: default 200 results, hard cap 1000 with pagination token. + +### 3) `GET /v2/projects/{project_id}/graph/health` + +Purpose: report deterministic graph quality and actionable issues. + +Query params: +1. `scope` optional directory prefix. +2. `timeframe` optional window like `30d`. + +Response schema: +```json +{ + "metrics": { + "orphan_rate": 0.0, + "stale_central_nodes": 0, + "overloaded_hubs": 0, + "contradiction_candidates": 0 + }, + "issues": [ + { + "issue_type": "orphan|stale_central|overloaded_hub|contradiction_candidate", + "entity_id": "string", + "severity": "low|medium|high", + "reason": "string", + "suggested_action": "string", + "confidence": 0.0 + } + ], + "computed_at": "RFC3339" +} +``` + +Deterministic fields: `metrics`, issue membership for fixed snapshot. +Probabilistic fields: contradiction confidence when applicable. +Latency target: p95 <= 1500ms project-wide, <= 700ms scoped. + +### 4) `POST /v2/projects/{project_id}/graph/reindex` + +Purpose: force project-wide graph index rebuild. + +Request schema: +```json +{ + "mode": "full|incremental", + "reason": "string|null" +} +``` + +Response schema: +```json +{ + "job_id": "string", + "status": "queued|running|completed|failed", + "scheduled_at": "RFC3339" +} +``` + +Deterministic fields: job metadata and status transitions. +Probabilistic fields: none. +Latency target: enqueue response p95 <= 120ms. + +### 5) `POST /v2/projects/{project_id}/fcm/simulate` + +Purpose: run FCM scenario simulation. + +Request schema: +```json +{ + "actions": [{"node_id": "string", "delta": 0.2}], + "scenario": { + "steps": 12, + "activation": "tanh|sigmoid|bounded_linear", + "decay": 0.05 + }, + "clamp_rules": [{"node_id": "string", "min": -1.0, "max": 1.0}] +} +``` + +Response schema: +```json +{ + "baseline": [{"node_id": "string", "state": 0.0}], + "projected": [{"node_id": "string", "state": 0.0}], + "deltas": [{"node_id": "string", "delta": 0.0}], + "stability": {"converged": true, "iterations_used": 0, "residual": 0.0}, + "confidence": 0.0, + "explanations": [{"node_id": "string", "top_influencers": [{"source": "string", "weight": 0.0}]}], + "evidence_refs": ["memory://..."] +} +``` + +Deterministic fields: baseline, projected, deltas, stability for fixed model and params. +Probabilistic fields: confidence. +Latency target: p95 <= 1000ms for <=500 nodes and <=5000 edges. + +### 6) `POST /v2/projects/{project_id}/fcm/rank-actions` + +Purpose: rank candidate interventions by expected outcome and risk. + +Request schema: +```json +{ + "goal": "string", + "constraints": { + "max_negative_impact": 0.25, + "required_tags": ["string"], + "disallowed_nodes": ["string"] + }, + "top_k": 10 +} +``` + +Response schema: +```json +{ + "goal": {"node_id": "string", "label": "string"}, + "recommendations": [ + { + "action_node_id": "string", + "expected_goal_delta": 0.0, + "risk_penalty": 0.0, + "net_score": 0.0, + "confidence": 0.0, + "rationale": ["string"], + "evidence_refs": ["memory://..."] + } + ] +} +``` + +Deterministic fields: candidate set and constraint compliance. +Probabilistic fields: expected delta, penalty, net score, confidence. +Latency target: p95 <= 1500ms for top-10 from <=100 candidates. + +### 7) `POST /v2/projects/{project_id}/fcm/import` + +Purpose: import FCM model from CSV-first contract. + +Request schema: +```json +{ + "source": "string", + "format": "csv_bundle_v1", + "merge_mode": "replace|upsert" +} +``` + +Response schema: +```json +{ + "import_id": "string", + "nodes_loaded": 0, + "edges_loaded": 0, + "warnings": ["string"], + "errors": ["string"] +} +``` + +Deterministic fields: counts and validation diagnostics. +Probabilistic fields: none. +Latency target: p95 <= 2500ms for 10k edges import. + +### 8) `POST /v2/projects/{project_id}/fcm/export` + +Purpose: export FCM model for interoperability. + +Request schema: +```json +{ + "format": "csv_bundle_v1", + "selection": { + "scope": "all|tag|subgraph", + "tag": "string|null", + "seed_nodes": ["string"] + } +} +``` + +Response schema: +```json +{ + "export_id": "string", + "format": "csv_bundle_v1", + "files": [{"name": "nodes.csv", "path": "string"}, {"name": "edges.csv", "path": "string"}], + "node_count": 0, + "edge_count": 0 +} +``` + +Deterministic fields: file names and counts for fixed selection. +Probabilistic fields: none. +Latency target: p95 <= 1800ms for 50k edges export. + +## Data Model and Storage Boundaries + +1. SQLite is mandatory operational source for entities, relations, embeddings, and project metadata. +2. Oxigraph stores derived knowledge graph index only. +3. FCM state persists in app database with scenario artifacts and run history. +4. Graph index is rebuildable and disposable by design. +5. Markdown files remain canonical source of truth. + +Implementation data boundaries: +1. Knowledge graph schema tracks descriptive nodes and typed edges plus provenance. +2. FCM schema tracks signed weighted causal edges and node states. +3. Provenance model requires `evidence_refs`, `confidence`, and `updated_at`. +4. Scenario model stores interventions, constraints, run parameters, and output deltas. +5. Interop schema starts with CSV-first Mental Modeler contract. + +## Background Jobs and Index Lifecycle + +Scheduler tasks to add: +1. `sync_graph_entity` +2. `sync_graph_project` +3. `reindex_graph_project` + +Lifecycle rules: +1. Note writes, edits, moves, and deletes schedule graph-index sync tasks. +2. Scheduling pattern mirrors existing vector sync behavior. +3. On stale or missing graph index, request path serves via SQL fallback and schedules catch-up. +4. Reindex is idempotent and safe to rerun. +5. Index version metadata is tracked per project for staleness checks. + +Operational behaviors: +1. Foreground requests never block on full reindex completion. +2. Background job failures surface in health endpoints with actionable status. +3. Reindex job can run incremental or full mode. +4. Phase 1 note: scheduler task names and reindex enqueue path are implemented; write/edit/move/delete sync hooks still need explicit wiring. + +## MCP and CLI Surface + +New MCP tools: +1. `graph_lineage` +2. `graph_impact` +3. `graph_health` +4. `fcm_simulate` +5. `fcm_rank_actions` +6. `fcm_import_model` +7. `fcm_export_model` + +CLI passthrough additions: +1. `bm tool graph-lineage ...` +2. `bm tool graph-impact ...` +3. `bm tool graph-health ...` +4. `bm tool fcm-simulate ...` +5. `bm tool fcm-rank-actions ...` +6. `bm tool fcm-import-model ...` +7. `bm tool fcm-export-model ...` + +Output conventions: +1. Default output is JSON for MCP and CLI. +2. MCP supports optional `output_format="text"` for human-readable summaries. +3. CLI remains JSON-first to keep agent integration deterministic. + +## Test Strategy (fast loop + gates) + +### Slice-by-slice loop + +For each vertical slice, implement in this order: +1. API contract and schema. +2. Typed MCP client. +3. MCP tool passthrough. +4. CLI passthrough. +5. Focused tests for API/MCP/CLI. + +Fast checks per slice: +1. Targeted `pytest` for changed API, MCP, and CLI modules. +2. `just fast-check`. +3. `just doctor`. +4. `just test-graph-intel` for graph/FCM-only iteration loop. + +Milestone gates: +1. SQLite unit and integration pass first. +2. Selective Postgres parity tests for new graph and FCM contracts. +3. Full `just check` before merge. + +### Required test cases and scenarios + +1. Casual user impact preview before note edit. +2. Decision audit: lineage plus evidence references explain recommendation. +3. Graph health deterministic output for fixed snapshot. +4. Research flow: import model -> simulate -> rank -> export. +5. Sparse and contradictory graph input degrades gracefully. +6. Interop round-trip preserves node/edge counts and signed weights. +7. Local and cloud parity on contract semantics and error model. +8. Stale index fallback path returns valid response and schedules catch-up. + +## Rollout and Feature Flagging + +Rollout controls: +1. Gate graph and FCM endpoints behind config flags first. +2. Add entitlement enforcement after behavior and reliability stabilize. +3. Keep existing tools and endpoints fully backward compatible. + +Suggested flags: +1. `feature_graph_intelligence_enabled` +2. `feature_fcm_enabled` +3. `feature_graph_oxigraph_provider_enabled` +4. `feature_graph_sql_fallback_enabled` + +Rollout sequence: +1. Enable contract skeleton in dev. +2. Enable graph features for internal alpha users. +3. Enable Oxigraph provider with fallback-on by default. +4. Enable FCM import/simulate/rank/export for research alpha users. +5. Promote to Local+ beta when acceptance criteria are met. + +## Risks and Mitigations + +1. Risk: graph query complexity increases p95 latency. +Mitigation: strict query caps, fallback path, and performance budgets per endpoint. +2. Risk: stale index produces confusing outputs. +Mitigation: explicit staleness checks, SQL fallback, and background catch-up scheduling. +3. Risk: FCM recommendations appear opaque. +Mitigation: require evidence references, confidence fields, and deterministic simulation metadata. +4. Risk: local/cloud contract drift. +Mitigation: shared schemas, contract tests, and parity checks in CI gates. +5. Risk: integration surface grows faster than team can validate. +Mitigation: phase gates and vertical-slice completion before opening next phase. + +## Improvement Backlog (Post-Phase 1) + +1. Refactor `bm tool` graph/FCM commands into a dedicated CLI module to reduce `tool.py` size and improve maintainability. +2. Consolidate repeated MCP text-formatting helpers for graph/FCM outputs. +3. Replace deterministic placeholder graph behavior with SQL-backed lineage/impact/health implementations. +4. Add explicit config/entitlement enforcement for graph/FCM endpoints and tools. +5. Add performance telemetry and p95 reporting for graph and FCM routes. +6. Add parity and degradation tests for stale-index fallback and contradictory/sparse inputs. + +## Acceptance Criteria + +1. All required sections in this document are complete with no unresolved decisions. +2. API/interface contracts are implementation-ready with request, response, error, latency, and scale details. +3. Architecture alignment is explicit: FastAPI logic core, MCP/CLI facades, shared local/cloud contracts. +4. Delivery phases define concrete outputs and exit criteria. +5. Test strategy includes tight iteration loop and milestone gates. +6. Required scenario matrix is covered in test plan and mapped to implementation phases. +7. Rollout plan includes feature flags and backward compatibility guarantees. +8. An implementer can execute this plan without additional architecture clarification. + +## Assumptions and Defaults + +1. Config-flag gating first; entitlement wiring later. +2. Graph-first vertical slices before deep FCM expansion. +3. Atomic tools first; orchestration layer deferred. +4. JSON-first contracts for agent usability. +5. No breaking changes to existing `build_context` and `search_notes`. + +## Out of Scope + +1. Implementation details unrelated to graph/FCM delivery phases in this document. +2. Migration execution. +3. Pricing and positioning rewrites. +4. Cloud infrastructure changes in this phase. diff --git a/docs/specs/SPEC-LOCAL-GRAPH-INTELLIGENCE-MASTER.md b/docs/specs/SPEC-LOCAL-GRAPH-INTELLIGENCE-MASTER.md new file mode 100644 index 00000000..663453cf --- /dev/null +++ b/docs/specs/SPEC-LOCAL-GRAPH-INTELLIGENCE-MASTER.md @@ -0,0 +1,782 @@ +# SPEC-LOCAL-GRAPH-INTELLIGENCE-MASTER: Local+ Graph Intelligence Blueprint + +**Status:** Draft (Iteration 2, Decision-Complete) +**Date:** 2026-03-05 +**Owner:** Basic Memory +**Primary Audience:** Internal build team (Product, Engineering, GTM) +**Current Phase (2026-03-05):** Implementation Plan Phase 1 is complete; Phase 2 (SQL-backed graph logic) is the active engineering phase. +**Related Specs:** +1. `/docs/specs/SPEC-LOCAL-GRAPH-INTELLIGENCE.md` +2. `/docs/specs/SPEC-LOCAL-GRAPH-INTELLIGENCE-TECHNICAL-ADDENDUM.md` +3. `/docs/specs/SPEC-LOCAL-GRAPH-INTELLIGENCE-IMPLEMENTATION-PLAN.md` + +Reading guide: +1. Sections 1-5 define the business and product decisions. +2. Sections 6-10 define architecture and interface contracts. +3. Sections 11-14 define pricing, rollout, and decision gates for execution. + +## 1) Executive Thesis + +Basic Memory will ship **Local+ Graph Intelligence** as a premium local capability that upgrades the product from retrieval to decision support. + +Positioning statement: +1. "Keep your local workflow. Add decision intelligence as complexity grows." +2. The product sells safer decisions and explainable recommendations, not graph database mechanics. + +Locked thesis decisions: +1. SQLite will remain the operational core. +2. Markdown will remain source of truth. +3. Graph and FCM indexes will be derived and rebuildable. +4. Oxigraph/pyoxigraph will be the v1 graph index path. +5. FCM simulation will run in a Python service layer. +6. SurrealDB and FalkorDB will not be core dependencies in v1 due license-roadmap mismatch. +7. Product messaging will sell outcomes (safer decisions, explainable recommendations), not database internals. + +## 2) Problem and Opportunity + +Current state after v0.19: +1. Recursive SQL traversal can retrieve connected notes but becomes expensive and noisy after a few hops. +2. Users still do manual synthesis for impact analysis, decision lineage, and contradiction resolution. +3. Researchers need causal reasoning and scenario modeling, not only graph navigation. + +Opportunity: +1. Deliver a premium local tier that materially improves decision quality while keeping data local. +2. Create a bridge from knowledge graph navigation to causal simulation (FCM). +3. Open a research-heavy market segment that values explainability and model interoperability. + +Business opportunity: +1. Add a middle tier between free OSS and cloud subscription. +2. Preserve an upgrade path to hosted collaboration for research teams later. +3. Differentiate Basic Memory for research-grade workflows without forcing cloud adoption. + +## 3) User Segments and Jobs-to-be-Done + +| Segment | Primary Job-to-be-Done | Pain Today | Value Trigger | +|---|---|---|---| +| Casual local builder | Avoid breaking related notes when editing | Hidden dependencies and rework | Impact preview before edits | +| Solo technical founder | Keep architecture and decision context coherent | Context overload and drift | Decision lineage + impact radius | +| Research user | Model and test intervention strategies | No integrated causal simulation with notes | FCM simulation + action ranking | +| Product/research lead | Synthesize evidence quickly across many docs | Fragmented understanding | Path exploration + priority briefs | + +## 4) Product Outcomes (not feature list) + +Local+ Graph Intelligence will optimize for these outcomes: +1. **Change Safety:** users catch downstream impacts before they edit. +2. **Decision Clarity:** users can explain why an answer or recommendation was produced. +3. **Knowledge Health:** users keep larger graphs coherent with less manual audit work. +4. **Research Leverage:** users run scenario-level reasoning tied to explicit evidence. + +Outcome metrics (for 30-day retained Local+ cohorts): +1. Median time-to-understanding for complex topics decreases by at least 35% for active Local+ users. +2. User-reported surprise side effects after note edits decrease by at least 30%. +3. At least 60% of active Local+ users invoke graph intelligence features weekly. +4. At least 40% of research-profile Local+ users invoke one FCM workflow weekly. + +## 5) Feature Set v1/v1.5/v2 + +### v1 (post-v0.19 launch scope) + +Included: +1. Decision Lineage +2. Impact Radius +3. Path Explorer (guided) +4. Graph Health (orphans, stale-central nodes, overloaded hubs) +5. CSV FCM import/export (nodes and edges) +6. FCM simulation for explicit action scenarios +7. FCM action ranking with evidence-linked rationale + +Excluded: +1. Native Mental Modeler project format write support +2. Team governance and shared model policy controls +3. Cloud-only enhancements + +### v1.5 + +Included: +1. Contradiction Watch with reconciliation queue +2. Priority Briefs (graph + FCM leverage summary) +3. Stronger uncertainty propagation in FCM scoring +4. Cloud execution optionality for heavy simulation jobs + +### v2 + +Included: +1. Team-shared model governance +2. Hosted collaboration features for research teams +3. Optional native model translators beyond CSV baseline + +Cut line policy: +1. If a capability cannot meet explainability requirements, it moves to v1.5+. +2. If a capability requires cloud to function, it cannot be marked v1. +3. If a capability cannot meet local performance envelopes, it cannot be promoted into default workflows. + +## 6) Technical Architecture (Two-Graph Model) + +### High-level architecture + +```mermaid +flowchart LR + A[Markdown Files Source of Truth] --> B[Parser + Sync Pipeline] + B --> C[SQLite Operational Store] + B --> D[Derived Knowledge Graph Index Oxigraph] + C --> E[Graph Intelligence Service] + D --> E + E --> F[Lineage Impact Path Health APIs] + C --> G[FCM Service Python] + D --> G + G --> H[Simulation Ranking Interop APIs] +``` + +### Two-graph model + +1. **Knowledge Graph (descriptive):** notes, decisions, concepts, and typed relations. +2. **FCM Graph (causal):** signed weighted influence links between goals, drivers, risks, and interventions. + +### Core architectural decisions + +1. SQLite is authoritative for entities, observations, relations, metadata, embeddings, and project state. +2. Oxigraph is a derived index for multi-hop graph traversal and graph-pattern retrieval. +3. FCM calculations run in Python using explicit model state and deterministic numerical steps. +4. Local mode runs fully offline. +5. Cloud mode can execute the same contracts via adjunct services while Neon remains system of record. + +## 7) Backend Decision and Trade-Offs + +### Final recommendation + +Use this stack for v1: +1. SQLite (existing): primary operational store. +2. Oxigraph/pyoxigraph: derived knowledge graph index. +3. Python FCM service: causal simulation and ranking. + +Decision rationale: +1. This preserves local-first UX while enabling deeper traversal and causal simulation. +2. This avoids restrictive licensing dependencies in the core product path. +3. This keeps a clean cloud portability path where Neon remains the hosted system of record. + +### Trade-off matrix + +| Option | Strengths | Risks | Decision | +|---|---|---|---| +| SQLite + Oxigraph + Python FCM | Local-first, permissive licensing, clear service boundaries, cloud-portable | Requires translation layer for query ergonomics | **Adopt v1** | +| Apache AGE on Postgres | SQL+graph in one engine, good cloud-side graph semantics | Neon extension support uncertainty, weaker local/cloud parity with SQLite local baseline | Defer | +| SurrealDB | Strong integrated multi-model experience | BSL posture conflicts with future hosted/open strategy timing | Reject for v1 core | +| FalkorDB | Graph performance and Redis ecosystem familiarity | SSPL posture conflicts with hosted/open strategy | Reject for v1 core | + +## 8) Public APIs / Interfaces + +All APIs are proposed MCP tool contracts for Local+ mode. + +### Common conventions + +1. `project` parameter is optional and follows existing Basic Memory project routing. +2. Deterministic fields are reproducible with identical inputs and index state. +3. Probabilistic fields are model-derived scores and include confidence metadata. +4. Error model uses structured codes and fail-fast behavior. + +Shared error codes: +1. `INVALID_ARGUMENT` +2. `NOT_FOUND` +3. `MODEL_INVALID` +4. `INDEX_NOT_READY` +5. `RESOURCE_LIMIT_EXCEEDED` +6. `INTERNAL_ERROR` + +--- + +### 8.1 `graph_lineage(start, goal?)` + +**Input schema:** +```json +{ + "start": "string (required, permalink or memory URL)", + "goal": "string (optional, concept or decision target)", + "max_hops": "integer (optional, default 4, range 1-6)", + "relation_filters": ["string"], + "project": "string (optional)" +} +``` + +**Output schema:** +```json +{ + "root": {"id": "string", "title": "string", "permalink": "string"}, + "paths": [ + { + "path_id": "string", + "nodes": [{"id": "string", "title": "string"}], + "edges": [{"relation": "string", "direction": "outgoing|incoming"}], + "deterministic_path_score": 0.0, + "confidence": 0.0, + "evidence_refs": ["memory://..."] + } + ], + "generated_at": "RFC3339" +} +``` + +**Deterministic fields:** root, nodes, edges, deterministic path score, generated timestamp. +**Probabilistic fields:** confidence. + +**Latency target:** p95 <= 450ms for `max_hops<=4`, graph envelope up to 50k nodes / 300k edges. + +**Scale envelope:** +1. Tested local baseline: 50k nodes, 300k edges. +2. Expected degradation: path expansion can exceed latency target when candidate paths > 20k. + +--- + +### 8.2 `graph_impact(target, horizon, relation_filters?)` + +**Input schema:** +```json +{ + "target": "string (required)", + "horizon": "integer (required, range 1-4)", + "relation_filters": ["string"], + "include_reasons": "boolean (default true)", + "project": "string (optional)" +} +``` + +**Output schema:** +```json +{ + "target": {"id": "string", "title": "string"}, + "affected": [ + { + "id": "string", + "title": "string", + "distance": 2, + "impact_score": 0.0, + "confidence": 0.0, + "reasons": ["string"] + } + ], + "summary": {"total_considered": 0, "total_returned": 0} +} +``` + +**Deterministic fields:** membership, distance, summary counts. +**Probabilistic fields:** impact score, confidence. + +**Latency target:** p95 <= 650ms for `horizon<=3` under baseline envelope. + +**Scale envelope:** +1. `affected` default cap: 200 items. +2. Hard cap: 1000 items with pagination token. + +--- + +### 8.3 `graph_health(scope?, timeframe?)` + +**Input schema:** +```json +{ + "scope": "string (optional, directory prefix or project-wide)", + "timeframe": "string (optional, e.g. 30d, 90d)", + "project": "string (optional)" +} +``` + +**Output schema:** +```json +{ + "metrics": { + "orphan_rate": 0.0, + "stale_central_nodes": 0, + "overloaded_hubs": 0, + "contradiction_candidates": 0 + }, + "issues": [ + { + "issue_type": "orphan|stale_central|overloaded_hub|contradiction_candidate", + "entity_id": "string", + "severity": "low|medium|high", + "reason": "string", + "suggested_action": "string" + } + ], + "computed_at": "RFC3339" +} +``` + +**Deterministic fields:** metrics and issue list membership for a fixed graph snapshot. +**Probabilistic fields:** contradiction candidate confidence when present. + +**Latency target:** p95 <= 1500ms project-wide; <= 700ms for scoped directory mode. + +**Scale envelope:** project-wide scans tested to 50k nodes. + +--- + +### 8.4 `fcm_simulate(actions, scenario?, clamp_rules?)` + +**Input schema:** +```json +{ + "actions": [ + {"node_id": "string", "delta": 0.2} + ], + "scenario": { + "steps": 12, + "activation": "tanh|sigmoid|bounded_linear", + "decay": 0.05 + }, + "clamp_rules": [ + {"node_id": "string", "min": -1.0, "max": 1.0} + ], + "project": "string (optional)" +} +``` + +**Output schema:** +```json +{ + "baseline": [{"node_id": "string", "state": 0.12}], + "projected": [{"node_id": "string", "state": 0.43}], + "deltas": [{"node_id": "string", "delta": 0.31}], + "stability": { + "converged": true, + "iterations_used": 9, + "residual": 0.002 + }, + "confidence": 0.0, + "explanations": [ + {"node_id": "string", "top_influencers": [{"source": "string", "weight": 0.7}]} + ] +} +``` + +**Deterministic fields:** baseline, projected, deltas, convergence metadata for fixed model and parameters. +**Probabilistic fields:** confidence (derived from edge confidence and evidence coverage). + +**Latency target:** p95 <= 1000ms for up to 500 nodes / 5000 edges and <=12 steps. + +**Scale envelope:** +1. Soft limit: 2000 nodes / 20000 edges. +2. Over soft limit: return `RESOURCE_LIMIT_EXCEEDED` with remediation guidance. + +--- + +### 8.5 `fcm_rank_actions(goal, constraints?, top_k?)` + +**Input schema:** +```json +{ + "goal": "string (required node_id)", + "constraints": { + "max_negative_impact": 0.25, + "required_tags": ["string"], + "disallowed_nodes": ["string"] + }, + "top_k": "integer (default 10, range 1-25)", + "project": "string (optional)" +} +``` + +**Output schema:** +```json +{ + "goal": {"node_id": "string", "label": "string"}, + "recommendations": [ + { + "action_node_id": "string", + "expected_goal_delta": 0.0, + "risk_penalty": 0.0, + "net_score": 0.0, + "confidence": 0.0, + "rationale": ["string"], + "evidence_refs": ["memory://..."] + } + ] +} +``` + +**Deterministic fields:** candidate action set, constraints compliance. +**Probabilistic fields:** expected goal delta, risk penalty, net score, confidence. + +**Latency target:** p95 <= 1500ms for top 10 from up to 100 candidate actions. + +**Scale envelope:** +1. Candidate actions hard cap: 1000. +2. For larger sets, require pre-filtering via tags/scope. + +--- + +### 8.6 `fcm_import_model(source, format)` + +**Input schema:** +```json +{ + "source": "string (required path or URI)", + "format": "csv_bundle_v1 (required)", + "merge_mode": "replace|upsert (default upsert)", + "project": "string (optional)" +} +``` + +**Output schema:** +```json +{ + "import_id": "string", + "nodes_loaded": 0, + "edges_loaded": 0, + "warnings": ["string"], + "errors": ["string"] +} +``` + +**Deterministic fields:** load counts and validation results. +**Probabilistic fields:** none. + +**Latency target:** p95 <= 2500ms for 10k edges CSV bundle. + +**Scale envelope:** +1. Maximum CSV rows per import: 250k. +2. Above limit returns `RESOURCE_LIMIT_EXCEEDED`. + +--- + +### 8.7 `fcm_export_model(format, selection?)` + +**Input schema:** +```json +{ + "format": "csv_bundle_v1 (required)", + "selection": { + "scope": "all|tag|subgraph", + "tag": "string (optional)", + "seed_nodes": ["string"] + }, + "project": "string (optional)" +} +``` + +**Output schema:** +```json +{ + "export_id": "string", + "format": "csv_bundle_v1", + "files": [ + {"name": "nodes.csv", "path": "string"}, + {"name": "edges.csv", "path": "string"} + ], + "node_count": 0, + "edge_count": 0 +} +``` + +**Deterministic fields:** file set and row counts for fixed selection. +**Probabilistic fields:** none. + +**Latency target:** p95 <= 1800ms for 50k edges export. + +**Scale envelope:** +1. Max export rows: 500k total. +2. Pagination or scoped export required above cap. + +## 9) Data Model and Storage Boundaries + +### 9.1 Knowledge graph schema (descriptive) + +`KnowledgeNode`: +1. `id: str` +2. `kind: note|decision|spec|concept|person|project` +3. `title: str` +4. `permalink: str` +5. `tags: list[str]` +6. `updated_at: datetime` + +`KnowledgeEdge`: +1. `id: str` +2. `src_id: str` +3. `dst_id: str` +4. `relation: str` +5. `directionality: directed|bidirectional` +6. `evidence_refs: list[str]` +7. `confidence: float [0,1]` +8. `updated_at: datetime` + +### 9.2 FCM schema (causal signed weighted) + +`FCMNode`: +1. `id: str` +2. `label: str` +3. `node_type: goal|driver|risk|intervention|context` +4. `state: float [-1,1]` +5. `clamp_min: float` +6. `clamp_max: float` +7. `metadata: map` + +`FCMEdge`: +1. `id: str` +2. `source_id: str` +3. `target_id: str` +4. `weight: float [-1,1]` +5. `confidence: float [0,1]` +6. `time_decay: float [0,1]` +7. `evidence_refs: list[str]` +8. `updated_at: datetime` + +### 9.3 Provenance model + +`ProvenanceRecord`: +1. `entity_id: str` +2. `evidence_refs: list[str]` +3. `confidence: float [0,1]` +4. `updated_at: datetime` +5. `source_type: extracted|user_authored|imported` + +### 9.4 Scenario model + +`Scenario`: +1. `id: str` +2. `name: str` +3. `interventions: list[{node_id, delta}]` +4. `constraints: list[{node_id, min, max}]` +5. `steps: int` +6. `activation: tanh|sigmoid|bounded_linear` +7. `created_at: datetime` +8. `created_by: str` + +`ScenarioResult`: +1. `scenario_id: str` +2. `converged: bool` +3. `iterations_used: int` +4. `residual: float` +5. `goal_deltas: list[{node_id, delta}]` +6. `confidence: float [0,1]` + +### 9.5 Storage boundaries + +| Layer | System of Record | Purpose | Rebuildable | +|---|---|---|---| +| Markdown files | File system | Canonical knowledge content | No | +| SQLite entities/relations/embeddings | SQLite | Operational queries and project state | Yes (from markdown + embedding pipeline) | +| Knowledge graph triples | Oxigraph | Fast graph traversal and pattern queries | Yes | +| FCM model and snapshots | SQLite + optional artifacts | Causal model state and scenario history | Yes (from imports and authored model definitions) | + +## 10) Mental Modeler Interoperability + +### v1 interoperability contract + +Format: `csv_bundle_v1` +1. `nodes.csv` +2. `edges.csv` +3. Optional `scenarios.csv` + +`nodes.csv` required columns: +1. `node_id` +2. `label` +3. `node_type` +4. `state` +5. `clamp_min` +6. `clamp_max` + +`edges.csv` required columns: +1. `edge_id` +2. `source_id` +3. `target_id` +4. `weight` +5. `confidence` +6. `evidence_refs` (semicolon-delimited) + +### Import rules + +1. Missing required columns fail with `MODEL_INVALID`. +2. Unknown node types fail fast. +3. Weight and confidence ranges are strictly validated. +4. Import returns warnings for dangling evidence references. + +### Export rules + +1. Preserve stable IDs for round-trip compatibility. +2. Preserve signed weights exactly. +3. Preserve confidence values exactly. +4. Non-portable metadata is emitted to `metadata.json` sidecar when present. + +### Native file translators + +1. Native project-format translation is deferred to v2. +2. CSV remains the guaranteed compatibility baseline in v1 and v1.5. + +## 11) Pricing and Packaging + +### Tier structure + +| Tier | Price Monthly | Price Annual | Beta Price (25% off) | Target Persona | Core Value | +|---|---:|---:|---:|---|---| +| OSS Local | $0 | $0 | $0 | Casual local users | Retrieval and memory basics | +| Local+ Graph Intelligence | $9 | $90 | $6.75 monthly / $67.50 annual | Founders, consultants, researchers | Safer changes + explainable graph + FCM simulation | +| Cloud Pro (current anchor) | $19 | $190 | $14.25 monthly / $142.50 annual | Users who need hosted sync and cloud workflows | Managed cloud + sync + collaboration path | + +Pricing principles: +1. Local+ is intentionally priced between free OSS and cloud to capture users who need deeper intelligence but not hosted sync. +2. Cloud Pro remains the hosted convenience anchor and future collaboration path. +3. Local+ must stand on standalone local value and cannot depend on cloud features. + +### Feature gate mapping + +| Capability | OSS Local | Local+ | Cloud Pro | +|---|---|---|---| +| Search and basic context tools | Yes | Yes | Yes | +| Decision Lineage | No | Yes | Yes | +| Impact Radius | No | Yes | Yes | +| Graph Health | No | Yes | Yes | +| FCM simulate + rank | No | Yes | Yes | +| CSV model import/export | No | Yes | Yes | +| Hosted collaboration controls | No | No | Future add-on | + +### Packaging decisions + +1. Local+ remains fully local-capable and does not require cloud auth to run. +2. Cloud Pro remains the hosted convenience and collaboration anchor. +3. Future hosted research add-on will layer on Cloud Pro after v2 readiness. + +## 12) Rollout Strategy + +### 12.1 Document production iterations (locked) + +Iteration 1 (draft complete): +1. Complete all 15 sections in one pass. +2. Include v1/v1.5/v2 cut lines. +3. Include pricing and scenario definitions. +4. Ensure no placeholders. + +Iteration 2 (hardening and decision lock): +1. Resolve cross-section contradictions. +2. Convert uncertain language to locked decisions. +3. Add measurable acceptance criteria and risk owners. +4. Finalize execution-ready API contracts. + +### 12.2 Product rollout phases + +Phase A: Foundation release (v1) +1. Graph lineage, impact, health. +2. CSV model import/export. +3. FCM simulation and ranking. +4. Advanced mode UX gating for research-grade controls. + +Phase B: Quality and confidence (v1.5) +1. Contradiction Watch. +2. Priority Briefs. +3. Improved uncertainty propagation. +4. Confidence calibration pass using real-world model feedback. + +Phase C: Team expansion (v2) +1. Hosted team governance. +2. Shared model controls. +3. Extended translator support. + +### 12.3 Go/No-Go release gates + +Gate to ship v1 default workflows: +1. p95 latency targets are met within the declared scale envelopes. +2. Scenario round-trip fidelity tests pass for CSV import/export. +3. Every recommendation and simulation path exposes evidence references and confidence. + +Gate to ship v1.5: +1. Contradiction Watch precision is acceptable for default-on use. +2. Confidence calibration reduces false-confidence reports in user testing. + +Gate to ship v2 team features: +1. Clear willingness-to-pay signal from team and research buyers. +2. Cloud execution path preserves local-cloud semantic parity for core contracts. + +## 13) Risks, Counterarguments, and Mitigations + +| Risk | Counterargument | Severity | Likelihood | Mitigation | Owner | +|---|---|---|---|---|---| +| "This is just better search" | Positioning can collapse into technical jargon | High | Medium | Lead with decision safety and explainability outcomes in product copy and onboarding | Product Lead | +| FCM feels opaque or invented | Users distrust black-box scoring | High | Medium | Require evidence refs and confidence disclosure on every recommendation | Applied AI Lead | +| Local performance regressions | Multi-hop and simulation can feel slow on laptops | Medium | Medium | Enforce envelopes, caps, and fail-fast limit errors with guidance | Engineering Lead | +| Research features overwhelm casual users | UX complexity can reduce adoption | Medium | High | Default to guided flows and hide advanced controls behind explicit advanced mode | Design Lead | +| License/roadmap conflict if backend changes | Later swap to restrictive engines creates GTM risk | High | Low | Lock permissive v1 stack and require leadership sign-off for any license-restricted dependency | Product + Legal | +| Interop mismatch with external tools | Round-trip drift harms trust with researchers | Medium | Medium | Validate node and edge parity in import/export tests and version interop schema | Integrations Lead | +| Pricing confusion between Local+ and Cloud Pro | Buyers may not understand which tier fits | Medium | Medium | Publish explicit tier comparison focused on local intelligence vs hosted collaboration | GTM Lead | + +## 14) Acceptance Criteria + +### 14.1 Document acceptance criteria + +1. Product, technical, and pricing decisions are explicit and unambiguous. +2. API contracts include input/output schemas, error models, deterministic versus probabilistic fields, and performance envelopes. +3. v1/v1.5/v2 cut lines are explicit and consistent. +4. Risk register includes severity, likelihood, owner, and mitigation. +5. Document can be handed to implementation without additional architecture decisions. +6. Leadership can use this document directly for pricing and positioning decisions. + +### 14.2 Product acceptance criteria for v1 delivery + +1. `graph_lineage`, `graph_impact`, `graph_health`, `fcm_simulate`, `fcm_rank_actions`, `fcm_import_model`, and `fcm_export_model` are available as Local+ contracts. +2. Local mode executes all v1 contracts without cloud dependency. +3. API p95 latency targets are met within defined scale envelopes. +4. Every ranked or simulated output includes evidence-linked rationale and confidence. +5. CSV round-trip preserves node count, edge count, and signed weights exactly. + +### 14.3 Scenario test matrix (required) + +1. **Casual local user impact check** +Expected pass: +`graph_impact` returns ranked affected notes with reasons before a note edit. + +2. **Research workflow simulation** +Expected pass: +User imports a model bundle, runs `fcm_simulate`, and receives converged deltas and rationale. + +3. **Decision audit traceability** +Expected pass: +`graph_lineage` returns path and evidence references that explain recommendation origin. + +4. **Cloud and local parity** +Expected pass: +Identical query inputs return semantically equivalent outputs in local and cloud modes with local fallback behavior when cloud is unavailable. + +5. **Sparse or contradictory graph behavior** +Expected pass: +System degrades gracefully with explicit uncertainty and does not fabricate high-confidence recommendations. + +6. **Interop round-trip fidelity** +Expected pass: +`fcm_export_model` then `fcm_import_model` preserves node and edge counts and signed weights without mutation. + +## 15) Appendix (license notes, terminology, examples) + +### 15.1 License notes (verified 2026-03-05) + +1. SurrealDB core licensing is published under BSL 1.1 with DBaaS-related restrictions in its conversion window. +2. FalkorDB is published under SSPLv1. +3. pyoxigraph is dual-licensed Apache-2.0 or MIT. +4. Neon extension catalog currently does not list Apache AGE as a supported extension. + +These notes support the v1 dependency decisions in this document. + +### 15.2 Terminology + +1. **Knowledge graph:** descriptive relation graph derived from markdown knowledge. +2. **FCM:** fuzzy cognitive model with signed weighted causal edges. +3. **Deterministic field:** reproducible output field from fixed input and fixed model/index snapshot. +4. **Probabilistic field:** score influenced by confidence weights and model uncertainty. + +### 15.3 Assumptions and defaults + +1. Markdown remains source of truth. +2. SQLite remains mandatory baseline. +3. Graph and FCM capabilities are premium Local+ features, not OSS defaults. +4. Research-heavy features are advanced mode, while mainstream UX stays guided. +5. Mental Modeler interoperability starts with CSV contract first; native translator support is deferred. + +### 15.4 Out of scope for this document + +1. Implementation code changes. +2. Database migrations. +3. Cloud infrastructure edits. +4. Full instrumentation pilot plan as the primary artifact. + +### 15.5 External references + +1. [SurrealDB licensing](https://surrealdb.com/license) +2. [FalkorDB licensing](https://docs.falkordb.com/References/license.html) +3. [pyoxigraph package and license](https://pypi.org/project/pyoxigraph/) +4. [Neon Postgres extension catalog](https://neon.com/docs/extensions/pg-extensions) diff --git a/docs/specs/SPEC-LOCAL-GRAPH-INTELLIGENCE-TECHNICAL-ADDENDUM.md b/docs/specs/SPEC-LOCAL-GRAPH-INTELLIGENCE-TECHNICAL-ADDENDUM.md new file mode 100644 index 00000000..34d29be0 --- /dev/null +++ b/docs/specs/SPEC-LOCAL-GRAPH-INTELLIGENCE-TECHNICAL-ADDENDUM.md @@ -0,0 +1,299 @@ +# SPEC-LOCAL-GRAPH-INTELLIGENCE: Technical Addendum (Graph + FCM) + +**Status:** Draft +**Date:** 2026-03-05 +**Owner:** Basic Memory +**Current Phase (2026-03-05):** Contract skeleton implementation is complete; next active phase is SQL-backed graph capabilities. + +Related product spec: +`/docs/specs/SPEC-LOCAL-GRAPH-INTELLIGENCE.md` + +Related execution spec: +`/docs/specs/SPEC-LOCAL-GRAPH-INTELLIGENCE-IMPLEMENTATION-PLAN.md` + +## Why This Addendum Exists + +The product spec defines user value. This addendum defines the technical shape that can deliver that value without +breaking local-first principles. + +This addendum also introduces a second graph layer: + +1. Knowledge graph for relationships between notes, entities, and decisions. +2. Fuzzy Cognitive Model (FCM) graph for weighted causal reasoning over actions and outcomes. + +Both are derived from markdown and optional user-provided models. + +## Strategic Reality Check + +This is a strong idea if we stage it correctly. + +It is not a pipe dream if we avoid one trap: building a big "graph platform" before proving users repeatedly use +decision simulation workflows. + +The correct strategy is: + +1. Launch high-precision graph insights first. +2. Add FCM scoring where it changes user behavior (not as a novelty dashboard). +3. Expand to hosted/team workflows only after local usage proves repeat value. + +## Constraints and Design Principles + +1. SQLite remains the operational source for entities, observations, relations, and embeddings. +2. Markdown remains source of truth. +3. Graph indexes are derived, rebuildable, and disposable. +4. Premium local mode must run fully offline. +5. Cloud deployment should support both single-tenant and SaaS later. +6. Avoid licenses that constrain hosted/open-source strategy. + +## Backend Recommendation + +### Primary Recommendation + +Use a dual-store architecture: + +1. SQLite (existing): operational data, metadata filters, embeddings, and most retrieval. +2. Oxigraph/pyoxigraph (new): derived graph index for graph traversal and graph-pattern queries. +3. Python simulation layer (new): FCM state propagation, scenario runs, and decision scoring. + +Why this is the best fit: + +1. Permissive licensing profile. +2. Works locally with low footprint. +3. Cloud-compatible as sidecar service while keeping Neon Postgres as core cloud store. +4. Clear boundary between graph query and numeric simulation concerns. + +### Candidate Trade-Offs + +#### Oxigraph/pyoxigraph + +Pros: + +1. Lightweight local embedding. +2. Good fit for derived-index strategy. +3. Strong path for standards-based graph representation. + +Cons: + +1. SPARQL fluency is less common than SQL/Cypher. +2. Requires a translation layer so product features are not query-language-coupled. + +#### Apache AGE (Postgres extension) + +Pros: + +1. SQL + graph in one engine. +2. Attractive for cloud-side graph operations. + +Cons: + +1. Neon support is uncertain for this extension. +2. Local/cloud parity is harder if local uses SQLite. + +#### SurrealDB / FalkorDB + +Pros: + +1. Strong graph-oriented developer experience. + +Cons: + +1. License posture is misaligned with a future hosted/open-source roadmap unless commercial terms are accepted. + +Decision: +Do not make these core dependencies for v1 of Local+ Graph Intelligence. + +## Two-Graph Model + +### A) Knowledge Graph (Descriptive) + +Node examples: + +1. Note +2. Decision +3. Spec +4. Person +5. Project +6. Concept + +Edge examples: + +1. `depends_on` +2. `informed_by` +3. `contradicts` +4. `supports` +5. `implements` +6. `derived_from` + +Purpose: +Power navigation, lineage, path explanation, impact radius, and health checks. + +### B) FCM Graph (Causal, Signed, Weighted) + +Node examples: + +1. Goal: "Reduce regressions" +2. Driver: "Test coverage" +3. Risk: "Scope creep" +4. Intervention: "Add review gate" +5. Context variable: "Team bandwidth" + +Edge attributes: + +1. `weight` in [-1.0, 1.0] +2. `confidence` in [0.0, 1.0] +3. `evidence_refs` (links to notes/specs) +4. `time_decay` (optional) + +Purpose: +Power scenario simulation and action ranking, not generic retrieval. + +## Premium Feature Mapping to Architecture + +### Decision Lineage + +Backed by: + +1. Knowledge graph path queries. +2. Evidence references stored on edges. + +### Impact Radius + +Backed by: + +1. Multi-hop neighborhood expansion with relation-type weights. +2. Risk ranking using centrality + recency + confidence. + +### Contradiction Watch + +Backed by: + +1. Candidate contradiction edges. +2. Confidence-scored reconciliation queue. + +### Priority Briefs + +Backed by: + +1. Health metrics (orphan rate, stale-central nodes, unresolved contradictions). +2. Optional FCM "top leverage actions" summary. + +### New Premium Feature: Action Simulator + +Backed by: + +1. FCM scenario runs over selected action nodes. +2. Ranked interventions with expected positive/negative downstream effects. +3. Explicit rationale graph for every recommendation. + +## Mental Modeler Interop Plan + +Goal: +Make Basic Memory the AI-enabled operating layer around existing researcher workflows, not a replacement for their tools. + +Interoperability phases: + +1. Import/export edge lists and node tables via CSV as the baseline interchange. +2. Preserve concept IDs and metadata so round-trips remain stable. +3. Add translator support for native model files if/when schema contracts are validated with partner data. + +Validation requirement: + +1. Round-trip tests must preserve node count, edge count, and signed weights. +2. Confidence/evidence metadata may be Basic Memory extensions and should degrade gracefully when exported. + +## Suggested Tool/API Surface (Product-Facing) + +1. `graph_lineage(start, goal?)` + Returns explainable evidence paths. +2. `graph_impact(target, horizon=2..4)` + Returns ranked affected nodes with reasons. +3. `graph_health()` + Returns actionable graph quality issues. +4. `fcm_simulate(actions, scenario?)` + Returns projected effects and uncertainty. +5. `fcm_rank_actions(goal, constraints?)` + Returns top candidate actions with trade-offs. +6. `fcm_import_model(source)` / `fcm_export_model(format)` + Handles interop with external cognitive mapping workflows. + +## Local and Cloud Deployment Shape + +### Local (Primary) + +1. SQLite + local embeddings. +2. Oxigraph as local sidecar/index library. +3. FCM simulation in process. + +### Cloud (Future-Compatible) + +1. Neon Postgres remains system of record in hosted mode. +2. Graph index service runs per tenant or shared multi-tenant with strict tenancy boundaries. +3. FCM simulation service can run stateless workers reading graph snapshots. + +Principle: +Do not require cloud to run premium local features. + +## Rollout Plan With Go/No-Go Gates + +### Phase 0: Proof of Utility (4-6 weeks) + +Deliver: + +1. Decision Lineage +2. Impact Radius +3. CSV FCM import + `fcm_simulate` prototype + +Gate to continue: + +1. Repeated weekly usage by pilot users. +2. Users report changed decisions, not just curiosity clicks. + +### Phase 1: Productized Local+ Beta + +Deliver: + +1. Graph health workflow +2. Contradiction Watch +3. Action ranking with explicit rationale + +Gate to continue: + +1. Retention of graph features after first month. +2. Measured reduction in "surprise side effects" after edits. + +### Phase 2: Hosted Expansion + +Deliver: + +1. Optional cloud execution for heavy simulations. +2. Team-shared model governance. + +Gate to continue: + +1. Clear willingness to pay for hosted collaboration. + +## Risks and Mitigations + +Risk: FCM outputs feel "made up." +Mitigation: Require evidence links and confidence scoring in every recommendation. + +Risk: Research-heavy feature alienates casual users. +Mitigation: Keep FCM features in an advanced mode; default to concise guidance workflows. + +Risk: Overengineering early graph stack. +Mitigation: Keep derived-index architecture and strict phase gates tied to behavior change. + +Risk: Interop friction with external tooling. +Mitigation: Start with transparent CSV contract and strict round-trip validation. + +## Candid Recommendation + +Pursue this. It is a high-upside differentiation path for Local+ if executed with staged validation. + +The key is to sell outcomes: + +1. "Safer decisions" +2. "Explainable recommendations" +3. "Faster synthesis for complex research" + +Avoid selling "graph DB" as the product. That is implementation detail. diff --git a/docs/specs/SPEC-LOCAL-GRAPH-INTELLIGENCE.md b/docs/specs/SPEC-LOCAL-GRAPH-INTELLIGENCE.md new file mode 100644 index 00000000..3880d316 --- /dev/null +++ b/docs/specs/SPEC-LOCAL-GRAPH-INTELLIGENCE.md @@ -0,0 +1,262 @@ +# SPEC-LOCAL-GRAPH-INTELLIGENCE: Premium Local Graph Intelligence + +**Status:** Draft +**Date:** 2026-03-05 +**Owner:** Basic Memory +**Current Phase (2026-03-05):** Phase 1 contract foundation shipped; engineering is now executing SQL-backed Phase 2 graph logic. + +Companion technical addendum: +`/docs/specs/SPEC-LOCAL-GRAPH-INTELLIGENCE-TECHNICAL-ADDENDUM.md` + +## Summary + +Add a premium local feature that turns Basic Memory from "search and recall" into "explain and guide." + +The value is not a new database. The value is better decisions for local users: + +1. Understand why something matters. +2. See what will be affected before making a change. +3. Detect weak spots in the knowledge base early. +4. Navigate complex knowledge intentionally instead of loading everything. + +This feature is additive. Existing local workflows remain intact. + +## Positioning + +Core message: +"Your notes do more than store knowledge. They reveal consequences, lineage, and blind spots." + +Local user promise: + +1. Keep files local. +2. Keep markdown as source of truth. +3. Get advanced graph intelligence as an opt-in premium capability. + +## Problem + +Today, deep graph navigation is possible but often expensive in context size and hard to steer for complex questions. +Users can find information, but they still do manual synthesis to answer: + +1. What changed because of this note? +2. Why did we decide this? +3. What might break if I update this? +4. Which parts of the graph are stale, isolated, or contradictory? + +The cost is time, cognitive load, and missed risk. + +## Goals + +1. Provide clear, explainable graph insights that users can act on. +2. Make deep navigation feel guided, not overwhelming. +3. Help users prevent mistakes before they happen. +4. Create premium local value that is easy to understand and justify. +5. Keep feature behavior transparent and trustworthy. + +## Non-Goals + +1. Replacing SQLite as the primary operational store. +2. Changing markdown as source of truth. +3. Forcing users to learn graph query languages. +4. Building a cloud-only feature set. +5. Turning Basic Memory into an enterprise BI product. + +## Product Frame: From Retrieval to Reasoning + +The feature should be framed as a shift in user outcome: + +1. Retrieval: "Find me the note." +2. Reasoning: "Show me the path, impact, and confidence around this note." + +This is the main narrative upgrade for premium local users. + +## Premium Value Pillars + +### 1) Decision Confidence + +Users can see decision lineage: + +1. What evidence supported a decision. +2. Which notes/specs informed it. +3. How that decision evolved over time. + +### 2) Change Safety + +Users can run impact-aware workflows: + +1. Estimate blast radius before editing. +2. Surface downstream dependencies. +3. Prioritize what to review first. + +### 3) Knowledge Quality + +Users can maintain graph health: + +1. Detect orphaned notes. +2. Detect overloaded hub notes. +3. Detect stale but high-centrality notes. +4. Detect likely contradictions. + +### 4) Guided Navigation + +Users can explore deeper relationships without context explosion: + +1. Follow promising branches. +2. Stop when confidence is sufficient. +3. Avoid "load everything and hope." + +## Feature Catalog (Value-First) + +### A. Decision Lineage + +What users get: + +1. A clear "why chain" for important conclusions. +2. Traceable connections to supporting notes. +3. Better handoffs and historical understanding. + +### B. Impact Radius + +What users get: + +1. A ranked list of likely affected notes before edits. +2. Safer refactors for docs, plans, and architecture. +3. Reduced accidental drift and inconsistency. + +### C. Knowledge Health Dashboard + +What users get: + +1. Weekly health signals for the graph. +2. Actionable cleanup targets. +3. Better long-term memory quality with less manual auditing. + +### D. Path Explorer + +What users get: + +1. "Show me how A connects to B" style explanations. +2. Multiple candidate paths with confidence cues. +3. Better discovery across large note collections. + +### E. Contradiction Watch + +What users get: + +1. Early warnings for conflicting statements. +2. Suggested reconciliation workflow. +3. Higher trust in the knowledge base. + +### F. Priority Briefs + +What users get: + +1. Periodic "what matters now" graph summaries. +2. Focused recommendations, not noisy activity dumps. +3. Better focus for solo builders and small teams. + +## User Personas and Why They Pay + +### Solo Technical Founder + +Pain: +Cannot hold full architecture and decision history in working memory. + +Premium value: +Impact Radius + Decision Lineage prevent rework and regressions. + +### Product/Research Lead + +Pain: +Knowledge is fragmented across specs, notes, and decisions. + +Premium value: +Path Explorer + Priority Briefs compress synthesis time. + +### Consultant/Fractional Operator + +Pain: +Frequent context switching across domains and clients. + +Premium value: +Knowledge Health + Decision Lineage speed onboarding and reporting. + +## Packaging Direction + +Suggested packaging: + +1. OSS Local: existing search + context tools. +2. Local+ Graph Intelligence: advanced graph insight features listed above. +3. Future Team Add-On: shared policies, shared graph health views, shared lineage views. + +Core upsell line: +"Keep your local workflow. Add graph intelligence when complexity grows." + +## Experience Principles + +1. Explainability first. +Every advanced result should show "why this was suggested." + +2. Actionability over novelty. +Insights should lead to concrete next steps, not abstract charts. + +3. Progressive disclosure. +Start with concise summaries, expand on demand. + +4. Deterministic where possible. +Users should trust repeated runs of the same workflow. + +5. Respect local-first expectations. +No surprise cloud dependency in premium local mode. + +## Success Criteria (Product) + +1. Users can describe the benefit in one sentence: +"It shows me what matters and what breaks before I change things." +2. Premium users report lower time-to-understanding for complex topics. +3. Premium users report fewer "surprise side effects" after edits. +4. Premium users keep larger knowledge graphs healthy with less manual effort. +5. Feature adoption is driven by outcomes, not by curiosity-only usage. + +## Risks and Mitigations + +Risk: Feature sounds like "just better search." +Mitigation: Lead messaging with decision confidence and change safety, not traversal depth. + +Risk: Feature feels too advanced for normal users. +Mitigation: Package as guided insights and reports, not as a query language. + +Risk: Insight quality feels noisy. +Mitigation: Focus launch scope on high-precision insight types and transparent rationale. + +Risk: Value is hard to prove. +Mitigation: Track user-facing outcomes (time saved, risk avoided, cleanup completed). + +## Rollout Narrative + +Phase 1: "Safer Changes" + +1. Impact Radius +2. Decision Lineage + +Phase 2: "Health and Clarity" + +1. Knowledge Health Dashboard +2. Contradiction Watch + +Phase 3: "Strategic Navigation" + +1. Path Explorer +2. Priority Briefs + +## One-Line Positioning Options + +1. "Local notes, strategic intelligence." +2. "Know what changed, why it matters, and what it affects." +3. "From note-taking to decision support." + +## Open Questions + +1. Which two features best define the paid tier at launch? +2. Which insight types should be guaranteed deterministic in v1? +3. Should Priority Briefs be bundled or separate as an add-on? +4. What is the simplest in-product education flow for first-time premium users?