mirror of
https://github.com/basicmachines-co/basic-memory
synced 2026-06-21 13:47:35 +00:00
feat: Schema system for Basic Memory (#549)
Signed-off-by: phernandez <paul@basicmachines.co> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Paul Hernandez
+494
@@ -0,0 +1,494 @@
|
||||
# Note Format Reference
|
||||
|
||||
Every document in Basic Memory is a plain Markdown file. Files are the source of truth — changes to files automatically update the knowledge graph in the database. You maintain complete ownership, files work with git, and knowledge persists independently of any AI conversation.
|
||||
|
||||
## Document Structure
|
||||
|
||||
A note has three parts: YAML frontmatter, content (observations), and relations.
|
||||
|
||||
```markdown
|
||||
---
|
||||
title: Coffee Brewing Methods
|
||||
type: note
|
||||
tags: [coffee, brewing]
|
||||
permalink: coffee-brewing-methods
|
||||
---
|
||||
|
||||
# Coffee Brewing Methods
|
||||
|
||||
## Observations
|
||||
- [method] Pour over provides more flavor clarity than French press
|
||||
- [technique] Water temperature at 205°F extracts optimal compounds #brewing
|
||||
- [preference] Ethiopian beans work well with lighter roasts (personal experience)
|
||||
|
||||
## Relations
|
||||
- relates_to [[Coffee Bean Origins]]
|
||||
- requires [[Proper Grinding Technique]]
|
||||
- contrasts_with [[Tea Brewing Methods]]
|
||||
```
|
||||
|
||||
The `## Observations` and `## Relations` headings are conventional but not required — the parser detects observations and relations by their syntax patterns anywhere in the document.
|
||||
|
||||
## Frontmatter
|
||||
|
||||
YAML metadata between `---` fences at the top of the file.
|
||||
|
||||
| Field | Required | Default | Description |
|
||||
|-------|----------|---------|-------------|
|
||||
| `title` | No | filename stem | Used for linking and references. Auto-set from filename if missing. |
|
||||
| `type` | No | `note` | Entity type. Used for schema resolution and filtering. |
|
||||
| `tags` | No | `[]` | List or comma-separated string. Used for organization and search. |
|
||||
| `permalink` | No | generated from title | Stable identifier. Persists even if the file moves. |
|
||||
| `schema` | No | none | Schema attachment — dict (inline), string (reference), or omitted (implicit). |
|
||||
|
||||
Custom fields are allowed. Any key not in the standard set is stored as `entity_metadata` and indexed for search and filtering.
|
||||
|
||||
```yaml
|
||||
---
|
||||
title: Paul Graham
|
||||
type: Person
|
||||
tags: [startups, essays, lisp]
|
||||
permalink: paul-graham
|
||||
status: active
|
||||
source: wikipedia
|
||||
---
|
||||
```
|
||||
|
||||
Here `status` and `source` are custom fields stored in `entity_metadata`.
|
||||
|
||||
### Frontmatter Value Handling
|
||||
|
||||
YAML automatically converts some values to native types. Basic Memory normalizes them:
|
||||
|
||||
- Date strings (`2025-10-24`) → kept as ISO format strings
|
||||
- Numbers (`1.0`) → converted to strings
|
||||
- Booleans (`true`) → converted to strings (`"True"`)
|
||||
- Lists and dicts → preserved, items normalized recursively
|
||||
|
||||
This prevents errors when downstream code expects string values.
|
||||
|
||||
## Observations
|
||||
|
||||
An observation is a categorized fact about the entity. Written as a Markdown list item.
|
||||
|
||||
**Syntax:**
|
||||
|
||||
```
|
||||
- [category] content text #tag1 #tag2 (context)
|
||||
```
|
||||
|
||||
| Part | Required | Description |
|
||||
|------|----------|-------------|
|
||||
| `[category]` | Yes | Classification in square brackets. Any text except `[]()` chars. |
|
||||
| content | Yes | The fact or statement. |
|
||||
| `#tags` | No | Inline tags. Space-separated, each starting with `#`. |
|
||||
| `(context)` | No | Parenthesized text at end of line. Supporting details or source. |
|
||||
|
||||
### Examples
|
||||
|
||||
```markdown
|
||||
- [tech] Uses SQLite for storage #database
|
||||
- [design] Follows local-first architecture #architecture
|
||||
- [decision] Selected bcrypt for passwords #security (based on OWASP audit)
|
||||
- [name] Paul Graham
|
||||
- [expertise] Startups
|
||||
- [expertise] Lisp
|
||||
- [expertise] Essay writing
|
||||
```
|
||||
|
||||
Array-like fields use repeated categories — multiple `[expertise]` observations above.
|
||||
|
||||
### What Is Not an Observation
|
||||
|
||||
The parser excludes these list item patterns:
|
||||
|
||||
| Pattern | Example | Reason |
|
||||
|---------|---------|--------|
|
||||
| Checkboxes | `- [ ] Todo item`, `- [x] Done`, `- [-] Cancelled` | Task list syntax |
|
||||
| Markdown links | `- [text](url)` | URL link syntax |
|
||||
| Bare wiki links | `- [[Target]]` | Treated as a relation instead |
|
||||
|
||||
A list item with `#tags` but no `[category]` is still parsed — the tags are extracted and the category defaults to `Note`.
|
||||
|
||||
## Relations
|
||||
|
||||
Relations connect documents to form the knowledge graph. There are two kinds.
|
||||
|
||||
### Explicit Relations
|
||||
|
||||
Written as list items with a relation type and a `[[wiki link]]` target.
|
||||
|
||||
**Syntax:**
|
||||
|
||||
```
|
||||
- relation_type [[Target Entity]] (context)
|
||||
```
|
||||
|
||||
| Part | Required | Description |
|
||||
|------|----------|-------------|
|
||||
| `relation_type` | No | Text before `[[`. Defaults to `relates_to` if omitted. |
|
||||
| `[[Target]]` | Yes | Wiki link to the target entity. Matched by title or permalink. |
|
||||
| `(context)` | No | Parenthesized text after `]]`. Supporting details. |
|
||||
|
||||
### Examples
|
||||
|
||||
```markdown
|
||||
- implements [[Search Design]]
|
||||
- depends_on [[Database Schema]]
|
||||
- works_at [[Y Combinator]] (co-founder)
|
||||
- [[Some Entity]]
|
||||
```
|
||||
|
||||
The last example — a bare `[[wiki link]]` in a list item — gets relation type `relates_to`.
|
||||
|
||||
Common relation types:
|
||||
- `implements`, `depends_on`, `relates_to`, `inspired_by`
|
||||
- `extends`, `part_of`, `contains`, `pairs_with`
|
||||
- `works_at`, `authored`, `collaborated_with`
|
||||
|
||||
Any text works as a relation type. These are conventions, not a fixed set.
|
||||
|
||||
### Inline References
|
||||
|
||||
Wiki links appearing in regular prose (not as list items) create implicit `links_to` relations.
|
||||
|
||||
```markdown
|
||||
This builds on [[Core Design]] and uses [[Utility Functions]].
|
||||
```
|
||||
|
||||
This creates two relations: `links_to [[Core Design]]` and `links_to [[Utility Functions]]`.
|
||||
|
||||
### Forward References
|
||||
|
||||
Relations can link to entities that don't exist yet. Basic Memory resolves them when the target is created.
|
||||
|
||||
## Permalinks and memory:// URLs
|
||||
|
||||
Every document has a unique **permalink** — a stable identifier derived from its title. You can set one explicitly in frontmatter, or let the system generate it.
|
||||
|
||||
```yaml
|
||||
permalink: auth-approaches-2024
|
||||
```
|
||||
|
||||
Permalinks form the basis of `memory://` URLs:
|
||||
|
||||
```
|
||||
memory://auth-approaches-2024 # By permalink
|
||||
memory://Authentication Approaches # By title (auto-resolves)
|
||||
memory://project/auth-approaches # By path
|
||||
```
|
||||
|
||||
Pattern matching is supported:
|
||||
|
||||
```
|
||||
memory://auth* # Starts with "auth"
|
||||
memory://*/approaches # Ends with "approaches"
|
||||
memory://project/*/requirements # Nested wildcard
|
||||
```
|
||||
|
||||
## Schemas
|
||||
|
||||
Schemas declare the expected structure of a note — which observation categories and relation types a well-formed note should have. They use Picoschema, a compact notation from Google's Dotprompt that fits naturally in YAML frontmatter.
|
||||
|
||||
### Picoschema Syntax
|
||||
|
||||
```yaml
|
||||
schema:
|
||||
name: string, full name # required field with description
|
||||
email?: string, contact email # ? = optional
|
||||
role?: string, job title
|
||||
works_at?: Organization, employer # capitalized type = entity reference
|
||||
tags?(array): string, categories # array of type
|
||||
status?(enum): [active, inactive] # enum with allowed values
|
||||
metadata?(object): # nested object
|
||||
updated_at?: string
|
||||
source?: string
|
||||
```
|
||||
|
||||
| Notation | Meaning | Example |
|
||||
|----------|---------|---------|
|
||||
| `field: type` | Required field | `name: string` |
|
||||
| `field?: type` | Optional field | `role?: string` |
|
||||
| `field(array): type` | Array of values | `expertise(array): string` |
|
||||
| `field?(enum): [vals]` | Enum with allowed values | `status?(enum): [active, inactive]` |
|
||||
| `field?(object):` | Nested object with sub-fields | `metadata?(object):` |
|
||||
| `, description` | Description after comma | `name: string, full name` |
|
||||
| `EntityName` | Capitalized type = entity reference | `works_at?: Organization` |
|
||||
|
||||
**Scalar types:** `string`, `integer`, `number`, `boolean`, `any`
|
||||
|
||||
Any type not in that set whose first letter is uppercase is treated as an entity reference (a relation target).
|
||||
|
||||
### Schema-to-Note Mapping
|
||||
|
||||
Schemas validate against existing observation/relation syntax. Note authors don't learn new syntax.
|
||||
|
||||
| Schema Declaration | Maps To | Example in Note |
|
||||
|--------------------|---------|-----------------|
|
||||
| `field: string` | Observation `[field] value` | `- [name] Paul Graham` |
|
||||
| `field?(array): string` | Multiple `[field]` observations | `- [expertise] Lisp` (repeated) |
|
||||
| `field?: EntityType` | Relation `field [[Target]]` | `- works_at [[Y Combinator]]` |
|
||||
| `field?(array): EntityType` | Multiple `field` relations | `- authored [[Book]]` (repeated) |
|
||||
| `tags` | Frontmatter `tags` array | `tags: [startups, essays]` |
|
||||
| `field?(enum): [vals]` | Observation `[field] value` where value is in the set | `- [status] active` |
|
||||
|
||||
Observations and relations not covered by the schema are valid — schemas describe a subset, not a straitjacket.
|
||||
|
||||
### Schema Attachment
|
||||
|
||||
Three ways to attach a schema to a note, resolved in priority order:
|
||||
|
||||
**1. Inline schema** — `schema` is a dict in frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
title: Team Standup 2024-01-15
|
||||
type: meeting
|
||||
schema:
|
||||
attendees(array): string, who was there
|
||||
decisions(array): string, what was decided
|
||||
action_items(array): string, follow-ups
|
||||
blockers?(array): string, anything stuck
|
||||
---
|
||||
```
|
||||
|
||||
Good for one-off structured notes or prototyping a schema before extracting it.
|
||||
|
||||
**2. Explicit reference** — `schema` is a string naming a schema note:
|
||||
|
||||
```yaml
|
||||
---
|
||||
title: Basic Memory
|
||||
schema: SoftwareProject
|
||||
---
|
||||
```
|
||||
|
||||
or by permalink:
|
||||
|
||||
```yaml
|
||||
---
|
||||
title: LLM Memory Patterns
|
||||
schema: schema/research-project
|
||||
---
|
||||
```
|
||||
|
||||
Use when the note's `type` differs from the schema it should validate against, or when multiple schema variants exist.
|
||||
|
||||
**3. Implicit by type** — no `schema` field, resolved by matching `type`:
|
||||
|
||||
```yaml
|
||||
---
|
||||
title: Paul Graham
|
||||
type: Person
|
||||
---
|
||||
```
|
||||
|
||||
The system looks up a schema note where `entity: Person`. If found, it applies. If not, no validation occurs.
|
||||
|
||||
**4. No schema** — perfectly fine. Most notes don't need one.
|
||||
|
||||
### Schema Notes
|
||||
|
||||
A schema is itself a Basic Memory note with `type: schema`. It lives anywhere (though `schema/` is the conventional directory).
|
||||
|
||||
```yaml
|
||||
# schema/Person.md
|
||||
---
|
||||
title: Person
|
||||
type: schema
|
||||
entity: Person
|
||||
version: 1
|
||||
schema:
|
||||
name: string, full name
|
||||
role?: string, job title or position
|
||||
works_at?: Organization, employer
|
||||
expertise?(array): string, areas of knowledge
|
||||
email?: string, contact email
|
||||
settings:
|
||||
validation: warn
|
||||
---
|
||||
|
||||
# Person
|
||||
|
||||
A human individual in the knowledge graph.
|
||||
```
|
||||
|
||||
| Field | Required | Description |
|
||||
|-------|----------|-------------|
|
||||
| `type` | Yes | Must be `schema` |
|
||||
| `entity` | Yes | The entity type this schema describes (e.g., `Person`) |
|
||||
| `version` | No | Schema version number (default: `1`) |
|
||||
| `schema` | Yes | Picoschema dict defining the fields |
|
||||
| `settings.validation` | No | Validation mode (default: `warn`) |
|
||||
|
||||
Schema notes are regular notes — they show up in search, can have observations and relations, and participate in the knowledge graph.
|
||||
|
||||
### Validation Modes
|
||||
|
||||
| Mode | Behavior |
|
||||
|------|----------|
|
||||
| `warn` | Warnings in output, doesn't block (default) |
|
||||
| `strict` | Errors that block sync, for CI/CD enforcement |
|
||||
| `off` | No validation |
|
||||
|
||||
### Validation Output
|
||||
|
||||
```
|
||||
$ bm schema validate people/ada-lovelace.md
|
||||
|
||||
⚠ Person schema validation:
|
||||
- Missing required field: name (expected [name] observation)
|
||||
- Missing optional field: role
|
||||
- Missing optional field: works_at (no relation found)
|
||||
|
||||
ℹ Unmatched observations: [fact] ×2, [born] ×1
|
||||
ℹ Unmatched relations: collaborated_with
|
||||
```
|
||||
|
||||
"Unmatched" items are informational — observations and relations the schema doesn't cover.
|
||||
|
||||
### Schema Inference
|
||||
|
||||
Generate schemas from existing notes by analyzing observation and relation frequency:
|
||||
|
||||
```
|
||||
$ bm schema infer Person
|
||||
|
||||
Analyzing 30 notes with type: Person...
|
||||
|
||||
Observations found:
|
||||
[name] 30/30 100% → name: string
|
||||
[role] 27/30 90% → role?: string
|
||||
[expertise] 18/30 60% → expertise?(array): string
|
||||
[email] 8/30 27% → email?: string
|
||||
|
||||
Relations found:
|
||||
works_at 22/30 73% → works_at?: Organization
|
||||
|
||||
Suggested schema:
|
||||
name: string, full name
|
||||
role?: string, job title
|
||||
expertise?(array): string, areas of knowledge
|
||||
email?: string, contact email
|
||||
works_at?: Organization, employer
|
||||
|
||||
Save to schema/Person.md? [y/n]
|
||||
```
|
||||
|
||||
Frequency thresholds:
|
||||
- **100% present** → required field
|
||||
- **25%+ present** → optional field
|
||||
- **Below 25%** → excluded from suggestion
|
||||
|
||||
### Schema Drift Detection
|
||||
|
||||
Track how usage patterns shift over time:
|
||||
|
||||
```
|
||||
$ bm schema diff Person
|
||||
|
||||
Schema drift detected:
|
||||
|
||||
+ expertise: now in 81% of notes (was 12%)
|
||||
- department: dropped to 3% of notes
|
||||
~ works_at: cardinality changed (one → many)
|
||||
|
||||
Update schema? [y/n/review]
|
||||
```
|
||||
|
||||
## Complete Examples
|
||||
|
||||
### Simple Note (No Schema)
|
||||
|
||||
```markdown
|
||||
---
|
||||
title: Project Ideas
|
||||
type: note
|
||||
tags: [ideas, brainstorm]
|
||||
---
|
||||
|
||||
# Project Ideas
|
||||
|
||||
## Observations
|
||||
- [idea] Build a CLI tool for markdown linting #tooling
|
||||
- [idea] Create a recipe knowledge base #cooking
|
||||
- [priority] Focus on developer tools first (Q1 goal)
|
||||
|
||||
## Relations
|
||||
- inspired_by [[Developer Workflow Research]]
|
||||
- part_of [[Q1 Planning]]
|
||||
```
|
||||
|
||||
### Schema-Validated Note
|
||||
|
||||
Schema at `schema/Person.md`:
|
||||
|
||||
```yaml
|
||||
---
|
||||
title: Person
|
||||
type: schema
|
||||
entity: Person
|
||||
version: 1
|
||||
schema:
|
||||
name: string, full name
|
||||
role?: string, job title or position
|
||||
works_at?: Organization, employer
|
||||
expertise?(array): string, areas of knowledge
|
||||
email?: string, contact email
|
||||
settings:
|
||||
validation: warn
|
||||
---
|
||||
|
||||
# Person
|
||||
|
||||
A human individual in the knowledge graph.
|
||||
```
|
||||
|
||||
Note at `people/paul-graham.md`:
|
||||
|
||||
```markdown
|
||||
---
|
||||
title: Paul Graham
|
||||
type: Person
|
||||
tags: [startups, essays, lisp]
|
||||
---
|
||||
|
||||
# Paul Graham
|
||||
|
||||
## Observations
|
||||
- [name] Paul Graham
|
||||
- [role] Essayist and investor
|
||||
- [expertise] Startups
|
||||
- [expertise] Lisp
|
||||
- [expertise] Essay writing
|
||||
- [fact] Created Viaweb, the first web app
|
||||
|
||||
## Relations
|
||||
- works_at [[Y Combinator]]
|
||||
- authored [[Hackers and Painters]]
|
||||
```
|
||||
|
||||
The `[fact]` observation and `authored` relation are not in the schema — they're valid, just unmatched. The schema only checks that `[name]` exists (required) and looks for optional fields like `[role]`, `[expertise]`, and `works_at`.
|
||||
|
||||
### Inline Schema Note
|
||||
|
||||
```markdown
|
||||
---
|
||||
title: Team Standup 2024-01-15
|
||||
type: meeting
|
||||
schema:
|
||||
attendees(array): string, who was there
|
||||
decisions(array): string, what was decided
|
||||
action_items(array): string, follow-ups
|
||||
blockers?(array): string, anything stuck
|
||||
---
|
||||
|
||||
# Team Standup 2024-01-15
|
||||
|
||||
## Observations
|
||||
- [attendees] Paul
|
||||
- [attendees] Sarah
|
||||
- [decisions] Ship v2 by Friday
|
||||
- [action_items] Paul to review PR #42
|
||||
- [blockers] Waiting on API credentials
|
||||
```
|
||||
@@ -0,0 +1,365 @@
|
||||
# SPEC-SCHEMA-IMPL: Schema System Implementation Plan
|
||||
|
||||
**Status:** Draft
|
||||
**Created:** 2025-02-06
|
||||
**Branch:** `feature/schema-system`
|
||||
**Depends on:** [SPEC-SCHEMA](SPEC-SCHEMA.md)
|
||||
|
||||
## Overview
|
||||
|
||||
Implementation plan for the Basic Memory Schema System. The system is entirely programmatic —
|
||||
no LLM agent runtime or API key required. The LLM already in the user's session (Claude Code,
|
||||
Claude Desktop, etc.) provides the intelligence layer by reading schema notes via existing
|
||||
MCP tools.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────┐
|
||||
│ Entry Points │
|
||||
│ CLI (bm schema ...) │ MCP (schema_validate) │
|
||||
└──────────┬────────────┴──────────┬──────────────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
┌─────────────────────────────────────────────────┐
|
||||
│ Schema Service Layer │
|
||||
│ resolve_schema · validate · infer · diff │
|
||||
└──────────┬────────────────────────┬──────────────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
┌──────────────────────┐ ┌────────────────────────┐
|
||||
│ Picoschema Parser │ │ Note/Entity Access │
|
||||
│ YAML → SchemaModel │ │ (existing repository) │
|
||||
└──────────────────────┘ └────────────────────────┘
|
||||
```
|
||||
|
||||
No new database tables. Schemas are notes with `type: schema` — they're already indexed.
|
||||
Validation reads observations and relations from existing data.
|
||||
|
||||
## Components
|
||||
|
||||
### 1. Picoschema Parser
|
||||
|
||||
**Location:** `src/basic_memory/schema/parser.py`
|
||||
|
||||
Parses Picoschema YAML into an internal representation.
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class SchemaField:
|
||||
name: str
|
||||
type: str # string, integer, number, boolean, any, or EntityName
|
||||
required: bool # True unless field name ends with ?
|
||||
is_array: bool # True if (array) notation
|
||||
is_enum: bool # True if (enum) notation
|
||||
enum_values: list[str] # Populated for enums
|
||||
description: str | None # Text after comma
|
||||
is_entity_ref: bool # True if type is capitalized (entity reference)
|
||||
children: list[SchemaField] # For (object) types
|
||||
|
||||
|
||||
@dataclass
|
||||
class SchemaDefinition:
|
||||
entity: str # The entity type this schema describes
|
||||
version: int # Schema version
|
||||
fields: list[SchemaField] # Parsed fields
|
||||
validation_mode: str # "warn" | "strict" | "off"
|
||||
|
||||
|
||||
def parse_picoschema(yaml_dict: dict) -> list[SchemaField]:
|
||||
"""Parse a Picoschema YAML dict into a list of SchemaField objects."""
|
||||
|
||||
|
||||
def parse_schema_note(frontmatter: dict) -> SchemaDefinition:
|
||||
"""Parse a full schema note's frontmatter into a SchemaDefinition."""
|
||||
```
|
||||
|
||||
**Input/Output:**
|
||||
```yaml
|
||||
# Input (YAML dict from frontmatter)
|
||||
schema:
|
||||
name: string, full name
|
||||
role?: string, job title
|
||||
works_at?: Organization, employer
|
||||
expertise?(array): string, areas of knowledge
|
||||
```
|
||||
|
||||
```python
|
||||
# Output
|
||||
[
|
||||
SchemaField(name="name", type="string", required=True, description="full name", ...),
|
||||
SchemaField(name="role", type="string", required=False, description="job title", ...),
|
||||
SchemaField(name="works_at", type="Organization", required=False, is_entity_ref=True, ...),
|
||||
SchemaField(name="expertise", type="string", required=False, is_array=True, ...),
|
||||
]
|
||||
```
|
||||
|
||||
### 2. Schema Resolver
|
||||
|
||||
**Location:** `src/basic_memory/schema/resolver.py`
|
||||
|
||||
Finds the applicable schema for a note using the resolution order.
|
||||
|
||||
```python
|
||||
async def resolve_schema(
|
||||
note_frontmatter: dict,
|
||||
search_fn: Callable, # injected search capability
|
||||
) -> SchemaDefinition | None:
|
||||
"""Resolve schema for a note.
|
||||
|
||||
Resolution order:
|
||||
1. Inline schema (frontmatter['schema'] is a dict)
|
||||
2. Explicit reference (frontmatter['schema'] is a string)
|
||||
3. Implicit by type (frontmatter['type'] → schema note with matching entity)
|
||||
4. No schema (returns None)
|
||||
"""
|
||||
```
|
||||
|
||||
### 3. Schema Validator
|
||||
|
||||
**Location:** `src/basic_memory/schema/validator.py`
|
||||
|
||||
Validates a note's observations and relations against a resolved schema.
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class FieldResult:
|
||||
field: SchemaField
|
||||
status: str # "present" | "missing" | "type_mismatch"
|
||||
values: list[str] # Matched observation values or relation targets
|
||||
message: str | None # Human-readable detail
|
||||
|
||||
|
||||
@dataclass
|
||||
class ValidationResult:
|
||||
note_identifier: str
|
||||
schema_entity: str
|
||||
passed: bool # True if no errors (warnings are OK)
|
||||
field_results: list[FieldResult]
|
||||
unmatched_observations: dict[str, int] # category → count
|
||||
unmatched_relations: list[str] # relation types not in schema
|
||||
warnings: list[str]
|
||||
errors: list[str]
|
||||
|
||||
|
||||
async def validate_note(
|
||||
note: Note,
|
||||
schema: SchemaDefinition,
|
||||
) -> ValidationResult:
|
||||
"""Validate a note against a schema definition.
|
||||
|
||||
Mapping rules:
|
||||
- field: string → observation [field] exists
|
||||
- field?(array): type → multiple [field] observations
|
||||
- field?: EntityType → relation 'field [[...]]' exists
|
||||
- field?(enum): [v] → observation [field] value ∈ enum values
|
||||
"""
|
||||
```
|
||||
|
||||
### 4. Schema Inference Engine
|
||||
|
||||
**Location:** `src/basic_memory/schema/inference.py`
|
||||
|
||||
Analyzes notes of a given type and suggests a schema based on usage frequency.
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class FieldFrequency:
|
||||
name: str
|
||||
source: str # "observation" | "relation"
|
||||
count: int # notes containing this field
|
||||
total: int # total notes analyzed
|
||||
percentage: float
|
||||
sample_values: list[str] # representative values
|
||||
is_array: bool # True if typically appears multiple times per note
|
||||
target_type: str | None # For relations, the most common target entity type
|
||||
|
||||
|
||||
@dataclass
|
||||
class InferenceResult:
|
||||
entity_type: str
|
||||
notes_analyzed: int
|
||||
field_frequencies: list[FieldFrequency]
|
||||
suggested_schema: dict # Ready-to-use Picoschema YAML dict
|
||||
suggested_required: list[str]
|
||||
suggested_optional: list[str]
|
||||
excluded: list[str] # Below threshold
|
||||
|
||||
|
||||
async def infer_schema(
|
||||
entity_type: str,
|
||||
notes: list[Note],
|
||||
required_threshold: float = 0.95, # 95%+ = required
|
||||
optional_threshold: float = 0.25, # 25%+ = optional
|
||||
) -> InferenceResult:
|
||||
"""Analyze notes and suggest a Picoschema definition."""
|
||||
```
|
||||
|
||||
### 5. Schema Diff
|
||||
|
||||
**Location:** `src/basic_memory/schema/diff.py`
|
||||
|
||||
Compares current note usage against an existing schema definition.
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class SchemaDrift:
|
||||
new_fields: list[FieldFrequency] # Fields not in schema but common in notes
|
||||
dropped_fields: list[FieldFrequency] # Fields in schema but rare in notes
|
||||
cardinality_changes: list[str] # one → many or many → one
|
||||
type_mismatches: list[str] # observation values don't match declared type
|
||||
|
||||
|
||||
async def diff_schema(
|
||||
schema: SchemaDefinition,
|
||||
notes: list[Note],
|
||||
) -> SchemaDrift:
|
||||
"""Compare a schema against actual note usage to detect drift."""
|
||||
```
|
||||
|
||||
## Entry Points
|
||||
|
||||
### CLI Commands
|
||||
|
||||
**Location:** `src/basic_memory/cli/schema.py`
|
||||
|
||||
```python
|
||||
import typer
|
||||
|
||||
schema_app = typer.Typer(name="schema", help="Schema management commands")
|
||||
|
||||
@schema_app.command()
|
||||
async def validate(
|
||||
target: str = typer.Argument(None, help="Note path or entity type"),
|
||||
strict: bool = typer.Option(False, help="Override to strict mode"),
|
||||
):
|
||||
"""Validate notes against their schemas."""
|
||||
|
||||
@schema_app.command()
|
||||
async def infer(
|
||||
entity_type: str = typer.Argument(..., help="Entity type to analyze"),
|
||||
threshold: float = typer.Option(0.25, help="Minimum frequency for optional fields"),
|
||||
save: bool = typer.Option(False, help="Save to schema/ directory"),
|
||||
):
|
||||
"""Infer schema from existing notes of a type."""
|
||||
|
||||
@schema_app.command()
|
||||
async def diff(
|
||||
entity_type: str = typer.Argument(..., help="Entity type to diff"),
|
||||
):
|
||||
"""Show drift between schema and actual usage."""
|
||||
```
|
||||
|
||||
Registered as subcommand: `bm schema validate`, `bm schema infer`, `bm schema diff`.
|
||||
|
||||
### MCP Tools
|
||||
|
||||
**Location:** `src/basic_memory/mcp/tools/schema.py`
|
||||
|
||||
```python
|
||||
@mcp_tool
|
||||
async def schema_validate(
|
||||
entity_type: str | None = None,
|
||||
identifier: str | None = None,
|
||||
project: str | None = None,
|
||||
) -> str:
|
||||
"""Validate notes against their resolved schema."""
|
||||
|
||||
@mcp_tool
|
||||
async def schema_infer(
|
||||
entity_type: str,
|
||||
threshold: float = 0.25,
|
||||
project: str | None = None,
|
||||
) -> str:
|
||||
"""Analyze existing notes and suggest a schema definition."""
|
||||
```
|
||||
|
||||
### API Endpoints
|
||||
|
||||
**Location:** `src/basic_memory/api/schema_router.py`
|
||||
|
||||
```python
|
||||
router = APIRouter(prefix="/schema", tags=["schema"])
|
||||
|
||||
@router.post("/validate")
|
||||
async def validate_schema(...) -> ValidationReport: ...
|
||||
|
||||
@router.post("/infer")
|
||||
async def infer_schema(...) -> InferenceResult: ...
|
||||
|
||||
@router.get("/diff/{entity_type}")
|
||||
async def diff_schema(...) -> SchemaDrift: ...
|
||||
```
|
||||
|
||||
MCP tools call these endpoints via the typed client pattern (consistent with existing
|
||||
architecture).
|
||||
|
||||
## Implementation Phases
|
||||
|
||||
### Phase 1: Parser + Resolver
|
||||
|
||||
Build the foundation — can parse Picoschema and find schemas for notes.
|
||||
|
||||
**Deliverables:**
|
||||
- `schema/parser.py` — Picoschema YAML → `SchemaDefinition`
|
||||
- `schema/resolver.py` — Resolution order (inline → explicit ref → implicit by type → none)
|
||||
- Unit tests for all Picoschema syntax variations
|
||||
- Unit tests for resolution order
|
||||
|
||||
**No external dependencies.** Pure Python parsing of YAML dicts. Can develop and test
|
||||
in isolation.
|
||||
|
||||
### Phase 2: Validator
|
||||
|
||||
Connect schemas to notes and produce validation results.
|
||||
|
||||
**Deliverables:**
|
||||
- `schema/validator.py` — Validate note observations/relations against schema fields
|
||||
- API endpoint: `POST /schema/validate`
|
||||
- MCP tool: `schema_validate`
|
||||
- CLI command: `bm schema validate`
|
||||
- Integration tests with real notes and schemas
|
||||
|
||||
**Depends on:** Phase 1 (parser + resolver)
|
||||
|
||||
### Phase 3: Inference
|
||||
|
||||
Analyze existing notes to suggest schemas.
|
||||
|
||||
**Deliverables:**
|
||||
- `schema/inference.py` — Frequency analysis across notes of a type
|
||||
- API endpoint: `POST /schema/infer`
|
||||
- MCP tool: `schema_infer`
|
||||
- CLI command: `bm schema infer`
|
||||
- Option to save inferred schema as a note via `write_note`
|
||||
|
||||
**Depends on:** Phase 1 (parser for output format)
|
||||
|
||||
### Phase 4: Diff
|
||||
|
||||
Compare schemas against current usage.
|
||||
|
||||
**Deliverables:**
|
||||
- `schema/diff.py` — Drift detection between schema and actual notes
|
||||
- API endpoint: `GET /schema/diff/{entity_type}`
|
||||
- CLI command: `bm schema diff`
|
||||
|
||||
**Depends on:** Phase 1 (parser), Phase 3 (inference, for frequency analysis)
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- **Unit tests** (`tests/schema/`): Parser edge cases, resolution logic, validation mapping,
|
||||
inference thresholds
|
||||
- **Integration tests** (`test-int/schema/`): End-to-end with real markdown files, schema notes
|
||||
on disk, CLI invocation
|
||||
- Coverage target: 100% (consistent with project standard)
|
||||
|
||||
## What This Does NOT Include
|
||||
|
||||
- No new database tables or migrations
|
||||
- No new markdown syntax (schemas validate existing observations/relations)
|
||||
- No LLM agent runtime or API key management
|
||||
- No hook integration (deferred)
|
||||
- No schema composition/inheritance (deferred)
|
||||
- No OWL/RDF export (deferred)
|
||||
- No built-in templates (deferred)
|
||||
@@ -0,0 +1,462 @@
|
||||
# SPEC-SCHEMA: Basic Memory Schema System
|
||||
|
||||
**Status:** Draft
|
||||
**Created:** 2025-02-06
|
||||
**Branch:** `feature/schema-system`
|
||||
|
||||
## Summary
|
||||
|
||||
A schema system for Basic Memory that uses [Picoschema](https://genkit.dev/docs/dotprompt/)
|
||||
syntax in YAML frontmatter. Schemas validate notes against their existing observation/relation
|
||||
structure — no new data model, no migration, just a declarative lens over what's already there.
|
||||
|
||||
## Core Principles
|
||||
|
||||
1. **Schemas are just notes** — A schema is a note with `type: schema`, lives anywhere
|
||||
2. **Use prior art** — Picoschema syntax in YAML frontmatter, no custom notation
|
||||
3. **Validation maps to existing format** — Observations and relations, not a parallel data model
|
||||
4. **Validation is soft** — Warnings by default, not blocking errors
|
||||
5. **Inference over prescription** — Schemas describe reality, emerge from usage
|
||||
6. **No built-in agent** — Programmatic core; the LLM already in the session provides intelligence
|
||||
|
||||
## Picoschema Syntax
|
||||
|
||||
Picoschema is a compact schema notation from Google's Dotprompt that fits naturally in YAML
|
||||
frontmatter.
|
||||
|
||||
### Supported Types
|
||||
|
||||
| Type | Description |
|
||||
|------|-------------|
|
||||
| `string` | Text value |
|
||||
| `integer` | Whole number |
|
||||
| `number` | Decimal number |
|
||||
| `boolean` | True/false |
|
||||
| `any` | Any scalar type |
|
||||
| `EntityName` | Reference to another entity (capitalized = entity reference) |
|
||||
|
||||
### Syntax Rules
|
||||
|
||||
```yaml
|
||||
schema:
|
||||
name: string, full name # required field with description
|
||||
email?: string, contact email # ? = optional
|
||||
role?: string, job title
|
||||
works_at?: Organization, employer # capitalized type = entity reference
|
||||
tags?(array): string, categories # array of type
|
||||
status?(enum): [active, inactive] # enum with allowed values
|
||||
metadata?(object): # nested object
|
||||
updated_at?: string
|
||||
source?: string
|
||||
```
|
||||
|
||||
- `field: type` — required field
|
||||
- `field?: type` — optional field
|
||||
- `field(array): type` — array of values
|
||||
- `field?(enum): [values]` — enumeration
|
||||
- `field?(object):` — nested object with sub-fields
|
||||
- `, description` — description after comma
|
||||
- `EntityName` as type (capitalized) — reference to another entity
|
||||
|
||||
## Schema-to-Note Mapping
|
||||
|
||||
Schemas validate against the existing Basic Memory note format. No new syntax for note
|
||||
authors to learn.
|
||||
|
||||
### Mapping Rules
|
||||
|
||||
| Schema Declaration | Grounded In | Example Match |
|
||||
|--------------------|-------------|---------------|
|
||||
| `field: string` | Observation `[field] value` | `- [name] Paul Graham` |
|
||||
| `field?(array): string` | Multiple `[field]` observations | `- [expertise] Lisp` (×N) |
|
||||
| `field?: EntityType` | Relation `field [[Target]]` | `- works_at [[Y Combinator]]` |
|
||||
| `field?(array): EntityType` | Multiple `field` relations | `- authored [[Book]]` (×N) |
|
||||
| `tags` | Frontmatter `tags` array | `tags: [startups, essays]` |
|
||||
| `field?(enum): [values]` | Observation `[field] value` where value ∈ set | `- [status] active` |
|
||||
|
||||
### Key Insight
|
||||
|
||||
Schemas don't introduce a new way to store data. They describe the patterns already present
|
||||
in observations and relations. A note doesn't have to change how it's written — the schema
|
||||
just says "a good Person note has a `[name]` observation and a `works_at` relation."
|
||||
|
||||
## Schema Definition
|
||||
|
||||
### As a Dedicated Schema Note
|
||||
|
||||
```yaml
|
||||
# schema/Person.md
|
||||
---
|
||||
title: Person
|
||||
type: schema
|
||||
entity: Person
|
||||
version: 1
|
||||
schema:
|
||||
name: string, full name
|
||||
email?: string, contact email
|
||||
role?: string, job title
|
||||
works_at?: Organization, employer
|
||||
expertise?(array): string, areas of knowledge
|
||||
settings:
|
||||
validation: warn # warn | strict | off
|
||||
---
|
||||
|
||||
# Person
|
||||
|
||||
A human individual in the knowledge graph.
|
||||
|
||||
Any documentation about this entity type goes here as prose.
|
||||
```
|
||||
|
||||
Schema notes are regular Basic Memory notes. They show up in search, can have their own
|
||||
observations and relations, and can be organized in any folder (though `schema/` is
|
||||
the suggested convention).
|
||||
|
||||
### Inline Schema in a Note
|
||||
|
||||
Notes can carry their own schema directly:
|
||||
|
||||
```yaml
|
||||
# meetings/2024-01-15-standup.md
|
||||
---
|
||||
title: Team Standup 2024-01-15
|
||||
type: meeting
|
||||
schema:
|
||||
attendees(array): string, who was there
|
||||
decisions(array): string, what was decided
|
||||
action_items(array): string, follow-ups
|
||||
blockers?(array): string, anything stuck
|
||||
---
|
||||
|
||||
# Team Standup 2024-01-15
|
||||
|
||||
## Observations
|
||||
- [attendees] Paul
|
||||
- [attendees] Sarah
|
||||
- [decisions] Ship v2 by Friday
|
||||
- [action_items] Paul to review PR #42
|
||||
- [blockers] Waiting on API credentials
|
||||
```
|
||||
|
||||
Good for one-off structured notes or prototyping a schema before extracting it.
|
||||
|
||||
### Explicit Schema Reference
|
||||
|
||||
A note can reference a schema by entity name or permalink:
|
||||
|
||||
```yaml
|
||||
# projects/basic-memory.md
|
||||
---
|
||||
title: Basic Memory
|
||||
schema: SoftwareProject # by entity name
|
||||
---
|
||||
|
||||
# research/llm-memory-patterns.md
|
||||
---
|
||||
title: LLM Memory Patterns
|
||||
schema: schema/research-project # by permalink
|
||||
---
|
||||
```
|
||||
|
||||
Use cases:
|
||||
- Note's `type` differs from the schema it should validate against
|
||||
- Multiple schema variants exist for the same domain
|
||||
- Applying structure to existing notes without changing their type
|
||||
|
||||
## Schema Resolution
|
||||
|
||||
When validating a note, schemas resolve in priority order:
|
||||
|
||||
```
|
||||
1. Inline schema → schema: { ... } (dict in frontmatter)
|
||||
2. Explicit ref → schema: Person (string in frontmatter)
|
||||
3. Implicit by type → type: Person (lookup schema note with entity: Person)
|
||||
4. No schema → no validation (perfectly fine)
|
||||
```
|
||||
|
||||
```python
|
||||
async def resolve_schema(note: Note) -> Schema | None:
|
||||
schema_value = note.frontmatter.get('schema')
|
||||
|
||||
# 1. Inline schema (dict)
|
||||
if isinstance(schema_value, dict):
|
||||
return parse_picoschema(schema_value)
|
||||
|
||||
# 2. Explicit reference (string)
|
||||
if isinstance(schema_value, str):
|
||||
schema_note = await find_schema_note(schema_value)
|
||||
if schema_note:
|
||||
return parse_picoschema(schema_note.frontmatter['schema'])
|
||||
|
||||
# 3. Implicit by type
|
||||
note_type = note.frontmatter.get('type')
|
||||
if note_type:
|
||||
results = await search_notes(f"type:schema entity:{note_type}")
|
||||
if results:
|
||||
return parse_picoschema(results[0].frontmatter['schema'])
|
||||
|
||||
# 4. No schema
|
||||
return None
|
||||
```
|
||||
|
||||
## Validation
|
||||
|
||||
### Modes
|
||||
|
||||
Configured in the schema's `settings.validation`:
|
||||
|
||||
| Mode | Behavior |
|
||||
|------|----------|
|
||||
| `off` | No validation |
|
||||
| `warn` | Warnings in output, doesn't block (default) |
|
||||
| `strict` | Errors that block sync, for CI/CD enforcement |
|
||||
|
||||
### Validation Output
|
||||
|
||||
For a note missing required fields:
|
||||
|
||||
```
|
||||
$ bm schema validate people/ada-lovelace.md
|
||||
|
||||
⚠ Person schema validation:
|
||||
- Missing required field: name (expected [name] observation)
|
||||
- Missing optional field: role
|
||||
- Missing optional field: works_at (no relation found)
|
||||
|
||||
ℹ Unmatched observations: [fact] ×2, [born] ×1
|
||||
ℹ Unmatched relations: collaborated_with
|
||||
```
|
||||
|
||||
"Unmatched" items are informational — observations and relations the schema doesn't cover.
|
||||
They're valid. Schemas are a subset, not a straitjacket.
|
||||
|
||||
### Batch Validation
|
||||
|
||||
```
|
||||
$ bm schema validate Person
|
||||
|
||||
Validating 30 notes against Person schema...
|
||||
|
||||
✓ people/paul-graham.md — all fields present
|
||||
✓ people/rich-hickey.md — all fields present
|
||||
⚠ people/ada-lovelace.md — missing: name
|
||||
⚠ people/alan-kay.md — missing: name, role
|
||||
✓ people/linus-torvalds.md — all fields present
|
||||
...
|
||||
|
||||
Summary: 22/30 valid, 8 warnings, 0 errors
|
||||
```
|
||||
|
||||
## Emerging Schemas
|
||||
|
||||
### The Problem with Traditional Schemas
|
||||
|
||||
Most schema systems require: define schema → create conforming content → fight the schema
|
||||
when reality doesn't match. This is backwards. Knowledge grows organically.
|
||||
|
||||
### The Basic Memory Approach
|
||||
|
||||
```
|
||||
Write notes freely → Patterns emerge → Crystallize into schema → Validate future notes
|
||||
```
|
||||
|
||||
### Schema Inference
|
||||
|
||||
Generate schemas from existing notes by analyzing observation and relation frequency:
|
||||
|
||||
```
|
||||
$ bm schema infer Person
|
||||
|
||||
Analyzing 30 notes with type: Person...
|
||||
|
||||
Observations found:
|
||||
[name] 30/30 100% → name: string
|
||||
[role] 27/30 90% → role?: string
|
||||
[fact] 25/30 83% (generic — no single field)
|
||||
[expertise] 18/30 60% → expertise?(array): string
|
||||
[email] 8/30 27% → email?: string
|
||||
[born] 6/30 20% (below threshold)
|
||||
|
||||
Relations found:
|
||||
works_at 22/30 73% → works_at?: Organization
|
||||
authored 11/30 37% → authored?(array): string
|
||||
|
||||
Suggested schema:
|
||||
name: string, full name
|
||||
role?: string, job title
|
||||
expertise?(array): string, areas of knowledge
|
||||
email?: string, contact email
|
||||
works_at?: Organization, employer
|
||||
|
||||
Save to schema/Person.md? [y/n]
|
||||
```
|
||||
|
||||
Frequency thresholds:
|
||||
- 100% present → required field
|
||||
- 25%+ present → optional field
|
||||
- Below 25% → excluded from suggestion (but noted)
|
||||
|
||||
### Schema Drift Detection
|
||||
|
||||
Track how usage patterns shift over time:
|
||||
|
||||
```
|
||||
$ bm schema diff Person
|
||||
|
||||
Schema drift detected:
|
||||
|
||||
+ expertise: now in 81% of notes (was 12%)
|
||||
- department: dropped to 3% of notes
|
||||
~ works_at: cardinality changed (one → many)
|
||||
|
||||
Update schema? [y/n/review]
|
||||
```
|
||||
|
||||
## LLM Integration (AI Guidance)
|
||||
|
||||
No agent runtime or API key required. The LLM already in the session uses schemas as
|
||||
context for note creation.
|
||||
|
||||
### Flow
|
||||
|
||||
1. User asks LLM to "write a note about Rich Hickey"
|
||||
2. LLM determines `type: Person` is appropriate
|
||||
3. LLM calls `search_notes("type:schema entity:Person")` → finds schema
|
||||
4. LLM reads schema fields: required `name`, optional `role`, `works_at`, `expertise`
|
||||
5. LLM calls `write_note` with observations and relations that satisfy the schema
|
||||
|
||||
The schema acts as a creation template. The LLM knows what a "complete" note looks like
|
||||
without any custom agent infrastructure.
|
||||
|
||||
### MCP Tools
|
||||
|
||||
```python
|
||||
@mcp_tool
|
||||
async def schema_validate(
|
||||
entity_type: str | None = None,
|
||||
identifier: str | None = None,
|
||||
project: str | None = None,
|
||||
) -> ValidationReport:
|
||||
"""Validate notes against their resolved schema.
|
||||
|
||||
Validates a specific note (by identifier) or all notes of a given type.
|
||||
Returns warnings/errors based on the schema's validation mode.
|
||||
"""
|
||||
|
||||
@mcp_tool
|
||||
async def schema_infer(
|
||||
entity_type: str,
|
||||
threshold: float = 0.25,
|
||||
project: str | None = None,
|
||||
) -> SuggestedSchema:
|
||||
"""Analyze existing notes and suggest a schema definition.
|
||||
|
||||
Examines observation categories and relation types across all notes
|
||||
of the given type. Returns frequency analysis and suggested Picoschema.
|
||||
"""
|
||||
```
|
||||
|
||||
## CLI Commands
|
||||
|
||||
```bash
|
||||
# Validate a specific note
|
||||
bm schema validate people/ada-lovelace.md
|
||||
|
||||
# Validate all notes of a type
|
||||
bm schema validate Person
|
||||
|
||||
# Validate everything with a schema
|
||||
bm schema validate
|
||||
|
||||
# Infer schema from existing notes
|
||||
bm schema infer Person
|
||||
|
||||
# Show schema drift from current definition
|
||||
bm schema diff Person
|
||||
|
||||
# List all schema notes
|
||||
bm search "type:schema"
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
### Complete Person Workflow
|
||||
|
||||
**Schema:**
|
||||
```yaml
|
||||
# schema/Person.md
|
||||
---
|
||||
title: Person
|
||||
type: schema
|
||||
entity: Person
|
||||
version: 1
|
||||
schema:
|
||||
name: string, full name
|
||||
role?: string, job title or position
|
||||
works_at?: Organization, employer
|
||||
expertise?(array): string, areas of knowledge
|
||||
email?: string, contact email
|
||||
settings:
|
||||
validation: warn
|
||||
---
|
||||
|
||||
# Person
|
||||
|
||||
A human individual in the knowledge graph.
|
||||
```
|
||||
|
||||
**Valid note:**
|
||||
```yaml
|
||||
# people/paul-graham.md
|
||||
---
|
||||
title: Paul Graham
|
||||
type: Person
|
||||
tags: [startups, essays, lisp]
|
||||
---
|
||||
|
||||
# Paul Graham
|
||||
|
||||
## Observations
|
||||
- [name] Paul Graham
|
||||
- [role] Essayist and investor
|
||||
- [expertise] Startups
|
||||
- [expertise] Lisp
|
||||
- [expertise] Essay writing
|
||||
- [fact] Created Viaweb, the first web app
|
||||
|
||||
## Relations
|
||||
- works_at [[Y Combinator]]
|
||||
- authored [[Hackers and Painters]]
|
||||
```
|
||||
|
||||
**Note with warnings:**
|
||||
```yaml
|
||||
# people/ada-lovelace.md
|
||||
---
|
||||
title: Ada Lovelace
|
||||
type: Person
|
||||
---
|
||||
|
||||
# Ada Lovelace
|
||||
|
||||
## Observations
|
||||
- [fact] Wrote the first computer program
|
||||
- [born] 1815
|
||||
|
||||
## Relations
|
||||
- collaborated_with [[Charles Babbage]]
|
||||
```
|
||||
|
||||
Validation: warns about missing required `[name]` observation. Everything else is optional
|
||||
or unmatched (which is fine).
|
||||
|
||||
## Future Considerations (Deferred)
|
||||
|
||||
These are interesting but out of scope for the initial implementation:
|
||||
|
||||
- **Multiple schema inheritance** — `schema: [Person, Author]`
|
||||
- **Hook integration** — Pre-write validation via the hooks system
|
||||
- **OWL/RDF export** — `bm schema export --format owl`
|
||||
- **SPARQL queries** — Schema-aware graph queries
|
||||
- **Built-in templates** — `bm schema use gtd`, `bm schema use zettelkasten`
|
||||
- **Schema versioning/migration** — Tracking breaking changes across versions
|
||||
@@ -18,6 +18,7 @@ from basic_memory.api.v2.routers import (
|
||||
directory_router as v2_directory,
|
||||
prompt_router as v2_prompt,
|
||||
importer_router as v2_importer,
|
||||
schema_router as v2_schema,
|
||||
)
|
||||
from basic_memory.api.v2.routers.project_router import (
|
||||
add_project,
|
||||
@@ -84,6 +85,7 @@ app.include_router(v2_resource, prefix="/v2/projects/{project_id}")
|
||||
app.include_router(v2_directory, prefix="/v2/projects/{project_id}")
|
||||
app.include_router(v2_prompt, prefix="/v2/projects/{project_id}")
|
||||
app.include_router(v2_importer, prefix="/v2/projects/{project_id}")
|
||||
app.include_router(v2_schema, prefix="/v2/projects/{project_id}")
|
||||
app.include_router(v2_project, prefix="/v2")
|
||||
|
||||
# Legacy web app proxy paths (compat with /proxy/projects/projects)
|
||||
|
||||
@@ -8,6 +8,7 @@ from basic_memory.api.v2.routers.resource_router import router as resource_route
|
||||
from basic_memory.api.v2.routers.directory_router import router as directory_router
|
||||
from basic_memory.api.v2.routers.prompt_router import router as prompt_router
|
||||
from basic_memory.api.v2.routers.importer_router import router as importer_router
|
||||
from basic_memory.api.v2.routers.schema_router import router as schema_router
|
||||
|
||||
__all__ = [
|
||||
"knowledge_router",
|
||||
@@ -18,4 +19,5 @@ __all__ = [
|
||||
"directory_router",
|
||||
"prompt_router",
|
||||
"importer_router",
|
||||
"schema_router",
|
||||
]
|
||||
|
||||
@@ -0,0 +1,305 @@
|
||||
"""V2 router for schema operations.
|
||||
|
||||
Provides endpoints for schema validation, inference, and drift detection.
|
||||
The schema system validates notes against Picoschema definitions without
|
||||
introducing any new data model -- it works entirely with existing
|
||||
observations and relations.
|
||||
|
||||
Flow: Entity loaded with eager observations/relations -> convert to tuples -> core functions.
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter, Path, Query
|
||||
|
||||
from basic_memory.deps import (
|
||||
SearchServiceV2ExternalDep,
|
||||
EntityRepositoryV2ExternalDep,
|
||||
)
|
||||
from basic_memory.models.knowledge import Entity
|
||||
from basic_memory.schemas.schema import (
|
||||
ValidationReport,
|
||||
InferenceReport,
|
||||
DriftReport,
|
||||
NoteValidationResponse,
|
||||
FieldResultResponse,
|
||||
FieldFrequencyResponse,
|
||||
DriftFieldResponse,
|
||||
)
|
||||
from basic_memory.schemas.search import SearchQuery
|
||||
from basic_memory.schema.resolver import resolve_schema
|
||||
from basic_memory.schema.validator import validate_note
|
||||
from basic_memory.schema.inference import infer_schema, NoteData, ObservationData, RelationData
|
||||
from basic_memory.schema.diff import diff_schema
|
||||
|
||||
# Note: No prefix here -- it's added during registration as /v2/{project_id}/schema
|
||||
router = APIRouter(tags=["schema"])
|
||||
|
||||
|
||||
# --- ORM to core data conversion ---
|
||||
|
||||
|
||||
def _entity_observations(entity: Entity) -> list[ObservationData]:
|
||||
"""Extract ObservationData from an entity's observations."""
|
||||
return [ObservationData(obs.category, obs.content) for obs in entity.observations]
|
||||
|
||||
|
||||
def _entity_relations(entity: Entity) -> list[RelationData]:
|
||||
"""Extract RelationData from an entity's outgoing relations.
|
||||
|
||||
Carries the target entity's type on each relation so the inference engine
|
||||
can suggest correct types (e.g. works_at -> Organization, not the source type).
|
||||
"""
|
||||
return [
|
||||
RelationData(
|
||||
relation_type=rel.relation_type,
|
||||
target_name=rel.to_name,
|
||||
target_entity_type=rel.to_entity.entity_type if rel.to_entity else None,
|
||||
)
|
||||
for rel in entity.outgoing_relations
|
||||
]
|
||||
|
||||
|
||||
def _entity_to_note_data(entity: Entity) -> NoteData:
|
||||
"""Convert an ORM Entity to a NoteData for inference/diff analysis."""
|
||||
return NoteData(
|
||||
identifier=entity.permalink or entity.file_path,
|
||||
observations=_entity_observations(entity),
|
||||
relations=_entity_relations(entity),
|
||||
)
|
||||
|
||||
|
||||
def _entity_frontmatter(entity: Entity) -> dict:
|
||||
"""Build a frontmatter dict from an entity for schema resolution."""
|
||||
frontmatter = dict(entity.entity_metadata) if entity.entity_metadata else {}
|
||||
if entity.entity_type:
|
||||
frontmatter.setdefault("type", entity.entity_type)
|
||||
return frontmatter
|
||||
|
||||
|
||||
# --- Validation ---
|
||||
|
||||
|
||||
@router.post("/schema/validate", response_model=ValidationReport)
|
||||
async def validate_schema(
|
||||
entity_repository: EntityRepositoryV2ExternalDep,
|
||||
search_service: SearchServiceV2ExternalDep,
|
||||
project_id: str = Path(..., description="Project external UUID"),
|
||||
entity_type: str | None = Query(None, description="Entity type to validate"),
|
||||
identifier: str | None = Query(None, description="Specific note identifier"),
|
||||
):
|
||||
"""Validate notes against their resolved schemas.
|
||||
|
||||
Validates a specific note (by identifier) or all notes of a given type.
|
||||
Returns warnings/errors based on the schema's validation mode.
|
||||
"""
|
||||
results: list[NoteValidationResponse] = []
|
||||
|
||||
async def search_fn(query: str) -> list:
|
||||
# Search for schema notes, then load full entity_metadata from the entity table.
|
||||
# The search index only stores minimal metadata (e.g., {"entity_type": "schema"}),
|
||||
# but parse_schema_note needs the full frontmatter with entity/schema/version keys.
|
||||
results = await search_service.search(SearchQuery(text=query, types=["schema"]), limit=5)
|
||||
frontmatters = []
|
||||
for row in results:
|
||||
if row.permalink:
|
||||
entity = await entity_repository.get_by_permalink(row.permalink)
|
||||
if entity:
|
||||
frontmatters.append(_entity_frontmatter(entity))
|
||||
return frontmatters
|
||||
|
||||
# --- Single note validation ---
|
||||
if identifier:
|
||||
entity = await entity_repository.get_by_permalink(identifier)
|
||||
if not entity:
|
||||
return ValidationReport(entity_type=entity_type, total_notes=0, results=[])
|
||||
|
||||
schema_def = await resolve_schema(_entity_frontmatter(entity), search_fn)
|
||||
if schema_def:
|
||||
result = validate_note(
|
||||
entity.permalink or identifier,
|
||||
schema_def,
|
||||
_entity_observations(entity),
|
||||
_entity_relations(entity),
|
||||
)
|
||||
results.append(_to_note_validation_response(result))
|
||||
|
||||
return ValidationReport(
|
||||
entity_type=entity_type or entity.entity_type,
|
||||
total_notes=1,
|
||||
valid_count=1 if (results and results[0].passed) else 0,
|
||||
warning_count=sum(len(r.warnings) for r in results),
|
||||
error_count=sum(len(r.errors) for r in results),
|
||||
results=results,
|
||||
)
|
||||
|
||||
# --- Batch validation by entity type ---
|
||||
entities = await _find_by_entity_type(entity_repository, entity_type) if entity_type else []
|
||||
|
||||
for entity in entities:
|
||||
schema_def = await resolve_schema(_entity_frontmatter(entity), search_fn)
|
||||
if schema_def:
|
||||
result = validate_note(
|
||||
entity.permalink or entity.file_path,
|
||||
schema_def,
|
||||
_entity_observations(entity),
|
||||
_entity_relations(entity),
|
||||
)
|
||||
results.append(_to_note_validation_response(result))
|
||||
|
||||
valid = sum(1 for r in results if r.passed)
|
||||
return ValidationReport(
|
||||
entity_type=entity_type,
|
||||
total_notes=len(results),
|
||||
valid_count=valid,
|
||||
warning_count=sum(len(r.warnings) for r in results),
|
||||
error_count=sum(len(r.errors) for r in results),
|
||||
results=results,
|
||||
)
|
||||
|
||||
|
||||
# --- Inference ---
|
||||
|
||||
|
||||
@router.post("/schema/infer", response_model=InferenceReport)
|
||||
async def infer_schema_endpoint(
|
||||
entity_repository: EntityRepositoryV2ExternalDep,
|
||||
project_id: str = Path(..., description="Project external UUID"),
|
||||
entity_type: str = Query(..., description="Entity type to analyze"),
|
||||
threshold: float = Query(0.25, description="Minimum frequency for optional fields"),
|
||||
):
|
||||
"""Infer a schema from existing notes of a given type.
|
||||
|
||||
Examines observation categories and relation types across all notes
|
||||
of the given type. Returns frequency analysis and suggested Picoschema.
|
||||
"""
|
||||
entities = await _find_by_entity_type(entity_repository, entity_type)
|
||||
notes_data = [_entity_to_note_data(entity) for entity in entities]
|
||||
|
||||
result = infer_schema(entity_type, notes_data, optional_threshold=threshold)
|
||||
|
||||
return InferenceReport(
|
||||
entity_type=result.entity_type,
|
||||
notes_analyzed=result.notes_analyzed,
|
||||
field_frequencies=[
|
||||
FieldFrequencyResponse(
|
||||
name=f.name,
|
||||
source=f.source,
|
||||
count=f.count,
|
||||
total=f.total,
|
||||
percentage=f.percentage,
|
||||
sample_values=f.sample_values,
|
||||
is_array=f.is_array,
|
||||
target_type=f.target_type,
|
||||
)
|
||||
for f in result.field_frequencies
|
||||
],
|
||||
suggested_schema=result.suggested_schema,
|
||||
suggested_required=result.suggested_required,
|
||||
suggested_optional=result.suggested_optional,
|
||||
excluded=result.excluded,
|
||||
)
|
||||
|
||||
|
||||
# --- Drift Detection ---
|
||||
|
||||
|
||||
@router.get("/schema/diff/{entity_type}", response_model=DriftReport)
|
||||
async def diff_schema_endpoint(
|
||||
entity_repository: EntityRepositoryV2ExternalDep,
|
||||
search_service: SearchServiceV2ExternalDep,
|
||||
entity_type: str = Path(..., description="Entity type to check for drift"),
|
||||
project_id: str = Path(..., description="Project external UUID"),
|
||||
):
|
||||
"""Show drift between a schema definition and actual note usage.
|
||||
|
||||
Compares the existing schema for an entity type against how notes
|
||||
of that type are actually structured. Identifies new fields, dropped
|
||||
fields, and cardinality changes.
|
||||
"""
|
||||
|
||||
async def search_fn(query: str) -> list:
|
||||
# Search for schema notes, then load full entity_metadata from the entity table.
|
||||
# The search index only stores minimal metadata (e.g., {"entity_type": "schema"}),
|
||||
# but parse_schema_note needs the full frontmatter with entity/schema/version keys.
|
||||
results = await search_service.search(SearchQuery(text=query, types=["schema"]), limit=5)
|
||||
frontmatters = []
|
||||
for row in results:
|
||||
if row.permalink:
|
||||
entity = await entity_repository.get_by_permalink(row.permalink)
|
||||
if entity:
|
||||
frontmatters.append(_entity_frontmatter(entity))
|
||||
return frontmatters
|
||||
|
||||
# Resolve schema by entity type
|
||||
schema_frontmatter = {"type": entity_type}
|
||||
schema_def = await resolve_schema(schema_frontmatter, search_fn)
|
||||
|
||||
if not schema_def:
|
||||
return DriftReport(entity_type=entity_type)
|
||||
|
||||
# Collect all notes of this type
|
||||
entities = await _find_by_entity_type(entity_repository, entity_type)
|
||||
notes_data = [_entity_to_note_data(entity) for entity in entities]
|
||||
|
||||
result = diff_schema(schema_def, notes_data)
|
||||
|
||||
return DriftReport(
|
||||
entity_type=entity_type,
|
||||
new_fields=[
|
||||
DriftFieldResponse(
|
||||
name=f.name,
|
||||
source=f.source,
|
||||
count=f.count,
|
||||
total=f.total,
|
||||
percentage=f.percentage,
|
||||
)
|
||||
for f in result.new_fields
|
||||
],
|
||||
dropped_fields=[
|
||||
DriftFieldResponse(
|
||||
name=f.name,
|
||||
source=f.source,
|
||||
count=f.count,
|
||||
total=f.total,
|
||||
percentage=f.percentage,
|
||||
)
|
||||
for f in result.dropped_fields
|
||||
],
|
||||
cardinality_changes=result.cardinality_changes,
|
||||
)
|
||||
|
||||
|
||||
# --- Helpers ---
|
||||
|
||||
|
||||
async def _find_by_entity_type(
|
||||
entity_repository: EntityRepositoryV2ExternalDep,
|
||||
entity_type: str,
|
||||
) -> list[Entity]:
|
||||
"""Find all entities of a given type using the repository's select pattern."""
|
||||
query = entity_repository.select().where(Entity.entity_type == entity_type)
|
||||
result = await entity_repository.execute_query(query)
|
||||
return list(result.scalars().all())
|
||||
|
||||
|
||||
def _to_note_validation_response(result) -> NoteValidationResponse:
|
||||
"""Convert a core ValidationResult to a Pydantic response model."""
|
||||
return NoteValidationResponse(
|
||||
note_identifier=result.note_identifier,
|
||||
schema_entity=result.schema_entity,
|
||||
passed=result.passed,
|
||||
field_results=[
|
||||
FieldResultResponse(
|
||||
field_name=fr.field.name,
|
||||
field_type=fr.field.type,
|
||||
required=fr.field.required,
|
||||
status=fr.status,
|
||||
values=fr.values,
|
||||
message=fr.message,
|
||||
)
|
||||
for fr in result.field_results
|
||||
],
|
||||
unmatched_observations=result.unmatched_observations,
|
||||
unmatched_relations=result.unmatched_relations,
|
||||
warnings=result.warnings,
|
||||
errors=result.errors,
|
||||
)
|
||||
@@ -1,7 +1,7 @@
|
||||
"""CLI commands for basic-memory."""
|
||||
|
||||
from . import status, db, doctor, import_memory_json, mcp, import_claude_conversations
|
||||
from . import import_claude_projects, import_chatgpt, tool, project, format
|
||||
from . import import_claude_projects, import_chatgpt, tool, project, format, schema
|
||||
|
||||
__all__ = [
|
||||
"status",
|
||||
@@ -15,4 +15,5 @@ __all__ = [
|
||||
"tool",
|
||||
"project",
|
||||
"format",
|
||||
"schema",
|
||||
]
|
||||
|
||||
@@ -0,0 +1,336 @@
|
||||
"""Schema management CLI commands for Basic Memory.
|
||||
|
||||
Provides CLI access to schema validation, inference, and drift detection.
|
||||
Registered as a subcommand group: `bm schema validate`, `bm schema infer`, `bm schema diff`.
|
||||
"""
|
||||
|
||||
import json
|
||||
from typing import Annotated, Optional
|
||||
|
||||
import typer
|
||||
from loguru import logger
|
||||
from rich.console import Console
|
||||
from rich.panel import Panel
|
||||
from rich.table import Table
|
||||
|
||||
from basic_memory.cli.app import app
|
||||
from basic_memory.cli.commands.command_utils import run_with_cleanup
|
||||
from basic_memory.cli.commands.routing import force_routing, validate_routing_flags
|
||||
from basic_memory.config import ConfigManager
|
||||
from basic_memory.mcp.async_client import get_client
|
||||
from basic_memory.mcp.project_context import get_active_project
|
||||
|
||||
console = Console()
|
||||
|
||||
schema_app = typer.Typer(help="Schema management commands")
|
||||
app.add_typer(schema_app, name="schema")
|
||||
|
||||
|
||||
def _resolve_project_name(project: Optional[str]) -> Optional[str]:
|
||||
"""Resolve project name from CLI argument or config default."""
|
||||
config_manager = ConfigManager()
|
||||
if project is not None:
|
||||
project_name, _ = config_manager.get_project(project)
|
||||
if not project_name:
|
||||
typer.echo(f"No project found named: {project}", err=True)
|
||||
raise typer.Exit(1)
|
||||
return project_name
|
||||
return config_manager.default_project
|
||||
|
||||
|
||||
# --- Validate ---
|
||||
|
||||
|
||||
async def _run_validate(
|
||||
target: Optional[str] = None,
|
||||
project: Optional[str] = None,
|
||||
strict: bool = False,
|
||||
):
|
||||
"""Run schema validation via the API."""
|
||||
from basic_memory.mcp.clients.schema import SchemaClient
|
||||
|
||||
async with get_client() as client:
|
||||
active_project = await get_active_project(client, project, None)
|
||||
schema_client = SchemaClient(client, active_project.external_id)
|
||||
|
||||
# Determine if target is a note identifier or entity type
|
||||
# Heuristic: if target contains / or ., treat as identifier
|
||||
entity_type = None
|
||||
identifier = None
|
||||
if target:
|
||||
if "/" in target or "." in target:
|
||||
identifier = target
|
||||
else:
|
||||
entity_type = target
|
||||
|
||||
report = await schema_client.validate(
|
||||
entity_type=entity_type,
|
||||
identifier=identifier,
|
||||
)
|
||||
|
||||
# --- Display results ---
|
||||
if report.total_notes == 0:
|
||||
console.print("[yellow]No notes matched for validation.[/yellow]")
|
||||
return
|
||||
|
||||
table = Table(title=f"Schema Validation: {entity_type or identifier or 'all'}")
|
||||
table.add_column("Note", style="cyan")
|
||||
table.add_column("Status", justify="center")
|
||||
table.add_column("Warnings", justify="right")
|
||||
table.add_column("Errors", justify="right")
|
||||
|
||||
for result in report.results:
|
||||
if result.passed and not result.warnings:
|
||||
status = "[green]pass[/green]"
|
||||
elif result.passed:
|
||||
status = "[yellow]warn[/yellow]"
|
||||
else:
|
||||
status = "[red]fail[/red]"
|
||||
|
||||
table.add_row(
|
||||
result.note_identifier,
|
||||
status,
|
||||
str(len(result.warnings)),
|
||||
str(len(result.errors)),
|
||||
)
|
||||
|
||||
console.print(table)
|
||||
console.print(
|
||||
f"\nSummary: {report.valid_count}/{report.total_notes} valid, "
|
||||
f"{report.warning_count} warnings, {report.error_count} errors"
|
||||
)
|
||||
|
||||
# Exit with error code in strict mode if there are failures
|
||||
if strict and report.error_count > 0:
|
||||
raise typer.Exit(1)
|
||||
|
||||
|
||||
@schema_app.command()
|
||||
def validate(
|
||||
target: Annotated[
|
||||
Optional[str],
|
||||
typer.Argument(help="Note path or entity type to validate"),
|
||||
] = None,
|
||||
project: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(help="The project name."),
|
||||
] = None,
|
||||
strict: bool = typer.Option(False, "--strict", help="Exit with error on validation failures"),
|
||||
local: bool = typer.Option(
|
||||
False, "--local", help="Force local API routing (ignore cloud mode)"
|
||||
),
|
||||
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
|
||||
):
|
||||
"""Validate notes against their schemas.
|
||||
|
||||
TARGET can be a note path (e.g., people/ada-lovelace.md) or an entity type
|
||||
(e.g., Person). If omitted, validates all notes that have schemas.
|
||||
|
||||
Use --strict to exit with error code 1 if any validation errors are found.
|
||||
Use --local to force local routing when cloud mode is enabled.
|
||||
Use --cloud to force cloud routing when cloud mode is disabled.
|
||||
"""
|
||||
try:
|
||||
validate_routing_flags(local, cloud)
|
||||
project_name = _resolve_project_name(project)
|
||||
with force_routing(local=local, cloud=cloud):
|
||||
run_with_cleanup(_run_validate(target, project_name, strict))
|
||||
except ValueError as e:
|
||||
console.print(f"[red]Error: {e}[/red]")
|
||||
raise typer.Exit(1)
|
||||
except Exception as e:
|
||||
if not isinstance(e, typer.Exit):
|
||||
logger.error(f"Error during schema validate: {e}")
|
||||
typer.echo(f"Error during schema validate: {e}", err=True)
|
||||
raise typer.Exit(1)
|
||||
raise
|
||||
|
||||
|
||||
# --- Infer ---
|
||||
|
||||
|
||||
async def _run_infer(
|
||||
entity_type: str,
|
||||
project: Optional[str] = None,
|
||||
threshold: float = 0.25,
|
||||
save: bool = False,
|
||||
):
|
||||
"""Run schema inference via the API."""
|
||||
from basic_memory.mcp.clients.schema import SchemaClient
|
||||
|
||||
async with get_client() as client:
|
||||
active_project = await get_active_project(client, project, None)
|
||||
schema_client = SchemaClient(client, active_project.external_id)
|
||||
|
||||
report = await schema_client.infer(entity_type, threshold=threshold)
|
||||
|
||||
if report.notes_analyzed == 0:
|
||||
console.print(f"[yellow]No notes found with type: {entity_type}[/yellow]")
|
||||
return
|
||||
|
||||
# --- Display frequency analysis ---
|
||||
console.print(
|
||||
f"\n[bold]Analyzing {report.notes_analyzed} notes with type: {entity_type}...[/bold]\n"
|
||||
)
|
||||
|
||||
table = Table(title="Field Frequencies")
|
||||
table.add_column("Field", style="cyan")
|
||||
table.add_column("Source")
|
||||
table.add_column("Count", justify="right")
|
||||
table.add_column("Percentage", justify="right")
|
||||
table.add_column("Suggested")
|
||||
|
||||
for freq in report.field_frequencies:
|
||||
pct = f"{freq.percentage:.0%}"
|
||||
if freq.name in report.suggested_required:
|
||||
suggested = "[green]required[/green]"
|
||||
elif freq.name in report.suggested_optional:
|
||||
suggested = "[yellow]optional[/yellow]"
|
||||
else:
|
||||
suggested = "[dim]excluded[/dim]"
|
||||
|
||||
table.add_row(
|
||||
freq.name,
|
||||
freq.source,
|
||||
str(freq.count),
|
||||
pct,
|
||||
suggested,
|
||||
)
|
||||
|
||||
console.print(table)
|
||||
|
||||
# --- Display suggested schema ---
|
||||
console.print("\n[bold]Suggested schema:[/bold]")
|
||||
console.print(Panel(json.dumps(report.suggested_schema, indent=2), title="Picoschema"))
|
||||
|
||||
if save:
|
||||
console.print(
|
||||
f"\n[yellow]--save not yet implemented. "
|
||||
f"Copy the schema above into schema/{entity_type}.md[/yellow]"
|
||||
)
|
||||
|
||||
|
||||
@schema_app.command()
|
||||
def infer(
|
||||
entity_type: Annotated[
|
||||
str,
|
||||
typer.Argument(help="Entity type to analyze (e.g., Person, meeting)"),
|
||||
],
|
||||
project: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(help="The project name."),
|
||||
] = None,
|
||||
threshold: float = typer.Option(
|
||||
0.25, "--threshold", help="Minimum frequency for optional fields (0-1)"
|
||||
),
|
||||
save: bool = typer.Option(False, "--save", help="Save inferred schema to schema/ directory"),
|
||||
local: bool = typer.Option(
|
||||
False, "--local", help="Force local API routing (ignore cloud mode)"
|
||||
),
|
||||
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
|
||||
):
|
||||
"""Infer schema from existing notes of a type.
|
||||
|
||||
Analyzes all notes with the given entity type and suggests a Picoschema
|
||||
definition based on observation and relation frequency.
|
||||
|
||||
Fields present in 95%+ of notes become required. Fields above the
|
||||
threshold (default 25%) become optional. Fields below threshold are excluded.
|
||||
|
||||
Use --local to force local routing when cloud mode is enabled.
|
||||
Use --cloud to force cloud routing when cloud mode is disabled.
|
||||
"""
|
||||
try:
|
||||
validate_routing_flags(local, cloud)
|
||||
project_name = _resolve_project_name(project)
|
||||
with force_routing(local=local, cloud=cloud):
|
||||
run_with_cleanup(_run_infer(entity_type, project_name, threshold, save))
|
||||
except ValueError as e:
|
||||
console.print(f"[red]Error: {e}[/red]")
|
||||
raise typer.Exit(1)
|
||||
except Exception as e:
|
||||
if not isinstance(e, typer.Exit):
|
||||
logger.error(f"Error during schema infer: {e}")
|
||||
typer.echo(f"Error during schema infer: {e}", err=True)
|
||||
raise typer.Exit(1)
|
||||
raise
|
||||
|
||||
|
||||
# --- Diff ---
|
||||
|
||||
|
||||
async def _run_diff(
|
||||
entity_type: str,
|
||||
project: Optional[str] = None,
|
||||
):
|
||||
"""Run schema drift detection via the API."""
|
||||
from basic_memory.mcp.clients.schema import SchemaClient
|
||||
|
||||
async with get_client() as client:
|
||||
active_project = await get_active_project(client, project, None)
|
||||
schema_client = SchemaClient(client, active_project.external_id)
|
||||
|
||||
report = await schema_client.diff(entity_type)
|
||||
|
||||
has_drift = report.new_fields or report.dropped_fields or report.cardinality_changes
|
||||
|
||||
if not has_drift:
|
||||
console.print(f"[green]No drift detected for {entity_type} schema.[/green]")
|
||||
return
|
||||
|
||||
console.print(f"\n[bold]Schema drift detected for {entity_type}:[/bold]\n")
|
||||
|
||||
if report.new_fields:
|
||||
console.print("[green]+ New fields (common in notes, not in schema):[/green]")
|
||||
for f in report.new_fields:
|
||||
console.print(f" + {f.name}: {f.percentage:.0%} of notes ({f.source})")
|
||||
|
||||
if report.dropped_fields:
|
||||
console.print("[red]- Dropped fields (in schema, rare in notes):[/red]")
|
||||
for f in report.dropped_fields:
|
||||
console.print(f" - {f.name}: {f.percentage:.0%} of notes ({f.source})")
|
||||
|
||||
if report.cardinality_changes:
|
||||
console.print("[yellow]~ Cardinality changes:[/yellow]")
|
||||
for change in report.cardinality_changes:
|
||||
console.print(f" ~ {change}")
|
||||
|
||||
|
||||
@schema_app.command()
|
||||
def diff(
|
||||
entity_type: Annotated[
|
||||
str,
|
||||
typer.Argument(help="Entity type to check for drift"),
|
||||
],
|
||||
project: Annotated[
|
||||
Optional[str],
|
||||
typer.Option(help="The project name."),
|
||||
] = None,
|
||||
local: bool = typer.Option(
|
||||
False, "--local", help="Force local API routing (ignore cloud mode)"
|
||||
),
|
||||
cloud: bool = typer.Option(False, "--cloud", help="Force cloud API routing"),
|
||||
):
|
||||
"""Show drift between schema and actual usage.
|
||||
|
||||
Compares the existing schema definition for an entity type against
|
||||
how notes of that type are actually structured. Identifies new fields,
|
||||
dropped fields, and cardinality changes.
|
||||
|
||||
Use --local to force local routing when cloud mode is enabled.
|
||||
Use --cloud to force cloud routing when cloud mode is disabled.
|
||||
"""
|
||||
try:
|
||||
validate_routing_flags(local, cloud)
|
||||
project_name = _resolve_project_name(project)
|
||||
with force_routing(local=local, cloud=cloud):
|
||||
run_with_cleanup(_run_diff(entity_type, project_name))
|
||||
except ValueError as e:
|
||||
console.print(f"[red]Error: {e}[/red]")
|
||||
raise typer.Exit(1)
|
||||
except Exception as e:
|
||||
if not isinstance(e, typer.Exit):
|
||||
logger.error(f"Error during schema diff: {e}")
|
||||
typer.echo(f"Error during schema diff: {e}", err=True)
|
||||
raise typer.Exit(1)
|
||||
raise
|
||||
@@ -13,6 +13,7 @@ from basic_memory.cli.commands import ( # noqa: F401 # pragma: no cover
|
||||
import_memory_json,
|
||||
mcp,
|
||||
project,
|
||||
schema,
|
||||
status,
|
||||
tool,
|
||||
)
|
||||
|
||||
@@ -17,6 +17,7 @@ from basic_memory.mcp.clients.memory import MemoryClient
|
||||
from basic_memory.mcp.clients.directory import DirectoryClient
|
||||
from basic_memory.mcp.clients.resource import ResourceClient
|
||||
from basic_memory.mcp.clients.project import ProjectClient
|
||||
from basic_memory.mcp.clients.schema import SchemaClient
|
||||
|
||||
__all__ = [
|
||||
"KnowledgeClient",
|
||||
@@ -25,4 +26,5 @@ __all__ = [
|
||||
"DirectoryClient",
|
||||
"ResourceClient",
|
||||
"ProjectClient",
|
||||
"SchemaClient",
|
||||
]
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
"""Typed client for schema API operations.
|
||||
|
||||
Encapsulates all /v2/projects/{project_id}/schema/* endpoints.
|
||||
"""
|
||||
|
||||
from httpx import AsyncClient
|
||||
|
||||
from basic_memory.mcp.tools.utils import call_post, call_get
|
||||
from basic_memory.schemas.schema import (
|
||||
ValidationReport,
|
||||
InferenceReport,
|
||||
DriftReport,
|
||||
)
|
||||
|
||||
|
||||
class SchemaClient:
|
||||
"""Typed client for schema operations.
|
||||
|
||||
Centralizes:
|
||||
- API path construction for /v2/projects/{project_id}/schema/*
|
||||
- Response validation via Pydantic models
|
||||
- Consistent error handling through call_* utilities
|
||||
|
||||
Usage:
|
||||
async with get_client() as http_client:
|
||||
client = SchemaClient(http_client, project_id)
|
||||
report = await client.validate(entity_type="Person")
|
||||
"""
|
||||
|
||||
def __init__(self, http_client: AsyncClient, project_id: str):
|
||||
"""Initialize the schema client.
|
||||
|
||||
Args:
|
||||
http_client: HTTPX AsyncClient for making requests
|
||||
project_id: Project external_id (UUID) for API calls
|
||||
"""
|
||||
self.http_client = http_client
|
||||
self.project_id = project_id
|
||||
self._base_path = f"/v2/projects/{project_id}/schema"
|
||||
|
||||
async def validate(
|
||||
self,
|
||||
*,
|
||||
entity_type: str | None = None,
|
||||
identifier: str | None = None,
|
||||
) -> ValidationReport:
|
||||
"""Validate notes against their resolved schemas.
|
||||
|
||||
Args:
|
||||
entity_type: Optional entity type to batch-validate
|
||||
identifier: Optional specific note to validate
|
||||
|
||||
Returns:
|
||||
ValidationReport with per-note results
|
||||
|
||||
Raises:
|
||||
ToolError: If the request fails
|
||||
"""
|
||||
params: dict[str, str] = {}
|
||||
if entity_type:
|
||||
params["entity_type"] = entity_type
|
||||
if identifier:
|
||||
params["identifier"] = identifier
|
||||
|
||||
response = await call_post(
|
||||
self.http_client,
|
||||
f"{self._base_path}/validate",
|
||||
params=params,
|
||||
)
|
||||
return ValidationReport.model_validate(response.json())
|
||||
|
||||
async def infer(
|
||||
self,
|
||||
entity_type: str,
|
||||
*,
|
||||
threshold: float = 0.25,
|
||||
) -> InferenceReport:
|
||||
"""Infer a schema from existing notes of a given type.
|
||||
|
||||
Args:
|
||||
entity_type: The entity type to analyze
|
||||
threshold: Minimum frequency for optional fields (0-1)
|
||||
|
||||
Returns:
|
||||
InferenceReport with frequency data and suggested schema
|
||||
|
||||
Raises:
|
||||
ToolError: If the request fails
|
||||
"""
|
||||
response = await call_post(
|
||||
self.http_client,
|
||||
f"{self._base_path}/infer",
|
||||
params={"entity_type": entity_type, "threshold": threshold},
|
||||
)
|
||||
return InferenceReport.model_validate(response.json())
|
||||
|
||||
async def diff(self, entity_type: str) -> DriftReport:
|
||||
"""Show drift between schema definition and actual usage.
|
||||
|
||||
Args:
|
||||
entity_type: The entity type to check for drift
|
||||
|
||||
Returns:
|
||||
DriftReport with detected differences
|
||||
|
||||
Raises:
|
||||
ToolError: If the request fails
|
||||
"""
|
||||
response = await call_get(
|
||||
self.http_client,
|
||||
f"{self._base_path}/diff/{entity_type}",
|
||||
)
|
||||
return DriftReport.model_validate(response.json())
|
||||
@@ -27,6 +27,9 @@ from basic_memory.mcp.tools.project_management import (
|
||||
# ChatGPT-compatible tools
|
||||
from basic_memory.mcp.tools.chatgpt_tools import search, fetch
|
||||
|
||||
# Schema tools
|
||||
from basic_memory.mcp.tools.schema import schema_validate, schema_infer, schema_diff
|
||||
|
||||
__all__ = [
|
||||
"build_context",
|
||||
"canvas",
|
||||
@@ -41,6 +44,9 @@ __all__ = [
|
||||
"read_content",
|
||||
"read_note",
|
||||
"recent_activity",
|
||||
"schema_diff",
|
||||
"schema_infer",
|
||||
"schema_validate",
|
||||
"search",
|
||||
"search_by_metadata",
|
||||
"search_notes",
|
||||
|
||||
@@ -0,0 +1,244 @@
|
||||
"""Schema tools for Basic Memory MCP server.
|
||||
|
||||
Provides tools for schema validation, inference, and drift detection through the MCP protocol.
|
||||
These tools call the schema API endpoints via the typed SchemaClient.
|
||||
"""
|
||||
|
||||
from typing import Optional
|
||||
|
||||
from loguru import logger
|
||||
from fastmcp import Context
|
||||
|
||||
from basic_memory.mcp.async_client import get_client
|
||||
from basic_memory.mcp.project_context import get_active_project
|
||||
from basic_memory.mcp.server import mcp
|
||||
from basic_memory.schemas.schema import ValidationReport, InferenceReport, DriftReport
|
||||
|
||||
|
||||
@mcp.tool(
|
||||
description="Validate notes against their Picoschema definitions.",
|
||||
)
|
||||
async def schema_validate(
|
||||
entity_type: Optional[str] = None,
|
||||
identifier: Optional[str] = None,
|
||||
project: Optional[str] = None,
|
||||
context: Context | None = None,
|
||||
) -> ValidationReport | str:
|
||||
"""Validate notes against their resolved schema.
|
||||
|
||||
Validates a specific note (by identifier) or all notes of a given type.
|
||||
Returns warnings/errors based on the schema's validation mode.
|
||||
|
||||
Schemas are resolved in priority order:
|
||||
1. Inline schema (dict in frontmatter)
|
||||
2. Explicit reference (string in frontmatter)
|
||||
3. Implicit by type (type field matches schema note entity field)
|
||||
4. No schema (no validation)
|
||||
|
||||
Project Resolution:
|
||||
Server resolves projects in this order: Single Project Mode -> project parameter -> default.
|
||||
If project unknown, use list_memory_projects() first.
|
||||
|
||||
Args:
|
||||
entity_type: Entity type to batch-validate (e.g., "Person").
|
||||
If provided, validates all notes of this type.
|
||||
identifier: Specific note to validate (permalink, title, or path).
|
||||
If provided, validates only this note.
|
||||
project: Project name. Optional -- server will resolve.
|
||||
context: Optional FastMCP context for performance caching.
|
||||
|
||||
Returns:
|
||||
ValidationReport with per-note results, or error guidance string
|
||||
|
||||
Examples:
|
||||
# Validate all Person notes
|
||||
schema_validate(entity_type="Person")
|
||||
|
||||
# Validate a specific note
|
||||
schema_validate(identifier="people/paul-graham")
|
||||
|
||||
# Validate in a specific project
|
||||
schema_validate(entity_type="Person", project="my-research")
|
||||
"""
|
||||
async with get_client() as client:
|
||||
active_project = await get_active_project(client, project, context)
|
||||
logger.info(
|
||||
f"MCP tool call tool=schema_validate project={active_project.name} "
|
||||
f"entity_type={entity_type} identifier={identifier}"
|
||||
)
|
||||
|
||||
try:
|
||||
from basic_memory.mcp.clients.schema import SchemaClient
|
||||
|
||||
schema_client = SchemaClient(client, active_project.external_id)
|
||||
result = await schema_client.validate(
|
||||
entity_type=entity_type,
|
||||
identifier=identifier,
|
||||
)
|
||||
|
||||
logger.info(
|
||||
f"MCP tool response: tool=schema_validate project={active_project.name} "
|
||||
f"total={result.total_notes} valid={result.valid_count} "
|
||||
f"warnings={result.warning_count} errors={result.error_count}"
|
||||
)
|
||||
return result
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Schema validation failed: {e}, project: {active_project.name}")
|
||||
return (
|
||||
f"# Schema Validation Failed\n\n"
|
||||
f"Error validating schemas: {e}\n\n"
|
||||
f"## Troubleshooting\n"
|
||||
f"1. Ensure schema notes exist (type: schema) for the target entity type\n"
|
||||
f"2. Check that notes have the correct type in frontmatter\n"
|
||||
f"3. Verify the project has been synced: `basic-memory status`\n"
|
||||
)
|
||||
|
||||
|
||||
@mcp.tool(
|
||||
description="Analyze existing notes and suggest a Picoschema definition.",
|
||||
)
|
||||
async def schema_infer(
|
||||
entity_type: str,
|
||||
threshold: float = 0.25,
|
||||
project: Optional[str] = None,
|
||||
context: Context | None = None,
|
||||
) -> InferenceReport | str:
|
||||
"""Analyze existing notes and suggest a schema definition.
|
||||
|
||||
Examines observation categories and relation types across all notes
|
||||
of the given type. Returns frequency analysis and suggested Picoschema
|
||||
YAML that can be saved as a schema note.
|
||||
|
||||
Frequency thresholds:
|
||||
- 95%+ present -> required field
|
||||
- threshold+ present -> optional field
|
||||
- Below threshold -> excluded (but noted)
|
||||
|
||||
Project Resolution:
|
||||
Server resolves projects in this order: Single Project Mode -> project parameter -> default.
|
||||
If project unknown, use list_memory_projects() first.
|
||||
|
||||
Args:
|
||||
entity_type: The entity type to analyze (e.g., "Person", "meeting").
|
||||
threshold: Minimum frequency (0-1) for a field to be suggested as optional.
|
||||
Default 0.25 (25%). Fields above 95% become required.
|
||||
project: Project name. Optional -- server will resolve.
|
||||
context: Optional FastMCP context for performance caching.
|
||||
|
||||
Returns:
|
||||
InferenceReport with frequency data and suggested schema, or error string
|
||||
|
||||
Examples:
|
||||
# Infer schema for Person notes
|
||||
schema_infer("Person")
|
||||
|
||||
# Use a higher threshold (50% minimum)
|
||||
schema_infer("meeting", threshold=0.5)
|
||||
|
||||
# Infer in a specific project
|
||||
schema_infer("Person", project="my-research")
|
||||
"""
|
||||
async with get_client() as client:
|
||||
active_project = await get_active_project(client, project, context)
|
||||
logger.info(
|
||||
f"MCP tool call tool=schema_infer project={active_project.name} "
|
||||
f"entity_type={entity_type} threshold={threshold}"
|
||||
)
|
||||
|
||||
try:
|
||||
from basic_memory.mcp.clients.schema import SchemaClient
|
||||
|
||||
schema_client = SchemaClient(client, active_project.external_id)
|
||||
result = await schema_client.infer(entity_type, threshold=threshold)
|
||||
|
||||
logger.info(
|
||||
f"MCP tool response: tool=schema_infer project={active_project.name} "
|
||||
f"entity_type={entity_type} notes_analyzed={result.notes_analyzed} "
|
||||
f"required={len(result.suggested_required)} "
|
||||
f"optional={len(result.suggested_optional)}"
|
||||
)
|
||||
return result
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Schema inference failed: {e}, project: {active_project.name}")
|
||||
return (
|
||||
f"# Schema Inference Failed\n\n"
|
||||
f"Error inferring schema for '{entity_type}': {e}\n\n"
|
||||
f"## Troubleshooting\n"
|
||||
f"1. Ensure notes of type '{entity_type}' exist in the project\n"
|
||||
f'2. Try searching: `search_notes("{entity_type}", types=["{entity_type}"])`\n'
|
||||
f"3. Verify the project has been synced: `basic-memory status`\n"
|
||||
)
|
||||
|
||||
|
||||
@mcp.tool(
|
||||
description="Detect drift between a schema definition and actual note usage.",
|
||||
)
|
||||
async def schema_diff(
|
||||
entity_type: str,
|
||||
project: Optional[str] = None,
|
||||
context: Context | None = None,
|
||||
) -> DriftReport | str:
|
||||
"""Detect drift between a schema definition and actual note usage.
|
||||
|
||||
Compares the existing schema for an entity type against how notes of
|
||||
that type are actually structured. Identifies new fields that have
|
||||
appeared, declared fields that are rarely used, and cardinality changes
|
||||
(single-value vs array).
|
||||
|
||||
Useful for evolving schemas as your knowledge base grows -- run
|
||||
periodically to see if your schema still matches reality.
|
||||
|
||||
Project Resolution:
|
||||
Server resolves projects in this order: Single Project Mode -> project parameter -> default.
|
||||
If project unknown, use list_memory_projects() first.
|
||||
|
||||
Args:
|
||||
entity_type: The entity type to check for drift (e.g., "Person").
|
||||
project: Project name. Optional -- server will resolve.
|
||||
context: Optional FastMCP context for performance caching.
|
||||
|
||||
Returns:
|
||||
DriftReport with new fields, dropped fields, and cardinality changes,
|
||||
or error guidance string
|
||||
|
||||
Examples:
|
||||
# Check drift for Person schema
|
||||
schema_diff("Person")
|
||||
|
||||
# Check drift in a specific project
|
||||
schema_diff("Person", project="my-research")
|
||||
"""
|
||||
async with get_client() as client:
|
||||
active_project = await get_active_project(client, project, context)
|
||||
logger.info(
|
||||
f"MCP tool call tool=schema_diff project={active_project.name} "
|
||||
f"entity_type={entity_type}"
|
||||
)
|
||||
|
||||
try:
|
||||
from basic_memory.mcp.clients.schema import SchemaClient
|
||||
|
||||
schema_client = SchemaClient(client, active_project.external_id)
|
||||
result = await schema_client.diff(entity_type)
|
||||
|
||||
logger.info(
|
||||
f"MCP tool response: tool=schema_diff project={active_project.name} "
|
||||
f"entity_type={entity_type} "
|
||||
f"new_fields={len(result.new_fields)} "
|
||||
f"dropped_fields={len(result.dropped_fields)} "
|
||||
f"cardinality_changes={len(result.cardinality_changes)}"
|
||||
)
|
||||
return result
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Schema diff failed: {e}, project: {active_project.name}")
|
||||
return (
|
||||
f"# Schema Diff Failed\n\n"
|
||||
f"Error detecting drift for '{entity_type}': {e}\n\n"
|
||||
f"## Troubleshooting\n"
|
||||
f"1. Ensure a schema note exists for entity type '{entity_type}'\n"
|
||||
f"2. Ensure notes of type '{entity_type}' exist in the project\n"
|
||||
f"3. Verify the project has been synced: `basic-memory status`\n"
|
||||
)
|
||||
@@ -0,0 +1,58 @@
|
||||
"""Schema system for Basic Memory.
|
||||
|
||||
Provides Picoschema-based validation for notes using observation/relation mapping.
|
||||
Schemas are just notes with type: schema — no new data model, no migration.
|
||||
"""
|
||||
|
||||
from basic_memory.schema.parser import (
|
||||
SchemaField,
|
||||
SchemaDefinition,
|
||||
parse_picoschema,
|
||||
parse_schema_note,
|
||||
)
|
||||
from basic_memory.schema.resolver import resolve_schema
|
||||
from basic_memory.schema.validator import (
|
||||
FieldResult,
|
||||
ValidationResult,
|
||||
validate_note,
|
||||
)
|
||||
from basic_memory.schema.inference import (
|
||||
FieldFrequency,
|
||||
InferenceResult,
|
||||
ObservationData,
|
||||
RelationData,
|
||||
NoteData,
|
||||
infer_schema,
|
||||
analyze_observations,
|
||||
analyze_relations,
|
||||
)
|
||||
from basic_memory.schema.diff import (
|
||||
SchemaDrift,
|
||||
diff_schema,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
# Parser
|
||||
"SchemaField",
|
||||
"SchemaDefinition",
|
||||
"parse_picoschema",
|
||||
"parse_schema_note",
|
||||
# Resolver
|
||||
"resolve_schema",
|
||||
# Validator
|
||||
"FieldResult",
|
||||
"ValidationResult",
|
||||
"validate_note",
|
||||
# Inference
|
||||
"FieldFrequency",
|
||||
"InferenceResult",
|
||||
"ObservationData",
|
||||
"RelationData",
|
||||
"NoteData",
|
||||
"infer_schema",
|
||||
"analyze_observations",
|
||||
"analyze_relations",
|
||||
# Diff
|
||||
"SchemaDrift",
|
||||
"diff_schema",
|
||||
]
|
||||
@@ -0,0 +1,111 @@
|
||||
"""Schema diff for Basic Memory.
|
||||
|
||||
Compares a schema definition against actual note usage to detect drift.
|
||||
Drift happens naturally as notes evolve -- new observation categories appear,
|
||||
old ones fall out of use, single-value fields become multi-value.
|
||||
|
||||
The diff engine reuses inference analysis internally, comparing inferred
|
||||
frequencies against the declared schema fields to surface:
|
||||
- New fields: common in notes but not declared in schema
|
||||
- Dropped fields: declared in schema but rare in actual notes
|
||||
- Cardinality changes: field changed from single to array or vice versa
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from basic_memory.schema.inference import (
|
||||
FieldFrequency,
|
||||
NoteData,
|
||||
analyze_observations,
|
||||
analyze_relations,
|
||||
)
|
||||
from basic_memory.schema.parser import SchemaDefinition
|
||||
|
||||
|
||||
@dataclass
|
||||
class SchemaDrift:
|
||||
"""Result of comparing a schema against actual note usage."""
|
||||
|
||||
new_fields: list[FieldFrequency] = field(default_factory=list)
|
||||
dropped_fields: list[FieldFrequency] = field(default_factory=list)
|
||||
cardinality_changes: list[str] = field(default_factory=list)
|
||||
|
||||
|
||||
def diff_schema(
|
||||
schema: SchemaDefinition,
|
||||
notes: list[NoteData],
|
||||
new_field_threshold: float = 0.25,
|
||||
dropped_field_threshold: float = 0.10,
|
||||
) -> SchemaDrift:
|
||||
"""Compare a schema against actual note usage to detect drift.
|
||||
|
||||
Args:
|
||||
schema: The current schema definition.
|
||||
notes: List of NoteData objects representing actual notes.
|
||||
new_field_threshold: Frequency above which an undeclared field is considered
|
||||
"new" and worth adding to the schema.
|
||||
dropped_field_threshold: Frequency below which a declared field is considered
|
||||
"dropped" and worth removing from the schema.
|
||||
|
||||
Returns:
|
||||
A SchemaDrift describing the differences between schema and reality.
|
||||
"""
|
||||
total = len(notes)
|
||||
if total == 0:
|
||||
return SchemaDrift()
|
||||
|
||||
# --- Analyze actual usage ---
|
||||
obs_frequencies = analyze_observations(notes, total, max_sample_values=3)
|
||||
rel_frequencies = analyze_relations(notes, total, max_sample_values=3)
|
||||
|
||||
# Build lookup from schema fields
|
||||
schema_field_names = {f.name for f in schema.fields}
|
||||
|
||||
# Build lookup from actual frequencies
|
||||
obs_freq_by_name = {f.name: f for f in obs_frequencies}
|
||||
rel_freq_by_name = {f.name: f for f in rel_frequencies}
|
||||
all_freq_by_name = {**obs_freq_by_name, **rel_freq_by_name}
|
||||
|
||||
result = SchemaDrift()
|
||||
|
||||
# --- Detect new fields ---
|
||||
# Fields that appear frequently in notes but aren't declared in the schema
|
||||
for freq in obs_frequencies + rel_frequencies:
|
||||
if freq.name not in schema_field_names and freq.percentage >= new_field_threshold:
|
||||
result.new_fields.append(freq)
|
||||
|
||||
# --- Detect dropped fields ---
|
||||
# Fields declared in the schema but rarely appearing in actual notes
|
||||
for schema_field in schema.fields:
|
||||
freq = all_freq_by_name.get(schema_field.name)
|
||||
if freq is None:
|
||||
# Field doesn't appear at all in any note
|
||||
result.dropped_fields.append(
|
||||
FieldFrequency(
|
||||
name=schema_field.name,
|
||||
source="relation" if schema_field.is_entity_ref else "observation",
|
||||
count=0,
|
||||
total=total,
|
||||
percentage=0.0,
|
||||
)
|
||||
)
|
||||
elif freq.percentage < dropped_field_threshold:
|
||||
result.dropped_fields.append(freq)
|
||||
|
||||
# --- Detect cardinality changes ---
|
||||
# Fields where the schema says single but usage shows array, or vice versa
|
||||
for schema_field in schema.fields:
|
||||
freq = all_freq_by_name.get(schema_field.name)
|
||||
if freq is None:
|
||||
continue
|
||||
|
||||
if schema_field.is_array and not freq.is_array:
|
||||
result.cardinality_changes.append(
|
||||
f"{schema_field.name}: schema declares array but usage is typically single-value"
|
||||
)
|
||||
elif not schema_field.is_array and freq.is_array:
|
||||
result.cardinality_changes.append(
|
||||
f"{schema_field.name}: schema declares single-value but usage is typically array"
|
||||
)
|
||||
|
||||
return result
|
||||
@@ -0,0 +1,323 @@
|
||||
"""Schema inference engine for Basic Memory.
|
||||
|
||||
Analyzes notes of a given type and suggests a schema based on observation
|
||||
and relation frequency. Instead of requiring users to define schemas upfront,
|
||||
schemas emerge from actual usage patterns:
|
||||
|
||||
Write notes freely -> Patterns emerge -> Crystallize into schema
|
||||
|
||||
Frequency thresholds:
|
||||
- 95%+ present -> required field
|
||||
- 25%+ present -> optional field
|
||||
- Below 25% -> excluded from suggestion (but noted)
|
||||
"""
|
||||
|
||||
from collections import Counter
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
|
||||
# --- Result Data Model ---
|
||||
|
||||
|
||||
@dataclass
|
||||
class FieldFrequency:
|
||||
"""Frequency analysis for a single field across notes of a type."""
|
||||
|
||||
name: str
|
||||
source: str # "observation" | "relation"
|
||||
count: int # notes containing this field
|
||||
total: int # total notes analyzed
|
||||
percentage: float
|
||||
sample_values: list[str] = field(default_factory=list)
|
||||
is_array: bool = False # True if typically appears multiple times per note
|
||||
target_type: str | None = None # For relations, the most common target entity type
|
||||
|
||||
|
||||
@dataclass
|
||||
class InferenceResult:
|
||||
"""Complete inference result with frequency analysis and suggested schema."""
|
||||
|
||||
entity_type: str
|
||||
notes_analyzed: int
|
||||
field_frequencies: list[FieldFrequency]
|
||||
suggested_schema: dict # Ready-to-use Picoschema YAML dict
|
||||
suggested_required: list[str]
|
||||
suggested_optional: list[str]
|
||||
excluded: list[str] # Below threshold
|
||||
|
||||
|
||||
# --- Note Data Abstraction ---
|
||||
# Instead of depending on the ORM Entity model, we accept simple data structures.
|
||||
# This keeps the inference engine decoupled from the data access layer.
|
||||
|
||||
|
||||
@dataclass
|
||||
class ObservationData:
|
||||
"""Lightweight observation for schema analysis. Decoupled from ORM."""
|
||||
|
||||
category: str
|
||||
content: str
|
||||
|
||||
|
||||
@dataclass
|
||||
class RelationData:
|
||||
"""Lightweight relation for schema analysis. Decoupled from ORM."""
|
||||
|
||||
relation_type: str
|
||||
target_name: str
|
||||
target_entity_type: str | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class NoteData:
|
||||
"""Minimal note representation for inference analysis.
|
||||
|
||||
Decoupled from ORM models so the inference engine can work with
|
||||
any data source (database, files, API responses).
|
||||
"""
|
||||
|
||||
identifier: str
|
||||
observations: list[ObservationData]
|
||||
relations: list[RelationData]
|
||||
|
||||
|
||||
# --- Inference Logic ---
|
||||
|
||||
|
||||
def infer_schema(
|
||||
entity_type: str,
|
||||
notes: list[NoteData],
|
||||
required_threshold: float = 0.95,
|
||||
optional_threshold: float = 0.25,
|
||||
max_sample_values: int = 5,
|
||||
) -> InferenceResult:
|
||||
"""Analyze notes and suggest a Picoschema definition.
|
||||
|
||||
Examines observation categories and relation types across all provided notes.
|
||||
Fields that appear in a high percentage of notes become required; those that
|
||||
appear less frequently become optional.
|
||||
|
||||
Args:
|
||||
entity_type: The entity type being analyzed (e.g., "Person").
|
||||
notes: List of NoteData objects to analyze.
|
||||
required_threshold: Frequency at or above which a field is required (default 0.95).
|
||||
optional_threshold: Frequency at or above which a field is optional (default 0.25).
|
||||
max_sample_values: Maximum number of sample values to include per field.
|
||||
|
||||
Returns:
|
||||
An InferenceResult with frequency analysis and suggested Picoschema dict.
|
||||
"""
|
||||
total = len(notes)
|
||||
if total == 0:
|
||||
return InferenceResult(
|
||||
entity_type=entity_type,
|
||||
notes_analyzed=0,
|
||||
field_frequencies=[],
|
||||
suggested_schema={},
|
||||
suggested_required=[],
|
||||
suggested_optional=[],
|
||||
excluded=[],
|
||||
)
|
||||
|
||||
# --- Analyze observation frequencies ---
|
||||
obs_frequencies = analyze_observations(notes, total, max_sample_values)
|
||||
|
||||
# --- Analyze relation frequencies ---
|
||||
rel_frequencies = analyze_relations(notes, total, max_sample_values)
|
||||
|
||||
# --- Classify fields by threshold ---
|
||||
all_frequencies = obs_frequencies + rel_frequencies
|
||||
suggested_required: list[str] = []
|
||||
suggested_optional: list[str] = []
|
||||
excluded: list[str] = []
|
||||
|
||||
for freq in all_frequencies:
|
||||
if freq.percentage >= required_threshold:
|
||||
suggested_required.append(freq.name)
|
||||
elif freq.percentage >= optional_threshold:
|
||||
suggested_optional.append(freq.name)
|
||||
else:
|
||||
excluded.append(freq.name)
|
||||
|
||||
# --- Build suggested Picoschema dict ---
|
||||
suggested_schema = _build_picoschema_dict(
|
||||
all_frequencies, required_threshold, optional_threshold
|
||||
)
|
||||
|
||||
return InferenceResult(
|
||||
entity_type=entity_type,
|
||||
notes_analyzed=total,
|
||||
field_frequencies=all_frequencies,
|
||||
suggested_schema=suggested_schema,
|
||||
suggested_required=suggested_required,
|
||||
suggested_optional=suggested_optional,
|
||||
excluded=excluded,
|
||||
)
|
||||
|
||||
|
||||
# --- Observation Analysis ---
|
||||
|
||||
|
||||
def analyze_observations(
|
||||
notes: list[NoteData],
|
||||
total: int,
|
||||
max_sample_values: int,
|
||||
) -> list[FieldFrequency]:
|
||||
"""Count observation category frequencies across notes.
|
||||
|
||||
A category is counted once per note (presence), not per occurrence.
|
||||
Array detection: if a category appears multiple times in a single note
|
||||
in more than half the notes where it appears, it's flagged as an array.
|
||||
"""
|
||||
# Count how many notes contain each category (presence per note)
|
||||
category_note_count: Counter[str] = Counter()
|
||||
# Count how many notes have multiple occurrences (for array detection)
|
||||
category_multi_count: Counter[str] = Counter()
|
||||
# Collect sample values
|
||||
category_samples: dict[str, list[str]] = {}
|
||||
|
||||
for note in notes:
|
||||
# Group observations by category within this note
|
||||
note_categories: dict[str, list[str]] = {}
|
||||
for obs in note.observations:
|
||||
note_categories.setdefault(obs.category, []).append(obs.content)
|
||||
|
||||
for category, values in note_categories.items():
|
||||
category_note_count[category] += 1
|
||||
if len(values) > 1:
|
||||
category_multi_count[category] += 1
|
||||
|
||||
# Collect sample values (deduplicated)
|
||||
samples = category_samples.setdefault(category, [])
|
||||
for v in values:
|
||||
if v not in samples and len(samples) < max_sample_values:
|
||||
samples.append(v)
|
||||
|
||||
# Build FieldFrequency objects
|
||||
frequencies: list[FieldFrequency] = []
|
||||
for category, count in category_note_count.most_common():
|
||||
# Array detection: if more than half of notes with this category have
|
||||
# multiple occurrences, treat it as an array field
|
||||
multi_count = category_multi_count.get(category, 0)
|
||||
is_array = multi_count > (count / 2)
|
||||
|
||||
frequencies.append(
|
||||
FieldFrequency(
|
||||
name=category,
|
||||
source="observation",
|
||||
count=count,
|
||||
total=total,
|
||||
percentage=count / total,
|
||||
sample_values=category_samples.get(category, []),
|
||||
is_array=is_array,
|
||||
)
|
||||
)
|
||||
|
||||
return frequencies
|
||||
|
||||
|
||||
# --- Relation Analysis ---
|
||||
|
||||
|
||||
def analyze_relations(
|
||||
notes: list[NoteData],
|
||||
total: int,
|
||||
max_sample_values: int,
|
||||
) -> list[FieldFrequency]:
|
||||
"""Count relation type frequencies across notes.
|
||||
|
||||
Similar to observations, a relation type is counted once per note.
|
||||
Array detection follows the same logic.
|
||||
"""
|
||||
rel_note_count: Counter[str] = Counter()
|
||||
rel_multi_count: Counter[str] = Counter()
|
||||
rel_samples: dict[str, list[str]] = {}
|
||||
# Track target entity types to suggest the type in the schema
|
||||
rel_target_types: dict[str, Counter[str]] = {}
|
||||
|
||||
for note in notes:
|
||||
note_rels: dict[str, list[str]] = {}
|
||||
note_rel_objects: dict[str, list[RelationData]] = {}
|
||||
for rel in note.relations:
|
||||
note_rels.setdefault(rel.relation_type, []).append(rel.target_name)
|
||||
note_rel_objects.setdefault(rel.relation_type, []).append(rel)
|
||||
|
||||
for rel_type, targets in note_rels.items():
|
||||
rel_note_count[rel_type] += 1
|
||||
if len(targets) > 1:
|
||||
rel_multi_count[rel_type] += 1
|
||||
|
||||
samples = rel_samples.setdefault(rel_type, [])
|
||||
for t in targets:
|
||||
if t not in samples and len(samples) < max_sample_values:
|
||||
samples.append(t)
|
||||
|
||||
# Track target entity types from individual relations (not the source note)
|
||||
target_counter = rel_target_types.setdefault(rel_type, Counter())
|
||||
for rel in note_rel_objects[rel_type]:
|
||||
if rel.target_entity_type:
|
||||
target_counter[rel.target_entity_type] += 1
|
||||
|
||||
frequencies: list[FieldFrequency] = []
|
||||
for rel_type, count in rel_note_count.most_common():
|
||||
multi_count = rel_multi_count.get(rel_type, 0)
|
||||
is_array = multi_count > (count / 2)
|
||||
|
||||
# Determine most common target type
|
||||
target_counter = rel_target_types.get(rel_type, Counter())
|
||||
most_common_target = target_counter.most_common(1)[0][0] if target_counter else None
|
||||
|
||||
frequencies.append(
|
||||
FieldFrequency(
|
||||
name=rel_type,
|
||||
source="relation",
|
||||
count=count,
|
||||
total=total,
|
||||
percentage=count / total,
|
||||
sample_values=rel_samples.get(rel_type, []),
|
||||
is_array=is_array,
|
||||
target_type=most_common_target,
|
||||
)
|
||||
)
|
||||
|
||||
return frequencies
|
||||
|
||||
|
||||
# --- Schema Generation ---
|
||||
|
||||
|
||||
def _build_picoschema_dict(
|
||||
frequencies: list[FieldFrequency],
|
||||
required_threshold: float,
|
||||
optional_threshold: float,
|
||||
) -> dict:
|
||||
"""Build a Picoschema YAML dict from field frequencies.
|
||||
|
||||
Only includes fields at or above the optional threshold.
|
||||
"""
|
||||
schema: dict = {}
|
||||
|
||||
for freq in frequencies:
|
||||
if freq.percentage < optional_threshold:
|
||||
continue
|
||||
|
||||
is_required = freq.percentage >= required_threshold
|
||||
|
||||
# --- Build the field key ---
|
||||
key = freq.name
|
||||
if not is_required:
|
||||
key += "?"
|
||||
if freq.is_array:
|
||||
key += "(array)"
|
||||
|
||||
# --- Build the field value ---
|
||||
if freq.source == "relation":
|
||||
# Relations become entity reference fields
|
||||
target = freq.target_type or "string"
|
||||
# Capitalize first letter for entity ref convention
|
||||
target = target[0].upper() + target[1:] if target != "string" else "string"
|
||||
schema[key] = target
|
||||
else:
|
||||
schema[key] = "string"
|
||||
|
||||
return schema
|
||||
@@ -0,0 +1,236 @@
|
||||
"""Picoschema parser for Basic Memory.
|
||||
|
||||
Parses Picoschema YAML dicts (from note frontmatter) into typed dataclass
|
||||
representations. Picoschema is a compact schema notation from Google's Dotprompt
|
||||
that fits naturally in YAML frontmatter.
|
||||
|
||||
Syntax reference:
|
||||
field: type, description # required field
|
||||
field?: type, description # optional field
|
||||
field(array): type # array of values
|
||||
field?(enum): [val1, val2] # enumeration
|
||||
field?(object): # nested object
|
||||
sub_field: type
|
||||
EntityName as type (capitalized) # entity reference
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
|
||||
# --- Data Model ---
|
||||
|
||||
|
||||
@dataclass
|
||||
class SchemaField:
|
||||
"""A single field in a Picoschema definition.
|
||||
|
||||
Maps to either an observation category or a relation type in Basic Memory notes.
|
||||
"""
|
||||
|
||||
name: str
|
||||
type: str # string, integer, number, boolean, any, or EntityName
|
||||
required: bool # True unless field name ends with ?
|
||||
is_array: bool = False # True if (array) notation
|
||||
is_enum: bool = False # True if (enum) notation
|
||||
enum_values: list[str] = field(default_factory=list)
|
||||
description: str | None = None # Text after comma
|
||||
is_entity_ref: bool = False # True if type is capitalized (entity reference)
|
||||
children: list["SchemaField"] = field(default_factory=list) # For (object) types
|
||||
|
||||
|
||||
@dataclass
|
||||
class SchemaDefinition:
|
||||
"""A complete schema definition parsed from a schema note's frontmatter.
|
||||
|
||||
Combines the parsed fields with metadata about the schema itself.
|
||||
"""
|
||||
|
||||
entity: str # The entity type this schema describes
|
||||
version: int # Schema version
|
||||
fields: list[SchemaField] # Parsed fields
|
||||
validation_mode: str # "warn" | "strict" | "off"
|
||||
|
||||
|
||||
# --- Built-in scalar types ---
|
||||
# Types that are NOT entity references. Anything not in this set and starting
|
||||
# with an uppercase letter is treated as an entity reference.
|
||||
|
||||
SCALAR_TYPES = frozenset({"string", "integer", "number", "boolean", "any"})
|
||||
|
||||
|
||||
# --- Field Name Parsing ---
|
||||
|
||||
|
||||
def _parse_field_key(key: str) -> tuple[str, bool, bool, bool, bool]:
|
||||
"""Parse a Picoschema field key into its components.
|
||||
|
||||
Returns (name, required, is_array, is_enum, is_object).
|
||||
The key format is: name[?][(array|enum|object)]
|
||||
|
||||
Examples:
|
||||
"name" -> ("name", True, False, False)
|
||||
"role?" -> ("role", False, False, False)
|
||||
"tags?(array)" -> ("tags", False, True, False)
|
||||
"status?(enum)" -> ("status", False, False, True)
|
||||
"metadata?(object)" -> ("metadata", False, False, False) + children
|
||||
"""
|
||||
required = True
|
||||
is_array = False
|
||||
is_enum = False
|
||||
is_object = False
|
||||
|
||||
# Check for modifier suffix: (array), (enum), (object)
|
||||
if key.endswith("(array)"):
|
||||
is_array = True
|
||||
key = key[: -len("(array)")]
|
||||
elif key.endswith("(enum)"):
|
||||
is_enum = True
|
||||
key = key[: -len("(enum)")]
|
||||
elif key.endswith("(object)"):
|
||||
is_object = True
|
||||
key = key[: -len("(object)")]
|
||||
|
||||
# Check for optional marker
|
||||
if key.endswith("?"):
|
||||
required = False
|
||||
key = key[:-1]
|
||||
|
||||
return key, required, is_array, is_enum, is_object
|
||||
|
||||
|
||||
def _parse_type_and_description(value: str) -> tuple[str, str | None]:
|
||||
"""Parse a type string that may include a comma-separated description.
|
||||
|
||||
Examples:
|
||||
"string" -> ("string", None)
|
||||
"string, full name" -> ("string", "full name")
|
||||
"Organization, employer" -> ("Organization", "employer")
|
||||
"""
|
||||
if "," in value:
|
||||
type_str, desc = value.split(",", 1)
|
||||
return type_str.strip(), desc.strip()
|
||||
return value.strip(), None
|
||||
|
||||
|
||||
def _is_entity_ref_type(type_str: str) -> bool:
|
||||
"""Determine if a type string represents an entity reference.
|
||||
|
||||
Entity references are capitalized type names that are not built-in scalar types.
|
||||
"""
|
||||
if type_str in SCALAR_TYPES:
|
||||
return False
|
||||
# Capitalized first letter = entity reference
|
||||
return len(type_str) > 0 and type_str[0].isupper()
|
||||
|
||||
|
||||
# --- Main Parser ---
|
||||
|
||||
|
||||
def parse_picoschema(yaml_dict: dict) -> list[SchemaField]:
|
||||
"""Parse a Picoschema YAML dict into a list of SchemaField objects.
|
||||
|
||||
This is the core parser that converts YAML frontmatter schema definitions
|
||||
into structured SchemaField dataclasses.
|
||||
|
||||
Args:
|
||||
yaml_dict: The schema dict from YAML frontmatter. Keys are field
|
||||
declarations (e.g., "name", "role?", "tags?(array)"), values are
|
||||
type declarations (e.g., "string", "string, description").
|
||||
|
||||
Returns:
|
||||
List of SchemaField objects representing the schema.
|
||||
"""
|
||||
fields: list[SchemaField] = []
|
||||
|
||||
for key, value in yaml_dict.items():
|
||||
name, required, is_array, is_enum, is_object = _parse_field_key(key)
|
||||
|
||||
# --- Enum fields ---
|
||||
# Trigger: value is a list (e.g., [active, inactive])
|
||||
# Why: enums declare allowed values directly as a YAML list
|
||||
# Outcome: SchemaField with is_enum=True and enum_values populated
|
||||
if is_enum:
|
||||
enum_values = value if isinstance(value, list) else [str(value)]
|
||||
fields.append(
|
||||
SchemaField(
|
||||
name=name,
|
||||
type="enum",
|
||||
required=required,
|
||||
is_enum=True,
|
||||
enum_values=[str(v) for v in enum_values],
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
# --- Object fields ---
|
||||
# Trigger: value is a dict (nested sub-fields)
|
||||
# Why: objects contain child fields parsed recursively
|
||||
# Outcome: SchemaField with children populated via recursive parse
|
||||
if is_object or (isinstance(value, dict) and not is_enum):
|
||||
children = parse_picoschema(value) if isinstance(value, dict) else []
|
||||
fields.append(
|
||||
SchemaField(
|
||||
name=name,
|
||||
type="object",
|
||||
required=required,
|
||||
children=children,
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
# --- Scalar and entity ref fields ---
|
||||
type_str, description = _parse_type_and_description(str(value))
|
||||
is_entity_ref = _is_entity_ref_type(type_str)
|
||||
|
||||
fields.append(
|
||||
SchemaField(
|
||||
name=name,
|
||||
type=type_str,
|
||||
required=required,
|
||||
is_array=is_array,
|
||||
description=description,
|
||||
is_entity_ref=is_entity_ref,
|
||||
)
|
||||
)
|
||||
|
||||
return fields
|
||||
|
||||
|
||||
def parse_schema_note(frontmatter: dict) -> SchemaDefinition:
|
||||
"""Parse a full schema note's frontmatter into a SchemaDefinition.
|
||||
|
||||
A schema note has type: schema and contains:
|
||||
- entity: the entity type this schema describes
|
||||
- version: schema version number
|
||||
- schema: the Picoschema dict
|
||||
- settings.validation: validation mode (warn/strict/off)
|
||||
|
||||
Args:
|
||||
frontmatter: The complete YAML frontmatter dict from a schema note.
|
||||
|
||||
Returns:
|
||||
A SchemaDefinition with parsed fields and metadata.
|
||||
|
||||
Raises:
|
||||
ValueError: If required fields (entity, schema) are missing.
|
||||
"""
|
||||
entity = frontmatter.get("entity")
|
||||
if not entity:
|
||||
raise ValueError("Schema note missing required 'entity' field in frontmatter")
|
||||
|
||||
schema_dict = frontmatter.get("schema")
|
||||
if not schema_dict or not isinstance(schema_dict, dict):
|
||||
raise ValueError("Schema note missing required 'schema' dict in frontmatter")
|
||||
|
||||
version = frontmatter.get("version", 1)
|
||||
settings = frontmatter.get("settings", {})
|
||||
validation_mode = settings.get("validation", "warn") if isinstance(settings, dict) else "warn"
|
||||
|
||||
fields = parse_picoschema(schema_dict)
|
||||
|
||||
return SchemaDefinition(
|
||||
entity=entity,
|
||||
version=version,
|
||||
fields=fields,
|
||||
validation_mode=validation_mode,
|
||||
)
|
||||
@@ -0,0 +1,118 @@
|
||||
"""Schema resolver for Basic Memory.
|
||||
|
||||
Finds the applicable schema for a note using a priority-based resolution order:
|
||||
1. Inline schema -> frontmatter['schema'] is a dict
|
||||
2. Explicit ref -> frontmatter['schema'] is a string (entity name or permalink)
|
||||
3. Implicit by type -> frontmatter['type'] matches a schema note's entity field
|
||||
4. No schema -> returns None (perfectly fine)
|
||||
|
||||
The resolver takes a search function as a dependency instead of importing
|
||||
repository code directly, keeping the schema package decoupled from the
|
||||
data access layer.
|
||||
"""
|
||||
|
||||
from collections.abc import Callable, Awaitable
|
||||
|
||||
from basic_memory.schema.parser import SchemaDefinition, parse_picoschema, parse_schema_note
|
||||
|
||||
|
||||
# Type alias for the search function dependency.
|
||||
# Given a query string, returns a list of frontmatter dicts from matching schema notes.
|
||||
type SchemaSearchFn = Callable[[str], Awaitable[list[dict]]]
|
||||
|
||||
|
||||
async def resolve_schema(
|
||||
note_frontmatter: dict,
|
||||
search_fn: SchemaSearchFn,
|
||||
) -> SchemaDefinition | None:
|
||||
"""Resolve the schema for a note based on its frontmatter.
|
||||
|
||||
Resolution order:
|
||||
1. Inline schema (frontmatter['schema'] is a dict) - parsed directly
|
||||
2. Explicit reference (frontmatter['schema'] is a string) - looked up via search_fn
|
||||
3. Implicit by type (frontmatter['type']) - searches for schema note with matching entity
|
||||
4. No schema - returns None
|
||||
|
||||
Args:
|
||||
note_frontmatter: The YAML frontmatter dict from the note being validated.
|
||||
search_fn: An async callable that takes a query string and returns a list
|
||||
of frontmatter dicts from matching schema notes. This keeps the resolver
|
||||
decoupled from the repository/service layer.
|
||||
|
||||
Returns:
|
||||
A SchemaDefinition if a schema is found, None otherwise.
|
||||
"""
|
||||
schema_value = note_frontmatter.get("schema")
|
||||
|
||||
# --- 1. Inline schema ---
|
||||
# Trigger: schema field is a dict (the Picoschema definition lives in this note)
|
||||
# Why: inline schemas are self-contained, no lookup needed
|
||||
# Outcome: parse and return immediately
|
||||
if isinstance(schema_value, dict):
|
||||
return _schema_from_inline(schema_value, note_frontmatter)
|
||||
|
||||
# --- 2. Explicit reference ---
|
||||
# Trigger: schema field is a string (entity name or permalink)
|
||||
# Why: the note points to a specific schema note by name
|
||||
# Outcome: search for the referenced schema note and parse it
|
||||
if isinstance(schema_value, str):
|
||||
result = await _schema_from_reference(schema_value, search_fn)
|
||||
if result is not None:
|
||||
return result
|
||||
|
||||
# --- 3. Implicit by type ---
|
||||
# Trigger: no schema field, but the note has a type field
|
||||
# Why: convention — a note with type: Person looks for a schema with entity: Person
|
||||
# Outcome: search for a schema note whose entity matches the note's type
|
||||
note_type = note_frontmatter.get("type")
|
||||
if note_type:
|
||||
result = await _schema_from_type(note_type, search_fn)
|
||||
if result is not None:
|
||||
return result
|
||||
|
||||
# --- 4. No schema ---
|
||||
return None
|
||||
|
||||
|
||||
def _schema_from_inline(schema_dict: dict, frontmatter: dict) -> SchemaDefinition:
|
||||
"""Build a SchemaDefinition from an inline schema dict.
|
||||
|
||||
For inline schemas, we derive metadata from the note's own frontmatter
|
||||
since there's no separate schema note.
|
||||
"""
|
||||
fields = parse_picoschema(schema_dict)
|
||||
entity = frontmatter.get("type", "unknown")
|
||||
settings = frontmatter.get("settings", {})
|
||||
validation_mode = settings.get("validation", "warn") if isinstance(settings, dict) else "warn"
|
||||
|
||||
return SchemaDefinition(
|
||||
entity=entity,
|
||||
version=1,
|
||||
fields=fields,
|
||||
validation_mode=validation_mode,
|
||||
)
|
||||
|
||||
|
||||
async def _schema_from_reference(
|
||||
ref: str,
|
||||
search_fn: SchemaSearchFn,
|
||||
) -> SchemaDefinition | None:
|
||||
"""Look up a schema by entity name or permalink reference.
|
||||
|
||||
The search function is expected to find schema notes matching the reference.
|
||||
"""
|
||||
results = await search_fn(ref)
|
||||
if not results:
|
||||
return None
|
||||
return parse_schema_note(results[0])
|
||||
|
||||
|
||||
async def _schema_from_type(
|
||||
note_type: str,
|
||||
search_fn: SchemaSearchFn,
|
||||
) -> SchemaDefinition | None:
|
||||
"""Look up a schema implicitly by matching the note's type to a schema's entity field."""
|
||||
results = await search_fn(note_type)
|
||||
if not results:
|
||||
return None
|
||||
return parse_schema_note(results[0])
|
||||
@@ -0,0 +1,259 @@
|
||||
"""Schema validator for Basic Memory.
|
||||
|
||||
Validates a note's observations and relations against a resolved schema definition.
|
||||
The mapping rules ground schema fields in the existing Basic Memory note format:
|
||||
|
||||
Schema Declaration -> Grounded In
|
||||
-----------------------------------------------
|
||||
field: string -> observation [field] value
|
||||
field?(array): string -> multiple [field] observations
|
||||
field?: EntityType -> relation 'field [[Target]]'
|
||||
field?(array): EntityType -> multiple 'field' relations
|
||||
field?(enum): [values] -> observation [field] value where value is in set
|
||||
|
||||
Validation is soft by default (warn mode). Unmatched observations and relations
|
||||
are informational, not errors -- schemas are a subset, not a straitjacket.
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass, field as dataclass_field
|
||||
|
||||
from basic_memory.schema.inference import ObservationData, RelationData
|
||||
from basic_memory.schema.parser import SchemaDefinition, SchemaField
|
||||
|
||||
|
||||
# --- Result Data Model ---
|
||||
|
||||
|
||||
@dataclass
|
||||
class FieldResult:
|
||||
"""Validation result for a single schema field."""
|
||||
|
||||
field: SchemaField
|
||||
status: str # "present" | "missing" | "enum_mismatch"
|
||||
values: list[str] = dataclass_field(default_factory=list) # Matched values
|
||||
message: str | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class ValidationResult:
|
||||
"""Complete validation result for a note against a schema."""
|
||||
|
||||
note_identifier: str
|
||||
schema_entity: str
|
||||
passed: bool # True if no errors (warnings are OK)
|
||||
field_results: list[FieldResult] = dataclass_field(default_factory=list)
|
||||
unmatched_observations: dict[str, int] = dataclass_field(default_factory=dict) # cat -> count
|
||||
unmatched_relations: list[str] = dataclass_field(default_factory=list) # types not in schema
|
||||
warnings: list[str] = dataclass_field(default_factory=list)
|
||||
errors: list[str] = dataclass_field(default_factory=list)
|
||||
|
||||
|
||||
# --- Validation Logic ---
|
||||
|
||||
|
||||
def validate_note(
|
||||
note_identifier: str,
|
||||
schema: SchemaDefinition,
|
||||
observations: list[ObservationData],
|
||||
relations: list[RelationData],
|
||||
) -> ValidationResult:
|
||||
"""Validate a note against a schema definition.
|
||||
|
||||
Args:
|
||||
note_identifier: The note's title, permalink, or file path for reporting.
|
||||
schema: The resolved SchemaDefinition to validate against.
|
||||
observations: List of ObservationData from the note's observations.
|
||||
relations: List of RelationData from the note's relations.
|
||||
|
||||
Returns:
|
||||
A ValidationResult with per-field results, unmatched items, and warnings/errors.
|
||||
"""
|
||||
result = ValidationResult(
|
||||
note_identifier=note_identifier,
|
||||
schema_entity=schema.entity,
|
||||
passed=True,
|
||||
)
|
||||
|
||||
# Build lookup structures from the note's actual content
|
||||
obs_by_category = _group_observations(observations)
|
||||
rel_by_type = _group_relations(relations)
|
||||
|
||||
# Track which observation categories and relation types are matched by schema fields
|
||||
matched_categories: set[str] = set()
|
||||
matched_relation_types: set[str] = set()
|
||||
|
||||
# --- Validate each schema field ---
|
||||
for schema_field in schema.fields:
|
||||
field_result = _validate_field(schema_field, obs_by_category, rel_by_type)
|
||||
result.field_results.append(field_result)
|
||||
|
||||
# Track which categories/relation types this field consumed
|
||||
if schema_field.is_entity_ref:
|
||||
matched_relation_types.add(schema_field.name)
|
||||
else:
|
||||
matched_categories.add(schema_field.name)
|
||||
|
||||
# --- Generate warnings or errors based on validation mode ---
|
||||
# Trigger: field declared in schema but not found in note
|
||||
# Why: required missing = warning (or error in strict); optional missing = silent
|
||||
# Outcome: only required missing fields produce diagnostics
|
||||
if field_result.status == "missing" and schema_field.required:
|
||||
msg = _missing_field_message(schema_field)
|
||||
if schema.validation_mode == "strict":
|
||||
result.errors.append(msg)
|
||||
result.passed = False
|
||||
else:
|
||||
result.warnings.append(msg)
|
||||
|
||||
elif field_result.status == "enum_mismatch":
|
||||
msg = field_result.message or f"Field '{schema_field.name}' has invalid enum value"
|
||||
if schema.validation_mode == "strict":
|
||||
result.errors.append(msg)
|
||||
result.passed = False
|
||||
else:
|
||||
result.warnings.append(msg)
|
||||
|
||||
# --- Collect unmatched observations ---
|
||||
for category, values in obs_by_category.items():
|
||||
if category not in matched_categories:
|
||||
result.unmatched_observations[category] = len(values)
|
||||
|
||||
# --- Collect unmatched relations ---
|
||||
for rel_type in rel_by_type:
|
||||
if rel_type not in matched_relation_types:
|
||||
result.unmatched_relations.append(rel_type)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
# --- Field Validation ---
|
||||
|
||||
|
||||
def _validate_field(
|
||||
schema_field: SchemaField,
|
||||
obs_by_category: dict[str, list[str]],
|
||||
rel_by_type: dict[str, list[str]],
|
||||
) -> FieldResult:
|
||||
"""Validate a single schema field against the note's data.
|
||||
|
||||
Entity ref fields map to relations; all other fields map to observations.
|
||||
"""
|
||||
# --- Entity reference fields map to relations ---
|
||||
if schema_field.is_entity_ref:
|
||||
return _validate_entity_ref_field(schema_field, rel_by_type)
|
||||
|
||||
# --- Enum fields require value membership check ---
|
||||
if schema_field.is_enum:
|
||||
return _validate_enum_field(schema_field, obs_by_category)
|
||||
|
||||
# --- Scalar and array fields map to observations ---
|
||||
return _validate_observation_field(schema_field, obs_by_category)
|
||||
|
||||
|
||||
def _validate_observation_field(
|
||||
schema_field: SchemaField,
|
||||
obs_by_category: dict[str, list[str]],
|
||||
) -> FieldResult:
|
||||
"""Validate a field that maps to observation categories."""
|
||||
values = obs_by_category.get(schema_field.name, [])
|
||||
|
||||
if not values:
|
||||
return FieldResult(
|
||||
field=schema_field,
|
||||
status="missing",
|
||||
message=_missing_field_message(schema_field),
|
||||
)
|
||||
|
||||
return FieldResult(
|
||||
field=schema_field,
|
||||
status="present",
|
||||
values=values,
|
||||
)
|
||||
|
||||
|
||||
def _validate_entity_ref_field(
|
||||
schema_field: SchemaField,
|
||||
rel_by_type: dict[str, list[str]],
|
||||
) -> FieldResult:
|
||||
"""Validate a field that maps to relations (entity references)."""
|
||||
targets = rel_by_type.get(schema_field.name, [])
|
||||
|
||||
if not targets:
|
||||
return FieldResult(
|
||||
field=schema_field,
|
||||
status="missing",
|
||||
message=f"Missing relation: {schema_field.name} (no '{schema_field.name} [[...]]' "
|
||||
f"relation found)",
|
||||
)
|
||||
|
||||
return FieldResult(
|
||||
field=schema_field,
|
||||
status="present",
|
||||
values=targets,
|
||||
)
|
||||
|
||||
|
||||
def _validate_enum_field(
|
||||
schema_field: SchemaField,
|
||||
obs_by_category: dict[str, list[str]],
|
||||
) -> FieldResult:
|
||||
"""Validate an enum field -- value must be in the allowed set."""
|
||||
values = obs_by_category.get(schema_field.name, [])
|
||||
|
||||
if not values:
|
||||
return FieldResult(
|
||||
field=schema_field,
|
||||
status="missing",
|
||||
message=_missing_field_message(schema_field),
|
||||
)
|
||||
|
||||
# Check each value against the allowed enum values
|
||||
invalid_values = [v for v in values if v not in schema_field.enum_values]
|
||||
if invalid_values:
|
||||
allowed = ", ".join(schema_field.enum_values)
|
||||
invalid = ", ".join(invalid_values)
|
||||
return FieldResult(
|
||||
field=schema_field,
|
||||
status="enum_mismatch",
|
||||
values=values,
|
||||
message=f"Field '{schema_field.name}' has invalid value(s): {invalid} "
|
||||
f"(allowed: {allowed})",
|
||||
)
|
||||
|
||||
return FieldResult(
|
||||
field=schema_field,
|
||||
status="present",
|
||||
values=values,
|
||||
)
|
||||
|
||||
|
||||
# --- Helper Functions ---
|
||||
|
||||
|
||||
def _group_observations(observations: list[ObservationData]) -> dict[str, list[str]]:
|
||||
"""Group observations by category."""
|
||||
result: dict[str, list[str]] = {}
|
||||
for obs in observations:
|
||||
result.setdefault(obs.category, []).append(obs.content)
|
||||
return result
|
||||
|
||||
|
||||
def _group_relations(relations: list[RelationData]) -> dict[str, list[str]]:
|
||||
"""Group relations by relation type."""
|
||||
result: dict[str, list[str]] = {}
|
||||
for rel in relations:
|
||||
result.setdefault(rel.relation_type, []).append(rel.target_name)
|
||||
return result
|
||||
|
||||
|
||||
def _missing_field_message(schema_field: SchemaField) -> str:
|
||||
"""Generate a human-readable message for a missing field."""
|
||||
kind = "required" if schema_field.required else "optional"
|
||||
|
||||
if schema_field.is_entity_ref:
|
||||
return (
|
||||
f"Missing {kind} field: {schema_field.name} "
|
||||
f"(no '{schema_field.name} [[...]]' relation found)"
|
||||
)
|
||||
|
||||
return f"Missing {kind} field: {schema_field.name} (expected [{schema_field.name}] observation)"
|
||||
@@ -0,0 +1,121 @@
|
||||
"""Pydantic response models for the schema system.
|
||||
|
||||
These models define the API response format for schema validation,
|
||||
inference, and drift detection operations. They mirror the dataclass
|
||||
structures in basic_memory.schema but are Pydantic models suitable
|
||||
for API serialization.
|
||||
"""
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
# --- Validation Response Models ---
|
||||
|
||||
|
||||
class FieldResultResponse(BaseModel):
|
||||
"""Result of validating a single schema field against a note."""
|
||||
|
||||
field_name: str
|
||||
field_type: str
|
||||
required: bool
|
||||
status: str = Field(description="One of: present, missing, type_mismatch")
|
||||
values: list[str] = Field(default_factory=list, description="Matched values from the note")
|
||||
message: str | None = None
|
||||
|
||||
|
||||
class NoteValidationResponse(BaseModel):
|
||||
"""Validation result for a single note against a schema."""
|
||||
|
||||
note_identifier: str
|
||||
schema_entity: str
|
||||
passed: bool = Field(description="True if no errors (warnings are OK)")
|
||||
field_results: list[FieldResultResponse] = Field(default_factory=list)
|
||||
unmatched_observations: dict[str, int] = Field(
|
||||
default_factory=dict,
|
||||
description="Observation categories not covered by schema, with counts",
|
||||
)
|
||||
unmatched_relations: list[str] = Field(
|
||||
default_factory=list,
|
||||
description="Relation types not covered by schema",
|
||||
)
|
||||
warnings: list[str] = Field(default_factory=list)
|
||||
errors: list[str] = Field(default_factory=list)
|
||||
|
||||
|
||||
class ValidationReport(BaseModel):
|
||||
"""Full validation report for one or more notes."""
|
||||
|
||||
entity_type: str | None = None
|
||||
total_notes: int = 0
|
||||
valid_count: int = 0
|
||||
warning_count: int = 0
|
||||
error_count: int = 0
|
||||
results: list[NoteValidationResponse] = Field(default_factory=list)
|
||||
|
||||
|
||||
# --- Inference Response Models ---
|
||||
|
||||
|
||||
class FieldFrequencyResponse(BaseModel):
|
||||
"""Frequency analysis for a single field across notes."""
|
||||
|
||||
name: str
|
||||
source: str = Field(description="One of: observation, relation")
|
||||
count: int = Field(description="Number of notes containing this field")
|
||||
total: int = Field(description="Total notes analyzed")
|
||||
percentage: float
|
||||
sample_values: list[str] = Field(default_factory=list)
|
||||
is_array: bool = Field(
|
||||
default=False,
|
||||
description="True if field typically appears multiple times per note",
|
||||
)
|
||||
target_type: str | None = Field(
|
||||
default=None,
|
||||
description="For relations, the most common target entity type",
|
||||
)
|
||||
|
||||
|
||||
class InferenceReport(BaseModel):
|
||||
"""Inference result with suggested schema definition."""
|
||||
|
||||
entity_type: str
|
||||
notes_analyzed: int
|
||||
field_frequencies: list[FieldFrequencyResponse] = Field(default_factory=list)
|
||||
suggested_schema: dict = Field(
|
||||
default_factory=dict,
|
||||
description="Ready-to-use Picoschema YAML dict",
|
||||
)
|
||||
suggested_required: list[str] = Field(default_factory=list)
|
||||
suggested_optional: list[str] = Field(default_factory=list)
|
||||
excluded: list[str] = Field(
|
||||
default_factory=list,
|
||||
description="Fields below the inclusion threshold",
|
||||
)
|
||||
|
||||
|
||||
# --- Drift Response Models ---
|
||||
|
||||
|
||||
class DriftFieldResponse(BaseModel):
|
||||
"""A field involved in schema drift."""
|
||||
|
||||
name: str
|
||||
source: str
|
||||
count: int
|
||||
total: int
|
||||
percentage: float
|
||||
|
||||
|
||||
class DriftReport(BaseModel):
|
||||
"""Schema drift analysis comparing schema definition to actual usage."""
|
||||
|
||||
entity_type: str
|
||||
new_fields: list[DriftFieldResponse] = Field(
|
||||
default_factory=list,
|
||||
description="Fields common in notes but not in schema",
|
||||
)
|
||||
dropped_fields: list[DriftFieldResponse] = Field(
|
||||
default_factory=list,
|
||||
description="Fields in schema but rare in notes",
|
||||
)
|
||||
cardinality_changes: list[str] = Field(default_factory=list)
|
||||
@@ -0,0 +1,17 @@
|
||||
---
|
||||
title: Alice Chen
|
||||
type: Person
|
||||
tags: [engineering, drift-test]
|
||||
---
|
||||
|
||||
# Alice Chen
|
||||
|
||||
## Observations
|
||||
- [name] Alice Chen
|
||||
- [github] alicechen
|
||||
- [expertise] Backend systems
|
||||
- [fact] Leads the platform team
|
||||
|
||||
## Relations
|
||||
- works_at [[Acme Corp]]
|
||||
- works_at [[Open Source Collective]]
|
||||
@@ -0,0 +1,16 @@
|
||||
---
|
||||
title: Bob Martinez
|
||||
type: Person
|
||||
tags: [design, drift-test]
|
||||
---
|
||||
|
||||
# Bob Martinez
|
||||
|
||||
## Observations
|
||||
- [name] Bob Martinez
|
||||
- [github] bobmartinez
|
||||
- [fact] Design systems advocate
|
||||
|
||||
## Relations
|
||||
- works_at [[DesignCo]]
|
||||
- works_at [[Figma Community]]
|
||||
@@ -0,0 +1,16 @@
|
||||
---
|
||||
title: Carol Wu
|
||||
type: Person
|
||||
tags: [data-science, drift-test]
|
||||
---
|
||||
|
||||
# Carol Wu
|
||||
|
||||
## Observations
|
||||
- [name] Carol Wu
|
||||
- [github] carolwu
|
||||
- [expertise] Machine learning
|
||||
|
||||
## Relations
|
||||
- works_at [[DataLab]]
|
||||
- works_at [[University Research Group]]
|
||||
@@ -0,0 +1,15 @@
|
||||
---
|
||||
title: David Kim
|
||||
type: Person
|
||||
tags: [security, drift-test]
|
||||
---
|
||||
|
||||
# David Kim
|
||||
|
||||
## Observations
|
||||
- [name] David Kim
|
||||
- [github] davidkim
|
||||
- [fact] Security researcher
|
||||
|
||||
## Relations
|
||||
- works_at [[SecureTech]]
|
||||
@@ -0,0 +1,16 @@
|
||||
---
|
||||
title: Elena Petrova
|
||||
type: Person
|
||||
tags: [devops, drift-test]
|
||||
---
|
||||
|
||||
# Elena Petrova
|
||||
|
||||
## Observations
|
||||
- [name] Elena Petrova
|
||||
- [github] elenap
|
||||
- [expertise] Infrastructure
|
||||
|
||||
## Relations
|
||||
- works_at [[CloudOps]]
|
||||
- works_at [[CNCF]]
|
||||
@@ -0,0 +1,16 @@
|
||||
---
|
||||
title: Frank Nguyen
|
||||
type: Person
|
||||
tags: [mobile, drift-test]
|
||||
---
|
||||
|
||||
# Frank Nguyen
|
||||
|
||||
## Observations
|
||||
- [name] Frank Nguyen
|
||||
- [github] frankn
|
||||
- [fact] iOS developer
|
||||
|
||||
## Relations
|
||||
- works_at [[AppWorks]]
|
||||
- works_at [[Swift Community]]
|
||||
@@ -0,0 +1,17 @@
|
||||
---
|
||||
title: Grace Okafor
|
||||
type: Person
|
||||
tags: [frontend, drift-test]
|
||||
---
|
||||
|
||||
# Grace Okafor
|
||||
|
||||
## Observations
|
||||
- [name] Grace Okafor
|
||||
- [github] graceo
|
||||
- [expertise] React and TypeScript
|
||||
- [fact] Open source contributor
|
||||
|
||||
## Relations
|
||||
- works_at [[WebDev Inc]]
|
||||
- works_at [[React Core Team]]
|
||||
@@ -0,0 +1,17 @@
|
||||
---
|
||||
title: Hassan Ali
|
||||
type: Person
|
||||
tags: [backend, drift-test]
|
||||
---
|
||||
|
||||
# Hassan Ali
|
||||
|
||||
## Observations
|
||||
- [name] Hassan Ali
|
||||
- [github] hassanali
|
||||
- [fact] Distributed systems engineer
|
||||
- [email] hassan@example.com
|
||||
|
||||
## Relations
|
||||
- works_at [[ScaleCo]]
|
||||
- works_at [[Apache Foundation]]
|
||||
@@ -0,0 +1,16 @@
|
||||
---
|
||||
title: Iris Zhang
|
||||
type: Person
|
||||
tags: [research, drift-test]
|
||||
---
|
||||
|
||||
# Iris Zhang
|
||||
|
||||
## Observations
|
||||
- [name] Iris Zhang
|
||||
- [github] irisz
|
||||
- [expertise] NLP research
|
||||
- [fact] Published 30+ papers
|
||||
|
||||
## Relations
|
||||
- works_at [[AI Research Lab]]
|
||||
@@ -0,0 +1,16 @@
|
||||
---
|
||||
title: Jake Thompson
|
||||
type: Person
|
||||
tags: [gaming, drift-test]
|
||||
---
|
||||
|
||||
# Jake Thompson
|
||||
|
||||
## Observations
|
||||
- [name] Jake Thompson
|
||||
- [github] jaket
|
||||
- [fact] Game engine developer
|
||||
|
||||
## Relations
|
||||
- works_at [[GameStudios]]
|
||||
- works_at [[Unreal Community]]
|
||||
@@ -0,0 +1,15 @@
|
||||
---
|
||||
title: Karen Liu
|
||||
type: Person
|
||||
tags: [database, drift-test]
|
||||
---
|
||||
|
||||
# Karen Liu
|
||||
|
||||
## Observations
|
||||
- [name] Karen Liu
|
||||
- [github] karenliu
|
||||
- [expertise] Database internals
|
||||
|
||||
## Relations
|
||||
- works_at [[DataStore]]
|
||||
@@ -0,0 +1,17 @@
|
||||
---
|
||||
title: Luis Rivera
|
||||
type: Person
|
||||
tags: [embedded, drift-test]
|
||||
---
|
||||
|
||||
# Luis Rivera
|
||||
|
||||
## Observations
|
||||
- [name] Luis Rivera
|
||||
- [github] luisr
|
||||
- [fact] Embedded systems specialist
|
||||
- [expertise] Real-time operating systems
|
||||
|
||||
## Relations
|
||||
- works_at [[IoT Solutions]]
|
||||
- works_at [[RISC-V Foundation]]
|
||||
@@ -0,0 +1,15 @@
|
||||
---
|
||||
title: Maria Santos
|
||||
type: Person
|
||||
tags: [product, drift-test]
|
||||
---
|
||||
|
||||
# Maria Santos
|
||||
|
||||
## Observations
|
||||
- [name] Maria Santos
|
||||
- [fact] Product manager
|
||||
- [role] Senior Product Manager
|
||||
|
||||
## Relations
|
||||
- works_at [[TechCorp]]
|
||||
@@ -0,0 +1,16 @@
|
||||
---
|
||||
title: Nate Johnson
|
||||
type: Person
|
||||
tags: [sre, drift-test]
|
||||
---
|
||||
|
||||
# Nate Johnson
|
||||
|
||||
## Observations
|
||||
- [name] Nate Johnson
|
||||
- [fact] Site reliability engineer
|
||||
- [expertise] Kubernetes
|
||||
|
||||
## Relations
|
||||
- works_at [[CloudPlatform]]
|
||||
- works_at [[SRE Community]]
|
||||
@@ -0,0 +1,15 @@
|
||||
---
|
||||
title: Olivia Park
|
||||
type: Person
|
||||
tags: [compiler, drift-test]
|
||||
---
|
||||
|
||||
# Olivia Park
|
||||
|
||||
## Observations
|
||||
- [name] Olivia Park
|
||||
- [fact] Compiler engineer
|
||||
- [expertise] LLVM
|
||||
|
||||
## Relations
|
||||
- works_at [[CompilerCo]]
|
||||
@@ -0,0 +1,15 @@
|
||||
---
|
||||
title: Paul Schmidt
|
||||
type: Person
|
||||
tags: [networking, drift-test]
|
||||
---
|
||||
|
||||
# Paul Schmidt
|
||||
|
||||
## Observations
|
||||
- [name] Paul Schmidt
|
||||
- [fact] Network protocol designer
|
||||
|
||||
## Relations
|
||||
- works_at [[NetDev]]
|
||||
- works_at [[IETF]]
|
||||
@@ -0,0 +1,14 @@
|
||||
---
|
||||
title: Quinn Taylor
|
||||
type: Person
|
||||
tags: [testing, drift-test]
|
||||
---
|
||||
|
||||
# Quinn Taylor
|
||||
|
||||
## Observations
|
||||
- [name] Quinn Taylor
|
||||
- [fact] Testing infrastructure lead
|
||||
|
||||
## Relations
|
||||
- works_at [[QualityCo]]
|
||||
@@ -0,0 +1,16 @@
|
||||
---
|
||||
title: Rachel Lee
|
||||
type: Person
|
||||
tags: [crypto, drift-test]
|
||||
---
|
||||
|
||||
# Rachel Lee
|
||||
|
||||
## Observations
|
||||
- [name] Rachel Lee
|
||||
- [fact] Cryptography researcher
|
||||
- [expertise] Post-quantum cryptography
|
||||
|
||||
## Relations
|
||||
- works_at [[CryptoLab]]
|
||||
- works_at [[NIST Advisory Board]]
|
||||
@@ -0,0 +1,15 @@
|
||||
---
|
||||
title: Sam Foster
|
||||
type: Person
|
||||
tags: [accessibility, drift-test]
|
||||
---
|
||||
|
||||
# Sam Foster
|
||||
|
||||
## Observations
|
||||
- [name] Sam Foster
|
||||
- [fact] Accessibility advocate
|
||||
- [email] sam@a11y.org
|
||||
|
||||
## Relations
|
||||
- works_at [[A11y Foundation]]
|
||||
@@ -0,0 +1,16 @@
|
||||
---
|
||||
title: Tanya Ivanova
|
||||
type: Person
|
||||
tags: [quantum, drift-test]
|
||||
---
|
||||
|
||||
# Tanya Ivanova
|
||||
|
||||
## Observations
|
||||
- [name] Tanya Ivanova
|
||||
- [fact] Quantum computing researcher
|
||||
- [expertise] Quantum algorithms
|
||||
|
||||
## Relations
|
||||
- works_at [[QuantumLab]]
|
||||
- works_at [[University of Waterloo]]
|
||||
@@ -0,0 +1,18 @@
|
||||
---
|
||||
title: Person
|
||||
type: schema
|
||||
entity: Person
|
||||
version: 1
|
||||
schema:
|
||||
name: string, full name
|
||||
role?: string, job title or position
|
||||
works_at?: Organization, employer
|
||||
email?: string, contact email
|
||||
settings:
|
||||
validation: warn
|
||||
---
|
||||
|
||||
# Person
|
||||
|
||||
A v1 Person schema used for drift detection testing. Usage has diverged
|
||||
from this definition over time.
|
||||
@@ -0,0 +1,17 @@
|
||||
---
|
||||
title: Solo Standup
|
||||
type: Meeting
|
||||
tags: [edge-case, array-single]
|
||||
---
|
||||
|
||||
# Solo Standup
|
||||
|
||||
An array field with only one observation. Should still pass validation.
|
||||
|
||||
## Observations
|
||||
- [attendees] Alice
|
||||
- [decisions] Continue current approach
|
||||
- [action_items] Alice to finish implementation
|
||||
|
||||
## Relations
|
||||
- part_of [[Q1 Sprint]]
|
||||
@@ -0,0 +1,5 @@
|
||||
---
|
||||
title: Empty Note
|
||||
type: Person
|
||||
tags: [test]
|
||||
---
|
||||
@@ -0,0 +1,20 @@
|
||||
---
|
||||
title: Project Kickoff Notes
|
||||
type: Person
|
||||
schema: Meeting
|
||||
tags: [edge-case, explicit-ref]
|
||||
---
|
||||
|
||||
# Project Kickoff Notes
|
||||
|
||||
Has schema: Meeting AND type: Person. The explicit schema reference should win.
|
||||
|
||||
## Observations
|
||||
- [attendees] Alice
|
||||
- [attendees] Bob
|
||||
- [decisions] Use microservices architecture
|
||||
- [action_items] Alice to draft architecture doc
|
||||
- [date] 2024-03-15
|
||||
|
||||
## Relations
|
||||
- followed_by [[Sprint 1 Standup]]
|
||||
@@ -0,0 +1,20 @@
|
||||
---
|
||||
title: Design Review 2024-03-01
|
||||
type: Person
|
||||
schema:
|
||||
topic: string, review subject
|
||||
outcome?: string, decision made
|
||||
reviewer?(array): string, who reviewed
|
||||
tags: [edge-case, inline-schema]
|
||||
---
|
||||
|
||||
# Design Review 2024-03-01
|
||||
|
||||
Has BOTH an inline schema dict AND type: Person. The inline schema should win.
|
||||
|
||||
## Observations
|
||||
- [topic] API versioning strategy
|
||||
- [outcome] Adopt URL-based versioning
|
||||
- [reviewer] Alice
|
||||
- [reviewer] Bob
|
||||
- [reviewer] Carol
|
||||
@@ -0,0 +1,11 @@
|
||||
# A Note Without Frontmatter
|
||||
|
||||
This markdown file has no YAML frontmatter at all. The schema system should
|
||||
handle this gracefully and skip validation entirely.
|
||||
|
||||
## Observations
|
||||
- [fact] Some observation here
|
||||
- [note] Another observation
|
||||
|
||||
## Relations
|
||||
- relates_to [[Some Other Note]]
|
||||
@@ -0,0 +1,15 @@
|
||||
---
|
||||
title: Architecture Links
|
||||
type: SoftwareProject
|
||||
tags: [edge-case, relation-only]
|
||||
---
|
||||
|
||||
# Architecture Links
|
||||
|
||||
A note with relations but no observations at all.
|
||||
|
||||
## Relations
|
||||
- maintained_by [[Pedro Hernandez]]
|
||||
- uses [[FastAPI]]
|
||||
- uses [[SQLite]]
|
||||
- depends_on [[Python]]
|
||||
@@ -0,0 +1,17 @@
|
||||
---
|
||||
title: Some Random Entity
|
||||
type: SomeRandomType
|
||||
tags: [untyped, edge-case]
|
||||
---
|
||||
|
||||
# Some Random Entity
|
||||
|
||||
This note has a type that does not match any schema definition. The schema
|
||||
system should recognize there is no matching schema and skip validation.
|
||||
|
||||
## Observations
|
||||
- [fact] This type has no corresponding schema note
|
||||
- [status] Active
|
||||
|
||||
## Relations
|
||||
- relates_to [[Another Entity]]
|
||||
@@ -0,0 +1,19 @@
|
||||
---
|
||||
title: Yukihiro Matsumoto Profile
|
||||
type: Person
|
||||
tags: [edge-case, unicode]
|
||||
---
|
||||
|
||||
# Yukihiro Matsumoto Profile
|
||||
|
||||
Observations with unicode content.
|
||||
|
||||
## Observations
|
||||
- [name] まつもと ゆきひろ (Yukihiro Matsumoto)
|
||||
- [role] プログラミング言語設計者
|
||||
- [expertise] オブジェクト指向プログラミング
|
||||
- [fact] Rubyの哲学: 「プログラマの幸福」
|
||||
|
||||
## Relations
|
||||
- works_at [[Ruby Association]]
|
||||
- created [[Ruby]]
|
||||
@@ -0,0 +1,20 @@
|
||||
---
|
||||
title: Ada Lovelace
|
||||
type: Person
|
||||
tags: [mathematics, computing, history]
|
||||
---
|
||||
|
||||
# Ada Lovelace
|
||||
|
||||
Mathematician and writer, often regarded as the first computer programmer.
|
||||
|
||||
## Observations
|
||||
- [name] Ada Lovelace
|
||||
- [role] Mathematician and writer
|
||||
- [born] 1815
|
||||
- [fact] Wrote the first computer program for Charles Babbage's Analytical Engine
|
||||
- [fact] Was the daughter of poet Lord Byron
|
||||
- [achievement] Recognized as the first person to see the potential of computers beyond calculation
|
||||
|
||||
## Relations
|
||||
- collaborated_with [[Charles Babbage]]
|
||||
@@ -0,0 +1,24 @@
|
||||
---
|
||||
title: Alan Turing
|
||||
type: Person
|
||||
tags: [mathematics, cryptography, computing, ai]
|
||||
---
|
||||
|
||||
# Alan Turing
|
||||
|
||||
Father of theoretical computer science and artificial intelligence.
|
||||
|
||||
## Observations
|
||||
- [name] Alan Turing
|
||||
- [role] Mathematician, logician, and cryptanalyst
|
||||
- [expertise] Mathematical logic
|
||||
- [expertise] Cryptanalysis
|
||||
- [expertise] Theoretical computer science
|
||||
- [born] 1912
|
||||
- [fact] Broke the Enigma code during World War II
|
||||
- [fact] Proposed the Turing test for machine intelligence
|
||||
|
||||
## Relations
|
||||
- works_at [[Bletchley Park]]
|
||||
- works_at [[University of Manchester]]
|
||||
- created [[Turing Machine]]
|
||||
@@ -0,0 +1,25 @@
|
||||
---
|
||||
title: Anders Hejlsberg
|
||||
type: Person
|
||||
tags: [typescript, csharp, turbo-pascal, microsoft]
|
||||
---
|
||||
|
||||
# Anders Hejlsberg
|
||||
|
||||
Creator of TypeScript, C#, and lead architect of Turbo Pascal and Delphi.
|
||||
|
||||
## Observations
|
||||
- [name] Anders Hejlsberg
|
||||
- [role] Technical Fellow and language designer
|
||||
- [expertise] Programming language design
|
||||
- [expertise] Compiler engineering
|
||||
- [expertise] Type systems
|
||||
- [email] anders@microsoft.com
|
||||
- [fact] Created TypeScript to bring static typing to JavaScript
|
||||
- [fact] Lead architect of C# and Turbo Pascal
|
||||
|
||||
## Relations
|
||||
- works_at [[Microsoft]]
|
||||
- created [[TypeScript]]
|
||||
- created [[C Sharp]]
|
||||
- created [[Turbo Pascal]]
|
||||
@@ -0,0 +1,21 @@
|
||||
---
|
||||
title: Andrej Karpathy
|
||||
type: Person
|
||||
tags: [ai, deep-learning, tesla, openai]
|
||||
---
|
||||
|
||||
# Andrej Karpathy
|
||||
|
||||
AI researcher and educator known for making deep learning accessible.
|
||||
|
||||
## Observations
|
||||
- [name] Andrej Karpathy
|
||||
- [role] AI researcher and educator
|
||||
- [expertise] Deep learning
|
||||
- [expertise] Computer vision
|
||||
- [expertise] Natural language processing
|
||||
- [fact] Former director of AI at Tesla for Autopilot
|
||||
- [fact] Created popular deep learning courses and tutorials
|
||||
|
||||
## Relations
|
||||
- works_at [[Eureka Labs]]
|
||||
@@ -0,0 +1,22 @@
|
||||
---
|
||||
title: Barbara Liskov
|
||||
type: Person
|
||||
tags: [programming-languages, oop, distributed-systems]
|
||||
---
|
||||
|
||||
# Barbara Liskov
|
||||
|
||||
Pioneer of programming language design and data abstraction.
|
||||
|
||||
## Observations
|
||||
- [name] Barbara Liskov
|
||||
- [role] Institute Professor at MIT
|
||||
- [expertise] Programming language design
|
||||
- [expertise] Distributed systems
|
||||
- [expertise] Data abstraction
|
||||
- [fact] Formulated the Liskov Substitution Principle
|
||||
- [achievement] Won the Turing Award in 2008
|
||||
|
||||
## Relations
|
||||
- works_at [[MIT]]
|
||||
- created [[CLU]]
|
||||
@@ -0,0 +1,20 @@
|
||||
---
|
||||
title: Bjarne Stroustrup
|
||||
type: Person
|
||||
tags: [cpp, programming-languages, bell-labs]
|
||||
---
|
||||
|
||||
# Bjarne Stroustrup
|
||||
|
||||
Creator of C++ and advocate for type-safe systems programming.
|
||||
|
||||
## Observations
|
||||
- [name] Bjarne Stroustrup
|
||||
- [role] Computer scientist and professor
|
||||
- [expertise] Programming language design
|
||||
- [expertise] Systems programming
|
||||
- [fact] Created C++ starting in 1979 as "C with Classes"
|
||||
|
||||
## Relations
|
||||
- works_at [[Columbia University]]
|
||||
- created [[C++]]
|
||||
@@ -0,0 +1,22 @@
|
||||
---
|
||||
title: Brendan Eich
|
||||
type: Person
|
||||
tags: [javascript, web, browsers, mozilla]
|
||||
---
|
||||
|
||||
# Brendan Eich
|
||||
|
||||
Creator of JavaScript and co-founder of the Mozilla project.
|
||||
|
||||
## Observations
|
||||
- [name] Brendan Eich
|
||||
- [role] Technology executive and programming language creator
|
||||
- [expertise] Programming language design
|
||||
- [expertise] Web browser technology
|
||||
- [fact] Created JavaScript in 10 days at Netscape in 1995
|
||||
- [fact] Co-founded the Mozilla project and later co-founded Brave Software
|
||||
|
||||
## Relations
|
||||
- works_at [[Brave Software]]
|
||||
- created [[JavaScript]]
|
||||
- co-founded [[Mozilla]]
|
||||
@@ -0,0 +1,22 @@
|
||||
---
|
||||
title: Claude Shannon
|
||||
type: Person
|
||||
tags: [information-theory, mathematics, bell-labs]
|
||||
---
|
||||
|
||||
# Claude Shannon
|
||||
|
||||
Father of information theory.
|
||||
|
||||
## Observations
|
||||
- [name] Claude Shannon
|
||||
- [role] Mathematician and electrical engineer
|
||||
- [expertise] Information theory
|
||||
- [expertise] Boolean algebra
|
||||
- [born] 1916
|
||||
- [fact] Published "A Mathematical Theory of Communication" in 1948
|
||||
|
||||
## Relations
|
||||
- works_at [[Bell Labs]]
|
||||
- works_at [[MIT]]
|
||||
- created [[Information Theory]]
|
||||
@@ -0,0 +1,23 @@
|
||||
---
|
||||
title: Dennis Ritchie
|
||||
type: Person
|
||||
tags: [c, unix, bell-labs, programming-languages]
|
||||
---
|
||||
|
||||
# Dennis Ritchie
|
||||
|
||||
Creator of the C programming language and co-creator of Unix.
|
||||
|
||||
## Observations
|
||||
- [name] Dennis Ritchie
|
||||
- [role] Computer scientist and language designer
|
||||
- [expertise] Programming language design
|
||||
- [expertise] Operating systems
|
||||
- [born] 1941
|
||||
- [fact] Created C in 1972 at Bell Labs
|
||||
- [fact] Co-authored "The C Programming Language" with Brian Kernighan
|
||||
|
||||
## Relations
|
||||
- works_at [[Bell Labs]]
|
||||
- created [[C Programming Language]]
|
||||
- co-created [[Unix]]
|
||||
@@ -0,0 +1,21 @@
|
||||
---
|
||||
title: Edsger Dijkstra
|
||||
type: Person
|
||||
tags: [algorithms, structured-programming, formal-methods]
|
||||
---
|
||||
|
||||
# Edsger Dijkstra
|
||||
|
||||
Pioneer of structured programming and graph algorithms.
|
||||
|
||||
## Observations
|
||||
- [name] Edsger Dijkstra
|
||||
- [role] Computer scientist and professor
|
||||
- [expertise] Algorithm design
|
||||
- [expertise] Formal verification
|
||||
- [fact] Developed the shortest path algorithm that bears his name
|
||||
- [fact] Advocated for structured programming and formal methods
|
||||
|
||||
## Relations
|
||||
- works_at [[University of Texas at Austin]]
|
||||
- created [[Dijkstra's Algorithm]]
|
||||
@@ -0,0 +1,22 @@
|
||||
---
|
||||
title: Donald Knuth
|
||||
type: Person
|
||||
tags: [algorithms, tex, computer-science, mathematics]
|
||||
---
|
||||
|
||||
# Donald Knuth
|
||||
|
||||
Author of The Art of Computer Programming and creator of TeX.
|
||||
|
||||
## Observations
|
||||
- [name] Donald Knuth
|
||||
- [role] Professor Emeritus of Computer Science
|
||||
- [expertise] Analysis of algorithms
|
||||
- [expertise] Digital typography
|
||||
- [fact] Has been writing The Art of Computer Programming since 1962
|
||||
- [fact] Created TeX typesetting system and METAFONT
|
||||
|
||||
## Relations
|
||||
- works_at [[Stanford University]]
|
||||
- created [[TeX]]
|
||||
- authored [[The Art of Computer Programming]]
|
||||
@@ -0,0 +1,20 @@
|
||||
---
|
||||
title: Fei-Fei Li
|
||||
type: Person
|
||||
tags: [ai, computer-vision, imagenet, stanford]
|
||||
---
|
||||
|
||||
# Fei-Fei Li
|
||||
|
||||
Creator of ImageNet and pioneer in computer vision research.
|
||||
|
||||
## Observations
|
||||
- [name] Fei-Fei Li
|
||||
- [role] Professor of computer science
|
||||
- [expertise] Computer vision
|
||||
- [expertise] Machine learning
|
||||
- [fact] Created ImageNet, the dataset that sparked the deep learning revolution
|
||||
|
||||
## Relations
|
||||
- works_at [[Stanford University]]
|
||||
- created [[ImageNet]]
|
||||
@@ -0,0 +1,23 @@
|
||||
---
|
||||
title: Geoffrey Hinton
|
||||
type: Person
|
||||
tags: [ai, deep-learning, neural-networks, backpropagation]
|
||||
---
|
||||
|
||||
# Geoffrey Hinton
|
||||
|
||||
Godfather of deep learning and pioneer of backpropagation.
|
||||
|
||||
## Observations
|
||||
- [name] Geoffrey Hinton
|
||||
- [role] Computer scientist and cognitive psychologist
|
||||
- [expertise] Neural networks
|
||||
- [expertise] Machine learning
|
||||
- [expertise] Deep learning
|
||||
- [born] 1947
|
||||
- [fact] Helped popularize backpropagation for training neural networks
|
||||
- [fact] Left Google in 2023 citing concerns about AI safety
|
||||
|
||||
## Relations
|
||||
- works_at [[University of Toronto]]
|
||||
- created [[Backpropagation Algorithm]]
|
||||
@@ -0,0 +1,24 @@
|
||||
---
|
||||
title: Grace Hopper
|
||||
type: Person
|
||||
tags: [computing, navy, compilers]
|
||||
---
|
||||
|
||||
# Grace Hopper
|
||||
|
||||
Pioneer of computer science and United States Navy rear admiral.
|
||||
|
||||
## Observations
|
||||
- [name] Grace Hopper
|
||||
- [role] Computer scientist and Navy rear admiral
|
||||
- [expertise] Compiler design
|
||||
- [expertise] Programming language theory
|
||||
- [email] grace.hopper@navy.mil
|
||||
- [fact] Developed one of the first linkers
|
||||
- [fact] Coined the term "debugging" after finding an actual bug in a computer
|
||||
- [achievement] Awarded the Presidential Medal of Freedom posthumously in 2016
|
||||
|
||||
## Relations
|
||||
- works_at [[United States Navy]]
|
||||
- contributed_to [[COBOL]]
|
||||
- contributed_to [[UNIVAC]]
|
||||
@@ -0,0 +1,22 @@
|
||||
---
|
||||
title: Guido van Rossum
|
||||
type: Person
|
||||
tags: [python, programming-languages, open-source]
|
||||
---
|
||||
|
||||
# Guido van Rossum
|
||||
|
||||
Creator of Python and former Benevolent Dictator for Life of the Python community.
|
||||
|
||||
## Observations
|
||||
- [name] Guido van Rossum
|
||||
- [role] Programming language designer and software engineer
|
||||
- [expertise] Programming language design
|
||||
- [expertise] Software architecture
|
||||
- [email] guido@python.org
|
||||
- [fact] Created Python in 1991 while working at CWI in the Netherlands
|
||||
- [fact] Served as Python's BDFL until stepping down in 2018
|
||||
|
||||
## Relations
|
||||
- works_at [[Microsoft]]
|
||||
- created [[Python]]
|
||||
@@ -0,0 +1,21 @@
|
||||
---
|
||||
title: James Gosling
|
||||
type: Person
|
||||
tags: [java, programming-languages, sun-microsystems]
|
||||
---
|
||||
|
||||
# James Gosling
|
||||
|
||||
Creator of the Java programming language.
|
||||
|
||||
## Observations
|
||||
- [name] James Gosling
|
||||
- [role] Computer scientist and software developer
|
||||
- [expertise] Programming language design
|
||||
- [expertise] Virtual machine architecture
|
||||
- [email] james.gosling@aws.com
|
||||
- [fact] Created Java at Sun Microsystems in the mid-1990s
|
||||
|
||||
## Relations
|
||||
- works_at [[Amazon Web Services]]
|
||||
- created [[Java]]
|
||||
@@ -0,0 +1,24 @@
|
||||
---
|
||||
title: John Carmack
|
||||
type: Person
|
||||
tags: [games, graphics, vr, ai]
|
||||
---
|
||||
|
||||
# John Carmack
|
||||
|
||||
Legendary game programmer and technical director.
|
||||
|
||||
## Observations
|
||||
- [name] John Carmack
|
||||
- [role] Programmer and technology entrepreneur
|
||||
- [expertise] 3D graphics programming
|
||||
- [expertise] Game engine development
|
||||
- [expertise] Virtual reality
|
||||
- [fact] Co-founded id Software and created the Doom and Quake engines
|
||||
- [fact] Served as CTO of Oculus VR
|
||||
|
||||
## Relations
|
||||
- works_at [[Keen Technologies]]
|
||||
- co-founded [[id Software]]
|
||||
- created [[Doom Engine]]
|
||||
- created [[Quake Engine]]
|
||||
@@ -0,0 +1,22 @@
|
||||
---
|
||||
title: Ken Thompson
|
||||
type: Person
|
||||
tags: [unix, go, bell-labs, plan9]
|
||||
---
|
||||
|
||||
# Ken Thompson
|
||||
|
||||
Co-creator of Unix, inventor of the B programming language, and co-creator of Go.
|
||||
|
||||
## Observations
|
||||
- [name] Ken Thompson
|
||||
- [role] Computer scientist and systems programmer
|
||||
- [expertise] Operating systems
|
||||
- [expertise] Programming language design
|
||||
- [fact] Co-created Unix with Dennis Ritchie at Bell Labs
|
||||
- [fact] Co-created the Go programming language at Google
|
||||
|
||||
## Relations
|
||||
- works_at [[Google]]
|
||||
- co-created [[Unix]]
|
||||
- co-created [[Go]]
|
||||
@@ -0,0 +1,18 @@
|
||||
---
|
||||
title: Larry Page
|
||||
type: Person
|
||||
tags: [google, search, alphabet]
|
||||
---
|
||||
|
||||
# Larry Page
|
||||
|
||||
Co-founder of Google and Alphabet.
|
||||
|
||||
## Observations
|
||||
- [name] Larry Page
|
||||
- [fact] Co-founded Google with Sergey Brin in 1998
|
||||
- [fact] Developed the PageRank algorithm as a PhD student at Stanford
|
||||
|
||||
## Relations
|
||||
- works_at [[Alphabet Inc]]
|
||||
- co-founded [[Google]]
|
||||
@@ -0,0 +1,24 @@
|
||||
---
|
||||
title: Leslie Lamport
|
||||
type: Person
|
||||
tags: [distributed-systems, latex, concurrency]
|
||||
---
|
||||
|
||||
# Leslie Lamport
|
||||
|
||||
Pioneer of distributed systems theory and creator of LaTeX.
|
||||
|
||||
## Observations
|
||||
- [name] Leslie Lamport
|
||||
- [role] Computer scientist and researcher
|
||||
- [expertise] Distributed systems
|
||||
- [expertise] Concurrency
|
||||
- [email] lamport@microsoft.com
|
||||
- [fact] Created LaTeX document preparation system
|
||||
- [fact] Invented the Paxos consensus algorithm
|
||||
- [achievement] Won the Turing Award in 2013
|
||||
|
||||
## Relations
|
||||
- works_at [[Microsoft Research]]
|
||||
- created [[LaTeX]]
|
||||
- created [[Paxos Algorithm]]
|
||||
@@ -0,0 +1,26 @@
|
||||
---
|
||||
title: Linus Torvalds
|
||||
type: Person
|
||||
tags: [linux, git, open-source, kernel]
|
||||
---
|
||||
|
||||
# Linus Torvalds
|
||||
|
||||
Creator of Linux and Git, two of the most influential open source projects in history.
|
||||
|
||||
## Observations
|
||||
- [name] Linus Torvalds
|
||||
- [role] Software engineer and open source leader
|
||||
- [expertise] Operating system kernel development
|
||||
- [expertise] Version control systems
|
||||
- [expertise] Low-level systems programming
|
||||
- [email] torvalds@linux-foundation.org
|
||||
- [born] 1969
|
||||
- [fact] Created Linux in 1991 as a university student in Helsinki
|
||||
- [fact] Created Git in 2005 to manage Linux kernel development
|
||||
- [quote] "Talk is cheap. Show me the code."
|
||||
|
||||
## Relations
|
||||
- works_at [[Linux Foundation]]
|
||||
- created [[Linux]]
|
||||
- created [[Git]]
|
||||
@@ -0,0 +1,24 @@
|
||||
---
|
||||
title: Margaret Hamilton
|
||||
type: Person
|
||||
tags: [nasa, software-engineering, apollo]
|
||||
---
|
||||
|
||||
# Margaret Hamilton
|
||||
|
||||
Pioneering software engineer who led the team that developed the Apollo flight software.
|
||||
|
||||
## Observations
|
||||
- [name] Margaret Hamilton
|
||||
- [role] Software engineer and systems designer
|
||||
- [expertise] Software engineering
|
||||
- [expertise] Systems design
|
||||
- [expertise] Error detection and recovery
|
||||
- [fact] Coined the term "software engineering"
|
||||
- [fact] Her software prevented an abort of the Apollo 11 moon landing
|
||||
- [achievement] Awarded the Presidential Medal of Freedom in 2016
|
||||
|
||||
## Relations
|
||||
- works_at [[MIT Instrumentation Laboratory]]
|
||||
- works_at [[NASA]]
|
||||
- created [[Apollo Flight Software]]
|
||||
@@ -0,0 +1,18 @@
|
||||
---
|
||||
title: Rasmus Lerdorf
|
||||
type: Person
|
||||
tags: [php, web, programming-languages]
|
||||
---
|
||||
|
||||
# Rasmus Lerdorf
|
||||
|
||||
Creator of the PHP programming language.
|
||||
|
||||
## Observations
|
||||
- [name] Rasmus Lerdorf
|
||||
- [expertise] Web development
|
||||
- [fact] Created PHP in 1994 originally as Personal Home Page Tools
|
||||
|
||||
## Relations
|
||||
- works_at [[Automattic]]
|
||||
- created [[PHP]]
|
||||
@@ -0,0 +1,23 @@
|
||||
---
|
||||
title: Richard Stallman
|
||||
type: Person
|
||||
tags: [free-software, gnu, emacs, gcc]
|
||||
---
|
||||
|
||||
# Richard Stallman
|
||||
|
||||
Founder of the Free Software Foundation and GNU Project.
|
||||
|
||||
## Observations
|
||||
- [name] Richard Stallman
|
||||
- [role] Software freedom activist and programmer
|
||||
- [expertise] Compiler design
|
||||
- [expertise] Software licensing
|
||||
- [fact] Founded the Free Software Foundation in 1985
|
||||
- [fact] Created GNU Emacs, GCC, and the GPL license
|
||||
|
||||
## Relations
|
||||
- works_at [[Free Software Foundation]]
|
||||
- created [[GNU Project]]
|
||||
- created [[GNU Emacs]]
|
||||
- created [[GCC]]
|
||||
@@ -0,0 +1,23 @@
|
||||
---
|
||||
title: Rob Pike
|
||||
type: Person
|
||||
tags: [go, plan9, utf8, bell-labs]
|
||||
---
|
||||
|
||||
# Rob Pike
|
||||
|
||||
Co-creator of Go, Plan 9, and UTF-8.
|
||||
|
||||
## Observations
|
||||
- [name] Rob Pike
|
||||
- [role] Software engineer and systems researcher
|
||||
- [expertise] Programming language design
|
||||
- [expertise] Distributed systems
|
||||
- [expertise] Text encoding
|
||||
- [fact] Co-created UTF-8 encoding with Ken Thompson
|
||||
|
||||
## Relations
|
||||
- works_at [[Google]]
|
||||
- co-created [[Go]]
|
||||
- co-created [[Plan 9]]
|
||||
- co-created [[UTF-8]]
|
||||
@@ -0,0 +1,18 @@
|
||||
---
|
||||
title: Satoshi Nakamoto
|
||||
type: Person
|
||||
tags: [bitcoin, cryptocurrency, blockchain]
|
||||
---
|
||||
|
||||
# Satoshi Nakamoto
|
||||
|
||||
Pseudonymous creator of Bitcoin and the blockchain concept.
|
||||
|
||||
## Observations
|
||||
- [name] Satoshi Nakamoto
|
||||
- [fact] Published the Bitcoin whitepaper in 2008
|
||||
- [fact] Mined the first Bitcoin block (genesis block) on January 3, 2009
|
||||
- [fact] True identity remains unknown
|
||||
|
||||
## Relations
|
||||
- created [[Bitcoin]]
|
||||
@@ -0,0 +1,25 @@
|
||||
---
|
||||
title: Tim Berners-Lee
|
||||
type: Person
|
||||
tags: [web, internet, w3c, html]
|
||||
---
|
||||
|
||||
# Tim Berners-Lee
|
||||
|
||||
Inventor of the World Wide Web and director of the W3C.
|
||||
|
||||
## Observations
|
||||
- [name] Tim Berners-Lee
|
||||
- [role] Computer scientist and web inventor
|
||||
- [expertise] Web architecture
|
||||
- [expertise] Distributed systems
|
||||
- [expertise] Information management
|
||||
- [email] timbl@w3.org
|
||||
- [fact] Invented the World Wide Web in 1989 while at CERN
|
||||
- [fact] Created the first web browser and web server
|
||||
|
||||
## Relations
|
||||
- works_at [[W3C]]
|
||||
- works_at [[MIT]]
|
||||
- created [[World Wide Web]]
|
||||
- created [[HTML]]
|
||||
@@ -0,0 +1,21 @@
|
||||
---
|
||||
title: Vint Cerf
|
||||
type: Person
|
||||
tags: [internet, tcp-ip, networking]
|
||||
---
|
||||
|
||||
# Vint Cerf
|
||||
|
||||
One of the fathers of the Internet, co-designer of TCP/IP.
|
||||
|
||||
## Observations
|
||||
- [name] Vint Cerf
|
||||
- [role] Internet pioneer and computer scientist
|
||||
- [expertise] Network protocols
|
||||
- [expertise] Internet architecture
|
||||
- [email] vcerf@google.com
|
||||
- [fact] Co-designed TCP/IP with Bob Kahn
|
||||
|
||||
## Relations
|
||||
- works_at [[Google]]
|
||||
- co-created [[TCP/IP]]
|
||||
@@ -0,0 +1,24 @@
|
||||
---
|
||||
title: Yann LeCun
|
||||
type: Person
|
||||
tags: [ai, deep-learning, convolutional-networks, meta]
|
||||
---
|
||||
|
||||
# Yann LeCun
|
||||
|
||||
Pioneer of convolutional neural networks and deep learning.
|
||||
|
||||
## Observations
|
||||
- [name] Yann LeCun
|
||||
- [role] VP and Chief AI Scientist
|
||||
- [expertise] Deep learning
|
||||
- [expertise] Convolutional neural networks
|
||||
- [expertise] Computer vision
|
||||
- [email] yann@cs.nyu.edu
|
||||
- [fact] Developed convolutional neural networks in the late 1980s
|
||||
- [achievement] Awarded the Turing Award in 2018
|
||||
|
||||
## Relations
|
||||
- works_at [[Meta AI]]
|
||||
- works_at [[New York University]]
|
||||
- created [[LeNet]]
|
||||
@@ -0,0 +1,19 @@
|
||||
---
|
||||
title: Yukihiro Matsumoto
|
||||
type: Person
|
||||
tags: [ruby, programming-languages, japan]
|
||||
---
|
||||
|
||||
# Yukihiro Matsumoto
|
||||
|
||||
Creator of the Ruby programming language, known as Matz.
|
||||
|
||||
## Observations
|
||||
- [name] Yukihiro Matsumoto
|
||||
- [role] Programming language designer and software engineer
|
||||
- [expertise] Programming language design
|
||||
- [expertise] Object-oriented programming
|
||||
- [fact] Created Ruby in 1995 with a focus on developer happiness
|
||||
|
||||
## Relations
|
||||
- created [[Ruby]]
|
||||
@@ -0,0 +1,22 @@
|
||||
---
|
||||
title: Book
|
||||
type: schema
|
||||
entity: Book
|
||||
version: 1
|
||||
schema:
|
||||
title: string, book title
|
||||
author?: string, author name
|
||||
published_year?: integer, year of publication
|
||||
genre?(enum): [fiction, nonfiction, technical]
|
||||
settings:
|
||||
validation: warn
|
||||
---
|
||||
|
||||
# Book
|
||||
|
||||
A book or publication tracked in the knowledge base. The genre field uses
|
||||
an enumeration to constrain values to fiction, nonfiction, or technical.
|
||||
|
||||
## Observations
|
||||
- [purpose] Tracks books with structured metadata including genre classification
|
||||
- [convention] published_year should be a four-digit year
|
||||
@@ -0,0 +1,22 @@
|
||||
---
|
||||
title: Meeting
|
||||
type: schema
|
||||
entity: Meeting
|
||||
version: 1
|
||||
schema:
|
||||
attendees(array): string, who was present
|
||||
decisions(array): string, what was decided
|
||||
action_items(array): string, follow-up tasks
|
||||
date?: string, when the meeting occurred
|
||||
settings:
|
||||
validation: warn
|
||||
---
|
||||
|
||||
# Meeting
|
||||
|
||||
A meeting or group discussion. Captures attendees, decisions made, and action items
|
||||
for follow-up. All array fields can have multiple entries.
|
||||
|
||||
## Observations
|
||||
- [purpose] Structured meeting notes that capture outcomes not just discussion
|
||||
- [convention] Action items should include the responsible person where possible
|
||||
@@ -0,0 +1,27 @@
|
||||
---
|
||||
title: Person
|
||||
type: schema
|
||||
entity: Person
|
||||
version: 1
|
||||
schema:
|
||||
name: string, full name
|
||||
role?: string, job title or position
|
||||
works_at?: Organization, employer
|
||||
expertise?(array): string, areas of knowledge
|
||||
email?: string, contact email
|
||||
settings:
|
||||
validation: warn
|
||||
---
|
||||
|
||||
# Person
|
||||
|
||||
A human individual in the knowledge graph. Represents people such as engineers,
|
||||
scientists, authors, founders, or any notable individual tracked in the knowledge base.
|
||||
|
||||
## Observations
|
||||
- [purpose] Defines the canonical structure for person notes
|
||||
- [convention] The name field is always required for identification
|
||||
|
||||
## Relations
|
||||
- used_by [[Paul Graham]]
|
||||
- used_by [[Rich Hickey]]
|
||||
@@ -0,0 +1,23 @@
|
||||
---
|
||||
title: SoftwareProject
|
||||
type: schema
|
||||
entity: SoftwareProject
|
||||
version: 1
|
||||
schema:
|
||||
name: string, project name
|
||||
language?: string, primary programming language
|
||||
license?: string, open source license type
|
||||
repo_url?: string, repository URL
|
||||
maintained_by?: Person, primary maintainer
|
||||
settings:
|
||||
validation: warn
|
||||
---
|
||||
|
||||
# SoftwareProject
|
||||
|
||||
A software project or library tracked in the knowledge base. Can be open source
|
||||
or proprietary. Links to the people who maintain it and the technologies used.
|
||||
|
||||
## Observations
|
||||
- [purpose] Tracks software projects and their key metadata
|
||||
- [convention] repo_url should be a full URL including protocol
|
||||
@@ -0,0 +1,22 @@
|
||||
---
|
||||
title: StrictSchema
|
||||
type: schema
|
||||
entity: StrictEntity
|
||||
version: 1
|
||||
schema:
|
||||
name: string, entity name
|
||||
category: string, required classification
|
||||
priority?: integer, priority level
|
||||
settings:
|
||||
validation: strict
|
||||
---
|
||||
|
||||
# StrictSchema
|
||||
|
||||
A schema with strict validation mode enabled. Notes validated against this schema
|
||||
will produce errors instead of warnings for missing required fields. Useful for
|
||||
CI/CD enforcement where structural compliance is mandatory.
|
||||
|
||||
## Observations
|
||||
- [purpose] Demonstrates strict validation mode for required field enforcement
|
||||
- [mode] Strict validation blocks sync on missing required fields
|
||||
@@ -0,0 +1,27 @@
|
||||
---
|
||||
title: Basic Memory
|
||||
type: SoftwareProject
|
||||
tags: [mcp, knowledge-management, local-first, python]
|
||||
---
|
||||
|
||||
# Basic Memory
|
||||
|
||||
Local-first knowledge management system built on the Model Context Protocol.
|
||||
Enables bidirectional communication between LLMs and markdown files to create
|
||||
a personal knowledge graph.
|
||||
|
||||
## Observations
|
||||
- [name] Basic Memory
|
||||
- [language] Python
|
||||
- [license] MIT
|
||||
- [repo_url] https://github.com/basicmachines-co/basic-memory
|
||||
- [fact] Uses SQLite for indexing and markdown files as source of truth
|
||||
- [fact] Provides 17 MCP tools for AI integration
|
||||
- [version] 0.13.8
|
||||
|
||||
## Relations
|
||||
- maintained_by [[Pedro Hernandez]]
|
||||
- uses [[Model Context Protocol]]
|
||||
- uses [[SQLite]]
|
||||
- uses [[FastAPI]]
|
||||
- integrates_with [[Claude Desktop]]
|
||||
@@ -0,0 +1,24 @@
|
||||
---
|
||||
title: Grace Hopper
|
||||
schema: Person
|
||||
tags: [computing, navy, compilers]
|
||||
---
|
||||
|
||||
# Grace Hopper
|
||||
|
||||
Pioneer of computer science and United States Navy rear admiral. This note uses
|
||||
an explicit schema reference rather than relying on type matching.
|
||||
|
||||
## Observations
|
||||
- [name] Grace Hopper
|
||||
- [role] Computer scientist and Navy rear admiral
|
||||
- [expertise] Compiler design
|
||||
- [expertise] Programming language theory
|
||||
- [email] grace.hopper@navy.mil
|
||||
- [fact] Developed one of the first linkers and popularized machine-independent languages
|
||||
- [fact] Coined the term "debugging" after finding an actual bug in a computer
|
||||
|
||||
## Relations
|
||||
- works_at [[United States Navy]]
|
||||
- contributed_to [[COBOL]]
|
||||
- contributed_to [[UNIVAC]]
|
||||
@@ -0,0 +1,17 @@
|
||||
---
|
||||
title: Random Thought About Coffee
|
||||
tags: [coffee, thoughts]
|
||||
---
|
||||
|
||||
# Random Thought About Coffee
|
||||
|
||||
Just a freeform note with no type and no schema. Should pass through validation
|
||||
with no checks applied.
|
||||
|
||||
## Observations
|
||||
- [thought] Ethiopian Yirgacheffe has a blueberry note that pairs well with dark chocolate
|
||||
- [preference] Pour over is superior to French press for lighter roasts
|
||||
- [fact] Coffee plants originated in Ethiopia
|
||||
|
||||
## Relations
|
||||
- relates_to [[Coffee Brewing Methods]]
|
||||
@@ -0,0 +1,29 @@
|
||||
---
|
||||
title: Paul Graham
|
||||
type: Person
|
||||
tags: [startups, essays, lisp, venture-capital]
|
||||
---
|
||||
|
||||
# Paul Graham
|
||||
|
||||
Co-founder of Y Combinator and prolific essayist whose writing on startups, programming,
|
||||
and technology has influenced a generation of founders and engineers.
|
||||
|
||||
## Observations
|
||||
- [name] Paul Graham
|
||||
- [role] Essayist, programmer, and venture capitalist
|
||||
- [expertise] Startups and early-stage investing
|
||||
- [expertise] Lisp and functional programming
|
||||
- [expertise] Essay writing and communication
|
||||
- [expertise] Programming language design
|
||||
- [email] pg@ycombinator.com
|
||||
- [fact] Co-created Viaweb, one of the first web applications, sold to Yahoo in 1998
|
||||
- [fact] Wrote "Hackers and Painters", a collection of essays on programming and startups
|
||||
- [born] 1964
|
||||
|
||||
## Relations
|
||||
- works_at [[Y Combinator]]
|
||||
- authored [[Hackers and Painters]]
|
||||
- authored [[On Lisp]]
|
||||
- founded [[Viaweb]]
|
||||
- influenced [[Sam Altman]]
|
||||
@@ -0,0 +1,24 @@
|
||||
---
|
||||
title: Rich Hickey
|
||||
type: Person
|
||||
tags: [clojure, functional-programming, databases]
|
||||
---
|
||||
|
||||
# Rich Hickey
|
||||
|
||||
Creator of Clojure and Datomic. Known for deep thinking about state, identity, and
|
||||
the problems with object-oriented programming.
|
||||
|
||||
## Observations
|
||||
- [name] Rich Hickey
|
||||
- [role] Language designer and software architect
|
||||
- [expertise] Programming language design
|
||||
- [expertise] Database architecture
|
||||
- [expertise] Functional programming
|
||||
- [fact] Spent two years developing Clojure before releasing it publicly
|
||||
- [fact] His talk "Simple Made Easy" is one of the most-watched programming talks
|
||||
|
||||
## Relations
|
||||
- works_at [[Nubank]]
|
||||
- created [[Clojure]]
|
||||
- created [[Datomic]]
|
||||
@@ -0,0 +1,32 @@
|
||||
---
|
||||
title: Team Standup 2024-01-15
|
||||
type: Meeting
|
||||
schema:
|
||||
attendees(array): string, who was there
|
||||
decisions(array): string, what was decided
|
||||
action_items(array): string, follow-ups
|
||||
blockers?(array): string, anything stuck
|
||||
date?: string, meeting date
|
||||
tags: [standup, team, weekly]
|
||||
---
|
||||
|
||||
# Team Standup 2024-01-15
|
||||
|
||||
Weekly standup meeting to review progress and plan the sprint.
|
||||
|
||||
## Observations
|
||||
- [attendees] Paul
|
||||
- [attendees] Sarah
|
||||
- [attendees] James
|
||||
- [attendees] Maria
|
||||
- [decisions] Ship v2 by Friday
|
||||
- [decisions] Defer mobile app to next quarter
|
||||
- [action_items] Paul to review PR #42
|
||||
- [action_items] Sarah to update API docs
|
||||
- [action_items] James to fix flaky CI tests
|
||||
- [blockers] Waiting on API credentials from vendor
|
||||
- [date] 2024-01-15
|
||||
|
||||
## Relations
|
||||
- part_of [[Q1 2024 Sprint]]
|
||||
- followed_by [[Team Standup 2024-01-22]]
|
||||
@@ -0,0 +1,27 @@
|
||||
---
|
||||
title: Nikola Tesla
|
||||
type: Person
|
||||
tags: [electricity, invention, engineering]
|
||||
---
|
||||
|
||||
# Nikola Tesla
|
||||
|
||||
Inventor and electrical engineer. This note has all required fields plus
|
||||
observations not defined in the Person schema. Extra observations should
|
||||
generate informational messages, not warnings.
|
||||
|
||||
## Observations
|
||||
- [name] Nikola Tesla
|
||||
- [role] Inventor and electrical engineer
|
||||
- [expertise] Electrical engineering
|
||||
- [expertise] Alternating current systems
|
||||
- [nationality] Serbian-American
|
||||
- [born] 1856
|
||||
- [died] 1943
|
||||
- [patent_count] Over 300 patents worldwide
|
||||
- [fact] Developed the alternating current electrical system
|
||||
- [fact] Had an eidetic memory and could visualize complex machines in his mind
|
||||
|
||||
## Relations
|
||||
- works_at [[Westinghouse Electric]]
|
||||
- rival_of [[Thomas Edison]]
|
||||
@@ -0,0 +1,19 @@
|
||||
---
|
||||
title: Alan Kay
|
||||
type: Person
|
||||
tags: [computing, oop, xerox-parc]
|
||||
---
|
||||
|
||||
# Alan Kay
|
||||
|
||||
Computer scientist known for pioneering object-oriented programming.
|
||||
This note is intentionally missing both required [name] AND optional [role].
|
||||
|
||||
## Observations
|
||||
- [fact] Coined the phrase "The best way to predict the future is to invent it"
|
||||
- [expertise] Object-oriented programming
|
||||
- [expertise] User interface design
|
||||
- [born] 1940
|
||||
|
||||
## Relations
|
||||
- works_at [[Xerox PARC]]
|
||||
@@ -0,0 +1,19 @@
|
||||
---
|
||||
title: Ada Lovelace
|
||||
type: Person
|
||||
tags: [computing, mathematics, history]
|
||||
---
|
||||
|
||||
# Ada Lovelace
|
||||
|
||||
Mathematician and writer, often regarded as the first computer programmer.
|
||||
This note is intentionally missing the required [name] observation.
|
||||
|
||||
## Observations
|
||||
- [fact] Wrote the first computer program for Charles Babbage's Analytical Engine
|
||||
- [fact] Was the daughter of poet Lord Byron
|
||||
- [born] 1815
|
||||
- [role] Mathematician and writer
|
||||
|
||||
## Relations
|
||||
- collaborated_with [[Charles Babbage]]
|
||||
@@ -0,0 +1,21 @@
|
||||
---
|
||||
title: The Art of War
|
||||
type: Book
|
||||
tags: [strategy, military, classic]
|
||||
---
|
||||
|
||||
# The Art of War
|
||||
|
||||
Ancient Chinese military treatise attributed to Sun Tzu. This note has a genre
|
||||
value that is not in the allowed enum set [fiction, nonfiction, technical].
|
||||
|
||||
## Observations
|
||||
- [title] The Art of War
|
||||
- [author] Sun Tzu
|
||||
- [published_year] -500
|
||||
- [genre] philosophy
|
||||
- [fact] One of the oldest and most influential works on military strategy
|
||||
|
||||
## Relations
|
||||
- written_by [[Sun Tzu]]
|
||||
- influenced [[Machiavelli]]
|
||||
@@ -0,0 +1 @@
|
||||
"""Integration tests for the Basic Memory schema system."""
|
||||
@@ -0,0 +1,85 @@
|
||||
"""Shared fixtures for schema integration tests.
|
||||
|
||||
Helper functions and path constants live in helpers.py for explicit import.
|
||||
This file only contains pytest fixtures (auto-injected by pytest).
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from test_schema.helpers import (
|
||||
SCHEMAS_DIR,
|
||||
VALID_DIR,
|
||||
WARNINGS_DIR,
|
||||
EDGE_CASES_DIR,
|
||||
INFERENCE_DIR,
|
||||
DRIFT_SCHEMA_DIR,
|
||||
DRIFT_PEOPLE_DIR,
|
||||
parse_frontmatter,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def schemas_dir() -> Path:
|
||||
return SCHEMAS_DIR
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def valid_dir() -> Path:
|
||||
return VALID_DIR
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def warnings_dir() -> Path:
|
||||
return WARNINGS_DIR
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def edge_cases_dir() -> Path:
|
||||
return EDGE_CASES_DIR
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def inference_dir() -> Path:
|
||||
return INFERENCE_DIR
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def drift_schema_dir() -> Path:
|
||||
return DRIFT_SCHEMA_DIR
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def drift_people_dir() -> Path:
|
||||
return DRIFT_PEOPLE_DIR
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def person_schema_frontmatter(schemas_dir) -> dict:
|
||||
"""Load Person schema frontmatter from fixture."""
|
||||
return parse_frontmatter(schemas_dir / "Person.md")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def book_schema_frontmatter(schemas_dir) -> dict:
|
||||
"""Load Book schema frontmatter from fixture."""
|
||||
return parse_frontmatter(schemas_dir / "Book.md")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def meeting_schema_frontmatter(schemas_dir) -> dict:
|
||||
"""Load Meeting schema frontmatter from fixture."""
|
||||
return parse_frontmatter(schemas_dir / "Meeting.md")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def software_project_schema_frontmatter(schemas_dir) -> dict:
|
||||
"""Load SoftwareProject schema frontmatter from fixture."""
|
||||
return parse_frontmatter(schemas_dir / "SoftwareProject.md")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def strict_schema_frontmatter(schemas_dir) -> dict:
|
||||
"""Load StrictSchema frontmatter from fixture."""
|
||||
return parse_frontmatter(schemas_dir / "StrictSchema.md")
|
||||
@@ -0,0 +1,61 @@
|
||||
"""Shared helpers and constants for schema integration tests.
|
||||
|
||||
Separated from conftest.py so they can be explicitly imported by test modules.
|
||||
Conftest fixtures are auto-injected by pytest and don't need explicit import.
|
||||
"""
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
from basic_memory.schema.inference import ObservationData, RelationData
|
||||
|
||||
|
||||
# --- Fixture Paths ---
|
||||
|
||||
FIXTURES_ROOT = Path(__file__).parent.parent / "fixtures" / "schema"
|
||||
SCHEMAS_DIR = FIXTURES_ROOT / "schemas"
|
||||
VALID_DIR = FIXTURES_ROOT / "valid"
|
||||
WARNINGS_DIR = FIXTURES_ROOT / "warnings"
|
||||
EDGE_CASES_DIR = FIXTURES_ROOT / "edge-cases"
|
||||
INFERENCE_DIR = FIXTURES_ROOT / "inference" / "people"
|
||||
DRIFT_SCHEMA_DIR = FIXTURES_ROOT / "drift" / "schema"
|
||||
DRIFT_PEOPLE_DIR = FIXTURES_ROOT / "drift" / "people"
|
||||
|
||||
|
||||
# --- Frontmatter Parsing ---
|
||||
|
||||
|
||||
def parse_frontmatter(filepath: Path) -> dict:
|
||||
"""Extract YAML frontmatter from a markdown file."""
|
||||
text = filepath.read_text(encoding="utf-8")
|
||||
match = re.match(r"^---\n(.*?)\n---", text, re.DOTALL)
|
||||
if not match:
|
||||
return {}
|
||||
return yaml.safe_load(match.group(1)) or {}
|
||||
|
||||
|
||||
def parse_observations(filepath: Path) -> list[ObservationData]:
|
||||
"""Extract ObservationData from a markdown file.
|
||||
|
||||
Parses lines matching: - [category] content
|
||||
"""
|
||||
text = filepath.read_text(encoding="utf-8")
|
||||
pattern = re.compile(r"^- \[([^\]]+)\] (.+)$", re.MULTILINE)
|
||||
return [ObservationData(m.group(1), m.group(2)) for m in pattern.finditer(text)]
|
||||
|
||||
|
||||
def parse_relations(filepath: Path) -> list[RelationData]:
|
||||
"""Extract RelationData from a markdown file.
|
||||
|
||||
Parses lines matching: - relation_type [[Target]]
|
||||
Only matches lines under a ## Relations heading.
|
||||
"""
|
||||
text = filepath.read_text(encoding="utf-8")
|
||||
relations_match = re.search(r"## Relations\n(.*?)(?=\n## |\Z)", text, re.DOTALL)
|
||||
if not relations_match:
|
||||
return []
|
||||
relations_text = relations_match.group(1)
|
||||
pattern = re.compile(r"^- (\S+) \[\[([^\]]+)\]\]$", re.MULTILINE)
|
||||
return [RelationData(m.group(1), m.group(2)) for m in pattern.finditer(relations_text)]
|
||||
@@ -0,0 +1,72 @@
|
||||
"""Integration tests for schema drift detection using drift fixtures."""
|
||||
|
||||
import pytest
|
||||
|
||||
from basic_memory.schema.parser import parse_schema_note
|
||||
from basic_memory.schema.diff import diff_schema
|
||||
from basic_memory.schema.inference import NoteData
|
||||
|
||||
from test_schema.helpers import (
|
||||
parse_frontmatter,
|
||||
parse_observations,
|
||||
parse_relations,
|
||||
DRIFT_SCHEMA_DIR,
|
||||
DRIFT_PEOPLE_DIR,
|
||||
)
|
||||
|
||||
|
||||
def load_drift_notes() -> list[NoteData]:
|
||||
"""Load all drift person fixture files as NoteData objects."""
|
||||
return [
|
||||
NoteData(
|
||||
identifier=filepath.stem,
|
||||
observations=parse_observations(filepath),
|
||||
relations=parse_relations(filepath),
|
||||
)
|
||||
for filepath in sorted(DRIFT_PEOPLE_DIR.glob("*.md"))
|
||||
]
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def drift_schema():
|
||||
frontmatter = parse_frontmatter(DRIFT_SCHEMA_DIR / "Person.md")
|
||||
return parse_schema_note(frontmatter)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def drift_notes():
|
||||
return load_drift_notes()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def drift_result(drift_schema, drift_notes):
|
||||
return diff_schema(drift_schema, drift_notes)
|
||||
|
||||
|
||||
class TestDriftCorpusSize:
|
||||
def test_minimum_drift_notes(self, drift_notes):
|
||||
assert len(drift_notes) >= 20
|
||||
|
||||
|
||||
class TestNewFieldDetection:
|
||||
"""github field appears in 60% of notes but is not in the schema."""
|
||||
|
||||
def test_github_detected_as_new_field(self, drift_result):
|
||||
new_field_names = [f.name for f in drift_result.new_fields]
|
||||
assert "github" in new_field_names
|
||||
|
||||
|
||||
class TestDroppedFieldDetection:
|
||||
"""role field is in schema but only 5% of notes have it."""
|
||||
|
||||
def test_role_detected_as_dropped(self, drift_result):
|
||||
dropped_names = [f.name for f in drift_result.dropped_fields]
|
||||
assert "role" in dropped_names
|
||||
|
||||
|
||||
class TestCardinalityChanges:
|
||||
"""works_at changed from single to multiple per note."""
|
||||
|
||||
def test_works_at_cardinality_change_detected(self, drift_result):
|
||||
changes = [msg for msg in drift_result.cardinality_changes if "works_at" in msg]
|
||||
assert len(changes) >= 1
|
||||
@@ -0,0 +1,112 @@
|
||||
"""Integration tests for schema inference using the 30+ person fixture corpus."""
|
||||
|
||||
import pytest
|
||||
|
||||
from basic_memory.schema.inference import infer_schema, NoteData
|
||||
|
||||
from test_schema.helpers import parse_observations, parse_relations, INFERENCE_DIR
|
||||
|
||||
|
||||
def load_inference_notes() -> list[NoteData]:
|
||||
"""Load all person fixture files as NoteData objects."""
|
||||
return [
|
||||
NoteData(
|
||||
identifier=filepath.stem,
|
||||
observations=parse_observations(filepath),
|
||||
relations=parse_relations(filepath),
|
||||
)
|
||||
for filepath in sorted(INFERENCE_DIR.glob("*.md"))
|
||||
]
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def person_notes() -> list[NoteData]:
|
||||
return load_inference_notes()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def inference_result(person_notes):
|
||||
return infer_schema("Person", person_notes)
|
||||
|
||||
|
||||
class TestInferenceCorpusSize:
|
||||
def test_minimum_notes_loaded(self, person_notes):
|
||||
assert len(person_notes) >= 30
|
||||
|
||||
def test_notes_analyzed_matches(self, inference_result, person_notes):
|
||||
assert inference_result.notes_analyzed == len(person_notes)
|
||||
|
||||
|
||||
class TestNameFieldInference:
|
||||
"""Name should be ~100% -> required."""
|
||||
|
||||
def test_name_is_required(self, inference_result):
|
||||
assert "name" in inference_result.suggested_required
|
||||
|
||||
def test_name_frequency_near_100(self, inference_result):
|
||||
name_freq = next(f for f in inference_result.field_frequencies if f.name == "name")
|
||||
assert name_freq.percentage >= 0.95
|
||||
|
||||
|
||||
class TestRoleFieldInference:
|
||||
"""Role should be ~90% -> optional."""
|
||||
|
||||
def test_role_is_optional(self, inference_result):
|
||||
assert "role" in inference_result.suggested_optional
|
||||
|
||||
def test_role_frequency_around_90(self, inference_result):
|
||||
role_freq = next(f for f in inference_result.field_frequencies if f.name == "role")
|
||||
assert 0.80 <= role_freq.percentage < 0.95
|
||||
|
||||
|
||||
class TestExpertiseFieldInference:
|
||||
"""Expertise should be ~60% -> optional, array."""
|
||||
|
||||
def test_expertise_is_optional(self, inference_result):
|
||||
assert "expertise" in inference_result.suggested_optional
|
||||
|
||||
def test_expertise_is_array(self, inference_result):
|
||||
freq = next(f for f in inference_result.field_frequencies if f.name == "expertise")
|
||||
assert freq.is_array is True
|
||||
|
||||
|
||||
class TestEmailFieldInference:
|
||||
"""Email should be ~27% -> optional."""
|
||||
|
||||
def test_email_is_optional(self, inference_result):
|
||||
assert "email" in inference_result.suggested_optional
|
||||
|
||||
|
||||
class TestBornFieldInference:
|
||||
"""Born should be ~17% -> excluded."""
|
||||
|
||||
def test_born_is_excluded(self, inference_result):
|
||||
assert "born" in inference_result.excluded
|
||||
|
||||
def test_born_frequency_below_threshold(self, inference_result):
|
||||
born_freq = next(f for f in inference_result.field_frequencies if f.name == "born")
|
||||
assert born_freq.percentage < 0.25
|
||||
|
||||
|
||||
class TestWorksAtRelationInference:
|
||||
"""works_at relation should be ~73% -> optional."""
|
||||
|
||||
def test_works_at_is_optional(self, inference_result):
|
||||
assert "works_at" in inference_result.suggested_optional
|
||||
|
||||
def test_works_at_is_relation(self, inference_result):
|
||||
freq = next(f for f in inference_result.field_frequencies if f.name == "works_at")
|
||||
assert freq.source == "relation"
|
||||
|
||||
|
||||
class TestSuggestedSchema:
|
||||
def test_suggested_schema_not_empty(self, inference_result):
|
||||
assert len(inference_result.suggested_schema) > 0
|
||||
|
||||
def test_excluded_fields_not_in_schema(self, inference_result):
|
||||
schema_names = set()
|
||||
for key in inference_result.suggested_schema:
|
||||
name = key.replace("?", "").split("(")[0]
|
||||
schema_names.add(name)
|
||||
for excluded in inference_result.excluded:
|
||||
assert excluded not in schema_names
|
||||
@@ -0,0 +1,83 @@
|
||||
"""Integration tests for the Picoschema parser using real fixture files."""
|
||||
|
||||
import pytest
|
||||
|
||||
from basic_memory.schema.parser import parse_schema_note
|
||||
|
||||
|
||||
class TestPersonSchemaParsing:
|
||||
"""Parse the Person.md schema fixture and verify all fields."""
|
||||
|
||||
def test_person_schema_entity(self, person_schema_frontmatter):
|
||||
schema = parse_schema_note(person_schema_frontmatter)
|
||||
assert schema.entity == "Person"
|
||||
|
||||
def test_person_schema_version(self, person_schema_frontmatter):
|
||||
schema = parse_schema_note(person_schema_frontmatter)
|
||||
assert schema.version == 1
|
||||
|
||||
def test_person_schema_validation_mode(self, person_schema_frontmatter):
|
||||
schema = parse_schema_note(person_schema_frontmatter)
|
||||
assert schema.validation_mode == "warn"
|
||||
|
||||
def test_person_schema_field_count(self, person_schema_frontmatter):
|
||||
schema = parse_schema_note(person_schema_frontmatter)
|
||||
assert len(schema.fields) == 5
|
||||
|
||||
def test_person_name_field_is_required(self, person_schema_frontmatter):
|
||||
schema = parse_schema_note(person_schema_frontmatter)
|
||||
name_field = next(f for f in schema.fields if f.name == "name")
|
||||
assert name_field.required is True
|
||||
assert name_field.type == "string"
|
||||
assert name_field.description == "full name"
|
||||
|
||||
def test_person_role_field_is_optional(self, person_schema_frontmatter):
|
||||
schema = parse_schema_note(person_schema_frontmatter)
|
||||
role_field = next(f for f in schema.fields if f.name == "role")
|
||||
assert role_field.required is False
|
||||
|
||||
def test_person_works_at_is_entity_ref(self, person_schema_frontmatter):
|
||||
schema = parse_schema_note(person_schema_frontmatter)
|
||||
works_at = next(f for f in schema.fields if f.name == "works_at")
|
||||
assert works_at.is_entity_ref is True
|
||||
assert works_at.type == "Organization"
|
||||
|
||||
def test_person_expertise_is_array(self, person_schema_frontmatter):
|
||||
schema = parse_schema_note(person_schema_frontmatter)
|
||||
expertise = next(f for f in schema.fields if f.name == "expertise")
|
||||
assert expertise.is_array is True
|
||||
|
||||
def test_person_email_is_optional(self, person_schema_frontmatter):
|
||||
schema = parse_schema_note(person_schema_frontmatter)
|
||||
email = next(f for f in schema.fields if f.name == "email")
|
||||
assert email.required is False
|
||||
|
||||
|
||||
class TestBookSchemaParsing:
|
||||
"""Parse the Book.md schema fixture."""
|
||||
|
||||
def test_book_genre_is_enum(self, book_schema_frontmatter):
|
||||
schema = parse_schema_note(book_schema_frontmatter)
|
||||
genre = next(f for f in schema.fields if f.name == "genre")
|
||||
assert genre.is_enum is True
|
||||
assert set(genre.enum_values) == {"fiction", "nonfiction", "technical"}
|
||||
|
||||
|
||||
class TestStrictSchemaParsing:
|
||||
"""Verify strict validation mode is parsed."""
|
||||
|
||||
def test_strict_validation_mode(self, strict_schema_frontmatter):
|
||||
schema = parse_schema_note(strict_schema_frontmatter)
|
||||
assert schema.validation_mode == "strict"
|
||||
|
||||
|
||||
class TestSchemaParsingErrors:
|
||||
"""Verify parser raises on invalid frontmatter."""
|
||||
|
||||
def test_missing_entity_raises(self):
|
||||
with pytest.raises(ValueError, match="entity"):
|
||||
parse_schema_note({"type": "schema", "schema": {"name": "string"}})
|
||||
|
||||
def test_missing_schema_dict_raises(self):
|
||||
with pytest.raises(ValueError, match="schema"):
|
||||
parse_schema_note({"type": "schema", "entity": "Person"})
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user