From 01d46727b40c24b017ea9db4b741daef565ac73e Mon Sep 17 00:00:00 2001 From: phernandez Date: Thu, 13 Feb 2025 21:18:25 -0600 Subject: [PATCH] fix: bump project patch version --- tasks.md | 458 ------------------------------------------------------- 1 file changed, 458 deletions(-) delete mode 100644 tasks.md diff --git a/tasks.md b/tasks.md deleted file mode 100644 index b68fa31b..00000000 --- a/tasks.md +++ /dev/null @@ -1,458 +0,0 @@ -# Basic Memory Tasks - -## Current Focus - -### Observation Management - -Implement update/remove functionality for observations with a focus on maintainability and consistency with our " -filesystem is source of truth" principle. - -Options under consideration: - -1. Bulk Update Approach - - Update all observations at once - - Pros: - - Simpler file operations - - No need to match on observation content - - Easier database synchronization - - Very consistent with "filesystem is source of truth" - - Cons: - - Less efficient - rewrites everything for small changes - - Potential concurrency implications - -2. Tracked Observations Approach - - Use markdown comments for observation IDs - ```markdown - # Entity Name - type: entity_type - - ## Observations - - - This is an observation - ``` - - Pros: - - Can track individual observations - - Enables precise updates/deletes - - Cons: - - More complex markdown parsing - - IDs visible in markdown - -3. Diff-based Approach - - Implement observation-aware diffing - - Track changes at observation level - - Pros: - - More efficient updates - - Preserves manual edits - - Cons: - - More complex implementation - - Need to handle merge conflicts - -4. Position-based Management - - Track observations by their position/order - - Pros: - - No need for explicit IDs - - Clean markdown - - Cons: - - Fragile if order changes - - Hard to handle concurrent edits - -## Completed - -- [x] Extract file operations to fileio.py module -- [x] Update EntityService to use fileio functions -- [x] Initial ObservationService implementation -- [x] Basic test coverage - -## Future Work - -- [ ] Implement observation updates/removals (exploring options above) -- [ ] Proper session management for concurrent operations -- [ ] EntityService tests using new fileio module -- [ ] More sophisticated search functionality -- [ ] Handle markdown formatting edge cases - -## TODO - -### refactor input schema - -1. Observations Format: - Old (JSON) way I tried first: - -```python -"observations": ["First observation", "Second observation"] # Simple string array -``` - -New required format: - -```python -"observations": [ - {"content": "First observation"}, - {"content": "Second observation"} -] # Array of objects with content field -``` - -2. Relations Format: - Old way: - -```python -"relations": [ - {"from": "EntityName", "to": "OtherEntity", "relationType": "relates_to"} # Using names -] -``` - -New format: - -```python -"relations": [ - {"fromId": "20241210-entity-id", "toId": "20241210-other-id", "relationType": "relates_to"} # Using IDs -] -``` - -My preferences: - -1. For observations: The simple string array felt more intuitive for basic use, but I can see the benefits of the object - format: - - Allows for additional metadata (context, timestamps, etc.) - - More explicit about what each field means - - Consistent with how we'd want to store this in a database - -2. For relations: Using IDs is technically better but requires an extra lookup step in my workflow: - - I have to first create the entities to get their IDs - - Then use those IDs to create relations - - Makes it harder to create entities and relations in a single step - -Suggestions for making it more intuitive: - -1. For observations: We could have a helper function that accepts either format: - -```python -# Both would work: -create_entities([{ - "name": "Entity", - "entityType": "type", - "observations": ["Simple string obs"] # Auto-converted to object format -}]) - -create_entities([{ - "name": "Entity", - "entityType": "type", - "observations": [{"content": "Full object obs"}] # Native format -}]) -``` - -2. For relations: Maybe allow a name-based helper function: - -```python -# Instead of requiring IDs: -create_relations_by_name([{ - "from": "EntityName", - "to": "OtherEntity", - "relationType": "relates_to" -}]) -``` - -3. A combined creation function for when we want to create entities and their relations together: - -```python -create_entity_with_relations({ - "entity": { - "name": "NewEntity", - "entityType": "type", - "observations": ["Obs 1", "Obs 2"] - }, - "relations": [{ - "to": "ExistingEntity", - "relationType": "relates_to" - }] -}) -``` - -The current schema is more "correct" from a software engineering perspective, but these helpers could make it more -natural to use while maintaining the rigorous underlying structure. What do you think about these suggestions? Would you -prefer to keep it strict for clarity, or add some convenience layers? - -### entity file organization - -- my problems - - wall of text is hard to scan visually - - files are not ordered or grouped granular enough. Only `_.md` - - hard to tell when a new entity file is added - -Possible fixes: - -- use subdirectories? - -## TASKS - -1. **Core Functionality Improvements** - - [x] entity.description addition - - [x] subdirectories - - Paul needs this for markdown view - - [x] refine directory sprawl - - [ ] improve tool api - - [ ] Choose and implement observation update/removal strategy - - [ ] Handle relationship updates in files - - [ ] Complete full CRUD operations - - delete - - [ ] Improve search functionality (currently broken as we discovered) - -### Suggested Sequence - -1. **Schema Update First** - - Add `entity.description` field - - rename entity.references? - - This affects database, Pydantic models, and file format - - Good foundation for other changes - -2. **File Organization** - - Add subdirectory support - - Affects: - - File path handling - - Entity loading/saving - - URI resolution - - Will make Paul's markdown viewing experience better - -3. **Tool API Improvements** - - Cleaner input/output schemas - - More consistent patterns - - Better error handling - - This sets us up for implementing the remaining operations - -4. **Core Operations** - - Implement delete operations - - Update/remove observations - - Relationship updates in files - - Building on the improved API - -5. **Search Fix** - - Can properly tackle this after file organization - - Will benefit from improved schema - -Would you like me to: -1. Start with the schema update for entity.description? -2. Plan out the subdirectory implementation? -3. Or focus on a different area? - -I think the schema update would be a clean, contained change to start with, but I'm happy to tackle whichever part you think would be most valuable first. - - -2. **Robustness & Testing** - - Fix DI issues - - Learn from fastmcp patterns - - Markdown service - - markdown.py - -python-frontmatter - - Complete test coverage - - Expand testing across services - - 100% coverate - - Improve error handling and logging - - Add comprehensive type hints - -3. **Architecture Improvements** - - Handle concurrent file operations safely - - Implement proper session management - - Balance file operations and DB sync - - Handle markdown formatting edge cases - -4. **Documentation & Infrastructure** - - Document core modules - - Implement proper backup strategy - - Add file change versioning - - Improve CLI interface - - -## Ideas - -- need update tool - -### 2-way sync - -- Enable updates to the markdown files to be able to be seen by AI - - possible via tool sync - - filesystem notifications via agent? - - Claude can use `file_write` tool to edit Entity files also - -### Projects - -- support multiple projects -- figure out flow - - load project at startup? - - switch project during chat? - -### References - -- better support for ref:// references -- should we call them memory:/// -- use Prompt to invoke reference uri - -### Entity File format - -- Consider using wiki link format: - [[20241210-ai_human_collaboration_model]] improves - -instead of -[20241210-basic-memory] validates_design_of - -- use frontmatter for file header - -### Code - -- cleanup server.py -- use markdown.py? -- fastmcp? -- Entity.description - -### Other Features - -- Search -- webui -- python notebook? -- vector eb extension for sqlite - -#### Obsidian - -- use as file viewer editor -- create knowledge graph? - -#### Save chats - -I'll sketch out a potential schema for chat sessions in our system: - -``` -{ - `entities`: [ - { - `name`: `Chat_Entity_Design`, - `entityType`: `design_proposal`, - `observations`: [ - { - `content`: `Entity Schema: -type: chat_session -date: ISO timestamp -participants: [human, ai] -topic: string -context_refs: [memory:///] -key_decisions: [string] -summary: string` - }, - { - `content`: `File Format (frontmatter): ---- -type: chat_session -date: 2024-12-10T10:30:00Z -participants: - - human: paul - - ai: claude -topic: Chat Session Storage Design -context_refs: - - memory://basic-memory/20241210-chat-entity-design -key_decisions: - - Store chats as first-class entities - - Use memory:// URI scheme -summary: Designed schema for chat storage ----` - }, - { - `content`: `Example Relations: -- chat_session references design_proposal -- chat_session implements feature -- chat_session discusses concept -- chat_session continues_from previous_chat` - }, - { - `content`: `Auto-generation approach: -1. At end of chat, Claude summarizes key points -2. Creates chat entity with metadata -3. Adds relations to referenced entities -4. Updates any entities modified during chat` - }, - { - `content`: `Integration points: -- memory:// URI scheme for references -- Relations to track context chain -- File format matches entity schema -- Automatic metadata generation` - } - ] - } - ] -} -``` - -``` -{ - `entities`: [ - { - `name`: `Chat_Session_Workflow`, - `entityType`: `process`, - `observations`: [ - { - `content`: `Start of chat: -- Load any referenced context via memory:// URIs -- Initialize new chat session metadata -- Track entities being discussed/modified` - }, - { - `content`: `During chat: -- Update references as new entities are mentioned -- Track key decisions and changes -- Note any new relations discovered` - }, - { - `content`: `End of chat: -- Generate summary and key points -- Create chat session entity -- Add relations to referenced entities -- Update any modified entities` - }, - { - `content`: `Future chat reference: -- Can reference previous chat via memory:// URI -- Loads relevant context and decisions -- Creates continues_from relation automatically` - } - ] - } - ] -} -``` - -Key aspects of this design: - -1. Chat Session as Entity - -- Clear metadata (participants, date, topic) -- Explicit references to other entities via memory:// URIs -- Captures key decisions and summary -- Uses frontmatter for clean file format - -2. Rich Relations - -- Can track chat-to-chat continuity -- Links to entities discussed/modified -- Captures purpose (discusses, implements, etc.) -- Maintains context chain - -3. Automated Workflow - -- I could generate metadata during chat -- Create summary and relations at end -- Update any modified entities -- Makes context loading explicit - -4. Integration - -- memory:// URI scheme fits naturally -- File format matches our entity pattern -- Two-way links between chats and entities -- Clear context persistence model - -The nice thing about this design is that it makes chat sessions "first-class citizens" in our knowledge graph while -keeping everything in human-readable markdown files. - -Would you like me to: - -1. Sketch out what this chat would look like as an entity? -2. Detail the MCP tool changes needed? -3. Mock up the file format more precisely? -4. Something else?