Replace delete-then-insert pattern with INSERT ... ON CONFLICT for
PostgreSQL search index operations. This fixes race conditions where
parallel entity indexing could cause UniqueViolationError on the
uix_search_index_permalink_project constraint.
Changes:
- Add index_item() override in PostgresSearchRepository with upsert
- Update bulk_index_items() to use ON CONFLICT (permalink, project_id)
- Add CREATE_POSTGRES_SEARCH_INDEX_PERMALINK DDL for test fixtures
- Add tests for upsert behavior on duplicate permalinks
Technical note: Use column-based ON CONFLICT syntax instead of
ON CONSTRAINT (which only works for table constraints, not indexes).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Importers now use relative file paths (based on permalink) instead of
absolute paths. This enables proper S3 key generation in cloud environments.
Changes:
- write_entity() accepts str | Path for file_path parameter
- ensure_folder_exists() uses relative paths directly
- All importers pass relative paths to FileService
- FileService handles base_path resolution internally
This prevents S3 keys from including container filesystem paths like
`/app/basic-memory/imports/...` and instead uses clean relative paths
like `imports/20241010-file.md`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Refactor importers to use FileService for all file operations instead of
direct filesystem calls. This enables cloud environments to override file
operations via dependency injection (e.g., S3FileService).
Changes:
- Add `to_markdown_string()` method to MarkdownProcessor for content
serialization without file I/O
- Update Importer base class to accept FileService and use it for:
- `write_entity()` - now uses FileService.write_file()
- `ensure_folder_exists()` - now async, uses FileService.ensure_directory()
- Fix direct `mkdir()` calls in:
- claude_projects_importer.py
- memory_json_importer.py
- Update deps.py to inject FileService into all importers (v1 and v2)
- Update CLI commands to create and pass FileService to importers
- Update tests to work with new FileService dependency
This follows the pattern used by /knowledge API and SyncService, enabling
cloud to override file operations by providing S3FileService via DI.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Add `allow_discovery` parameter to `resolve_project_parameter()` that
enables tools like `recent_activity` to work across all projects in
cloud mode without requiring a project parameter.
- Add `allow_discovery: bool = False` param to resolve_project_parameter
- In cloud mode with allow_discovery=True, return None instead of error
- Update recent_activity to use allow_discovery=True
- Fix circular import by deferring call_get import inside functions
- Add comprehensive tests for resolve_project_parameter
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Fixes#452 - Imported conversations not fully indexed
Files with UTF-8 BOM (Byte Order Mark) at the start would fail frontmatter
detection, causing:
- Title to fall back to filename instead of frontmatter value
- Permalink to be null in the database
Added strip_bom() helper function and updated all frontmatter-related
functions to strip BOM before processing:
- has_frontmatter()
- parse_frontmatter()
- remove_frontmatter()
- EntityParser.parse_markdown_content()
Added comprehensive tests for BOM handling with various scenarios.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
The API Pydantic schema had a MaxLen(1000) constraint on observation
content while the database uses SQLAlchemy's Text type (unlimited).
This mismatch caused validation errors when observations exceeded
1000 characters (e.g., JSON schemas with 1458+ chars).
Removed the MaxLen constraint to match the DB schema. Retained:
- BeforeValidator(str.strip) for whitespace cleaning
- MinLen(1) to ensure non-empty content
Added comprehensive tests to verify:
- Long content (10K+ chars) is accepted
- Very long content (50K+ chars) is accepted
- Empty/whitespace-only content is still rejected
- Whitespace stripping still works
Fixes#385🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
When a file exists in the database but is missing from the filesystem,
the sync worker now treats this as a deletion instead of crashing.
The sync_file() method catches FileNotFoundError specifically and calls
handle_delete() to clean up the orphaned database record. This prevents
the sync from failing on database/filesystem inconsistencies that can
occur due to race conditions, manual file deletions, or cloud storage
caching issues.
Includes a test to verify the graceful handling behavior.
Fixes#386🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Denormalizes project_id onto Relation and Observation tables to enable
efficient project-scoped queries without joins. Migration backfills
from associated entity and adds pg_trgm extension with GIN indexes
for fuzzy link resolution on PostgreSQL.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Add test confirming entity_repository.update() returns the entity with
observations and relations eagerly loaded, eliminating the need for a
separate find_by_id() call after update.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Add add_all_ignore_duplicates() method to RelationRepository for bulk
inserting relations with ON CONFLICT DO NOTHING. This handles cases
where the same [[wiki link]] appears multiple times in a document,
silently ignoring duplicates based on the (from_id, to_name, relation_type)
unique constraint.
Works with both SQLite and PostgreSQL dialects.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Add optimized repository methods for resolve_permalink() that skip
eager loading of observations and relations:
- permalink_exists(): Check existence without loading entity
- get_file_path_for_permalink(): Get only file_path column
- get_permalink_for_file_path(): Get only permalink column
- get_all_permalinks(): Get all permalinks as strings
- get_permalink_to_file_path_map(): Bulk lookup mapping
- get_file_path_to_permalink_map(): Reverse mapping
Updated entity_service.resolve_permalink() to use these lightweight
methods instead of loading full entities with all relationships.
Also added logfire instrumentation to markdown utils.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
1. Hashtag detection now checks for standalone words starting with #
instead of just checking if # appears anywhere in content.
This prevents HTML color codes like #4285F4 from being
interpreted as hashtags.
2. Observation permalinks now truncate content to 200 chars
to stay under PostgreSQL's btree index limit of 2704 bytes.
Added tests for both fixes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Paul Hernandez