admin/basicmachines-co-basic-memory
1
0
Fork 0
mirror of https://github.com/basicmachines-co/basic-memory synced 2026-06-21 13:47:35 +00:00
Code Issues Packages Projects Releases Wiki Activity
959 Commits 129 Branches 104 Tags
optimization/postgres-scan-batching
Commit Graph

81 Commits

Author SHA1 Message Date
phernandez 615d8ba685 perf: Add batch processing for Postgres sync optimization 
Implements streaming batch processing to reduce database roundtrips from 50K-80K to ~4K-6K for large projects (10K files).

**Phase 1: Scan Optimization**
- Add entity_repository.get_by_file_paths_batch() for bulk entity fetching
- Reduces scan phase from N queries to 1 batched query
- Impact: 427 files scanned with 2 queries vs 427 before

**Phase 2: Batch Infrastructure**
- Add sync_batch_size config (default: 100 files per batch)
- Add chunks() utility for streaming batch processing
- Add entity_repository.upsert_entities() for bulk inserts/updates
- Add observation_repository.delete_by_entity_ids() for batch deletes
- Add relation_repository.delete_outgoing_relations_from_entities() for batch deletes

**Phase 3: Sync Phase Optimization**
- Add sync_markdown_batch() method with 3-phase processing:
  1. Parse all files in batch (no DB operations)
  2. Bulk upsert entities in single transaction
  3. Post-process relations, checksums, search indexing per file
- Update new/modified file loops to use batch processing
- Add exception handling for circuit breaker and fatal errors
- Separate markdown/regular file processing in batches

**Test Updates**
- Update circuit breaker tests to work with batch architecture
- Change mocks from sync_markdown_file to sync_markdown_batch
- Update fatal error test to mock upsert_entities
- All circuit breaker tests passing (8/8)

**Expected Performance**
- Initial bulk import: ~10-15 queries/file (vs 43 before)
- Incremental sync: Massive scan improvement + batch upsert benefits
- Handles both new files and existing files efficiently

Addresses N+1 query patterns and transaction overhead with remote Postgres databases while maintaining circuit breaker functionality and proper error handling.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-11-20 21:16:04 -06:00
Paul Hernandez fb5e9e1d77 feat: Add PostgreSQL database backend support (#439) 
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-11-20 11:20:29 -06:00
Paul Hernandez f0d7398815 fix: Quote string values in YAML frontmatter to handle special characters (#418) 
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
 2025-11-10 18:13:48 -06:00
Drew Cain 0ba6f219f1 fix: Windows CLI Unicode encoding errors (#411) 
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Co-authored-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
 2025-11-02 10:20:48 -06:00
Paul Hernandez 0b3272ae6e feat: SPEC-20 Simplified Project-Scoped Rclone Sync (#405) 
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-11-02 09:35:26 -06:00
Paul Hernandez e78345ff25 feat: Streaming Foundation & Async I/O Consolidation (SPEC-19) (#384) 
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-10-21 09:03:59 -05:00
Paul Hernandez 449b62d947 fix: Prevent deleted projects from being recreated by background sync (#193) (#370) 
Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
 2025-10-16 15:16:24 -05:00
Paul Hernandez b7497d7484 fix: Use filesystem timestamps for entity sync instead of database operation time (#138) (#369) 
Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
 2025-10-16 14:21:27 -05:00
Paul Hernandez 171bef717f fix: Resolve UNIQUE constraint violation in entity upsert with observations (#187) (#367) 
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-10-16 12:30:56 -05:00
Paul Hernandez 729a5a3b8d fix: Terminate sync immediately when project is deleted (#366) 
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-10-16 11:07:12 -05:00
Paul Hernandez 434cdf24dd feat: Add circuit breaker for file sync failures (#364) 
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-10-16 09:47:48 -05:00
Paul Hernandez 5da97e4820 feat: implement SPEC-11 API performance optimizations (#315) 
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-09-26 14:34:46 -05:00
Paul Hernandez 17a6733c9d fix: remove obsolete update_current_project function and --project flag reference (#310) 
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
 2025-09-26 11:46:33 -05:00
Paul Hernandez e0d8aeb149 feat: Basic memory cloud upload (#296) 
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: Drew Cain <groksrc@gmail.com>
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Drew Cain <groksrc@gmail.com>
 2025-09-16 15:07:14 -05:00
jope-bm ec2fa07350 chore: apply lint and formatting fixes for 0.14.4 release (#290) 
Signed-off-by: Joe P <joe@basicmemory.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-09-05 10:00:52 -06:00
Drew Cain 7a8b08d11e fix: Windows test failures and add Windows CI support (#273) 
Signed-off-by: Drew Cain <groksrc@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-08-25 08:58:24 -05:00
manuelbliemel 9aa40246a8 Addressed issues when running basic-memory on the Windows platform (#252) 
Signed-off-by: Manuel Bliemel <manuel.bliemel@gmail.com>
 2025-08-24 19:12:40 -07:00
jope-bm 2cd2a62f30 fix: Ensure all datetime operations return timezone-aware objects (#268) 
Signed-off-by: Joe P <joe@basicmemory.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-08-22 13:43:55 -06:00
Paul Hernandez 2438094914 fix: handle vim atomic write DELETE events without ADD (#249) 
Signed-off-by: Joe P <joe@basicmemory.com>
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
Co-authored-by: Joe P <joe@basicmemory.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-08-20 14:36:43 -05:00
Drew Cain 7460a938df fix: make case sensitivity test platform-aware 
- Add platform detection to handle case-insensitive file systems
- Test now passes on macOS and Windows while maintaining Linux behavior
- Fixes test failure on case-insensitive file systems
 2025-08-01 22:02:53 -05:00
Drew Cain 43fa5762a8 ruff checks 
Signed-off-by: Drew Cain <groksrc@gmail.com>
 2025-08-01 21:50:18 -05:00
Paul Hernandez fb1350b294 fix: enhance character conflict detection and error handling for sync operations (#201) 
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
 2025-08-01 21:35:56 -05:00
phernandez 2abf626c46 fix: resolve unused variable lint warnings in tests 
- Remove unused variables in test mock functions
- Clean up test code per ruff linting rules
 2025-06-09 15:15:05 -05:00
phernandez 7108a7baf1 fix: resolve sync race conditions and search errors 
- Add IntegrityError handling in entity_service.create_entity_from_markdown for file_path/permalink constraint violations
- Add IntegrityError handling in sync_service.sync_regular_file for concurrent sync race conditions
- Fix FTS "unknown special query" error when searching for wildcard "*" patterns
- Add comprehensive test coverage for race condition edge cases and error handling
- Gracefully handle concurrent sync processes with fallback to update operations

Fixes sync errors from beta testing including:
- "UNIQUE constraint failed: entity.file_path"
- "UNIQUE constraint failed: entity.permalink"
- "unknown special query" FTS errors

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-06-08 15:20:27 -05:00
phernandez c5c70cb0f4 improve validation for memory:// urls, add examples to build_context 
Signed-off-by: phernandez <paul@basicmachines.co>
 2025-06-04 00:16:33 -05:00
phernandez ac08a8d024 fix: update FastMCP initialization for API changes 
- Remove deprecated auth_server_provider parameter
- Use auth parameter correctly with OAuthProvider instead of AuthSettings
- Fixes type error after dependency updates

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-06-03 09:08:17 -05:00
Paul Hernandez 634486107b feat: v0.13.0 pre (#122) 
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-06-03 01:00:40 -05:00
bm-claudeai 020957cd76 feat: Multi-project support, OAuth authentication, and major improvements (#119) 
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-05-25 10:07:34 -05:00
Paul Hernandez 2934176331 fix: utf8 for all file reads/write/open instead of default platform encoding (#91) 
Signed-off-by: phernandez <paul@basicmachines.co>
 2025-04-08 10:55:01 -05:00
Paul Hernandez 78a3412bcf fix: run migrations and sync when starting mcp (#88) 2025-04-07 17:45:26 -05:00
Paul Hernandez 617e60bda4 feat: permalink enhancements (#82) 
- Avoiding "useless permalink values" for files without metadata
- Enable permalinks to be updated on move via config setting
 2025-04-06 14:54:59 -05:00
github-actions[bot] 7930ddb291 fix: [BUG] Some notes never exit "modified" status (#77) 2025-04-05 22:03:38 -05:00
phernandez b667bca5a2 fix test coverage and type checks 
Signed-off-by: phernandez <paul@basicmachines.co>
 2025-03-24 22:46:47 -05:00
phernandez 9c791259a0 \fix: improve tags handling in write_note tool to fix #38\n\nThis change makes the write_note tool more flexible with tag inputs by:\n1. Adding a parse_tags helper function to handle various input formats\n2. Removing explicit type annotation from the write_note signature\n3. Adding documentation on how to pass tags from external MCP clients\n4. Adding a test that reproduces the issue from bug report #38\n\nThe issue occurred in external MCP clients like Cursor where the tags\nparameter was causing type mismatch errors.\n\n\ud83e\udd16 Generated with [Claude Code](https://claude.ai/code)\n\nCo-Authored-By: Claude <noreply@anthropic.com>\ 
Signed-off-by: phernandez <paul@basicmachines.co>
 2025-03-24 22:46:47 -05:00
Paul Hernandez 6b110b28dd fix: don't sync *.tmp files on watch (#31) 
Fixes #30

Signed-off-by: phernandez <paul@basicmachines.co>
 2025-03-15 15:43:28 -05:00
Paul Hernandez e6496df595 feat: Beta work (#17) 
feat: Add multiple projects support 
feat: enhanced read_note for when initial result is not found
fix: merge frontmatter when updating note
fix: handle directory removed on sync watch
 2025-03-05 18:46:04 -06:00
Paul Hernandez 37a01b806d feat: Incremental sync on watch (#13) 
- incremental sync on watch
- sync non-markdown files in knowledge base
- experimental `read_resource` tool for reading non-markdown files in raw form (pdf, image)
 2025-02-24 22:13:15 -06:00
phernandez 355519a786 refactor cli commands 2025-02-13 15:42:50 -06:00
phernandez 28513c8b8d lint code 2025-02-12 22:39:22 -06:00
Paul Hernandez 5b4f0eafcc Pre release fixups (#5) 
* configure logging

* set mcp output logging also

* fix type check errors

* fix type check

* rename Permalink schema type

* fix type errors

* add typechecks to ci workflow

* pytest coverage setup

* add tests for status cli

* sync tests coverage

* watch_service test coverage

* tests for tool_utils.py

* clean up imports

* file_utils coverage

* markdown plugins coverage

* 99% test coverage

* more test coverage, remove ObservationCategory

* more tool coverage

* fix type-check

* format, upgrade deps

---------

Co-authored-by: phernandez <phernandez@basicmachines.co>
 2025-02-12 08:23:10 -06:00
phernandez aad2c750bc fix sync test 2025-02-07 16:12:25 -06:00
phernandez ffbd091ad3 add db check at startup 2025-02-07 16:03:31 -06:00
phernandez 5e501b5440 fix sync 2025-02-05 23:05:15 -06:00
phernandez 135bec181d fix: fix permalink uniqueness violations on create/update/sync 2025-02-04 16:43:18 -06:00
phernandez 3484e26631 feat: add memory-json importer, tweak observation content 2025-02-04 10:59:28 -06:00
phernandez a88fc99663 add file watcher to sync command 2025-02-02 20:07:04 -06:00
phernandez 4afa368a6b finish watch_service.py 2025-02-02 19:08:31 -06:00
phernandez 0af60e7fdc watch service 2025-02-02 16:35:46 -06:00
phernandez d2536f7345 remove dup test code 2025-01-31 14:24:13 -06:00
phernandez 9983271e94 Refactor created/modified handling in markdown entities. 
Replace frontmatter properties with top-level attributes for `created` and `modified` to simplify handling and ensure consistent attribute usage. Added a utility function for schema-to-markdown conversion and updated related tests and code to reflect these changes.
 2025-01-31 13:59:06 -06:00