admin/basicmachines-co-basic-memory
1
0
Fork 0
mirror of https://github.com/basicmachines-co/basic-memory synced 2026-06-21 13:47:35 +00:00
Code Issues Packages Projects Releases Wiki Activity
959 Commits 129 Branches 104 Tags
optimization/postgres-scan-batching
Commit Graph

959 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
phernandez 615d8ba685 perf: Add batch processing for Postgres sync optimization 
Implements streaming batch processing to reduce database roundtrips from 50K-80K to ~4K-6K for large projects (10K files).

**Phase 1: Scan Optimization**
- Add entity_repository.get_by_file_paths_batch() for bulk entity fetching
- Reduces scan phase from N queries to 1 batched query
- Impact: 427 files scanned with 2 queries vs 427 before

**Phase 2: Batch Infrastructure**
- Add sync_batch_size config (default: 100 files per batch)
- Add chunks() utility for streaming batch processing
- Add entity_repository.upsert_entities() for bulk inserts/updates
- Add observation_repository.delete_by_entity_ids() for batch deletes
- Add relation_repository.delete_outgoing_relations_from_entities() for batch deletes

**Phase 3: Sync Phase Optimization**
- Add sync_markdown_batch() method with 3-phase processing:
  1. Parse all files in batch (no DB operations)
  2. Bulk upsert entities in single transaction
  3. Post-process relations, checksums, search indexing per file
- Update new/modified file loops to use batch processing
- Add exception handling for circuit breaker and fatal errors
- Separate markdown/regular file processing in batches

**Test Updates**
- Update circuit breaker tests to work with batch architecture
- Change mocks from sync_markdown_file to sync_markdown_batch
- Update fatal error test to mock upsert_entities
- All circuit breaker tests passing (8/8)

**Expected Performance**
- Initial bulk import: ~10-15 queries/file (vs 43 before)
- Incremental sync: Massive scan improvement + batch upsert benefits
- Handles both new files and existing files efficiently

Addresses N+1 query patterns and transaction overhead with remote Postgres databases while maintaining circuit breaker functionality and proper error handling.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-11-20 21:16:04 -06:00
Paul Hernandez fb5e9e1d77 feat: Add PostgreSQL database backend support (#439) 
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-11-20 11:20:29 -06:00
phernandez 66b91b2847 ci: Add PostgreSQL testing to GitHub Actions workflow 
Add Postgres service container and separate test step for PostgreSQL backend testing.
The Postgres tests only run on Linux runners since GitHub Actions service containers
are only available on Linux.

- Add postgres:17 service container with health checks
- Add 'Run tests (Postgres)' step with Linux-only condition
- Rename existing test step to 'Run tests (SQLite)' for clarity

This enables CI testing of dual database backend support introduced in the
postgres-support feature branch.

Signed-off-by: phernandez <paul@basicmachines.co>
 2025-11-18 12:25:18 -06:00
Cedric Hurst b004565df9 fix: handle periods in kebab_filenames mode (#424) 2025-11-18 06:49:55 -05:00
Drew Cain a258b73e1d chore: update version to 0.16.2 for v0.16.2 release v0.16.2 2025-11-16 21:30:59 -06:00
Drew Cain 9a845f2906 docs: prepare for v0.16.2 release 2025-11-16 21:28:13 -06:00
Drew Cain 6517e9845f fix: Use platform-native path separators in config.json (#429) 
Signed-off-by: Drew Cain <groksrc@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-11-13 09:12:26 -06:00
Drew Cain 1af05392ee fix: Add rclone installation checks for Windows bisync commands (#427) 2025-11-12 14:22:14 -06:00
Brandon Mayes cad7019c89 fix: main project always recreated on project list command (#421) 2025-11-12 09:57:08 -05:00
phernandez 099c334e3d chore: update version to 0.16.1 for v0.16.1 release v0.16.1 2025-11-11 09:21:47 -06:00
phernandez 7685586178 docs: Add v0.16.1 CHANGELOG entry for Windows line ending fix 
Signed-off-by: phernandez <paul@basicmachines.co>
 2025-11-11 09:10:26 -06:00
Paul Hernandez e9d0a944a9 fix: Handle Windows line endings in rclone bisync (#422) 
Signed-off-by: phernandez <paul@basicmachines.co>
 2025-11-11 09:08:20 -06:00
phernandez caf3c14bb1 chore: update version to 0.16.0 for v0.16.0 release v0.16.0 2025-11-10 19:19:48 -06:00
phernandez c5d9067754 docs: Add v0.16.0 CHANGELOG entry with comprehensive release notes 
Signed-off-by: phernandez <paul@basicmachines.co>
 2025-11-10 19:17:29 -06:00
phernandez e0fc59ea97 style: Format upload.py for better readability 
Signed-off-by: phernandez <paul@basicmachines.co>
 2025-11-10 19:15:03 -06:00
Paul Hernandez 49b2adc35c fix: skip archive files during cloud upload (#420) 
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
 2025-11-10 19:04:49 -06:00
Paul Hernandez 1646572f69 fix: Rename write_note entity_type to note_type for clarity (#419) 
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
 2025-11-10 19:02:50 -06:00
Paul Hernandez f0d7398815 fix: Quote string values in YAML frontmatter to handle special characters (#418) 
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
 2025-11-10 18:13:48 -06:00
Paul Hernandez 581b7b17c6 fix: Add explicit type annotations to MCP tool parameters (#394) 
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
 2025-11-10 17:02:17 -06:00
Paul Hernandez d775f7bab9 fix: Simplify search_notes schema by removing Optional wrappers (#395) 
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
 2025-11-10 16:30:59 -06:00
Drew Cain fc01f6abaf fix: Replace Unicode arrows with ASCII for Windows compatibility (#414) 2025-11-10 16:30:41 -06:00
Paul Hernandez 4614fd09d5 fix: Handle dict objects in write_resource endpoint (#415) 
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
 2025-11-10 16:30:27 -06:00
phernandez 0d4ad7bbf0 remove v0.15.0 info from assistant-guide 
Signed-off-by: phernandez <paul@basicmachines.co>
 2025-11-10 16:25:49 -06:00
jope-bm 7ccec7eba2 feat: Add run_in_background parameter to sync endpoint with tests (#417) 
Co-authored-by: Claude <noreply@anthropic.com>
 2025-11-07 09:04:43 -07:00
Paul Hernandez 021af74545 fix: Strip duplicate headers in edit_note replace_section (#396) 
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Drew Cain <groksrc@users.noreply.github.com>
 2025-11-02 14:20:11 -06:00
phernandez 2ad0ee9d5d fix: Use force_full=true for database sync after project sync/bisync 
After rclone synchronizes files between local and cloud storage, the
database needs to perform a full scan to ensure it captures all changes.
Previously, incremental sync (watermark optimization) could miss files
that were changed remotely.

Changes:
- project sync command now calls /project/sync?force_full=true
- project bisync command now calls /project/sync?force_full=true
- Ensures complete database refresh after file synchronization

This guarantees the database is fully in sync with the filesystem
after any rclone sync or bisync operation.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
 2025-11-02 13:26:08 -06:00
Brandon Mayes c9946ecf1e fix: Various rclone fixes for cloud sync on Windows (#410) 
Signed-off-by: Brandon Mayes <5610870+bdmayes@users.noreply.github.com>
Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Co-authored-by: phernandez <paul@basicmachines.co>
 2025-11-02 11:26:57 -06:00
Drew Cain 0ba6f219f1 fix: Windows CLI Unicode encoding errors (#411) 
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
Co-authored-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
 2025-11-02 10:20:48 -06:00
Paul Hernandez 0b3272ae6e feat: SPEC-20 Simplified Project-Scoped Rclone Sync (#405) 
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-11-02 09:35:26 -06:00
Paul Hernandez a7d7cc5ee6 fix: Normalize YAML frontmatter types to prevent AttributeError (#236) (#402) 
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-10-27 09:19:50 -05:00
Paul Hernandez a7e696b039 Add free trial information to README 
Added information about a 7-day free trial.

Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
 2025-10-24 10:05:11 -05:00
Paul Hernandez 8aaddb6d45 Add free trial information to README 
Added information about a 7-day free trial to the README.

Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
 2025-10-24 10:04:41 -05:00
Paul Hernandez d7565312fc Announce Basic Memory Cloud launch in README 
Added a section announcing the launch of Basic Memory Cloud with details on cross-device support and early supporter pricing.

Signed-off-by: Paul Hernandez <60959+phernandez@users.noreply.github.com>
 2025-10-24 09:53:14 -05:00
Paul Hernandez c7e6eab02f feat: Add delete_notes parameter to remove project endpoint (#391) 
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-10-21 14:09:20 -05:00
Paul Hernandez bb8da31472 fix: Handle null, empty, and string 'None' title in markdown frontmatter (#387) (#389) 
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
 2025-10-21 09:29:19 -05:00
Paul Hernandez e78345ff25 feat: Streaming Foundation & Async I/O Consolidation (SPEC-19) (#384) 
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-10-21 09:03:59 -05:00
Paul Hernandez 32236cd247 fix: Handle YAML parsing errors gracefully in update_frontmatter (#378) (#379) 
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-10-16 20:17:58 -05:00
Paul Hernandez 4fd6d0c648 fix: Optimize sync memory usage to prevent OOM on large projects (#380) 
Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
 2025-10-16 20:17:34 -05:00
Paul Hernandez e6c8e3662c fix: preserve mtime webdav upload 376 (#377) 
Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-10-16 17:18:10 -05:00
Paul Hernandez 449b62d947 fix: Prevent deleted projects from being recreated by background sync (#193) (#370) 
Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
 2025-10-16 15:16:24 -05:00
Paul Hernandez b7497d7484 fix: Use filesystem timestamps for entity sync instead of database operation time (#138) (#369) 
Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: phernandez <paul@basicmachines.co>
 2025-10-16 14:21:27 -05:00
Paul Hernandez d1431bdb1b fix: Handle YAML parsing errors and missing entity_type in markdown files (#368) 
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-10-16 13:18:58 -05:00
Paul Hernandez 171bef717f fix: Resolve UNIQUE constraint violation in entity upsert with observations (#187) (#367) 
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-10-16 12:30:56 -05:00
Paul Hernandez 729a5a3b8d fix: Terminate sync immediately when project is deleted (#366) 
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-10-16 11:07:12 -05:00
Paul Hernandez 434cdf24dd feat: Add circuit breaker for file sync failures (#364) 
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-10-16 09:47:48 -05:00
Paul Hernandez 7f9c1a97a4 feat: Add --verbose and --no-gitignore options to cloud upload (#362) 
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-10-15 20:03:36 -05:00
Paul Hernandez 53fb13b054 fix: Make project creation endpoint idempotent (#357) 
Signed-off-by: phernandez <paul@basicmachines.co>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-10-15 19:27:41 -05:00
Paul Hernandez bd6c8348b8 fix: Handle None text values in Claude conversations importer (#353) 
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>
 2025-10-15 16:08:57 -05:00
phernandez 994c8b8e7e chore: update version to 0.15.2 for v0.15.2 release v0.15.2 2025-10-14 09:36:47 -05:00
phernandez a78e8c3ac5 style: Apply linter formatting changes 2025-10-14 09:34:10 -05:00