mirror of
https://github.com/basicmachines-co/basic-memory
synced 2026-06-21 13:47:35 +00:00
bb02091aa3
Implements streaming checksum computation for files larger than 1MB to significantly reduce memory usage during sync operations. This addresses high memory consumption (1.5-1.7GB) when syncing projects with large binary files (PDFs, images, videos). Changes: - Add compute_checksum_streaming() function that reads files in 64KB chunks - Update sync_service._compute_checksum_async() to use streaming for files >1MB - Add streaming_checksum_threshold_mb config option (default: 1MB) - Add comprehensive tests for streaming checksum functionality Impact: - Reduces memory usage for large file syncs from ~80MB to <1MB per file - Maintains backward compatibility with existing checksum behavior - Configurable threshold allows tuning for different deployment scenarios Fixes #382 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com>