bb02091aa3
feat: Optimize memory usage for large file syncs with streaming checksums
...
Implements streaming checksum computation for files larger than 1MB to
significantly reduce memory usage during sync operations. This addresses
high memory consumption (1.5-1.7GB) when syncing projects with large
binary files (PDFs, images, videos).
Changes:
- Add compute_checksum_streaming() function that reads files in 64KB chunks
- Update sync_service._compute_checksum_async() to use streaming for files >1MB
- Add streaming_checksum_threshold_mb config option (default: 1MB)
- Add comprehensive tests for streaming checksum functionality
Impact:
- Reduces memory usage for large file syncs from ~80MB to <1MB per file
- Maintains backward compatibility with existing checksum behavior
- Configurable threshold allows tuning for different deployment scenarios
Fixes #382
🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com >
2025-10-17 02:00:14 +00:00
32236cd247
fix: Handle YAML parsing errors gracefully in update_frontmatter ( #378 ) ( #379 )
...
Signed-off-by: phernandez <paul@basicmachines.co >
Co-authored-by: Claude <noreply@anthropic.com >
2025-10-16 20:17:58 -05:00
17b929446a
fix: Sanitize folder names and properly join paths ( #292 )
2025-09-15 23:26:56 -04:00
ec2fa07350
chore: apply lint and formatting fixes for 0.14.4 release ( #290 )
...
Signed-off-by: Joe P <joe@basicmemory.com >
Co-authored-by: Claude <noreply@anthropic.com >
2025-09-05 10:00:52 -06:00
7e024a8674
fix: resolve linting errors for release preparation
...
- Replace bare except clauses with Exception in legal_file_inventory.py
- Remove unused variables in test files
- Prepare codebase for v0.14.4 release
2025-09-04 14:01:06 -06:00
22f7bfa398
fix: Update YAML frontmatter tag formatting for Obsidian compatibility ( #280 )
...
Signed-off-by: Drew Cain <groksrc@users.noreply.github.com >
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: Paul Hernandez <phernandez@users.noreply.github.com >
Co-authored-by: Paul Hernandez <60959+phernandez@users.noreply.github.com >
2025-09-04 09:57:45 -05:00
cd7cee650f
fix: complete project management special character support ( #272 ) ( #279 )
...
Signed-off-by: Joe P <joe@basicmemory.com >
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: jope-bm <jope-bm@users.noreply.github.com >
Co-authored-by: Claude <noreply@anthropic.com >
2025-08-29 10:57:13 -06:00
74e12eb782
fix: Sanitize filenames and allow optional kebab case ( #260 )
...
Signed-off-by: Brandon Mayes <5610870+bdmayes@users.noreply.github.com >
2025-08-27 19:29:01 -04:00
a4a3b1b689
fix: handle missing 'name' key in memory JSON import ( #241 )
...
Signed-off-by: Joe P <joe@basicmemory.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: jope-bm <jope-bm@users.noreply.github.com >
2025-07-28 14:52:15 -06:00
24a1d6195d
fix: path traversal security vulnerability in mcp tools ( #223 )
...
Signed-off-by: Joe P <joe@basicmemory.com >
2025-07-15 09:05:11 -06:00
634486107b
feat: v0.13.0 pre ( #122 )
...
Signed-off-by: phernandez <paul@basicmachines.co >
Co-authored-by: Claude <noreply@anthropic.com >
2025-06-03 01:00:40 -05:00
73ea91fe0d
fix: add extra logic for permalink generation with mixed Latin unicode and Chinese characters
...
Signed-off-by: phernandez <paul@basicmachines.co >
2025-04-17 10:00:40 -05:00
2934176331
fix: utf8 for all file reads/write/open instead of default platform encoding ( #91 )
...
Signed-off-by: phernandez <paul@basicmachines.co >
2025-04-08 10:55:01 -05:00
78a3412bcf
fix: run migrations and sync when starting mcp ( #88 )
2025-04-07 17:45:26 -05:00
617e60bda4
feat: permalink enhancements ( #82 )
...
- Avoiding "useless permalink values" for files without metadata
- Enable permalinks to be updated on move via config setting
2025-04-06 14:54:59 -05:00
6c19c9edf5
fix: [BUG] # character accumulation in markdown frontmatter tags prop ( #79 )
2025-04-05 23:07:53 -05:00
3bffb2e190
Add updated getting started docs and utf8 test
...
Signed-off-by: phernandez <paul@basicmachines.co >
2025-03-15 16:52:17 -05:00
e6496df595
feat: Beta work ( #17 )
...
feat: Add multiple projects support
feat: enhanced read_note for when initial result is not found
fix: merge frontmatter when updating note
fix: handle directory removed on sync watch
2025-03-05 18:46:04 -06:00
28513c8b8d
lint code
2025-02-12 22:39:22 -06:00
5b4f0eafcc
Pre release fixups ( #5 )
...
* configure logging
* set mcp output logging also
* fix type check errors
* fix type check
* rename Permalink schema type
* fix type errors
* add typechecks to ci workflow
* pytest coverage setup
* add tests for status cli
* sync tests coverage
* watch_service test coverage
* tests for tool_utils.py
* clean up imports
* file_utils coverage
* markdown plugins coverage
* 99% test coverage
* more test coverage, remove ObservationCategory
* more tool coverage
* fix type-check
* format, upgrade deps
---------
Co-authored-by: phernandez <phernandez@basicmachines.co >
2025-02-12 08:23:10 -06:00
5e501b5440
fix sync
2025-02-05 23:05:15 -06:00
32606d771d
fix tests for context and permalinks
2025-01-18 23:54:28 -06:00
db9da9051a
add ai_edit tool
2025-01-18 21:17:34 -06:00
eda2a30099
fix yml parsing/writing in document service
2025-01-03 23:59:09 -06:00
f62f5677b0
add file_utils.py
2024-12-22 14:07:45 -06:00
claude[bot]