basicmachines-co-basic-memory/test-int/BENCHMARKS.md

Performance Benchmarks

This directory contains performance benchmark tests for Basic Memory's sync/indexing operations.

Purpose

These benchmarks measure baseline performance to track improvements from optimizations. They are particularly important for:

• Cloud deployments with ephemeral databases that need fast re-indexing
• Large repositories (100s to 1000s of files)
• Validating optimization efforts

Running Benchmarks

Run all benchmarks (excluding slow ones)

pytest test-int/test_sync_performance_benchmark.py -v -m "benchmark and not slow"

Run specific benchmark

# 100 files (fast, ~10-30 seconds)
pytest test-int/test_sync_performance_benchmark.py::test_benchmark_sync_100_files -v

# 500 files (medium, ~1-3 minutes)
pytest test-int/test_sync_performance_benchmark.py::test_benchmark_sync_500_files -v

# 1000 files (slow, ~3-10 minutes)
pytest test-int/test_sync_performance_benchmark.py::test_benchmark_sync_1000_files -v

# Re-sync with no changes (tests scan performance)
pytest test-int/test_sync_performance_benchmark.py::test_benchmark_resync_no_changes -v

Run all benchmarks including slow ones

pytest test-int/test_sync_performance_benchmark.py -v -m benchmark

Skip benchmarks in regular test runs

pytest -m "not benchmark"

Benchmark Output

Each benchmark provides detailed metrics including:

• Performance Metrics:

  • Total sync time
  • Files processed per second
  • Milliseconds per file

• Database Metrics:

  • Initial database size
  • Final database size
  • Database growth (total and per file)

• Operation Counts:

  • New files indexed
  • Modified files processed
  • Deleted files handled
  • Moved files tracked

Example Output

======================================================================
BENCHMARK: Sync 100 files (small repository)
======================================================================

Generating 100 test files...
  Created files 0-100 (100/100)
  File generation completed in 0.15s (666.7 files/sec)

Initial database size: 120.00 KB

Starting sync of 100 files...

----------------------------------------------------------------------
RESULTS:
----------------------------------------------------------------------
Files processed:      100
  New:                100
  Modified:           0
  Deleted:            0
  Moved:              0

Performance:
  Total time:         12.34s
  Files/sec:          8.1
  ms/file:            123.4

Database:
  Initial size:       120.00 KB
  Final size:         5.23 MB
  Growth:             5.11 MB
  Growth per file:    52.31 KB
======================================================================

Interpreting Results

Good Performance Indicators

• Files/sec > 10: Good indexing speed for small-medium repos
• Files/sec > 5: Acceptable for large repos with complex relations
• DB growth < 100KB per file: Reasonable index size

Areas for Improvement

• Files/sec < 5: May benefit from batch operations
• ms/file > 200: High latency per file, check for N+1 queries
• DB growth > 200KB per file: Search index may be bloated (trigrams?)

Tracking Improvements

Before making optimizations:

1. Run benchmarks to establish baseline
2. Save output for comparison
3. Note any particular pain points (e.g., slow search indexing)

After optimizations:

1. Run the same benchmarks
2. Compare metrics:
   • Files/sec should increase
   • ms/file should decrease
   • DB growth per file may decrease (with search optimizations)
3. Document improvements in PR

Related Issues

• #351: Performance: Optimize sync/indexing for cloud deployments

Test File Generation

Benchmarks generate realistic markdown files with:

• YAML frontmatter with tags
• 3-10 observations per file with categories
• 1-3 relations per file (including forward references)
• Varying content to simulate real usage
• Files organized in category subdirectories