Infrastructure:
- Enable pg_stat_statements extension for query monitoring
- Increase Postgres max connections and shared buffers
- Bump RabbitMQ prefetchCount to 100
- Add curl/netstat/htop to dev Docker image
Database:
- Add indexes for findings and workflows
- Add workflow status constraint and is_purged column
- Index workflows by object_id for active workflows
Workflow tracking:
- Add WorkflowStatus enum
- Added finalize_workflow activity
- Generate instance IDs with name prefix pattern
(<name>.<instance>.<object_id>)
Workflow purging:
- Create WorkflowPurger to purge Dapr workflow data (kills perf over
time)
- added to file_enrichment/document_conversion
- New workflows column to track purge workflows
File linking:
- Optimize upserts to single query (perf)
Configuration:
- PII info in env.example
Performance and configuration improvements:
- Tune OpenTelemetry Collector with batching, queuing, and memory limits
- Optimize Jaeger for write-heavy workloads with consistency disabled
- Add Prometheus scraping for OTel Collector internal metrics
- Increase document conversion prefetch count from 1 to 5
- Add CPU limit (2 cores) for document-conversion service
- Disable Tesseract OCR in default Tika config for faster processing
Workflow tracking enhancements:
- Add WorkflowTrackingService for centralized workflow state management
- Implement finalize_workflow activities for both services
- Track enrichment success/failure per workflow instance
- Support updating workflow status by object_id for subscriptions
Code improvements:
- Refactor PDF analyzer with modular extraction functions
- Add Unicode surrogate sanitization to prevent JSONB errors
- Update PII analyzer with improved error handling
- Add documentation links to Tika and Tesseract configuration
Frontend:
- Update StatsOverview component for workflow metrics display
- PII analyzer disabled by default, can be enabled via ENV variable
- Created `enrichment_configuration.md` documentation documenting all enrichment module configurations
- Restructure document_conversion to match into workflow structure
- PDF analyzer: use PyMuPDF and extract extensive metadata
- Add PDF test suite with encrypted/unencrypted fixtures
- Add Tika Tesseract OCR configuration support example
- Update Dapr configurations for new document_conversion component
- Remove deprecated helpers from common lib
- Update queue constants and workflow setup
Lee Chagolla-Christensen