Infrastructure:
- Enable pg_stat_statements extension for query monitoring
- Increase Postgres max connections and shared buffers
- Bump RabbitMQ prefetchCount to 100
- Add curl/netstat/htop to dev Docker image
Database:
- Add indexes for findings and workflows
- Add workflow status constraint and is_purged column
- Index workflows by object_id for active workflows
Workflow tracking:
- Add WorkflowStatus enum
- Added finalize_workflow activity
- Generate instance IDs with name prefix pattern
(<name>.<instance>.<object_id>)
Workflow purging:
- Create WorkflowPurger to purge Dapr workflow data (kills perf over
time)
- added to file_enrichment/document_conversion
- New workflows column to track purge workflows
File linking:
- Optimize upserts to single query (perf)
Configuration:
- PII info in env.example
Performance and configuration improvements:
- Tune OpenTelemetry Collector with batching, queuing, and memory limits
- Optimize Jaeger for write-heavy workloads with consistency disabled
- Add Prometheus scraping for OTel Collector internal metrics
- Increase document conversion prefetch count from 1 to 5
- Add CPU limit (2 cores) for document-conversion service
- Disable Tesseract OCR in default Tika config for faster processing
Workflow tracking enhancements:
- Add WorkflowTrackingService for centralized workflow state management
- Implement finalize_workflow activities for both services
- Track enrichment success/failure per workflow instance
- Support updating workflow status by object_id for subscriptions
Code improvements:
- Refactor PDF analyzer with modular extraction functions
- Add Unicode surrogate sanitization to prevent JSONB errors
- Update PII analyzer with improved error handling
- Add documentation links to Tika and Tesseract configuration
Frontend:
- Update StatsOverview component for workflow metrics display
- PII analyzer disabled by default, can be enabled via ENV variable
- Created `enrichment_configuration.md` documentation documenting all enrichment module configurations
- Restructure document_conversion to match into workflow structure
- PDF analyzer: use PyMuPDF and extract extensive metadata
- Add PDF test suite with encrypted/unencrypted fixtures
- Add Tika Tesseract OCR configuration support example
- Update Dapr configurations for new document_conversion component
- Remove deprecated helpers from common lib
- Update queue constants and workflow setup
* upgrade to dapr postgresv2 statestore
* actually make it v2
* dapr state table name, cleanup subscriptions/globals
* proper exceptions
* formatting/lint
* Refactor workflow tracking and improve activity input handling
- Extract workflow tracking logic into dedicated WorkflowTrackingService
- Simplify activity signatures to accept specific parameters instead of generic dicts
- Remove unused asyncio event loop references from enrichment modules
- Update YaraRuleManager initialization and method names
* re-added workflow tracking in the DB
* update uvicorn prod options
* enrichment work parallelism, convert queues from broadcast to task queues
* Refactor pubsub and improve workflow parallelism
- Split Dapr pubsub Dapr yaml components into topic-specific queues (alerting, dotnet, dpapi, files, noseyparker, workflow_monitor)
- Update all Dapr volume mounts to reference new topic-specific pubsub components
- Converted queues to task queues
- Use YAML anchors to reduce duplication for file-enrichment replicas
- Pass asyncpg pool to enrichment modules instead of creating connections
- Add asyncpg_pool parameter throughout chromium and enrichment module analyzers
- Update VSCode workspace (removed InspectAssembly, renamed dotnet_api to dotnet_service)
- Added curl commands for Jaeger API to performance docs to help with perf troubleshooting
- Created common.queues module to centralize pubsub/topic names (eases
future refactoring)
* worker mods
* Workflow performance tuning, fix pubsub config, CLI arg changes
- Fix pubsub deleteWhenUnused typo (deletedWhenUnused)
- Add LOG_LEVEL environment variable support across services
- CLI: Rename --repeat to --times, add --max-files option
- Increase files pubsub prefetchCount from 25 to 50
- Add MAX_PARALLEL_WORKFLOWS configuration
- Fix DotNetAssemblyAnalysis null handling with field validators
- Update dashboard to show cumulative files/findings over time
- Add RUST_LOG environment variable support to noseyparker
- Update CHANGELOG for 2.1.4 release notes
* Dapr 1.16.2 and use db transactions
- Upgrade all Dapr containers from 1.16.1 to 1.16.2
- Reduce enrichment parallelism default from 25 to 5 workflows
- Reduce healthcheck intervals from 10s to 5s for alerting and document conversion
- Fixed DPAPI eventing to use new pubsubs
- Refactor file_linking database operations to use atomic upserts and avoid deadlocks
- Add WriteOnceViolationError handling in DPAPI masterkey analyzer
- Wrap database operations in transactions for enrichment storage and plaintext indexing
- Fix postgres notification handler closure variable capture
* remove unused start_time
* Scheduler persistence, workflow concurrency tuning, and config cleanup
- Add volume for Dapr scheduler and init service
- Add scheduler dependency to file enrichment service
- Add async workflow client libraries
- Format and cleanup compose.yaml (spacing, indentation, empty lines)
* Migrate file_enrichment to async Dapr client and optimize Dockerfile
- Use async DaprClient where possible in file_enrichment
- Improve Dockerfile caching
- Add asyncpg connection pool helper and fix typo in secret store name
- Include VS Code debug configuration for document_conversion
- Remove unused dapr_client from DpapiBlobAnalyzer
- Clean up activity return types and better handle exceptions
* Enrichment tracking for NoseyParker and logging cleanup
- Add workflow_id to NoseyParkerInput and NoseyParkerOutput models
- Remove workflow lookup query in noseyparker subscription handler
- Adjust jaeger_perf_stats.sh output formatting and precision
- Add type hints for async functions
* noseyparker scanner perf, tracing for update_enrichment_results
---------
Co-authored-by: Lee Chagolla-Christensen <lee@localhost>
- Findings/alert filtering options when LLM functionality enabled
- Fixed race-condition for noseyparker/dotnet analysis findings + alerts when these containers insert findings _after_ the main file_enrichment workflow completes
- Changed layout of alerting settings display
- Move JWT from agents noseyparker subscription in file_enrichment
- Add new fields to published alert in publish_findings.py
- Store alerting settings in database, accessible from Settings.jsx page
- Have alerting apply alerting settings before publishing
- Move Arize Phoenix and Agents into `llm` docker compose profile
* make all enrichment (should)process methods async
* fix office2john resource leak
* check for none
* more resource errors, add ole hash tests, pytest configs
* update chromium to use async pool
* update registry module to use asyncpg pool
* more asyncpg
* use async get_file_enriched
* update sleep
* fix dapr configs, new ones for workflow services
* remove deprecate/redundant httprequestmax setting
* disable workflow tracking (for now)
---------
Co-authored-by: Lee Chagolla-Christensen <lee@localhost>
Lee Chagolla-Christensen