* Conversion of NoseyParker scanner to Titus
- Old `noseyparker_scanner` project removed, new `titus_scaner` Golang project built
- Replaced `noseyparker` references with `titus` as appropriate
- Bumped various built-in docker image versions (Dapr, Gotenberg, etc.)
* Replace ZIP-only extraction with Titus library archive support
Replace custom ZIP extraction code with Titus library's enum.ExtractText()
to support scanning secrets inside zip, jar, war, ear, apk, ipa, xpi, crx,
tar, tar.gz/tgz, and 7z archives. Document formats (xlsx, docx, pdf, etc.)
are intentionally excluded since Nemesis handles those via document_conversion.
- Add original_path to TitusInput so Titus can dispatch by file extension
(MinIO stores files by UUID without extensions)
- Add magic-byte fallback detection for ZIP, 7z, gzip, and tar
- Preserve git-repo-in-ZIP detection as a special case before archive scan
- Rename env vars: DECOMPRESS_ZIPS -> EXTRACT_ARCHIVES,
MAX_EXTRACT_SIZE_MB -> EXTRACT_MAX_TOTAL_SIZE_MB, and add
EXTRACT_MAX_FILE_SIZE_MB and EXTRACT_MAX_DEPTH
- Add unit tests for archive detection and config loading
* Integrated (optional) Titus secret validation
- Integrated (optional) Titus secret validation that can be enabled
* Added Titus rule ID display and ability to disable specific rules
- Added Titus rule ID display and ability to disable specific built-in rules (by ID)
* `titus_scanner` fixes
- One scanner per concurrent file to prevent timeouts
- Fix custom rule regexes to prevent timeouts
- Exclude a few default run-away rules causing timeouts
- Bumped cores for titus scanner to 2
* Enable Vectorscan/Hyperscan acceleration for titus_scanner
Switch from the pure-Go regexp2 engine to the Vectorscan (Hyperscan)
SIMD-accelerated engine, using a fork with parallelized regexp2
confirmation and content-based deduplication
Key changes:
- Dockerfile: install libvectorscan-dev/libvectorscan5, build with
CGO_ENABLED=1 and -tags vectorscan
- go.mod: replace praetorian-inc/titus with HarmJ0y/titus fork that
parallelizes the regexp2 confirmation loop and uses content-based
dedup (matching portable engine behavior)
- Add bounded-concurrency semaphore in handler to prevent OOM from
unbounded goroutines while still returning 200 to Dapr immediately
- Add content-based deduplication, match count cap, snippet truncation,
and payload size logging to prevent oversized publishes
- Replace O(N*M) line/column computation with precomputed line index
using binary search
- Add GOMEMLIMIT and memory limit to container resources
* Update README.md
* remove `noseyparker_scanner` container
* Add EVTX (Windows Event Log) enrichment module
Parses .evtx files and extracts security-relevant data:
- Markdown summary transform with event counts, timeline, unique accounts/IPs
- Downloadable CSV transforms for account changes, group membership changes,
explicit credential use (4648), process creation (4688), and task changes
- Findings for high-signal events: audit log cleared, new services, admin group
changes, account changes, explicit creds, scheduled tasks, PS script blocks
- PowerShell 4104 script blocks reassembled and resubmitted to enrichment
pipeline as child .ps1 files for full Titus/YARA scanning
- Added de-duping for duplicate PS scripts carved from EVTXs (hash-based)
* EVTX mods
- Power timelining done for SYSTEM
- Inbound/outbound auth for SECURITY
- Filtering for default system accounts
- CSV viewer now can download files, transforms for EVTX transformed to the viewer
* Convert titus scanner to Dapr bulk subscribe for batch processing
- Add bulk subscribe types to models (BulkMessagePayload, BulkResponse, etc.)
- Replace async single-event handler with synchronous HandleBulkEvent
- Process batches with bounded concurrency via semaphore + WaitGroup
- Handle edge cases: duplicate entryId, empty objectID/entryId, panic recovery
- Add BulkMaxMessages and BulkMaxAwaitDurationMs config with clamping
- Set app-max-concurrency to 1 (parallelism within work batch via scanner pool)
---------
Co-authored-by: Lee Chagolla-Christensen <lee@localhost>
- Upgrade cryptography to 46.0.5 across all projects (SECT curve subgroup attack)
- Upgrade pydantic-ai from 0.7.x to 1.58.0 (SSRF in URL download handling)
- Migrate AsyncTenacityTransport to use RetryConfig instead of AsyncRetrying
- Rename OpenAIModel to OpenAIChatModel (deprecated in pydantic-ai 1.x)
- Fix model_manager type safety by guarding against None model_name
- Relax uvicorn and websockets pins for pydantic-ai 1.x compatibility
* fix: upgrade python-multipart to 0.0.22 to address CVE-2026-24486 (Dependabot #683)
* update deps+skill, pyright
* pyright + tests in CLI
* Add pyright type checking and enhance registry hive analysis
Registry Hive Analyzer Enhancements:
- Extract machine SID from SAM domain V value with binary SID decoding
- Extract per-user metadata via regipy (ACB flags, timestamps, full
name, comment via USER_ACCOUNT_V)
- Compute password expiration from domain max password age policy
- Detect empty LM/NT hashes via well-known constants
- Parse DCC cached domain credentials into structured entries
- Store DPAPI system machine_key/user_key as separate fields
- Add structured secret_type field to LSA secrets (dcc, dpapi_system,
hex_blob, generic)
- Add detailed markdown and plain-text formatters for SAM accounts
and LSA secrets, replacing _get_lsa_secret_output_string
- Fix SYSTEM hive attribute names: computer_name -> machinename,
current_control_set -> currentcontrol
- Switch SAM user iteration from sam.users to sam.secrets (pypykatz API)
- Add FILETIME-to-UTC and regipy value-to-bytes helpers
New Tests:
- Add test_registry_hive.py with SAM, SECURITY, and SYSTEM hive fixtures
- Add test_container.py for container analyzer
- Add SAM/SECURITY/SYSTEM binary test fixtures
Pyright Setup:
- Add pyrightconfig.json (basic mode, Python 3.13) to all libs and projects
- Add pyright>=1.1 as dev dependency to all pyproject.toml files
- Update all uv.lock files accordingly
- Update lint.sh to deactivate active venvs and verify pyright availability
Type Annotation Fixes:
- Fix globals initialized as None without Optional type across
file_enrichment, document_conversion, and agents global_vars
- Fix StorageMinio return types: upload/upload_file/upload_uploadfile
return str, not uuid.UUID
- Fix MockStorageMinio to match updated StorageMinio return types
- Add explicit type annotations to dict literals in chromekey.py,
pdf/analyzer.py, registry_hive/analyzer.py, and publish_findings.py
- Fix kubeconfig current_context parameter to accept str | None
- Fix container_contents allowed_extensions: set = None -> set | None = None
- Fix file subscription file_queue: asyncio.Queue = None -> asyncio.Queue | None = None
- Fix web_api upload_file to wrap object_id in uuid.UUID() for response
None-safety Assertions:
- Add assert statements for asyncpg_pool, tracking_service,
workflow_client, workflow_manager, file_linking_engine, file_queue,
gotenberg_url, and process.stdout across all activity, subscription,
route, and workflow files in file_enrichment and document_conversion
- Add assertions for asyncpg_pool in all chromium processors
- Add assertions for File.from_metadata timestamp/expiration fields
- Add assertion for alerting GQL client session type
Pyright Ignore Annotations:
- Suppress third-party type issues in Dapr workflow/activity APIs,
gRPC subscription imports, ccache/lnk/office_doc attribute access,
and nemesis_dpapi FlagMixin operators
- Add file-level suppression for office2john.py, pdf2john.py,
pe/analyzer.py, and test harness files
Bug Fixes:
- Fix office2john.py format string: bare % filename -> % (filename, stream)
- Fix container analyzer 7z iteration: iterate sz.files list instead
of calling .items()
- Fix file_linking rules_engine: store match result to avoid double call
- Fix logger.exception calls: remove exception object as first arg in
storage.py, cookies.py, enrichments.py, housekeeping/main.py
- Fix document_conversion lifespan: use stack.callback() for sync shutdown
- Fix NoseyParkerOutput fallback: add missing workflow_id field
- Fix regipy hive_type: handle None return from RegistryHive.hive_type
DPAPI Manager:
- Remove unused guid parameter from get_system_credentials across
DpapiManager, NullDpapiManager, and DpapiManagerProtocol
Added Missing Dependencies (file_enrichment_modules):
- pypykatz>=0.6.11, pyarrow>=19.0.1, msoffcrypto-tool>=5.4.2,
oletools>=0.60.2, regipy>=5.2.0, pillow>=11.3.0
* quiet console logs
* feat: lazy file loading with backend range request support
Add offset/length query params to the download endpoint so the frontend
can request partial file content. FileViewer now fetches data on demand
— hex, transform (Strings, etc.), ZIP, SQLite, and image tabs only load
when activated. Text-based content is capped at 10 MB previews.
* fix: transform tabs stuck on "Loading content..."
- Prevent stale WS subscription from overwriting fetched content
- Show retry button when transform content fails to load
- Reset fetch guard on non-OK HTTP responses
* fix: hex tab deferred loading with full file content
* feat: truncation dropdown, spinner overlay, and tab render refactor
- Add truncation dropdown to MonacoContentViewer for files > 10MB,
replacing the old banner alert and hex "Load Hex View" button
- Add spinner overlay on Monaco editor while loading full content
- Wire truncation support to transform tabs (monaco/json types),
enrichment tabs, and text tabs
- Change hex tab to auto-load first 10MB preview with dropdown for
full file instead of requiring manual load
- Default word wrap to off
- Remove "File is too large" warning message
- Refactor ~150-line ternary chain into explicit per-tab render
functions (renderPreviewTab, renderZipTab, renderSqliteTab,
renderTextTab, renderHexTab, renderTabContent) with shared helpers
(renderFullFileContent, getTextContent)
* feat: improve Yara Rules UI
- Compact table rows with tighter padding
- Add X button and Esc key to close editor dialog
- Disable Save button when rule content is unchanged
- Disable Create button with inline warning when rule name already exists
- Default source to "Created manually by <user>" for new rules
- Update source placeholder to "e.g. /yara_rules/custom.yara"
* update gitignore
---------
Co-authored-by: Lee Chagolla-Christensen <lee@localhost>
- Raise requires-python to >=3.13 across all projects/libs; set ruff target to py313
- Fix bare excepts, unused variables/imports, Optional→union syntax, and import ordering
- Refactor chatbot MCP server to async subprocess with health checks and graceful shutdown
- Simplify file-enrichment prod Dockerfile to use uv sync with UV_PROJECT_ENVIRONMENT
- Fix module_loader to use fully qualified analyzer module name
- Make YARA _analyze_yara async and fix rule_manager access in workflow
- Enable LiteLLM healthcheck and add agents→litellm dependency
- Add PE analyzer tests with PyInstaller fixture;
- Update PDF tests for nested metadata
- Suppress third-party DeprecationWarnings, use async-lru for LLM status cache
- Add PHOENIX_ENABLED validation to nemesis-ctl.sh; remove debug pprint in agents
- Add Claude Code managing-packages skill, update CLAUDE.md guidance
Performance and configuration improvements:
- Tune OpenTelemetry Collector with batching, queuing, and memory limits
- Optimize Jaeger for write-heavy workloads with consistency disabled
- Add Prometheus scraping for OTel Collector internal metrics
- Increase document conversion prefetch count from 1 to 5
- Add CPU limit (2 cores) for document-conversion service
- Disable Tesseract OCR in default Tika config for faster processing
Workflow tracking enhancements:
- Add WorkflowTrackingService for centralized workflow state management
- Implement finalize_workflow activities for both services
- Track enrichment success/failure per workflow instance
- Support updating workflow status by object_id for subscriptions
Code improvements:
- Refactor PDF analyzer with modular extraction functions
- Add Unicode surrogate sanitization to prevent JSONB errors
- Update PII analyzer with improved error handling
- Add documentation links to Tika and Tesseract configuration
Frontend:
- Update StatsOverview component for workflow metrics display
- Restructure document_conversion to match into workflow structure
- PDF analyzer: use PyMuPDF and extract extensive metadata
- Add PDF test suite with encrypted/unencrypted fixtures
- Add Tika Tesseract OCR configuration support example
- Update Dapr configurations for new document_conversion component
- Remove deprecated helpers from common lib
- Update queue constants and workflow setup
* upgrade to dapr postgresv2 statestore
* actually make it v2
* dapr state table name, cleanup subscriptions/globals
* proper exceptions
* formatting/lint
* Refactor workflow tracking and improve activity input handling
- Extract workflow tracking logic into dedicated WorkflowTrackingService
- Simplify activity signatures to accept specific parameters instead of generic dicts
- Remove unused asyncio event loop references from enrichment modules
- Update YaraRuleManager initialization and method names
* re-added workflow tracking in the DB
* update uvicorn prod options
* enrichment work parallelism, convert queues from broadcast to task queues
* Refactor pubsub and improve workflow parallelism
- Split Dapr pubsub Dapr yaml components into topic-specific queues (alerting, dotnet, dpapi, files, noseyparker, workflow_monitor)
- Update all Dapr volume mounts to reference new topic-specific pubsub components
- Converted queues to task queues
- Use YAML anchors to reduce duplication for file-enrichment replicas
- Pass asyncpg pool to enrichment modules instead of creating connections
- Add asyncpg_pool parameter throughout chromium and enrichment module analyzers
- Update VSCode workspace (removed InspectAssembly, renamed dotnet_api to dotnet_service)
- Added curl commands for Jaeger API to performance docs to help with perf troubleshooting
- Created common.queues module to centralize pubsub/topic names (eases
future refactoring)
* worker mods
* Workflow performance tuning, fix pubsub config, CLI arg changes
- Fix pubsub deleteWhenUnused typo (deletedWhenUnused)
- Add LOG_LEVEL environment variable support across services
- CLI: Rename --repeat to --times, add --max-files option
- Increase files pubsub prefetchCount from 25 to 50
- Add MAX_PARALLEL_WORKFLOWS configuration
- Fix DotNetAssemblyAnalysis null handling with field validators
- Update dashboard to show cumulative files/findings over time
- Add RUST_LOG environment variable support to noseyparker
- Update CHANGELOG for 2.1.4 release notes
* Dapr 1.16.2 and use db transactions
- Upgrade all Dapr containers from 1.16.1 to 1.16.2
- Reduce enrichment parallelism default from 25 to 5 workflows
- Reduce healthcheck intervals from 10s to 5s for alerting and document conversion
- Fixed DPAPI eventing to use new pubsubs
- Refactor file_linking database operations to use atomic upserts and avoid deadlocks
- Add WriteOnceViolationError handling in DPAPI masterkey analyzer
- Wrap database operations in transactions for enrichment storage and plaintext indexing
- Fix postgres notification handler closure variable capture
* remove unused start_time
* Scheduler persistence, workflow concurrency tuning, and config cleanup
- Add volume for Dapr scheduler and init service
- Add scheduler dependency to file enrichment service
- Add async workflow client libraries
- Format and cleanup compose.yaml (spacing, indentation, empty lines)
* Migrate file_enrichment to async Dapr client and optimize Dockerfile
- Use async DaprClient where possible in file_enrichment
- Improve Dockerfile caching
- Add asyncpg connection pool helper and fix typo in secret store name
- Include VS Code debug configuration for document_conversion
- Remove unused dapr_client from DpapiBlobAnalyzer
- Clean up activity return types and better handle exceptions
* Enrichment tracking for NoseyParker and logging cleanup
- Add workflow_id to NoseyParkerInput and NoseyParkerOutput models
- Remove workflow lookup query in noseyparker subscription handler
- Adjust jaeger_perf_stats.sh output formatting and precision
- Add type hints for async functions
* noseyparker scanner perf, tracing for update_enrichment_results
---------
Co-authored-by: Lee Chagolla-Christensen <lee@localhost>
- Switch from using POSTGRES_CONNECTION_STRING to
POSTGRES_USER/POSTGRES_PASSWORD/POSTGRES_DATABASE/etc.
- When no apprise URL is configured, use the test echo'ing endpoint
- Add centralized PostgreSQL connection string helper in common/db.py
- Replace ntpath with posixpath
- Add get_drive_from_path() helper for extracting drive letters from paths
- Update all modules to use centralized DB connection helper instead of individual Dapr client calls
- Improve path normalization across Chromium, DPAPI, and file enrichment modules
- Added dependencies to local libraries
- Expand file linking tests with comprehensive rule engine test cases
- Refactor masterkey path construction to use posixpath.normpath for proper path joining
- Update state helpers and API models to use centralized connection
- Updated web API to use strongly typed models instead of dicts
internally
- Improve DPAPI core functionality with better error handling and path normalization
Lee Chagolla-Christensen