* fix mythic connector with unwaited coroutine
* Change download_callback type from Awaitable to Coroutine for pyright detection
---------
Co-authored-by: Lee Chagolla-Christensen <lee@localhost>
* Add k8s/k3d/k3s deployment with Helm, KEDA autoscaling, and connection pool fixes
- Helm chart for full Nemesis deployment (Traefik, Dapr, KEDA, all services)
- KEDA autoscaling for file-enrichment and document-conversion on RabbitMQ queue depth
- Fix PostgreSQL connection exhaustion: configurable pool size (default 20, was 100)
- Cap Dapr statestore connections (maxConns=10), increase postgres max_connections to 300
- Add Prometheus metrics and /system/pool-stats endpoint for connection pool monitoring
- Background pool stats logger with pressure warnings
- Fix Prometheus scrape targets to correct service ports
- Helm-based Dapr install (v1.16.9) instead of CLI
- k8s setup/deploy/build/verify/teardown scripts
- Kubernetes deployment docs and references in CLAUDE.md, AGENTS.md, mkdocs
* Add KEDA autoscaling for titus-scanner and dotnet-service
* Add PgBouncer connection pooling, KEDA gotenberg CPU scaling, and tuning
- Add PgBouncer deployment between services and PostgreSQL (transaction pooling)
- Route all DB connections through pgbouncer instead of postgres directly
- Add KEDA CPU-based autoscaling for gotenberg (1-3 replicas)
- Mirror external images (pgbouncer) into k3d registry via build-and-push.sh
- Reduce queue-based cooldowns to 60s, gotenberg stabilization to 30s
- Add 180s terminationGracePeriodSeconds to gotenberg for in-flight conversions
- Update docs with PgBouncer architecture, connection tuning, and troubleshooting
* Add k3s support, unify Traefik on port 7443, rename scripts
- Add setup-cluster-k3s.sh and teardown-cluster-k3s.sh for native k3s
- Rename scripts with -k3d suffix to distinguish runtimes
- Unify k3s Traefik to HTTPS-only on port 7443 (matching k3d)
- Add k3d registry check for --build flag in deploy.sh
- Fix verify.sh arithmetic under set -e with || true
- Update docs with k3s quick start, comparison table, and teardown
- Add image-sync workflow and pgbouncer registry override for values-dev
* Add monitoring, Jupyter, and LLM optional stacks to Helm chart
- Monitoring: Prometheus, Grafana (with dashboards), Loki, Promtail,
Jaeger, OTEL Collector, Node Exporter, Postgres Exporter
- Jupyter: notebook service with Hasura integration
- LLM: Agents (Dapr-enabled), LiteLLM proxy, Phoenix observability
- Add ingress routes, strip-prefix middleware, litellm database
- Update deploy.sh, build script, README, and test connectivity checks
* Add k3s --build support with auto-detection in deploy.sh
- deploy.sh --build auto-detects k3d registry vs k3s and uses appropriate build script
- Add build-and-load-k3s.sh for building and importing images into k3s containerd
- Add values-dev-k3s.yaml with imagePullPolicy: Never (no pgbouncer registry override)
- Fix jupyter Dockerfile context path in build-and-push-k3d.sh
- Update docs for k3s local build support
---------
Co-authored-by: Lee Chagolla-Christensen <lee@localhost>
* fix: upgrade python-multipart to 0.0.22 to address CVE-2026-24486 (Dependabot #683)
* update deps+skill, pyright
* pyright + tests in CLI
* Add pyright type checking and enhance registry hive analysis
Registry Hive Analyzer Enhancements:
- Extract machine SID from SAM domain V value with binary SID decoding
- Extract per-user metadata via regipy (ACB flags, timestamps, full
name, comment via USER_ACCOUNT_V)
- Compute password expiration from domain max password age policy
- Detect empty LM/NT hashes via well-known constants
- Parse DCC cached domain credentials into structured entries
- Store DPAPI system machine_key/user_key as separate fields
- Add structured secret_type field to LSA secrets (dcc, dpapi_system,
hex_blob, generic)
- Add detailed markdown and plain-text formatters for SAM accounts
and LSA secrets, replacing _get_lsa_secret_output_string
- Fix SYSTEM hive attribute names: computer_name -> machinename,
current_control_set -> currentcontrol
- Switch SAM user iteration from sam.users to sam.secrets (pypykatz API)
- Add FILETIME-to-UTC and regipy value-to-bytes helpers
New Tests:
- Add test_registry_hive.py with SAM, SECURITY, and SYSTEM hive fixtures
- Add test_container.py for container analyzer
- Add SAM/SECURITY/SYSTEM binary test fixtures
Pyright Setup:
- Add pyrightconfig.json (basic mode, Python 3.13) to all libs and projects
- Add pyright>=1.1 as dev dependency to all pyproject.toml files
- Update all uv.lock files accordingly
- Update lint.sh to deactivate active venvs and verify pyright availability
Type Annotation Fixes:
- Fix globals initialized as None without Optional type across
file_enrichment, document_conversion, and agents global_vars
- Fix StorageMinio return types: upload/upload_file/upload_uploadfile
return str, not uuid.UUID
- Fix MockStorageMinio to match updated StorageMinio return types
- Add explicit type annotations to dict literals in chromekey.py,
pdf/analyzer.py, registry_hive/analyzer.py, and publish_findings.py
- Fix kubeconfig current_context parameter to accept str | None
- Fix container_contents allowed_extensions: set = None -> set | None = None
- Fix file subscription file_queue: asyncio.Queue = None -> asyncio.Queue | None = None
- Fix web_api upload_file to wrap object_id in uuid.UUID() for response
None-safety Assertions:
- Add assert statements for asyncpg_pool, tracking_service,
workflow_client, workflow_manager, file_linking_engine, file_queue,
gotenberg_url, and process.stdout across all activity, subscription,
route, and workflow files in file_enrichment and document_conversion
- Add assertions for asyncpg_pool in all chromium processors
- Add assertions for File.from_metadata timestamp/expiration fields
- Add assertion for alerting GQL client session type
Pyright Ignore Annotations:
- Suppress third-party type issues in Dapr workflow/activity APIs,
gRPC subscription imports, ccache/lnk/office_doc attribute access,
and nemesis_dpapi FlagMixin operators
- Add file-level suppression for office2john.py, pdf2john.py,
pe/analyzer.py, and test harness files
Bug Fixes:
- Fix office2john.py format string: bare % filename -> % (filename, stream)
- Fix container analyzer 7z iteration: iterate sz.files list instead
of calling .items()
- Fix file_linking rules_engine: store match result to avoid double call
- Fix logger.exception calls: remove exception object as first arg in
storage.py, cookies.py, enrichments.py, housekeeping/main.py
- Fix document_conversion lifespan: use stack.callback() for sync shutdown
- Fix NoseyParkerOutput fallback: add missing workflow_id field
- Fix regipy hive_type: handle None return from RegistryHive.hive_type
DPAPI Manager:
- Remove unused guid parameter from get_system_credentials across
DpapiManager, NullDpapiManager, and DpapiManagerProtocol
Added Missing Dependencies (file_enrichment_modules):
- pypykatz>=0.6.11, pyarrow>=19.0.1, msoffcrypto-tool>=5.4.2,
oletools>=0.60.2, regipy>=5.2.0, pillow>=11.3.0
* quiet console logs
* feat: lazy file loading with backend range request support
Add offset/length query params to the download endpoint so the frontend
can request partial file content. FileViewer now fetches data on demand
— hex, transform (Strings, etc.), ZIP, SQLite, and image tabs only load
when activated. Text-based content is capped at 10 MB previews.
* fix: transform tabs stuck on "Loading content..."
- Prevent stale WS subscription from overwriting fetched content
- Show retry button when transform content fails to load
- Reset fetch guard on non-OK HTTP responses
* fix: hex tab deferred loading with full file content
* feat: truncation dropdown, spinner overlay, and tab render refactor
- Add truncation dropdown to MonacoContentViewer for files > 10MB,
replacing the old banner alert and hex "Load Hex View" button
- Add spinner overlay on Monaco editor while loading full content
- Wire truncation support to transform tabs (monaco/json types),
enrichment tabs, and text tabs
- Change hex tab to auto-load first 10MB preview with dropdown for
full file instead of requiring manual load
- Default word wrap to off
- Remove "File is too large" warning message
- Refactor ~150-line ternary chain into explicit per-tab render
functions (renderPreviewTab, renderZipTab, renderSqliteTab,
renderTextTab, renderHexTab, renderTabContent) with shared helpers
(renderFullFileContent, getTextContent)
* feat: improve Yara Rules UI
- Compact table rows with tighter padding
- Add X button and Esc key to close editor dialog
- Disable Save button when rule content is unchanged
- Disable Create button with inline warning when rule name already exists
- Default source to "Created manually by <user>" for new rules
- Update source placeholder to "e.g. /yara_rules/custom.yara"
* update gitignore
---------
Co-authored-by: Lee Chagolla-Christensen <lee@localhost>
- Raise requires-python to >=3.13 across all projects/libs; set ruff target to py313
- Fix bare excepts, unused variables/imports, Optional→union syntax, and import ordering
- Refactor chatbot MCP server to async subprocess with health checks and graceful shutdown
- Simplify file-enrichment prod Dockerfile to use uv sync with UV_PROJECT_ENVIRONMENT
- Fix module_loader to use fully qualified analyzer module name
- Make YARA _analyze_yara async and fix rule_manager access in workflow
- Enable LiteLLM healthcheck and add agents→litellm dependency
- Add PE analyzer tests with PyInstaller fixture;
- Update PDF tests for nested metadata
- Suppress third-party DeprecationWarnings, use async-lru for LLM status cache
- Add PHOENIX_ENABLED validation to nemesis-ctl.sh; remove debug pprint in agents
- Add Claude Code managing-packages skill, update CLAUDE.md guidance
* Added Cobalt Strike Connector
* Updated cli readme
* Add Cobalt Strike to C2 connectors list in README
Updated README to include Cobalt Strike in C2 connectors.
* Update usage guide to include Cobalt Strike
* Document Cobalt Strike Connector setup and usage
Added Cobalt Strike Connector section with configuration and usage instructions.
* Updated documentation on REST API startup and modified Path to include the file name
* Update Cobalt Strike URL to include port 50443
* adding code 403 to force reauthentication to mathc with the CS response
- Restructure document_conversion to match into workflow structure
- PDF analyzer: use PyMuPDF and extract extensive metadata
- Add PDF test suite with encrypted/unencrypted fixtures
- Add Tika Tesseract OCR configuration support example
- Update Dapr configurations for new document_conversion component
- Remove deprecated helpers from common lib
- Update queue constants and workflow setup
* upgrade to dapr postgresv2 statestore
* actually make it v2
* dapr state table name, cleanup subscriptions/globals
* proper exceptions
* formatting/lint
* Refactor workflow tracking and improve activity input handling
- Extract workflow tracking logic into dedicated WorkflowTrackingService
- Simplify activity signatures to accept specific parameters instead of generic dicts
- Remove unused asyncio event loop references from enrichment modules
- Update YaraRuleManager initialization and method names
* re-added workflow tracking in the DB
* update uvicorn prod options
* enrichment work parallelism, convert queues from broadcast to task queues
* Refactor pubsub and improve workflow parallelism
- Split Dapr pubsub Dapr yaml components into topic-specific queues (alerting, dotnet, dpapi, files, noseyparker, workflow_monitor)
- Update all Dapr volume mounts to reference new topic-specific pubsub components
- Converted queues to task queues
- Use YAML anchors to reduce duplication for file-enrichment replicas
- Pass asyncpg pool to enrichment modules instead of creating connections
- Add asyncpg_pool parameter throughout chromium and enrichment module analyzers
- Update VSCode workspace (removed InspectAssembly, renamed dotnet_api to dotnet_service)
- Added curl commands for Jaeger API to performance docs to help with perf troubleshooting
- Created common.queues module to centralize pubsub/topic names (eases
future refactoring)
* worker mods
* Workflow performance tuning, fix pubsub config, CLI arg changes
- Fix pubsub deleteWhenUnused typo (deletedWhenUnused)
- Add LOG_LEVEL environment variable support across services
- CLI: Rename --repeat to --times, add --max-files option
- Increase files pubsub prefetchCount from 25 to 50
- Add MAX_PARALLEL_WORKFLOWS configuration
- Fix DotNetAssemblyAnalysis null handling with field validators
- Update dashboard to show cumulative files/findings over time
- Add RUST_LOG environment variable support to noseyparker
- Update CHANGELOG for 2.1.4 release notes
* Dapr 1.16.2 and use db transactions
- Upgrade all Dapr containers from 1.16.1 to 1.16.2
- Reduce enrichment parallelism default from 25 to 5 workflows
- Reduce healthcheck intervals from 10s to 5s for alerting and document conversion
- Fixed DPAPI eventing to use new pubsubs
- Refactor file_linking database operations to use atomic upserts and avoid deadlocks
- Add WriteOnceViolationError handling in DPAPI masterkey analyzer
- Wrap database operations in transactions for enrichment storage and plaintext indexing
- Fix postgres notification handler closure variable capture
* remove unused start_time
* Scheduler persistence, workflow concurrency tuning, and config cleanup
- Add volume for Dapr scheduler and init service
- Add scheduler dependency to file enrichment service
- Add async workflow client libraries
- Format and cleanup compose.yaml (spacing, indentation, empty lines)
* Migrate file_enrichment to async Dapr client and optimize Dockerfile
- Use async DaprClient where possible in file_enrichment
- Improve Dockerfile caching
- Add asyncpg connection pool helper and fix typo in secret store name
- Include VS Code debug configuration for document_conversion
- Remove unused dapr_client from DpapiBlobAnalyzer
- Clean up activity return types and better handle exceptions
* Enrichment tracking for NoseyParker and logging cleanup
- Add workflow_id to NoseyParkerInput and NoseyParkerOutput models
- Remove workflow lookup query in noseyparker subscription handler
- Adjust jaeger_perf_stats.sh output formatting and precision
- Add type hints for async functions
* noseyparker scanner perf, tracing for update_enrichment_results
---------
Co-authored-by: Lee Chagolla-Christensen <lee@localhost>
- Add centralized PostgreSQL connection string helper in common/db.py
- Replace ntpath with posixpath
- Add get_drive_from_path() helper for extracting drive letters from paths
- Update all modules to use centralized DB connection helper instead of individual Dapr client calls
- Improve path normalization across Chromium, DPAPI, and file enrichment modules
- Added dependencies to local libraries
- Expand file linking tests with comprehensive rule engine test cases
- Refactor masterkey path construction to use posixpath.normpath for proper path joining
- Update state helpers and API models to use centralized connection
- Updated web API to use strongly typed models instead of dicts
internally
- Improve DPAPI core functionality with better error handling and path normalization
Lee Chagolla-Christensen