* Initial EKS deployment work
- Initial EKS deployment documentation and scripts
* Added EFS for EKS `/mounted-containers`
- Added EFS for EKS `/mounted-containers` for large container processing
* Fix EKS Helm deployment bugs and agents API change (#109)
* fix: pods crash on startup due to psql credential path mismatch
templates referenced .Values.postgres which don't exist in values. Updated to .Values.credentials.postgres.
* fix: jaeger container crash backoff due to v1 vs v2 service config diff
* fix: pod crash back off due to loki require of delete_request_store also securityContext updated
* fix: pod crash back off due to non valid config keys
* fix: chatbot 503 mcp cannot start due to chatbot_readonly missing
* nit: add aws session token for STS creds for testing if blank should not make a difference
* fix: deps issue with pydantic-ai getting modelmanager fail with unexpected keywaord ardument controller
* Change default action build trigger
---------
Co-authored-by: Jake <93880195+aggr0cr4g@users.noreply.github.com>
* Add k8s/k3d/k3s deployment with Helm, KEDA autoscaling, and connection pool fixes
- Helm chart for full Nemesis deployment (Traefik, Dapr, KEDA, all services)
- KEDA autoscaling for file-enrichment and document-conversion on RabbitMQ queue depth
- Fix PostgreSQL connection exhaustion: configurable pool size (default 20, was 100)
- Cap Dapr statestore connections (maxConns=10), increase postgres max_connections to 300
- Add Prometheus metrics and /system/pool-stats endpoint for connection pool monitoring
- Background pool stats logger with pressure warnings
- Fix Prometheus scrape targets to correct service ports
- Helm-based Dapr install (v1.16.9) instead of CLI
- k8s setup/deploy/build/verify/teardown scripts
- Kubernetes deployment docs and references in CLAUDE.md, AGENTS.md, mkdocs
* Add KEDA autoscaling for titus-scanner and dotnet-service
* Add PgBouncer connection pooling, KEDA gotenberg CPU scaling, and tuning
- Add PgBouncer deployment between services and PostgreSQL (transaction pooling)
- Route all DB connections through pgbouncer instead of postgres directly
- Add KEDA CPU-based autoscaling for gotenberg (1-3 replicas)
- Mirror external images (pgbouncer) into k3d registry via build-and-push.sh
- Reduce queue-based cooldowns to 60s, gotenberg stabilization to 30s
- Add 180s terminationGracePeriodSeconds to gotenberg for in-flight conversions
- Update docs with PgBouncer architecture, connection tuning, and troubleshooting
* Add k3s support, unify Traefik on port 7443, rename scripts
- Add setup-cluster-k3s.sh and teardown-cluster-k3s.sh for native k3s
- Rename scripts with -k3d suffix to distinguish runtimes
- Unify k3s Traefik to HTTPS-only on port 7443 (matching k3d)
- Add k3d registry check for --build flag in deploy.sh
- Fix verify.sh arithmetic under set -e with || true
- Update docs with k3s quick start, comparison table, and teardown
- Add image-sync workflow and pgbouncer registry override for values-dev
* Add monitoring, Jupyter, and LLM optional stacks to Helm chart
- Monitoring: Prometheus, Grafana (with dashboards), Loki, Promtail,
Jaeger, OTEL Collector, Node Exporter, Postgres Exporter
- Jupyter: notebook service with Hasura integration
- LLM: Agents (Dapr-enabled), LiteLLM proxy, Phoenix observability
- Add ingress routes, strip-prefix middleware, litellm database
- Update deploy.sh, build script, README, and test connectivity checks
* Add k3s --build support with auto-detection in deploy.sh
- deploy.sh --build auto-detects k3d registry vs k3s and uses appropriate build script
- Add build-and-load-k3s.sh for building and importing images into k3s containerd
- Add values-dev-k3s.yaml with imagePullPolicy: Never (no pgbouncer registry override)
- Fix jupyter Dockerfile context path in build-and-push-k3d.sh
- Update docs for k3s local build support
---------
Co-authored-by: Lee Chagolla-Christensen <lee@localhost>
* Conversion of NoseyParker scanner to Titus
- Old `noseyparker_scanner` project removed, new `titus_scaner` Golang project built
- Replaced `noseyparker` references with `titus` as appropriate
- Bumped various built-in docker image versions (Dapr, Gotenberg, etc.)
* Replace ZIP-only extraction with Titus library archive support
Replace custom ZIP extraction code with Titus library's enum.ExtractText()
to support scanning secrets inside zip, jar, war, ear, apk, ipa, xpi, crx,
tar, tar.gz/tgz, and 7z archives. Document formats (xlsx, docx, pdf, etc.)
are intentionally excluded since Nemesis handles those via document_conversion.
- Add original_path to TitusInput so Titus can dispatch by file extension
(MinIO stores files by UUID without extensions)
- Add magic-byte fallback detection for ZIP, 7z, gzip, and tar
- Preserve git-repo-in-ZIP detection as a special case before archive scan
- Rename env vars: DECOMPRESS_ZIPS -> EXTRACT_ARCHIVES,
MAX_EXTRACT_SIZE_MB -> EXTRACT_MAX_TOTAL_SIZE_MB, and add
EXTRACT_MAX_FILE_SIZE_MB and EXTRACT_MAX_DEPTH
- Add unit tests for archive detection and config loading
* Integrated (optional) Titus secret validation
- Integrated (optional) Titus secret validation that can be enabled
* Added Titus rule ID display and ability to disable specific rules
- Added Titus rule ID display and ability to disable specific built-in rules (by ID)
* `titus_scanner` fixes
- One scanner per concurrent file to prevent timeouts
- Fix custom rule regexes to prevent timeouts
- Exclude a few default run-away rules causing timeouts
- Bumped cores for titus scanner to 2
* Enable Vectorscan/Hyperscan acceleration for titus_scanner
Switch from the pure-Go regexp2 engine to the Vectorscan (Hyperscan)
SIMD-accelerated engine, using a fork with parallelized regexp2
confirmation and content-based deduplication
Key changes:
- Dockerfile: install libvectorscan-dev/libvectorscan5, build with
CGO_ENABLED=1 and -tags vectorscan
- go.mod: replace praetorian-inc/titus with HarmJ0y/titus fork that
parallelizes the regexp2 confirmation loop and uses content-based
dedup (matching portable engine behavior)
- Add bounded-concurrency semaphore in handler to prevent OOM from
unbounded goroutines while still returning 200 to Dapr immediately
- Add content-based deduplication, match count cap, snippet truncation,
and payload size logging to prevent oversized publishes
- Replace O(N*M) line/column computation with precomputed line index
using binary search
- Add GOMEMLIMIT and memory limit to container resources
* Update README.md
* remove `noseyparker_scanner` container
* Add EVTX (Windows Event Log) enrichment module
Parses .evtx files and extracts security-relevant data:
- Markdown summary transform with event counts, timeline, unique accounts/IPs
- Downloadable CSV transforms for account changes, group membership changes,
explicit credential use (4648), process creation (4688), and task changes
- Findings for high-signal events: audit log cleared, new services, admin group
changes, account changes, explicit creds, scheduled tasks, PS script blocks
- PowerShell 4104 script blocks reassembled and resubmitted to enrichment
pipeline as child .ps1 files for full Titus/YARA scanning
- Added de-duping for duplicate PS scripts carved from EVTXs (hash-based)
* EVTX mods
- Power timelining done for SYSTEM
- Inbound/outbound auth for SECURITY
- Filtering for default system accounts
- CSV viewer now can download files, transforms for EVTX transformed to the viewer
* Convert titus scanner to Dapr bulk subscribe for batch processing
- Add bulk subscribe types to models (BulkMessagePayload, BulkResponse, etc.)
- Replace async single-event handler with synchronous HandleBulkEvent
- Process batches with bounded concurrency via semaphore + WaitGroup
- Handle edge cases: duplicate entryId, empty objectID/entryId, panic recovery
- Add BulkMaxMessages and BulkMaxAwaitDurationMs config with clamping
- Set app-max-concurrency to 1 (parallelism within work batch via scanner pool)
---------
Co-authored-by: Lee Chagolla-Christensen <lee@localhost>
- Raise requires-python to >=3.13 across all projects/libs; set ruff target to py313
- Fix bare excepts, unused variables/imports, Optional→union syntax, and import ordering
- Refactor chatbot MCP server to async subprocess with health checks and graceful shutdown
- Simplify file-enrichment prod Dockerfile to use uv sync with UV_PROJECT_ENVIRONMENT
- Fix module_loader to use fully qualified analyzer module name
- Make YARA _analyze_yara async and fix rule_manager access in workflow
- Enable LiteLLM healthcheck and add agents→litellm dependency
- Add PE analyzer tests with PyInstaller fixture;
- Update PDF tests for nested metadata
- Suppress third-party DeprecationWarnings, use async-lru for LLM status cache
- Add PHOENIX_ENABLED validation to nemesis-ctl.sh; remove debug pprint in agents
- Add Claude Code managing-packages skill, update CLAUDE.md guidance
* Added Cobalt Strike Connector
* Updated cli readme
* Add Cobalt Strike to C2 connectors list in README
Updated README to include Cobalt Strike in C2 connectors.
* Update usage guide to include Cobalt Strike
* Document Cobalt Strike Connector setup and usage
Added Cobalt Strike Connector section with configuration and usage instructions.
* Updated documentation on REST API startup and modified Path to include the file name
* Update Cobalt Strike URL to include port 50443
* adding code 403 to force reauthentication to mathc with the CS response
- PII analyzer disabled by default, can be enabled via ENV variable
- Created `enrichment_configuration.md` documentation documenting all enrichment module configurations
* upgrade to dapr postgresv2 statestore
* actually make it v2
* dapr state table name, cleanup subscriptions/globals
* proper exceptions
* formatting/lint
* Refactor workflow tracking and improve activity input handling
- Extract workflow tracking logic into dedicated WorkflowTrackingService
- Simplify activity signatures to accept specific parameters instead of generic dicts
- Remove unused asyncio event loop references from enrichment modules
- Update YaraRuleManager initialization and method names
* re-added workflow tracking in the DB
* update uvicorn prod options
* enrichment work parallelism, convert queues from broadcast to task queues
* Refactor pubsub and improve workflow parallelism
- Split Dapr pubsub Dapr yaml components into topic-specific queues (alerting, dotnet, dpapi, files, noseyparker, workflow_monitor)
- Update all Dapr volume mounts to reference new topic-specific pubsub components
- Converted queues to task queues
- Use YAML anchors to reduce duplication for file-enrichment replicas
- Pass asyncpg pool to enrichment modules instead of creating connections
- Add asyncpg_pool parameter throughout chromium and enrichment module analyzers
- Update VSCode workspace (removed InspectAssembly, renamed dotnet_api to dotnet_service)
- Added curl commands for Jaeger API to performance docs to help with perf troubleshooting
- Created common.queues module to centralize pubsub/topic names (eases
future refactoring)
* worker mods
* Workflow performance tuning, fix pubsub config, CLI arg changes
- Fix pubsub deleteWhenUnused typo (deletedWhenUnused)
- Add LOG_LEVEL environment variable support across services
- CLI: Rename --repeat to --times, add --max-files option
- Increase files pubsub prefetchCount from 25 to 50
- Add MAX_PARALLEL_WORKFLOWS configuration
- Fix DotNetAssemblyAnalysis null handling with field validators
- Update dashboard to show cumulative files/findings over time
- Add RUST_LOG environment variable support to noseyparker
- Update CHANGELOG for 2.1.4 release notes
* Dapr 1.16.2 and use db transactions
- Upgrade all Dapr containers from 1.16.1 to 1.16.2
- Reduce enrichment parallelism default from 25 to 5 workflows
- Reduce healthcheck intervals from 10s to 5s for alerting and document conversion
- Fixed DPAPI eventing to use new pubsubs
- Refactor file_linking database operations to use atomic upserts and avoid deadlocks
- Add WriteOnceViolationError handling in DPAPI masterkey analyzer
- Wrap database operations in transactions for enrichment storage and plaintext indexing
- Fix postgres notification handler closure variable capture
* remove unused start_time
* Scheduler persistence, workflow concurrency tuning, and config cleanup
- Add volume for Dapr scheduler and init service
- Add scheduler dependency to file enrichment service
- Add async workflow client libraries
- Format and cleanup compose.yaml (spacing, indentation, empty lines)
* Migrate file_enrichment to async Dapr client and optimize Dockerfile
- Use async DaprClient where possible in file_enrichment
- Improve Dockerfile caching
- Add asyncpg connection pool helper and fix typo in secret store name
- Include VS Code debug configuration for document_conversion
- Remove unused dapr_client from DpapiBlobAnalyzer
- Clean up activity return types and better handle exceptions
* Enrichment tracking for NoseyParker and logging cleanup
- Add workflow_id to NoseyParkerInput and NoseyParkerOutput models
- Remove workflow lookup query in noseyparker subscription handler
- Adjust jaeger_perf_stats.sh output formatting and precision
- Add type hints for async functions
* noseyparker scanner perf, tracing for update_enrichment_results
---------
Co-authored-by: Lee Chagolla-Christensen <lee@localhost>
- Switch from using POSTGRES_CONNECTION_STRING to
POSTGRES_USER/POSTGRES_PASSWORD/POSTGRES_DATABASE/etc.
- When no apprise URL is configured, use the test echo'ing endpoint
Lee Chagolla-Christensen