* fix mythic connector with unwaited coroutine
* Change download_callback type from Awaitable to Coroutine for pyright detection
---------
Co-authored-by: Lee Chagolla-Christensen <lee@localhost>
* Add k8s/k3d/k3s deployment with Helm, KEDA autoscaling, and connection pool fixes
- Helm chart for full Nemesis deployment (Traefik, Dapr, KEDA, all services)
- KEDA autoscaling for file-enrichment and document-conversion on RabbitMQ queue depth
- Fix PostgreSQL connection exhaustion: configurable pool size (default 20, was 100)
- Cap Dapr statestore connections (maxConns=10), increase postgres max_connections to 300
- Add Prometheus metrics and /system/pool-stats endpoint for connection pool monitoring
- Background pool stats logger with pressure warnings
- Fix Prometheus scrape targets to correct service ports
- Helm-based Dapr install (v1.16.9) instead of CLI
- k8s setup/deploy/build/verify/teardown scripts
- Kubernetes deployment docs and references in CLAUDE.md, AGENTS.md, mkdocs
* Add KEDA autoscaling for titus-scanner and dotnet-service
* Add PgBouncer connection pooling, KEDA gotenberg CPU scaling, and tuning
- Add PgBouncer deployment between services and PostgreSQL (transaction pooling)
- Route all DB connections through pgbouncer instead of postgres directly
- Add KEDA CPU-based autoscaling for gotenberg (1-3 replicas)
- Mirror external images (pgbouncer) into k3d registry via build-and-push.sh
- Reduce queue-based cooldowns to 60s, gotenberg stabilization to 30s
- Add 180s terminationGracePeriodSeconds to gotenberg for in-flight conversions
- Update docs with PgBouncer architecture, connection tuning, and troubleshooting
* Add k3s support, unify Traefik on port 7443, rename scripts
- Add setup-cluster-k3s.sh and teardown-cluster-k3s.sh for native k3s
- Rename scripts with -k3d suffix to distinguish runtimes
- Unify k3s Traefik to HTTPS-only on port 7443 (matching k3d)
- Add k3d registry check for --build flag in deploy.sh
- Fix verify.sh arithmetic under set -e with || true
- Update docs with k3s quick start, comparison table, and teardown
- Add image-sync workflow and pgbouncer registry override for values-dev
* Add monitoring, Jupyter, and LLM optional stacks to Helm chart
- Monitoring: Prometheus, Grafana (with dashboards), Loki, Promtail,
Jaeger, OTEL Collector, Node Exporter, Postgres Exporter
- Jupyter: notebook service with Hasura integration
- LLM: Agents (Dapr-enabled), LiteLLM proxy, Phoenix observability
- Add ingress routes, strip-prefix middleware, litellm database
- Update deploy.sh, build script, README, and test connectivity checks
* Add k3s --build support with auto-detection in deploy.sh
- deploy.sh --build auto-detects k3d registry vs k3s and uses appropriate build script
- Add build-and-load-k3s.sh for building and importing images into k3s containerd
- Add values-dev-k3s.yaml with imagePullPolicy: Never (no pgbouncer registry override)
- Fix jupyter Dockerfile context path in build-and-push-k3d.sh
- Update docs for k3s local build support
---------
Co-authored-by: Lee Chagolla-Christensen <lee@localhost>
- Alerting: return "sent"/"filtered"/"failed" from send_alert_with_retries() so filtered alerts don't log as delivery failures
- file-enrichment: parse raw_data JSON string from asyncpg before indexing, add TypeError to except clause
- titus-scanner: verify gzip contains tar archive before classifying as .tar.gz to avoid invalid tar header errors on plain .gz files
* Conversion of NoseyParker scanner to Titus
- Old `noseyparker_scanner` project removed, new `titus_scaner` Golang project built
- Replaced `noseyparker` references with `titus` as appropriate
- Bumped various built-in docker image versions (Dapr, Gotenberg, etc.)
* Replace ZIP-only extraction with Titus library archive support
Replace custom ZIP extraction code with Titus library's enum.ExtractText()
to support scanning secrets inside zip, jar, war, ear, apk, ipa, xpi, crx,
tar, tar.gz/tgz, and 7z archives. Document formats (xlsx, docx, pdf, etc.)
are intentionally excluded since Nemesis handles those via document_conversion.
- Add original_path to TitusInput so Titus can dispatch by file extension
(MinIO stores files by UUID without extensions)
- Add magic-byte fallback detection for ZIP, 7z, gzip, and tar
- Preserve git-repo-in-ZIP detection as a special case before archive scan
- Rename env vars: DECOMPRESS_ZIPS -> EXTRACT_ARCHIVES,
MAX_EXTRACT_SIZE_MB -> EXTRACT_MAX_TOTAL_SIZE_MB, and add
EXTRACT_MAX_FILE_SIZE_MB and EXTRACT_MAX_DEPTH
- Add unit tests for archive detection and config loading
* Integrated (optional) Titus secret validation
- Integrated (optional) Titus secret validation that can be enabled
* Added Titus rule ID display and ability to disable specific rules
- Added Titus rule ID display and ability to disable specific built-in rules (by ID)
* `titus_scanner` fixes
- One scanner per concurrent file to prevent timeouts
- Fix custom rule regexes to prevent timeouts
- Exclude a few default run-away rules causing timeouts
- Bumped cores for titus scanner to 2
* Enable Vectorscan/Hyperscan acceleration for titus_scanner
Switch from the pure-Go regexp2 engine to the Vectorscan (Hyperscan)
SIMD-accelerated engine, using a fork with parallelized regexp2
confirmation and content-based deduplication
Key changes:
- Dockerfile: install libvectorscan-dev/libvectorscan5, build with
CGO_ENABLED=1 and -tags vectorscan
- go.mod: replace praetorian-inc/titus with HarmJ0y/titus fork that
parallelizes the regexp2 confirmation loop and uses content-based
dedup (matching portable engine behavior)
- Add bounded-concurrency semaphore in handler to prevent OOM from
unbounded goroutines while still returning 200 to Dapr immediately
- Add content-based deduplication, match count cap, snippet truncation,
and payload size logging to prevent oversized publishes
- Replace O(N*M) line/column computation with precomputed line index
using binary search
- Add GOMEMLIMIT and memory limit to container resources
* Update README.md
* remove `noseyparker_scanner` container
* Add EVTX (Windows Event Log) enrichment module
Parses .evtx files and extracts security-relevant data:
- Markdown summary transform with event counts, timeline, unique accounts/IPs
- Downloadable CSV transforms for account changes, group membership changes,
explicit credential use (4648), process creation (4688), and task changes
- Findings for high-signal events: audit log cleared, new services, admin group
changes, account changes, explicit creds, scheduled tasks, PS script blocks
- PowerShell 4104 script blocks reassembled and resubmitted to enrichment
pipeline as child .ps1 files for full Titus/YARA scanning
- Added de-duping for duplicate PS scripts carved from EVTXs (hash-based)
* EVTX mods
- Power timelining done for SYSTEM
- Inbound/outbound auth for SECURITY
- Filtering for default system accounts
- CSV viewer now can download files, transforms for EVTX transformed to the viewer
* Convert titus scanner to Dapr bulk subscribe for batch processing
- Add bulk subscribe types to models (BulkMessagePayload, BulkResponse, etc.)
- Replace async single-event handler with synchronous HandleBulkEvent
- Process batches with bounded concurrency via semaphore + WaitGroup
- Handle edge cases: duplicate entryId, empty objectID/entryId, panic recovery
- Add BulkMaxMessages and BulkMaxAwaitDurationMs config with clamping
- Set app-max-concurrency to 1 (parallelism within work batch via scanner pool)
---------
Co-authored-by: Lee Chagolla-Christensen <lee@localhost>
- Upgrade cryptography to 46.0.5 across all projects (SECT curve subgroup attack)
- Upgrade pydantic-ai from 0.7.x to 1.58.0 (SSRF in URL download handling)
- Migrate AsyncTenacityTransport to use RetryConfig instead of AsyncRetrying
- Rename OpenAIModel to OpenAIChatModel (deprecated in pydantic-ai 1.x)
- Fix model_manager type safety by guarding against None model_name
- Relax uvicorn and websockets pins for pydantic-ai 1.x compatibility
* fix: upgrade python-multipart to 0.0.22 to address CVE-2026-24486 (Dependabot #683)
* update deps+skill, pyright
* pyright + tests in CLI
* Add pyright type checking and enhance registry hive analysis
Registry Hive Analyzer Enhancements:
- Extract machine SID from SAM domain V value with binary SID decoding
- Extract per-user metadata via regipy (ACB flags, timestamps, full
name, comment via USER_ACCOUNT_V)
- Compute password expiration from domain max password age policy
- Detect empty LM/NT hashes via well-known constants
- Parse DCC cached domain credentials into structured entries
- Store DPAPI system machine_key/user_key as separate fields
- Add structured secret_type field to LSA secrets (dcc, dpapi_system,
hex_blob, generic)
- Add detailed markdown and plain-text formatters for SAM accounts
and LSA secrets, replacing _get_lsa_secret_output_string
- Fix SYSTEM hive attribute names: computer_name -> machinename,
current_control_set -> currentcontrol
- Switch SAM user iteration from sam.users to sam.secrets (pypykatz API)
- Add FILETIME-to-UTC and regipy value-to-bytes helpers
New Tests:
- Add test_registry_hive.py with SAM, SECURITY, and SYSTEM hive fixtures
- Add test_container.py for container analyzer
- Add SAM/SECURITY/SYSTEM binary test fixtures
Pyright Setup:
- Add pyrightconfig.json (basic mode, Python 3.13) to all libs and projects
- Add pyright>=1.1 as dev dependency to all pyproject.toml files
- Update all uv.lock files accordingly
- Update lint.sh to deactivate active venvs and verify pyright availability
Type Annotation Fixes:
- Fix globals initialized as None without Optional type across
file_enrichment, document_conversion, and agents global_vars
- Fix StorageMinio return types: upload/upload_file/upload_uploadfile
return str, not uuid.UUID
- Fix MockStorageMinio to match updated StorageMinio return types
- Add explicit type annotations to dict literals in chromekey.py,
pdf/analyzer.py, registry_hive/analyzer.py, and publish_findings.py
- Fix kubeconfig current_context parameter to accept str | None
- Fix container_contents allowed_extensions: set = None -> set | None = None
- Fix file subscription file_queue: asyncio.Queue = None -> asyncio.Queue | None = None
- Fix web_api upload_file to wrap object_id in uuid.UUID() for response
None-safety Assertions:
- Add assert statements for asyncpg_pool, tracking_service,
workflow_client, workflow_manager, file_linking_engine, file_queue,
gotenberg_url, and process.stdout across all activity, subscription,
route, and workflow files in file_enrichment and document_conversion
- Add assertions for asyncpg_pool in all chromium processors
- Add assertions for File.from_metadata timestamp/expiration fields
- Add assertion for alerting GQL client session type
Pyright Ignore Annotations:
- Suppress third-party type issues in Dapr workflow/activity APIs,
gRPC subscription imports, ccache/lnk/office_doc attribute access,
and nemesis_dpapi FlagMixin operators
- Add file-level suppression for office2john.py, pdf2john.py,
pe/analyzer.py, and test harness files
Bug Fixes:
- Fix office2john.py format string: bare % filename -> % (filename, stream)
- Fix container analyzer 7z iteration: iterate sz.files list instead
of calling .items()
- Fix file_linking rules_engine: store match result to avoid double call
- Fix logger.exception calls: remove exception object as first arg in
storage.py, cookies.py, enrichments.py, housekeeping/main.py
- Fix document_conversion lifespan: use stack.callback() for sync shutdown
- Fix NoseyParkerOutput fallback: add missing workflow_id field
- Fix regipy hive_type: handle None return from RegistryHive.hive_type
DPAPI Manager:
- Remove unused guid parameter from get_system_credentials across
DpapiManager, NullDpapiManager, and DpapiManagerProtocol
Added Missing Dependencies (file_enrichment_modules):
- pypykatz>=0.6.11, pyarrow>=19.0.1, msoffcrypto-tool>=5.4.2,
oletools>=0.60.2, regipy>=5.2.0, pillow>=11.3.0
* quiet console logs
* feat: lazy file loading with backend range request support
Add offset/length query params to the download endpoint so the frontend
can request partial file content. FileViewer now fetches data on demand
— hex, transform (Strings, etc.), ZIP, SQLite, and image tabs only load
when activated. Text-based content is capped at 10 MB previews.
* fix: transform tabs stuck on "Loading content..."
- Prevent stale WS subscription from overwriting fetched content
- Show retry button when transform content fails to load
- Reset fetch guard on non-OK HTTP responses
* fix: hex tab deferred loading with full file content
* feat: truncation dropdown, spinner overlay, and tab render refactor
- Add truncation dropdown to MonacoContentViewer for files > 10MB,
replacing the old banner alert and hex "Load Hex View" button
- Add spinner overlay on Monaco editor while loading full content
- Wire truncation support to transform tabs (monaco/json types),
enrichment tabs, and text tabs
- Change hex tab to auto-load first 10MB preview with dropdown for
full file instead of requiring manual load
- Default word wrap to off
- Remove "File is too large" warning message
- Refactor ~150-line ternary chain into explicit per-tab render
functions (renderPreviewTab, renderZipTab, renderSqliteTab,
renderTextTab, renderHexTab, renderTabContent) with shared helpers
(renderFullFileContent, getTextContent)
* feat: improve Yara Rules UI
- Compact table rows with tighter padding
- Add X button and Esc key to close editor dialog
- Disable Save button when rule content is unchanged
- Disable Create button with inline warning when rule name already exists
- Default source to "Created manually by <user>" for new rules
- Update source placeholder to "e.g. /yara_rules/custom.yara"
* update gitignore
---------
Co-authored-by: Lee Chagolla-Christensen <lee@localhost>
- Raise requires-python to >=3.13 across all projects/libs; set ruff target to py313
- Fix bare excepts, unused variables/imports, Optional→union syntax, and import ordering
- Refactor chatbot MCP server to async subprocess with health checks and graceful shutdown
- Simplify file-enrichment prod Dockerfile to use uv sync with UV_PROJECT_ENVIRONMENT
- Fix module_loader to use fully qualified analyzer module name
- Make YARA _analyze_yara async and fix rule_manager access in workflow
- Enable LiteLLM healthcheck and add agents→litellm dependency
- Add PE analyzer tests with PyInstaller fixture;
- Update PDF tests for nested metadata
- Suppress third-party DeprecationWarnings, use async-lru for LLM status cache
- Add PHOENIX_ENABLED validation to nemesis-ctl.sh; remove debug pprint in agents
- Add Claude Code managing-packages skill, update CLAUDE.md guidance
* Claude module development skill (drafting, test case generation, end-to-end testing)
* enrichment-module-builder skill triggered by /new-enrichment-module command for rapid file enrichment module development
* Added CLAUDE.md project file
* Added prefetch and ccache enrichment modules (developed by Claude skill)
* Added Proper GitHub issue templates
* Added Cobalt Strike Connector
* Updated cli readme
* Add Cobalt Strike to C2 connectors list in README
Updated README to include Cobalt Strike in C2 connectors.
* Update usage guide to include Cobalt Strike
* Document Cobalt Strike Connector setup and usage
Added Cobalt Strike Connector section with configuration and usage instructions.
* Updated documentation on REST API startup and modified Path to include the file name
* Update Cobalt Strike URL to include port 50443
* adding code 403 to force reauthentication to mathc with the CS response
Lee Chagolla-Christensen