34 Commits

Author SHA1 Message Date
Lee Chagolla-Christensen b5b00ca72a Migrate to uv from poetry (#99)
* initial uv migration

* update github workflows

* use python module not system commands

* remove poetry references

* fix uv in prod images

---------

Co-authored-by: Lee Chagolla-Christensen <lee@localhost>
2026-01-14 15:20:31 -08:00
Lee Chagolla-Christensen ff6447216d update deps 2026-01-12 11:09:02 -08:00
Lee Chagolla-Christensen 2527e38b32 update dep versions (#98) 2026-01-06 16:28:09 -08:00
Pablo A. Zurro c8e469baea Cobalt Strike Connector (#97)
* Added Cobalt Strike Connector

* Updated cli readme

* Add Cobalt Strike to C2 connectors list in README

Updated README to include Cobalt Strike in C2 connectors.

* Update usage guide to include Cobalt Strike

* Document Cobalt Strike Connector setup and usage

Added Cobalt Strike Connector section with configuration and usage instructions.

* Updated documentation on REST API startup and modified Path to include the file name

* Update Cobalt Strike URL to include port 50443

* adding code 403 to force reauthentication to mathc with the CS response
2026-01-06 09:36:09 -08:00
harmj0y 6760542f96 Updates for vulnerable packages
- Redid `poetry lock` to fix some vulnerable package versions
2025-12-05 11:09:53 -08:00
Lee Chagolla-Christensen 922e8e4142 more verbose CLI errors, ignore swp files 2025-11-14 10:23:08 -08:00
Lee Chagolla-Christensen 58dd461532 Refactor document conversion service into Dapr workflow
- Restructure document_conversion to match into workflow structure
- PDF analyzer: use PyMuPDF and extract extensive metadata
- Add PDF test suite with encrypted/unencrypted fixtures
- Add Tika Tesseract OCR configuration support example
- Update Dapr configurations for new document_conversion component
- Remove deprecated helpers from common lib
- Update queue constants and workflow setup
2025-11-06 01:30:10 -08:00
Lee Chagolla-Christensen 6ec0a3b61a Workflow performance improvements (#87)
* upgrade to dapr postgresv2 statestore

* actually make it v2

* dapr state table name, cleanup subscriptions/globals

* proper exceptions

* formatting/lint

* Refactor workflow tracking and improve activity input handling

- Extract workflow tracking logic into dedicated WorkflowTrackingService
- Simplify activity signatures to accept specific parameters instead of generic dicts
- Remove unused asyncio event loop references from enrichment modules
- Update YaraRuleManager initialization and method names

* re-added workflow tracking in the DB

* update uvicorn prod options

* enrichment work parallelism, convert queues from broadcast to task queues

* Refactor pubsub and improve workflow parallelism

- Split Dapr pubsub Dapr yaml components into topic-specific queues (alerting, dotnet, dpapi, files, noseyparker, workflow_monitor)
- Update all Dapr volume mounts to reference new topic-specific pubsub components
- Converted queues to task queues
- Use YAML anchors to reduce duplication for file-enrichment replicas
- Pass asyncpg pool to enrichment modules instead of creating connections
- Add asyncpg_pool parameter throughout chromium and enrichment module analyzers
- Update VSCode workspace (removed InspectAssembly, renamed dotnet_api to dotnet_service)
- Added curl commands for Jaeger API to performance docs to help with perf troubleshooting
- Created common.queues module to centralize pubsub/topic names (eases
  future refactoring)

* worker mods

* Workflow performance tuning, fix pubsub config, CLI arg changes

- Fix pubsub deleteWhenUnused typo (deletedWhenUnused)
- Add LOG_LEVEL environment variable support across services
- CLI: Rename --repeat to --times, add --max-files option
- Increase files pubsub prefetchCount from 25 to 50
- Add MAX_PARALLEL_WORKFLOWS configuration
- Fix DotNetAssemblyAnalysis null handling with field validators
- Update dashboard to show cumulative files/findings over time
- Add RUST_LOG environment variable support to noseyparker
- Update CHANGELOG for 2.1.4 release notes

* Dapr 1.16.2 and use db transactions

- Upgrade all Dapr containers from 1.16.1 to 1.16.2
- Reduce enrichment parallelism default from 25 to 5 workflows
- Reduce healthcheck intervals from 10s to 5s for alerting and document conversion
- Fixed DPAPI eventing to use new pubsubs
- Refactor file_linking database operations to use atomic upserts and avoid deadlocks
- Add WriteOnceViolationError handling in DPAPI masterkey analyzer
- Wrap database operations in transactions for enrichment storage and plaintext indexing
- Fix postgres notification handler closure variable capture

* remove unused start_time

* Scheduler persistence, workflow concurrency tuning, and config cleanup

- Add volume for Dapr scheduler and init service
- Add scheduler dependency to file enrichment service
- Add async workflow client libraries
- Format and cleanup compose.yaml (spacing, indentation, empty lines)

* Migrate file_enrichment to async Dapr client and optimize Dockerfile

- Use async DaprClient where possible in file_enrichment
- Improve Dockerfile caching
- Add asyncpg connection pool helper and fix typo in secret store name
- Include VS Code debug configuration for document_conversion
- Remove unused dapr_client from DpapiBlobAnalyzer
- Clean up activity return types and better handle exceptions

* Enrichment tracking for NoseyParker and logging cleanup

- Add workflow_id to NoseyParkerInput and NoseyParkerOutput models
- Remove workflow lookup query in noseyparker subscription handler
- Adjust jaeger_perf_stats.sh output formatting and precision
- Add type hints for async functions

* noseyparker scanner perf, tracing for update_enrichment_results

---------

Co-authored-by: Lee Chagolla-Christensen <lee@localhost>
2025-11-04 14:06:59 -08:00
Will Schroeder d4049fe199 Dependabot updates 03 11 25 (#84)
* starlette fixes

starlette fixes

* pypdf fixes

pypdf fixes

* bump urllib3

bump urllib3

* vite bump

vite bump

* Bump requests

Bump requests

* Replaced pypdf2 with pypdf

-Replaced pypdf2 with pypdf
2025-11-03 10:42:51 -08:00
Lee Chagolla-Christensen 91a3ce12d2 improve submit pooling/stats, remove bucket checks 2025-10-24 15:47:58 -07:00
Lee Chagolla-Christensen 2e48a0d3a5 ruff - auto fixes 2025-10-17 14:29:45 -07:00
Lee Chagolla-Christensen fa918973ba make SQL strings private, lint warnings, dead code 2025-10-17 14:20:55 -07:00
Lee Chagolla-Christensen 4c70844fdd linting 2025-10-16 18:17:10 -07:00
Lee Chagolla-Christensen 4cc62007cc type warnings 2025-10-16 18:15:08 -07:00
Lee Chagolla-Christensen 5bdf80b2aa update CLI to support explicitly empty --folder 2025-10-16 18:13:49 -07:00
Lee Chagolla-Christensen b63813e300 Standardize path handling and centralize database connection management
- Add centralized PostgreSQL connection string helper in common/db.py
- Replace ntpath with posixpath
- Add get_drive_from_path() helper for extracting drive letters from paths
- Update all modules to use centralized DB connection helper instead of individual Dapr client calls
- Improve path normalization across Chromium, DPAPI, and file enrichment modules
- Added dependencies to local libraries
- Expand file linking tests with comprehensive rule engine test cases
- Refactor masterkey path construction to use posixpath.normpath for proper path joining
- Update state helpers and API models to use centralized connection
- Updated web API to use strongly typed models instead of dicts
  internally
- Improve DPAPI core functionality with better error handling and path normalization
2025-10-15 13:48:50 -07:00
Lee Chagolla-Christensen 6099a6515b update dapr to 1.16.1 2025-10-10 17:32:12 -07:00
Lee Chagolla-Christensen a22218fcef fix ruff/tests in project, install venv script, local state link to user masterkey, masterkey type from path 2025-10-10 14:16:52 -07:00
Lee Chagolla-Christensen 2baa7d5d6a add CLI --folder option, linkings foreign key, linkage UI bug, plaintext checks in analyzers 2025-10-02 23:02:36 -07:00
Lee Chagolla-Christensen 3a144b1426 update cli shorthands 2025-09-30 10:00:58 -07:00
harmj0y 8034abcedb Update README.md 2025-08-29 16:09:16 -07:00
harmj0y 40902d9d8f Version 2.1.3
- See CHANGELOG.md for summary of changes
2025-08-29 15:24:32 -07:00
Lee Chagolla-Christensen 2374c6b1d1 allow pass or token mythic creds 2025-08-22 10:03:58 -07:00
harmj0y 8a5441a21e Don't exit when cli folder monitoring starts on an empty folder
-Don't exit when `cli` folder monitoring starts on an empty folder
2025-07-01 11:59:50 -07:00
Lee Chagolla-Christensen 46269a9eea update CLI readme docs 2025-06-26 18:11:04 -07:00
Lee Chagolla-Christensen 60d57379a6 update helper scripts + cli.md 2025-06-26 17:52:47 -07:00
harmj0y c88917263a Update Dockerfile 2025-06-26 15:50:51 -07:00
Lee Chagolla-Christensen a2796a940b nemesis-ctl, fix cli compose 2025-06-26 12:29:48 -07:00
Lee Chagolla-Christensen 166878f4e9 formatting 2025-06-25 02:19:25 -07:00
Lee Chagolla-Christensen 4fc163a31f docker compose updates 2025-06-24 18:04:52 -07:00
Lee Chagolla-Christensen 2ee9a37061 add project to mythic connector 2025-06-19 06:07:49 -07:00
Lee Chagolla-Christensen ff95494215 log mythic url in errors 2025-06-16 14:24:49 -07:00
Lee Chagolla-Christensen b796025e8d fix aiohttp/protbuf dependabot complaints 2025-06-16 13:04:45 -07:00
Lee Christensen 8035a6a168 add code 2025-06-13 11:33:07 +02:00