mirror of
https://github.com/iimp0ster/detection-chokepoints
synced 2026-08-09 12:41:00 +00:00
Claude
f859bbd3d4
Apply three coding standards across all 6 enrichment pipeline scripts: pathlib - Replace all os.path.join/exists/dirname/makedirs with Path / operator, path.exists(), path.mkdir(), path.read_text(), path.write_text() httpx + async - Replace requests.Session with httpx.AsyncClient throughout - Drop the hand-rolled blocking RateLimiter class in enrich_infra.py; replace with _VTThrottle (async lock + sliding window, same semantics) - Parallelize: hostname enrichment (asyncio.gather + Semaphore), favicon fetches (configurable FAVICON_CONCURRENCY), HA IOC lookups (Semaphore of 6), LFS file downloads (asyncio.gather per file), sandbox poll loops (concurrent per-job coroutines) - All main() functions are now async; entry point is asyncio.run(main()) .env - Add .env.example with all tunable constants (URLs, timeouts, rate limits, concurrency, environment IDs); values load via python-dotenv with sane defaults so scripts work without a local .env file - Add .env to .gitignore - Add httpx python-dotenv to workflow pip install line https://claude.ai/code/session_01A9ipwynqdYKxToTJPMW8uf
39 lines
1.9 KiB
Bash
39 lines
1.9 KiB
Bash
# Detection Chokepoints — Enrichment Pipeline Configuration
|
|
#
|
|
# Copy this file to .env and fill in your API keys for local development.
|
|
# In GitHub Actions, these are injected as repository secrets/variables.
|
|
# .env is gitignored and never committed.
|
|
|
|
# ── GitHub (ClickGrab LFS access) ─────────────────────────────────────────
|
|
GITHUB_TOKEN=
|
|
|
|
# ── ClickGrab ingest ──────────────────────────────────────────────────────
|
|
CLICKGRAB_LOOKBACK_DAYS=7
|
|
CLICKGRAB_REQUEST_TIMEOUT=30
|
|
|
|
# ── IPinfo ────────────────────────────────────────────────────────────────
|
|
IPINFO_TOKEN=
|
|
IPINFO_URL=https://ipinfo.io/{ip}/json
|
|
IPINFO_SLEEP=0.05
|
|
|
|
# ── VirusTotal ────────────────────────────────────────────────────────────
|
|
VT_API_KEY=
|
|
VT_DOMAIN_URL=https://www.virustotal.com/api/v3/domains/{domain}
|
|
VT_IP_URL=https://www.virustotal.com/api/v3/ip_addresses/{ip}
|
|
VT_RATE_PER_MIN=4
|
|
VT_DAILY_BUDGET=500
|
|
REQUEST_TIMEOUT=10
|
|
|
|
# ── Hybrid Analysis ───────────────────────────────────────────────────────
|
|
HA_API_KEY=
|
|
HA_BASE_URL=https://www.hybrid-analysis.com/api/v2
|
|
HA_REQUEST_DELAY=0.8
|
|
HA_MAX_RESULTS_PER_TERM=5
|
|
HA_ENVIRONMENT_ID=160
|
|
HA_POLL_INTERVAL=30
|
|
HA_POLL_TIMEOUT=900
|
|
|
|
# ── Campaign clustering ───────────────────────────────────────────────────
|
|
FAVICON_TIMEOUT=5
|
|
FAVICON_CONCURRENCY=10
|