Bumps clistats, fastdialer, retryablehttp-go, dsl, networkpolicy,
ratelimit, sarif, utils, wappalyzergo, cdncheck. dsl v0.8.18 moves to
the projectdiscovery/govaluate fork, and utils v0.11.0 pulls aurora v4,
so the corresponding imports and type references are migrated.
Refs #7380.
WithConfigFile/Bytes now decode YAML directly into a typed RuntimeConfig
struct and apply it via MergeOptions(opts). The goflags-backed reflection
overlay had too much surface area (266-line flag inventory duplicate plus
a flag-default vs DefaultOptions diff) for the actual SDK use case, which
is accepting a known set of scan knobs from cloud-shipped YAML.
RuntimeConfig schema covers tags/severity filtering, headers/vars,
interactsh, socks5-proxy, plus rate-limit/bulk-size/concurrency/timeout/
retries/rate-limit-host. Scalar knobs use *int so omitted keys preserve
the engine's existing value.
Drops internal/runner/flags.go (266 lines) and lib/config_load.go (86
lines). Reverts internal/runner/runner.go to dev (no SetupPDCPUpload
export; SDK inlines its own PDCP wrap). Keeps the LoadReportingOptionsFromBytes
extract in internal/runner/options.go for WithReportingConfigFile/Bytes.
DisableUpdateCheck() mutates DefaultConfig process-globally. config_test.go
files run alphabetically before sdk_test.go, so my tests were disabling the
update-check globally and TestContextCancelNucleiEngine (which depends on
template auto-install) was hitting an empty template store on first run.
Drop the option from the seven test sites. First test in the binary now
triggers template install (sync.Once gated; subsequent tests free).
- L1: sdk_private.go references SetupPDCPUpload (was lowercase).
- L2: config_test.go test comment describes how tmpEngine is constructed
rather than incorrectly claiming "inherits from parent".
- Trim godocs and inline comments across the PR — drop restated WHAT,
keep WHY and limitations.
- Inline (*Runner).setupPDCPUpload — wrapper just delegated to the
package-level SetupPDCPUpload. Call site inlines the three lines.
- Dedupe reporting-config file load: WithReportingConfigFile and
loadImplicitReportingConfig share loadReportingConfigFromPath.
- Move config-overlay/load helpers (newConfigFlagSet, overlayConfigFromFile,
applyOverlay, loadReportingConfigFromPath, loadImplicitReportingConfig)
out of lib/config.go into lib/config_load.go.
- Drop threadSafePerScan engine mode and isThreadSafe helper. The four
new With* options no longer gate per-scan use. pd-agent doesn't enter
thread-safe mode; reintroducing the gate ships as a follow-up PR with
its own scope.
SDK consumers had to rebuild types.Options by hand or shell out to the
CLI to get config-file parity. This shares BindOptionFlags,
LoadReportingOptionsFromBytes, ApplyExporterOptionsFromTypes, and
SetupPDCPUpload between cmd/nuclei and lib/ so both paths produce
identical engine state.
WithConfigFile/Bytes apply only YAML-set fields (reflection diff against
goflags defaults) so prior With* and DefaultOptions() values survive
YAML that omits them. Per-scan ExecuteNucleiWithOpts rejects these
options via a new threadSafePerScan mode, matching the pattern for
options that only make sense at construction.
when creating default limiter.
The engine could be set up with custom rate limits
through `WithOptions()`, but internal was still
hardcoding `e.rateLimiter` to 150 RPS when no
global limiter was explicitly set.
Now it builds the default limiter from the
resolved options:
1. Applies `RateLimitMinute` translation,
2. Defaults `RateLimitDuration` to 1s when
`RateLimit > 0` and no duration is given, then
3. Creates the limiter via `GetRateLimiter()`.
Fixes#7341
Signed-off-by: Dwi Siswanto <git@dw1.io>
This commit fixes a race condition where templates start executing before
the secret file's login flow completes when using -secret-file flag.
Changes:
- pkg/authprovider/authx/dynamic.go: Replace atomic.Bool-based synchronization
with sync.Once to ensure fetch callback runs exactly once and all concurrent
callers block until completion. This prevents the race where multiple goroutines
would see fetching=true and return early with nil error before secrets were ready.
- internal/runner/runner.go: Always prefetch secrets when AuthProvider exists,
removing the PreFetchSecrets flag check. This ensures authentication completes
before scanning starts regardless of flag settings.
- lib/sdk_private.go: Same change as runner.go for SDK mode consistency.
- pkg/authprovider/file_test.go: Add test for concurrent dynamic secret access
to verify fetch happens exactly once and all workers get authenticated.
- pkg/authprovider/authx/dynamic_test.go: Add concurrent fetch tests to verify
proper synchronization behavior.
Fixes#6592
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
`MergeMaps` accounts for 11.41% of allocs (13.8
GB) in clusterbomb mode. With 1,305 combinations
per target, this function is called millions of
times in the hot path.
RCA:
* Request generator calls `MergeMaps` with single
arg on every payload combination, incurring
variadic overhead.
* Build request merges same maps multiple times
per request.
* `BuildPayloadFromOptions` recomputes static CLI
options on every call.
* Variables calls `MergeMaps` $$2×N$$ times per
variable evaluation (once in loop, once in
`evaluateVariableValue`)
Changes:
Core optimizations in maps.go:
* Pre-size merged map to avoid rehashing (30-40%
reduction)
* Add `CopyMap` for efficient single-map copy
without variadic overhead.
* Add `MergeMapsInto` for in-place mutation when
caller owns destination.
Hot path fixes:
* Replace `MergeMaps(r.currentPayloads)` with
`CopyMap(r.currentPayloads)` to eliminates
allocation on every combination iteration.
* Pre-allocate combined map once, extend in-place
during `ForEach` loop instead of creating new
map per variable (eliminates $$2×N$$ allocations
per request).
Caching with concurrency safety:
* Cache `BuildPayloadFromOptions` computation in
`sync.Map` keyed by `types.Options` ptr, but
return copy to prevent concurrent modification.
* Cost: shallow copy of ~10-20 entries vs. full
merge of vars + env (85-90% savings in typical
case)
* Clear cache in `closeInternal()` to prevent
memory leaks when SDK instances are created or
destroyed.
Estimated impact: 40-60% reduction in `MergeMaps`
allocations (5.5-8.3 GB savings from original
13.8 GB). Safe for concurrent execution and SDK
usage with multiple instances.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(lib): segfault when init engine with `EnableHeadlessWithOpts`
The panic was caused by attempting to log a
sandbox warning before the logger was initialized.
RCA:
* SDK option funcs were exec'd before logger init.
* `EnableHeadlessWithOpts()` attempted to create
browser instance & log warnings during the
config phase.
* `Logger` was only init'd later in `init()`
phase.
* This caused nil pointer dereference when
`MustDisableSandbox()` returned true (root on
Linux/Unix or Windows).
Changes:
* Init `Logger` in `types.DefaultOptions()` to
ensure it's always available before any option
functions execute.
* Init `Logger` field in both
`NewNucleiEngineCtx()` and
`NewThreadSafeNucleiEngineCtx()` from
`defaultOptions.Logger`.
* Move browser instance creation from
`EnableHeadlessWithOpts()` to the `init()` phase
where `Logger` is guaranteed to be available.
* Simplify logger sync logic in `init()` to only
update if changed by `WithLogger` option.
* Add test case to verify headless initialization
works without panic.
The fix maintains backward compatibility while
make sure the logger is always available when
needed by any SDK option function.
Fixes#6601.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* build(make): adds `-timeout 30m -count 1` GOFLAGS in `test` cmd
Signed-off-by: Dwi Siswanto <git@dw1.io>
* Revert "fix(lib): segfault when init engine with `EnableHeadlessWithOpts`"
let see if this pass flaky test.
This reverts commit 63fcb6a1cbe7a4db7a78be766affc70eb237e57e.
* test(engine): let see if this pass flaky test
Signed-off-by: Dwi Siswanto <git@dw1.io>
* Revert "Revert "fix(lib): segfault when init engine with `EnableHeadlessWithOpts`""
This reverts commit 62b4223803ccb1e93593e2e08e39923d76aa20b1.
* test(engine): increase `TestActionNavigate` timeout
Signed-off-by: Dwi Siswanto <git@dw1.io>
* Revert "test(engine): let see if this pass flaky test"
This reverts commit d27cd985cff1b06aa1965ea11f8aa32f00778ab5.
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix: remove undefined errorutil.ShowStackTrace
* feat: add make lint support and integrate with test
* refactor: migrate errorutil to errkit across codebase
- Replace deprecated errorutil with modern errkit
- Convert error declarations from var to func for better compatibility
- Fix all SA1019 deprecation warnings
- Maintain error chain support and stack traces
* fix: improve DNS test reliability using Google DNS
- Configure test to use Google DNS (8.8.8.8) for stability
- Fix nil pointer issue in DNS client initialization
- Keep production defaults unchanged
* fixing logic
* removing unwanted branches in makefile
---------
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
Dogan Can Bakir