* fix(js): prevent pool slot starvation under load
Zombie goroutines from timed-out JS executions held pool slots
indefinitely: Add() blocked with context.Background(), and defer Done()
only ran when the goroutine eventually completed. Under load, both
pools (80 pooled + 20 non-pooled slots) filled with zombies, silently
dropping all subsequent matches.
Three changes fix slot lifecycle management:
1. Propagate the 20s deadline context into ExecuteProgram (compiler.go)
so both execution paths can respect the deadline.
2. Replace Add() with AddWithContext(ctx) in both pool.go and
non-pool.go so goroutines waiting for a slot fail fast when the
deadline expires instead of blocking indefinitely.
3. Add a watchdog goroutine that releases the slot when the deadline
expires, even if the zombie is still running. An atomic.Bool
ensures exactly one Done() call between the watchdog and the
normal defer path.
* adding context
* refactor(js): derived the ctx to remaining tractable deadline leaks (#7302)
* refactor(js): use `context.Background` as default instead
Signed-off-by: Dwi Siswanto <git@dw1.io>
* refactor(js): derived the ctx to remaining tractable deadline leaks
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test(js): add `NucleiJS.Context` tests
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(cmd): context param exclusion in memoization hash
The memoization template condition for excluding
context parameters from hash keys was incorrect.
The memoize package represents context.Context
types as "&{context Context}" (AST string
representation), not "context.Context".
Signed-off-by: Dwi Siswanto <git@dw1.io>
* chore(js): memogen'ed
Signed-off-by: Dwi Siswanto <git@dw1.io>
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(js): hangs in checkRDPEncryption
by bounding socket I/O
Add `setConnDeadlineFromContext` helper to set
deadlines on conns derived from context timeouts.
Move conn cleanup out of loop-scoped defers to
make sure immediate cleanup per probe attempt.
Signed-off-by: Dwi Siswanto <git@dw1.io>
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
Co-authored-by: Niek den Breeje <AuditeMarlow@users.noreply.github.com>
Co-authored-by: Dwi Siswanto <25837540+dwisiswant0@users.noreply.github.com>
Co-authored-by: Dwi Siswanto <git@dw1.io>
- Update -omit-raw help text to include PDF in scope
- Move require.NoError out of goroutine in concurrency test (FailNow semantics)
- Truncate raw request/response at buffer time in Export() to prevent memory accumulation
* fix(expressions): only eval template-authored expressions
`expressions.Evaluate()` currently replaces
placeholders first and then scans the substituted
output for expressions which allows
response-derived values, including extractor
output, to be reinterpreted as DSL/helper syntax
on a second pass, which then becomes more serious
when env-backed variables are enabled.
So making this by collecting expressions from the
original template text before placeholder
substitution and then evaluating only those
original expressions after normal replacement to
keep trusted template-authored helpers working,
including placeholders nested inside helper
arguments while treating substituted response data
as literal text.
Fixes#7210
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test: add resp data literal reuse test
Signed-off-by: Dwi Siswanto <git@dw1.io>
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
Resume files are runtime artifacts and SHOULD NOT
be presented as persistent config.
Update reset messaging to separate config vs cache
and explicitly include resume state under cache
cleanup. Also delete the cache dir during `-reset`
to keep behavior aligned with the new semantics.
Close#6792
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(http): interactsh matching with `payloads`
in parallel execution.
Templates using `payloads` with Interactsh
matchers failed to detect OAST interactions
because the parallel HTTP execution path (used
when `payloads` are present) did not register
Interactsh request events, unlike the seq path.
This caused incoming interactions to lack
associated request context, preventing matchers
from running and resulting in missed detections.
Fix#5485 by wiring
`(*interactsh.Client).RequestEvent` registration
into the parallel worker goroutine, make sure both
execution paths handle Interactsh correlation
equally.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test: add interactsh with `payloads` integration
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test: disable interactsh-with-payloads
Signed-off-by: Dwi Siswanto <git@dw1.io>
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
* adding min auth support
* adding unauth list modules + auth list files in module
* example
* adding rsync test
* bump go.mod
---------
Co-authored-by: Dwi Siswanto <git@dw1.io>
Make sure postgres Exec/ExecContext are invoked with the correct
argument order, preventing context from being passed as the query.
* fixing pg syntax
* adding test
* feat: add site-url to optionally provide jira server URL for oauth
* chore(cmd): add `site-url` config option
Adds optional `site-url` field to JIRA issue
tracker configuration for specifying browsable URL
when it differs from the API endpoint. This is
particularly useful for OAuth-based JIRA Cloud
integrations where `issue.Self` contains
"api.atlassian.com" instead of the user-facing
domain.
Signed-off-by: Dwi Siswanto <git@dw1.io>
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
Co-authored-by: Dwi Siswanto <git@dw1.io>
with configurable limits
This patch fixes duplicate issue detection for
GitLab trackers by implementing paginated search
with configurable page size and max pages. Adds
`duplicate-issue-page-size` and
`duplicate-issue-max-pages` options to the config.
Fixes#6711.
Signed-off-by: Dwi Siswanto <git@dw1.io>
Restore backwards compat for JavaScript protocol
templates that omit the `Port` argument.
Regression was introduced in f4f2e9f2, which
removed the fallback for empty `Port` in
`(*Request).ExecuteWithResults`, causing templates
without `Port` to be silently skipped.
Now, if no `Port` is specified, the engine
executes the JavaScript block using the target
URL's port.
Fixes#6708.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(http): pass `dynamicValues` to `EvaluateWithInteractsh`
When `LazyEval` is true (triggered by `variables`
containing `BaseURL`, `Hostname`,
`interactsh-url`, etc.), variable expressions are not
eval'ed during YAML parsing & remain as raw exprs
like "{{rand_base(5)}}".
At request build time, `EvaluateWithInteractsh()`
checks if a variable already has a value in the
passed map before re-evaluating its expression.
But, `dynamicValues` (which contains the template
context with previously eval'ed values) was not
being passed, causing exprs like `rand_*` to be
re-evaluated on each request, producing different
values.
Fixes#6684 by including `dynamicValues` in the
map passed to `EvaluateWithInteractsh()`, so
variables evaluated in earlier requests retain
their values in subsequent requests.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* chore(http): rm early eval in `(*Request).ExecuteWithResults()`
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test: adds variables-threads-previous integration test
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test: adds constants-with-threads integration test
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test: adds race-with-variables integration test
Signed-off-by: Dwi Siswanto <git@dw1.io>
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
* feat(loader): implement persistent metadata cache
for template filtering optimization.
Introduce a new template metadata indexing system
with persistent caching to dramatically improve
template loading perf when filters are applied.
The implementation adds a new index pkg that
caches lightweight template metadata (ID, tags,
authors, severity, .etc) and enables filtering
templates before expensive YAML parsing occurs.
The index uses an in-memory LRU cache backed by
`otter` pkg for efficient memory management with
adaptive sizing based on entry weight, defaulting
to approx. 40MB for 50K templates.
Metadata is persisted to disk using gob encoding
at "~/.cache/nuclei/index.gob" with atomic writes
to prevent corruption. The cache automatically
invalidates stale entries using `ModTime` to
detect file modifications, ensuring metadata
freshness w/o manual intervention.
Filtering has been refactored from the previous
`TagFilter` and `PathFilter` approach into a
unified `index.Filter` type that handles all basic
filtering ops including severity, authors, tags,
template IDs with wildcard support, protocol
types, and path-based inclusion and exclusion. The
filter implements OR logic within each field type
and AND logic across different field types, with
exclusion filters taking precedence over inclusion
filters and forced inclusion via
`IncludeTemplates` and `IncludeTags` overriding
exclusions.
The `loader` integration creates an index filter
from store configuration via `buildIndexFilter`
and manages the cache lifecycle through
`loadTemplatesIndex` and `saveTemplatesIndex`
methods. When `LoadTemplatesOnlyMetadata` or
`LoadTemplatesWithTags` is called, the system
first checks the metadata cache for each template
path. If cached metadata exists and passes
validation, the filter is applied directly against
the metadata without parsing. Only templates
matching the filter criteria proceed to full YAML
parsing, resulting in significant performance
gains.
Advanced filtering via "-tc" flag
(`IncludeConditions`) still requires template
parsing as these are expression-based filters that
cannot be evaluated from metadata alone. The
`TagFilter` has been simplified to handle only
`IncludeConditions` while all other filtering ops
are delegated to the index-based filtering system.
Cache management is fully automatic with no user
configuration required. The cache gracefully
handles errors by logging warnings & falling back
to normal op w/o caching. Cache files use schema
versioning to invalidate incompatible cache
formats across nuclei updates (well, specifically
`Index` and `Metadata` changes).
This optimization particularly benefits repeated
scans with the same filters, CI/CD pipelines
running nuclei regularly, development and testing
workflows with frequent template loading, and any
scenario with large template collections where
filtering would exclude most templates.
* test(loader): adds `BenchmarkLoadTemplates{,OnlyMetadata}` benchs
Signed-off-by: Dwi Siswanto <git@dw1.io>
* ci: cache nuclei-templates index
Signed-off-by: Dwi Siswanto <git@dw1.io>
* chore(index): satisfy lints
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(index): correct metadata filter logic
for proper template matching.
The `filter.matchesIncludes()` was using OR logic
across different filter types, causing incorrect
template matching. Additionally, ID matching was
case-sensitive, failing to match patterns like
'CVE-2021-*'.
The filter now correctly implements: (author1 OR
author2) AND (tag1 OR tag2) AND (severity1 OR
severity2) - using OR within each filter type and
AND across different types.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test(index): resolve test timing issue
in CI environments.
Some test was failing in CI due to filesystem
timestamp resolution limitations. On filesystems
with 1s ModTime granularity (common in CI),
modifying a file immediately after capturing its
timestamp resulted in identical ModTime values,
causing IsValid() to incorrectly return true.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* ci: cache nuclei with composite action
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(index): file locking issue on Windows
during cache save/load.
Explicitly close file handles before performing
rename/remove ops in `Save` and `Load` methods.
* In `Save`, close temp file before rename.
* In `Load`, close file before remove during error
handling/version mismatch.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test(index): flaky index tests on Windows
Fix path separator mismatch in `TestCacheSize`
and `TestCachePersistenceWithLargeDataset` by
using `filepath.Join` consistently instead of
hardcoded forward slashes.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test(cmd): init logger to prevent nil pointer deref
The integration tests were panicking with a nil
pointer dereference in `pkg/catalog/loader`
because the logger was not init'ed.
When `store.saveMetadataIndexOnce` attempted to
log the result of the metadata cache op, it
dereferenced the nil logger, causing a crash.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(loader): resolve include/exclude paths
for metadata cache filter.
The `indexFilter` was previously init'ed using raw
relative paths from the config for
`IncludeTemplates` and `ExcludeTemplates`.
But the persistent metadata cache stores templates
using their absolute paths. This mismatch caused
the `matchesPath` check to fail, leading to
templates being incorrectly excluded even when
explicitly included via flags
(e.g., "-include-templates
loader/excluded-template.yaml").
This commit updates `buildIndexFilter` to resolve
these paths to their absolute versions using
`store.config.Catalog.GetTemplatesPath` before
creating the filter, ensuring consistent path
matching against the metadata cache.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* feat(index): adds `NewMetadataFromTemplate` func
Signed-off-by: Dwi Siswanto <git@dw1.io>
* refactor(index): return metadata when `(*Index).cache` is nil
Signed-off-by: Dwi Siswanto <git@dw1.io>
* refactor(loader): restore pre‑index behavior semantics
Signed-off-by: Dwi Siswanto <git@dw1.io>
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
* replace hardcoded `DEBUG` env var check with
extensible helper func.
* add support for GitHub Actions Runner env var.
* accept multiple truthy value variants.
Signed-off-by: Dwi Siswanto <git@dw1.io>
Mzack9999