* feat(filepath): add IsPathWithinAnyDirectory func
Signed-off-by: Dwi Siswanto <git@dw1.io>
* refactor: use path-aware filesystem containment checks
Several filesystem trust boundaries relied on
lexical prefix checks to decide whether a path
fell under an allowed directory. That let sibling
paths be treated as if they were children of
trusted directories.
Replace those checks with canonical path
containment checks for helper payload loading,
template archive extraction, custom template
metadata, template path classification, and
headless screenshot output validation.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* adding more tests
* fixing tests
* adding test
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
Bumps clistats, fastdialer, retryablehttp-go, dsl, networkpolicy,
ratelimit, sarif, utils, wappalyzergo, cdncheck. dsl v0.8.18 moves to
the projectdiscovery/govaluate fork, and utils v0.11.0 pulls aurora v4,
so the corresponding imports and type references are migrated.
Refs #7380.
Uses metadata caching via catalog index to reduce
runtime overhead.
Add `(*Store).LoadTemplateTags` for efficient tag
aggregation.
Signed-off-by: Dwi Siswanto <git@dw1.io>
Prev, the template exclusion logic checked if the
full file path contained any of the known
miscellaneous directory names (e.g., helpers,
.git). This caused false positives when valid
templates were stored in paths where a parent
directory matched one of these names (e.g.,
/path/to/somewhere/that/has/helpers/dir/name/).
This commit introduces `IsTemplateWithRoot`, which
checks for excluded directories relative to a
provided root directory. It splits the relative
path into components.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* feat(loader): implement persistent metadata cache
for template filtering optimization.
Introduce a new template metadata indexing system
with persistent caching to dramatically improve
template loading perf when filters are applied.
The implementation adds a new index pkg that
caches lightweight template metadata (ID, tags,
authors, severity, .etc) and enables filtering
templates before expensive YAML parsing occurs.
The index uses an in-memory LRU cache backed by
`otter` pkg for efficient memory management with
adaptive sizing based on entry weight, defaulting
to approx. 40MB for 50K templates.
Metadata is persisted to disk using gob encoding
at "~/.cache/nuclei/index.gob" with atomic writes
to prevent corruption. The cache automatically
invalidates stale entries using `ModTime` to
detect file modifications, ensuring metadata
freshness w/o manual intervention.
Filtering has been refactored from the previous
`TagFilter` and `PathFilter` approach into a
unified `index.Filter` type that handles all basic
filtering ops including severity, authors, tags,
template IDs with wildcard support, protocol
types, and path-based inclusion and exclusion. The
filter implements OR logic within each field type
and AND logic across different field types, with
exclusion filters taking precedence over inclusion
filters and forced inclusion via
`IncludeTemplates` and `IncludeTags` overriding
exclusions.
The `loader` integration creates an index filter
from store configuration via `buildIndexFilter`
and manages the cache lifecycle through
`loadTemplatesIndex` and `saveTemplatesIndex`
methods. When `LoadTemplatesOnlyMetadata` or
`LoadTemplatesWithTags` is called, the system
first checks the metadata cache for each template
path. If cached metadata exists and passes
validation, the filter is applied directly against
the metadata without parsing. Only templates
matching the filter criteria proceed to full YAML
parsing, resulting in significant performance
gains.
Advanced filtering via "-tc" flag
(`IncludeConditions`) still requires template
parsing as these are expression-based filters that
cannot be evaluated from metadata alone. The
`TagFilter` has been simplified to handle only
`IncludeConditions` while all other filtering ops
are delegated to the index-based filtering system.
Cache management is fully automatic with no user
configuration required. The cache gracefully
handles errors by logging warnings & falling back
to normal op w/o caching. Cache files use schema
versioning to invalidate incompatible cache
formats across nuclei updates (well, specifically
`Index` and `Metadata` changes).
This optimization particularly benefits repeated
scans with the same filters, CI/CD pipelines
running nuclei regularly, development and testing
workflows with frequent template loading, and any
scenario with large template collections where
filtering would exclude most templates.
* test(loader): adds `BenchmarkLoadTemplates{,OnlyMetadata}` benchs
Signed-off-by: Dwi Siswanto <git@dw1.io>
* ci: cache nuclei-templates index
Signed-off-by: Dwi Siswanto <git@dw1.io>
* chore(index): satisfy lints
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(index): correct metadata filter logic
for proper template matching.
The `filter.matchesIncludes()` was using OR logic
across different filter types, causing incorrect
template matching. Additionally, ID matching was
case-sensitive, failing to match patterns like
'CVE-2021-*'.
The filter now correctly implements: (author1 OR
author2) AND (tag1 OR tag2) AND (severity1 OR
severity2) - using OR within each filter type and
AND across different types.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test(index): resolve test timing issue
in CI environments.
Some test was failing in CI due to filesystem
timestamp resolution limitations. On filesystems
with 1s ModTime granularity (common in CI),
modifying a file immediately after capturing its
timestamp resulted in identical ModTime values,
causing IsValid() to incorrectly return true.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* ci: cache nuclei with composite action
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(index): file locking issue on Windows
during cache save/load.
Explicitly close file handles before performing
rename/remove ops in `Save` and `Load` methods.
* In `Save`, close temp file before rename.
* In `Load`, close file before remove during error
handling/version mismatch.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test(index): flaky index tests on Windows
Fix path separator mismatch in `TestCacheSize`
and `TestCachePersistenceWithLargeDataset` by
using `filepath.Join` consistently instead of
hardcoded forward slashes.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test(cmd): init logger to prevent nil pointer deref
The integration tests were panicking with a nil
pointer dereference in `pkg/catalog/loader`
because the logger was not init'ed.
When `store.saveMetadataIndexOnce` attempted to
log the result of the metadata cache op, it
dereferenced the nil logger, causing a crash.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(loader): resolve include/exclude paths
for metadata cache filter.
The `indexFilter` was previously init'ed using raw
relative paths from the config for
`IncludeTemplates` and `ExcludeTemplates`.
But the persistent metadata cache stores templates
using their absolute paths. This mismatch caused
the `matchesPath` check to fail, leading to
templates being incorrectly excluded even when
explicitly included via flags
(e.g., "-include-templates
loader/excluded-template.yaml").
This commit updates `buildIndexFilter` to resolve
these paths to their absolute versions using
`store.config.Catalog.GetTemplatesPath` before
creating the filter, ensuring consistent path
matching against the metadata cache.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* feat(index): adds `NewMetadataFromTemplate` func
Signed-off-by: Dwi Siswanto <git@dw1.io>
* refactor(index): return metadata when `(*Index).cache` is nil
Signed-off-by: Dwi Siswanto <git@dw1.io>
* refactor(loader): restore pre‑index behavior semantics
Signed-off-by: Dwi Siswanto <git@dw1.io>
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
* perf(loader): reuse cached parsed templates
in `(*Store).areWorkflowOrTemplatesValid`, which
is being called during template `-validate`-ion.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* refactor(testutils): optionally assign template info
in `NewMockExecuterOptions`, which is not
required for specific case, like when we want to
`(*Store).ValidateTemplates`.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test(loader): adds `(*Store).ValidateTemplates` bench
Signed-off-by: Dwi Siswanto <git@dw1.io>
* refactor(templates): adds fast read parser
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test(templates): adds `Parser*` benchs
Signed-off-by: Dwi Siswanto <git@dw1.io>
* chore(templates): satisfy lints
Signed-off-by: Dwi Siswanto <git@dw1.io>
* revert(templates): rm fast read parser
Signed-off-by: Dwi Siswanto <git@dw1.io>
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
* normalize file `fpath` in `IsTemplate` using
filepath.FromSlash to ensure consistent matching
across platforms.
* update `GetKnownMiscDirectories` docs to clarify
that trailing slashes prevent false positives,
since `IsTemplate` compares against normalized
full paths.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* remove the stack trace when the nuclei-ignore file does not exist
* removing useless debug stack
---------
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
Dwi Siswanto