`MergeMaps` accounts for 11.41% of allocs (13.8
GB) in clusterbomb mode. With 1,305 combinations
per target, this function is called millions of
times in the hot path.
RCA:
* Request generator calls `MergeMaps` with single
arg on every payload combination, incurring
variadic overhead.
* Build request merges same maps multiple times
per request.
* `BuildPayloadFromOptions` recomputes static CLI
options on every call.
* Variables calls `MergeMaps` $$2×N$$ times per
variable evaluation (once in loop, once in
`evaluateVariableValue`)
Changes:
Core optimizations in maps.go:
* Pre-size merged map to avoid rehashing (30-40%
reduction)
* Add `CopyMap` for efficient single-map copy
without variadic overhead.
* Add `MergeMapsInto` for in-place mutation when
caller owns destination.
Hot path fixes:
* Replace `MergeMaps(r.currentPayloads)` with
`CopyMap(r.currentPayloads)` to eliminates
allocation on every combination iteration.
* Pre-allocate combined map once, extend in-place
during `ForEach` loop instead of creating new
map per variable (eliminates $$2×N$$ allocations
per request).
Caching with concurrency safety:
* Cache `BuildPayloadFromOptions` computation in
`sync.Map` keyed by `types.Options` ptr, but
return copy to prevent concurrent modification.
* Cost: shallow copy of ~10-20 entries vs. full
merge of vars + env (85-90% savings in typical
case)
* Clear cache in `closeInternal()` to prevent
memory leaks when SDK instances are created or
destroyed.
Estimated impact: 40-60% reduction in `MergeMaps`
allocations (5.5-8.3 GB savings from original
13.8 GB). Safe for concurrent execution and SDK
usage with multiple instances.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* Enhance matcher compilation with caching for regex and DSL expressions to improve performance. Update template parsing to conditionally retain raw templates based on size constraints.
* Implement caching for regex and DSL expressions in extractors and matchers to enhance performance. Introduce a buffer pool in raw requests to reduce memory allocations. Update template cache management for improved efficiency.
* feat: improve concurrency to be bound
* refactor: replace fmt.Sprintf with fmt.Fprintf for improved performance in header handling
* feat: add regex matching tests and benchmarks for performance evaluation
* feat: add prefix check in regex extraction to optimize matching process
* feat: implement regex caching mechanism to enhance performance in extractors and matchers, along with tests and benchmarks for validation
* feat: add unit tests for template execution in the core engine, enhancing test coverage and reliability
* feat: enhance error handling in template execution and improve regex caching logic for better performance
* Implement caching for regex and DSL expressions in the cache package, replacing previous sync.Map usage. Add unit tests for cache functionality, including eviction by capacity and retrieval of cached items. Update extractors and matchers to utilize the new cache system for improved performance and memory efficiency.
* Add tests for SetCapacities in cache package to ensure cache behavior on capacity changes
- Implemented TestSetCapacities_NoRebuildOnZero to verify that setting capacities to zero does not clear existing caches.
- Added TestSetCapacities_BeforeFirstUse to confirm that initial cache settings are respected and not overridden by subsequent capacity changes.
* Refactor matchers and update load test generator to use io package
- Removed maxRegexScanBytes constant from match.go.
- Replaced ioutil with io package in load_test.go for NopCloser usage.
- Restored TestValidate_AllowsInlineMultiline in load_test.go to ensure inline validation functionality.
* Add cancellation support in template execution and enhance test coverage
- Updated executeTemplateWithTargets to respect context cancellation.
- Introduced fakeTargetProvider and slowExecuter for testing.
- Added Test_executeTemplateWithTargets_RespectsCancellation to validate cancellation behavior during template execution.
* misc update
* chore(deps): bump github.com/gin-gonic/gin from 1.9.0 to 1.9.1 (#4252)
Bumps [github.com/gin-gonic/gin](https://github.com/gin-gonic/gin) from 1.9.0 to 1.9.1.
- [Release notes](https://github.com/gin-gonic/gin/releases)
- [Changelog](https://github.com/gin-gonic/gin/blob/master/CHANGELOG.md)
- [Commits](https://github.com/gin-gonic/gin/compare/v1.9.0...v1.9.1)
---
updated-dependencies:
- dependency-name: github.com/gin-gonic/gin
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(deps): bump github.com/docker/docker (#4316)
Bumps [github.com/docker/docker](https://github.com/docker/docker) from 24.0.5+incompatible to 24.0.7+incompatible.
- [Release notes](https://github.com/docker/docker/releases)
- [Commits](https://github.com/docker/docker/compare/v24.0.5...v24.0.7)
---
updated-dependencies:
- dependency-name: github.com/docker/docker
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix README_CN.md typos (#4369)
* version update
* Add more support for `fs.FS` in the disk catalog
This adds more support for `fs.FS` in the disk catalog. This
fixes some places where direct `os` file-related calls were being
made to use the catalog interface instead.
Note that the JavaScript compiler *still* does not work in any
context where the `pkg/js/libs/fs` package is used. In particular,
the `ReadFilesFromDir` function is hard-coded to use the `os`
package and not respect the catalog.
* Remove some testing artifacts
* Wrap up
* Unwind other changes
* Add a LoadHelperFileFunction to Options
* Use a direct func
* Tweak validation
* Use a function type
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xc1Ym <xuedongyuming2233@gmail.com>
Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
* feat: added fuzzing output enhancements
* changes as requested
* misc
* feat: added dfp flag to display fuzz points + misc additions
* feat: added support for fuzzing nested path segments
* feat: added parts to fuzzing requests
* feat: added tracking for parameter occurence frequency in fuzzing
* added cli flag for fuzz frequency
* fixed broken tests
* fixed path based sqli integration test
* feat: added configurable fuzzing aggression level for payloads
* fixed failing test
Dwi Siswanto