Two bugs in rate limit handling:
1. Global -rl flag was ignored for sources without a per-source override.
Only sources explicitly listed in -rls got rate-limited; all others
ran unlimited, defeating the purpose of -rl.
2. Per-source duration from -rls (e.g. hackertarget=2/m) was silently
dropped. runner.go stored MaxCount but discarded Duration, so all
sources defaulted to per-second regardless of the /m or /s suffix.
Root cause: runner.go converted goflags.RateLimitMap → CustomRateLimit
but only copied MaxCount. buildMultiRateLimiter only checked per-source
limits, never falling back to the global rate.
Changes:
- Add CustomDuration field to CustomRateLimit struct
- Store both MaxCount and Duration from -rls in runner.go
- Add resolveSourceRateLimit() with clear priority:
per-source > global > unlimited
- Guard against nil rateLimit in buildMultiRateLimiter
- Add 9 unit tests covering all combinations
Fixes#1434
Add URLScan.io as a new passive subdomain enumeration source with full
pagination support and robust rate limiting handling.
Features:
- Fetches subdomains from URLScan.io Search API
- Implements cursor-based pagination using search_after parameter
- Extracts domains from task.domain, task.url, page.domain, page.url fields
- Requires API key (free tier available at urlscan.io)
Rate Limiting:
- Conservative pagination delay (10s between pages) to respect strict burst limits
- Exponential backoff retry logic for 429/503 responses
- Respects X-Rate-Limit-Reset-After header for dynamic backoff
- Limited to 5 pages max (500 results) to avoid quota exhaustion
Configuration:
- Max 5 pages per enumeration (configurable via maxPages constant)
- 100 results per page (configurable via maxPerPage constant)
- 2 retry attempts for rate-limited requests
- 20 second initial backoff, doubles on each retry
Changes:
- pkg/subscraping/sources/urlscan/urlscan.go: New URLScan source implementation
- pkg/passive/sources.go: Register URLScan source
- pkg/passive/sources_test.go: Add URLScan to test lists
- pkg/runner/options.go: Add urlscan to source options
- .github/workflows/build-test.yml: Add URLSCAN_API_KEY secret
Closes: Feature request for URLScan.io integration
- add KeyRequirement enum with three states: NoKey, OptionalKey, RequiredKey
- add KeyRequirement() method to Source interface
- update config generation to include optional sources
- update env var loading for optional sources
- update source listing with ~ marker for optional sources
- mark hackertarget, leakix, reconeer as OptionalKey
- fix hackertarget and reconeer to work without API key
closes#1695
Adds request counting to all sources to help users monitor API usage
and debug quota consumption issues like #1562.
Changes:
- Add Requests field to Statistics struct
- Track HTTP requests in all 53 sources
- Display request count in stats output
Closes#1698
* Add wildcard certificate detection in JSON output
This commit implements wildcard certificate detection for subdomains that
are covered by wildcard certificates (e.g., *.example.com). When sources
return results containing wildcard patterns, the subdomain is now marked
with a wildcard_certificate field in JSON output mode.
Key changes:
- Added WildcardCertificate field to HostEntry and Result structs
- Detection logic: checks if result.Value contains "*.subdomain" pattern
- Propagates wildcard flag through resolution pipeline
- JSON output includes wildcard_certificate field (omitted if false)
- Updates version to v2.9.1-dev
- Fix .goreleaser.yml 386 architecture quoting
- Add /subfinder to .gitignore
Note: wildcard_certificate field is not included when using -cs flag to
avoid breaking changes to the library API.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix wildcard certificate flag propagation with -nW flag
Fixed a bug where the wildcard_certificate field was being lost when using
the -nW flag (DNS resolution with wildcard filtering). The issue occurred
because when multiple sources find the same subdomain, only the first
occurrence is sent to the resolution pool. If a later source marks the
subdomain as having a wildcard certificate, that information was stored
in uniqueMap but never propagated to foundResults.
Solution: After resolution completes, merge wildcard certificate information
from uniqueMap into foundResults. This ensures that if any source marked a
subdomain as having a wildcard certificate, that flag is preserved in the
final output.
This ensures consistent behavior - wildcard_certificate field appears in
JSON output regardless of whether -nW flag is used.
Validation:
- Without -nW: api.nuclei.sh shows wildcard_certificate:true ✓
- With -nW: api.nuclei.sh shows wildcard_certificate:true ✓
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
Keeps module path as github.com/projectdiscovery/subfinder/v2 to
preserve import compatibility.
Signed-off-by: Mikel Olasagasti Uranga <mikel@olasagasti.info>
Dogan Can Bakir