Two bugs in rate limit handling:
1. Global -rl flag was ignored for sources without a per-source override.
Only sources explicitly listed in -rls got rate-limited; all others
ran unlimited, defeating the purpose of -rl.
2. Per-source duration from -rls (e.g. hackertarget=2/m) was silently
dropped. runner.go stored MaxCount but discarded Duration, so all
sources defaulted to per-second regardless of the /m or /s suffix.
Root cause: runner.go converted goflags.RateLimitMap → CustomRateLimit
but only copied MaxCount. buildMultiRateLimiter only checked per-source
limits, never falling back to the global rate.
Changes:
- Add CustomDuration field to CustomRateLimit struct
- Store both MaxCount and Duration from -rls in runner.go
- Add resolveSourceRateLimit() with clear priority:
per-source > global > unlimited
- Guard against nil rateLimit in buildMultiRateLimiter
- Add 9 unit tests covering all combinations
Fixes#1434
- add KeyRequirement enum with three states: NoKey, OptionalKey, RequiredKey
- add KeyRequirement() method to Source interface
- update config generation to include optional sources
- update env var loading for optional sources
- update source listing with ~ marker for optional sources
- mark hackertarget, leakix, reconeer as OptionalKey
- fix hackertarget and reconeer to work without API key
closes#1695
Adds request counting to all sources to help users monitor API usage
and debug quota consumption issues like #1562.
Changes:
- Add Requests field to Statistics struct
- Track HTTP requests in all 53 sources
- Display request count in stats output
Closes#1698
Keeps module path as github.com/projectdiscovery/subfinder/v2 to
preserve import compatibility.
Signed-off-by: Mikel Olasagasti Uranga <mikel@olasagasti.info>
- Support multiple tokens to randomly pick one to avoid exceeding request rate limit
- Retry after "Retry-After" seconds header value if rate limit abuse is detected.
Charles Wong