- PerSourceExceedsGlobal: per-source limit > global should be honoured
(verifies sourceRateLimitOrDefault returns source, not global)
- CaseInsensitive: source names stored lowercase must match mixed-case
lookups (documents and verifies the ToLower normalization contract)
Per CodeRabbit nitpick: the blank identifier assignment and math import
in TestBuildMultiRateLimiter_UnlimitedWhenNoLimits served no purpose.
The test already verifies correct behavior via NoError + NotNil assertions.
CodeRabbit identified that the zero-valued CustomRateLimit struct
left Custom.Map and CustomDuration.Map as nil, which could cause
a panic when resolveSourceRateLimit calls .Get() on these maps.
Mirror the initialization pattern from runner.go by explicitly
creating the underlying maps.
Two bugs in rate limit handling:
1. Global -rl flag was ignored for sources without a per-source override.
Only sources explicitly listed in -rls got rate-limited; all others
ran unlimited, defeating the purpose of -rl.
2. Per-source duration from -rls (e.g. hackertarget=2/m) was silently
dropped. runner.go stored MaxCount but discarded Duration, so all
sources defaulted to per-second regardless of the /m or /s suffix.
Root cause: runner.go converted goflags.RateLimitMap → CustomRateLimit
but only copied MaxCount. buildMultiRateLimiter only checked per-source
limits, never falling back to the global rate.
Changes:
- Add CustomDuration field to CustomRateLimit struct
- Store both MaxCount and Duration from -rls in runner.go
- Add resolveSourceRateLimit() with clear priority:
per-source > global > unlimited
- Guard against nil rateLimit in buildMultiRateLimiter
- Add 9 unit tests covering all combinations
Fixes#1434
Add leakix, reconeer, and sitedossier to ignored sources list:
- leakix: now requires API key (returns 401)
- reconeer: now requires API key (returns 401)
- sitedossier: flaky, returns no results in CI
Add URLScan.io as a new passive subdomain enumeration source with full
pagination support and robust rate limiting handling.
Features:
- Fetches subdomains from URLScan.io Search API
- Implements cursor-based pagination using search_after parameter
- Extracts domains from task.domain, task.url, page.domain, page.url fields
- Requires API key (free tier available at urlscan.io)
Rate Limiting:
- Conservative pagination delay (10s between pages) to respect strict burst limits
- Exponential backoff retry logic for 429/503 responses
- Respects X-Rate-Limit-Reset-After header for dynamic backoff
- Limited to 5 pages max (500 results) to avoid quota exhaustion
Configuration:
- Max 5 pages per enumeration (configurable via maxPages constant)
- 100 results per page (configurable via maxPerPage constant)
- 2 retry attempts for rate-limited requests
- 20 second initial backoff, doubles on each retry
Changes:
- pkg/subscraping/sources/urlscan/urlscan.go: New URLScan source implementation
- pkg/passive/sources.go: Register URLScan source
- pkg/passive/sources_test.go: Add URLScan to test lists
- pkg/runner/options.go: Add urlscan to source options
- .github/workflows/build-test.yml: Add URLSCAN_API_KEY secret
Closes: Feature request for URLScan.io integration
- add KeyRequirement enum with three states: NoKey, OptionalKey, RequiredKey
- add KeyRequirement() method to Source interface
- update config generation to include optional sources
- update env var loading for optional sources
- update source listing with ~ marker for optional sources
- mark hackertarget, leakix, reconeer as OptionalKey
- fix hackertarget and reconeer to work without API key
closes#1695
* fix: context cancellation for the major sources
* fix: handle context cancellation for the minor sources
* fix: handle context cancellation for the minor sources
Keeps module path as github.com/projectdiscovery/subfinder/v2 to
preserve import compatibility.
Signed-off-by: Mikel Olasagasti Uranga <mikel@olasagasti.info>
Dogan Can Bakir