- PerSourceExceedsGlobal: per-source limit > global should be honoured
(verifies sourceRateLimitOrDefault returns source, not global)
- CaseInsensitive: source names stored lowercase must match mixed-case
lookups (documents and verifies the ToLower normalization contract)
Per CodeRabbit nitpick: the blank identifier assignment and math import
in TestBuildMultiRateLimiter_UnlimitedWhenNoLimits served no purpose.
The test already verifies correct behavior via NoError + NotNil assertions.
CodeRabbit identified that the zero-valued CustomRateLimit struct
left Custom.Map and CustomDuration.Map as nil, which could cause
a panic when resolveSourceRateLimit calls .Get() on these maps.
Mirror the initialization pattern from runner.go by explicitly
creating the underlying maps.
Two bugs in rate limit handling:
1. Global -rl flag was ignored for sources without a per-source override.
Only sources explicitly listed in -rls got rate-limited; all others
ran unlimited, defeating the purpose of -rl.
2. Per-source duration from -rls (e.g. hackertarget=2/m) was silently
dropped. runner.go stored MaxCount but discarded Duration, so all
sources defaulted to per-second regardless of the /m or /s suffix.
Root cause: runner.go converted goflags.RateLimitMap → CustomRateLimit
but only copied MaxCount. buildMultiRateLimiter only checked per-source
limits, never falling back to the global rate.
Changes:
- Add CustomDuration field to CustomRateLimit struct
- Store both MaxCount and Duration from -rls in runner.go
- Add resolveSourceRateLimit() with clear priority:
per-source > global > unlimited
- Guard against nil rateLimit in buildMultiRateLimiter
- Add 9 unit tests covering all combinations
Fixes#1434
Remove custom pagination delay and retry logic since the session
already handles rate limiting via MultiRateLimiter. This aligns
with how other sources (shodan, virustotal) are implemented.
Add leakix, reconeer, and sitedossier to ignored sources list:
- leakix: now requires API key (returns 401)
- reconeer: now requires API key (returns 401)
- sitedossier: flaky, returns no results in CI
Replace deprecated github.com/projectdiscovery/utils/errors package
with standard Go error wrapping using fmt.Errorf to fix staticcheck
SA1019 linter errors.
Add URLScan.io as a new passive subdomain enumeration source with full
pagination support and robust rate limiting handling.
Features:
- Fetches subdomains from URLScan.io Search API
- Implements cursor-based pagination using search_after parameter
- Extracts domains from task.domain, task.url, page.domain, page.url fields
- Requires API key (free tier available at urlscan.io)
Rate Limiting:
- Conservative pagination delay (10s between pages) to respect strict burst limits
- Exponential backoff retry logic for 429/503 responses
- Respects X-Rate-Limit-Reset-After header for dynamic backoff
- Limited to 5 pages max (500 results) to avoid quota exhaustion
Configuration:
- Max 5 pages per enumeration (configurable via maxPages constant)
- 100 results per page (configurable via maxPerPage constant)
- 2 retry attempts for rate-limited requests
- 20 second initial backoff, doubles on each retry
Changes:
- pkg/subscraping/sources/urlscan/urlscan.go: New URLScan source implementation
- pkg/passive/sources.go: Register URLScan source
- pkg/passive/sources_test.go: Add URLScan to test lists
- pkg/runner/options.go: Add urlscan to source options
- .github/workflows/build-test.yml: Add URLSCAN_API_KEY secret
Closes: Feature request for URLScan.io integration
- add KeyRequirement enum with three states: NoKey, OptionalKey, RequiredKey
- add KeyRequirement() method to Source interface
- update config generation to include optional sources
- update env var loading for optional sources
- update source listing with ~ marker for optional sources
- mark hackertarget, leakix, reconeer as OptionalKey
- fix hackertarget and reconeer to work without API key
closes#1695
Adds request counting to all sources to help users monitor API usage
and debug quota consumption issues like #1562.
Changes:
- Add Requests field to Statistics struct
- Track HTTP requests in all 53 sources
- Display request count in stats output
Closes#1698
Mzack9999