Remove custom pagination delay and retry logic since the session
already handles rate limiting via MultiRateLimiter. This aligns
with how other sources (shodan, virustotal) are implemented.
Add leakix, reconeer, and sitedossier to ignored sources list:
- leakix: now requires API key (returns 401)
- reconeer: now requires API key (returns 401)
- sitedossier: flaky, returns no results in CI
Replace deprecated github.com/projectdiscovery/utils/errors package
with standard Go error wrapping using fmt.Errorf to fix staticcheck
SA1019 linter errors.
Add URLScan.io as a new passive subdomain enumeration source with full
pagination support and robust rate limiting handling.
Features:
- Fetches subdomains from URLScan.io Search API
- Implements cursor-based pagination using search_after parameter
- Extracts domains from task.domain, task.url, page.domain, page.url fields
- Requires API key (free tier available at urlscan.io)
Rate Limiting:
- Conservative pagination delay (10s between pages) to respect strict burst limits
- Exponential backoff retry logic for 429/503 responses
- Respects X-Rate-Limit-Reset-After header for dynamic backoff
- Limited to 5 pages max (500 results) to avoid quota exhaustion
Configuration:
- Max 5 pages per enumeration (configurable via maxPages constant)
- 100 results per page (configurable via maxPerPage constant)
- 2 retry attempts for rate-limited requests
- 20 second initial backoff, doubles on each retry
Changes:
- pkg/subscraping/sources/urlscan/urlscan.go: New URLScan source implementation
- pkg/passive/sources.go: Register URLScan source
- pkg/passive/sources_test.go: Add URLScan to test lists
- pkg/runner/options.go: Add urlscan to source options
- .github/workflows/build-test.yml: Add URLSCAN_API_KEY secret
Closes: Feature request for URLScan.io integration
Jigardjain