The Postgres query inherited the verbatim form crt.sh's web UI uses to
render its certificate browser. subfinder consumes NAME_VALUE only, so
GROUP BY sub.CERTIFICATE on multi-KB DER blobs, the four x509_*
function calls per group, the correlated LEFT JOIN LATERAL into
ct_log_entry, and the trailing ORDER BY ENTRY_TIMESTAMP are all
discarded.
Replacing the WITH/JOIN with a direct DISTINCT NAME_VALUE preserves the
result set (verified parity on iana.org: 24 unique names from both
queries) and lets the LIMIT 10000 clause bound unique subdomains
instead of raw certificate rows. The downstream SplitSeq + Extractor
path is unchanged.
Co-authored-by: Chris (ChrisJr404) <11917633+ChrisJr404@users.noreply.github.com>
Addresses CodeRabbit review: jsoniter's ToBool returns false for empty arrays as well as missing keys, so a valid response with ContributingSubdomainList: [] would incorrectly trip the shape-mismatch branch. Switch to ValueType() != ArrayValue; the iteration loop handles empty arrays naturally (0 results, no error emitted).
Two bugs in one spot: io.ReadAll's error was never checked, so a partial/empty body fell into the else branch unnoticed; and that else branch emitted Error: err where err was whatever ReadAll returned — almost always nil. The real signal ("response didn't contain Result.ContributingSubdomainList") was lost. Check ReadAll up front and synthesize a real error for the shape-mismatch case. Also flattens the happy path.
On session.SimpleGet failure the source returned without emitting an Error result, so stats under-reported and callers saw a silent zero. Emit the error and bump s.errors like every other source does.
jsoniter.NewDecoder(resp.Body).Decode(...) drains the body, so the subsequent io.ReadAll read 0 bytes and its error branch was unreachable. Delete the dead read and drop the now-unused io import.
Widens two 429-only branches to any non-200, moves status status check before ReadAll, and addresses "ressponse" typos; no compat breaking introductions; see #1765.
- PerSourceExceedsGlobal: per-source limit > global should be honoured
(verifies sourceRateLimitOrDefault returns source, not global)
- CaseInsensitive: source names stored lowercase must match mixed-case
lookups (documents and verifies the ToLower normalization contract)
Per CodeRabbit nitpick: the blank identifier assignment and math import
in TestBuildMultiRateLimiter_UnlimitedWhenNoLimits served no purpose.
The test already verifies correct behavior via NoError + NotNil assertions.
CodeRabbit identified that the zero-valued CustomRateLimit struct
left Custom.Map and CustomDuration.Map as nil, which could cause
a panic when resolveSourceRateLimit calls .Get() on these maps.
Mirror the initialization pattern from runner.go by explicitly
creating the underlying maps.
Two bugs in rate limit handling:
1. Global -rl flag was ignored for sources without a per-source override.
Only sources explicitly listed in -rls got rate-limited; all others
ran unlimited, defeating the purpose of -rl.
2. Per-source duration from -rls (e.g. hackertarget=2/m) was silently
dropped. runner.go stored MaxCount but discarded Duration, so all
sources defaulted to per-second regardless of the /m or /s suffix.
Root cause: runner.go converted goflags.RateLimitMap → CustomRateLimit
but only copied MaxCount. buildMultiRateLimiter only checked per-source
limits, never falling back to the global rate.
Changes:
- Add CustomDuration field to CustomRateLimit struct
- Store both MaxCount and Duration from -rls in runner.go
- Add resolveSourceRateLimit() with clear priority:
per-source > global > unlimited
- Guard against nil rateLimit in buildMultiRateLimiter
- Add 9 unit tests covering all combinations
Fixes#1434
Remove custom pagination delay and retry logic since the session
already handles rate limiting via MultiRateLimiter. This aligns
with how other sources (shodan, virustotal) are implemented.
Add leakix, reconeer, and sitedossier to ignored sources list:
- leakix: now requires API key (returns 401)
- reconeer: now requires API key (returns 401)
- sitedossier: flaky, returns no results in CI
Replace deprecated github.com/projectdiscovery/utils/errors package
with standard Go error wrapping using fmt.Errorf to fix staticcheck
SA1019 linter errors.
Add URLScan.io as a new passive subdomain enumeration source with full
pagination support and robust rate limiting handling.
Features:
- Fetches subdomains from URLScan.io Search API
- Implements cursor-based pagination using search_after parameter
- Extracts domains from task.domain, task.url, page.domain, page.url fields
- Requires API key (free tier available at urlscan.io)
Rate Limiting:
- Conservative pagination delay (10s between pages) to respect strict burst limits
- Exponential backoff retry logic for 429/503 responses
- Respects X-Rate-Limit-Reset-After header for dynamic backoff
- Limited to 5 pages max (500 results) to avoid quota exhaustion
Configuration:
- Max 5 pages per enumeration (configurable via maxPages constant)
- 100 results per page (configurable via maxPerPage constant)
- 2 retry attempts for rate-limited requests
- 20 second initial backoff, doubles on each retry
Changes:
- pkg/subscraping/sources/urlscan/urlscan.go: New URLScan source implementation
- pkg/passive/sources.go: Register URLScan source
- pkg/passive/sources_test.go: Add URLScan to test lists
- pkg/runner/options.go: Add urlscan to source options
- .github/workflows/build-test.yml: Add URLSCAN_API_KEY secret
Closes: Feature request for URLScan.io integration
- add KeyRequirement enum with three states: NoKey, OptionalKey, RequiredKey
- add KeyRequirement() method to Source interface
- update config generation to include optional sources
- update env var loading for optional sources
- update source listing with ~ marker for optional sources
- mark hackertarget, leakix, reconeer as OptionalKey
- fix hackertarget and reconeer to work without API key
closes#1695
Adds request counting to all sources to help users monitor API usage
and debug quota consumption issues like #1562.
Changes:
- Add Requests field to Statistics struct
- Track HTTP requests in all 53 sources
- Display request count in stats output
Closes#1698
ChrisJr404