fba1cc3d27
fix timeout flag not working for crtsh source
2026-02-02 18:13:11 +03:00
58c1e5ac82
refactor: simplify urlscan source to use session rate limiting
...
Remove custom pagination delay and retry logic since the session
already handles rate limiting via MultiRateLimiter. This aligns
with how other sources (shodan, virustotal) are implemented.
2026-01-30 18:05:47 +05:30
675be65d47
test: skip flaky sources in TestSourcesWithoutKeys
...
Add leakix, reconeer, and sitedossier to ignored sources list:
- leakix: now requires API key (returns 401)
- reconeer: now requires API key (returns 401)
- sitedossier: flaky, returns no results in CI
2026-01-30 17:57:10 +05:30
e8c0e16d6d
fix: replace deprecated errorutil with fmt.Errorf in facebook source
...
Replace deprecated github.com/projectdiscovery/utils/errors package
with standard Go error wrapping using fmt.Errorf to fix staticcheck
SA1019 linter errors.
2026-01-30 17:42:20 +05:30
978addb8bd
feat: add URLScan.io as passive subdomain source
...
Add URLScan.io as a new passive subdomain enumeration source with full
pagination support and robust rate limiting handling.
Features:
- Fetches subdomains from URLScan.io Search API
- Implements cursor-based pagination using search_after parameter
- Extracts domains from task.domain, task.url, page.domain, page.url fields
- Requires API key (free tier available at urlscan.io)
Rate Limiting:
- Conservative pagination delay (10s between pages) to respect strict burst limits
- Exponential backoff retry logic for 429/503 responses
- Respects X-Rate-Limit-Reset-After header for dynamic backoff
- Limited to 5 pages max (500 results) to avoid quota exhaustion
Configuration:
- Max 5 pages per enumeration (configurable via maxPages constant)
- 100 results per page (configurable via maxPerPage constant)
- 2 retry attempts for rate-limited requests
- 20 second initial backoff, doubles on each retry
Changes:
- pkg/subscraping/sources/urlscan/urlscan.go: New URLScan source implementation
- pkg/passive/sources.go: Register URLScan source
- pkg/passive/sources_test.go: Add URLScan to test lists
- pkg/runner/options.go: Add urlscan to source options
- .github/workflows/build-test.yml: Add URLSCAN_API_KEY secret
Closes: Feature request for URLScan.io integration
2026-01-30 17:02:44 +05:30
b71a21bd9b
Sort sources alphabetically
...
Ensure the order of -ls matches provider-config.yaml to
avoid confusion when comparing them.
2026-01-29 18:04:58 +01:00
227ef90b2f
chore(deps): bump the modules group with 10 updates ( #1714 )
...
Bumps the modules group with 10 updates:
| Package | From | To |
| --- | --- | --- |
| [github.com/projectdiscovery/dnsx](https://github.com/projectdiscovery/dnsx ) | `1.2.2` | `1.2.3` |
| [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger ) | `1.1.54` | `1.1.62` |
| [github.com/projectdiscovery/ratelimit](https://github.com/projectdiscovery/ratelimit ) | `0.0.81` | `0.0.82` |
| [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go ) | `1.0.115` | `1.1.0` |
| [github.com/projectdiscovery/utils](https://github.com/projectdiscovery/utils ) | `0.4.21` | `0.7.3` |
| [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck ) | `1.1.24` | `1.2.13` |
| [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer ) | `0.4.1` | `0.4.19` |
| [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap ) | `0.0.90` | `0.0.98` |
| [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy ) | `0.1.16` | `0.1.31` |
| [github.com/projectdiscovery/retryabledns](https://github.com/projectdiscovery/retryabledns ) | `1.0.102` | `1.0.111` |
Updates `github.com/projectdiscovery/dnsx` from 1.2.2 to 1.2.3
- [Release notes](https://github.com/projectdiscovery/dnsx/releases )
- [Commits](https://github.com/projectdiscovery/dnsx/compare/v1.2.2...v1.2.3 )
Updates `github.com/projectdiscovery/gologger` from 1.1.54 to 1.1.62
- [Release notes](https://github.com/projectdiscovery/gologger/releases )
- [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.54...v1.1.62 )
Updates `github.com/projectdiscovery/ratelimit` from 0.0.81 to 0.0.82
- [Release notes](https://github.com/projectdiscovery/ratelimit/releases )
- [Commits](https://github.com/projectdiscovery/ratelimit/compare/v0.0.81...v0.0.82 )
Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.115 to 1.1.0
- [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases )
- [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.115...v1.1.0 )
Updates `github.com/projectdiscovery/utils` from 0.4.21 to 0.7.3
- [Release notes](https://github.com/projectdiscovery/utils/releases )
- [Changelog](https://github.com/projectdiscovery/utils/blob/main/CHANGELOG.md )
- [Commits](https://github.com/projectdiscovery/utils/compare/v0.4.21...v0.7.3 )
Updates `github.com/projectdiscovery/cdncheck` from 1.1.24 to 1.2.13
- [Release notes](https://github.com/projectdiscovery/cdncheck/releases )
- [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.1.24...v1.2.13 )
Updates `github.com/projectdiscovery/fastdialer` from 0.4.1 to 0.4.19
- [Release notes](https://github.com/projectdiscovery/fastdialer/releases )
- [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.4.1...v0.4.19 )
Updates `github.com/projectdiscovery/hmap` from 0.0.90 to 0.0.98
- [Release notes](https://github.com/projectdiscovery/hmap/releases )
- [Commits](https://github.com/projectdiscovery/hmap/compare/v0.0.90...v0.0.98 )
Updates `github.com/projectdiscovery/networkpolicy` from 0.1.16 to 0.1.31
- [Release notes](https://github.com/projectdiscovery/networkpolicy/releases )
- [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.16...v0.1.31 )
Updates `github.com/projectdiscovery/retryabledns` from 1.0.102 to 1.0.111
- [Release notes](https://github.com/projectdiscovery/retryabledns/releases )
- [Commits](https://github.com/projectdiscovery/retryabledns/compare/v1.0.102...v1.0.111 )
---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/dnsx
dependency-version: 1.2.3
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/gologger
dependency-version: 1.1.62
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/ratelimit
dependency-version: 0.0.82
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/retryablehttp-go
dependency-version: 1.1.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: modules
- dependency-name: github.com/projectdiscovery/utils
dependency-version: 0.7.3
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: modules
- dependency-name: github.com/projectdiscovery/cdncheck
dependency-version: 1.2.13
dependency-type: indirect
update-type: version-update:semver-minor
dependency-group: modules
- dependency-name: github.com/projectdiscovery/fastdialer
dependency-version: 0.4.19
dependency-type: indirect
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/hmap
dependency-version: 0.0.98
dependency-type: indirect
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/networkpolicy
dependency-version: 0.1.31
dependency-type: indirect
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/retryabledns
dependency-version: 1.0.111
dependency-type: indirect
update-type: version-update:semver-patch
dependency-group: modules
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-01-24 15:11:49 +05:30
d9f08bdc64
dependanbot: change path after #1613 ( #1708 )
2026-01-24 15:06:27 +05:30
1eac4a2b13
fix: mark LeakIX as requiring an API key
2026-01-21 21:38:35 +01:00
c846de75ac
Merge pull request #1696 from projectdiscovery/dev
...
v2.12.0
2026-01-12 15:24:30 +07:00
4984bfb3a7
Merge pull request #1699 from projectdiscovery/feature/request-tracking-stats
...
Add request tracking to -stats flag
2026-01-07 18:10:35 +04:00
4bba544dfe
Merge branch 'dev' into feature/request-tracking-stats
2026-01-07 18:07:37 +04:00
a2615faa73
Merge pull request #1700 from projectdiscovery/feature/optional-api-key-support
...
Add optional API key support for sources
2026-01-06 01:01:22 +07:00
1ca2a67f92
add optional API key support for sources
...
- add KeyRequirement enum with three states: NoKey, OptionalKey, RequiredKey
- add KeyRequirement() method to Source interface
- update config generation to include optional sources
- update env var loading for optional sources
- update source listing with ~ marker for optional sources
- mark hackertarget, leakix, reconeer as OptionalKey
- fix hackertarget and reconeer to work without API key
closes #1695
2026-01-05 15:00:59 +03:00
9b4705252e
fix race condition in github and gitlab sources
...
use atomic.Int32 for counter fields to prevent data races when
multiple goroutines concurrently increment requests/results/errors
2026-01-05 14:23:20 +03:00
e96320ec38
add request tracking to -stats flag
...
Adds request counting to all sources to help users monitor API usage
and debug quota consumption issues like #1562 .
Changes:
- Add Requests field to Statistics struct
- Track HTTP requests in all 53 sources
- Display request count in stats output
Closes #1698
2026-01-05 14:14:20 +03:00
c1977c0315
Merge pull request #1694 from projectdiscovery/add_reconeer
...
add reconeer
2026-01-05 17:29:41 +07:00
8eacfb143a
bump version
2026-01-05 11:42:17 +03:00
9d815b78a4
Merge pull request #1622 from projectdiscovery/1576_add_apikey_support_for_hackertarget
...
add apikey support for hackertarget
2026-01-04 22:29:27 +07:00
e5a61a6db7
Merge branch 'dev' into 1576_add_apikey_support_for_hackertarget
2026-01-04 22:26:01 +07:00
fe41f49040
add reconeer
2026-01-04 15:56:45 +03:00
933918a3cb
Merge pull request #1689 from projectdiscovery/dev
...
v2.11.0
2025-12-20 00:29:28 +07:00
2e0982cb0e
Merge pull request #1683 from nohehf/main
...
feat: add merklemap source
2025-12-20 00:12:02 +07:00
322a4336ed
Merge branch 'dev' into main
2025-12-19 17:33:49 +01:00
e2d7ecfd14
fix: tests
2025-12-19 17:06:04 +01:00
f3b1870f36
Merge branch 'main' into dev
2025-12-18 19:30:17 +07:00
79a426007b
bump version
2025-12-18 15:28:48 +03:00
2e32220a39
Merge pull request #1685 from projectdiscovery/add_thc
2025-12-18 17:27:27 +07:00
692e30a592
add thc
2025-12-18 12:24:51 +03:00
e18ed93e52
fix: implement feedback
2025-12-16 18:03:34 +01:00
7ac8b6cfcc
feat: skip merklemap if no api key
2025-12-16 17:55:13 +01:00
e7738aa6bd
feat: merklemap source
2025-12-16 16:35:04 +01:00
7842ebfa64
Merge pull request #1682 from projectdiscovery/1681_add_profundis
...
add profundis
2025-12-10 13:13:31 +04:00
5ad10c65a9
Merge pull request #1654 from projectdiscovery/feature/censys-api-key
...
Update Censys PAT integration and docs
2025-12-09 15:06:33 +07:00
0b6432d8cc
fix: linter errors
2025-12-08 20:09:59 +05:30
b7193b46c3
fix: response strucutre
2025-12-08 20:04:27 +05:30
8cf49e7e31
fix: added go doc
2025-12-08 19:33:54 +05:30
6b3df63b66
fix: added unit test and tested e2e
2025-12-08 19:32:53 +05:30
037d063603
minor
2025-12-08 15:25:17 +03:00
074497b536
add profundis
2025-12-08 10:31:36 +03:00
deccd62f59
fix:context cancellation sources ( #1680 )
...
* fix: context cancellation for the major sources
* fix: handle context cancellation for the minor sources
* fix: handle context cancellation for the minor sources
2025-12-04 23:04:40 +05:30
bfdb4a8340
Merge branch 'main' of https://github.com/projectdiscovery/subfinder
2025-11-22 18:43:15 +05:30
786a6464b5
v2.10.1 ( #1674 )
...
* chore(deps): bump golang.org/x/crypto from 0.36.0 to 0.45.0
Bumps [golang.org/x/crypto](https://github.com/golang/crypto ) from 0.36.0 to 0.45.0.
- [Commits](https://github.com/golang/crypto/compare/v0.36.0...v0.45.0 )
---
updated-dependencies:
- dependency-name: golang.org/x/crypto
dependency-version: 0.45.0
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com >
* bump version
---------
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-11-21 22:12:56 +05:30
fb6b520a25
Merge pull request #1673 from projectdiscovery/bump_version
...
bump version
2025-11-21 13:11:20 +09:00
c83e0b62b4
bump version
2025-11-21 13:09:54 +09:00
deeaf0cf1f
chore(deps): bump golang.org/x/crypto from 0.36.0 to 0.45.0
...
Bumps [golang.org/x/crypto](https://github.com/golang/crypto ) from 0.36.0 to 0.45.0.
- [Commits](https://github.com/golang/crypto/compare/v0.36.0...v0.45.0 )
---
updated-dependencies:
- dependency-name: golang.org/x/crypto
dependency-version: 0.45.0
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com >
2025-11-20 02:03:22 +00:00
cc04a596e5
Merge pull request #1668 from projectdiscovery/dev
...
v2.10.0
2025-11-19 11:28:53 +09:00
5bcfca5358
Merge branch 'main' into dev
2025-11-18 11:02:40 +09:00
92bd205d21
Merge pull request #1666 from sunnyraindy/dev
...
refactor: use strings.Builder to improve performance
2025-11-12 08:07:08 +03:00
d5255517a3
refactor: use strings.Builder to improve performance
...
Signed-off-by: sunnyraindy <sunnyraindy@outlook.com >
2025-11-11 23:06:17 +08:00
Recep Gunes