4bba544dfe
Merge branch 'dev' into feature/request-tracking-stats
2026-01-07 18:07:37 +04:00
1ca2a67f92
add optional API key support for sources
...
- add KeyRequirement enum with three states: NoKey, OptionalKey, RequiredKey
- add KeyRequirement() method to Source interface
- update config generation to include optional sources
- update env var loading for optional sources
- update source listing with ~ marker for optional sources
- mark hackertarget, leakix, reconeer as OptionalKey
- fix hackertarget and reconeer to work without API key
closes #1695
2026-01-05 15:00:59 +03:00
9b4705252e
fix race condition in github and gitlab sources
...
use atomic.Int32 for counter fields to prevent data races when
multiple goroutines concurrently increment requests/results/errors
2026-01-05 14:23:20 +03:00
e96320ec38
add request tracking to -stats flag
...
Adds request counting to all sources to help users monitor API usage
and debug quota consumption issues like #1562 .
Changes:
- Add Requests field to Statistics struct
- Track HTTP requests in all 53 sources
- Display request count in stats output
Closes #1698
2026-01-05 14:14:20 +03:00
deccd62f59
fix:context cancellation sources ( #1680 )
...
* fix: context cancellation for the major sources
* fix: handle context cancellation for the minor sources
* fix: handle context cancellation for the minor sources
2025-12-04 23:04:40 +05:30
6867e076f1
refactor: move v2 module code to root directory for cleaner structure
...
Keeps module path as github.com/projectdiscovery/subfinder/v2 to
preserve import compatibility.
Signed-off-by: Mikel Olasagasti Uranga <mikel@olasagasti.info >
2025-07-10 15:27:42 +02:00
7d89b5facb
go mod v2 fix
2020-09-24 19:02:02 +02:00
1cac2e87b9
Fixes #306
...
Invalid return
2020-09-09 11:39:24 +02:00
016b69a9df
Fixed GitHub search query term + sort results
...
The double quotes arround the search term does not return any results for some terms.
2020-08-13 00:48:31 +02:00
e828cee8a0
Memory improvement for sources
2020-08-10 10:15:30 +02:00
caf8062b92
Simplified subscraping HTTP operations with custom agent methods
2020-07-23 22:34:24 +02:00
77eb7090f3
Fixed funlen
2020-07-22 17:52:35 +02:00
ec5b0cebdb
Format code with gofmt
2020-07-22 17:47:58 +02:00
2e8ee83606
Fixed misspelling of Process
2020-07-22 17:12:47 +02:00
9798cd9d3f
Fixed importShadowing (gocritic)
2020-07-22 16:17:55 +02:00
75c3682b6e
Fixed S1002: should omit comparison to bool constant (gosimple)
2020-07-22 15:53:59 +02:00
56a83256e4
Fixed golint style issues
2020-07-22 15:52:41 +02:00
47a37a7dde
Fixed File is not gofmt-ed with -s (gofmt)
...
Fixed running 'gofmt -s -w file'
2020-07-22 15:38:38 +02:00
736dff1c64
Fixed misspellings (misspell)
2020-07-22 15:35:01 +02:00
7fbbd79dca
Fixed assignOp (gocritic)
2020-07-22 15:24:02 +02:00
eb9bb0c930
Fixed wrapperFunc
...
use strings.ReplaceAll method instead off strings.Replace (gocritic)
2020-07-22 15:05:10 +02:00
2a73ce2811
Fixed incorrect handling of 404 status code
...
Fixes a bug introduced on #274
2020-07-21 13:55:15 +02:00
c0cec04a04
Fixed incorrect handling on item 404 status code
...
We can not use continue because skip the current item text match results
2020-07-21 13:05:55 +02:00
9d6d79f19d
Report request item error only if status code is 404
2020-07-20 21:41:52 +02:00
937ad6a441
Fixed isNotFound check
2020-07-20 21:34:01 +02:00
8e2ce0aa6e
Continue on 404 error while reading item code before matching subdomains
2020-07-20 20:59:35 +02:00
3af1d48d5a
New method to discard a HTTP response on unexpected status
2020-07-20 20:49:50 +02:00
a4576376d7
Return the response on unexpected status code
...
Sometimes we may have to check the status code to act accordingly, ej: GitHub source needs to check for 403 to know when a request rate limit exceeded occurs, invalidate the current token and get the next in the pool.
2020-07-20 19:38:52 +02:00
798815c519
Fixed code format
2020-07-17 22:59:27 +02:00
0cfb970e60
Round-robin token managment for distributed GitHub token use
2020-07-17 22:55:32 +02:00
c8ae7a5bd3
Better regular expression to matching subdomains
...
Avoid results starting with "." for wildcard subdomains in content
2020-07-15 22:13:44 +02:00
f0ccebbd8d
Find subdomains in TextMatches too, normalize content before matching
2020-07-15 22:07:45 +02:00
06b7137439
Simplified regular expression
2020-07-15 20:38:38 +02:00
d2cf41d939
FindAllString to get all matches ignoring submatches
...
We do not need submatches because we are not using it.
2020-07-15 20:27:09 +02:00
eadd2e571a
Fixed request rate limit exceeded message
2020-07-15 15:07:25 +02:00
eda27f6934
Handle GitHub Search request rate limit
...
- Support multiple tokens to randomly pick one to avoid exceeding request rate limit
- Retry after "Retry-After" seconds header value if rate limit abuse is detected.
2020-07-15 13:19:31 +02:00
44c58f4d53
Verbose info for next GitHub search results URL
2020-07-05 20:50:45 +02:00
032ee977dd
Added GitHub search source
...
Search for subdomains in GitHub code providing a Personal access token
2020-07-05 20:04:20 +02:00
Mzack9999