- add KeyRequirement enum with three states: NoKey, OptionalKey, RequiredKey
- add KeyRequirement() method to Source interface
- update config generation to include optional sources
- update env var loading for optional sources
- update source listing with ~ marker for optional sources
- mark hackertarget, leakix, reconeer as OptionalKey
- fix hackertarget and reconeer to work without API key
closes#1695
Adds request counting to all sources to help users monitor API usage
and debug quota consumption issues like #1562.
Changes:
- Add Requests field to Statistics struct
- Track HTTP requests in all 53 sources
- Display request count in stats output
Closes#1698
* fix: context cancellation for the major sources
* fix: handle context cancellation for the minor sources
* fix: handle context cancellation for the minor sources
Keeps module path as github.com/projectdiscovery/subfinder/v2 to
preserve import compatibility.
Signed-off-by: Mikel Olasagasti Uranga <mikel@olasagasti.info>
Sometimes we may have to check the status code to act accordingly, ej: GitHub source needs to check for 403 to know when a request rate limit exceeded occurs, invalidate the current token and get the next in the pool.
- Support multiple tokens to randomly pick one to avoid exceeding request rate limit
- Retry after "Retry-After" seconds header value if rate limit abuse is detected.
Mzack9999