admin/projectdiscovery-subfinder
1
0
Fork 0
mirror of https://github.com/projectdiscovery/subfinder synced 2026-06-21 14:05:24 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,205 Commits 5 Branches 61 Tags
dev
Commit Graph

88 Commits

Author SHA1 Message Date
Pontus Lindblom e6ed1e1b72 Chore - Code cleanup in workflows (#1768) 
* Change file extension to .yml for consistency

* Move permissions for clarity and consistency

* Change compat-checks action from @master to @v1

The action should be pinned to a stable ref such as @v1, consistent with
the documentation and how other PD projects reference it.

* Fix indentations and whitespaces in workflows
 2026-03-24 02:26:08 +01:00
Mzack9999 7bd72ff7f7 use org actions 2026-03-20 23:09:48 +01:00
PontusLindblom 0dfd80add5 Update workflow dependencies 2026-02-07 08:21:53 +01:00
Jigardjain 978addb8bd feat: add URLScan.io as passive subdomain source 
Add URLScan.io as a new passive subdomain enumeration source with full
pagination support and robust rate limiting handling.

Features:
- Fetches subdomains from URLScan.io Search API
- Implements cursor-based pagination using search_after parameter
- Extracts domains from task.domain, task.url, page.domain, page.url fields
- Requires API key (free tier available at urlscan.io)

Rate Limiting:
- Conservative pagination delay (10s between pages) to respect strict burst limits
- Exponential backoff retry logic for 429/503 responses
- Respects X-Rate-Limit-Reset-After header for dynamic backoff
- Limited to 5 pages max (500 results) to avoid quota exhaustion

Configuration:
- Max 5 pages per enumeration (configurable via maxPages constant)
- 100 results per page (configurable via maxPerPage constant)
- 2 retry attempts for rate-limited requests
- 20 second initial backoff, doubles on each retry

Changes:
- pkg/subscraping/sources/urlscan/urlscan.go: New URLScan source implementation
- pkg/passive/sources.go: Register URLScan source
- pkg/passive/sources_test.go: Add URLScan to test lists
- pkg/runner/options.go: Add urlscan to source options
- .github/workflows/build-test.yml: Add URLSCAN_API_KEY secret

Closes: Feature request for URLScan.io integration
 2026-01-30 17:02:44 +05:30
Doğan Can Bakır bc7edda8b6 remove hunter 2025-10-20 11:31:10 +03:00
Mzack9999 68a34884ab latest osx 2025-09-24 20:23:39 +02:00
Douglas Danger Manley f0ec4032b2 Fix the build-test action (#1631) 
This fixes the `build-test` action by restoring the `examples`
working directory.  This was incorrectly removed in the previous
commit.
 2025-08-15 19:43:22 +05:30
Mikel Olasagasti Uranga 0d9014e177 refactor: adapt workflows to new non-v2 paths 2025-07-10 15:39:25 +02:00
Dogan Can Bakir b669adbfd2 modernize subfinder (#1601) 
* update gh lint action

* modernize

* minor

* update lint

* fix lint errs

* fix MORE lint errs

* join errs

* fix MOOORE lint errs

* add go version file

* ai review

* ignore anubis tests
 2025-06-20 22:02:13 +05:30
Doğan Can Bakır ab4e8852b7 update lint action 2025-04-28 15:10:46 +03:00
Dwi Siswanto 232dbdef90 ci(compat-checks): define go version file 
Signed-off-by: Dwi Siswanto <git@dw1.io>
 2025-04-01 15:04:37 +07:00
Dwi Siswanto b974fbdd74 ci: adds compatibility checks 
also merge lint to build test

Signed-off-by: Dwi Siswanto <git@dw1.io>
 2025-03-31 18:45:44 +07:00
Dogan Can Bakir 099637fff4 remove passivetotal (#1515) 2025-01-25 19:58:02 +05:30
sandeep ec0060f34c fix outdated option 2024-10-26 01:27:30 +05:30
Ramana Reddy d3d7121298 Fix build failure on termux/android (#1053) 
* update utils pkg

* update go version to 1.21
 2023-11-20 13:36:44 +05:30
Mzack9999 36d102b42e Merge pull request #948 from projectdiscovery/use_short_for_long_tests 
use `-short`mode for excluding long running tests
 2023-08-29 08:01:04 +02:00
Dogan Can Bakir 1d37afff6f remove zoomeye (#943) 2023-08-24 18:08:07 +05:30
Doğan Can Bakır a7764d0904 minor 2023-08-21 11:49:25 +00:00
Doğan Can Bakır 43ebc2ef33 use -shortmode for excluding long running tests 2023-08-21 11:45:27 +00:00
Dogan Can Bakir 2aff5497fc add rate limit per source option (#891) 
* add rate limit per source option

* check for err

* check for lower and uppper bounds

* fix tests

* redo lint directive

* fix lint err

* fix tests

* removing deprecated code

* []string => nil

* add new type for ratelimit cfg

* move rate limit init in NewRunner func

* commit to last commit

* little renaming

* flagging hackertarget as ignored

* attempting non breaking change

* use ratelimit flag

* some code cleanup

* validate keys

* update README

---------

Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
 2023-08-07 19:27:57 +05:30
Tarun Koyalwar 70d0a5c8ba adds new source facebook (#912) 
* add new source: facebook

* misc update

* updated example key format

---------

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
 2023-07-27 16:41:29 +05:30
sandeep 0d90c635f6 added dep auto merge workflow 2023-07-23 15:31:35 +05:30
sandeep 51687c5eaa workflow fix 2023-07-23 15:21:08 +05:30
sandeep da47f3f58b added release test 2023-07-23 15:10:53 +05:30
dependabot[bot] 56dd9426d8 chore(deps): bump golangci/golangci-lint-action from 3.5.0 to 3.6.0 (#876) 
Bumps [golangci/golangci-lint-action](https://github.com/golangci/golangci-lint-action) from 3.5.0 to 3.6.0.
- [Release notes](https://github.com/golangci/golangci-lint-action/releases)
- [Commits](https://github.com/golangci/golangci-lint-action/compare/v3.5.0...v3.6.0)

---
updated-dependencies:
- dependency-name: golangci/golangci-lint-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-17 12:30:37 +05:30
dependabot[bot] 5ac9d83f56 chore(deps): bump golangci/golangci-lint-action from 3.4.0 to 3.5.0 (#869) 
Bumps [golangci/golangci-lint-action](https://github.com/golangci/golangci-lint-action) from 3.4.0 to 3.5.0.
- [Release notes](https://github.com/golangci/golangci-lint-action/releases)
- [Commits](https://github.com/golangci/golangci-lint-action/compare/v3.4.0...v3.5.0)

---
updated-dependencies:
- dependency-name: golangci/golangci-lint-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-05 21:57:47 +05:30
nth347 94b8115c90 fix provider config file not used in library mode (#864) 
* Give library user the ability to parse their own Options

* Rename the module

* Make changes according to projectdiscovery/subfinder/pull/864

* fix import paths

* load providers in library mode

* add example library usage

* update example to show providerConfig option

---------

Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
 2023-06-05 20:19:09 +05:30
Dogan Can Bakir 8d75460c41 Test sources require an API key (#852) 
* test providers require key

* remove redundant tests

* ignore failing(often) test

* added crtsh to ignore list

---------

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
 2023-05-31 01:02:33 +05:30
sandeep c3b493b064 version update 2023-05-01 18:40:58 +05:30
dependabot[bot] c62f7fd2a8 chore(deps): bump actions/setup-go from 3 to 4 (#794) 
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 3 to 4.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](https://github.com/actions/setup-go/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/setup-go
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-04-15 18:28:13 +05:30
Sandeep Singh d577ef206d workflow update (#790) 2023-03-17 21:33:56 +05:30
Sandeep Singh 6bbd5ca94c workflow updates (#767) 2023-02-22 20:17:46 +05:30
sandeep c493f2ff15 workflow update 2023-02-21 19:35:48 +05:30
dependabot[bot] b86e652d7b chore(deps): bump docker/build-push-action from 3 to 4 (#754) 
Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 3 to 4.
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](https://github.com/docker/build-push-action/compare/v3...v4)

---
updated-dependencies:
- dependency-name: docker/build-push-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-02-18 04:02:45 +05:30
dependabot[bot] ea4119920a chore(deps): bump golangci/golangci-lint-action from 3.3.1 to 3.4.0 (#748) 
Bumps [golangci/golangci-lint-action](https://github.com/golangci/golangci-lint-action) from 3.3.1 to 3.4.0.
- [Release notes](https://github.com/golangci/golangci-lint-action/releases)
- [Commits](https://github.com/golangci/golangci-lint-action/compare/v3.3.1...v3.4.0)

---
updated-dependencies:
- dependency-name: golangci/golangci-lint-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-02-18 03:58:26 +05:30
Shubham Rasal 8cc8f2562e resolve failing cases (#720) 
* Resolve the failing cases

* Update test case for commenting reconcloud and threatminer

* add retry for test command

* update retry for test command

* update retry to cd v2

* add ; for windows after cd

* add ; for windows after cd

* add retry

* Build is not starting testing

* Update test

* Update test

* Update timeout

* Add dail timeout

* Update github action to remove on push build & lint
 2022-11-29 18:19:36 +05:30
dependabot[bot] 929ae28e9a chore(deps): bump golangci/golangci-lint-action from 3.3.0 to 3.3.1 (#705) 
Bumps [golangci/golangci-lint-action](https://github.com/golangci/golangci-lint-action) from 3.3.0 to 3.3.1.
- [Release notes](https://github.com/golangci/golangci-lint-action/releases)
- [Commits](https://github.com/golangci/golangci-lint-action/compare/v3.3.0...v3.3.1)

---
updated-dependencies:
- dependency-name: golangci/golangci-lint-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-11-17 19:47:32 +05:30
dependabot[bot] c6eca3ba5b chore(deps): bump golangci/golangci-lint-action from 3.2.0 to 3.3.0 (#681) 
Bumps [golangci/golangci-lint-action](https://github.com/golangci/golangci-lint-action) from 3.2.0 to 3.3.0.
- [Release notes](https://github.com/golangci/golangci-lint-action/releases)
- [Commits](https://github.com/golangci/golangci-lint-action/compare/v3.2.0...v3.3.0)

---
updated-dependencies:
- dependency-name: golangci/golangci-lint-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-10-29 14:57:26 +02:00
sandeep 34236b08d1 release workflow update 2022-10-15 15:34:04 +05:30
Sami b6ed45cb14 new source dnsrepo (#501) 
* new source dnsrepo

* added api keys for dnsrepo

* source added to test case

* integration test set up and test case for dnsrepo

* lint error fix

* misc update

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
 2022-09-20 22:53:30 +05:30
Mzack9999 aa6c83f36e Merge pull request #543 from projectdiscovery/dependabot/github_actions/dev/docker/login-action-2 
chore(deps): bump docker/login-action from 1 to 2
 2022-05-23 16:58:11 +02:00
Mzack9999 2013dddd15 Merge pull request #544 from projectdiscovery/dependabot/github_actions/dev/docker/setup-qemu-action-2 
chore(deps): bump docker/setup-qemu-action from 1 to 2
 2022-05-23 16:58:00 +02:00
Mzack9999 7d9b321a24 Merge pull request #551 from projectdiscovery/dependabot/github_actions/dev/goreleaser/goreleaser-action-3 
chore(deps): bump goreleaser/goreleaser-action from 2 to 3
 2022-05-23 16:57:51 +02:00
dependabot[bot] 9260556fb6 chore(deps): bump docker/setup-qemu-action from 1 to 2 
Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 1 to 2.
- [Release notes](https://github.com/docker/setup-qemu-action/releases)
- [Commits](https://github.com/docker/setup-qemu-action/compare/v1...v2)

---
updated-dependencies:
- dependency-name: docker/setup-qemu-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-05-23 14:52:52 +00:00
dependabot[bot] 413bbeafe9 chore(deps): bump docker/login-action from 1 to 2 
Bumps [docker/login-action](https://github.com/docker/login-action) from 1 to 2.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v1...v2)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-05-23 14:52:49 +00:00
Mzack9999 f965254527 Merge pull request #545 from projectdiscovery/dependabot/github_actions/dev/docker/setup-buildx-action-2 
chore(deps): bump docker/setup-buildx-action from 1 to 2
 2022-05-23 16:52:30 +02:00
dependabot[bot] 5c10972d86 chore(deps): bump goreleaser/goreleaser-action from 2 to 3 
Bumps [goreleaser/goreleaser-action](https://github.com/goreleaser/goreleaser-action) from 2 to 3.
- [Release notes](https://github.com/goreleaser/goreleaser-action/releases)
- [Commits](https://github.com/goreleaser/goreleaser-action/compare/v2...v3)

---
updated-dependencies:
- dependency-name: goreleaser/goreleaser-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-05-23 14:52:26 +00:00
Mzack9999 b9c72b3a43 Merge pull request #546 from projectdiscovery/dependabot/github_actions/dev/docker/build-push-action-3 
chore(deps): bump docker/build-push-action from 2 to 3
 2022-05-23 16:52:21 +02:00
dependabot[bot] f933b57426 chore(deps): bump golangci/golangci-lint-action from 3.1.0 to 3.2.0 
Bumps [golangci/golangci-lint-action](https://github.com/golangci/golangci-lint-action) from 3.1.0 to 3.2.0.
- [Release notes](https://github.com/golangci/golangci-lint-action/releases)
- [Commits](https://github.com/golangci/golangci-lint-action/compare/v3.1.0...v3.2.0)

---
updated-dependencies:
- dependency-name: golangci/golangci-lint-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-05-16 14:05:04 +00:00
dependabot[bot] 35b9c4b4ad chore(deps): bump docker/build-push-action from 2 to 3 
Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 2 to 3.
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](https://github.com/docker/build-push-action/compare/v2...v3)

---
updated-dependencies:
- dependency-name: docker/build-push-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-05-09 14:04:53 +00:00