admin/projectdiscovery-httpx
1
0
Fork 0
mirror of https://github.com/projectdiscovery/httpx synced 2026-06-08 16:50:17 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,774 Commits 15 Branches 65 Tags
dev
Commit Graph

196 Commits

Author SHA1 Message Date
Ayan Rajpoot 7fd08b7f8d Add support for duplicate custom request headers (#2504) 
* refactor: change CustomHeaders to support multiple values per header

* fix: clear existing header values before setting custom headers

* harden headers

* expand tests

---------

Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
 2026-06-04 14:21:55 +02:00
Kral d0081af3ec fix: improve error handling and fix bugs (#2499) 
* code quality: improve error handling and fix bugs

- Fix bug: return err instead of closeErr in DecodeData error path (httpx.go)
- Improve error handling for strconv.Atoi calls in multiple locations
- Change hammingDistanceThreshold from var to const since it's never modified
- Fix redundant return statement in ListFilesWithPattern
- Improve port parsing to handle empty port strings explicitly

* runner: keep input counters incrementing under -skip-dedupe

Revert the defensive strconv.Atoi guard around the SkipDedupe branches:
values stored in r.hm are always integer strings written by this same
code path, so the parseErr/cnt>0 check is unreachable in practice, and
when taken literally it skipped the numHosts/numTargets increment for
duplicate inputs.

---------

Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
 2026-05-20 06:41:57 +02:00
Dogan Can Bakir 6c34d93168 feat: honor HTTP_PROXY/HTTPS_PROXY env vars when -proxy is unset (#2493) 
Falls back to http.ProxyFromEnvironment for the HTTP transport and
reads the same vars manually for the headless launcher (chromedp
does not read env).

Closes #2492
 2026-05-13 21:24:40 +02:00
Mzack9999 9836829c62 Merge pull request #2455 from tal7aouy/fix/unsafe-head-nil-pointer 
fix: prevent nil panic for unsafe HEAD requests
 2026-03-21 00:10:59 +01:00
Mzack9999 6425eb5ed0 fmt 2026-03-21 00:01:56 +01:00
Mhammed Talhaouy 8ae53cb9ae fix: prevent nil panic for unsafe HEAD requests 2026-03-19 00:12:21 +00:00
Veirt b7dabf7456 fix(httpx): keep retry fallback on HTTP/1.1 when -pr http11 is set 2026-02-28 02:45:47 +08:00
Mzack9999 7e6003ca51 fixing comment 2025-12-24 15:11:53 +04:00
Mzack9999 78bf9c393a restoring original logic + limiting read to 512Mb + lint 2025-12-24 15:00:10 +04:00
Connor Larson 85695760cb fix: removed temp logger 2025-12-22 12:37:23 -07:00
Connor Larson ec2b7154bc changed resp.Raw contstruction to cap the body, avoiding oom issue 2025-12-22 12:22:10 -07:00
Mzack9999 05c6364d90 better error handling 2025-11-18 16:46:15 +04:00
ayanrajpoot10 497c39920d fix: ensure Host header is set in unsafe mode for rawhttp requests 2025-10-22 20:44:53 +05:30
Mzack9999 2b688a5587 lint 2025-08-28 17:04:33 +02:00
Mzack9999 f2881db0cb improving autoref 2025-08-28 16:46:38 +02:00
siamak a.m.o f0e464f6ea Implement auto-referer option 
The option -auto-referer, sets the referer HTTP header
of the current request to it's URL.
 2025-08-18 12:08:42 +03:30
dependabot[bot] f973821811 chore(deps): bump the modules group with 7 updates (#2216) 
* chore(deps): bump the modules group with 7 updates

Bumps the modules group with 7 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.1.27` | `1.1.28` |
| [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.4.1` | `0.4.2` |
| [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.17` | `0.1.18` |
| [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.0.117` | `1.0.118` |
| [github.com/projectdiscovery/tlsx](https://github.com/projectdiscovery/tlsx) | `1.1.9` | `1.2.0` |
| [github.com/projectdiscovery/utils](https://github.com/projectdiscovery/utils) | `0.4.21` | `0.4.22` |
| [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.37` | `0.2.38` |


Updates `github.com/projectdiscovery/cdncheck` from 1.1.27 to 1.1.28
- [Release notes](https://github.com/projectdiscovery/cdncheck/releases)
- [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml)
- [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.1.27...v1.1.28)

Updates `github.com/projectdiscovery/fastdialer` from 0.4.1 to 0.4.2
- [Release notes](https://github.com/projectdiscovery/fastdialer/releases)
- [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.4.1...v0.4.2)

Updates `github.com/projectdiscovery/networkpolicy` from 0.1.17 to 0.1.18
- [Release notes](https://github.com/projectdiscovery/networkpolicy/releases)
- [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.17...v0.1.18)

Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.117 to 1.0.118
- [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases)
- [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.117...v1.0.118)

Updates `github.com/projectdiscovery/tlsx` from 1.1.9 to 1.2.0
- [Release notes](https://github.com/projectdiscovery/tlsx/releases)
- [Changelog](https://github.com/projectdiscovery/tlsx/blob/main/.goreleaser.yml)
- [Commits](https://github.com/projectdiscovery/tlsx/compare/v1.1.9...v1.2.0)

Updates `github.com/projectdiscovery/utils` from 0.4.21 to 0.4.22
- [Release notes](https://github.com/projectdiscovery/utils/releases)
- [Changelog](https://github.com/projectdiscovery/utils/blob/main/CHANGELOG.md)
- [Commits](https://github.com/projectdiscovery/utils/compare/v0.4.21...v0.4.22)

Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.37 to 0.2.38
- [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases)
- [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.37...v0.2.38)

---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/cdncheck
  dependency-version: 1.1.28
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/fastdialer
  dependency-version: 0.4.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/networkpolicy
  dependency-version: 0.1.18
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/retryablehttp-go
  dependency-version: 1.0.118
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/tlsx
  dependency-version: 1.2.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/utils
  dependency-version: 0.4.22
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/wappalyzergo
  dependency-version: 0.2.38
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
...

Signed-off-by: dependabot[bot] <support@github.com>

* lint

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
 2025-07-22 13:27:25 +05:30
yutianqaq fdd4d68f11 fix: resolve Traditional Chinese page garbled text issue 2025-06-28 14:53:29 +08:00
geeknik 9f61b8b8ef Update title.go 
Fix crash from malformed <title> tags on remote hosts.
 2025-06-24 10:50:13 -05:00
Kian-Meng Ang f08a003d29 Fix typos 
Found via `codespell -L ines,te,ue,occured,ans`
 2025-03-11 00:04:31 +08:00
Mzack9999 c52a923407 Refactoring proxy falg 2024-11-25 12:17:39 +01:00
Mzack9999 db72bb2902 Adding trace (#1883) 
* adding trace

* using retryablehttp

* syntax error

---------

Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
 2024-09-16 23:56:21 +05:30
Ramana Reddy 493e493cd2 use options timeout 2024-08-10 10:28:12 +05:30
Ramana Reddy 6c0c0e8320 fix ztls context deadline error 2024-08-08 16:05:07 +05:30
Ice3man 3b5554af36 feat: added enhancements to favicon + made API public (#1774) 
* more additions and enhancements to httpx

* feat: added enhancements to favicon + made API public

* update deps

* misc additions

* fix hasChain with 1 status code not working

* Revert "fix hasChain with 1 status code not working"

This reverts commit fabadcddbc.

---------

Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
 2024-06-24 19:07:26 +05:30
Ramana Reddy 9330887a58 Extract body_domains and body_fqdns to jsonl (#1750) 
* Extract body-domains and body-fqdns

* remvove port in domains

* Add test for domains extraction

* misc update

* improve domain regex

* fix test

* extract domain inside quotes

* sanitize urls

* fix test

* minor

* do not embed

* remove js variables fp + improve regex

---------

Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
 2024-06-23 01:01:43 +05:30
Ice3man 80d5532b15 misc fixing tests 2024-06-17 19:58:12 +05:30
Ice3man ca3e696834 feat: use networkpolicy + wappalyzer singletons from library 2024-06-17 18:11:01 +05:30
Ramana Reddy 8d31a457b8 fix edge cases 2024-05-30 16:52:12 +05:30
Ramana Reddy de9375e643 remove wildcard in domain 2024-05-30 15:03:15 +05:30
Ramana Reddy c5ca655207 exclude root domains from fqdn 2024-05-30 14:12:01 +05:30
Ramana Reddy ff262a084f Add extract-fqdn option 2024-05-30 09:16:23 +05:30
Ramana Reddy 91afa0cf76 Add fqdn to csp output 2024-05-30 08:34:58 +05:30
Mzack9999 7bb09b1fb1 File writing ops respect filters/matchers (#1720) 2024-05-29 20:45:19 +05:30
mzack 8b93f27dfe fixing example 2024-04-15 22:52:21 +02:00
mzack ce8c01891c adding http11 mode 2024-04-15 21:45:25 +02:00
Aviv Keller c72d98628e Introducing CanHaveTitleTag Function for MIME Type Validation (#1608) 
* chore(deps): bump golang.org/x/crypto from 0.14.0 to 0.17.0 (#1493)

Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.14.0 to 0.17.0.
- [Commits](https://github.com/golang/crypto/compare/v0.14.0...v0.17.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump github.com/cloudflare/circl from 1.3.3 to 1.3.7 (#1526)

Bumps [github.com/cloudflare/circl](https://github.com/cloudflare/circl) from 1.3.3 to 1.3.7.
- [Release notes](https://github.com/cloudflare/circl/releases)
- [Commits](https://github.com/cloudflare/circl/compare/v1.3.3...v1.3.7)

---
updated-dependencies:
- dependency-name: github.com/cloudflare/circl
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump github.com/quic-go/quic-go from 0.37.4 to 0.37.7 (#1531)

Bumps [github.com/quic-go/quic-go](https://github.com/quic-go/quic-go) from 0.37.4 to 0.37.7.
- [Release notes](https://github.com/quic-go/quic-go/releases)
- [Changelog](https://github.com/quic-go/quic-go/blob/master/Changelog.md)
- [Commits](https://github.com/quic-go/quic-go/compare/v0.37.4...v0.37.7)

---
updated-dependencies:
- dependency-name: github.com/quic-go/quic-go
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Update Title Extraction

* Make Requsted Changes

* Import slices

* optional asn

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
Co-authored-by: mzack <marco.rivoli.nvh@gmail.com>
 2024-03-23 21:47:40 +05:30
Dogan Can Bakir a4b026111a Fix content rel issues (#1630) 
* do not omit content length

* fix words and lines len
 2024-03-20 22:51:10 +05:30
Ramana Reddy dd8b546b35 Make cdn and tech-detect as default in json output (#1614) 
* Make cdn and tech-detect as default in json output

* Reflect changes only in jsonl output
 2024-03-06 17:16:58 +05:30
Dogan Can Bakir 2e35ad15af use networkpolicy (#1597) 
* use networkpolicy

* fix test
 2024-02-27 18:22:55 +05:30
Dogan Can Bakir b9a444795d Use useragent (#1562) 
* use projectdiscovery/useragent

* fix go.mod

* update useragent

* commit to last commit
 2024-01-31 20:53:17 +05:30
Mzack9999 7594c8bf5d Merge branch 'dev' into orig-1469 2024-01-15 22:03:56 +01:00
mzack 0061bc3a20 small description change 2024-01-15 21:57:41 +01:00
Ramana Reddy db1ea9fad3 Fix issue with host redirect on ports flag (#1529) 2024-01-15 15:27:24 +05:30
Ramana Reddy 64b6a0789c Fix: Pipeline and Websocket not working for some targets (#1528) 
* Fix pipeline issue for port included targets

* Add more checks to websocket
 2024-01-15 15:06:28 +05:30
mzack e557bf8662 using fastdialer deny 2024-01-07 21:10:56 +01:00
Karel f3a6eadfce Do not use local resolvers when given resolver input 2023-11-27 16:37:06 +01:00
Doğan Can Bakır d55d459f1d extend char set 2023-11-08 07:54:22 +00:00
Doğan Can Bakır 37591e1dd5 fix -title newline issue 2023-11-07 19:50:28 +00:00
Cody Cline 7dee9988ca Feature: optional flag to skip a host if it has a private IP address. (#1408) 
* feat: Implement flag which when enabled, will skip any host with a private ip address instead of timing out.

* fix: Amend private ip check to include loopback and link-local addresses

* fix: Amend debug message to be more concise

* fix: Account for port numbers in skipPrivateHosts feature

* docs: Remove uncessary  debug statement

* readme update

---------

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
 2023-11-02 14:21:22 +05:30