admin/projectdiscovery-httpx
1
0
Fork 0
mirror of https://github.com/projectdiscovery/httpx synced 2026-06-08 16:50:17 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,774 Commits 15 Branches 65 Tags
dev
Commit Graph

294 Commits

Author SHA1 Message Date
Ayan Rajpoot 7fd08b7f8d Add support for duplicate custom request headers (#2504) 
* refactor: change CustomHeaders to support multiple values per header

* fix: clear existing header values before setting custom headers

* harden headers

* expand tests

---------

Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
 2026-06-04 14:21:55 +02:00
Kral d0081af3ec fix: improve error handling and fix bugs (#2499) 
* code quality: improve error handling and fix bugs

- Fix bug: return err instead of closeErr in DecodeData error path (httpx.go)
- Improve error handling for strconv.Atoi calls in multiple locations
- Change hammingDistanceThreshold from var to const since it's never modified
- Fix redundant return statement in ListFilesWithPattern
- Improve port parsing to handle empty port strings explicitly

* runner: keep input counters incrementing under -skip-dedupe

Revert the defensive strconv.Atoi guard around the SkipDedupe branches:
values stored in r.hm are always integer strings written by this same
code path, so the parseErr/cnt>0 check is unreachable in practice, and
when taken literally it skipped the numHosts/numTargets increment for
duplicate inputs.

---------

Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
 2026-05-20 06:41:57 +02:00
Dogan Can Bakir 6c34d93168 feat: honor HTTP_PROXY/HTTPS_PROXY env vars when -proxy is unset (#2493) 
Falls back to http.ProxyFromEnvironment for the HTTP transport and
reads the same vars manually for the headless launcher (chromedp
does not read env).

Closes #2492
 2026-05-13 21:24:40 +02:00
Mzack9999 9836829c62 Merge pull request #2455 from tal7aouy/fix/unsafe-head-nil-pointer 
fix: prevent nil panic for unsafe HEAD requests
 2026-03-21 00:10:59 +01:00
Mzack9999 6425eb5ed0 fmt 2026-03-21 00:01:56 +01:00
Mzack9999 ecef472ab6 Merge pull request #2424 from Veirt/fix/http11-retry-fallback-2240 
fix(httpx): keep retry fallback on HTTP/1.1 when -pr http11 is set
 2026-03-19 16:49:49 +01:00
Mhammed Talhaouy 8ae53cb9ae fix: prevent nil panic for unsafe HEAD requests 2026-03-19 00:12:21 +00:00
Doğan Can Bakır 1c4561ba07 feat: replace page classifier with dit, add -fpt flag 
Closes #2403
 2026-03-03 21:03:32 +03:00
Veirt b7dabf7456 fix(httpx): keep retry fallback on HTTP/1.1 when -pr http11 is set 2026-02-28 02:45:47 +08:00
Mzack9999 fb1ea1c9aa Merge branch 'dev' into feature/secret-file-auth 2026-01-12 15:41:54 +04:00
Mzack9999 a4ae407118 lint 2026-01-12 15:40:40 +04:00
Mzack9999 f445b0aa72 docs update + flags validation + debug logging 2026-01-12 15:34:24 +04:00
Doğan Can Bakır a780d059ee test: add comprehensive tests for auth provider 
- Test Cookie.Parse edge cases (equals in value, spaces, empty fields)
- Test Secret validation for all auth types
- Test file loading with different extensions (case-insensitive)
- Test all auth strategies (Apply and ApplyOnRR methods)
- Test FileAuthProvider domain lookup (exact and regex)
- Test MultiAuthProvider delegation

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-12 14:33:44 +03:00
Doğan Can Bakır 62eed15d6d fix: address CodeRabbitAI review comments 
- Fix slices.Delete while iterating bug in cookies_auth.go
- Fix Cookie.Parse to handle '=' chars in values using SplitN
- Add case-insensitive file extension handling in GetAuthDataFromFile
- Remove unreachable default case in Secret.Validate
- Make LookupURL/LookupURLX consistent with LookupAddr (use len>0)
- Improve comments for clarity

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-12 14:31:40 +03:00
Doğan Can Bakır 430e33f9a1 fix: address coderabbit review comments 
- Handle error from format.Parse in streamInput
- Add InputMode validation to ValidateOptions for early error detection
- Extract duplicate format validation to getInputFormat helper
- Fix shadow variable err in loadFromFormat callback
- Add bounds check in test before index access
- Add test for malformed XML input
- Run go mod tidy to fix dependency marking
 2026-01-12 14:23:54 +03:00
Doğan Can Bakır 0f5141b0d7 feat: add burp xml input mode support (-im flag) 2026-01-12 14:13:31 +03:00
Doğan Can Bakır b60bfa24db feat: add secret file authentication support (-sf flag) 2026-01-12 14:03:07 +03:00
Mzack9999 7e6003ca51 fixing comment 2025-12-24 15:11:53 +04:00
Mzack9999 78bf9c393a restoring original logic + limiting read to 512Mb + lint 2025-12-24 15:00:10 +04:00
Connor Larson 85695760cb fix: removed temp logger 2025-12-22 12:37:23 -07:00
Connor Larson ec2b7154bc changed resp.Raw contstruction to cap the body, avoiding oom issue 2025-12-22 12:22:10 -07:00
Sandeep Singh 599441ef15 fix: HTML parser panic protection with multiple fallback (#2330) 
fix: enhance HTML parser panic protection with multiple fallback strategies

- Add ultra-aggressive HTML sanitization to reduce nesting depth
- Implement size limiting (1MB) to prevent processing huge documents
- Add plain text extraction fallback for complex HTML structures
- Enhance panic recovery with comprehensive error handling
- Remove deeply nestable elements (div, span, ul, ol, li) from sanitizer
- Add comprehensive test coverage for edge cases

Resolves HTML parser panic: 'html: open stack of elements exceeds 512 nodes'
that occurred after switching to html-to-markdown/v2 library in PR #2255
 2025-12-06 23:39:09 +05:30
Sandeep Singh d11472205c feat: update -ldp option to show default ports in CLI output (#2331) 
feat: update -ldp option to show default ports in CLI output

- Modified URL formatting in runner.go to respect LeaveDefaultPorts option
- Fixed AddURLDefaultPort function to actually add default ports (80/443)
- When -ldp is used, CLI output now shows https://example.com:443 instead of https://example.com
- Maintains backward compatibility - default behavior unchanged

Fixes CLI output inconsistency where -ldp flag only affected Host headers
but not the displayed URL in console output.
 2025-12-06 22:52:16 +05:30
Mzack9999 275d63fa0b fix test 2025-11-28 19:12:40 +04:00
Mzack9999 5f13beb39c lint 2025-11-20 14:55:35 +04:00
Mzack9999 0309110c89 adding panic guard + tests 2025-11-20 14:47:47 +04:00
Mzack9999 05c6364d90 better error handling 2025-11-18 16:46:15 +04:00
@gdattacker 7c94f19571 Improve error handling in htmlToText function 
Enhance htmlToText function to handle panics and errors safely.

panic: html: open stack of elements exceeds 512 nodes

goroutine 5523922 [running]:
github.com/projectdiscovery/httpx/common/pagetypeclassifier.htmlToText(...)
	/home/runner/work/httpx/httpx/common/pagetypeclassifier/pagetypeclassifier.go:36
github.com/projectdiscovery/httpx/common/pagetypeclassifier.(*PageTypeClassifier).Classify(0xc0005164d8, {0xc0ba03a000?, 0xd?})
	/home/runner/work/httpx/httpx/common/pagetypeclassifier/pagetypeclassifier.go:26 +0x6f
github.com/projectdiscovery/httpx/runner.(*Runner).analyze(_, _, {_, _}, {{0xc00470c450, 0xb}, {0x0, 0x0}, {0x0, 0x0}}, ...)
	/home/runner/work/httpx/httpx/runner/runner.go:2349 +0x7555
github.com/projectdiscovery/httpx/runner.(*Runner).process.func1({{0xc00470c450, 0xb}, {0x0, 0x0}, {0x0, 0x0}}, {0x1686161?, 0x10?}, {0x16ace2d, 0xa})
	/home/runner/work/httpx/httpx/runner/runner.go:1444 +0x125
created by github.com/projectdiscovery/httpx/runner.(*Runner).process in goroutine 1
	/home/runner/work/httpx/httpx/runner/runner.go:1442 +0x8a6
 2025-11-17 08:55:50 +05:30
ayanrajpoot10 497c39920d fix: ensure Host header is set in unsafe mode for rawhttp requests 2025-10-22 20:44:53 +05:30
Sandeep Singh 8d6c265866 Html2text change library (#2255) 
* change library for html2text

* misc

* misc

---------

Co-authored-by: Ice3man <nizamulrana@gmail.com>
 2025-09-02 23:14:07 +05:30
Mzack9999 2b688a5587 lint 2025-08-28 17:04:33 +02:00
Mzack9999 f2881db0cb improving autoref 2025-08-28 16:46:38 +02:00
siamak a.m.o f0e464f6ea Implement auto-referer option 
The option -auto-referer, sets the referer HTTP header
of the current request to it's URL.
 2025-08-18 12:08:42 +03:30
dependabot[bot] f973821811 chore(deps): bump the modules group with 7 updates (#2216) 
* chore(deps): bump the modules group with 7 updates

Bumps the modules group with 7 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.1.27` | `1.1.28` |
| [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.4.1` | `0.4.2` |
| [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.17` | `0.1.18` |
| [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.0.117` | `1.0.118` |
| [github.com/projectdiscovery/tlsx](https://github.com/projectdiscovery/tlsx) | `1.1.9` | `1.2.0` |
| [github.com/projectdiscovery/utils](https://github.com/projectdiscovery/utils) | `0.4.21` | `0.4.22` |
| [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.37` | `0.2.38` |


Updates `github.com/projectdiscovery/cdncheck` from 1.1.27 to 1.1.28
- [Release notes](https://github.com/projectdiscovery/cdncheck/releases)
- [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml)
- [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.1.27...v1.1.28)

Updates `github.com/projectdiscovery/fastdialer` from 0.4.1 to 0.4.2
- [Release notes](https://github.com/projectdiscovery/fastdialer/releases)
- [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.4.1...v0.4.2)

Updates `github.com/projectdiscovery/networkpolicy` from 0.1.17 to 0.1.18
- [Release notes](https://github.com/projectdiscovery/networkpolicy/releases)
- [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.17...v0.1.18)

Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.117 to 1.0.118
- [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases)
- [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.117...v1.0.118)

Updates `github.com/projectdiscovery/tlsx` from 1.1.9 to 1.2.0
- [Release notes](https://github.com/projectdiscovery/tlsx/releases)
- [Changelog](https://github.com/projectdiscovery/tlsx/blob/main/.goreleaser.yml)
- [Commits](https://github.com/projectdiscovery/tlsx/compare/v1.1.9...v1.2.0)

Updates `github.com/projectdiscovery/utils` from 0.4.21 to 0.4.22
- [Release notes](https://github.com/projectdiscovery/utils/releases)
- [Changelog](https://github.com/projectdiscovery/utils/blob/main/CHANGELOG.md)
- [Commits](https://github.com/projectdiscovery/utils/compare/v0.4.21...v0.4.22)

Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.37 to 0.2.38
- [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases)
- [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.37...v0.2.38)

---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/cdncheck
  dependency-version: 1.1.28
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/fastdialer
  dependency-version: 0.4.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/networkpolicy
  dependency-version: 0.1.18
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/retryablehttp-go
  dependency-version: 1.0.118
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/tlsx
  dependency-version: 1.2.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/utils
  dependency-version: 0.4.22
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/wappalyzergo
  dependency-version: 0.2.38
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
...

Signed-off-by: dependabot[bot] <support@github.com>

* lint

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
 2025-07-22 13:27:25 +05:30
yutianqaq fdd4d68f11 fix: resolve Traditional Chinese page garbled text issue 2025-06-28 14:53:29 +08:00
geeknik 9f61b8b8ef Update title.go 
Fix crash from malformed <title> tags on remote hosts.
 2025-06-24 10:50:13 -05:00
Kian-Meng Ang f08a003d29 Fix typos 
Found via `codespell -L ines,te,ue,occured,ans`
 2025-03-11 00:04:31 +08:00
Mzack9999 c52a923407 Refactoring proxy falg 2024-11-25 12:17:39 +01:00
Dogan Can Bakir 0c9bba73d7 introduce page type classifier (#1924) 
* introduce page type classifier

* add more data
 2024-10-21 02:30:36 +05:30
Mzack9999 db72bb2902 Adding trace (#1883) 
* adding trace

* using retryablehttp

* syntax error

---------

Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
 2024-09-16 23:56:21 +05:30
Mzack9999 f07cf4c288 Merge pull request #1860 from projectdiscovery/issue-1858-fix-ztls-ctx-err 
fix `ztls` context deadline error
 2024-08-12 21:10:52 +02:00
Ramana Reddy 493e493cd2 use options timeout 2024-08-10 10:28:12 +05:30
Ramana Reddy 1af4647336 fix missing favicon hash for targets (#1848) 
* fix missing favicon hash for targets

* improve base64 favicon check
 2024-08-10 00:24:33 +05:30
Ramana Reddy 6c0c0e8320 fix ztls context deadline error 2024-08-08 16:05:07 +05:30
Dogan Can Bakir 7ed4cd19ae add md5 hash support for favicon (#1799) 2024-07-10 18:43:14 +05:30
Ice3man 3b5554af36 feat: added enhancements to favicon + made API public (#1774) 
* more additions and enhancements to httpx

* feat: added enhancements to favicon + made API public

* update deps

* misc additions

* fix hasChain with 1 status code not working

* Revert "fix hasChain with 1 status code not working"

This reverts commit fabadcddbc.

---------

Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
 2024-06-24 19:07:26 +05:30
Ramana Reddy 9330887a58 Extract body_domains and body_fqdns to jsonl (#1750) 
* Extract body-domains and body-fqdns

* remvove port in domains

* Add test for domains extraction

* misc update

* improve domain regex

* fix test

* extract domain inside quotes

* sanitize urls

* fix test

* minor

* do not embed

* remove js variables fp + improve regex

---------

Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
 2024-06-23 01:01:43 +05:30
Ice3man 80d5532b15 misc fixing tests 2024-06-17 19:58:12 +05:30
Ice3man ca3e696834 feat: use networkpolicy + wappalyzer singletons from library 2024-06-17 18:11:01 +05:30
Ramana Reddy 8d31a457b8 fix edge cases 2024-05-30 16:52:12 +05:30