admin/projectdiscovery-httpx
1
0
Fork 0
mirror of https://github.com/projectdiscovery/httpx synced 2026-06-08 16:50:17 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,721 Commits 15 Branches 65 Tags
main
Commit Graph

286 Commits

Author SHA1 Message Date
Doğan Can Bakır 1c4561ba07 feat: replace page classifier with dit, add -fpt flag 
Closes #2403
 2026-03-03 21:03:32 +03:00
Mzack9999 fb1ea1c9aa Merge branch 'dev' into feature/secret-file-auth 2026-01-12 15:41:54 +04:00
Mzack9999 a4ae407118 lint 2026-01-12 15:40:40 +04:00
Mzack9999 f445b0aa72 docs update + flags validation + debug logging 2026-01-12 15:34:24 +04:00
Doğan Can Bakır a780d059ee test: add comprehensive tests for auth provider 
- Test Cookie.Parse edge cases (equals in value, spaces, empty fields)
- Test Secret validation for all auth types
- Test file loading with different extensions (case-insensitive)
- Test all auth strategies (Apply and ApplyOnRR methods)
- Test FileAuthProvider domain lookup (exact and regex)
- Test MultiAuthProvider delegation

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-12 14:33:44 +03:00
Doğan Can Bakır 62eed15d6d fix: address CodeRabbitAI review comments 
- Fix slices.Delete while iterating bug in cookies_auth.go
- Fix Cookie.Parse to handle '=' chars in values using SplitN
- Add case-insensitive file extension handling in GetAuthDataFromFile
- Remove unreachable default case in Secret.Validate
- Make LookupURL/LookupURLX consistent with LookupAddr (use len>0)
- Improve comments for clarity

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-12 14:31:40 +03:00
Doğan Can Bakır 430e33f9a1 fix: address coderabbit review comments 
- Handle error from format.Parse in streamInput
- Add InputMode validation to ValidateOptions for early error detection
- Extract duplicate format validation to getInputFormat helper
- Fix shadow variable err in loadFromFormat callback
- Add bounds check in test before index access
- Add test for malformed XML input
- Run go mod tidy to fix dependency marking
 2026-01-12 14:23:54 +03:00
Doğan Can Bakır 0f5141b0d7 feat: add burp xml input mode support (-im flag) 2026-01-12 14:13:31 +03:00
Doğan Can Bakır b60bfa24db feat: add secret file authentication support (-sf flag) 2026-01-12 14:03:07 +03:00
Mzack9999 7e6003ca51 fixing comment 2025-12-24 15:11:53 +04:00
Mzack9999 78bf9c393a restoring original logic + limiting read to 512Mb + lint 2025-12-24 15:00:10 +04:00
Connor Larson 85695760cb fix: removed temp logger 2025-12-22 12:37:23 -07:00
Connor Larson ec2b7154bc changed resp.Raw contstruction to cap the body, avoiding oom issue 2025-12-22 12:22:10 -07:00
Sandeep Singh 599441ef15 fix: HTML parser panic protection with multiple fallback (#2330) 
fix: enhance HTML parser panic protection with multiple fallback strategies

- Add ultra-aggressive HTML sanitization to reduce nesting depth
- Implement size limiting (1MB) to prevent processing huge documents
- Add plain text extraction fallback for complex HTML structures
- Enhance panic recovery with comprehensive error handling
- Remove deeply nestable elements (div, span, ul, ol, li) from sanitizer
- Add comprehensive test coverage for edge cases

Resolves HTML parser panic: 'html: open stack of elements exceeds 512 nodes'
that occurred after switching to html-to-markdown/v2 library in PR #2255
 2025-12-06 23:39:09 +05:30
Sandeep Singh d11472205c feat: update -ldp option to show default ports in CLI output (#2331) 
feat: update -ldp option to show default ports in CLI output

- Modified URL formatting in runner.go to respect LeaveDefaultPorts option
- Fixed AddURLDefaultPort function to actually add default ports (80/443)
- When -ldp is used, CLI output now shows https://example.com:443 instead of https://example.com
- Maintains backward compatibility - default behavior unchanged

Fixes CLI output inconsistency where -ldp flag only affected Host headers
but not the displayed URL in console output.
 2025-12-06 22:52:16 +05:30
Mzack9999 275d63fa0b fix test 2025-11-28 19:12:40 +04:00
Mzack9999 5f13beb39c lint 2025-11-20 14:55:35 +04:00
Mzack9999 0309110c89 adding panic guard + tests 2025-11-20 14:47:47 +04:00
Mzack9999 05c6364d90 better error handling 2025-11-18 16:46:15 +04:00
@gdattacker 7c94f19571 Improve error handling in htmlToText function 
Enhance htmlToText function to handle panics and errors safely.

panic: html: open stack of elements exceeds 512 nodes

goroutine 5523922 [running]:
github.com/projectdiscovery/httpx/common/pagetypeclassifier.htmlToText(...)
	/home/runner/work/httpx/httpx/common/pagetypeclassifier/pagetypeclassifier.go:36
github.com/projectdiscovery/httpx/common/pagetypeclassifier.(*PageTypeClassifier).Classify(0xc0005164d8, {0xc0ba03a000?, 0xd?})
	/home/runner/work/httpx/httpx/common/pagetypeclassifier/pagetypeclassifier.go:26 +0x6f
github.com/projectdiscovery/httpx/runner.(*Runner).analyze(_, _, {_, _}, {{0xc00470c450, 0xb}, {0x0, 0x0}, {0x0, 0x0}}, ...)
	/home/runner/work/httpx/httpx/runner/runner.go:2349 +0x7555
github.com/projectdiscovery/httpx/runner.(*Runner).process.func1({{0xc00470c450, 0xb}, {0x0, 0x0}, {0x0, 0x0}}, {0x1686161?, 0x10?}, {0x16ace2d, 0xa})
	/home/runner/work/httpx/httpx/runner/runner.go:1444 +0x125
created by github.com/projectdiscovery/httpx/runner.(*Runner).process in goroutine 1
	/home/runner/work/httpx/httpx/runner/runner.go:1442 +0x8a6
 2025-11-17 08:55:50 +05:30
ayanrajpoot10 497c39920d fix: ensure Host header is set in unsafe mode for rawhttp requests 2025-10-22 20:44:53 +05:30
Sandeep Singh 8d6c265866 Html2text change library (#2255) 
* change library for html2text

* misc

* misc

---------

Co-authored-by: Ice3man <nizamulrana@gmail.com>
 2025-09-02 23:14:07 +05:30
Mzack9999 2b688a5587 lint 2025-08-28 17:04:33 +02:00
Mzack9999 f2881db0cb improving autoref 2025-08-28 16:46:38 +02:00
siamak a.m.o f0e464f6ea Implement auto-referer option 
The option -auto-referer, sets the referer HTTP header
of the current request to it's URL.
 2025-08-18 12:08:42 +03:30
dependabot[bot] f973821811 chore(deps): bump the modules group with 7 updates (#2216) 
* chore(deps): bump the modules group with 7 updates

Bumps the modules group with 7 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.1.27` | `1.1.28` |
| [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.4.1` | `0.4.2` |
| [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.17` | `0.1.18` |
| [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.0.117` | `1.0.118` |
| [github.com/projectdiscovery/tlsx](https://github.com/projectdiscovery/tlsx) | `1.1.9` | `1.2.0` |
| [github.com/projectdiscovery/utils](https://github.com/projectdiscovery/utils) | `0.4.21` | `0.4.22` |
| [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.37` | `0.2.38` |


Updates `github.com/projectdiscovery/cdncheck` from 1.1.27 to 1.1.28
- [Release notes](https://github.com/projectdiscovery/cdncheck/releases)
- [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml)
- [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.1.27...v1.1.28)

Updates `github.com/projectdiscovery/fastdialer` from 0.4.1 to 0.4.2
- [Release notes](https://github.com/projectdiscovery/fastdialer/releases)
- [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.4.1...v0.4.2)

Updates `github.com/projectdiscovery/networkpolicy` from 0.1.17 to 0.1.18
- [Release notes](https://github.com/projectdiscovery/networkpolicy/releases)
- [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.17...v0.1.18)

Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.117 to 1.0.118
- [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases)
- [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.117...v1.0.118)

Updates `github.com/projectdiscovery/tlsx` from 1.1.9 to 1.2.0
- [Release notes](https://github.com/projectdiscovery/tlsx/releases)
- [Changelog](https://github.com/projectdiscovery/tlsx/blob/main/.goreleaser.yml)
- [Commits](https://github.com/projectdiscovery/tlsx/compare/v1.1.9...v1.2.0)

Updates `github.com/projectdiscovery/utils` from 0.4.21 to 0.4.22
- [Release notes](https://github.com/projectdiscovery/utils/releases)
- [Changelog](https://github.com/projectdiscovery/utils/blob/main/CHANGELOG.md)
- [Commits](https://github.com/projectdiscovery/utils/compare/v0.4.21...v0.4.22)

Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.37 to 0.2.38
- [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases)
- [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.37...v0.2.38)

---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/cdncheck
  dependency-version: 1.1.28
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/fastdialer
  dependency-version: 0.4.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/networkpolicy
  dependency-version: 0.1.18
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/retryablehttp-go
  dependency-version: 1.0.118
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/tlsx
  dependency-version: 1.2.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/utils
  dependency-version: 0.4.22
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/wappalyzergo
  dependency-version: 0.2.38
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
...

Signed-off-by: dependabot[bot] <support@github.com>

* lint

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
 2025-07-22 13:27:25 +05:30
yutianqaq fdd4d68f11 fix: resolve Traditional Chinese page garbled text issue 2025-06-28 14:53:29 +08:00
geeknik 9f61b8b8ef Update title.go 
Fix crash from malformed <title> tags on remote hosts.
 2025-06-24 10:50:13 -05:00
Kian-Meng Ang f08a003d29 Fix typos 
Found via `codespell -L ines,te,ue,occured,ans`
 2025-03-11 00:04:31 +08:00
Mzack9999 c52a923407 Refactoring proxy falg 2024-11-25 12:17:39 +01:00
Dogan Can Bakir 0c9bba73d7 introduce page type classifier (#1924) 
* introduce page type classifier

* add more data
 2024-10-21 02:30:36 +05:30
Mzack9999 db72bb2902 Adding trace (#1883) 
* adding trace

* using retryablehttp

* syntax error

---------

Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
 2024-09-16 23:56:21 +05:30
Mzack9999 f07cf4c288 Merge pull request #1860 from projectdiscovery/issue-1858-fix-ztls-ctx-err 
fix `ztls` context deadline error
 2024-08-12 21:10:52 +02:00
Ramana Reddy 493e493cd2 use options timeout 2024-08-10 10:28:12 +05:30
Ramana Reddy 1af4647336 fix missing favicon hash for targets (#1848) 
* fix missing favicon hash for targets

* improve base64 favicon check
 2024-08-10 00:24:33 +05:30
Ramana Reddy 6c0c0e8320 fix ztls context deadline error 2024-08-08 16:05:07 +05:30
Dogan Can Bakir 7ed4cd19ae add md5 hash support for favicon (#1799) 2024-07-10 18:43:14 +05:30
Ice3man 3b5554af36 feat: added enhancements to favicon + made API public (#1774) 
* more additions and enhancements to httpx

* feat: added enhancements to favicon + made API public

* update deps

* misc additions

* fix hasChain with 1 status code not working

* Revert "fix hasChain with 1 status code not working"

This reverts commit fabadcddbc.

---------

Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
 2024-06-24 19:07:26 +05:30
Ramana Reddy 9330887a58 Extract body_domains and body_fqdns to jsonl (#1750) 
* Extract body-domains and body-fqdns

* remvove port in domains

* Add test for domains extraction

* misc update

* improve domain regex

* fix test

* extract domain inside quotes

* sanitize urls

* fix test

* minor

* do not embed

* remove js variables fp + improve regex

---------

Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
 2024-06-23 01:01:43 +05:30
Ice3man 80d5532b15 misc fixing tests 2024-06-17 19:58:12 +05:30
Ice3man ca3e696834 feat: use networkpolicy + wappalyzer singletons from library 2024-06-17 18:11:01 +05:30
Ramana Reddy 8d31a457b8 fix edge cases 2024-05-30 16:52:12 +05:30
Ramana Reddy de9375e643 remove wildcard in domain 2024-05-30 15:03:15 +05:30
Ramana Reddy c5ca655207 exclude root domains from fqdn 2024-05-30 14:12:01 +05:30
Ramana Reddy ff262a084f Add extract-fqdn option 2024-05-30 09:16:23 +05:30
Ramana Reddy 91afa0cf76 Add fqdn to csp output 2024-05-30 08:34:58 +05:30
Mzack9999 7bb09b1fb1 File writing ops respect filters/matchers (#1720) 2024-05-29 20:45:19 +05:30
mzack 8b93f27dfe fixing example 2024-04-15 22:52:21 +02:00
mzack ce8c01891c adding http11 mode 2024-04-15 21:45:25 +02:00
Aviv Keller c72d98628e Introducing CanHaveTitleTag Function for MIME Type Validation (#1608) 
* chore(deps): bump golang.org/x/crypto from 0.14.0 to 0.17.0 (#1493)

Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.14.0 to 0.17.0.
- [Commits](https://github.com/golang/crypto/compare/v0.14.0...v0.17.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump github.com/cloudflare/circl from 1.3.3 to 1.3.7 (#1526)

Bumps [github.com/cloudflare/circl](https://github.com/cloudflare/circl) from 1.3.3 to 1.3.7.
- [Release notes](https://github.com/cloudflare/circl/releases)
- [Commits](https://github.com/cloudflare/circl/compare/v1.3.3...v1.3.7)

---
updated-dependencies:
- dependency-name: github.com/cloudflare/circl
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump github.com/quic-go/quic-go from 0.37.4 to 0.37.7 (#1531)

Bumps [github.com/quic-go/quic-go](https://github.com/quic-go/quic-go) from 0.37.4 to 0.37.7.
- [Release notes](https://github.com/quic-go/quic-go/releases)
- [Changelog](https://github.com/quic-go/quic-go/blob/master/Changelog.md)
- [Commits](https://github.com/quic-go/quic-go/compare/v0.37.4...v0.37.7)

---
updated-dependencies:
- dependency-name: github.com/quic-go/quic-go
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Update Title Extraction

* Make Requsted Changes

* Import slices

* optional asn

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
Co-authored-by: mzack <marco.rivoli.nvh@gmail.com>
 2024-03-23 21:47:40 +05:30