admin/projectdiscovery-httpx
1
0
Fork 0
mirror of https://github.com/projectdiscovery/httpx synced 2026-06-08 16:50:17 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,721 Commits 15 Branches 65 Tags
main
Commit Graph

161 Commits

Author SHA1 Message Date
Dogan Can Bakir 4311f8b736 Merge pull request #2363 from projectdiscovery/feature/database-output 
feat: add database output support for storing scan results
 2026-03-03 21:01:43 +03:00
Adel Assakaf fa950239f2 fix: prevent data loss on interrupt by implementing graceful shutdown 2026-02-05 05:03:40 +01:00
Doğan Can Bakır 2ea576f175 feat: add database output support for storing scan results 
Add support for storing httpx scan results directly to databases with
both CLI flags and YAML config file options.

Supported databases:
- MongoDB
- PostgreSQL
- MySQL

Features:
- Batched writes for performance (configurable batch size)
- Auto-flush with configurable interval
- Individual columns for each Result field (not JSON blob)
- Support for environment variable HTTPX_DB_CONNECTION_STRING
- Option to omit raw request/response data (-rdbor)

New CLI flags under OUTPUT group:
- -rdb, -result-db: enable database storage
- -rdbc, -result-db-config: path to YAML config file
- -rdbt, -result-db-type: database type (mongodb, postgres, mysql)
- -rdbcs, -result-db-conn: connection string
- -rdbn, -result-db-name: database name (default: httpx)
- -rdbtb, -result-db-table: table/collection name (default: results)
- -rdbbs, -result-db-batch-size: batch size (default: 100)
- -rdbor, -result-db-omit-raw: omit raw data

Closes #1973
Closes #2360
Closes #2361
Closes #2362
 2026-01-12 12:25:22 +03:00
Mzack9999 6ddf7ca9ac Enable screenshot upload (#2307) 
enable screenshot upload
 2025-11-14 23:51:58 +05:30
Mzack9999 27a22c6ac6 Fixing custom asset id upload (#2305) 
fixing custom asset id
 2025-11-08 02:23:37 +05:30
Mzack9999 2b688a5587 lint 2025-08-28 17:04:33 +02:00
dependabot[bot] f973821811 chore(deps): bump the modules group with 7 updates (#2216) 
* chore(deps): bump the modules group with 7 updates

Bumps the modules group with 7 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.1.27` | `1.1.28` |
| [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.4.1` | `0.4.2` |
| [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.17` | `0.1.18` |
| [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.0.117` | `1.0.118` |
| [github.com/projectdiscovery/tlsx](https://github.com/projectdiscovery/tlsx) | `1.1.9` | `1.2.0` |
| [github.com/projectdiscovery/utils](https://github.com/projectdiscovery/utils) | `0.4.21` | `0.4.22` |
| [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.37` | `0.2.38` |


Updates `github.com/projectdiscovery/cdncheck` from 1.1.27 to 1.1.28
- [Release notes](https://github.com/projectdiscovery/cdncheck/releases)
- [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml)
- [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.1.27...v1.1.28)

Updates `github.com/projectdiscovery/fastdialer` from 0.4.1 to 0.4.2
- [Release notes](https://github.com/projectdiscovery/fastdialer/releases)
- [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.4.1...v0.4.2)

Updates `github.com/projectdiscovery/networkpolicy` from 0.1.17 to 0.1.18
- [Release notes](https://github.com/projectdiscovery/networkpolicy/releases)
- [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.17...v0.1.18)

Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.117 to 1.0.118
- [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases)
- [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.117...v1.0.118)

Updates `github.com/projectdiscovery/tlsx` from 1.1.9 to 1.2.0
- [Release notes](https://github.com/projectdiscovery/tlsx/releases)
- [Changelog](https://github.com/projectdiscovery/tlsx/blob/main/.goreleaser.yml)
- [Commits](https://github.com/projectdiscovery/tlsx/compare/v1.1.9...v1.2.0)

Updates `github.com/projectdiscovery/utils` from 0.4.21 to 0.4.22
- [Release notes](https://github.com/projectdiscovery/utils/releases)
- [Changelog](https://github.com/projectdiscovery/utils/blob/main/CHANGELOG.md)
- [Commits](https://github.com/projectdiscovery/utils/compare/v0.4.21...v0.4.22)

Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.37 to 0.2.38
- [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases)
- [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.37...v0.2.38)

---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/cdncheck
  dependency-version: 1.1.28
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/fastdialer
  dependency-version: 0.4.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/networkpolicy
  dependency-version: 0.1.18
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/retryablehttp-go
  dependency-version: 1.0.118
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/tlsx
  dependency-version: 1.2.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/utils
  dependency-version: 0.4.22
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/wappalyzergo
  dependency-version: 0.2.38
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
...

Signed-off-by: dependabot[bot] <support@github.com>

* lint

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
 2025-07-22 13:27:25 +05:30
Ramana Reddy eb2b540324 Add team-id option (#1886) 
add team-id option
 2024-08-22 19:26:31 +05:30
Tarun Koyalwar 63b64cb4e6 feat: httpx results dashboard (#1834) 
* misc updates

* WIP: add dashboard support

* feat dashboard support

* screenshot with upload validation

* misc textual update

* interactive api key login

* feat: add -pdu local results file upload

* fix missing tech info in dashboard

* update flag description

---------

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
 2024-07-26 20:57:19 +05:30
Dogan Can Bakir 02558cacad change -tech-detect flag var type (#1702) 2024-05-15 17:36:47 +05:30
Ramana Reddy dd8b546b35 Make cdn and tech-detect as default in json output (#1614) 
* Make cdn and tech-detect as default in json output

* Reflect changes only in jsonl output
 2024-03-06 17:16:58 +05:30
mzack 13f76a94c0 adding sdk stream test 2024-02-05 21:45:21 +01:00
Doğan Can Bakır ae856b4028 fix func test 2024-01-04 15:27:54 +03:00
Doğan Can Bakır d55d459f1d extend char set 2023-11-08 07:54:22 +00:00
Ramana Reddy 76adadfeb4 Add flag to preview first N characters (#1316) 
* Add flag to preview first N characters

* use dynamic var for body-preview flag

* strip html tags from response

* add stripe option to remove html or xml tags

* typo

* resolve merge conflicts

* add test and update readme

* update readme

* typo

* typo no 2

* small refactor

* include body preview in jsonl output

* minor changes

---------

Co-authored-by: mzack <marco.rivoli.nvh@gmail.com>
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
 2023-09-07 15:26:54 +05:30
Dogan Can Bakir c3eb8897e7 Support multiple output format (#1242) 
* support multiple output format

* output plain text when -oA

* separate the conditions

* fail output-all when it's not used with output

* update doc

* remove hashes for plain output when -hash isn't used

* add integration test

* use filepath

* fix no hash std output

* remove json/csv extension for json/csv only output

* add ext when both -csv and -json is used

* fix assertion

* commit to last commit

* print filenames

* no tmpdir

* remove files

* remove prints

* lowercase options

* test update

---------

Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
 2023-07-17 16:09:10 +05:30
Mzack9999 8ea85e901d Adding prototype of tls impersonate (#1199) 
* Adding prototype of tls impersonate

* updating docs

* misc update

* dep update

---------

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
 2023-06-17 16:34:19 +05:30
shubhamrasal 827ed16461 remove fastdialer from runner object 2023-05-16 17:37:46 +05:30
Mzack9999 fbae1abf98 Fixing asn with ip:port 2023-02-28 11:35:27 +01:00
Tarun Koyalwar c7456929b4 fix integration tests and upgrade deps 2023-02-08 19:50:03 +05:30
xm1k3 88c49b812b added ztls flag (#896) 
* added ztls to httpx

* adding missing dialtls

* updated readme.md

* add ztls functional test

Co-authored-by: mzack <marco.rivoli.nvh@gmail.com>
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
 2022-12-28 23:40:22 +05:30
Mzack9999 986b2184c5 Enforcing wildcard check to prefix only (#911) 
enforcing wildcard check to prefix only
 2022-12-23 23:26:38 +05:30
vrenzolaverace 686a2cced8 Update helpers libraries #850 (#851) 
Use utils helpers libraries (#850)
 2022-11-07 01:52:47 +05:30
foxcores baa0fa0d29 Fix issue #849 2022-11-05 21:37:53 +01:00
foxcores e40367d904 Updating httpx as a library integration test (#847) 
* Update library.go

Go 1.18 + Some missing tests #826

* using standard http lib

Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
 2022-11-02 14:03:19 +05:30
Sajad e31c1250a2 add dsl match/filter condtion (#782) 
* add dsl match/filter condtion

* update README

* add test cases
 2022-10-06 17:11:12 +05:30
Sami d491201eff Issue 735+issue 739 (#757) 
* adding strings.ToLower to headers

* added validation logic for flags

* validated silent flag with incompatible flags

* silent flag bug fix

* added test case

* updated test case to include multiple headers
 2022-09-13 16:46:38 +05:30
Ice3man e236531eac Added profile-mem flag for memory profiling (#738) 2022-08-30 17:05:14 +05:30
forgedhallpass c48909f64f Minor fixes and improvements (#726) 
* Added .idea to .gitignore

* Updated the README

* Added more verbose information on preset extractors

* removing deprecated ioutil

Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
 2022-08-17 03:16:06 +04:00
Mzack9999 21ca2621bd Fixing and improving custom ports handling (#714) 
* Fixing and improving custom ports

* adding test case from https://github.com/projectdiscovery/httpx/pull/713
 2022-07-31 20:54:34 +05:30
Mzack9999 e4e2aab402 Adding httpx as a library minimal example (#670) 2022-06-28 16:19:46 +05:30
Mzack9999 32930d2cc6 Using scanme.sh in functional tests + Fixing gosec action error (#667) 
* Using scanme.sh in functional tests

* using gosec action
 2022-06-27 13:38:11 +05:30
Mzack9999 2c0802dc7b Fixing wrong app name (#663) 2022-06-13 17:48:28 +05:30
LuitelSamikshya db3a27a1ac frt/mrt improvemets 2022-05-09 15:01:38 -05:00
Mzack9999 5a562107f4 Updating GH workflows + Sonar (#514) 
* Updating GH workflows + Sonar

* adding missing step to build
 2022-02-09 14:39:30 +05:30
Mzack9999 f5935ebe61 Improving paths loading from file (#483) 
* Improving paths loading from file

* improving invalid uri handling
 2022-01-10 16:41:47 +05:30
LuitelSamikshya 69b0bb5fec test case addition 2021-11-29 13:16:37 -06:00
sandeep 56535375a5 testcases update 2021-11-18 19:12:17 +05:30
LuitelSamikshya a4512bcca6 removed commas and added missing arguments to flags 2021-11-17 10:24:14 -06:00
LuitelSamikshya 77ec79caaf github action for functional test 2021-11-16 18:23:43 -06:00
LuitelSamikshya 4d4db87346 functional test 2021-11-11 20:51:31 -06:00
mzack 20d51a1f75 Fixing stream mode with path parameter 2021-10-12 09:33:50 +02:00
LuitelSamikshya a39845bd39 Added regression test for POST request with body 2021-09-23 13:09:47 -05:00
mzack 4a2a6d31d1 adding more tests 2021-08-23 15:37:46 +02:00
mzack 020b719e4d Adding integration/regression tests 2021-08-23 12:50:37 +02:00
mzack b7afe803c3 Adding stop-resume support 2021-08-06 12:38:42 +02:00
mzack a786957e19 Making runner public 
Implements #258
 2021-05-06 10:35:02 +02:00
Ice3man543 63cb1277fe Fix conflict 2021-04-06 18:33:40 +05:30
Ice3man543 07bff31c45 Added wappalyzer based technology detection 2021-04-06 18:13:55 +05:30
Mzack9999 dcb7ee9adc gologger syntax refactor 2021-02-16 23:40:01 +01:00