Add support for storing httpx scan results directly to databases with
both CLI flags and YAML config file options.
Supported databases:
- MongoDB
- PostgreSQL
- MySQL
Features:
- Batched writes for performance (configurable batch size)
- Auto-flush with configurable interval
- Individual columns for each Result field (not JSON blob)
- Support for environment variable HTTPX_DB_CONNECTION_STRING
- Option to omit raw request/response data (-rdbor)
New CLI flags under OUTPUT group:
- -rdb, -result-db: enable database storage
- -rdbc, -result-db-config: path to YAML config file
- -rdbt, -result-db-type: database type (mongodb, postgres, mysql)
- -rdbcs, -result-db-conn: connection string
- -rdbn, -result-db-name: database name (default: httpx)
- -rdbtb, -result-db-table: table/collection name (default: results)
- -rdbbs, -result-db-batch-size: batch size (default: 100)
- -rdbor, -result-db-omit-raw: omit raw data
Closes#1973Closes#2360Closes#2361Closes#2362
Add support for passive detection of CPE (Common Platform Enumeration)
identifiers and WordPress plugins/themes using awesome-search-queries.
CPE Detection (-cpe flag):
- Matches response title, body, and favicon hash against patterns
- Extracts product, vendor, and generates CPE 2.3 identifiers
- Uses patterns from Shodan, FOFA, Google dorks
WordPress Detection (-wp flag):
- Detects plugins via /wp-content/plugins/[name]/ patterns
- Detects themes via /wp-content/themes/[name]/ patterns
- Validates against known plugins/themes list
New CLI flags in PROBES group:
- -cpe: display CPE based on awesome-search-queries
- -wp, -wordpress: display WordPress plugins and themes
Both are automatically included in JSON/CSV output.
Closes#1975
* feat: Implement flag which when enabled, will skip any host with a private ip address instead of timing out.
* fix: Amend private ip check to include loopback and link-local addresses
* fix: Amend debug message to be more concise
* fix: Account for port numbers in skipPrivateHosts feature
* docs: Remove uncessary debug statement
* readme update
---------
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
* Add flag to preview first N characters
* use dynamic var for body-preview flag
* strip html tags from response
* add stripe option to remove html or xml tags
* typo
* resolve merge conflicts
* add test and update readme
* update readme
* typo
* typo no 2
* small refactor
* include body preview in jsonl output
* minor changes
---------
Co-authored-by: mzack <marco.rivoli.nvh@gmail.com>
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
* support multiple output format
* output plain text when -oA
* separate the conditions
* fail output-all when it's not used with output
* update doc
* remove hashes for plain output when -hash isn't used
* add integration test
* use filepath
* fix no hash std output
* remove json/csv extension for json/csv only output
* add ext when both -csv and -json is used
* fix assertion
* commit to last commit
* print filenames
* no tmpdir
* remove files
* remove prints
* lowercase options
* test update
---------
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
husler