* code quality: improve error handling and fix bugs
- Fix bug: return err instead of closeErr in DecodeData error path (httpx.go)
- Improve error handling for strconv.Atoi calls in multiple locations
- Change hammingDistanceThreshold from var to const since it's never modified
- Fix redundant return statement in ListFilesWithPattern
- Improve port parsing to handle empty port strings explicitly
* runner: keep input counters incrementing under -skip-dedupe
Revert the defensive strconv.Atoi guard around the SkipDedupe branches:
values stored in r.hm are always integer strings written by this same
code path, so the parseErr/cnt>0 check is unreachable in practice, and
when taken literally it skipped the numHosts/numTargets increment for
duplicate inputs.
---------
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
Falls back to http.ProxyFromEnvironment for the HTTP transport and
reads the same vars manually for the headless launcher (chromedp
does not read env).
Closes#2492
* fix(db): persist CPE in postgres and mysql (#2487)
Adds cpe column to both SQL schemas, idempotent migration for
existing tables, and JSON binding in InsertBatch.
* adding comment note
* fixing lint
---------
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
testAndSet performs a non-atomic read-then-write sequence:
1. seen(k) - check if key exists
2. setSeen(k) - set the key
Without synchronization, two concurrent goroutines can both pass the
seen() check before either calls setSeen(), causing duplicate
processing of the same target.
This race can occur in processTargets() where multiple goroutines
call testAndSet() for discovered TLS SubjectAN/SubjectCN values.
The fix adds a sync.Mutex to serialize testAndSet operations.
- Handle error from format.Parse in streamInput
- Add InputMode validation to ValidateOptions for early error detection
- Extract duplicate format validation to getInputFormat helper
- Fix shadow variable err in loadFromFormat callback
- Add bounds check in test before index access
- Add test for malformed XML input
- Run go mod tidy to fix dependency marking
Add support for storing httpx scan results directly to databases with
both CLI flags and YAML config file options.
Supported databases:
- MongoDB
- PostgreSQL
- MySQL
Features:
- Batched writes for performance (configurable batch size)
- Auto-flush with configurable interval
- Individual columns for each Result field (not JSON blob)
- Support for environment variable HTTPX_DB_CONNECTION_STRING
- Option to omit raw request/response data (-rdbor)
New CLI flags under OUTPUT group:
- -rdb, -result-db: enable database storage
- -rdbc, -result-db-config: path to YAML config file
- -rdbt, -result-db-type: database type (mongodb, postgres, mysql)
- -rdbcs, -result-db-conn: connection string
- -rdbn, -result-db-name: database name (default: httpx)
- -rdbtb, -result-db-table: table/collection name (default: results)
- -rdbbs, -result-db-batch-size: batch size (default: 100)
- -rdbor, -result-db-omit-raw: omit raw data
Closes#1973Closes#2360Closes#2361Closes#2362
When FollowRedirects is enabled and the target redirects to a different host
(e.g., hackerone.com -> www.hackerone.com), the favicon URL is correctly
resolved to the final host, but the cloned request's Host header still
contains the original host. This causes the server to return 404 because
the Host header doesn't match the URL.
This fix updates the Host header to match the resolved URL host before
making the favicon request.
Add support for passive detection of CPE (Common Platform Enumeration)
identifiers and WordPress plugins/themes using awesome-search-queries.
CPE Detection (-cpe flag):
- Matches response title, body, and favicon hash against patterns
- Extracts product, vendor, and generates CPE 2.3 identifiers
- Uses patterns from Shodan, FOFA, Google dorks
WordPress Detection (-wp flag):
- Detects plugins via /wp-content/plugins/[name]/ patterns
- Detects themes via /wp-content/themes/[name]/ patterns
- Validates against known plugins/themes list
New CLI flags in PROBES group:
- -cpe: display CPE based on awesome-search-queries
- -wp, -wordpress: display WordPress plugins and themes
Both are automatically included in JSON/CSV output.
Closes#1975
Use URL.Hostname() instead of URL.Host to extract the hostname for DNS
resolution. This ensures the port number is not included in the DNS
query when probing all IPs for a URL like http://example.com:8080.
Fixes#2346
Signed-off-by: majiayu000 <1835304752@qq.com>
* Consolidate output‑type checks into a single `jsonOrCsvOrMD` variable.
* Adjust file‑opening logic to add the “.md” suffix only when needed.
* Update related conditionals to respect the new combined flag handling.
* Ensure markdown output is correctly written without creating “.md.md” files.
- Split Markdown generation into `MarkdownHeader` and `MarkdownRow` for clearer separation.
- Remove body‑preview block from Markdown rows.
- Add guard in runner to write the header only once per run.
- Introduce `MarkDownOutput` flag and CLI option (`-markdown` / `-md`).
- Implement `Result.MarkdownOutput` to generate a formatted Markdown table with optional title and CDN columns, and a response body preview.
- Add markdown escaping helper.
- Update runner to create `.md` output file, handle markdown generation per result, and write to file or stdout.
- Adjust option parsing and related logic to include markdown handling.
String concatenation doesn't properly handle IPv6 addresses which need brackets.
Example: [::1]:443 would become ::1:80 instead of [::1]:80
Uses net.JoinHostPort() which correctly formats IPv6 addresses.
Ayan Rajpoot