* code quality: improve error handling and fix bugs
- Fix bug: return err instead of closeErr in DecodeData error path (httpx.go)
- Improve error handling for strconv.Atoi calls in multiple locations
- Change hammingDistanceThreshold from var to const since it's never modified
- Fix redundant return statement in ListFilesWithPattern
- Improve port parsing to handle empty port strings explicitly
* runner: keep input counters incrementing under -skip-dedupe
Revert the defensive strconv.Atoi guard around the SkipDedupe branches:
values stored in r.hm are always integer strings written by this same
code path, so the parseErr/cnt>0 check is unreachable in practice, and
when taken literally it skipped the numHosts/numTargets increment for
duplicate inputs.
---------
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
* fix(db): persist CPE in postgres and mysql (#2487)
Adds cpe column to both SQL schemas, idempotent migration for
existing tables, and JSON binding in InsertBatch.
* adding comment note
* fixing lint
---------
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
testAndSet performs a non-atomic read-then-write sequence:
1. seen(k) - check if key exists
2. setSeen(k) - set the key
Without synchronization, two concurrent goroutines can both pass the
seen() check before either calls setSeen(), causing duplicate
processing of the same target.
This race can occur in processTargets() where multiple goroutines
call testAndSet() for discovered TLS SubjectAN/SubjectCN values.
The fix adds a sync.Mutex to serialize testAndSet operations.
- Handle error from format.Parse in streamInput
- Add InputMode validation to ValidateOptions for early error detection
- Extract duplicate format validation to getInputFormat helper
- Fix shadow variable err in loadFromFormat callback
- Add bounds check in test before index access
- Add test for malformed XML input
- Run go mod tidy to fix dependency marking
When FollowRedirects is enabled and the target redirects to a different host
(e.g., hackerone.com -> www.hackerone.com), the favicon URL is correctly
resolved to the final host, but the cloned request's Host header still
contains the original host. This causes the server to return 404 because
the Host header doesn't match the URL.
This fix updates the Host header to match the resolved URL host before
making the favicon request.
Add support for passive detection of CPE (Common Platform Enumeration)
identifiers and WordPress plugins/themes using awesome-search-queries.
CPE Detection (-cpe flag):
- Matches response title, body, and favicon hash against patterns
- Extracts product, vendor, and generates CPE 2.3 identifiers
- Uses patterns from Shodan, FOFA, Google dorks
WordPress Detection (-wp flag):
- Detects plugins via /wp-content/plugins/[name]/ patterns
- Detects themes via /wp-content/themes/[name]/ patterns
- Validates against known plugins/themes list
New CLI flags in PROBES group:
- -cpe: display CPE based on awesome-search-queries
- -wp, -wordpress: display WordPress plugins and themes
Both are automatically included in JSON/CSV output.
Closes#1975
Use URL.Hostname() instead of URL.Host to extract the hostname for DNS
resolution. This ensures the port number is not included in the DNS
query when probing all IPs for a URL like http://example.com:8080.
Fixes#2346
Signed-off-by: majiayu000 <1835304752@qq.com>
* Consolidate output‑type checks into a single `jsonOrCsvOrMD` variable.
* Adjust file‑opening logic to add the “.md” suffix only when needed.
* Update related conditionals to respect the new combined flag handling.
* Ensure markdown output is correctly written without creating “.md.md” files.
- Split Markdown generation into `MarkdownHeader` and `MarkdownRow` for clearer separation.
- Remove body‑preview block from Markdown rows.
- Add guard in runner to write the header only once per run.
- Introduce `MarkDownOutput` flag and CLI option (`-markdown` / `-md`).
- Implement `Result.MarkdownOutput` to generate a formatted Markdown table with optional title and CDN columns, and a response body preview.
- Add markdown escaping helper.
- Update runner to create `.md` output file, handle markdown generation per result, and write to file or stdout.
- Adjust option parsing and related logic to include markdown handling.
String concatenation doesn't properly handle IPv6 addresses which need brackets.
Example: [::1]:443 would become ::1:80 instead of [::1]:80
Uses net.JoinHostPort() which correctly formats IPv6 addresses.
- When falling back from HTTPS to HTTP on port 443, now switches to port 80
- When falling back from HTTP to HTTPS on port 80, now switches to port 443
- Prevents 'plain HTTP request sent to HTTPS port' errors during fallback
- Added test cases for determineMostLikelySchemeOrder and port fallback logic
- Changed 'host' field to return actual hostname (e.g., example.com)
- Added new 'host_ip' field for the resolved/dialed IP address
- Fixes semantic issue where 'host' was incorrectly returning IP
feat: update -ldp option to show default ports in CLI output
- Modified URL formatting in runner.go to respect LeaveDefaultPorts option
- Fixed AddURLDefaultPort function to actually add default ports (80/443)
- When -ldp is used, CLI output now shows https://example.com:443 instead of https://example.com
- Maintains backward compatibility - default behavior unchanged
Fixes CLI output inconsistency where -ldp flag only affected Host headers
but not the displayed URL in console output.
Ayan Rajpoot