* fix(db): persist CPE in postgres and mysql (#2487)
Adds cpe column to both SQL schemas, idempotent migration for
existing tables, and JSON binding in InsertBatch.
* adding comment note
* fixing lint
---------
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
- Fix SQL injection in postgres.go using pq.QuoteIdentifier for table/index names
- Fix SQL injection in mysql.go using custom quoteIdentifier function
- Fix race condition in writer.go by checking closed state before channel send
- Add missing idx_url index in mysql.go for parity with PostgreSQL schema
- Include RawHeaders in omitRaw check for consistency
Add support for storing httpx scan results directly to databases with
both CLI flags and YAML config file options.
Supported databases:
- MongoDB
- PostgreSQL
- MySQL
Features:
- Batched writes for performance (configurable batch size)
- Auto-flush with configurable interval
- Individual columns for each Result field (not JSON blob)
- Support for environment variable HTTPX_DB_CONNECTION_STRING
- Option to omit raw request/response data (-rdbor)
New CLI flags under OUTPUT group:
- -rdb, -result-db: enable database storage
- -rdbc, -result-db-config: path to YAML config file
- -rdbt, -result-db-type: database type (mongodb, postgres, mysql)
- -rdbcs, -result-db-conn: connection string
- -rdbn, -result-db-name: database name (default: httpx)
- -rdbtb, -result-db-table: table/collection name (default: results)
- -rdbbs, -result-db-batch-size: batch size (default: 100)
- -rdbor, -result-db-omit-raw: omit raw data
Closes#1973Closes#2360Closes#2361Closes#2362
* adding strings.ToLower to headers
* added validation logic for flags
* validated silent flag with incompatible flags
* silent flag bug fix
* added test case
* updated test case to include multiple headers
When the url's format like: https://www.google.com:443, the domain pass
to the resolver will be www.google.com:443 and can't get the right
result
This patch split the port at the start of the function and rewrite the
domain to the right format, which makes the result of IP/Cname be
right.
Dogan Can Bakir