* fix(db): persist CPE in postgres and mysql (#2487)
Adds cpe column to both SQL schemas, idempotent migration for
existing tables, and JSON binding in InsertBatch.
* adding comment note
* fixing lint
---------
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
- Fix SQL injection in postgres.go using pq.QuoteIdentifier for table/index names
- Fix SQL injection in mysql.go using custom quoteIdentifier function
- Fix race condition in writer.go by checking closed state before channel send
- Add missing idx_url index in mysql.go for parity with PostgreSQL schema
- Include RawHeaders in omitRaw check for consistency
Add support for storing httpx scan results directly to databases with
both CLI flags and YAML config file options.
Supported databases:
- MongoDB
- PostgreSQL
- MySQL
Features:
- Batched writes for performance (configurable batch size)
- Auto-flush with configurable interval
- Individual columns for each Result field (not JSON blob)
- Support for environment variable HTTPX_DB_CONNECTION_STRING
- Option to omit raw request/response data (-rdbor)
New CLI flags under OUTPUT group:
- -rdb, -result-db: enable database storage
- -rdbc, -result-db-config: path to YAML config file
- -rdbt, -result-db-type: database type (mongodb, postgres, mysql)
- -rdbcs, -result-db-conn: connection string
- -rdbn, -result-db-name: database name (default: httpx)
- -rdbtb, -result-db-table: table/collection name (default: results)
- -rdbbs, -result-db-batch-size: batch size (default: 100)
- -rdbor, -result-db-omit-raw: omit raw data
Closes#1973Closes#2360Closes#2361Closes#2362
Dogan Can Bakir