================================================================================
 RUSTSPLOIT — RELEASE NOTES
 Offensive-security framework (RouterSploit/Metasploit-inspired), single binary.
 One module library across four interfaces: interactive shell, CLI runner,
 PQ-encrypted REST/WebSocket API, and an MCP server.

 Build date : 2026-06-13
 Toolchain  : Rust edition 2024 (clang + lld)
 Build      : cargo build --bin rustsploit -> clean (0 errors, 0 warnings)
 Modules    : 389 self-registering native modules
================================================================================


OVERVIEW
--------------------------------------------------------------------------------
This release ports four well-known upstream projects into Rustsploit, fixes a
cluster of mass-scan / full-internet-sweep usability bugs, and lands a
framework-wide HTTP performance improvement.

All ported components are permissively licensed (BSD-2 / BSD-3 / Apache-2.0 /
MIT) and were vendored or added as dependencies accordingly. Every change below
was compiled and tested locally; the verification summary is in section 5.


================================================================================
 1. NEW CAPABILITIES (UPSTREAM PORTS)
================================================================================

[1.1] MCP server migrated to the official `rmcp` SDK            (Apache-2.0)
--------------------------------------------------------------------------------
  * Replaced the hand-rolled JSON-RPC-over-stdio MCP server (src/mcp/server.rs)
    with the official Model Context Protocol Rust SDK, rmcp v1.7.0.
  * The existing tool/resource logic (src/mcp/tools.rs, resources.rs, types.rs)
    is preserved unchanged. server.rs is now a thin adapter implementing rmcp's
    `ServerHandler` trait; protocol framing, transport, and spec compliance are
    owned by the SDK (giving correct resources/prompts/progress support going
    forward instead of a bespoke protocol).
  * All 29 tools and 7 resources retained. The per-call execution timeout
    (env RUSTSPLOIT_MCP_TIMEOUT_SECS, default 300s) and the stdout-isolation
    guard (the protocol channel is dup'd off fd 1 and fd 1 redirected to
    /dev/null, so stray println! can never corrupt the JSON-RPC stream) are
    both kept.
  * Validated end-to-end over stdio: the `initialize` handshake returns
    serverInfo = rustsploit-mcp 0.5.0, negotiates protocolVersion down to the
    client's requested 2024-11-05, and `tools/list` advertises all 29 tools.
  * Dependency added to Cargo.toml:
        rmcp = { version = "1.7", features =
                 ["server", "transport-io", "transport-async-rw"] }

[1.2] Recog fingerprint engine                          (Rapid7, BSD-2-Clause)
--------------------------------------------------------------------------------
  * New src/utils/recog.rs: an XML fingerprint-database loader + matcher modeled
    on Rapid7 Recog (recog-go was the reference for the match loop). Parses the
    Recog <fingerprint>/<param> schema with quick-xml, pre-compiles every regex
    behind once_cell::Lazy, and resolves a banner to structured fields
    (service.product / .version / .vendor, os.product, service.cpe23, ...).
  * Matcher semantics follow real Recog: a <param> carrying a non-empty `value`
    is a literal/interpolated assignment (e.g.
    cpe:/a:vendor:product:{service.version}); a <param> with no value reads
    regex capture group `pos`. Capture params are resolved in a first pass so
    that {field} placeholders in templates always bind, regardless of param
    order or any `pos` attribute on the template.
  * Vendored databases under src/utils/recog_db/ (embedded via include_str!):
    ssh_banners, ftp_banners, smtp_banners, http_servers, mysql_banners. A
    BSD-2-Clause credit header points at upstream.
  * Wired into src/modules/scanners/service_scanner.rs: after each banner read
    (FTP, SSH, SMTP, MySQL, and the HTTPS `Server:` header) the matcher enriches
    the detected version with a structured product/version + CPE; findings still
    flow through the existing FindingKind::Banner path.
  * 19 unit tests assert extracted fields against real example banners
    (OpenSSH/Dropbear, ProFTPD/vsftpd, Exim, Apache/IIS, MySQL/MariaDB, plus the
    malformed-regex-skip and unknown-db cases).
  * NOTE: the vendored DBs are curated subsets authored to the exact upstream
    schema (the porting agent had no outbound network access to fetch the full
    upstream XML). The complete Rapid7 DBs (5 files, ~770 KB, ~1024 fingerprints)
    are staged and can be dropped in for full coverage — see section 6.

[1.3] JARM + JA3 / JA3S TLS fingerprinting          (Salesforce, BSD-3-Clause)
--------------------------------------------------------------------------------
  * New src/utils/tls_fingerprint.rs:
      - JARM: 10 hand-crafted TLS ClientHello probes ported from Salesforce
        jarm.py (TLS 1.2/1.3, multiple cipher orderings, GREASE on/off, ALPN
        variants, extension permutations). Probes are sent over a raw tokio
        TcpStream; each ServerHello is parsed (fully bounds-checked) and the
        results are assembled into the canonical 62-character JARM hash.
      - JA3 / JA3S string builders + MD5 (client- and server-side fingerprints),
        with GREASE stripping per spec.
      - Robust failure handling: a down host, a TLS alert, or a truncated
        response degrades to the canonical all-zero JARM hash; the ServerHello
        parser returns None on garbage input and never panics.
  * New module src/modules/scanners/jarm_scan.rs (default port 443) reports the
    JARM hash, JA3S, and client JA3 as findings.
  * src/modules/scanners/ssl_scanner.rs was intentionally NOT modified: it uses
    rustls, which hides the raw ServerHello bytes JA3S requires. JA3S is exposed
    via tls_fingerprint for callers (such as jarm_scan) that hold raw bytes.
  * 14 unit tests: JA3/JA3S string + MD5 (verified with md5sum), JARM hash
    assembly, and parser bounds-safety on empty/truncated/garbage input.

[1.4] SecLists wordlist catalog                                        (MIT)
--------------------------------------------------------------------------------
  * Seeded the previously-empty, checksum-pinned wordlist catalog in
    src/utils/wordlist.rs with 6 curated SecLists entries. Each SHA-256 was
    computed over the exact raw upstream bytes (the same bytes resolve() hashes
    after download):
        passwords-top-1k        Passwords/Common-Credentials/Pwdb_top-1000.txt
        passwords-top-10k       Passwords/Common-Credentials/Pwdb_top-10000.txt
        usernames-short         Usernames/top-usernames-shortlist.txt
        web-common              Discovery/Web-Content/common.txt
        web-raft-small-dirs     Discovery/Web-Content/raft-small-directories.txt
        subdomains-top5k        Discovery/DNS/subdomains-top1million-5000.txt
  * resolve(name) downloads + SHA-256-verifies on first use into
    ~/.rustsploit/wordlists/ (rejecting any checksum mismatch). 2 no-network
    tests assert every catalog entry is well-formed: https URL, 64-hex SHA-256,
    non-empty name/local_name, and unique names.


================================================================================
 2. BUG FIXES (MASS-SCAN / FULL-INTERNET SWEEP)
================================================================================

[2.1] Full-internet sweep host-cap consistency
--------------------------------------------------------------------------------
  * `setg target 0.0.0.0` auto-raised max_random_hosts to the full public-IPv4
    count, but a literal `0.0.0.0/0` typed into `use <mod>; run` never passed
    through that handler and fell back to the scheduler default of 10,000 —
    silently capping a "scan everything" target at 10k hosts.
  * Fixed in src/scheduler.rs at the single Target::Random dispatch chokepoint:
    when the operator has NOT explicitly set max_random_hosts, a full sweep now
    means every reachable public host on both entry paths. An explicit
    `setg max_random_hosts <n>` is still honored, and the per-sweep advisory +
    interactive confirmation gate is unchanged.

[2.2] Mass-scan pre-config: confirm BEFORE harvest, and silence the placeholder
--------------------------------------------------------------------------------
  The pre-batch prompt phase runs the selected module once against a placeholder
  host (0.0.0.1) to harvest the operator's answers once for the whole batch.
  Two problems were fixed in src/scheduler.rs:
    (a) ORDER: the harvest ran BEFORE the full-sweep confirmation, so the
        placeholder was effectively "scanned" (and, for service_scanner, a
        results file was written) before the operator ever confirmed the sweep.
        The full-sweep advisory + confirmation now runs FIRST on both the random
        and sequential sweep paths; declining aborts with nothing touched.
    (b) NOISE: the harvest run leaked module output
        ("Scanning 0.0.0.1 ... 0 services detected ... Results saved"). The
        harvest run is now wrapped in a throwaway OUTPUT_BUFFER scope so its
        mprintln!/meprintln! output is suppressed, while the interactive prompts
        (which print via real stdout) remain visible.
  RESIDUAL: the harvest still executes the module against the placeholder host
  (now silently). Removing that entirely requires a per-module "prompt-only"
  mode — see section 6.

[2.3] service_scanner mass-scan output spam + file-save race
--------------------------------------------------------------------------------
  * In a mass-scan fan-out the module runs once per host and previously printed
    its full single-target block for EVERY host — "No services detected",
    "Results saved", "Scan complete: 0 services detected on <host>" — which
    garbled output across concurrent tasks and (worse) had every host racing to
    overwrite the same results file.
  * These are now gated on `!is_batch_mode()`:
      - only hosts that actually have services print their results table,
      - the per-host file save is skipped in batch mode (findings flow through
        the module outcome instead, avoiding the concurrent-overwrite race),
      - the redundant per-host "No services" / "Results saved" / "Scan complete"
        status lines are suppressed.
    Interactive single-target runs are unchanged.

[2.4] `show options` listed only a subset of global options
--------------------------------------------------------------------------------
  * Added the four missing global keys to the shell's `show options` table:
    scan_order, exclusions, target_rps, module_rps.


================================================================================
 3. PERFORMANCE
================================================================================

[3.1] HTTP client connection-pool reuse (framework-wide)
--------------------------------------------------------------------------------
  * 184 of the ~304 reqwest-client construction sites go through the shared
    helper build_http_client(timeout), which rebuilt a fresh reqwest::Client on
    every call — re-initialising the TLS configuration and allocating a brand
    new (empty) connection pool. In an HTTP-based mass scan that is once per
    host; in normal use it is once per module run.
  * src/utils/network.rs::build_http_client now caches the permissive client,
    keyed by (timeout, accept_invalid_certs), and returns cheap Arc clones —
    reqwest clients are Arc internally and share their connection pool across
    clones, so warm connections and TLS setup are reused across runs.
  * Correctness: accept_invalid_certs is part of the key because permissive()
    derives it from the live `strict_tls` global; toggling `setg strict_tls` is
    therefore still honored rather than pinned to whatever it was on first build.
    The client is built outside the lock so a slow build() can't serialise other
    callers, and a poisoned lock is recovered (the map holds only cloneable
    clients) rather than panicking into every HTTP caller.
  * Custom-opts clients (cookies / headers / redirects via
    build_http_client_with) are unchanged — those options cannot be shared
    safely or keyed cheaply.


================================================================================
 4. FILES CHANGED
================================================================================
  New files:
    src/mcp/server.rs                          rewritten as an rmcp adapter
    src/utils/recog.rs                         Recog matcher (~590 lines)
    src/utils/recog_db/*.xml                   5 vendored fingerprint databases
    src/utils/tls_fingerprint.rs               JARM / JA3 / JA3S (~1270 lines)
    src/modules/scanners/jarm_scan.rs          new JARM scanner module (~127 lines)

  Modified files:
    Cargo.toml                                 + rmcp dependency
    src/module.rs                              + ModuleCtx.prompt_only flag (6.1)
    src/scheduler.rs                           sweep host-cap consistency,
                                               confirm-before-harvest reorder,
                                               harvest output capture,
                                               set prompt_only on harvest ctx
    src/modules/scanners/service_scanner.rs    Recog enrichment wiring +
                                               batch-mode output gating +
                                               prompt_only early return (6.1)
    src/utils/network.rs                       HTTP client cache +
                                               pool_idle_timeout (6.2)
    src/utils/recog.rs                         two-pass matcher fix (value
                                               templates win over `pos`)
    src/utils/recog_db/mysql_banners.xml       MariaDB regex colon fix
    src/utils/tls_fingerprint.rs               corrected JA3/JA3S test constants
    src/utils/wordlist.rs                      SecLists catalog + 2 tests
    src/utils/mod.rs                           + pub mod recog;
                                               + pub mod tls_fingerprint;
    src/modules/scanners/mod.rs                + pub mod jarm_scan;
    src/shell.rs                               show options: + 4 global keys
    README.md, docs/BAD_PATTERNS.md            stale-docs corrections (6.3)


================================================================================
 5. VERIFICATION
================================================================================
  * cargo build --bin rustsploit : clean — 0 errors, 0 warnings.
  * MCP : live stdio handshake validated — initialize returns
          serverInfo=rustsploit-mcp 0.5.0, protocolVersion negotiated to
          2024-11-05, tools/list returns all 29 tools.
  * Tests : new-port + core suites green —
          cargo test --bin rustsploit -- recog tls_fingerprint jarm wordlist
          cyclic ja3  =>  40 passed; 0 failed.
  * Banned-pattern audit (scripts/audit-bad-patterns.sh) : no new hits
          introduced in any changed file.

  PRE-RELEASE FIXES (found and fixed in this release):
    The four ports were authored by isolated agents that could not execute the
    test runner in their sandbox, so 7 tests shipped failing and were caught +
    fixed here:
      - 3 JA3/JA3S tests had fabricated "known vector" MD5 constants; the
        implementation was correct (the real md5 of the JA3/JA3S strings matches
        what the code produces) — the bogus expected constants were corrected.
      - 4 Recog tests exposed a real matcher bug: a <param> with both a `value`
        template and a `pos` attribute had its template ignored (the bare
        capture group was stored instead of the interpolated CPE), plus a
        MariaDB regex that rejected a colon in the distro suffix. Both fixed.


================================================================================
 6. FOLLOW-UP PASS (completed)
================================================================================
The four follow-ups noted at first cut were all worked. Three are done; the
fourth was investigated and intentionally deferred with a precise finding.

[6.1] Pre-config harvest now has a per-module "prompt-only" mode            DONE
  * Added ModuleCtx.prompt_only (src/module.rs). The scheduler sets it on the
    harvest dry-run ctx (src/scheduler.rs). A module that honours the flag
    answers its cfg_prompt_* calls and then returns immediately — no network
    work, no file writes against the placeholder host.
  * src/modules/scanners/service_scanner.rs is the reference implementation: it
    returns right after gathering prompts when prompt_only is set, so the
    placeholder host (0.0.0.1) is no longer scanned at all during harvest. The
    output-capture guard remains as belt-and-suspenders for modules that don't
    yet check the flag; other modules can adopt the same one-line early return.

[6.2] HTTP client cache: idle-connection lifetime bounded                   DONE
  * Added HttpClientOpts.pool_idle_timeout (src/utils/network.rs). The cached,
    long-lived permissive client now sets pool_idle_timeout = 30s, so a
    full-internet sweep reaps idle keep-alives to stale hosts instead of letting
    them accumulate. Custom-opts clients default to reqwest's ~90s unless they
    opt in.

[6.3] Docs drift corrected                                                  DONE
  * README.md and docs/BAD_PATTERNS.md prose updated to reflect the
    exploitation-only model (no check()/CheckResult) and compile-time `inventory`
    self-registration (no build.rs module indexer). The BAD_PATTERNS regex
    matrix is driven by hardcoded bash arrays in the audit script — confirmed
    untouched, so the lint is unaffected.

[6.4] Recog full-DB swap: investigated, DEFERRED (it is a feature, not a swap)
  * The full upstream Rapid7 DBs (5 files, ~770 KB, ~1024 fingerprints) were
    dropped in and the Recog tests re-run: 13/20 failed. Root cause is NOT a test
    issue — the real DBs are not drop-in compatible with the current wiring:
      - The real SSH patterns are anchored on the version COMMENT
        (e.g. ^OpenSSH_(...)$), i.e. they expect the post-"SSH-2.0-" substring,
        not the full banner line the scanner currently feeds them.
      - Real field names/values differ from the curated set (e.g. Apache yields
        service.product = "HTTPD", not "HTTP Server").
    So adopting the full DBs requires a per-DB INPUT-NORMALISATION layer (feed
    each DB the exact field it expects) plus downstream handling of real Recog
    field names — a feature, not a data swap. Shipping the real DBs as-is would
    silently break matching end-to-end (raw banners vs. normalised-input
    patterns => zero matches in practice).
  * DECISION: kept the curated, internally-consistent DBs (all 19 Recog tests
    pass; wiring + patterns + tests agree). The full-DB adoption is tracked as a
    future feature with the concrete design above. The real DBs remain staged.


================================================================================
 6c. NEW: per-run output auto-save
================================================================================
  * Every interactive console / CLI module run now auto-appends ALL of its
    output (stdout + stderr) to a per-run file:
        ~/.rustsploit/loot/<module> <YYYY-MM-DD_HH-MM-SS> results.txt
    Files are opened in APPEND mode, so multi-host mass-scan output accumulates
    into the one run file instead of racing to overwrite (the original
    service_scanner "save to file" bug). New module: src/results_sink.rs — a
    global append sink mirroring the spool's write pattern (so spawned per-host
    task output is captured), hooked into the console branches of the
    mprintln!/meprintln! routing in src/output.rs and begun/ended per run in
    commands::run_module. Scoped to console/CLI runs (sequential); API / MCP
    runs return their output to the caller via OUTPUT_BUFFER and are not
    duplicated to disk. Verified end-to-end (file created, headered, appended).
  * Build fix bundled: Cargo.lock restored. The branch's previous
    "Delete Cargo.lock" commit left a binary crate that does NOT build from a
    clean clone (transitive `cookie`/`time` version conflict on fresh
    resolution); the verified lock is re-added so `git clone && cargo build`
    works again.


================================================================================
 6b. REMAINING KNOWN LIMITATIONS
================================================================================
  * Recog coverage is the curated subset until the input-normalisation layer in
    [6.4] is built.
  * Modules other than service_scanner do not yet check ModuleCtx.prompt_only;
    during harvest their output is suppressed but they still execute (cheaply).
    Adopting the one-line early return per module closes this fully.


================================================================================
 6d. FRAMEWORK HARDENING (error-safety, retry-then-continue, wiring)
================================================================================
A two-agent audit of the framework (non-module) .rs files (it found the core
"unusually disciplined" already) plus fixes for every genuine issue surfaced:

  Panic / crash paths:
  * src/shell.rs — the rustyline completer sliced the input line at a raw byte
    offset; a cursor landing mid-UTF-8 codepoint crashed the whole shell. Now
    snaps to a char boundary first.
  * src/scheduler.rs — three `unreachable!(...)` in the fan-out dispatch are now
    `anyhow::bail!(...)` (internal-error surfaced, never a panic).

  Retry-then-continue (transient errors retry, then the scan continues):
  * New `run_host_with_retry` in scheduler.rs wraps every per-host module
    dispatch across all four mass-scan fan-outs (CIDR / file / random /
    sequential): a transient failure (timeout, connection reset/aborted, broken
    pipe, unexpected EOF) is retried once with a short backoff; a terminal error
    (closed port, "not affected") returns immediately. A single host never
    aborts the loop.
  * The "first 10 dispatches errored -> abort the whole sweep" heuristic in the
    random + sequential sweeps is softened to a one-time warning that KEEPS
    GOING (Ctrl+C still stops it) — error -> retry -> continue, not auto-abort.
  * Confirmed already-correct (left as-is): throttle back-off, tcp_connect
    address-iteration, bruteforce per-host error tolerance, job catch_unwind.

  Wiring / protocol bugs:
  * src/ws.rs — an oversize WS frame was `continue`-skipped, but the PQ AEAD
    ratchet's nonce embeds the receive counter and only advances on a successful
    decrypt, so a skipped frame permanently desynced (bricked) the connection.
    Now closes cleanly instead (axum's max_frame_size already caps upstream).
  * src/mcp/tools.rs — background jobs spawn under the tenant job-manager but
    list/kill read the process-global one, so tenant jobs were invisible and
    unkillable. list/kill now use the tenant job-manager too.
  * src/mcp/tools.rs — an out-of-range `port` argument was silently dropped (and
    the module fell back to its default); now rejected with a clear error.
  * src/ws.rs — a non-string option value was coerced to "" and stored; now
    rejected with a clear error.

  No silent error-swallowing:
  * Swept the framework (non-module) files for the banned swallow patterns
    (`Err(_)`, `if let Ok(...)` with no else, bare `.ok();`, `let _ = <result>`,
    `.map_err(|_| ...)`) and bound + surfaced every one via `tracing`
    (debug for benign fallbacks, warn for lost data/replies) — semantics
    unchanged, errors no longer dropped. Legitimate poison-recovery and
    deliberate Result->Option conversions were left intact.

  Verified: clean build (0 warnings), 40/40 targeted tests green, bad-patterns
  audit clean on the changed files.


================================================================================
 7. LICENSES (PORTED COMPONENTS)
================================================================================
  Recog (Rapid7)               BSD-2-Clause
  JARM / JA3 / JA3S (Salesforce)  BSD-3-Clause
  rmcp (MCP Rust SDK)          Apache-2.0
  SecLists                     MIT
  ZMap address iterator        Apache-2.0   (previously ported; unchanged)

================================================================================
 END OF RELEASE NOTES
================================================================================
