Add cookie jar persistence and improve cookie handling
Key Changes:
Cookie jar file persistence: --cookie-jar now saves cookies to file and loads them on startup, maintaining cookies across WhatWeb sessions
Domain-aware cookie storage: New format tracks cookies per domain (backwards compatible with legacy single-domain format)
Automatic file creation: Cookie jar file is created if it doesn't exist
Thread-safe implementation: Mutex-protected cookie operations for concurrent scanning
Implementation Details:
Modified SimpleCookieJar to support load_from_file() and save_to_file() methods
Cookie jar file uses domain-aware format: # Domain: example.com followed by cookie string
Maintains backward compatibility with legacy cookie jar format
Saves cookies at program exit, preserving session state
Added proper error handling and directory creation for cookie jar files
Changelog:
* Updated version from 0.5.5 to 0.6.1
* Improved SSL/TLS security with comprehensive cipher suite support (TLSv1, TLSv1.1, TLSv1.2, SSLv3, SSLv2)
- Added SSL_ATTRIBUTES constant for verify_mode
- Replaced dynamic SSL parameter collection with explicit SSLContext configuration
- Set verify_mode to VERIFY_NONE for maximum compatibility
- Enabled negotiation of all SSL/TLS versions including legacy SSLv2/SSLv3
* Enhanced HTTPS connection handling with optimized verification settings
- Cleared minimum version restrictions and disabled protocol exclusion flags
- Added fallback handling for different OpenSSL configurations
* Added dual-protocol feature: simple hostnames (without paths or ports) are now scanned using both HTTP and HTTPS protocols
- Automatically creates and tests both HTTP and HTTPS versions of simple hostnames
- Shows informational message when dual-protocol scanning is performed
- Intelligently detects and avoids duplicate URL scanning with redirect handling
* Added robust input validation and error handling
- Automatically aborts processing after 10 consecutive parsing errors
- Prevents wasted resources when processing invalid data sources
- Provides helpful error messages to guide users in troubleshooting input issues
- Handles various error cases including malformed URLs and non-URL input data
* Improved dependency management with Bundler group support
- Reorganized Gemfile with proper optional group definitions for mongo and rchardet
- Added clear installation instructions for optional dependencies
- Set explicit version constraints for test dependencies
- Added support for older Bundler versions through GEMFILE_GROUPS environment variable
* Expanded documentation for optional dependencies
- Added detailed installation instructions for MongoDB support
- Added clear steps for character set detection configuration
- Provided combined installation commands for all optional features
- Improved explanation of performance implications
This release introduces three new search contexts for plugin authors to use:
- uri.path
- uri.query
- uri.extension
An example of how to use this is from the PHP plugin.
```
{ :name=>"File extension", :regexp=>/^(php|phtml|php3|php4|php5|phps)$/, :search=>"uri.extension" }
```
* #311 - JSON Logging fails... UTF-8 can't modify frozen String (@juananpe)
* Refactored Helper::convert_to_utf8. Droped Ruby 2.0 support (@urbanadventurer)
* Added new search contexts for plugins: uri.path, uri.query, uri.extension (@urbanadventurer)
* Plex Media Server (@urbanadventurer)
* Meta-Facebook-Infrastructure (@urbanadventurer)
* Netflix-Platform (@urbanadventurer)
* VKontakte-Platform (@urbanadventurer)
* Distributed-Tracing (@urbanadventurer)
* Modern-Security-Headers (@urbanadventurer)
* Baidu-Platform (@urbanadventurer)
* Alibaba-Aliyun (@urbanadventurer)
* Weibo-Platform (@urbanadventurer)
* Adobe-Flash (@urbanadventurer)
* ASP_NET (@urbanadventurer)
* ColdFusion (@urbanadventurer)
* Drupal (@gboddin)
* Java (@urbanadventurer)
* Perl (@urbanadventurer)
* PHP (@urbanadventurer)
* Python (@urbanadventurer)
* Ruby (@urbanadventurer)
* TYPO3 (@definity)
* WordPress (@juananpe)
* Shopify (enhanced HTTP header detection) (@urbanadventurer)
* CloudFlare (enhanced with CF-RAY and other headers) (@urbanadventurer)
* CloudFront (renamed to Amazon-CloudFront and enhanced) (@urbanadventurer)
* Tengine Web Server (added Alibaba-specific headers) (@urbanadventurer)
* Content-Security-Policy (added modern CSP headers) (@urbanadventurer)
* Azure (renamed to Microsoft-Azure and enhanced) (@urbanadventurer)
Redirection temporarily disabled.
Moved run_plugins into Parser.
Changed Scan initialize to only set up variables. Created Scan.scan to execute scanning and yields target objects.
Made opts and targets instance variables in Scan
No longer passing plugin list to Scan
Moved log and plugin shutown to whatweb
Andrew Horton