Files
urbanadventurer-WhatWeb/lib/whatweb/parser.rb
T
Andrew Horton 29aa301273 Version 0.6.1 - August 2, 2025
Changelog:
* Updated version from 0.5.5 to 0.6.1
* Improved SSL/TLS security with comprehensive cipher suite support (TLSv1, TLSv1.1, TLSv1.2, SSLv3, SSLv2)
  - Added SSL_ATTRIBUTES constant for verify_mode
  - Replaced dynamic SSL parameter collection with explicit SSLContext configuration
  - Set verify_mode to VERIFY_NONE for maximum compatibility
  - Enabled negotiation of all SSL/TLS versions including legacy SSLv2/SSLv3
* Enhanced HTTPS connection handling with optimized verification settings
  - Cleared minimum version restrictions and disabled protocol exclusion flags
  - Added fallback handling for different OpenSSL configurations
* Added dual-protocol feature: simple hostnames (without paths or ports) are now scanned using both HTTP and HTTPS protocols
  - Automatically creates and tests both HTTP and HTTPS versions of simple hostnames
  - Shows informational message when dual-protocol scanning is performed
  - Intelligently detects and avoids duplicate URL scanning with redirect handling
* Added robust input validation and error handling
  - Automatically aborts processing after 10 consecutive parsing errors
  - Prevents wasted resources when processing invalid data sources
  - Provides helpful error messages to guide users in troubleshooting input issues
  - Handles various error cases including malformed URLs and non-URL input data
* Improved dependency management with Bundler group support
  - Reorganized Gemfile with proper optional group definitions for mongo and rchardet
  - Added clear installation instructions for optional dependencies
  - Set explicit version constraints for test dependencies
  - Added support for older Bundler versions through GEMFILE_GROUPS environment variable
* Expanded documentation for optional dependencies
  - Added detailed installation instructions for MongoDB support
  - Added clear steps for character set detection configuration
  - Provided combined installation commands for all optional features
  - Improved explanation of performance implications

This release introduces three new search contexts for plugin authors to use:
- uri.path
- uri.query
- uri.extension

An example of how to use this is from the PHP plugin.

```
{ :name=>"File extension", :regexp=>/^(php|phtml|php3|php4|php5|phps)$/, :search=>"uri.extension" }
```

* #311 - JSON Logging fails... UTF-8 can't modify frozen String (@juananpe)

* Refactored Helper::convert_to_utf8. Droped Ruby 2.0 support (@urbanadventurer)
* Added new search contexts for plugins: uri.path, uri.query, uri.extension (@urbanadventurer)

* Plex Media Server (@urbanadventurer)
* Meta-Facebook-Infrastructure (@urbanadventurer)
* Netflix-Platform (@urbanadventurer)
* VKontakte-Platform (@urbanadventurer)
* Distributed-Tracing (@urbanadventurer)
* Modern-Security-Headers (@urbanadventurer)
* Baidu-Platform (@urbanadventurer)
* Alibaba-Aliyun (@urbanadventurer)
* Weibo-Platform (@urbanadventurer)

* Adobe-Flash (@urbanadventurer)
* ASP_NET (@urbanadventurer)
* ColdFusion (@urbanadventurer)
* Drupal (@gboddin)
* Java (@urbanadventurer)
* Perl (@urbanadventurer)
* PHP (@urbanadventurer)
* Python (@urbanadventurer)
* Ruby (@urbanadventurer)
* TYPO3 (@definity)
* WordPress (@juananpe)
* Shopify (enhanced HTTP header detection) (@urbanadventurer)
* CloudFlare (enhanced with CF-RAY and other headers) (@urbanadventurer)
* CloudFront (renamed to Amazon-CloudFront and enhanced) (@urbanadventurer)
* Tengine Web Server (added Alibaba-specific headers) (@urbanadventurer)
* Content-Security-Policy (added modern CSP headers) (@urbanadventurer)
* Azure (renamed to Microsoft-Azure and enhanced) (@urbanadventurer)
2025-08-02 03:04:19 +10:00

69 lines
2.2 KiB
Ruby

# Copyright 2009 to 2025 Andrew Horton and Brendan Coles
#
# This file is part of WhatWeb.
#
# WhatWeb is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 2 of the License, or
# at your option) any later version.
#
# WhatWeb is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with WhatWeb. If not, see <http://www.gnu.org/licenses/>.
module WhatWeb
class Parser
def self.run_plugins(target, plugins, scanner: nil)
results = []
raise('No plugins selected') if plugins.empty?
plugins.each do |name, plugin|
begin
# execute the plugin
# start_time = Time.now
result = plugin.scan(target)
# end_time = Time.now
# $PLUGIN_TIMES[name] += end_time - start_time
rescue => e
error("ERROR: Plugin #{name} failed for #{target}. #{e}")
raise if $WWDEBUG == true
end
results << [name, result] unless result.nil? || result.empty?
end
results
end
def self.parse(target, result, logging_list: nil, grep_plugin: false)
logging_list.each do |log|
begin
# Hide log if Grep plugin did not match
if grep_plugin
if result.map { |plugin_name, _plugin_result| plugin_name }.include? 'Grep'
log.out(target, target.status, result)
end
else
log.out(target, target.status, result)
end
rescue => e
error("ERROR Logging failed: #{target} - #{e}")
raise if $WWDEBUG == true
end
end unless logging_list.nil?
result = {
'target' => target,
'status' => target.status,
'result' => result
}
return result unless grep_plugin
return result if result.map { |plugin_name, _plugin_result| plugin_name }.include? 'Grep'
end
end
end