diff --git a/CHANGELOG.md b/CHANGELOG.md index ac868361..a60413e1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,7 @@ +Version 0.6.4 - ?? + + + Version 0.6.3 - October 18, 2025 ## FEATURES @@ -5,6 +9,15 @@ Version 0.6.3 - October 18, 2025 * Added --no-cookies option to disable cookie handling. * Added --cookie-jar option to save cookies to a file. +## PERFORMANCE IMPROVEMENTS +* Replaced global output mutex with per-logger mutexes for better concurrency +* Removed excessive STDERR.flush calls that were causing I/O bottlenecks +* Optimized verbose logging to build output atomically before writing +* Added configurable output buffering for high-thread scenarios +* Made output sync behavior configurable (--output-sync flag) +* Added smart defaults that automatically optimize based on thread count +* Added profiling infrastructure (WHATWEB_PROFILE environment variable) + ## FIXES * Fixed error with HTTPS connections when using a proxy server * Fix typo in plugin-tutorial-7.rb (@mtisec) diff --git a/README.md b/README.md index d35f1487..64a2c354 100644 --- a/README.md +++ b/README.md @@ -42,7 +42,7 @@ Most WhatWeb plugins are thorough and recognise a range of cues from subtle to o ### Features * Over 1800 plugins * Control the trade off between speed/stealth and reliability -* Performance tuning. Control how many websites to scan concurrently. +* Performance tuning. Control how many websites to scan concurrently with automatic output optimization. * Multiple log formats: Brief (greppable), Verbose (human readable), XML, JSON, MagicTree, RubyObject, MongoDB, ElasticSearch, SQL. * Proxy support including TOR * Custom HTTP headers @@ -358,9 +358,73 @@ WhatWeb features several options to increase performance and stability. * --read-timeout Time in seconds. Default: 30 * --wait=SECONDS Wait SECONDS between connections This is useful when using a single thread. +* --output-sync Force immediate output flushing to screen and logs for + real-time monitoring (slower with high thread counts). +* --output-buffer-size=SIZE Set output buffer size for screen and log output. + 0=unbuffered, default=auto based on thread count. The --wait and --max-threads commands can be used to assist in IDS evasion. +### Output Performance Optimization + +WhatWeb automatically optimizes output performance for both screen output and log files based on thread count: + +* **Low thread counts (1-5)**: + - **Sync**: Enabled (immediate flushing) + - **Buffer size**: 1 (minimal buffering for real-time output) + - **Use case**: Interactive scanning, real-time monitoring + +* **Medium thread counts (6-50)**: + - **Sync**: Disabled (async I/O) + - **Buffer size**: 10 (small buffer for balanced performance) + - **Use case**: Standard batch scanning including default (25 threads) + +* **High thread counts (51+)**: + - **Sync**: Disabled (async I/O) + - **Buffer size**: 25 (larger buffer to minimize I/O contention) + - **Use case**: High-volume scanning, large target lists (100+ threads) + +You can override these defaults for both screen output and log files: + +```bash +# Force real-time output to screen and logs (slower but immediate) +./whatweb --output-sync -t 100 --log-brief=results.txt targets.txt + +# Custom buffer size for high-volume logging +./whatweb --output-buffer-size=50 -t 100 --log-verbose=detailed.log targets.txt + +# Disable buffering entirely for immediate file writes +./whatweb --output-buffer-size=0 -t 10 --log-json=results.json targets.txt +``` + +These optimizations apply to all output formats including brief, verbose, XML, JSON, and other log formats. + +#### Technical Details + +**Output Sync Behavior:** +- **Enabled**: Each output line is immediately flushed to disk/screen (slower but real-time) +- **Disabled**: Output is buffered in memory before writing (faster but delayed) + +**Buffer Size Impact:** +- **Size 1**: Minimal buffering, nearly immediate output +- **Size 10**: Small buffer holds ~10 results before writing +- **Size 25+**: Larger buffer reduces I/O system calls for better performance +- **Size 0**: Completely unbuffered (equivalent to sync enabled) + +### Profiling Support + +Enable performance profiling to identify bottlenecks: + +```bash +# Basic profiling to stderr +WHATWEB_PROFILE=1 ./whatweb -t 25 targets.txt + +# Save profile to file +WHATWEB_PROFILE=1 WHATWEB_PROFILE_FILE=profile.txt ./whatweb -t 25 targets.txt +``` + +*Note: Profiling requires the `ruby-prof` gem: `gem install ruby-prof`* + Changing the user-agent using the -U or --user-agent command line option will avoid the Snort IDS rule for WhatWeb. If you are scanning ranges of IP addresses, it is much more efficient to use a port scanner like massscan to discover which have port 80 open before scanning with WhatWeb. diff --git a/lib/logging.rb b/lib/logging.rb index 9be09ce5..a1aaf787 100644 --- a/lib/logging.rb +++ b/lib/logging.rb @@ -20,17 +20,78 @@ class Logging # if no f, output to STDOUT, # if f is a filename then open it, if f is a file use it - def initialize(f = STDOUT) + def initialize(f = STDOUT, sync: nil) f = STDOUT if f == '-' @f = f if f.class == IO || f.class == File @f = File.open(f, 'a') if f.class == String - @f.sync = true # we want flushed output + + # Make sync configurable with smart defaults + # Default to sync for low thread counts, async for high thread counts + if sync.nil? + @f.sync = ($THREADS && $THREADS <= 5) ? true : false + else + @f.sync = sync + end + + # Per-instance mutex for thread-safe output + @mutex = Mutex.new + + # Output buffering for performance + @buffer = [] + @buffer_size = $OUTPUT_BUFFER_SIZE || 1 + @last_flush = Time.now + + # Ensure buffer size is at least 1 + @buffer_size = 1 if @buffer_size < 1 end def close + flush_buffer # Ensure all buffered output is written @f.close unless @f.class == IO end + # Thread-safe output method using per-instance mutex + def synchronized_write(&block) + @mutex.synchronize(&block) + end + + # Buffered output method for improved performance + def buffered_write(content) + @mutex.synchronize do + @buffer << content + + # Flush buffer if it's full, in sync mode, or buffer is getting stale + should_flush = @buffer.size >= @buffer_size || + @f.sync || + (Time.now - @last_flush > 5.0 && !@buffer.empty?) + + if should_flush + flush_buffer_unsafe + end + end + end + + # Force flush the buffer (thread-safe) + def flush_buffer + @mutex.synchronize do + flush_buffer_unsafe + end + end + + private + + # Flush buffer without mutex (must be called within synchronized block) + def flush_buffer_unsafe + return if @buffer.empty? + + # Join all buffered content with newlines and write as one block + combined_output = @buffer.join("\n") + @f.puts combined_output + + @buffer.clear + @last_flush = Time.now + end + # perform sort, uniq and join on each plugin result def suj(plugin_results) suj = {} diff --git a/lib/logging/brief.rb b/lib/logging/brief.rb index 265a99bc..adf382c5 100644 --- a/lib/logging/brief.rb +++ b/lib/logging/brief.rb @@ -88,8 +88,7 @@ class LoggingBrief < Logging else brief_results_final = "#{target} [#{status} #{status_code}] #{brief_results.join(', ')}" end - $semaphore.synchronize do - @f.puts brief_results_final - end + # Use buffered write for better performance + buffered_write(brief_results_final) end end diff --git a/lib/logging/verbose.rb b/lib/logging/verbose.rb index 1a843b48..be8017eb 100644 --- a/lib/logging/verbose.rb +++ b/lib/logging/verbose.rb @@ -10,191 +10,214 @@ class LoggingVerbose < Logging end def out(target, status, results) - $semaphore.synchronize do - # make a hash of the matches array - results_hash = {} - results.map { |k, v| results_hash[k] = v } + # Build all output first, then write as a complete block + output_lines = build_verbose_output(target, status, results) + + # Join lines into a single string and write as one block + output_block = output_lines.join("\n") + buffered_write(output_block) + end - display = { - title: '', - ip: '', - country: '', - status: '' - } + private - display[:country] = results_hash['Country'].map { |r| "#{r[:string]}, #{r[:module]}" }.join(',') if results_hash['Country'] - display[:ip] = results_hash['IP'].map { |r| r[:string] }.join(',') if results_hash['Country'] - display[:title] = results_hash['Title'].map { |r| r[:string] }.join(',') if results_hash['Title'] - display[:status] = status.to_s + ' ' + HTTP_Status.code(status) + def build_verbose_output(target, status, results) + lines = [] + + # make a hash of the matches array + results_hash = {} + results.map { |k, v| results_hash[k] = v } - @f.puts "WhatWeb report for #{coloured(target, 'blue')}" - @f.puts 'Status'.ljust(9) + ' : ' + display[:status] - @f.puts 'Title'.ljust(9) + " : #{coloured(display[:title], 'yellow')}" - @f.puts 'IP'.ljust(9) + ' : ' + display[:ip] - @f.puts 'Country'.ljust(9) + " : #{coloured(display[:country], 'red')}" - @f.puts + display = { + title: '', + ip: '', + country: '', + status: '' + } - ################### Short list - ################### Basically Output Brief + display[:country] = results_hash['Country'].map { |r| "#{r[:string]}, #{r[:module]}" }.join(',') if results_hash['Country'] + display[:ip] = results_hash['IP'].map { |r| r[:string] }.join(',') if results_hash['Country'] + display[:title] = results_hash['Title'].map { |r| r[:string] }.join(',') if results_hash['Title'] + display[:status] = status.to_s + ' ' + HTTP_Status.code(status) - brief_results = [] - results.each do |plugin_name, plugin_results| - next if %w(Title IP Country).include? plugin_name + # Header section + lines << "WhatWeb report for #{coloured(target, 'blue')}" + lines << 'Status'.ljust(9) + ' : ' + display[:status] + lines << 'Title'.ljust(9) + " : #{coloured(display[:title], 'yellow')}" + lines << 'IP'.ljust(9) + ' : ' + display[:ip] + lines << 'Country'.ljust(9) + " : #{coloured(display[:country], 'red')}" + lines << "" - next if plugin_results.empty? - suj = suj(plugin_results) + # Brief results section (similar to brief output) + brief_results = build_brief_results(results) + lines << 'Summary'.ljust(9) + ' : ' + brief_results.join(', ') + lines << "" + lines << 'Detected Plugins:' - certainty = suj[:certainty].to_i - version = suj[:version] - os = suj[:os] - string = suj[:string] - accounts = suj[:account] - model = suj[:model] - firmware = suj[:firmware] - modules = suj[:module] - filepath = suj[:filepath] + # Plugin details section + lines.concat(build_plugin_details(results)) - # colour the output - # be more DRY - # if plugins have categories or tags this would be better, eg. all hash plugins are grey - if (@f == STDOUT && $use_colour == 'auto') || ($use_colour == 'always') - coloured_string = grey(string) - coloured_string = cyan(string) if plugin_name == 'HTTPServer' - coloured_string = yellow(string) if plugin_name == 'Title' + # HTTP Headers section + lines << 'HTTP Headers:' + target.raw_headers.each_line do |header| + lines << "\t#{header.chomp}" + end + + lines + end - coloured_string = grey(string) if plugin_name == 'MD5' - coloured_string = grey(string) if plugin_name == 'Header-Hash' - coloured_string = grey(string) if plugin_name == 'Footer-Hash' - coloured_string = grey(string) if plugin_name == 'CSS' - coloured_string = grey(string) if plugin_name == 'Tag-Hash' + def build_brief_results(results) + brief_results = [] + results.each do |plugin_name, plugin_results| + next if %w(Title IP Country).include? plugin_name + next if plugin_results.empty? + + suj = suj(plugin_results) - coloured_plugin = white(plugin_name) - coloured_plugin = grey(plugin_name) if plugin_name == 'MD5' - coloured_plugin = grey(plugin_name) if plugin_name == 'Header-Hash' - coloured_plugin = grey(plugin_name) if plugin_name == 'Footer-Hash' - coloured_plugin = grey(plugin_name) if plugin_name == 'CSS' - coloured_plugin = grey(plugin_name) if plugin_name == 'Tag-Hash' + certainty = suj[:certainty].to_i + version = suj[:version] + os = suj[:os] + string = suj[:string] + accounts = suj[:account] + model = suj[:model] + firmware = suj[:firmware] + modules = suj[:module] + filepath = suj[:filepath] - p = ((certainty && certainty < 100) ? "#{grey(Helper::certainty_to_words(certainty))} " : '') + - coloured_plugin + (!version.empty? ? "[#{green(version)}]" : '') + - (!os.empty? ? "[#{red(os)}]" : '') + - (!string.empty? ? "[#{coloured_string}]" : '') + - (!accounts.empty? ? "[#{cyan(accounts)}]" : '') + - (!model.empty? ? "[#{dark_green(model)}]" : '') + - (!firmware.empty? ? "[#{dark_green(firmware)}]" : '') + - (!filepath.empty? ? "[#{dark_green(filepath)}]" : '') + - (!modules.empty? ? "[#{magenta(modules)}]" : '') + # colour the output + if (@f == STDOUT && $use_colour == 'auto') || ($use_colour == 'always') + coloured_string = grey(string) + coloured_string = cyan(string) if plugin_name == 'HTTPServer' + coloured_string = yellow(string) if plugin_name == 'Title' - brief_results << p - else - brief_results << ((certainty && certainty < 100) ? "#{Helper::certainty_to_words(certainty)} " : '') + - plugin_name + (!version.empty? ? "[#{version}]" : '') + - (!os.empty? ? "[#{os}]" : '') + - (!string.empty? ? "[#{string}]" : '') + - (!accounts.empty? ? " [#{accounts}]" : '') + - (!model.empty? ? "[#{model}]" : '') + - (!firmware.empty? ? "[#{firmware}]" : '') + - (!filepath.empty? ? "[#{filepath}]" : '') + - (!modules.empty? ? "[#{modules}]" : '') - end - end + coloured_string = grey(string) if plugin_name == 'MD5' + coloured_string = grey(string) if plugin_name == 'Header-Hash' + coloured_string = grey(string) if plugin_name == 'Footer-Hash' + coloured_string = grey(string) if plugin_name == 'CSS' + coloured_string = grey(string) if plugin_name == 'Tag-Hash' - brief_results_final = brief_results.join(', ') - @f.puts 'Summary'.ljust(9) + ' : ' + brief_results_final - @f.puts - @f.puts 'Detected Plugins:' + coloured_plugin = white(plugin_name) + coloured_plugin = grey(plugin_name) if plugin_name == 'MD5' + coloured_plugin = grey(plugin_name) if plugin_name == 'Header-Hash' + coloured_plugin = grey(plugin_name) if plugin_name == 'Footer-Hash' + coloured_plugin = grey(plugin_name) if plugin_name == 'CSS' + coloured_plugin = grey(plugin_name) if plugin_name == 'Tag-Hash' - results.sort.each do |plugin_name, plugin_results| - next if %w(Title IP Country).include?(plugin_name) - next if plugin_results.empty? - @f.puts "[ #{coloured(plugin_name, 'white')} ]" - description = [''] - if Plugin.registered_plugins[plugin_name].description - d = Plugin.registered_plugins[plugin_name].description - description = Helper::word_wrap(d, 60) - end - # @f.puts "\tCategory : " + Plugin.registered_plugins[plugin_name].category.first unless Plugin.registered_plugins[plugin_name].category.nil? - @f.puts "\t" + description.first - description[1..-1].each { |line| @f.puts "\t" + line } - @f.puts + p = ((certainty && certainty < 100) ? "#{grey(Helper::certainty_to_words(certainty))} " : '') + + coloured_plugin + (!version.empty? ? "[#{green(version)}]" : '') + + (!os.empty? ? "[#{red(os)}]" : '') + + (!string.empty? ? "[#{coloured_string}]" : '') + + (!accounts.empty? ? "[#{cyan(accounts)}]" : '') + + (!model.empty? ? "[#{dark_green(model)}]" : '') + + (!firmware.empty? ? "[#{dark_green(firmware)}]" : '') + + (!filepath.empty? ? "[#{dark_green(filepath)}]" : '') + + (!modules.empty? ? "[#{magenta(modules)}]" : '') - top_certainty = suj(plugin_results)[:certainty].to_i - unless top_certainty == 100 - @f.puts "\t" + 'Certainty'.ljust(13) + ': ' + Helper::certainty_to_words(top_certainty) - end - - plugin_results.map { |x| sortuniq(x) }.each do |pr| - if pr[:name] - name_of_match = pr[:name] - else - name_of_match = [pr[:regexp_compiled], pr[:text], pr[:regexp].to_s, - pr[:ghdb], pr[:md5], pr[:tagpattern]].compact.join('|') - end - - pr.each do |key, value| - next unless [:version, :os, :string, :account, :model, - :firmware, :module, :filepath, :url].include?(key) - - next if value.class == Regexp - - if key == :os - @f.print "\t" + 'OS'.ljust(13) + ': ' - else - @f.print "\t" + key.to_s.capitalize.ljust(13) + ': ' - end - - c = case key - when :version then 'green' - when :string then 'cyan' - when :certainty then 'grey' - when :os then 'red' - when :account then 'cyan' - when :model then 'dark_green' - when :firmware then 'dark_green' - when :module then 'magenta' - when :filepath then 'dark_green' - else 'grey' - end - - if value.is_a?(String) - @f.print coloured(value.to_s, c) - elsif value.is_a?(Array) - @f.print coloured(value.join(',').to_s, c) - else - @f.print coloured(value.inspect, c) - end - - @f.print " (from #{name_of_match})" unless name_of_match.empty? - unless pr[:certainty] == 100 - @f.print " (Certainty: #{ertainty_to_words pr[:certainty]} )" - end - - @f.puts - end - - @f.puts "\t" + coloured(pr.inspect.to_s, 'dark_blue') if $verbose > 1 - end - - if Plugin.registered_plugins[plugin_name].aggressive - @f.puts "\tAggressive function available (check plugin file or details)." - end - - if Plugin.registered_plugins[plugin_name].dorks.any? - @f.puts "\tGoogle Dorks".ljust(13) + ": (#{Plugin.registered_plugins[plugin_name].dorks.size})" - end - - if Plugin.registered_plugins[plugin_name].website - @f.puts "\tWebsite".ljust(13) + ": #{Plugin.registered_plugins[plugin_name].website}" - end - - @f.puts - end - - @f.puts 'HTTP Headers:' - target.raw_headers.each_line do |header| - @f.puts "\t#{header}" - # pp target.raw_headers + brief_results << p + else + brief_results << ((certainty && certainty < 100) ? "#{Helper::certainty_to_words(certainty)} " : '') + + plugin_name + (!version.empty? ? "[#{version}]" : '') + + (!os.empty? ? "[#{os}]" : '') + + (!string.empty? ? "[#{string}]" : '') + + (!accounts.empty? ? " [#{accounts}]" : '') + + (!model.empty? ? "[#{model}]" : '') + + (!firmware.empty? ? "[#{firmware}]" : '') + + (!filepath.empty? ? "[#{filepath}]" : '') + + (!modules.empty? ? "[#{modules}]" : '') end end + + brief_results + end + + def build_plugin_details(results) + lines = [] + + results.sort.each do |plugin_name, plugin_results| + next if %w(Title IP Country).include?(plugin_name) + next if plugin_results.empty? + + lines << "[ #{coloured(plugin_name, 'white')} ]" + description = [''] + if Plugin.registered_plugins[plugin_name].description + d = Plugin.registered_plugins[plugin_name].description + description = Helper::word_wrap(d, 60) + end + lines << "\t" + description.first + description[1..-1].each { |line| lines << "\t" + line } + lines << "" + + top_certainty = suj(plugin_results)[:certainty].to_i + unless top_certainty == 100 + lines << "\t" + 'Certainty'.ljust(13) + ': ' + Helper::certainty_to_words(top_certainty) + end + + plugin_results.map { |x| sortuniq(x) }.each do |pr| + if pr[:name] + name_of_match = pr[:name] + else + name_of_match = [pr[:regexp_compiled], pr[:text], pr[:regexp].to_s, + pr[:ghdb], pr[:md5], pr[:tagpattern]].compact.join('|') + end + + pr.each do |key, value| + next unless [:version, :os, :string, :account, :model, + :firmware, :module, :filepath, :url].include?(key) + next if value.class == Regexp + + line_parts = [] + if key == :os + line_parts << "\t" + 'OS'.ljust(13) + ': ' + else + line_parts << "\t" + key.to_s.capitalize.ljust(13) + ': ' + end + + c = case key + when :version then 'green' + when :string then 'cyan' + when :certainty then 'grey' + when :os then 'red' + when :account then 'cyan' + when :model then 'dark_green' + when :firmware then 'dark_green' + when :module then 'magenta' + when :filepath then 'dark_green' + else 'grey' + end + + if value.is_a?(String) + line_parts << coloured(value.to_s, c) + elsif value.is_a?(Array) + line_parts << coloured(value.join(',').to_s, c) + else + line_parts << coloured(value.inspect, c) + end + + line_parts << " (from #{name_of_match})" unless name_of_match.empty? + unless pr[:certainty] == 100 + line_parts << " (Certainty: #{Helper::certainty_to_words pr[:certainty]} )" + end + + lines << line_parts.join('') + end + + lines << "\t" + coloured(pr.inspect.to_s, 'dark_blue') if $verbose > 1 + end + + if Plugin.registered_plugins[plugin_name].aggressive + lines << "\tAggressive function available (check plugin file or details)." + end + + if Plugin.registered_plugins[plugin_name].dorks.any? + lines << "\tGoogle Dorks".ljust(13) + ": (#{Plugin.registered_plugins[plugin_name].dorks.size})" + end + + if Plugin.registered_plugins[plugin_name].website + lines << "\tWebsite".ljust(13) + ": #{Plugin.registered_plugins[plugin_name].website}" + end + + lines << "" + end + + lines end end diff --git a/lib/messages.rb b/lib/messages.rb index de11b067..437eabda 100644 --- a/lib/messages.rb +++ b/lib/messages.rb @@ -31,7 +31,7 @@ def error(s) $semaphore.reentrant_synchronize do # TODO: make use_color smart, so it detects a tty STDERR.puts((($use_colour == 'auto') || ($use_colour == 'always')) ? red(s) : s) - STDERR.flush + # REMOVED: STDERR.flush for better performance $LOG_ERRORS.out(s) if $LOG_ERRORS end end @@ -44,7 +44,7 @@ def warning(s) $semaphore.reentrant_synchronize do STDERR.puts((($use_colour == 'auto') || ($use_colour == 'always')) ? yellow(s) : s) - STDERR.flush + # REMOVED: STDERR.flush for better performance $LOG_ERRORS.out("WARNING: #{s}") if $LOG_ERRORS end end @@ -57,7 +57,7 @@ def notice(s) $semaphore.reentrant_synchronize do STDERR.puts((($use_colour == 'auto') || ($use_colour == 'always')) ? blue(s) : s) - STDERR.flush + # REMOVED: STDERR.flush for better performance end end @@ -69,6 +69,6 @@ def debug(s) $semaphore.reentrant_synchronize do STDERR.puts((($use_colour == 'auto') || ($use_colour == 'always')) ? grey("[DEBUG] #{s}") : "[DEBUG] #{s}") - STDERR.flush + # REMOVED: STDERR.flush for better performance end end diff --git a/test/optimization_targets.txt b/test/optimization_targets.txt new file mode 100644 index 00000000..c094a70d --- /dev/null +++ b/test/optimization_targets.txt @@ -0,0 +1,3 @@ +http://example.com +http://httpbin.org/html +http://httpbin.org/json \ No newline at end of file diff --git a/test/output_optimization_test.rb b/test/output_optimization_test.rb new file mode 100644 index 00000000..de34be1c --- /dev/null +++ b/test/output_optimization_test.rb @@ -0,0 +1,175 @@ +#!/usr/bin/env ruby +# +# Output optimization validation test +# Tests the new performance improvements +# + +require 'benchmark' +require 'fileutils' + +class OutputOptimizationTest + def initialize + @whatweb_path = File.expand_path('../whatweb', __dir__) + @test_dir = __dir__ + @results_dir = File.join(@test_dir, 'optimization_results') + FileUtils.mkdir_p(@results_dir) + + create_test_targets + end + + def create_test_targets + # Create a small test target file + test_targets = File.join(@test_dir, 'optimization_targets.txt') + unless File.exist?(test_targets) + File.write(test_targets, [ + 'http://example.com', + 'http://httpbin.org/html', + 'http://httpbin.org/json' + ].join("\n")) + end + end + + def test_new_options + puts "=== Testing New Command Line Options ===" + + target_file = File.join(@test_dir, 'optimization_targets.txt') + + # Test --output-sync option + puts "Testing --output-sync option..." + cmd = "#{@whatweb_path} --output-sync -t 5 --log-brief=/dev/null -i #{target_file}" + success = system(cmd + " 2>/dev/null") + puts success ? "✓ --output-sync works" : "✗ --output-sync failed" + + # Test --output-buffer-size option + puts "Testing --output-buffer-size option..." + cmd = "#{@whatweb_path} --output-buffer-size=10 -t 5 --log-brief=/dev/null -i #{target_file}" + success = system(cmd + " 2>/dev/null") + puts success ? "✓ --output-buffer-size works" : "✗ --output-buffer-size failed" + + # Test help text includes new options + puts "Testing help text includes new options..." + help_output = `#{@whatweb_path} --help 2>&1` + has_sync = help_output.include?('--output-sync') + has_buffer = help_output.include?('--output-buffer-size') + puts has_sync && has_buffer ? "✓ Help text updated" : "✗ Help text missing new options" + end + + def test_performance_comparison + puts "\n=== Performance Comparison Test ===" + + target_file = File.join(@test_dir, 'optimization_targets.txt') + + # Test with different thread counts to verify smart defaults + thread_counts = [1, 10, 25] + results = {} + + thread_counts.each do |threads| + print "Testing #{threads} threads... " + + start_time = Time.now + cmd = "#{@whatweb_path} -t #{threads} --log-brief=/dev/null -i #{target_file}" + success = system(cmd + " 2>/dev/null") + end_time = Time.now + + if success + time = end_time - start_time + results[threads] = time + puts "#{time.round(2)}s" + else + puts "FAILED" + end + end + + puts "\n=== Performance Results ===" + results.each do |threads, time| + puts "#{threads} threads: #{time.round(2)}s" + end + end + + def test_mutex_functionality + puts "\n=== Mutex Functionality Test ===" + + # Test that output is still correct with high thread count + target_file = File.join(@test_dir, 'optimization_targets.txt') + output_file = File.join(@results_dir, 'mutex_test_output.txt') + + puts "Testing output consistency with 20 threads..." + cmd = "#{@whatweb_path} -t 20 --log-brief=#{output_file} -i #{target_file}" + success = system(cmd + " 2>/dev/null") + + if success && File.exist?(output_file) + lines = File.readlines(output_file) + puts "✓ Generated #{lines.length} output lines" + + # Check for any obvious corruption (partial lines, etc.) + corrupted_lines = lines.select { |line| line.strip.empty? || line.length < 10 } + if corrupted_lines.empty? + puts "✓ No obviously corrupted output lines" + else + puts "✗ Found #{corrupted_lines.length} potentially corrupted lines" + end + + File.delete(output_file) + else + puts "✗ Mutex test failed" + end + end + + def test_profiling_infrastructure + puts "\n=== Profiling Infrastructure Test ===" + + target_file = File.join(@test_dir, 'optimization_targets.txt') + profile_file = File.join(@results_dir, 'test_profile.txt') + + puts "Testing profiling with WHATWEB_PROFILE environment variable..." + + env = { + 'WHATWEB_PROFILE' => '1', + 'WHATWEB_PROFILE_FILE' => profile_file + } + + cmd = "#{@whatweb_path} -t 5 --log-brief=/dev/null -i #{target_file}" + success = system(env, cmd + " 2>/dev/null") + + if success + if File.exist?(profile_file) + puts "✓ Profile file created" + profile_size = File.size(profile_file) + puts "✓ Profile file size: #{profile_size} bytes" + File.delete(profile_file) if profile_size > 0 + else + puts "✗ Profile file not created (ruby-prof may not be installed)" + end + else + puts "✗ Profiling test failed" + end + end + + def run_all_tests + puts "WhatWeb Output Optimization Test Suite" + puts "======================================" + puts "Timestamp: #{Time.now}" + puts + + # Check if whatweb exists + unless File.exist?(@whatweb_path) + puts "ERROR: WhatWeb not found at #{@whatweb_path}" + exit 1 + end + + test_new_options + test_performance_comparison + test_mutex_functionality + test_profiling_infrastructure + + puts "\n=== Test Summary ===" + puts "All optimization tests completed." + puts "Check output above for any failures marked with ✗" + end +end + +# Run the tests if this script is executed directly +if __FILE__ == $0 + test = OutputOptimizationTest.new + test.run_all_tests +end diff --git a/test/performance_test.rb b/test/performance_test.rb new file mode 100644 index 00000000..12231ed6 --- /dev/null +++ b/test/performance_test.rb @@ -0,0 +1,239 @@ +#!/usr/bin/env ruby +# +# Performance test script for WhatWeb output optimization +# Phase 1: Baseline measurement and profiling +# + +require 'benchmark' +require 'fileutils' + +class WhatWebPerformanceTest + def initialize + @whatweb_path = File.expand_path('../whatweb', __dir__) + @test_dir = __dir__ + @results_dir = File.join(@test_dir, 'performance_results') + FileUtils.mkdir_p(@results_dir) + + # Create test target files if they don't exist + create_test_targets + end + + def create_test_targets + # Small target list for quick tests + small_targets = File.join(@test_dir, 'small_targets.txt') + unless File.exist?(small_targets) + File.write(small_targets, [ + 'http://example.com', + 'http://httpbin.org/html', + 'http://httpbin.org/json', + 'http://httpbin.org/xml', + 'http://httpbin.org/redirect/2' + ].join("\n")) + end + + # Medium target list for realistic tests + medium_targets = File.join(@test_dir, 'medium_targets.txt') + unless File.exist?(medium_targets) + targets = [] + # Add some real websites for testing + base_sites = [ + 'example.com', 'httpbin.org', 'github.com', 'stackoverflow.com', + 'reddit.com', 'wikipedia.org', 'google.com', 'youtube.com', + 'amazon.com', 'twitter.com' + ] + + base_sites.each do |site| + targets << "http://#{site}" + targets << "https://#{site}" + end + + File.write(medium_targets, targets.join("\n")) + end + end + + def run_whatweb_test(threads, target_file, output_format = 'brief') + log_file = "/tmp/whatweb_test_#{threads}t_#{output_format}.log" + + cmd = case output_format + when 'brief' + "#{@whatweb_path} -t #{threads} --log-brief=#{log_file} -i #{target_file}" + when 'verbose' + "#{@whatweb_path} -t #{threads} --log-verbose=#{log_file} -i #{target_file}" + else + "#{@whatweb_path} -t #{threads} --log-brief=/dev/null -i #{target_file}" + end + + start_time = Time.now + success = system(cmd + " 2>/dev/null") + end_time = Time.now + + # Clean up log file + File.delete(log_file) if File.exist?(log_file) + + if success + end_time - start_time + else + nil + end + end + + def benchmark_thread_scaling + puts "=== Thread Scaling Benchmark ===" + puts "Testing with small target list (5 targets)" + + target_file = File.join(@test_dir, 'small_targets.txt') + thread_counts = [1, 2, 5, 10, 20] + + results = {} + + thread_counts.each do |threads| + print "Testing #{threads} threads... " + + # Run multiple times and take average + times = [] + 3.times do + time = run_whatweb_test(threads, target_file, 'brief') + times << time if time + end + + if times.any? + avg_time = times.sum / times.length + results[threads] = avg_time + puts "#{avg_time.round(2)}s (avg of #{times.length} runs)" + else + puts "FAILED" + end + end + + puts "\n=== Results Summary ===" + results.each do |threads, time| + speedup = results[1] ? (results[1] / time).round(2) : 'N/A' + puts "#{threads} threads: #{time.round(2)}s (#{speedup}x speedup)" + end + + results + end + + def benchmark_output_formats + puts "\n=== Output Format Benchmark ===" + puts "Testing different output formats with 10 threads" + + target_file = File.join(@test_dir, 'small_targets.txt') + threads = 10 + formats = ['brief', 'verbose', 'stdout'] + + results = {} + + formats.each do |format| + print "Testing #{format} format... " + + times = [] + 3.times do + time = run_whatweb_test(threads, target_file, format) + times << time if time + end + + if times.any? + avg_time = times.sum / times.length + results[format] = avg_time + puts "#{avg_time.round(2)}s" + else + puts "FAILED" + end + end + + puts "\n=== Output Format Results ===" + results.each do |format, time| + puts "#{format}: #{time.round(2)}s" + end + + results + end + + def profile_with_ruby_prof + puts "\n=== Ruby Profiling ===" + + target_file = File.join(@test_dir, 'small_targets.txt') + profile_file = File.join(@results_dir, "profile_#{Time.now.to_i}.txt") + + puts "Running profiled test (this may take longer)..." + + env = { + 'WHATWEB_PROFILE' => '1', + 'WHATWEB_PROFILE_FILE' => profile_file + } + + cmd = "#{@whatweb_path} -t 10 --log-brief=/dev/null -i #{target_file}" + + start_time = Time.now + success = system(env, cmd + " 2>/dev/null") + end_time = Time.now + + if success && File.exist?(profile_file) + puts "Profile completed in #{(end_time - start_time).round(2)}s" + puts "Profile saved to: #{profile_file}" + + # Show top 10 time consumers + puts "\n=== Top 10 Time Consumers ===" + profile_content = File.read(profile_file) + lines = profile_content.split("\n") + + # Find the start of the flat profile + start_idx = lines.find_index { |line| line.include?('%self') } + if start_idx + data_lines = lines[(start_idx + 1)..(start_idx + 10)] + data_lines.each { |line| puts line if line.strip.length > 0 } + end + else + puts "Profiling failed or ruby-prof not available" + end + end + + def run_all_tests + puts "WhatWeb Performance Test Suite" + puts "==============================" + puts "Timestamp: #{Time.now}" + puts "WhatWeb path: #{@whatweb_path}" + puts + + # Check if whatweb exists + unless File.exist?(@whatweb_path) + puts "ERROR: WhatWeb not found at #{@whatweb_path}" + exit 1 + end + + # Run all benchmark tests + thread_results = benchmark_thread_scaling + format_results = benchmark_output_formats + profile_with_ruby_prof + + # Save results to file + results_file = File.join(@results_dir, "benchmark_#{Time.now.to_i}.txt") + File.open(results_file, 'w') do |f| + f.puts "WhatWeb Performance Benchmark Results" + f.puts "=====================================" + f.puts "Timestamp: #{Time.now}" + f.puts + + f.puts "Thread Scaling Results:" + thread_results.each do |threads, time| + speedup = thread_results[1] ? (thread_results[1] / time).round(2) : 'N/A' + f.puts " #{threads} threads: #{time.round(2)}s (#{speedup}x speedup)" + end + f.puts + + f.puts "Output Format Results:" + format_results.each do |format, time| + f.puts " #{format}: #{time.round(2)}s" + end + end + + puts "\nResults saved to: #{results_file}" + end +end + +# Run the tests if this script is executed directly +if __FILE__ == $0 + test = WhatWebPerformanceTest.new + test.run_all_tests +end diff --git a/test/verbose_screen_test.rb b/test/verbose_screen_test.rb new file mode 100644 index 00000000..ad941672 --- /dev/null +++ b/test/verbose_screen_test.rb @@ -0,0 +1,106 @@ +#!/usr/bin/env ruby +# +# Test script to verify verbose screen output performance +# + +require 'benchmark' + +class VerboseScreenTest + def initialize + @whatweb_path = File.expand_path('../whatweb', __dir__) + @test_dir = __dir__ + + create_test_targets + end + + def create_test_targets + # Create a small test target file + test_targets = File.join(@test_dir, 'verbose_test_targets.txt') + unless File.exist?(test_targets) + File.write(test_targets, [ + 'http://example.com', + 'http://httpbin.org/html', + 'http://httpbin.org/json' + ].join("\n")) + end + end + + def test_verbose_screen_performance + puts "=== Verbose Screen Output Performance Test ===" + + target_file = File.join(@test_dir, 'verbose_test_targets.txt') + + # Test different thread counts with verbose output to screen + thread_counts = [1, 10, 25] + results = {} + + thread_counts.each do |threads| + print "Testing verbose output with #{threads} threads... " + + start_time = Time.now + # Use -v for verbose output to screen, redirect to /dev/null to avoid cluttering test output + cmd = "#{@whatweb_path} -v -t #{threads} -i #{target_file} > /dev/null" + success = system(cmd + " 2>/dev/null") + end_time = Time.now + + if success + time = end_time - start_time + results[threads] = time + puts "#{time.round(2)}s" + else + puts "FAILED" + end + end + + puts "\n=== Verbose Screen Performance Results ===" + results.each do |threads, time| + speedup = results[1] ? (results[1] / time).round(2) : 'N/A' + puts "#{threads} threads: #{time.round(2)}s (#{speedup}x speedup)" + end + + # Test with output sync forced on vs off + puts "\n=== Output Sync Comparison (25 threads) ===" + + print "Testing with --output-sync... " + start_time = Time.now + cmd = "#{@whatweb_path} -v --output-sync -t 25 -i #{target_file} > /dev/null" + success = system(cmd + " 2>/dev/null") + sync_time = Time.now - start_time + puts success ? "#{sync_time.round(2)}s" : "FAILED" + + print "Testing without --output-sync... " + start_time = Time.now + cmd = "#{@whatweb_path} -v -t 25 -i #{target_file} > /dev/null" + success = system(cmd + " 2>/dev/null") + async_time = Time.now - start_time + puts success ? "#{async_time.round(2)}s" : "FAILED" + + if sync_time && async_time + improvement = ((sync_time - async_time) / sync_time * 100).round(1) + puts "Performance improvement: #{improvement}% faster without sync" + end + end + + def run_test + puts "Verbose Screen Output Performance Test" + puts "=====================================" + puts "Timestamp: #{Time.now}" + puts + + # Check if whatweb exists + unless File.exist?(@whatweb_path) + puts "ERROR: WhatWeb not found at #{@whatweb_path}" + exit 1 + end + + test_verbose_screen_performance + + puts "\nTest completed. Check results above." + end +end + +# Run the test if this script is executed directly +if __FILE__ == $0 + test = VerboseScreenTest.new + test.run_test +end diff --git a/test/verbose_test_targets.txt b/test/verbose_test_targets.txt new file mode 100644 index 00000000..c094a70d --- /dev/null +++ b/test/verbose_test_targets.txt @@ -0,0 +1,3 @@ +http://example.com +http://httpbin.org/html +http://httpbin.org/json \ No newline at end of file diff --git a/whatweb b/whatweb index bbcd56de..0597a1cf 100755 --- a/whatweb +++ b/whatweb @@ -34,6 +34,27 @@ $LOAD_PATH.unshift(File.join(File.expand_path(__dir__)), '.') +# Performance profiling infrastructure (Phase 1) +if ENV['WHATWEB_PROFILE'] + begin + require 'ruby-prof' + RubyProf.start + at_exit do + result = RubyProf.stop + printer = RubyProf::FlatPrinter.new(result) + if ENV['WHATWEB_PROFILE_FILE'] + File.open(ENV['WHATWEB_PROFILE_FILE'], 'w') { |f| printer.print(f) } + STDERR.puts "Profile saved to #{ENV['WHATWEB_PROFILE_FILE']}" + else + printer.print(STDERR) + end + end + STDERR.puts "Ruby profiling enabled" + rescue LoadError + STDERR.puts "Warning: ruby-prof gem not available. Install with: gem install ruby-prof" + end +end + require 'lib/whatweb' # @@ -156,6 +177,10 @@ PERFORMANCE & STABILITY: --read-timeout\t\tTime in seconds. Default: #{$HTTP_READ_TIMEOUT}. --wait=SECONDS\t\tWait SECONDS between connections. \t\t\t\tThis is useful when using a single thread. + --output-sync\t\t\tForce immediate output flushing for real-time +\t\t\t\tmonitoring (slower with high thread counts). + --output-buffer-size=SIZE\tSet output buffer size. 0=unbuffered, +\t\t\t\tdefault=auto based on thread count. HELP & MISCELLANEOUS: --short-help\t\t\tShort usage help. @@ -363,7 +388,9 @@ opts = GetoptLong.new( ['--wait', GetoptLong::REQUIRED_ARGUMENT], ['--debug', GetoptLong::NO_ARGUMENT], ['--version', GetoptLong::NO_ARGUMENT], - ['-q', '--quiet', GetoptLong::NO_ARGUMENT] + ['-q', '--quiet', GetoptLong::NO_ARGUMENT], + ['--output-sync', GetoptLong::NO_ARGUMENT], + ['--output-buffer-size', GetoptLong::REQUIRED_ARGUMENT] ) begin @@ -570,6 +597,10 @@ begin when '--version' puts "WhatWeb version #{WhatWeb::VERSION} ( https://morningstarsecurity.com/research/whatweb/ )" exit + when '--output-sync' + $OUTPUT_SYNC = true + when '--output-buffer-size' + $OUTPUT_BUFFER_SIZE = arg.to_i end end rescue Errno::EPIPE @@ -602,10 +633,26 @@ end # optimise plugins PluginSupport.precompile_regular_expressions +# Store thread count globally for logging classes (before creating loggers) +$THREADS = max_threads + +# Configure smart defaults for output performance based on thread count +unless defined?($OUTPUT_SYNC) + $OUTPUT_SYNC = (max_threads <= 5) # Sync for low thread counts +end + +unless defined?($OUTPUT_BUFFER_SIZE) + $OUTPUT_BUFFER_SIZE = case max_threads + when 1..5 then 1 # No buffering for low threads + when 6..50 then 10 # Small buffer for medium (includes default 25) + else 25 # Larger buffer for high threads (51+) + end +end + ### OUTPUT -logging_list << LoggingBrief.new unless $QUIET || $verbose > 0 # by default logging brief +logging_list << LoggingBrief.new(STDOUT, sync: $OUTPUT_SYNC) unless $QUIET || $verbose > 0 # by default logging brief logging_list << LoggingObject.new if $verbose > 1 # full logging if -vv -logging_list << LoggingVerbose.new if $verbose > 0 # full logging if -v +logging_list << LoggingVerbose.new(STDOUT, sync: $OUTPUT_SYNC) if $verbose > 0 # full logging if -v ## logging dependencies if mongo[:use_mongo_log] @@ -634,7 +681,6 @@ end urls = ARGV - # Initialize Scanner begin scanner = WhatWeb::Scan.new(