performance improvements!!!

CHANGELOG.md

@@ -1,3 +1,7 @@
+Version 0.6.4 - ??
+
+
+
 Version 0.6.3 - October 18, 2025
 
 ## FEATURES
@@ -5,6 +9,15 @@ Version 0.6.3 - October 18, 2025
 * Added --no-cookies option to disable cookie handling.
 * Added --cookie-jar option to save cookies to a file.
 
+## PERFORMANCE IMPROVEMENTS
+* Replaced global output mutex with per-logger mutexes for better concurrency
+* Removed excessive STDERR.flush calls that were causing I/O bottlenecks  
+* Optimized verbose logging to build output atomically before writing
+* Added configurable output buffering for high-thread scenarios
+* Made output sync behavior configurable (--output-sync flag)
+* Added smart defaults that automatically optimize based on thread count
+* Added profiling infrastructure (WHATWEB_PROFILE environment variable)
+
 ## FIXES
 * Fixed error with HTTPS connections when using a proxy server
 * Fix typo in plugin-tutorial-7.rb (@mtisec)

README.md

@@ -42,7 +42,7 @@ Most WhatWeb plugins are thorough and recognise a range of cues from subtle to o
 ### Features
 * Over 1800 plugins
 * Control the trade off between speed/stealth and reliability
-* Performance tuning. Control how many websites to scan concurrently.
+* Performance tuning. Control how many websites to scan concurrently with automatic output optimization.
 * Multiple log formats: Brief (greppable), Verbose (human readable), XML, JSON, MagicTree, RubyObject, MongoDB, ElasticSearch, SQL.
 * Proxy support including TOR
 * Custom HTTP headers
@@ -358,9 +358,73 @@ WhatWeb features several options to increase performance and stability.
 *  --read-timeout        Time in seconds. Default: 30
 *  --wait=SECONDS        Wait SECONDS between connections
                         This is useful when using a single thread.
+*  --output-sync         Force immediate output flushing to screen and logs for 
+                        real-time monitoring (slower with high thread counts).
+*  --output-buffer-size=SIZE  Set output buffer size for screen and log output. 
+                        0=unbuffered, default=auto based on thread count.
 
 The --wait and --max-threads commands can be used to assist in IDS evasion.
 
+### Output Performance Optimization
+
+WhatWeb automatically optimizes output performance for both screen output and log files based on thread count:
+
+* **Low thread counts (1-5)**: 
+  - **Sync**: Enabled (immediate flushing)
+  - **Buffer size**: 1 (minimal buffering for real-time output)
+  - **Use case**: Interactive scanning, real-time monitoring
+
+* **Medium thread counts (6-50)**: 
+  - **Sync**: Disabled (async I/O)
+  - **Buffer size**: 10 (small buffer for balanced performance)
+  - **Use case**: Standard batch scanning including default (25 threads)
+
+* **High thread counts (51+)**: 
+  - **Sync**: Disabled (async I/O)
+  - **Buffer size**: 25 (larger buffer to minimize I/O contention)
+  - **Use case**: High-volume scanning, large target lists (100+ threads)
+
+You can override these defaults for both screen output and log files:
+
+```bash
+# Force real-time output to screen and logs (slower but immediate)
+./whatweb --output-sync -t 100 --log-brief=results.txt targets.txt
+
+# Custom buffer size for high-volume logging
+./whatweb --output-buffer-size=50 -t 100 --log-verbose=detailed.log targets.txt
+
+# Disable buffering entirely for immediate file writes
+./whatweb --output-buffer-size=0 -t 10 --log-json=results.json targets.txt
+```
+
+These optimizations apply to all output formats including brief, verbose, XML, JSON, and other log formats.
+
+#### Technical Details
+
+**Output Sync Behavior:**
+- **Enabled**: Each output line is immediately flushed to disk/screen (slower but real-time)
+- **Disabled**: Output is buffered in memory before writing (faster but delayed)
+
+**Buffer Size Impact:**
+- **Size 1**: Minimal buffering, nearly immediate output
+- **Size 10**: Small buffer holds ~10 results before writing
+- **Size 25+**: Larger buffer reduces I/O system calls for better performance
+- **Size 0**: Completely unbuffered (equivalent to sync enabled)
+
+### Profiling Support
+
+Enable performance profiling to identify bottlenecks:
+
+```bash
+# Basic profiling to stderr
+WHATWEB_PROFILE=1 ./whatweb -t 25 targets.txt
+
+# Save profile to file
+WHATWEB_PROFILE=1 WHATWEB_PROFILE_FILE=profile.txt ./whatweb -t 25 targets.txt
+```
+
+*Note: Profiling requires the `ruby-prof` gem: `gem install ruby-prof`*
+
 Changing the user-agent using the -U or --user-agent command line option will avoid the Snort IDS rule for WhatWeb.
 
 If you are scanning ranges of IP addresses, it is much more efficient to use a port scanner like massscan to discover which have port 80 open before scanning with WhatWeb.

lib/logging.rb

@@ -20,17 +20,78 @@ class Logging
 
   # if no f, output to STDOUT,
   # if f is a filename then open it, if f is a file use it
-  def initialize(f = STDOUT)
+  def initialize(f = STDOUT, sync: nil)
     f = STDOUT if f == '-'
     @f = f if f.class == IO || f.class == File
     @f = File.open(f, 'a') if f.class == String
-    @f.sync = true # we want flushed output
+    
+    # Make sync configurable with smart defaults
+    # Default to sync for low thread counts, async for high thread counts
+    if sync.nil?
+      @f.sync = ($THREADS && $THREADS <= 5) ? true : false
+    else
+      @f.sync = sync
+    end
+    
+    # Per-instance mutex for thread-safe output
+    @mutex = Mutex.new
+    
+    # Output buffering for performance
+    @buffer = []
+    @buffer_size = $OUTPUT_BUFFER_SIZE || 1
+    @last_flush = Time.now
+    
+    # Ensure buffer size is at least 1
+    @buffer_size = 1 if @buffer_size < 1
   end
 
   def close
+    flush_buffer # Ensure all buffered output is written
     @f.close unless @f.class == IO
   end
 
+  # Thread-safe output method using per-instance mutex
+  def synchronized_write(&block)
+    @mutex.synchronize(&block)
+  end
+  
+  # Buffered output method for improved performance
+  def buffered_write(content)
+    @mutex.synchronize do
+      @buffer << content
+      
+      # Flush buffer if it's full, in sync mode, or buffer is getting stale
+      should_flush = @buffer.size >= @buffer_size || 
+                     @f.sync || 
+                     (Time.now - @last_flush > 5.0 && !@buffer.empty?)
+      
+      if should_flush
+        flush_buffer_unsafe
+      end
+    end
+  end
+  
+  # Force flush the buffer (thread-safe)
+  def flush_buffer
+    @mutex.synchronize do
+      flush_buffer_unsafe
+    end
+  end
+  
+  private
+  
+  # Flush buffer without mutex (must be called within synchronized block)
+  def flush_buffer_unsafe
+    return if @buffer.empty?
+    
+    # Join all buffered content with newlines and write as one block
+    combined_output = @buffer.join("\n")
+    @f.puts combined_output
+    
+    @buffer.clear
+    @last_flush = Time.now
+  end
+
   # perform sort, uniq and join on each plugin result
   def suj(plugin_results)
     suj = {}

lib/logging/brief.rb

@@ -88,8 +88,7 @@ class LoggingBrief < Logging
     else
       brief_results_final = "#{target} [#{status} #{status_code}] #{brief_results.join(', ')}"
     end
-    $semaphore.synchronize do
-      @f.puts brief_results_final
-    end
+    # Use buffered write for better performance
+    buffered_write(brief_results_final)
   end
 end

lib/logging/verbose.rb

@@ -10,191 +10,214 @@ class LoggingVerbose < Logging
   end
 
   def out(target, status, results)
-    $semaphore.synchronize do
-      # make a hash of the matches array
-      results_hash = {}
-      results.map { |k, v| results_hash[k] = v }
+    # Build all output first, then write as a complete block
+    output_lines = build_verbose_output(target, status, results)
+    
+    # Join lines into a single string and write as one block
+    output_block = output_lines.join("\n")
+    buffered_write(output_block)
+  end
 
-      display = {
-        title: '<None>',
-        ip: '<Unknown>',
-        country: '<Unknown>',
-        status: '<Unknown>'
-      }
+  private
 
-      display[:country] = results_hash['Country'].map { |r| "#{r[:string]}, #{r[:module]}" }.join(',') if results_hash['Country']
-      display[:ip] = results_hash['IP'].map { |r| r[:string] }.join(',') if results_hash['Country']
-      display[:title] = results_hash['Title'].map { |r| r[:string] }.join(',') if results_hash['Title']
-      display[:status] = status.to_s + ' ' + HTTP_Status.code(status)
+  def build_verbose_output(target, status, results)
+    lines = []
+    
+    # make a hash of the matches array
+    results_hash = {}
+    results.map { |k, v| results_hash[k] = v }
 
-      @f.puts "WhatWeb report for #{coloured(target, 'blue')}"
-      @f.puts 'Status'.ljust(9) + ' : ' + display[:status]
-      @f.puts 'Title'.ljust(9) + " : #{coloured(display[:title], 'yellow')}"
-      @f.puts 'IP'.ljust(9) + ' : ' + display[:ip]
-      @f.puts 'Country'.ljust(9) + " : #{coloured(display[:country], 'red')}"
-      @f.puts
+    display = {
+      title: '<None>',
+      ip: '<Unknown>',
+      country: '<Unknown>',
+      status: '<Unknown>'
+    }
 
-      ################### Short list
-      ################### Basically Output Brief
+    display[:country] = results_hash['Country'].map { |r| "#{r[:string]}, #{r[:module]}" }.join(',') if results_hash['Country']
+    display[:ip] = results_hash['IP'].map { |r| r[:string] }.join(',') if results_hash['Country']
+    display[:title] = results_hash['Title'].map { |r| r[:string] }.join(',') if results_hash['Title']
+    display[:status] = status.to_s + ' ' + HTTP_Status.code(status)
 
-      brief_results = []
-      results.each do |plugin_name, plugin_results|
-        next if %w(Title IP Country).include? plugin_name
+    # Header section
+    lines << "WhatWeb report for #{coloured(target, 'blue')}"
+    lines << 'Status'.ljust(9) + ' : ' + display[:status]
+    lines << 'Title'.ljust(9) + " : #{coloured(display[:title], 'yellow')}"
+    lines << 'IP'.ljust(9) + ' : ' + display[:ip]
+    lines << 'Country'.ljust(9) + " : #{coloured(display[:country], 'red')}"
+    lines << ""
 
-        next if plugin_results.empty?
-        suj = suj(plugin_results)
+    # Brief results section (similar to brief output)
+    brief_results = build_brief_results(results)
+    lines << 'Summary'.ljust(9) + ' : ' + brief_results.join(', ')
+    lines << ""
+    lines << 'Detected Plugins:'
 
-        certainty = suj[:certainty].to_i
-        version = suj[:version]
-        os = suj[:os]
-        string = suj[:string]
-        accounts = suj[:account]
-        model = suj[:model]
-        firmware = suj[:firmware]
-        modules = suj[:module]
-        filepath = suj[:filepath]
+    # Plugin details section
+    lines.concat(build_plugin_details(results))
 
-        # colour the output
-        # be more DRY
-        # if plugins have categories or tags this would be better, eg. all hash plugins are grey
-        if (@f == STDOUT && $use_colour == 'auto') || ($use_colour == 'always')
-          coloured_string = grey(string)
-          coloured_string = cyan(string) if plugin_name == 'HTTPServer'
-          coloured_string = yellow(string) if plugin_name == 'Title'
+    # HTTP Headers section
+    lines << 'HTTP Headers:'
+    target.raw_headers.each_line do |header|
+      lines << "\t#{header.chomp}"
+    end
+    
+    lines
+  end
 
-          coloured_string = grey(string) if plugin_name == 'MD5'
-          coloured_string = grey(string) if plugin_name == 'Header-Hash'
-          coloured_string = grey(string) if plugin_name == 'Footer-Hash'
-          coloured_string = grey(string) if plugin_name == 'CSS'
-          coloured_string = grey(string) if plugin_name == 'Tag-Hash'
+  def build_brief_results(results)
+    brief_results = []
+    results.each do |plugin_name, plugin_results|
+      next if %w(Title IP Country).include? plugin_name
+      next if plugin_results.empty?
+      
+      suj = suj(plugin_results)
 
-          coloured_plugin = white(plugin_name)
-          coloured_plugin = grey(plugin_name) if plugin_name == 'MD5'
-          coloured_plugin = grey(plugin_name) if plugin_name == 'Header-Hash'
-          coloured_plugin = grey(plugin_name) if plugin_name == 'Footer-Hash'
-          coloured_plugin = grey(plugin_name) if plugin_name == 'CSS'
-          coloured_plugin = grey(plugin_name) if plugin_name == 'Tag-Hash'
+      certainty = suj[:certainty].to_i
+      version = suj[:version]
+      os = suj[:os]
+      string = suj[:string]
+      accounts = suj[:account]
+      model = suj[:model]
+      firmware = suj[:firmware]
+      modules = suj[:module]
+      filepath = suj[:filepath]
 
-          p = ((certainty && certainty < 100) ? "#{grey(Helper::certainty_to_words(certainty))} " : '') +
-              coloured_plugin + (!version.empty? ? "[#{green(version)}]" : '') +
-              (!os.empty? ? "[#{red(os)}]" : '') +
-              (!string.empty? ? "[#{coloured_string}]" : '') +
-              (!accounts.empty? ? "[#{cyan(accounts)}]" : '') +
-              (!model.empty? ? "[#{dark_green(model)}]" : '') +
-              (!firmware.empty? ? "[#{dark_green(firmware)}]" : '') +
-              (!filepath.empty? ? "[#{dark_green(filepath)}]" : '') +
-              (!modules.empty? ? "[#{magenta(modules)}]" : '')
+      # colour the output
+      if (@f == STDOUT && $use_colour == 'auto') || ($use_colour == 'always')
+        coloured_string = grey(string)
+        coloured_string = cyan(string) if plugin_name == 'HTTPServer'
+        coloured_string = yellow(string) if plugin_name == 'Title'
 
-          brief_results << p
-        else
-          brief_results << ((certainty && certainty < 100) ? "#{Helper::certainty_to_words(certainty)} " : '') +
-                           plugin_name + (!version.empty? ? "[#{version}]" : '') +
-                           (!os.empty? ? "[#{os}]" : '') +
-                           (!string.empty? ? "[#{string}]" : '') +
-                           (!accounts.empty? ? " [#{accounts}]" : '') +
-                           (!model.empty? ? "[#{model}]" : '') +
-                           (!firmware.empty? ? "[#{firmware}]" : '') +
-                           (!filepath.empty? ? "[#{filepath}]" : '') +
-                           (!modules.empty? ? "[#{modules}]" : '')
-        end
-      end
+        coloured_string = grey(string) if plugin_name == 'MD5'
+        coloured_string = grey(string) if plugin_name == 'Header-Hash'
+        coloured_string = grey(string) if plugin_name == 'Footer-Hash'
+        coloured_string = grey(string) if plugin_name == 'CSS'
+        coloured_string = grey(string) if plugin_name == 'Tag-Hash'
 
-      brief_results_final = brief_results.join(', ')
-      @f.puts 'Summary'.ljust(9) + ' : ' + brief_results_final
-      @f.puts
-      @f.puts 'Detected Plugins:'
+        coloured_plugin = white(plugin_name)
+        coloured_plugin = grey(plugin_name) if plugin_name == 'MD5'
+        coloured_plugin = grey(plugin_name) if plugin_name == 'Header-Hash'
+        coloured_plugin = grey(plugin_name) if plugin_name == 'Footer-Hash'
+        coloured_plugin = grey(plugin_name) if plugin_name == 'CSS'
+        coloured_plugin = grey(plugin_name) if plugin_name == 'Tag-Hash'
 
-      results.sort.each do |plugin_name, plugin_results|
-        next if %w(Title IP Country).include?(plugin_name)
-        next if plugin_results.empty?
-        @f.puts "[ #{coloured(plugin_name, 'white')} ]"
-        description = ['']
-        if Plugin.registered_plugins[plugin_name].description
-          d = Plugin.registered_plugins[plugin_name].description
-          description = Helper::word_wrap(d, 60)
-        end
-        #          @f.puts "\tCategory   : " + Plugin.registered_plugins[plugin_name].category.first unless Plugin.registered_plugins[plugin_name].category.nil?
-        @f.puts "\t" + description.first
-        description[1..-1].each { |line| @f.puts "\t" + line }
-        @f.puts
+        p = ((certainty && certainty < 100) ? "#{grey(Helper::certainty_to_words(certainty))} " : '') +
+            coloured_plugin + (!version.empty? ? "[#{green(version)}]" : '') +
+            (!os.empty? ? "[#{red(os)}]" : '') +
+            (!string.empty? ? "[#{coloured_string}]" : '') +
+            (!accounts.empty? ? "[#{cyan(accounts)}]" : '') +
+            (!model.empty? ? "[#{dark_green(model)}]" : '') +
+            (!firmware.empty? ? "[#{dark_green(firmware)}]" : '') +
+            (!filepath.empty? ? "[#{dark_green(filepath)}]" : '') +
+            (!modules.empty? ? "[#{magenta(modules)}]" : '')
 
-        top_certainty = suj(plugin_results)[:certainty].to_i
-        unless top_certainty == 100
-          @f.puts "\t" + 'Certainty'.ljust(13) + ': ' + Helper::certainty_to_words(top_certainty)
-        end
-
-        plugin_results.map { |x| sortuniq(x) }.each do |pr|
-          if pr[:name]
-            name_of_match = pr[:name]
-          else
-            name_of_match = [pr[:regexp_compiled], pr[:text], pr[:regexp].to_s,
-                             pr[:ghdb], pr[:md5], pr[:tagpattern]].compact.join('|')
-          end
-
-          pr.each do |key, value|
-            next unless [:version, :os, :string, :account, :model,
-                         :firmware, :module, :filepath, :url].include?(key)
-
-            next if value.class == Regexp
-
-            if key == :os
-              @f.print "\t" + 'OS'.ljust(13) + ': '
-            else
-              @f.print "\t" + key.to_s.capitalize.ljust(13) + ': '
-            end
-
-            c = case key
-                when :version then 'green'
-                when :string then 'cyan'
-                when :certainty then 'grey'
-                when :os then 'red'
-                when :account then 'cyan'
-                when :model then 'dark_green'
-                when :firmware then 'dark_green'
-                when :module then 'magenta'
-                when :filepath then 'dark_green'
-                else 'grey'
-                end
-
-            if value.is_a?(String)
-              @f.print coloured(value.to_s, c)
-            elsif value.is_a?(Array)
-              @f.print coloured(value.join(',').to_s, c)
-            else
-              @f.print coloured(value.inspect, c)
-            end
-
-            @f.print " (from #{name_of_match})" unless name_of_match.empty?
-            unless pr[:certainty] == 100
-              @f.print " (Certainty: #{ertainty_to_words pr[:certainty]} )"
-            end
-
-            @f.puts
-          end
-
-          @f.puts "\t" + coloured(pr.inspect.to_s, 'dark_blue') if $verbose > 1
-        end
-
-        if Plugin.registered_plugins[plugin_name].aggressive
-          @f.puts "\tAggressive function available (check plugin file or details)."
-        end
-
-        if Plugin.registered_plugins[plugin_name].dorks.any?
-          @f.puts "\tGoogle Dorks".ljust(13) + ": (#{Plugin.registered_plugins[plugin_name].dorks.size})"
-        end
-
-        if Plugin.registered_plugins[plugin_name].website
-          @f.puts "\tWebsite".ljust(13) + ": #{Plugin.registered_plugins[plugin_name].website}"
-        end
-
-        @f.puts
-      end
-
-      @f.puts 'HTTP Headers:'
-      target.raw_headers.each_line do |header|
-        @f.puts "\t#{header}"
-        # pp target.raw_headers
+        brief_results << p
+      else
+        brief_results << ((certainty && certainty < 100) ? "#{Helper::certainty_to_words(certainty)} " : '') +
+                         plugin_name + (!version.empty? ? "[#{version}]" : '') +
+                         (!os.empty? ? "[#{os}]" : '') +
+                         (!string.empty? ? "[#{string}]" : '') +
+                         (!accounts.empty? ? " [#{accounts}]" : '') +
+                         (!model.empty? ? "[#{model}]" : '') +
+                         (!firmware.empty? ? "[#{firmware}]" : '') +
+                         (!filepath.empty? ? "[#{filepath}]" : '') +
+                         (!modules.empty? ? "[#{modules}]" : '')
       end
     end
+    
+    brief_results
+  end
+
+  def build_plugin_details(results)
+    lines = []
+    
+    results.sort.each do |plugin_name, plugin_results|
+      next if %w(Title IP Country).include?(plugin_name)
+      next if plugin_results.empty?
+      
+      lines << "[ #{coloured(plugin_name, 'white')} ]"
+      description = ['']
+      if Plugin.registered_plugins[plugin_name].description
+        d = Plugin.registered_plugins[plugin_name].description
+        description = Helper::word_wrap(d, 60)
+      end
+      lines << "\t" + description.first
+      description[1..-1].each { |line| lines << "\t" + line }
+      lines << ""
+
+      top_certainty = suj(plugin_results)[:certainty].to_i
+      unless top_certainty == 100
+        lines << "\t" + 'Certainty'.ljust(13) + ': ' + Helper::certainty_to_words(top_certainty)
+      end
+
+      plugin_results.map { |x| sortuniq(x) }.each do |pr|
+        if pr[:name]
+          name_of_match = pr[:name]
+        else
+          name_of_match = [pr[:regexp_compiled], pr[:text], pr[:regexp].to_s,
+                           pr[:ghdb], pr[:md5], pr[:tagpattern]].compact.join('|')
+        end
+
+        pr.each do |key, value|
+          next unless [:version, :os, :string, :account, :model,
+                       :firmware, :module, :filepath, :url].include?(key)
+          next if value.class == Regexp
+
+          line_parts = []
+          if key == :os
+            line_parts << "\t" + 'OS'.ljust(13) + ': '
+          else
+            line_parts << "\t" + key.to_s.capitalize.ljust(13) + ': '
+          end
+
+          c = case key
+              when :version then 'green'
+              when :string then 'cyan'
+              when :certainty then 'grey'
+              when :os then 'red'
+              when :account then 'cyan'
+              when :model then 'dark_green'
+              when :firmware then 'dark_green'
+              when :module then 'magenta'
+              when :filepath then 'dark_green'
+              else 'grey'
+              end
+
+          if value.is_a?(String)
+            line_parts << coloured(value.to_s, c)
+          elsif value.is_a?(Array)
+            line_parts << coloured(value.join(',').to_s, c)
+          else
+            line_parts << coloured(value.inspect, c)
+          end
+
+          line_parts << " (from #{name_of_match})" unless name_of_match.empty?
+          unless pr[:certainty] == 100
+            line_parts << " (Certainty: #{Helper::certainty_to_words pr[:certainty]} )"
+          end
+
+          lines << line_parts.join('')
+        end
+
+        lines << "\t" + coloured(pr.inspect.to_s, 'dark_blue') if $verbose > 1
+      end
+
+      if Plugin.registered_plugins[plugin_name].aggressive
+        lines << "\tAggressive function available (check plugin file or details)."
+      end
+
+      if Plugin.registered_plugins[plugin_name].dorks.any?
+        lines << "\tGoogle Dorks".ljust(13) + ": (#{Plugin.registered_plugins[plugin_name].dorks.size})"
+      end
+
+      if Plugin.registered_plugins[plugin_name].website
+        lines << "\tWebsite".ljust(13) + ": #{Plugin.registered_plugins[plugin_name].website}"
+      end
+
+      lines << ""
+    end
+    
+    lines
   end
 end

lib/messages.rb

@@ -31,7 +31,7 @@ def error(s)
   $semaphore.reentrant_synchronize do
     # TODO: make use_color smart, so it detects a tty
     STDERR.puts((($use_colour == 'auto') || ($use_colour == 'always')) ? red(s) : s)
-    STDERR.flush
+    # REMOVED: STDERR.flush for better performance
     $LOG_ERRORS.out(s) if $LOG_ERRORS
   end
 end
@@ -44,7 +44,7 @@ def warning(s)
 
   $semaphore.reentrant_synchronize do
     STDERR.puts((($use_colour == 'auto') || ($use_colour == 'always')) ? yellow(s) : s)
-    STDERR.flush
+    # REMOVED: STDERR.flush for better performance
     $LOG_ERRORS.out("WARNING: #{s}") if $LOG_ERRORS
   end
 end
@@ -57,7 +57,7 @@ def notice(s)
 
   $semaphore.reentrant_synchronize do
     STDERR.puts((($use_colour == 'auto') || ($use_colour == 'always')) ? blue(s) : s)
-    STDERR.flush
+    # REMOVED: STDERR.flush for better performance
   end
 end
 
@@ -69,6 +69,6 @@ def debug(s)
 
   $semaphore.reentrant_synchronize do
     STDERR.puts((($use_colour == 'auto') || ($use_colour == 'always')) ? grey("[DEBUG] #{s}") : "[DEBUG] #{s}")
-    STDERR.flush
+    # REMOVED: STDERR.flush for better performance
   end
 end

test/optimization_targets.txt

@@ -0,0 +1,3 @@
+http://example.com
+http://httpbin.org/html
+http://httpbin.org/json

test/output_optimization_test.rb

@@ -0,0 +1,175 @@
+#!/usr/bin/env ruby
+#
+# Output optimization validation test
+# Tests the new performance improvements
+#
+
+require 'benchmark'
+require 'fileutils'
+
+class OutputOptimizationTest
+  def initialize
+    @whatweb_path = File.expand_path('../whatweb', __dir__)
+    @test_dir = __dir__
+    @results_dir = File.join(@test_dir, 'optimization_results')
+    FileUtils.mkdir_p(@results_dir)
+    
+    create_test_targets
+  end
+
+  def create_test_targets
+    # Create a small test target file
+    test_targets = File.join(@test_dir, 'optimization_targets.txt')
+    unless File.exist?(test_targets)
+      File.write(test_targets, [
+        'http://example.com',
+        'http://httpbin.org/html',
+        'http://httpbin.org/json'
+      ].join("\n"))
+    end
+  end
+
+  def test_new_options
+    puts "=== Testing New Command Line Options ==="
+    
+    target_file = File.join(@test_dir, 'optimization_targets.txt')
+    
+    # Test --output-sync option
+    puts "Testing --output-sync option..."
+    cmd = "#{@whatweb_path} --output-sync -t 5 --log-brief=/dev/null -i #{target_file}"
+    success = system(cmd + " 2>/dev/null")
+    puts success ? "✓ --output-sync works" : "✗ --output-sync failed"
+    
+    # Test --output-buffer-size option
+    puts "Testing --output-buffer-size option..."
+    cmd = "#{@whatweb_path} --output-buffer-size=10 -t 5 --log-brief=/dev/null -i #{target_file}"
+    success = system(cmd + " 2>/dev/null")
+    puts success ? "✓ --output-buffer-size works" : "✗ --output-buffer-size failed"
+    
+    # Test help text includes new options
+    puts "Testing help text includes new options..."
+    help_output = `#{@whatweb_path} --help 2>&1`
+    has_sync = help_output.include?('--output-sync')
+    has_buffer = help_output.include?('--output-buffer-size')
+    puts has_sync && has_buffer ? "✓ Help text updated" : "✗ Help text missing new options"
+  end
+
+  def test_performance_comparison
+    puts "\n=== Performance Comparison Test ==="
+    
+    target_file = File.join(@test_dir, 'optimization_targets.txt')
+    
+    # Test with different thread counts to verify smart defaults
+    thread_counts = [1, 10, 25]
+    results = {}
+    
+    thread_counts.each do |threads|
+      print "Testing #{threads} threads... "
+      
+      start_time = Time.now
+      cmd = "#{@whatweb_path} -t #{threads} --log-brief=/dev/null -i #{target_file}"
+      success = system(cmd + " 2>/dev/null")
+      end_time = Time.now
+      
+      if success
+        time = end_time - start_time
+        results[threads] = time
+        puts "#{time.round(2)}s"
+      else
+        puts "FAILED"
+      end
+    end
+    
+    puts "\n=== Performance Results ==="
+    results.each do |threads, time|
+      puts "#{threads} threads: #{time.round(2)}s"
+    end
+  end
+
+  def test_mutex_functionality
+    puts "\n=== Mutex Functionality Test ==="
+    
+    # Test that output is still correct with high thread count
+    target_file = File.join(@test_dir, 'optimization_targets.txt')
+    output_file = File.join(@results_dir, 'mutex_test_output.txt')
+    
+    puts "Testing output consistency with 20 threads..."
+    cmd = "#{@whatweb_path} -t 20 --log-brief=#{output_file} -i #{target_file}"
+    success = system(cmd + " 2>/dev/null")
+    
+    if success && File.exist?(output_file)
+      lines = File.readlines(output_file)
+      puts "✓ Generated #{lines.length} output lines"
+      
+      # Check for any obvious corruption (partial lines, etc.)
+      corrupted_lines = lines.select { |line| line.strip.empty? || line.length < 10 }
+      if corrupted_lines.empty?
+        puts "✓ No obviously corrupted output lines"
+      else
+        puts "✗ Found #{corrupted_lines.length} potentially corrupted lines"
+      end
+      
+      File.delete(output_file)
+    else
+      puts "✗ Mutex test failed"
+    end
+  end
+
+  def test_profiling_infrastructure
+    puts "\n=== Profiling Infrastructure Test ==="
+    
+    target_file = File.join(@test_dir, 'optimization_targets.txt')
+    profile_file = File.join(@results_dir, 'test_profile.txt')
+    
+    puts "Testing profiling with WHATWEB_PROFILE environment variable..."
+    
+    env = {
+      'WHATWEB_PROFILE' => '1',
+      'WHATWEB_PROFILE_FILE' => profile_file
+    }
+    
+    cmd = "#{@whatweb_path} -t 5 --log-brief=/dev/null -i #{target_file}"
+    success = system(env, cmd + " 2>/dev/null")
+    
+    if success
+      if File.exist?(profile_file)
+        puts "✓ Profile file created"
+        profile_size = File.size(profile_file)
+        puts "✓ Profile file size: #{profile_size} bytes"
+        File.delete(profile_file) if profile_size > 0
+      else
+        puts "✗ Profile file not created (ruby-prof may not be installed)"
+      end
+    else
+      puts "✗ Profiling test failed"
+    end
+  end
+
+  def run_all_tests
+    puts "WhatWeb Output Optimization Test Suite"
+    puts "======================================"
+    puts "Timestamp: #{Time.now}"
+    puts
+    
+    # Check if whatweb exists
+    unless File.exist?(@whatweb_path)
+      puts "ERROR: WhatWeb not found at #{@whatweb_path}"
+      exit 1
+    end
+    
+    test_new_options
+    test_performance_comparison
+    test_mutex_functionality
+    test_profiling_infrastructure
+    
+    puts "\n=== Test Summary ==="
+    puts "All optimization tests completed."
+    puts "Check output above for any failures marked with ✗"
+  end
+end
+
+# Run the tests if this script is executed directly
+if __FILE__ == $0
+  test = OutputOptimizationTest.new
+  test.run_all_tests
+end

test/performance_test.rb

@@ -0,0 +1,239 @@
+#!/usr/bin/env ruby
+#
+# Performance test script for WhatWeb output optimization
+# Phase 1: Baseline measurement and profiling
+#
+
+require 'benchmark'
+require 'fileutils'
+
+class WhatWebPerformanceTest
+  def initialize
+    @whatweb_path = File.expand_path('../whatweb', __dir__)
+    @test_dir = __dir__
+    @results_dir = File.join(@test_dir, 'performance_results')
+    FileUtils.mkdir_p(@results_dir)
+    
+    # Create test target files if they don't exist
+    create_test_targets
+  end
+
+  def create_test_targets
+    # Small target list for quick tests
+    small_targets = File.join(@test_dir, 'small_targets.txt')
+    unless File.exist?(small_targets)
+      File.write(small_targets, [
+        'http://example.com',
+        'http://httpbin.org/html',
+        'http://httpbin.org/json',
+        'http://httpbin.org/xml',
+        'http://httpbin.org/redirect/2'
+      ].join("\n"))
+    end
+
+    # Medium target list for realistic tests
+    medium_targets = File.join(@test_dir, 'medium_targets.txt')
+    unless File.exist?(medium_targets)
+      targets = []
+      # Add some real websites for testing
+      base_sites = [
+        'example.com', 'httpbin.org', 'github.com', 'stackoverflow.com',
+        'reddit.com', 'wikipedia.org', 'google.com', 'youtube.com',
+        'amazon.com', 'twitter.com'
+      ]
+      
+      base_sites.each do |site|
+        targets << "http://#{site}"
+        targets << "https://#{site}"
+      end
+      
+      File.write(medium_targets, targets.join("\n"))
+    end
+  end
+
+  def run_whatweb_test(threads, target_file, output_format = 'brief')
+    log_file = "/tmp/whatweb_test_#{threads}t_#{output_format}.log"
+    
+    cmd = case output_format
+          when 'brief'
+            "#{@whatweb_path} -t #{threads} --log-brief=#{log_file} -i #{target_file}"
+          when 'verbose'
+            "#{@whatweb_path} -t #{threads} --log-verbose=#{log_file} -i #{target_file}"
+          else
+            "#{@whatweb_path} -t #{threads} --log-brief=/dev/null -i #{target_file}"
+          end
+
+    start_time = Time.now
+    success = system(cmd + " 2>/dev/null")
+    end_time = Time.now
+    
+    # Clean up log file
+    File.delete(log_file) if File.exist?(log_file)
+    
+    if success
+      end_time - start_time
+    else
+      nil
+    end
+  end
+
+  def benchmark_thread_scaling
+    puts "=== Thread Scaling Benchmark ==="
+    puts "Testing with small target list (5 targets)"
+    
+    target_file = File.join(@test_dir, 'small_targets.txt')
+    thread_counts = [1, 2, 5, 10, 20]
+    
+    results = {}
+    
+    thread_counts.each do |threads|
+      print "Testing #{threads} threads... "
+      
+      # Run multiple times and take average
+      times = []
+      3.times do
+        time = run_whatweb_test(threads, target_file, 'brief')
+        times << time if time
+      end
+      
+      if times.any?
+        avg_time = times.sum / times.length
+        results[threads] = avg_time
+        puts "#{avg_time.round(2)}s (avg of #{times.length} runs)"
+      else
+        puts "FAILED"
+      end
+    end
+    
+    puts "\n=== Results Summary ==="
+    results.each do |threads, time|
+      speedup = results[1] ? (results[1] / time).round(2) : 'N/A'
+      puts "#{threads} threads: #{time.round(2)}s (#{speedup}x speedup)"
+    end
+    
+    results
+  end
+
+  def benchmark_output_formats
+    puts "\n=== Output Format Benchmark ==="
+    puts "Testing different output formats with 10 threads"
+    
+    target_file = File.join(@test_dir, 'small_targets.txt')
+    threads = 10
+    formats = ['brief', 'verbose', 'stdout']
+    
+    results = {}
+    
+    formats.each do |format|
+      print "Testing #{format} format... "
+      
+      times = []
+      3.times do
+        time = run_whatweb_test(threads, target_file, format)
+        times << time if time
+      end
+      
+      if times.any?
+        avg_time = times.sum / times.length
+        results[format] = avg_time
+        puts "#{avg_time.round(2)}s"
+      else
+        puts "FAILED"
+      end
+    end
+    
+    puts "\n=== Output Format Results ==="
+    results.each do |format, time|
+      puts "#{format}: #{time.round(2)}s"
+    end
+    
+    results
+  end
+
+  def profile_with_ruby_prof
+    puts "\n=== Ruby Profiling ==="
+    
+    target_file = File.join(@test_dir, 'small_targets.txt')
+    profile_file = File.join(@results_dir, "profile_#{Time.now.to_i}.txt")
+    
+    puts "Running profiled test (this may take longer)..."
+    
+    env = {
+      'WHATWEB_PROFILE' => '1',
+      'WHATWEB_PROFILE_FILE' => profile_file
+    }
+    
+    cmd = "#{@whatweb_path} -t 10 --log-brief=/dev/null -i #{target_file}"
+    
+    start_time = Time.now
+    success = system(env, cmd + " 2>/dev/null")
+    end_time = Time.now
+    
+    if success && File.exist?(profile_file)
+      puts "Profile completed in #{(end_time - start_time).round(2)}s"
+      puts "Profile saved to: #{profile_file}"
+      
+      # Show top 10 time consumers
+      puts "\n=== Top 10 Time Consumers ==="
+      profile_content = File.read(profile_file)
+      lines = profile_content.split("\n")
+      
+      # Find the start of the flat profile
+      start_idx = lines.find_index { |line| line.include?('%self') }
+      if start_idx
+        data_lines = lines[(start_idx + 1)..(start_idx + 10)]
+        data_lines.each { |line| puts line if line.strip.length > 0 }
+      end
+    else
+      puts "Profiling failed or ruby-prof not available"
+    end
+  end
+
+  def run_all_tests
+    puts "WhatWeb Performance Test Suite"
+    puts "=============================="
+    puts "Timestamp: #{Time.now}"
+    puts "WhatWeb path: #{@whatweb_path}"
+    puts
+    
+    # Check if whatweb exists
+    unless File.exist?(@whatweb_path)
+      puts "ERROR: WhatWeb not found at #{@whatweb_path}"
+      exit 1
+    end
+    
+    # Run all benchmark tests
+    thread_results = benchmark_thread_scaling
+    format_results = benchmark_output_formats
+    profile_with_ruby_prof
+    
+    # Save results to file
+    results_file = File.join(@results_dir, "benchmark_#{Time.now.to_i}.txt")
+    File.open(results_file, 'w') do |f|
+      f.puts "WhatWeb Performance Benchmark Results"
+      f.puts "====================================="
+      f.puts "Timestamp: #{Time.now}"
+      f.puts
+      
+      f.puts "Thread Scaling Results:"
+      thread_results.each do |threads, time|
+        speedup = thread_results[1] ? (thread_results[1] / time).round(2) : 'N/A'
+        f.puts "  #{threads} threads: #{time.round(2)}s (#{speedup}x speedup)"
+      end
+      f.puts
+      
+      f.puts "Output Format Results:"
+      format_results.each do |format, time|
+        f.puts "  #{format}: #{time.round(2)}s"
+      end
+    end
+    
+    puts "\nResults saved to: #{results_file}"
+  end
+end
+
+# Run the tests if this script is executed directly
+if __FILE__ == $0
+  test = WhatWebPerformanceTest.new
+  test.run_all_tests
+end

test/verbose_screen_test.rb

@@ -0,0 +1,106 @@
+#!/usr/bin/env ruby
+#
+# Test script to verify verbose screen output performance
+#
+
+require 'benchmark'
+
+class VerboseScreenTest
+  def initialize
+    @whatweb_path = File.expand_path('../whatweb', __dir__)
+    @test_dir = __dir__
+    
+    create_test_targets
+  end
+
+  def create_test_targets
+    # Create a small test target file
+    test_targets = File.join(@test_dir, 'verbose_test_targets.txt')
+    unless File.exist?(test_targets)
+      File.write(test_targets, [
+        'http://example.com',
+        'http://httpbin.org/html',
+        'http://httpbin.org/json'
+      ].join("\n"))
+    end
+  end
+
+  def test_verbose_screen_performance
+    puts "=== Verbose Screen Output Performance Test ==="
+    
+    target_file = File.join(@test_dir, 'verbose_test_targets.txt')
+    
+    # Test different thread counts with verbose output to screen
+    thread_counts = [1, 10, 25]
+    results = {}
+    
+    thread_counts.each do |threads|
+      print "Testing verbose output with #{threads} threads... "
+      
+      start_time = Time.now
+      # Use -v for verbose output to screen, redirect to /dev/null to avoid cluttering test output
+      cmd = "#{@whatweb_path} -v -t #{threads} -i #{target_file} > /dev/null"
+      success = system(cmd + " 2>/dev/null")
+      end_time = Time.now
+      
+      if success
+        time = end_time - start_time
+        results[threads] = time
+        puts "#{time.round(2)}s"
+      else
+        puts "FAILED"
+      end
+    end
+    
+    puts "\n=== Verbose Screen Performance Results ==="
+    results.each do |threads, time|
+      speedup = results[1] ? (results[1] / time).round(2) : 'N/A'
+      puts "#{threads} threads: #{time.round(2)}s (#{speedup}x speedup)"
+    end
+    
+    # Test with output sync forced on vs off
+    puts "\n=== Output Sync Comparison (25 threads) ==="
+    
+    print "Testing with --output-sync... "
+    start_time = Time.now
+    cmd = "#{@whatweb_path} -v --output-sync -t 25 -i #{target_file} > /dev/null"
+    success = system(cmd + " 2>/dev/null")
+    sync_time = Time.now - start_time
+    puts success ? "#{sync_time.round(2)}s" : "FAILED"
+    
+    print "Testing without --output-sync... "
+    start_time = Time.now
+    cmd = "#{@whatweb_path} -v -t 25 -i #{target_file} > /dev/null"
+    success = system(cmd + " 2>/dev/null")
+    async_time = Time.now - start_time
+    puts success ? "#{async_time.round(2)}s" : "FAILED"
+    
+    if sync_time && async_time
+      improvement = ((sync_time - async_time) / sync_time * 100).round(1)
+      puts "Performance improvement: #{improvement}% faster without sync"
+    end
+  end
+
+  def run_test
+    puts "Verbose Screen Output Performance Test"
+    puts "====================================="
+    puts "Timestamp: #{Time.now}"
+    puts
+    
+    # Check if whatweb exists
+    unless File.exist?(@whatweb_path)
+      puts "ERROR: WhatWeb not found at #{@whatweb_path}"
+      exit 1
+    end
+    
+    test_verbose_screen_performance
+    
+    puts "\nTest completed. Check results above."
+  end
+end
+
+# Run the test if this script is executed directly
+if __FILE__ == $0
+  test = VerboseScreenTest.new
+  test.run_test
+end

test/verbose_test_targets.txt

@@ -0,0 +1,3 @@
+http://example.com
+http://httpbin.org/html
+http://httpbin.org/json

whatweb

@@ -34,6 +34,27 @@
 
 $LOAD_PATH.unshift(File.join(File.expand_path(__dir__)), '.')
 
+# Performance profiling infrastructure (Phase 1)
+if ENV['WHATWEB_PROFILE']
+  begin
+    require 'ruby-prof'
+    RubyProf.start
+    at_exit do
+      result = RubyProf.stop
+      printer = RubyProf::FlatPrinter.new(result)
+      if ENV['WHATWEB_PROFILE_FILE']
+        File.open(ENV['WHATWEB_PROFILE_FILE'], 'w') { |f| printer.print(f) }
+        STDERR.puts "Profile saved to #{ENV['WHATWEB_PROFILE_FILE']}"
+      else
+        printer.print(STDERR)
+      end
+    end
+    STDERR.puts "Ruby profiling enabled"
+  rescue LoadError
+    STDERR.puts "Warning: ruby-prof gem not available. Install with: gem install ruby-prof"
+  end
+end
+
 require 'lib/whatweb'
 
 #
@@ -156,6 +177,10 @@ PERFORMANCE & STABILITY:
   --read-timeout\t\tTime in seconds. Default: #{$HTTP_READ_TIMEOUT}.
   --wait=SECONDS\t\tWait SECONDS between connections.
 \t\t\t\tThis is useful when using a single thread.
+  --output-sync\t\t\tForce immediate output flushing for real-time
+\t\t\t\tmonitoring (slower with high thread counts).
+  --output-buffer-size=SIZE\tSet output buffer size. 0=unbuffered,
+\t\t\t\tdefault=auto based on thread count.
 
 HELP & MISCELLANEOUS:
   --short-help\t\t\tShort usage help.
@@ -363,7 +388,9 @@ opts = GetoptLong.new(
   ['--wait', GetoptLong::REQUIRED_ARGUMENT],
   ['--debug', GetoptLong::NO_ARGUMENT],
   ['--version', GetoptLong::NO_ARGUMENT],
-  ['-q', '--quiet', GetoptLong::NO_ARGUMENT]
+  ['-q', '--quiet', GetoptLong::NO_ARGUMENT],
+  ['--output-sync', GetoptLong::NO_ARGUMENT],
+  ['--output-buffer-size', GetoptLong::REQUIRED_ARGUMENT]
 )
 
 begin
@@ -570,6 +597,10 @@ begin
     when '--version'
       puts "WhatWeb version #{WhatWeb::VERSION} ( https://morningstarsecurity.com/research/whatweb/ )"
       exit
+    when '--output-sync'
+      $OUTPUT_SYNC = true
+    when '--output-buffer-size'
+      $OUTPUT_BUFFER_SIZE = arg.to_i
     end
   end
 rescue Errno::EPIPE
@@ -602,10 +633,26 @@ end
 # optimise plugins
 PluginSupport.precompile_regular_expressions
 
+# Store thread count globally for logging classes (before creating loggers)
+$THREADS = max_threads
+
+# Configure smart defaults for output performance based on thread count
+unless defined?($OUTPUT_SYNC)
+  $OUTPUT_SYNC = (max_threads <= 5)  # Sync for low thread counts
+end
+
+unless defined?($OUTPUT_BUFFER_SIZE)
+  $OUTPUT_BUFFER_SIZE = case max_threads
+                        when 1..5 then 1      # No buffering for low threads
+                        when 6..50 then 10    # Small buffer for medium (includes default 25)
+                        else 25               # Larger buffer for high threads (51+)
+                        end
+end
+
 ### OUTPUT
-logging_list << LoggingBrief.new unless $QUIET || $verbose > 0 # by default logging brief
+logging_list << LoggingBrief.new(STDOUT, sync: $OUTPUT_SYNC) unless $QUIET || $verbose > 0 # by default logging brief
 logging_list << LoggingObject.new if $verbose > 1 # full logging if -vv
-logging_list << LoggingVerbose.new if $verbose > 0 # full logging if -v
+logging_list << LoggingVerbose.new(STDOUT, sync: $OUTPUT_SYNC) if $verbose > 0 # full logging if -v
 
 ## logging dependencies
 if mongo[:use_mongo_log]
@@ -634,7 +681,6 @@ end
 
 urls = ARGV
 
-
 # Initialize Scanner
 begin
   scanner = WhatWeb::Scan.new(