From 2829cfb53b7b66d126ae06b83b766b1f72cbe51f Mon Sep 17 00:00:00 2001 From: Andrew Horton Date: Thu, 30 Sep 2010 23:03:42 +1300 Subject: [PATCH] first commit --- CHANGELOG | 137 ++ How-to-develop-WhatWeb-plugins-1.1.txt | 2129 ++++++++++++++++++++++++ INSTALL | 23 + LICENSE | 339 ++++ Makefile | 13 + README | 782 +++++++++ TODO | 778 +++++++++ whatweb | 1066 ++++++++++++ whatweb.1 | 140 ++ whatweb.xsl | 37 + 10 files changed, 5444 insertions(+) create mode 100644 CHANGELOG create mode 100644 How-to-develop-WhatWeb-plugins-1.1.txt create mode 100644 INSTALL create mode 100644 LICENSE create mode 100644 Makefile create mode 100644 README create mode 100644 TODO create mode 100755 whatweb create mode 100644 whatweb.1 create mode 100644 whatweb.xsl diff --git a/CHANGELOG b/CHANGELOG new file mode 100644 index 00000000..015a03d7 --- /dev/null +++ b/CHANGELOG @@ -0,0 +1,137 @@ +Version 0.4.6 Released ? 2010 +* Added x plugins from Brendan Coles. They are: ClipShare, PhpMesFilms, wpQuiz, phpQuestionnaire, RevSense, linkSpheric, WebspotBlogging, XchangeBoard, SazCart, AirvaeCommerce, Cartweaver, BosClassifieds, zFeeder, CMScout, Ultrastats, Connectix-Boards +* Added Escenic CMS plugin from Erik Inge Bolsø +* Changed $ANEMONE_SKIP_REGEX=Regexp.union line to be compatible with Ruby 1.8.6. Thanks to Michal Ambroz +* Added plugin reporting support for :model=>, :firmware=>, :modules=> +* Modified Joomla plugin to use :modules instead of :string, changed certainty of a regexp +* Modified PHP-Nuke plugin to use :modules +* Modified XML logging to record modules separately +* Added --wait SECONDS between connections. Combine with -t 1 if preferred. +* Added meta-refresh redirect support. eg. . Only for non-spidering +* Added {:version=>/regexp/, :version_regex_offset} to remove cargo cult programming. eg. +{:version=>/2, :name=>"meta generator tag" } +* Updated plugins to use :version=>/regexp/: Advanced-Guestbook, ASP-Nuke, Concrete5, Coppermine, FormMail, InvisionPowerBoard, FormMail, MikroTik, MovableType, SearchFitShoppingCart, SquirrelMail, VBulletin, VP-ASP, VSNSLemon, WordPress +* Replaced :probability with :certainty in my-plugins/plugin-template.rb.txt. Thanks Erik Inge Bolsø +* Added support for em-resolv-replace which speeds up whatweb many times. http://github.com/mperham/em-resolv-replace +* whatweb.xsl added by Brendan Coles +* Added reporting of version detection with matches to the Plugin Info, eg. whatweb -I +* Changed whatweb -I behaviour to search plugins for keywords. eg. './whatweb -I nuke' brings up ASP-Nuke, PHPNuke, DotNetNuke, etc. +* Bugfix: Changed webpage data for when working with files, not URIs. Now it passes empty hashes, etc instead of nil which caused plugins to report errors. +* Added JSON logging. Must have the json ruby gem installed or be using Ruby 1.9 +* Added error logging. +* Added XML header and footer to XML logs + + +Version 0.4.5 Released August 17th 2010 +* Added 5 plugins from Tonmoy Saikia. They are: Commonspot, TextPattern, Mediawiki, DUclassified and Mailman +* Added 119 plugins from Brendan Coles. They are: Alcatel-Lucent-Omniswitch, Allinta-CMS, anyInventory, Arab-Portal, AVTech-Video-Web-Server, Barracuda-Spam-Firewall, Basilic, Biromsoft-WebCam, BlueNet-Video-Server, BM-Classifieds, Brother-Printer, BusinessSpace, BXR, Campsite, Canon-Network-Camera, Cisco-VPN-3000-Concentrator, CMSQLite, ColdFusion, coWiki, cpCommerce, CruxCMS, CruxPA, Dell-Printer, D-Link-Network-Camera, DMXReady, DT-Centrepiece, EazyCMS, eLitius, EMO-Realty-Manager, Empire-CMS, envezion~media, eSyndiCat, Evo-Cam, FestOS, Flax-Article-Manager, FluentNET, Forest-Blog, GuppY, HP-LaserJet-Printer, i-Catcher-Console, iDVR, Intellinet-IP-Camera, Interspire-Shopping-Cart, IPCop-Firewall, IQeye-Netcam, iRealty, iScripts-CyberMatch, iScripts-EasySnaps, iScripts-MultiCart, iScripts-ReserveLogic, iScripts-SocialWare, JAMM-CMS, Jamroom, Linksys-NAS, Linksys-Network-Camera, Linksys-Wireless-G-Camera, LocazoList-Classifieds, Lucky-Tech-iGuard, Mobotix-Network-Camera, MyioSoft-Ajax-Portal, My-PHP-Indexer, My-WebCamXP-Server, NetBotz-Network-Monitoring-Device, Netious-CMS, Netsnap-Web-Camera, Nukedit, Open-Blog, ORCA-Platform, ORITE-301-Camera, PageUp-People, Panasonic-Network-Camera, Parked-Domain, PHPDirector, PHPEasyData, phPhotoAlbum, Pixel-Ads-Script, Pixie, Pligg-CMS, PortalApp, Pressflow, RunCMS, sabros.us, samPHPweb, SHOUTcast-Administrator, SimpNews, SkaLinks, SmodCMS, Snap-Appliance-Server, Softbiz-Freelancers-Script, Softbiz-Online-Auctions-Script, Softbiz-Online-Classifieds, Sony-Network-Camera, Sony-Video-Network-Station, Stardot-Express, StarDot-NetCam, Star-Network, Subdreamer-CMS, Subrion-CMS, SyndeoCMS, syntaxCMS, TaskFreak, Team-Board, The-PHP-Real-Estate-Script, TomatoCMS, Toshiba-Network-Camera, Veo-Observer, VisionGS-Webcam, WebDVR, WebEye-Network-Camera, WebPress, WhiteBoard, Winamp-Web-Interface, Windows-Internet-Printing, Xerox-Printers, xGB, XHP-CMS, Zeus-Cart, Zoph, Zyxel-Vantage-Service-Gateway +* Added 11 plugins from Caleb Anderson. They are: AdobeFlash, AtomFeed, CodeIgniterProfiler, DublinCore, MicrosoftODBCError, MysqlSyntaxError, OpenGraphProtocol, OpenID, OpenSearch, PasswordField, RSSFeed +* Updated plugins: Aardvark-Topsites-PHP, Confluence, Open-Source-Ticket-Request-System, PHP-Link-Directory, PHP-Shell, Vulnerable-to-XSS, Zoph +* Updated mailto plugin +* Verbose output now shows which patterns were matched within a plugin +* Fixed bug: Removed Makefile reference to 'disabled-plugins' folder +* Ruby 1.9 compatability fix. requires digest/md5 instead of md5 +* Ruby 1.9 compatability fix. Replace UTF8 chars in frog-cms, dotnetnuke and mno-go-search and wordpress-supercache +* Fixed spelling error of verion in help information +* Fixed a typo where -t is shown as the command line option for proxies +* Modified command line usage and is now in 80x24 terminal format +* MD5sum of body is now available as @md5sum to all plugins +* :md5 is available in matches[], eg. {:name=>"must be treshna.com",:md5=>"8666257030b94d3bdb46e05945f60b42"} +* tag pattern of HTML elements in body is now available as @tagpattern to all plugins +* :tagpattern is available in matches[], eg. {:name=>"must be google.com",:tagpattern=>""!doctype,html,head,meta,title,/title,script,/script,style,/style, etc...."} +* :url is available in plugins. eg. {:url=>"/wp-login.php", :text=>'action=lostpassword'}, this will match the url and the text passively and when scanning aggressively, it will request the specified url and check for the text. Another example, {:url=>"/readme.html", :md5=>'9ea06ab0184049bf4ea2410bf51ce402', :version=>"3.0"}, +* Added --url-prefix, eg. whatweb --url-prefix www.morningstarsecurity.com/ -i ./guess-files +* Added --url-suffix, eg. whatweb --url-suffix /robots.txt -i ./target-urls +* Added --url-pattern, eg. whatweb --url-pattern www.example.com/%insert%/.htaccess -i ./folder-list +* Added --custom-plugin to define a plugin on the command line. eg, ./whatweb --custom-plugin ":text=>'powered by abc'" -i ./targets or --custom-plugin "{:text=>'powered by abc'},{:regexp=>/meta abc/i}" -i ./targets +* Plugin errors are now in red, added target name +* Added --open-timeout and --read-timeout +* Removed div-span plugin, replaced with HTML tag pattern hash +* Added --spider-skip-extensions. Redefine the file extensions that Anemone will skip. The list is comma delimited. +* Moved plugin-template.rb to my-plugins and added more example, comments, etc +* Added $DEBUG = false. If set to true, it will raise errors in plugins to assist plugin development. + +Version 0.4.4 Released June 29th 2010 +* :probability is renamed to :certainty. :certainty in plugins is no longer required, it defaults to 100 if not specified. +* Fixed bug with ruby 1.8.5 when loading plugins +* Added author names to plugin info, eg. whatweb -I +* Added 67 plugins from Brendan Coles, bringing WhatWeb up to 163 plugins. 360-Web-Manager,ANECMS,AWStats,Aardvark-Topsites-PHP,ArGoSoft-Mail-Server,Axis-Network-Camera,BeEF,BlognPlus,Burning-Board-Lite,CGI,CGIProxy,CMScontrol,CMSimple,Confluence,DUforum,DUgallery,F3Site,File-Upload-Manager,Google-API,Google-Hack-Honeypot,IMGallery,JGS-Portal,Kloxo,Liferay,Lime-Survey,Linksys-USB-HDD,Loggix,Microsoft-Sharepoint,Open-Freeway,Open-Source-Ticket-Request-System,PG-Roomate-Finder-Solution,PHP-Fusion,PHP-Layers,PHP-Link-Directory,PHP-Shell,PHPFM,PHPraid,PhilBoard,Piwik,QNAP-NAS,Saurus-CMS,Site-Sift,TWiki,Trac,Turbo-Seek,Umbraco,VideoShareEnterprise,Virtualmin,Vulnerable-To-XSS,WWWBoard,Web-Calendar-System,Web-Data-Administrator,WoW-Raid-Manager,X7-Chat,Zen-Cart,Zikula,boastMachine,ezBOO-WebStats,jobberBase,mojoPortal,php-ping,phpFreeChat,phpMyAdmin,phpPgAdmin,phpSysInfo,phpinfo,uPortal +* Added references to Security-Assessment.com +* Updates to README, CHANGELOG, plugin-template.rb.txt + +Version 0.4.3 Released May 24th 2010 +* Added GPLv2 notices +* Added Makefile (Thanks Michal Ambroz ) +* Added man pages (Thanks Michal Ambroz ) +* Added --version +* Added Invalid command line argument handling +* Added @cookie variable to plugins but is not availble for recursive use +* Changed output colour of page titles +* Changed plugin names to use a CamelCase convention +* Merged the google analytics GA and Urchin plugins +* Modified MovableType plugin +* Added Cookie names plugin +* Added Concrete5 CMS plugin +* Added CushyCMS plugin +* Added FrogCMS plugin +* Added ModxCMS plugin +* Added TypoLight plugin +* Added ExpressionEngine plugin +* Fixed a bug in Tomcat plugin +* New feature, my-plugins/ folder. Keep your personal plugins separate. +* Usage info shows correct defaults +* Fixed a bug where aggressive plugins didn't use the proxy settings +* Added XML (naive) logging +* Updated usage to show how to pipe HTML to /dev/stdin +* Added --no-redirect option. Do not follow HTTP 3xx redirects + +Version 0.4.2 Released April 30th 2010 +* Added header-hash plugin. Makes a hash of the first 500 characters. This is useful to identify unknown systems +* Added footer-hash plugin. Makes a hash of the last 500 characters, only if the page has > 1000 characters. This is useful to identify unknown systems +* Added div-span-structure plugin. Makes a hash of a signature of div and span tags. This is useful to identify unknown systems +* Added MikroTik Router plugin. Recognises version +* Fixed a bug where the URL had a ? suffix. This caused some types of http servers to repspond incorrectly. +* Added SquirrelMail plugin. Recognises version +* Added SearchFitShoppingCart plugin. Recognises version +* Added RoundCube plugin. +* Modified OSCommerce plugin. Recognises security warnings about file permissions and installation directory. +* Changed output colour to be more readable. Plugins that create hashes are in grey +* Changed output order of plugins, so plugins that create hashes come last + +Version 0.4.1 Released April 28th 2010 +* Removed dependency on rubygems and libxslt by modifying and locally including the Anemone gem. This also simplified installation +* Fixed a bug which didn't send URL parameters. eg. would send /index.php instead of /index.php?q=foo +* Improved installation instructions. Henri Salo contacted me to say ruby-dev is required for Anemone +* Removed UTF-8 character in formmail +* Changed require 'md5' to require 'digest/md5' for compatibility with ruby 1.9 +* Fixed bug in Tomcat plugin +* Added SilverStripe plugin +* Added DotNetNuke plugin +* Added HTML5 plugin +* Added PHP error plugin +* Modified PHP-Nuke plugin +* Changed the plugin development script, wget-list to retry only twice +* Added proxy support +* Default threads is now 25 +* Default max recursive spidering depth is now 10 +* Default max number of links to follow on a single page is now 250 + +Version 0.4 Released March 13th 2010 +* Added HTTPS support +* Improved installation instructions +* Improved documentation +* Better compatibility with ruby 1.9. Changed a case statement syntax, changed when 0: to when 0 then. +* Removed UTF-8 characters in plugins that were causing crashes +* Added php-nuke plugin, passively recognises modules +* Added Fluxbb plugin, can identify versions aggressively +* Added meta powered-by plugin. Matches tags like +* Added powered by plugin. Matches "Powered by BobsCMS", any text following powered by +* Improved plugin info listing invoked by ./whatweb -I. Shows number of examples and matches, and shows presence of passive and aggressive functions +* Changed output style. Before strings are surrounded by single quotes, now all strings are surrounded by square brackets +* Added OpenCMS plugin submitted by Emilio Casbas +* Added TomCat plugin submitted by Louis Nyffenegger +* Improved meta-generator plugin +* Fixed a bug in processing a target list from a file where a trailing space would be interpreted incorrectly + +Version 0.3 Released November 2nd 2009 at Kiwicon III + diff --git a/How-to-develop-WhatWeb-plugins-1.1.txt b/How-to-develop-WhatWeb-plugins-1.1.txt new file mode 100644 index 00000000..c0aee44d --- /dev/null +++ b/How-to-develop-WhatWeb-plugins-1.1.txt @@ -0,0 +1,2129 @@ + +How to develop WhatWeb 0.4 plugins +---------------------------------- +by Andrew Horton aka urbanadventurer. MorningStar Security http://www.morningstarsecurity.com/ +Revision 1.1, 29th March 2010. + + +Contents +================================================= +1. Introduction to WhatWeb +2. Introduction to WhatWeb plugins + General aims of a plugin + Methods to identify systems + Important files and folders + Anatomy of a plugin +3. Research background information +4. Collect samples + Website Showcases + Using Search Engines + Forums for website development with the cms +5. Analyze samples + Read the source of a couple of samples + Collect HTML and HTTP headers from samples + Remove incorrectly identified samples + Examine the samples with WhatWeb + Remove more incorrectly identified samples with the whatweb report + Use find-common-stuff to automatically identify common strings in the samples + Analyse HTTP headers and cookies + Read more HTML source +6. Review of unique patterns identified +7. Write the plugin +8. Closing notes +9. Resources + + +1. Introduction to WhatWeb +================================================= + +WhatWeb lets you identify content management systems (CMS), blogging platforms, stats/analytics packages, javascript libraries, servers and more. When you visit a website in your browser the transaction includes many unseen hints about how the webserver is set up and what software is delivering the webpage. Some of these hints are obvious, eg. "Powered by XYZ" and others are more subtle. WhatWeb recognises these hints and reports what it finds. + +WhatWeb has many plugins and needs community support to develop more. Plugins can identify systems with obvious identifying hints removed by also looking for subtle clues. For example, a WordPress site might remove the tag but the WordPress plugin also looks for "wp-content" which is less easy to disguise. Plugins are flexible and can return any datatype, for example plugins can return version numbers, email addresses, account ID's and more. + +There are both passive and aggressive plugins, passive plugins use information on the page, in cookies and in the URL to identify the system. A passive request is as light weight as a simple GET / HTTP/1.1 request so it is suitable for large scale scanning of websites. Aggressive plugins guess URLs and request more files. + + +2. Introduction to WhatWeb Plugins +================================================= + +Plugins are easy to write, you don't need to know ruby to make them but it helps. + +General aims of a plugin +------------------------ + +Most plugins have a primary aim which is to identify a type of system based on signatures. The system could be a: + + * Content Management System + * Javascript Library + * HTTP Server + * Application Framework + +Some plugins do not have the aim to identify a specific type of system. Instead they try to give information that can be used to identify unanticipated systems or can be used for all types of websites. These plugins are: + + * Title + * MD5 hash + * Meta generator tag name + * Uncommon HTTP headers + + +Methods to identify systems +--------------------------- +There are 4 main methods to identify a CMS or web application. They are: + + 1. Matching patterns in the HTTP headers and HTML of a simple webpage request + 2. Testing for URLs and identifying patterns in the HTML + 3. Testing for URLs and recognising the MD5 hash of the HTML + 4. Testing for URLs and simply noting they exist or return an HTTP status 200 code. + +WhatWeb supports all 4 methods however the 1st method is the most useful in large scale scanning. It is also the most efficient by trading off knowledge for network bandwidth and time. +Support for the first method is the most developed method within WhatWeb and is discussed in detail in this document. Future development of WhatWeb will add more user friendly support for methods 2 through 4 which come under the purview of aggressive plugins. + + +Important files and folders +--------------------------- + +The important folders to plugins are: + + * disabled-plugins/ + * plugin-development/ + * plugin-development/tests/ + * plugins/ + +All .rb files in the plugins/ folder are loaded by WhatWeb. To disable a plugin, move it into the disabled-plugins/ folder. + +The plugin-development folder contains some tools that are useful in developing plugins. +The tools are: + + * find-common-stuff - This searches for common strings among a set of HTML files + * wget-list - This downloads a list of example websites + +The plugin-development/tests folder contains example webpages of CMS's to study. The wget-list will create two files for each example webpage. A .html file and a .meta file. + + + + +Anatomy of a plugin +------------------- + +This is a typical plugin. It identifies the Drupal framework and it's split into sections and given line numbers. + + +->----------------------------------------------------------------------------------------------------------- +1 Plugin.define "Drupal" do +2 author "Andrew Horton" +3 version "0.1" +4 description "Drupal is an opensource CMS written in PHP. Homepage: http://www.drupal.org" +-<----------------------------------------------------------------------------------------------------------- + +Line 1. has the name. This name can be referred to on the commandline in a case insensitive way. + +For example, the following works: + + $ ./whatweb -pdrupal www.example.com + +Line 2. has the author. Just fill in your name between the double quotes. +Line 3. contains the version number. It's up to you what number to choose. +Line 4. Contains the description. This should contain a description of what the plugin identifies that anyone can understand. It can be many lines but must start and end with double quotes. + +Note that the author, version and description follow the format: + + field-name field-content + +On the left is the name of the variable and on the right, separated by a space is the value. This type of variable declaration isn't ruby code, it's specific to the plugins and only works for certain variable names. + +The list of variable names that can be declared in a plugin in this manner are: + + * author + * version + * description + * examples + * matches + + +->----------------------------------------------------------------------------------------------------------- +5 # hard to identify +6 #Powered by Drupal, an open source content management system +7 # +8 # +9 # @import "/misc/drupal.css"; +10 # Set-Cookie: SESS6bdd09d4debccdc3a0f49becc449e8d5=2sq674vjn6vig48e3podh3j8e2; expires=Fri, 11 Dec 2009 15:37:52 GMT; path=/; domain=.moby.com +11 # Set-Cookie: SESS9795bcd4ea70e3f846e84f29f9491636=57eafcca6400d894772a136fb5889b92; expires=Fri, 11-Dec-2009 15:38:25 GMT; path=/; domain=.save-your-future.com +12 +13 +14 examples %w| amnesty.org/ appel.nasa.gov/ beta.worldbank.org/ entergy.pewclimate.org/ labs.divx.com/ lindenlab.com/ littlestarprints.com moby.com/ myplay.com/ sequelnaturals.com/ teen.secondlife.com/ www.artwaves.de www.asys.com.br/ www.atomicbop.net www.cristal.com.pe/?adulto=si www.dutchbutnotfromholland.eu/ www.elespectador.com/ www.ensembles.com.ph/ www.foxsearchlight.com/index.php www.freshbrain.org/ www.icsalabs.com/ www.johnnycashonline.com/ www.journalismcenter.org/ www.jovenscriativos.com.br/ www.koalafoundation.org.au/ www.la2day.com/ www.moove.be www.mtv.co.uk/channel/flux www.mulinobianco.it/ www.multiways.com/ www.nowpublic.com/ www.pravda.lt/ www.realismssoftware.com/ www.save-your-future.com www.shock.com.co/ www.sosojuicy.com/ www.spreadfirefox.com/ www.tidningenresultat.se www.ubuntu.com/ www.universitytowers.net/ www.warnerbrosrecords.com | +15 +-<----------------------------------------------------------------------------------------------------------- + +Lines 5 through to 11 are comments. Each commented line must begin with a # character and this is a standard ruby way to comment code. + +Line 14 is a list of example websites. The examples prefix of %w| means an array of elements separated by whitespace. The individual examples are URLs. If they are missing the http:// or https:// then http:// is assumed. + +If you prefer you can list the examples like this: + + examples %w| + http://www.example.com + http://www.example2.com + http://www.site.com/blah/ + | + +->----------------------------------------------------------------------------------------------------------- +16 matches [ +17 {:name=>"/misc/drupal.js", +18 :probability=>100, +19 :regexp=>/