From a780dc0c142d9417593b3952fcd0189be4084495 Mon Sep 17 00:00:00 2001 From: Efrain Torres Date: Fri, 3 Feb 2012 16:27:24 -0600 Subject: [PATCH] Wmap 1.5 documentation --- documentation/wmap.txt | 570 +++++++++++++++++++++++++++++++++-------- 1 file changed, 461 insertions(+), 109 deletions(-) diff --git a/documentation/wmap.txt b/documentation/wmap.txt index f8e4c719b5..178c64d85f 100644 --- a/documentation/wmap.txt +++ b/documentation/wmap.txt @@ -1,150 +1,502 @@ -=[ WMAP v1.0 -=[ Efrain Torres - et[]metasploit.com ---------------------------------------------------------------------------- - "Metasploit goes Web", H D Moore. + .-.-.-..-.-.-..---..---. + | | | || | | || | || |-' + `-----'`-'-'-'`-^-'`-' + Metasploit Wmap 1.5 +============================================================================== +Efrain Torres et [ ] metasploit.com 2012 +------------------------------------------------------------------------------ -=[ Intro. +=[ 0. Intro ] ---------------------------------------------------------------- -WMAP is a general purpose web application scanning framework for -Metasploit 3. The architecture is simple and its simplicity is what makes -it powerful. It's a different approach compared to other open source -alternatives and commercial scanners, as WMAP is not build around any browser -or spider for data capture and manipulation. +So its 2012 and before the Mayans are proven to be right I was able to create +a new version of this plugin. If you have read the old wmap documentation this +is what is going on: +Wmap is still a general purpose web application scanning framework for +Metasploit. Still is a different approach compared to other open source +alternatives and commercial scanners, as Wmap is not build around any browser +or spider for data capture and manipulation. And the best thing is that still +is FR33. Lots of bugs are gone and the new code allows for faster and more +efficient execution. -=[ How it works. - -In the WMAP design, any tool can become a data gathering tool. In the -general case an attack proxy can be modified to store all the traffic between -the client(s) (i.e. favorite browser and/or spider). (See figure.) - -Notice that a client may be used to store data too. +=[ 1. How it works ] --------------------------------------------------------- +The old architecture (versions < 1.5): [CLIENT] ----- [ATTACK PROXY] ----- [TARGET] | | ^ - ---------->[METASPLOIT DB] | + +--------->[METASPLOIT DB] | | | - [MSF 3 - WMAP SCANNER] | - [MSF 3 - WMAP MODULES] -----+ + [MSF 3 - Wmap SCANNER] | + [MSF 3 - Wmap MODULES] -----+ -WMAP is a Metasploit plugin and will interact with the database, reading all -gathered traffic, processing it and launching the different tests -implemented as modules. As tests are MSF Modules they can be easily -implemented, and can be run manually from the command line or automatically -via WMAP. +The new architecture: -As you may see this simple architecture allows you to have different -distributed clients and even different proxies all storing data to the -central repository. Remember everything is based on Metasploit, the test -modules are implemented as auxiliary modules and they can interact with any -other MSF component including the database, exploits and plugins. + [CLIENTS] + | + | + +-------[Wmap PLUGIN]<-----+----->[METASPLOIT DB] + | | | | + | | | | +[NODE 1] [NODE 2] [NODE n] ---------+ + | | | \ + | | | [Wmap MODULES] + +---------[TARGETS]--------+ -=[ WMAP Modules. - -The test modules implemented at this time are basic and will improve over -time not only in quality and quantity, so you are more than welcome to -submit new modules. +Wmap is a Metasploit plugin and will interact with the database, reading all +gathered traffic from any client you have configured/adapted or duck taped to +store web sites, requests, responses and forms in the Metasploit DB. -Each module has a WMAP type, this determine when the module is launched and -to a certain degree, the minimum type of information it requires to be -executed. The best way to develop a new test for WMAP, is to use already -implemented modules as a base and then develop a normal MSF module that can -be run manually from the command line. To enable a module to be run -automatically via WMAP just include the mixin that determine the type -of the module. +The test performed are all Metasploit modules which WMAP execute in a +configurable order. The test modules are implemented as auxiliary or exploit +modules and they can interact with any other MSF component including the +database other exploits and plugins. - Example: +The new architecture allows to have different distributed clients and nodes +all storing results and data to a central database. This means that large +enviorments can be tested using multiple metasploit msfrcpd servers (nodes) +controled from one (or more) WMAP consoles. Wmap will execute the tests to be +launched from each node distributing evenly the job load across all configured +nodes. - include Auxiliary::WMAPScanFile +In case you dont want to use a distributed model wmap will detect that no +nodes have been configured and will run the modules from the local host. + + +=[ 2. Crawlers,proxies and other clients ] ----------------------------------- + +At this time Metasploit have 3 components that may be used as clients +to store web sites in the database: + +(1) If you have configured your database properly and use the +auxiliary/scanner/http/crawler module, this module will create a web site +(with related host and service) and store all requests,responses and forms +automatically. + +(2) Less known is that metasploit has a different crawler called msfcrawler +and besides supporting modules to parse the responses in any way you want +it will also store the required data in the database. + +(3) Also any module that creates a web_site in the database (e.g. +auxiliary/scanner/http/http_version module) +will add a site to the database that can be selected as a target in Wmap, +however the only path you will be storing will be the root path of the +website '/'. + +Or you can add a site manually to the database using the 'wmap_sites -a' +command (after loading the wmap plugin. See '4. Wmap Plugin'): + + msf > wmap_sites -a www.blah.net,http://192.168.0.2/ + [*] Site created. + + Note: www.blah.net,http://192.168.0.2/ <-- is one site vhost,url + +For other tools to store web data in the database the only hard part is to +deal with the ruby marshalling of parameters in the web_forms table. (Topic +for another paper). But this is one of the main issues regarding the use of +other tools to connect to the database. However any Ruby based tool can be +modified easily to do this. + +If you noticed the previous architecture the ATTACK PROXY has gone the way of +the Dodo (actually not as is just another client). But i will stop mention it +because i have been unable to create a good Ruby based MITM proxy (Sorry) +and second because i dont want to maintain plugins for every type of proxy +out there. Is your exercise to create custom plugins for your tools to connect +to the database, after that Wmap does not care where the data comes from. + +=[ 3. The Wmap cycle ]-------------------------------------------------------- + +Or how every other scanner works but in wmap fancy terms. + +0. Gather data from (See Crawlers,proxies and other clients) + targets +1. Load the scanner (Load wmap plugin. See "4. Wmap Plugin") +2. Define nodes (Use 'wmap_nodes'. See "Wmap Nodes") +3. Define targets (Use 'wmap_sites' and 'wmap_targets'.See "5. Wmap Targets") +4. Configure (Lots of things here) +5. Launch (Use 'wmap_run'. See "6. Launch a scan") +6. Enjoy WTF Moments(Priceless...) + +=[ 4. Wmap Plugin ]----------------------------------------------------------- + +To launch wmap open a Metasploit console and load the wmap plugin. + +msf > load wmap +[*] [Wmap 1.5] === et [ ] metasploit.com 2012 +[*] Successfully loaded plugin: wmap + +Now that the plugin is loaded lets go through the list of basic commands: + +msf > help wmap + +Wmap Commands +============= + + Command Description + ------- ----------- + wmap_nodes Manage nodes + wmap_run Test targets + wmap_sites Manage sites + wmap_targets Manage targets + wmap_modules Manage wmap modules + +=[ 5. Wmap Targets ]---------------------------------------------------------- + +The targets are selected from the sites already stored in the database. For +example after crawling a site (See "2. Crawlers,proxies and other clients") +now we can use the command 'wmap_sites' to list them: + +msf > wmap_sites +[*] Usage: wmap_sites [options] + -h Display this help text + -a [url] Add site (vhost,url) + -l List all available sites + -s [id] Display site structure (vhost,url|ids) (level) + +msf > wmap_sites -l +[*] Available sites +=============== + + Id Host Vhost Port Proto # Pages # Forms + -- ---- ----- ---- ----- ------- ------- + 0 10.10.10.1 blah.xyz.com 443 https 3 2 + 1 10.10.10.2 blah.xyz.com 443 https 3 2 + 2 10.1.2.2 nah.test.com 443 https 1 0 + 3 10.4.3.10 test.abcd.com 80 http 1 1 + +Note 1: Metasploit/wmap supports multiple Vhosts/IPs. + +Note 2: If you want to check the web site structure use the '-s site_id' flag +like this (also especify an optional level to display): + +msf > wmap_sites -s 0 1 + (First level of site 0) +msf > wmap_sites -s 0 + + [10.10.10.1] (blah.xyz.com) + | + +-------- dir1 + | + +------ login.php + +-------- dir2 + .... + +Then from the table we can select the targets we want to scan with the +'wmap_targets' command: + +msf > wmap_targets +[*] Usage: Wmap_targets [options] + -h Display this help text + -t [urls] Define target sites (vhost1,url[space]vhost2,url) + -d [ids] Define target sites (id1, id2, id3 ...) + -c Clean target sites list + -l List all target sites + +You can define targets in two ways, using the vhost,url syntax (-t) or the +table ids (-d) + +msf > wmap_targets -t test.abcd.com,http://10.4.3.10/ + +msf > wmap_targets -d 0,1 +[*] Loading blah.xyz.com,https://10.10.10.1:443/. +[*] Loading blah.xyz.com,https://10.10.10.2:443/. + +To see the list of all the targets to scan at this time run the +command with the (-l) flag. + +msf > wmap_targets -l +[*] Defined targets +=============== + + Id Vhost Host Port SSL Path + -- ----- ---- ---- --- ---- + 0 blah.xyz.com 10.10.10.1 443 true / + 1 blah.xyz.com 10.10.10.2 443 true / + 2 test.abcd.com 10.4.3.10 80 false / + + +=[ 6. Wmap Nodes ]------------------------------------------------------------ + +Wmap uses 'nodes' as a way to distribute the execution of the test against +one or more targets. Nodes are not required to run wmap . if nodes are not +configured and a scan is launched wmap will detect this and launch all tests +from the local host. + +The nodes are just msfrpcd servers that are created the following way: + +msf>ruby msfrpcd -h + +Usage: msfrpcd + +OPTIONS: + + -P Specify the password to access msfrpcd + -S Disable SSL on the RPC socket + -U Specify the username to access msfrpcd + -a Bind to this IP address + -f Run the daemon in the foreground + -h Help banner + -n Disable database + -p Bind to this port instead of 55553 + -u URI for Web server + +msf>ruby msfrpcd -U msf -P nodepass +[*] MSGRPC starting on 192.168.0.1:55553 (SSL):Msg... +[*] MSGRPC backgrounding at 2012-01-17 11:01:01 -0600... + +if you want to create a msfrpc server from the msfconsole you can do it by +loading the msgrpc plugin: + +msf > load msgrpc User=msf Pass=nodepass +[*] MSGRPC Service: 127.0.0.1:55552 +[*] MSGRPC Username: msf +[*] MSGRPC Password: nodepass +[*] Successfully loaded plugin: msgrpc + +On a later stage in the wmap console we will add such nodes so the scans can +be distributed across all the configured nodes. so remember how you deployed +your nodes so they can be configured in wmap. + +In the metasploit console after you have loaded the wmap plugin you can add +the previous nodes with the 'wmap_nodes' command: + +msf > wmap_nodes +[*] Usage: wmap_nodes [options] + -h Display this help text + -c id Remove id node (Use ALL for ALL nodes + -a host port ssl user pass Add node + -d host port user pass db Force all nodes to connect to a db + -j View detailed jobs + -k ALL|id ALL|job_id Kill jobs on node + -l List all current nodes + +msf > wmap_nodes -a 192.168.0.1 55553 true msf nodepass +[*] Connected to 192.168.0.1:55553 [4.2.0-dev]. +[*] Node created. + +Note: When launching msfrpcd waiht for a couple of seconds beofr adding it to +wmap as msfrpcd sometimes is slow to start accepting connections. + +Add as many nodes you want. To see the list use 'wmap_nodes -l': + +msf > wmap_nodes -l +[*] Nodes +===== + + Id Host Port SSL User Pass Status #jobs + -- ---- ---- --- ---- ---- ------ ----- + 0 127.0.0.1 55553 true msf nodepass 4.2.0-dev 0 + 1 192.168.0.1 55553 true msf nodepass 4.2.0-dev 0 + +Note: After launching all tests this command will allow you to see if all +your jobs have been completed (#jobs == 0). + +Remember that all these commands can be added as a .rc file so you dont +have to type again and again the loading of wmap and the configuration of +nodes. + +After you have your nodes connected then you can force them to connect to +the central metasploit database: + +msf > wmap_nodes -d 127.0.0.1 7175 dbuser dbpass msf3 +[*] db_connect {"driver"=>"postgresql", "db"=>"msf3"} 127.0.0.1:7175 OK +[*] db_connect {"driver"=>"postgresql", "db"=>"msf3"} 192.168.0.1:7175 OK +[*] OK. + + +=[ 7. Launch a scan ]--------------------------------------------------------- + +Now that database,targets and maybe nodes are set we run a scan with the +'wmap_run' command: + +msf > wmap_run +[*] Usage: wmap_run [options] + -h Display this help text + -t Show all enabled modules + -m [regex] Launch only modules that match provided regex. + -e [/path/to/profile] Launch profile modules against all targets. + No file runs all enabled modules. + +msf > wmap_run -e +[*] Using ALL wmap enabled modules. +[*] Testing target: +[*] Site: test.abcd.com (10.4.3.10) +[*] Port: 80 SSL: false +============================================================ +[*] Testing started. 2012-12-21 0:0:0 -0600 +[*] +=[ SSL testing ]= +============================================================ +[*] Target is not SSL. SSL modules disabled. +[*] +=[ Web Server testing ]= +============================================================ +[*] Module auxiliary/admin/http/http_version +[*] Module auxiliary/admin/http/tomcat_administration +[*] Module auxiliary/admin/http/tomcat_utf8_traversal +[*] Module auxiliary/admin/http/trendmicro_dlp_traversal +[*] Module auxiliary/scanner/http/cisco_nac_manager_traversal +.... +msf > + +As you see here wmap executes each of the modules against the defined targets. +(See "8. Wmap Modules") If nodes were configured it will go thru the list of +nodes and will send a job to the less loaded node to execute the especific +module with the required options. To force a good job distribution across the +nodes wmap has a limit of 25 jobs per node. If a node has reached the limit it +will try with the next node until there is a slot available. + +Check periodically with the 'wmap_nodes -l' command to see the current job +status. After the asssement is complete now you can use the normal metasploit +commands to see the results. + +To view detailed job information on each node use the 'wmap_nodes' (-j) flag: + +msf >wmap_nodes -j +[*] [Node #0: 127.0.0.1 Port:55553 SSL:true User:msf3] +[*] Jobs + ==== + + Id Job name + -- -------- + 42 Auxiliary: scanner/http/brute_dirs + + +[*] [Node #1: 127.0.0.1 Port:80 SSL:true User:msf] +[*] Jobs + ==== + + Id Job name + -- -------- + 22 Auxiliary: scanner/http/files_dir + +Also you can kill especific jobs or all jobs from one or all nodes: + +msf > wmap_nodes -k 0 ALL +[*] Node 0 Killed job id 262 Auxiliary: admin/http/tomcat_administration +[*] Node 0 Killed job id 263 Auxiliary: admin/http/tomcat_utf8_traversal +[*] Node 0 Killed job id 271 Auxiliary: scanner/http/soap_xml +[*] Node 0 Killed job id 299 Auxiliary: scanner/http/brute_dirs +[*] Node 0 Killed job id 300 Auxiliary: scanner/http/brute_dirs +[*] Node 0 Killed job id 301 Auxiliary: scanner/http/brute_dirs +.... + +If during the scan a node dies wmap will disable the node and will keep +sending the jobs to the other active nodes. + +If nodes were not configured wmap will launch the tests from the +local host the old fashion. + +=[ 8. Wmap Modules ] --------------------------------------------------------- + +Wmap modules are normal Metasploit modules. Each module has a WMAP type, +this determine when the module is launched and to a certain degree,the minimum +type of information it requires to be executed. The best way to develop a new +test for wmap, is to use already implemented modules as a base and then +develop a normal MSF module that can be run manually from the command line. To +enable a module to be run automatically via wmap just include the mixin that +determine the type of the module (Means: just add the example string to a +module and use the correct type). + + Example: + + include Auxiliary::WMAPScanFile The following are the types of modules implemented at this time and they are listed in the order WMAP runs them: -WMAPScanSSL - Run once against SSL server -WMAPScanServer - Run once against the target Web Server -WMAPScanDir - Runs for every directory found in the target -WMAPScanFile - Runs for every file found in the target -WMAPScanUniqueQuery - Runs for every unique query found in each request to the +WmapScanSSL - Run once against a SSL server +WmapScanServer - Run once against a target Web Server +WmapScanDir - Runs for every directory found in the target +WmapScanFile - Runs for every file found in the target +WmapScanUniqueQuery - Runs for every unique query found in each request to the target -WMAPScanQuery - Runs for every query found in each request to the target -WMAPScanGeneric - Modules to be run after all tests complete. Good place to +WmapScanQuery - Runs for every query found in each request to the target +WmapScanGeneric - Modules to be run after all tests complete.Good place to perform passive analysis of responses, analysis of test results to launch other modules (i.e. exploits). -=[ Reporting. +The execution order not only is handled by the wmap type but also it can be +adjusted across all modules by defining a wmap orderid number using the +'register_wmap_options' method. -It uses the native reporting capabilities of MEtasploit. +Using http_version.rb module as an example: -=[ Database +Class Metasploit3 < Msf::Auxiliary -No more sqlite. it uses whatever the framework is using. + # Exploit mixins should be called first + include Msf::Exploit::Remote::HttpClient + include Msf::Auxiliary::WmapScanServer + # Scanner mixin should be near last + include Msf::Auxiliary::Scanner -=[ Simple example. - -The following are the basic steps for testing a web server/app using WMAP: - - 1. Crawl a web site using the /auxiliary/scanner/http/crawler module - 2. Load the wmap plugin - 3. View available sites to test: - - wmap_sites -l - - 4. View site structure - - wmap_sites -s - - Example: wmap_sites -s www.testsite.org,http://192.168.1.1 - - 5. Define targets from available sites - - wmap_targets -t - - 6. Test it. - - wmap_run -e - -=[ Additional Stuff - - Before runing the test you may need to set certain variables - required by some modules. - - Example: - - msf > setg DOMAIN targetco.com - DOMAIN => targetco.com - - msf > setg EXT .asp - EXT => .asp - - msf > setg WMAP_EXCLUDE_FILE - - NOTE: By default image files are not included in the tests. - - If required profiles can be defined in the following way: - - wmap_run -e path/to/profile/file - - The profile file contains the list of modules to execute. - See data/wmap/wmap_sample.profile for a sample. - -=[ TODO. + def initialize + super( + 'Name' => 'HTTP Version Detection', + ... + ) + register_wmap_options({ + 'OrderID' => 0, + 'Require' => {}, + }) + ... -This is first real release version of WMAP and as you know, the Metasploit project -welcomes feedback, comments, ideas, patches, modules, etc. +'OrderID' Numeric value that represents the order ALL modules will be executed + You can see the modules and orderid with the 'wmap_modules -l' + command: + msf > wmap_modules -l + -=[ EOF. - +'Require' Array of all the modules orderids that are required to be executed + and finished first before the curent module. (This specific + funtionality is still in the works, but the objective is to have + modules to provide results as input to other modules.) + +Wmap enabled modules can be reloaded using the wmap_modules -r command. + +=[ 9. RANDOM NOTES ]---------------------------------------------------------- + +Because every test is a module the datastore is sent to the module in the node +for execution. If a module you create needs a specific option set before +launch just set it in the console as a regular variable. For example: +msf > set DOMAIN abcd.com +DOMAIN => abcd.com +msf > +This is usefull if you want to include exploits in the testing and not only +auxiliary modules. WMAP looks for wmap enabled modules in ALL auxiliary and +exploit modules. +Also if you have asked yourself why there are commands that receive not only +table id but the ugly vhost,url syntax for site and target definition is +because this allows to do complex scripts, so be creative. +If you see a Reauth message in WMAP is because the XMLRPC token is not valid +and a reauthentication to the nodes is required. But dont worry wmap does that +automaticaly for you. + +=[ 10. TO DO ]---------------------------------------------------------------- +- The quality of the scan depends on the quality of the modules. So please +contribute more modules and improvements. If you dont contribute , you dont +have the right to complain. The only key issues to consider are: + + The module should follow metasploit guidelines + + Add the right mixin + + The module should store the right data and results in the database + + The module has to have a clear purpose! +- Also if you want to take the big task of developing a ruby MITM proxy for +metasploit that will be very helpfull for the project. + +=[ Disclaimer ]--------------------------------------------------------------- +I dont work for R7. XD +============================================================================== + et [ ] metasploit.com 2012 + \ No newline at end of file