metasploit-framework/documentation/wmap.txt

                         .-.-.-..-.-.-..---..---.
                         | | | || | | || | || |-'
                         `-----'`-'-'-'`-^-'`-' 
                           Metasploit Wmap 1.5
==============================================================================
Efrain Torres                                      et [  ] metasploit.com 2012
------------------------------------------------------------------------------

=[ 0. Intro ] ----------------------------------------------------------------

So its 2012 and before the Mayans are proven to be right I was able to create
a new version of this plugin. If you have read the old wmap documentation this
is what is going on: 

Wmap is still a general purpose web application scanning framework for 
Metasploit.  Still is a different approach compared to other open source 
alternatives and commercial scanners, as Wmap is not build around any browser 
or spider for data capture and manipulation. And the best thing is that still 
is FR33. Lots of bugs are gone and the new code allows for faster and more 
efficient execution.

=[ 1. How it works ] ---------------------------------------------------------

The old architecture (versions < 1.5):

[CLIENT] ----- [ATTACK PROXY] ----- [TARGET]
    |                 |                 ^ 
    +--------->[METASPLOIT DB]          |
                      |                 |
            [MSF 3 - Wmap SCANNER]      | 
            [MSF 3 - Wmap MODULES] -----+

The new architecture:

                                        [CLIENTS]
                                            |
                                            | 
   +-------[Wmap PLUGIN]<-----+----->[METASPLOIT DB]
   |             |            |             |
   |             |            |             | 
[NODE 1]     [NODE 2]     [NODE n] ---------+
   |             |            |   \
   |             |            |   [Wmap MODULES] 
   +---------[TARGETS]--------+    


Wmap is a Metasploit plugin and will interact with the database, reading all
gathered traffic from any client you have configured/adapted or duct taped to
store web sites, requests, responses and forms in the Metasploit DB. 

The test performed are all Metasploit modules which WMAP execute in a 
configurable order. The test modules are implemented as auxiliary or exploit 
modules and they can interact with any other MSF component including the 
database other exploits and plugins.

The new architecture allows to have different distributed clients and nodes 
all storing results and data to a central database. This means that large 
enviorments can be tested using multiple metasploit msfrcpd servers (nodes) 
controled from one (or more) WMAP consoles. Wmap will execute the tests to be 
launched from each node distributing evenly the job load across all configured 
nodes.

In case you dont want to use a distributed model wmap will detect that no 
nodes have been configured and will run the modules from the local host.


=[ 2. Crawlers,proxies and other clients ] -----------------------------------

At this time Metasploit have 4 components that may be used as clients 
to store web sites in the database: 

(1) If you have configured your database properly and use the 
auxiliary/scanner/http/crawler module, this module will create a web site 
(with related host and service) and store all requests,responses and forms 
automatically.

(2) Less known is that metasploit has a different crawler called msfcrawler 
and besides  supporting  modules to parse the responses in any way you want 
it will also  store the required data in the database.

(3) Also any module that creates a web_site in the database (e.g. 
auxiliary/scanner/http/http_version module) 
will add a site to the database that can be selected as a target in Wmap,
however the only path you will be storing will be the root path of the 
website '/'.

(4) Metasploit has the awesome 'db_import' command that allows to import
multiple scan results from many sources. For tools like Acunetix,
Burp and Appscan the results contains web_pages and forms. For the rest 
(at this time) the results will import services (no web sites, pages or 
forms associated to them). 

    msf > db_import
    Usage: db_import <filename> [file2...]

    Filenames can be globs like *.xml, or **/*.xml which will search recursively
    Currently supported file types include:
    Acunetix XML
    Amap Log
    Amap Log -m
    Appscan XML
    Burp Session XML
    ... 


Or you can add a site manually to the database using the 'wmap_sites -a' 
command (after loading the wmap plugin. See '4. Wmap Plugin'):

 msf > wmap_sites -a www.blah.net,http://192.168.0.2/
 [*] Site created. 

    Note: www.blah.net,http://192.168.0.2/ <-- is one site vhost,url
 
For other tools to store web data in the database the only hard part is to 
deal with the ruby marshalling of parameters in the web_forms table. (Topic 
for another paper). But this is one of the main issues regarding the use of 
other tools to connect to the database. However any Ruby based tool can be 
modified easily to do this. 

If you noticed the previous architecture the ATTACK PROXY has gone the way of 
the Dodo (actually not as is just another client). But i will stop mention it 
because i have been unable to create a good Ruby based MITM proxy (Sorry) 
and second because i dont want to maintain plugins for every type of proxy 
out there. Is your exercise to create custom plugins for your tools to connect
to the database, after that Wmap does not care where the data comes from.

=[ 3. The Wmap cycle ]--------------------------------------------------------

Or how every other scanner works but in wmap fancy terms.

0. Gather data from (See Crawlers,proxies and other clients) 
   targets      
1. Load the scanner (Load wmap plugin. See "4. Wmap Plugin")
2. Optional: Define nodes     (Use 'wmap_nodes'. See "6. Wmap Nodes")
3. Define targets   (Use 'wmap_sites' and 'wmap_targets'.See "5. Wmap Targets")
4. Configure        (Lots of things here)
5. Launch           (Use 'wmap_run'. See "7. Launch a scan") 
6. Enjoy WTF Moments(Priceless...)

Note: Step 2 is optional as Wmap allows you to perform distributed scans 
using multiple nodes. If no nodes are configured the scan runs as usual
running all tests from the local host.

=[ 4. Wmap Plugin ]-----------------------------------------------------------

To launch wmap open a Metasploit console and load the wmap plugin.

msf > load wmap
[*] [Wmap 1.5] ===  et [  ] metasploit.com 2012
[*] Successfully loaded plugin: wmap

Now that the plugin is loaded lets go through the list of basic commands:

msf > help wmap

Wmap Commands
=============

    Command       Description
    -------       -----------
    wmap_nodes    Manage nodes
    wmap_run      Test targets
    wmap_sites    Manage sites
    wmap_targets  Manage targets
    wmap_modules  Manage wmap modules
    wmap_vulns    Display web vulns

=[ 5. Wmap Targets ]----------------------------------------------------------

The targets are selected from the sites already stored in the database. For 
example after crawling a site (See "2. Crawlers,proxies and other clients"). 
Now we can use the command 'wmap_sites' to list them:

msf > wmap_sites
[*] Usage: wmap_sites [options]
        -h              Display this help text
        -a [url]        Add site (vhost,url)
        -l              List all available sites
        -s [id]         Display site structure (vhost,url|ids) (level)

msf > wmap_sites -l
[*] Available sites
===============

     Id  Host         Vhost          Port  Proto  # Pages  # Forms
     --  ----         -----          ----  -----  -------  -------
     0   10.10.10.1   blah.xyz.com   443   https  3        2
     1   10.10.10.2   blah.xyz.com   443   https  3        2
     2   10.1.2.2     nah.test.com   443   https  1        0
     3   10.4.3.10    test.abcd.com  80    http   1        1  

Note 1: Metasploit/Wmap supports multiple Vhosts/IPs. 

Note 2: If you want to check the web site structure use the '-s site_id' flag 
like this (also especify an optional level to display):

msf > wmap_sites -s 0 1
      (First level of site 0)
msf > wmap_sites -s 0

   [10.10.10.1] (blah.xyz.com)
         |
         +-------- dir1
                    | 
                    +------ login.php
         +-------- dir2
         ....

Then from the table we can select the targets we want to scan with the
'wmap_targets' command:

msf > wmap_targets
[*] Usage: Wmap_targets [options]
        -h              Display this help text
        -t [urls]       Define target sites (vhost1,url[space]vhost2,url)
        -d [ids]        Define target sites (id1, id2, id3 ...)
        -c              Clean target sites list
        -l              List all target sites

You can define targets in two ways, using the vhost,url syntax (-t) or the 
table ids (-d)

msf > wmap_targets -t test.abcd.com,http://10.4.3.10/

msf > wmap_targets -d 0,1
[*] Loading blah.xyz.com,https://10.10.10.1:443/.
[*] Loading blah.xyz.com,https://10.10.10.2:443/.

To see the list of all the targets to scan at this time run the 
command with the (-l) flag.

msf > wmap_targets -l
[*] Defined targets
===============

     Id  Vhost            Host            Port  SSL    Path
     --  -----            ----            ----  ---    ----
     0   blah.xyz.com     10.10.10.1      443   true   /
     1   blah.xyz.com     10.10.10.2      443   true   /
     2   test.abcd.com    10.4.3.10       80    false  /


=[ 6. Wmap Nodes ]------------------------------------------------------------

Wmap uses 'nodes' as a way to distribute the execution of the test against 
one or more targets. Nodes are not required to run wmap . if nodes are not 
configured and a scan is launched wmap will detect this and launch all tests 
from the local host.

The nodes are just msfrpcd servers that are created the following way:

msf>ruby msfrpcd -h

Usage: msfrpcd <options>

OPTIONS:

    -P <opt>  Specify the password to access msfrpcd
    -S        Disable SSL on the RPC socket
    -U <opt>  Specify the username to access msfrpcd
    -a <opt>  Bind to this IP address
    -f        Run the daemon in the foreground
    -h        Help banner
    -n        Disable database
    -p <opt>  Bind to this port instead of 55553
    -u <opt>  URI for Web server

msf>ruby msfrpcd -U msf -P nodepass
[*] MSGRPC starting on 192.168.0.1:55553 (SSL):Msg...
[*] MSGRPC backgrounding at 2012-01-17 11:01:01 -0600...

if you want to create a msfrpc server from the msfconsole you can do it by
loading the msgrpc plugin:

msf > load msgrpc User=msf Pass=nodepass
[*] MSGRPC Service:  127.0.0.1:55552
[*] MSGRPC Username: msf
[*] MSGRPC Password: nodepass
[*] Successfully loaded plugin: msgrpc

On a later stage in the wmap console we will add such nodes so the scans can 
be distributed across all the configured nodes. so remember how you deployed 
your nodes so they can be configured in wmap.

In the metasploit console after you have loaded the wmap plugin you can add 
the previous nodes with the 'wmap_nodes' command:

msf > wmap_nodes
[*] Usage: wmap_nodes [options]
        -h                              Display this help text
        -c id                           Remove id node (Use ALL for ALL nodes
        -a host port ssl user pass      Add node
        -d host port user pass db       Force all nodes to connect to a db
        -j                              View detailed jobs
        -k ALL|id ALL|job_id            Kill jobs on node
        -l List all current nodes

msf > wmap_nodes -a 192.168.0.1 55553 true msf nodepass
[*] Connected to 192.168.0.1:55553 [4.2.0-dev].
[*] Node created.

Note: When launching msfrpcd waiht for a couple of seconds beofr adding it to 
wmap as msfrpcd sometimes is slow to start accepting connections. 

Add as many nodes you want. To see the list use 'wmap_nodes -l':

msf > wmap_nodes -l
[*] Nodes
=====

     Id  Host        Port   SSL   User  Pass      Status     #jobs
     --  ----        ----   ---   ----  ----      ------     -----
     0   127.0.0.1   55553  true  msf   nodepass  4.2.0-dev  0
     1   192.168.0.1 55553  true  msf   nodepass  4.2.0-dev  0

Note: After launching all tests this command will allow you to see if all 
your jobs have been completed (#jobs == 0).

Remember that all these commands can be added as a .rc file so you dont 
have to type again and again the loading of wmap and the configuration of 
nodes. 

After you have your nodes connected then you can force them to connect to 
the central metasploit database:

msf > wmap_nodes -d 127.0.0.1 7175 dbuser dbpass msf3
[*] db_connect {"driver"=>"postgresql", "db"=>"msf3"} 127.0.0.1:7175 OK
[*] db_connect {"driver"=>"postgresql", "db"=>"msf3"} 192.168.0.1:7175 OK
[*] OK.


=[ 7. Launch a scan ]---------------------------------------------------------

Now that database,targets and maybe nodes are set we run a scan with the 
'wmap_run' command:

msf > wmap_run
[*] Usage: wmap_run [options]
        -h                    Display this help text
        -t                    Show all enabled modules
        -m [regex]            Launch only modules that match provided regex
        -p [regex]            Only test path defined by regex..
        -e [/path/to/profile] Launch profile modules against all targets.
                              No file runs all enabled modules.

msf > wmap_run -e
[*] Using ALL wmap enabled modules.
[*] Testing target:
[*]     Site: test.abcd.com (10.4.3.10)
[*]     Port: 80 SSL: false
============================================================
[*] Testing started. 2012-12-21 0:0:0 -0600
[*]
=[ SSL testing ]=
============================================================
[*] Target is not SSL. SSL modules disabled.
[*]
=[ Web Server testing ]=
============================================================
[*] Module auxiliary/admin/http/http_version
[*] Module auxiliary/admin/http/tomcat_administration
[*] Module auxiliary/admin/http/tomcat_utf8_traversal
[*] Module auxiliary/admin/http/trendmicro_dlp_traversal
[*] Module auxiliary/scanner/http/cisco_nac_manager_traversal
....
msf >

As you see here wmap executes each of the modules against the defined targets.
(See "8. Wmap Modules") If nodes were configured it will go thru the list of 
nodes and will send a job to the less loaded node to execute the especific 
module with the required options. To force a good job distribution across the
nodes wmap has a limit of 25 jobs per node. If a node has reached the limit it
will try with the next node until there is a slot available.  

Check periodically with the 'wmap_nodes -l' command to see the current job 
status. After the asssement is complete now you can use the normal metasploit 
commands to see the results.

To view detailed job information on each node use the 'wmap_nodes' (-j) flag:

msf >wmap_nodes -j
[*] [Node #0: 127.0.0.1 Port:55553 SSL:true User:msf]
[*] Jobs
    ====

     Id  Job name                             Target          PATH
     --  --------                             ------          ----
     0   Auxiliary: scanner/http/dir_scanner  192.168.0.1:80  /
     ...


[*] [Node #1: 127.0.0.1 Port:55555 SSL:true User:msf]
[*] Jobs
    ====

     Id  Job name                             Target          PATH
     --  --------                             ------          ----
     2   Auxiliary: scanner/http/dir_scanner  192.168.0.2:80  /
     ...

Also you can kill especific jobs or all jobs from one or all nodes:

msf > wmap_nodes -k 0 ALL
[*] Node 0 Killed job id 262 Auxiliary: admin/http/tomcat_administration
[*] Node 0 Killed job id 263 Auxiliary: admin/http/tomcat_utf8_traversal
[*] Node 0 Killed job id 271 Auxiliary: scanner/http/soap_xml
[*] Node 0 Killed job id 299 Auxiliary: scanner/http/brute_dirs
[*] Node 0 Killed job id 300 Auxiliary: scanner/http/brute_dirs
[*] Node 0 Killed job id 301 Auxiliary: scanner/http/brute_dirs
....

If during the scan a node dies wmap will disable the node and will keep 
sending the jobs to the other active nodes. 

If nodes were not configured wmap will launch the tests from the 
local host the old fashion.

=[ 8. Wmap Modules ] ---------------------------------------------------------

Wmap modules are normal Metasploit modules. Each module has a WMAP type, 
this determine when the module is launched and to a certain degree,the minimum 
type of information it requires to be executed. The best way to develop a new 
test for wmap, is to use already implemented modules as a base and then 
develop a normal MSF module that can be run manually from the command line. To
enable a module to be run automatically via wmap just include the mixin that 
determine the type of the module (Means: just add the example string to a 
module and use the correct type).  

    Example:

    include Auxiliary::WmapScanFile

The following are the types of modules implemented at this time and they are
listed in the order WMAP runs them:

WmapScanSSL         - Run once against a SSL server
WmapScanServer      - Run once against a target Web Server
WmapScanDir         - Runs for every directory found in the target 
WmapScanFile        - Runs for every file found in the target
WmapScanUniqueQuery - Runs for every unique query found in each request to the 
                      target 
WmapScanQuery       - Runs for every query found in each request to the target
WmapScanGeneric     - Modules to be run after all tests complete.Good place to
                      perform passive analysis of responses, analysis of test
                      results to launch other modules (i.e. exploits). 

Note: Multiple mixins can be included in a module if needed.

The execution order not only is handled by the wmap type but also it can be 
adjusted across all modules by defining a wmap orderid number using the 
'register_wmap_options' method. 

Using http_version.rb module as an example:

Class Metasploit3 < Msf::Auxiliary

 # Exploit mixins should be called first
 include Msf::Exploit::Remote::HttpClient
 include Msf::Auxiliary::WmapScanServer
 # Scanner mixin should be near last
 include Msf::Auxiliary::Scanner

 def initialize
  super(
   'Name'        => 'HTTP Version Detection',
   ...
  )
  
  register_wmap_options({
   'OrderID' => 0,
   'Require' => {},
  })
  ...    

'OrderID' Numeric value that represents the order ALL modules will be executed
          You can see the modules and orderid with the 'wmap_modules -l' 
          command:

          msf > wmap_modules -l
    

'Require' Array of all the modules orderids that are required to be executed 
          and finished first before the curent module. (This specific 
          funtionality is still in the works, but the objective is to have 
          modules to provide results as input to other modules.)
    
Wmap enabled modules can be reloaded using the wmap_modules -r command. 
  
=[ 9. RANDOM NOTES ]----------------------------------------------------------
       
Because every test is a module the datastore is sent to the module in the node
for execution. If a module you create needs a specific option set before 
launch just set it in the console as a regular variable. For example:

msf > set DOMAIN abcd.com
DOMAIN => abcd.com
msf >

This is usefull if you want to include exploits in the testing and not only 
auxiliary modules. WMAP looks for wmap enabled modules in ALL auxiliary and 
exploit modules.  

Also if you have asked yourself why there are commands that receive not only 
table id but the ugly vhost,url syntax for site and target definition is 
because this allows to do complex scripts, so be creative.

If you see a Reauth message in WMAP is because the XMLRPC token is not valid 
and a reauthentication to the nodes is required. But dont worry wmap does that
automaticaly for you.
  
=[ 10. Results ]--------------------------------------------------------------

Modules may report results as notes (notes) , vulnerabilities (vulns) and/or
web vulnerabilities (web vulns). As notes and general vulnerabilities can be
displayed using the metasploit commands 'notes' and 'vulns', Wmap implements
'wmap_vulns' to display the results stored in the web_vulns db table. The 
reporting is basic at this time , however the Metasploit database can be
easily accessed to fullfill your reporting needs.

Note: Always check 'notes', 'vulns' and 'wmap_vulns' for results.
  

=[ 11. TO DO ]----------------------------------------------------------------

- The quality of the scan depends on the quality of the modules. So please 
contribute more modules and improvements. If you dont contribute , you dont 
have the right to complain. The only key issues to consider are:

 + The module should follow metasploit guidelines
 + Add the right mixin(s)
 + The module should store the right data and results in the database
 + Always use report_vuln or report_web_vuln to report output from a 
   module as report_note overwrites results if 'type' is the same.
 + The module has to have a clear purpose!!!! 
   Is better to have multiple modules with simple tasks/objectives that 
   one that does everything. 
 + The variables/options used in the datastore have to be the same between 
   wmap and the module so wmap can pass the right information to it.
   Usually this is the naming convention to use for the options:

   OptString:

   'VHOST'  = HTTP Virtual Host
   'METHOD' = HTTP Method
   'PATH'   = HTTP URI Path.
   'QUERY'  = HTTP URI Query usually in the param1=value1& form.
   'DATA'   = HTTP Data. In a POST request is the body of the request.
              Usually in the param1=value1& form.
   'HEADERS'= HTTP headers (header1=value1;..)
   
   OptBool:
   
   'VERBOSE'= Verbose flag.	 

   Note: This naming convention may change. However if this naming 
   convention is used any changes can be implemented easily.

- Also if you want to take the big task of developing a ruby MITM proxy for 
metasploit that will be very helpfull for the project. 
 
=[ Disclaimer ]---------------------------------------------------------------
I dont work for R7. XD 
==============================================================================
                                                   et [  ] metasploit.com 2012