MSP-11145
**NOTE: Failing specs**
Add a tool for reading `log/untested-payload.log` and
`framework.payloads` to determine `context`s to add
`spec/modules/payloads_spec.rb` to test the untested payloads.
See the complaint on #4039. This doesn't fix that particular
issue (it's somewhat unrelated), but does solve around
a file parsing problem reported by @void-in
MSP-11368
MSP-11143
Remove fastlib as it slows down the code loading process. From the
previous commit, the mean loading for
`METASPLOIT_FRAMEWORK_PROFILE=true msfconsole -q -x exit` was
27.9530±0.3485 seconds (N=10). The mean after removal of fastlib
was 17.9820±0.6497 seconds (N=10). This means an average 35.67%
reduction in boot time.
Rubocop replaces the default YAML library which makes development
testing difficult. It does not cause problems on Travis, but according
to reports, it does cause instability with many individual dev
environments.
While I would love to have a more solid source of this bug report, right
now this was an oral report from @shuckins-r7 (who I tend to believe a
lot).
There are still strong reservations about using Nokogiri to parse
untrusted XML data.
http://www.wireharbor.com/hidden-security-risks-of-xml-parsing-xxe-attack/
It is also believed that many desktop operating systems are still
shipping out-of-date and vulnerable libxml2 libraries, which become
exposed via Nokogiri. For example:
http://stackoverflow.com/questions/18627075/nokogiri-1-6-0-still-pulls-in-wrong-version-of-libxml-on-os-x
While this isn't a problem for binary builds of Metasploit (Metasploit
Community, Express, or Pro) it can be a problem for development
versions or Kali's / Backtrack's version.
So, the compromise here is to allow for modules that don't directly
expose XML parsing. I can't say for sure that the various libxml2
vulnerabilities (current and future) aren't also exposed via
`Nokogiri::HTML` but I also can't come up with a reasonable demo.
Metasploit committers should still look at any module that relies on
Nokogiri very carefully, and suggest alternatives if there are any. But,
it's sometimes going to be required for complex HTML parsing.
tl;dr: Use REXML for XML parsing, and Nokogiri for HTML parsing if you
absolutely must.
It's often you want counts of just WARN and ERROR messages, and don't
want to spam yourself with INFO messages that you don't intend to
address anyway. This is most often the case with CI, such as with
https://travis-ci.org/todb-r7/metasploit-framework
This correct msftidy's disclosure date check to do the following:
1. If the module has a disclosure date, the check should kick in.
2. If the module is an exploit, and doesn't have a disclosure
date, then it will be flagged.
3. If the module is an auxiliary, and doesn't have a disclosure
date, then it will NOT be flgged (because not all aux modules
target bugs/vulns like exploits do).
This will make it possible to run a post-merge check when
pre-commit-hook.rb is referenced as a symlink from .git/hooks/post-merge
The kind of check you're going to do is entirely dependant on the
basename of the file, which is a little weird but convenient.
Verification is a little tricky on this. Coming soon.
Nokogiri has a habit of shipping vulnerable builds of libxml2. For
example, see this:
http://www.ubuntu.com/usn/usn-1904-1/
and compare to Nokogiri's bundled requirements:
https://github.com/sparklemotion/nokogiri/blob/master/dependencies.yml
While Nokogiri is quite pleasant to use, it really shouldn't be trusted
to handle potentially malicious data. Imagine if a "vulnerable" target
was actually a malicious honeypot, lying in wait for a poor Metasploit
user to come along and parse out its payload. (OT: does such a thing
have a clever name? If not, I propose "beehive" to imply the offensive
capabilities of such a honeypot.)
Nokogiri is used elsewhere in Metasploit, but those functions handle
data sourced from the Metasploit user herself, so those XML hunks are
nominally trustworthy.
This change updates msftidy to be run automatically for new modules
added since the last tag release because we can't rely on folks using
tools/dev/pre-commit-hook before submitting a PR. Now, when one attempts
to open a PR with a non-tidy'ed module, the build will fail out of the
gate.
Related to the 100s of msftidy errors extant today.
[SeeRM #8498]
commit c894e52de5705a1133191be5e9caf3ebdee33621
Author: Tod Beardsley <tod_beardsley@rapid7.com>
Date: Fri Jan 31 14:17:02 2014 -0600
Add a jacked up title to test travis. Revert this!
commit 2f00c190be71aeb456a7a546071286fd6d670bc1
Author: Tod Beardsley <tod_beardsley@rapid7.com>
Date: Fri Jan 31 11:39:42 2014 -0600
Allow for checking and spotchecking.
commit db11e8dfad5381030b08c431a183dbafe7a5f304
Author: Tod Beardsley <tod_beardsley@rapid7.com>
Date: Thu Jan 30 17:16:37 2014 -0600
Whoops, need to exit an Integer always.
commit 12d131d3157a78ff11e597476138323ed0a062fc
Author: Tod Beardsley <tod_beardsley@rapid7.com>
Date: Thu Jan 30 16:59:35 2014 -0600
Allow for exit statuses from msftidy.
commit 2c3b294ff17416f49935472caf2b6be3dbdd93a4
Author: Tod Beardsley <tod_beardsley@rapid7.com>
Date: Thu Jan 30 15:36:43 2014 -0600
Be more dynamic about tag checking years
commit d5d8a0b05ac17fb18666a9c252dbb6928d6b5e56
Author: Tod Beardsley <tod_beardsley@rapid7.com>
Date: Thu Jan 30 14:36:44 2014 -0600
Don't warn when there's really nothing
commit fb44a3142fb01eb2647c1c240bb1cc2e7bf59120
Author: Tod Beardsley <tod_beardsley@rapid7.com>
Date: Thu Jan 30 14:21:50 2014 -0600
Revert the intentional failure
This reverts commit 99a7630b0da301b27ac495cb027009a8cd9e2caf.
Fun fact: Reverting a commit does not automatically sign with my current
aliases, one must git revert then git c --amend.
commit 99a7630b0da301b27ac495cb027009a8cd9e2caf
Author: Tod Beardsley <tod_beardsley@rapid7.com>
Date: Thu Jan 30 14:08:05 2014 -0600
Cause an exit status in precommit check
Maybe travis will see these and fail the build.
Don't forget to revert this commit @todb-r7 !
commit 5a3b2fcd9598fae51a0dd2c7c87680c703a85448
Author: Tod Beardsley <tod_beardsley@rapid7.com>
Date: Thu Jan 30 13:11:04 2014 -0600
Update msftidy pre-commit-hook for spotchecking
commit 3f255e36dad9ed3081aaf359f845525d96872ef0
Author: Tod Beardsley <tod_beardsley@rapid7.com>
Date: Thu Jan 30 12:35:16 2014 -0600
Travis should run msftidy via precommit hook
commit 0959d9d2d281590a94c0ac960e43b74354e4e21b
Author: Tod Beardsley <tod_beardsley@rapid7.com>
Date: Thu Jan 30 12:25:53 2014 -0600
Add SPOTCHECK_RECENT to msftidy.rb
In case it takes too long to get a report, the method will give up
checking after one hour. The user can still manually check the report
from the analysis link given earlier.
[SeeRM #8733] This a tool that uses VirusTotal's public API to submit
a malware sample for analysis. As an offensive tool developer, this
would provide a convenient way to check and see how AVs react to
something we write.
Also, enforce binary encoding like the other Metasploit tools.
This opens the door to fixing files that have things that could be fixed
programmatically.
[SeeRM #8497]
Instead of just relying on a filename of *.rb, use the file utility to
determine file type.
For systems that lack lack 'which' and 'file', fall back to filename
matching.
This is useful for retabbing things like 'msfconsole' that don't have a
.rb extension.
Local backups are generally not needed since you can just git checkout
old versions anyway before committing. It was nice to have during dev
but generally shouldn't be done now.
This signals a move to allowing for normal Ruby indentation (2 space
soft tabs). This change will check files for indentation of spaces or of
tabs, since we don't want to fail out all modules quite yet.
For more, see
https://github.com/rapid7/metasploit-framework/wiki/Indentation-Standards
where all details of the conversion plan will be documented in order to
minimize the amount of whitespace conflict we are sure to encounter over
this conversion.
Usage: tools/dev/retab.rb directory
will retab with 2-width spaces rather than tabs for indentation.
This utility should be used by the @tabassassin account when it's
unleashed on the Metasploit code base in order to make git blame a
little easier to spot. (diffs should use -b or -w to avoid seeing
@tabassassin's changes)
This merge adds four new tools:
* .mailmap : allows for easier identification of committers
* tools/module_count.rb : Spits out a current count of modules
* tools/module_commits.rb: Spits out who commited to a module
* tools/committer_counts.rb : Spits out commiters by commit counts
This was part of a long-running feature branch, which is why it's now
bundled up in one big squash merge.
Squashed commit of the following:
commit de201ff6a5b304d0fedec56d9f1930abf1a10d9e
Author: Tod Beardsley <todb@metasploit.com>
Date: Thu Jan 24 14:48:24 2013 -0600
Rename from scorecard to merely a count
commit 8028cf838b0b560831602e3163e92d0751a4c0a9
Author: Tod Beardsley <todb@metasploit.com>
Date: Thu Jan 24 14:36:42 2013 -0600
Some final comment docs
commit a69fd7883837849664bc8777d119ac760de4a43d
Merge: e288f13 3faf4b3
Author: Tod Beardsley <todb@metasploit.com>
Date: Thu Jan 24 13:21:14 2013 -0600
Merge branch 'master' into committer-scorecard
I think these conflicts came from a move or a rename or something.
Conflicts:
external/source/exploits/cve-2012-5076_2/Makefile
external/source/exploits/cve-2012-5088/Makefile
modules/exploits/multi/browser/java_jre17_method_handle.rb
modules/exploits/multi/http/jenkins_script_console.rb
commit e288f13d7f7bca7aa4ceddd555b88d971a9f65a2
Author: Tod Beardsley <todb@metasploit.com>
Date: Wed Jan 16 14:06:23 2013 -0600
Add FireFart's mail alias
commit 1b1792e84febf015a79c3beb3d2473953da56935
Author: Tod Beardsley <todb@metasploit.com>
Date: Fri Jan 18 22:41:44 2013 -0600
Fix grammar on description for webcam
commit 276388fac541f0eebb9a18a980c5b474f438d117
Author: Robin Wood <robin@digininja.org>
Date: Tue Jan 22 15:42:23 2013 +0000
added extra checking for strict databases
commit a40ea3d73e52ab822cb89052ef7575f7ac52abb6
Author: jvazquez-r7 <juan.vazquez@metasploit.com>
Date: Tue Jan 22 12:07:16 2013 +0100
fix data added to table
commit 738d2fad5fccfbff23967ce219ad6bd4af90bbea
Author: HD Moore <hd_moore@rapid7.com>
Date: Tue Jan 22 00:27:03 2013 -0600
Fix a stack overflow in bidirectional pipe
commit aeec5a816b2f09f517930cdff074ea4b42ed5088
Author: jvazquez-r7 <juan.vazquez@metasploit.com>
Date: Mon Jan 21 12:26:35 2013 +0100
Cleanup for mysql_file_enum.rb
commit 13f68f089b4f3dd7c58bb4d5cb5767ff3df12852
Author: sinn3r <msfsinn3r@gmail.com>
Date: Mon Jan 21 00:30:43 2013 -0600
Updates the progress function
Because the previous one was wrong.
commit d971fe0bb5f34667b6a621043838f7472e7255cd
Author: Robin Wood <robin@digininja.org>
Date: Sun Jan 20 21:32:02 2013 +0000
Brute force directory and file names with MySQL
commit a96ca2e96a3a34e302a6759ba48706c60b9724cd
Author: Robin Wood <robin@digininja.org>
Date: Sun Jan 20 00:13:42 2013 +0000
added a warning and using optpath
commit aa98d85abbc30166ce7d69a446bf78cddff92e0a
Author: Robin Wood <robin@digininja.org>
Date: Sun Jan 20 00:12:38 2013 +0000
added a warning and using optpath
commit 6dd5bb8532d0f68d44ca80099780428e0a3ad872
Author: Robin Wood <robin@digininja.org>
Date: Sun Jan 20 00:02:07 2013 +0000
stopped using fixed table name
commit 520aeb93119a77b4eb8d1187cac4084690d45613
Author: Robin Wood <robin@digininja.org>
Date: Sat Jan 19 23:41:38 2013 +0000
Fixed msftidy stuff
commit cec6a06c56444f12dc8b8985c2505b2d259d5077
Author: Robin Wood <robin@digininja.org>
Date: Sat Jan 19 22:48:00 2013 +0000
File/dir brute forcer using MySQL
commit 3cc0f3feaed87df11ab3695342af304d3b13d056
Author: jvazquez-r7 <juan.vazquez@metasploit.com>
Date: Sun Jan 20 19:54:24 2013 +0100
finally it doesn't use FileDropper atm
commit 2670d5ca8fbe2b26b2073445537bf0bfacd079dd
Author: jvazquez-r7 <juan.vazquez@metasploit.com>
Date: Sun Jan 20 17:38:37 2013 +0100
references and date updated
commit 1230d5267b3a8b33cfd64f6efb613986d6d13b31
Author: bcoles <bcoles@gmail.com>
Date: Mon Jan 21 02:12:42 2013 +1030
update php_charts_exec metadata
commit cf37c594e55b0130640f5aaea240b3aa936b7c8d
Author: bcoles <bcoles@gmail.com>
Date: Mon Jan 21 02:10:48 2013 +1030
move and update php_charts_exec metadata
commit 1e86429fa16a2f5d5003fbe6e69a74cac5efd767
Author: bcoles <bcoles@gmail.com>
Date: Sun Jan 20 23:51:17 2013 +1030
Add PHP-Charts v1.0 PHP Code Execution Exploit
commit fe60ee6dffc60a53b28bcfd08b5aada8bc8d4000
Author: jvazquez-r7 <juan.vazquez@metasploit.com>
Date: Sun Jan 20 13:42:02 2013 +0100
linux stager plus little cleanup
commit 5900248f585e7a5e10d93a0672aa8d330d5581ee
Author: Spencer McIntyre <zeroSteiner@gmail.com>
Date: Sat Jan 19 19:10:56 2013 -0500
use target_uri and normalize_uri as well as fix a cookie problem
commit a7ce0a500fe1ae4c71652191ee97ba1757cf65e0
Author: Spencer McIntyre <zeroSteiner@gmail.com>
Date: Fri Jan 18 14:56:52 2013 -0500
add module to execute commands via Jenkins Script Console
commit 33b8aa49f4dbbfbcc275b5cc0dfc43db9fec08f8
Author: jvazquez-r7 <juan.vazquez@metasploit.com>
Date: Fri Jan 18 18:42:27 2013 +0100
title updated
commit 63fe457fadf66ac27eac6210a26880c1f816d0ce
Author: Charles Smith <charles.smith@n2netsec.com>
Date: Thu Jan 17 16:52:02 2013 -0500
Fixed loot formatting so data is under the proper column
The credentials table was defined with the columns "User", "Password", "Host", "Port", and "SSL". Credentials were not added in that order, however. They were added in the order "host, port, user, password, ssl" in this line:
credentials << [cred['host'], cred['port'], cred['user'], cred['password'], cred['ssl']]
I changed the order the columns were defined to fix this.
The permissions table had a similar issue. The "FileWrite" column was missing, so I added it. I also moved the "Home" column to after the "AutoCreate" column. Now the line:
permissions << [perm['host'], perm['user'], perm['dir'], perm['fileread'], perm['filewrite'], perm['filedelete'], perm['fileappend'],perm['dircreate'], perm['dirdelete'], perm['dirlist'], perm['dirsubdirs'], perm['autocreate']]
works correctly.
commit b948559b5ae0090c9ecb704bfba2da219577d4f4
Author: jvazquez-r7 <juan.vazquez@metasploit.com>
Date: Thu Jan 17 21:45:13 2013 +0100
cleanup
commit 199ab00a9c46295776b3f9c47d941721d5777a65
Author: jvazquez-r7 <juan.vazquez@metasploit.com>
Date: Thu Jan 17 21:39:41 2013 +0100
cleanup
commit 8d5504475dbce315581e87f395c9453bbe624d2e
Author: jvazquez-r7 <juan.vazquez@metasploit.com>
Date: Thu Jan 17 21:27:47 2013 +0100
Added new module for cve-2012-5076
commit 31ae18f392dea9fcfc4e1e6e1ec627aed2513d09
Author: jvazquez-r7 <juan.vazquez@metasploit.com>
Date: Thu Jan 17 21:14:49 2013 +0100
Added module for CVE-2012-5088
commit 6ac99f3db8f464767d15aaf60a2a5796b4ae8b30
Author: Tod Beardsley <todb@metasploit.com>
Date: Sat Jan 19 09:08:31 2013 -0600
Add a quick comment doc
commit 0c18f1c7cb53a77b4338e6014b76ea74749b41f9
Author: Tod Beardsley <todb@metasploit.com>
Date: Sat Jan 19 09:06:34 2013 -0600
Adds a per-module commit counter.
commit 44fa22832bb2e229f5a96a62658d7c4b0b88b966
Merge: fa288ff 9f42abd
Author: Tod Beardsley <todb@metasploit.com>
Date: Sat Jan 19 08:30:37 2013 -0600
Merge remote-tracking branch 'origin/master' into committer-scorecard
commit fa288ff007c1ead48ca011cda2488164d5103715
Author: Tod Beardsley <todb@metasploit.com>
Date: Fri Jan 18 14:05:47 2013 -0600
Make module_count execable
commit 6c1625ed709f505ec9e8be89820f9d6827a52567
Author: Tod Beardsley <todb@metasploit.com>
Date: Tue Jan 8 09:56:48 2013 -0600
Wrote a quick module counter, by type
commit af07ddc8184b85ecd43fb9e2cb2c607d54fb0c1b
Merge: 2ee5df8 2c3ccb5
Author: Tod Beardsley <todb@metasploit.com>
Date: Tue Jan 8 09:35:28 2013 -0600
Merge remote-tracking branch 'origin/master' into committer-scorecard
commit 2ee5df810313290a753344b83a9b9e591c30ef05
Merge: 501c678 b50e040
Author: Tod Beardsley <todb@metasploit.com>
Date: Fri Jan 4 10:24:27 2013 -0600
Merge remote-tracking branch 'origin/master' into committer-scorecard
commit 501c678b2ca6f67639d7d7425469d380ba6534cf
Merge: 8001401 c2586d0
Author: Tod Beardsley <todb@metasploit.com>
Date: Thu Dec 27 15:42:25 2012 -0600
Merge branch 'master' into committer-scorecard
commit 800140176686c8aa4e41629b259a1bcb8b7c9e0c
Author: Tod Beardsley <todb@metasploit.com>
Date: Thu Dec 27 11:13:04 2012 -0600
Adding shuckins and cjr to the mailmap
commit ab2db49c17b78616dc9199d62928e65d624e9e12
Merge: 8b6ecb3 daf5465
Author: Tod Beardsley <todb@metasploit.com>
Date: Thu Dec 27 10:29:19 2012 -0600
Merge remote branch 'origin/master' into committer-scorecard
commit 8b6ecb34bd2a1719bc51ab136cb9de1a8cd5c782
Author: Tod Beardsley <todb@metasploit.com>
Date: Mon Dec 17 21:58:37 2012 -0600
Comment docs on .mailmap
commit 8e245a086c2e91a80be31accdb6349837cba3dff
Author: Tod Beardsley <todb@metasploit.com>
Date: Mon Dec 17 21:56:06 2012 -0600
Another alias for h0ng10
commit aff6169602791a048cff2e41bac5cbb565abd341
Author: Tod Beardsley <todb@metasploit.com>
Date: Mon Dec 17 17:02:35 2012 -0600
A more useful committer score card
Now with aliases for anyone who hit the top 20 list of the last year,
six months, and twelve weeks.
Still needs some optparsey niceities, but it's good enough for an
intial push to GitHub.
commit bd4e00ee019cedfed2eb8af6b52786f5184193ca
Author: Tod Beardsley <todb@metasploit.com>
Date: Mon Dec 17 15:22:33 2012 -0600
Initial commit of a git commit scorecard
Sometimes the first letter of a word shouldn't be capitalized.
If you do, it may actually be technically incorrect. For example:
a function name, a filename, or even a software name like freeFTPd.
We should ignore scenarios like those.
It's not always possible or technically correct to uppercase every
word. For example, sometimes a module may contain a filename or
function name. We should allow these conditions to pass.
When File.executable?() is used, it checks if the file is an
executable AND that exe permission is set. Well, we don't really
have to set the permission we can still load/use the module. So
really, all we care is the module must be a ".rb" to make sure
it loads at startup. So that's what we need to check.
Changes include:
* Make invalid ranking an error, because it may affect module
search in msfconsole, or the metasploit site.
* Make invalid disclosure format an error, same concern as above.
* Plus other changes
- Adds a call to mprotect(2) to the reverse and bind stagers
- Adds accurate source for some other linux shellcode, including some
comments to make it more maintainable
- Adds tools/module_payload.rb for listing all payloads for each exploit
in a greppable format. Makes it easy to find out if a payload change
causes a payload to no longer be compatible with a given exploit.
- Missing from this commit is source for reverse_ipv6_tcp
sinn3r
Luke Imhoff
URI Assassin
William Vu
Tod Beardsley
Josh
Joshua Smith
Jon Hart
HD Moore
Christian Mehlmauer
Sagi Shahar
Rob Fuller
Joe Vennix
jvazquez-r7
Ethan Robish
James Lee
lsanchez-r7
jvazquez-r7
h0ng10
Brandon Turner
Trevor Rosen
Tod Beardsley
sinn3r
TADA Tadashi
Alexander Klink