MSP-11145
**NOTE: Failing specs**
Add a tool for reading `log/untested-payload.log` and
`framework.payloads` to determine `context`s to add
`spec/modules/payloads_spec.rb` to test the untested payloads.
See the complaint on #4039. This doesn't fix that particular
issue (it's somewhat unrelated), but does solve around
a file parsing problem reported by @void-in
MSP-11368
MSP-11143
Remove fastlib as it slows down the code loading process. From the
previous commit, the mean loading for
`METASPLOIT_FRAMEWORK_PROFILE=true msfconsole -q -x exit` was
27.9530±0.3485 seconds (N=10). The mean after removal of fastlib
was 17.9820±0.6497 seconds (N=10). This means an average 35.67%
reduction in boot time.
Rubocop replaces the default YAML library which makes development
testing difficult. It does not cause problems on Travis, but according
to reports, it does cause instability with many individual dev
environments.
While I would love to have a more solid source of this bug report, right
now this was an oral report from @shuckins-r7 (who I tend to believe a
lot).
There are still strong reservations about using Nokogiri to parse
untrusted XML data.
http://www.wireharbor.com/hidden-security-risks-of-xml-parsing-xxe-attack/
It is also believed that many desktop operating systems are still
shipping out-of-date and vulnerable libxml2 libraries, which become
exposed via Nokogiri. For example:
http://stackoverflow.com/questions/18627075/nokogiri-1-6-0-still-pulls-in-wrong-version-of-libxml-on-os-x
While this isn't a problem for binary builds of Metasploit (Metasploit
Community, Express, or Pro) it can be a problem for development
versions or Kali's / Backtrack's version.
So, the compromise here is to allow for modules that don't directly
expose XML parsing. I can't say for sure that the various libxml2
vulnerabilities (current and future) aren't also exposed via
`Nokogiri::HTML` but I also can't come up with a reasonable demo.
Metasploit committers should still look at any module that relies on
Nokogiri very carefully, and suggest alternatives if there are any. But,
it's sometimes going to be required for complex HTML parsing.
tl;dr: Use REXML for XML parsing, and Nokogiri for HTML parsing if you
absolutely must.
It's often you want counts of just WARN and ERROR messages, and don't
want to spam yourself with INFO messages that you don't intend to
address anyway. This is most often the case with CI, such as with
https://travis-ci.org/todb-r7/metasploit-framework
This correct msftidy's disclosure date check to do the following:
1. If the module has a disclosure date, the check should kick in.
2. If the module is an exploit, and doesn't have a disclosure
date, then it will be flagged.
3. If the module is an auxiliary, and doesn't have a disclosure
date, then it will NOT be flgged (because not all aux modules
target bugs/vulns like exploits do).
This will make it possible to run a post-merge check when
pre-commit-hook.rb is referenced as a symlink from .git/hooks/post-merge
The kind of check you're going to do is entirely dependant on the
basename of the file, which is a little weird but convenient.
Verification is a little tricky on this. Coming soon.
Nokogiri has a habit of shipping vulnerable builds of libxml2. For
example, see this:
http://www.ubuntu.com/usn/usn-1904-1/
and compare to Nokogiri's bundled requirements:
https://github.com/sparklemotion/nokogiri/blob/master/dependencies.yml
While Nokogiri is quite pleasant to use, it really shouldn't be trusted
to handle potentially malicious data. Imagine if a "vulnerable" target
was actually a malicious honeypot, lying in wait for a poor Metasploit
user to come along and parse out its payload. (OT: does such a thing
have a clever name? If not, I propose "beehive" to imply the offensive
capabilities of such a honeypot.)
Nokogiri is used elsewhere in Metasploit, but those functions handle
data sourced from the Metasploit user herself, so those XML hunks are
nominally trustworthy.
This change updates msftidy to be run automatically for new modules
added since the last tag release because we can't rely on folks using
tools/dev/pre-commit-hook before submitting a PR. Now, when one attempts
to open a PR with a non-tidy'ed module, the build will fail out of the
gate.
Related to the 100s of msftidy errors extant today.
[SeeRM #8498]
commit c894e52de5705a1133191be5e9caf3ebdee33621
Author: Tod Beardsley <tod_beardsley@rapid7.com>
Date: Fri Jan 31 14:17:02 2014 -0600
Add a jacked up title to test travis. Revert this!
commit 2f00c190be71aeb456a7a546071286fd6d670bc1
Author: Tod Beardsley <tod_beardsley@rapid7.com>
Date: Fri Jan 31 11:39:42 2014 -0600
Allow for checking and spotchecking.
commit db11e8dfad5381030b08c431a183dbafe7a5f304
Author: Tod Beardsley <tod_beardsley@rapid7.com>
Date: Thu Jan 30 17:16:37 2014 -0600
Whoops, need to exit an Integer always.
commit 12d131d3157a78ff11e597476138323ed0a062fc
Author: Tod Beardsley <tod_beardsley@rapid7.com>
Date: Thu Jan 30 16:59:35 2014 -0600
Allow for exit statuses from msftidy.
commit 2c3b294ff17416f49935472caf2b6be3dbdd93a4
Author: Tod Beardsley <tod_beardsley@rapid7.com>
Date: Thu Jan 30 15:36:43 2014 -0600
Be more dynamic about tag checking years
commit d5d8a0b05ac17fb18666a9c252dbb6928d6b5e56
Author: Tod Beardsley <tod_beardsley@rapid7.com>
Date: Thu Jan 30 14:36:44 2014 -0600
Don't warn when there's really nothing
commit fb44a3142fb01eb2647c1c240bb1cc2e7bf59120
Author: Tod Beardsley <tod_beardsley@rapid7.com>
Date: Thu Jan 30 14:21:50 2014 -0600
Revert the intentional failure
This reverts commit 99a7630b0da301b27ac495cb027009a8cd9e2caf.
Fun fact: Reverting a commit does not automatically sign with my current
aliases, one must git revert then git c --amend.
commit 99a7630b0da301b27ac495cb027009a8cd9e2caf
Author: Tod Beardsley <tod_beardsley@rapid7.com>
Date: Thu Jan 30 14:08:05 2014 -0600
Cause an exit status in precommit check
Maybe travis will see these and fail the build.
Don't forget to revert this commit @todb-r7 !
commit 5a3b2fcd9598fae51a0dd2c7c87680c703a85448
Author: Tod Beardsley <tod_beardsley@rapid7.com>
Date: Thu Jan 30 13:11:04 2014 -0600
Update msftidy pre-commit-hook for spotchecking
commit 3f255e36dad9ed3081aaf359f845525d96872ef0
Author: Tod Beardsley <tod_beardsley@rapid7.com>
Date: Thu Jan 30 12:35:16 2014 -0600
Travis should run msftidy via precommit hook
commit 0959d9d2d281590a94c0ac960e43b74354e4e21b
Author: Tod Beardsley <tod_beardsley@rapid7.com>
Date: Thu Jan 30 12:25:53 2014 -0600
Add SPOTCHECK_RECENT to msftidy.rb
In case it takes too long to get a report, the method will give up
checking after one hour. The user can still manually check the report
from the analysis link given earlier.
[SeeRM #8733] This a tool that uses VirusTotal's public API to submit
a malware sample for analysis. As an offensive tool developer, this
would provide a convenient way to check and see how AVs react to
something we write.
Luke Imhoff
URI Assassin
William Vu
Tod Beardsley
Josh
Joshua Smith
Jon Hart
HD Moore
Christian Mehlmauer
sinn3r
Sagi Shahar
Rob Fuller
Joe Vennix
jvazquez-r7
Ethan Robish