MSP-11368
MSP-11143
Remove fastlib as it slows down the code loading process. From the
previous commit, the mean loading for
`METASPLOIT_FRAMEWORK_PROFILE=true msfconsole -q -x exit` was
27.9530±0.3485 seconds (N=10). The mean after removal of fastlib
was 17.9820±0.6497 seconds (N=10). This means an average 35.67%
reduction in boot time.
On August 15, shuckins-r7 merged the Metasploit 4.10.0 branch
(staging/electro-release) into master. Rather than merging with
history, he squashed all history into two commits (see
149c3ecc63 and
82760bf5b3).
We want to preserve history (for things like git blame, git log, etc.).
So on August 22, we reverted the commits above (see
19ba7772f3).
This merge commit merges the staging/electro-release branch
(62b81d6814) into master
(48f0743d1b). It ensures that any changes
committed to master since the original squashed merge are retained.
As a side effect, you may see this merge commit in history/blame for the
time period between August 15 and August 22.
This fixes a huge number of hard-to-detect runtime bugs
that occur when a default utf-8 string from one of these
libraries is passed into a method expecting ascii-8bit
MSP-10998
It's compacting of the version parts into a single float doesn't work
with APIMinor over 10, so replace with Gem::Version, which compares
parts correctly.
so we shouldn't have to load all of them to run this utility. The
overall goal of this PR is to narrow down what modules
(exploit/aux + payload + encoder + nop) you possibly need in order
to shave off loading time. By doing this, on my box this is 5-6
seconds faster than the original one.
I actually tried to avoid making too many changes in the library
(such as Module Manager), because we don't have test cases for them,
and we can't really afford to risk breaking it. I also developed
a test script to actually be able to test msfcli.
Instead of deleting all non-symbolics before the re-adding phase of
PayloadSet#recalculate, store a list of old module names, populate a
list of new ones during the re-adding phase, and finally remove any
non-symbolic module that was in the old list but wasn't in the new list.
Also includes a minor refactoring to make ModuleManager its own thing
instead of being an awkard subclass of ModuleSet. Now PayloadSet doesn't
need to know about the existence of framework.modules, which makes the
separation a little more natural.
[FixRM #7037]
[Fixes#38426061, #38097411]
Msf::Modules::Loader::Directory#read_module_content may calculate a non-existent
module_path that gets passed to File.open causing an Errno::ENOENT exception
to be raised when using the module cache with a module that has been
moved to a new path (as is the case that originally found this bug) or
deleted. Now, the exception is rescued and read_module_content returns
an empty string (''), which load_module detects with
module_content.empty? and returns earlier without attempting to module
eval the (empty) content.
As having Msf::Modules::Loader::Directory#read_module_content rescue the
exception, meant there was another place that needed to log and error
and store an error in Msf::ModuleManager#module_load_error_by_path, I
refactored the error reporting to call
Msf::Modules::Loader::Base#load_error, which handles writing to the log
and setting the Hash, so the error reporting is consistent across the
loaders.
The exception hierarchy was also refactored so that
namespace_module.metasploit_class now has an error raising counter-part:
namespace_module.metasploit_class! that can be used with
Msf::Modules::Loader::Base#load_error as it requires an exception, and
not just a string so the exception class, message, and backtrace can be
logged.
Msf::Modules::Loader::Archive#each_module_reference_name tried to check
the enabled types for the module_manager by accessing the
enabledment_by_type Hash, which is protected. Instead, it should use
the public type_enabled? method.
Add specs to test all of Msf::Modules::Loader::Archive while testing
each_module_reference_name. In order to properly test that modules
could be found in archives, I had to produce a fastlib archive, so there
is now a spec for FastLib.dump and FastLib.load. Some specs are marked
pending as I found a bug in FastLib, which has a work-around. The bug
is filed in PivotalTracker as
https://www.pivotaltracker.com/story/show/38730815 and the pending tests
include the URL also in their tags.
[Fixes#37630057]
Modules were always being detected as having file changes because the
parent_path directory, instead of the actual module_path, was being
passed to module_manager.file_changed?, which caused the modification
times to not match.
To ensure this change fixes the ambiguous module warnings, a full spec
for Msf::Core::Modules::Loader::Base has been written.
spec/msf has moved to spec/lib/msf to match conventional spec layout and
allow for the spec/support directory to not be confused as a lib
subdirectory being tested.
[#36737359]
The merging of reload_module and the various load_module methods
resulted in the module loading from disk, but because the Hash entry in
the module manager was not deleted before on_module_load was called, the
newly reloaded module was logged as an ambiguous module name instead of
a reload. In order to report the reload errors correctly, I determined
that module_load_error_by_reference_name should really be
module_load_error_by_path. I eliminated faild in favor of this new name
since failed was just calling the attribute and the attribute's name is
clearer about the format of the data.
Tested by run rexploit and then exiting over and over with
ms08_067_netapi. When I messed up the file so it couldn't load, by
adding `inclde Exploit` (note mispelling of `include`), it reported the
error to msfconsole. When I removed the bad line and added a puts
"RELOADING <n>", where I kept incrementing n and saving the file, the
new number appeared during each rexploit.
[#36737359]
Write the module_eval backtrace to the log, but only the error's class
and name to Msf::ModuleManager#module_load_error_by_reference_name as
the contents of the Hash are printed in the console, which should never
recieve backtraces.
[#36737359]
Refactor the behavior of loading symbolic modules from cache by renaming
methods so it's clearer what they do and ensure that cached modules from
Fastlibs and directories can both be loaded, which was not previously
possible since the demand_load_module only called load_module_from_file.
[Fixes#36737359]
Refactor Msf::ModuleManager into concerns so its easier to understand and
duplicate code can be made DRY. The refactoring also ensures that when
loading from directories, Fastlibs, or reloading, the wrapper module will
always be named so that activesupport/dependencies will function.
Luke Imhoff
Brandon Turner
HD Moore
Samuel Huckins
Christian Mehlmauer
Meatballs
Tab Assassin
sinn3r
James Lee
sinn3r
Tod Beardsley