MSP-11368
MSP-11143
Remove fastlib as it slows down the code loading process. From the
previous commit, the mean loading for
`METASPLOIT_FRAMEWORK_PROFILE=true msfconsole -q -x exit` was
27.9530±0.3485 seconds (N=10). The mean after removal of fastlib
was 17.9820±0.6497 seconds (N=10). This means an average 35.67%
reduction in boot time.
This fixes a huge number of hard-to-detect runtime bugs
that occur when a default utf-8 string from one of these
libraries is passed into a method expecting ascii-8bit
so we shouldn't have to load all of them to run this utility. The
overall goal of this PR is to narrow down what modules
(exploit/aux + payload + encoder + nop) you possibly need in order
to shave off loading time. By doing this, on my box this is 5-6
seconds faster than the original one.
I actually tried to avoid making too many changes in the library
(such as Module Manager), because we don't have test cases for them,
and we can't really afford to risk breaking it. I also developed
a test script to actually be able to test msfcli.
[#47720609]
Msf::PayloadSet#add_module does NOT return an annotated module class as
Msf::ModuleSet#add_module does because a payload module is defined as a
ruby Module instead of a ruby Class. Since add_module doesn't always
return an annotated_class, the logic in
Msf::ModuleManager#on_module_load needed to change to NOT use
annotated_class and create #add_module as return [void]. Thus, it is
necessary to pass in all the metasploit module metadata to
Msf::ModuleManager#cache_in_memory instead of assuming they can be
derived from the (payload) Module or (other) Class.
[#47720609]
Msf::ModuleManager#module_info_by_path was not being updated when a
module was loaded, so if a load_module was called again, say during
start up of prosvc, the module would reload even though there was no
change in the file because file_changed? couldn't find an entry for the
module's path in module_info_by_path.
[#47720609]
Fix some docs and variable names to make it clearer when methods are
expecting module instance and module classes. Change some 'name'
variables to 'reference_name' since that's the proper terminology.
Instead of deleting all non-symbolics before the re-adding phase of
PayloadSet#recalculate, store a list of old module names, populate a
list of new ones during the re-adding phase, and finally remove any
non-symbolic module that was in the old list but wasn't in the new list.
Also includes a minor refactoring to make ModuleManager its own thing
instead of being an awkard subclass of ModuleSet. Now PayloadSet doesn't
need to know about the existence of framework.modules, which makes the
separation a little more natural.
[FixRM #7037]
Just had a brain fart when converting the hash key names and translated
:mtype to :modification_time instead of the correct :type. Correct key
names are in
Msf::ModuleManager::Cache#module_info_by_path_from_database!.
[Fixes#37630057]
Modules were always being detected as having file changes because the
parent_path directory, instead of the actual module_path, was being
passed to module_manager.file_changed?, which caused the modification
times to not match.
To ensure this change fixes the ambiguous module warnings, a full spec
for Msf::Core::Modules::Loader::Base has been written.
spec/msf has moved to spec/lib/msf to match conventional spec layout and
allow for the spec/support directory to not be confused as a lib
subdirectory being tested.
I missed a spot where I referenced the nested_paths as nested_pathnams
after I renamed the variable. Now, Msf::ModuleManager#add_module_paths
has rspec tests.
Rspec can be invoked with `rake` as the default task or `rake spec`
explicitly.
I changed RuntimeError to ArgumentError since that error was more
specific to having a bad argument error. I adding missing dependencies
to the Gemfile and a require to msf/core/db_manager.rb where it errored
out trying to access Msf::Config when I just did require 'msf/core' in
the spec.
[#36737359]
The merging of reload_module and the various load_module methods
resulted in the module loading from disk, but because the Hash entry in
the module manager was not deleted before on_module_load was called, the
newly reloaded module was logged as an ambiguous module name instead of
a reload. In order to report the reload errors correctly, I determined
that module_load_error_by_reference_name should really be
module_load_error_by_path. I eliminated faild in favor of this new name
since failed was just calling the attribute and the attribute's name is
clearer about the format of the data.
Tested by run rexploit and then exiting over and over with
ms08_067_netapi. When I messed up the file so it couldn't load, by
adding `inclde Exploit` (note mispelling of `include`), it reported the
error to msfconsole. When I removed the bad line and added a puts
"RELOADING <n>", where I kept incrementing n and saving the file, the
new number appeared during each rexploit.
[#36737359]
Refactor the behavior of loading symbolic modules from cache by renaming
methods so it's clearer what they do and ensure that cached modules from
Fastlibs and directories can both be loaded, which was not previously
possible since the demand_load_module only called load_module_from_file.
[#36737359]
On Linux, some of the unqualified constants that resolve on Mac OS X,
don't resolve, so to prevent errors (and because I can't justify why the
unqualified constants should resolve on OS X), I'm qualifying all the
Msf constants that are referenced in the code I've refactored.
[#36737359]
Fix the YARD docs to document the return values and make them consistent
with the modules being called. Ensure the force flag is passed as an
option to load_modules instead of a positional argument.
[#36737359]
When starting msfconsole, 'bundler/setup' is not required, the
'msf/env/gemcache' is required instead. Unlike 'bundler/setup' the
msf/env gemcache does not do the automatic requires for gems in the
cache, so explicit requires on 'active_support/concern' is needed to get
ActiveSupport::Concern defined. (I could have done require
'active_support' to match the behavior of 'bundler/setup', but a smaller
require seemed more appropriate.
I wasn't using any the features of ActiveSupport::Concern in
Msf::ModuleManager::Reloading, so remove the extend and just include it
as a regular module.
[Fixes#36737359]
Refactor Msf::ModuleManager into concerns so its easier to understand and
duplicate code can be made DRY. The refactoring also ensures that when
loading from directories, Fastlibs, or reloading, the wrapper module will
always be named so that activesupport/dependencies will function.
Luke Imhoff
HD Moore
Tab Assassin
sinn3r
James Lee
Tod Beardsley