7e5e0f7fc8
Retab lib
2013-08-30 16:28:33 -05:00
d341b825d0
Rename dirbust option to conform to style
2013-06-14 12:58:08 -05:00
b509ac8504
Crawler mixin: Dirbusting opt moved to advanced
2013-06-13 00:04:31 +03:00
b474cda4aa
Crawler/Anemone: Dirbusting now optional
...
[FIXRM #8030 ]
Anemone updated to make dirbusting optional (on by default) and the Crawler core
module updated to provide an option to do so.
2013-06-13 00:00:09 +03:00
6aa7c74fdd
make anemone also rspect domain
2013-06-07 14:24:14 -05:00
b2563dd6c2
trying to clean up the mess from the revert
2013-02-19 21:25:37 -06:00
8ddc19e842
Unmerge #1476 and #1444
...
In that order. #1476 was an attempt to salvage the functionality, but
sinn3r found some more bugs. So, undoing that, and undoing #1444 as
well.
First, do no harm. It's obvious we cannot be making sweeping changes in
libraries like this without a minimum of testing available. #1478 starts
to address that, by the way.
FixRM #7752
2013-02-11 20:49:55 -06:00
c71b803413
Add invisible auth to web crawler
...
the anemone web crawler now properly supports our invisible auth scheme
for rex http.
2013-02-04 14:38:08 -06:00
b53e8c794f
Fix indent level
2013-01-07 13:36:55 -06:00
3d4d6e9860
Crawler aux mixin updated to catch the mysterious and anonymous timeout exception and re-raise it as a Timeout::Error
2013-01-04 21:32:18 +02:00
d656e3185f
Mark all libraries as defaulting to 8-bit strings
2012-06-29 00:18:28 -05:00
9c887eb457
Fix displayed host name for IPv6 targets
2011-12-10 13:24:58 -06:00
79c2264263
more spaces at EOL and bad tabs
2011-11-20 12:05:14 +11:00
1bb9470e59
Adds the ability to set the cookies, authentication, and arbitrary headers
...
git-svn-id: file:///home/svn/framework3/trunk@13195 4d416f70-5f16-0410-b530-b9f4589650da
2011-07-16 13:29:28 +00:00
c5c679cdb7
Remove all calls to framework.db.sync and make this method a no-op now that the task manager is no longer used.
...
git-svn-id: file:///home/svn/framework3/trunk@12708 4d416f70-5f16-0410-b530-b9f4589650da
2011-05-25 00:30:03 +00:00
1904e446b2
Change this back to cooincide with autoload revert
...
git-svn-id: file:///home/svn/framework3/trunk@12602 4d416f70-5f16-0410-b530-b9f4589650da
2011-05-12 20:11:43 +00:00
95d802754b
Rework to avoid a warning
...
git-svn-id: file:///home/svn/framework3/trunk@12598 4d416f70-5f16-0410-b530-b9f4589650da
2011-05-12 19:26:11 +00:00
f2057f54a8
De-dup'ed get_link_filter() in the crawler module. Added some comment
...
docs on crawler_process_page(). Nothing exciting.
git-svn-id: file:///home/svn/framework3/trunk@12010 4d416f70-5f16-0410-b530-b9f4589650da
2011-03-17 21:21:21 +00:00
a8de0591b2
vhost should be used when it's available
...
git-svn-id: file:///home/svn/framework3/trunk@11761 4d416f70-5f16-0410-b530-b9f4589650da
2011-02-17 02:16:20 +00:00
070f48e16a
Work around some weirdness between report_website and report_host not accepting hostnames as :host by forcing :host to be the address
...
git-svn-id: file:///home/svn/framework3/trunk@11723 4d416f70-5f16-0410-b530-b9f4589650da
2011-02-08 17:32:37 +00:00
4c69fc7b84
Fix typo
...
git-svn-id: file:///home/svn/framework3/trunk@11028 4d416f70-5f16-0410-b530-b9f4589650da
2010-11-13 06:58:24 +00:00
dc5116f9df
Move the crawler mixin to an auxiliary
...
git-svn-id: file:///home/svn/framework3/trunk@11026 4d416f70-5f16-0410-b530-b9f4589650da
2010-11-13 06:55:19 +00:00
Tab Assassin
Tod Beardsley
Tasos Laskos
David Maloney
Tod Beardsley
sinn3r
HD Moore
HD Moore
James Lee
Jonathan Cran