Commit Graph

607 Commits (42a59189f39cf62e275a8057643fdf183fe575a9)

Author SHA1 Message Date
LuitelSamikshya 944d24a252 mics changes 2022-10-13 09:30:25 -05:00
LuitelSamikshya cb0da81a14 ratelimit library 2022-10-12 22:04:37 -05:00
Ice3man 09ceb29ba3 Fixed build error on 32bit arch 2022-10-08 01:55:18 +05:30
Ice3man 6c93d99745 Fixed failing lint tests 2022-10-08 00:10:32 +05:30
Ice3man 4f37c0b1f5 Added pagination support to result retrieval 2022-09-29 04:52:03 +05:30
Mzack9999 18f14b631c
Adding same host redirect support (#2655)
* simplifying test syntax

* adding same host redirect + refactoring redirect handling

* adding missing file

* adding support for template syntax

* adding integration test

* updating options

* fixing issue on same host redirect
2022-09-29 04:11:28 +05:30
Ice3man fc27fc94a5
Added default config generation for reporting options (#2605) 2022-09-27 02:40:34 +05:30
Sami 512eab7b21
tl flag improvements (#2596)
* tl flag improvements

* tl flag enhancement with additional filters

* added ExcluedTags filter

* tl flas to list template paths

* using stdout

* misc update

Co-authored-by: sandeep <sandeep@projectdiscovery.io>
2022-09-21 03:03:57 +05:30
Mzack9999 99c14f4c9c
implementation of rate limiter with bucket refill and unrestricted token burst (#2536)
* implementation of rate limiter with interval burst

* fixing import path

* fixing syntax

* adding tests

* fixing lint errors

* adding support for context

* moving rate limiter earlier to avoid hitting timeout
2022-09-19 17:09:28 +05:30
mzack 37c016a5f0 fixing lint errors 2022-09-19 08:38:52 +02:00
Ice3man 04b47b0309
Added custom json-unmarshaller + misc updates (#2556)
* Added custom json-unmarshaller + misc updates

* Added support for nuclei-cloud based scan execution

* Removed unnecessary files

* Misc

* Changes as per review comments

* misc option update

Co-authored-by: sandeep <sandeep@projectdiscovery.io>
2022-09-19 01:13:59 +05:30
forgedhallpass fc0763641f
New dsl functions (#2545)
* Update GO version to 1.18

* Removed redundant entry from the .gitignore file

* Added new DSL functions

to_unix_time(input string, optionalLayout string) int64
hex_to_dec(input string) float64
oct_to_dec(input string|number) float64
bin_to_dec(intput string|number) float64

* Notify if debug is enabled when a proxy cannot be validated

* Documentation: Go version requirement updated to 1.18

* test fix: Timezone agnostic date expectation in the assertion

* code review: extracted the default date-time layouts into a global variable
2022-09-07 00:44:29 +05:30
Ice3man 466176e9e8
Merge pull request #2500 from projectdiscovery/goflags-update
Updated goflags to latest + misc
2022-08-30 11:52:25 +05:30
Sandeep Singh e329428684
fix: make headless template validation optional (#2507) 2022-08-29 03:00:58 +05:30
Ice3man 8892250583 Updated goflags to latest + misc 2022-08-27 19:35:17 +05:30
Mzack9999 30054d1fb6
Adding advanced template filtering (#2374)
* Adding advanced template filtering

* fixing bug in slice

* refactoring tests

* adding test cases

* increasing error verbosity

* fixing quoted fields with spaces

* adding more test cases

* fixing merge error

* fixing lint errors

* switching to []string

* updating tag filter tests

* updating functional tests

* fixing functional test cases

* updating syntax
2022-08-25 16:52:08 +05:30
Sami 0aac36a44b
added custom config flag (#2399)
* added custom config flag

* config.yaml file in custom  directory

* lint error fix

* few updates and error checks

* fix lint error

* copy config.yaml file if the dest folder does not exist

* lint error check

* added integration test

* improved test cases

* lint error fix
2022-08-25 16:10:07 +05:30
Ice3man 7b7936b7a5
Added show-actions flag to display headless actions (#2456)
* Added show-actions flag to display headless actions

* misc update

* readme update

Co-authored-by: sandeep <sandeep@projectdiscovery.io>
2022-08-25 10:43:32 +05:30
Ice3man 77c81834b2
Fixed loader templateConfig nil pointer crash (#2486) 2022-08-24 19:32:56 +05:30
Ice3man 67d5769cd9
Added initial catalog interface implementation (#2318)
* Added initial catalog interface implementation

* Added OpenFile to Catalog + disk catalog implementation

* Fixed merge issues

Co-authored-by: sandeep <sandeep@projectdiscovery.io>
2022-08-10 11:05:58 -07:00
Mzack9999 b942ddc6ad
Fixing map race condition (#2340) 2022-07-26 18:30:15 +05:30
Mike Rheinheimer 9efba05e0c
expose hosterrorscache.Cache as an interface (#2291)
* expose hosterrorscache as an interface, change signature to capture the error reason

* use the hosterrorscache.CacheInterface as struct field so users of Nuclei embedded can provide their own cache implementation

Co-authored-by: Mike Rheinheimer <mrheinheimer@atlassian.com>
2022-07-19 02:05:53 +05:30
invist db727db006
Optionally disable templates syntax strict check (#2266)
* nuclei::templates|define strict option (default)

* renaming flag and internal variable

* misc flag update

Co-authored-by: c-f <you@example.com>
Co-authored-by: mzack <marco.rivoli.nvh@gmail.com>
Co-authored-by: sandeep <sandeep@projectdiscovery.io>
2022-07-13 17:00:11 +05:30
Sami 6c2fdd3387
Issue 2227 ntv flag run new templates added in specific version (#2275)
* ntv flag to run templates added in specified version

* added missing arguments

* misc update

* added functional test and err check

* updated the min version

Co-authored-by: sandeep <sandeep@projectdiscovery.io>
2022-07-13 16:49:06 +05:30
Sajad be73fde0db
use go-homedir instead of standard os.userhomedir (#2262)
* use go-homedir instead of standard os.userhomedir

* set r.templatesConfig before write attempt to avoid panic
2022-07-13 13:33:13 +05:30
Mzack9999 476773ff8c
Replacing hasstdin with helper library (#2191)
* Replacing hasstdin with helper library

* adding timeout reader on stdin

* adding large input read timeout

* reducing stdin timeout + nostdin flag

* go mod update

* readme update

* go mod tidy

Co-authored-by: sandeep <sandeep@projectdiscovery.io>
2022-06-30 17:51:54 +05:30
Ice3man 7875b06fc8
Added exclude-matchers support for template & matchers (#2218)
* Added exclude-matchers support for template & matchers

* Fixed panics due to typo

* Added support for only template ID + misc cleanup
2022-06-24 23:09:27 +05:30
Mzack9999 0b351e83f3
Add self diagnostic functionality (#2178)
* Adding Self-Diagnostic

* adding comment
2022-06-22 00:40:10 +05:30
Ice3man 3648c47e35
Fixed template validation edge cases (#2051) 2022-05-25 11:26:05 +05:30
Mzack9999 184f3fdc30
Fixing concurrent read/write map (#1989) 2022-05-17 14:38:48 +05:30
Mzack9999 a534b9f06c
Fixing directory walk error check on windows (#1951)
* Fixing directory walk error check on windows

* moving check to helper package

* replacing godirwalk with standard library
2022-05-08 12:22:21 +05:30
LuitelSamikshya 3ccbfe4626 disable redirects flag 2022-04-27 11:19:44 -05:00
sandeep 1b953655d1 misc updates 2022-04-19 16:14:49 +05:30
LuitelSamikshya 5eae7159c7 nil value check 2022-04-18 17:21:33 -05:00
Sami ce79a8dc57
Template folder exists changes (#1825) 2022-04-11 11:59:22 +05:30
Sandeep Singh c5712c98f4
Fixed the default behavior of srd flag (#1817)
* fix: fixed automatic storing of debug response

* small code refactor

Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
2022-04-06 16:49:36 +05:30
Mzack9999 0bab297328
improving default template path logic (#1806) 2022-04-05 04:03:49 +05:30
Sami 301307bb77
Issue 1705 save responses on disk (#1727)
* save response on disk

* lint error check

* store raw request/response

* lint error fix

* file path

* mock test fix

* readme update

* .txt extension

Co-authored-by: sandeep <sandeep@projectdiscovery.io>
2022-04-02 00:59:02 +05:30
Sandeep Singh 8ee4f9f797
Merge branch 'dev' into proxy-internal-requests 2022-03-23 23:12:51 +05:30
mzack e9125f8fca Fix for nil unexpected return value in workflow execution 2022-03-21 11:33:10 +01:00
sandeep 49c5cca78b Merge branch 'dev' of https://github.com/projectdiscovery/nuclei into proxy-internal-requests 2022-03-21 13:03:17 +05:30
mzack 6e5e7f23e5 Implements github request with token fallback 2022-03-17 15:57:35 +01:00
Ice3man 660a81c1ab Fixed crash + added unique tags for execution 2022-03-14 16:25:27 +05:30
Ice3man 32bc6c1718 Misc changes, moving things around as per review 2022-03-14 12:32:05 +05:30
Ice3man 14dab085ac Merge branch 'dev' of https://github.com/projectdiscovery/nuclei into automatic-workflows 2022-03-14 12:17:39 +05:30
mzack 3f5c055aca adding flag check 2022-03-09 12:38:26 +01:00
mzack 6f5489b8b6 restoring code for catch all cases 2022-03-09 12:37:03 +01:00
mzack d2b5579303 Adding proxy support for internal standard HTTP requests 2022-03-09 12:31:12 +01:00
sandeep 8096737e1a Merge branch 'dev' of https://github.com/projectdiscovery/nuclei into update-custom-dir 2022-03-09 01:52:08 +05:30
sandeep 78bb099412 Update runner.go 2022-03-08 10:11:06 +05:30
Sandeep Singh ce2a5c3d81
Merge branch 'dev' into issue-1594-regexp 2022-03-08 10:05:06 +05:30
Ice3man d9a121344c Merge branch 'dev' of https://github.com/projectdiscovery/nuclei into automatic-workflows 2022-03-07 13:19:37 +05:30
Ice3man 88c5ca68eb Ignore error on pprof server close 2022-03-07 10:28:25 +05:30
Ice3man f6dc6264dd Added pprof-server support with ena
ble-pprof flag
2022-03-03 19:44:29 +05:30
mzack 64bea4da8c Merge branch 'dev' into issue-1594-regexp 2022-03-02 17:28:04 +01:00
mzack 630bf44c8b adding gh token 2022-03-02 17:02:51 +01:00
mzack 6e9ea761a7 Adding GH token detection 2022-03-02 16:56:32 +01:00
Sajad Parra bdc1b337e0 fix setting custom nuclei-templates directory #1611 2022-03-02 16:50:20 +05:30
mzack 6cb522c9b2 Merge branch 'dev' into issue-1594-regexp 2022-02-28 11:42:46 +01:00
mzack 06ab0dfa99 making code more uniform with chosen architecture 2022-02-26 09:06:43 +01:00
Ice3man fb6ab39d65 Added interaction event highlighting support in debug mode 2022-02-25 23:11:56 +05:30
mzack e8780b7db1 Merge branch 'dev' into issue-1594-regexp 2022-02-25 13:04:46 +01:00
mzack 2409d7a713 dropping multiple scan 2022-02-24 22:41:33 +01:00
Ice3man 42b3a5d3e2 Allow random resume file generation on conflict + made resume string flag 2022-02-24 14:32:41 +05:30
Sandeep Singh bf7a9aa708
Merge branch 'dev' into proxy-to-stringslice 2022-02-24 01:43:53 +05:30
Ice3man 010bab5aaf Fixed failing tests due to missing field declaration 2022-02-23 21:45:55 +05:30
mzack 6746071979 Refactoring file templates to handle large files in chunks + removing deprecated io methods 2022-02-23 13:54:46 +01:00
Sajad 94d8c1d8e8
remove new-addtions file creation (#1624) 2022-02-22 13:10:57 +05:30
Mzack9999 0732406a68
Uniform behavior for input CLI flags (#1569)
* Adding file normalized string slice support

* updating goflags
2022-02-09 10:16:17 +05:30
mzack 5889a39fb4 Disable internal standard library logger 2022-02-01 14:46:40 +01:00
Ice3man 8dfa9cee39
Added debug-req/resp support for interactsh interactions (#1491)
* Added debug-req/resp support for interactsh interactions

* Added format function for interact debug logs + misc fixes

* Added function for interact debug header

* Typo fix

* Enable debug logging for req/resp debug flag
2022-01-27 12:14:32 +05:30
Ice3man be75480256 Added initial smart workflow implementation 2022-01-18 20:59:37 +05:30
Ice3man bd7f32d92f Fixed stats calculation for payloads in http base request 2022-01-17 03:33:10 +05:30
mzack 11df6da209 Excluding non yaml file from new additions loading 2022-01-14 11:20:19 +01:00
Ice3man 24326389a4
Added interactsh additional variables support (#1468)
* Added interactsh additional variables support

* Added support for interact server rotation

* Misc fixes to interact related modules

* Fixed a crash with interactsh

* Perform deduplication of output extract data before printing
2022-01-13 13:22:43 +05:30
Mzack9999 39519c01a6
Adding support for template id based execution (#1448) 2022-01-07 17:30:20 +05:30
Sandeep Singh 88bb6cb046
Merge pull request #1308 from projectdiscovery/issue-614-scan-resume
Adding stop-resume support
2022-01-03 15:07:30 +05:30
mzack 1f878192ec Merge branch 'dev' into issue-614-scan-resume 2021-12-23 13:15:47 +01:00
sandeep 50bad75cb4 Merge branch 'dev' of https://github.com/projectdiscovery/nuclei into interactsh-first-match-stop 2021-12-21 15:46:37 +05:30
Sajad Parra 911045ae9a add stop at first match for interactsh matchers 2021-12-21 14:20:03 +05:30
mzack 0a9f890fdd Removing invalid negative paths for windows 2021-12-20 14:11:55 +01:00
mzack 63b194c10a refactoring stop-resume logic with better tracking 2021-12-16 22:32:03 +01:00
Ice3man fea3fabdf2
Misc changes to update logic (#1212)
* Misc changes to update logic

* Misc adjustments to update logic

* update: build check

* update: revert test update

Co-authored-by: sandeep <sandeep@projectdiscovery.io>
2021-12-16 13:59:19 +05:30
mzack e279e0437d moving resume file to config folder 2021-12-10 12:45:29 +01:00
mzack 2f4b131f26 fixing resume skipping on-going scans 2021-12-10 12:27:34 +01:00
mzack9999 edc6e5f9b3 fix cross-os chunks calculation 2021-12-06 18:56:52 +01:00
Mzack9999 40d01412cb Fixing templates bundle unzip on windows 2021-12-06 18:12:39 +01:00
Sandeep Singh df55f7a2eb
Disabling no-sandbox in headless engine (#1135)
* Disabling no-sandbox in headless engine

* limiting disabling sandbox to bare minimum

* adding warnings related to linux os and root user requirement

Co-authored-by: mzack <marco.rivoli.nvh@gmail.com>
2021-12-05 20:14:16 +05:30
Mzack9999 3c88afac0c
Fixing payloads path during validation (#1320)
* Fixing payloads path during validation

* Added GH Action for public template parsing / validation

* tracking payload errors as syntax warnings

* improving path parsing + introducing hard failure for runtime errors on validation

Co-authored-by: sandeep <sandeep@projectdiscovery.io>
2021-12-05 19:41:14 +05:30
Sandeep Singh a0da7452e3
Merge pull request #1315 from projectdiscovery/validate-flag
validate flag updates
2021-12-02 17:23:34 +05:30
LuitelSamikshya 6d5146e540 validate flag updates 2021-12-01 10:35:18 -06:00
Ice3man d6cfa556b8
Merge pull request #1306 from projectdiscovery/fix-stdin-reading
fix: #1274 spawned nuclei child process hangs reading stdin
2021-11-30 00:28:00 +05:30
mzack 2140ffa382 Adding stop-resume support 2021-11-29 14:38:45 +01:00
Ice3man543 c9943c0b2a fix: #1274 spawned nuclei child process hangs reading stdin
Next and final attempt at fixing the stdin parsing issue when spawing as child from nodejs
2021-11-29 14:38:55 +05:30
forgedhallpass 7e22d70ded refactor/documentation: typos and grammatical errors 2021-11-25 18:54:16 +02:00
forgedhallpass ec6889931d refactor: linter driven fixes
* x = x + ""  => x += ""
* pre-allocating slice with known size
* added t.Helper() methods in test helpers
* complex if-else conditions replaced by switches
* errors should be checked using error.Is() instead of ==
* function parameter should start with lower case letter
* removed unnecessary type definition
* variable/label naming convention: camelCase instead of snake_case
2021-11-25 17:57:22 +02:00
forgedhallpass 3fd1f57b96 refactor: godoc and comment uniformization
Adding space after // and before the godoc/comment
2021-11-25 17:03:56 +02:00
forgedhallpass fdd22ab668 refactor: Wrap errors using %w in fmt.Errorf
see:
 * https://github.com/xxpxxxxp/intellij-plugin-golangci-lint/blob/master/explanation/goerr113.md
 * https://go.dev/blog/go1.13-errors#wrapping-errors-with-w
2021-11-25 16:24:37 +02:00
forgedhallpass f9c214a66f refactor: update logic refactor to make it more testable
Introduced logic to test for zip slip (path traversal)
2021-11-25 16:24:28 +02:00
forgedhallpass 4bccb6cf8a fix: File and directory creation permission changes
Directories: 0755 (5 - group and other can read cd into the directory and read it's content)
Files: 0644 (4 - group and other can only read the created files)
Tests files: 0777
2021-11-25 14:37:57 +02:00
Ice3man 1581c96e4e
Added matched-status flag + template-path and url to output (#1272)
* Added matched-status flag + template-path and url to output
2021-11-22 17:53:25 +05:30
Mzack9999 ca9676f52e
Adding support for clustering within workflow (#1255)
* Adding support for clustering within workflow
2021-11-22 04:49:53 +05:30
Sajad f74ff3fc49
Tag based struct validation (#1256)
* Added tag based struct validation
2021-11-20 13:25:27 +05:30
Ice3man 50a816fce9
Fixed a crash with uninitialized interactsh client (#1251)
* Fixed a crash with uninitialized interactsh client
2021-11-16 20:02:39 +05:30
Ice3man543 c1a35b3ff9 Merge branch 'dev' of https://github.com/projectdiscovery/nuclei into more-protocols 2021-11-11 17:21:25 +05:30
LuitelSamikshya 0e46d3e041
feat: Checking socks5 proxy before launching a scan #1001 (#1225)
* Proxy validation and list input support

Co-authored-by: Sajad Parra <parrasajad@gmail.com>
Co-authored-by: sandeep <sandeep@projectdiscovery.io>
2021-11-10 21:30:03 +05:30
Ice3man543 213853c45d Merge from dev 2021-11-08 15:40:18 +05:30
Sandeep Singh 3c53e262e3
Merge pull request #1219 from projectdiscovery/uniq-interact
Unique Interactsh URL
2021-11-06 03:33:09 +05:30
Sajad Parra 4d8eaad0a3 add unit test for unique interactsh url #1068 2021-11-05 15:27:49 +05:30
Ice3man543 8ad3ebcd05 Made code changes as per review comments 2021-11-05 03:01:41 +05:30
Ice3man543 390ca8b3c6 Merge from dev 2021-11-03 18:58:00 +05:30
Sandeep Singh 0f897a3977
Merge pull request #1186 from projectdiscovery/protocol-types
Added new type and exclude-type flag
2021-11-03 18:47:26 +05:30
Ice3man543 47949c0b52 Use separate type enum for protocol types 2021-11-03 17:18:35 +05:30
Ice3man 09cad2557c
Merge pull request #1123 from EndPositive/remote-template-workflow-lists
Remote template workflow lists
2021-11-02 16:12:29 +05:30
Ice3man 331ae0f03b
Merge pull request #1171 from kchason/client-cert-auth
Add Support for Client Certificate Authentication
2021-10-31 02:08:18 +05:30
Alexey Zhuchkov 3f1186da2b Add error log support 2021-10-30 14:12:21 +03:00
Ice3man543 1ca2cf3bea Misc 2021-10-28 23:17:05 +05:30
Ice3man543 0abc7202b1 Misc fixes with goflags 2021-10-28 17:45:38 +05:30
Ice3man543 d124dbacc7 Moved all important execution stuff to engine 2021-10-28 17:20:07 +05:30
kchason 4a1440a17b Merge dev branch 2021-10-27 12:13:23 -04:00
Ice3man543 c16c93fe7c refactor the modules to core 2021-10-27 16:50:36 +05:30
Ice3man543 97645dde52 Added new workpool package + Misc refactor 2021-10-27 15:53:04 +05:30
Jop Zitman 86cf09fa3f Merge branch 'dev' into remote-template-workflow-lists
# Conflicts:
#	v2/cmd/nuclei/main.go
2021-10-26 15:33:39 +02:00
Ice3man543 6541b04f4c Added new type and exclude-type flag 2021-10-25 23:24:42 +05:30
Ice3man543 ca6cc51302 Color change for warning disclaimer 2021-10-25 17:47:39 +05:30
kchason 4b70560cc9 Merge branch 'master' into client-cert-auth 2021-10-21 13:59:39 -04:00
sandeep fa79a16a9d reverting #1017 2021-10-21 05:56:15 +05:30
kchason 9c77f15012 Argument checks for presence and validity 2021-10-20 11:32:26 -04:00
Ice3man543 de01158556 Removed non-used code 2021-10-19 21:52:40 +05:30
Ice3man543 8c9d9a77bf Added self-contained to network protocol 2021-10-17 18:50:07 +05:30
Ice3man543 a820594506 Merge branch 'dev' of https://github.com/projectdiscovery/nuclei into self-contained-templates 2021-10-17 10:25:25 +05:30
Sandeep Singh 680a2ad76b
Merge branch 'dev' into feature-preserve-input-order 2021-10-16 18:47:26 +05:30
Jop Zitman 14bb1b7b21 Implement `-template-url` and `-workflow-url` for retrieving lists of templates/workflows to run. 2021-10-14 23:33:08 +02:00
Ice3man543 736a530bde Added self-contained http requests implementation 2021-10-14 22:26:01 +05:30
mzack 30558eabf4 Merge branch 'dev' into feature-preserve-input-order 2021-10-12 09:42:23 +02:00
forgedhallpass 8392143944 Merge remote-tracking branch 'origin/dev' into colorize_responses 2021-10-11 14:00:53 +03:00
Ice3man e79c6262b9
Merge branch 'dev' into cli-variables-as-payload 2021-10-09 19:57:48 +05:30
Alexey Zhuchkov 3e8a0af36f Add exclude severity filter 2021-10-08 22:27:27 +03:00
forgedhallpass f96168fc2c [feature] Add coloring to debug information #999
* Reverted the bug introduced by mistake (checking file exists using os.IsExist vs !os.IsNotExist)
2021-10-08 20:17:37 +03:00
forgedhallpass 0a9704ebd8 [feature] Add coloring to debug information #999
* code-review fixes
2021-10-07 20:54:12 +03:00
mzack fc31688f69 fixing lint errors 2021-10-07 12:40:18 +02:00
mzack 12b6b2ca89 Add support for CLI payload variables 2021-10-07 12:36:27 +02:00
forgedhallpass 742a5c44fc Merge remote-tracking branch 'origin/dev' into colorize_responses 2021-10-05 13:20:21 +03:00
forgedhallpass d10dfb550e Implicitly set the headless flag if template validation was requested, in order to correctly validate headless templates instead of complaining about "cannot create template executer" 2021-09-30 19:07:59 +03:00
forgedhallpass 8fdc9a7164 Minor optimization: short-circuit the updateTemplates logic if template update was not requested 2021-09-29 19:44:37 +03:00
mzack fc868aba4a misc 2021-09-26 22:36:44 +02:00
mzack 71e4396bd4 Adding support to preserve input order 2021-09-26 22:33:12 +02:00
sandeep 76b86ba2a0 disk cleanups 2021-09-19 16:26:47 +05:30
forgedhallpass 2baf695709 Merge remote-tracking branch 'origin/dev' into code_smells 2021-09-16 20:00:21 +03:00
Sandeep Singh 4845b85328
Merge pull request #1026 from projectdiscovery/github-rate-limit-fix
GitHub rate limit bug fix
2021-09-16 21:27:46 +05:30