Commit Graph

171 Commits (dev)

Author SHA1 Message Date
Dogan Can Bakir a4ca2021cd
Add headless header and status matchers (#3794)
* add headless header and status matchers

* rename headers as header

* add integration test for header+status

* fix typo
2023-06-09 15:03:03 +05:30
Shubham Rasal a34b94e62f
Issue 3339 headless fuzz (#3790)
* Basic headless fuzzing

* Remove debug statements

* Add integration tests

* Update template

* Fix recognize payload value in matcher

* Update tempalte

* use req.SetURL()

---------

Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
2023-06-09 05:50:44 +05:30
Mzack9999 0d2d510689
Adding support for constants (#3692)
* adding support for constants

* fixing typo

* adding integration test

* fixing lint issues

* fixing template syntax
2023-05-25 22:02:35 +05:30
Shubham Rasal 9c2fa8f9c4
Add payload in dns protocol (#3632)
* add execute function in dns

* Add payload in dns protocol

* Add integration test to cover dns payload

- also check command line overriding a payload variable

* Update matchedAt and remove trailing dot

* Consider payload data for request count

- Update verbose output to print question
- Update dns requests Requests function to consider payload data

* update gitignore

* bump nuclei version to v2.9.4-dev

---------

Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
2023-05-11 03:26:29 +05:30
Shubham Rasal 449afc0c5c
Issue 3564 var override (#3599)
* Check if the variables are override by other means

- you can override the template variable value using command line flags

* Update lazy eval logic

- previously, we were checking any function/expression in variable
- now, update the logic, lazy eval only if variable contains any
  protocol variable(global)

* add integration tests

* Add test to check the dsl function working in variable

* gather all generate variables logic in utils

* go mod update

* Refactor the generate variables function

* go mod update+ fix typo

---------

Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
2023-05-02 23:49:56 +05:30
Shubham Rasal f640187709
Expose DNS fields for matchers and extractors (#3613)
* Extend dns extractor to dns answer records

* add test template

* Ignore error for dns variables are not found

* Add all the records of answer section

* Fixed the wrong typecasting
2023-05-02 17:13:11 +05:30
Shubham Rasal 6ebb8e98f4
Fix wrong template loading in dev branch (#3629)
* Templates wrong loading

* Add tests to cover following scenarios

- check optional fields only if template loaded
- it should return warning only if template is loaded
2023-05-02 15:12:55 +05:30
Tarun Koyalwar 7f5e4e2336
aws signer: fix missing x-content-sha256 header (#3601)
* fix missing x-content-sha256 header

* fix variable priority in self-contained templates

* remove debug statement

* adds generic raw request parser for self-contained req

* more integration tests

* bug fix: 10x faster race requests

* fix failing integration test
2023-05-01 12:15:35 +05:30
Tarun Koyalwar 4e6ef4490e
duplicated params in self contained requests (#3608)
* fix duplicated params in self-contained+ export extracted values to file

* add integration tests + fix percentage overflow in pb

* fix integration test template id

* integration test: validate if file exists
2023-04-26 12:35:07 +05:30
Mzack9999 6f4b1ae48a
Replacing ccache with generic gcache (#3523)
* Replacing ccache with generic gcache

* fixing lint issues

* removing unecessary hashing + using errorutils

* making test more tolerant

* removing dead code + refactor

* removing redundant code

* removing race

* maint

* moving code

* adding more iterations

* note + typo

* temporary fixing stop-at-first-match with interact

* wrapping internal map with mux

* sort before running integration test

* fix deadlock in requestShouldStopAtFirstMatch

* add timeout to integration_test workflow

* attempting to remove outer lock

* adds interactsh protocol tests in integration_test

---------

Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
2023-04-16 23:19:35 +05:30
Shubham Rasal 45cc676f96
Evaluate payload variables (#3503)
* Evaluate payload variables

* Add variables evaluation

* Extend variables test

- to check evaluation of global variables in variables
- to check evaluation of golbal variables in payload

* Add default and cli variables to websocket, whois and dns proto

- use url.Parse with urlutil.Parse
2023-04-12 01:50:58 +05:30
Ramana Reddy c9634fae72
Issue 3350 matcher condition or not work (#3397)
* fix or condition match even interactsh includes as matcher-part (#3350)

* add integration test

* add new template to integration test

* matcher-condtion: test case for both conditions

* fix lint errors

* upgrade dependencies

---------

Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
2023-03-15 20:45:44 +05:30
Shubham Rasal 572c8eb780
Issue 2987 fuzz options (#3355)
* Add override fuzzing type and mode flags

* Update english readme

* Fix failing tests

* Add the integration tests

- validate the command line overriding type and mode for fuzzing
2023-03-06 16:56:38 +05:30
Tarun Koyalwar d9e953acfa
fix file input in custom vars for self contained http template (#3385)
* fix file input in variables(-V)

* fix lint error

* fix nuclei-ignore file failures
2023-03-04 04:57:27 +05:30
xm1k3 f26e01551e
resolve() helper function (#3321)
* started the implementation of resolve helper function

* fixes go mod and sum

* fixes and use makeDslWithOptionalArgsFunction signature

* added tests

* added more dnsTypes based on dnsx docs

* used dns client pool

* dsl functions

* fixes on dnsclientpool, added init()

* go mod tidy

* go mod tidy

* dsl signature makeMultiSignatureDslFunction

* error on len

* managed mx

* fix on mx record

* dns types managed with FirstNonZero func

* error handling

* utils to stable version

* version bump

* fixing var name

---------

Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
2023-03-04 04:21:24 +05:30
Mzack9999 d80cbef51d
Merge pull request #3333 from CodFrm/main
fix some json deserialization issues
2023-02-28 09:02:04 +01:00
Tarun Koyalwar 21b03a2e8a
bug fix in url path and adds integration tests (#3331)
* fix unsafe edgecases+ adds integration test

* bug fixes and more url testcases

* upgrade cfssl

* fix template id in integration test
2023-02-20 22:26:04 +05:30
王一之 10693efa05 add integration tests 2023-02-17 14:23:28 +08:00
xm1k3 cc2f796d2f
Helper function for IP Formats (#3286)
* implemented ip_format helper function

* added tests on ip_format() helper

* fixes on logic

* fixes related to mapcidr docs

* better error value

* fixes + unit test to check index 11

* added call for integration tests

* fixes on dsl-functions number

---------

Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
2023-02-17 01:47:40 +05:30
Tarun Koyalwar c49383de98 detect data race panics in integration_tests 2023-02-10 19:28:07 +05:30
Tarun Koyalwar d18fa6f6b2 fix data race in race requests 2023-02-06 16:18:30 +05:30
Tarun Koyalwar 628628893c
fix make http request inconsistencies (#3243)
* fix make http request inconsistencies

* remove parameters from http vars

* fix trailingslash unit test

* naming conventions: best practices

* fix publish docs action

* remove branch ref from push
2023-02-01 17:23:28 +05:30
xm1k3 a81c754db5
support env var from report yaml (#3188)
* added support yaml for report options

* better to use .HasPrefix()

* wip: working on unmarshal YAML optimization

* managed yaml tag + nil pointers + unit test

* implemented tests

* removed unused code from reporting + code refactoring

* WIP: code refactoring and tests

* check on env var

* more test coverage and added callback func

* docs + renaming func

* moved callback logic + removed yaml validation

* used yaml decoder

* struct typo

* refactoring walk method with generic signature

* removed yamlwrapper refs, used yaml2 + docs

implemented test to check also fields without yaml tag

* used DecodeAndValidate()

* removed double import reference

---------

Co-authored-by: mzack <marco.rivoli.nvh@gmail.com>
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
2023-02-01 02:38:17 +05:30
Tarun Koyalwar 0b2a3e296a
fix url encoding issues and inconsistencies (#3211)
* fix url encoding issues

* complete requested changes and improvements

* fix missing issue-tracker-config.yaml

* fuzz: deepcopy and use urlutil.Params
2023-01-24 22:04:52 +05:30
Mzack9999 d956275e98
Adding content length edge cases (#3147)
* adding content length edge cases

* fixing CL behavior

* suppressing -1 error

* fixing path
2023-01-07 18:06:44 +05:30
Mzack9999 a96f764959
extending request/response hijacking with native calls (#3091)
* extending request/response hijacking with native calls

* fixing tests
2023-01-05 17:26:18 +05:30
Tarun Koyalwar e66ed30cec
fix missing trailing slash (#3127)
* raw: fix missing trailing slash

* adds rawpath integration test

* rename trailing slash test
2023-01-03 23:45:34 +05:30
Mzack9999 a4ce231983
Replacing unstable interactsh with scanme.sh (#3107) 2022-12-30 20:13:07 +05:30
Taufik Mulyana 7093180b6d
Add support to query DNS TLSA record (#3054)
* Add support to query DNS TLSA record

* fix build test

* fix ci-lint

* set expected to 0

* test domain update

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2022-12-28 21:12:02 +05:30
xm1k3 bbb561b097
CLI variables are not accessible in SSL Protocol (#3069)
* added vars payload also in ssl

* fix on ssl.go, moved function on payloadValues creation

* added integration test

* rebase + minor changes

Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
2022-12-21 02:02:18 +05:30
forgedhallpass 0295ca19bc
Add `split` DSL function (#2838)
* Add support for showing overloaded DSL method signatures

* Add `split` DSL function #2837

* fixing lint warnings

* replacing faulty regex with strings methods

Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
Co-authored-by: mzack <marco.rivoli.nvh@gmail.com>
2022-11-14 06:08:12 +05:30
Mzack9999 1fbbca66f9
Adding support to scan all v4/v6 IPs (#2709)
* Adding support to scan all v4/v6 IPs

* adding tests

* metainput prototype

* using new signature

* fixing nil pointer

* adding request context with metadata

* removing log instruction

* fixing merge conflicts

* adding clone helpers

* attempting to fix ipv6 square parenthesis wrap

* fixing dialed ip info

* fixing syntax

* fixing output ip selection

* adding integration tests

* disabling test due to gh ipv6 issue

* using ipv4 only due to GH limited networking

* extending metainput marshaling

* fixing hmap key

* adding test for httpx integration

* fixing lint error

* reworking marshaling/id-calculation

* adding ip version validation

* improving handling non url targets

* fixing condition check
2022-11-09 18:48:56 +05:30
Ice3man b9472cf7e1
Added fuzzing support for query params + var dump feature (#2679)
* Added fuzzing support for query params + var dump feature

* Added query-fuzz integration test

* Fixed payloads + added keys-regex fuzz parameter

* Fixed interactsh not working + misc

* Fixed evaluation + added global variables/dsl support to payloads

* Misc fixes related to variables evaluations

* Added http variables support to fuzz

* misc

* Misc

* Added testing playground + misc renaming

* Added support for path and raw request to fuzzing

* Fixed fuzz integration test

* Fixed variable unresolved issue

* Add multiple parameter support with same name

* Added parameter value as 'value' dsl variable for parts

Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
2022-11-01 20:28:50 +05:30
Ice3man d956f08cb9
Added attack-type option to override template attack-type (#2724)
* Added attack-type option to override template attack-type

* Added docs + integration tests
2022-10-19 03:51:45 +05:30
Mzack9999 9493dfdb20
Adding automatic request condition detection (#2707)
* Adding automatic request condition detection

* adding missing checks on part

* test update as per latest change

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2022-10-15 15:19:04 +05:30
Mzack9999 781e4e6105
Shared Execution Context Prototype (#2576)
* renaming var

* Introducing shared execution context prototype

* fixing field name

* adding shared values propagation

* adding shared context lock

* add slice values normalization

* adding integration tests

* adding metadata support for dns

* adding multi-protocol context sharing test

* removing debug test files

* moving contextargs around

* adding comments

* refactoring code

- getter/setter for complex types
- using pointers to avoid heap allocations
2022-10-03 15:42:20 +05:30
Mzack9999 18f14b631c
Adding same host redirect support (#2655)
* simplifying test syntax

* adding same host redirect + refactoring redirect handling

* adding missing file

* adding support for template syntax

* adding integration test

* updating options

* fixing issue on same host redirect
2022-09-29 04:11:28 +05:30
Myung-jong Kim 9eea441b0e
Add `sort(list)`, `sort(string)`, `uniq(list)`, `uniq(string)` helper functions (#2372)
* Add feature in join() to sort a single string slice

Signed-off-by: Myung-jong Kim <mjkim610@gmail.com>

* Add sort helper function and related tests

Signed-off-by: Myung-jong Kim <mjkim610@gmail.com>

* Add uniq helper function and related tests

Signed-off-by: mjkim610 <mjkim610@gmail.com>

Signed-off-by: Myung-jong Kim <mjkim610@gmail.com>
Signed-off-by: mjkim610 <mjkim610@gmail.com>
2022-09-27 02:59:13 +05:30
Sami f3eb0daa39
additional dsl functions (#2550)
* additional dsl functions

* avoid conversion at each iteration
2022-09-08 14:25:34 +05:30
Ice3man e193e7c87e
Added tlsx integration to nuclei (#2522)
* Added tlsx integration to nuclei

* tls tests fix

* Added helper functions + upgrade tlsx to fix

* go mod update

* workflow fix to race test on windows

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2022-09-01 23:56:55 +05:30
James Turner 42a0732d68
Add sha512 support (#2517) 2022-08-31 12:36:02 +05:30
Ice3man e7cffad312
Fixed request annotation based timeout bugs + tests + misc (#2476) 2022-08-23 12:45:55 +05:30
Sajad c4ba2b4edb
Additional helpers (#2359)
* add starts_with, ends_with helper functions

* add test cases, update dsl signatures

* change split_starts_with to line_starts_with
2022-08-06 23:16:03 +05:30
Mzack9999 57b2ec7ade
Adding offline http tests (#2276) 2022-07-12 02:58:36 +05:30
Mzack9999 1c332bb85b
Improving RFC request/response passive parsing (#2192)
* Improving RFC request/response passive parsing

* adding test
2022-07-11 22:43:10 +05:30
Ice3man ffe6ab04b3
Added include-templates force-loading for templates (#2232)
* Added include-templates force-loading for templates

* Fixed loader case with include-templates

* Added integration test for excluded-template in loader
2022-06-27 18:09:29 +05:30
Ice3man 9073b753ca
Added aes_gcm implementation for DSL function (#2196)
* Added aes_gcm implementation for DSL function

* Added integration test for dsl-functions.yaml
2022-06-23 16:16:24 +05:30
forgedhallpass cadba90a5a test: extended the DSL helper integration test cases 2022-06-10 15:58:54 +03:00
forgedhallpass 179cf908a8 Revert "test: extended the DSL helper integration test cases"
This reverts commit ea0b6eae64.
2022-06-10 13:48:34 +03:00
forgedhallpass ea0b6eae64 test: extended the DSL helper integration test cases 2022-06-10 13:33:16 +03:00
forgedhallpass 9fd9892f49 test: extra DSL function test cases 2022-06-08 20:58:46 +03:00
Ice3man be5f1a7623
Added redirected matched URL + stop-at-first-match for redirect chains (#2050)
* Added redirected matched URL + stop-at-first-match for redirect chains

* Pleasing go-linter
2022-05-30 15:19:09 +05:30
Mzack9999 dd3b0a3cfc
Increasing buffer size on AND condition (#2030)
* Increasing buffer size on AND condition

* adding integration tests

* fixing tests data path
2022-05-30 13:45:28 +05:30
Mzack9999 16a05d0aa2
Adding CLI SNI support to unsafe http (#2077)
* Adding CLI SNI support to unsafe http

* adding http unsafe sni test
2022-05-27 21:53:07 +05:30
Mzack9999 39c7317ec3
Adding SNI override via request annotations (#1970)
* Adding SNI override via request annotations

* adding cli flag priority
2022-05-12 16:43:56 +05:30
Mzack9999 2f1330345f
Adding global SNI support for HTTP protocol via CLI (#1964)
* Adding global SNI support via CLI

* adding integration test

* adding cli option to docs

* reverting deleted test
2022-05-11 16:00:39 +05:30
Sandeep Singh 8b7e5535f8
Merge pull request #1902 from projectdiscovery/go-example-tests
Fixed nuclei go example + added relevant integration test
2022-04-28 18:09:11 +05:30
Ice3man a031ef32b4 Fixed nuclei go example + added relevant integration test 2022-04-28 01:02:06 +05:30
LuitelSamikshya 3ccbfe4626 disable redirects flag 2022-04-27 11:19:44 -05:00
Ice3man b91bad813b Misc changes to PR 2022-04-20 15:36:02 +05:30
Ice3man d0d65f8d6b Added integration tests for variables + misc changes 2022-04-02 02:14:00 +05:30
Sandeep Singh e8690f378d
Merge pull request #1746 from projectdiscovery/issue-1703-minor-changes
Stats counter + Mime type filter
2022-03-29 17:49:44 +05:30
Xavier Stevens 4a65097194 Squashed commit of the following:
commit b590de2de14923e4cb35dd19845e12833a4ccbfa
Author: Xavier Stevens <xstevens@users.noreply.github.com>
Date:   Wed Mar 23 08:29:37 2022 -0700

    Updated rand_ip to use variadic args

commit ea883be8c0aa56174a1301252129289334659f3f
Author: Xavier Stevens <xstevens@users.noreply.github.com>
Date:   Tue Mar 22 16:59:19 2022 -0700

    Updated rand_ip function to handle multiple CIDRs
2022-03-23 08:37:05 -07:00
mzack 5cd25bd069 more checks + test cases fix 2022-03-22 14:18:01 +01:00
skhalsa-sigsci 63eccc85ba random ip helper function 2022-03-21 17:39:10 -07:00
Sandeep Singh 34841358b3
Merge pull request #1694 from projectdiscovery/#1332-integration-test-for-template-loading
#1332 integration test for template loading
2022-03-10 17:28:07 +05:30
LuitelSamikshya 8e767951f8 added test template 2022-03-08 09:57:56 -06:00
Ice3man 2a054290c6 Added ciphersuite to tls data + misc 2022-03-08 19:41:52 +05:30
Ice3man 90b4c09f80 Added more tests and ciphersuits for SSL protocol 2022-03-07 14:07:30 +05:30
Ice3man 4d6071c1e0 Run template update once during functional and integration test 2022-03-03 19:10:03 +05:30
Sandeep Singh 145faeef8d
Merge pull request #1574 from projectdiscovery/issue-582-headless-payloads
Adding support for payloads in headless protocol
2022-02-05 20:28:46 +05:30
mzack 54d8f9fdb5 Adding support for payloads in headless protocol 2022-02-04 11:43:42 +01:00
mzack 62fc7a8110 Merge branch 'dev' into issue-1231-zcrypto 2022-02-03 09:02:09 +01:00
Sandeep Singh 04f8f84c9d
Merge pull request #1564 from projectdiscovery/issue-1542-caa-query
Adding support for DNS CAA query
2022-02-03 11:23:03 +05:30
mzack 4d6a8a71a7 Adding support for DNS CAA query 2022-02-02 07:54:15 +01:00
mzack 3807e648c7 Fixing exit at first match with extractors 2022-02-01 11:25:29 +01:00
mzack 19d2b80a1c adding ssl test cases + tcpserver refactor 2022-01-27 21:29:28 +01:00
Sandeep Singh 93616a9e80
Merge pull request #1511 from projectdiscovery/issue-1464-dns-ptr-query
Adding auto-PTR IP to FQDN
2022-01-19 17:30:12 +05:30
Sandeep Singh 59762ae266
Merge pull request #1503 from projectdiscovery/dsl_concat
DSL concat function
2022-01-18 21:34:21 +05:30
mzack aec007e1b4 adding integration tests for single/multiple race conditions requests 2022-01-18 14:15:15 +01:00
mzack 994bc9e789 adding PTR integration test 2022-01-18 13:47:15 +01:00
forgedhallpass 286e27510d test: introduce 'concat' DSL function
* correct order and number of expected results
2022-01-17 13:50:42 +02:00
forgedhallpass d4a015fb47 test: introduce 'concat' DSL function
* fixed typo in test
2022-01-17 13:45:16 +02:00
forgedhallpass 7f6c72853f feat: introduce 'concat' DSL function #1502 2022-01-17 13:32:15 +02:00
sandeep 50bad75cb4 Merge branch 'dev' of https://github.com/projectdiscovery/nuclei into interactsh-first-match-stop 2021-12-21 15:46:37 +05:30
Sajad Parra c0f9c1da70 interactsh stopAtFirstMatch intergration test 2021-12-21 15:24:16 +05:30
Mzack9999 75946f7c0f fixing ip/hostname for windows self-contained templates 2021-12-20 16:23:36 +01:00
Sajad 5200bcd94f
Whois Protocol Support (using rdap) (#1354)
* init rdap

* add an integration test, option to supply RDAP server to execute the request on

* add rdap protocolMappings

* add debug info, add IP, ASN query type support

* rename rdap to whois, Host to Query in template

* rename pending rdap to whois

* remove port from whois varaiables

* set Host variable even if input is not a parsable url
2021-12-16 17:08:02 +05:30
forgedhallpass 85e0b96d51
bug: fixed couple of bugs in the DSL functions (#1372)
* feat: Improve DSL function UX #1295

Sort the output signatures

* feat: Improve DSL function UX #1295

Sort the output signatures.
Lint: simplified the sorting.

* bug: fixed couple of bugs in the DSL functions

Input number parameters are stored as float64 types, hence the type conversion should happen accordingly. Affected functions:
* rand_int
* wait_for
* unix_time
* rand_text_numeric

Added tests for all functions.
Related: #1261

* bug: fixed couple of bugs in the DSL functions

Handle cases when the optional input character set is an empty string. Affected methods:
* rand_char
* rand_base

* bug: fixed couple of bugs in the DSL functions

Change rand_char to return a one character string, instead of the character code

* refactor: Minor integration test changes to show the actual and expected result numbers

* test: Added integration test for all existing DSL functions

* test: Added integration test for all existing DSL functions

Fixing linter issues.

* feat: Add "repeat" DSL function

* test: Add "repeat" DSL function
2021-12-15 19:33:57 +05:30
forgedhallpass 974cbfb35e
feat: Re-run failed integration tests in debug mode (#1367)
* If the tests are executed through GitHub actions and there are failed integration tests, they will be re-executed with verbose output to help figuring out the underlying issues.
* Added some grouping to make reading the logs easier

Ticket: #1365
2021-12-14 18:13:53 +02:00
Mzack9999 8336421179
Improving headless test cases (#1313)
* Adding headless test cases
2021-12-04 23:10:11 +05:30
Sajad Parra 5b99921d75 add unit and integration tests for dsl variable #555 2021-11-30 20:20:43 +05:30
forgedhallpass 7e22d70ded refactor/documentation: typos and grammatical errors 2021-11-25 18:54:16 +02:00
Sajad f74ff3fc49
Tag based struct validation (#1256)
* Added tag based struct validation
2021-11-20 13:25:27 +05:30
Ice3man543 c1a35b3ff9 Merge branch 'dev' of https://github.com/projectdiscovery/nuclei into more-protocols 2021-11-11 17:21:25 +05:30
Sandeep Singh 66074a1842
Merge pull request #1232 from projectdiscovery/fix-redirect-response-bug
fix #1173: perform matching on all redirect responses instead of final
2021-11-10 17:38:35 +05:30
Sandeep Singh f78ff42dbe
Merge branch 'dev' into more-protocols 2021-11-09 18:27:13 +05:30
Ice3man543 9b364080ea Fixed some edge cases with status codes in redirect chains 2021-11-09 17:55:42 +05:30
sandeep f2c81ee9a2 fixed unsafe test 2021-11-09 16:11:11 +05:30
Ice3man543 8396413f9d Added integration test for the redirect chain http functionality 2021-11-09 06:10:07 +05:30