mirror of https://github.com/daffainfo/nuclei.git
Merge from dev
commit
72a387c40a
|
@ -33,4 +33,4 @@ Example: steps to reproduce the behavior:
|
||||||
|
|
||||||
|
|
||||||
### Anything else:
|
### Anything else:
|
||||||
<!-- Links? References? Screnshots? Anything that will give us more context about the issue that you are encountering! -->
|
<!-- Links? References? Screenshots? Anything that will give us more context about the issue that you are encountering! -->
|
||||||
|
|
43
DESIGN.md
43
DESIGN.md
|
@ -10,9 +10,9 @@ Template is the basic unit of input to the engine which describes the requests t
|
||||||
|
|
||||||
The template structure is described here. Template level attributes are defined here as well as convenience methods to validate, parse and compile templates creating executers.
|
The template structure is described here. Template level attributes are defined here as well as convenience methods to validate, parse and compile templates creating executers.
|
||||||
|
|
||||||
Any attributes etc required for the template, engine or requests to function are also set here.
|
Any attributes etc. required for the template, engine or requests to function are also set here.
|
||||||
|
|
||||||
Workflows are also compiled, their templates are loaded and compiled as well. Any validations etc on the paths provided are also done here.
|
Workflows are also compiled, their templates are loaded and compiled as well. Any validations etc. on the paths provided are also done here.
|
||||||
|
|
||||||
`Parse` function is the main entry point which returns a template for a `filePath` and `executorOptions`. It compiles all the requests for the templates, all the workflows, as well as any self-contained request etc. It also caches the templates in an in-memory cache.
|
`Parse` function is the main entry point which returns a template for a `filePath` and `executorOptions`. It compiles all the requests for the templates, all the workflows, as well as any self-contained request etc. It also caches the templates in an in-memory cache.
|
||||||
|
|
||||||
|
@ -106,7 +106,7 @@ The default executer is provided in `pkg/protocols/common/executer` . It takes a
|
||||||
|
|
||||||
A different executer is the Clustered Requests executer which implements the Nuclei Request clustering functionality in `pkg/templates` We have a single HTTP request in cases where multiple templates can be clustered and multiple operator lists to match/extract. The first HTTP request is executed while all the template matcher/extractor are evaluated separately.
|
A different executer is the Clustered Requests executer which implements the Nuclei Request clustering functionality in `pkg/templates` We have a single HTTP request in cases where multiple templates can be clustered and multiple operator lists to match/extract. The first HTTP request is executed while all the template matcher/extractor are evaluated separately.
|
||||||
|
|
||||||
For Workflow execution, a separate RunWorkflow function is used which executes the workflow independently from the template execution.
|
For Workflow execution, a separate RunWorkflow function is used which executes the workflow independently of the template execution.
|
||||||
|
|
||||||
With this basic premise set, we can now start exploring the current runner implementation which will also walk us through the architecture of nuclei.
|
With this basic premise set, we can now start exploring the current runner implementation which will also walk us through the architecture of nuclei.
|
||||||
|
|
||||||
|
@ -118,7 +118,7 @@ The first process after all CLI specific initialisation is the loading of templa
|
||||||
|
|
||||||
#### pkg/catalog
|
#### pkg/catalog
|
||||||
|
|
||||||
This package is used to get paths using mixed syntax. It takes a template directory and performs resolving for template paths both from provided template directory as well as the current user directory.
|
This package is used to get paths using mixed syntax. It takes a template directory and performs resolving for template paths both from provided template and current user directory.
|
||||||
|
|
||||||
The syntax is very versatile and can include filenames, glob patterns, directories, absolute paths, and relative-paths.
|
The syntax is very versatile and can include filenames, glob patterns, directories, absolute paths, and relative-paths.
|
||||||
|
|
||||||
|
@ -177,7 +177,7 @@ ResultEvent structure is passed to the Nuclei Output Writer which contains the e
|
||||||
|
|
||||||
#### pkg/protocols/common/interactsh
|
#### pkg/protocols/common/interactsh
|
||||||
|
|
||||||
Interactsh module is used to provide automatic Out of Band vulnerability identification in Nuclei.
|
Interactsh module is used to provide automatic Out-of-Band vulnerability identification in Nuclei.
|
||||||
|
|
||||||
It uses two LRU caches, one for storing interactions for request URLs and one for storing requests for interaction URL. These both caches are used to correlated requests received to the Interactsh OOB server and Nuclei Instance. [Interactsh Client](https://github.com/projectdiscovery/interactsh/pkg/client) package does most of the heavy lifting of this module.
|
It uses two LRU caches, one for storing interactions for request URLs and one for storing requests for interaction URL. These both caches are used to correlated requests received to the Interactsh OOB server and Nuclei Instance. [Interactsh Client](https://github.com/projectdiscovery/interactsh/pkg/client) package does most of the heavy lifting of this module.
|
||||||
|
|
||||||
|
@ -193,13 +193,13 @@ Next we arrive in the `RunEnumeration` function of the runner.
|
||||||
|
|
||||||
Next the `WorkflowLoader` is initialised which used to load workflows. It exists in `v2/pkg/parsers/workflow_loader.go`
|
Next the `WorkflowLoader` is initialised which used to load workflows. It exists in `v2/pkg/parsers/workflow_loader.go`
|
||||||
|
|
||||||
The loader is initialised moving forward which is responsible for Using Catalog, Passed Tags, Filters, Paths, etc to return compiled `Templates` and `Workflows`.
|
The loader is initialised moving forward which is responsible for Using Catalog, Passed Tags, Filters, Paths, etc. to return compiled `Templates` and `Workflows`.
|
||||||
|
|
||||||
#### pkg/catalog/loader
|
#### pkg/catalog/loader
|
||||||
|
|
||||||
First the input passed by the user as paths is normalised to absolute paths which is done by the `pkg/catalog` module. Next the path filter module is used to removed the excluded template/workflows paths.
|
First the input passed by the user as paths is normalised to absolute paths which is done by the `pkg/catalog` module. Next the path filter module is used to remove the excluded template/workflows paths.
|
||||||
|
|
||||||
`pkg/parsers` module's `LoadTemplate`,`LoadWorkflow` functions are used to check if the templates pass the validation + are not excluded via tags/severity/etc filters. If all checks are passed, then the template/workflow is parsed and returned in a compiled form by the `pkg/templates`'s `Parse` function.
|
`pkg/parsers` module's `LoadTemplate`,`LoadWorkflow` functions are used to check if the templates pass the validation + are not excluded via tags/severity/etc. filters. If all checks are passed, then the template/workflow is parsed and returned in a compiled form by the `pkg/templates`'s `Parse` function.
|
||||||
|
|
||||||
`Parse` function performs compilation of all the requests in a template + creates Executers from them returning a runnable Template/Workflow structure.
|
`Parse` function performs compilation of all the requests in a template + creates Executers from them returning a runnable Template/Workflow structure.
|
||||||
|
|
||||||
|
@ -207,10 +207,10 @@ Clustering module comes in next whose job is to cluster identical HTTP GET reque
|
||||||
|
|
||||||
### pkg/operators
|
### pkg/operators
|
||||||
|
|
||||||
Operators package implements all of the matching and extracting logic of Nuclei.
|
Operators package implements all the matching and extracting logic of Nuclei.
|
||||||
|
|
||||||
```go
|
```go
|
||||||
// Operators contains the operators that can be applied on protocols
|
// Operators contain the operators that can be applied on protocols
|
||||||
type Operators struct {
|
type Operators struct {
|
||||||
Matchers []*matchers.Matcher
|
Matchers []*matchers.Matcher
|
||||||
Extractors []*extractors.Extractor
|
Extractors []*extractors.Extractor
|
||||||
|
@ -218,7 +218,7 @@ type Operators struct {
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
A protocol only needs to embed the `operators.Operators` type shown above and it can utilise all the matching/extracting functionality of nuclei.
|
A protocol only needs to embed the `operators.Operators` type shown above, and it can utilise all the matching/extracting functionality of nuclei.
|
||||||
|
|
||||||
```go
|
```go
|
||||||
// MatchFunc performs matching operation for a matcher on model and returns true or false.
|
// MatchFunc performs matching operation for a matcher on model and returns true or false.
|
||||||
|
@ -246,7 +246,7 @@ type Result struct {
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
The internal logics for matching and extracting for things like words, regexes, jq, paths, etc is specified in `pkg/operators/matchers`, `pkg/operators/extractors`. Those packages should be investigated for further look into the topic.
|
The internal logics for matching and extracting for things like words, regexes, jq, paths, etc. is specified in `pkg/operators/matchers`, `pkg/operators/extractors`. Those packages should be investigated for further look into the topic.
|
||||||
|
|
||||||
|
|
||||||
### Template Execution
|
### Template Execution
|
||||||
|
@ -275,6 +275,8 @@ import (
|
||||||
"path"
|
"path"
|
||||||
|
|
||||||
"github.com/logrusorgru/aurora"
|
"github.com/logrusorgru/aurora"
|
||||||
|
"go.uber.org/ratelimit"
|
||||||
|
|
||||||
"github.com/projectdiscovery/goflags"
|
"github.com/projectdiscovery/goflags"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/catalog"
|
"github.com/projectdiscovery/nuclei/v2/pkg/catalog"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/catalog/config"
|
"github.com/projectdiscovery/nuclei/v2/pkg/catalog/config"
|
||||||
|
@ -291,7 +293,6 @@ import (
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/reporting"
|
"github.com/projectdiscovery/nuclei/v2/pkg/reporting"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
|
"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/types"
|
"github.com/projectdiscovery/nuclei/v2/pkg/types"
|
||||||
"go.uber.org/ratelimit"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
func main() {
|
func main() {
|
||||||
|
@ -356,7 +357,7 @@ func main() {
|
||||||
|
|
||||||
### Adding a New Protocol
|
### Adding a New Protocol
|
||||||
|
|
||||||
Protocols form the core of Nuclei Engine. All the request types like `http`, `dns`, etc are implemented in form of protocol requests.
|
Protocols form the core of Nuclei Engine. All the request types like `http`, `dns`, etc. are implemented in form of protocol requests.
|
||||||
|
|
||||||
A protocol must implement the `Protocol` and `Request` interfaces described above in `pkg/protocols`. We'll take the example of an existing protocol implementation - websocket for this short reference around Nuclei internals.
|
A protocol must implement the `Protocol` and `Request` interfaces described above in `pkg/protocols`. We'll take the example of an existing protocol implementation - websocket for this short reference around Nuclei internals.
|
||||||
|
|
||||||
|
@ -474,13 +475,13 @@ func (r *Request) Type() templateTypes.ProtocolType {
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
Almost all of these protocols have boilerplate functions for which default implementations have been provided in the `providers` package. Examples are the implementation of `Match`, `Extract`, `MakeResultEvent`, GetCompiledOperators`, etc which are almost same throughout Nuclei protocols code. It is enough to copy-paste them unless customization is required.
|
Almost all of these protocols have boilerplate functions for which default implementations have been provided in the `providers` package. Examples are the implementation of `Match`, `Extract`, `MakeResultEvent`, GetCompiledOperators`, etc. which are almost same throughout Nuclei protocols code. It is enough to copy-paste them unless customization is required.
|
||||||
|
|
||||||
`eventcreator` package offers `CreateEventWithAdditionalOptions` function which can be used to create result events after doing request execution.
|
`eventcreator` package offers `CreateEventWithAdditionalOptions` function which can be used to create result events after doing request execution.
|
||||||
|
|
||||||
Step by step description of how to add a new protocol to Nuclei -
|
Step by step description of how to add a new protocol to Nuclei -
|
||||||
|
|
||||||
1. Add the protocol implementation in `pkg/protocols` directory. If it's a small protocol with less number of options, considering adding it to the `pkg/protocols/others` directory. Add the enum for the new protocol to `v2/pkg/templates/types/types.go`.
|
1. Add the protocol implementation in `pkg/protocols` directory. If it's a small protocol with fewer options, considering adding it to the `pkg/protocols/others` directory. Add the enum for the new protocol to `v2/pkg/templates/types/types.go`.
|
||||||
|
|
||||||
2. Add the protocol request structure to the `Template` structure fields. This is done in `pkg/templates/templates.go` with the corresponding import line.
|
2. Add the protocol request structure to the `Template` structure fields. This is done in `pkg/templates/templates.go` with the corresponding import line.
|
||||||
|
|
||||||
|
@ -526,7 +527,7 @@ func (t *Template) Type() templateTypes.ProtocolType {
|
||||||
|
|
||||||
```go
|
```go
|
||||||
|
|
||||||
// Requests returns the total request count for the template
|
// Requests return the total request count for the template
|
||||||
func (template *Template) Requests() int {
|
func (template *Template) Requests() int {
|
||||||
return len(template.RequestsDNS) +
|
return len(template.RequestsDNS) +
|
||||||
...
|
...
|
||||||
|
@ -554,11 +555,11 @@ That's it, you've added a new protocol to Nuclei. The next good step would be to
|
||||||
- [v2/pkg/reporting](./v2/pkg/reporting) - Reporting modules for nuclei.
|
- [v2/pkg/reporting](./v2/pkg/reporting) - Reporting modules for nuclei.
|
||||||
- [v2/pkg/reporting/exporters/sarif](./v2/pkg/reporting/exporters/sarif) - Sarif Result Exporter
|
- [v2/pkg/reporting/exporters/sarif](./v2/pkg/reporting/exporters/sarif) - Sarif Result Exporter
|
||||||
- [v2/pkg/reporting/exporters/markdown](./v2/pkg/reporting/exporters/markdown) - Markdown Result Exporter
|
- [v2/pkg/reporting/exporters/markdown](./v2/pkg/reporting/exporters/markdown) - Markdown Result Exporter
|
||||||
- [v2/pkg/reporting/exporters/es](./v2/pkg/reporting/exporters/e) - Elasticsearch Result Exporter
|
- [v2/pkg/reporting/exporters/es](./v2/pkg/reporting/exporters/es) - Elasticsearch Result Exporter
|
||||||
- [v2/pkg/reporting/dedupe](./v2/pkg/reporting/dedupe) - Dedupe module for Results
|
- [v2/pkg/reporting/dedupe](./v2/pkg/reporting/dedupe) - Dedupe module for Results
|
||||||
- [v2/pkg/reporting/trackers/gitlab](./v2/pkg/reporting/trackers/gitlab) - Gitlab Issue Tracker Exporter
|
- [v2/pkg/reporting/trackers/gitlab](./v2/pkg/reporting/trackers/gitlab) - Gitlab Issue Tracker Exporter
|
||||||
- [v2/pkg/reporting/trackers/jira](./v2/pkg/reporting/trackers/jira) - Jira Issue Tracker Exporter
|
- [v2/pkg/reporting/trackers/jira](./v2/pkg/reporting/trackers/jira) - Jira Issue Tracker Exporter
|
||||||
- [v2/pkg/reporting/trackers/github](./v2/pkg/reporting/trackers/github) - Github Issue Tracker Exporter
|
- [v2/pkg/reporting/trackers/github](./v2/pkg/reporting/trackers/github) - GitHub Issue Tracker Exporter
|
||||||
- [v2/pkg/reporting/format](./v2/pkg/reporting/format) - Result Formatting Functions
|
- [v2/pkg/reporting/format](./v2/pkg/reporting/format) - Result Formatting Functions
|
||||||
- [v2/pkg/parsers](./v2/pkg/parsers) - Implements template as well as workflow loader for initial template discovery, validation and - loading.
|
- [v2/pkg/parsers](./v2/pkg/parsers) - Implements template as well as workflow loader for initial template discovery, validation and - loading.
|
||||||
- [v2/pkg/types](./v2/pkg/types) - Contains CLI options as well as misc helper functions.
|
- [v2/pkg/types](./v2/pkg/types) - Contains CLI options as well as misc helper functions.
|
||||||
|
@ -577,12 +578,12 @@ That's it, you've added a new protocol to Nuclei. The next good step would be to
|
||||||
- [v2/pkg/model](./v2/pkg/model) - Template Info + misc
|
- [v2/pkg/model](./v2/pkg/model) - Template Info + misc
|
||||||
- [v2/pkg/templates](./v2/pkg/templates) - Templates core starting point
|
- [v2/pkg/templates](./v2/pkg/templates) - Templates core starting point
|
||||||
- [v2/pkg/templates/cache](./v2/pkg/templates/cache) - Templates cache
|
- [v2/pkg/templates/cache](./v2/pkg/templates/cache) - Templates cache
|
||||||
- [v2/pkg/protocols](./v2/pkg/protocol) - Protocol Specification
|
- [v2/pkg/protocols](./v2/pkg/protocols) - Protocol Specification
|
||||||
- [v2/pkg/protocols/file](./v2/pkg/protocols/file) - File protocol
|
- [v2/pkg/protocols/file](./v2/pkg/protocols/file) - File protocol
|
||||||
- [v2/pkg/protocols/network](./v2/pkg/protocols/network) - Network protocol
|
- [v2/pkg/protocols/network](./v2/pkg/protocols/network) - Network protocol
|
||||||
- [v2/pkg/protocols/common/expressions](./v2/pkg/protocols/common/expressions) - Expression evaluation + Templating Support
|
- [v2/pkg/protocols/common/expressions](./v2/pkg/protocols/common/expressions) - Expression evaluation + Templating Support
|
||||||
- [v2/pkg/protocols/common/interactsh](./v2/pkg/protocols/common/interactsh) - Interactsh integration
|
- [v2/pkg/protocols/common/interactsh](./v2/pkg/protocols/common/interactsh) - Interactsh integration
|
||||||
- [v2/pkg/protocols/common/generators](./v2/pkg/protocols/common/generators) - Payload support for Requests (Sniper, etc)
|
- [v2/pkg/protocols/common/generators](./v2/pkg/protocols/common/generators) - Payload support for Requests (Sniper, etc.)
|
||||||
- [v2/pkg/protocols/common/executer](./v2/pkg/protocols/common/executer) - Default Template Executer
|
- [v2/pkg/protocols/common/executer](./v2/pkg/protocols/common/executer) - Default Template Executer
|
||||||
- [v2/pkg/protocols/common/replacer](./v2/pkg/protocols/common/replacer) - Template replacement helpers
|
- [v2/pkg/protocols/common/replacer](./v2/pkg/protocols/common/replacer) - Template replacement helpers
|
||||||
- [v2/pkg/protocols/common/helpers/eventcreator](./v2/pkg/protocols/common/helpers/eventcreator) - Result event creator
|
- [v2/pkg/protocols/common/helpers/eventcreator](./v2/pkg/protocols/common/helpers/eventcreator) - Result event creator
|
||||||
|
|
|
@ -1,7 +1,7 @@
|
||||||
FROM golang:1.17.3-alpine as build-env
|
FROM golang:1.17.3-alpine as build-env
|
||||||
RUN go install -v github.com/projectdiscovery/nuclei/v2/cmd/nuclei@latest
|
RUN go install -v github.com/projectdiscovery/nuclei/v2/cmd/nuclei@latest
|
||||||
|
|
||||||
FROM alpine:3.14
|
FROM alpine:3.15.0
|
||||||
RUN apk add --no-cache bind-tools ca-certificates chromium
|
RUN apk add --no-cache bind-tools ca-certificates chromium
|
||||||
COPY --from=build-env /go/bin/nuclei /usr/local/bin/nuclei
|
COPY --from=build-env /go/bin/nuclei /usr/local/bin/nuclei
|
||||||
ENTRYPOINT ["nuclei"]
|
ENTRYPOINT ["nuclei"]
|
||||||
|
|
|
@ -58,7 +58,7 @@ go install -v github.com/projectdiscovery/nuclei/v2/cmd/nuclei@latest
|
||||||
|
|
||||||
### Nuclei Templates
|
### Nuclei Templates
|
||||||
|
|
||||||
Nuclei has had built-in support for automatic template download/update as default since version [v2.5.2](https://github.com/projectdiscovery/nuclei/releases/tag/v2.5.2). [**Nuclei-Templates**](https://github.com/projectdiscovery/nuclei-templates) project provides a community-contributed list of ready-to-use templates that is constantly updated.
|
Nuclei has built-in support for automatic template download/update as default since version [v2.5.2](https://github.com/projectdiscovery/nuclei/releases/tag/v2.5.2). [**Nuclei-Templates**](https://github.com/projectdiscovery/nuclei-templates) project provides a community-contributed list of ready-to-use templates that is constantly updated.
|
||||||
|
|
||||||
You may still use the `update-templates` flag to update the nuclei templates at any time; You can write your own checks for your individual workflow and needs following Nuclei's [templating guide](https://nuclei.projectdiscovery.io/templating-guide/).
|
You may still use the `update-templates` flag to update the nuclei templates at any time; You can write your own checks for your individual workflow and needs following Nuclei's [templating guide](https://nuclei.projectdiscovery.io/templating-guide/).
|
||||||
|
|
||||||
|
@ -250,7 +250,7 @@ Please check our other open-source projects that might fit into your bug bounty
|
||||||
|
|
||||||
Nuclei immensely improve how you approach security assessment by augmenting the manual, repetitive processes. Consultancies are already converting their manual assessment steps with Nuclei, it allows them to run set of their custom assessment approach across thousands of hosts in an automated manner.
|
Nuclei immensely improve how you approach security assessment by augmenting the manual, repetitive processes. Consultancies are already converting their manual assessment steps with Nuclei, it allows them to run set of their custom assessment approach across thousands of hosts in an automated manner.
|
||||||
|
|
||||||
Pen-testers get the full power of our public templates and customization capabilities to speed-up their assessment process, and specifically with the regression cycle where you can easily verify the fix.
|
Pen-testers get the full power of our public templates and customization capabilities to speed up their assessment process, and specifically with the regression cycle where you can easily verify the fix.
|
||||||
|
|
||||||
- Easily create your compliance, standards suite (e.g. OWASP Top 10) checklist.
|
- Easily create your compliance, standards suite (e.g. OWASP Top 10) checklist.
|
||||||
- With capabilities like [fuzz](https://nuclei.projectdiscovery.io/templating-guide/#advance-fuzzing) and [workflows](https://nuclei.projectdiscovery.io/templating-guide/#workflows), complex manual steps and repetitive assessment can be easily automated with Nuclei.
|
- With capabilities like [fuzz](https://nuclei.projectdiscovery.io/templating-guide/#advance-fuzzing) and [workflows](https://nuclei.projectdiscovery.io/templating-guide/#workflows), complex manual steps and repetitive assessment can be easily automated with Nuclei.
|
||||||
|
|
|
@ -29,7 +29,7 @@ Nuclei是一个基于模板的、可配置攻击目标的扫描快速工具,
|
||||||
- [安装](#安装)
|
- [安装](#安装)
|
||||||
- [Nuclei模板](#nuclei模板)
|
- [Nuclei模板](#nuclei模板)
|
||||||
- [用法](#用法)
|
- [用法](#用法)
|
||||||
- [运行Nuclei](#运行nuclei)
|
- [运行Nuclei](#运行Nuclei)
|
||||||
- [排除模板](#排除模板)
|
- [排除模板](#排除模板)
|
||||||
- [致谢](#致谢)
|
- [致谢](#致谢)
|
||||||
|
|
||||||
|
|
|
@ -847,7 +847,7 @@ ID is the optional id of the request
|
||||||
Name is the optional name of the request.
|
Name is the optional name of the request.
|
||||||
|
|
||||||
If a name is specified, all the named request in a template can be matched upon
|
If a name is specified, all the named request in a template can be matched upon
|
||||||
in a combined manner allowing multirequest based matchers.
|
in a combined manner allowing multi-request based matchers.
|
||||||
|
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
@ -862,7 +862,7 @@ in a combined manner allowing multirequest based matchers.
|
||||||
|
|
||||||
Attack is the type of payload combinations to perform.
|
Attack is the type of payload combinations to perform.
|
||||||
|
|
||||||
batteringram is same payload into all of the defined payload positions at once, pitchfork combines multiple payload sets and clusterbomb generates
|
batteringram is inserts the same payload into all defined payload positions at once, pitchfork combines multiple payload sets and clusterbomb generates
|
||||||
permutations and combinations for all payloads.
|
permutations and combinations for all payloads.
|
||||||
|
|
||||||
|
|
||||||
|
@ -1454,7 +1454,7 @@ Examples:
|
||||||
|
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
# Match for outlook mail protection domain
|
# Match for Outlook mail protection domain
|
||||||
words:
|
words:
|
||||||
- mail.protection.outlook.com
|
- mail.protection.outlook.com
|
||||||
```
|
```
|
||||||
|
@ -2452,7 +2452,7 @@ host:
|
||||||
|
|
||||||
Attack is the type of payload combinations to perform.
|
Attack is the type of payload combinations to perform.
|
||||||
|
|
||||||
Batteringram is same payload into all of the defined payload positions at once, pitchfork combines multiple payload sets and clusterbomb generates
|
Batteringram is inserts the same payload into all defined payload positions at once, pitchfork combines multiple payload sets and clusterbomb generates
|
||||||
permutations and combinations for all payloads.
|
permutations and combinations for all payloads.
|
||||||
|
|
||||||
|
|
||||||
|
@ -2676,7 +2676,7 @@ Valid values:
|
||||||
Read is the number of bytes to read from socket.
|
Read is the number of bytes to read from socket.
|
||||||
|
|
||||||
This can be used for protocols which expect an immediate response. You can
|
This can be used for protocols which expect an immediate response. You can
|
||||||
read and write responses one after another and evetually perform matching
|
read and write responses one after another and eventually perform matching
|
||||||
on every data captured with `name` attribute.
|
on every data captured with `name` attribute.
|
||||||
|
|
||||||
The [network docs](https://nuclei.projectdiscovery.io/templating-guide/protocols/network/) highlight more on how to do this.
|
The [network docs](https://nuclei.projectdiscovery.io/templating-guide/protocols/network/) highlight more on how to do this.
|
||||||
|
|
|
@ -3,35 +3,35 @@ allow-list:
|
||||||
deny-list:
|
deny-list:
|
||||||
severity: low
|
severity: low
|
||||||
|
|
||||||
# github contains configuration options for github issue tracker
|
# GitHub contains configuration options for GitHub issue tracker
|
||||||
github:
|
github:
|
||||||
# base-url is the optional self-hosted github application url
|
# base-url is the optional self-hosted GitHub application url
|
||||||
base-url: https://localhost:8443/github
|
base-url: https://localhost:8443/github
|
||||||
# username is the username of the github user
|
# username is the username of the GitHub user
|
||||||
username: test-username
|
username: test-username
|
||||||
# owner is the owner name of the repository for issues.
|
# owner is the owner name of the repository for issues
|
||||||
owner: test-owner
|
owner: test-owner
|
||||||
# token is the token for github account.
|
# token is the token for GitHub account
|
||||||
token: test-token
|
token: test-token
|
||||||
# project-name is the name of the repository.
|
# project-name is the name of the repository
|
||||||
project-name: test-project
|
project-name: test-project
|
||||||
# issue-label is the label of the created issue type
|
# issue-label is the label of the created issue type
|
||||||
issue-label: bug
|
issue-label: bug
|
||||||
|
|
||||||
# gitlab contains configuration options for gitlab issue tracker
|
# GitLab contains configuration options for gitlab issue tracker
|
||||||
gitlab:
|
gitlab:
|
||||||
# base-url is the optional self-hosted gitlab application url
|
# base-url is the optional self-hosted GitLab application url
|
||||||
base-url: https://localhost:8443/gitlab
|
base-url: https://localhost:8443/gitlab
|
||||||
# username is the username of the gitlab user
|
# username is the username of the GitLab user
|
||||||
username: test-username
|
username: test-username
|
||||||
# token is the token for gitlab account.
|
# token is the token for GitLab account
|
||||||
token: test-token
|
token: test-token
|
||||||
# project-name is the name/id of the project(repository).
|
# project-name is the name/id of the project(repository)
|
||||||
project-name: "1234"
|
project-name: "1234"
|
||||||
# issue-label is the label of the created issue type
|
# issue-label is the label of the created issue type
|
||||||
issue-label: bug
|
issue-label: bug
|
||||||
|
|
||||||
# jira contains configuration options for jira issue tracker
|
# Jira contains configuration options for Jira issue tracker
|
||||||
jira:
|
jira:
|
||||||
# cloud is the boolean which tells if Jira instance is running in the cloud or on-prem version is used
|
# cloud is the boolean which tells if Jira instance is running in the cloud or on-prem version is used
|
||||||
cloud: true
|
cloud: true
|
||||||
|
@ -39,11 +39,11 @@ jira:
|
||||||
update-existing: false
|
update-existing: false
|
||||||
# URL is the jira application url
|
# URL is the jira application url
|
||||||
url: https://localhost/jira
|
url: https://localhost/jira
|
||||||
# account-id is the account-id of the jira user or username in case of on-prem Jira
|
# account-id is the account-id of the Jira user or username in case of on-prem Jira
|
||||||
account-id: test-account-id
|
account-id: test-account-id
|
||||||
# email is the email of the user for jira instance
|
# email is the email of the user for Jira instance
|
||||||
email: test@test.com
|
email: test@test.com
|
||||||
# token is the token for jira instance or password in case of on-prem Jira
|
# token is the token for Jira instance or password in case of on-prem Jira
|
||||||
token: test-token
|
token: test-token
|
||||||
# project-name is the name of the project.
|
# project-name is the name of the project.
|
||||||
project-name: test-project-name
|
project-name: test-project-name
|
||||||
|
|
|
@ -5,47 +5,47 @@ allow-list:
|
||||||
deny-list:
|
deny-list:
|
||||||
severity: low
|
severity: low
|
||||||
|
|
||||||
# github contains configuration options for github issue tracker
|
# GitHub contains configuration options for GitHub issue tracker
|
||||||
github:
|
GitHub:
|
||||||
# base-url is the optional self-hosted github application url
|
# base-url is the optional self-hosted GitHub application url
|
||||||
base-url: https://localhost:8443/github
|
base-url: https://localhost:8443/GitHub
|
||||||
# username is the username of the github user
|
# username is the username of the GitHub user
|
||||||
username: test-username
|
username: test-username
|
||||||
# owner is the owner name of the repository for issues.
|
# owner is the owner name of the repository for issues.
|
||||||
owner: test-owner
|
owner: test-owner
|
||||||
# token is the token for github account.
|
# token is the token for GitHub account.
|
||||||
token: test-token
|
token: test-token
|
||||||
# project-name is the name of the repository.
|
# project-name is the name of the repository.
|
||||||
project-name: test-project
|
project-name: test-project
|
||||||
# issue-label is the label of the created issue type
|
# issue-label is the label of the created issue type
|
||||||
issue-label: bug
|
issue-label: bug
|
||||||
|
|
||||||
# gitlab contains configuration options for gitlab issue tracker
|
# GitLab contains configuration options for GitLab issue tracker
|
||||||
gitlab:
|
GitLab:
|
||||||
# base-url is the optional self-hosted gitlab application url
|
# base-url is the optional self-hosted GitLab application url
|
||||||
base-url: https://localhost:8443/gitlab
|
base-url: https://localhost:8443/GitLab
|
||||||
# username is the username of the gitlab user
|
# username is the username of the GitLab user
|
||||||
username: test-username
|
username: test-username
|
||||||
# token is the token for gitlab account.
|
# token is the token for GitLab account.
|
||||||
token: test-token
|
token: test-token
|
||||||
# project-name is the name/id of the project(repository).
|
# project-name is the name/id of the project(repository).
|
||||||
project-name: "1234"
|
project-name: "1234"
|
||||||
# issue-label is the label of the created issue type
|
# issue-label is the label of the created issue type
|
||||||
issue-label: bug
|
issue-label: bug
|
||||||
|
|
||||||
# jira contains configuration options for jira issue tracker
|
# Jira contains configuration options for Jira issue tracker
|
||||||
jira:
|
Jira:
|
||||||
# cloud is the boolean which tells if Jira instance is running in the cloud or on-prem version is used
|
# cloud is the boolean which tells if Jira instance is running in the cloud or on-prem version is used
|
||||||
cloud: true
|
cloud: true
|
||||||
# update-existing is the boolean which tells if the existing, opened issue should be updated or new one should be created
|
# update-existing is the boolean which tells if the existing, opened issue should be updated or new one should be created
|
||||||
update-existing: false
|
update-existing: false
|
||||||
# URL is the jira application url
|
# URL is the Jira application url
|
||||||
url: https://localhost/jira
|
url: https://localhost/Jira
|
||||||
# account-id is the account-id of the jira user or username in case of on-prem Jira
|
# account-id is the account-id of the Jira user or username in case of on-prem Jira
|
||||||
account-id: test-account-id
|
account-id: test-account-id
|
||||||
# email is the email of the user for jira instance
|
# email is the email of the user for Jira instance
|
||||||
email: test@test.com
|
email: test@test.com
|
||||||
# token is the token for jira instance or password in case of on-prem Jira
|
# token is the token for Jira instance or password in case of on-prem Jira
|
||||||
token: test-token
|
token: test-token
|
||||||
# project-name is the name of the project.
|
# project-name is the name of the project.
|
||||||
project-name: test-project-name
|
project-name: test-project-name
|
||||||
|
|
|
@ -10,6 +10,7 @@ import (
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"github.com/Ice3man543/nvd"
|
"github.com/Ice3man543/nvd"
|
||||||
|
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/catalog"
|
"github.com/projectdiscovery/nuclei/v2/pkg/catalog"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
@ -116,26 +117,26 @@ func getCVEData(client *nvd.Client, filePath, data string) {
|
||||||
}
|
}
|
||||||
if !strings.Contains(infoBlockClean, "classification") && (cvssScore != 0 && cvssMetrics != "") {
|
if !strings.Contains(infoBlockClean, "classification") && (cvssScore != 0 && cvssMetrics != "") {
|
||||||
changed = true
|
changed = true
|
||||||
newInfoBlock = newInfoBlock + fmt.Sprintf("\n classification:\n cvss-metrics: %s\n cvss-score: %.2f\n cve-id: %s", cvssMetrics, cvssScore, cveName)
|
newInfoBlock += fmt.Sprintf("\n classification:\n cvss-metrics: %s\n cvss-score: %.2f\n cve-id: %s", cvssMetrics, cvssScore, cveName)
|
||||||
if len(cweID) > 0 && (cweID[0] != "NVD-CWE-Other" && cweID[0] != "NVD-CWE-noinfo") {
|
if len(cweID) > 0 && (cweID[0] != "NVD-CWE-Other" && cweID[0] != "NVD-CWE-noinfo") {
|
||||||
newInfoBlock = newInfoBlock + fmt.Sprintf("\n cwe-id: %s", strings.Join(cweID, ","))
|
newInfoBlock += fmt.Sprintf("\n cwe-id: %s", strings.Join(cweID, ","))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
// If there is no description field, fill the description from CVE information
|
// If there is no description field, fill the description from CVE information
|
||||||
if !strings.Contains(infoBlockClean, "description:") && len(cveItem.CVE.Description.DescriptionData) > 0 {
|
if !strings.Contains(infoBlockClean, "description:") && len(cveItem.CVE.Description.DescriptionData) > 0 {
|
||||||
changed = true
|
changed = true
|
||||||
newInfoBlock = newInfoBlock + fmt.Sprintf("\n description: %s", fmt.Sprintf("%q", cveItem.CVE.Description.DescriptionData[0].Value))
|
newInfoBlock += fmt.Sprintf("\n description: %s", fmt.Sprintf("%q", cveItem.CVE.Description.DescriptionData[0].Value))
|
||||||
}
|
}
|
||||||
if !strings.Contains(infoBlockClean, "reference:") && len(cveItem.CVE.References.ReferenceData) > 0 {
|
if !strings.Contains(infoBlockClean, "reference:") && len(cveItem.CVE.References.ReferenceData) > 0 {
|
||||||
changed = true
|
changed = true
|
||||||
newInfoBlock = newInfoBlock + "\n reference:"
|
newInfoBlock += "\n reference:"
|
||||||
for _, reference := range cveItem.CVE.References.ReferenceData {
|
for _, reference := range cveItem.CVE.References.ReferenceData {
|
||||||
newInfoBlock = newInfoBlock + fmt.Sprintf("\n - %s", reference.URL)
|
newInfoBlock += fmt.Sprintf("\n - %s", reference.URL)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
newTemplate := strings.ReplaceAll(data, infoBlockClean, newInfoBlock)
|
newTemplate := strings.ReplaceAll(data, infoBlockClean, newInfoBlock)
|
||||||
if changed {
|
if changed {
|
||||||
_ = ioutil.WriteFile(filePath, []byte(newTemplate), 0777)
|
_ = ioutil.WriteFile(filePath, []byte(newTemplate), 0644)
|
||||||
fmt.Printf("Wrote updated template to %s\n", filePath)
|
fmt.Printf("Wrote updated template to %s\n", filePath)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
@ -10,10 +10,11 @@ import (
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"github.com/alecthomas/jsonschema"
|
"github.com/alecthomas/jsonschema"
|
||||||
|
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/templates"
|
"github.com/projectdiscovery/nuclei/v2/pkg/templates"
|
||||||
)
|
)
|
||||||
|
|
||||||
var pathRegex = regexp.MustCompile(`github.com/projectdiscovery/nuclei/v2/(?:internal|pkg)/(?:.*/)?([A-Za-z\.]+)`)
|
var pathRegex = regexp.MustCompile(`github\.com/projectdiscovery/nuclei/v2/(?:internal|pkg)/(?:.*/)?([A-Za-z.]+)`)
|
||||||
|
|
||||||
func main() {
|
func main() {
|
||||||
// Generate yaml syntax documentation
|
// Generate yaml syntax documentation
|
||||||
|
@ -21,7 +22,7 @@ func main() {
|
||||||
if err != nil {
|
if err != nil {
|
||||||
log.Fatalf("Could not encode docs: %s\n", err)
|
log.Fatalf("Could not encode docs: %s\n", err)
|
||||||
}
|
}
|
||||||
err = ioutil.WriteFile(os.Args[1], data, 0777)
|
err = ioutil.WriteFile(os.Args[1], data, 0644)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
log.Fatalf("Could not write docs: %s\n", err)
|
log.Fatalf("Could not write docs: %s\n", err)
|
||||||
}
|
}
|
||||||
|
@ -43,7 +44,7 @@ func main() {
|
||||||
for _, match := range pathRegex.FindAllStringSubmatch(schema, -1) {
|
for _, match := range pathRegex.FindAllStringSubmatch(schema, -1) {
|
||||||
schema = strings.ReplaceAll(schema, match[0], match[1])
|
schema = strings.ReplaceAll(schema, match[0], match[1])
|
||||||
}
|
}
|
||||||
err = ioutil.WriteFile(os.Args[2], []byte(schema), 0777)
|
err = ioutil.WriteFile(os.Args[2], []byte(schema), 0644)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
log.Fatalf("Could not write jsonschema: %s\n", err)
|
log.Fatalf("Could not write jsonschema: %s\n", err)
|
||||||
}
|
}
|
||||||
|
|
|
@ -40,17 +40,17 @@ var httpTestcases = map[string]testutils.TestCase{
|
||||||
|
|
||||||
type httpInteractshRequest struct{}
|
type httpInteractshRequest struct{}
|
||||||
|
|
||||||
// Executes executes a test case and returns an error if occurred
|
// Execute executes a test case and returns an error if occurred
|
||||||
func (h *httpInteractshRequest) Execute(filePath string) error {
|
func (h *httpInteractshRequest) Execute(filePath string) error {
|
||||||
router := httprouter.New()
|
router := httprouter.New()
|
||||||
router.GET("/", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
|
router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
|
||||||
value := r.Header.Get("url")
|
value := r.Header.Get("url")
|
||||||
if value != "" {
|
if value != "" {
|
||||||
if resp, _ := http.DefaultClient.Get(value); resp != nil {
|
if resp, _ := http.DefaultClient.Get(value); resp != nil {
|
||||||
resp.Body.Close()
|
resp.Body.Close()
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}))
|
})
|
||||||
ts := httptest.NewServer(router)
|
ts := httptest.NewServer(router)
|
||||||
defer ts.Close()
|
defer ts.Close()
|
||||||
|
|
||||||
|
|
|
@ -6,6 +6,7 @@ import (
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"github.com/logrusorgru/aurora"
|
"github.com/logrusorgru/aurora"
|
||||||
|
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
|
"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
|
@ -8,6 +8,7 @@ import (
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"github.com/julienschmidt/httprouter"
|
"github.com/julienschmidt/httprouter"
|
||||||
|
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
|
"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
@ -24,14 +25,14 @@ type remoteTemplateList struct{}
|
||||||
func (h *remoteTemplateList) Execute(templateList string) error {
|
func (h *remoteTemplateList) Execute(templateList string) error {
|
||||||
router := httprouter.New()
|
router := httprouter.New()
|
||||||
|
|
||||||
router.GET("/", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
|
router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
|
||||||
fmt.Fprintf(w, "This is test matcher text")
|
fmt.Fprintf(w, "This is test matcher text")
|
||||||
if strings.EqualFold(r.Header.Get("test"), "nuclei") {
|
if strings.EqualFold(r.Header.Get("test"), "nuclei") {
|
||||||
fmt.Fprintf(w, "This is test headers matcher text")
|
fmt.Fprintf(w, "This is test headers matcher text")
|
||||||
}
|
}
|
||||||
}))
|
})
|
||||||
|
|
||||||
router.GET("/template_list", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
|
router.GET("/template_list", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
|
||||||
file, err := os.ReadFile(templateList)
|
file, err := os.ReadFile(templateList)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
w.WriteHeader(500)
|
w.WriteHeader(500)
|
||||||
|
@ -40,7 +41,7 @@ func (h *remoteTemplateList) Execute(templateList string) error {
|
||||||
if err != nil {
|
if err != nil {
|
||||||
w.WriteHeader(500)
|
w.WriteHeader(500)
|
||||||
}
|
}
|
||||||
}))
|
})
|
||||||
ts := httptest.NewServer(router)
|
ts := httptest.NewServer(router)
|
||||||
defer ts.Close()
|
defer ts.Close()
|
||||||
|
|
||||||
|
@ -60,14 +61,14 @@ type remoteWorkflowList struct{}
|
||||||
func (h *remoteWorkflowList) Execute(workflowList string) error {
|
func (h *remoteWorkflowList) Execute(workflowList string) error {
|
||||||
router := httprouter.New()
|
router := httprouter.New()
|
||||||
|
|
||||||
router.GET("/", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
|
router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
|
||||||
fmt.Fprintf(w, "This is test matcher text")
|
fmt.Fprintf(w, "This is test matcher text")
|
||||||
if strings.EqualFold(r.Header.Get("test"), "nuclei") {
|
if strings.EqualFold(r.Header.Get("test"), "nuclei") {
|
||||||
fmt.Fprintf(w, "This is test headers matcher text")
|
fmt.Fprintf(w, "This is test headers matcher text")
|
||||||
}
|
}
|
||||||
}))
|
})
|
||||||
|
|
||||||
router.GET("/workflow_list", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
|
router.GET("/workflow_list", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
|
||||||
file, err := os.ReadFile(workflowList)
|
file, err := os.ReadFile(workflowList)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
w.WriteHeader(500)
|
w.WriteHeader(500)
|
||||||
|
@ -76,7 +77,7 @@ func (h *remoteWorkflowList) Execute(workflowList string) error {
|
||||||
if err != nil {
|
if err != nil {
|
||||||
w.WriteHeader(500)
|
w.WriteHeader(500)
|
||||||
}
|
}
|
||||||
}))
|
})
|
||||||
ts := httptest.NewServer(router)
|
ts := httptest.NewServer(router)
|
||||||
defer ts.Close()
|
defer ts.Close()
|
||||||
|
|
||||||
|
|
|
@ -5,6 +5,7 @@ import (
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"github.com/gobwas/ws/wsutil"
|
"github.com/gobwas/ws/wsutil"
|
||||||
|
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
|
"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
@ -22,7 +23,7 @@ func (h *websocketBasic) Execute(filePath string) error {
|
||||||
connHandler := func(conn net.Conn) {
|
connHandler := func(conn net.Conn) {
|
||||||
for {
|
for {
|
||||||
msg, op, _ := wsutil.ReadClientData(conn)
|
msg, op, _ := wsutil.ReadClientData(conn)
|
||||||
if string(msg) != string("hello") {
|
if string(msg) != "hello" {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
_ = wsutil.WriteServerMessage(conn, op, []byte("world"))
|
_ = wsutil.WriteServerMessage(conn, op, []byte("world"))
|
||||||
|
|
|
@ -5,51 +5,51 @@
|
||||||
#deny-list:
|
#deny-list:
|
||||||
# severity: info, low, medium
|
# severity: info, low, medium
|
||||||
|
|
||||||
# github contains configuration options for github issue tracker
|
# GitHub contains configuration options for GitHub issue tracker
|
||||||
#github:
|
#GitHub:
|
||||||
# # base-url (optional) is the self-hosted github application url
|
# # base-url (optional) is the self-hosted GitHub application url
|
||||||
# base-url: ""
|
# base-url: ""
|
||||||
# # username is the username of the github user
|
# # username is the username of the GitHub user
|
||||||
# username: ""
|
# username: ""
|
||||||
# # owner is the owner name of the repository for issues.
|
# # owner is the owner name of the repository for issues.
|
||||||
# owner: ""
|
# owner: ""
|
||||||
# # token is the token for github account.
|
# # token is the token for GitHub account.
|
||||||
# token: ""
|
# token: ""
|
||||||
# # project-name is the name of the repository.
|
# # project-name is the name of the repository.
|
||||||
# project-name: ""
|
# project-name: ""
|
||||||
# # issue-label (optional) is the label of the created issue type
|
# # issue-label (optional) is the label of the created issue type
|
||||||
# issue-label: ""
|
# issue-label: ""
|
||||||
# # severity-as-label (optional) sets the sevetiry as the label of the created issue type
|
# # severity-as-label (optional) sets the severity as the label of the created issue type
|
||||||
# severity-as-label: false
|
# severity-as-label: false
|
||||||
|
|
||||||
# gitlab contains configuration options for gitlab issue tracker
|
# GitLab contains configuration options for GitLab issue tracker
|
||||||
#gitlab:
|
#GitLab:
|
||||||
# # base-url (optional) is the self-hosted gitlab application url
|
# # base-url (optional) is the self-hosted GitLab application url
|
||||||
# base-url: ""
|
# base-url: ""
|
||||||
# # username is the username of the gitlab user
|
# # username is the username of the GitLab user
|
||||||
# username: ""
|
# username: ""
|
||||||
# # token is the token for gitlab account.
|
# # token is the token for GitLab account.
|
||||||
# token: ""
|
# token: ""
|
||||||
# # project-id is the ID of the repository.
|
# # project-id is the ID of the repository.
|
||||||
# project-id: ""
|
# project-id: ""
|
||||||
# # issue-label (optional) is the label of the created issue type
|
# # issue-label (optional) is the label of the created issue type
|
||||||
# issue-label: ""
|
# issue-label: ""
|
||||||
# # severity-as-label (optional) sets the sevetiry as the label of the created issue type
|
# # severity-as-label (optional) sets the severity as the label of the created issue type
|
||||||
# severity-as-label: false
|
# severity-as-label: false
|
||||||
|
|
||||||
# jira contains configuration options for jira issue tracker
|
# Jira contains configuration options for Jira issue tracker
|
||||||
#jira:
|
#Jira:
|
||||||
# # cloud (optional) is the boolean which tells if Jira instance is running in the cloud or on-prem version is used
|
# # cloud (optional) is the boolean which tells if Jira instance is running in the cloud or on-prem version is used
|
||||||
# cloud: true
|
# cloud: true
|
||||||
# # update-existing (optional) is the boolean which tells if the existing, opened issue should be updated or new one should be created
|
# # update-existing (optional) is the boolean which tells if the existing, opened issue should be updated or new one should be created
|
||||||
# update-existing: false
|
# update-existing: false
|
||||||
# # URL is the jira application url
|
# # URL is the Jira application URL
|
||||||
# url: ""
|
# url: ""
|
||||||
# # account-id is the account-id of the jira user or username in case of on-prem Jira
|
# # account-id is the account-id of the Jira user or username in case of on-prem Jira
|
||||||
# account-id: ""
|
# account-id: ""
|
||||||
# # email is the email of the user for jira instance
|
# # email is the email of the user for Jira instance
|
||||||
# email: ""
|
# email: ""
|
||||||
# # token is the token for jira instance or password in case of on-prem Jira
|
# # token is the token for Jira instance or password in case of on-prem Jira
|
||||||
# token: ""
|
# token: ""
|
||||||
# # project-name is the name of the project.
|
# # project-name is the name of the project.
|
||||||
# project-name: ""
|
# project-name: ""
|
||||||
|
|
|
@ -110,7 +110,7 @@ on extensive configurability, massive extensibility and ease of use.`)
|
||||||
flagSet.IntVar(&options.InteractionsCacheSize, "interactions-cache-size", 5000, "number of requests to keep in the interactions cache"),
|
flagSet.IntVar(&options.InteractionsCacheSize, "interactions-cache-size", 5000, "number of requests to keep in the interactions cache"),
|
||||||
flagSet.IntVar(&options.InteractionsEviction, "interactions-eviction", 60, "number of seconds to wait before evicting requests from cache"),
|
flagSet.IntVar(&options.InteractionsEviction, "interactions-eviction", 60, "number of seconds to wait before evicting requests from cache"),
|
||||||
flagSet.IntVar(&options.InteractionsPollDuration, "interactions-poll-duration", 5, "number of seconds to wait before each interaction poll request"),
|
flagSet.IntVar(&options.InteractionsPollDuration, "interactions-poll-duration", 5, "number of seconds to wait before each interaction poll request"),
|
||||||
flagSet.IntVar(&options.InteractionsCooldownPeriod, "interactions-cooldown-period", 5, "extra time for interaction polling before exiting"),
|
flagSet.IntVar(&options.InteractionsCoolDownPeriod, "interactions-cooldown-period", 5, "extra time for interaction polling before exiting"),
|
||||||
flagSet.BoolVarP(&options.NoInteractsh, "no-interactsh", "ni", false, "disable interactsh server for OAST testing, exclude OAST based templates"),
|
flagSet.BoolVarP(&options.NoInteractsh, "no-interactsh", "ni", false, "disable interactsh server for OAST testing, exclude OAST based templates"),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
|
@ -9,6 +9,7 @@ import (
|
||||||
"github.com/pkg/errors"
|
"github.com/pkg/errors"
|
||||||
|
|
||||||
"github.com/go-playground/validator/v10"
|
"github.com/go-playground/validator/v10"
|
||||||
|
|
||||||
"github.com/projectdiscovery/fileutil"
|
"github.com/projectdiscovery/fileutil"
|
||||||
"github.com/projectdiscovery/gologger"
|
"github.com/projectdiscovery/gologger"
|
||||||
"github.com/projectdiscovery/gologger/formatter"
|
"github.com/projectdiscovery/gologger/formatter"
|
||||||
|
@ -66,15 +67,14 @@ func ParseOptions(options *types.Options) {
|
||||||
|
|
||||||
// hasStdin returns true if we have stdin input
|
// hasStdin returns true if we have stdin input
|
||||||
func hasStdin() bool {
|
func hasStdin() bool {
|
||||||
stat, err := os.Stdin.Stat()
|
fi, err := os.Stdin.Stat()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
|
if fi.Mode()&os.ModeNamedPipe == 0 {
|
||||||
isPipedFromChrDev := (stat.Mode() & os.ModeCharDevice) == 0
|
return false
|
||||||
isPipedFromFIFO := (stat.Mode() & os.ModeNamedPipe) != 0
|
}
|
||||||
|
return true
|
||||||
return isPipedFromChrDev || isPipedFromFIFO
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// validateOptions validates the configuration options passed
|
// validateOptions validates the configuration options passed
|
||||||
|
|
|
@ -28,7 +28,7 @@ func loadProxyServers(options *types.Options) error {
|
||||||
} else if fileutil.FileExists(p) {
|
} else if fileutil.FileExists(p) {
|
||||||
file, err := os.Open(p)
|
file, err := os.Open(p)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("could not open proxy file: %s", err)
|
return fmt.Errorf("could not open proxy file: %w", err)
|
||||||
}
|
}
|
||||||
defer file.Close()
|
defer file.Close()
|
||||||
scanner := bufio.NewScanner(file)
|
scanner := bufio.NewScanner(file)
|
||||||
|
|
|
@ -151,7 +151,7 @@ func New(options *types.Options) (*Runner, error) {
|
||||||
opts.Authorization = options.InteractshToken
|
opts.Authorization = options.InteractshToken
|
||||||
opts.CacheSize = int64(options.InteractionsCacheSize)
|
opts.CacheSize = int64(options.InteractionsCacheSize)
|
||||||
opts.Eviction = time.Duration(options.InteractionsEviction) * time.Second
|
opts.Eviction = time.Duration(options.InteractionsEviction) * time.Second
|
||||||
opts.ColldownPeriod = time.Duration(options.InteractionsCooldownPeriod) * time.Second
|
opts.ColldownPeriod = time.Duration(options.InteractionsCoolDownPeriod) * time.Second
|
||||||
opts.PollDuration = time.Duration(options.InteractionsPollDuration) * time.Second
|
opts.PollDuration = time.Duration(options.InteractionsPollDuration) * time.Second
|
||||||
opts.NoInteractsh = runner.options.NoInteractsh
|
opts.NoInteractsh = runner.options.NoInteractsh
|
||||||
|
|
||||||
|
|
|
@ -54,7 +54,7 @@ func (r *Runner) updateTemplates() error { // TODO this method does more than ju
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
configDir := filepath.Join(home, ".config", "nuclei")
|
configDir := filepath.Join(home, ".config", "nuclei")
|
||||||
_ = os.MkdirAll(configDir, os.ModePerm)
|
_ = os.MkdirAll(configDir, 0755)
|
||||||
|
|
||||||
if err := r.readInternalConfigurationFile(home, configDir); err != nil {
|
if err := r.readInternalConfigurationFile(home, configDir); err != nil {
|
||||||
return errors.Wrap(err, "could not read configuration file")
|
return errors.Wrap(err, "could not read configuration file")
|
||||||
|
@ -242,12 +242,12 @@ func (r *Runner) getLatestReleaseFromGithub(latestTag string) (*github.Repositor
|
||||||
func (r *Runner) downloadReleaseAndUnzip(ctx context.Context, version, downloadURL string) (*templateUpdateResults, error) {
|
func (r *Runner) downloadReleaseAndUnzip(ctx context.Context, version, downloadURL string) (*templateUpdateResults, error) {
|
||||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, downloadURL, nil)
|
req, err := http.NewRequestWithContext(ctx, http.MethodGet, downloadURL, nil)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("failed to create HTTP request to %s: %s", downloadURL, err)
|
return nil, fmt.Errorf("failed to create HTTP request to %s: %w", downloadURL, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
res, err := http.DefaultClient.Do(req)
|
res, err := http.DefaultClient.Do(req)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("failed to download a release file from %s: %s", downloadURL, err)
|
return nil, fmt.Errorf("failed to download a release file from %s: %w", downloadURL, err)
|
||||||
}
|
}
|
||||||
defer res.Body.Close()
|
defer res.Body.Close()
|
||||||
if res.StatusCode != http.StatusOK {
|
if res.StatusCode != http.StatusOK {
|
||||||
|
@ -256,23 +256,23 @@ func (r *Runner) downloadReleaseAndUnzip(ctx context.Context, version, downloadU
|
||||||
|
|
||||||
buf, err := ioutil.ReadAll(res.Body)
|
buf, err := ioutil.ReadAll(res.Body)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("failed to create buffer for zip file: %s", err)
|
return nil, fmt.Errorf("failed to create buffer for zip file: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
reader := bytes.NewReader(buf)
|
reader := bytes.NewReader(buf)
|
||||||
zipReader, err := zip.NewReader(reader, reader.Size())
|
zipReader, err := zip.NewReader(reader, reader.Size())
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("failed to uncompress zip file: %s", err)
|
return nil, fmt.Errorf("failed to uncompress zip file: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Create the template folder if it doesn't exist
|
// Create the template folder if it doesn't exist
|
||||||
if err := os.MkdirAll(r.templatesConfig.TemplatesDirectory, os.ModePerm); err != nil {
|
if err := os.MkdirAll(r.templatesConfig.TemplatesDirectory, 0755); err != nil {
|
||||||
return nil, fmt.Errorf("failed to create template base folder: %s", err)
|
return nil, fmt.Errorf("failed to create template base folder: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
results, err := r.compareAndWriteTemplates(zipReader)
|
results, err := r.compareAndWriteTemplates(zipReader)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("failed to write templates: %s", err)
|
return nil, fmt.Errorf("failed to write templates: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
if r.options.Verbose {
|
if r.options.Verbose {
|
||||||
|
@ -291,7 +291,7 @@ func (r *Runner) downloadReleaseAndUnzip(ctx context.Context, version, downloadU
|
||||||
buffer.WriteString("\n")
|
buffer.WriteString("\n")
|
||||||
}
|
}
|
||||||
|
|
||||||
if err := ioutil.WriteFile(additionsFile, buffer.Bytes(), os.ModePerm); err != nil {
|
if err := ioutil.WriteFile(additionsFile, buffer.Bytes(), 0644); err != nil {
|
||||||
return nil, errors.Wrap(err, "could not write new additions file")
|
return nil, errors.Wrap(err, "could not write new additions file")
|
||||||
}
|
}
|
||||||
return results, err
|
return results, err
|
||||||
|
@ -316,60 +316,42 @@ func (r *Runner) compareAndWriteTemplates(zipReader *zip.Reader) (*templateUpdat
|
||||||
// If the path isn't found in new update after being read from the previous checksum,
|
// If the path isn't found in new update after being read from the previous checksum,
|
||||||
// it is removed. This allows us fine-grained control over the download process
|
// it is removed. This allows us fine-grained control over the download process
|
||||||
// as well as solves a long problem with nuclei-template updates.
|
// as well as solves a long problem with nuclei-template updates.
|
||||||
checksumFile := filepath.Join(r.templatesConfig.TemplatesDirectory, ".checksum")
|
configuredTemplateDirectory := r.templatesConfig.TemplatesDirectory
|
||||||
|
checksumFile := filepath.Join(configuredTemplateDirectory, ".checksum")
|
||||||
templateChecksumsMap, _ := createTemplateChecksumsMap(checksumFile)
|
templateChecksumsMap, _ := createTemplateChecksumsMap(checksumFile)
|
||||||
for _, zipTemplateFile := range zipReader.File {
|
for _, zipTemplateFile := range zipReader.File {
|
||||||
directory, name := filepath.Split(zipTemplateFile.Name)
|
templateAbsolutePath, skipFile, err := calculateTemplateAbsolutePath(zipTemplateFile.Name, configuredTemplateDirectory)
|
||||||
if name == "" {
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if skipFile {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
paths := strings.Split(directory, string(os.PathSeparator))
|
|
||||||
finalPath := filepath.Join(paths[1:]...)
|
|
||||||
|
|
||||||
if strings.HasPrefix(name, ".") || strings.HasPrefix(finalPath, ".") || strings.EqualFold(name, "README.md") {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
results.totalCount++
|
|
||||||
templateDirectory := filepath.Join(r.templatesConfig.TemplatesDirectory, finalPath)
|
|
||||||
if err := os.MkdirAll(templateDirectory, os.ModePerm); err != nil {
|
|
||||||
return nil, fmt.Errorf("failed to create template folder %s : %s", templateDirectory, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
templatePath := filepath.Join(templateDirectory, name)
|
|
||||||
|
|
||||||
isAddition := false
|
isAddition := false
|
||||||
if _, statErr := os.Stat(templatePath); os.IsNotExist(statErr) {
|
if _, statErr := os.Stat(templateAbsolutePath); os.IsNotExist(statErr) {
|
||||||
isAddition = true
|
isAddition = true
|
||||||
}
|
}
|
||||||
templateFile, err := os.OpenFile(templatePath, os.O_TRUNC|os.O_CREATE|os.O_WRONLY, 0777)
|
|
||||||
|
newTemplateChecksum, err := writeUnZippedTemplateFile(templateAbsolutePath, zipTemplateFile)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
templateFile.Close()
|
return nil, err
|
||||||
return nil, fmt.Errorf("could not create uncompressed file: %s", err)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
zipTemplateFileReader, err := zipTemplateFile.Open()
|
oldTemplateChecksum, checksumOk := templateChecksumsMap[templateAbsolutePath]
|
||||||
|
|
||||||
|
relativeTemplatePath, err := filepath.Rel(configuredTemplateDirectory, templateAbsolutePath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
templateFile.Close()
|
return nil, fmt.Errorf("could not calculate relative path for template: %s. %w", templateAbsolutePath, err)
|
||||||
return nil, fmt.Errorf("could not open archive to extract file: %s", err)
|
|
||||||
}
|
}
|
||||||
hasher := md5.New()
|
|
||||||
|
|
||||||
// Save file and also read into hasher for md5
|
|
||||||
if _, err := io.Copy(templateFile, io.TeeReader(zipTemplateFileReader, hasher)); err != nil {
|
|
||||||
templateFile.Close()
|
|
||||||
return nil, fmt.Errorf("could not write template file: %s", err)
|
|
||||||
}
|
|
||||||
templateFile.Close()
|
|
||||||
|
|
||||||
oldChecksum, checksumOK := templateChecksumsMap[templatePath]
|
|
||||||
|
|
||||||
checksum := hex.EncodeToString(hasher.Sum(nil))
|
|
||||||
if isAddition {
|
if isAddition {
|
||||||
results.additions = append(results.additions, filepath.Join(finalPath, name))
|
results.additions = append(results.additions, relativeTemplatePath)
|
||||||
} else if checksumOK && oldChecksum[0] != checksum {
|
} else if checksumOk && oldTemplateChecksum[0] != newTemplateChecksum {
|
||||||
results.modifications = append(results.modifications, filepath.Join(finalPath, name))
|
results.modifications = append(results.modifications, relativeTemplatePath)
|
||||||
}
|
}
|
||||||
results.checksums[templatePath] = checksum
|
results.checksums[templateAbsolutePath] = newTemplateChecksum
|
||||||
|
results.totalCount++
|
||||||
}
|
}
|
||||||
|
|
||||||
// If we don't find the previous file in the newly downloaded list,
|
// If we don't find the previous file in the newly downloaded list,
|
||||||
|
@ -378,12 +360,63 @@ func (r *Runner) compareAndWriteTemplates(zipReader *zip.Reader) (*templateUpdat
|
||||||
_, ok := results.checksums[templatePath]
|
_, ok := results.checksums[templatePath]
|
||||||
if !ok && templateChecksums[0] == templateChecksums[1] {
|
if !ok && templateChecksums[0] == templateChecksums[1] {
|
||||||
_ = os.Remove(templatePath)
|
_ = os.Remove(templatePath)
|
||||||
results.deletions = append(results.deletions, strings.TrimPrefix(strings.TrimPrefix(templatePath, r.templatesConfig.TemplatesDirectory), string(os.PathSeparator)))
|
results.deletions = append(results.deletions, strings.TrimPrefix(strings.TrimPrefix(templatePath, configuredTemplateDirectory), string(os.PathSeparator)))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return results, nil
|
return results, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func writeUnZippedTemplateFile(templateAbsolutePath string, zipTemplateFile *zip.File) (string, error) {
|
||||||
|
templateFile, err := os.OpenFile(templateAbsolutePath, os.O_TRUNC|os.O_CREATE|os.O_WRONLY, 0644)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("could not create template file: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
zipTemplateFileReader, err := zipTemplateFile.Open()
|
||||||
|
if err != nil {
|
||||||
|
_ = templateFile.Close()
|
||||||
|
return "", fmt.Errorf("could not open archive to extract file: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
md5Hash := md5.New()
|
||||||
|
|
||||||
|
// Save file and also read into hash.Hash for md5
|
||||||
|
if _, err := io.Copy(templateFile, io.TeeReader(zipTemplateFileReader, md5Hash)); err != nil {
|
||||||
|
_ = templateFile.Close()
|
||||||
|
return "", fmt.Errorf("could not write template file: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := templateFile.Close(); err != nil {
|
||||||
|
return "", fmt.Errorf("could not close file newly created template file: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
checksum := hex.EncodeToString(md5Hash.Sum(nil))
|
||||||
|
return checksum, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func calculateTemplateAbsolutePath(zipFilePath, configuredTemplateDirectory string) (string, bool, error) {
|
||||||
|
directory, fileName := filepath.Split(zipFilePath)
|
||||||
|
|
||||||
|
if strings.TrimSpace(fileName) == "" || strings.HasPrefix(fileName, ".") || strings.EqualFold(fileName, "README.md") {
|
||||||
|
return "", true, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
directoryPathChunks := strings.Split(directory, string(os.PathSeparator))
|
||||||
|
relativeDirectoryPathWithoutZipRoot := filepath.Join(directoryPathChunks[1:]...)
|
||||||
|
|
||||||
|
if strings.HasPrefix(relativeDirectoryPathWithoutZipRoot, ".") {
|
||||||
|
return "", true, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
templateDirectory := filepath.Join(configuredTemplateDirectory, relativeDirectoryPathWithoutZipRoot)
|
||||||
|
|
||||||
|
if err := os.MkdirAll(templateDirectory, 0755); err != nil {
|
||||||
|
return "", false, fmt.Errorf("failed to create template folder: %s. %w", templateDirectory, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return filepath.Join(templateDirectory, fileName), false, nil
|
||||||
|
}
|
||||||
|
|
||||||
// createTemplateChecksumsMap reads the previous checksum file from the disk.
|
// createTemplateChecksumsMap reads the previous checksum file from the disk.
|
||||||
// Creates a map of template paths and their previous and currently calculated checksums as values.
|
// Creates a map of template paths and their previous and currently calculated checksums as values.
|
||||||
func createTemplateChecksumsMap(checksumsFilePath string) (map[string][2]string, error) {
|
func createTemplateChecksumsMap(checksumsFilePath string) (map[string][2]string, error) {
|
||||||
|
|
|
@ -12,10 +12,11 @@ import (
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
|
||||||
"github.com/projectdiscovery/gologger"
|
"github.com/projectdiscovery/gologger"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/catalog/config"
|
"github.com/projectdiscovery/nuclei/v2/pkg/catalog/config"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
|
"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestDownloadReleaseAndUnzipAddition(t *testing.T) {
|
func TestDownloadReleaseAndUnzipAddition(t *testing.T) {
|
||||||
|
@ -25,7 +26,7 @@ func TestDownloadReleaseAndUnzipAddition(t *testing.T) {
|
||||||
require.Nil(t, err, "could not create temp directory")
|
require.Nil(t, err, "could not create temp directory")
|
||||||
defer os.RemoveAll(baseTemplates)
|
defer os.RemoveAll(baseTemplates)
|
||||||
|
|
||||||
err = ioutil.WriteFile(filepath.Join(baseTemplates, "base.yaml"), []byte("id: test"), 0777)
|
err = ioutil.WriteFile(filepath.Join(baseTemplates, "base.yaml"), []byte("id: test"), os.ModePerm)
|
||||||
require.Nil(t, err, "could not create write base file")
|
require.Nil(t, err, "could not create write base file")
|
||||||
|
|
||||||
err = zipFromDirectory("base.zip", baseTemplates)
|
err = zipFromDirectory("base.zip", baseTemplates)
|
||||||
|
@ -50,9 +51,9 @@ func TestDownloadReleaseAndUnzipAddition(t *testing.T) {
|
||||||
require.Nil(t, err, "could not create temp directory")
|
require.Nil(t, err, "could not create temp directory")
|
||||||
defer os.RemoveAll(newTempDir)
|
defer os.RemoveAll(newTempDir)
|
||||||
|
|
||||||
err = ioutil.WriteFile(filepath.Join(newTempDir, "base.yaml"), []byte("id: test"), 0777)
|
err = ioutil.WriteFile(filepath.Join(newTempDir, "base.yaml"), []byte("id: test"), os.ModePerm)
|
||||||
require.Nil(t, err, "could not create base file")
|
require.Nil(t, err, "could not create base file")
|
||||||
err = ioutil.WriteFile(filepath.Join(newTempDir, "new.yaml"), []byte("id: test"), 0777)
|
err = ioutil.WriteFile(filepath.Join(newTempDir, "new.yaml"), []byte("id: test"), os.ModePerm)
|
||||||
require.Nil(t, err, "could not create new file")
|
require.Nil(t, err, "could not create new file")
|
||||||
|
|
||||||
err = zipFromDirectory("new.zip", newTempDir)
|
err = zipFromDirectory("new.zip", newTempDir)
|
||||||
|
@ -77,7 +78,7 @@ func TestDownloadReleaseAndUnzipDeletion(t *testing.T) {
|
||||||
require.Nil(t, err, "could not create temp directory")
|
require.Nil(t, err, "could not create temp directory")
|
||||||
defer os.RemoveAll(baseTemplates)
|
defer os.RemoveAll(baseTemplates)
|
||||||
|
|
||||||
err = ioutil.WriteFile(filepath.Join(baseTemplates, "base.yaml"), []byte("id: test"), 0777)
|
err = ioutil.WriteFile(filepath.Join(baseTemplates, "base.yaml"), []byte("id: test"), os.ModePerm)
|
||||||
require.Nil(t, err, "could not create write base file")
|
require.Nil(t, err, "could not create write base file")
|
||||||
|
|
||||||
err = zipFromDirectory("base.zip", baseTemplates)
|
err = zipFromDirectory("base.zip", baseTemplates)
|
||||||
|
@ -118,6 +119,43 @@ func TestDownloadReleaseAndUnzipDeletion(t *testing.T) {
|
||||||
require.Equal(t, "base.yaml", results.deletions[0], "could not get correct new deletions")
|
require.Equal(t, "base.yaml", results.deletions[0], "could not get correct new deletions")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestCalculateTemplateAbsolutePath(t *testing.T) {
|
||||||
|
configuredTemplateDirectory := filepath.Join(os.TempDir(), "templates")
|
||||||
|
defer os.RemoveAll(configuredTemplateDirectory)
|
||||||
|
|
||||||
|
t.Run("positive scenarios", func(t *testing.T) {
|
||||||
|
zipFilePathsExpectedPathsMap := map[string]string{
|
||||||
|
"nuclei-templates/cve/test.yaml": filepath.Join(configuredTemplateDirectory, "cve/test.yaml"),
|
||||||
|
"nuclei-templates/cve/test/test.yaml": filepath.Join(configuredTemplateDirectory, "cve/test/test.yaml"),
|
||||||
|
}
|
||||||
|
|
||||||
|
for filePathFromZip, expectedTemplateAbsPath := range zipFilePathsExpectedPathsMap {
|
||||||
|
calculatedTemplateAbsPath, skipFile, err := calculateTemplateAbsolutePath(filePathFromZip, configuredTemplateDirectory)
|
||||||
|
require.Nil(t, err)
|
||||||
|
require.Equal(t, expectedTemplateAbsPath, calculatedTemplateAbsPath)
|
||||||
|
require.False(t, skipFile)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("negative scenarios", func(t *testing.T) {
|
||||||
|
filePathsFromZip := []string{
|
||||||
|
"./../nuclei-templates/../cve/test.yaml",
|
||||||
|
"nuclei-templates/../cve/test.yaml",
|
||||||
|
"nuclei-templates/cve/../test.yaml",
|
||||||
|
"nuclei-templates/././../cve/test.yaml",
|
||||||
|
"nuclei-templates/.././../cve/test.yaml",
|
||||||
|
"nuclei-templates/.././../cve/../test.yaml",
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, filePathFromZip := range filePathsFromZip {
|
||||||
|
calculatedTemplateAbsPath, skipFile, err := calculateTemplateAbsolutePath(filePathFromZip, configuredTemplateDirectory)
|
||||||
|
require.Nil(t, err)
|
||||||
|
require.True(t, skipFile)
|
||||||
|
require.Equal(t, "", calculatedTemplateAbsPath)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
func zipFromDirectory(zipPath, directory string) error {
|
func zipFromDirectory(zipPath, directory string) error {
|
||||||
file, err := os.Create(zipPath)
|
file, err := os.Create(zipPath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|
|
@ -34,7 +34,7 @@ func getConfigDetails() (string, error) {
|
||||||
return "", errors.Wrap(err, "could not get home directory")
|
return "", errors.Wrap(err, "could not get home directory")
|
||||||
}
|
}
|
||||||
configDir := filepath.Join(homeDir, ".config", "nuclei")
|
configDir := filepath.Join(homeDir, ".config", "nuclei")
|
||||||
_ = os.MkdirAll(configDir, os.ModePerm)
|
_ = os.MkdirAll(configDir, 0755)
|
||||||
templatesConfigFile := filepath.Join(configDir, nucleiConfigFilename)
|
templatesConfigFile := filepath.Join(configDir, nucleiConfigFilename)
|
||||||
return templatesConfigFile, nil
|
return templatesConfigFile, nil
|
||||||
}
|
}
|
||||||
|
@ -67,7 +67,7 @@ func WriteConfiguration(config *Config) error {
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
file, err := os.OpenFile(templatesConfigFile, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0777)
|
file, err := os.OpenFile(templatesConfigFile, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
@ -112,7 +112,7 @@ func getIgnoreFilePath() string {
|
||||||
home, err := os.UserHomeDir()
|
home, err := os.UserHomeDir()
|
||||||
if err == nil {
|
if err == nil {
|
||||||
configDir := filepath.Join(home, ".config", "nuclei")
|
configDir := filepath.Join(home, ".config", "nuclei")
|
||||||
_ = os.MkdirAll(configDir, os.ModePerm)
|
_ = os.MkdirAll(configDir, 0755)
|
||||||
|
|
||||||
defIgnoreFilePath = filepath.Join(configDir, nucleiIgnoreFile)
|
defIgnoreFilePath = filepath.Join(configDir, nucleiIgnoreFile)
|
||||||
return defIgnoreFilePath
|
return defIgnoreFilePath
|
||||||
|
|
|
@ -7,6 +7,7 @@ import (
|
||||||
|
|
||||||
"github.com/karrick/godirwalk"
|
"github.com/karrick/godirwalk"
|
||||||
"github.com/pkg/errors"
|
"github.com/pkg/errors"
|
||||||
|
|
||||||
"github.com/projectdiscovery/gologger"
|
"github.com/projectdiscovery/gologger"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
@ -79,7 +80,7 @@ func (c *Catalog) GetTemplatePath(target string) ([]string, error) {
|
||||||
}
|
}
|
||||||
|
|
||||||
// convertPathToAbsolute resolves the paths provided to absolute paths
|
// convertPathToAbsolute resolves the paths provided to absolute paths
|
||||||
// before doing any operations on them regardless of them being blob, folders, files, etc.
|
// before doing any operations on them regardless of them being BLOB, folders, files, etc.
|
||||||
func (c *Catalog) convertPathToAbsolute(t string) (string, error) {
|
func (c *Catalog) convertPathToAbsolute(t string) (string, error) {
|
||||||
if strings.Contains(t, "*") {
|
if strings.Contains(t, "*") {
|
||||||
file := filepath.Base(t)
|
file := filepath.Base(t)
|
||||||
|
|
|
@ -230,9 +230,9 @@ func splitCommaTrim(value string) []string {
|
||||||
if !strings.Contains(value, ",") {
|
if !strings.Contains(value, ",") {
|
||||||
return []string{strings.ToLower(value)}
|
return []string{strings.ToLower(value)}
|
||||||
}
|
}
|
||||||
splitted := strings.Split(value, ",")
|
split := strings.Split(value, ",")
|
||||||
final := make([]string, len(splitted))
|
final := make([]string, len(split))
|
||||||
for i, value := range splitted {
|
for i, value := range split {
|
||||||
final[i] = strings.ToLower(strings.TrimSpace(value))
|
final[i] = strings.ToLower(strings.TrimSpace(value))
|
||||||
}
|
}
|
||||||
return final
|
return final
|
||||||
|
|
|
@ -98,8 +98,8 @@ func New(config *Config) (*Store, error) {
|
||||||
finalWorkflows: config.Workflows,
|
finalWorkflows: config.Workflows,
|
||||||
}
|
}
|
||||||
|
|
||||||
urlbasedTemplatesProvided := len(config.TemplateURLs) > 0 || len(config.WorkflowURLs) > 0
|
urlBasedTemplatesProvided := len(config.TemplateURLs) > 0 || len(config.WorkflowURLs) > 0
|
||||||
if urlbasedTemplatesProvided {
|
if urlBasedTemplatesProvided {
|
||||||
remoteTemplates, remoteWorkflows, err := getRemoteTemplatesAndWorkflows(config.TemplateURLs, config.WorkflowURLs)
|
remoteTemplates, remoteWorkflows, err := getRemoteTemplatesAndWorkflows(config.TemplateURLs, config.WorkflowURLs)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return store, err
|
return store, err
|
||||||
|
@ -109,7 +109,7 @@ func New(config *Config) (*Store, error) {
|
||||||
}
|
}
|
||||||
|
|
||||||
// Handle a case with no templates or workflows, where we use base directory
|
// Handle a case with no templates or workflows, where we use base directory
|
||||||
if len(store.finalTemplates) == 0 && len(store.finalWorkflows) == 0 && !urlbasedTemplatesProvided {
|
if len(store.finalTemplates) == 0 && len(store.finalWorkflows) == 0 && !urlBasedTemplatesProvided {
|
||||||
store.finalTemplates = []string{config.TemplatesDirectory}
|
store.finalTemplates = []string{config.TemplatesDirectory}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
@ -1,11 +1,12 @@
|
||||||
package core
|
package core
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"github.com/remeh/sizedwaitgroup"
|
||||||
|
"go.uber.org/atomic"
|
||||||
|
|
||||||
"github.com/projectdiscovery/gologger"
|
"github.com/projectdiscovery/gologger"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/templates"
|
"github.com/projectdiscovery/nuclei/v2/pkg/templates"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/templates/types"
|
"github.com/projectdiscovery/nuclei/v2/pkg/templates/types"
|
||||||
"github.com/remeh/sizedwaitgroup"
|
|
||||||
"go.uber.org/atomic"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// Execute takes a list of templates/workflows that have been compiled
|
// Execute takes a list of templates/workflows that have been compiled
|
||||||
|
@ -17,7 +18,7 @@ func (e *Engine) Execute(templates []*templates.Template, target InputProvider)
|
||||||
return e.ExecuteWithOpts(templates, target, false)
|
return e.ExecuteWithOpts(templates, target, false)
|
||||||
}
|
}
|
||||||
|
|
||||||
// ExecuteWithOpts is execute with the full options
|
// ExecuteWithOpts executes with the full options
|
||||||
func (e *Engine) ExecuteWithOpts(templatesList []*templates.Template, target InputProvider, noCluster bool) *atomic.Bool {
|
func (e *Engine) ExecuteWithOpts(templatesList []*templates.Template, target InputProvider, noCluster bool) *atomic.Bool {
|
||||||
var finalTemplates []*templates.Template
|
var finalTemplates []*templates.Template
|
||||||
if !noCluster {
|
if !noCluster {
|
||||||
|
@ -38,15 +39,17 @@ func (e *Engine) ExecuteWithOpts(templatesList []*templates.Template, target Inp
|
||||||
}
|
}
|
||||||
|
|
||||||
wg.Add()
|
wg.Add()
|
||||||
|
go func(tpl *templates.Template) {
|
||||||
switch {
|
switch {
|
||||||
case template.SelfContained:
|
case tpl.SelfContained:
|
||||||
// Self Contained requests are executed here separately
|
// Self Contained requests are executed here separately
|
||||||
e.executeSelfContainedTemplateWithInput(template, results)
|
e.executeSelfContainedTemplateWithInput(tpl, results)
|
||||||
default:
|
default:
|
||||||
// All other request types are executed here
|
// All other request types are executed here
|
||||||
e.executeModelWithInput(templateType, template, target, results)
|
e.executeModelWithInput(templateType, tpl, target, results)
|
||||||
}
|
}
|
||||||
wg.Done()
|
wg.Done()
|
||||||
|
}(template)
|
||||||
}
|
}
|
||||||
e.workPool.Wait()
|
e.workPool.Wait()
|
||||||
return results
|
return results
|
||||||
|
@ -71,9 +74,9 @@ func (e *Engine) executeModelWithInput(templateType types.ProtocolType, template
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
wg.Waitgroup.Add()
|
wg.WaitGroup.Add()
|
||||||
go func(value string) {
|
go func(value string) {
|
||||||
defer wg.Waitgroup.Done()
|
defer wg.WaitGroup.Done()
|
||||||
|
|
||||||
var match bool
|
var match bool
|
||||||
var err error
|
var err error
|
||||||
|
@ -89,5 +92,5 @@ func (e *Engine) executeModelWithInput(templateType types.ProtocolType, template
|
||||||
results.CAS(false, match)
|
results.CAS(false, match)
|
||||||
}(scannedValue)
|
}(scannedValue)
|
||||||
})
|
})
|
||||||
wg.Waitgroup.Wait()
|
wg.WaitGroup.Wait()
|
||||||
}
|
}
|
||||||
|
|
|
@ -9,6 +9,7 @@ import (
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"github.com/pkg/errors"
|
"github.com/pkg/errors"
|
||||||
|
|
||||||
"github.com/projectdiscovery/filekv"
|
"github.com/projectdiscovery/filekv"
|
||||||
"github.com/projectdiscovery/fileutil"
|
"github.com/projectdiscovery/fileutil"
|
||||||
"github.com/projectdiscovery/gologger"
|
"github.com/projectdiscovery/gologger"
|
||||||
|
|
|
@ -1,11 +1,12 @@
|
||||||
package core
|
package core
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"github.com/remeh/sizedwaitgroup"
|
||||||
|
"go.uber.org/atomic"
|
||||||
|
|
||||||
"github.com/projectdiscovery/gologger"
|
"github.com/projectdiscovery/gologger"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/output"
|
"github.com/projectdiscovery/nuclei/v2/pkg/output"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/workflows"
|
"github.com/projectdiscovery/nuclei/v2/pkg/workflows"
|
||||||
"github.com/remeh/sizedwaitgroup"
|
|
||||||
"go.uber.org/atomic"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// executeWorkflow runs a workflow on an input and returns true or false
|
// executeWorkflow runs a workflow on an input and returns true or false
|
||||||
|
|
|
@ -1,12 +1,13 @@
|
||||||
package core
|
package core
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/templates/types"
|
|
||||||
"github.com/remeh/sizedwaitgroup"
|
"github.com/remeh/sizedwaitgroup"
|
||||||
|
|
||||||
|
"github.com/projectdiscovery/nuclei/v2/pkg/templates/types"
|
||||||
)
|
)
|
||||||
|
|
||||||
// WorkPool implements an execution pool for executing different
|
// WorkPool implements an execution pool for executing different
|
||||||
// types of task with different concurreny requirements.
|
// types of task with different concurrency requirements.
|
||||||
//
|
//
|
||||||
// It also allows Configuration of such requirements. This is used
|
// It also allows Configuration of such requirements. This is used
|
||||||
// for per-module like separate headless concurrency etc.
|
// for per-module like separate headless concurrency etc.
|
||||||
|
@ -48,7 +49,7 @@ func (w *WorkPool) Wait() {
|
||||||
|
|
||||||
// InputWorkPool is a work pool per-input
|
// InputWorkPool is a work pool per-input
|
||||||
type InputWorkPool struct {
|
type InputWorkPool struct {
|
||||||
Waitgroup *sizedwaitgroup.SizedWaitGroup
|
WaitGroup *sizedwaitgroup.SizedWaitGroup
|
||||||
}
|
}
|
||||||
|
|
||||||
// InputPool returns a work pool for an input type
|
// InputPool returns a work pool for an input type
|
||||||
|
@ -60,5 +61,5 @@ func (w *WorkPool) InputPool(templateType types.ProtocolType) *InputWorkPool {
|
||||||
count = w.config.InputConcurrency
|
count = w.config.InputConcurrency
|
||||||
}
|
}
|
||||||
swg := sizedwaitgroup.New(count)
|
swg := sizedwaitgroup.New(count)
|
||||||
return &InputWorkPool{Waitgroup: &swg}
|
return &InputWorkPool{WaitGroup: &swg}
|
||||||
}
|
}
|
||||||
|
|
|
@ -72,6 +72,7 @@ func TestUnmarshal(t *testing.T) {
|
||||||
}
|
}
|
||||||
|
|
||||||
assertUnmarshalledTemplateInfo := func(t *testing.T, yamlPayload string) Info {
|
assertUnmarshalledTemplateInfo := func(t *testing.T, yamlPayload string) Info {
|
||||||
|
t.Helper()
|
||||||
info := Info{}
|
info := Info{}
|
||||||
err := yaml.Unmarshal([]byte(yamlPayload), &info)
|
err := yaml.Unmarshal([]byte(yamlPayload), &info)
|
||||||
assert.Nil(t, err)
|
assert.Nil(t, err)
|
||||||
|
|
|
@ -43,7 +43,7 @@ func (severities *Severities) UnmarshalYAML(unmarshal func(interface{}) error) e
|
||||||
}
|
}
|
||||||
|
|
||||||
func (severities Severities) String() string {
|
func (severities Severities) String() string {
|
||||||
var stringSeverities []string
|
var stringSeverities = make([]string, 0, len(severities))
|
||||||
for _, severity := range severities {
|
for _, severity := range severities {
|
||||||
stringSeverities = append(stringSeverities, severity.String())
|
stringSeverities = append(stringSeverities, severity.String())
|
||||||
}
|
}
|
||||||
|
|
|
@ -6,7 +6,6 @@ import (
|
||||||
"github.com/alecthomas/jsonschema"
|
"github.com/alecthomas/jsonschema"
|
||||||
)
|
)
|
||||||
|
|
||||||
//nolint:exported,revive //prefer to be explicit about the name, and make it refactor-safe
|
|
||||||
// Holder holds a Severity type. Required for un/marshalling purposes
|
// Holder holds a Severity type. Required for un/marshalling purposes
|
||||||
type Holder struct {
|
type Holder struct {
|
||||||
Severity Severity
|
Severity Severity
|
||||||
|
|
|
@ -30,6 +30,7 @@ func TestGetSupportedSeverities(t *testing.T) {
|
||||||
}
|
}
|
||||||
|
|
||||||
func testUnmarshal(t *testing.T, unmarshaller func(data []byte, v interface{}) error, payloadCreator func(value string) string) {
|
func testUnmarshal(t *testing.T, unmarshaller func(data []byte, v interface{}) error, payloadCreator func(value string) string) {
|
||||||
|
t.Helper()
|
||||||
payloads := [...]string{
|
payloads := [...]string{
|
||||||
payloadCreator("Info"),
|
payloadCreator("Info"),
|
||||||
payloadCreator("info"),
|
payloadCreator("info"),
|
||||||
|
@ -48,6 +49,7 @@ func testUnmarshal(t *testing.T, unmarshaller func(data []byte, v interface{}) e
|
||||||
}
|
}
|
||||||
|
|
||||||
func testUnmarshalFail(t *testing.T, unmarshaller func(data []byte, v interface{}) error, payloadCreator func(value string) string) {
|
func testUnmarshalFail(t *testing.T, unmarshaller func(data []byte, v interface{}) error, payloadCreator func(value string) string) {
|
||||||
|
t.Helper()
|
||||||
assert.Panics(t, func() { unmarshal(payloadCreator("invalid"), unmarshaller) })
|
assert.Panics(t, func() { unmarshal(payloadCreator("invalid"), unmarshaller) })
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
@ -80,11 +80,12 @@ func marshalStringToSlice(unmarshal func(interface{}) error) ([]string, error) {
|
||||||
}
|
}
|
||||||
|
|
||||||
var result []string
|
var result []string
|
||||||
if len(marshalledValuesAsSlice) > 0 {
|
switch {
|
||||||
|
case len(marshalledValuesAsSlice) > 0:
|
||||||
result = marshalledValuesAsSlice
|
result = marshalledValuesAsSlice
|
||||||
} else if utils.IsNotBlank(marshalledValueAsString) {
|
case utils.IsNotBlank(marshalledValueAsString):
|
||||||
result = strings.Split(marshalledValueAsString, ",")
|
result = strings.Split(marshalledValueAsString, ",")
|
||||||
} else {
|
default:
|
||||||
result = []string{}
|
result = []string{}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
@ -19,10 +19,11 @@ import (
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/Knetic/govaluate"
|
"github.com/Knetic/govaluate"
|
||||||
|
"github.com/spaolacci/murmur3"
|
||||||
|
|
||||||
"github.com/projectdiscovery/gologger"
|
"github.com/projectdiscovery/gologger"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/helpers/deserialization"
|
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/helpers/deserialization"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/types"
|
"github.com/projectdiscovery/nuclei/v2/pkg/types"
|
||||||
"github.com/spaolacci/murmur3"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
|
|
|
@ -8,8 +8,9 @@ import (
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/Knetic/govaluate"
|
"github.com/Knetic/govaluate"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/types"
|
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
|
|
||||||
|
"github.com/projectdiscovery/nuclei/v2/pkg/types"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestDSLURLEncodeDecode(t *testing.T) {
|
func TestDSLURLEncodeDecode(t *testing.T) {
|
||||||
|
|
|
@ -1,9 +1,8 @@
|
||||||
package extractors
|
package extractors
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"strings"
|
|
||||||
|
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
|
"strings"
|
||||||
|
|
||||||
"github.com/antchfx/htmlquery"
|
"github.com/antchfx/htmlquery"
|
||||||
|
|
||||||
|
|
|
@ -20,7 +20,6 @@ const (
|
||||||
XPathExtractor
|
XPathExtractor
|
||||||
// JSONExtractor extracts responses with json
|
// JSONExtractor extracts responses with json
|
||||||
JSONExtractor
|
JSONExtractor
|
||||||
//limit
|
|
||||||
limit
|
limit
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
|
@ -12,74 +12,73 @@ import (
|
||||||
)
|
)
|
||||||
|
|
||||||
// CompileMatchers performs the initial setup operation on a matcher
|
// CompileMatchers performs the initial setup operation on a matcher
|
||||||
func (m *Matcher) CompileMatchers() error {
|
func (matcher *Matcher) CompileMatchers() error {
|
||||||
var ok bool
|
var ok bool
|
||||||
|
|
||||||
// Support hexadecimal encoding for matchers too.
|
// Support hexadecimal encoding for matchers too.
|
||||||
if m.Encoding == "hex" {
|
if matcher.Encoding == "hex" {
|
||||||
for i, word := range m.Words {
|
for i, word := range matcher.Words {
|
||||||
if decoded, err := hex.DecodeString(word); err == nil && len(decoded) > 0 {
|
if decoded, err := hex.DecodeString(word); err == nil && len(decoded) > 0 {
|
||||||
m.Words[i] = string(decoded)
|
matcher.Words[i] = string(decoded)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Set up the matcher type
|
// Set up the matcher type
|
||||||
computedType, err := toMatcherTypes(m.GetType().String())
|
computedType, err := toMatcherTypes(matcher.GetType().String())
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("unknown matcher type specified: %s", m.Type)
|
return fmt.Errorf("unknown matcher type specified: %s", matcher.Type)
|
||||||
}
|
}
|
||||||
|
|
||||||
m.matcherType = computedType
|
matcher.matcherType = computedType
|
||||||
// By default, match on body if user hasn't provided any specific items
|
// By default, match on body if user hasn't provided any specific items
|
||||||
if m.Part == "" {
|
if matcher.Part == "" {
|
||||||
m.Part = "body"
|
matcher.Part = "body"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
// Compile the regexes
|
// Compile the regexes
|
||||||
for _, regex := range m.Regex {
|
for _, regex := range matcher.Regex {
|
||||||
compiled, err := regexp.Compile(regex)
|
compiled, err := regexp.Compile(regex)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("could not compile regex: %s", regex)
|
return fmt.Errorf("could not compile regex: %s", regex)
|
||||||
}
|
}
|
||||||
m.regexCompiled = append(m.regexCompiled, compiled)
|
matcher.regexCompiled = append(matcher.regexCompiled, compiled)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Compile and validate binary Values in matcher
|
// Compile and validate binary Values in matcher
|
||||||
for _, value := range m.Binary {
|
for _, value := range matcher.Binary {
|
||||||
if decoded, err := hex.DecodeString(value); err != nil {
|
if decoded, err := hex.DecodeString(value); err != nil {
|
||||||
return fmt.Errorf("could not hex decode binary: %s", value)
|
return fmt.Errorf("could not hex decode binary: %s", value)
|
||||||
} else {
|
} else {
|
||||||
m.binaryDecoded = append(m.binaryDecoded, string(decoded))
|
matcher.binaryDecoded = append(matcher.binaryDecoded, string(decoded))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Compile the dsl expressions
|
// Compile the dsl expressions
|
||||||
for _, expr := range m.DSL {
|
for _, expr := range matcher.DSL {
|
||||||
compiled, err := govaluate.NewEvaluableExpressionWithFunctions(expr, dsl.HelperFunctions())
|
compiled, err := govaluate.NewEvaluableExpressionWithFunctions(expr, dsl.HelperFunctions())
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("could not compile dsl: %s", expr)
|
return fmt.Errorf("could not compile dsl: %s", expr)
|
||||||
}
|
}
|
||||||
m.dslCompiled = append(m.dslCompiled, compiled)
|
matcher.dslCompiled = append(matcher.dslCompiled, compiled)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Set up the condition type, if any.
|
// Set up the condition type, if any.
|
||||||
if m.Condition != "" {
|
if matcher.Condition != "" {
|
||||||
m.condition, ok = ConditionTypes[m.Condition]
|
matcher.condition, ok = ConditionTypes[matcher.Condition]
|
||||||
if !ok {
|
if !ok {
|
||||||
return fmt.Errorf("unknown condition specified: %s", m.Condition)
|
return fmt.Errorf("unknown condition specified: %s", matcher.Condition)
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
m.condition = ORCondition
|
matcher.condition = ORCondition
|
||||||
}
|
}
|
||||||
|
|
||||||
if m.CaseInsensitive {
|
if matcher.CaseInsensitive {
|
||||||
if m.GetType() != WordsMatcher {
|
if matcher.GetType() != WordsMatcher {
|
||||||
return fmt.Errorf("case-insensitive flag is supported only for 'word' matchers (not '%s')", m.Type)
|
return fmt.Errorf("case-insensitive flag is supported only for 'word' matchers (not '%s')", matcher.Type)
|
||||||
}
|
}
|
||||||
for i := range m.Words {
|
for i := range matcher.Words {
|
||||||
m.Words[i] = strings.ToLower(m.Words[i])
|
matcher.Words[i] = strings.ToLower(matcher.Words[i])
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
|
|
|
@ -7,11 +7,11 @@ import (
|
||||||
)
|
)
|
||||||
|
|
||||||
// MatchStatusCode matches a status code check against a corpus
|
// MatchStatusCode matches a status code check against a corpus
|
||||||
func (m *Matcher) MatchStatusCode(statusCode int) bool {
|
func (matcher *Matcher) MatchStatusCode(statusCode int) bool {
|
||||||
// Iterate over all the status codes accepted as valid
|
// Iterate over all the status codes accepted as valid
|
||||||
//
|
//
|
||||||
// Status codes don't support AND conditions.
|
// Status codes don't support AND conditions.
|
||||||
for _, status := range m.Status {
|
for _, status := range matcher.Status {
|
||||||
// Continue if the status codes don't match
|
// Continue if the status codes don't match
|
||||||
if statusCode != status {
|
if statusCode != status {
|
||||||
continue
|
continue
|
||||||
|
@ -23,11 +23,11 @@ func (m *Matcher) MatchStatusCode(statusCode int) bool {
|
||||||
}
|
}
|
||||||
|
|
||||||
// MatchSize matches a size check against a corpus
|
// MatchSize matches a size check against a corpus
|
||||||
func (m *Matcher) MatchSize(length int) bool {
|
func (matcher *Matcher) MatchSize(length int) bool {
|
||||||
// Iterate over all the sizes accepted as valid
|
// Iterate over all the sizes accepted as valid
|
||||||
//
|
//
|
||||||
// Sizes codes don't support AND conditions.
|
// Sizes codes don't support AND conditions.
|
||||||
for _, size := range m.Size {
|
for _, size := range matcher.Size {
|
||||||
// Continue if the size doesn't match
|
// Continue if the size doesn't match
|
||||||
if length != size {
|
if length != size {
|
||||||
continue
|
continue
|
||||||
|
@ -39,14 +39,14 @@ func (m *Matcher) MatchSize(length int) bool {
|
||||||
}
|
}
|
||||||
|
|
||||||
// MatchWords matches a word check against a corpus.
|
// MatchWords matches a word check against a corpus.
|
||||||
func (m *Matcher) MatchWords(corpus string, dynamicValues map[string]interface{}) (bool, []string) {
|
func (matcher *Matcher) MatchWords(corpus string, dynamicValues map[string]interface{}) (bool, []string) {
|
||||||
if m.CaseInsensitive {
|
if matcher.CaseInsensitive {
|
||||||
corpus = strings.ToLower(corpus)
|
corpus = strings.ToLower(corpus)
|
||||||
}
|
}
|
||||||
|
|
||||||
var matchedWords []string
|
var matchedWords []string
|
||||||
// Iterate over all the words accepted as valid
|
// Iterate over all the words accepted as valid
|
||||||
for i, word := range m.Words {
|
for i, word := range matcher.Words {
|
||||||
if dynamicValues == nil {
|
if dynamicValues == nil {
|
||||||
dynamicValues = make(map[string]interface{})
|
dynamicValues = make(map[string]interface{})
|
||||||
}
|
}
|
||||||
|
@ -60,7 +60,7 @@ func (m *Matcher) MatchWords(corpus string, dynamicValues map[string]interface{}
|
||||||
if !strings.Contains(corpus, word) {
|
if !strings.Contains(corpus, word) {
|
||||||
// If we are in an AND request and a match failed,
|
// If we are in an AND request and a match failed,
|
||||||
// return false as the AND condition fails on any single mismatch.
|
// return false as the AND condition fails on any single mismatch.
|
||||||
if m.condition == ANDCondition {
|
if matcher.condition == ANDCondition {
|
||||||
return false, []string{}
|
return false, []string{}
|
||||||
}
|
}
|
||||||
// Continue with the flow since it's an OR Condition.
|
// Continue with the flow since it's an OR Condition.
|
||||||
|
@ -68,14 +68,14 @@ func (m *Matcher) MatchWords(corpus string, dynamicValues map[string]interface{}
|
||||||
}
|
}
|
||||||
|
|
||||||
// If the condition was an OR, return on the first match.
|
// If the condition was an OR, return on the first match.
|
||||||
if m.condition == ORCondition {
|
if matcher.condition == ORCondition {
|
||||||
return true, []string{word}
|
return true, []string{word}
|
||||||
}
|
}
|
||||||
|
|
||||||
matchedWords = append(matchedWords, word)
|
matchedWords = append(matchedWords, word)
|
||||||
|
|
||||||
// If we are at the end of the words, return with true
|
// If we are at the end of the words, return with true
|
||||||
if len(m.Words)-1 == i {
|
if len(matcher.Words)-1 == i {
|
||||||
return true, matchedWords
|
return true, matchedWords
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
@ -83,15 +83,15 @@ func (m *Matcher) MatchWords(corpus string, dynamicValues map[string]interface{}
|
||||||
}
|
}
|
||||||
|
|
||||||
// MatchRegex matches a regex check against a corpus
|
// MatchRegex matches a regex check against a corpus
|
||||||
func (m *Matcher) MatchRegex(corpus string) (bool, []string) {
|
func (matcher *Matcher) MatchRegex(corpus string) (bool, []string) {
|
||||||
var matchedRegexes []string
|
var matchedRegexes []string
|
||||||
// Iterate over all the regexes accepted as valid
|
// Iterate over all the regexes accepted as valid
|
||||||
for i, regex := range m.regexCompiled {
|
for i, regex := range matcher.regexCompiled {
|
||||||
// Continue if the regex doesn't match
|
// Continue if the regex doesn't match
|
||||||
if !regex.MatchString(corpus) {
|
if !regex.MatchString(corpus) {
|
||||||
// If we are in an AND request and a match failed,
|
// If we are in an AND request and a match failed,
|
||||||
// return false as the AND condition fails on any single mismatch.
|
// return false as the AND condition fails on any single mismatch.
|
||||||
if m.condition == ANDCondition {
|
if matcher.condition == ANDCondition {
|
||||||
return false, []string{}
|
return false, []string{}
|
||||||
}
|
}
|
||||||
// Continue with the flow since it's an OR Condition.
|
// Continue with the flow since it's an OR Condition.
|
||||||
|
@ -100,14 +100,14 @@ func (m *Matcher) MatchRegex(corpus string) (bool, []string) {
|
||||||
|
|
||||||
currentMatches := regex.FindAllString(corpus, -1)
|
currentMatches := regex.FindAllString(corpus, -1)
|
||||||
// If the condition was an OR, return on the first match.
|
// If the condition was an OR, return on the first match.
|
||||||
if m.condition == ORCondition {
|
if matcher.condition == ORCondition {
|
||||||
return true, currentMatches
|
return true, currentMatches
|
||||||
}
|
}
|
||||||
|
|
||||||
matchedRegexes = append(matchedRegexes, currentMatches...)
|
matchedRegexes = append(matchedRegexes, currentMatches...)
|
||||||
|
|
||||||
// If we are at the end of the regex, return with true
|
// If we are at the end of the regex, return with true
|
||||||
if len(m.regexCompiled)-1 == i {
|
if len(matcher.regexCompiled)-1 == i {
|
||||||
return true, matchedRegexes
|
return true, matchedRegexes
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
@ -115,14 +115,14 @@ func (m *Matcher) MatchRegex(corpus string) (bool, []string) {
|
||||||
}
|
}
|
||||||
|
|
||||||
// MatchBinary matches a binary check against a corpus
|
// MatchBinary matches a binary check against a corpus
|
||||||
func (m *Matcher) MatchBinary(corpus string) (bool, []string) {
|
func (matcher *Matcher) MatchBinary(corpus string) (bool, []string) {
|
||||||
var matchedBinary []string
|
var matchedBinary []string
|
||||||
// Iterate over all the words accepted as valid
|
// Iterate over all the words accepted as valid
|
||||||
for i, binary := range m.binaryDecoded {
|
for i, binary := range matcher.binaryDecoded {
|
||||||
if !strings.Contains(corpus, binary) {
|
if !strings.Contains(corpus, binary) {
|
||||||
// If we are in an AND request and a match failed,
|
// If we are in an AND request and a match failed,
|
||||||
// return false as the AND condition fails on any single mismatch.
|
// return false as the AND condition fails on any single mismatch.
|
||||||
if m.condition == ANDCondition {
|
if matcher.condition == ANDCondition {
|
||||||
return false, []string{}
|
return false, []string{}
|
||||||
}
|
}
|
||||||
// Continue with the flow since it's an OR Condition.
|
// Continue with the flow since it's an OR Condition.
|
||||||
|
@ -130,14 +130,14 @@ func (m *Matcher) MatchBinary(corpus string) (bool, []string) {
|
||||||
}
|
}
|
||||||
|
|
||||||
// If the condition was an OR, return on the first match.
|
// If the condition was an OR, return on the first match.
|
||||||
if m.condition == ORCondition {
|
if matcher.condition == ORCondition {
|
||||||
return true, []string{binary}
|
return true, []string{binary}
|
||||||
}
|
}
|
||||||
|
|
||||||
matchedBinary = append(matchedBinary, binary)
|
matchedBinary = append(matchedBinary, binary)
|
||||||
|
|
||||||
// If we are at the end of the words, return with true
|
// If we are at the end of the words, return with true
|
||||||
if len(m.Binary)-1 == i {
|
if len(matcher.Binary)-1 == i {
|
||||||
return true, matchedBinary
|
return true, matchedBinary
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
@ -145,9 +145,9 @@ func (m *Matcher) MatchBinary(corpus string) (bool, []string) {
|
||||||
}
|
}
|
||||||
|
|
||||||
// MatchDSL matches on a generic map result
|
// MatchDSL matches on a generic map result
|
||||||
func (m *Matcher) MatchDSL(data map[string]interface{}) bool {
|
func (matcher *Matcher) MatchDSL(data map[string]interface{}) bool {
|
||||||
// Iterate over all the expressions accepted as valid
|
// Iterate over all the expressions accepted as valid
|
||||||
for i, expression := range m.dslCompiled {
|
for i, expression := range matcher.dslCompiled {
|
||||||
result, err := expression.Evaluate(data)
|
result, err := expression.Evaluate(data)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
continue
|
continue
|
||||||
|
@ -160,7 +160,7 @@ func (m *Matcher) MatchDSL(data map[string]interface{}) bool {
|
||||||
if !ok || !bResult {
|
if !ok || !bResult {
|
||||||
// If we are in an AND request and a match failed,
|
// If we are in an AND request and a match failed,
|
||||||
// return false as the AND condition fails on any single mismatch.
|
// return false as the AND condition fails on any single mismatch.
|
||||||
if m.condition == ANDCondition {
|
if matcher.condition == ANDCondition {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
// Continue with the flow since it's an OR Condition.
|
// Continue with the flow since it's an OR Condition.
|
||||||
|
@ -168,12 +168,12 @@ func (m *Matcher) MatchDSL(data map[string]interface{}) bool {
|
||||||
}
|
}
|
||||||
|
|
||||||
// If the condition was an OR, return on the first match.
|
// If the condition was an OR, return on the first match.
|
||||||
if m.condition == ORCondition {
|
if matcher.condition == ORCondition {
|
||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
|
|
||||||
// If we are at the end of the dsl, return with true
|
// If we are at the end of the dsl, return with true
|
||||||
if len(m.dslCompiled)-1 == i {
|
if len(matcher.dslCompiled)-1 == i {
|
||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
@ -62,7 +62,7 @@ type Matcher struct {
|
||||||
// description: |
|
// description: |
|
||||||
// Words contains word patterns required to be present in the response part.
|
// Words contains word patterns required to be present in the response part.
|
||||||
// examples:
|
// examples:
|
||||||
// - name: Match for outlook mail protection domain
|
// - name: Match for Outlook mail protection domain
|
||||||
// value: >
|
// value: >
|
||||||
// []string{"mail.protection.outlook.com"}
|
// []string{"mail.protection.outlook.com"}
|
||||||
// - name: Match for application/json in response headers
|
// - name: Match for application/json in response headers
|
||||||
|
@ -120,7 +120,6 @@ type Matcher struct {
|
||||||
dslCompiled []*govaluate.EvaluableExpression
|
dslCompiled []*govaluate.EvaluableExpression
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
// ConditionType is the type of condition for matcher
|
// ConditionType is the type of condition for matcher
|
||||||
type ConditionType int
|
type ConditionType int
|
||||||
|
|
||||||
|
@ -138,18 +137,17 @@ var ConditionTypes = map[string]ConditionType{
|
||||||
}
|
}
|
||||||
|
|
||||||
// Result reverts the results of the match if the matcher is of type negative.
|
// Result reverts the results of the match if the matcher is of type negative.
|
||||||
func (m *Matcher) Result(data bool) bool {
|
func (matcher *Matcher) Result(data bool) bool {
|
||||||
if m.Negative {
|
if matcher.Negative {
|
||||||
return !data
|
return !data
|
||||||
}
|
}
|
||||||
return data
|
return data
|
||||||
}
|
}
|
||||||
|
|
||||||
// ResultWithMatchedSnippet returns true and the matched snippet, or false and an empty string
|
// ResultWithMatchedSnippet returns true and the matched snippet, or false and an empty string
|
||||||
func (m *Matcher) ResultWithMatchedSnippet(data bool, matchedSnippet []string) (bool, []string) {
|
func (matcher *Matcher) ResultWithMatchedSnippet(data bool, matchedSnippet []string) (bool, []string) {
|
||||||
if m.Negative {
|
if matcher.Negative {
|
||||||
return !data, []string{}
|
return !data, []string{}
|
||||||
}
|
}
|
||||||
return data, matchedSnippet
|
return data, matchedSnippet
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
@ -24,7 +24,6 @@ const (
|
||||||
SizeMatcher
|
SizeMatcher
|
||||||
// DSLMatcher matches based upon dsl syntax
|
// DSLMatcher matches based upon dsl syntax
|
||||||
DSLMatcher
|
DSLMatcher
|
||||||
//limit
|
|
||||||
limit
|
limit
|
||||||
)
|
)
|
||||||
|
|
||||||
|
@ -39,8 +38,8 @@ var MatcherTypes = map[MatcherType]string{
|
||||||
}
|
}
|
||||||
|
|
||||||
//GetType returns the type of the matcher
|
//GetType returns the type of the matcher
|
||||||
func (e *Matcher) GetType() MatcherType {
|
func (matcher *Matcher) GetType() MatcherType {
|
||||||
return e.Type.MatcherType
|
return matcher.Type.MatcherType
|
||||||
}
|
}
|
||||||
|
|
||||||
// GetSupportedMatcherTypes returns list of supported types
|
// GetSupportedMatcherTypes returns list of supported types
|
||||||
|
|
|
@ -5,11 +5,12 @@ import (
|
||||||
"fmt"
|
"fmt"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/catalog/loader/filter"
|
"github.com/projectdiscovery/nuclei/v2/pkg/catalog/loader/filter"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/model"
|
"github.com/projectdiscovery/nuclei/v2/pkg/model"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/model/types/stringslice"
|
"github.com/projectdiscovery/nuclei/v2/pkg/model/types/stringslice"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/templates"
|
"github.com/projectdiscovery/nuclei/v2/pkg/templates"
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestLoadTemplate(t *testing.T) {
|
func TestLoadTemplate(t *testing.T) {
|
||||||
|
|
|
@ -79,23 +79,6 @@ func newInternalResponse() *InternalResponse {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Unused
|
|
||||||
// func toInternalRequest(req *http.Request, target string, body []byte) *InternalRequest {
|
|
||||||
// intReq := newInternalRquest()
|
|
||||||
|
|
||||||
// intReq.Target = target
|
|
||||||
// intReq.HTTPMajor = req.ProtoMajor
|
|
||||||
// intReq.HTTPMinor = req.ProtoMinor
|
|
||||||
// for k, v := range req.Header {
|
|
||||||
// intReq.Headers[k] = v
|
|
||||||
// }
|
|
||||||
// intReq.Headers = req.Header
|
|
||||||
// intReq.Method = req.Method
|
|
||||||
// intReq.Body = body
|
|
||||||
|
|
||||||
// return intReq
|
|
||||||
// }
|
|
||||||
|
|
||||||
func toInternalResponse(resp *http.Response, body []byte) *InternalResponse {
|
func toInternalResponse(resp *http.Response, body []byte) *InternalResponse {
|
||||||
intResp := newInternalResponse()
|
intResp := newInternalResponse()
|
||||||
|
|
||||||
|
@ -125,14 +108,3 @@ func fromInternalResponse(intResp *InternalResponse) *http.Response {
|
||||||
Body: ioutil.NopCloser(bytes.NewReader(intResp.Body)),
|
Body: ioutil.NopCloser(bytes.NewReader(intResp.Body)),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Unused
|
|
||||||
// func fromInternalRequest(intReq *InternalRequest) *http.Request {
|
|
||||||
// return &http.Request{
|
|
||||||
// ProtoMinor: intReq.HTTPMinor,
|
|
||||||
// ProtoMajor: intReq.HTTPMajor,
|
|
||||||
// Header: intReq.Headers,
|
|
||||||
// ContentLength: int64(len(intReq.Body)),
|
|
||||||
// Body: ioutil.NopCloser(bytes.NewReader(intReq.Body)),
|
|
||||||
// }
|
|
||||||
// }
|
|
||||||
|
|
|
@ -42,13 +42,13 @@ func (pf *ProjectFile) Get(req []byte) (*http.Response, error) {
|
||||||
return nil, fmt.Errorf("not found")
|
return nil, fmt.Errorf("not found")
|
||||||
}
|
}
|
||||||
|
|
||||||
var httprecord HTTPRecord
|
var httpRecord HTTPRecord
|
||||||
httprecord.Response = newInternalResponse()
|
httpRecord.Response = newInternalResponse()
|
||||||
if err := unmarshal(data, &httprecord); err != nil {
|
if err := unmarshal(data, &httpRecord); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
return fromInternalResponse(httprecord.Response), nil
|
return fromInternalResponse(httpRecord.Response), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (pf *ProjectFile) Set(req []byte, resp *http.Response, data []byte) error {
|
func (pf *ProjectFile) Set(req []byte, resp *http.Response, data []byte) error {
|
||||||
|
@ -57,10 +57,10 @@ func (pf *ProjectFile) Set(req []byte, resp *http.Response, data []byte) error {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
var httprecord HTTPRecord
|
var httpRecord HTTPRecord
|
||||||
httprecord.Request = req
|
httpRecord.Request = req
|
||||||
httprecord.Response = toInternalResponse(resp, data)
|
httpRecord.Response = toInternalResponse(resp, data)
|
||||||
data, err = marshal(httprecord)
|
data, err = marshal(httpRecord)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
|
@ -4,12 +4,13 @@ import (
|
||||||
"regexp"
|
"regexp"
|
||||||
|
|
||||||
"github.com/Knetic/govaluate"
|
"github.com/Knetic/govaluate"
|
||||||
|
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/operators/common/dsl"
|
"github.com/projectdiscovery/nuclei/v2/pkg/operators/common/dsl"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/generators"
|
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/generators"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/replacer"
|
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/replacer"
|
||||||
)
|
)
|
||||||
|
|
||||||
var templateExpressionRegex = regexp.MustCompile(`(?m)\{\{[^}]+\}\}["'\)\}]*`)
|
var templateExpressionRegex = regexp.MustCompile(`(?m){{[^}]+}}["')}]*`)
|
||||||
|
|
||||||
// Evaluate checks if the match contains a dynamic variable, for each
|
// Evaluate checks if the match contains a dynamic variable, for each
|
||||||
// found one we will check if it's an expression and can
|
// found one we will check if it's an expression and can
|
||||||
|
|
|
@ -6,7 +6,7 @@ import (
|
||||||
"strings"
|
"strings"
|
||||||
)
|
)
|
||||||
|
|
||||||
var unresolvedVariablesRegex = regexp.MustCompile(`(?:%7[B|b]|\{){2}([^}]+)(?:%7[D|d]|\}){2}["'\)\}]*`)
|
var unresolvedVariablesRegex = regexp.MustCompile(`(?:%7[B|b]|{){2}([^}]+)(?:%7[D|d]|}){2}["')}]*`)
|
||||||
|
|
||||||
// ContainsUnresolvedVariables returns an error with variable names if the passed
|
// ContainsUnresolvedVariables returns an error with variable names if the passed
|
||||||
// input contains unresolved {{<pattern-here>}} variables.
|
// input contains unresolved {{<pattern-here>}} variables.
|
||||||
|
|
|
@ -13,12 +13,12 @@ type AttackType int
|
||||||
|
|
||||||
// Supported values for the ProtocolType
|
// Supported values for the ProtocolType
|
||||||
const (
|
const (
|
||||||
// BatteringRamAttack replaces same payload into all of the defined payload positions at once.
|
// BatteringRamAttack replaces same payload in all the defined payload positions at once
|
||||||
BatteringRamAttack AttackType = iota + 1
|
BatteringRamAttack AttackType = iota + 1
|
||||||
// PitchForkAttack replaces variables with positional value from multiple wordlists
|
// PitchForkAttack replaces variables with positional value from multiple wordlists
|
||||||
PitchForkAttack
|
PitchForkAttack
|
||||||
// ClusterbombAttack replaces variables with all possible combinations of values
|
// ClusterBombAttack replaces variables with all possible combinations of values
|
||||||
ClusterbombAttack
|
ClusterBombAttack
|
||||||
limit
|
limit
|
||||||
)
|
)
|
||||||
|
|
||||||
|
@ -26,7 +26,7 @@ const (
|
||||||
var attackTypeMappings = map[AttackType]string{
|
var attackTypeMappings = map[AttackType]string{
|
||||||
BatteringRamAttack: "batteringram",
|
BatteringRamAttack: "batteringram",
|
||||||
PitchForkAttack: "pitchfork",
|
PitchForkAttack: "pitchfork",
|
||||||
ClusterbombAttack: "clusterbomb",
|
ClusterBombAttack: "clusterbomb",
|
||||||
}
|
}
|
||||||
|
|
||||||
func GetSupportedAttackTypes() []AttackType {
|
func GetSupportedAttackTypes() []AttackType {
|
||||||
|
|
|
@ -4,6 +4,7 @@ package generators
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"github.com/pkg/errors"
|
"github.com/pkg/errors"
|
||||||
|
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/catalog"
|
"github.com/projectdiscovery/nuclei/v2/pkg/catalog"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
@ -110,7 +111,7 @@ func (i *Iterator) Total() int {
|
||||||
count = len(p.values)
|
count = len(p.values)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
case ClusterbombAttack:
|
case ClusterBombAttack:
|
||||||
count = 1
|
count = 1
|
||||||
for _, p := range i.payloads {
|
for _, p := range i.payloads {
|
||||||
count *= len(p.values)
|
count *= len(p.values)
|
||||||
|
@ -126,7 +127,7 @@ func (i *Iterator) Value() (map[string]interface{}, bool) {
|
||||||
return i.batteringRamValue()
|
return i.batteringRamValue()
|
||||||
case PitchForkAttack:
|
case PitchForkAttack:
|
||||||
return i.pitchforkValue()
|
return i.pitchforkValue()
|
||||||
case ClusterbombAttack:
|
case ClusterBombAttack:
|
||||||
return i.clusterbombValue()
|
return i.clusterbombValue()
|
||||||
default:
|
default:
|
||||||
return i.batteringRamValue()
|
return i.batteringRamValue()
|
||||||
|
|
|
@ -3,8 +3,9 @@ package generators
|
||||||
import (
|
import (
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/catalog"
|
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
|
|
||||||
|
"github.com/projectdiscovery/nuclei/v2/pkg/catalog"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestBatteringRamGenerator(t *testing.T) {
|
func TestBatteringRamGenerator(t *testing.T) {
|
||||||
|
@ -53,7 +54,7 @@ func TestClusterbombGenerator(t *testing.T) {
|
||||||
passwords := []string{"admin", "password", "token"}
|
passwords := []string{"admin", "password", "token"}
|
||||||
|
|
||||||
catalogInstance := catalog.New("")
|
catalogInstance := catalog.New("")
|
||||||
generator, err := New(map[string]interface{}{"username": usernames, "password": passwords}, ClusterbombAttack, "", catalogInstance)
|
generator, err := New(map[string]interface{}{"username": usernames, "password": passwords}, ClusterBombAttack, "", catalogInstance)
|
||||||
require.Nil(t, err, "could not create generator")
|
require.Nil(t, err, "could not create generator")
|
||||||
|
|
||||||
iterator := generator.NewIterator()
|
iterator := generator.NewIterator()
|
||||||
|
|
|
@ -53,7 +53,7 @@ func loadPayloadsFromFile(filepath string) ([]string, error) {
|
||||||
}
|
}
|
||||||
lines = append(lines, text)
|
lines = append(lines, text)
|
||||||
}
|
}
|
||||||
if err := scanner.Err(); err != nil && err != io.EOF {
|
if err := scanner.Err(); err != nil && !errors.Is(err, io.EOF) {
|
||||||
return lines, scanner.Err()
|
return lines, scanner.Err()
|
||||||
}
|
}
|
||||||
return lines, nil
|
return lines, nil
|
||||||
|
|
|
@ -13,15 +13,15 @@ import (
|
||||||
// validate validates the payloads if any.
|
// validate validates the payloads if any.
|
||||||
func (g *PayloadGenerator) validate(payloads map[string]interface{}, templatePath string) error {
|
func (g *PayloadGenerator) validate(payloads map[string]interface{}, templatePath string) error {
|
||||||
for name, payload := range payloads {
|
for name, payload := range payloads {
|
||||||
switch pt := payload.(type) {
|
switch payloadType := payload.(type) {
|
||||||
case string:
|
case string:
|
||||||
// check if it's a multiline string list
|
// check if it's a multiline string list
|
||||||
if len(strings.Split(pt, "\n")) != 1 {
|
if len(strings.Split(payloadType, "\n")) != 1 {
|
||||||
return errors.New("invalid number of lines in payload")
|
return errors.New("invalid number of lines in payload")
|
||||||
}
|
}
|
||||||
|
|
||||||
// check if it's a worldlist file and try to load it
|
// check if it's a worldlist file and try to load it
|
||||||
if fileExists(pt) {
|
if fileExists(payloadType) {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -29,18 +29,18 @@ func (g *PayloadGenerator) validate(payloads map[string]interface{}, templatePat
|
||||||
pathTokens := strings.Split(templatePath, string(os.PathSeparator))
|
pathTokens := strings.Split(templatePath, string(os.PathSeparator))
|
||||||
|
|
||||||
for i := range pathTokens {
|
for i := range pathTokens {
|
||||||
tpath := filepath.Join(filepath.Join(pathTokens[:i]...), pt)
|
payloadPath := filepath.Join(filepath.Join(pathTokens[:i]...), payloadType)
|
||||||
if fileExists(tpath) {
|
if fileExists(payloadPath) {
|
||||||
payloads[name] = tpath
|
payloads[name] = payloadPath
|
||||||
changed = true
|
changed = true
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if !changed {
|
if !changed {
|
||||||
return fmt.Errorf("the %s file for payload %s does not exist or does not contain enough elements", pt, name)
|
return fmt.Errorf("the %s file for payload %s does not exist or does not contain enough elements", payloadType, name)
|
||||||
}
|
}
|
||||||
case interface{}:
|
case interface{}:
|
||||||
loadedPayloads := types.ToStringSlice(pt)
|
loadedPayloads := types.ToStringSlice(payloadType)
|
||||||
if len(loadedPayloads) == 0 {
|
if len(loadedPayloads) == 0 {
|
||||||
return fmt.Errorf("the payload %s does not contain enough elements", name)
|
return fmt.Errorf("the payload %s does not contain enough elements", name)
|
||||||
}
|
}
|
||||||
|
|
|
@ -7,6 +7,7 @@ import (
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"github.com/bluele/gcache"
|
"github.com/bluele/gcache"
|
||||||
|
|
||||||
"github.com/projectdiscovery/gologger"
|
"github.com/projectdiscovery/gologger"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
@ -24,11 +25,11 @@ type Cache struct {
|
||||||
const DefaultMaxHostsCount = 10000
|
const DefaultMaxHostsCount = 10000
|
||||||
|
|
||||||
// New returns a new host max errors cache
|
// New returns a new host max errors cache
|
||||||
func New(MaxHostError, maxHostsCount int) *Cache {
|
func New(maxHostError, maxHostsCount int) *Cache {
|
||||||
gc := gcache.New(maxHostsCount).
|
gc := gcache.New(maxHostsCount).
|
||||||
ARC().
|
ARC().
|
||||||
Build()
|
Build()
|
||||||
return &Cache{failedTargets: gc, MaxHostError: MaxHostError}
|
return &Cache{failedTargets: gc, MaxHostError: maxHostError}
|
||||||
}
|
}
|
||||||
|
|
||||||
// SetVerbose sets the cache to log at verbose level
|
// SetVerbose sets the cache to log at verbose level
|
||||||
|
@ -46,7 +47,6 @@ func (c *Cache) normalizeCacheValue(value string) string {
|
||||||
finalValue := value
|
finalValue := value
|
||||||
if strings.HasPrefix(value, "http") {
|
if strings.HasPrefix(value, "http") {
|
||||||
if parsed, err := url.Parse(value); err == nil {
|
if parsed, err := url.Parse(value); err == nil {
|
||||||
|
|
||||||
hostname := parsed.Host
|
hostname := parsed.Host
|
||||||
finalPort := parsed.Port()
|
finalPort := parsed.Port()
|
||||||
if finalPort == "" {
|
if finalPort == "" {
|
||||||
|
|
|
@ -11,15 +11,15 @@ func TestCacheCheckMarkFailed(t *testing.T) {
|
||||||
|
|
||||||
cache.MarkFailed("http://example.com:80")
|
cache.MarkFailed("http://example.com:80")
|
||||||
if value, err := cache.failedTargets.Get("http://example.com:80"); err == nil && value != nil {
|
if value, err := cache.failedTargets.Get("http://example.com:80"); err == nil && value != nil {
|
||||||
require.Equal(t, 1, value, "could not get correct markfailed")
|
require.Equal(t, 1, value, "could not get correct number of marked failed hosts")
|
||||||
}
|
}
|
||||||
cache.MarkFailed("example.com:80")
|
cache.MarkFailed("example.com:80")
|
||||||
if value, err := cache.failedTargets.Get("example.com:80"); err == nil && value != nil {
|
if value, err := cache.failedTargets.Get("example.com:80"); err == nil && value != nil {
|
||||||
require.Equal(t, 2, value, "could not get correct markfailed")
|
require.Equal(t, 2, value, "could not get correct number of marked failed hosts")
|
||||||
}
|
}
|
||||||
cache.MarkFailed("example.com")
|
cache.MarkFailed("example.com")
|
||||||
if value, err := cache.failedTargets.Get("example.com"); err == nil && value != nil {
|
if value, err := cache.failedTargets.Get("example.com"); err == nil && value != nil {
|
||||||
require.Equal(t, 1, value, "could not get correct markfailed")
|
require.Equal(t, 1, value, "could not get correct number of marked failed hosts")
|
||||||
}
|
}
|
||||||
for i := 0; i < 3; i++ {
|
for i := 0; i < 3; i++ {
|
||||||
cache.MarkFailed("test")
|
cache.MarkFailed("test")
|
||||||
|
|
|
@ -2,6 +2,7 @@ package protocolinit
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"github.com/corpix/uarand"
|
"github.com/corpix/uarand"
|
||||||
|
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/protocolstate"
|
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/protocolstate"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/dns/dnsclientpool"
|
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/dns/dnsclientpool"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/http/httpclientpool"
|
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/http/httpclientpool"
|
||||||
|
|
|
@ -2,6 +2,7 @@ package protocolstate
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"github.com/pkg/errors"
|
"github.com/pkg/errors"
|
||||||
|
|
||||||
"github.com/projectdiscovery/fastdialer/fastdialer"
|
"github.com/projectdiscovery/fastdialer/fastdialer"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/types"
|
"github.com/projectdiscovery/nuclei/v2/pkg/types"
|
||||||
)
|
)
|
||||||
|
|
|
@ -21,7 +21,6 @@ const (
|
||||||
MX
|
MX
|
||||||
TXT
|
TXT
|
||||||
AAAA
|
AAAA
|
||||||
//limit
|
|
||||||
limit
|
limit
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
|
@ -76,7 +76,7 @@ func (request *Request) getMatchPart(part string, data output.InternalEvent) (in
|
||||||
}
|
}
|
||||||
|
|
||||||
// responseToDSLMap converts a DNS response to a map for use in DSL matching
|
// responseToDSLMap converts a DNS response to a map for use in DSL matching
|
||||||
func (request *Request) responseToDSLMap(req, resp *dns.Msg, host, matched string, tracedata *retryabledns.TraceData) output.InternalEvent {
|
func (request *Request) responseToDSLMap(req, resp *dns.Msg, host, matched string, traceData *retryabledns.TraceData) output.InternalEvent {
|
||||||
return output.InternalEvent{
|
return output.InternalEvent{
|
||||||
"host": host,
|
"host": host,
|
||||||
"matched": matched,
|
"matched": matched,
|
||||||
|
@ -91,7 +91,7 @@ func (request *Request) responseToDSLMap(req, resp *dns.Msg, host, matched strin
|
||||||
"template-info": request.options.TemplateInfo,
|
"template-info": request.options.TemplateInfo,
|
||||||
"template-path": request.options.TemplatePath,
|
"template-path": request.options.TemplatePath,
|
||||||
"type": request.Type().String(),
|
"type": request.Type().String(),
|
||||||
"trace": traceToString(tracedata, false),
|
"trace": traceToString(traceData, false),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -133,10 +133,10 @@ func questionToString(resourceRecords []dns.Question) string {
|
||||||
return buffer.String()
|
return buffer.String()
|
||||||
}
|
}
|
||||||
|
|
||||||
func traceToString(tracedata *retryabledns.TraceData, withSteps bool) string {
|
func traceToString(traceData *retryabledns.TraceData, withSteps bool) string {
|
||||||
buffer := &bytes.Buffer{}
|
buffer := &bytes.Buffer{}
|
||||||
if tracedata != nil {
|
if traceData != nil {
|
||||||
for i, dnsRecord := range tracedata.DNSData {
|
for i, dnsRecord := range traceData.DNSData {
|
||||||
if withSteps {
|
if withSteps {
|
||||||
buffer.WriteString(fmt.Sprintf("request %d to resolver %s:\n", i, strings.Join(dnsRecord.Resolver, ",")))
|
buffer.WriteString(fmt.Sprintf("request %d to resolver %s:\n", i, strings.Join(dnsRecord.Resolver, ",")))
|
||||||
}
|
}
|
||||||
|
|
|
@ -12,8 +12,8 @@ import (
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/expressions"
|
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/expressions"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/helpers/eventcreator"
|
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/helpers/eventcreator"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/helpers/responsehighlighter"
|
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/helpers/responsehighlighter"
|
||||||
"github.com/projectdiscovery/retryabledns"
|
|
||||||
templateTypes "github.com/projectdiscovery/nuclei/v2/pkg/templates/types"
|
templateTypes "github.com/projectdiscovery/nuclei/v2/pkg/templates/types"
|
||||||
|
"github.com/projectdiscovery/retryabledns"
|
||||||
)
|
)
|
||||||
|
|
||||||
var _ protocols.Request = &Request{}
|
var _ protocols.Request = &Request{}
|
||||||
|
@ -74,15 +74,15 @@ func (request *Request) ExecuteWithResults(input string, metadata /*TODO review
|
||||||
gologger.Verbose().Msgf("[%s] Sent DNS request to %s\n", request.options.TemplateID, domain)
|
gologger.Verbose().Msgf("[%s] Sent DNS request to %s\n", request.options.TemplateID, domain)
|
||||||
|
|
||||||
// perform trace if necessary
|
// perform trace if necessary
|
||||||
var tracedata *retryabledns.TraceData
|
var traceData *retryabledns.TraceData
|
||||||
if request.Trace {
|
if request.Trace {
|
||||||
tracedata, err = request.dnsClient.Trace(domain, request.question, request.TraceMaxRecursion)
|
traceData, err = request.dnsClient.Trace(domain, request.question, request.TraceMaxRecursion)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
request.options.Output.Request(request.options.TemplatePath, domain, "dns", err)
|
request.options.Output.Request(request.options.TemplatePath, domain, "dns", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
outputEvent := request.responseToDSLMap(compiledRequest, response, input, input, tracedata)
|
outputEvent := request.responseToDSLMap(compiledRequest, response, input, input, traceData)
|
||||||
for k, v := range previous {
|
for k, v := range previous {
|
||||||
outputEvent[k] = v
|
outputEvent[k] = v
|
||||||
}
|
}
|
||||||
|
@ -92,7 +92,7 @@ func (request *Request) ExecuteWithResults(input string, metadata /*TODO review
|
||||||
|
|
||||||
dumpResponse(event, request.options, response.String(), domain)
|
dumpResponse(event, request.options, response.String(), domain)
|
||||||
if request.Trace {
|
if request.Trace {
|
||||||
dumpTraceData(event, request.options, traceToString(tracedata, true), domain)
|
dumpTraceData(event, request.options, traceToString(traceData, true), domain)
|
||||||
}
|
}
|
||||||
|
|
||||||
callback(event)
|
callback(event)
|
||||||
|
@ -112,15 +112,15 @@ func dumpResponse(event *output.InternalWrappedEvent, requestOptions *protocols.
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func dumpTraceData(event *output.InternalWrappedEvent, requestOptions *protocols.ExecuterOptions, tracedata, domain string) {
|
func dumpTraceData(event *output.InternalWrappedEvent, requestOptions *protocols.ExecuterOptions, traceData, domain string) {
|
||||||
cliOptions := requestOptions.Options
|
cliOptions := requestOptions.Options
|
||||||
if cliOptions.Debug || cliOptions.DebugResponse {
|
if cliOptions.Debug || cliOptions.DebugResponse {
|
||||||
hexDump := false
|
hexDump := false
|
||||||
if responsehighlighter.HasBinaryContent(tracedata) {
|
if responsehighlighter.HasBinaryContent(traceData) {
|
||||||
hexDump = true
|
hexDump = true
|
||||||
tracedata = hex.Dump([]byte(tracedata))
|
traceData = hex.Dump([]byte(traceData))
|
||||||
}
|
}
|
||||||
highlightedResponse := responsehighlighter.Highlight(event.OperatorsResult, tracedata, cliOptions.NoColor, hexDump)
|
highlightedResponse := responsehighlighter.Highlight(event.OperatorsResult, traceData, cliOptions.NoColor, hexDump)
|
||||||
gologger.Debug().Msgf("[%s] Dumped DNS Trace data for %s\n\n%s", requestOptions.TemplateID, domain, highlightedResponse)
|
gologger.Debug().Msgf("[%s] Dumped DNS Trace data for %s\n\n%s", requestOptions.TemplateID, domain, highlightedResponse)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
@ -44,7 +44,7 @@ func TestFindInputPaths(t *testing.T) {
|
||||||
"test.js": "TEST",
|
"test.js": "TEST",
|
||||||
}
|
}
|
||||||
for k, v := range files {
|
for k, v := range files {
|
||||||
err = ioutil.WriteFile(filepath.Join(tempDir, k), []byte(v), 0777)
|
err = ioutil.WriteFile(filepath.Join(tempDir, k), []byte(v), os.ModePerm)
|
||||||
require.Nil(t, err, "could not write temporary file")
|
require.Nil(t, err, "could not write temporary file")
|
||||||
}
|
}
|
||||||
expected := []string{"config.yaml", "final.yaml", "test.js"}
|
expected := []string{"config.yaml", "final.yaml", "test.js"}
|
||||||
|
|
|
@ -57,7 +57,7 @@ func TestFileExecuteWithResults(t *testing.T) {
|
||||||
"config.yaml": "TEST\r\n1.1.1.1\r\n",
|
"config.yaml": "TEST\r\n1.1.1.1\r\n",
|
||||||
}
|
}
|
||||||
for k, v := range files {
|
for k, v := range files {
|
||||||
err = ioutil.WriteFile(filepath.Join(tempDir, k), []byte(v), 0777)
|
err = ioutil.WriteFile(filepath.Join(tempDir, k), []byte(v), os.ModePerm)
|
||||||
require.Nil(t, err, "could not write temporary file")
|
require.Nil(t, err, "could not write temporary file")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
@ -21,7 +21,7 @@ import (
|
||||||
type Browser struct {
|
type Browser struct {
|
||||||
customAgent string
|
customAgent string
|
||||||
tempDir string
|
tempDir string
|
||||||
previouspids map[int32]struct{} // track already running pids
|
previousPIDs map[int32]struct{} // track already running PIDs
|
||||||
engine *rod.Browser
|
engine *rod.Browser
|
||||||
httpclient *http.Client
|
httpclient *http.Client
|
||||||
options *types.Options
|
options *types.Options
|
||||||
|
@ -33,7 +33,7 @@ func New(options *types.Options) (*Browser, error) {
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, errors.Wrap(err, "could not create temporary directory")
|
return nil, errors.Wrap(err, "could not create temporary directory")
|
||||||
}
|
}
|
||||||
previouspids := findChromeProcesses()
|
previousPIDs := findChromeProcesses()
|
||||||
|
|
||||||
chromeLauncher := launcher.New().
|
chromeLauncher := launcher.New().
|
||||||
Leakless(false).
|
Leakless(false).
|
||||||
|
@ -89,7 +89,7 @@ func New(options *types.Options) (*Browser, error) {
|
||||||
customAgent = uarand.GetRandom()
|
customAgent = uarand.GetRandom()
|
||||||
}
|
}
|
||||||
|
|
||||||
httpclient, err := newhttpClient(options)
|
httpclient, err := newHttpClient(options)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
@ -101,7 +101,7 @@ func New(options *types.Options) (*Browser, error) {
|
||||||
httpclient: httpclient,
|
httpclient: httpclient,
|
||||||
options: options,
|
options: options,
|
||||||
}
|
}
|
||||||
engine.previouspids = previouspids
|
engine.previousPIDs = previousPIDs
|
||||||
return engine, nil
|
return engine, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -123,7 +123,7 @@ func (b *Browser) killChromeProcesses() {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
// skip chrome processes that were already running
|
// skip chrome processes that were already running
|
||||||
if _, ok := b.previouspids[process.Pid]; ok {
|
if _, ok := b.previousPIDs[process.Pid]; ok {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
_ = process.Kill()
|
_ = process.Kill()
|
||||||
|
|
|
@ -12,13 +12,14 @@ import (
|
||||||
|
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/utils"
|
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/utils"
|
||||||
|
|
||||||
|
"golang.org/x/net/proxy"
|
||||||
|
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/protocolstate"
|
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/protocolstate"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/types"
|
"github.com/projectdiscovery/nuclei/v2/pkg/types"
|
||||||
"golang.org/x/net/proxy"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// newhttpClient creates a new http client for headless communication with a timeout
|
// newHttpClient creates a new http client for headless communication with a timeout
|
||||||
func newhttpClient(options *types.Options) (*http.Client, error) {
|
func newHttpClient(options *types.Options) (*http.Client, error) {
|
||||||
dialer := protocolstate.Dialer
|
dialer := protocolstate.Dialer
|
||||||
|
|
||||||
// Set the base TLS configuration definition
|
// Set the base TLS configuration definition
|
||||||
|
|
|
@ -399,12 +399,12 @@ func (p *Page) SelectInputElement(act *Action, out map[string]string /*TODO revi
|
||||||
return errors.Wrap(err, "could not scroll into view")
|
return errors.Wrap(err, "could not scroll into view")
|
||||||
}
|
}
|
||||||
|
|
||||||
selectedbool := false
|
selectedBool := false
|
||||||
if act.GetArg("selected") == "true" {
|
if act.GetArg("selected") == "true" {
|
||||||
selectedbool = true
|
selectedBool = true
|
||||||
}
|
}
|
||||||
by := act.GetArg("selector")
|
by := act.GetArg("selector")
|
||||||
if err := element.Select([]string{value}, selectedbool, selectorBy(by)); err != nil {
|
if err := element.Select([]string{value}, selectedBool, selectorBy(by)); err != nil {
|
||||||
return errors.Wrap(err, "could not select input")
|
return errors.Wrap(err, "could not select input")
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
|
@ -509,7 +509,7 @@ func (p *Page) WaitEvent(act *Action, out map[string]string /*TODO review unused
|
||||||
protoEvent := &protoEvent{event: event}
|
protoEvent := &protoEvent{event: event}
|
||||||
|
|
||||||
// Uses another instance in order to be able to chain the timeout only to the wait operation
|
// Uses another instance in order to be able to chain the timeout only to the wait operation
|
||||||
pagec := p.page
|
pageCopy := p.page
|
||||||
timeout := act.GetArg("timeout")
|
timeout := act.GetArg("timeout")
|
||||||
if timeout != "" {
|
if timeout != "" {
|
||||||
ts, err := strconv.Atoi(timeout)
|
ts, err := strconv.Atoi(timeout)
|
||||||
|
@ -517,11 +517,11 @@ func (p *Page) WaitEvent(act *Action, out map[string]string /*TODO review unused
|
||||||
return errors.Wrap(err, "could not get timeout")
|
return errors.Wrap(err, "could not get timeout")
|
||||||
}
|
}
|
||||||
if ts > 0 {
|
if ts > 0 {
|
||||||
pagec = p.page.Timeout(time.Duration(ts) * time.Second)
|
pageCopy = p.page.Timeout(time.Duration(ts) * time.Second)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
// Just wait the event to happen
|
// Just wait the event to happen
|
||||||
pagec.WaitEvent(protoEvent)()
|
pageCopy.WaitEvent(protoEvent)()
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
@ -28,7 +28,7 @@ func TestActionNavigate(t *testing.T) {
|
||||||
</body>
|
</body>
|
||||||
</html>`
|
</html>`
|
||||||
|
|
||||||
actions := []*Action{{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}}, {ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitLoad)}}}
|
actions := []*Action{{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}}, {ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}}}
|
||||||
|
|
||||||
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
|
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
|
||||||
require.Nil(t, err, "could not run page actions")
|
require.Nil(t, err, "could not run page actions")
|
||||||
|
@ -50,9 +50,9 @@ func TestActionScript(t *testing.T) {
|
||||||
|
|
||||||
t.Run("run-and-results", func(t *testing.T) {
|
t.Run("run-and-results", func(t *testing.T) {
|
||||||
actions := []*Action{
|
actions := []*Action{
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitLoad)}},
|
{ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionScript)}, Name: "test", Data: map[string]string{"code": "window.test"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionScript}, Name: "test", Data: map[string]string{"code": "window.test"}},
|
||||||
}
|
}
|
||||||
|
|
||||||
testHeadlessSimpleResponse(t, response, actions, timeout, func(page *Page, err error, out map[string]string) {
|
testHeadlessSimpleResponse(t, response, actions, timeout, func(page *Page, err error, out map[string]string) {
|
||||||
|
@ -64,10 +64,10 @@ func TestActionScript(t *testing.T) {
|
||||||
|
|
||||||
t.Run("hook", func(t *testing.T) {
|
t.Run("hook", func(t *testing.T) {
|
||||||
actions := []*Action{
|
actions := []*Action{
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionScript)}, Data: map[string]string{"code": "window.test = 'some-data';", "hook": "true"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionScript}, Data: map[string]string{"code": "window.test = 'some-data';", "hook": "true"}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitLoad)}},
|
{ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionScript)}, Name: "test", Data: map[string]string{"code": "window.test"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionScript}, Name: "test", Data: map[string]string{"code": "window.test"}},
|
||||||
}
|
}
|
||||||
testHeadlessSimpleResponse(t, response, actions, timeout, func(page *Page, err error, out map[string]string) {
|
testHeadlessSimpleResponse(t, response, actions, timeout, func(page *Page, err error, out map[string]string) {
|
||||||
require.Nil(t, err, "could not run page actions")
|
require.Nil(t, err, "could not run page actions")
|
||||||
|
@ -88,9 +88,9 @@ func TestActionClick(t *testing.T) {
|
||||||
</html>`
|
</html>`
|
||||||
|
|
||||||
actions := []*Action{
|
actions := []*Action{
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitLoad)}},
|
{ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionClick)}, Data: map[string]string{"selector": "button"}}, // Use css selector for clicking
|
{ActionType: ActionTypeHolder{ActionType: ActionClick}, Data: map[string]string{"selector": "button"}}, // Use css selector for clicking
|
||||||
}
|
}
|
||||||
|
|
||||||
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
|
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
|
||||||
|
@ -121,9 +121,9 @@ func TestActionRightClick(t *testing.T) {
|
||||||
</html>`
|
</html>`
|
||||||
|
|
||||||
actions := []*Action{
|
actions := []*Action{
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitLoad)}},
|
{ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionRightClick)}, Data: map[string]string{"selector": "button"}}, // Use css selector for clicking
|
{ActionType: ActionTypeHolder{ActionType: ActionRightClick}, Data: map[string]string{"selector": "button"}}, // Use css selector for clicking
|
||||||
}
|
}
|
||||||
|
|
||||||
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
|
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
|
||||||
|
@ -146,9 +146,9 @@ func TestActionTextInput(t *testing.T) {
|
||||||
</html>`
|
</html>`
|
||||||
|
|
||||||
actions := []*Action{
|
actions := []*Action{
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitLoad)}},
|
{ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionTextInput)}, Data: map[string]string{"selector": "input", "value": "test"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionTextInput}, Data: map[string]string{"selector": "input", "value": "test"}},
|
||||||
}
|
}
|
||||||
|
|
||||||
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
|
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
|
||||||
|
@ -163,9 +163,9 @@ func TestActionTextInput(t *testing.T) {
|
||||||
|
|
||||||
func TestActionHeadersChange(t *testing.T) {
|
func TestActionHeadersChange(t *testing.T) {
|
||||||
actions := []*Action{
|
actions := []*Action{
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionSetHeader)}, Data: map[string]string{"part": "request", "key": "Test", "value": "Hello"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionSetHeader}, Data: map[string]string{"part": "request", "key": "Test", "value": "Hello"}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitLoad)}},
|
{ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
|
||||||
}
|
}
|
||||||
|
|
||||||
handler := func(w http.ResponseWriter, r *http.Request) {
|
handler := func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
@ -190,9 +190,9 @@ func TestActionScreenshot(t *testing.T) {
|
||||||
</html>`
|
</html>`
|
||||||
|
|
||||||
actions := []*Action{
|
actions := []*Action{
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitLoad)}},
|
{ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionScreenshot)}, Data: map[string]string{"to": "test"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionScreenshot}, Data: map[string]string{"to": "test"}},
|
||||||
}
|
}
|
||||||
|
|
||||||
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
|
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
|
||||||
|
@ -215,9 +215,9 @@ func TestActionTimeInput(t *testing.T) {
|
||||||
</html>`
|
</html>`
|
||||||
|
|
||||||
actions := []*Action{
|
actions := []*Action{
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitLoad)}},
|
{ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionTimeInput)}, Data: map[string]string{"selector": "input", "value": "2006-01-02T15:04:05Z"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionTimeInput}, Data: map[string]string{"selector": "input", "value": "2006-01-02T15:04:05Z"}},
|
||||||
}
|
}
|
||||||
|
|
||||||
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
|
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
|
||||||
|
@ -243,9 +243,9 @@ func TestActionSelectInput(t *testing.T) {
|
||||||
</html>`
|
</html>`
|
||||||
|
|
||||||
actions := []*Action{
|
actions := []*Action{
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitLoad)}},
|
{ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionSelectInput)}, Data: map[string]string{"by": "x", "xpath": "//select[@id='test']", "value": "Test2", "selected": "true"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionSelectInput}, Data: map[string]string{"by": "x", "xpath": "//select[@id='test']", "value": "Test2", "selected": "true"}},
|
||||||
}
|
}
|
||||||
|
|
||||||
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
|
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
|
||||||
|
@ -266,9 +266,9 @@ func TestActionFilesInput(t *testing.T) {
|
||||||
</html>`
|
</html>`
|
||||||
|
|
||||||
actions := []*Action{
|
actions := []*Action{
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitLoad)}},
|
{ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionFilesInput)}, Data: map[string]string{"selector": "input", "value": "test1.pdf"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionFilesInput}, Data: map[string]string{"selector": "input", "value": "test1.pdf"}},
|
||||||
}
|
}
|
||||||
|
|
||||||
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
|
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
|
||||||
|
@ -292,8 +292,8 @@ func TestActionWaitLoad(t *testing.T) {
|
||||||
</html>`
|
</html>`
|
||||||
|
|
||||||
actions := []*Action{
|
actions := []*Action{
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitLoad)}},
|
{ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
|
||||||
}
|
}
|
||||||
|
|
||||||
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
|
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
|
||||||
|
@ -317,8 +317,8 @@ func TestActionGetResource(t *testing.T) {
|
||||||
</html>`
|
</html>`
|
||||||
|
|
||||||
actions := []*Action{
|
actions := []*Action{
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionGetResource)}, Data: map[string]string{"by": "x", "xpath": "//img[@id='test']"}, Name: "src"},
|
{ActionType: ActionTypeHolder{ActionType: ActionGetResource}, Data: map[string]string{"by": "x", "xpath": "//img[@id='test']"}, Name: "src"},
|
||||||
}
|
}
|
||||||
|
|
||||||
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
|
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
|
||||||
|
@ -337,8 +337,8 @@ func TestActionExtract(t *testing.T) {
|
||||||
</html>`
|
</html>`
|
||||||
|
|
||||||
actions := []*Action{
|
actions := []*Action{
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionExtract)}, Data: map[string]string{"by": "x", "xpath": "//button[@id='test']"}, Name: "extract"},
|
{ActionType: ActionTypeHolder{ActionType: ActionExtract}, Data: map[string]string{"by": "x", "xpath": "//button[@id='test']"}, Name: "extract"},
|
||||||
}
|
}
|
||||||
|
|
||||||
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
|
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
|
||||||
|
@ -356,8 +356,8 @@ func TestActionSetMethod(t *testing.T) {
|
||||||
</html>`
|
</html>`
|
||||||
|
|
||||||
actions := []*Action{
|
actions := []*Action{
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionSetMethod)}, Data: map[string]string{"part": "x", "method": "SET"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionSetMethod}, Data: map[string]string{"part": "x", "method": "SET"}},
|
||||||
}
|
}
|
||||||
|
|
||||||
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
|
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
|
||||||
|
@ -368,9 +368,9 @@ func TestActionSetMethod(t *testing.T) {
|
||||||
|
|
||||||
func TestActionAddHeader(t *testing.T) {
|
func TestActionAddHeader(t *testing.T) {
|
||||||
actions := []*Action{
|
actions := []*Action{
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionAddHeader)}, Data: map[string]string{"part": "request", "key": "Test", "value": "Hello"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionAddHeader}, Data: map[string]string{"part": "request", "key": "Test", "value": "Hello"}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitLoad)}},
|
{ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
|
||||||
}
|
}
|
||||||
|
|
||||||
handler := func(w http.ResponseWriter, r *http.Request) {
|
handler := func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
@ -387,11 +387,11 @@ func TestActionAddHeader(t *testing.T) {
|
||||||
|
|
||||||
func TestActionDeleteHeader(t *testing.T) {
|
func TestActionDeleteHeader(t *testing.T) {
|
||||||
actions := []*Action{
|
actions := []*Action{
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionAddHeader)}, Data: map[string]string{"part": "request", "key": "Test1", "value": "Hello"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionAddHeader}, Data: map[string]string{"part": "request", "key": "Test1", "value": "Hello"}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionAddHeader)}, Data: map[string]string{"part": "request", "key": "Test2", "value": "World"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionAddHeader}, Data: map[string]string{"part": "request", "key": "Test2", "value": "World"}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionDeleteHeader)}, Data: map[string]string{"part": "request", "key": "Test2"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionDeleteHeader}, Data: map[string]string{"part": "request", "key": "Test2"}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitLoad)}},
|
{ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
|
||||||
}
|
}
|
||||||
|
|
||||||
handler := func(w http.ResponseWriter, r *http.Request) {
|
handler := func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
@ -408,9 +408,9 @@ func TestActionDeleteHeader(t *testing.T) {
|
||||||
|
|
||||||
func TestActionSetBody(t *testing.T) {
|
func TestActionSetBody(t *testing.T) {
|
||||||
actions := []*Action{
|
actions := []*Action{
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionSetBody)}, Data: map[string]string{"part": "request", "body": "hello"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionSetBody}, Data: map[string]string{"part": "request", "body": "hello"}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitLoad)}},
|
{ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
|
||||||
}
|
}
|
||||||
|
|
||||||
handler := func(w http.ResponseWriter, r *http.Request) {
|
handler := func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
@ -436,10 +436,10 @@ func TestActionKeyboard(t *testing.T) {
|
||||||
</html>`
|
</html>`
|
||||||
|
|
||||||
actions := []*Action{
|
actions := []*Action{
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitLoad)}},
|
{ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionClick)}, Data: map[string]string{"selector": "input"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionClick}, Data: map[string]string{"selector": "input"}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionKeyboard)}, Data: map[string]string{"keys": "Test2"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionKeyboard}, Data: map[string]string{"keys": "Test2"}},
|
||||||
}
|
}
|
||||||
|
|
||||||
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
|
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
|
||||||
|
@ -462,8 +462,8 @@ func TestActionSleep(t *testing.T) {
|
||||||
</html>`
|
</html>`
|
||||||
|
|
||||||
actions := []*Action{
|
actions := []*Action{
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionSleep)}, Data: map[string]string{"duration": "2"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionSleep}, Data: map[string]string{"duration": "2"}},
|
||||||
}
|
}
|
||||||
|
|
||||||
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
|
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
|
||||||
|
@ -485,8 +485,8 @@ func TestActionWaitVisible(t *testing.T) {
|
||||||
</html>`
|
</html>`
|
||||||
|
|
||||||
actions := []*Action{
|
actions := []*Action{
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
|
||||||
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitVisible)}, Data: map[string]string{"by": "x", "xpath": "//button[@id='test']"}},
|
{ActionType: ActionTypeHolder{ActionType: ActionWaitVisible}, Data: map[string]string{"by": "x", "xpath": "//button[@id='test']"}},
|
||||||
}
|
}
|
||||||
|
|
||||||
t.Run("wait for an element being visible", func(t *testing.T) {
|
t.Run("wait for an element being visible", func(t *testing.T) {
|
||||||
|
|
|
@ -8,7 +8,7 @@ import (
|
||||||
|
|
||||||
// routingRuleHandler handles proxy rule for actions related to request/response modification
|
// routingRuleHandler handles proxy rule for actions related to request/response modification
|
||||||
func (p *Page) routingRuleHandler(ctx *rod.Hijack) {
|
func (p *Page) routingRuleHandler(ctx *rod.Hijack) {
|
||||||
// usually browsers don't use chunked transfer encoding so we set the content-length nevertheless
|
// usually browsers don't use chunked transfer encoding, so we set the content-length nevertheless
|
||||||
ctx.Request.Req().ContentLength = int64(len(ctx.Request.Body()))
|
ctx.Request.Req().ContentLength = int64(len(ctx.Request.Body()))
|
||||||
|
|
||||||
for _, rule := range p.rules {
|
for _, rule := range p.rules {
|
||||||
|
@ -16,15 +16,16 @@ func (p *Page) routingRuleHandler(ctx *rod.Hijack) {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
if rule.Action == ActionSetMethod {
|
switch rule.Action {
|
||||||
|
case ActionSetMethod:
|
||||||
ctx.Request.Req().Method = rule.Args["method"]
|
ctx.Request.Req().Method = rule.Args["method"]
|
||||||
} else if rule.Action == ActionAddHeader {
|
case ActionAddHeader:
|
||||||
ctx.Request.Req().Header.Add(rule.Args["key"], rule.Args["value"])
|
ctx.Request.Req().Header.Add(rule.Args["key"], rule.Args["value"])
|
||||||
} else if rule.Action == ActionSetHeader {
|
case ActionSetHeader:
|
||||||
ctx.Request.Req().Header.Set(rule.Args["key"], rule.Args["value"])
|
ctx.Request.Req().Header.Set(rule.Args["key"], rule.Args["value"])
|
||||||
} else if rule.Action == ActionDeleteHeader {
|
case ActionDeleteHeader:
|
||||||
ctx.Request.Req().Header.Del(rule.Args["key"])
|
ctx.Request.Req().Header.Del(rule.Args["key"])
|
||||||
} else if rule.Action == ActionSetBody {
|
case ActionSetBody:
|
||||||
body := rule.Args["body"]
|
body := rule.Args["body"]
|
||||||
ctx.Request.Req().ContentLength = int64(len(body))
|
ctx.Request.Req().ContentLength = int64(len(body))
|
||||||
ctx.Request.SetBody(body)
|
ctx.Request.SetBody(body)
|
||||||
|
@ -36,13 +37,15 @@ func (p *Page) routingRuleHandler(ctx *rod.Hijack) {
|
||||||
if rule.Part != "response" {
|
if rule.Part != "response" {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
if rule.Action == ActionAddHeader {
|
|
||||||
|
switch rule.Action {
|
||||||
|
case ActionAddHeader:
|
||||||
ctx.Response.Headers().Add(rule.Args["key"], rule.Args["value"])
|
ctx.Response.Headers().Add(rule.Args["key"], rule.Args["value"])
|
||||||
} else if rule.Action == ActionSetHeader {
|
case ActionSetHeader:
|
||||||
ctx.Response.Headers().Set(rule.Args["key"], rule.Args["value"])
|
ctx.Response.Headers().Set(rule.Args["key"], rule.Args["value"])
|
||||||
} else if rule.Action == ActionDeleteHeader {
|
case ActionDeleteHeader:
|
||||||
ctx.Response.Headers().Del(rule.Args["key"])
|
ctx.Response.Headers().Del(rule.Args["key"])
|
||||||
} else if rule.Action == ActionSetBody {
|
case ActionSetBody:
|
||||||
body := rule.Args["body"]
|
body := rule.Args["body"]
|
||||||
ctx.Response.Headers().Set("Content-Length", fmt.Sprintf("%d", len(body)))
|
ctx.Response.Headers().Set("Content-Length", fmt.Sprintf("%d", len(body)))
|
||||||
ctx.Response.SetBody(rule.Args["body"])
|
ctx.Response.SetBody(rule.Args["body"])
|
||||||
|
|
|
@ -114,7 +114,7 @@ func (r *requestGenerator) makeSelfContainedRequest(data string, payloads, dynam
|
||||||
reader := bufio.NewReader(strings.NewReader(data))
|
reader := bufio.NewReader(strings.NewReader(data))
|
||||||
s, err := reader.ReadString('\n')
|
s, err := reader.ReadString('\n')
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("could not read request: %s", err)
|
return nil, fmt.Errorf("could not read request: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
parts := strings.Split(s, " ")
|
parts := strings.Split(s, " ")
|
||||||
|
@ -123,7 +123,7 @@ func (r *requestGenerator) makeSelfContainedRequest(data string, payloads, dynam
|
||||||
}
|
}
|
||||||
parsed, err := url.Parse(parts[1])
|
parsed, err := url.Parse(parts[1])
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("could not parse request URL: %s", err)
|
return nil, fmt.Errorf("could not parse request URL: %w", err)
|
||||||
}
|
}
|
||||||
values := generators.MergeMaps(
|
values := generators.MergeMaps(
|
||||||
generators.MergeMaps(dynamicValues, generateVariables(parsed, false)),
|
generators.MergeMaps(dynamicValues, generateVariables(parsed, false)),
|
||||||
|
@ -297,10 +297,12 @@ func (r *requestGenerator) fillRequest(req *http.Request, values map[string]inte
|
||||||
}
|
}
|
||||||
req.Body = ioutil.NopCloser(strings.NewReader(body))
|
req.Body = ioutil.NopCloser(strings.NewReader(body))
|
||||||
}
|
}
|
||||||
|
if !r.request.Unsafe {
|
||||||
setHeader(req, "User-Agent", uarand.GetRandom())
|
setHeader(req, "User-Agent", uarand.GetRandom())
|
||||||
|
}
|
||||||
|
|
||||||
// Only set these headers on non-raw requests
|
// Only set these headers on non-raw requests
|
||||||
if len(r.request.Raw) == 0 {
|
if len(r.request.Raw) == 0 && !r.request.Unsafe {
|
||||||
setHeader(req, "Accept", "*/*")
|
setHeader(req, "Accept", "*/*")
|
||||||
setHeader(req, "Accept-Language", "en")
|
setHeader(req, "Accept-Language", "en")
|
||||||
}
|
}
|
||||||
|
|
|
@ -5,12 +5,13 @@ import (
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/model"
|
"github.com/projectdiscovery/nuclei/v2/pkg/model"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/model/types/severity"
|
"github.com/projectdiscovery/nuclei/v2/pkg/model/types/severity"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/generators"
|
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/generators"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/interactsh"
|
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/interactsh"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
|
"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestBaseURLWithTemplatePrefs(t *testing.T) {
|
func TestBaseURLWithTemplatePrefs(t *testing.T) {
|
||||||
|
@ -138,7 +139,7 @@ func TestMakeRequestFromRawWithPayloads(t *testing.T) {
|
||||||
"username": []string{"admin"},
|
"username": []string{"admin"},
|
||||||
"password": []string{"admin", "guest", "password", "test", "12345", "123456"},
|
"password": []string{"admin", "guest", "password", "test", "12345", "123456"},
|
||||||
},
|
},
|
||||||
AttackType: generators.AttackTypeHolder{Value: generators.ClusterbombAttack},
|
AttackType: generators.AttackTypeHolder{Value: generators.ClusterBombAttack},
|
||||||
Raw: []string{`GET /manager/html HTTP/1.1
|
Raw: []string{`GET /manager/html HTTP/1.1
|
||||||
Host: {{Hostname}}
|
Host: {{Hostname}}
|
||||||
User-Agent: Nuclei - Open-source project (github.com/projectdiscovery/nuclei)
|
User-Agent: Nuclei - Open-source project (github.com/projectdiscovery/nuclei)
|
||||||
|
@ -179,7 +180,7 @@ func TestMakeRequestFromRawPayloadExpressions(t *testing.T) {
|
||||||
"username": []string{"admin"},
|
"username": []string{"admin"},
|
||||||
"password": []string{"admin", "guest", "password", "test", "12345", "123456"},
|
"password": []string{"admin", "guest", "password", "test", "12345", "123456"},
|
||||||
},
|
},
|
||||||
AttackType: generators.AttackTypeHolder{Value: generators.ClusterbombAttack},
|
AttackType: generators.AttackTypeHolder{Value: generators.ClusterBombAttack},
|
||||||
Raw: []string{`GET /manager/html HTTP/1.1
|
Raw: []string{`GET /manager/html HTTP/1.1
|
||||||
Host: {{Hostname}}
|
Host: {{Hostname}}
|
||||||
User-Agent: Nuclei - Open-source project (github.com/projectdiscovery/nuclei)
|
User-Agent: Nuclei - Open-source project (github.com/projectdiscovery/nuclei)
|
||||||
|
@ -233,7 +234,7 @@ func TestMakeRequestFromModelUniqueInteractsh(t *testing.T) {
|
||||||
ServerURL: options.InteractshURL,
|
ServerURL: options.InteractshURL,
|
||||||
CacheSize: int64(options.InteractionsCacheSize),
|
CacheSize: int64(options.InteractionsCacheSize),
|
||||||
Eviction: time.Duration(options.InteractionsEviction) * time.Second,
|
Eviction: time.Duration(options.InteractionsEviction) * time.Second,
|
||||||
ColldownPeriod: time.Duration(options.InteractionsCooldownPeriod) * time.Second,
|
ColldownPeriod: time.Duration(options.InteractionsCoolDownPeriod) * time.Second,
|
||||||
PollDuration: time.Duration(options.InteractionsPollDuration) * time.Second,
|
PollDuration: time.Duration(options.InteractionsPollDuration) * time.Second,
|
||||||
})
|
})
|
||||||
require.Nil(t, err, "could not create interactsh client")
|
require.Nil(t, err, "could not create interactsh client")
|
||||||
|
|
|
@ -41,12 +41,12 @@ type Request struct {
|
||||||
// Name is the optional name of the request.
|
// Name is the optional name of the request.
|
||||||
//
|
//
|
||||||
// If a name is specified, all the named request in a template can be matched upon
|
// If a name is specified, all the named request in a template can be matched upon
|
||||||
// in a combined manner allowing multirequest based matchers.
|
// in a combined manner allowing multi-request based matchers.
|
||||||
Name string `yaml:"name,omitempty" jsonschema:"title=name for the http request,description=Optional name for the HTTP Request"`
|
Name string `yaml:"name,omitempty" jsonschema:"title=name for the http request,description=Optional name for the HTTP Request"`
|
||||||
// description: |
|
// description: |
|
||||||
// Attack is the type of payload combinations to perform.
|
// Attack is the type of payload combinations to perform.
|
||||||
//
|
//
|
||||||
// batteringram is same payload into all of the defined payload positions at once, pitchfork combines multiple payload sets and clusterbomb generates
|
// batteringram is inserts the same payload into all defined payload positions at once, pitchfork combines multiple payload sets and clusterbomb generates
|
||||||
// permutations and combinations for all payloads.
|
// permutations and combinations for all payloads.
|
||||||
// values:
|
// values:
|
||||||
// - "batteringram"
|
// - "batteringram"
|
||||||
|
@ -137,7 +137,7 @@ type Request struct {
|
||||||
dynamicValues map[string]interface{}
|
dynamicValues map[string]interface{}
|
||||||
|
|
||||||
// description: |
|
// description: |
|
||||||
// SelfContained specifies if the request is self contained.
|
// SelfContained specifies if the request is self-contained.
|
||||||
SelfContained bool `yaml:"-" json:"-"`
|
SelfContained bool `yaml:"-" json:"-"`
|
||||||
|
|
||||||
// description: |
|
// description: |
|
||||||
|
|
|
@ -22,7 +22,6 @@ const (
|
||||||
HTTPTrace
|
HTTPTrace
|
||||||
HTTPPatch
|
HTTPPatch
|
||||||
HTTPPurge
|
HTTPPurge
|
||||||
//limit
|
|
||||||
limit
|
limit
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
|
@ -23,7 +23,7 @@ func TestHTTPCompile(t *testing.T) {
|
||||||
"username": []string{"admin"},
|
"username": []string{"admin"},
|
||||||
"password": []string{"admin", "guest", "password", "test", "12345", "123456"},
|
"password": []string{"admin", "guest", "password", "test", "12345", "123456"},
|
||||||
},
|
},
|
||||||
AttackType: generators.AttackTypeHolder{Value: generators.ClusterbombAttack},
|
AttackType: generators.AttackTypeHolder{Value: generators.ClusterBombAttack},
|
||||||
Raw: []string{`GET /manager/html HTTP/1.1
|
Raw: []string{`GET /manager/html HTTP/1.1
|
||||||
Host: {{Hostname}}
|
Host: {{Hostname}}
|
||||||
User-Agent: Nuclei - Open-source project (github.com/projectdiscovery/nuclei)
|
User-Agent: Nuclei - Open-source project (github.com/projectdiscovery/nuclei)
|
||||||
|
|
|
@ -30,7 +30,7 @@ var (
|
||||||
// Dialer is a copy of the fastdialer from protocolstate
|
// Dialer is a copy of the fastdialer from protocolstate
|
||||||
Dialer *fastdialer.Dialer
|
Dialer *fastdialer.Dialer
|
||||||
|
|
||||||
rawhttpClient *rawhttp.Client
|
rawHttpClient *rawhttp.Client
|
||||||
poolMutex *sync.RWMutex
|
poolMutex *sync.RWMutex
|
||||||
normalClient *retryablehttp.Client
|
normalClient *retryablehttp.Client
|
||||||
clientPool map[string]*retryablehttp.Client
|
clientPool map[string]*retryablehttp.Client
|
||||||
|
@ -98,12 +98,12 @@ func (c *Configuration) HasStandardOptions() bool {
|
||||||
|
|
||||||
// GetRawHTTP returns the rawhttp request client
|
// GetRawHTTP returns the rawhttp request client
|
||||||
func GetRawHTTP(options *types.Options) *rawhttp.Client {
|
func GetRawHTTP(options *types.Options) *rawhttp.Client {
|
||||||
if rawhttpClient == nil {
|
if rawHttpClient == nil {
|
||||||
rawhttpOptions := rawhttp.DefaultOptions
|
rawHttpOptions := rawhttp.DefaultOptions
|
||||||
rawhttpOptions.Timeout = time.Duration(options.Timeout) * time.Second
|
rawHttpOptions.Timeout = time.Duration(options.Timeout) * time.Second
|
||||||
rawhttpClient = rawhttp.NewClient(rawhttpOptions)
|
rawHttpClient = rawhttp.NewClient(rawHttpOptions)
|
||||||
}
|
}
|
||||||
return rawhttpClient
|
return rawHttpClient
|
||||||
}
|
}
|
||||||
|
|
||||||
// Get creates or gets a client for the protocol based on custom configuration
|
// Get creates or gets a client for the protocol based on custom configuration
|
||||||
|
@ -138,7 +138,7 @@ func wrappedGet(options *types.Options, configuration *Configuration) (*retryabl
|
||||||
}
|
}
|
||||||
|
|
||||||
// Multiple Host
|
// Multiple Host
|
||||||
retryablehttpOptions := retryablehttp.DefaultOptionsSpraying
|
retryableHttpOptions := retryablehttp.DefaultOptionsSpraying
|
||||||
disableKeepAlives := true
|
disableKeepAlives := true
|
||||||
maxIdleConns := 0
|
maxIdleConns := 0
|
||||||
maxConnsPerHost := 0
|
maxConnsPerHost := 0
|
||||||
|
@ -146,14 +146,14 @@ func wrappedGet(options *types.Options, configuration *Configuration) (*retryabl
|
||||||
|
|
||||||
if configuration.Threads > 0 {
|
if configuration.Threads > 0 {
|
||||||
// Single host
|
// Single host
|
||||||
retryablehttpOptions = retryablehttp.DefaultOptionsSingle
|
retryableHttpOptions = retryablehttp.DefaultOptionsSingle
|
||||||
disableKeepAlives = false
|
disableKeepAlives = false
|
||||||
maxIdleConnsPerHost = 500
|
maxIdleConnsPerHost = 500
|
||||||
maxConnsPerHost = 500
|
maxConnsPerHost = 500
|
||||||
}
|
}
|
||||||
|
|
||||||
retryablehttpOptions.RetryWaitMax = 10 * time.Second
|
retryableHttpOptions.RetryWaitMax = 10 * time.Second
|
||||||
retryablehttpOptions.RetryMax = options.Retries
|
retryableHttpOptions.RetryMax = options.Retries
|
||||||
followRedirects := configuration.FollowRedirects
|
followRedirects := configuration.FollowRedirects
|
||||||
maxRedirects := configuration.MaxRedirects
|
maxRedirects := configuration.MaxRedirects
|
||||||
|
|
||||||
|
@ -185,7 +185,7 @@ func wrappedGet(options *types.Options, configuration *Configuration) (*retryabl
|
||||||
if proxyURL != nil {
|
if proxyURL != nil {
|
||||||
// Attempts to overwrite the dial function with the socks proxied version
|
// Attempts to overwrite the dial function with the socks proxied version
|
||||||
if proxyURL.Scheme == types.SOCKS5 {
|
if proxyURL.Scheme == types.SOCKS5 {
|
||||||
var proxyAuth *proxy.Auth = &proxy.Auth{}
|
var proxyAuth = &proxy.Auth{}
|
||||||
proxyAuth.User = proxyURL.User.Username()
|
proxyAuth.User = proxyURL.User.Username()
|
||||||
proxyAuth.Password, _ = proxyURL.User.Password()
|
proxyAuth.Password, _ = proxyURL.User.Password()
|
||||||
|
|
||||||
|
@ -213,7 +213,7 @@ func wrappedGet(options *types.Options, configuration *Configuration) (*retryabl
|
||||||
Transport: transport,
|
Transport: transport,
|
||||||
Timeout: time.Duration(options.Timeout) * time.Second,
|
Timeout: time.Duration(options.Timeout) * time.Second,
|
||||||
CheckRedirect: makeCheckRedirectFunc(followRedirects, maxRedirects),
|
CheckRedirect: makeCheckRedirectFunc(followRedirects, maxRedirects),
|
||||||
}, retryablehttpOptions)
|
}, retryableHttpOptions)
|
||||||
if jar != nil {
|
if jar != nil {
|
||||||
client.HTTPClient.Jar = jar
|
client.HTTPClient.Jar = jar
|
||||||
}
|
}
|
||||||
|
|
|
@ -13,7 +13,7 @@ type SyncedReadCloser struct {
|
||||||
data []byte
|
data []byte
|
||||||
p int64
|
p int64
|
||||||
length int64
|
length int64
|
||||||
opengate chan struct{}
|
openGate chan struct{}
|
||||||
enableBlocking bool
|
enableBlocking bool
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -29,7 +29,7 @@ func NewSyncedReadCloser(r io.ReadCloser) *SyncedReadCloser {
|
||||||
}
|
}
|
||||||
r.Close()
|
r.Close()
|
||||||
s.length = int64(len(s.data))
|
s.length = int64(len(s.data))
|
||||||
s.opengate = make(chan struct{})
|
s.openGate = make(chan struct{})
|
||||||
s.enableBlocking = true
|
s.enableBlocking = true
|
||||||
return &s
|
return &s
|
||||||
}
|
}
|
||||||
|
@ -48,13 +48,13 @@ func (s *SyncedReadCloser) SetOpenGate(status bool) {
|
||||||
|
|
||||||
// OpenGate opens the gate allowing all requests to be completed
|
// OpenGate opens the gate allowing all requests to be completed
|
||||||
func (s *SyncedReadCloser) OpenGate() {
|
func (s *SyncedReadCloser) OpenGate() {
|
||||||
s.opengate <- struct{}{}
|
s.openGate <- struct{}{}
|
||||||
}
|
}
|
||||||
|
|
||||||
// OpenGateAfter schedules gate to be opened after a duration
|
// OpenGateAfter schedules gate to be opened after a duration
|
||||||
func (s *SyncedReadCloser) OpenGateAfter(d time.Duration) {
|
func (s *SyncedReadCloser) OpenGateAfter(d time.Duration) {
|
||||||
time.AfterFunc(d, func() {
|
time.AfterFunc(d, func() {
|
||||||
s.opengate <- struct{}{}
|
s.openGate <- struct{}{}
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -84,7 +84,7 @@ func (s *SyncedReadCloser) Seek(offset int64, whence int) (int64, error) {
|
||||||
func (s *SyncedReadCloser) Read(p []byte) (n int, err error) {
|
func (s *SyncedReadCloser) Read(p []byte) (n int, err error) {
|
||||||
// If the data fits in the buffer blocks awaiting the sync instruction
|
// If the data fits in the buffer blocks awaiting the sync instruction
|
||||||
if s.p+int64(len(p)) >= s.length && s.enableBlocking {
|
if s.p+int64(len(p)) >= s.length && s.enableBlocking {
|
||||||
<-s.opengate
|
<-s.openGate
|
||||||
}
|
}
|
||||||
n = copy(p, s.data[s.p:])
|
n = copy(p, s.data[s.p:])
|
||||||
s.p += int64(n)
|
s.p += int64(n)
|
||||||
|
@ -94,7 +94,7 @@ func (s *SyncedReadCloser) Read(p []byte) (n int, err error) {
|
||||||
return n, err
|
return n, err
|
||||||
}
|
}
|
||||||
|
|
||||||
// Close implements close method for io.ReadSeeker
|
// Close closes an io.ReadSeeker
|
||||||
func (s *SyncedReadCloser) Close() error {
|
func (s *SyncedReadCloser) Close() error {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
|
@ -33,7 +33,7 @@ func Parse(request, baseURL string, unsafe bool) (*Request, error) {
|
||||||
|
|
||||||
parsedURL, err := url.Parse(baseURL)
|
parsedURL, err := url.Parse(baseURL)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("could not parse request URL: %s", err)
|
return nil, fmt.Errorf("could not parse request URL: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
if unsafe {
|
if unsafe {
|
||||||
|
@ -42,7 +42,7 @@ func Parse(request, baseURL string, unsafe bool) (*Request, error) {
|
||||||
reader := bufio.NewReader(strings.NewReader(request))
|
reader := bufio.NewReader(strings.NewReader(request))
|
||||||
s, err := reader.ReadString('\n')
|
s, err := reader.ReadString('\n')
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("could not read request: %s", err)
|
return nil, fmt.Errorf("could not read request: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
parts := strings.Split(s, " ")
|
parts := strings.Split(s, " ")
|
||||||
|
@ -57,7 +57,7 @@ func Parse(request, baseURL string, unsafe bool) (*Request, error) {
|
||||||
// Set the request Method
|
// Set the request Method
|
||||||
rawRequest.Method = parts[0]
|
rawRequest.Method = parts[0]
|
||||||
|
|
||||||
var mutlipartRequest bool
|
var multiPartRequest bool
|
||||||
// Accepts all malformed headers
|
// Accepts all malformed headers
|
||||||
var key, value string
|
var key, value string
|
||||||
for {
|
for {
|
||||||
|
@ -76,7 +76,7 @@ func Parse(request, baseURL string, unsafe bool) (*Request, error) {
|
||||||
value = p[1]
|
value = p[1]
|
||||||
}
|
}
|
||||||
if strings.Contains(key, "Content-Type") && strings.Contains(value, "multipart/") {
|
if strings.Contains(key, "Content-Type") && strings.Contains(value, "multipart/") {
|
||||||
mutlipartRequest = true
|
multiPartRequest = true
|
||||||
}
|
}
|
||||||
|
|
||||||
// in case of unsafe requests multiple headers should be accepted
|
// in case of unsafe requests multiple headers should be accepted
|
||||||
|
@ -101,7 +101,7 @@ func Parse(request, baseURL string, unsafe bool) (*Request, error) {
|
||||||
if !unsafe && strings.HasPrefix(parts[1], "http") {
|
if !unsafe && strings.HasPrefix(parts[1], "http") {
|
||||||
parsed, parseErr := url.Parse(parts[1])
|
parsed, parseErr := url.Parse(parts[1])
|
||||||
if parseErr != nil {
|
if parseErr != nil {
|
||||||
return nil, fmt.Errorf("could not parse request URL: %s", parseErr)
|
return nil, fmt.Errorf("could not parse request URL: %w", parseErr)
|
||||||
}
|
}
|
||||||
|
|
||||||
rawRequest.Path = parsed.Path
|
rawRequest.Path = parsed.Path
|
||||||
|
@ -133,10 +133,10 @@ func Parse(request, baseURL string, unsafe bool) (*Request, error) {
|
||||||
// Set the request body
|
// Set the request body
|
||||||
b, err := ioutil.ReadAll(reader)
|
b, err := ioutil.ReadAll(reader)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("could not read request body: %s", err)
|
return nil, fmt.Errorf("could not read request body: %w", err)
|
||||||
}
|
}
|
||||||
rawRequest.Data = string(b)
|
rawRequest.Data = string(b)
|
||||||
if !mutlipartRequest {
|
if !multiPartRequest {
|
||||||
rawRequest.Data = strings.TrimSuffix(rawRequest.Data, "\r\n")
|
rawRequest.Data = strings.TrimSuffix(rawRequest.Data, "\r\n")
|
||||||
}
|
}
|
||||||
return rawRequest, nil
|
return rawRequest, nil
|
||||||
|
|
|
@ -250,7 +250,8 @@ func (request *Request) ExecuteWithResults(reqURL string, dynamicValues, previou
|
||||||
reqURL = generatedHttpRequest.URL()
|
reqURL = generatedHttpRequest.URL()
|
||||||
}
|
}
|
||||||
request.dynamicValues = generatedHttpRequest.dynamicValues
|
request.dynamicValues = generatedHttpRequest.dynamicValues
|
||||||
// Check if hosts just keep erroring
|
|
||||||
|
// Check if hosts keep erroring
|
||||||
if request.options.HostErrorsCache != nil && request.options.HostErrorsCache.Check(reqURL) {
|
if request.options.HostErrorsCache != nil && request.options.HostErrorsCache.Check(reqURL) {
|
||||||
return true, nil
|
return true, nil
|
||||||
}
|
}
|
||||||
|
@ -275,6 +276,7 @@ func (request *Request) ExecuteWithResults(reqURL string, dynamicValues, previou
|
||||||
callback(event)
|
callback(event)
|
||||||
}
|
}
|
||||||
}, requestCount)
|
}, requestCount)
|
||||||
|
|
||||||
// If a variable is unresolved, skip all further requests
|
// If a variable is unresolved, skip all further requests
|
||||||
if err == errStopExecution {
|
if err == errStopExecution {
|
||||||
return true, nil
|
return true, nil
|
||||||
|
@ -345,7 +347,7 @@ func (request *Request) executeRequest(reqURL string, generatedRequest *generate
|
||||||
}
|
}
|
||||||
dumpedRequestString := string(dumpedRequest)
|
dumpedRequestString := string(dumpedRequest)
|
||||||
|
|
||||||
// Check if are there any unresolved variables. If yes, skip unless overriden by user.
|
// Check if are there any unresolved variables. If yes, skip unless overridden by user.
|
||||||
if varErr := expressions.ContainsUnresolvedVariables(dumpedRequestString); varErr != nil && !request.SkipVariablesCheck {
|
if varErr := expressions.ContainsUnresolvedVariables(dumpedRequestString); varErr != nil && !request.SkipVariablesCheck {
|
||||||
gologger.Warning().Msgf("[%s] Could not make http request for %s: %v\n", request.options.TemplateID, reqURL, varErr)
|
gologger.Warning().Msgf("[%s] Could not make http request for %s: %v\n", request.options.TemplateID, reqURL, varErr)
|
||||||
return errStopExecution
|
return errStopExecution
|
||||||
|
@ -356,7 +358,6 @@ func (request *Request) executeRequest(reqURL string, generatedRequest *generate
|
||||||
gologger.Print().Msgf("%s", dumpedRequestString)
|
gologger.Print().Msgf("%s", dumpedRequestString)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
var formedURL string
|
var formedURL string
|
||||||
var hostname string
|
var hostname string
|
||||||
timeStart := time.Now()
|
timeStart := time.Now()
|
||||||
|
@ -449,6 +450,7 @@ func (request *Request) executeRequest(reqURL string, generatedRequest *generate
|
||||||
}
|
}
|
||||||
|
|
||||||
var dumpedResponse []redirectedResponse
|
var dumpedResponse []redirectedResponse
|
||||||
|
var gotData []byte
|
||||||
// If the status code is HTTP 101, we should not proceed with reading body.
|
// If the status code is HTTP 101, we should not proceed with reading body.
|
||||||
if resp.StatusCode != http.StatusSwitchingProtocols {
|
if resp.StatusCode != http.StatusSwitchingProtocols {
|
||||||
var bodyReader io.Reader
|
var bodyReader io.Reader
|
||||||
|
@ -466,6 +468,7 @@ func (request *Request) executeRequest(reqURL string, generatedRequest *generate
|
||||||
return errors.Wrap(err, "could not read http body")
|
return errors.Wrap(err, "could not read http body")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
gotData = data
|
||||||
resp.Body.Close()
|
resp.Body.Close()
|
||||||
|
|
||||||
dumpedResponse, err = dumpResponseWithRedirectChain(resp, data)
|
dumpedResponse, err = dumpResponseWithRedirectChain(resp, data)
|
||||||
|
@ -476,14 +479,17 @@ func (request *Request) executeRequest(reqURL string, generatedRequest *generate
|
||||||
dumpedResponse = []redirectedResponse{{fullResponse: dumpedResponseHeaders, headers: dumpedResponseHeaders}}
|
dumpedResponse = []redirectedResponse{{fullResponse: dumpedResponseHeaders, headers: dumpedResponseHeaders}}
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, response := range dumpedResponse {
|
|
||||||
// if nuclei-project is enabled store the response if not previously done
|
// if nuclei-project is enabled store the response if not previously done
|
||||||
if request.options.ProjectFile != nil && !fromCache {
|
if request.options.ProjectFile != nil && !fromCache {
|
||||||
if err := request.options.ProjectFile.Set(dumpedRequest, resp, response.body); err != nil {
|
if err := request.options.ProjectFile.Set(dumpedRequest, resp, gotData); err != nil {
|
||||||
return errors.Wrap(err, "could not store in project file")
|
return errors.Wrap(err, "could not store in project file")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
for _, response := range dumpedResponse {
|
||||||
|
if response.resp == nil {
|
||||||
|
continue // Skip nil responses
|
||||||
|
}
|
||||||
matchedURL := reqURL
|
matchedURL := reqURL
|
||||||
if generatedRequest.rawRequest != nil && generatedRequest.rawRequest.FullURL != "" {
|
if generatedRequest.rawRequest != nil && generatedRequest.rawRequest.FullURL != "" {
|
||||||
matchedURL = generatedRequest.rawRequest.FullURL
|
matchedURL = generatedRequest.rawRequest.FullURL
|
||||||
|
|
|
@ -3,9 +3,10 @@ package http
|
||||||
import (
|
import (
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/catalog"
|
"github.com/projectdiscovery/nuclei/v2/pkg/catalog"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/generators"
|
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/generators"
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestRequestGeneratorPaths(t *testing.T) {
|
func TestRequestGeneratorPaths(t *testing.T) {
|
||||||
|
@ -29,7 +30,7 @@ func TestRequestGeneratorClusterBombSingle(t *testing.T) {
|
||||||
|
|
||||||
req := &Request{
|
req := &Request{
|
||||||
Payloads: map[string]interface{}{"username": []string{"admin", "tomcat", "manager"}, "password": []string{"password", "test", "secret"}},
|
Payloads: map[string]interface{}{"username": []string{"admin", "tomcat", "manager"}, "password": []string{"password", "test", "secret"}},
|
||||||
AttackType: generators.AttackTypeHolder{Value: generators.ClusterbombAttack},
|
AttackType: generators.AttackTypeHolder{Value: generators.ClusterBombAttack},
|
||||||
Raw: []string{`GET /{{username}}:{{password}} HTTP/1.1`},
|
Raw: []string{`GET /{{username}}:{{password}} HTTP/1.1`},
|
||||||
}
|
}
|
||||||
catalogInstance := catalog.New("")
|
catalogInstance := catalog.New("")
|
||||||
|
@ -53,7 +54,7 @@ func TestRequestGeneratorClusterBombMultipleRaw(t *testing.T) {
|
||||||
|
|
||||||
req := &Request{
|
req := &Request{
|
||||||
Payloads: map[string]interface{}{"username": []string{"admin", "tomcat", "manager"}, "password": []string{"password", "test", "secret"}},
|
Payloads: map[string]interface{}{"username": []string{"admin", "tomcat", "manager"}, "password": []string{"password", "test", "secret"}},
|
||||||
AttackType: generators.AttackTypeHolder{Value: generators.ClusterbombAttack},
|
AttackType: generators.AttackTypeHolder{Value: generators.ClusterBombAttack},
|
||||||
Raw: []string{`GET /{{username}}:{{password}} HTTP/1.1`, `GET /{{username}}@{{password}} HTTP/1.1`},
|
Raw: []string{`GET /{{username}}:{{password}} HTTP/1.1`, `GET /{{username}}@{{password}} HTTP/1.1`},
|
||||||
}
|
}
|
||||||
catalogInstance := catalog.New("")
|
catalogInstance := catalog.New("")
|
||||||
|
|
|
@ -11,11 +11,12 @@ import (
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"github.com/pkg/errors"
|
"github.com/pkg/errors"
|
||||||
|
"golang.org/x/text/encoding/simplifiedchinese"
|
||||||
|
"golang.org/x/text/transform"
|
||||||
|
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/generators"
|
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/generators"
|
||||||
"github.com/projectdiscovery/rawhttp"
|
"github.com/projectdiscovery/rawhttp"
|
||||||
"github.com/projectdiscovery/stringsutil"
|
"github.com/projectdiscovery/stringsutil"
|
||||||
"golang.org/x/text/encoding/simplifiedchinese"
|
|
||||||
"golang.org/x/text/transform"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
type redirectedResponse struct {
|
type redirectedResponse struct {
|
||||||
|
@ -97,13 +98,13 @@ func normalizeResponseBody(resp *http.Response, response *redirectedResponse) er
|
||||||
// gb18030 supersedes gb2312
|
// gb18030 supersedes gb2312
|
||||||
responseContentType := resp.Header.Get("Content-Type")
|
responseContentType := resp.Header.Get("Content-Type")
|
||||||
if isContentTypeGbk(responseContentType) {
|
if isContentTypeGbk(responseContentType) {
|
||||||
response.fullResponse, err = decodegbk(response.fullResponse)
|
response.fullResponse, err = decodeGBK(response.fullResponse)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return errors.Wrap(err, "could not gbk decode")
|
return errors.Wrap(err, "could not gbk decode")
|
||||||
}
|
}
|
||||||
|
|
||||||
// the uncompressed body needs to be decoded to standard utf8
|
// the uncompressed body needs to be decoded to standard utf8
|
||||||
response.body, err = decodegbk(response.body)
|
response.body, err = decodeGBK(response.body)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return errors.Wrap(err, "could not gbk decode")
|
return errors.Wrap(err, "could not gbk decode")
|
||||||
}
|
}
|
||||||
|
@ -149,8 +150,8 @@ func handleDecompression(resp *http.Response, bodyOrig []byte) (bodyDec []byte,
|
||||||
return bodyDec, nil
|
return bodyDec, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// decodegbk converts GBK to UTF-8
|
// decodeGBK converts GBK to UTF-8
|
||||||
func decodegbk(s []byte) ([]byte, error) {
|
func decodeGBK(s []byte) ([]byte, error) {
|
||||||
I := bytes.NewReader(s)
|
I := bytes.NewReader(s)
|
||||||
O := transform.NewReader(I, simplifiedchinese.GBK.NewDecoder())
|
O := transform.NewReader(I, simplifiedchinese.GBK.NewDecoder())
|
||||||
d, e := ioutil.ReadAll(O)
|
d, e := ioutil.ReadAll(O)
|
||||||
|
|
|
@ -1,7 +1,6 @@
|
||||||
package network
|
package network
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"net"
|
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"github.com/pkg/errors"
|
"github.com/pkg/errors"
|
||||||
|
@ -34,7 +33,7 @@ type Request struct {
|
||||||
// description: |
|
// description: |
|
||||||
// Attack is the type of payload combinations to perform.
|
// Attack is the type of payload combinations to perform.
|
||||||
//
|
//
|
||||||
// Batteringram is same payload into all of the defined payload positions at once, pitchfork combines multiple payload sets and clusterbomb generates
|
// Batteringram is inserts the same payload into all defined payload positions at once, pitchfork combines multiple payload sets and clusterbomb generates
|
||||||
// permutations and combinations for all payloads.
|
// permutations and combinations for all payloads.
|
||||||
// values:
|
// values:
|
||||||
// - "batteringram"
|
// - "batteringram"
|
||||||
|
@ -68,7 +67,7 @@ type Request struct {
|
||||||
ReadAll bool `yaml:"read-all,omitempty" jsonschema:"title=read all response stream,description=Read all response stream till the server stops sending"`
|
ReadAll bool `yaml:"read-all,omitempty" jsonschema:"title=read all response stream,description=Read all response stream till the server stops sending"`
|
||||||
|
|
||||||
// description: |
|
// description: |
|
||||||
// SelfContained specifies if the request is self contained.
|
// SelfContained specifies if the request is self-contained.
|
||||||
SelfContained bool `yaml:"-" json:"-"`
|
SelfContained bool `yaml:"-" json:"-"`
|
||||||
|
|
||||||
// Operators for the current request go here.
|
// Operators for the current request go here.
|
||||||
|
@ -83,8 +82,7 @@ type Request struct {
|
||||||
}
|
}
|
||||||
|
|
||||||
type addressKV struct {
|
type addressKV struct {
|
||||||
ip string
|
address string
|
||||||
port string
|
|
||||||
tls bool
|
tls bool
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -110,7 +108,7 @@ type Input struct {
|
||||||
// Read is the number of bytes to read from socket.
|
// Read is the number of bytes to read from socket.
|
||||||
//
|
//
|
||||||
// This can be used for protocols which expect an immediate response. You can
|
// This can be used for protocols which expect an immediate response. You can
|
||||||
// read and write responses one after another and evetually perform matching
|
// read and write responses one after another and eventually perform matching
|
||||||
// on every data captured with `name` attribute.
|
// on every data captured with `name` attribute.
|
||||||
//
|
//
|
||||||
// The [network docs](https://nuclei.projectdiscovery.io/templating-guide/protocols/network/) highlight more on how to do this.
|
// The [network docs](https://nuclei.projectdiscovery.io/templating-guide/protocols/network/) highlight more on how to do this.
|
||||||
|
@ -141,15 +139,7 @@ func (request *Request) Compile(options *protocols.ExecuterOptions) error {
|
||||||
shouldUseTLS = true
|
shouldUseTLS = true
|
||||||
address = strings.TrimPrefix(address, "tls://")
|
address = strings.TrimPrefix(address, "tls://")
|
||||||
}
|
}
|
||||||
if strings.Contains(address, ":") {
|
request.addresses = append(request.addresses, addressKV{address: address, tls: shouldUseTLS})
|
||||||
addressHost, addressPort, portErr := net.SplitHostPort(address)
|
|
||||||
if portErr != nil {
|
|
||||||
return errors.Wrap(portErr, "could not parse address")
|
|
||||||
}
|
|
||||||
request.addresses = append(request.addresses, addressKV{ip: addressHost, port: addressPort, tls: shouldUseTLS})
|
|
||||||
} else {
|
|
||||||
request.addresses = append(request.addresses, addressKV{ip: address, tls: shouldUseTLS})
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
// Pre-compile any input dsl functions before executing the request.
|
// Pre-compile any input dsl functions before executing the request.
|
||||||
for _, input := range request.Inputs {
|
for _, input := range request.Inputs {
|
||||||
|
|
|
@ -14,7 +14,6 @@ type NetworkInputType int
|
||||||
const (
|
const (
|
||||||
hexType NetworkInputType = iota + 1
|
hexType NetworkInputType = iota + 1
|
||||||
textType
|
textType
|
||||||
//limit
|
|
||||||
limit
|
limit
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
|
@ -17,7 +17,7 @@ func TestNetworkCompileMake(t *testing.T) {
|
||||||
templateID := "testing-network"
|
templateID := "testing-network"
|
||||||
request := &Request{
|
request := &Request{
|
||||||
ID: templateID,
|
ID: templateID,
|
||||||
Address: []string{"{{Hostname}}", "{{Hostname}}:8082", "tls://{{Hostname}}:443"},
|
Address: []string{"tls://{{Host}}:443"},
|
||||||
ReadSize: 1024,
|
ReadSize: 1024,
|
||||||
Inputs: []*Input{{Data: "test-data"}},
|
Inputs: []*Input{{Data: "test-data"}},
|
||||||
}
|
}
|
||||||
|
@ -28,17 +28,8 @@ func TestNetworkCompileMake(t *testing.T) {
|
||||||
err := request.Compile(executerOpts)
|
err := request.Compile(executerOpts)
|
||||||
require.Nil(t, err, "could not compile network request")
|
require.Nil(t, err, "could not compile network request")
|
||||||
|
|
||||||
require.Equal(t, 3, len(request.addresses), "could not get correct number of input address")
|
require.Equal(t, 1, len(request.addresses), "could not get correct number of input address")
|
||||||
t.Run("check-host", func(t *testing.T) {
|
|
||||||
require.Equal(t, "{{Hostname}}", request.addresses[0].ip, "could not get correct host")
|
|
||||||
})
|
|
||||||
t.Run("check-host-with-port", func(t *testing.T) {
|
|
||||||
require.Equal(t, "{{Hostname}}", request.addresses[1].ip, "could not get correct host with port")
|
|
||||||
require.Equal(t, "8082", request.addresses[1].port, "could not get correct port for host")
|
|
||||||
})
|
|
||||||
t.Run("check-tls-with-port", func(t *testing.T) {
|
t.Run("check-tls-with-port", func(t *testing.T) {
|
||||||
require.Equal(t, "{{Hostname}}", request.addresses[2].ip, "could not get correct host with port")
|
require.True(t, request.addresses[0].tls, "could not get correct port for host")
|
||||||
require.Equal(t, "443", request.addresses[2].port, "could not get correct port for host")
|
|
||||||
require.True(t, request.addresses[2].tls, "could not get correct port for host")
|
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
|
@ -49,18 +49,10 @@ func (request *Request) ExecuteWithResults(input string, metadata /*TODO review
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, kv := range request.addresses {
|
for _, kv := range request.addresses {
|
||||||
actualAddress := replacer.Replace(kv.ip, map[string]interface{}{"Hostname": address})
|
variables := generateNetworkVariables(address)
|
||||||
if kv.port != "" {
|
actualAddress := replacer.Replace(kv.address, variables)
|
||||||
if strings.Contains(address, ":") {
|
|
||||||
actualAddress, _, _ = net.SplitHostPort(actualAddress)
|
|
||||||
}
|
|
||||||
actualAddress = net.JoinHostPort(actualAddress, kv.port)
|
|
||||||
}
|
|
||||||
if input != "" {
|
|
||||||
input = actualAddress
|
|
||||||
}
|
|
||||||
|
|
||||||
if err := request.executeAddress(actualAddress, address, input, kv.tls, previous, callback); err != nil {
|
if err := request.executeAddress(variables, actualAddress, address, input, kv.tls, previous, callback); err != nil {
|
||||||
gologger.Verbose().Label("ERR").Msgf("Could not make network request for %s: %s\n", actualAddress, err)
|
gologger.Verbose().Label("ERR").Msgf("Could not make network request for %s: %s\n", actualAddress, err)
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
@ -69,7 +61,7 @@ func (request *Request) ExecuteWithResults(input string, metadata /*TODO review
|
||||||
}
|
}
|
||||||
|
|
||||||
// executeAddress executes the request for an address
|
// executeAddress executes the request for an address
|
||||||
func (request *Request) executeAddress(actualAddress, address, input string, shouldUseTLS bool, previous output.InternalEvent, callback protocols.OutputEventCallback) error {
|
func (request *Request) executeAddress(variables map[string]interface{}, actualAddress, address, input string, shouldUseTLS bool, previous output.InternalEvent, callback protocols.OutputEventCallback) error {
|
||||||
if !strings.Contains(actualAddress, ":") {
|
if !strings.Contains(actualAddress, ":") {
|
||||||
err := errors.New("no port provided in network protocol request")
|
err := errors.New("no port provided in network protocol request")
|
||||||
request.options.Output.Request(request.options.TemplatePath, address, request.Type().String(), err)
|
request.options.Output.Request(request.options.TemplatePath, address, request.Type().String(), err)
|
||||||
|
@ -88,27 +80,27 @@ func (request *Request) executeAddress(actualAddress, address, input string, sho
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
value = generators.MergeMaps(value, payloads)
|
value = generators.MergeMaps(value, payloads)
|
||||||
if err := request.executeRequestWithPayloads(actualAddress, address, input, shouldUseTLS, value, previous, callback); err != nil {
|
if err := request.executeRequestWithPayloads(variables, actualAddress, address, input, shouldUseTLS, value, previous, callback); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
value := generators.MergeMaps(map[string]interface{}{}, payloads)
|
value := generators.CopyMap(payloads)
|
||||||
if err := request.executeRequestWithPayloads(actualAddress, address, input, shouldUseTLS, value, previous, callback); err != nil {
|
if err := request.executeRequestWithPayloads(variables, actualAddress, address, input, shouldUseTLS, value, previous, callback); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (request *Request) executeRequestWithPayloads(actualAddress, address, input string, shouldUseTLS bool, payloads map[string]interface{}, previous output.InternalEvent, callback protocols.OutputEventCallback) error {
|
func (request *Request) executeRequestWithPayloads(variables map[string]interface{}, actualAddress, address, input string, shouldUseTLS bool, payloads map[string]interface{}, previous output.InternalEvent, callback protocols.OutputEventCallback) error {
|
||||||
var (
|
var (
|
||||||
hostname string
|
hostname string
|
||||||
conn net.Conn
|
conn net.Conn
|
||||||
err error
|
err error
|
||||||
)
|
)
|
||||||
|
|
||||||
request.dynamicValues = generators.MergeMaps(payloads, map[string]interface{}{"Hostname": address})
|
request.dynamicValues = generators.MergeMaps(payloads, variables)
|
||||||
|
|
||||||
if host, _, splitErr := net.SplitHostPort(actualAddress); splitErr == nil {
|
if host, _, splitErr := net.SplitHostPort(actualAddress); splitErr == nil {
|
||||||
hostname = host
|
hostname = host
|
||||||
|
@ -223,12 +215,12 @@ func (request *Request) executeRequestWithPayloads(actualAddress, address, input
|
||||||
<-t.C
|
<-t.C
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
read_socket:
|
readSocket:
|
||||||
for {
|
for {
|
||||||
select {
|
select {
|
||||||
case <-readInterval.C:
|
case <-readInterval.C:
|
||||||
closeTimer(readInterval)
|
closeTimer(readInterval)
|
||||||
break read_socket
|
break readSocket
|
||||||
default:
|
default:
|
||||||
buf := make([]byte, bufferSize)
|
buf := make([]byte, bufferSize)
|
||||||
nBuf, err := conn.Read(buf)
|
nBuf, err := conn.Read(buf)
|
||||||
|
@ -328,3 +320,18 @@ func getAddress(toTest string) (string, error) {
|
||||||
}
|
}
|
||||||
return toTest, nil
|
return toTest, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func generateNetworkVariables(input string) map[string]interface{} {
|
||||||
|
if !strings.Contains(input, ":") {
|
||||||
|
return map[string]interface{}{"Hostname": input, "Host": input}
|
||||||
|
}
|
||||||
|
host, port, err := net.SplitHostPort(input)
|
||||||
|
if err != nil {
|
||||||
|
return map[string]interface{}{"Hostname": input}
|
||||||
|
}
|
||||||
|
return map[string]interface{}{
|
||||||
|
"Host": host,
|
||||||
|
"Port": port,
|
||||||
|
"Hostname": input,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
|
@ -50,7 +50,7 @@ func TestNetworkExecuteWithResults(t *testing.T) {
|
||||||
|
|
||||||
parsed, err := url.Parse(ts.URL)
|
parsed, err := url.Parse(ts.URL)
|
||||||
require.Nil(t, err, "could not parse url")
|
require.Nil(t, err, "could not parse url")
|
||||||
request.Address[0] = "{{Hostname}}:" + parsed.Port()
|
request.Address[0] = "{{Hostname}}"
|
||||||
|
|
||||||
request.Inputs = append(request.Inputs, &Input{Data: fmt.Sprintf("GET / HTTP/1.1\r\nHost: %s\r\n\r\n", parsed.Host)})
|
request.Inputs = append(request.Inputs, &Input{Data: fmt.Sprintf("GET / HTTP/1.1\r\nHost: %s\r\n\r\n", parsed.Host)})
|
||||||
executerOpts := testutils.NewMockExecuterOptions(options, &testutils.TemplateInfo{
|
executerOpts := testutils.NewMockExecuterOptions(options, &testutils.TemplateInfo{
|
||||||
|
@ -84,12 +84,7 @@ func TestNetworkExecuteWithResults(t *testing.T) {
|
||||||
})
|
})
|
||||||
require.Nil(t, err, "could not execute network request")
|
require.Nil(t, err, "could not execute network request")
|
||||||
})
|
})
|
||||||
require.NotNil(t, finalEvent, "could not get event output from request")
|
require.Nil(t, finalEvent, "could not get event output from request")
|
||||||
require.Equal(t, 1, len(finalEvent.Results), "could not get correct number of results")
|
|
||||||
require.Equal(t, "test", finalEvent.Results[0].MatcherName, "could not get correct matcher name of results")
|
|
||||||
require.Equal(t, 1, len(finalEvent.Results[0].ExtractedResults), "could not get correct number of extracted results")
|
|
||||||
require.Equal(t, "<h1>Example Domain</h1>", finalEvent.Results[0].ExtractedResults[0], "could not get correct extracted results")
|
|
||||||
finalEvent = nil
|
|
||||||
|
|
||||||
request.Inputs[0].Type = NetworkInputTypeHolder{NetworkInputType: hexType}
|
request.Inputs[0].Type = NetworkInputTypeHolder{NetworkInputType: hexType}
|
||||||
request.Inputs[0].Data = hex.EncodeToString([]byte(fmt.Sprintf("GET / HTTP/1.1\r\nHost: %s\r\n\r\n", parsed.Host)))
|
request.Inputs[0].Data = hex.EncodeToString([]byte(fmt.Sprintf("GET / HTTP/1.1\r\nHost: %s\r\n\r\n", parsed.Host)))
|
||||||
|
|
|
@ -40,7 +40,7 @@ func TestFindResponses(t *testing.T) {
|
||||||
"test.txt": "TEST",
|
"test.txt": "TEST",
|
||||||
}
|
}
|
||||||
for k, v := range files {
|
for k, v := range files {
|
||||||
err = ioutil.WriteFile(filepath.Join(tempDir, k), []byte(v), 0777)
|
err = ioutil.WriteFile(filepath.Join(tempDir, k), []byte(v), os.ModePerm)
|
||||||
require.Nil(t, err, "could not write temporary file")
|
require.Nil(t, err, "could not write temporary file")
|
||||||
}
|
}
|
||||||
expected := []string{"config.txt", "final.txt", "test.txt"}
|
expected := []string{"config.txt", "final.txt", "test.txt"}
|
||||||
|
|
|
@ -8,7 +8,7 @@ import (
|
||||||
"strings"
|
"strings"
|
||||||
)
|
)
|
||||||
|
|
||||||
var noMinor = regexp.MustCompile(`HTTP\/([0-9]) `)
|
var noMinor = regexp.MustCompile(`HTTP/([0-9]) `)
|
||||||
|
|
||||||
// readResponseFromString reads a raw http response from a string.
|
// readResponseFromString reads a raw http response from a string.
|
||||||
func readResponseFromString(data string) (*http.Response, error) {
|
func readResponseFromString(data string) (*http.Response, error) {
|
||||||
|
|
|
@ -159,7 +159,7 @@ Server: Google Frontend
|
||||||
t.Run("test-live-response-with-content-length", func(t *testing.T) {
|
t.Run("test-live-response-with-content-length", func(t *testing.T) {
|
||||||
var ts *httptest.Server
|
var ts *httptest.Server
|
||||||
router := httprouter.New()
|
router := httprouter.New()
|
||||||
router.GET("/", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, p httprouter.Params) {
|
router.GET("/", func(w http.ResponseWriter, r *http.Request, p httprouter.Params) {
|
||||||
w.Header().Add("Server", "Google Frontend")
|
w.Header().Add("Server", "Google Frontend")
|
||||||
fmt.Fprintf(w, "%s", `<!DOCTYPE html>
|
fmt.Fprintf(w, "%s", `<!DOCTYPE html>
|
||||||
<html>
|
<html>
|
||||||
|
@ -172,7 +172,7 @@ Server: Google Frontend
|
||||||
<p>
|
<p>
|
||||||
</body>
|
</body>
|
||||||
</html>`)
|
</html>`)
|
||||||
}))
|
})
|
||||||
ts = httptest.NewServer(router)
|
ts = httptest.NewServer(router)
|
||||||
defer ts.Close()
|
defer ts.Close()
|
||||||
|
|
||||||
|
|
|
@ -4,6 +4,7 @@ import (
|
||||||
"go.uber.org/ratelimit"
|
"go.uber.org/ratelimit"
|
||||||
|
|
||||||
"github.com/logrusorgru/aurora"
|
"github.com/logrusorgru/aurora"
|
||||||
|
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/catalog"
|
"github.com/projectdiscovery/nuclei/v2/pkg/catalog"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/model"
|
"github.com/projectdiscovery/nuclei/v2/pkg/model"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/operators"
|
"github.com/projectdiscovery/nuclei/v2/pkg/operators"
|
||||||
|
|
|
@ -10,6 +10,7 @@ import (
|
||||||
|
|
||||||
jsoniter "github.com/json-iterator/go"
|
jsoniter "github.com/json-iterator/go"
|
||||||
"github.com/pkg/errors"
|
"github.com/pkg/errors"
|
||||||
|
|
||||||
"github.com/projectdiscovery/cryptoutil"
|
"github.com/projectdiscovery/cryptoutil"
|
||||||
"github.com/projectdiscovery/fastdialer/fastdialer"
|
"github.com/projectdiscovery/fastdialer/fastdialer"
|
||||||
"github.com/projectdiscovery/gologger"
|
"github.com/projectdiscovery/gologger"
|
||||||
|
|
|
@ -3,11 +3,12 @@ package ssl
|
||||||
import (
|
import (
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/model"
|
"github.com/projectdiscovery/nuclei/v2/pkg/model"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/model/types/severity"
|
"github.com/projectdiscovery/nuclei/v2/pkg/model/types/severity"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/output"
|
"github.com/projectdiscovery/nuclei/v2/pkg/output"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
|
"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
|
||||||
"github.com/stretchr/testify/require"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestSSLProtocol(t *testing.T) {
|
func TestSSLProtocol(t *testing.T) {
|
||||||
|
|
|
@ -15,6 +15,7 @@ import (
|
||||||
"github.com/gobwas/ws"
|
"github.com/gobwas/ws"
|
||||||
"github.com/gobwas/ws/wsutil"
|
"github.com/gobwas/ws/wsutil"
|
||||||
"github.com/pkg/errors"
|
"github.com/pkg/errors"
|
||||||
|
|
||||||
"github.com/projectdiscovery/fastdialer/fastdialer"
|
"github.com/projectdiscovery/fastdialer/fastdialer"
|
||||||
"github.com/projectdiscovery/gologger"
|
"github.com/projectdiscovery/gologger"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/operators"
|
"github.com/projectdiscovery/nuclei/v2/pkg/operators"
|
||||||
|
@ -299,7 +300,7 @@ func (request *Request) readWriteInputWebsocket(conn net.Conn, payloadValues map
|
||||||
requestOptions.Progress.IncrementFailedRequestsBy(1)
|
requestOptions.Progress.IncrementFailedRequestsBy(1)
|
||||||
return nil, "", errors.Wrap(err, "could not write request to server")
|
return nil, "", errors.Wrap(err, "could not write request to server")
|
||||||
}
|
}
|
||||||
// Only perform matching and writes in case we recieve
|
// Only perform matching and writes in case we receive
|
||||||
// text or binary opcode from the websocket server.
|
// text or binary opcode from the websocket server.
|
||||||
if opCode != ws.OpText && opCode != ws.OpBinary {
|
if opCode != ws.OpText && opCode != ws.OpBinary {
|
||||||
continue
|
continue
|
||||||
|
|
|
@ -5,8 +5,9 @@ import (
|
||||||
"os"
|
"os"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/output"
|
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
|
|
||||||
|
"github.com/projectdiscovery/nuclei/v2/pkg/output"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestDedupeDuplicates(t *testing.T) {
|
func TestDedupeDuplicates(t *testing.T) {
|
||||||
|
|
|
@ -3,20 +3,20 @@ package es
|
||||||
import (
|
import (
|
||||||
"bytes"
|
"bytes"
|
||||||
"crypto/tls"
|
"crypto/tls"
|
||||||
|
"encoding/base64"
|
||||||
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io/ioutil"
|
"io/ioutil"
|
||||||
"net/http"
|
"net/http"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"encoding/base64"
|
|
||||||
"encoding/json"
|
|
||||||
|
|
||||||
"github.com/pkg/errors"
|
"github.com/pkg/errors"
|
||||||
|
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/output"
|
"github.com/projectdiscovery/nuclei/v2/pkg/output"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/protocolstate"
|
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/protocolstate"
|
||||||
)
|
)
|
||||||
|
|
||||||
// Options contains necessary options required for elasticsearch communicaiton
|
// Options contains necessary options required for elasticsearch communication
|
||||||
type Options struct {
|
type Options struct {
|
||||||
// IP for elasticsearch instance
|
// IP for elasticsearch instance
|
||||||
IP string `yaml:"ip" validate:"required,ip"`
|
IP string `yaml:"ip" validate:"required,ip"`
|
||||||
|
|
|
@ -32,7 +32,7 @@ func New(options *Options) (*Exporter, error) {
|
||||||
}
|
}
|
||||||
directory = dir
|
directory = dir
|
||||||
}
|
}
|
||||||
_ = os.MkdirAll(directory, os.ModePerm)
|
_ = os.MkdirAll(directory, 0755)
|
||||||
return &Exporter{options: options, directory: directory}, nil
|
return &Exporter{options: options, directory: directory}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
@ -10,8 +10,8 @@ import (
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/model/types/stringslice"
|
"github.com/projectdiscovery/nuclei/v2/pkg/model/types/stringslice"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/output"
|
"github.com/projectdiscovery/nuclei/v2/pkg/output"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/reporting/dedupe"
|
"github.com/projectdiscovery/nuclei/v2/pkg/reporting/dedupe"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/reporting/exporters/markdown"
|
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/reporting/exporters/es"
|
"github.com/projectdiscovery/nuclei/v2/pkg/reporting/exporters/es"
|
||||||
|
"github.com/projectdiscovery/nuclei/v2/pkg/reporting/exporters/markdown"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/reporting/exporters/sarif"
|
"github.com/projectdiscovery/nuclei/v2/pkg/reporting/exporters/sarif"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/reporting/trackers/github"
|
"github.com/projectdiscovery/nuclei/v2/pkg/reporting/trackers/github"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/reporting/trackers/gitlab"
|
"github.com/projectdiscovery/nuclei/v2/pkg/reporting/trackers/gitlab"
|
||||||
|
@ -24,10 +24,10 @@ type Options struct {
|
||||||
AllowList *Filter `yaml:"allow-list"`
|
AllowList *Filter `yaml:"allow-list"`
|
||||||
// DenyList contains a list of denied events for reporting module
|
// DenyList contains a list of denied events for reporting module
|
||||||
DenyList *Filter `yaml:"deny-list"`
|
DenyList *Filter `yaml:"deny-list"`
|
||||||
// Github contains configuration options for Github Issue Tracker
|
// GitHub contains configuration options for GitHub Issue Tracker
|
||||||
Github *github.Options `yaml:"github"`
|
GitHub *github.Options `yaml:"github"`
|
||||||
// Gitlab contains configuration options for Gitlab Issue Tracker
|
// GitLab contains configuration options for GitLab Issue Tracker
|
||||||
Gitlab *gitlab.Options `yaml:"gitlab"`
|
GitLab *gitlab.Options `yaml:"gitlab"`
|
||||||
// Jira contains configuration options for Jira Issue Tracker
|
// Jira contains configuration options for Jira Issue Tracker
|
||||||
Jira *jira.Options `yaml:"jira"`
|
Jira *jira.Options `yaml:"jira"`
|
||||||
// MarkdownExporter contains configuration options for Markdown Exporter Module
|
// MarkdownExporter contains configuration options for Markdown Exporter Module
|
||||||
|
@ -107,15 +107,15 @@ type Client struct {
|
||||||
// New creates a new nuclei issue tracker reporting client
|
// New creates a new nuclei issue tracker reporting client
|
||||||
func New(options *Options, db string) (*Client, error) {
|
func New(options *Options, db string) (*Client, error) {
|
||||||
client := &Client{options: options}
|
client := &Client{options: options}
|
||||||
if options.Github != nil {
|
if options.GitHub != nil {
|
||||||
tracker, err := github.New(options.Github)
|
tracker, err := github.New(options.GitHub)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, errors.Wrap(err, "could not create reporting client")
|
return nil, errors.Wrap(err, "could not create reporting client")
|
||||||
}
|
}
|
||||||
client.trackers = append(client.trackers, tracker)
|
client.trackers = append(client.trackers, tracker)
|
||||||
}
|
}
|
||||||
if options.Gitlab != nil {
|
if options.GitLab != nil {
|
||||||
tracker, err := gitlab.New(options.Gitlab)
|
tracker, err := gitlab.New(options.GitLab)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, errors.Wrap(err, "could not create reporting client")
|
return nil, errors.Wrap(err, "could not create reporting client")
|
||||||
}
|
}
|
||||||
|
|
|
@ -21,15 +21,15 @@ type Integration struct {
|
||||||
options *Options
|
options *Options
|
||||||
}
|
}
|
||||||
|
|
||||||
// Options contains the configuration options for github issue tracker client
|
// Options contains the configuration options for GitHub issue tracker client
|
||||||
type Options struct {
|
type Options struct {
|
||||||
// BaseURL (optional) is the self-hosted github application url
|
// BaseURL (optional) is the self-hosted GitHub application url
|
||||||
BaseURL string `yaml:"base-url" validate:"omitempty,url"`
|
BaseURL string `yaml:"base-url" validate:"omitempty,url"`
|
||||||
// Username is the username of the github user
|
// Username is the username of the github user
|
||||||
Username string `yaml:"username" validate:"required"`
|
Username string `yaml:"username" validate:"required"`
|
||||||
// Owner (manadatory) is the owner name of the repository for issues.
|
// Owner is the owner name of the repository for issues.
|
||||||
Owner string `yaml:"owner" validate:"required"`
|
Owner string `yaml:"owner" validate:"required"`
|
||||||
// Token is the token for github account.
|
// Token is the token for GitHub account.
|
||||||
Token string `yaml:"token" validate:"required"`
|
Token string `yaml:"token" validate:"required"`
|
||||||
// ProjectName is the name of the repository.
|
// ProjectName is the name of the repository.
|
||||||
ProjectName string `yaml:"project-name" validate:"required"`
|
ProjectName string `yaml:"project-name" validate:"required"`
|
||||||
|
|
|
@ -3,9 +3,10 @@ package gitlab
|
||||||
import (
|
import (
|
||||||
"fmt"
|
"fmt"
|
||||||
|
|
||||||
|
"github.com/xanzy/go-gitlab"
|
||||||
|
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/output"
|
"github.com/projectdiscovery/nuclei/v2/pkg/output"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/reporting/format"
|
"github.com/projectdiscovery/nuclei/v2/pkg/reporting/format"
|
||||||
"github.com/xanzy/go-gitlab"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// Integration is a client for an issue tracker integration
|
// Integration is a client for an issue tracker integration
|
||||||
|
|
|
@ -7,6 +7,7 @@ import (
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"github.com/andygrunwald/go-jira"
|
"github.com/andygrunwald/go-jira"
|
||||||
|
|
||||||
"github.com/projectdiscovery/gologger"
|
"github.com/projectdiscovery/gologger"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/catalog/config"
|
"github.com/projectdiscovery/nuclei/v2/pkg/catalog/config"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/output"
|
"github.com/projectdiscovery/nuclei/v2/pkg/output"
|
||||||
|
@ -103,7 +104,7 @@ func (i *Integration) CreateNewIssue(event *output.ResultEvent) error {
|
||||||
d, _ := ioutil.ReadAll(resp.Body)
|
d, _ := ioutil.ReadAll(resp.Body)
|
||||||
data = string(d)
|
data = string(d)
|
||||||
}
|
}
|
||||||
return fmt.Errorf("%s => %s", err, data)
|
return fmt.Errorf("%w => %s", err, data)
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
@ -140,7 +141,7 @@ func (i *Integration) FindExistingIssue(event *output.ResultEvent) (string, erro
|
||||||
d, _ := ioutil.ReadAll(resp.Body)
|
d, _ := ioutil.ReadAll(resp.Body)
|
||||||
data = string(d)
|
data = string(d)
|
||||||
}
|
}
|
||||||
return "", fmt.Errorf("%s => %s", err, data)
|
return "", fmt.Errorf("%w => %s", err, data)
|
||||||
}
|
}
|
||||||
|
|
||||||
switch resp.Total {
|
switch resp.Total {
|
||||||
|
|
|
@ -3,6 +3,8 @@ package templates
|
||||||
import (
|
import (
|
||||||
"fmt"
|
"fmt"
|
||||||
|
|
||||||
|
"github.com/rs/xid"
|
||||||
|
|
||||||
"github.com/projectdiscovery/gologger"
|
"github.com/projectdiscovery/gologger"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/model"
|
"github.com/projectdiscovery/nuclei/v2/pkg/model"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/operators"
|
"github.com/projectdiscovery/nuclei/v2/pkg/operators"
|
||||||
|
@ -10,7 +12,6 @@ import (
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/protocols"
|
"github.com/projectdiscovery/nuclei/v2/pkg/protocols"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/helpers/writer"
|
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/helpers/writer"
|
||||||
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/http"
|
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/http"
|
||||||
"github.com/rs/xid"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// Cluster clusters a list of templates into a lesser number if possible based
|
// Cluster clusters a list of templates into a lesser number if possible based
|
||||||
|
|
|
@ -12,10 +12,10 @@ type Preprocessor interface {
|
||||||
Process(data []byte) []byte
|
Process(data []byte) []byte
|
||||||
}
|
}
|
||||||
|
|
||||||
var preprocessorRegex = regexp.MustCompile(`\{\{([a-z0-9_]+)\}\}`)
|
var preprocessorRegex = regexp.MustCompile(`{{([a-z0-9_]+)}}`)
|
||||||
|
|
||||||
// expandPreprocessors expands the pre-processors if any for a template data.
|
// expandPreprocessors expands the pre-processors if any for a template data.
|
||||||
func (t *Template) expandPreprocessors(data []byte) []byte {
|
func (template *Template) expandPreprocessors(data []byte) []byte {
|
||||||
foundMap := make(map[string]struct{})
|
foundMap := make(map[string]struct{})
|
||||||
|
|
||||||
for _, expression := range preprocessorRegex.FindAllStringSubmatch(string(data), -1) {
|
for _, expression := range preprocessorRegex.FindAllStringSubmatch(string(data), -1) {
|
||||||
|
|
|
@ -96,23 +96,23 @@ var TemplateProtocols = []string{
|
||||||
}
|
}
|
||||||
|
|
||||||
// Type returns the type of the template
|
// Type returns the type of the template
|
||||||
func (t *Template) Type() types.ProtocolType {
|
func (template *Template) Type() types.ProtocolType {
|
||||||
switch {
|
switch {
|
||||||
case len(t.RequestsDNS) > 0:
|
case len(template.RequestsDNS) > 0:
|
||||||
return types.DNSProtocol
|
return types.DNSProtocol
|
||||||
case len(t.RequestsFile) > 0:
|
case len(template.RequestsFile) > 0:
|
||||||
return types.FileProtocol
|
return types.FileProtocol
|
||||||
case len(t.RequestsHTTP) > 0:
|
case len(template.RequestsHTTP) > 0:
|
||||||
return types.HTTPProtocol
|
return types.HTTPProtocol
|
||||||
case len(t.RequestsHeadless) > 0:
|
case len(template.RequestsHeadless) > 0:
|
||||||
return types.HeadlessProtocol
|
return types.HeadlessProtocol
|
||||||
case len(t.RequestsNetwork) > 0:
|
case len(template.RequestsNetwork) > 0:
|
||||||
return types.NetworkProtocol
|
return types.NetworkProtocol
|
||||||
case len(t.Workflow.Workflows) > 0:
|
case len(template.Workflow.Workflows) > 0:
|
||||||
return types.WorkflowProtocol
|
return types.WorkflowProtocol
|
||||||
case len(t.RequestsSSL) > 0:
|
case len(template.RequestsSSL) > 0:
|
||||||
return types.SSLProtocol
|
return types.SSLProtocol
|
||||||
case len(t.RequestsWebsocket) > 0:
|
case len(template.RequestsWebsocket) > 0:
|
||||||
return types.WebsocketProtocol
|
return types.WebsocketProtocol
|
||||||
default:
|
default:
|
||||||
return types.InvalidProtocol
|
return types.InvalidProtocol
|
||||||
|
|
|
@ -331,12 +331,12 @@ func init() {
|
||||||
HTTPRequestDoc.Fields[6].Name = "name"
|
HTTPRequestDoc.Fields[6].Name = "name"
|
||||||
HTTPRequestDoc.Fields[6].Type = "string"
|
HTTPRequestDoc.Fields[6].Type = "string"
|
||||||
HTTPRequestDoc.Fields[6].Note = ""
|
HTTPRequestDoc.Fields[6].Note = ""
|
||||||
HTTPRequestDoc.Fields[6].Description = "Name is the optional name of the request.\n\nIf a name is specified, all the named request in a template can be matched upon\nin a combined manner allowing multirequest based matchers."
|
HTTPRequestDoc.Fields[6].Description = "Name is the optional name of the request.\n\nIf a name is specified, all the named request in a template can be matched upon\nin a combined manner allowing multi-request based matchers."
|
||||||
HTTPRequestDoc.Fields[6].Comments[encoder.LineComment] = "Name is the optional name of the request."
|
HTTPRequestDoc.Fields[6].Comments[encoder.LineComment] = "Name is the optional name of the request."
|
||||||
HTTPRequestDoc.Fields[7].Name = "attack"
|
HTTPRequestDoc.Fields[7].Name = "attack"
|
||||||
HTTPRequestDoc.Fields[7].Type = "generators.AttackTypeHolder"
|
HTTPRequestDoc.Fields[7].Type = "generators.AttackTypeHolder"
|
||||||
HTTPRequestDoc.Fields[7].Note = ""
|
HTTPRequestDoc.Fields[7].Note = ""
|
||||||
HTTPRequestDoc.Fields[7].Description = "Attack is the type of payload combinations to perform.\n\nbatteringram is same payload into all of the defined payload positions at once, pitchfork combines multiple payload sets and clusterbomb generates\npermutations and combinations for all payloads."
|
HTTPRequestDoc.Fields[7].Description = "Attack is the type of payload combinations to perform.\n\nbatteringram is inserts the same payload into all defined payload positions at once, pitchfork combines multiple payload sets and clusterbomb generates\npermutations and combinations for all payloads."
|
||||||
HTTPRequestDoc.Fields[7].Comments[encoder.LineComment] = "Attack is the type of payload combinations to perform."
|
HTTPRequestDoc.Fields[7].Comments[encoder.LineComment] = "Attack is the type of payload combinations to perform."
|
||||||
HTTPRequestDoc.Fields[7].Values = []string{
|
HTTPRequestDoc.Fields[7].Values = []string{
|
||||||
"batteringram",
|
"batteringram",
|
||||||
|
@ -559,7 +559,7 @@ func init() {
|
||||||
MATCHERSMatcherDoc.Fields[7].Description = "Words contains word patterns required to be present in the response part."
|
MATCHERSMatcherDoc.Fields[7].Description = "Words contains word patterns required to be present in the response part."
|
||||||
MATCHERSMatcherDoc.Fields[7].Comments[encoder.LineComment] = "Words contains word patterns required to be present in the response part."
|
MATCHERSMatcherDoc.Fields[7].Comments[encoder.LineComment] = "Words contains word patterns required to be present in the response part."
|
||||||
|
|
||||||
MATCHERSMatcherDoc.Fields[7].AddExample("Match for outlook mail protection domain", []string{"mail.protection.outlook.com"})
|
MATCHERSMatcherDoc.Fields[7].AddExample("Match for Outlook mail protection domain", []string{"mail.protection.outlook.com"})
|
||||||
|
|
||||||
MATCHERSMatcherDoc.Fields[7].AddExample("Match for application/json in response headers", []string{"application/json"})
|
MATCHERSMatcherDoc.Fields[7].AddExample("Match for application/json in response headers", []string{"application/json"})
|
||||||
MATCHERSMatcherDoc.Fields[8].Name = "regex"
|
MATCHERSMatcherDoc.Fields[8].Name = "regex"
|
||||||
|
@ -938,7 +938,7 @@ func init() {
|
||||||
NETWORKRequestDoc.Fields[2].Name = "attack"
|
NETWORKRequestDoc.Fields[2].Name = "attack"
|
||||||
NETWORKRequestDoc.Fields[2].Type = "generators.AttackTypeHolder"
|
NETWORKRequestDoc.Fields[2].Type = "generators.AttackTypeHolder"
|
||||||
NETWORKRequestDoc.Fields[2].Note = ""
|
NETWORKRequestDoc.Fields[2].Note = ""
|
||||||
NETWORKRequestDoc.Fields[2].Description = "Attack is the type of payload combinations to perform.\n\nBatteringram is same payload into all of the defined payload positions at once, pitchfork combines multiple payload sets and clusterbomb generates\npermutations and combinations for all payloads."
|
NETWORKRequestDoc.Fields[2].Description = "Attack is the type of payload combinations to perform.\n\nBatteringram is inserts the same payload into all defined payload positions at once, pitchfork combines multiple payload sets and clusterbomb generates\npermutations and combinations for all payloads."
|
||||||
NETWORKRequestDoc.Fields[2].Comments[encoder.LineComment] = "Attack is the type of payload combinations to perform."
|
NETWORKRequestDoc.Fields[2].Comments[encoder.LineComment] = "Attack is the type of payload combinations to perform."
|
||||||
NETWORKRequestDoc.Fields[2].Values = []string{
|
NETWORKRequestDoc.Fields[2].Values = []string{
|
||||||
"batteringram",
|
"batteringram",
|
||||||
|
@ -1020,7 +1020,7 @@ func init() {
|
||||||
NETWORKInputDoc.Fields[2].Name = "read"
|
NETWORKInputDoc.Fields[2].Name = "read"
|
||||||
NETWORKInputDoc.Fields[2].Type = "int"
|
NETWORKInputDoc.Fields[2].Type = "int"
|
||||||
NETWORKInputDoc.Fields[2].Note = ""
|
NETWORKInputDoc.Fields[2].Note = ""
|
||||||
NETWORKInputDoc.Fields[2].Description = "Read is the number of bytes to read from socket.\n\nThis can be used for protocols which expect an immediate response. You can\nread and write responses one after another and evetually perform matching\non every data captured with `name` attribute.\n\nThe [network docs](https://nuclei.projectdiscovery.io/templating-guide/protocols/network/) highlight more on how to do this."
|
NETWORKInputDoc.Fields[2].Description = "Read is the number of bytes to read from socket.\n\nThis can be used for protocols which expect an immediate response. You can\nread and write responses one after another and eventually perform matching\non every data captured with `name` attribute.\n\nThe [network docs](https://nuclei.projectdiscovery.io/templating-guide/protocols/network/) highlight more on how to do this."
|
||||||
NETWORKInputDoc.Fields[2].Comments[encoder.LineComment] = "Read is the number of bytes to read from socket."
|
NETWORKInputDoc.Fields[2].Comments[encoder.LineComment] = "Read is the number of bytes to read from socket."
|
||||||
|
|
||||||
NETWORKInputDoc.Fields[2].AddExample("", 1024)
|
NETWORKInputDoc.Fields[2].AddExample("", 1024)
|
||||||
|
|
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue