daffainfo
/
nuclei
mirror of https://github.com/daffainfo/nuclei.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'dev' into dsl_signatures

dev
forgedhallpass 2021-12-07 18:17:34 +02:00
parent d05c4dc122 8bb3b173f2
commit 2d5784d992
144 changed files with 2947 additions and 1246 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.github/ISSUE_TEMPLATE/issue-report.md vendored
View File

@ -33,4 +33,4 @@ Example: steps to reproduce the behavior:
### Anything else:
<!-- Links? References? Screnshots? Anything that will give us more context about the issue that you are encountering! -->
<!-- Links? References? Screenshots? Anything that will give us more context about the issue that you are encountering! -->

29
.github/workflows/template-validate.yml vendored Normal file
View File

@ -0,0 +1,29 @@
name: 🛠 Template Validate
on: [ push, pull_request ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@master
- uses: actions/setup-go@v2
with:
go-version: 1.17
- name: Cache Go
id: cache-go
uses: actions/cache@v2
with:
path: /home/runner/go
key: ${{ runner.os }}-go
- name: Installing Nuclei
if: steps.cache-go.outputs.cache-hit != 'true'
run: |
go install github.com/projectdiscovery/nuclei/v2/cmd/nuclei@latest
- name: Template Validation
run: |
nuclei -validate
nuclei -validate -w ./workflows

43
DESIGN.md
View File

@ -10,9 +10,9 @@ Template is the basic unit of input to the engine which describes the requests t
The template structure is described here. Template level attributes are defined here as well as convenience methods to validate, parse and compile templates creating executers.
Any attributes etc required for the template, engine or requests to function are also set here.
Any attributes etc. required for the template, engine or requests to function are also set here.
Workflows are also compiled, their templates are loaded and compiled as well. Any validations etc on the paths provided are also done here.
Workflows are also compiled, their templates are loaded and compiled as well. Any validations etc. on the paths provided are also done here.
`Parse` function is the main entry point which returns a template for a `filePath` and `executorOptions`. It compiles all the requests for the templates, all the workflows, as well as any self-contained request etc. It also caches the templates in an in-memory cache.
@ -106,7 +106,7 @@ The default executer is provided in `pkg/protocols/common/executer` . It takes a
A different executer is the Clustered Requests executer which implements the Nuclei Request clustering functionality in `pkg/templates` We have a single HTTP request in cases where multiple templates can be clustered and multiple operator lists to match/extract. The first HTTP request is executed while all the template matcher/extractor are evaluated separately.
For Workflow execution, a separate RunWorkflow function is used which executes the workflow independently from the template execution.
For Workflow execution, a separate RunWorkflow function is used which executes the workflow independently of the template execution.
With this basic premise set, we can now start exploring the current runner implementation which will also walk us through the architecture of nuclei.
@ -118,7 +118,7 @@ The first process after all CLI specific initialisation is the loading of templa
#### pkg/catalog
This package is used to get paths using mixed syntax. It takes a template directory and performs resolving for template paths both from provided template directory as well as the current user directory.
This package is used to get paths using mixed syntax. It takes a template directory and performs resolving for template paths both from provided template and current user directory.
The syntax is very versatile and can include filenames, glob patterns, directories, absolute paths, and relative-paths.
@ -177,7 +177,7 @@ ResultEvent structure is passed to the Nuclei Output Writer which contains the e
#### pkg/protocols/common/interactsh
Interactsh module is used to provide automatic Out of Band vulnerability identification in Nuclei.
Interactsh module is used to provide automatic Out-of-Band vulnerability identification in Nuclei.
It uses two LRU caches, one for storing interactions for request URLs and one for storing requests for interaction URL. These both caches are used to correlated requests received to the Interactsh OOB server and Nuclei Instance. [Interactsh Client](https://github.com/projectdiscovery/interactsh/pkg/client) package does most of the heavy lifting of this module.
@ -193,13 +193,13 @@ Next we arrive in the `RunEnumeration` function of the runner.
Next the `WorkflowLoader` is initialised which used to load workflows. It exists in `v2/pkg/parsers/workflow_loader.go`
The loader is initialised moving forward which is responsible for Using Catalog, Passed Tags, Filters, Paths, etc to return compiled `Templates` and `Workflows`.
The loader is initialised moving forward which is responsible for Using Catalog, Passed Tags, Filters, Paths, etc. to return compiled `Templates` and `Workflows`.
#### pkg/catalog/loader
First the input passed by the user as paths is normalised to absolute paths which is done by the `pkg/catalog` module. Next the path filter module is used to removed the excluded template/workflows paths.
First the input passed by the user as paths is normalised to absolute paths which is done by the `pkg/catalog` module. Next the path filter module is used to remove the excluded template/workflows paths.
`pkg/parsers` module's `LoadTemplate`,`LoadWorkflow` functions are used to check if the templates pass the validation + are not excluded via tags/severity/etc filters. If all checks are passed, then the template/workflow is parsed and returned in a compiled form by the `pkg/templates`'s `Parse` function.
`pkg/parsers` module's `LoadTemplate`,`LoadWorkflow` functions are used to check if the templates pass the validation + are not excluded via tags/severity/etc. filters. If all checks are passed, then the template/workflow is parsed and returned in a compiled form by the `pkg/templates`'s `Parse` function.
`Parse` function performs compilation of all the requests in a template + creates Executers from them returning a runnable Template/Workflow structure.
@ -207,10 +207,10 @@ Clustering module comes in next whose job is to cluster identical HTTP GET reque
### pkg/operators
Operators package implements all of the matching and extracting logic of Nuclei.
Operators package implements all the matching and extracting logic of Nuclei.
```go
// Operators contains the operators that can be applied on protocols
// Operators contain the operators that can be applied on protocols
type Operators struct {
Matchers []*matchers.Matcher
Extractors []*extractors.Extractor
@ -218,7 +218,7 @@ type Operators struct {
}
```
A protocol only needs to embed the `operators.Operators` type shown above and it can utilise all the matching/extracting functionality of nuclei.
A protocol only needs to embed the `operators.Operators` type shown above, and it can utilise all the matching/extracting functionality of nuclei.
```go
// MatchFunc performs matching operation for a matcher on model and returns true or false.
@ -246,7 +246,7 @@ type Result struct {
}
```
The internal logics for matching and extracting for things like words, regexes, jq, paths, etc is specified in `pkg/operators/matchers`, `pkg/operators/extractors`. Those packages should be investigated for further look into the topic.
The internal logics for matching and extracting for things like words, regexes, jq, paths, etc. is specified in `pkg/operators/matchers`, `pkg/operators/extractors`. Those packages should be investigated for further look into the topic.
### Template Execution
@ -275,6 +275,8 @@ import (
"path"
"github.com/logrusorgru/aurora"
"go.uber.org/ratelimit"
"github.com/projectdiscovery/goflags"
"github.com/projectdiscovery/nuclei/v2/pkg/catalog"
"github.com/projectdiscovery/nuclei/v2/pkg/catalog/config"
@ -291,7 +293,6 @@ import (
"github.com/projectdiscovery/nuclei/v2/pkg/reporting"
"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
"github.com/projectdiscovery/nuclei/v2/pkg/types"
"go.uber.org/ratelimit"
)
func main() {
@ -356,7 +357,7 @@ func main() {
### Adding a New Protocol
Protocols form the core of Nuclei Engine. All the request types like `http`, `dns`, etc are implemented in form of protocol requests.
Protocols form the core of Nuclei Engine. All the request types like `http`, `dns`, etc. are implemented in form of protocol requests.
A protocol must implement the `Protocol` and `Request` interfaces described above in `pkg/protocols`. We'll take the example of an existing protocol implementation - websocket for this short reference around Nuclei internals.
@ -474,13 +475,13 @@ func (r *Request) Type() templateTypes.ProtocolType {
}
```
Almost all of these protocols have boilerplate functions for which default implementations have been provided in the `providers` package. Examples are the implementation of `Match`, `Extract`, `MakeResultEvent`, GetCompiledOperators`, etc which are almost same throughout Nuclei protocols code. It is enough to copy-paste them unless customization is required.
Almost all of these protocols have boilerplate functions for which default implementations have been provided in the `providers` package. Examples are the implementation of `Match`, `Extract`, `MakeResultEvent`, GetCompiledOperators`, etc. which are almost same throughout Nuclei protocols code. It is enough to copy-paste them unless customization is required.
`eventcreator` package offers `CreateEventWithAdditionalOptions` function which can be used to create result events after doing request execution.
Step by step description of how to add a new protocol to Nuclei -
1. Add the protocol implementation in `pkg/protocols` directory. If it's a small protocol with less number of options, considering adding it to the `pkg/protocols/others` directory. Add the enum for the new protocol to `v2/pkg/templates/types/types.go`.
1. Add the protocol implementation in `pkg/protocols` directory. If it's a small protocol with fewer options, considering adding it to the `pkg/protocols/others` directory. Add the enum for the new protocol to `v2/pkg/templates/types/types.go`.
2. Add the protocol request structure to the `Template` structure fields. This is done in `pkg/templates/templates.go` with the corresponding import line.
@ -526,7 +527,7 @@ func (t *Template) Type() templateTypes.ProtocolType {
```go
// Requests returns the total request count for the template
// Requests return the total request count for the template
func (template *Template) Requests() int {
return len(template.RequestsDNS) +
...
@ -554,11 +555,11 @@ That's it, you've added a new protocol to Nuclei. The next good step would be to
- [v2/pkg/reporting](./v2/pkg/reporting) - Reporting modules for nuclei.
- [v2/pkg/reporting/exporters/sarif](./v2/pkg/reporting/exporters/sarif) - Sarif Result Exporter
- [v2/pkg/reporting/exporters/markdown](./v2/pkg/reporting/exporters/markdown) - Markdown Result Exporter
- [v2/pkg/reporting/exporters/es](./v2/pkg/reporting/exporters/e) - Elasticsearch Result Exporter
- [v2/pkg/reporting/exporters/es](./v2/pkg/reporting/exporters/es) - Elasticsearch Result Exporter
- [v2/pkg/reporting/dedupe](./v2/pkg/reporting/dedupe) - Dedupe module for Results
- [v2/pkg/reporting/trackers/gitlab](./v2/pkg/reporting/trackers/gitlab) - Gitlab Issue Tracker Exporter
- [v2/pkg/reporting/trackers/jira](./v2/pkg/reporting/trackers/jira) - Jira Issue Tracker Exporter
- [v2/pkg/reporting/trackers/github](./v2/pkg/reporting/trackers/github) - Github Issue Tracker Exporter
- [v2/pkg/reporting/trackers/github](./v2/pkg/reporting/trackers/github) - GitHub Issue Tracker Exporter
- [v2/pkg/reporting/format](./v2/pkg/reporting/format) - Result Formatting Functions
- [v2/pkg/parsers](./v2/pkg/parsers) - Implements template as well as workflow loader for initial template discovery, validation and - loading.
- [v2/pkg/types](./v2/pkg/types) - Contains CLI options as well as misc helper functions.
@ -577,12 +578,12 @@ That's it, you've added a new protocol to Nuclei. The next good step would be to
- [v2/pkg/model](./v2/pkg/model) - Template Info + misc
- [v2/pkg/templates](./v2/pkg/templates) - Templates core starting point
- [v2/pkg/templates/cache](./v2/pkg/templates/cache) - Templates cache
- [v2/pkg/protocols](./v2/pkg/protocol) - Protocol Specification
- [v2/pkg/protocols](./v2/pkg/protocols) - Protocol Specification
- [v2/pkg/protocols/file](./v2/pkg/protocols/file) - File protocol
- [v2/pkg/protocols/network](./v2/pkg/protocols/network) - Network protocol
- [v2/pkg/protocols/common/expressions](./v2/pkg/protocols/common/expressions) - Expression evaluation + Templating Support
- [v2/pkg/protocols/common/interactsh](./v2/pkg/protocols/common/interactsh) - Interactsh integration
- [v2/pkg/protocols/common/generators](./v2/pkg/protocols/common/generators) - Payload support for Requests (Sniper, etc)
- [v2/pkg/protocols/common/generators](./v2/pkg/protocols/common/generators) - Payload support for Requests (Sniper, etc.)
- [v2/pkg/protocols/common/executer](./v2/pkg/protocols/common/executer) - Default Template Executer
- [v2/pkg/protocols/common/replacer](./v2/pkg/protocols/common/replacer) - Template replacement helpers
- [v2/pkg/protocols/common/helpers/eventcreator](./v2/pkg/protocols/common/helpers/eventcreator) - Result event creator

4
Dockerfile
View File

@ -1,7 +1,7 @@
FROM golang:1.17.3-alpine as build-env
FROM golang:1.17.4-alpine as build-env
RUN go install -v github.com/projectdiscovery/nuclei/v2/cmd/nuclei@latest
FROM alpine:3.14
FROM alpine:3.15.0
RUN apk add --no-cache bind-tools ca-certificates chromium
COPY --from=build-env /go/bin/nuclei /usr/local/bin/nuclei
ENTRYPOINT ["nuclei"]

50
README.md
View File

@ -26,6 +26,11 @@
<a href="https://discord.gg/projectdiscovery">Join Discord</a>
</p>
<p align="center">
<a href="https://github.com/projectdiscovery/nuclei/blob/master/README.md">English</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/master/README_CN.md">中文</a>
</p>
---
Nuclei is used to send requests across targets based on a template leading to zero false positives and providing fast scanning on large number of hosts. Nuclei offers scanning for a variety of protocols including TCP, DNS, HTTP, File, etc. With powerful and flexible templating, all kinds of security checks can be modelled with Nuclei.
@ -58,7 +63,7 @@ go install -v github.com/projectdiscovery/nuclei/v2/cmd/nuclei@latest
### Nuclei Templates
Nuclei has had built-in support for automatic template download/update as default since version [v2.5.2](https://github.com/projectdiscovery/nuclei/releases/tag/v2.5.2). [**Nuclei-Templates**](https://github.com/projectdiscovery/nuclei-templates) project provides a community-contributed list of ready-to-use templates that is constantly updated.
Nuclei has built-in support for automatic template download/update as default since version [v2.5.2](https://github.com/projectdiscovery/nuclei/releases/tag/v2.5.2). [**Nuclei-Templates**](https://github.com/projectdiscovery/nuclei-templates) project provides a community-contributed list of ready-to-use templates that is constantly updated.
You may still use the `update-templates` flag to update the nuclei templates at any time; You can write your own checks for your individual workflow and needs following Nuclei's [templating guide](https://nuclei.projectdiscovery.io/templating-guide/).
@ -90,20 +95,24 @@ TARGET:
-l, -list string path to file containing a list of target URLs/hosts to scan (one per line)
TEMPLATES:
-t, -templates string[] template or template directory paths to include in the scan
-nt, -new-templates run only new templates added in latest nuclei-templates release
-w, -workflows string[] workflow or workflow directory paths to include in the scan
-validate validate the passed templates to nuclei
-tl list all available templates
-t, -templates string[] template or template directory paths to include in the scan
-tu, -template-url string[] URL containing list of templates to run
-nt, -new-templates run only new templates added in latest nuclei-templates release
-w, -workflows string[] workflow or workflow directory paths to include in the scan
-wu, -workflow-url string[] URL containing list of workflows to run
-validate validate the passed templates to nuclei
-tl list all available templates
FILTERING:
-tags string[] execute a subset of templates that contain the provided tags
-etags, -exclude-tags string[] exclude templates with the provided tags
-itags, -include-tags string[] tags from the default deny list that permit executing more intrusive templates
-et, -exclude-templates string[] template or template directory paths to exclude
-etags, -exclude-tags string[] exclude templates with the provided tags
-it, -include-templates string[] templates to be executed even if they are excluded either by default or configuration
-s, -severity value[] Templates to run based on severity. Possible values - info,low,medium,high,critical
-es, -exclude-severity value[] Templates to exclude based on severity. Possible values - info,low,medium,high,critical
-et, -exclude-templates string[] template or template directory paths to exclude
-s, -severity value[] Templates to run based on severity. Possible values info,low,medium,high,critical
-es, -exclude-severity value[] Templates to exclude based on severity. Possible values info,low,medium,high,critical
-pt, -type value[] protocol types to be executed. Possible values dns,file,http,headless,network,workflow,ssl,websocket
-ept, -exclude-type value[] protocol types to not be executed. Possible values dns,file,http,headless,network,workflow,ssl,websocket
-a, -author string[] execute templates that are (co-)created by the specified authors
OUTPUT:
@ -115,6 +124,7 @@ OUTPUT:
-nm, -no-meta don't display match metadata
-nts, -no-timestamp don't display timestamp metadata in CLI output
-rdb, -report-db string local nuclei reporting database (always use this to persist report data)
-ms, -matcher-status show optional match failure status
-me, -markdown-export string directory to export results in markdown format
-se, -sarif-export string file to export results in SARIF format
@ -126,10 +136,10 @@ CONFIGURATIONS:
-r, -resolvers string file containing resolver list for nuclei
-sr, -system-resolvers use system DNS resolving as error fallback
-passive enable passive HTTP response processing mode
-ev, -env-vars enable environment variables support to be used in template
-cc, -client-cert client certificate file (PEM-encoded) used for authenticating against scanned hosts
-ck, -client-key client key file (PEM-encoded) used for authenticating against scanned hosts
-ca, -client-ca client certificate authority file (PEM-encoded) used for authenticating against scanned hosts
-ev, -env-vars enable environment variables to be used in template
-cc, -client-cert string client certificate file (PEM-encoded) used for authenticating against scanned hosts
-ck, -client-key string client key file (PEM-encoded) used for authenticating against scanned hosts
-ca, -client-ca string client certificate authority file (PEM-encoded) used for authenticating against scanned hosts
INTERACTSH:
-iserver, -interactsh-server string interactsh server url for self-hosted instance (default "https://interactsh.com")
@ -141,10 +151,12 @@ INTERACTSH:
-ni, -no-interactsh disable interactsh server for OAST testing, exclude OAST based templates
RATE-LIMIT:
-rl, -rate-limit int maximum number of requests to send per second (default 150)
-rlm, -rate-limit-minute int maximum number of requests to send per minute
-bs, -bulk-size int maximum number of hosts to be analyzed in parallel per template (default 25)
-c, -concurrency int maximum number of templates to be executed in parallel (default 25)
-rl, -rate-limit int maximum number of requests to send per second (default 150)
-rlm, -rate-limit-minute int maximum number of requests to send per minute
-bs, -bulk-size int maximum number of hosts to be analyzed in parallel per template (default 25)
-c, -concurrency int maximum number of templates to be executed in parallel (default 25)
-hbs, -headless-bulk-size int maximum number of headless hosts to be analyzed in parallel per template (default 10)
-hc, -headless-concurrency int maximum number of headless templates to be executed in parallel (default 10)
OPTIMIZATIONS:
-timeout int time to wait in seconds before timeout (default 5)
@ -250,7 +262,7 @@ Please check our other open-source projects that might fit into your bug bounty
Nuclei immensely improve how you approach security assessment by augmenting the manual, repetitive processes. Consultancies are already converting their manual assessment steps with Nuclei, it allows them to run set of their custom assessment approach across thousands of hosts in an automated manner.
Pen-testers get the full power of our public templates and customization capabilities to speed-up their assessment process, and specifically with the regression cycle where you can easily verify the fix.
Pen-testers get the full power of our public templates and customization capabilities to speed up their assessment process, and specifically with the regression cycle where you can easily verify the fix.
- Easily create your compliance, standards suite (e.g. OWASP Top 10) checklist.
- With capabilities like [fuzz](https://nuclei.projectdiscovery.io/templating-guide/#advance-fuzzing) and [workflows](https://nuclei.projectdiscovery.io/templating-guide/#workflows), complex manual steps and repetitive assessment can be easily automated with Nuclei.

397
README_CN.md
View File

@ -1,252 +1,303 @@
<h1 align="center">
<img src="static/nuclei-logo.png" alt="nuclei" width="200px"></a>
<br>
<a href="https://nuclei.projectdiscovery.io"><img src="static/nuclei-logo.png" width="200px" alt="Nuclei"></a>
</h1>
[![License](https://img.shields.io/badge/license-MIT-_red.svg)](https://opensource.org/licenses/MIT)
[![Go Report Card](https://goreportcard.com/badge/github.com/projectdiscovery/nuclei)](https://goreportcard.com/report/github.com/projectdiscovery/nuclei)
[![contributions welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat)](https://github.com/projectdiscovery/nuclei/issues)
[![GitHub Release](https://img.shields.io/github/release/projectdiscovery/nuclei)](https://github.com/projectdiscovery/nuclei/releases)
[![Follow on Twitter](https://img.shields.io/twitter/follow/pdnuclei.svg?logo=twitter)](https://twitter.com/pdnuclei)
[![Docker Images](https://img.shields.io/docker/pulls/projectdiscovery/nuclei.svg)](https://hub.docker.com/r/projectdiscovery/nuclei)
[![Chat on Discord](https://img.shields.io/discord/695645237418131507.svg?logo=discord)](https://discord.gg/KECAGdH)
<h4 align="center">基于YAML语法模板的定制化快速漏洞扫描器</h4>
<p align="center">
<a href="https://nuclei.projectdiscovery.io/templating-guide/" target="_blank"><img src="static/read-the-docs-button.png" height="42px"/></center></a> <a href="https://github.com/projectdiscovery/nuclei-templates" target="_blank"><img src="static/download-templates-button.png" height="42px"/></a>
<a href="https://goreportcard.com/report/github.com/projectdiscovery/nuclei"><img src="https://goreportcard.com/badge/github.com/projectdiscovery/nuclei"></a>
<a href="https://github.com/projectdiscovery/nuclei/issues"><img src="https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat"></a>
<a href="https://github.com/projectdiscovery/nuclei/releases"><img src="https://img.shields.io/github/release/projectdiscovery/nuclei"></a>
<a href="https://twitter.com/pdnuclei"><img src="https://img.shields.io/twitter/follow/pdnuclei.svg?logo=twitter"></a>
<a href="https://discord.gg/projectdiscovery"><img src="https://img.shields.io/discord/695645237418131507.svg?logo=discord"></a>
<a href="https://github.com/projectdiscovery/nuclei/actions/workflows/build-test.yml"><img src="https://github.com/projectdiscovery/nuclei/actions/workflows/build-test.yml/badge.svg?branch=master"></a>
</p>
<p align="center">
<a href="#工作流程">工作流程</a> •
<a href="#安装Nuclei">安装</a> •
<a href="#对于安全工程师">对于安全工程师</a> •
<a href="#对于开发者和组织">对于开发者</a> •
<a href="https://nuclei.projectdiscovery.io/nuclei/get-started/">文档</a> •
<a href="#c致谢">致谢</a> •
<a href="https://nuclei.projectdiscovery.io/faq/nuclei/">常见问题</a> •
<a href="https://discord.gg/projectdiscovery">加入Discord</a>
</p>
<p align="center">
<a href="https://github.com/projectdiscovery/nuclei/blob/master/README.md">English</a> •
<a href="https://github.com/projectdiscovery/nuclei/blob/master/README_CN.md">中文</a>
</p>
Nuclei是一个基于模板的、可配置攻击目标的扫描快速工具，同时还提供了强大的可扩展性和易用性。
---
基于模板的nuclei被用来发送请求给目标，有着实现零误报的优点，并且可以对已知的路径进行有效的扫描。nuclei的主要用于在初期的探测阶段快速地对已知的且易于检测的漏洞或者CVE进行扫描。如果存在WAF的话，nuclei使用[retryablehttp-go库](https://github.com/projectdiscovery/retryablehttp-go)来处理各种错误，并且重新尝试攻击，这也是我们自定义功能的核心模块之一。
Nuclei使用零误报的定制模板向目标发送请求，同时可以对大量主机进行快速扫描。Nuclei提供TCP、DNS、HTTP、FILE等各类协议的扫描，通过强大且灵活的模板，可以使用Nuclei模拟各种安全检查。
我们也维护一个具有各个类型的模板的[开源库](https://github.com/projectdiscovery/nuclei-templates)，我们希望你也能贡献一些模板，贡献的这些模板最好是有效的，并且能允许每个人基于你的模板重新构建。查看[**nuclei.projectdiscovery.io**](https://nuclei.projectdiscovery.io/templating-guide/)这个网站去学习制作模板的入门知识。
我们的[模板仓库](https://github.com/projectdiscovery/nuclei-templates)包含**超过200**安全研究员和工程师提供的模板。
## 目录
- [目录](#目录)
- [功能](#功能)
- [安装](#安装)
- [Nuclei模板](#nuclei模板)
- [用法](#用法)
- [运行Nuclei](#运行nuclei)
- [排除模板](#排除模板)
- [致谢](#致谢)
## 功能
## 工作流程
<h1 align="left">
<img src="static/nuclei-run.png" alt="nuclei" width="700px"></a>
<br>
</h1>
- 有着易于开发的、简单的、模块化的代码库
- 使用了基于模板的引擎，运行速度极快，可以修改所以配置
- 可以对特殊情况处理、重试、绕过等，可以绕过WAF
- 智能匹配，零误报
<h3 align="center">
<img src="static/nuclei-flow.jpg" alt="nuclei-flow" width="700px"></a>
</h3>
## 安装
### 二进制文件安装
# 安装Nuclei
二进制文件安装很简单，你可以从[Releases](https://github.com/projectdiscovery/nuclei/releases/)页面下载已经构建好的二进制文件压缩包，使用解压工具提取下载的压缩包，并将解压的文件夹移动到$PATH目录，就可以直接使用了。
Nuclei需要**go1.17**才能安装成功。执行下列命令安装最新版本的Nuclei
```sh
Download latest binary from https://github.com/projectdiscovery/nuclei/releases
▶ tar -xzvf nuclei-linux-amd64.tar.gz
▶ mv nuclei /usr/local/bin/
▶ nuclei -version
go install -v github.com/projectdiscovery/nuclei/v2/cmd/nuclei@latest
```
### 源码安装
**更多的安装方式 [请点击此处](https://nuclei.projectdiscovery.io/nuclei/get-started/).**
nuclei需要**go1.14+**才能成功安装，运行以下命令获取repo
<table>
<tr>
<td>
```sh
▶ GO111MODULE=on go get -v github.com/projectdiscovery/nuclei/v2/cmd/nuclei
```
### Nuclei模板
### GitHub安装
自从[v2.5.2]((https://github.com/projectdiscovery/nuclei/releases/tag/v2.5.2))起，Nuclei就内置了自动下载和更新模板的功能。[**Nuclei模板**](https://github.com/projectdiscovery/nuclei-templates)仓库随时更新社区中可用的模板列表。
```sh
▶ git clone https://github.com/projectdiscovery/nuclei.git; cd nuclei/v2/cmd/nuclei/; go build; mv nuclei /usr/local/bin/; nuclei -version
```
您仍然可以随时使用`update-templates`命令更新模板，您可以根据[模板指南](https://nuclei.projectdiscovery.io/templating-guide/)编写您自己的模板。
## Nuclei模板
YAML的语法规范在[这里](SYNTAX-REFERENCE.md)。
你可以使用`update-templates`来下载和更新nuclei模板，该命令会从unclei的[模板库]()中下载最新版本，这个由社区来维护的库是可以随时使用的。
</td>
</tr>
</table>
```sh
▶ nuclei -update-templates
```
此外您可以根据自己的工作情况或者需求编写模板，请参阅**nuclei[模板向导](https://nuclei.projectdiscovery.io/templating-guide/)去编写自定义模板**
## 用法
### 用法
```sh
nuclei -h
```
这条命令会显示帮助，以下是nuclei支持的所有命令
这将显示Nuclei的帮助，以下是所有支持的命令
|命令|描述|例子|
|-----|-----|-----|
|bulk-size|每个模板最大并行的主机数(默认25)|nuclei -bulk-size 25|
|burp-collaborator-biid|使用burp-collaborator插件|nuclei -burp-collaborator-biid XXXX|
|c|并行的最大模板数量(默认10)|nuclei -c 10|
|l|对URL列表进行测试|nuclei -l urls.txt|
|target|对目标进行测试|nuclei -target hxxps://example.com -target hxxps://example2.com|
|t|要检测的模板种类|nuclei -t git-core.yaml -t cves/|
|no-color|输出不显示颜色|nuclei -no-color|
|no-meta|不显示匹配的元数据|nuclei -no-meta|
|json|输出为json格式|nuclei -json|
|include-rr|json输出格式中包含请求和响应数据|nuclei -json -include-rr|
|o|输出为文件|nuclei -o output.txt|
|project|避免发送相同的请求|nuclei -project|
|stats|使用进度条|nuclei -stats|
|silent|只输出测试成功的结果|nuclei -silent|
|retries|失败后的重试次数|nuclei -retries 1|
|timeout|超时时间(默认为5秒)|nuclei -timeout 5|
|trace-log|输出日志到log文件|nuclei -trace-log logs|
|rate-limit|每秒最大请求数(默认150)|nuclei -rate-limit 150|
|severity|根据严重性选择模板|nuclei -severity critical,high|
|stop-at-first-match|第一次匹配不要处理HTTP请求|nuclei -stop-at-frst-match|
|exclude|排除的模板或文件夹|nuclei -exclude panels -exclude tokens|
|debug|调试请求或者响应|nuclei -debug|
|update-templates|下载或者升级模板|nuclei -update-templates|
|update-directory|选择储存模板的目录(可选)|nuclei -update-directory templates|
|tl|列出可用的模板|nuclei -tl|
|templates-version|显示已安装的模板版本|nuclei -templates-version|
|v|显示发送请求的详细信息|nuclei -v|
|version|显示nuclei的版本号|nuclei -version|
|proxy|输入代理地址|nuclei -proxy ./proxy.txt|
|random-agent|使用随机的UA|nuclei -random-agent|
|H|自定义请求头|nuclei -H “x-bug-bounty:hacker”|
## 运行Nuclei
```yaml
Nuclei是一款注重于可配置性、可扩展性和易用性的基于模板的快速漏洞扫描器。
### 运行单个模板
用法：
nuclei [命令]
这将对`urls.txt`中所有的主机运行`git-core.yaml`并返回结果到`results.txt`
命令：
目标：
-u, -target string[] 指定扫描的URL/主机
-l, -list string 指定需要扫描的URL/主机文件（一行一个）
模板：
-t, -templates string[] 指定需要扫描的模板或者模板的路径
-nt, -new-templates 只扫描最新版本中添加的模板
-w, -workflows string[] 指定扫描中的工作流或者工作流目录
-validate 验证通过的模板
-tl 列出所有可用的模板
过滤：
-tags string[] 执行有标记的模板子集
-etags, -exclude-tags string[] 执行标记为排除的模板
-itags, -include-tags string[] 不执行具有攻击性的模板
-et, -exclude-templates string[] 要排除的模板或者模板目录
-it, -include-templates string[] 执行默认或配置中排除的模板
-s, -severity value[] 根据严重性运行模板，允许的值有：info,low,medium,high,critical
-es, -exclude-severity value[] 根据严重性排除模板，允许的值有：info,low,medium,high,critical
-a, -author string[] 执行指定作者的模板
输出：
-o, -output string 输出发现的问题到文件
-silent 只显示结果
-nc, -no-color 禁用输出内容着色（ANSI转义码）
-json 输出为jsonL（ines）
-irr, -include-rr 在JSONL中输出对应的请求和相应（仅结果）
-nm, -no-meta 不显示匹配的元数据
-nts, -no-timestamp 不在输出中显示时间戳
-rdb, -report-db string 本地的Nuclei结果数据库（始终使用该数据库保存结果）
-me, -markdown-export string 以markdown导出结果
-se, -sarif-export string 以SARIF导出结果
配置：
-config string 指定Nuclei的配置文件
-rc, -report-config string 指定Nuclei报告模板文件
-H, -header string[] 指定报告中的标题：value格式
-V, -var value 通过var=value指定var值
-r, -resolvers string 指定Nuclei的解析文件
-sr, -system-resolvers 当DNS错误时使用系统DNS
-passive 启用被动扫描处理HTTP响应
-ev, env-vars 在模板中使用环境变量
交互：
-inserver, -ineractsh-server string 使用interactsh反连检测平台（默认为"https://interactsh.com"）
-itoken, -interactsh-token string 指定反连检测平台的身份凭证
-interactions-cache-size int 指定保存在交互缓存中的请求数（默认：5000）
-interactions-eviction int 聪缓存中删除请求前等待的时间（默认为60秒）
-interactions-poll-duration int 每个轮询前等待时间（默认为5秒）
-interactions-cooldown-period int 退出轮询前的等待时间（默认为5秒）
-ni, -no-interactsh 禁用反连检测平台，同时排除基于反连检测的模板
限速：
-r1, -rate-limit int 每秒最大请求量（默认：150）
-rlm, -rate-limit-minute int 每分钟最大请求量
-bs, -bulk-size int 每个模板最大并行检测数（默认：25）
-c, -concurrency int 并行执行的最大模板数量（默认：25）
优化：
-timeout int 超时时间（默认为5秒）
-retries int 重试次数（默认：1）
-mhe, -max-host-error int 某主机扫描失败次数，跳过该主机（默认：30）
-project 使用项目文件夹避免多次发送同一请求
-project-path string 设置特定的项目文件夹
-spm, -stop-at-first-path 得到一个结果后停止（或许会中断模板和工作流的逻辑）
-stream 流模式 - 在不整理输入的情况下详细描述
无界面浏览器：
-headless 启用需要无界面浏览器的模板
-page-timeout int 在无界面下超时秒数（默认：20）
-sb, -show-brower 在无界面浏览器运行模板时，显示浏览器
-sc, -system-chrome 不使用Nuclei自带的浏览器，使用本地浏览器
调试：
-debug 显示所有请求和响应
-debug-req 显示所有请求
-debug-resp 显示所有响应
-proxy, -proxy-url string 使用HTTP代理
-proxy-socks-url string 使用SOCK5代理
-tlog, -trace-log string 写入请求日志到文件
-version 显示版本信息
-v, -verbose 显示详细信息
-vv 显示额外的详细信息
-tv, -templates-version 显示已安装的模板版本
升级：
-update 更新Nuclei到最新版本
-ut, -update-templates 更新Nuclei模板到最新版
-ud, -update-directory string 覆盖安装模板
-duc, -disable-update-check 禁用更新
统计：
-stats 显示正在扫描的统计信息
-sj, -stats-json 将统计信息以JSONL格式输出到文件
-si, -stats-inerval int 显示统计信息更新的间隔秒数（默认：5）
-m, -metrics 显示Nuclei端口信息
-mp, -metrics-port int 更改Nuclei默认端口（默认：9092）
```
### 运行Nuclei
使用[社区提供的模板](https://github.com/projectdiscovery/nuclei-templates)扫描单个目标
```sh
▶ nuclei -l urls.txt -t files/git-core.yaml -o results.txt
nuclei -u https://example.com
```
你可以轻松的通过管道使用标准的输入(STDIN)传递URL列表。
使用[社区提供的模板](https://github.com/projectdiscovery/nuclei-templates)扫描多个目标
```sh
▶ cat urls.txt | nuclei -t files/git-core.yaml -o results.txt
nuclei -list urls.txt
```
💡 Nuclei可以接受如下列表的URL作为输入，例如以下URL：
Example of `urls.txt`:
```
https://test.some-site.com
http://vuls-testing.com
https://test.com
```
### 运行多个模板
这将会对`urls.txt`中所有的URL运行`cves`和`files`模板检查，并返回输出到`results.txt`
```sh
▶ nuclei -l urls.txt -t cves/ -t files/ -o results.txt
```yaml
http://example.com
http://app.example.com
http://test.example.com
http://uat.example.com
```
### 使用subfinder运行
**更多关于Nuclei的详细实例可以在[这里](https://nuclei.projectdiscovery.io/nuclei/get-started/#running-nuclei)找到**
```sh
▶ subfinder -d hackerone.com -silent | httpx -silent | nuclei -t cves/ -o results.txt
```
# 对于安全工程师
### 在docker中运行
Nuclei提供了大量有助于安全工程师在工作流定制相关的功能。通过各种扫描功能（如DNS、HTTP、TCP），安全工程师可以更轻松的使用Nuclei创建一套自定义的检查方式。
你需要使用[nuclei的docker镜像](https://hub.docker.com/r/projectdiscovery/nuclei)来运行
- 支持多种协议：TCP、DNS、HTTP、FILE等
- 通过工作流和动态请求实现复杂的漏洞扫描
- 易于集成到CI/CD，旨在可以轻松的集成到周期扫描中，以主动检测漏洞的修复和重新出现
```sh
▶ docker pull projectdiscovery/nuclei
```
<h1 align="left">
<a href="https://nuclei.projectdiscovery.io/nuclei/get-started/"><img src="static/learn-more-button.png" width="170px" alt="Learn More"></a>
</h1>
下载并构建完成后，运行以下命令：
<table>
<tr>
<td>
```sh
▶ docker run -it projectdiscovery/nuclei
```
**对于赏金猎人：**
这将会对`urls.txt`中的URL通过docker中的nuclei进行检测，并将结果输出到本机的`results.txt`文件的：
Nuclei允许您定制自己的测试方法，可以轻松的运行您的程序。此外Nuclei可以更容易的集成到您的漏扫设备中。
```sh
▶ cat urls.txt | docker run -v /path/to/nuclei-templates:/app/nuclei-templates -v /path/to/nuclei/config:/app/.nuclei-config.json -i projectdiscovery/nuclei -t /app/nuclei-templates/files/git-config.yaml > results.txt
```
记住更改的模板路径到本机
- 可以集成到其他工作流中
- 可以在几分钟处理上千台主机
- 使用YAML语法定制自动化测试
### 速率限制
欢迎查看我们其他的开源项目，可能有适合您的赏金猎人工作流：[github.com/projectdiscovery](http://github.com/projectdiscovery)，我们还使用[Chaos绘制了每日的DNS数据](http://chaos.projectdiscovery.io)。
Nuclei有多种控制速率的方法，包括并行执行多个模板、并行检查多个主机，以及使nuclei限制全局的请求速率，下面就是示例。
</td>
</tr>
</table>
- `-c`参数 => 限制并行的模板数
- `-bulk-size`参数 => 限制并行的主机数
- `-rate-limit`参数 => 全局速率限制
<table>
<tr>
<td>
如果你想快速扫描或者控制扫描，请使用这些标志并输入限制数，`速率限制`只保证控制传出的请求，与其他参数无关。
**对于渗透测试：**
### 排除模板
Nuclei通过增加手动、自动的过程，极大地改变了安全评估的方式。一些公司已经在用Nuclei升级他们的手动测试步骤，可以使用Nulcei对数千台主机使用同样的流程自动化测试。
[Nuclei模板](https://github.com/projectdiscovery/nuclei-templates)包含多种检查，其中有许多对攻击有用的检查，但并不是都有用的。如果您只希望扫描少数特定的模板或目录，则可以使用如下的参数筛选模板，或将某些模板排除。
渗透测试员可以使用公共模板或者自定义模板来更快的完成渗透测试，特别是漏洞验证时，可以轻松的验证漏洞是否修复。
#### 排除模板运行
- 轻松根据您的要求创建标准清单（例如：OWASP TOP 10）
- 通过[FUZZ](https://nuclei.projectdiscovery.io/templating-guide/#advance-fuzzing)和[工作流](https://nuclei.projectdiscovery.io/templating-guide/#workflows)等功能，可以使用Nuclei完成复杂的手动步骤和重复性渗透测试
- 只需要重新运行Nuclei即可验证漏洞修复情况
我们不建议同时运行所有的nuclei模板，如果要排除模板，可以使用`exclude`参数来排除特定的目录或模板。
</td>
</tr>
</table>
```sh
nuclei -l urls.txt -t nuclei-templates -exclude panels/ -exclude technologies -exclude files/wp-xmlrpc.yaml
```
# 对于开发和组织
注意：如上述示例中显示的那样，目录和特定模板都将不会扫描
Nuclei构建很简单，通过数百名安全研究员的社区模板，Nuclei可以随时扫描来了解安全威胁。Nuclei通常用来用于复测，以确定漏洞是否被修复。
#### 基于严重性运行模板
- **CI/CD：**工程师已经支持了CI/CD，可以使用Nuclei来监控生产环境
- **周期性扫描：**使用Nuclei创建新发现的漏洞模板，通过Nuclei可以周期性扫描消除漏洞
您可以根据模板的严重性运行模板，扫描时可以选择单个严重性或多个严重性。
我们有个[讨论组](https://github.com/projectdiscovery/nuclei-templates/discussions/693)，黑客提交自己的模板后可以获得赏金，这可以减少资产的漏洞，并且减少重复。如果你想实行该计划，可以[联系我](mailto:contact@projectdiscovery.io)。我们非常乐意提供帮助，或者在[讨论组](https://github.com/projectdiscovery/nuclei-templates/discussions/693)中发布相关信息。
```sh
nuclei -l urls.txt -t cves/ -severity critical,medium
```
<h3 align="center">
<img src="static/regression-with-nuclei.jpg" alt="regression-cycle-with-nuclei" width="1100px"></a>
</h3>
上面的例子将运行`cves`目录下所有`严重`和`中等`的模板。
<h1 align="left">
<a href="https://github.com/projectdiscovery/nuclei-action"><img src="static/learn-more-button.png" width="170px" alt="Learn More"></a>
</h1>
```sh
nuclei -l urls.txt -t panels/ -t technologies -severity info
```
### 资源
- [使用Nuclei扫描](https://blog.projectdiscovery.io/community-powered-scanning-with-nuclei/)
- [Nuclei Unleashed - 快速编写复杂漏洞](https://blog.projectdiscovery.io/nuclei-unleashed-quickly-write-complex-exploits/)
- [Nuclei - FUZZ一切](https://blog.projectdiscovery.io/nuclei-fuzz-all-the-things/)
- [Nuclei + Interactsh Integration，用于自动化OOB测试](https://blog.projectdiscovery.io/nuclei-interactsh-integration/)
- [武器化Nuclei](https://medium.com/@dwisiswant0/weaponizes-nuclei-workflows-to-pwn-all-the-things-cd01223feb77) 作者：[@dwisiswant0](https://github.com/dwisiswant0)
- [如何使用Nuclei连续扫描？](https://medium.com/@dwisiswant0/how-to-scan-continuously-with-nuclei-fcb7e9d8b8b9) 作者：[@dwisiswant0](https://github.com/dwisiswant0)
- [自动化攻击](https://dhiyaneshgeek.github.io/web/security/2021/07/19/hack-with-automation/) 作者：[@DhiyaneshGeek](https://github.com/DhiyaneshGeek)
上面的例子将运行`panels`和`technologies`目录下严重性标记为`info`的模板
### 致谢
#### 使用`.nuclei-ignore`文件排除模板
感谢所有[社区贡献者提供的PR](https://github.com/projectdiscovery/nuclei/graphs/contributors)，另外您可以其他类似的开源项目：
自从nuclei的[v2.1.1版本](https://github.com/projectdiscovery/nuclei/releases/tag/v2.1.1)以来，我们添加了对`.nuclei-ignore`文件的支持，该文件与`update-templates`参数一起使用，在 **.nuclei-ignore** 文件中，您可以定义要从nuclei扫描中排除的所有模板目录或者模板路径，要开始使用此功能，请确保使用`nuclei-update-templates`参数安装nuclei模板，现在可以根据`.nuclei-ignore`的文件来添加、更新、删除模板文件。
[FFuF](https://github.com/ffuf/ffuf), [Qsfuzz](https://github.com/ameenmaali/qsfuzz), [Inception](https://github.com/proabiral/inception), [Snallygaster](https://github.com/hannob/snallygaster), [Gofingerprint](https://github.com/Static-Flow/gofingerprint), [Sn1per](https://github.com/1N3/Sn1per/tree/master/templates), [Google tsunami](https://github.com/google/tsunami-security-scanner), [Jaeles](https://github.com/jaeles-project/jaeles), [ChopChop](https://github.com/michelin/ChopChop)
```
nano ~/nuclei-templates/.nuclei-ignore
```
### 许可证
默认的**nuclei忽略**列表可以访问[这里]((https://github.com/projectdiscovery/nuclei-templates/blob/master/.nuclei-ignore)，如果不想排除任何内容，只需要删除`.nuclei-ignore`文件。
Nuclei使用[MIT许可证](https://github.com/projectdiscovery/nuclei/blob/master/LICENSE.md)
* * *
### 📋 笔记
- 进度条是实验性功能，在某些情况下可能无法使用。
- 进度条不适用于工作流，因为是条件执行，所以不准确。
## 致谢
也要看看这些类似的好项目，或许它们也适合你：
[Burp Suite](https://portswigger.net/burp), [FFuF](https://github.com/ffuf/ffuf), [Jaeles](https://github.com/jaeles-project/jaeles), [Qsfuzz](https://github.com/ameenmaali/qsfuzz), [Inception](https://github.com/proabiral/inception), [Snallygaster](https://github.com/hannob/snallygaster), [Gofingerprint](https://github.com/Static-Flow/gofingerprint), [Sn1per](https://github.com/1N3/Sn1per/tree/master/templates), [Google tsunami](https://github.com/google/tsunami-security-scanner), [ChopChop](https://github.com/michelin/ChopChop)
--------
Nuclei是由[projectdiscovery](https://projectdiscovery.io)团队用🖤制作的，当然社区也贡献了很多，通过 **[Thanks.md](https://github.com/projectdiscovery/nuclei/blob/master/THANKS.md)**文件以获取更多详细信息。
<h1 align="left">
<a href="https://discord.gg/projectdiscovery"><img src="static/Join-Discord.png" width="380" alt="Join Discord"></a> <a href="https://nuclei.projectdiscovery.io"><img src="static/check-nuclei-documentation.png" width="380" alt="Check Nuclei Documentation"></a>
</h1>

599
SYNTAX-REFERENCE.md
View File

@ -9,6 +9,8 @@ Template is a YAML input file which defines all the requests and
<hr />
<div class="dd">
@ -134,7 +136,7 @@ dns:
type: CNAME
class: inet
retries: 2
recursion: true
recursion: false
```
@ -269,6 +271,19 @@ Self Contained marks Requests for the template as self-contained
<hr />
<div class="dd">
<code>stop-at-first-match</code> <i>bool</i>
</div>
<div class="dt">
Stop execution once first match is found
</div>
<hr />
@ -290,6 +305,8 @@ reference: https://zxsecurity.co.nz/research/argunment-injection-ruby-dragonfly/
severity: high
```
<hr />
<div class="dd">
@ -435,19 +452,6 @@ reference:
Severity of the template.
Valid values:
- <code>info</code>
- <code>low</code>
- <code>medium</code>
- <code>high</code>
- <code>critical</code>
</div>
<hr />
@ -558,6 +562,8 @@ CWE-22
## severity.Holder
Holder holds a Severity type. Required for un/marshalling purposes
@ -570,6 +576,40 @@ Appears in:
<hr />
<div class="dd">
<code></code> <i>Severity</i>
</div>
<div class="dt">
Enum Values:
- <code>undefined</code>
- <code>info</code>
- <code>low</code>
- <code>medium</code>
- <code>high</code>
- <code>critical</code>
</div>
<hr />
## model.Classification
Appears in:
@ -579,6 +619,8 @@ Appears in:
<hr />
<div class="dd">
@ -705,6 +747,26 @@ path:
method: GET
```
Part Definitions:
- <code>template-id</code> - ID of the template executed
- <code>template-info</code> - Info Block of the template executed
- <code>template-path</code> - Path of the template executed
- <code>host</code> - Host is the input to the template
- <code>matched</code> - Matched is the input which was matched upon
- <code>type</code> - Type is the type of request made
- <code>request</code> - HTTP request made from the client
- <code>response</code> - HTTP response recieved from server
- <code>status_code</code> - Status Code received from the Server
- <code>body</code> - HTTP response body received from server (default)
- <code>content_length</code> - HTTP Response content length
- <code>header,all_headers</code> - HTTP response headers
- <code>duration</code> - HTTP request time duration
- <code>all</code> - HTTP response body + headers
- <code>cookies_from_response</code> - HTTP response cookies in name:value format
- <code>headers_from_response</code> - HTTP response headers in name:value format
<hr />
<div class="dd">
@ -847,7 +909,7 @@ ID is the optional id of the request
Name is the optional name of the request.
If a name is specified, all the named request in a template can be matched upon
in a combined manner allowing multirequest based matchers.
in a combined manner allowing multi-request based matchers.
</div>
@ -862,7 +924,7 @@ in a combined manner allowing multirequest based matchers.
Attack is the type of payload combinations to perform.
batteringram is same payload into all of the defined payload positions at once, pitchfork combines multiple payload sets and clusterbomb generates
batteringram is inserts the same payload into all defined payload positions at once, pitchfork combines multiple payload sets and clusterbomb generates
permutations and combinations for all payloads.
@ -880,36 +942,13 @@ Valid values:
<div class="dd">
<code>method</code> <i>HTTPMethodTypeHolder</i>
<code>method</code> <i><a href="#httpmethodtypeholder">HTTPMethodTypeHolder</a></i>
</div>
<div class="dt">
Method is the HTTP Request Method.
Valid values:
- <code>GET</code>
- <code>HEAD</code>
- <code>POST</code>
- <code>PUT</code>
- <code>DELETE</code>
- <code>CONNECT</code>
- <code>OPTIONS</code>
- <code>TRACE</code>
- <code>PATCH</code>
- <code>PURGE</code>
</div>
<hr />
@ -1244,6 +1283,19 @@ SkipVariablesCheck skips the check for unresolved variables in request
<hr />
<div class="dd">
<code>iterate-all</code> <i>bool</i>
</div>
<div class="dt">
IterateAll iterates all the values extracted from internal extractors
</div>
<hr />
@ -1270,32 +1322,19 @@ Appears in:
<hr />
<div class="dd">
<code>type</code> <i>MatcherTypeHolder</i>
<code>type</code> <i><a href="#matchertypeholder">MatcherTypeHolder</a></i>
</div>
<div class="dt">
Type is the type of the matcher.
Valid values:
- <code>status</code>
- <code>size</code>
- <code>word</code>
- <code>regex</code>
- <code>binary</code>
- <code>dsl</code>
</div>
<hr />
@ -1454,7 +1493,7 @@ Examples:
```yaml
# Match for outlook mail protection domain
# Match for Outlook mail protection domain
words:
- mail.protection.outlook.com
```
@ -1608,6 +1647,52 @@ Valid values:
## MatcherTypeHolder
MatcherTypeHolder is used to hold internal type of the matcher
Appears in:
- <code><a href="#matchersmatcher">matchers.Matcher</a>.type</code>
<hr />
<div class="dd">
<code></code> <i>MatcherType</i>
</div>
<div class="dt">
Enum Values:
- <code>word</code>
- <code>regex</code>
- <code>binary</code>
- <code>status</code>
- <code>size</code>
- <code>dsl</code>
</div>
<hr />
## extractors.Extractor
Extractor is used to extract part of response using a regex.
@ -1630,6 +1715,8 @@ Appears in:
<hr />
<div class="dd">
@ -1658,24 +1745,13 @@ name: cookie-extractor
<div class="dd">
<code>type</code> <i>TypeHolder</i>
<code>type</code> <i><a href="#extractortypeholder">ExtractorTypeHolder</a></i>
</div>
<div class="dt">
Type is the type of the extractor.
Valid values:
- <code>regex</code>
- <code>kval</code>
- <code>json</code>
- <code>xpath</code>
</div>
<hr />
@ -1911,6 +1987,48 @@ Valid values:
## ExtractorTypeHolder
ExtractorTypeHolder is used to hold internal type of the extractor
Appears in:
- <code><a href="#extractorsextractor">extractors.Extractor</a>.type</code>
<hr />
<div class="dd">
<code></code> <i>ExtractorType</i>
</div>
<div class="dt">
Enum Values:
- <code>regex</code>
- <code>kval</code>
- <code>xpath</code>
- <code>json</code>
</div>
<hr />
## generators.AttackTypeHolder
AttackTypeHolder is used to hold internal type of the protocol
@ -1927,6 +2045,88 @@ Appears in:
<hr />
<div class="dd">
<code></code> <i>AttackType</i>
</div>
<div class="dt">
Enum Values:
- <code>batteringram</code>
- <code>pitchfork</code>
- <code>clusterbomb</code>
</div>
<hr />
## HTTPMethodTypeHolder
HTTPMethodTypeHolder is used to hold internal type of the HTTP Method
Appears in:
- <code><a href="#httprequest">http.Request</a>.method</code>
<hr />
<div class="dd">
<code></code> <i>HTTPMethodType</i>
</div>
<div class="dt">
Enum Values:
- <code>GET</code>
- <code>GET</code>
- <code>POST</code>
- <code>PUT</code>
- <code>DELETE</code>
- <code>CONNECT</code>
- <code>OPTIONS</code>
- <code>TRACE</code>
- <code>PATCH</code>
- <code>PURGE</code>
</div>
<hr />
## dns.Request
Request contains a DNS protocol request to be made from a template
@ -1946,9 +2146,27 @@ name: '{{FQDN}}'
type: CNAME
class: inet
retries: 2
recursion: true
recursion: false
```
Part Definitions:
- <code>template-id</code> - ID of the template executed
- <code>template-info</code> - Info Block of the template executed
- <code>template-path</code> - Path of the template executed
- <code>host</code> - Host is the input to the template
- <code>matched</code> - Matched is the input which was matched upon
- <code>request</code> - Request contains the DNS request in text format
- <code>type</code> - Type is the type of request made
- <code>rcode</code> - Rcode field returned for the DNS request
- <code>question</code> - Question contains the DNS question field
- <code>extra</code> - Extra contains the DNS response extra field
- <code>answer</code> - Answer contains the DNS response answer field
- <code>ns</code> - NS contains the DNS response NS field
- <code>raw,body,all</code> - Raw contains the raw DNS response (default)
- <code>trace</code> - Trace contains trace data for DNS request if enabled
<hr />
<div class="dd">
@ -2043,34 +2261,13 @@ name: '{{FQDN}}'
<div class="dd">
<code>type</code> <i>DNSRequestTypeHolder</i>
<code>type</code> <i><a href="#dnsrequesttypeholder">DNSRequestTypeHolder</a></i>
</div>
<div class="dt">
RequestType is the type of DNS request to make.
Valid values:
- <code>A</code>
- <code>NS</code>
- <code>DS</code>
- <code>CNAME</code>
- <code>SOA</code>
- <code>PTR</code>
- <code>MX</code>
- <code>TXT</code>
- <code>AAAA</code>
</div>
<hr />
@ -2168,7 +2365,7 @@ trace-max-recursion: 100
<div class="dd">
<code>recursion</code> <i>bool</i>
<code>recursion</code> <i>dns.bool</i>
</div>
<div class="dt">
@ -2196,6 +2393,58 @@ Resolvers to use for the dns requests
## DNSRequestTypeHolder
DNSRequestTypeHolder is used to hold internal type of the DNS type
Appears in:
- <code><a href="#dnsrequest">dns.Request</a>.type</code>
<hr />
<div class="dd">
<code></code> <i>DNSRequestType</i>
</div>
<div class="dt">
Enum Values:
- <code>A</code>
- <code>NS</code>
- <code>DS</code>
- <code>CNAME</code>
- <code>SOA</code>
- <code>PTR</code>
- <code>MX</code>
- <code>TXT</code>
- <code>AAAA</code>
</div>
<hr />
## file.Request
Request contains a File matching mechanism for local disk operations.
@ -2214,6 +2463,17 @@ extensions:
- all
```
Part Definitions:
- <code>template-id</code> - ID of the template executed
- <code>template-info</code> - Info Block of the template executed
- <code>template-path</code> - Path of the template executed
- <code>matched</code> - Matched is the input which was matched upon
- <code>path</code> - Path is the path of file on local filesystem
- <code>type</code> - Type is the type of request made
- <code>raw,body,all,data</code> - Raw contains the raw file contents
<hr />
<div class="dd">
@ -2401,6 +2661,19 @@ matchers:
- zookeeper.version
```
Part Definitions:
- <code>template-id</code> - ID of the template executed
- <code>template-info</code> - Info Block of the template executed
- <code>template-path</code> - Path of the template executed
- <code>host</code> - Host is the input to the template
- <code>matched</code> - Matched is the input which was matched upon
- <code>type</code> - Type is the type of request made
- <code>request</code> - Network request made from the client
- <code>body,all,data</code> - Network response recieved from server (default)
- <code>raw</code> - Full Network protocol data
<hr />
<div class="dd">
@ -2452,18 +2725,9 @@ host:
Attack is the type of payload combinations to perform.
Batteringram is same payload into all of the defined payload positions at once, pitchfork combines multiple payload sets and clusterbomb generates
Batteringram is inserts the same payload into all defined payload positions at once, pitchfork combines multiple payload sets and clusterbomb generates
permutations and combinations for all payloads.
Valid values:
- <code>batteringram</code>
- <code>pitchfork</code>
- <code>clusterbomb</code>
</div>
<hr />
@ -2613,6 +2877,8 @@ Appears in:
<hr />
<div class="dd">
@ -2646,7 +2912,7 @@ data: hex_decode('50494e47')
<div class="dd">
<code>type</code> <i>NetworkInputTypeHolder</i>
<code>type</code> <i><a href="#networkinputtypeholder">NetworkInputTypeHolder</a></i>
</div>
<div class="dt">
@ -2676,7 +2942,7 @@ Valid values:
Read is the number of bytes to read from socket.
This can be used for protocols which expect an immediate response. You can
read and write responses one after another and evetually perform matching
read and write responses one after another and eventually perform matching
on every data captured with `name` attribute.
The [network docs](https://nuclei.projectdiscovery.io/templating-guide/protocols/network/) highlight more on how to do this.
@ -2722,6 +2988,44 @@ name: prefix
## NetworkInputTypeHolder
NetworkInputTypeHolder is used to hold internal type of the Network type
Appears in:
- <code><a href="#networkinput">network.Input</a>.type</code>
<hr />
<div class="dd">
<code></code> <i>NetworkInputType</i>
</div>
<div class="dt">
Enum Values:
- <code>hex</code>
- <code>text</code>
</div>
<hr />
## headless.Request
Request contains a Headless protocol request to be made from a template
@ -2732,6 +3036,18 @@ Appears in:
Part Definitions:
- <code>template-id</code> - ID of the template executed
- <code>template-info</code> - Info Block of the template executed
- <code>template-path</code> - Path of the template executed
- <code>host</code> - Host is the input to the template
- <code>matched</code> - Matched is the input which was matched upon
- <code>type</code> - Type is the type of request made
- <code>req</code> - Headless request made from the client
- <code>resp,body,data</code> - Headless response recieved from client (default)
<hr />
<div class="dd">
@ -2831,6 +3147,8 @@ Appears in:
<hr />
<div class="dd">
@ -2879,15 +3197,46 @@ Description is the optional description of the headless action
<div class="dd">
<code>action</code> <i>ActionTypeHolder</i>
<code>action</code> <i><a href="#actiontypeholder">ActionTypeHolder</a></i>
</div>
<div class="dt">
Action is the type of the action to perform.
</div>
Valid values:
<hr />
## ActionTypeHolder
ActionTypeHolder is used to hold internal type of the action
Appears in:
- <code><a href="#engineaction">engine.Action</a>.action</code>
<hr />
<div class="dd">
<code></code> <i>ActionType</i>
</div>
<div class="dt">
Enum Values:
- <code>navigate</code>
@ -2931,6 +3280,8 @@ Valid values:
- <code>debug</code>
- <code>sleep</code>
- <code>waitvisible</code>
</div>
<hr />
@ -2949,6 +3300,15 @@ Appears in:
Part Definitions:
- <code>type</code> - Type is the type of request made
- <code>response</code> - JSON SSL protocol handshake details
- <code>not_after</code> - Timestamp after which the remote cert expires
- <code>host</code> - Host is the input to the template
- <code>matched</code> - Matched is the input which was matched upon
<hr />
<div class="dd">
@ -3030,6 +3390,16 @@ Appears in:
Part Definitions:
- <code>type</code> - Type is the type of request made
- <code>success</code> - Success specifies whether websocket connection was successful
- <code>request</code> - Websocket request made to the server
- <code>response</code> - Websocket response recieved from the server
- <code>host</code> - Host is the input to the template
- <code>matched</code> - Matched is the input which was matched upon
<hr />
<div class="dd">
@ -3135,15 +3505,6 @@ Attack is the type of payload combinations to perform.
Sniper is each payload once, pitchfork combines multiple payload sets and clusterbomb generates
permutations and combinations for all payloads.
Valid values:
- <code>sniper</code>
- <code>pitchfork</code>
- <code>clusterbomb</code>
</div>
<hr />
@ -3178,6 +3539,8 @@ Appears in:
<hr />
<div class="dd">
@ -3249,6 +3612,8 @@ Appears in:
<hr />
<div class="dd">
@ -3332,6 +3697,8 @@ Appears in:
<hr />
<div class="dd">

18
integration_tests/headless/headless-basic.yaml Normal file
View File

@ -0,0 +1,18 @@
id: headless-basic
info:
name: Headless Basic
author: pdteam
severity: info
tags: headless
headless:
- steps:
- action: navigate
args:
url: "{{BaseURL}}/"
- action: waitload
matchers:
- type: word
words:
- "<html>"

31
integration_tests/headless/headless-extract-values.yaml Normal file
View File

@ -0,0 +1,31 @@
id: headless-extract-values
info:
name: Headless Extract Value
author: pdteam
severity: info
tags: headless
headless:
- steps:
- action: navigate
args:
url: "{{BaseURL}}"
- action: waitload
# From headless/extract-urls.yaml
- action: script
name: extract
args:
code: |
'\n' + [...new Set(Array.from(document.querySelectorAll('[src], [href], [url], [action]')).map(i => i.src || i.href || i.url || i.action))].join('\r\n') + '\n'
matchers:
- type: word
words:
- "test.html"
extractors:
- type: kval
part: extract
kval:
- extract

24
integration_tests/headless/headless-header-action.yaml Normal file
View File

@ -0,0 +1,24 @@
id: headless-header-action
info:
name: Headless Header Action
author: pdteam
severity: info
tags: headless
headless:
- steps:
- action: setheader
args:
part: request
key: Test
value: test value
- action: navigate
args:
url: "{{BaseURL}}/"
- action: waitload
matchers:
- type: word
words:
- "test value"

23
integration_tests/http/dsl-matcher-variable.yaml Normal file
View File

@ -0,0 +1,23 @@
id: dsl-matcher-variable
info:
name: dsl-matcher-variable
author: pd-team
severity: info
requests:
-
path:
- "{{BaseURL}}"
payloads:
VALUES:
- This
- is
- test
- matcher
- text
matchers:
-
dsl:
- 'contains(body,"{{VALUES}}")'
type: dsl

30
integration_tests/test-issue-tracker-config1.yaml
View File

@ -3,35 +3,35 @@ allow-list:
deny-list:
severity: low
# github contains configuration options for github issue tracker
# GitHub contains configuration options for GitHub issue tracker
github:
# base-url is the optional self-hosted github application url
# base-url is the optional self-hosted GitHub application url
base-url: https://localhost:8443/github
# username is the username of the github user
# username is the username of the GitHub user
username: test-username
# owner is the owner name of the repository for issues.
# owner is the owner name of the repository for issues
owner: test-owner
# token is the token for github account.
# token is the token for GitHub account
token: test-token
# project-name is the name of the repository.
# project-name is the name of the repository
project-name: test-project
# issue-label is the label of the created issue type
issue-label: bug
# gitlab contains configuration options for gitlab issue tracker
# GitLab contains configuration options for gitlab issue tracker
gitlab:
# base-url is the optional self-hosted gitlab application url
# base-url is the optional self-hosted GitLab application url
base-url: https://localhost:8443/gitlab
# username is the username of the gitlab user
# username is the username of the GitLab user
username: test-username
# token is the token for gitlab account.
# token is the token for GitLab account
token: test-token
# project-name is the name/id of the project(repository).
# project-name is the name/id of the project(repository)
project-name: "1234"
# issue-label is the label of the created issue type
issue-label: bug
# jira contains configuration options for jira issue tracker
# Jira contains configuration options for Jira issue tracker
jira:
# cloud is the boolean which tells if Jira instance is running in the cloud or on-prem version is used
cloud: true
@ -39,11 +39,11 @@ jira:
update-existing: false
# URL is the jira application url
url: https://localhost/jira
# account-id is the account-id of the jira user or username in case of on-prem Jira
# account-id is the account-id of the Jira user or username in case of on-prem Jira
account-id: test-account-id
# email is the email of the user for jira instance
# email is the email of the user for Jira instance
email: test@test.com
# token is the token for jira instance or password in case of on-prem Jira
# token is the token for Jira instance or password in case of on-prem Jira
token: test-token
# project-name is the name of the project.
project-name: test-project-name

38
integration_tests/test-issue-tracker-config2.yaml
View File

@ -5,47 +5,47 @@ allow-list:
deny-list:
severity: low
# github contains configuration options for github issue tracker
github:
# base-url is the optional self-hosted github application url
base-url: https://localhost:8443/github
# username is the username of the github user
# GitHub contains configuration options for GitHub issue tracker
GitHub:
# base-url is the optional self-hosted GitHub application url
base-url: https://localhost:8443/GitHub
# username is the username of the GitHub user
username: test-username
# owner is the owner name of the repository for issues.
owner: test-owner
# token is the token for github account.
# token is the token for GitHub account.
token: test-token
# project-name is the name of the repository.
project-name: test-project
# issue-label is the label of the created issue type
issue-label: bug
# gitlab contains configuration options for gitlab issue tracker
gitlab:
# base-url is the optional self-hosted gitlab application url
base-url: https://localhost:8443/gitlab
# username is the username of the gitlab user
# GitLab contains configuration options for GitLab issue tracker
GitLab:
# base-url is the optional self-hosted GitLab application url
base-url: https://localhost:8443/GitLab
# username is the username of the GitLab user
username: test-username
# token is the token for gitlab account.
# token is the token for GitLab account.
token: test-token
# project-name is the name/id of the project(repository).
project-name: "1234"
# issue-label is the label of the created issue type
issue-label: bug
# jira contains configuration options for jira issue tracker
jira:
# Jira contains configuration options for Jira issue tracker
Jira:
# cloud is the boolean which tells if Jira instance is running in the cloud or on-prem version is used
cloud: true
# update-existing is the boolean which tells if the existing, opened issue should be updated or new one should be created
update-existing: false
# URL is the jira application url
url: https://localhost/jira
# account-id is the account-id of the jira user or username in case of on-prem Jira
# URL is the Jira application url
url: https://localhost/Jira
# account-id is the account-id of the Jira user or username in case of on-prem Jira
account-id: test-account-id
# email is the email of the user for jira instance
# email is the email of the user for Jira instance
email: test@test.com
# token is the token for jira instance or password in case of on-prem Jira
# token is the token for Jira instance or password in case of on-prem Jira
token: test-token
# project-name is the name of the project.
project-name: test-project-name

14
nuclei-jsonschema.json
View File

@ -131,7 +131,7 @@
},
"type": {
"$schema": "http://json-schema.org/draft-04/schema#",
"$ref": "#/definitions/extractors.TypeHolder"
"$ref": "#/definitions/extractors.ExtractorTypeHolder"
},
"regex": {
"items": {
@ -194,7 +194,7 @@
"additionalProperties": false,
"type": "object"
},
"extractors.TypeHolder": {
"extractors.ExtractorTypeHolder": {
"enum": [
"regex",
"kval",
@ -780,6 +780,11 @@
"type": "boolean",
"title": "skip variable checks",
"description": "Skips the check for unresolved variables in request"
},
"iterate-all": {
"type": "boolean",
"title": "iterate all the values",
"description": "Iterates all the values extracted from internal extractors"
}
},
"additionalProperties": false,
@ -1118,6 +1123,11 @@
"type": "boolean",
"title": "mark requests as self-contained",
"description": "Mark Requests for the template as self-contained"
},
"stop-at-first-match": {
"type": "boolean",
"title": "stop at first match",
"description": "Stop at first match for the template"
}
},
"additionalProperties": false,

4
v2/Makefile
View File

@ -18,5 +18,9 @@ docs:
./cmd/docgen/docgen docs.md nuclei-jsonschema.json
test:
$(GOTEST) -v ./...
integration:
bash ../integration_tests/run.sh
functional:
bash cmd/functional-tests/run.sh
tidy:
$(GOMOD) tidy

13
v2/cmd/cve-annotate/main.go
View File

@ -10,6 +10,7 @@ import (
"strings"
"github.com/Ice3man543/nvd"
"github.com/projectdiscovery/nuclei/v2/pkg/catalog"
)
@ -116,26 +117,26 @@ func getCVEData(client *nvd.Client, filePath, data string) {
}
if !strings.Contains(infoBlockClean, "classification") && (cvssScore != 0 && cvssMetrics != "") {
changed = true
newInfoBlock = newInfoBlock + fmt.Sprintf("\n classification:\n cvss-metrics: %s\n cvss-score: %.2f\n cve-id: %s", cvssMetrics, cvssScore, cveName)
newInfoBlock += fmt.Sprintf("\n classification:\n cvss-metrics: %s\n cvss-score: %.2f\n cve-id: %s", cvssMetrics, cvssScore, cveName)
if len(cweID) > 0 && (cweID[0] != "NVD-CWE-Other" && cweID[0] != "NVD-CWE-noinfo") {
newInfoBlock = newInfoBlock + fmt.Sprintf("\n cwe-id: %s", strings.Join(cweID, ","))
newInfoBlock += fmt.Sprintf("\n cwe-id: %s", strings.Join(cweID, ","))
}
}
// If there is no description field, fill the description from CVE information
if !strings.Contains(infoBlockClean, "description:") && len(cveItem.CVE.Description.DescriptionData) > 0 {
changed = true
newInfoBlock = newInfoBlock + fmt.Sprintf("\n description: %s", fmt.Sprintf("%q", cveItem.CVE.Description.DescriptionData[0].Value))
newInfoBlock += fmt.Sprintf("\n description: %s", fmt.Sprintf("%q", cveItem.CVE.Description.DescriptionData[0].Value))
}
if !strings.Contains(infoBlockClean, "reference:") && len(cveItem.CVE.References.ReferenceData) > 0 {
changed = true
newInfoBlock = newInfoBlock + "\n reference:"
newInfoBlock += "\n reference:"
for _, reference := range cveItem.CVE.References.ReferenceData {
newInfoBlock = newInfoBlock + fmt.Sprintf("\n - %s", reference.URL)
newInfoBlock += fmt.Sprintf("\n - %s", reference.URL)
}
}
newTemplate := strings.ReplaceAll(data, infoBlockClean, newInfoBlock)
if changed {
_ = ioutil.WriteFile(filePath, []byte(newTemplate), 0777)
_ = ioutil.WriteFile(filePath, []byte(newTemplate), 0644)
fmt.Printf("Wrote updated template to %s\n", filePath)
}
}

7
v2/cmd/docgen/docgen.go
View File

@ -10,10 +10,11 @@ import (
"strings"
"github.com/alecthomas/jsonschema"
"github.com/projectdiscovery/nuclei/v2/pkg/templates"
)
var pathRegex = regexp.MustCompile(`github.com/projectdiscovery/nuclei/v2/(?:internal|pkg)/(?:.*/)?([A-Za-z\.]+)`)
var pathRegex = regexp.MustCompile(`github\.com/projectdiscovery/nuclei/v2/(?:internal|pkg)/(?:.*/)?([A-Za-z.]+)`)
func main() {
// Generate yaml syntax documentation
@ -21,7 +22,7 @@ func main() {
if err != nil {
log.Fatalf("Could not encode docs: %s\n", err)
}
err = ioutil.WriteFile(os.Args[1], data, 0777)
err = ioutil.WriteFile(os.Args[1], data, 0644)
if err != nil {
log.Fatalf("Could not write docs: %s\n", err)
}
@ -43,7 +44,7 @@ func main() {
for _, match := range pathRegex.FindAllStringSubmatch(schema, -1) {
schema = strings.ReplaceAll(schema, match[0], match[1])
}
err = ioutil.WriteFile(os.Args[2], []byte(schema), 0777)
err = ioutil.WriteFile(os.Args[2], []byte(schema), 0644)
if err != nil {
log.Fatalf("Could not write jsonschema: %s\n", err)
}

81
v2/cmd/integration-test/headless.go Normal file
View File

@ -0,0 +1,81 @@
package main
import (
"net/http"
"net/http/httptest"
"github.com/julienschmidt/httprouter"
"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
)
var headlessTestcases = map[string]testutils.TestCase{
"headless/headless-basic.yaml": &headlessBasic{},
"headless/headless-header-action.yaml": &headlessHeaderActions{},
"headless/headless-extract-values.yaml": &headlessExtractValues{},
}
type headlessBasic struct{}
// Execute executes a test case and returns an error if occurred
func (h *headlessBasic) Execute(filePath string) error {
router := httprouter.New()
router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
_, _ = w.Write([]byte("<html><body></body></html>"))
})
ts := httptest.NewServer(router)
defer ts.Close()
results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug, "-headless")
if err != nil {
return err
}
if len(results) != 1 {
return errIncorrectResultsCount(results)
}
return nil
}
type headlessHeaderActions struct{}
// Execute executes a test case and returns an error if occurred
func (h *headlessHeaderActions) Execute(filePath string) error {
router := httprouter.New()
router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
testValue := r.Header.Get("test")
if r.Header.Get("test") != "" {
_, _ = w.Write([]byte("<html><body>" + testValue + "</body></html>"))
}
})
ts := httptest.NewServer(router)
defer ts.Close()
results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug, "-headless")
if err != nil {
return err
}
if len(results) != 1 {
return errIncorrectResultsCount(results)
}
return nil
}
type headlessExtractValues struct{}
// Execute executes a test case and returns an error if occurred
func (h *headlessExtractValues) Execute(filePath string) error {
router := httprouter.New()
router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
_, _ = w.Write([]byte("<html><body><a href='/test.html'>test</a></body></html>"))
})
ts := httptest.NewServer(router)
defer ts.Close()
results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug, "-headless")
if err != nil {
return err
}
if len(results) != 3 {
return errIncorrectResultsCount(results)
}
return nil
}

28
v2/cmd/integration-test/http.go
View File

@ -36,21 +36,22 @@ var httpTestcases = map[string]testutils.TestCase{
"http/get-case-insensitive.yaml": &httpGetCaseInsensitive{},
"http/get.yaml,http/get-case-insensitive.yaml": &httpGetCaseInsensitiveCluster{},
"http/get-redirects-chain-headers.yaml": &httpGetRedirectsChainHeaders{},
"http/dsl-matcher-variable.yaml": &httpDSLVariable{},
}
type httpInteractshRequest struct{}
// Executes executes a test case and returns an error if occurred
// Execute executes a test case and returns an error if occurred
func (h *httpInteractshRequest) Execute(filePath string) error {
router := httprouter.New()
router.GET("/", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
value := r.Header.Get("url")
if value != "" {
if resp, _ := http.DefaultClient.Get(value); resp != nil {
resp.Body.Close()
}
}
}))
})
ts := httptest.NewServer(router)
defer ts.Close()
@ -155,6 +156,27 @@ func (h *httpGet) Execute(filePath string) error {
return nil
}
type httpDSLVariable struct{}
// Execute executes a test case and returns an error if occurred
func (h *httpDSLVariable) Execute(filePath string) error {
router := httprouter.New()
router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
fmt.Fprintf(w, "This is test matcher text")
})
ts := httptest.NewServer(router)
defer ts.Close()
results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
if err != nil {
return err
}
if len(results) != 5 {
return errIncorrectResultsCount(results)
}
return nil
}
type httpPostBody struct{}
// Execute executes a test case and returns an error if occurred

2
v2/cmd/integration-test/integration-test.go
View File

@ -6,6 +6,7 @@ import (
"strings"
"github.com/logrusorgru/aurora"
"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
)
@ -28,6 +29,7 @@ func main() {
"workflow": workflowTestcases,
"loader": loaderTestcases,
"websocket": websocketTestCases,
"headless": headlessTestcases,
}
for proto, tests := range protocolTests {
if protocol == "" || protocol == proto {

17
v2/cmd/integration-test/loader.go
View File

@ -8,6 +8,7 @@ import (
"strings"
"github.com/julienschmidt/httprouter"
"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
)
@ -24,14 +25,14 @@ type remoteTemplateList struct{}
func (h *remoteTemplateList) Execute(templateList string) error {
router := httprouter.New()
router.GET("/", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
fmt.Fprintf(w, "This is test matcher text")
if strings.EqualFold(r.Header.Get("test"), "nuclei") {
fmt.Fprintf(w, "This is test headers matcher text")
}
}))
})
router.GET("/template_list", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
router.GET("/template_list", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
file, err := os.ReadFile(templateList)
if err != nil {
w.WriteHeader(500)
@ -40,7 +41,7 @@ func (h *remoteTemplateList) Execute(templateList string) error {
if err != nil {
w.WriteHeader(500)
}
}))
})
ts := httptest.NewServer(router)
defer ts.Close()
@ -60,14 +61,14 @@ type remoteWorkflowList struct{}
func (h *remoteWorkflowList) Execute(workflowList string) error {
router := httprouter.New()
router.GET("/", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
fmt.Fprintf(w, "This is test matcher text")
if strings.EqualFold(r.Header.Get("test"), "nuclei") {
fmt.Fprintf(w, "This is test headers matcher text")
}
}))
})
router.GET("/workflow_list", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
router.GET("/workflow_list", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
file, err := os.ReadFile(workflowList)
if err != nil {
w.WriteHeader(500)
@ -76,7 +77,7 @@ func (h *remoteWorkflowList) Execute(workflowList string) error {
if err != nil {
w.WriteHeader(500)
}
}))
})
ts := httptest.NewServer(router)
defer ts.Close()

3
v2/cmd/integration-test/websocket.go
View File

@ -5,6 +5,7 @@ import (
"strings"
"github.com/gobwas/ws/wsutil"
"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
)
@ -22,7 +23,7 @@ func (h *websocketBasic) Execute(filePath string) error {
connHandler := func(conn net.Conn) {
for {
msg, op, _ := wsutil.ReadClientData(conn)
if string(msg) != string("hello") {
if string(msg) != "hello" {
return
}
_ = wsutil.WriteServerMessage(conn, op, []byte("world"))

36
v2/cmd/nuclei/issue-tracker-config.yaml
View File

@ -5,51 +5,51 @@
#deny-list:
# severity: info, low, medium
# github contains configuration options for github issue tracker
#github:
# # base-url (optional) is the self-hosted github application url
# GitHub contains configuration options for GitHub issue tracker
#GitHub:
# # base-url (optional) is the self-hosted GitHub application url
# base-url: ""
# # username is the username of the github user
# # username is the username of the GitHub user
# username: ""
# # owner is the owner name of the repository for issues.
# owner: ""
# # token is the token for github account.
# # token is the token for GitHub account.
# token: ""
# # project-name is the name of the repository.
# project-name: ""
# # issue-label (optional) is the label of the created issue type
# issue-label: ""
# # severity-as-label (optional) sets the sevetiry as the label of the created issue type
# # severity-as-label (optional) sets the severity as the label of the created issue type
# severity-as-label: false
# gitlab contains configuration options for gitlab issue tracker
#gitlab:
# # base-url (optional) is the self-hosted gitlab application url
# GitLab contains configuration options for GitLab issue tracker
#GitLab:
# # base-url (optional) is the self-hosted GitLab application url
# base-url: ""
# # username is the username of the gitlab user
# # username is the username of the GitLab user
# username: ""
# # token is the token for gitlab account.
# # token is the token for GitLab account.
# token: ""
# # project-id is the ID of the repository.
# project-id: ""
# # issue-label (optional) is the label of the created issue type
# issue-label: ""
# # severity-as-label (optional) sets the sevetiry as the label of the created issue type
# # severity-as-label (optional) sets the severity as the label of the created issue type
# severity-as-label: false
# jira contains configuration options for jira issue tracker
#jira:
# Jira contains configuration options for Jira issue tracker
#Jira:
# # cloud (optional) is the boolean which tells if Jira instance is running in the cloud or on-prem version is used
# cloud: true
# # update-existing (optional) is the boolean which tells if the existing, opened issue should be updated or new one should be created
# update-existing: false
# # URL is the jira application url
# # URL is the Jira application URL
# url: ""
# # account-id is the account-id of the jira user or username in case of on-prem Jira
# # account-id is the account-id of the Jira user or username in case of on-prem Jira
# account-id: ""
# # email is the email of the user for jira instance
# # email is the email of the user for Jira instance
# email: ""
# # token is the token for jira instance or password in case of on-prem Jira
# # token is the token for Jira instance or password in case of on-prem Jira
# token: ""
# # project-name is the name of the project.
# project-name: ""

4
v2/cmd/nuclei/main.go
View File

@ -110,7 +110,7 @@ on extensive configurability, massive extensibility and ease of use.`)
flagSet.IntVar(&options.InteractionsCacheSize, "interactions-cache-size", 5000, "number of requests to keep in the interactions cache"),
flagSet.IntVar(&options.InteractionsEviction, "interactions-eviction", 60, "number of seconds to wait before evicting requests from cache"),
flagSet.IntVar(&options.InteractionsPollDuration, "interactions-poll-duration", 5, "number of seconds to wait before each interaction poll request"),
flagSet.IntVar(&options.InteractionsCooldownPeriod, "interactions-cooldown-period", 5, "extra time for interaction polling before exiting"),
flagSet.IntVar(&options.InteractionsCoolDownPeriod, "interactions-cooldown-period", 5, "extra time for interaction polling before exiting"),
flagSet.BoolVarP(&options.NoInteractsh, "no-interactsh", "ni", false, "disable interactsh server for OAST testing, exclude OAST based templates"),
)
@ -134,7 +134,7 @@ on extensive configurability, massive extensibility and ease of use.`)
)
createGroup(flagSet, "headless", "Headless",
flagSet.BoolVar(&options.Headless, "headless", false, "enable templates that require headless browser support"),
flagSet.BoolVar(&options.Headless, "headless", false, "enable templates that require headless browser support (root user on linux will disable sandbox)"),
flagSet.IntVar(&options.PageTimeout, "page-timeout", 20, "seconds to wait for each page in headless mode"),
flagSet.BoolVarP(&options.ShowBrowser, "show-browser", "sb", false, "show the browser on the screen when running templates with headless mode"),
flagSet.BoolVarP(&options.UseInstalledChrome, "system-chrome", "sc", false, "Use local installed chrome browser instead of nuclei installed"),

4
v2/go.mod
View File

@ -40,7 +40,7 @@ require (
github.com/projectdiscovery/retryabledns v1.0.13-0.20211109182249-43d38df59660
github.com/projectdiscovery/retryablehttp-go v1.0.2
github.com/projectdiscovery/stringsutil v0.0.0-20210830151154-f567170afdd9
github.com/projectdiscovery/yamldoc-go v1.0.2
github.com/projectdiscovery/yamldoc-go v1.0.3-0.20211126104922-00d2c6bb43b6
github.com/remeh/sizedwaitgroup v1.0.0
github.com/rs/xid v1.3.0
github.com/segmentio/ksuid v1.0.4
@ -65,6 +65,8 @@ require (
moul.io/http2curl v1.0.0
)
require github.com/projectdiscovery/folderutil v0.0.0-20211206150108-b4e7ea80f36e
require (
git.mills.io/prologic/smtpd v0.0.0-20210710122116-a525b76c287a // indirect
github.com/PuerkitoBio/goquery v1.6.0 // indirect

7
v2/go.sum
View File

@ -595,6 +595,10 @@ github.com/projectdiscovery/fileutil v0.0.0-20210914153648-31f843feaad4/go.mod h
github.com/projectdiscovery/fileutil v0.0.0-20210926202739-6050d0acf73c/go.mod h1:U+QCpQnX8o2N2w0VUGyAzjM3yBAe4BKedVElxiImsx0=
github.com/projectdiscovery/fileutil v0.0.0-20210928100737-cab279c5d4b5 h1:2dbm7UhrAKnccZttr78CAmG768sSCd+MBn4ayLVDeqA=
github.com/projectdiscovery/fileutil v0.0.0-20210928100737-cab279c5d4b5/go.mod h1:U+QCpQnX8o2N2w0VUGyAzjM3yBAe4BKedVElxiImsx0=
github.com/projectdiscovery/folderutil v0.0.0-20211206102047-d6bf8e7490ff h1:ci7/Pq9xvrVFb94jeARYb45oSzs85NWG+Fxp/kjgHVc=
github.com/projectdiscovery/folderutil v0.0.0-20211206102047-d6bf8e7490ff/go.mod h1:BMqXH4jNGByVdE2iLtKvc/6XStaiZRuCIaKv1vw9PnI=
github.com/projectdiscovery/folderutil v0.0.0-20211206150108-b4e7ea80f36e h1:RJJuYyuwskYtzZi2gziy6SE/b7saWEzyskaA252E0VY=
github.com/projectdiscovery/folderutil v0.0.0-20211206150108-b4e7ea80f36e/go.mod h1:BMqXH4jNGByVdE2iLtKvc/6XStaiZRuCIaKv1vw9PnI=
github.com/projectdiscovery/goflags v0.0.7/go.mod h1:Jjwsf4eEBPXDSQI2Y+6fd3dBumJv/J1U0nmpM+hy2YY=
github.com/projectdiscovery/goflags v0.0.8-0.20211028121123-edf02bc05b1a h1:EzwVm8i4zmzqZX55vrDtyfogwHh8AAZ3cWCJe4fEduk=
github.com/projectdiscovery/goflags v0.0.8-0.20211028121123-edf02bc05b1a/go.mod h1:Jjwsf4eEBPXDSQI2Y+6fd3dBumJv/J1U0nmpM+hy2YY=
@ -639,8 +643,9 @@ github.com/projectdiscovery/stringsutil v0.0.0-20210804142656-fd3c28dbaafe/go.mo
github.com/projectdiscovery/stringsutil v0.0.0-20210823090203-2f5f137e8e1d/go.mod h1:oTRc18WBv9t6BpaN9XBY+QmG28PUpsyDzRht56Qf49I=
github.com/projectdiscovery/stringsutil v0.0.0-20210830151154-f567170afdd9 h1:xbL1/7h0k6HE3RzPdYk9W/8pUxESrGWewTaZdIB5Pes=
github.com/projectdiscovery/stringsutil v0.0.0-20210830151154-f567170afdd9/go.mod h1:oTRc18WBv9t6BpaN9XBY+QmG28PUpsyDzRht56Qf49I=
github.com/projectdiscovery/yamldoc-go v1.0.2 h1:SKb7PHgSOXm27Zci05ba0FxpyQiu6bGEiVMEcjCK1rQ=
github.com/projectdiscovery/yamldoc-go v1.0.2/go.mod h1:7uSxfMXaBmzvw8m5EhOEjB6nhz0rK/H9sUjq1ciZu24=
github.com/projectdiscovery/yamldoc-go v1.0.3-0.20211126104922-00d2c6bb43b6 h1:DvWRQpw7Ib2CRL3ogYm/BWM+X0UGPfz1n9Ix9YKgFM8=
github.com/projectdiscovery/yamldoc-go v1.0.3-0.20211126104922-00d2c6bb43b6/go.mod h1:8OfZj8p/axkUM/TJoS/O9LDjj/S8u17rxRbqluE9CU4=
github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
github.com/prometheus/client_golang v0.9.3-0.20190127221311-3c4408c8b829/go.mod h1:p2iRAGwDERtqlqzRXnrOVns+ignqQo//hLXqYxZYVNs=
github.com/prometheus/client_golang v1.0.0/go.mod h1:db9x61etRT2tGnBNRi70OPL5FsnadC4Ky3P0J6CfImo=

16
v2/internal/runner/options.go
View File

@ -9,6 +9,7 @@ import (
"github.com/pkg/errors"
"github.com/go-playground/validator/v10"
"github.com/projectdiscovery/fileutil"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/gologger/formatter"
@ -66,15 +67,14 @@ func ParseOptions(options *types.Options) {
// hasStdin returns true if we have stdin input
func hasStdin() bool {
stat, err := os.Stdin.Stat()
fi, err := os.Stdin.Stat()
if err != nil {
return false
}
isPipedFromChrDev := (stat.Mode() & os.ModeCharDevice) == 0
isPipedFromFIFO := (stat.Mode() & os.ModeNamedPipe) != 0
return isPipedFromChrDev || isPipedFromFIFO
if fi.Mode()&os.ModeNamedPipe == 0 {
return false
}
return true
}
// validateOptions validates the configuration options passed
@ -93,7 +93,7 @@ func validateOptions(options *types.Options) error {
if options.Verbose && options.Silent {
return errors.New("both verbose and silent mode specified")
}
//loading the proxy server list from file or cli and test the connectivity
// loading the proxy server list from file or cli and test the connectivity
if err := loadProxyServers(options); err != nil {
return err
}
@ -116,7 +116,7 @@ func validateOptions(options *types.Options) error {
// configureOutput configures the output logging levels to be displayed on the screen
func configureOutput(options *types.Options) {
// If the user desires verbose output, show verbose output
if options.Verbose {
if options.Verbose || options.Validate {
gologger.DefaultLogger.SetMaxLevel(levels.LevelVerbose)
}
if options.Debug {

4
v2/internal/runner/proxy.go
View File

@ -28,7 +28,7 @@ func loadProxyServers(options *types.Options) error {
} else if fileutil.FileExists(p) {
file, err := os.Open(p)
if err != nil {
return fmt.Errorf("could not open proxy file: %s", err)
return fmt.Errorf("could not open proxy file: %w", err)
}
defer file.Close()
scanner := bufio.NewScanner(file)
@ -117,7 +117,7 @@ func validateProxyURL(proxy string) (url.URL, error) {
return url.URL{}, errors.New("invalid proxy format (It should be http[s]/socks5://[username:password@]host:port)")
}
//isSupportedProtocol checks given protocols are supported
// isSupportedProtocol checks given protocols are supported
func isSupportedProtocol(value string) bool {
return value == types.HTTP || value == types.HTTPS || value == types.SOCKS5
}

20
v2/internal/runner/runner.go
View File

@ -69,11 +69,16 @@ func New(options *types.Options) (*Runner, error) {
}
if options.Validate {
parsers.ShouldValidate = true
// Does not update the templates when validate flag is used
options.NoUpdateTemplates = true
}
if err := runner.updateTemplates(); err != nil {
gologger.Warning().Msgf("Could not update templates: %s\n", err)
}
if options.Headless {
if engine.MustDisableSandbox() {
gologger.Warning().Msgf("The current platform and privileged user will run the browser without sandbox\n")
}
browser, err := engine.New(options)
if err != nil {
return nil, err
@ -151,7 +156,7 @@ func New(options *types.Options) (*Runner, error) {
opts.Authorization = options.InteractshToken
opts.CacheSize = int64(options.InteractionsCacheSize)
opts.Eviction = time.Duration(options.InteractionsEviction) * time.Second
opts.ColldownPeriod = time.Duration(options.InteractionsCooldownPeriod) * time.Second
opts.ColldownPeriod = time.Duration(options.InteractionsCoolDownPeriod) * time.Second
opts.PollDuration = time.Duration(options.InteractionsPollDuration) * time.Second
opts.NoInteractsh = runner.options.NoInteractsh
@ -231,10 +236,12 @@ func (r *Runner) RunEnumeration() error {
}
r.options.Templates = append(r.options.Templates, templatesLoaded...)
}
ignoreFile := config.ReadIgnoreFile()
r.options.ExcludeTags = append(r.options.ExcludeTags, ignoreFile.Tags...)
r.options.ExcludedTemplates = append(r.options.ExcludedTemplates, ignoreFile.Files...)
// Exclude ignored file for validation
if !r.options.Validate {
ignoreFile := config.ReadIgnoreFile()
r.options.ExcludeTags = append(r.options.ExcludeTags, ignoreFile.Tags...)
r.options.ExcludedTemplates = append(r.options.ExcludedTemplates, ignoreFile.Files...)
}
var cache *hosterrorscache.Cache
if r.options.MaxHostError > 0 {
cache = hosterrorscache.New(r.options.MaxHostError, hosterrorscache.DefaultMaxHostsCount).SetVerbose(r.options.Verbose)
@ -275,7 +282,7 @@ func (r *Runner) RunEnumeration() error {
if err := store.ValidateTemplates(r.options.Templates, r.options.Workflows); err != nil {
return err
}
if stats.GetValue(parsers.SyntaxErrorStats) == 0 && stats.GetValue(parsers.SyntaxWarningStats) == 0 {
if stats.GetValue(parsers.SyntaxErrorStats) == 0 && stats.GetValue(parsers.SyntaxWarningStats) == 0 && stats.GetValue(parsers.RuntimeWarningsStats) == 0 {
gologger.Info().Msgf("All templates validated successfully\n")
} else {
return errors.New("encountered errors while performing template validation")
@ -358,6 +365,7 @@ func (r *Runner) displayExecutionInfo(store *loader.Store) {
// Display stats for any loaded templates' syntax warnings or errors
stats.Display(parsers.SyntaxWarningStats)
stats.Display(parsers.SyntaxErrorStats)
stats.Display(parsers.RuntimeWarningsStats)
builder := &strings.Builder{}
if r.templatesConfig != nil && r.templatesConfig.NucleiLatestVersion != "" {

129
v2/internal/runner/update.go
View File

@ -54,7 +54,7 @@ func (r *Runner) updateTemplates() error { // TODO this method does more than ju
return err
}
configDir := filepath.Join(home, ".config", "nuclei")
_ = os.MkdirAll(configDir, os.ModePerm)
_ = os.MkdirAll(configDir, 0755)
if err := r.readInternalConfigurationFile(home, configDir); err != nil {
return errors.Wrap(err, "could not read configuration file")
@ -242,12 +242,12 @@ func (r *Runner) getLatestReleaseFromGithub(latestTag string) (*github.Repositor
func (r *Runner) downloadReleaseAndUnzip(ctx context.Context, version, downloadURL string) (*templateUpdateResults, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, downloadURL, nil)
if err != nil {
return nil, fmt.Errorf("failed to create HTTP request to %s: %s", downloadURL, err)
return nil, fmt.Errorf("failed to create HTTP request to %s: %w", downloadURL, err)
}
res, err := http.DefaultClient.Do(req)
if err != nil {
return nil, fmt.Errorf("failed to download a release file from %s: %s", downloadURL, err)
return nil, fmt.Errorf("failed to download a release file from %s: %w", downloadURL, err)
}
defer res.Body.Close()
if res.StatusCode != http.StatusOK {
@ -256,23 +256,23 @@ func (r *Runner) downloadReleaseAndUnzip(ctx context.Context, version, downloadU
buf, err := ioutil.ReadAll(res.Body)
if err != nil {
return nil, fmt.Errorf("failed to create buffer for zip file: %s", err)
return nil, fmt.Errorf("failed to create buffer for zip file: %w", err)
}
reader := bytes.NewReader(buf)
zipReader, err := zip.NewReader(reader, reader.Size())
if err != nil {
return nil, fmt.Errorf("failed to uncompress zip file: %s", err)
return nil, fmt.Errorf("failed to uncompress zip file: %w", err)
}
// Create the template folder if it doesn't exist
if err := os.MkdirAll(r.templatesConfig.TemplatesDirectory, os.ModePerm); err != nil {
return nil, fmt.Errorf("failed to create template base folder: %s", err)
if err := os.MkdirAll(r.templatesConfig.TemplatesDirectory, 0755); err != nil {
return nil, fmt.Errorf("failed to create template base folder: %w", err)
}
results, err := r.compareAndWriteTemplates(zipReader)
if err != nil {
return nil, fmt.Errorf("failed to write templates: %s", err)
return nil, fmt.Errorf("failed to write templates: %w", err)
}
if r.options.Verbose {
@ -291,7 +291,7 @@ func (r *Runner) downloadReleaseAndUnzip(ctx context.Context, version, downloadU
buffer.WriteString("\n")
}
if err := ioutil.WriteFile(additionsFile, buffer.Bytes(), os.ModePerm); err != nil {
if err := ioutil.WriteFile(additionsFile, buffer.Bytes(), 0644); err != nil {
return nil, errors.Wrap(err, "could not write new additions file")
}
return results, err
@ -316,60 +316,42 @@ func (r *Runner) compareAndWriteTemplates(zipReader *zip.Reader) (*templateUpdat
// If the path isn't found in new update after being read from the previous checksum,
// it is removed. This allows us fine-grained control over the download process
// as well as solves a long problem with nuclei-template updates.
checksumFile := filepath.Join(r.templatesConfig.TemplatesDirectory, ".checksum")
configuredTemplateDirectory := r.templatesConfig.TemplatesDirectory
checksumFile := filepath.Join(configuredTemplateDirectory, ".checksum")
templateChecksumsMap, _ := createTemplateChecksumsMap(checksumFile)
for _, zipTemplateFile := range zipReader.File {
directory, name := filepath.Split(zipTemplateFile.Name)
if name == "" {
templateAbsolutePath, skipFile, err := calculateTemplateAbsolutePath(zipTemplateFile.Name, configuredTemplateDirectory)
if err != nil {
return nil, err
}
if skipFile {
continue
}
paths := strings.Split(directory, string(os.PathSeparator))
finalPath := filepath.Join(paths[1:]...)
if strings.HasPrefix(name, ".") || strings.HasPrefix(finalPath, ".") || strings.EqualFold(name, "README.md") {
continue
}
results.totalCount++
templateDirectory := filepath.Join(r.templatesConfig.TemplatesDirectory, finalPath)
if err := os.MkdirAll(templateDirectory, os.ModePerm); err != nil {
return nil, fmt.Errorf("failed to create template folder %s : %s", templateDirectory, err)
}
templatePath := filepath.Join(templateDirectory, name)
isAddition := false
if _, statErr := os.Stat(templatePath); os.IsNotExist(statErr) {
if _, statErr := os.Stat(templateAbsolutePath); os.IsNotExist(statErr) {
isAddition = true
}
templateFile, err := os.OpenFile(templatePath, os.O_TRUNC|os.O_CREATE|os.O_WRONLY, 0777)
newTemplateChecksum, err := writeUnZippedTemplateFile(templateAbsolutePath, zipTemplateFile)
if err != nil {
templateFile.Close()
return nil, fmt.Errorf("could not create uncompressed file: %s", err)
return nil, err
}
zipTemplateFileReader, err := zipTemplateFile.Open()
oldTemplateChecksum, checksumOk := templateChecksumsMap[templateAbsolutePath]
relativeTemplatePath, err := filepath.Rel(configuredTemplateDirectory, templateAbsolutePath)
if err != nil {
templateFile.Close()
return nil, fmt.Errorf("could not open archive to extract file: %s", err)
return nil, fmt.Errorf("could not calculate relative path for template: %s. %w", templateAbsolutePath, err)
}
hasher := md5.New()
// Save file and also read into hasher for md5
if _, err := io.Copy(templateFile, io.TeeReader(zipTemplateFileReader, hasher)); err != nil {
templateFile.Close()
return nil, fmt.Errorf("could not write template file: %s", err)
}
templateFile.Close()
oldChecksum, checksumOK := templateChecksumsMap[templatePath]
checksum := hex.EncodeToString(hasher.Sum(nil))
if isAddition {
results.additions = append(results.additions, filepath.Join(finalPath, name))
} else if checksumOK && oldChecksum[0] != checksum {
results.modifications = append(results.modifications, filepath.Join(finalPath, name))
results.additions = append(results.additions, relativeTemplatePath)
} else if checksumOk && oldTemplateChecksum[0] != newTemplateChecksum {
results.modifications = append(results.modifications, relativeTemplatePath)
}
results.checksums[templatePath] = checksum
results.checksums[templateAbsolutePath] = newTemplateChecksum
results.totalCount++
}
// If we don't find the previous file in the newly downloaded list,
@ -378,12 +360,63 @@ func (r *Runner) compareAndWriteTemplates(zipReader *zip.Reader) (*templateUpdat
_, ok := results.checksums[templatePath]
if !ok && templateChecksums[0] == templateChecksums[1] {
_ = os.Remove(templatePath)
results.deletions = append(results.deletions, strings.TrimPrefix(strings.TrimPrefix(templatePath, r.templatesConfig.TemplatesDirectory), string(os.PathSeparator)))
results.deletions = append(results.deletions, strings.TrimPrefix(strings.TrimPrefix(templatePath, configuredTemplateDirectory), string(os.PathSeparator)))
}
}
return results, nil
}
func writeUnZippedTemplateFile(templateAbsolutePath string, zipTemplateFile *zip.File) (string, error) {
templateFile, err := os.OpenFile(templateAbsolutePath, os.O_TRUNC|os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
return "", fmt.Errorf("could not create template file: %w", err)
}
zipTemplateFileReader, err := zipTemplateFile.Open()
if err != nil {
_ = templateFile.Close()
return "", fmt.Errorf("could not open archive to extract file: %w", err)
}
md5Hash := md5.New()
// Save file and also read into hash.Hash for md5
if _, err := io.Copy(templateFile, io.TeeReader(zipTemplateFileReader, md5Hash)); err != nil {
_ = templateFile.Close()
return "", fmt.Errorf("could not write template file: %w", err)
}
if err := templateFile.Close(); err != nil {
return "", fmt.Errorf("could not close file newly created template file: %w", err)
}
checksum := hex.EncodeToString(md5Hash.Sum(nil))
return checksum, nil
}
func calculateTemplateAbsolutePath(zipFilePath, configuredTemplateDirectory string) (string, bool, error) {
directory, fileName := filepath.Split(zipFilePath)
if strings.TrimSpace(fileName) == "" || strings.HasPrefix(fileName, ".") || strings.EqualFold(fileName, "README.md") {
return "", true, nil
}
directoryPathChunks := strings.Split(directory, string(os.PathSeparator))
relativeDirectoryPathWithoutZipRoot := filepath.Join(directoryPathChunks[1:]...)
if strings.HasPrefix(relativeDirectoryPathWithoutZipRoot, ".") {
return "", true, nil
}
templateDirectory := filepath.Join(configuredTemplateDirectory, relativeDirectoryPathWithoutZipRoot)
if err := os.MkdirAll(templateDirectory, 0755); err != nil {
return "", false, fmt.Errorf("failed to create template folder: %s. %w", templateDirectory, err)
}
return filepath.Join(templateDirectory, fileName), false, nil
}
// createTemplateChecksumsMap reads the previous checksum file from the disk.
// Creates a map of template paths and their previous and currently calculated checksums as values.
func createTemplateChecksumsMap(checksumsFilePath string) (map[string][2]string, error) {

48
v2/internal/runner/update_test.go
View File

@ -12,10 +12,11 @@ import (
"strings"
"testing"
"github.com/stretchr/testify/require"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v2/pkg/catalog/config"
"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
"github.com/stretchr/testify/require"
)
func TestDownloadReleaseAndUnzipAddition(t *testing.T) {
@ -25,7 +26,7 @@ func TestDownloadReleaseAndUnzipAddition(t *testing.T) {
require.Nil(t, err, "could not create temp directory")
defer os.RemoveAll(baseTemplates)
err = ioutil.WriteFile(filepath.Join(baseTemplates, "base.yaml"), []byte("id: test"), 0777)
err = ioutil.WriteFile(filepath.Join(baseTemplates, "base.yaml"), []byte("id: test"), os.ModePerm)
require.Nil(t, err, "could not create write base file")
err = zipFromDirectory("base.zip", baseTemplates)
@ -50,9 +51,9 @@ func TestDownloadReleaseAndUnzipAddition(t *testing.T) {
require.Nil(t, err, "could not create temp directory")
defer os.RemoveAll(newTempDir)
err = ioutil.WriteFile(filepath.Join(newTempDir, "base.yaml"), []byte("id: test"), 0777)
err = ioutil.WriteFile(filepath.Join(newTempDir, "base.yaml"), []byte("id: test"), os.ModePerm)
require.Nil(t, err, "could not create base file")
err = ioutil.WriteFile(filepath.Join(newTempDir, "new.yaml"), []byte("id: test"), 0777)
err = ioutil.WriteFile(filepath.Join(newTempDir, "new.yaml"), []byte("id: test"), os.ModePerm)
require.Nil(t, err, "could not create new file")
err = zipFromDirectory("new.zip", newTempDir)
@ -77,7 +78,7 @@ func TestDownloadReleaseAndUnzipDeletion(t *testing.T) {
require.Nil(t, err, "could not create temp directory")
defer os.RemoveAll(baseTemplates)
err = ioutil.WriteFile(filepath.Join(baseTemplates, "base.yaml"), []byte("id: test"), 0777)
err = ioutil.WriteFile(filepath.Join(baseTemplates, "base.yaml"), []byte("id: test"), os.ModePerm)
require.Nil(t, err, "could not create write base file")
err = zipFromDirectory("base.zip", baseTemplates)
@ -118,6 +119,43 @@ func TestDownloadReleaseAndUnzipDeletion(t *testing.T) {
require.Equal(t, "base.yaml", results.deletions[0], "could not get correct new deletions")
}
func TestCalculateTemplateAbsolutePath(t *testing.T) {
configuredTemplateDirectory := filepath.Join(os.TempDir(), "templates")
defer os.RemoveAll(configuredTemplateDirectory)
t.Run("positive scenarios", func(t *testing.T) {
zipFilePathsExpectedPathsMap := map[string]string{
"nuclei-templates/cve/test.yaml": filepath.Join(configuredTemplateDirectory, "cve/test.yaml"),
"nuclei-templates/cve/test/test.yaml": filepath.Join(configuredTemplateDirectory, "cve/test/test.yaml"),
}
for filePathFromZip, expectedTemplateAbsPath := range zipFilePathsExpectedPathsMap {
calculatedTemplateAbsPath, skipFile, err := calculateTemplateAbsolutePath(filePathFromZip, configuredTemplateDirectory)
require.Nil(t, err)
require.Equal(t, expectedTemplateAbsPath, calculatedTemplateAbsPath)
require.False(t, skipFile)
}
})
t.Run("negative scenarios", func(t *testing.T) {
filePathsFromZip := []string{
"./../nuclei-templates/../cve/test.yaml",
"nuclei-templates/../cve/test.yaml",
"nuclei-templates/cve/../test.yaml",
"nuclei-templates/././../cve/test.yaml",
"nuclei-templates/.././../cve/test.yaml",
"nuclei-templates/.././../cve/../test.yaml",
}
for _, filePathFromZip := range filePathsFromZip {
calculatedTemplateAbsPath, skipFile, err := calculateTemplateAbsolutePath(filePathFromZip, configuredTemplateDirectory)
require.Nil(t, err)
require.True(t, skipFile)
require.Equal(t, "", calculatedTemplateAbsPath)
}
})
}
func zipFromDirectory(zipPath, directory string) error {
file, err := os.Create(zipPath)
if err != nil {

8
v2/pkg/catalog/config/config.go
View File

@ -26,7 +26,7 @@ type Config struct {
const nucleiConfigFilename = ".templates-config.json"
// Version is the current version of nuclei
const Version = `2.5.4-dev`
const Version = `2.5.4`
func getConfigDetails() (string, error) {
homeDir, err := os.UserHomeDir()
@ -34,7 +34,7 @@ func getConfigDetails() (string, error) {
return "", errors.Wrap(err, "could not get home directory")
}
configDir := filepath.Join(homeDir, ".config", "nuclei")
_ = os.MkdirAll(configDir, os.ModePerm)
_ = os.MkdirAll(configDir, 0755)
templatesConfigFile := filepath.Join(configDir, nucleiConfigFilename)
return templatesConfigFile, nil
}
@ -67,7 +67,7 @@ func WriteConfiguration(config *Config) error {
if err != nil {
return err
}
file, err := os.OpenFile(templatesConfigFile, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0777)
file, err := os.OpenFile(templatesConfigFile, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
if err != nil {
return err
}
@ -112,7 +112,7 @@ func getIgnoreFilePath() string {
home, err := os.UserHomeDir()
if err == nil {
configDir := filepath.Join(home, ".config", "nuclei")
_ = os.MkdirAll(configDir, os.ModePerm)
_ = os.MkdirAll(configDir, 0755)
defIgnoreFilePath = filepath.Join(configDir, nucleiIgnoreFile)
return defIgnoreFilePath

3
v2/pkg/catalog/find.go
View File

@ -7,6 +7,7 @@ import (
"github.com/karrick/godirwalk"
"github.com/pkg/errors"
"github.com/projectdiscovery/gologger"
)
@ -79,7 +80,7 @@ func (c *Catalog) GetTemplatePath(target string) ([]string, error) {
}
// convertPathToAbsolute resolves the paths provided to absolute paths
// before doing any operations on them regardless of them being blob, folders, files, etc.
// before doing any operations on them regardless of them being BLOB, folders, files, etc.
func (c *Catalog) convertPathToAbsolute(t string) (string, error) {
if strings.Contains(t, "*") {
file := filepath.Base(t)

6
v2/pkg/catalog/loader/filter/tag_filter.go
View File

@ -230,9 +230,9 @@ func splitCommaTrim(value string) []string {
if !strings.Contains(value, ",") {
return []string{strings.ToLower(value)}
}
splitted := strings.Split(value, ",")
final := make([]string, len(splitted))
for i, value := range splitted {
split := strings.Split(value, ",")
final := make([]string, len(split))
for i, value := range split {
final[i] = strings.ToLower(strings.TrimSpace(value))
}
return final

12
v2/pkg/catalog/loader/loader.go
View File

@ -12,6 +12,7 @@ import (
"github.com/projectdiscovery/nuclei/v2/pkg/templates"
templateTypes "github.com/projectdiscovery/nuclei/v2/pkg/templates/types"
"github.com/projectdiscovery/nuclei/v2/pkg/types"
"github.com/projectdiscovery/nuclei/v2/pkg/utils/stats"
)
// Config contains the configuration options for the loader
@ -98,8 +99,8 @@ func New(config *Config) (*Store, error) {
finalWorkflows: config.Workflows,
}
urlbasedTemplatesProvided := len(config.TemplateURLs) > 0 || len(config.WorkflowURLs) > 0
if urlbasedTemplatesProvided {
urlBasedTemplatesProvided := len(config.TemplateURLs) > 0 || len(config.WorkflowURLs) > 0
if urlBasedTemplatesProvided {
remoteTemplates, remoteWorkflows, err := getRemoteTemplatesAndWorkflows(config.TemplateURLs, config.WorkflowURLs)
if err != nil {
return store, err
@ -109,7 +110,7 @@ func New(config *Config) (*Store, error) {
}
// Handle a case with no templates or workflows, where we use base directory
if len(store.finalTemplates) == 0 && len(store.finalWorkflows) == 0 && !urlbasedTemplatesProvided {
if len(store.finalTemplates) == 0 && len(store.finalWorkflows) == 0 && !urlBasedTemplatesProvided {
store.finalTemplates = []string{config.TemplatesDirectory}
}
@ -141,6 +142,10 @@ func (store *Store) Load() {
// ValidateTemplates takes a list of templates and validates them
// erroring out on discovering any faulty templates.
func (store *Store) ValidateTemplates(templatesList, workflowsList []string) error {
// consider all the templates by default if no templates passed by user
if len(templatesList) == 0 {
templatesList = store.finalTemplates
}
templatePaths := store.config.Catalog.GetTemplatesPath(templatesList)
workflowPaths := store.config.Catalog.GetTemplatesPath(workflowsList)
@ -214,6 +219,7 @@ func (store *Store) LoadTemplates(templatesList []string) []*templates.Template
if loaded {
parsed, err := templates.Parse(templatePath, store.preprocessor, store.config.ExecutorOptions)
if err != nil {
stats.Increment(parsers.RuntimeWarningsStats)
gologger.Warning().Msgf("Could not parse template %s: %s\n", templatePath, err)
} else if parsed != nil {
loadedTemplates = append(loadedTemplates, parsed)

39
v2/pkg/catalog/path.go
View File

@ -4,6 +4,10 @@ import (
"fmt"
"os"
"path/filepath"
"github.com/pkg/errors"
"github.com/projectdiscovery/folderutil"
)
// ResolvePath resolves the path to an absolute one in various ways.
@ -15,11 +19,10 @@ func (c *Catalog) ResolvePath(templateName, second string) (string, error) {
if filepath.IsAbs(templateName) {
return templateName, nil
}
if second != "" {
secondBasePath := filepath.Join(filepath.Dir(second), templateName)
if _, err := os.Stat(secondBasePath); !os.IsNotExist(err) {
return secondBasePath, nil
if potentialPath, err := c.tryResolve(secondBasePath); err != errNoValidCombination {
return potentialPath, nil
}
}
@ -29,15 +32,37 @@ func (c *Catalog) ResolvePath(templateName, second string) (string, error) {
}
templatePath := filepath.Join(curDirectory, templateName)
if _, err := os.Stat(templatePath); !os.IsNotExist(err) {
return templatePath, nil
if potentialPath, err := c.tryResolve(templatePath); err != errNoValidCombination {
return potentialPath, nil
}
if c.templatesDirectory != "" {
templatePath := filepath.Join(c.templatesDirectory, templateName)
if _, err := os.Stat(templatePath); !os.IsNotExist(err) {
return templatePath, nil
if potentialPath, err := c.tryResolve(templatePath); err != errNoValidCombination {
return potentialPath, nil
}
}
return "", fmt.Errorf("no such path found: %s", templateName)
}
var errNoValidCombination = errors.New("no valid combination found")
// tryResolve attempts to load locate the target by iterating across all the folders tree
func (c *Catalog) tryResolve(fullpath string) (string, error) {
dir, filename := filepath.Split(fullpath)
pathInfo, err := folderutil.NewPathInfo(dir)
if err != nil {
return "", err
}
pathInfoItems, err := pathInfo.MeshWith(filename)
if err != nil {
return "", err
}
for _, pathInfoItem := range pathInfoItems {
if _, err := os.Stat(pathInfoItem); !os.IsNotExist(err) {
return pathInfoItem, nil
}
}
return "", errNoValidCombination
}

33
v2/pkg/core/execute.go
View File

@ -1,11 +1,12 @@
package core
import (
"github.com/remeh/sizedwaitgroup"
"go.uber.org/atomic"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v2/pkg/templates"
"github.com/projectdiscovery/nuclei/v2/pkg/templates/types"
"github.com/remeh/sizedwaitgroup"
"go.uber.org/atomic"
)
// Execute takes a list of templates/workflows that have been compiled
@ -17,7 +18,7 @@ func (e *Engine) Execute(templates []*templates.Template, target InputProvider)
return e.ExecuteWithOpts(templates, target, false)
}
// ExecuteWithOpts is execute with the full options
// ExecuteWithOpts executes with the full options
func (e *Engine) ExecuteWithOpts(templatesList []*templates.Template, target InputProvider, noCluster bool) *atomic.Bool {
var finalTemplates []*templates.Template
if !noCluster {
@ -38,15 +39,17 @@ func (e *Engine) ExecuteWithOpts(templatesList []*templates.Template, target Inp
}
wg.Add()
switch {
case template.SelfContained:
// Self Contained requests are executed here separately
e.executeSelfContainedTemplateWithInput(template, results)
default:
// All other request types are executed here
e.executeModelWithInput(templateType, template, target, results)
}
wg.Done()
go func(tpl *templates.Template) {
switch {
case tpl.SelfContained:
// Self Contained requests are executed here separately
e.executeSelfContainedTemplateWithInput(tpl, results)
default:
// All other request types are executed here
e.executeModelWithInput(templateType, tpl, target, results)
}
wg.Done()
}(template)
}
e.workPool.Wait()
return results
@ -71,9 +74,9 @@ func (e *Engine) executeModelWithInput(templateType types.ProtocolType, template
return
}
wg.Waitgroup.Add()
wg.WaitGroup.Add()
go func(value string) {
defer wg.Waitgroup.Done()
defer wg.WaitGroup.Done()
var match bool
var err error
@ -89,5 +92,5 @@ func (e *Engine) executeModelWithInput(templateType types.ProtocolType, template
results.CAS(false, match)
}(scannedValue)
})
wg.Waitgroup.Wait()
wg.WaitGroup.Wait()
}

1
v2/pkg/core/inputs/hybrid/hmap.go
View File

@ -9,6 +9,7 @@ import (
"strings"
"github.com/pkg/errors"
"github.com/projectdiscovery/filekv"
"github.com/projectdiscovery/fileutil"
"github.com/projectdiscovery/gologger"

5
v2/pkg/core/workflow_execute.go
View File

@ -1,11 +1,12 @@
package core
import (
"github.com/remeh/sizedwaitgroup"
"go.uber.org/atomic"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v2/pkg/output"
"github.com/projectdiscovery/nuclei/v2/pkg/workflows"
"github.com/remeh/sizedwaitgroup"
"go.uber.org/atomic"
)
// executeWorkflow runs a workflow on an input and returns true or false

17
v2/pkg/core/workpool.go
View File

@ -1,12 +1,13 @@
package core
import (
"github.com/projectdiscovery/nuclei/v2/pkg/templates/types"
"github.com/remeh/sizedwaitgroup"
"github.com/projectdiscovery/nuclei/v2/pkg/templates/types"
)
// WorkPool implements an execution pool for executing different
// types of task with different concurreny requirements.
// types of task with different concurrency requirements.
//
// It also allows Configuration of such requirements. This is used
// for per-module like separate headless concurrency etc.
@ -16,7 +17,7 @@ type WorkPool struct {
config WorkPoolConfig
}
// WorkPoolConfig is the configuration for workpool
// WorkPoolConfig is the configuration for work pool
type WorkPoolConfig struct {
// InputConcurrency is the concurrency for inputs values.
InputConcurrency int
@ -40,18 +41,18 @@ func NewWorkPool(config WorkPoolConfig) *WorkPool {
}
}
// Wait waits for all the workpool waitgroups to finish
// Wait waits for all the work pool wait groups to finish
func (w *WorkPool) Wait() {
w.Default.Wait()
w.Headless.Wait()
}
// InputWorkPool is a workpool per-input
// InputWorkPool is a work pool per-input
type InputWorkPool struct {
Waitgroup *sizedwaitgroup.SizedWaitGroup
WaitGroup *sizedwaitgroup.SizedWaitGroup
}
// InputPool returns a workpool for an input type
// InputPool returns a work pool for an input type
func (w *WorkPool) InputPool(templateType types.ProtocolType) *InputWorkPool {
var count int
if templateType == types.HeadlessProtocol {
@ -60,5 +61,5 @@ func (w *WorkPool) InputPool(templateType types.ProtocolType) *InputWorkPool {
count = w.config.InputConcurrency
}
swg := sizedwaitgroup.New(count)
return &InputWorkPool{Waitgroup: &swg}
return &InputWorkPool{WaitGroup: &swg}
}

7
v2/pkg/model/model.go
View File

@ -50,13 +50,6 @@ type Info struct {
Reference stringslice.StringSlice `json:"reference,omitempty" yaml:"reference,omitempty" jsonschema:"title=references for the template,description=Links relevant to the template"`
// description: |
// Severity of the template.
//
// values:
// - info
// - low
// - medium
// - high
// - critical
SeverityHolder severity.Holder `json:"severity,omitempty" yaml:"severity,omitempty"`
// description: |
// Metadata of the template.

1
v2/pkg/model/model_test.go
View File

@ -72,6 +72,7 @@ func TestUnmarshal(t *testing.T) {
}
assertUnmarshalledTemplateInfo := func(t *testing.T, yamlPayload string) Info {
t.Helper()
info := Info{}
err := yaml.Unmarshal([]byte(yamlPayload), &info)
assert.Nil(t, err)

2
v2/pkg/model/types/severity/severities.go
View File

@ -43,7 +43,7 @@ func (severities *Severities) UnmarshalYAML(unmarshal func(interface{}) error) e
}
func (severities Severities) String() string {
var stringSeverities []string
var stringSeverities = make([]string, 0, len(severities))
for _, severity := range severities {
stringSeverities = append(stringSeverities, severity.String())
}

50
v2/pkg/model/types/severity/severity.go
View File

@ -1,19 +1,28 @@
package severity
import (
"encoding/json"
"strings"
"github.com/alecthomas/jsonschema"
"github.com/pkg/errors"
)
type Severity int
// name:Severity
const (
// name:undefined
Undefined Severity = iota
// name:info
Info
// name:low
Low
// name:medium
Medium
// name:high
High
// name:critical
Critical
limit
)
@ -51,3 +60,44 @@ func normalizeValue(value string) string {
func (severity Severity) String() string {
return severityMappings[severity]
}
//nolint:exported,revive //prefer to be explicit about the name, and make it refactor-safe
// Holder holds a Severity type. Required for un/marshalling purposes
type Holder struct {
Severity Severity `mapping:"true"`
}
func (severityHolder Holder) JSONSchemaType() *jsonschema.Type {
gotType := &jsonschema.Type{
Type: "string",
Title: "severity of the template",
Description: "Seriousness of the implications of the template",
}
for _, severity := range GetSupportedSeverities() {
gotType.Enum = append(gotType.Enum, severity.String())
}
return gotType
}
func (severityHolder *Holder) UnmarshalYAML(unmarshal func(interface{}) error) error {
var marshalledSeverity string
if err := unmarshal(&marshalledSeverity); err != nil {
return err
}
computedSeverity, err := toSeverity(marshalledSeverity)
if err != nil {
return err
}
severityHolder.Severity = computedSeverity
return nil
}
func (severityHolder *Holder) MarshalJSON() ([]byte, error) {
return json.Marshal(severityHolder.Severity.String())
}
func (severityHolder Holder) MarshalYAML() (interface{}, error) {
return severityHolder.Severity.String(), nil
}

48
v2/pkg/model/types/severity/severity_holder.go
View File

@ -1,48 +0,0 @@
package severity
import (
"encoding/json"
"github.com/alecthomas/jsonschema"
)
//nolint:exported,revive //prefer to be explicit about the name, and make it refactor-safe
// Holder holds a Severity type. Required for un/marshalling purposes
type Holder struct {
Severity Severity
}
func (severityHolder Holder) JSONSchemaType() *jsonschema.Type {
gotType := &jsonschema.Type{
Type: "string",
Title: "severity of the template",
Description: "Seriousness of the implications of the template",
}
for _, severity := range GetSupportedSeverities() {
gotType.Enum = append(gotType.Enum, severity.String())
}
return gotType
}
func (severityHolder *Holder) UnmarshalYAML(unmarshal func(interface{}) error) error {
var marshalledSeverity string
if err := unmarshal(&marshalledSeverity); err != nil {
return err
}
computedSeverity, err := toSeverity(marshalledSeverity)
if err != nil {
return err
}
severityHolder.Severity = computedSeverity
return nil
}
func (severityHolder *Holder) MarshalJSON() ([]byte, error) {
return json.Marshal(severityHolder.Severity.String())
}
func (severityHolder Holder) MarshalYAML() (interface{}, error) {
return severityHolder.Severity.String(), nil
}

2
v2/pkg/model/types/severity/severity_test.go
View File

@ -30,6 +30,7 @@ func TestGetSupportedSeverities(t *testing.T) {
}
func testUnmarshal(t *testing.T, unmarshaller func(data []byte, v interface{}) error, payloadCreator func(value string) string) {
t.Helper()
payloads := [...]string{
payloadCreator("Info"),
payloadCreator("info"),
@ -48,6 +49,7 @@ func testUnmarshal(t *testing.T, unmarshaller func(data []byte, v interface{}) e
}
func testUnmarshalFail(t *testing.T, unmarshaller func(data []byte, v interface{}) error, payloadCreator func(value string) string) {
t.Helper()
assert.Panics(t, func() { unmarshal(payloadCreator("invalid"), unmarshaller) })
}

7
v2/pkg/model/types/stringslice/stringslice.go
View File

@ -80,11 +80,12 @@ func marshalStringToSlice(unmarshal func(interface{}) error) ([]string, error) {
}
var result []string
if len(marshalledValuesAsSlice) > 0 {
switch {
case len(marshalledValuesAsSlice) > 0:
result = marshalledValuesAsSlice
} else if utils.IsNotBlank(marshalledValueAsString) {
case utils.IsNotBlank(marshalledValueAsString):
result = strings.Split(marshalledValueAsString, ",")
} else {
default:
result = []string{}
}

3
v2/pkg/operators/extractors/extract.go
View File

@ -1,9 +1,8 @@
package extractors
import (
"strings"
"encoding/json"
"strings"
"github.com/antchfx/htmlquery"

24
v2/pkg/operators/extractors/extractor_types.go
View File

@ -11,16 +11,16 @@ import (
// ExtractorType is the type of the extractor specified
type ExtractorType int
// name:ExtractorType
const (
// RegexExtractor extracts responses with regexes
// name:regex
RegexExtractor ExtractorType = iota + 1
// KValExtractor extracts responses with key:value
// name:kval
KValExtractor
// XPathExtractor extracts responses with Xpath selectors
// name:xpath
XPathExtractor
// JSONExtractor extracts responses with json
// name:json
JSONExtractor
//limit
limit
)
@ -64,12 +64,12 @@ func (t ExtractorType) String() string {
return extractorMappings[t]
}
// TypeHolder is used to hold internal type of the extractor
type TypeHolder struct {
ExtractorType ExtractorType
// ExtractorTypeHolder is used to hold internal type of the extractor
type ExtractorTypeHolder struct {
ExtractorType ExtractorType `mapping:"true"`
}
func (holder TypeHolder) JSONSchemaType() *jsonschema.Type {
func (holder ExtractorTypeHolder) JSONSchemaType() *jsonschema.Type {
gotType := &jsonschema.Type{
Type: "string",
Title: "type of the extractor",
@ -81,7 +81,7 @@ func (holder TypeHolder) JSONSchemaType() *jsonschema.Type {
return gotType
}
func (holder *TypeHolder) UnmarshalYAML(unmarshal func(interface{}) error) error {
func (holder *ExtractorTypeHolder) UnmarshalYAML(unmarshal func(interface{}) error) error {
var marshalledTypes string
if err := unmarshal(&marshalledTypes); err != nil {
return err
@ -96,10 +96,10 @@ func (holder *TypeHolder) UnmarshalYAML(unmarshal func(interface{}) error) error
return nil
}
func (holder *TypeHolder) MarshalJSON() ([]byte, error) {
func (holder *ExtractorTypeHolder) MarshalJSON() ([]byte, error) {
return json.Marshal(holder.ExtractorType.String())
}
func (holder TypeHolder) MarshalYAML() (interface{}, error) {
func (holder ExtractorTypeHolder) MarshalYAML() (interface{}, error) {
return holder.ExtractorType.String(), nil
}

7
v2/pkg/operators/extractors/extractors.go
View File

@ -16,12 +16,7 @@ type Extractor struct {
Name string `yaml:"name,omitempty" jsonschema:"title=name of the extractor,description=Name of the extractor"`
// description: |
// Type is the type of the extractor.
// values:
// - "regex"
// - "kval"
// - "json"
// - "xpath"
Type TypeHolder `json:"name,omitempty" yaml:"type"`
Type ExtractorTypeHolder `json:"name,omitempty" yaml:"type"`
// extractorType is the internal type of the extractor
extractorType ExtractorType

48
v2/pkg/operators/matchers/compile.go
View File

@ -12,73 +12,73 @@ import (
)
// CompileMatchers performs the initial setup operation on a matcher
func (m *Matcher) CompileMatchers() error {
func (matcher *Matcher) CompileMatchers() error {
var ok bool
// Support hexadecimal encoding for matchers too.
if m.Encoding == "hex" {
for i, word := range m.Words {
if matcher.Encoding == "hex" {
for i, word := range matcher.Words {
if decoded, err := hex.DecodeString(word); err == nil && len(decoded) > 0 {
m.Words[i] = string(decoded)
matcher.Words[i] = string(decoded)
}
}
}
// Set up the matcher type
computedType, err := toMatcherTypes(m.GetType().String())
computedType, err := toMatcherTypes(matcher.GetType().String())
if err != nil {
return fmt.Errorf("unknown matcher type specified: %s", m.Type)
return fmt.Errorf("unknown matcher type specified: %s", matcher.Type)
}
m.matcherType = computedType
matcher.matcherType = computedType
// By default, match on body if user hasn't provided any specific items
if m.Part == "" {
m.Part = "body"
if matcher.Part == "" {
matcher.Part = "body"
}
// Compile the regexes
for _, regex := range m.Regex {
for _, regex := range matcher.Regex {
compiled, err := regexp.Compile(regex)
if err != nil {
return fmt.Errorf("could not compile regex: %s", regex)
}
m.regexCompiled = append(m.regexCompiled, compiled)
matcher.regexCompiled = append(matcher.regexCompiled, compiled)
}
// Compile and validate binary Values in matcher
for _, value := range m.Binary {
for _, value := range matcher.Binary {
if decoded, err := hex.DecodeString(value); err != nil {
return fmt.Errorf("could not hex decode binary: %s", value)
} else {
m.binaryDecoded = append(m.binaryDecoded, string(decoded))
matcher.binaryDecoded = append(matcher.binaryDecoded, string(decoded))
}
}
// Compile the dsl expressions
for _, dslExpression := range m.DSL {
for _, dslExpression := range matcher.DSL {
compiledExpression, err := govaluate.NewEvaluableExpressionWithFunctions(dslExpression, dsl.HelperFunctions())
if err != nil {
return &DslCompilationError{DslSignature: dslExpression, WrappedError: err}
}
m.dslCompiled = append(m.dslCompiled, compiledExpression)
matcher.dslCompiled = append(matcher.dslCompiled, compiledExpression)
}
// Set up the condition type, if any.
if m.Condition != "" {
m.condition, ok = ConditionTypes[m.Condition]
if matcher.Condition != "" {
matcher.condition, ok = ConditionTypes[matcher.Condition]
if !ok {
return fmt.Errorf("unknown condition specified: %s", m.Condition)
return fmt.Errorf("unknown condition specified: %s", matcher.Condition)
}
} else {
m.condition = ORCondition
matcher.condition = ORCondition
}
if m.CaseInsensitive {
if m.GetType() != WordsMatcher {
return fmt.Errorf("case-insensitive flag is supported only for 'word' matchers (not '%s')", m.Type)
if matcher.CaseInsensitive {
if matcher.GetType() != WordsMatcher {
return fmt.Errorf("case-insensitive flag is supported only for 'word' matchers (not '%s')", matcher.Type)
}
for i := range m.Words {
m.Words[i] = strings.ToLower(m.Words[i])
for i := range matcher.Words {
matcher.Words[i] = strings.ToLower(matcher.Words[i])
}
}
return nil

76
v2/pkg/operators/matchers/match.go
View File

@ -3,16 +3,19 @@ package matchers
import (
"strings"
"github.com/Knetic/govaluate"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v2/pkg/operators/common/dsl"
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/expressions"
)
// MatchStatusCode matches a status code check against a corpus
func (m *Matcher) MatchStatusCode(statusCode int) bool {
func (matcher *Matcher) MatchStatusCode(statusCode int) bool {
// Iterate over all the status codes accepted as valid
//
// Status codes don't support AND conditions.
for _, status := range m.Status {
for _, status := range matcher.Status {
// Continue if the status codes don't match
if statusCode != status {
continue
@ -24,11 +27,11 @@ func (m *Matcher) MatchStatusCode(statusCode int) bool {
}
// MatchSize matches a size check against a corpus
func (m *Matcher) MatchSize(length int) bool {
func (matcher *Matcher) MatchSize(length int) bool {
// Iterate over all the sizes accepted as valid
//
// Sizes codes don't support AND conditions.
for _, size := range m.Size {
for _, size := range matcher.Size {
// Continue if the size doesn't match
if length != size {
continue
@ -40,20 +43,20 @@ func (m *Matcher) MatchSize(length int) bool {
}
// MatchWords matches a word check against a corpus.
func (m *Matcher) MatchWords(corpus string, dynamicValues map[string]interface{}) (bool, []string) {
if m.CaseInsensitive {
func (matcher *Matcher) MatchWords(corpus string, data map[string]interface{}) (bool, []string) {
if matcher.CaseInsensitive {
corpus = strings.ToLower(corpus)
}
var matchedWords []string
// Iterate over all the words accepted as valid
for i, word := range m.Words {
if dynamicValues == nil {
dynamicValues = make(map[string]interface{})
for i, word := range matcher.Words {
if data == nil {
data = make(map[string]interface{})
}
var err error
word, err = expressions.Evaluate(word, dynamicValues)
word, err = expressions.Evaluate(word, data)
if err != nil {
gologger.Warning().Msgf("Error while evaluating word matcher: %q", word)
continue
@ -62,7 +65,7 @@ func (m *Matcher) MatchWords(corpus string, dynamicValues map[string]interface{}
if !strings.Contains(corpus, word) {
// If we are in an AND request and a match failed,
// return false as the AND condition fails on any single mismatch.
switch m.condition {
switch matcher.condition {
case ANDCondition:
return false, []string{}
case ORCondition:
@ -71,14 +74,14 @@ func (m *Matcher) MatchWords(corpus string, dynamicValues map[string]interface{}
}
// If the condition was an OR, return on the first match.
if m.condition == ORCondition {
if matcher.condition == ORCondition {
return true, []string{word}
}
matchedWords = append(matchedWords, word)
// If we are at the end of the words, return with true
if len(m.Words)-1 == i {
if len(matcher.Words)-1 == i {
return true, matchedWords
}
}
@ -86,15 +89,15 @@ func (m *Matcher) MatchWords(corpus string, dynamicValues map[string]interface{}
}
// MatchRegex matches a regex check against a corpus
func (m *Matcher) MatchRegex(corpus string) (bool, []string) {
func (matcher *Matcher) MatchRegex(corpus string) (bool, []string) {
var matchedRegexes []string
// Iterate over all the regexes accepted as valid
for i, regex := range m.regexCompiled {
for i, regex := range matcher.regexCompiled {
// Continue if the regex doesn't match
if !regex.MatchString(corpus) {
// If we are in an AND request and a match failed,
// return false as the AND condition fails on any single mismatch.
switch m.condition {
switch matcher.condition {
case ANDCondition:
return false, []string{}
case ORCondition:
@ -104,14 +107,14 @@ func (m *Matcher) MatchRegex(corpus string) (bool, []string) {
currentMatches := regex.FindAllString(corpus, -1)
// If the condition was an OR, return on the first match.
if m.condition == ORCondition {
if matcher.condition == ORCondition {
return true, currentMatches
}
matchedRegexes = append(matchedRegexes, currentMatches...)
// If we are at the end of the regex, return with true
if len(m.regexCompiled)-1 == i {
if len(matcher.regexCompiled)-1 == i {
return true, matchedRegexes
}
}
@ -119,14 +122,14 @@ func (m *Matcher) MatchRegex(corpus string) (bool, []string) {
}
// MatchBinary matches a binary check against a corpus
func (m *Matcher) MatchBinary(corpus string) (bool, []string) {
func (matcher *Matcher) MatchBinary(corpus string) (bool, []string) {
var matchedBinary []string
// Iterate over all the words accepted as valid
for i, binary := range m.binaryDecoded {
for i, binary := range matcher.binaryDecoded {
if !strings.Contains(corpus, binary) {
// If we are in an AND request and a match failed,
// return false as the AND condition fails on any single mismatch.
switch m.condition {
switch matcher.condition {
case ANDCondition:
return false, []string{}
case ORCondition:
@ -135,14 +138,14 @@ func (m *Matcher) MatchBinary(corpus string) (bool, []string) {
}
// If the condition was an OR, return on the first match.
if m.condition == ORCondition {
if matcher.condition == ORCondition {
return true, []string{binary}
}
matchedBinary = append(matchedBinary, binary)
// If we are at the end of the words, return with true
if len(m.Binary)-1 == i {
if len(matcher.Binary)-1 == i {
return true, matchedBinary
}
}
@ -150,9 +153,26 @@ func (m *Matcher) MatchBinary(corpus string) (bool, []string) {
}
// MatchDSL matches on a generic map result
func (m *Matcher) MatchDSL(data map[string]interface{}) bool {
func (matcher *Matcher) MatchDSL(data map[string]interface{}) bool {
logExpressionEvaluationFailure := func (matcherName string, err error) {
gologger.Warning().Msgf("Could not evaluate expression: %s, error: %s", matcherName, err.Error())
}
// Iterate over all the expressions accepted as valid
for i, expression := range m.dslCompiled {
for i, expression := range matcher.dslCompiled {
if varErr := expressions.ContainsUnresolvedVariables(expression.String()); varErr != nil {
resolvedExpression, err := expressions.Evaluate(expression.String(), data)
if err != nil {
logExpressionEvaluationFailure(matcher.Name, err)
return false
}
expression, err = govaluate.NewEvaluableExpressionWithFunctions(resolvedExpression, dsl.HelperFunctions())
if err != nil {
logExpressionEvaluationFailure(matcher.Name, err)
return false
}
}
result, err := expression.Evaluate(data)
if err != nil {
gologger.Warning().Msgf(err.Error())
@ -165,7 +185,7 @@ func (m *Matcher) MatchDSL(data map[string]interface{}) bool {
} else if !boolResult {
// If we are in an AND request and a match failed,
// return false as the AND condition fails on any single mismatch.
switch m.condition {
switch matcher.condition {
case ANDCondition:
return false
case ORCondition:
@ -174,12 +194,12 @@ func (m *Matcher) MatchDSL(data map[string]interface{}) bool {
}
// If the condition was an OR, return on the first match.
if m.condition == ORCondition {
if matcher.condition == ORCondition {
return true
}
// If we are at the end of the dsl, return with true
if len(m.dslCompiled)-1 == i {
if len(matcher.dslCompiled)-1 == i {
return true
}
}

18
v2/pkg/operators/matchers/match_test.go
View File

@ -3,6 +3,8 @@ package matchers
import (
"testing"
"github.com/Knetic/govaluate"
"github.com/projectdiscovery/nuclei/v2/pkg/operators/common/dsl"
"github.com/stretchr/testify/require"
)
@ -71,3 +73,19 @@ func TestHexEncoding(t *testing.T) {
require.True(t, isMatched, "Could not match valid Hex condition")
require.Equal(t, m.Words, matched)
}
func TestMatcher_MatchDSL(t *testing.T) {
compiled, err := govaluate.NewEvaluableExpressionWithFunctions("contains(body, \"{{VARIABLE}}\")", dsl.HelperFunctions())
require.Nil(t, err, "couldn't compile expression")
m := &Matcher{Type: MatcherTypeHolder{MatcherType: DSLMatcher}, dslCompiled: []*govaluate.EvaluableExpression{compiled}}
err = m.CompileMatchers()
require.Nil(t, err, "could not compile matcher")
values := []string{"PING", "pong"}
for value := range values {
isMatched := m.MatchDSL(map[string]interface{}{"body": value, "VARIABLE": value})
require.True(t, isMatched)
}
}

19
v2/pkg/operators/matchers/matchers.go
View File

@ -10,13 +10,6 @@ import (
type Matcher struct {
// description: |
// Type is the type of the matcher.
// values:
// - "status"
// - "size"
// - "word"
// - "regex"
// - "binary"
// - "dsl"
Type MatcherTypeHolder `yaml:"type" jsonschema:"title=type of matcher,description=Type of the matcher,enum=status,enum=size,enum=word,enum=regex,enum=binary,enum=dsl"`
// description: |
// Condition is the optional condition between two matcher variables. By default,
@ -62,7 +55,7 @@ type Matcher struct {
// description: |
// Words contains word patterns required to be present in the response part.
// examples:
// - name: Match for outlook mail protection domain
// - name: Match for Outlook mail protection domain
// value: >
// []string{"mail.protection.outlook.com"}
// - name: Match for application/json in response headers
@ -120,7 +113,6 @@ type Matcher struct {
dslCompiled []*govaluate.EvaluableExpression
}
// ConditionType is the type of condition for matcher
type ConditionType int
@ -138,18 +130,17 @@ var ConditionTypes = map[string]ConditionType{
}
// Result reverts the results of the match if the matcher is of type negative.
func (m *Matcher) Result(data bool) bool {
if m.Negative {
func (matcher *Matcher) Result(data bool) bool {
if matcher.Negative {
return !data
}
return data
}
// ResultWithMatchedSnippet returns true and the matched snippet, or false and an empty string
func (m *Matcher) ResultWithMatchedSnippet(data bool, matchedSnippet []string) (bool, []string) {
if m.Negative {
func (matcher *Matcher) ResultWithMatchedSnippet(data bool, matchedSnippet []string) (bool, []string) {
if matcher.Negative {
return !data, []string{}
}
return data, matchedSnippet
}

20
v2/pkg/operators/matchers/matchers_types.go
View File

@ -11,20 +11,20 @@ import (
// MatcherType is the type of the matcher specified
type MatcherType int
// name:MatcherType
const (
// WordsMatcher matches responses with words
// name:word
WordsMatcher MatcherType = iota + 1
// RegexMatcher matches responses with regexes
// name:regex
RegexMatcher
// BinaryMatcher matches responses with words
// name:binary
BinaryMatcher
// StatusMatcher matches responses with status codes
// name:status
StatusMatcher
// SizeMatcher matches responses with response size
// name:size
SizeMatcher
// DSLMatcher matches based upon dsl syntax
// name:dsl
DSLMatcher
//limit
limit
)
@ -39,8 +39,8 @@ var MatcherTypes = map[MatcherType]string{
}
//GetType returns the type of the matcher
func (m *Matcher) GetType() MatcherType {
return m.Type.MatcherType
func (matcher *Matcher) GetType() MatcherType {
return matcher.Type.MatcherType
}
// GetSupportedMatcherTypes returns list of supported types
@ -72,7 +72,7 @@ func (t MatcherType) String() string {
// MatcherTypeHolder is used to hold internal type of the matcher
type MatcherTypeHolder struct {
MatcherType MatcherType
MatcherType MatcherType `mapping:"true"`
}
func (t MatcherTypeHolder) String() string {

63
v2/pkg/operators/operators.go
View File

@ -72,11 +72,64 @@ type Result struct {
// OutputExtracts is the list of extracts to be displayed on screen.
OutputExtracts []string
// DynamicValues contains any dynamic values to be templated
DynamicValues map[string]interface{}
DynamicValues map[string][]string
// PayloadValues contains payload values provided by user. (Optional)
PayloadValues map[string]interface{}
}
// MakeDynamicValuesCallback takes an input dynamic values map and calls
// the callback function with all variations of the data in input in form
// of map[string]string (interface{}).
func MakeDynamicValuesCallback(input map[string][]string, iterateAllValues bool, callback func(map[string]interface{}) bool) {
output := make(map[string]interface{}, len(input))
if !iterateAllValues {
for k, v := range input {
if len(v) > 0 {
output[k] = v[0]
}
}
callback(output)
return
}
inputIndex := make(map[string]int, len(input))
var maxValue int
for _, v := range input {
if len(v) > maxValue {
maxValue = len(v)
}
}
for i := 0; i < maxValue; i++ {
for k, v := range input {
if len(v) == 0 {
continue
}
if len(v) == 1 {
output[k] = v[0]
continue
}
if gotIndex, ok := inputIndex[k]; !ok {
inputIndex[k] = 0
output[k] = v[0]
} else {
newIndex := gotIndex + 1
if newIndex >= len(v) {
output[k] = v[len(v)-1]
continue
}
output[k] = v[newIndex]
inputIndex[k] = newIndex
}
}
// skip if the callback says so
if callback(output) {
return
}
}
}
// Merge merges a result structure into the other.
func (r *Result) Merge(result *Result) {
if !r.Matched && result.Matched {
@ -115,7 +168,7 @@ func (operators *Operators) Execute(data map[string]interface{}, match MatchFunc
result := &Result{
Matches: make(map[string][]string),
Extracts: make(map[string][]string),
DynamicValues: make(map[string]interface{}),
DynamicValues: make(map[string][]string),
}
// Start with the extractors first and evaluate them.
@ -126,8 +179,10 @@ func (operators *Operators) Execute(data map[string]interface{}, match MatchFunc
extractorResults = append(extractorResults, match)
if extractor.Internal {
if _, ok := result.DynamicValues[extractor.Name]; !ok {
result.DynamicValues[extractor.Name] = match
if data, ok := result.DynamicValues[extractor.Name]; !ok {
result.DynamicValues[extractor.Name] = []string{match}
} else {
result.DynamicValues[extractor.Name] = append(data, match)
}
} else {
result.OutputExtracts = append(result.OutputExtracts, match)

57
v2/pkg/operators/operators_test.go Normal file
View File

@ -0,0 +1,57 @@
package operators
import (
"testing"
"github.com/stretchr/testify/require"
)
func TestMakeDynamicValuesCallback(t *testing.T) {
input := map[string][]string{
"a": []string{"1", "2"},
"b": []string{"3"},
"c": []string{},
"d": []string{"A", "B", "C"},
}
count := 0
MakeDynamicValuesCallback(input, true, func(data map[string]interface{}) bool {
count++
require.Len(t, data, 3, "could not get correct output length")
return false
})
require.Equal(t, 3, count, "could not get correct result count")
t.Run("all", func(t *testing.T) {
input := map[string][]string{
"a": []string{"1"},
"b": []string{"2"},
"c": []string{"3"},
}
count := 0
MakeDynamicValuesCallback(input, true, func(data map[string]interface{}) bool {
count++
require.Len(t, data, 3, "could not get correct output length")
return false
})
require.Equal(t, 1, count, "could not get correct result count")
})
t.Run("first", func(t *testing.T) {
input := map[string][]string{
"a": []string{"1", "2"},
"b": []string{"3"},
"c": []string{},
"d": []string{"A", "B", "C"},
}
count := 0
MakeDynamicValuesCallback(input, false, func(data map[string]interface{}) bool {
count++
require.Len(t, data, 3, "could not get correct output length")
return false
})
require.Equal(t, 1, count, "could not get correct result count")
})
}

6
v2/pkg/parsers/parser.go
View File

@ -108,8 +108,9 @@ var (
)
const (
SyntaxWarningStats = "syntax-warnings"
SyntaxErrorStats = "syntax-errors"
SyntaxWarningStats = "syntax-warnings"
SyntaxErrorStats = "syntax-errors"
RuntimeWarningsStats = "runtime-warnings"
)
func init() {
@ -118,6 +119,7 @@ func init() {
stats.NewEntry(SyntaxWarningStats, "Found %d templates with syntax warning (use -validate flag for further examination)")
stats.NewEntry(SyntaxErrorStats, "Found %d templates with syntax error (use -validate flag for further examination)")
stats.NewEntry(RuntimeWarningsStats, "Found %d templates with runtime error (use -validate flag for further examination)")
}
// ParseTemplate parses a template and returns a *templates.Template structure

3
v2/pkg/parsers/parser_test.go
View File

@ -5,11 +5,12 @@ import (
"fmt"
"testing"
"github.com/stretchr/testify/require"
"github.com/projectdiscovery/nuclei/v2/pkg/catalog/loader/filter"
"github.com/projectdiscovery/nuclei/v2/pkg/model"
"github.com/projectdiscovery/nuclei/v2/pkg/model/types/stringslice"
"github.com/projectdiscovery/nuclei/v2/pkg/templates"
"github.com/stretchr/testify/require"
)
func TestLoadTemplate(t *testing.T) {

28
v2/pkg/projectfile/httputil.go
View File

@ -79,23 +79,6 @@ func newInternalResponse() *InternalResponse {
}
}
// Unused
// func toInternalRequest(req *http.Request, target string, body []byte) *InternalRequest {
// intReq := newInternalRquest()
// intReq.Target = target
// intReq.HTTPMajor = req.ProtoMajor
// intReq.HTTPMinor = req.ProtoMinor
// for k, v := range req.Header {
// intReq.Headers[k] = v
// }
// intReq.Headers = req.Header
// intReq.Method = req.Method
// intReq.Body = body
// return intReq
// }
func toInternalResponse(resp *http.Response, body []byte) *InternalResponse {
intResp := newInternalResponse()
@ -125,14 +108,3 @@ func fromInternalResponse(intResp *InternalResponse) *http.Response {
Body: ioutil.NopCloser(bytes.NewReader(intResp.Body)),
}
}
// Unused
// func fromInternalRequest(intReq *InternalRequest) *http.Request {
// return &http.Request{
// ProtoMinor: intReq.HTTPMinor,
// ProtoMajor: intReq.HTTPMajor,
// Header: intReq.Headers,
// ContentLength: int64(len(intReq.Body)),
// Body: ioutil.NopCloser(bytes.NewReader(intReq.Body)),
// }
// }

16
v2/pkg/projectfile/project.go
View File

@ -42,13 +42,13 @@ func (pf *ProjectFile) Get(req []byte) (*http.Response, error) {
return nil, fmt.Errorf("not found")
}
var httprecord HTTPRecord
httprecord.Response = newInternalResponse()
if err := unmarshal(data, &httprecord); err != nil {
var httpRecord HTTPRecord
httpRecord.Response = newInternalResponse()
if err := unmarshal(data, &httpRecord); err != nil {
return nil, err
}
return fromInternalResponse(httprecord.Response), nil
return fromInternalResponse(httpRecord.Response), nil
}
func (pf *ProjectFile) Set(req []byte, resp *http.Response, data []byte) error {
@ -57,10 +57,10 @@ func (pf *ProjectFile) Set(req []byte, resp *http.Response, data []byte) error {
return err
}
var httprecord HTTPRecord
httprecord.Request = req
httprecord.Response = toInternalResponse(resp, data)
data, err = marshal(httprecord)
var httpRecord HTTPRecord
httpRecord.Request = req
httpRecord.Response = toInternalResponse(resp, data)
data, err = marshal(httpRecord)
if err != nil {
return err
}

10
v2/pkg/protocols/common/executer/executer.go
View File

@ -98,6 +98,10 @@ func (e *Executer) Execute(input string) (bool, error) {
}
gologger.Warning().Msgf("[%s] Could not execute request for %s: %s\n", e.options.TemplateID, input, err)
}
// If a match was found and stop at first match is set, break out of the loop and return
if results && (e.options.StopAtFirstMatch || e.options.Options.StopAtFirstMatch) {
break
}
}
return results, nil
}
@ -106,6 +110,7 @@ func (e *Executer) Execute(input string) (bool, error) {
func (e *Executer) ExecuteWithResults(input string, callback protocols.OutputEventCallback) error {
dynamicValues := make(map[string]interface{})
previous := make(map[string]interface{})
var results bool
for _, req := range e.requests {
req := req
@ -125,6 +130,7 @@ func (e *Executer) ExecuteWithResults(input string, callback protocols.OutputEve
if event.OperatorsResult == nil {
return
}
results = true
callback(event)
})
if err != nil {
@ -135,6 +141,10 @@ func (e *Executer) ExecuteWithResults(input string, callback protocols.OutputEve
}
gologger.Warning().Msgf("[%s] Could not execute request for %s: %s\n", e.options.TemplateID, input, err)
}
// If a match was found and stop at first match is set, break out of the loop and return
if results && (e.options.StopAtFirstMatch || e.options.Options.StopAtFirstMatch) {
break
}
}
return nil
}

3
v2/pkg/protocols/common/expressions/expressions.go
View File

@ -4,12 +4,13 @@ import (
"regexp"
"github.com/Knetic/govaluate"
"github.com/projectdiscovery/nuclei/v2/pkg/operators/common/dsl"
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/generators"
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/replacer"
)
var templateExpressionRegex = regexp.MustCompile(`(?m)\{\{[^}]+\}\}["'\)\}]*`)
var templateExpressionRegex = regexp.MustCompile(`(?m){{[^}]+}}["')}]*`)
// Evaluate checks if the match contains a dynamic variable, for each
// found one we will check if it's an expression and can

2
v2/pkg/protocols/common/expressions/variables.go
View File

@ -6,7 +6,7 @@ import (
"strings"
)
var unresolvedVariablesRegex = regexp.MustCompile(`(?:%7[B|b]|\{){2}([^}]+)(?:%7[D|d]|\}){2}["'\)\}]*`)
var unresolvedVariablesRegex = regexp.MustCompile(`(?:%7[B|b]|{){2}([^}]+)(?:%7[D|d]|}){2}["')}]*`)
// ContainsUnresolvedVariables returns an error with variable names if the passed
// input contains unresolved {{<pattern-here>}} variables.

15
v2/pkg/protocols/common/generators/attack_types.go
View File

@ -11,14 +11,15 @@ import (
// AttackType is the type of attack for payloads
type AttackType int
// Supported values for the ProtocolType
// Supported values for the AttackType
// name:AttackType
const (
// BatteringRamAttack replaces same payload into all of the defined payload positions at once.
// name:batteringram
BatteringRamAttack AttackType = iota + 1
// PitchForkAttack replaces variables with positional value from multiple wordlists
// name:pitchfork
PitchForkAttack
// ClusterbombAttack replaces variables with all possible combinations of values
ClusterbombAttack
// name:clusterbomb
ClusterBombAttack
limit
)
@ -26,7 +27,7 @@ const (
var attackTypeMappings = map[AttackType]string{
BatteringRamAttack: "batteringram",
PitchForkAttack: "pitchfork",
ClusterbombAttack: "clusterbomb",
ClusterBombAttack: "clusterbomb",
}
func GetSupportedAttackTypes() []AttackType {
@ -57,7 +58,7 @@ func (t AttackType) String() string {
// AttackTypeHolder is used to hold internal type of the protocol
type AttackTypeHolder struct {
Value AttackType
Value AttackType `mapping:"true"`
}
func (holder AttackTypeHolder) JSONSchemaType() *jsonschema.Type {

7
v2/pkg/protocols/common/generators/generators.go
View File

@ -4,6 +4,7 @@ package generators
import (
"github.com/pkg/errors"
"github.com/projectdiscovery/nuclei/v2/pkg/catalog"
)
@ -110,7 +111,7 @@ func (i *Iterator) Total() int {
count = len(p.values)
}
}
case ClusterbombAttack:
case ClusterBombAttack:
count = 1
for _, p := range i.payloads {
count *= len(p.values)
@ -126,7 +127,7 @@ func (i *Iterator) Value() (map[string]interface{}, bool) {
return i.batteringRamValue()
case PitchForkAttack:
return i.pitchforkValue()
case ClusterbombAttack:
case ClusterBombAttack:
return i.clusterbombValue()
default:
return i.batteringRamValue()
@ -183,7 +184,7 @@ func (i *Iterator) clusterbombValue() (map[string]interface{}, bool) {
signalNext = false
}
if !p.next() {
// No more inputs in this inputprovider
// No more inputs in this input provider
if index == i.msbIterator {
// Reset all previous wordlists and increment the msb counter
i.msbIterator++

5
v2/pkg/protocols/common/generators/generators_test.go
View File

@ -3,8 +3,9 @@ package generators
import (
"testing"
"github.com/projectdiscovery/nuclei/v2/pkg/catalog"
"github.com/stretchr/testify/require"
"github.com/projectdiscovery/nuclei/v2/pkg/catalog"
)
func TestBatteringRamGenerator(t *testing.T) {
@ -53,7 +54,7 @@ func TestClusterbombGenerator(t *testing.T) {
passwords := []string{"admin", "password", "token"}
catalogInstance := catalog.New("")
generator, err := New(map[string]interface{}{"username": usernames, "password": passwords}, ClusterbombAttack, "", catalogInstance)
generator, err := New(map[string]interface{}{"username": usernames, "password": passwords}, ClusterBombAttack, "", catalogInstance)
require.Nil(t, err, "could not create generator")
iterator := generator.NewIterator()

2
v2/pkg/protocols/common/generators/load.go
View File

@ -53,7 +53,7 @@ func loadPayloadsFromFile(filepath string) ([]string, error) {
}
lines = append(lines, text)
}
if err := scanner.Err(); err != nil && err != io.EOF {
if err := scanner.Err(); err != nil && !errors.Is(err, io.EOF) {
return lines, scanner.Err()
}
return lines, nil

40
v2/pkg/protocols/common/generators/maps.go
View File

@ -1,9 +1,49 @@
package generators
import (
"reflect"
"strings"
)
// MergeMapsMany merges many maps into a new map
func MergeMapsMany(maps ...interface{}) map[string][]string {
m := make(map[string][]string)
for _, gotMap := range maps {
val := reflect.ValueOf(gotMap)
if val.Kind() != reflect.Map {
continue
}
appendToSlice := func(key, value string) {
if values, ok := m[key]; !ok {
m[key] = []string{value}
} else {
m[key] = append(values, value)
}
}
for _, e := range val.MapKeys() {
v := val.MapIndex(e)
switch v.Kind() {
case reflect.Slice, reflect.Array:
for i := 0; i < v.Len(); i++ {
appendToSlice(e.String(), v.Index(i).String())
}
case reflect.String:
appendToSlice(e.String(), v.String())
case reflect.Interface:
switch data := v.Interface().(type) {
case string:
appendToSlice(e.String(), data)
case []string:
for _, value := range data {
appendToSlice(e.String(), value)
}
}
}
}
}
return m
}
// MergeMaps merges two maps into a new map
func MergeMaps(m1, m2 map[string]interface{}) map[string]interface{} {
m := make(map[string]interface{}, len(m1)+len(m2))

16
v2/pkg/protocols/common/generators/maps_test.go Normal file
View File

@ -0,0 +1,16 @@
package generators
import (
"testing"
"github.com/stretchr/testify/require"
)
func TestMergeMapsMany(t *testing.T) {
got := MergeMapsMany(map[string]interface{}{"a": []string{"1", "2"}, "c": "5"}, map[string][]string{"b": []string{"3", "4"}})
require.Equal(t, map[string][]string{
"a": []string{"1", "2"},
"b": []string{"3", "4"},
"c": []string{"5"},
}, got, "could not get correct merged map")
}

36
v2/pkg/protocols/common/generators/validate.go
View File

@ -7,40 +7,54 @@ import (
"path/filepath"
"strings"
"github.com/projectdiscovery/folderutil"
"github.com/projectdiscovery/nuclei/v2/pkg/types"
)
// validate validates the payloads if any.
func (g *PayloadGenerator) validate(payloads map[string]interface{}, templatePath string) error {
for name, payload := range payloads {
switch pt := payload.(type) {
switch payloadType := payload.(type) {
case string:
// check if it's a multiline string list
if len(strings.Split(pt, "\n")) != 1 {
if len(strings.Split(payloadType, "\n")) != 1 {
return errors.New("invalid number of lines in payload")
}
// check if it's a worldlist file and try to load it
if fileExists(pt) {
// check if it's a file and try to load it
if fileExists(payloadType) {
continue
}
changed := false
pathTokens := strings.Split(templatePath, string(os.PathSeparator))
for i := range pathTokens {
tpath := filepath.Join(filepath.Join(pathTokens[:i]...), pt)
if fileExists(tpath) {
payloads[name] = tpath
var dir string
if folderutil.IsWindowsOS() {
dir, payloadType = filepath.Split(filepath.Join(templatePath, payloadType))
} else {
dir, _ = filepath.Split(templatePath)
}
templatePathInfo, err := folderutil.NewPathInfo(dir)
if err != nil {
return err
}
payloadPathsToProbe, err := templatePathInfo.MeshWith(payloadType)
if err != nil {
return err
}
for _, payloadPath := range payloadPathsToProbe {
if fileExists(payloadPath) {
payloads[name] = payloadPath
changed = true
break
}
}
if !changed {
return fmt.Errorf("the %s file for payload %s does not exist or does not contain enough elements", pt, name)
return fmt.Errorf("the %s file for payload %s does not exist or does not contain enough elements", payloadType, name)
}
case interface{}:
loadedPayloads := types.ToStringSlice(pt)
loadedPayloads := types.ToStringSlice(payloadType)
if len(loadedPayloads) == 0 {
return fmt.Errorf("the payload %s does not contain enough elements", name)
}

10
v2/pkg/protocols/common/hosterrorscache/hosterrorscache.go
View File

@ -7,6 +7,7 @@ import (
"strings"
"github.com/bluele/gcache"
"github.com/projectdiscovery/gologger"
)
@ -16,7 +17,7 @@ import (
// It uses an LRU cache internally for skipping unresponsive hosts
// that remain so for a duration.
type Cache struct {
MaxHostError int
MaxHostError int
verbose bool
failedTargets gcache.Cache
}
@ -24,11 +25,11 @@ type Cache struct {
const DefaultMaxHostsCount = 10000
// New returns a new host max errors cache
func New(MaxHostError, maxHostsCount int) *Cache {
func New(maxHostError, maxHostsCount int) *Cache {
gc := gcache.New(maxHostsCount).
ARC().
Build()
return &Cache{failedTargets: gc, MaxHostError: MaxHostError}
return &Cache{failedTargets: gc, MaxHostError: maxHostError}
}
// SetVerbose sets the cache to log at verbose level
@ -46,7 +47,6 @@ func (c *Cache) normalizeCacheValue(value string) string {
finalValue := value
if strings.HasPrefix(value, "http") {
if parsed, err := url.Parse(value); err == nil {
hostname := parsed.Host
finalPort := parsed.Port()
if finalPort == "" {
@ -64,7 +64,7 @@ func (c *Cache) normalizeCacheValue(value string) string {
}
// ErrUnresponsiveHost is returned when a host is unresponsive
//var ErrUnresponsiveHost = errors.New("skipping as host is unresponsive")
// var ErrUnresponsiveHost = errors.New("skipping as host is unresponsive")
// Check returns true if a host should be skipped as it has been
// unresponsive for a certain number of times.

6
v2/pkg/protocols/common/hosterrorscache/hosterrorscache_test.go
View File

@ -11,15 +11,15 @@ func TestCacheCheckMarkFailed(t *testing.T) {
cache.MarkFailed("http://example.com:80")
if value, err := cache.failedTargets.Get("http://example.com:80"); err == nil && value != nil {
require.Equal(t, 1, value, "could not get correct markfailed")
require.Equal(t, 1, value, "could not get correct number of marked failed hosts")
}
cache.MarkFailed("example.com:80")
if value, err := cache.failedTargets.Get("example.com:80"); err == nil && value != nil {
require.Equal(t, 2, value, "could not get correct markfailed")
require.Equal(t, 2, value, "could not get correct number of marked failed hosts")
}
cache.MarkFailed("example.com")
if value, err := cache.failedTargets.Get("example.com"); err == nil && value != nil {
require.Equal(t, 1, value, "could not get correct markfailed")
require.Equal(t, 1, value, "could not get correct number of marked failed hosts")
}
for i := 0; i < 3; i++ {
cache.MarkFailed("test")

1
v2/pkg/protocols/common/protocolinit/init.go
View File

@ -2,6 +2,7 @@ package protocolinit
import (
"github.com/corpix/uarand"
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/protocolstate"
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/dns/dnsclientpool"
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/http/httpclientpool"

1
v2/pkg/protocols/common/protocolstate/state.go
View File

@ -2,6 +2,7 @@ package protocolstate
import (
"github.com/pkg/errors"
"github.com/projectdiscovery/fastdialer/fastdialer"
"github.com/projectdiscovery/nuclei/v2/pkg/types"
)

41
v2/pkg/protocols/dns/dns.go
View File

@ -34,16 +34,6 @@ type Request struct {
Name string `yaml:"name,omitempty" jsonschema:"title=hostname to make dns request for,description=Name is the Hostname to make DNS request for"`
// description: |
// RequestType is the type of DNS request to make.
// values:
// - "A"
// - "NS"
// - "DS"
// - "CNAME"
// - "SOA"
// - "PTR"
// - "MX"
// - "TXT"
// - "AAAA"
RequestType DNSRequestTypeHolder `yaml:"type,omitempty" jsonschema:"title=type of dns request to make,description=Type is the type of DNS request to make,enum=A,enum=NS,enum=DS,enum=CNAME,enum=SOA,enum=PTR,enum=MX,enum=TXT,enum=AAAA"`
// description: |
// Class is the class of the DNS request.
@ -83,11 +73,31 @@ type Request struct {
// description: |
// Recursion determines if resolver should recurse all records to get fresh results.
Recursion bool `yaml:"recursion,omitempty" jsonschema:"title=recurse all servers,description=Recursion determines if resolver should recurse all records to get fresh results"`
Recursion *bool `yaml:"recursion,omitempty" jsonschema:"title=recurse all servers,description=Recursion determines if resolver should recurse all records to get fresh results"`
// Resolvers to use for the dns requests
Resolvers []string `yaml:"resolvers,omitempty" jsonschema:"title=Resolvers,description=Define resolvers to use within the template"`
}
// RequestPartDefinitions contains a mapping of request part definitions and their
// description. Multiple definitions are separated by commas.
// Definitions not having a name (generated on runtime) are prefixed & suffixed by <>.
var RequestPartDefinitions = map[string]string{
"template-id": "ID of the template executed",
"template-info": "Info Block of the template executed",
"template-path": "Path of the template executed",
"host": "Host is the input to the template",
"matched": "Matched is the input which was matched upon",
"request": "Request contains the DNS request in text format",
"type": "Type is the type of request made",
"rcode": "Rcode field returned for the DNS request",
"question": "Question contains the DNS question field",
"extra": "Extra contains the DNS response extra field",
"answer": "Answer contains the DNS response answer field",
"ns": "NS contains the DNS response NS field",
"raw,body,all": "Raw contains the raw DNS response (default)",
"trace": "Trace contains trace data for DNS request if enabled",
}
func (request *Request) GetCompiledOperators() []*operators.Operators {
return []*operators.Operators{request.CompiledOperators}
}
@ -99,6 +109,13 @@ func (request *Request) GetID() string {
// Compile compiles the protocol request for further execution.
func (request *Request) Compile(options *protocols.ExecuterOptions) error {
if request.Retries == 0 {
request.Retries = 3
}
if request.Recursion == nil {
recursion := true
request.Recursion = &recursion
}
dnsClientOptions := &dnsclientpool.Configuration{
Retries: request.Retries,
}
@ -162,7 +179,7 @@ func (request *Request) Make(domain string) (*dns.Msg, error) {
// Build a request on the specified URL
req := new(dns.Msg)
req.Id = dns.Id()
req.RecursionDesired = request.Recursion
req.RecursionDesired = *request.Recursion
var q dns.Question

3
v2/pkg/protocols/dns/dns_test.go
View File

@ -24,6 +24,7 @@ func TestGenerateDNSVariables(t *testing.T) {
func TestDNSCompileMake(t *testing.T) {
options := testutils.DefaultOptions
recursion := false
testutils.Init(options)
const templateID = "testing-dns"
request := &Request{
@ -31,7 +32,7 @@ func TestDNSCompileMake(t *testing.T) {
Class: "INET",
Retries: 5,
ID: templateID,
Recursion: false,
Recursion: &recursion,
Name: "{{FQDN}}",
}
executerOpts := testutils.NewMockExecuterOptions(options, &testutils.TemplateInfo{

13
v2/pkg/protocols/dns/dns_types.go
View File

@ -11,17 +11,26 @@ import (
// DNSRequestType is the type of the method specified
type DNSRequestType int
// name:DNSRequestType
const (
// name:A
A DNSRequestType = iota + 1
// name:NS
NS
// name:DS
DS
// name:CNAME
CNAME
// name:SOA
SOA
// name:PTR
PTR
// name:MX
MX
// name:TXT
TXT
// name:AAAA
AAAA
//limit
limit
)
@ -67,7 +76,7 @@ func (t DNSRequestType) String() string {
// DNSRequestTypeHolder is used to hold internal type of the DNS type
type DNSRequestTypeHolder struct {
DNSRequestType DNSRequestType
DNSRequestType DNSRequestType `mapping:"true"`
}
func (holder DNSRequestTypeHolder) String() string {

2
v2/pkg/protocols/dns/dnsclientpool/clientpool.go
View File

@ -23,7 +23,7 @@ var defaultResolvers = []string{
"8.8.4.4:53", // Google
}
// Init initializes the clientpool implementation
// Init initializes the client pool implementation
func Init(options *types.Options) error {
// Don't create clients if already created in the past.
if normalClient != nil {

10
v2/pkg/protocols/dns/operators.go
View File

@ -76,7 +76,7 @@ func (request *Request) getMatchPart(part string, data output.InternalEvent) (in
}
// responseToDSLMap converts a DNS response to a map for use in DSL matching
func (request *Request) responseToDSLMap(req, resp *dns.Msg, host, matched string, tracedata *retryabledns.TraceData) output.InternalEvent {
func (request *Request) responseToDSLMap(req, resp *dns.Msg, host, matched string, traceData *retryabledns.TraceData) output.InternalEvent {
return output.InternalEvent{
"host": host,
"matched": matched,
@ -91,7 +91,7 @@ func (request *Request) responseToDSLMap(req, resp *dns.Msg, host, matched strin
"template-info": request.options.TemplateInfo,
"template-path": request.options.TemplatePath,
"type": request.Type().String(),
"trace": traceToString(tracedata, false),
"trace": traceToString(traceData, false),
}
}
@ -133,10 +133,10 @@ func questionToString(resourceRecords []dns.Question) string {
return buffer.String()
}
func traceToString(tracedata *retryabledns.TraceData, withSteps bool) string {
func traceToString(traceData *retryabledns.TraceData, withSteps bool) string {
buffer := &bytes.Buffer{}
if tracedata != nil {
for i, dnsRecord := range tracedata.DNSData {
if traceData != nil {
for i, dnsRecord := range traceData.DNSData {
if withSteps {
buffer.WriteString(fmt.Sprintf("request %d to resolver %s:\n", i, strings.Join(dnsRecord.Resolver, ",")))
}

18
v2/pkg/protocols/dns/operators_test.go
View File

@ -20,6 +20,7 @@ import (
func TestResponseToDSLMap(t *testing.T) {
options := testutils.DefaultOptions
recursion := false
testutils.Init(options)
templateID := "testing-dns"
request := &Request{
@ -27,7 +28,7 @@ func TestResponseToDSLMap(t *testing.T) {
Class: "INET",
Retries: 5,
ID: templateID,
Recursion: false,
Recursion: &recursion,
Name: "{{FQDN}}",
}
executerOpts := testutils.NewMockExecuterOptions(options, &testutils.TemplateInfo{
@ -52,6 +53,7 @@ func TestResponseToDSLMap(t *testing.T) {
func TestDNSOperatorMatch(t *testing.T) {
options := testutils.DefaultOptions
recursion := false
testutils.Init(options)
templateID := "testing-dns"
request := &Request{
@ -59,7 +61,7 @@ func TestDNSOperatorMatch(t *testing.T) {
Class: "INET",
Retries: 5,
ID: templateID,
Recursion: false,
Recursion: &recursion,
Name: "{{FQDN}}",
}
executerOpts := testutils.NewMockExecuterOptions(options, &testutils.TemplateInfo{
@ -163,6 +165,7 @@ func TestDNSOperatorMatch(t *testing.T) {
func TestDNSOperatorExtract(t *testing.T) {
options := testutils.DefaultOptions
recursion := false
testutils.Init(options)
templateID := "testing-dns"
request := &Request{
@ -170,7 +173,7 @@ func TestDNSOperatorExtract(t *testing.T) {
Class: "INET",
Retries: 5,
ID: templateID,
Recursion: false,
Recursion: &recursion,
Name: "{{FQDN}}",
}
executerOpts := testutils.NewMockExecuterOptions(options, &testutils.TemplateInfo{
@ -192,7 +195,7 @@ func TestDNSOperatorExtract(t *testing.T) {
t.Run("extract", func(t *testing.T) {
extractor := &extractors.Extractor{
Part: "raw",
Type: extractors.TypeHolder{ExtractorType: extractors.RegexExtractor},
Type: extractors.ExtractorTypeHolder{ExtractorType: extractors.RegexExtractor},
Regex: []string{"[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+"},
}
err = extractor.CompileExtractors()
@ -205,7 +208,7 @@ func TestDNSOperatorExtract(t *testing.T) {
t.Run("kval", func(t *testing.T) {
extractor := &extractors.Extractor{
Type: extractors.TypeHolder{ExtractorType: extractors.KValExtractor},
Type: extractors.ExtractorTypeHolder{ExtractorType: extractors.KValExtractor},
KVal: []string{"rcode"},
}
err = extractor.CompileExtractors()
@ -220,6 +223,7 @@ func TestDNSOperatorExtract(t *testing.T) {
func TestDNSMakeResult(t *testing.T) {
options := testutils.DefaultOptions
recursion := false
testutils.Init(options)
templateID := "testing-dns"
request := &Request{
@ -227,7 +231,7 @@ func TestDNSMakeResult(t *testing.T) {
Class: "INET",
Retries: 5,
ID: templateID,
Recursion: false,
Recursion: &recursion,
Name: "{{FQDN}}",
Operators: operators.Operators{
Matchers: []*matchers.Matcher{{
@ -238,7 +242,7 @@ func TestDNSMakeResult(t *testing.T) {
}},
Extractors: []*extractors.Extractor{{
Part: "raw",
Type: extractors.TypeHolder{ExtractorType: extractors.RegexExtractor},
Type: extractors.ExtractorTypeHolder{ExtractorType: extractors.RegexExtractor},
Regex: []string{"[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+"},
}},
},

18
v2/pkg/protocols/dns/request.go
View File

@ -12,8 +12,8 @@ import (
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/expressions"
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/helpers/eventcreator"
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/helpers/responsehighlighter"
"github.com/projectdiscovery/retryabledns"
templateTypes "github.com/projectdiscovery/nuclei/v2/pkg/templates/types"
"github.com/projectdiscovery/retryabledns"
)
var _ protocols.Request = &Request{}
@ -74,15 +74,15 @@ func (request *Request) ExecuteWithResults(input string, metadata /*TODO review
gologger.Verbose().Msgf("[%s] Sent DNS request to %s\n", request.options.TemplateID, domain)
// perform trace if necessary
var tracedata *retryabledns.TraceData
var traceData *retryabledns.TraceData
if request.Trace {
tracedata, err = request.dnsClient.Trace(domain, request.question, request.TraceMaxRecursion)
traceData, err = request.dnsClient.Trace(domain, request.question, request.TraceMaxRecursion)
if err != nil {
request.options.Output.Request(request.options.TemplatePath, domain, "dns", err)
}
}
outputEvent := request.responseToDSLMap(compiledRequest, response, input, input, tracedata)
outputEvent := request.responseToDSLMap(compiledRequest, response, input, input, traceData)
for k, v := range previous {
outputEvent[k] = v
}
@ -92,7 +92,7 @@ func (request *Request) ExecuteWithResults(input string, metadata /*TODO review
dumpResponse(event, request.options, response.String(), domain)
if request.Trace {
dumpTraceData(event, request.options, traceToString(tracedata, true), domain)
dumpTraceData(event, request.options, traceToString(traceData, true), domain)
}
callback(event)
@ -112,15 +112,15 @@ func dumpResponse(event *output.InternalWrappedEvent, requestOptions *protocols.
}
}
func dumpTraceData(event *output.InternalWrappedEvent, requestOptions *protocols.ExecuterOptions, tracedata, domain string) {
func dumpTraceData(event *output.InternalWrappedEvent, requestOptions *protocols.ExecuterOptions, traceData, domain string) {
cliOptions := requestOptions.Options
if cliOptions.Debug || cliOptions.DebugResponse {
hexDump := false
if responsehighlighter.HasBinaryContent(tracedata) {
if responsehighlighter.HasBinaryContent(traceData) {
hexDump = true
tracedata = hex.Dump([]byte(tracedata))
traceData = hex.Dump([]byte(traceData))
}
highlightedResponse := responsehighlighter.Highlight(event.OperatorsResult, tracedata, cliOptions.NoColor, hexDump)
highlightedResponse := responsehighlighter.Highlight(event.OperatorsResult, traceData, cliOptions.NoColor, hexDump)
gologger.Debug().Msgf("[%s] Dumped DNS Trace data for %s\n\n%s", requestOptions.TemplateID, domain, highlightedResponse)
}
}

5
v2/pkg/protocols/dns/request_test.go
View File

@ -17,6 +17,7 @@ import (
func TestDNSExecuteWithResults(t *testing.T) {
options := testutils.DefaultOptions
recursion := false
testutils.Init(options)
templateID := "testing-dns"
request := &Request{
@ -24,7 +25,7 @@ func TestDNSExecuteWithResults(t *testing.T) {
Class: "INET",
Retries: 5,
ID: templateID,
Recursion: false,
Recursion: &recursion,
Name: "{{FQDN}}",
Operators: operators.Operators{
Matchers: []*matchers.Matcher{{
@ -35,7 +36,7 @@ func TestDNSExecuteWithResults(t *testing.T) {
}},
Extractors: []*extractors.Extractor{{
Part: "raw",
Type: extractors.TypeHolder{ExtractorType: extractors.RegexExtractor},
Type: extractors.ExtractorTypeHolder{ExtractorType: extractors.RegexExtractor},
Regex: []string{"[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+"},
}},
},

13
v2/pkg/protocols/file/file.go
View File

@ -52,6 +52,19 @@ type Request struct {
allExtensions bool
}
// RequestPartDefinitions contains a mapping of request part definitions and their
// description. Multiple definitions are separated by commas.
// Definitions not having a name (generated on runtime) are prefixed & suffixed by <>.
var RequestPartDefinitions = map[string]string{
"template-id": "ID of the template executed",
"template-info": "Info Block of the template executed",
"template-path": "Path of the template executed",
"matched": "Matched is the input which was matched upon",
"path": "Path is the path of file on local filesystem",
"type": "Type is the type of request made",
"raw,body,all,data": "Raw contains the raw file contents",
}
// defaultDenylist is the default list of extensions to be denied
var defaultDenylist = []string{".3g2", ".3gp", ".7z", ".apk", ".arj", ".avi", ".axd", ".bmp", ".css", ".csv", ".deb", ".dll", ".doc", ".drv", ".eot", ".exe", ".flv", ".gif", ".gifv", ".gz", ".h264", ".ico", ".iso", ".jar", ".jpeg", ".jpg", ".lock", ".m4a", ".m4v", ".map", ".mkv", ".mov", ".mp3", ".mp4", ".mpeg", ".mpg", ".msi", ".ogg", ".ogm", ".ogv", ".otf", ".pdf", ".pkg", ".png", ".ppt", ".psd", ".rar", ".rm", ".rpm", ".svg", ".swf", ".sys", ".tar.gz", ".tar", ".tif", ".tiff", ".ttf", ".vob", ".wav", ".webm", ".wmv", ".woff", ".woff2", ".xcf", ".xls", ".xlsx", ".zip"}

2
v2/pkg/protocols/file/find_test.go
View File

@ -44,7 +44,7 @@ func TestFindInputPaths(t *testing.T) {
"test.js": "TEST",
}
for k, v := range files {
err = ioutil.WriteFile(filepath.Join(tempDir, k), []byte(v), 0777)
err = ioutil.WriteFile(filepath.Join(tempDir, k), []byte(v), os.ModePerm)
require.Nil(t, err, "could not write temporary file")
}
expected := []string{"config.yaml", "final.yaml", "test.js"}

6
v2/pkg/protocols/file/operators_test.go
View File

@ -154,7 +154,7 @@ func TestFileOperatorExtract(t *testing.T) {
t.Run("extract", func(t *testing.T) {
extractor := &extractors.Extractor{
Part: "raw",
Type: extractors.TypeHolder{ExtractorType: extractors.RegexExtractor},
Type: extractors.ExtractorTypeHolder{ExtractorType: extractors.RegexExtractor},
Regex: []string{"[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+"},
}
err = extractor.CompileExtractors()
@ -167,7 +167,7 @@ func TestFileOperatorExtract(t *testing.T) {
t.Run("kval", func(t *testing.T) {
extractor := &extractors.Extractor{
Type: extractors.TypeHolder{ExtractorType: extractors.KValExtractor},
Type: extractors.ExtractorTypeHolder{ExtractorType: extractors.KValExtractor},
KVal: []string{"raw"},
}
err = extractor.CompileExtractors()
@ -250,7 +250,7 @@ func testFileMakeResult(t *testing.T, matchers []*matchers.Matcher, matcherCondi
Matchers: matchers,
Extractors: []*extractors.Extractor{{
Part: "raw",
Type: extractors.TypeHolder{ExtractorType: extractors.RegexExtractor},
Type: extractors.ExtractorTypeHolder{ExtractorType: extractors.RegexExtractor},
Regex: []string{"[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+"},
}},
},

4
v2/pkg/protocols/file/request_test.go
View File

@ -37,7 +37,7 @@ func TestFileExecuteWithResults(t *testing.T) {
}},
Extractors: []*extractors.Extractor{{
Part: "raw",
Type: extractors.TypeHolder{ExtractorType: extractors.RegexExtractor},
Type: extractors.ExtractorTypeHolder{ExtractorType: extractors.RegexExtractor},
Regex: []string{"[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+"},
}},
},
@ -57,7 +57,7 @@ func TestFileExecuteWithResults(t *testing.T) {
"config.yaml": "TEST\r\n1.1.1.1\r\n",
}
for k, v := range files {
err = ioutil.WriteFile(filepath.Join(tempDir, k), []byte(v), 0777)
err = ioutil.WriteFile(filepath.Join(tempDir, k), []byte(v), os.ModePerm)
require.Nil(t, err, "could not write temporary file")
}

22
v2/pkg/protocols/headless/engine/action.go
View File

@ -26,28 +26,6 @@ type Action struct {
Description string `yaml:"description,omitempty" jsonschema:"title=description for headless action,description=Description of the headless action"`
// description: |
// Action is the type of the action to perform.
// values:
// - "navigate"
// - "script"
// - "click"
// - "rightclick"
// - "text"
// - "screenshot"
// - "time"
// - "select"
// - "files"
// - "waitload"
// - "getresource"
// - "extract"
// - "setmethod"
// - "addheader"
// - "setheader"
// - "deleteheader"
// - "setbody"
// - "waitevent"
// - "keyboard"
// - "debug"
// - "sleep"
ActionType ActionTypeHolder `yaml:"action" jsonschema:"title=action to perform,description=Type of actions to perform,enum=navigate,enum=script,enum=click,enum=rightclick,enum=text,enum=screenshot,enum=time,enum=select,enum=files,enum=waitload,enum=getresource,enum=extract,enum=setmethod,enum=addheader,enum=setheader,enum=deleteheader,enum=setbody,enum=waitevent,enum=keyboard,enum=debug,enum=sleep"`
}

26
v2/pkg/protocols/headless/engine/action_types.go
View File

@ -12,51 +12,73 @@ import (
type ActionType int8
// Types to be executed by the user.
// name:ActionType
const (
// ActionNavigate performs a navigation to the specified URL
// URL can include nuclei payload data such as URL, Hostname, etc.
// name:navigate
ActionNavigate ActionType = iota + 1
// ActionScript executes a JS snippet on the page.
// name:script
ActionScript
// ActionClick performs the left-click action on an Element.
// name:click
ActionClick
// ActionRightClick performs the right-click action on an Element.
// name:rightclick
ActionRightClick
// ActionTextInput performs an action for a text input
// name:text
ActionTextInput
// ActionScreenshot performs the screenshot action writing to a file.
// name:screenshot
ActionScreenshot
// ActionTimeInput performs an action on a time input.
// name:time
ActionTimeInput
// ActionSelectInput performs an action on a select input.
// name:select
ActionSelectInput
// ActionFilesInput performs an action on a file input.
// name:files
ActionFilesInput
// ActionWaitLoad waits for the page to stop loading.
// name:waitload
ActionWaitLoad
// ActionGetResource performs a get resource action on an element
// name:getresource
ActionGetResource
// ActionExtract performs an extraction on an element
// name:extract
ActionExtract
// ActionSetMethod sets the request method
// name:setmethod
ActionSetMethod
// ActionAddHeader adds a header to the request
// name:addheader
ActionAddHeader
// ActionSetHeader sets a header in the request
// name:setheader
ActionSetHeader
// ActionDeleteHeader deletes a header from the request
// name:deleteheader
ActionDeleteHeader
// ActionSetBody sets the value of the request body
// name:setbody
ActionSetBody
// ActionWaitEvent waits for a specific event.
// name:waitevent
ActionWaitEvent
// ActionKeyboard performs a keyboard action event on a page.
// name:keyboard
ActionKeyboard
// ActionDebug debug slows down headless and adds a sleep to each page.
// name:debug
ActionDebug
// ActionSleep executes a sleep for a specified duration
// name:sleep
ActionSleep
// ActionWaitVisible waits until an element appears.
// name:waitvisible
ActionWaitVisible
// limit
limit
@ -143,7 +165,7 @@ func (t ActionType) String() string {
// ActionTypeHolder is used to hold internal type of the action
type ActionTypeHolder struct {
ActionType ActionType
ActionType ActionType `mapping:"true"`
}
func (holder ActionTypeHolder) String() string {

23
v2/pkg/protocols/headless/engine/engine.go
View File

@ -5,6 +5,7 @@ import (
"io/ioutil"
"net/http"
"os"
"runtime"
"strings"
"github.com/corpix/uarand"
@ -21,7 +22,7 @@ import (
type Browser struct {
customAgent string
tempDir string
previouspids map[int32]struct{} // track already running pids
previousPIDs map[int32]struct{} // track already running PIDs
engine *rod.Browser
httpclient *http.Client
options *types.Options
@ -33,7 +34,7 @@ func New(options *types.Options) (*Browser, error) {
if err != nil {
return nil, errors.Wrap(err, "could not create temporary directory")
}
previouspids := findChromeProcesses()
previousPIDs := findChromeProcesses()
chromeLauncher := launcher.New().
Leakless(false).
@ -44,12 +45,15 @@ func New(options *types.Options) (*Browser, error) {
Set("disable-notifications", "true").
Set("hide-scrollbars", "true").
Set("window-size", fmt.Sprintf("%d,%d", 1080, 1920)).
Set("no-sandbox", "true").
Set("mute-audio", "true").
Set("incognito", "true").
Delete("use-mock-keychain").
UserDataDir(dataStore)
if MustDisableSandbox() {
chromeLauncher = chromeLauncher.NoSandbox(true)
}
if options.UseInstalledChrome {
if chromePath, hasChrome := launcher.LookPath(); hasChrome {
chromeLauncher.Bin(chromePath)
@ -89,7 +93,7 @@ func New(options *types.Options) (*Browser, error) {
customAgent = uarand.GetRandom()
}
httpclient, err := newhttpClient(options)
httpclient, err := newHttpClient(options)
if err != nil {
return nil, err
}
@ -101,10 +105,17 @@ func New(options *types.Options) (*Browser, error) {
httpclient: httpclient,
options: options,
}
engine.previouspids = previouspids
engine.previousPIDs = previousPIDs
return engine, nil
}
// MustDisableSandbox determines if the current os and user needs sandbox mode disabled
func MustDisableSandbox() bool {
// linux with root user needs "--no-sandbox" option
// https://github.com/chromium/chromium/blob/c4d3c31083a2e1481253ff2d24298a1dfe19c754/chrome/test/chromedriver/client/chromedriver.py#L209
return runtime.GOOS == "linux" && os.Geteuid() == 0
}
// Close closes the browser engine
func (b *Browser) Close() {
b.engine.Close()
@ -123,7 +134,7 @@ func (b *Browser) killChromeProcesses() {
continue
}
// skip chrome processes that were already running
if _, ok := b.previouspids[process.Pid]; ok {
if _, ok := b.previousPIDs[process.Pid]; ok {
continue
}
_ = process.Kill()

7
v2/pkg/protocols/headless/engine/http_client.go
View File

@ -12,13 +12,14 @@ import (
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/utils"
"golang.org/x/net/proxy"
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/protocolstate"
"github.com/projectdiscovery/nuclei/v2/pkg/types"
"golang.org/x/net/proxy"
)
// newhttpClient creates a new http client for headless communication with a timeout
func newhttpClient(options *types.Options) (*http.Client, error) {
// newHttpClient creates a new http client for headless communication with a timeout
func newHttpClient(options *types.Options) (*http.Client, error) {
dialer := protocolstate.Dialer
// Set the base TLS configuration definition

12
v2/pkg/protocols/headless/engine/page_actions.go
View File

@ -399,12 +399,12 @@ func (p *Page) SelectInputElement(act *Action, out map[string]string /*TODO revi
return errors.Wrap(err, "could not scroll into view")
}
selectedbool := false
selectedBool := false
if act.GetArg("selected") == "true" {
selectedbool = true
selectedBool = true
}
by := act.GetArg("selector")
if err := element.Select([]string{value}, selectedbool, selectorBy(by)); err != nil {
if err := element.Select([]string{value}, selectedBool, selectorBy(by)); err != nil {
return errors.Wrap(err, "could not select input")
}
return nil
@ -509,7 +509,7 @@ func (p *Page) WaitEvent(act *Action, out map[string]string /*TODO review unused
protoEvent := &protoEvent{event: event}
// Uses another instance in order to be able to chain the timeout only to the wait operation
pagec := p.page
pageCopy := p.page
timeout := act.GetArg("timeout")
if timeout != "" {
ts, err := strconv.Atoi(timeout)
@ -517,11 +517,11 @@ func (p *Page) WaitEvent(act *Action, out map[string]string /*TODO review unused
return errors.Wrap(err, "could not get timeout")
}
if ts > 0 {
pagec = p.page.Timeout(time.Duration(ts) * time.Second)
pageCopy = p.page.Timeout(time.Duration(ts) * time.Second)
}
}
// Just wait the event to happen
pagec.WaitEvent(protoEvent)()
pageCopy.WaitEvent(protoEvent)()
return nil
}

118
v2/pkg/protocols/headless/engine/page_actions_test.go
View File

@ -28,7 +28,7 @@ func TestActionNavigate(t *testing.T) {
</body>
</html>`
actions := []*Action{{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}}, {ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitLoad)}}}
actions := []*Action{{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}}, {ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}}}
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
require.Nil(t, err, "could not run page actions")
@ -50,9 +50,9 @@ func TestActionScript(t *testing.T) {
t.Run("run-and-results", func(t *testing.T) {
actions := []*Action{
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitLoad)}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionScript)}, Name: "test", Data: map[string]string{"code": "window.test"}},
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
{ActionType: ActionTypeHolder{ActionType: ActionScript}, Name: "test", Data: map[string]string{"code": "window.test"}},
}
testHeadlessSimpleResponse(t, response, actions, timeout, func(page *Page, err error, out map[string]string) {
@ -64,10 +64,10 @@ func TestActionScript(t *testing.T) {
t.Run("hook", func(t *testing.T) {
actions := []*Action{
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionScript)}, Data: map[string]string{"code": "window.test = 'some-data';", "hook": "true"}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitLoad)}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionScript)}, Name: "test", Data: map[string]string{"code": "window.test"}},
{ActionType: ActionTypeHolder{ActionType: ActionScript}, Data: map[string]string{"code": "window.test = 'some-data';", "hook": "true"}},
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
{ActionType: ActionTypeHolder{ActionType: ActionScript}, Name: "test", Data: map[string]string{"code": "window.test"}},
}
testHeadlessSimpleResponse(t, response, actions, timeout, func(page *Page, err error, out map[string]string) {
require.Nil(t, err, "could not run page actions")
@ -88,9 +88,9 @@ func TestActionClick(t *testing.T) {
</html>`
actions := []*Action{
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitLoad)}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionClick)}, Data: map[string]string{"selector": "button"}}, // Use css selector for clicking
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
{ActionType: ActionTypeHolder{ActionType: ActionClick}, Data: map[string]string{"selector": "button"}}, // Use css selector for clicking
}
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
@ -121,9 +121,9 @@ func TestActionRightClick(t *testing.T) {
</html>`
actions := []*Action{
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitLoad)}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionRightClick)}, Data: map[string]string{"selector": "button"}}, // Use css selector for clicking
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
{ActionType: ActionTypeHolder{ActionType: ActionRightClick}, Data: map[string]string{"selector": "button"}}, // Use css selector for clicking
}
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
@ -146,9 +146,9 @@ func TestActionTextInput(t *testing.T) {
</html>`
actions := []*Action{
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitLoad)}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionTextInput)}, Data: map[string]string{"selector": "input", "value": "test"}},
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
{ActionType: ActionTypeHolder{ActionType: ActionTextInput}, Data: map[string]string{"selector": "input", "value": "test"}},
}
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
@ -163,9 +163,9 @@ func TestActionTextInput(t *testing.T) {
func TestActionHeadersChange(t *testing.T) {
actions := []*Action{
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionSetHeader)}, Data: map[string]string{"part": "request", "key": "Test", "value": "Hello"}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitLoad)}},
{ActionType: ActionTypeHolder{ActionType: ActionSetHeader}, Data: map[string]string{"part": "request", "key": "Test", "value": "Hello"}},
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
}
handler := func(w http.ResponseWriter, r *http.Request) {
@ -190,9 +190,9 @@ func TestActionScreenshot(t *testing.T) {
</html>`
actions := []*Action{
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitLoad)}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionScreenshot)}, Data: map[string]string{"to": "test"}},
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
{ActionType: ActionTypeHolder{ActionType: ActionScreenshot}, Data: map[string]string{"to": "test"}},
}
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
@ -215,9 +215,9 @@ func TestActionTimeInput(t *testing.T) {
</html>`
actions := []*Action{
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitLoad)}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionTimeInput)}, Data: map[string]string{"selector": "input", "value": "2006-01-02T15:04:05Z"}},
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
{ActionType: ActionTypeHolder{ActionType: ActionTimeInput}, Data: map[string]string{"selector": "input", "value": "2006-01-02T15:04:05Z"}},
}
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
@ -243,9 +243,9 @@ func TestActionSelectInput(t *testing.T) {
</html>`
actions := []*Action{
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitLoad)}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionSelectInput)}, Data: map[string]string{"by": "x", "xpath": "//select[@id='test']", "value": "Test2", "selected": "true"}},
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
{ActionType: ActionTypeHolder{ActionType: ActionSelectInput}, Data: map[string]string{"by": "x", "xpath": "//select[@id='test']", "value": "Test2", "selected": "true"}},
}
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
@ -266,9 +266,9 @@ func TestActionFilesInput(t *testing.T) {
</html>`
actions := []*Action{
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitLoad)}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionFilesInput)}, Data: map[string]string{"selector": "input", "value": "test1.pdf"}},
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
{ActionType: ActionTypeHolder{ActionType: ActionFilesInput}, Data: map[string]string{"selector": "input", "value": "test1.pdf"}},
}
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
@ -292,8 +292,8 @@ func TestActionWaitLoad(t *testing.T) {
</html>`
actions := []*Action{
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitLoad)}},
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
}
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
@ -317,8 +317,8 @@ func TestActionGetResource(t *testing.T) {
</html>`
actions := []*Action{
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionGetResource)}, Data: map[string]string{"by": "x", "xpath": "//img[@id='test']"}, Name: "src"},
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionGetResource}, Data: map[string]string{"by": "x", "xpath": "//img[@id='test']"}, Name: "src"},
}
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
@ -337,8 +337,8 @@ func TestActionExtract(t *testing.T) {
</html>`
actions := []*Action{
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionExtract)}, Data: map[string]string{"by": "x", "xpath": "//button[@id='test']"}, Name: "extract"},
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionExtract}, Data: map[string]string{"by": "x", "xpath": "//button[@id='test']"}, Name: "extract"},
}
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
@ -356,8 +356,8 @@ func TestActionSetMethod(t *testing.T) {
</html>`
actions := []*Action{
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionSetMethod)}, Data: map[string]string{"part": "x", "method": "SET"}},
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionSetMethod}, Data: map[string]string{"part": "x", "method": "SET"}},
}
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
@ -368,9 +368,9 @@ func TestActionSetMethod(t *testing.T) {
func TestActionAddHeader(t *testing.T) {
actions := []*Action{
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionAddHeader)}, Data: map[string]string{"part": "request", "key": "Test", "value": "Hello"}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitLoad)}},
{ActionType: ActionTypeHolder{ActionType: ActionAddHeader}, Data: map[string]string{"part": "request", "key": "Test", "value": "Hello"}},
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
}
handler := func(w http.ResponseWriter, r *http.Request) {
@ -387,11 +387,11 @@ func TestActionAddHeader(t *testing.T) {
func TestActionDeleteHeader(t *testing.T) {
actions := []*Action{
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionAddHeader)}, Data: map[string]string{"part": "request", "key": "Test1", "value": "Hello"}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionAddHeader)}, Data: map[string]string{"part": "request", "key": "Test2", "value": "World"}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionDeleteHeader)}, Data: map[string]string{"part": "request", "key": "Test2"}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitLoad)}},
{ActionType: ActionTypeHolder{ActionType: ActionAddHeader}, Data: map[string]string{"part": "request", "key": "Test1", "value": "Hello"}},
{ActionType: ActionTypeHolder{ActionType: ActionAddHeader}, Data: map[string]string{"part": "request", "key": "Test2", "value": "World"}},
{ActionType: ActionTypeHolder{ActionType: ActionDeleteHeader}, Data: map[string]string{"part": "request", "key": "Test2"}},
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
}
handler := func(w http.ResponseWriter, r *http.Request) {
@ -408,9 +408,9 @@ func TestActionDeleteHeader(t *testing.T) {
func TestActionSetBody(t *testing.T) {
actions := []*Action{
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionSetBody)}, Data: map[string]string{"part": "request", "body": "hello"}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitLoad)}},
{ActionType: ActionTypeHolder{ActionType: ActionSetBody}, Data: map[string]string{"part": "request", "body": "hello"}},
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
}
handler := func(w http.ResponseWriter, r *http.Request) {
@ -436,10 +436,10 @@ func TestActionKeyboard(t *testing.T) {
</html>`
actions := []*Action{
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitLoad)}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionClick)}, Data: map[string]string{"selector": "input"}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionKeyboard)}, Data: map[string]string{"keys": "Test2"}},
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionWaitLoad}},
{ActionType: ActionTypeHolder{ActionType: ActionClick}, Data: map[string]string{"selector": "input"}},
{ActionType: ActionTypeHolder{ActionType: ActionKeyboard}, Data: map[string]string{"keys": "Test2"}},
}
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
@ -462,8 +462,8 @@ func TestActionSleep(t *testing.T) {
</html>`
actions := []*Action{
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionSleep)}, Data: map[string]string{"duration": "2"}},
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionSleep}, Data: map[string]string{"duration": "2"}},
}
testHeadlessSimpleResponse(t, response, actions, 20*time.Second, func(page *Page, err error, out map[string]string) {
@ -485,8 +485,8 @@ func TestActionWaitVisible(t *testing.T) {
</html>`
actions := []*Action{
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionNavigate)}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionType(ActionWaitVisible)}, Data: map[string]string{"by": "x", "xpath": "//button[@id='test']"}},
{ActionType: ActionTypeHolder{ActionType: ActionNavigate}, Data: map[string]string{"url": "{{BaseURL}}"}},
{ActionType: ActionTypeHolder{ActionType: ActionWaitVisible}, Data: map[string]string{"by": "x", "xpath": "//button[@id='test']"}},
}
t.Run("wait for an element being visible", func(t *testing.T) {

23
v2/pkg/protocols/headless/engine/rules.go
View File

@ -8,7 +8,7 @@ import (
// routingRuleHandler handles proxy rule for actions related to request/response modification
func (p *Page) routingRuleHandler(ctx *rod.Hijack) {
// usually browsers don't use chunked transfer encoding so we set the content-length nevertheless
// usually browsers don't use chunked transfer encoding, so we set the content-length nevertheless
ctx.Request.Req().ContentLength = int64(len(ctx.Request.Body()))
for _, rule := range p.rules {
@ -16,15 +16,16 @@ func (p *Page) routingRuleHandler(ctx *rod.Hijack) {
continue
}
if rule.Action == ActionSetMethod {
switch rule.Action {
case ActionSetMethod:
ctx.Request.Req().Method = rule.Args["method"]
} else if rule.Action == ActionAddHeader {
case ActionAddHeader:
ctx.Request.Req().Header.Add(rule.Args["key"], rule.Args["value"])
} else if rule.Action == ActionSetHeader {
case ActionSetHeader:
ctx.Request.Req().Header.Set(rule.Args["key"], rule.Args["value"])
} else if rule.Action == ActionDeleteHeader {
case ActionDeleteHeader:
ctx.Request.Req().Header.Del(rule.Args["key"])
} else if rule.Action == ActionSetBody {
case ActionSetBody:
body := rule.Args["body"]
ctx.Request.Req().ContentLength = int64(len(body))
ctx.Request.SetBody(body)
@ -36,13 +37,15 @@ func (p *Page) routingRuleHandler(ctx *rod.Hijack) {
if rule.Part != "response" {
continue
}
if rule.Action == ActionAddHeader {
switch rule.Action {
case ActionAddHeader:
ctx.Response.Headers().Add(rule.Args["key"], rule.Args["value"])
} else if rule.Action == ActionSetHeader {
case ActionSetHeader:
ctx.Response.Headers().Set(rule.Args["key"], rule.Args["value"])
} else if rule.Action == ActionDeleteHeader {
case ActionDeleteHeader:
ctx.Response.Headers().Del(rule.Args["key"])
} else if rule.Action == ActionSetBody {
case ActionSetBody:
body := rule.Args["body"]
ctx.Response.Headers().Set("Content-Length", fmt.Sprintf("%d", len(body)))
ctx.Response.SetBody(rule.Args["body"])

14
v2/pkg/protocols/headless/headless.go
View File

@ -25,6 +25,20 @@ type Request struct {
options *protocols.ExecuterOptions
}
// RequestPartDefinitions contains a mapping of request part definitions and their
// description. Multiple definitions are separated by commas.
// Definitions not having a name (generated on runtime) are prefixed & suffixed by <>.
var RequestPartDefinitions = map[string]string{
"template-id": "ID of the template executed",
"template-info": "Info Block of the template executed",
"template-path": "Path of the template executed",
"host": "Host is the input to the template",
"matched": "Matched is the input which was matched upon",
"type": "Type is the type of request made",
"req": "Headless request made from the client",
"resp,body,data": "Headless response recieved from client (default)",
}
// Step is a headless protocol request step.
type Step struct {
// Action is the headless action to execute for the script

26
v2/pkg/protocols/http/build_request.go
View File

@ -53,14 +53,9 @@ func (g *generatedRequest) URL() string {
// Make creates a http request for the provided input.
// It returns io.EOF as error when all the requests have been exhausted.
func (r *requestGenerator) Make(baseURL string, dynamicValues map[string]interface{}) (*generatedRequest, error) {
func (r *requestGenerator) Make(baseURL, data string, payloads, dynamicValues map[string]interface{}) (*generatedRequest, error) {
if r.request.SelfContained {
return r.makeSelfContainedRequest(dynamicValues)
}
// We get the next payload for the request.
data, payloads, ok := r.nextValue()
if !ok {
return nil, io.EOF
return r.makeSelfContainedRequest(data, payloads, dynamicValues)
}
ctx := context.Background()
@ -107,12 +102,7 @@ func (r *requestGenerator) Make(baseURL string, dynamicValues map[string]interfa
return r.makeHTTPRequestFromModel(ctx, data, values, payloads)
}
func (r *requestGenerator) makeSelfContainedRequest(dynamicValues map[string]interface{}) (*generatedRequest, error) {
// We get the next payload for the request.
data, payloads, ok := r.nextValue()
if !ok {
return nil, io.EOF
}
func (r *requestGenerator) makeSelfContainedRequest(data string, payloads, dynamicValues map[string]interface{}) (*generatedRequest, error) {
ctx := context.Background()
isRawRequest := r.request.isRaw()
@ -124,7 +114,7 @@ func (r *requestGenerator) makeSelfContainedRequest(dynamicValues map[string]int
reader := bufio.NewReader(strings.NewReader(data))
s, err := reader.ReadString('\n')
if err != nil {
return nil, fmt.Errorf("could not read request: %s", err)
return nil, fmt.Errorf("could not read request: %w", err)
}
parts := strings.Split(s, " ")
@ -133,7 +123,7 @@ func (r *requestGenerator) makeSelfContainedRequest(dynamicValues map[string]int
}
parsed, err := url.Parse(parts[1])
if err != nil {
return nil, fmt.Errorf("could not parse request URL: %s", err)
return nil, fmt.Errorf("could not parse request URL: %w", err)
}
values := generators.MergeMaps(
generators.MergeMaps(dynamicValues, generateVariables(parsed, false)),
@ -307,10 +297,12 @@ func (r *requestGenerator) fillRequest(req *http.Request, values map[string]inte
}
req.Body = ioutil.NopCloser(strings.NewReader(body))
}
setHeader(req, "User-Agent", uarand.GetRandom())
if !r.request.Unsafe {
setHeader(req, "User-Agent", uarand.GetRandom())
}
// Only set these headers on non-raw requests
if len(r.request.Raw) == 0 {
if len(r.request.Raw) == 0 && !r.request.Unsafe {
setHeader(req, "Accept", "*/*")
setHeader(req, "Accept-Language", "en")
}

33
v2/pkg/protocols/http/build_request_test.go
View File

@ -5,12 +5,13 @@ import (
"testing"
"time"
"github.com/stretchr/testify/require"
"github.com/projectdiscovery/nuclei/v2/pkg/model"
"github.com/projectdiscovery/nuclei/v2/pkg/model/types/severity"
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/generators"
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/interactsh"
"github.com/projectdiscovery/nuclei/v2/pkg/testutils"
"github.com/stretchr/testify/require"
)
func TestBaseURLWithTemplatePrefs(t *testing.T) {
@ -86,7 +87,8 @@ func TestMakeRequestFromModal(t *testing.T) {
require.Nil(t, err, "could not compile http request")
generator := request.newGenerator()
req, err := generator.Make("https://example.com", map[string]interface{}{})
inputData, payloads, _ := generator.nextValue()
req, err := generator.Make("https://example.com", inputData, payloads, map[string]interface{}{})
require.Nil(t, err, "could not make http request")
bodyBytes, _ := req.request.BodyBytes()
@ -113,12 +115,14 @@ func TestMakeRequestFromModalTrimSuffixSlash(t *testing.T) {
require.Nil(t, err, "could not compile http request")
generator := request.newGenerator()
req, err := generator.Make("https://example.com/test.php", map[string]interface{}{})
inputData, payloads, _ := generator.nextValue()
req, err := generator.Make("https://example.com/test.php", inputData, payloads, map[string]interface{}{})
require.Nil(t, err, "could not make http request")
require.Equal(t, "https://example.com/test.php?query=example", req.request.URL.String(), "could not get correct request path")
generator = request.newGenerator()
req, err = generator.Make("https://example.com/test/", map[string]interface{}{})
inputData, payloads, _ = generator.nextValue()
req, err = generator.Make("https://example.com/test/", inputData, payloads, map[string]interface{}{})
require.Nil(t, err, "could not make http request")
require.Equal(t, "https://example.com/test/?query=example", req.request.URL.String(), "could not get correct request path")
}
@ -135,7 +139,7 @@ func TestMakeRequestFromRawWithPayloads(t *testing.T) {
"username": []string{"admin"},
"password": []string{"admin", "guest", "password", "test", "12345", "123456"},
},
AttackType: generators.AttackTypeHolder{Value: generators.ClusterbombAttack},
AttackType: generators.AttackTypeHolder{Value: generators.ClusterBombAttack},
Raw: []string{`GET /manager/html HTTP/1.1
Host: {{Hostname}}
User-Agent: Nuclei - Open-source project (github.com/projectdiscovery/nuclei)
@ -151,12 +155,14 @@ Accept-Encoding: gzip`},
require.Nil(t, err, "could not compile http request")
generator := request.newGenerator()
req, err := generator.Make("https://example.com", map[string]interface{}{})
inputData, payloads, _ := generator.nextValue()
req, err := generator.Make("https://example.com", inputData, payloads, map[string]interface{}{})
require.Nil(t, err, "could not make http request")
authorization := req.request.Header.Get("Authorization")
require.Equal(t, "Basic admin:admin", authorization, "could not get correct authorization headers from raw")
req, err = generator.Make("https://example.com", map[string]interface{}{})
inputData, payloads, _ = generator.nextValue()
req, err = generator.Make("https://example.com", inputData, payloads, map[string]interface{}{})
require.Nil(t, err, "could not make http request")
authorization = req.request.Header.Get("Authorization")
require.Equal(t, "Basic admin:guest", authorization, "could not get correct authorization headers from raw")
@ -174,7 +180,7 @@ func TestMakeRequestFromRawPayloadExpressions(t *testing.T) {
"username": []string{"admin"},
"password": []string{"admin", "guest", "password", "test", "12345", "123456"},
},
AttackType: generators.AttackTypeHolder{Value: generators.ClusterbombAttack},
AttackType: generators.AttackTypeHolder{Value: generators.ClusterBombAttack},
Raw: []string{`GET /manager/html HTTP/1.1
Host: {{Hostname}}
User-Agent: Nuclei - Open-source project (github.com/projectdiscovery/nuclei)
@ -190,12 +196,14 @@ Accept-Encoding: gzip`},
require.Nil(t, err, "could not compile http request")
generator := request.newGenerator()
req, err := generator.Make("https://example.com", map[string]interface{}{})
inputData, payloads, _ := generator.nextValue()
req, err := generator.Make("https://example.com", inputData, payloads, map[string]interface{}{})
require.Nil(t, err, "could not make http request")
authorization := req.request.Header.Get("Authorization")
require.Equal(t, "Basic YWRtaW46YWRtaW4=", authorization, "could not get correct authorization headers from raw")
req, err = generator.Make("https://example.com", map[string]interface{}{})
inputData, payloads, _ = generator.nextValue()
req, err = generator.Make("https://example.com", inputData, payloads, map[string]interface{}{})
require.Nil(t, err, "could not make http request")
authorization = req.request.Header.Get("Authorization")
require.Equal(t, "Basic YWRtaW46Z3Vlc3Q=", authorization, "could not get correct authorization headers from raw")
@ -226,12 +234,13 @@ func TestMakeRequestFromModelUniqueInteractsh(t *testing.T) {
ServerURL: options.InteractshURL,
CacheSize: int64(options.InteractionsCacheSize),
Eviction: time.Duration(options.InteractionsEviction) * time.Second,
ColldownPeriod: time.Duration(options.InteractionsCooldownPeriod) * time.Second,
ColldownPeriod: time.Duration(options.InteractionsCoolDownPeriod) * time.Second,
PollDuration: time.Duration(options.InteractionsPollDuration) * time.Second,
})
require.Nil(t, err, "could not create interactsh client")
got, err := generator.Make("https://example.com", map[string]interface{}{})
inputData, payloads, _ := generator.nextValue()
got, err := generator.Make("https://example.com", inputData, payloads, map[string]interface{}{})
require.Nil(t, err, "could not make http request")
// check if all the interactsh markers are replaced with unique urls

43
v2/pkg/protocols/http/http.go
View File

@ -41,12 +41,12 @@ type Request struct {
// Name is the optional name of the request.
//
// If a name is specified, all the named request in a template can be matched upon
// in a combined manner allowing multirequest based matchers.
// in a combined manner allowing multi-request based matchers.
Name string `yaml:"name,omitempty" jsonschema:"title=name for the http request,description=Optional name for the HTTP Request"`
// description: |
// Attack is the type of payload combinations to perform.
//
// batteringram is same payload into all of the defined payload positions at once, pitchfork combines multiple payload sets and clusterbomb generates
// batteringram is inserts the same payload into all defined payload positions at once, pitchfork combines multiple payload sets and clusterbomb generates
// permutations and combinations for all payloads.
// values:
// - "batteringram"
@ -55,17 +55,6 @@ type Request struct {
AttackType generators.AttackTypeHolder `yaml:"attack,omitempty" jsonschema:"title=attack is the payload combination,description=Attack is the type of payload combinations to perform,enum=batteringram,enum=pitchfork,enum=clusterbomb"`
// description: |
// Method is the HTTP Request Method.
// values:
// - "GET"
// - "HEAD"
// - "POST"
// - "PUT"
// - "DELETE"
// - "CONNECT"
// - "OPTIONS"
// - "TRACE"
// - "PATCH"
// - "PURGE"
Method HTTPMethodTypeHolder `yaml:"method,omitempty" jsonschema:"title=method is the http request method,description=Method is the HTTP Request Method,enum=GET,enum=HEAD,enum=POST,enum=PUT,enum=DELETE,enum=CONNECT,enum=OPTIONS,enum=TRACE,enum=PATCH,enum=PURGE"`
// description: |
// Body is an optional parameter which contains HTTP Request body.
@ -134,10 +123,9 @@ type Request struct {
generator *generators.PayloadGenerator // optional, only enabled when using payloads
httpClient *retryablehttp.Client
rawhttpClient *rawhttp.Client
dynamicValues map[string]interface{}
// description: |
// SelfContained specifies if the request is self contained.
// SelfContained specifies if the request is self-contained.
SelfContained bool `yaml:"-" json:"-"`
// description: |
@ -176,6 +164,31 @@ type Request struct {
// description: |
// SkipVariablesCheck skips the check for unresolved variables in request
SkipVariablesCheck bool `yaml:"skip-variables-check,omitempty" jsonschema:"title=skip variable checks,description=Skips the check for unresolved variables in request"`
// description: |
// IterateAll iterates all the values extracted from internal extractors
IterateAll bool `yaml:"iterate-all,omitempty" jsonschema:"title=iterate all the values,description=Iterates all the values extracted from internal extractors"`
}
// RequestPartDefinitions contains a mapping of request part definitions and their
// description. Multiple definitions are separated by commas.
// Definitions not having a name (generated on runtime) are prefixed & suffixed by <>.
var RequestPartDefinitions = map[string]string{
"template-id": "ID of the template executed",
"template-info": "Info Block of the template executed",
"template-path": "Path of the template executed",
"host": "Host is the input to the template",
"matched": "Matched is the input which was matched upon",
"type": "Type is the type of request made",
"request": "HTTP request made from the client",
"response": "HTTP response recieved from server",
"status_code": "Status Code received from the Server",
"body": "HTTP response body received from server (default)",
"content_length": "HTTP Response content length",
"header,all_headers": "HTTP response headers",
"duration": "HTTP request time duration",
"all": "HTTP response body + headers",
"cookies_from_response": "HTTP response cookies in name:value format",
"headers_from_response": "HTTP response headers in name:value format",
}
// GetID returns the unique ID of the request if any.

14
v2/pkg/protocols/http/http_method_types.go
View File

@ -11,18 +11,28 @@ import (
// HTTPMethodType is the type of the method specified
type HTTPMethodType int
// name:HTTPMethodType
const (
// name:GET
HTTPGet HTTPMethodType = iota + 1
// name:GET
HTTPHead
// name:POST
HTTPPost
// name:PUT
HTTPPut
// name:DELETE
HTTPDelete
// name:CONNECT
HTTPConnect
// name:OPTIONS
HTTPOptions
// name:TRACE
HTTPTrace
// name:PATCH
HTTPPatch
// name:PURGE
HTTPPurge
//limit
limit
)
@ -69,7 +79,7 @@ func (t HTTPMethodType) String() string {
// HTTPMethodTypeHolder is used to hold internal type of the HTTP Method
type HTTPMethodTypeHolder struct {
MethodType HTTPMethodType
MethodType HTTPMethodType `mapping:"true"`
}
func (holder HTTPMethodTypeHolder) String() string {

2
v2/pkg/protocols/http/http_test.go
View File

@ -23,7 +23,7 @@ func TestHTTPCompile(t *testing.T) {
"username": []string{"admin"},
"password": []string{"admin", "guest", "password", "test", "12345", "123456"},
},
AttackType: generators.AttackTypeHolder{Value: generators.ClusterbombAttack},
AttackType: generators.AttackTypeHolder{Value: generators.ClusterBombAttack},
Raw: []string{`GET /manager/html HTTP/1.1
Host: {{Hostname}}
User-Agent: Nuclei - Open-source project (github.com/projectdiscovery/nuclei)

Some files were not shown because too many files have changed in this diff Show More