mirror of https://github.com/daffainfo/nuclei.git
bauthard
GitHub
commit
d0525e4542
121
README.md
121
README.md
|
|
@ -11,39 +11,40 @@
|
|||
[](https://hub.docker.com/r/projectdiscovery/nuclei)
|
||||
[](https://discord.gg/KECAGdH)
|
||||
|
||||
Nuclei is a fast tool for configurable targeted scanning based on templates offering massive extensibility and ease of use.
|
||||
Nuclei is a fast tool for configurable targeted scanning based on templates offering massive extensibility and ease of use.
|
||||
|
||||
Nuclei is used to send requests across targets based on a template leading to zero false positives and providing effective scanning for known paths. Main use cases for nuclei are during initial reconnaissance phase to quickly check for low hanging fruits or CVEs across targets that are known and easily detectable. It uses [retryablehttp-go library](https://github.com/projectdiscovery/retryablehttp-go) designed to handle various errors and retries in case of blocking by WAFs, this is also one of our core modules from custom-queries.
|
||||
|
||||
We have also [open-sourced a template repository](https://github.com/projectdiscovery/nuclei-templates) to maintain various type of templates, we hope that you will contribute there too. Templates are provided in hopes that these will be useful and will allow everyone to build their own templates for the scanner. Checkout the templating guide at [**nuclei.projectdiscovery.io**](https://nuclei.projectdiscovery.io/templating-guide/) for a primer on nuclei templates.
|
||||
|
||||
# Resources
|
||||
- [Resources](#resources)
|
||||
- [Features](#features)
|
||||
- [Usage](#usage)
|
||||
- [Installation Instructions](#installation-instructions)
|
||||
- [From Binary](#from-binary)
|
||||
- [From Source](#from-source)
|
||||
- [From Github](#from-github)
|
||||
- [Nuclei templates](#nuclei-templates)
|
||||
- [Running nuclei](#running-nuclei)
|
||||
- [1. Running nuclei with a single template.](#1-running-nuclei-with-a-single-template)
|
||||
- [2. Running nuclei with multiple templates.](#2-running-nuclei-with-multiple-templates)
|
||||
- [3. Automating nuclei with subfinder and any other similar tool.](#3-automating-nuclei-with-subfinder-and-any-other-similar-tool)
|
||||
- [4. Running nuclei in a Docker](#running-in-a-docker-container)
|
||||
- [Thanks](#thanks)
|
||||
|
||||
# Features
|
||||
- [Resources](#resources)
|
||||
- [Features](#features)
|
||||
- [Usage](#usage)
|
||||
- [Installation Instructions](#installation-instructions)
|
||||
- [From Binary](#from-binary)
|
||||
- [From Source](#from-source)
|
||||
- [From Github](#from-github)
|
||||
- [Nuclei templates](#nuclei-templates)
|
||||
- [Running nuclei](#running-nuclei)
|
||||
- [1. Running nuclei with a single template.](#1-running-nuclei-with-a-single-template)
|
||||
- [2. Running nuclei with multiple templates.](#2-running-nuclei-with-multiple-templates)
|
||||
- [3. Automating nuclei with subfinder and any other similar tool.](#3-automating-nuclei-with-subfinder-and-any-other-similar-tool)
|
||||
- [4. Running nuclei in a Docker](#running-in-a-docker-container)
|
||||
- [Thanks](#thanks)
|
||||
|
||||
# Features
|
||||
|
||||
<h1 align="left">
|
||||
<img src="static/nuclei-run.png" alt="nuclei" width="700px"></a>
|
||||
<br>
|
||||
</h1>
|
||||
|
||||
- Simple and modular code base making it easy to contribute.
|
||||
- Fast And fully configurable using a template based engine.
|
||||
- Handles edge cases doing retries, backoffs etc for handling WAFs.
|
||||
- Smart matching functionality for zero false positive scanning.
|
||||
- Simple and modular code base making it easy to contribute.
|
||||
- Fast And fully configurable using a template based engine.
|
||||
- Handles edge cases doing retries, backoffs etc for handling WAFs.
|
||||
- Smart matching functionality for zero false positive scanning.
|
||||
|
||||
# Usage
|
||||
|
||||
|
|
@ -53,34 +54,33 @@ nuclei -h
|
|||
|
||||
This will display help for the tool. Here are all the switches it supports.
|
||||
|
||||
| Flag | Description | Example |
|
||||
|:-------------------:|:-------------------------------------------------------:|:----------------------------------------------------:|
|
||||
| -c | Number of concurrent requests (default 10) | nuclei -c 100 |
|
||||
| -l | List of urls to run templates | nuclei -l urls.txt |
|
||||
| -target | Target to scan using templates | nuclei -target hxxps://example.com |
|
||||
| -t | Templates input file/files to check across hosts | nuclei -t git-core.yaml |
|
||||
| -t | Templates input file/files to check across hosts | nuclei -t nuclei-templates/cves/ |
|
||||
| -nC | Don't Use colors in output | nuclei -nC |
|
||||
| -json | Prints and write output in json format | nuclei -json |
|
||||
| -json-requests | Write requests/responses for matches in JSON output | nuclei -json -json-requests |
|
||||
| -o | File to save output result (optional) | nuclei -o output.txt |
|
||||
| -pbar | Enable the progress bar (optional) | nuclei -pbar |
|
||||
| -silent | Show only found results in output | nuclei -silent |
|
||||
| -retries | Number of times to retry a failed request (default 1) | nuclei -retries 1 |
|
||||
| -timeout | Seconds to wait before timeout (default 5) | nuclei -timeout 5 |
|
||||
| -debug | Allow debugging of request/responses. | nuclei -debug |
|
||||
| -update-templates | Download and updates nuclei templates | nuclei -update-templates |
|
||||
| -update-directory | Directory for storing nuclei-templates(optional) | nuclei -update-directory templates |
|
||||
| -v | Shows verbose output of all sent requests | nuclei -v |
|
||||
| -version | Show version of nuclei | nuclei -version |
|
||||
| -proxy-url | Proxy URL | nuclei -proxy-url hxxp://127.0.0.1:8080 |
|
||||
| -proxy-socks-url | Socks proxy URL | nuclei -proxy-socks-url socks5://127.0.0.1:8080 |
|
||||
| -H | Custom Header | nuclei -H "x-bug-bounty: hacker" |
|
||||
|
||||
| Flag | Description | Example |
|
||||
| :---------------: | :---------------------------------------------------: | :---------------------------------------------: |
|
||||
| -c | Number of concurrent requests (default 10) | nuclei -c 100 |
|
||||
| -l | List of urls to run templates | nuclei -l urls.txt |
|
||||
| -target | Target to scan using templates | nuclei -target hxxps://example.com |
|
||||
| -t | Templates input file/files to check across hosts | nuclei -t git-core.yaml |
|
||||
| -t | Templates input file/files to check across hosts | nuclei -t nuclei-templates/cves/ |
|
||||
| -nC | Don't Use colors in output | nuclei -nC |
|
||||
| -json | Prints and write output in json format | nuclei -json |
|
||||
| -json-requests | Write requests/responses for matches in JSON output | nuclei -json -json-requests |
|
||||
| -o | File to save output result (optional) | nuclei -o output.txt |
|
||||
| -pbar | Enable the progress bar (optional) | nuclei -pbar |
|
||||
| -silent | Show only found results in output | nuclei -silent |
|
||||
| -retries | Number of times to retry a failed request (default 1) | nuclei -retries 1 |
|
||||
| -timeout | Seconds to wait before timeout (default 5) | nuclei -timeout 5 |
|
||||
| -debug | Allow debugging of request/responses. | nuclei -debug |
|
||||
| -update-templates | Download and updates nuclei templates | nuclei -update-templates |
|
||||
| -update-directory | Directory for storing nuclei-templates(optional) | nuclei -update-directory templates |
|
||||
| -lt | List available templates | nuclei -lt |
|
||||
| -v | Shows verbose output of all sent requests | nuclei -v |
|
||||
| -version | Show version of nuclei | nuclei -version |
|
||||
| -proxy-url | Proxy URL | nuclei -proxy-url hxxp://127.0.0.1:8080 |
|
||||
| -proxy-socks-url | Socks proxy URL | nuclei -proxy-socks-url socks5://127.0.0.1:8080 |
|
||||
| -H | Custom Header | nuclei -H "x-bug-bounty: hacker" |
|
||||
|
||||
# Installation Instructions
|
||||
|
||||
|
||||
### From Binary
|
||||
|
||||
The installation is easy. You can download the pre-built binaries for your platform from the [Releases](https://github.com/projectdiscovery/nuclei/releases/) page. Extract them using tar, move it to your `$PATH`and you're ready to go.
|
||||
|
|
@ -94,7 +94,7 @@ nuclei -h
|
|||
|
||||
### From Source
|
||||
|
||||
nuclei requires **go1.14+** to install successfully. Run the following command to get the repo -
|
||||
nuclei requires **go1.14+** to install successfully. Run the following command to get the repo -
|
||||
|
||||
```bash
|
||||
> GO111MODULE=on go get -u -v github.com/projectdiscovery/nuclei/v2/cmd/nuclei
|
||||
|
|
@ -102,7 +102,6 @@ nuclei requires **go1.14+** to install successfully. Run the following command t
|
|||
|
||||
In order to update the tool, you can use -u flag with `go get` command.
|
||||
|
||||
|
||||
### From Github
|
||||
|
||||
```bash
|
||||
|
|
@ -115,7 +114,7 @@ nuclei -h
|
|||
|
||||
# Nuclei templates
|
||||
|
||||
You can download or update the nuclei templates using `update-templates` flag.
|
||||
You can download or update the nuclei templates using `update-templates` flag.
|
||||
|
||||
```bash
|
||||
> nuclei -update-templates
|
||||
|
|
@ -131,9 +130,9 @@ or download it from [nuclei templates](https://github.com/projectdiscovery/nucle
|
|||
|
||||
# Running nuclei
|
||||
|
||||
### 1. Running nuclei with a single template.
|
||||
### 1. Running nuclei with a single template.
|
||||
|
||||
This will run the tool against all the hosts in `urls.txt` and returns the matched results.
|
||||
This will run the tool against all the hosts in `urls.txt` and returns the matched results.
|
||||
|
||||
```bash
|
||||
> nuclei -l urls.txt -t files/git-core.yaml -o results.txt
|
||||
|
|
@ -141,23 +140,22 @@ This will run the tool against all the hosts in `urls.txt` and returns the match
|
|||
|
||||
You can also pass the list of hosts at standard input (STDIN). This allows for easy integration in automation pipelines.
|
||||
|
||||
This will run the tool against all the hosts in `urls.txt` and returns the matched results.
|
||||
This will run the tool against all the hosts in `urls.txt` and returns the matched results.
|
||||
|
||||
```bash
|
||||
> cat urls.txt | nuclei -t files/git-core.yaml -o results.txt
|
||||
```
|
||||
|
||||
### 2. Running nuclei with multiple templates.
|
||||
### 2. Running nuclei with multiple templates.
|
||||
|
||||
This will run the tool against all the hosts in `urls.txt` with all the templates in the `path-to-templates` directory and returns the matched results.
|
||||
This will run the tool against all the hosts in `urls.txt` with all the templates in the `path-to-templates` directory and returns the matched results.
|
||||
|
||||
```bash
|
||||
> nuclei -l urls.txt -t cves/ -o results.txt
|
||||
> nuclei -l urls.txt -t cves/ -o results.txt
|
||||
```
|
||||
|
||||
### 3. Automating nuclei with subfinder and any other similar tool.
|
||||
|
||||
|
||||
```bash
|
||||
> subfinder -d hackerone.com -silent | httpx -silent | nuclei -t cves/ -o results.txt
|
||||
```
|
||||
|
|
@ -170,25 +168,26 @@ You can use the [nuclei dockerhub image](https://hub.docker.com/r/projectdiscove
|
|||
> docker pull projectdiscovery/nuclei
|
||||
```
|
||||
|
||||
- After downloading or building the container, run the following:
|
||||
- After downloading or building the container, run the following:
|
||||
|
||||
```bash
|
||||
> docker run -it projectdiscovery/nuclei
|
||||
```
|
||||
|
||||
For example, this will run the tool against all the hosts in `urls.txt` and output the results to your host file system:
|
||||
|
||||
```bash
|
||||
> cat urls.txt | docker run -v /path-to-nuclei-templates:/go/src/app/ -i projectdiscovery/nuclei -t ./files/git-config.yaml > results.txt
|
||||
```
|
||||
|
||||
Remember to change `/path-to-nuclei-templates` to the real path on your host file system.
|
||||
|
||||
|
||||
-------
|
||||
|
||||
* * *
|
||||
|
||||
# Thanks
|
||||
|
||||
nuclei is made with 🖤 by the [projectdiscovery](https://projectdiscovery.io) team. Community contributions have made the project what it is. See the **[Thanks.md](https://github.com/projectdiscovery/nuclei/blob/master/THANKS.md)** file for more details.
|
||||
nuclei is made with 🖤 by the [projectdiscovery](https://projectdiscovery.io) team. Community contributions have made the project what it is. See the **[Thanks.md](https://github.com/projectdiscovery/nuclei/blob/master/THANKS.md)** file for more details.
|
||||
|
||||
Do also check out these similar awesome projects that may fit in your workflow:
|
||||
|
||||
- [Burp Suite](https://portswigger.net/burp), [FFuF](https://github.com/ffuf/ffuf), [Jaeles](https://github.com/jaeles-project/jaeles), [Qsfuzz](https://github.com/ameenmaali/qsfuzz), [Inception](https://github.com/proabiral/inception), [Snallygaster](https://github.com/hannob/snallygaster), [Gofingerprint](https://github.com/Static-Flow/gofingerprint), [Sn1per](https://github.com/1N3/Sn1per/tree/master/templates), [Google tsunami](https://github.com/google/tsunami-security-scanner), [ChopChop](https://github.com/michelin/ChopChop)
|
||||
- [Burp Suite](https://portswigger.net/burp), [FFuF](https://github.com/ffuf/ffuf), [Jaeles](https://github.com/jaeles-project/jaeles), [Qsfuzz](https://github.com/ameenmaali/qsfuzz), [Inception](https://github.com/proabiral/inception), [Snallygaster](https://github.com/hannob/snallygaster), [Gofingerprint](https://github.com/Static-Flow/gofingerprint), [Sn1per](https://github.com/1N3/Sn1per/tree/master/templates), [Google tsunami](https://github.com/google/tsunami-security-scanner), [ChopChop](https://github.com/michelin/ChopChop)
|
||||
|
|
|
|||
|
|
@ -1,26 +1,14 @@
|
|||
package runner
|
||||
|
||||
import (
|
||||
"archive/zip"
|
||||
"bufio"
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"io/ioutil"
|
||||
"net/http"
|
||||
"os"
|
||||
"path"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/blang/semver"
|
||||
"github.com/google/go-github/v32/github"
|
||||
jsoniter "github.com/json-iterator/go"
|
||||
"github.com/projectdiscovery/gologger"
|
||||
)
|
||||
|
||||
// nucleiConfig contains some configuration options for nuclei
|
||||
|
|
@ -134,301 +122,3 @@ func (r *Runner) checkIfInNucleiIgnore(item string) bool {
|
|||
|
||||
return false
|
||||
}
|
||||
|
||||
// updateTemplates checks if the default list of nuclei-templates
|
||||
// exist in the users home directory, if not the latest revision
|
||||
// is downloaded from github.
|
||||
//
|
||||
// If the path exists but is not latest, the new version is downloaded
|
||||
// from github and replaced with the templates directory.
|
||||
func (r *Runner) updateTemplates() error {
|
||||
home, err := os.UserHomeDir()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
templatesConfigFile := path.Join(home, nucleiConfigFilename)
|
||||
if _, statErr := os.Stat(templatesConfigFile); !os.IsNotExist(statErr) {
|
||||
config, readErr := r.readConfiguration()
|
||||
|
||||
if readErr != nil {
|
||||
return readErr
|
||||
}
|
||||
|
||||
r.templatesConfig = config
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
if r.templatesConfig == nil || (r.options.TemplatesDirectory != "" && r.templatesConfig.TemplatesDirectory != r.options.TemplatesDirectory) {
|
||||
if !r.options.UpdateTemplates {
|
||||
gologger.Labelf("nuclei-templates are not installed, use update-templates flag.\n")
|
||||
return nil
|
||||
}
|
||||
|
||||
// Use custom location if user has given a template directory
|
||||
if r.options.TemplatesDirectory != "" {
|
||||
home = r.options.TemplatesDirectory
|
||||
}
|
||||
|
||||
r.templatesConfig = &nucleiConfig{TemplatesDirectory: path.Join(home, "nuclei-templates")}
|
||||
|
||||
// Download the repository and also write the revision to a HEAD file.
|
||||
version, asset, getErr := r.getLatestReleaseFromGithub()
|
||||
if getErr != nil {
|
||||
return getErr
|
||||
}
|
||||
|
||||
gologger.Verbosef("Downloading nuclei-templates (v%s) to %s\n", "update-templates", version.String(), r.templatesConfig.TemplatesDirectory)
|
||||
|
||||
err = r.downloadReleaseAndUnzip(ctx, asset.GetZipballURL())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
r.templatesConfig.CurrentVersion = version.String()
|
||||
|
||||
err = r.writeConfiguration(r.templatesConfig)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
gologger.Infof("Successfully downloaded nuclei-templates (v%s). Enjoy!\n", version.String())
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Check if last checked is more than 24 hours.
|
||||
// If not, return since we don't want to do anything now.
|
||||
if time.Since(r.templatesConfig.LastChecked) < 24*time.Hour && !r.options.UpdateTemplates {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Get the configuration currently on disk.
|
||||
verText := r.templatesConfig.CurrentVersion
|
||||
indices := reVersion.FindStringIndex(verText)
|
||||
|
||||
if indices == nil {
|
||||
return fmt.Errorf("invalid release found with tag %s", err)
|
||||
}
|
||||
|
||||
if indices[0] > 0 {
|
||||
verText = verText[indices[0]:]
|
||||
}
|
||||
|
||||
oldVersion, err := semver.Make(verText)
|
||||
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
version, asset, err := r.getLatestReleaseFromGithub()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if version.EQ(oldVersion) {
|
||||
gologger.Labelf("Latest version of nuclei-templates installed: v%s\n", oldVersion.String())
|
||||
return r.writeConfiguration(r.templatesConfig)
|
||||
}
|
||||
|
||||
if version.GT(oldVersion) {
|
||||
if !r.options.UpdateTemplates {
|
||||
gologger.Labelf("You're using outdated nuclei-templates. Latest v%s\n", version.String())
|
||||
return r.writeConfiguration(r.templatesConfig)
|
||||
}
|
||||
|
||||
if r.options.TemplatesDirectory != "" {
|
||||
home = r.options.TemplatesDirectory
|
||||
r.templatesConfig.TemplatesDirectory = path.Join(home, "nuclei-templates")
|
||||
}
|
||||
|
||||
r.templatesConfig.CurrentVersion = version.String()
|
||||
|
||||
gologger.Verbosef("Downloading nuclei-templates (v%s) to %s\n", "update-templates", version.String(), r.templatesConfig.TemplatesDirectory)
|
||||
|
||||
err = r.downloadReleaseAndUnzip(ctx, asset.GetZipballURL())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
err = r.writeConfiguration(r.templatesConfig)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
gologger.Infof("Successfully updated nuclei-templates (v%s). Enjoy!\n", version.String())
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
const (
|
||||
userName = "projectdiscovery"
|
||||
repoName = "nuclei-templates"
|
||||
)
|
||||
|
||||
// getLatestReleaseFromGithub returns the latest release from github
|
||||
func (r *Runner) getLatestReleaseFromGithub() (semver.Version, *github.RepositoryRelease, error) {
|
||||
client := github.NewClient(nil)
|
||||
|
||||
rels, _, err := client.Repositories.ListReleases(context.Background(), userName, repoName, nil)
|
||||
if err != nil {
|
||||
return semver.Version{}, nil, err
|
||||
}
|
||||
|
||||
// Find the most recent version based on semantic versioning.
|
||||
var latestRelease semver.Version
|
||||
|
||||
var latestPublish *github.RepositoryRelease
|
||||
|
||||
for _, release := range rels {
|
||||
verText := release.GetTagName()
|
||||
indices := reVersion.FindStringIndex(verText)
|
||||
|
||||
if indices == nil {
|
||||
return semver.Version{}, nil, fmt.Errorf("invalid release found with tag %s", err)
|
||||
}
|
||||
|
||||
if indices[0] > 0 {
|
||||
verText = verText[indices[0]:]
|
||||
}
|
||||
|
||||
ver, err := semver.Make(verText)
|
||||
if err != nil {
|
||||
return semver.Version{}, nil, err
|
||||
}
|
||||
|
||||
if latestPublish == nil || ver.GTE(latestRelease) {
|
||||
latestRelease = ver
|
||||
latestPublish = release
|
||||
}
|
||||
}
|
||||
|
||||
if latestPublish == nil {
|
||||
return semver.Version{}, nil, errors.New("no version found for the templates")
|
||||
}
|
||||
|
||||
return latestRelease, latestPublish, nil
|
||||
}
|
||||
|
||||
// downloadReleaseAndUnzip downloads and unzips the release in a directory
|
||||
func (r *Runner) downloadReleaseAndUnzip(ctx context.Context, downloadURL string) error {
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, downloadURL, nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to create HTTP request to %s: %s", downloadURL, err)
|
||||
}
|
||||
|
||||
res, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to download a release file from %s: %s", downloadURL, err)
|
||||
}
|
||||
defer res.Body.Close()
|
||||
|
||||
if res.StatusCode != http.StatusOK {
|
||||
return fmt.Errorf("failed to download a release file from %s: Not successful status %d", downloadURL, res.StatusCode)
|
||||
}
|
||||
|
||||
buf, err := ioutil.ReadAll(res.Body)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to create buffer for zip file: %s", err)
|
||||
}
|
||||
|
||||
reader := bytes.NewReader(buf)
|
||||
z, err := zip.NewReader(reader, reader.Size())
|
||||
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to uncompress zip file: %s", err)
|
||||
}
|
||||
|
||||
// Create the template folder if it doesn't exists
|
||||
err = os.MkdirAll(r.templatesConfig.TemplatesDirectory, os.ModePerm)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to create template base folder: %s", err)
|
||||
}
|
||||
|
||||
for _, file := range z.File {
|
||||
directory, name := filepath.Split(file.Name)
|
||||
if name == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
paths := strings.Split(directory, "/")
|
||||
finalPath := strings.Join(paths[1:], "/")
|
||||
|
||||
templateDirectory := path.Join(r.templatesConfig.TemplatesDirectory, finalPath)
|
||||
err = os.MkdirAll(templateDirectory, os.ModePerm)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to create template folder %s : %s", templateDirectory, err)
|
||||
}
|
||||
|
||||
f, err := os.OpenFile(path.Join(templateDirectory, name), os.O_TRUNC|os.O_CREATE|os.O_WRONLY, 0777)
|
||||
if err != nil {
|
||||
f.Close()
|
||||
return fmt.Errorf("could not create uncompressed file: %s", err)
|
||||
}
|
||||
|
||||
reader, err := file.Open()
|
||||
if err != nil {
|
||||
f.Close()
|
||||
return fmt.Errorf("could not open archive to extract file: %s", err)
|
||||
}
|
||||
|
||||
_, err = io.Copy(f, reader)
|
||||
if err != nil {
|
||||
f.Close()
|
||||
return fmt.Errorf("could not write template file: %s", err)
|
||||
}
|
||||
f.Close()
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// isRelative checks if a given path is a relative path
|
||||
func (r *Runner) isRelative(thePath string) bool {
|
||||
if strings.HasPrefix(thePath, "/") || strings.Contains(thePath, ":\\") {
|
||||
return false
|
||||
}
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
// resolvePath gets the absolute path to the template by either
|
||||
// looking in the current directory or checking the nuclei templates directory.
|
||||
//
|
||||
// Current directory is given preference over the nuclei-templates directory.
|
||||
func (r *Runner) resolvePath(templateName string) (string, error) {
|
||||
curDirectory, err := os.Getwd()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
templatePath := path.Join(curDirectory, templateName)
|
||||
if _, err := os.Stat(templatePath); !os.IsNotExist(err) {
|
||||
gologger.Debugf("Found template in current directory: %s\n", templatePath)
|
||||
|
||||
return templatePath, nil
|
||||
}
|
||||
|
||||
if r.templatesConfig != nil {
|
||||
templatePath := path.Join(r.templatesConfig.TemplatesDirectory, templateName)
|
||||
if _, err := os.Stat(templatePath); !os.IsNotExist(err) {
|
||||
gologger.Debugf("Found template in nuclei-templates directory: %s\n", templatePath)
|
||||
|
||||
return templatePath, nil
|
||||
}
|
||||
}
|
||||
|
||||
return "", fmt.Errorf("no such path found: %s", templateName)
|
||||
}
|
||||
|
||||
func (r *Runner) resolvePathWithBaseFolder(baseFolder, templateName string) (string, error) {
|
||||
templatePath := path.Join(baseFolder, templateName)
|
||||
if _, err := os.Stat(templatePath); !os.IsNotExist(err) {
|
||||
gologger.Debugf("Found template in current directory: %s\n", templatePath)
|
||||
return templatePath, nil
|
||||
}
|
||||
|
||||
return "", fmt.Errorf("no such path found: %s", templateName)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,7 +1,9 @@
|
|||
package runner
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"flag"
|
||||
"net/url"
|
||||
"os"
|
||||
|
||||
"github.com/projectdiscovery/gologger"
|
||||
|
|
@ -20,6 +22,7 @@ type Options struct {
|
|||
JSON bool // JSON writes json output to files
|
||||
JSONRequests bool // write requests/responses for matches in JSON output
|
||||
EnableProgressBar bool // Enable progrss bar
|
||||
TemplateList bool // List available templates
|
||||
|
||||
Stdin bool // Stdin specifies whether stdin input was given to the process
|
||||
Templates multiStringFlag // Signature specifies the template/templates to use
|
||||
|
|
@ -74,6 +77,7 @@ func ParseOptions() *Options {
|
|||
flag.BoolVar(&options.JSON, "json", false, "Write json output to files")
|
||||
flag.BoolVar(&options.JSONRequests, "json-requests", false, "Write requests/responses for matches in JSON output")
|
||||
flag.BoolVar(&options.EnableProgressBar, "pbar", false, "Enable the progress bar")
|
||||
flag.BoolVar(&options.TemplateList, "tl", false, "List available templates")
|
||||
|
||||
flag.Parse()
|
||||
|
||||
|
|
@ -113,3 +117,71 @@ func hasStdin() bool {
|
|||
|
||||
return true
|
||||
}
|
||||
|
||||
// validateOptions validates the configuration options passed
|
||||
func (options *Options) validateOptions() error {
|
||||
// Both verbose and silent flags were used
|
||||
if options.Verbose && options.Silent {
|
||||
return errors.New("both verbose and silent mode specified")
|
||||
}
|
||||
|
||||
if !options.TemplateList {
|
||||
// Check if a list of templates was provided and it exists
|
||||
if len(options.Templates) == 0 && !options.UpdateTemplates {
|
||||
return errors.New("no template/templates provided")
|
||||
}
|
||||
|
||||
if options.Targets == "" && !options.Stdin && options.Target == "" && !options.UpdateTemplates {
|
||||
return errors.New("no target input provided")
|
||||
}
|
||||
}
|
||||
|
||||
// Validate proxy options if provided
|
||||
err := validateProxyURL(
|
||||
options.ProxyURL,
|
||||
"invalid http proxy format (It should be http://username:password@host:port)",
|
||||
)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
err = validateProxyURL(
|
||||
options.ProxySocksURL,
|
||||
"invalid socks proxy format (It should be socks5://username:password@host:port)",
|
||||
)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateProxyURL(proxyURL, message string) error {
|
||||
if proxyURL != "" && !isValidURL(proxyURL) {
|
||||
return errors.New(message)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func isValidURL(urlString string) bool {
|
||||
_, err := url.Parse(urlString)
|
||||
|
||||
return err == nil
|
||||
}
|
||||
|
||||
// configureOutput configures the output on the screen
|
||||
func (options *Options) configureOutput() {
|
||||
// If the user desires verbose output, show verbose output
|
||||
if options.Verbose {
|
||||
gologger.MaxLevel = gologger.Verbose
|
||||
}
|
||||
|
||||
if options.NoColor {
|
||||
gologger.UseColors = false
|
||||
}
|
||||
|
||||
if options.Silent {
|
||||
gologger.MaxLevel = gologger.Silent
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,48 @@
|
|||
package runner
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path"
|
||||
"strings"
|
||||
|
||||
"github.com/projectdiscovery/gologger"
|
||||
)
|
||||
|
||||
// isRelative checks if a given path is a relative path
|
||||
func isRelative(filePath string) bool {
|
||||
if strings.HasPrefix(filePath, "/") || strings.Contains(filePath, ":\\") {
|
||||
return false
|
||||
}
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
// resolvePath gets the absolute path to the template by either
|
||||
// looking in the current directory or checking the nuclei templates directory.
|
||||
//
|
||||
// Current directory is given preference over the nuclei-templates directory.
|
||||
func (r *Runner) resolvePath(templateName string) (string, error) {
|
||||
curDirectory, err := os.Getwd()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
templatePath := path.Join(curDirectory, templateName)
|
||||
if _, err := os.Stat(templatePath); !os.IsNotExist(err) {
|
||||
gologger.Debugf("Found template in current directory: %s\n", templatePath)
|
||||
|
||||
return templatePath, nil
|
||||
}
|
||||
|
||||
if r.templatesConfig != nil {
|
||||
templatePath := path.Join(r.templatesConfig.TemplatesDirectory, templateName)
|
||||
if _, err := os.Stat(templatePath); !os.IsNotExist(err) {
|
||||
gologger.Debugf("Found template in nuclei-templates directory: %s\n", templatePath)
|
||||
|
||||
return templatePath, nil
|
||||
}
|
||||
}
|
||||
|
||||
return "", fmt.Errorf("no such path found: %s", templateName)
|
||||
}
|
||||
|
|
@ -0,0 +1,320 @@
|
|||
package runner
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"fmt"
|
||||
"net/http/cookiejar"
|
||||
"os"
|
||||
"path"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
tengo "github.com/d5/tengo/v2"
|
||||
"github.com/d5/tengo/v2/stdlib"
|
||||
"github.com/karrick/godirwalk"
|
||||
"github.com/projectdiscovery/gologger"
|
||||
"github.com/projectdiscovery/nuclei/v2/internal/progress"
|
||||
"github.com/projectdiscovery/nuclei/v2/pkg/atomicboolean"
|
||||
"github.com/projectdiscovery/nuclei/v2/pkg/executer"
|
||||
"github.com/projectdiscovery/nuclei/v2/pkg/requests"
|
||||
"github.com/projectdiscovery/nuclei/v2/pkg/templates"
|
||||
"github.com/projectdiscovery/nuclei/v2/pkg/workflows"
|
||||
)
|
||||
|
||||
// workflowTemplates contains the initialized workflow templates per template group
|
||||
type workflowTemplates struct {
|
||||
Name string
|
||||
Templates []*workflows.Template
|
||||
}
|
||||
|
||||
// processTemplateWithList processes a template and runs the enumeration on all the targets
|
||||
func (r *Runner) processTemplateWithList(ctx context.Context, p progress.IProgress, template *templates.Template, request interface{}) bool {
|
||||
var writer *bufio.Writer
|
||||
if r.output != nil {
|
||||
writer = bufio.NewWriter(r.output)
|
||||
defer writer.Flush()
|
||||
}
|
||||
|
||||
var httpExecuter *executer.HTTPExecuter
|
||||
|
||||
var dnsExecuter *executer.DNSExecuter
|
||||
|
||||
var err error
|
||||
|
||||
// Create an executer based on the request type.
|
||||
switch value := request.(type) {
|
||||
case *requests.DNSRequest:
|
||||
dnsExecuter = executer.NewDNSExecuter(&executer.DNSOptions{
|
||||
Debug: r.options.Debug,
|
||||
Template: template,
|
||||
DNSRequest: value,
|
||||
Writer: writer,
|
||||
JSON: r.options.JSON,
|
||||
JSONRequests: r.options.JSONRequests,
|
||||
ColoredOutput: !r.options.NoColor,
|
||||
Colorizer: r.colorizer,
|
||||
Decolorizer: r.decolorizer,
|
||||
})
|
||||
case *requests.BulkHTTPRequest:
|
||||
httpExecuter, err = executer.NewHTTPExecuter(&executer.HTTPOptions{
|
||||
Debug: r.options.Debug,
|
||||
Template: template,
|
||||
BulkHTTPRequest: value,
|
||||
Writer: writer,
|
||||
Timeout: r.options.Timeout,
|
||||
Retries: r.options.Retries,
|
||||
ProxyURL: r.options.ProxyURL,
|
||||
ProxySocksURL: r.options.ProxySocksURL,
|
||||
CustomHeaders: r.options.CustomHeaders,
|
||||
JSON: r.options.JSON,
|
||||
JSONRequests: r.options.JSONRequests,
|
||||
CookieReuse: value.CookieReuse,
|
||||
ColoredOutput: !r.options.NoColor,
|
||||
Colorizer: r.colorizer,
|
||||
Decolorizer: r.decolorizer,
|
||||
})
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
p.Drop(request.(*requests.BulkHTTPRequest).GetRequestCount())
|
||||
gologger.Warningf("Could not create http client: %s\n", err)
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
var globalresult atomicboolean.AtomBool
|
||||
|
||||
var wg sync.WaitGroup
|
||||
|
||||
scanner := bufio.NewScanner(strings.NewReader(r.input))
|
||||
for scanner.Scan() {
|
||||
text := scanner.Text()
|
||||
|
||||
r.limiter <- struct{}{}
|
||||
|
||||
wg.Add(1)
|
||||
|
||||
go func(URL string) {
|
||||
defer wg.Done()
|
||||
|
||||
var result executer.Result
|
||||
|
||||
if httpExecuter != nil {
|
||||
result = httpExecuter.ExecuteHTTP(ctx, p, URL)
|
||||
globalresult.Or(result.GotResults)
|
||||
}
|
||||
|
||||
if dnsExecuter != nil {
|
||||
result = dnsExecuter.ExecuteDNS(p, URL)
|
||||
globalresult.Or(result.GotResults)
|
||||
}
|
||||
|
||||
if result.Error != nil {
|
||||
gologger.Warningf("Could not execute step: %s\n", result.Error)
|
||||
}
|
||||
|
||||
<-r.limiter
|
||||
}(text)
|
||||
}
|
||||
|
||||
wg.Wait()
|
||||
|
||||
// See if we got any results from the executers
|
||||
return globalresult.Get()
|
||||
}
|
||||
|
||||
// ProcessWorkflowWithList coming from stdin or list of targets
|
||||
func (r *Runner) processWorkflowWithList(p progress.IProgress, workflow *workflows.Workflow) {
|
||||
workflowTemplatesList, err := r.preloadWorkflowTemplates(p, workflow)
|
||||
if err != nil {
|
||||
gologger.Warningf("Could not preload templates for workflow %s: %s\n", workflow.ID, err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
logicBytes := []byte(workflow.Logic)
|
||||
|
||||
var wg sync.WaitGroup
|
||||
|
||||
scanner := bufio.NewScanner(strings.NewReader(r.input))
|
||||
for scanner.Scan() {
|
||||
targetURL := scanner.Text()
|
||||
r.limiter <- struct{}{}
|
||||
|
||||
wg.Add(1)
|
||||
|
||||
go func(targetURL string) {
|
||||
defer wg.Done()
|
||||
|
||||
script := tengo.NewScript(logicBytes)
|
||||
script.SetImports(stdlib.GetModuleMap(stdlib.AllModuleNames()...))
|
||||
|
||||
for _, workflowTemplate := range *workflowTemplatesList {
|
||||
err := script.Add(workflowTemplate.Name, &workflows.NucleiVar{Templates: workflowTemplate.Templates, URL: targetURL})
|
||||
if err != nil {
|
||||
gologger.Errorf("Could not initialize script for workflow '%s': %s\n", workflow.ID, err)
|
||||
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
_, err := script.RunContext(context.Background())
|
||||
if err != nil {
|
||||
gologger.Errorf("Could not execute workflow '%s': %s\n", workflow.ID, err)
|
||||
}
|
||||
|
||||
<-r.limiter
|
||||
}(targetURL)
|
||||
}
|
||||
|
||||
wg.Wait()
|
||||
}
|
||||
|
||||
func (r *Runner) preloadWorkflowTemplates(p progress.IProgress, workflow *workflows.Workflow) (*[]workflowTemplates, error) {
|
||||
var jar *cookiejar.Jar
|
||||
|
||||
if workflow.CookieReuse {
|
||||
var err error
|
||||
jar, err = cookiejar.New(nil)
|
||||
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
// Single yaml provided
|
||||
var wflTemplatesList []workflowTemplates
|
||||
|
||||
for name, value := range workflow.Variables {
|
||||
var writer *bufio.Writer
|
||||
if r.output != nil {
|
||||
writer = bufio.NewWriter(r.output)
|
||||
defer writer.Flush()
|
||||
}
|
||||
|
||||
// Check if the template is an absolute path or relative path.
|
||||
// If the path is absolute, use it. Otherwise,
|
||||
if isRelative(value) {
|
||||
newPath, err := r.resolvePath(value)
|
||||
if err != nil {
|
||||
newPath, err = resolvePathWithBaseFolder(filepath.Dir(workflow.GetPath()), value)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
value = newPath
|
||||
}
|
||||
|
||||
var wtlst []*workflows.Template
|
||||
|
||||
if strings.HasSuffix(value, ".yaml") {
|
||||
t, err := templates.Parse(value)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
template := &workflows.Template{Progress: p}
|
||||
if len(t.BulkRequestsHTTP) > 0 {
|
||||
template.HTTPOptions = &executer.HTTPOptions{
|
||||
Debug: r.options.Debug,
|
||||
Writer: writer,
|
||||
Template: t,
|
||||
Timeout: r.options.Timeout,
|
||||
Retries: r.options.Retries,
|
||||
ProxyURL: r.options.ProxyURL,
|
||||
ProxySocksURL: r.options.ProxySocksURL,
|
||||
CustomHeaders: r.options.CustomHeaders,
|
||||
CookieJar: jar,
|
||||
ColoredOutput: !r.options.NoColor,
|
||||
Colorizer: r.colorizer,
|
||||
Decolorizer: r.decolorizer,
|
||||
}
|
||||
} else if len(t.RequestsDNS) > 0 {
|
||||
template.DNSOptions = &executer.DNSOptions{
|
||||
Debug: r.options.Debug,
|
||||
Template: t,
|
||||
Writer: writer,
|
||||
ColoredOutput: !r.options.NoColor,
|
||||
Colorizer: r.colorizer,
|
||||
Decolorizer: r.decolorizer,
|
||||
}
|
||||
}
|
||||
|
||||
if template.DNSOptions != nil || template.HTTPOptions != nil {
|
||||
wtlst = append(wtlst, template)
|
||||
}
|
||||
} else {
|
||||
matches := []string{}
|
||||
|
||||
err := godirwalk.Walk(value, &godirwalk.Options{
|
||||
Callback: func(path string, d *godirwalk.Dirent) error {
|
||||
if !d.IsDir() && strings.HasSuffix(path, ".yaml") {
|
||||
matches = append(matches, path)
|
||||
}
|
||||
|
||||
return nil
|
||||
},
|
||||
ErrorCallback: func(path string, err error) godirwalk.ErrorAction {
|
||||
return godirwalk.SkipNode
|
||||
},
|
||||
Unsorted: true,
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 0 matches means no templates were found in directory
|
||||
if len(matches) == 0 {
|
||||
return nil, fmt.Errorf("no match found in the directory %s", value)
|
||||
}
|
||||
|
||||
for _, match := range matches {
|
||||
t, err := templates.Parse(match)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
template := &workflows.Template{Progress: p}
|
||||
if len(t.BulkRequestsHTTP) > 0 {
|
||||
template.HTTPOptions = &executer.HTTPOptions{
|
||||
Debug: r.options.Debug,
|
||||
Writer: writer,
|
||||
Template: t,
|
||||
Timeout: r.options.Timeout,
|
||||
Retries: r.options.Retries,
|
||||
ProxyURL: r.options.ProxyURL,
|
||||
ProxySocksURL: r.options.ProxySocksURL,
|
||||
CustomHeaders: r.options.CustomHeaders,
|
||||
CookieJar: jar,
|
||||
}
|
||||
} else if len(t.RequestsDNS) > 0 {
|
||||
template.DNSOptions = &executer.DNSOptions{
|
||||
Debug: r.options.Debug,
|
||||
Template: t,
|
||||
Writer: writer,
|
||||
}
|
||||
}
|
||||
if template.DNSOptions != nil || template.HTTPOptions != nil {
|
||||
wtlst = append(wtlst, template)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
wflTemplatesList = append(wflTemplatesList, workflowTemplates{Name: name, Templates: wtlst})
|
||||
}
|
||||
|
||||
return &wflTemplatesList, nil
|
||||
}
|
||||
|
||||
func resolvePathWithBaseFolder(baseFolder, templateName string) (string, error) {
|
||||
templatePath := path.Join(baseFolder, templateName)
|
||||
if _, err := os.Stat(templatePath); !os.IsNotExist(err) {
|
||||
gologger.Debugf("Found template in current directory: %s\n", templatePath)
|
||||
return templatePath, nil
|
||||
}
|
||||
|
||||
return "", fmt.Errorf("no such path found: %s", templateName)
|
||||
}
|
||||
|
|
@ -3,27 +3,18 @@ package runner
|
|||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"io/ioutil"
|
||||
"net/http/cookiejar"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
"github.com/logrusorgru/aurora"
|
||||
|
||||
tengo "github.com/d5/tengo/v2"
|
||||
"github.com/d5/tengo/v2/stdlib"
|
||||
"github.com/karrick/godirwalk"
|
||||
"github.com/projectdiscovery/gologger"
|
||||
"github.com/projectdiscovery/nuclei/v2/internal/progress"
|
||||
"github.com/projectdiscovery/nuclei/v2/pkg/atomicboolean"
|
||||
"github.com/projectdiscovery/nuclei/v2/pkg/executer"
|
||||
"github.com/projectdiscovery/nuclei/v2/pkg/requests"
|
||||
"github.com/projectdiscovery/nuclei/v2/pkg/templates"
|
||||
"github.com/projectdiscovery/nuclei/v2/pkg/workflows"
|
||||
)
|
||||
|
|
@ -51,12 +42,6 @@ type Runner struct {
|
|||
decolorizer *regexp.Regexp
|
||||
}
|
||||
|
||||
// WorkflowTemplates contains the initialized workflow templates per template group
|
||||
type WorkflowTemplates struct {
|
||||
Name string
|
||||
Templates []*workflows.Template
|
||||
}
|
||||
|
||||
// New creates a new client for running enumeration process.
|
||||
func New(options *Options) (*Runner, error) {
|
||||
runner := &Runner{
|
||||
|
|
@ -68,6 +53,11 @@ func New(options *Options) (*Runner, error) {
|
|||
gologger.Warningf("Could not update templates: %s\n", err)
|
||||
}
|
||||
|
||||
if options.TemplateList {
|
||||
runner.listAvailableTemplates()
|
||||
os.Exit(0)
|
||||
}
|
||||
|
||||
if (len(options.Templates) == 0 || (options.Targets == "" && !options.Stdin && options.Target == "")) && options.UpdateTemplates {
|
||||
os.Exit(0)
|
||||
}
|
||||
|
|
@ -184,206 +174,6 @@ func (r *Runner) Close() {
|
|||
os.Remove(r.tempFile)
|
||||
}
|
||||
|
||||
func isFilePath(path string) (bool, error) {
|
||||
info, err := os.Stat(path)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
|
||||
return info.Mode().IsRegular(), nil
|
||||
}
|
||||
|
||||
func (r *Runner) resolvePathIfRelative(path string) (string, error) {
|
||||
if r.isRelative(path) {
|
||||
newPath, err := r.resolvePath(path)
|
||||
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
return newPath, nil
|
||||
}
|
||||
|
||||
return path, nil
|
||||
}
|
||||
|
||||
func isNewPath(path string, pathMap map[string]bool) bool {
|
||||
if _, already := pathMap[path]; already {
|
||||
gologger.Warningf("Skipping already specified path '%s'", path)
|
||||
return false
|
||||
}
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
func hasMatchingSeverity(templateSeverity string, allowedSeverities []string) bool {
|
||||
for _, s := range allowedSeverities {
|
||||
if s != "" && strings.HasPrefix(templateSeverity, s) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
func (r *Runner) logTemplateLoaded(id, name, author, severity string) {
|
||||
// Display the message for the template
|
||||
message := fmt.Sprintf("[%s] %s (%s)",
|
||||
r.colorizer.BrightBlue(id).String(), r.colorizer.Bold(name).String(), r.colorizer.BrightYellow("@"+author).String())
|
||||
if severity != "" {
|
||||
message += " [" + r.colorizer.Yellow(severity).String() + "]"
|
||||
}
|
||||
|
||||
gologger.Infof("%s\n", message)
|
||||
}
|
||||
|
||||
// getParsedTemplatesFor parse the specified templates and returns a slice of the parsable ones, optionally filtered
|
||||
// by severity, along with a flag indicating if workflows are present.
|
||||
func (r *Runner) getParsedTemplatesFor(templatePaths []string, severities string) (parsedTemplates []interface{}, workflowCount int) {
|
||||
workflowCount = 0
|
||||
severities = strings.ToLower(severities)
|
||||
allSeverities := strings.Split(severities, ",")
|
||||
filterBySeverity := len(severities) > 0
|
||||
|
||||
gologger.Infof("Loading templates...")
|
||||
|
||||
for _, match := range templatePaths {
|
||||
t, err := r.parse(match)
|
||||
switch tp := t.(type) {
|
||||
case *templates.Template:
|
||||
id := tp.ID
|
||||
|
||||
// only include if severity matches or no severity filtering
|
||||
sev := strings.ToLower(tp.Info.Severity)
|
||||
if !filterBySeverity || hasMatchingSeverity(sev, allSeverities) {
|
||||
parsedTemplates = append(parsedTemplates, tp)
|
||||
r.logTemplateLoaded(tp.ID, tp.Info.Name, tp.Info.Author, tp.Info.Severity)
|
||||
} else {
|
||||
gologger.Warningf("Excluding template %s due to severity filter (%s not in [%s])", id, sev, severities)
|
||||
}
|
||||
case *workflows.Workflow:
|
||||
parsedTemplates = append(parsedTemplates, tp)
|
||||
r.logTemplateLoaded(tp.ID, tp.Info.Name, tp.Info.Author, tp.Info.Severity)
|
||||
workflowCount++
|
||||
default:
|
||||
gologger.Errorf("Could not parse file '%s': %s\n", match, err)
|
||||
}
|
||||
}
|
||||
|
||||
return parsedTemplates, workflowCount
|
||||
}
|
||||
|
||||
// getTemplatesFor parses the specified input template definitions and returns a list of unique, absolute template paths.
|
||||
func (r *Runner) getTemplatesFor(definitions []string) []string {
|
||||
// keeps track of processed dirs and files
|
||||
processed := make(map[string]bool)
|
||||
allTemplates := []string{}
|
||||
|
||||
// parses user input, handle file/directory cases and produce a list of unique templates
|
||||
for _, t := range definitions {
|
||||
var absPath string
|
||||
|
||||
var err error
|
||||
|
||||
if strings.Contains(t, "*") {
|
||||
dirs := strings.Split(t, "/")
|
||||
priorDir := strings.Join(dirs[:len(dirs)-1], "/")
|
||||
absPath, err = r.resolvePathIfRelative(priorDir)
|
||||
absPath += "/" + dirs[len(dirs)-1]
|
||||
} else {
|
||||
// resolve and convert relative to absolute path
|
||||
absPath, err = r.resolvePathIfRelative(t)
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
gologger.Errorf("Could not find template file '%s': %s\n", t, err)
|
||||
continue
|
||||
}
|
||||
|
||||
// Template input includes a wildcard
|
||||
if strings.Contains(absPath, "*") {
|
||||
var matches []string
|
||||
matches, err = filepath.Glob(absPath)
|
||||
|
||||
if err != nil {
|
||||
gologger.Labelf("Wildcard found, but unable to glob '%s': %s\n", absPath, err)
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
// couldn't find templates in directory
|
||||
if len(matches) == 0 {
|
||||
gologger.Labelf("Error, no templates were found with '%s'.\n", absPath)
|
||||
continue
|
||||
} else {
|
||||
gologger.Labelf("Identified %d templates\n", len(matches))
|
||||
}
|
||||
|
||||
for _, match := range matches {
|
||||
if !r.checkIfInNucleiIgnore(match) {
|
||||
processed[match] = true
|
||||
|
||||
allTemplates = append(allTemplates, match)
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// determine file/directory
|
||||
isFile, err := isFilePath(absPath)
|
||||
if err != nil {
|
||||
gologger.Errorf("Could not stat '%s': %s\n", absPath, err)
|
||||
continue
|
||||
}
|
||||
// test for uniqueness
|
||||
if !isNewPath(absPath, processed) {
|
||||
continue
|
||||
}
|
||||
// mark this absolute path as processed
|
||||
// - if it's a file, we'll never process it again
|
||||
// - if it's a dir, we'll never walk it again
|
||||
processed[absPath] = true
|
||||
|
||||
if isFile {
|
||||
allTemplates = append(allTemplates, absPath)
|
||||
} else {
|
||||
matches := []string{}
|
||||
|
||||
// Recursively walk down the Templates directory and run all the template file checks
|
||||
err = godirwalk.Walk(absPath, &godirwalk.Options{
|
||||
Callback: func(path string, d *godirwalk.Dirent) error {
|
||||
if !d.IsDir() && strings.HasSuffix(path, ".yaml") {
|
||||
if !r.checkIfInNucleiIgnore(path) && isNewPath(path, processed) {
|
||||
matches = append(matches, path)
|
||||
processed[path] = true
|
||||
}
|
||||
}
|
||||
return nil
|
||||
},
|
||||
ErrorCallback: func(path string, err error) godirwalk.ErrorAction {
|
||||
return godirwalk.SkipNode
|
||||
},
|
||||
Unsorted: true,
|
||||
})
|
||||
|
||||
// directory couldn't be walked
|
||||
if err != nil {
|
||||
gologger.Labelf("Could not find templates in directory '%s': %s\n", absPath, err)
|
||||
continue
|
||||
}
|
||||
|
||||
// couldn't find templates in directory
|
||||
if len(matches) == 0 {
|
||||
gologger.Labelf("Error, no templates were found in '%s'.\n", absPath)
|
||||
continue
|
||||
}
|
||||
|
||||
allTemplates = append(allTemplates, matches...)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return allTemplates
|
||||
}
|
||||
|
||||
// RunEnumeration sets up the input layer for giving input nuclei.
|
||||
// binary and runs the actual enumeration
|
||||
func (r *Runner) RunEnumeration() {
|
||||
|
|
@ -465,7 +255,7 @@ func (r *Runner) RunEnumeration() {
|
|||
}
|
||||
case *workflows.Workflow:
|
||||
workflow := template.(*workflows.Workflow)
|
||||
r.ProcessWorkflowWithList(p, workflow)
|
||||
r.processWorkflowWithList(p, workflow)
|
||||
}
|
||||
}(t)
|
||||
}
|
||||
|
|
@ -484,308 +274,3 @@ func (r *Runner) RunEnumeration() {
|
|||
gologger.Infof("No results found. Happy hacking!")
|
||||
}
|
||||
}
|
||||
|
||||
// processTemplateWithList processes a template and runs the enumeration on all the targets
|
||||
func (r *Runner) processTemplateWithList(ctx context.Context, p progress.IProgress, template *templates.Template, request interface{}) bool {
|
||||
var writer *bufio.Writer
|
||||
if r.output != nil {
|
||||
writer = bufio.NewWriter(r.output)
|
||||
defer writer.Flush()
|
||||
}
|
||||
|
||||
var httpExecuter *executer.HTTPExecuter
|
||||
|
||||
var dnsExecuter *executer.DNSExecuter
|
||||
|
||||
var err error
|
||||
|
||||
// Create an executer based on the request type.
|
||||
switch value := request.(type) {
|
||||
case *requests.DNSRequest:
|
||||
dnsExecuter = executer.NewDNSExecuter(&executer.DNSOptions{
|
||||
Debug: r.options.Debug,
|
||||
Template: template,
|
||||
DNSRequest: value,
|
||||
Writer: writer,
|
||||
JSON: r.options.JSON,
|
||||
JSONRequests: r.options.JSONRequests,
|
||||
ColoredOutput: !r.options.NoColor,
|
||||
Colorizer: r.colorizer,
|
||||
Decolorizer: r.decolorizer,
|
||||
})
|
||||
case *requests.BulkHTTPRequest:
|
||||
httpExecuter, err = executer.NewHTTPExecuter(&executer.HTTPOptions{
|
||||
Debug: r.options.Debug,
|
||||
Template: template,
|
||||
BulkHTTPRequest: value,
|
||||
Writer: writer,
|
||||
Timeout: r.options.Timeout,
|
||||
Retries: r.options.Retries,
|
||||
ProxyURL: r.options.ProxyURL,
|
||||
ProxySocksURL: r.options.ProxySocksURL,
|
||||
CustomHeaders: r.options.CustomHeaders,
|
||||
JSON: r.options.JSON,
|
||||
JSONRequests: r.options.JSONRequests,
|
||||
CookieReuse: value.CookieReuse,
|
||||
ColoredOutput: !r.options.NoColor,
|
||||
Colorizer: r.colorizer,
|
||||
Decolorizer: r.decolorizer,
|
||||
})
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
p.Drop(request.(*requests.BulkHTTPRequest).GetRequestCount())
|
||||
gologger.Warningf("Could not create http client: %s\n", err)
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
var globalresult atomicboolean.AtomBool
|
||||
|
||||
var wg sync.WaitGroup
|
||||
|
||||
scanner := bufio.NewScanner(strings.NewReader(r.input))
|
||||
for scanner.Scan() {
|
||||
text := scanner.Text()
|
||||
|
||||
r.limiter <- struct{}{}
|
||||
|
||||
wg.Add(1)
|
||||
|
||||
go func(URL string) {
|
||||
defer wg.Done()
|
||||
|
||||
var result executer.Result
|
||||
|
||||
if httpExecuter != nil {
|
||||
result = httpExecuter.ExecuteHTTP(ctx, p, URL)
|
||||
globalresult.Or(result.GotResults)
|
||||
}
|
||||
|
||||
if dnsExecuter != nil {
|
||||
result = dnsExecuter.ExecuteDNS(p, URL)
|
||||
globalresult.Or(result.GotResults)
|
||||
}
|
||||
|
||||
if result.Error != nil {
|
||||
gologger.Warningf("Could not execute step: %s\n", result.Error)
|
||||
}
|
||||
|
||||
<-r.limiter
|
||||
}(text)
|
||||
}
|
||||
|
||||
wg.Wait()
|
||||
|
||||
// See if we got any results from the executers
|
||||
return globalresult.Get()
|
||||
}
|
||||
|
||||
// ProcessWorkflowWithList coming from stdin or list of targets
|
||||
func (r *Runner) ProcessWorkflowWithList(p progress.IProgress, workflow *workflows.Workflow) {
|
||||
workflowTemplatesList, err := r.PreloadTemplates(p, workflow)
|
||||
if err != nil {
|
||||
gologger.Warningf("Could not preload templates for workflow %s: %s\n", workflow.ID, err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
logicBytes := []byte(workflow.Logic)
|
||||
|
||||
var wg sync.WaitGroup
|
||||
|
||||
scanner := bufio.NewScanner(strings.NewReader(r.input))
|
||||
for scanner.Scan() {
|
||||
targetURL := scanner.Text()
|
||||
r.limiter <- struct{}{}
|
||||
|
||||
wg.Add(1)
|
||||
|
||||
go func(targetURL string) {
|
||||
defer wg.Done()
|
||||
|
||||
script := tengo.NewScript(logicBytes)
|
||||
script.SetImports(stdlib.GetModuleMap(stdlib.AllModuleNames()...))
|
||||
|
||||
for _, workflowTemplate := range *workflowTemplatesList {
|
||||
err := script.Add(workflowTemplate.Name, &workflows.NucleiVar{Templates: workflowTemplate.Templates, URL: targetURL})
|
||||
if err != nil {
|
||||
gologger.Errorf("Could not initialize script for workflow '%s': %s\n", workflow.ID, err)
|
||||
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
_, err := script.RunContext(context.Background())
|
||||
if err != nil {
|
||||
gologger.Errorf("Could not execute workflow '%s': %s\n", workflow.ID, err)
|
||||
}
|
||||
|
||||
<-r.limiter
|
||||
}(targetURL)
|
||||
}
|
||||
|
||||
wg.Wait()
|
||||
}
|
||||
|
||||
// PreloadTemplates preload the workflow templates once
|
||||
func (r *Runner) PreloadTemplates(p progress.IProgress, workflow *workflows.Workflow) (*[]WorkflowTemplates, error) {
|
||||
var jar *cookiejar.Jar
|
||||
|
||||
if workflow.CookieReuse {
|
||||
var err error
|
||||
jar, err = cookiejar.New(nil)
|
||||
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
// Single yaml provided
|
||||
var wflTemplatesList []WorkflowTemplates
|
||||
|
||||
for name, value := range workflow.Variables {
|
||||
var writer *bufio.Writer
|
||||
if r.output != nil {
|
||||
writer = bufio.NewWriter(r.output)
|
||||
defer writer.Flush()
|
||||
}
|
||||
|
||||
// Check if the template is an absolute path or relative path.
|
||||
// If the path is absolute, use it. Otherwise,
|
||||
if r.isRelative(value) {
|
||||
newPath, err := r.resolvePath(value)
|
||||
if err != nil {
|
||||
newPath, err = r.resolvePathWithBaseFolder(filepath.Dir(workflow.GetPath()), value)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
value = newPath
|
||||
}
|
||||
|
||||
var wtlst []*workflows.Template
|
||||
|
||||
if strings.HasSuffix(value, ".yaml") {
|
||||
t, err := templates.Parse(value)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
template := &workflows.Template{Progress: p}
|
||||
if len(t.BulkRequestsHTTP) > 0 {
|
||||
template.HTTPOptions = &executer.HTTPOptions{
|
||||
Debug: r.options.Debug,
|
||||
Writer: writer,
|
||||
Template: t,
|
||||
Timeout: r.options.Timeout,
|
||||
Retries: r.options.Retries,
|
||||
ProxyURL: r.options.ProxyURL,
|
||||
ProxySocksURL: r.options.ProxySocksURL,
|
||||
CustomHeaders: r.options.CustomHeaders,
|
||||
CookieJar: jar,
|
||||
ColoredOutput: !r.options.NoColor,
|
||||
Colorizer: r.colorizer,
|
||||
Decolorizer: r.decolorizer,
|
||||
}
|
||||
} else if len(t.RequestsDNS) > 0 {
|
||||
template.DNSOptions = &executer.DNSOptions{
|
||||
Debug: r.options.Debug,
|
||||
Template: t,
|
||||
Writer: writer,
|
||||
ColoredOutput: !r.options.NoColor,
|
||||
Colorizer: r.colorizer,
|
||||
Decolorizer: r.decolorizer,
|
||||
}
|
||||
}
|
||||
|
||||
if template.DNSOptions != nil || template.HTTPOptions != nil {
|
||||
wtlst = append(wtlst, template)
|
||||
}
|
||||
} else {
|
||||
matches := []string{}
|
||||
|
||||
err := godirwalk.Walk(value, &godirwalk.Options{
|
||||
Callback: func(path string, d *godirwalk.Dirent) error {
|
||||
if !d.IsDir() && strings.HasSuffix(path, ".yaml") {
|
||||
matches = append(matches, path)
|
||||
}
|
||||
|
||||
return nil
|
||||
},
|
||||
ErrorCallback: func(path string, err error) godirwalk.ErrorAction {
|
||||
return godirwalk.SkipNode
|
||||
},
|
||||
Unsorted: true,
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 0 matches means no templates were found in directory
|
||||
if len(matches) == 0 {
|
||||
return nil, fmt.Errorf("no match found in the directory %s", value)
|
||||
}
|
||||
|
||||
for _, match := range matches {
|
||||
t, err := templates.Parse(match)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
template := &workflows.Template{Progress: p}
|
||||
if len(t.BulkRequestsHTTP) > 0 {
|
||||
template.HTTPOptions = &executer.HTTPOptions{
|
||||
Debug: r.options.Debug,
|
||||
Writer: writer,
|
||||
Template: t,
|
||||
Timeout: r.options.Timeout,
|
||||
Retries: r.options.Retries,
|
||||
ProxyURL: r.options.ProxyURL,
|
||||
ProxySocksURL: r.options.ProxySocksURL,
|
||||
CustomHeaders: r.options.CustomHeaders,
|
||||
CookieJar: jar,
|
||||
}
|
||||
} else if len(t.RequestsDNS) > 0 {
|
||||
template.DNSOptions = &executer.DNSOptions{
|
||||
Debug: r.options.Debug,
|
||||
Template: t,
|
||||
Writer: writer,
|
||||
}
|
||||
}
|
||||
if template.DNSOptions != nil || template.HTTPOptions != nil {
|
||||
wtlst = append(wtlst, template)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
wflTemplatesList = append(wflTemplatesList, WorkflowTemplates{Name: name, Templates: wtlst})
|
||||
}
|
||||
|
||||
return &wflTemplatesList, nil
|
||||
}
|
||||
|
||||
func (r *Runner) parse(file string) (interface{}, error) {
|
||||
// check if it's a template
|
||||
template, errTemplate := templates.Parse(file)
|
||||
if errTemplate == nil {
|
||||
return template, nil
|
||||
}
|
||||
|
||||
// check if it's a workflow
|
||||
workflow, errWorkflow := workflows.Parse(file)
|
||||
if errWorkflow == nil {
|
||||
return workflow, nil
|
||||
}
|
||||
|
||||
if errTemplate != nil {
|
||||
return nil, errTemplate
|
||||
}
|
||||
|
||||
if errWorkflow != nil {
|
||||
return nil, errWorkflow
|
||||
}
|
||||
|
||||
return nil, errors.New("unknown error occurred")
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,311 @@
|
|||
package runner
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"github.com/karrick/godirwalk"
|
||||
"github.com/logrusorgru/aurora"
|
||||
"github.com/projectdiscovery/gologger"
|
||||
"github.com/projectdiscovery/nuclei/v2/pkg/templates"
|
||||
"github.com/projectdiscovery/nuclei/v2/pkg/workflows"
|
||||
)
|
||||
|
||||
var severityMap = map[string]string{
|
||||
"info": aurora.Cyan("info").String(),
|
||||
"low": aurora.Green("low").String(),
|
||||
"medium": aurora.Yellow("medium").String(),
|
||||
"high": aurora.Red("high").String(),
|
||||
}
|
||||
|
||||
// getTemplatesFor parses the specified input template definitions and returns a list of unique, absolute template paths.
|
||||
func (r *Runner) getTemplatesFor(definitions []string) []string {
|
||||
// keeps track of processed dirs and files
|
||||
processed := make(map[string]bool)
|
||||
allTemplates := []string{}
|
||||
|
||||
// parses user input, handle file/directory cases and produce a list of unique templates
|
||||
for _, t := range definitions {
|
||||
var absPath string
|
||||
|
||||
var err error
|
||||
|
||||
if strings.Contains(t, "*") {
|
||||
dirs := strings.Split(t, "/")
|
||||
priorDir := strings.Join(dirs[:len(dirs)-1], "/")
|
||||
absPath, err = r.resolvePathIfRelative(priorDir)
|
||||
absPath += "/" + dirs[len(dirs)-1]
|
||||
} else {
|
||||
// resolve and convert relative to absolute path
|
||||
absPath, err = r.resolvePathIfRelative(t)
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
gologger.Errorf("Could not find template file '%s': %s\n", t, err)
|
||||
continue
|
||||
}
|
||||
|
||||
// Template input includes a wildcard
|
||||
if strings.Contains(absPath, "*") {
|
||||
var matches []string
|
||||
matches, err = filepath.Glob(absPath)
|
||||
|
||||
if err != nil {
|
||||
gologger.Labelf("Wildcard found, but unable to glob '%s': %s\n", absPath, err)
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
// couldn't find templates in directory
|
||||
if len(matches) == 0 {
|
||||
gologger.Labelf("Error, no templates were found with '%s'.\n", absPath)
|
||||
continue
|
||||
} else {
|
||||
gologger.Labelf("Identified %d templates\n", len(matches))
|
||||
}
|
||||
|
||||
for _, match := range matches {
|
||||
if !r.checkIfInNucleiIgnore(match) {
|
||||
processed[match] = true
|
||||
|
||||
allTemplates = append(allTemplates, match)
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// determine file/directory
|
||||
isFile, err := isFilePath(absPath)
|
||||
if err != nil {
|
||||
gologger.Errorf("Could not stat '%s': %s\n", absPath, err)
|
||||
continue
|
||||
}
|
||||
// test for uniqueness
|
||||
if !isNewPath(absPath, processed) {
|
||||
continue
|
||||
}
|
||||
// mark this absolute path as processed
|
||||
// - if it's a file, we'll never process it again
|
||||
// - if it's a dir, we'll never walk it again
|
||||
processed[absPath] = true
|
||||
|
||||
if isFile {
|
||||
allTemplates = append(allTemplates, absPath)
|
||||
} else {
|
||||
matches := []string{}
|
||||
|
||||
// Recursively walk down the Templates directory and run all the template file checks
|
||||
err := directoryWalker(
|
||||
absPath,
|
||||
func(path string, d *godirwalk.Dirent) error {
|
||||
if !d.IsDir() && strings.HasSuffix(path, ".yaml") {
|
||||
if !r.checkIfInNucleiIgnore(path) && isNewPath(path, processed) {
|
||||
matches = append(matches, path)
|
||||
processed[path] = true
|
||||
}
|
||||
}
|
||||
return nil
|
||||
},
|
||||
)
|
||||
|
||||
// directory couldn't be walked
|
||||
if err != nil {
|
||||
gologger.Labelf("Could not find templates in directory '%s': %s\n", absPath, err)
|
||||
continue
|
||||
}
|
||||
|
||||
// couldn't find templates in directory
|
||||
if len(matches) == 0 {
|
||||
gologger.Labelf("Error, no templates were found in '%s'.\n", absPath)
|
||||
continue
|
||||
}
|
||||
|
||||
allTemplates = append(allTemplates, matches...)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return allTemplates
|
||||
}
|
||||
|
||||
// getParsedTemplatesFor parse the specified templates and returns a slice of the parsable ones, optionally filtered
|
||||
// by severity, along with a flag indicating if workflows are present.
|
||||
func (r *Runner) getParsedTemplatesFor(templatePaths []string, severities string) (parsedTemplates []interface{}, workflowCount int) {
|
||||
workflowCount = 0
|
||||
severities = strings.ToLower(severities)
|
||||
allSeverities := strings.Split(severities, ",")
|
||||
filterBySeverity := len(severities) > 0
|
||||
|
||||
gologger.Infof("Loading templates...")
|
||||
|
||||
for _, match := range templatePaths {
|
||||
t, err := r.parseTemplateFile(match)
|
||||
switch tp := t.(type) {
|
||||
case *templates.Template:
|
||||
// only include if severity matches or no severity filtering
|
||||
sev := strings.ToLower(tp.Info.Severity)
|
||||
if !filterBySeverity || hasMatchingSeverity(sev, allSeverities) {
|
||||
parsedTemplates = append(parsedTemplates, tp)
|
||||
gologger.Infof("%s\n", r.templateLogMsg(tp.ID, tp.Info.Name, tp.Info.Author, tp.Info.Severity))
|
||||
} else {
|
||||
gologger.Warningf("Excluding template %s due to severity filter (%s not in [%s])", tp.ID, sev, severities)
|
||||
}
|
||||
case *workflows.Workflow:
|
||||
parsedTemplates = append(parsedTemplates, tp)
|
||||
gologger.Infof("%s\n", r.templateLogMsg(tp.ID, tp.Info.Name, tp.Info.Author, tp.Info.Severity))
|
||||
workflowCount++
|
||||
default:
|
||||
gologger.Errorf("Could not parse file '%s': %s\n", match, err)
|
||||
}
|
||||
}
|
||||
|
||||
return parsedTemplates, workflowCount
|
||||
}
|
||||
|
||||
func (r *Runner) parseTemplateFile(file string) (interface{}, error) {
|
||||
// check if it's a template
|
||||
template, errTemplate := templates.Parse(file)
|
||||
if errTemplate == nil {
|
||||
return template, nil
|
||||
}
|
||||
|
||||
// check if it's a workflow
|
||||
workflow, errWorkflow := workflows.Parse(file)
|
||||
if errWorkflow == nil {
|
||||
return workflow, nil
|
||||
}
|
||||
|
||||
if errTemplate != nil {
|
||||
return nil, errTemplate
|
||||
}
|
||||
|
||||
if errWorkflow != nil {
|
||||
return nil, errWorkflow
|
||||
}
|
||||
|
||||
return nil, errors.New("unknown error occurred")
|
||||
}
|
||||
|
||||
func (r *Runner) templateLogMsg(id, name, author, severity string) string {
|
||||
// Display the message for the template
|
||||
message := fmt.Sprintf("[%s] %s (%s)",
|
||||
r.colorizer.BrightBlue(id).String(),
|
||||
r.colorizer.Bold(name).String(),
|
||||
r.colorizer.BrightYellow("@"+author).String())
|
||||
|
||||
if severity != "" {
|
||||
message += " [" + severityMap[severity] + "]"
|
||||
}
|
||||
|
||||
return message
|
||||
}
|
||||
|
||||
func (r *Runner) logAvailableTemplate(tplPath string) {
|
||||
t, err := r.parseTemplateFile(tplPath)
|
||||
if t != nil {
|
||||
switch tp := t.(type) {
|
||||
case *templates.Template:
|
||||
gologger.Silentf("%s\n", r.templateLogMsg(tp.ID, tp.Info.Name, tp.Info.Author, tp.Info.Severity))
|
||||
case *workflows.Workflow:
|
||||
gologger.Silentf("%s\n", r.templateLogMsg(tp.ID, tp.Info.Name, tp.Info.Author, tp.Info.Severity))
|
||||
default:
|
||||
gologger.Errorf("Could not parse file '%s': %s\n", tplPath, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ListAvailableTemplates prints available templates to stdout
|
||||
func (r *Runner) listAvailableTemplates() {
|
||||
if r.templatesConfig == nil {
|
||||
return
|
||||
}
|
||||
|
||||
if _, err := os.Stat(r.templatesConfig.TemplatesDirectory); os.IsNotExist(err) {
|
||||
gologger.Errorf("%s does not exists", r.templatesConfig.TemplatesDirectory)
|
||||
return
|
||||
}
|
||||
|
||||
gologger.Silentf(
|
||||
"\nListing available v.%s nuclei templates for %s",
|
||||
r.templatesConfig.CurrentVersion,
|
||||
r.templatesConfig.TemplatesDirectory,
|
||||
)
|
||||
r.colorizer = aurora.NewAurora(true)
|
||||
err := directoryWalker(
|
||||
r.templatesConfig.TemplatesDirectory,
|
||||
func(path string, d *godirwalk.Dirent) error {
|
||||
if d.IsDir() && path != r.templatesConfig.TemplatesDirectory {
|
||||
gologger.Silentf("\n%s:\n\n", r.colorizer.Bold(r.colorizer.BgBrightBlue(strings.Title(d.Name()))).String())
|
||||
} else if strings.HasSuffix(path, ".yaml") {
|
||||
r.logAvailableTemplate(path)
|
||||
}
|
||||
|
||||
return nil
|
||||
},
|
||||
)
|
||||
|
||||
// directory couldn't be walked
|
||||
if err != nil {
|
||||
gologger.Labelf("Could not find templates in directory '%s': %s\n", r.templatesConfig.TemplatesDirectory, err)
|
||||
}
|
||||
}
|
||||
|
||||
func (r *Runner) resolvePathIfRelative(filePath string) (string, error) {
|
||||
if isRelative(filePath) {
|
||||
newPath, err := r.resolvePath(filePath)
|
||||
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
return newPath, nil
|
||||
}
|
||||
|
||||
return filePath, nil
|
||||
}
|
||||
|
||||
func hasMatchingSeverity(templateSeverity string, allowedSeverities []string) bool {
|
||||
for _, s := range allowedSeverities {
|
||||
if s != "" && strings.HasPrefix(templateSeverity, s) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
func directoryWalker(fsPath string, callback func(fsPath string, d *godirwalk.Dirent) error) error {
|
||||
err := godirwalk.Walk(fsPath, &godirwalk.Options{
|
||||
Callback: callback,
|
||||
ErrorCallback: func(fsPath string, err error) godirwalk.ErrorAction {
|
||||
return godirwalk.SkipNode
|
||||
},
|
||||
Unsorted: true,
|
||||
})
|
||||
|
||||
// directory couldn't be walked
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func isFilePath(filePath string) (bool, error) {
|
||||
info, err := os.Stat(filePath)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
|
||||
return info.Mode().IsRegular(), nil
|
||||
}
|
||||
|
||||
func isNewPath(filePath string, pathMap map[string]bool) bool {
|
||||
if _, already := pathMap[filePath]; already {
|
||||
gologger.Warningf("Skipping already specified path '%s'", filePath)
|
||||
return false
|
||||
}
|
||||
|
||||
return true
|
||||
}
|
||||
|
|
@ -0,0 +1,273 @@
|
|||
package runner
|
||||
|
||||
import (
|
||||
"archive/zip"
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"io/ioutil"
|
||||
"net/http"
|
||||
"os"
|
||||
"path"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/blang/semver"
|
||||
"github.com/google/go-github/v32/github"
|
||||
"github.com/projectdiscovery/gologger"
|
||||
)
|
||||
|
||||
const (
|
||||
userName = "projectdiscovery"
|
||||
repoName = "nuclei-templates"
|
||||
)
|
||||
|
||||
// updateTemplates checks if the default list of nuclei-templates
|
||||
// exist in the users home directory, if not the latest revision
|
||||
// is downloaded from github.
|
||||
//
|
||||
// If the path exists but is not latest, the new version is downloaded
|
||||
// from github and replaced with the templates directory.
|
||||
func (r *Runner) updateTemplates() error {
|
||||
home, err := os.UserHomeDir()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
templatesConfigFile := path.Join(home, nucleiConfigFilename)
|
||||
if _, statErr := os.Stat(templatesConfigFile); !os.IsNotExist(statErr) {
|
||||
config, readErr := r.readConfiguration()
|
||||
|
||||
if readErr != nil {
|
||||
return readErr
|
||||
}
|
||||
|
||||
r.templatesConfig = config
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
if r.templatesConfig == nil || (r.options.TemplatesDirectory != "" && r.templatesConfig.TemplatesDirectory != r.options.TemplatesDirectory) {
|
||||
if !r.options.UpdateTemplates {
|
||||
gologger.Labelf("nuclei-templates are not installed, use update-templates flag.\n")
|
||||
return nil
|
||||
}
|
||||
|
||||
// Use custom location if user has given a template directory
|
||||
if r.options.TemplatesDirectory != "" {
|
||||
home = r.options.TemplatesDirectory
|
||||
}
|
||||
|
||||
r.templatesConfig = &nucleiConfig{TemplatesDirectory: path.Join(home, "nuclei-templates")}
|
||||
|
||||
// Download the repository and also write the revision to a HEAD file.
|
||||
version, asset, getErr := r.getLatestReleaseFromGithub()
|
||||
if getErr != nil {
|
||||
return getErr
|
||||
}
|
||||
|
||||
gologger.Verbosef("Downloading nuclei-templates (v%s) to %s\n", "update-templates", version.String(), r.templatesConfig.TemplatesDirectory)
|
||||
|
||||
err = r.downloadReleaseAndUnzip(ctx, asset.GetZipballURL())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
r.templatesConfig.CurrentVersion = version.String()
|
||||
|
||||
err = r.writeConfiguration(r.templatesConfig)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
gologger.Infof("Successfully downloaded nuclei-templates (v%s). Enjoy!\n", version.String())
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Check if last checked is more than 24 hours.
|
||||
// If not, return since we don't want to do anything now.
|
||||
if time.Since(r.templatesConfig.LastChecked) < 24*time.Hour && !r.options.UpdateTemplates {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Get the configuration currently on disk.
|
||||
verText := r.templatesConfig.CurrentVersion
|
||||
indices := reVersion.FindStringIndex(verText)
|
||||
|
||||
if indices == nil {
|
||||
return fmt.Errorf("invalid release found with tag %s", err)
|
||||
}
|
||||
|
||||
if indices[0] > 0 {
|
||||
verText = verText[indices[0]:]
|
||||
}
|
||||
|
||||
oldVersion, err := semver.Make(verText)
|
||||
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
version, asset, err := r.getLatestReleaseFromGithub()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if version.EQ(oldVersion) {
|
||||
gologger.Labelf("Latest version of nuclei-templates installed: v%s\n", oldVersion.String())
|
||||
return r.writeConfiguration(r.templatesConfig)
|
||||
}
|
||||
|
||||
if version.GT(oldVersion) {
|
||||
if !r.options.UpdateTemplates {
|
||||
gologger.Labelf("You're using outdated nuclei-templates. Latest v%s\n", version.String())
|
||||
return r.writeConfiguration(r.templatesConfig)
|
||||
}
|
||||
|
||||
if r.options.TemplatesDirectory != "" {
|
||||
home = r.options.TemplatesDirectory
|
||||
r.templatesConfig.TemplatesDirectory = path.Join(home, "nuclei-templates")
|
||||
}
|
||||
|
||||
r.templatesConfig.CurrentVersion = version.String()
|
||||
|
||||
gologger.Verbosef("Downloading nuclei-templates (v%s) to %s\n", "update-templates", version.String(), r.templatesConfig.TemplatesDirectory)
|
||||
|
||||
err = r.downloadReleaseAndUnzip(ctx, asset.GetZipballURL())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
err = r.writeConfiguration(r.templatesConfig)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
gologger.Infof("Successfully updated nuclei-templates (v%s). Enjoy!\n", version.String())
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// getLatestReleaseFromGithub returns the latest release from github
|
||||
func (r *Runner) getLatestReleaseFromGithub() (semver.Version, *github.RepositoryRelease, error) {
|
||||
client := github.NewClient(nil)
|
||||
|
||||
rels, _, err := client.Repositories.ListReleases(context.Background(), userName, repoName, nil)
|
||||
if err != nil {
|
||||
return semver.Version{}, nil, err
|
||||
}
|
||||
|
||||
// Find the most recent version based on semantic versioning.
|
||||
var latestRelease semver.Version
|
||||
|
||||
var latestPublish *github.RepositoryRelease
|
||||
|
||||
for _, release := range rels {
|
||||
verText := release.GetTagName()
|
||||
indices := reVersion.FindStringIndex(verText)
|
||||
|
||||
if indices == nil {
|
||||
return semver.Version{}, nil, fmt.Errorf("invalid release found with tag %s", err)
|
||||
}
|
||||
|
||||
if indices[0] > 0 {
|
||||
verText = verText[indices[0]:]
|
||||
}
|
||||
|
||||
ver, err := semver.Make(verText)
|
||||
if err != nil {
|
||||
return semver.Version{}, nil, err
|
||||
}
|
||||
|
||||
if latestPublish == nil || ver.GTE(latestRelease) {
|
||||
latestRelease = ver
|
||||
latestPublish = release
|
||||
}
|
||||
}
|
||||
|
||||
if latestPublish == nil {
|
||||
return semver.Version{}, nil, errors.New("no version found for the templates")
|
||||
}
|
||||
|
||||
return latestRelease, latestPublish, nil
|
||||
}
|
||||
|
||||
// downloadReleaseAndUnzip downloads and unzips the release in a directory
|
||||
func (r *Runner) downloadReleaseAndUnzip(ctx context.Context, downloadURL string) error {
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, downloadURL, nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to create HTTP request to %s: %s", downloadURL, err)
|
||||
}
|
||||
|
||||
res, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to download a release file from %s: %s", downloadURL, err)
|
||||
}
|
||||
defer res.Body.Close()
|
||||
|
||||
if res.StatusCode != http.StatusOK {
|
||||
return fmt.Errorf("failed to download a release file from %s: Not successful status %d", downloadURL, res.StatusCode)
|
||||
}
|
||||
|
||||
buf, err := ioutil.ReadAll(res.Body)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to create buffer for zip file: %s", err)
|
||||
}
|
||||
|
||||
reader := bytes.NewReader(buf)
|
||||
z, err := zip.NewReader(reader, reader.Size())
|
||||
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to uncompress zip file: %s", err)
|
||||
}
|
||||
|
||||
// Create the template folder if it doesn't exists
|
||||
err = os.MkdirAll(r.templatesConfig.TemplatesDirectory, os.ModePerm)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to create template base folder: %s", err)
|
||||
}
|
||||
|
||||
for _, file := range z.File {
|
||||
directory, name := filepath.Split(file.Name)
|
||||
if name == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
paths := strings.Split(directory, "/")
|
||||
finalPath := strings.Join(paths[1:], "/")
|
||||
|
||||
templateDirectory := path.Join(r.templatesConfig.TemplatesDirectory, finalPath)
|
||||
err = os.MkdirAll(templateDirectory, os.ModePerm)
|
||||
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to create template folder %s : %s", templateDirectory, err)
|
||||
}
|
||||
|
||||
f, err := os.OpenFile(path.Join(templateDirectory, name), os.O_TRUNC|os.O_CREATE|os.O_WRONLY, 0777)
|
||||
if err != nil {
|
||||
f.Close()
|
||||
return fmt.Errorf("could not create uncompressed file: %s", err)
|
||||
}
|
||||
|
||||
reader, err := file.Open()
|
||||
if err != nil {
|
||||
f.Close()
|
||||
return fmt.Errorf("could not open archive to extract file: %s", err)
|
||||
}
|
||||
|
||||
_, err = io.Copy(f, reader)
|
||||
if err != nil {
|
||||
f.Close()
|
||||
return fmt.Errorf("could not write template file: %s", err)
|
||||
}
|
||||
|
||||
f.Close()
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
|
@ -1,58 +0,0 @@
|
|||
package runner
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"net/url"
|
||||
|
||||
"github.com/projectdiscovery/gologger"
|
||||
)
|
||||
|
||||
// validateOptions validates the configuration options passed
|
||||
func (options *Options) validateOptions() error {
|
||||
// Both verbose and silent flags were used
|
||||
if options.Verbose && options.Silent {
|
||||
return errors.New("both verbose and silent mode specified")
|
||||
}
|
||||
|
||||
// Check if a list of templates was provided and it exists
|
||||
if len(options.Templates) == 0 && !options.UpdateTemplates {
|
||||
return errors.New("no template/templates provided")
|
||||
}
|
||||
|
||||
if options.Targets == "" && !options.Stdin && options.Target == "" && !options.UpdateTemplates {
|
||||
return errors.New("no target input provided")
|
||||
}
|
||||
|
||||
// Validate proxy options if provided
|
||||
if options.ProxyURL != "" && !isValidProxyURL(options.ProxyURL) {
|
||||
return errors.New("invalid http proxy format (It should be http://username:password@host:port)")
|
||||
}
|
||||
|
||||
if options.ProxySocksURL != "" && !isValidProxyURL(options.ProxySocksURL) {
|
||||
return errors.New("invalid socks proxy format (It should be socks5://username:password@host:port)")
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func isValidProxyURL(proxyURL string) bool {
|
||||
_, err := url.Parse(proxyURL)
|
||||
|
||||
return err == nil
|
||||
}
|
||||
|
||||
// configureOutput configures the output on the screen
|
||||
func (options *Options) configureOutput() {
|
||||
// If the user desires verbose output, show verbose output
|
||||
if options.Verbose {
|
||||
gologger.MaxLevel = gologger.Verbose
|
||||
}
|
||||
|
||||
if options.NoColor {
|
||||
gologger.UseColors = false
|
||||
}
|
||||
|
||||
if options.Silent {
|
||||
gologger.MaxLevel = gologger.Silent
|
||||
}
|
||||
}
|
||||
|
|
@ -215,6 +215,13 @@ func (r *BulkHTTPRequest) handleRawWithPaylods(ctx context.Context, raw, baseURL
|
|||
return &HTTPRequest{Request: request, Meta: genValues}, nil
|
||||
}
|
||||
|
||||
func setHeader(req *http.Request, name, value string) {
|
||||
// Set some headers only if the header wasn't supplied by the user
|
||||
if _, ok := req.Header[name]; !ok {
|
||||
req.Header.Set(name, value)
|
||||
}
|
||||
}
|
||||
|
||||
func (r *BulkHTTPRequest) fillRequest(req *http.Request, values map[string]interface{}) (*retryablehttp.Request, error) {
|
||||
req.Header.Set("Connection", "close")
|
||||
req.Close = true
|
||||
|
|
@ -230,23 +237,15 @@ func (r *BulkHTTPRequest) fillRequest(req *http.Request, values map[string]inter
|
|||
req.Header[header] = []string{replacer.Replace(value)}
|
||||
}
|
||||
|
||||
// Set some headers only if the header wasn't supplied by the user
|
||||
if _, ok := req.Header["User-Agent"]; !ok {
|
||||
req.Header.Set("User-Agent", "Nuclei - Open-source project (github.com/projectdiscovery/nuclei)")
|
||||
}
|
||||
setHeader(req, "User-Agent", "Nuclei - Open-source project (github.com/projectdiscovery/nuclei)")
|
||||
|
||||
// raw requests are left untouched
|
||||
if len(r.Raw) > 0 {
|
||||
return retryablehttp.FromRequest(req)
|
||||
}
|
||||
|
||||
if _, ok := req.Header["Accept"]; !ok {
|
||||
req.Header.Set("Accept", "*/*")
|
||||
}
|
||||
|
||||
if _, ok := req.Header["Accept-Language"]; !ok {
|
||||
req.Header.Set("Accept-Language", "en")
|
||||
}
|
||||
setHeader(req, "Accept", "*/*")
|
||||
setHeader(req, "Accept-Language", "en")
|
||||
|
||||
return retryablehttp.FromRequest(req)
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in New Issue