daffainfo
/
nuclei
mirror of https://github.com/daffainfo/nuclei.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'dev' into patch-4

dev
forgedhallpass 2021-09-16 20:35:43 +03:00 committed by GitHub
parent 0d31e965ef 818838ade6
commit c5a1753548
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
82 changed files with 446 additions and 512 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
Dockerfile
View File

@ -1,4 +1,4 @@
FROM golang:1.17.0-alpine as build-env
FROM golang:1.17.1-alpine as build-env
RUN GO111MODULE=on go get -v github.com/projectdiscovery/nuclei/v2/cmd/nuclei
FROM alpine:3.14

8
README.md
View File

@ -219,7 +219,7 @@ Nuclei offers great number of features that are helpful for security engineers t
<tr>
<td>
**For bugbounty hunters:**
**For Bug Bounty hunters:**
Nuclei allows you to customise your testing approach with your own suite of checks and easily run across your bug bounty programs. Moreover, Nuclei can be easily integrated into any continuous scanning workflow.
@ -237,9 +237,9 @@ Please check our other open-source projects that might fit into your bug bounty
<tr>
<td>
**For pentesters:**
**For Penetration Testers:**
Nuclei immensely improve how you approach security assessment by augmenting the manual repetitve processes. Consultancies are already converting their manual assessment steps with Nuclei, it allows them to run set of their custom assessment approach across thousands of hosts in an automated manner.
Nuclei immensely improve how you approach security assessment by augmenting the manual, repetitive processes. Consultancies are already converting their manual assessment steps with Nuclei, it allows them to run set of their custom assessment approach across thousands of hosts in an automated manner.
Pen-testers get the full power of our public templates and customization capabilities to speed-up their assessment process, and specifically with the regression cycle where you can easily verify the fix.
@ -254,7 +254,7 @@ Pen-testers get the full power of our public templates and customization capabil
# For Developers and Organisations
Nuclei is built with simplicity in mind, with the community backed templates by hundreds of security researchers, it allows you to stay updated with latest security threats using continuous Nuclei scanning on the hosts. It is designed to be easily integrated into regression tests cycle, to verify the fixes and eliminate vulnerabilities from occurring in future.
Nuclei is built with simplicity in mind, with the community backed templates by hundreds of security researchers, it allows you to stay updated with the latest security threats using continuous Nuclei scanning on the hosts. It is designed to be easily integrated into regression tests cycle, to verify the fixes and eliminate vulnerabilities from occurring in the future.
- **CI/CD:** Engineers are already utilising Nuclei within their CI/CD pipeline, it allows them to constantly monitor their staging and production environments with customised templates.
- **Continuous Regression Cycle:** With Nuclei, you can create your custom template on every new identified vulnerability and put into Nuclei engine to eliminate in the continuous regression cycle.

2
SECURITY.md
View File

@ -2,4 +2,4 @@
## Reporting a Vulnerability
DO NOT CREATE AN ISSUE to report a security problem. Instead, please send an email to security@projectdiscovery.io and we will acknowledge it within 3 working days.
DO NOT CREATE AN ISSUE to report a security problem. Instead, please send an email to security@projectdiscovery.io, and we will acknowledge it within 3 working days.

34
SYNTAX-REFERENCE.md
View File

@ -415,12 +415,12 @@ Valid values:
<div class="dd">
<code>additional-fields</code> <i>map[string]string</i>
<code>metadata</code> <i>map[string]string</i>
</div>
<div class="dt">
AdditionalFields regarding metadata of the template.
Metadata of the template.
@ -428,7 +428,7 @@ Examples:
```yaml
additional-fields:
metadata:
customField1: customValue1
```
@ -679,7 +679,7 @@ Matchers contains the detection mechanism for the request to identify
whether the request was successful by doing pattern matching
on request/responses.
Multiple matchers can be combined together with `matcher-condition` flag
Multiple matchers can be combined with `matcher-condition` flag
which accepts either `and` or `or` as argument.
</div>
@ -792,7 +792,7 @@ raw:
</div>
<div class="dt">
ID is the the optional id of the request
ID is the optional id of the request
</div>
@ -869,6 +869,8 @@ Valid values:
- <code>TRACE</code>
- <code>PATCH</code>
- <code>PURGE</code>
</div>
<hr />
@ -1666,7 +1668,7 @@ group: 1
description: |
kval contains the key-value pairs present in the HTTP response header.
kval extractor can be used to extract HTTP response header and cookie key-value pairs.
kval extractor inputs are case insensitive, and does not support dash (-) in input which can replaced with underscores (_)
kval extractor inputs are case-insensitive, and does not support dash (-) in input which can replaced with underscores (_)
For example, Content-Type should be replaced with content_type
A list of supported parts is available in docs for request types.
@ -1844,7 +1846,7 @@ Matchers contains the detection mechanism for the request to identify
whether the request was successful by doing pattern matching
on request/responses.
Multiple matchers can be combined together with `matcher-condition` flag
Multiple matchers can be combined with `matcher-condition` flag
which accepts either `and` or `or` as argument.
</div>
@ -1892,7 +1894,7 @@ Valid values:
</div>
<div class="dt">
ID is the the optional id of the request
ID is the optional id of the request
</div>
@ -2059,7 +2061,7 @@ Matchers contains the detection mechanism for the request to identify
whether the request was successful by doing pattern matching
on request/responses.
Multiple matchers can be combined together with `matcher-condition` flag
Multiple matchers can be combined with `matcher-condition` flag
which accepts either `and` or `or` as argument.
</div>
@ -2162,7 +2164,7 @@ denylist:
</div>
<div class="dt">
ID is the the optional id of the request
ID is the optional id of the request
</div>
@ -2242,7 +2244,7 @@ matchers:
</div>
<div class="dt">
ID is the the optional id of the request
ID is the optional id of the request
</div>
@ -2366,7 +2368,7 @@ Matchers contains the detection mechanism for the request to identify
whether the request was successful by doing pattern matching
on request/responses.
Multiple matchers can be combined together with `matcher-condition` flag
Multiple matchers can be combined with `matcher-condition` flag
which accepts either `and` or `or` as argument.
</div>
@ -2548,7 +2550,7 @@ Appears in:
</div>
<div class="dt">
ID is the the optional id of the request
ID is the optional id of the request
</div>
@ -2578,7 +2580,7 @@ Matchers contains the detection mechanism for the request to identify
whether the request was successful by doing pattern matching
on request/responses.
Multiple matchers can be combined together with `matcher-condition` flag
Multiple matchers can be combined with `matcher-condition` flag
which accepts either `and` or `or` as argument.
</div>
@ -2823,7 +2825,7 @@ Matchers perform name based matching to run subtemplates for a workflow.
</div>
<div class="dt">
Subtemplates are ran if the `template` field Template matches.
Subtemplates are run if the `template` field Template matches.
</div>
@ -2864,7 +2866,7 @@ Name is the name of the item to match.
</div>
<div class="dt">
Subtemplates are ran if the name of matcher matches.
Subtemplates are run if the name of matcher matches.
</div>

6
nuclei-jsonschema.json
View File

@ -69,7 +69,7 @@
"$schema": "http://json-schema.org/draft-04/schema#",
"$ref": "#/definitions/severity.Holder"
},
"additional-fields": {
"metadata": {
"patternProperties": {
".*": {
"type": "string"
@ -207,6 +207,7 @@
"size",
"word",
"regex",
"binary",
"dsl"
],
"type": "string",
@ -615,7 +616,8 @@
"CONNECT",
"OPTIONS",
"TRACE",
"PATCH"
"PATCH",
"PURGE"
],
"type": "string",
"title": "method is the http request method",

5
v2/cmd/cve-annotate/main.go
View File

@ -75,6 +75,10 @@ func getCVEData(client *nvd.Client, filePath, data string) {
}
severityValue := severityMatches[0][1]
// Skip if there's classification data already
if strings.Contains(data, "classification:") {
return
}
cveItem, err := client.FetchCVE(cveName)
if err != nil {
log.Printf("Could not fetch cve %s: %s\n", cveName, err)
@ -110,7 +114,6 @@ func getCVEData(client *nvd.Client, filePath, data string) {
newInfoBlock = strings.ReplaceAll(newInfoBlock, severityMatches[0][0], "severity: "+newSeverity)
fmt.Printf("Adjusting severity for %s from %s=>%s (%.2f)\n", filePath, severityValue, newSeverity, cvssScore)
}
// Start with additional-fields as that is the one most likely to break stuff.
if !strings.Contains(infoBlockClean, "classification") && (cvssScore != 0 && cvssMetrics != "") {
changed = true
newInfoBlock = newInfoBlock + fmt.Sprintf("\n classification:\n cvss-metrics: %s\n cvss-score: %.2f\n cve-id: %s", cvssMetrics, cvssScore, cveName)

3
v2/cmd/functional-test/main.go
View File

@ -10,6 +10,7 @@ import (
"github.com/logrusorgru/aurora"
"github.com/pkg/errors"
"github.com/projectdiscovery/nuclei/v2/internal/testutils"
)
@ -75,5 +76,5 @@ func runIndividualTestCase(testcase string) error {
if mainOutput == devOutput {
return nil
}
return fmt.Errorf("%s main is not equal to %s dev", mainOutput, devOutput)
return fmt.Errorf("%s main is not equal to %s dev", mainOutput, devOutput)
}

2
v2/cmd/integration-test/dns.go
View File

@ -10,7 +10,7 @@ var dnsTestCases = map[string]testutils.TestCase{
type dnsBasic struct{}
// Executes executes a test case and returns an error if occurred
// Execute executes a test case and returns an error if occurred
func (h *dnsBasic) Execute(filePath string) error {
var routerErr error

30
v2/cmd/integration-test/http.go
View File

@ -44,7 +44,7 @@ func httpDebugRequestDump(r *http.Request) {
type httpGetHeaders struct{}
// Executes executes a test case and returns an error if occurred
// Execute executes a test case and returns an error if occurred
func (h *httpGetHeaders) Execute(filePath string) error {
router := httprouter.New()
router.GET("/", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
@ -68,7 +68,7 @@ func (h *httpGetHeaders) Execute(filePath string) error {
type httpGetQueryString struct{}
// Executes executes a test case and returns an error if occurred
// Execute executes a test case and returns an error if occurred
func (h *httpGetQueryString) Execute(filePath string) error {
router := httprouter.New()
router.GET("/", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
@ -92,7 +92,7 @@ func (h *httpGetQueryString) Execute(filePath string) error {
type httpGetRedirects struct{}
// Executes executes a test case and returns an error if occurred
// Execute executes a test case and returns an error if occurred
func (h *httpGetRedirects) Execute(filePath string) error {
router := httprouter.New()
router.GET("/", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
@ -118,7 +118,7 @@ func (h *httpGetRedirects) Execute(filePath string) error {
type httpGet struct{}
// Executes executes a test case and returns an error if occurred
// Execute executes a test case and returns an error if occurred
func (h *httpGet) Execute(filePath string) error {
router := httprouter.New()
router.GET("/", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
@ -140,7 +140,7 @@ func (h *httpGet) Execute(filePath string) error {
type httpPostBody struct{}
// Executes executes a test case and returns an error if occurred
// Execute executes a test case and returns an error if occurred
func (h *httpPostBody) Execute(filePath string) error {
router := httprouter.New()
var routerErr error
@ -173,7 +173,7 @@ func (h *httpPostBody) Execute(filePath string) error {
type httpPostJSONBody struct{}
// Executes executes a test case and returns an error if occurred
// Execute executes a test case and returns an error if occurred
func (h *httpPostJSONBody) Execute(filePath string) error {
router := httprouter.New()
var routerErr error
@ -212,7 +212,7 @@ func (h *httpPostJSONBody) Execute(filePath string) error {
type httpPostMultipartBody struct{}
// Executes executes a test case and returns an error if occurred
// Execute executes a test case and returns an error if occurred
func (h *httpPostMultipartBody) Execute(filePath string) error {
router := httprouter.New()
var routerErr error
@ -255,7 +255,7 @@ func (h *httpPostMultipartBody) Execute(filePath string) error {
type httpRawDynamicExtractor struct{}
// Executes executes a test case and returns an error if occurred
// Execute executes a test case and returns an error if occurred
func (h *httpRawDynamicExtractor) Execute(filePath string) error {
router := httprouter.New()
var routerErr error
@ -294,7 +294,7 @@ func (h *httpRawDynamicExtractor) Execute(filePath string) error {
type httpRawGetQuery struct{}
// Executes executes a test case and returns an error if occurred
// Execute executes a test case and returns an error if occurred
func (h *httpRawGetQuery) Execute(filePath string) error {
router := httprouter.New()
var routerErr error
@ -323,7 +323,7 @@ func (h *httpRawGetQuery) Execute(filePath string) error {
type httpRawGet struct{}
// Executes executes a test case and returns an error if occurred
// Execute executes a test case and returns an error if occurred
func (h *httpRawGet) Execute(filePath string) error {
router := httprouter.New()
var routerErr error
@ -351,7 +351,7 @@ func (h *httpRawGet) Execute(filePath string) error {
type httpRawPayload struct{}
// Executes executes a test case and returns an error if occurred
// Execute executes a test case and returns an error if occurred
func (h *httpRawPayload) Execute(filePath string) error {
router := httprouter.New()
var routerErr error
@ -387,7 +387,7 @@ func (h *httpRawPayload) Execute(filePath string) error {
type httpRawPostBody struct{}
// Executes executes a test case and returns an error if occurred
// Execute executes a test case and returns an error if occurred
func (h *httpRawPostBody) Execute(filePath string) error {
router := httprouter.New()
var routerErr error
@ -420,7 +420,7 @@ func (h *httpRawPostBody) Execute(filePath string) error {
type httpRawCookieReuse struct{}
// Executes executes a test case and returns an error if occurred
// Execute executes a test case and returns an error if occurred
func (h *httpRawCookieReuse) Execute(filePath string) error {
router := httprouter.New()
var routerErr error
@ -469,7 +469,7 @@ func (h *httpRawCookieReuse) Execute(filePath string) error {
type httpRawUnsafeRequest struct{}
// Executes executes a test case and returns an error if occurred
// Execute executes a test case and returns an error if occurred
func (h *httpRawUnsafeRequest) Execute(filePath string) error {
var routerErr error
@ -494,7 +494,7 @@ func (h *httpRawUnsafeRequest) Execute(filePath string) error {
type httpRequestCondition struct{}
// Executes executes a test case and returns an error if occurred
// Execute executes a test case and returns an error if occurred
func (h *httpRequestCondition) Execute(filePath string) error {
router := httprouter.New()
var routerErr error

3
v2/cmd/integration-test/integration-test.go
View File

@ -35,8 +35,7 @@ func main() {
if customTest != "" && !strings.Contains(file, customTest) {
continue // only run tests user asked
}
err := test.Execute(file)
if err != nil {
if err := test.Execute(file); err != nil {
fmt.Fprintf(os.Stderr, "%s Test \"%s\" failed: %s\n", failed, file, err)
errored = true
} else {

4
v2/cmd/integration-test/network.go
View File

@ -14,7 +14,7 @@ var networkTestcases = map[string]testutils.TestCase{
type networkBasic struct{}
// Executes executes a test case and returns an error if occurred
// Execute executes a test case and returns an error if occurred
func (h *networkBasic) Execute(filePath string) error {
var routerErr error
@ -47,7 +47,7 @@ func (h *networkBasic) Execute(filePath string) error {
type networkMultiStep struct{}
// Executes executes a test case and returns an error if occurred
// Execute executes a test case and returns an error if occurred
func (h *networkMultiStep) Execute(filePath string) error {
var routerErr error

25
v2/cmd/integration-test/workflow.go
View File

@ -6,6 +6,7 @@ import (
"net/http/httptest"
"github.com/julienschmidt/httprouter"
"github.com/projectdiscovery/nuclei/v2/internal/testutils"
)
@ -18,13 +19,13 @@ var workflowTestcases = map[string]testutils.TestCase{
type workflowBasic struct{}
// Executes executes a test case and returns an error if occurred
// Execute executes a test case and returns an error if occurred
func (h *workflowBasic) Execute(filePath string) error {
router := httprouter.New()
router.GET("/", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
httpDebugRequestDump(r)
fmt.Fprintf(w, "This is test matcher text")
}))
})
ts := httptest.NewServer(router)
defer ts.Close()
@ -40,13 +41,13 @@ func (h *workflowBasic) Execute(filePath string) error {
type workflowConditionMatched struct{}
// Executes executes a test case and returns an error if occurred
// Execute executes a test case and returns an error if occurred
func (h *workflowConditionMatched) Execute(filePath string) error {
router := httprouter.New()
router.GET("/", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
httpDebugRequestDump(r)
fmt.Fprintf(w, "This is test matcher text")
}))
})
ts := httptest.NewServer(router)
defer ts.Close()
@ -62,13 +63,13 @@ func (h *workflowConditionMatched) Execute(filePath string) error {
type workflowConditionUnmatch struct{}
// Executes executes a test case and returns an error if occurred
// Execute executes a test case and returns an error if occurred
func (h *workflowConditionUnmatch) Execute(filePath string) error {
router := httprouter.New()
router.GET("/", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
httpDebugRequestDump(r)
fmt.Fprintf(w, "This is test matcher text")
}))
})
ts := httptest.NewServer(router)
defer ts.Close()
@ -84,13 +85,13 @@ func (h *workflowConditionUnmatch) Execute(filePath string) error {
type workflowMatcherName struct{}
// Executes executes a test case and returns an error if occurred
// Execute executes a test case and returns an error if occurred
func (h *workflowMatcherName) Execute(filePath string) error {
router := httprouter.New()
router.GET("/", httprouter.Handle(func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
router.GET("/", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
httpDebugRequestDump(r)
fmt.Fprintf(w, "This is test matcher text")
}))
})
ts := httptest.NewServer(router)
defer ts.Close()

7
v2/go.mod
View File

@ -19,7 +19,7 @@ require (
github.com/gosuri/uilive v0.0.4 // indirect
github.com/gosuri/uiprogress v0.0.1 // indirect
github.com/itchyny/gojq v0.12.4
github.com/json-iterator/go v1.1.11
github.com/json-iterator/go v1.1.12
github.com/julienschmidt/httprouter v1.3.0
github.com/karlseguin/ccache v2.0.3+incompatible
github.com/karrick/godirwalk v1.16.1
@ -35,6 +35,7 @@ require (
github.com/projectdiscovery/gologger v1.1.4
github.com/projectdiscovery/hmap v0.0.2-0.20210616215655-7b78e7f33d1f
github.com/projectdiscovery/interactsh v0.0.4
github.com/projectdiscovery/nuclei-updatecheck-api v0.0.0-20210914222811-0a072d262f77
github.com/projectdiscovery/rawhttp v0.0.7
github.com/projectdiscovery/retryabledns v1.0.12
github.com/projectdiscovery/retryablehttp-go v1.0.2
@ -51,8 +52,6 @@ require (
github.com/tj/go-update v2.2.5-0.20200519121640-62b4b798fd68+incompatible
github.com/valyala/fasttemplate v1.2.1
github.com/xanzy/go-gitlab v0.50.3
github.com/ysmood/got v0.14.1 // indirect
github.com/ysmood/gotrace v0.2.2 // indirect
github.com/ysmood/gson v0.6.4 // indirect
github.com/ysmood/leakless v0.7.0 // indirect
go.uber.org/atomic v1.9.0
@ -102,7 +101,7 @@ require (
github.com/klauspost/pgzip v1.2.5 // indirect
github.com/mattn/go-isatty v0.0.13 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.1 // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/projectdiscovery/iputil v0.0.0-20210429152401-c18a5408ca46 // indirect
github.com/projectdiscovery/mapcidr v0.0.6 // indirect

10
v2/go.sum
View File

@ -224,6 +224,7 @@ github.com/hashicorp/go-hclog v0.9.2 h1:CG6TE5H9/JXsFWJCfoIVpKFIkFe6ysEuHirp4DxC
github.com/hashicorp/go-hclog v0.9.2/go.mod h1:5CU+agLiy3J7N7QjHK5d05KxGsuXiQLrjA0H7acj2lQ=
github.com/hashicorp/go-retryablehttp v0.6.8 h1:92lWxgpa+fF3FozM4B3UZtHZMJX8T5XT+TFdCxsPyWs=
github.com/hashicorp/go-retryablehttp v0.6.8/go.mod h1:vAew36LZh98gCBJNLH42IQ1ER/9wtLZZ8meHqQvEYWY=
github.com/hashicorp/go-version v1.3.0/go.mod h1:fltr4n8CU8Ke44wwGCBoEymUuxUHl09ZGVZPK5anwXA=
github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
@ -244,8 +245,9 @@ github.com/jasonlvhit/gocron v0.0.1/go.mod h1:k9a3TV8VcU73XZxfVHCHWMWF9SOqgoku0/
github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k=
github.com/jpillora/backoff v0.0.0-20180909062703-3050d21c67d7/go.mod h1:2iMrUgbbvHEiQClaW2NsSzMyGHqN+rDFqY705q49KG0=
github.com/json-iterator/go v1.1.10/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/json-iterator/go v1.1.11 h1:uVUAXhF2To8cbw/3xN3pxj6kk7TYKs98NIrTqPlMWAQ=
github.com/json-iterator/go v1.1.11/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU=
github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk=
github.com/julienschmidt/httprouter v1.3.0 h1:U0609e9tgbseu3rBINet9P48AI/D3oJs4dN7jwJOQ1U=
@ -298,8 +300,9 @@ github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJ
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI=
github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M=
github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
github.com/ngdinhtoan/glide-cleanup v0.2.0/go.mod h1:UQzsmiDOb8YV3nOsCxK/c9zPpCZVNoHScRE3EO9pVMM=
github.com/nxadm/tail v1.4.4/go.mod h1:kenIhsEOeOJmVchQTgglprH7qJGnHDVpk1VPCcaMI8A=
github.com/nxadm/tail v1.4.8 h1:nPr65rt6Y5JFSKQO7qToXr7pePgD6Gwiw05lkbyAQTE=
@ -350,6 +353,9 @@ github.com/projectdiscovery/mapcidr v0.0.6 h1:RRIrqNakUEF/pstIXWTD6yvCMF9N6SnOb9
github.com/projectdiscovery/mapcidr v0.0.6/go.mod h1:ZEBhMmBU3laUl3g9QGTrzJku1VJOzjdFwW01f/zVVzM=
github.com/projectdiscovery/networkpolicy v0.0.1 h1:RGRuPlxE8WLFF9tdKSjTsYiTIKHNHW20Kl0nGGiRb1I=
github.com/projectdiscovery/networkpolicy v0.0.1/go.mod h1:asvdg5wMy3LPVMGALatebKeOYH5n5fV5RCTv6DbxpIs=
github.com/projectdiscovery/nuclei-updatecheck-api v0.0.0-20210914222811-0a072d262f77 h1:SNtAiRRrJtDJJDroaa/bFXt/Tix2LA6+rHRib0ORlJQ=
github.com/projectdiscovery/nuclei-updatecheck-api v0.0.0-20210914222811-0a072d262f77/go.mod h1:pxWVDgq88t9dWv4+J2AIaWgY+EqOE1AyfHS0Tn23w4M=
github.com/projectdiscovery/nuclei/v2 v2.5.1/go.mod h1:sU2qcY0MQFS0CqP1BgkR8ZnUyFhqK0BdnY6bvTKNjXY=
github.com/projectdiscovery/rawhttp v0.0.7 h1:5m4peVgjbl7gqDcRYMTVEuX+Xs/nh76ohTkkvufucLg=
github.com/projectdiscovery/rawhttp v0.0.7/go.mod h1:PQERZAhAv7yxI/hR6hdDPgK1WTU56l204BweXrBec+0=
github.com/projectdiscovery/retryabledns v1.0.11/go.mod h1:4sMC8HZyF01HXukRleSQYwz4870bwgb4+hTSXTMrkf4=

19
v2/internal/runner/options.go
View File

@ -21,6 +21,11 @@ func ParseOptions(options *types.Options) {
// Check if stdin pipe was given
options.Stdin = hasStdin()
// if VerboseVerbose is set, it implicitly enables the Verbose option as well
if options.VerboseVerbose {
options.Verbose = true
}
// Read the inputs and configure the logging
configureOutput(options)
@ -36,7 +41,7 @@ func ParseOptions(options *types.Options) {
if err != nil {
gologger.Fatal().Msgf("Could not read template configuration: %s\n", err)
}
gologger.Info().Msgf("Current nuclei-templates version: %s (%s)\n", configuration.CurrentVersion, configuration.TemplatesDirectory)
gologger.Info().Msgf("Current nuclei-templates version: %s (%s)\n", configuration.TemplateVersion, configuration.TemplatesDirectory)
os.Exit(0)
}
@ -77,19 +82,15 @@ func hasStdin() bool {
// validateOptions validates the configuration options passed
func validateOptions(options *types.Options) error {
// Both verbose and silent flags were used
if options.Verbose && options.Silent {
return errors.New("both verbose and silent mode specified")
}
// Validate proxy options if provided
err := validateProxyURL(options.ProxyURL, "invalid http proxy format (It should be http://username:password@host:port)")
if err != nil {
if err := validateProxyURL(options.ProxyURL, "invalid http proxy format (It should be http://username:password@host:port)"); err != nil {
return err
}
err = validateProxyURL(options.ProxySocksURL, "invalid socks proxy format (It should be socks5://username:password@host:port)")
if err != nil {
if err := validateProxyURL(options.ProxySocksURL, "invalid socks proxy format (It should be socks5://username:password@host:port)"); err != nil {
return err
}
@ -113,10 +114,10 @@ func isValidURL(urlString string) bool {
return err == nil
}
// configureOutput configures the output on the screen
// configureOutput configures the output logging levels to be displayed on the screen
func configureOutput(options *types.Options) {
// If the user desires verbose output, show verbose output
if options.Verbose {
if options.Verbose || options.VerboseVerbose {
gologger.DefaultLogger.SetMaxLevel(levels.LevelVerbose)
}
if options.Debug {

2
v2/internal/runner/processor.go
View File

@ -7,7 +7,7 @@ import (
"go.uber.org/atomic"
)
// processTemplateWithList process a template on the URL list
// processTemplateWithList execute a template against the list of user provided targets
func (r *Runner) processTemplateWithList(template *templates.Template) bool {
results := &atomic.Bool{}
wg := sizedwaitgroup.New(r.options.BulkSize)

14
v2/internal/runner/runner.go
View File

@ -208,7 +208,7 @@ func New(options *types.Options) (*Runner, error) {
return nil, progressErr
}
// create project file if requested or load existing one
// create project file if requested or load the existing one
if options.Project {
var projectFileErr error
runner.projectFile, projectFileErr = projectfile.New(&projectfile.Options{Path: options.ProjectPath, Cleanup: utils.IsBlank(options.ProjectPath)})
@ -298,7 +298,7 @@ func (r *Runner) Close() {
func (r *Runner) RunEnumeration() error {
defer r.Close()
// If user asked for new templates to be executed, collect the list from template directory.
// If user asked for new templates to be executed, collect the list from the templates' directory.
if r.options.NewTemplates {
templatesLoaded, err := r.readNewTemplatesFile()
if err != nil {
@ -367,7 +367,7 @@ func (r *Runner) RunEnumeration() error {
return nil // exit
}
// Display stats for any loaded templates syntax warnings or errors
// Display stats for any loaded templates' syntax warnings or errors
stats.Display(parsers.SyntaxWarningStats)
stats.Display(parsers.SyntaxErrorStats)
@ -392,7 +392,7 @@ func (r *Runner) RunEnumeration() error {
if r.templatesConfig != nil && r.templatesConfig.NucleiTemplatesLatestVersion != "" { // TODO extract duplicated logic
builder.WriteString(" (")
if r.templatesConfig.CurrentVersion == r.templatesConfig.NucleiTemplatesLatestVersion {
if r.templatesConfig.TemplateVersion == r.templatesConfig.NucleiTemplatesLatestVersion {
builder.WriteString(r.colorizer.Green("latest").String())
} else {
builder.WriteString(r.colorizer.Red("outdated").String())
@ -403,7 +403,7 @@ func (r *Runner) RunEnumeration() error {
builder.Reset()
if r.templatesConfig != nil {
gologger.Info().Msgf("Using Nuclei Templates %s%s", r.templatesConfig.CurrentVersion, messageStr)
gologger.Info().Msgf("Using Nuclei Templates %s%s", r.templatesConfig.TemplateVersion, messageStr)
}
if r.interactsh != nil {
gologger.Info().Msgf("Using Interactsh Server %s", r.options.InteractshURL)
@ -489,7 +489,7 @@ func (r *Runner) RunEnumeration() error {
// 0 matches means no templates were found in directory
if templateCount == 0 {
return errors.New("no templates were found")
return errors.New("no valid templates were found")
}
/*
@ -561,7 +561,7 @@ func (r *Runner) readNewTemplatesFile() ([]string, error) {
return templatesList, nil
}
// readNewTemplatesFile reads newly added templates from directory if it exists
// countNewTemplates returns the number of newly added templates
func (r *Runner) countNewTemplates() int {
if r.templatesConfig == nil {
return 0

24
v2/internal/runner/templates.go
View File

@ -13,30 +13,24 @@ import (
"github.com/projectdiscovery/nuclei/v2/pkg/types"
)
func (r *Runner) templateLogMsg(id, name, author string, templateSeverity severity.Severity) string {
func (r *Runner) templateLogMsg(id, name string, authors []string, templateSeverity severity.Severity) string {
// Display the message for the template
return fmt.Sprintf("[%s] %s (%s) [%s]",
r.colorizer.BrightBlue(id).String(),
r.colorizer.Bold(name).String(),
r.colorizer.BrightYellow(appendAtSignToAuthors(author)).String(),
r.colorizer.BrightYellow(appendAtSignToAuthors(authors)).String(),
r.addColor(templateSeverity))
}
// appendAtSignToAuthors appends @ before each author and returns final string
func appendAtSignToAuthors(author string) string {
authors := strings.Split(author, ",")
// appendAtSignToAuthors appends @ before each author and returns the final string
func appendAtSignToAuthors(authors []string) string {
if len(authors) == 0 {
return "@none"
}
if len(authors) == 1 {
if !strings.HasPrefix(authors[0], "@") {
return fmt.Sprintf("@%s", authors[0])
}
return authors[0]
}
values := make([]string, 0, len(authors))
for _, k := range authors {
if !strings.HasPrefix(authors[0], "@") {
if !strings.HasPrefix(k, "@") {
values = append(values, fmt.Sprintf("@%s", k))
} else {
values = append(values, k)
@ -52,12 +46,12 @@ func (r *Runner) logAvailableTemplate(tplPath string) {
} else {
gologger.Print().Msgf("%s\n", r.templateLogMsg(t.ID,
types.ToString(t.Info.Name),
types.ToString(t.Info.Authors),
t.Info.Authors.ToSlice(),
t.Info.SeverityHolder.Severity))
}
}
// ListAvailableTemplates prints available templates to stdout
// listAvailableTemplates prints available templates to stdout
func (r *Runner) listAvailableTemplates() {
if r.templatesConfig == nil {
return
@ -70,7 +64,7 @@ func (r *Runner) listAvailableTemplates() {
gologger.Print().Msgf(
"\nListing available v.%s nuclei templates for %s",
r.templatesConfig.CurrentVersion,
r.templatesConfig.TemplateVersion,
r.templatesConfig.TemplatesDirectory,
)
err := directoryWalker(

25
v2/internal/runner/templates_test.go Normal file
View File

@ -0,0 +1,25 @@
package runner
import (
"testing"
"github.com/stretchr/testify/assert"
)
func Test_appendAtSignToAuthors(t *testing.T) {
result := appendAtSignToAuthors([]string{"user1", "user2", "user3"})
assert.Equal(t, result, "@user1,@user2,@user3")
}
func Test_appendAtSignToMissingAuthors(t *testing.T) {
result := appendAtSignToAuthors([]string{})
assert.Equal(t, result, "@none")
result = appendAtSignToAuthors(nil)
assert.Equal(t, result, "@none")
}
func Test_appendAtSignToOneAuthor(t *testing.T) {
result := appendAtSignToAuthors([]string{"user1"})
assert.Equal(t, result, "@user1")
}

313
v2/internal/runner/update.go
View File

@ -7,7 +7,6 @@ import (
"context"
"crypto/md5"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"io/ioutil"
@ -18,14 +17,15 @@ import (
"runtime"
"strconv"
"strings"
"time"
"github.com/apex/log"
"github.com/blang/semver"
"github.com/google/go-github/github"
"github.com/olekukonko/tablewriter"
"github.com/pkg/errors"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei-updatecheck-api/client"
"github.com/projectdiscovery/nuclei/v2/pkg/catalog/config"
"github.com/tj/go-update"
@ -38,17 +38,16 @@ const (
repoName = "nuclei-templates"
nucleiIgnoreFile = ".nuclei-ignore"
nucleiConfigFilename = ".templates-config.json"
defaultIgnoreURL = "https://raw.githubusercontent.com/projectdiscovery/nuclei-templates/master/.nuclei-ignore"
)
var reVersion = regexp.MustCompile(`\d+\.\d+\.\d+`)
// updateTemplates checks if the default list of nuclei-templates
// exist in the users home directory, if not the latest revision
// is downloaded from github.
// exist in the user's home directory, if not the latest revision
// is downloaded from GitHub.
//
// If the path exists but is not latest, the new version is downloaded
// from github and replaced with the templates directory.
// If the path exists but does not contain the latest version of public templates,
// the new version is downloaded from GitHub to the templates' directory, overwriting the old content.
func (r *Runner) updateTemplates() error {
home, err := os.UserHomeDir()
if err != nil {
@ -61,75 +60,69 @@ func (r *Runner) updateTemplates() error {
return errors.Wrap(err, "could not read configuration file")
}
// If the config doesn't exist, write it now.
// If the config doesn't exist, create it now.
if r.templatesConfig == nil {
currentConfig := &config.Config{
TemplatesDirectory: filepath.Join(home, "nuclei-templates"),
IgnoreURL: defaultIgnoreURL,
NucleiVersion: config.Version,
}
if writeErr := config.WriteConfiguration(currentConfig, false, false); writeErr != nil {
if writeErr := config.WriteConfiguration(currentConfig); writeErr != nil {
return errors.Wrap(writeErr, "could not write template configuration")
}
r.templatesConfig = currentConfig
}
if r.options.NoUpdateTemplates {
if r.options.NoUpdateTemplates && !r.options.UpdateTemplates {
return nil
}
// Check if last checked for nuclei-ignore is more than 1 hours.
// and if true, run the check.
//
// Also at the same time fetch latest version from github to do outdated nuclei
// and templates check.
checkedIgnore := false
if r.templatesConfig == nil || time.Since(r.templatesConfig.LastCheckedIgnore) > 1*time.Hour {
checkedIgnore = r.checkNucleiIgnoreFileUpdates(configDir)
}
client.InitNucleiVersion(config.Version)
r.fetchLatestVersionsFromGithub(configDir) // also fetch the latest versions
ctx := context.Background()
if r.templatesConfig.CurrentVersion == "" || (r.options.TemplatesDirectory != "" && r.templatesConfig.TemplatesDirectory != r.options.TemplatesDirectory) {
var noTemplatesFound bool
if _, err := os.Stat(r.templatesConfig.TemplatesDirectory); os.IsNotExist(err) {
noTemplatesFound = true
}
if r.templatesConfig.TemplateVersion == "" || (r.options.TemplatesDirectory != "" && r.templatesConfig.TemplatesDirectory != r.options.TemplatesDirectory) || noTemplatesFound {
gologger.Info().Msgf("nuclei-templates are not installed, installing...\n")
// Use custom location if user has given a template directory
// Use the custom location if the user has given a template directory
r.templatesConfig = &config.Config{
TemplatesDirectory: filepath.Join(home, "nuclei-templates"),
}
if r.options.TemplatesDirectory != "" && r.options.TemplatesDirectory != filepath.Join(home, "nuclei-templates") {
r.templatesConfig.TemplatesDirectory, _ = filepath.Abs(r.options.TemplatesDirectory)
}
r.fetchLatestVersionsFromGithub(configDir) // also fetch the latest versions
// Download the repository and also write the revision to a HEAD file.
version, asset, getErr := r.getLatestReleaseFromGithub()
version, err := semver.Parse(r.templatesConfig.NucleiTemplatesLatestVersion)
if err != nil {
return err
}
// Download the repository and write the revision to a HEAD file.
asset, getErr := r.getLatestReleaseFromGithub(r.templatesConfig.NucleiTemplatesLatestVersion)
if getErr != nil {
return getErr
}
gologger.Verbose().Msgf("Downloading nuclei-templates (v%s) to %s\n", version.String(), r.templatesConfig.TemplatesDirectory)
r.fetchLatestVersionsFromGithub() // also fetch latest versions
_, err = r.downloadReleaseAndUnzip(ctx, version.String(), asset.GetZipballURL())
if err != nil {
if _, err := r.downloadReleaseAndUnzip(ctx, version.String(), asset.GetZipballURL()); err != nil {
return err
}
r.templatesConfig.CurrentVersion = version.String()
r.templatesConfig.TemplateVersion = version.String()
err = config.WriteConfiguration(r.templatesConfig, true, checkedIgnore)
if err != nil {
if err := config.WriteConfiguration(r.templatesConfig); err != nil {
return err
}
gologger.Info().Msgf("Successfully downloaded nuclei-templates (v%s). GoodLuck!\n", version.String())
return nil
}
// Check if last checked is more than 24 hours and we don't have updateTemplates flag.
// If not, return since we don't want to do anything now.
if time.Since(r.templatesConfig.LastChecked) < 24*time.Hour && !r.options.UpdateTemplates {
return nil
}
// Get the configuration currently on disk.
verText := r.templatesConfig.CurrentVersion
verText := r.templatesConfig.TemplateVersion
indices := reVersion.FindStringIndex(verText)
if indices == nil {
return fmt.Errorf("invalid release found with tag %s", err)
@ -143,13 +136,16 @@ func (r *Runner) updateTemplates() error {
return err
}
version, asset, err := r.getLatestReleaseFromGithub()
version, err := semver.Parse(r.templatesConfig.NucleiTemplatesLatestVersion)
if err != nil {
return err
}
if version.EQ(oldVersion) {
return config.WriteConfiguration(r.templatesConfig, false, checkedIgnore)
if r.options.UpdateTemplates {
gologger.Info().Msgf("No new updates found for nuclei templates")
}
return config.WriteConfiguration(r.templatesConfig)
}
if version.GT(oldVersion) {
@ -159,16 +155,18 @@ func (r *Runner) updateTemplates() error {
if r.options.TemplatesDirectory != "" {
r.templatesConfig.TemplatesDirectory = r.options.TemplatesDirectory
}
r.templatesConfig.CurrentVersion = version.String()
r.templatesConfig.TemplateVersion = version.String()
gologger.Verbose().Msgf("Downloading nuclei-templates (v%s) to %s\n", version.String(), r.templatesConfig.TemplatesDirectory)
r.fetchLatestVersionsFromGithub()
_, err = r.downloadReleaseAndUnzip(ctx, version.String(), asset.GetZipballURL())
asset, err := r.getLatestReleaseFromGithub(r.templatesConfig.NucleiTemplatesLatestVersion)
if err != nil {
return err
}
err = config.WriteConfiguration(r.templatesConfig, true, checkedIgnore)
if err != nil {
if _, err := r.downloadReleaseAndUnzip(ctx, version.String(), asset.GetZipballURL()); err != nil {
return err
}
if err := config.WriteConfiguration(r.templatesConfig); err != nil {
return err
}
gologger.Info().Msgf("Successfully updated nuclei-templates (v%s). GoodLuck!\n", version.String())
@ -193,77 +191,35 @@ func (r *Runner) readInternalConfigurationFile(home, configDir string) error {
return nil
}
// checkNucleiIgnoreFileUpdates checks .nuclei-ignore file for updates from github
// checkNucleiIgnoreFileUpdates checks .nuclei-ignore file for updates from GitHub
func (r *Runner) checkNucleiIgnoreFileUpdates(configDir string) bool {
ignoreURL := defaultIgnoreURL
if r.templatesConfig != nil && r.templatesConfig.IgnoreURL != "" {
ignoreURL = r.templatesConfig.IgnoreURL
data, err := client.GetLatestIgnoreFile()
if err != nil {
return false
}
gologger.Verbose().Msgf("Downloading config file from %s", ignoreURL)
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
req, reqErr := http.NewRequestWithContext(ctx, http.MethodGet, ignoreURL, nil)
if reqErr == nil {
resp, httpGetErr := http.DefaultClient.Do(req)
if httpGetErr != nil {
if resp != nil && resp.Body != nil {
resp.Body.Close()
}
gologger.Warning().Msgf("Could not get ignore-file from %s: %s", ignoreURL, httpGetErr)
} else {
data, _ := ioutil.ReadAll(resp.Body)
resp.Body.Close()
if len(data) > 0 {
_ = ioutil.WriteFile(filepath.Join(configDir, nucleiIgnoreFile), data, 0644)
}
if r.templatesConfig != nil {
if err := config.WriteConfiguration(r.templatesConfig, false, true); err != nil {
gologger.Warning().Msgf("Could not get ignore-file from %s: %s", ignoreURL, err)
if err := config.WriteConfiguration(r.templatesConfig); err != nil {
gologger.Warning().Msgf("Could not get ignore-file from server: %s", err)
}
}
}
}
cancel()
return true
}
// getLatestReleaseFromGithub returns the latest release from github
func (r *Runner) getLatestReleaseFromGithub() (semver.Version, *github.RepositoryRelease, error) {
// getLatestReleaseFromGithub returns the latest release from GitHub
func (r *Runner) getLatestReleaseFromGithub(latestTag string) (*github.RepositoryRelease, error) {
client := github.NewClient(nil)
rels, _, err := client.Repositories.ListReleases(context.Background(), userName, repoName, nil)
release, _, err := client.Repositories.GetReleaseByTag(context.Background(), userName, repoName, "v"+latestTag)
if err != nil {
return semver.Version{}, nil, err
return nil, err
}
// Find the most recent version based on semantic versioning.
var latestRelease semver.Version
var latestPublish *github.RepositoryRelease
for _, release := range rels {
verText := release.GetTagName()
indices := reVersion.FindStringIndex(verText)
if indices == nil {
return semver.Version{}, nil, fmt.Errorf("invalid release found with tag %s", err)
if release == nil {
return nil, errors.New("no version found for the templates")
}
if indices[0] > 0 {
verText = verText[indices[0]:]
}
ver, err := semver.Make(verText)
if err != nil {
return semver.Version{}, nil, err
}
if latestPublish == nil || ver.GTE(latestRelease) {
latestRelease = ver
latestPublish = release
}
}
if latestPublish == nil {
return semver.Version{}, nil, errors.New("no version found for the templates")
}
return latestRelease, latestPublish, nil
return release, nil
}
// downloadReleaseAndUnzip downloads and unzips the release in a directory
@ -288,18 +244,17 @@ func (r *Runner) downloadReleaseAndUnzip(ctx context.Context, version, downloadU
}
reader := bytes.NewReader(buf)
z, err := zip.NewReader(reader, reader.Size())
zipReader, err := zip.NewReader(reader, reader.Size())
if err != nil {
return nil, fmt.Errorf("failed to uncompress zip file: %s", err)
}
// Create the template folder if it doesn't exists
err = os.MkdirAll(r.templatesConfig.TemplatesDirectory, os.ModePerm)
if err != nil {
// Create the template folder if it doesn't exist
if err := os.MkdirAll(r.templatesConfig.TemplatesDirectory, os.ModePerm); err != nil {
return nil, fmt.Errorf("failed to create template base folder: %s", err)
}
results, err := r.compareAndWriteTemplates(z)
results, err := r.compareAndWriteTemplates(zipReader)
if err != nil {
return nil, fmt.Errorf("failed to write templates: %s", err)
}
@ -308,8 +263,7 @@ func (r *Runner) downloadReleaseAndUnzip(ctx context.Context, version, downloadU
r.printUpdateChangelog(results, version)
}
checksumFile := filepath.Join(r.templatesConfig.TemplatesDirectory, ".checksum")
err = writeTemplatesChecksum(checksumFile, results.checksums)
if err != nil {
if err := writeTemplatesChecksum(checksumFile, results.checksums); err != nil {
return nil, errors.Wrap(err, "could not write checksum")
}
@ -320,8 +274,8 @@ func (r *Runner) downloadReleaseAndUnzip(ctx context.Context, version, downloadU
buffer.WriteString(addition)
buffer.WriteString("\n")
}
err = ioutil.WriteFile(additionsFile, buffer.Bytes(), os.ModePerm)
if err != nil {
if err := ioutil.WriteFile(additionsFile, buffer.Bytes(), os.ModePerm); err != nil {
return nil, errors.Wrap(err, "could not write new additions file")
}
return results, err
@ -335,9 +289,8 @@ type templateUpdateResults struct {
checksums map[string]string
}
// compareAndWriteTemplates compares and returns the stats of a template
// update operations.
func (r *Runner) compareAndWriteTemplates(z *zip.Reader) (*templateUpdateResults, error) {
// compareAndWriteTemplates compares and returns the stats of a template update operations.
func (r *Runner) compareAndWriteTemplates(zipReader *zip.Reader) (*templateUpdateResults, error) {
results := &templateUpdateResults{
checksums: make(map[string]string),
}
@ -348,9 +301,9 @@ func (r *Runner) compareAndWriteTemplates(z *zip.Reader) (*templateUpdateResults
// it is removed. This allows us fine-grained control over the download process
// as well as solves a long problem with nuclei-template updates.
checksumFile := filepath.Join(r.templatesConfig.TemplatesDirectory, ".checksum")
previousChecksum, _ := readPreviousTemplatesChecksum(checksumFile)
for _, file := range z.File {
directory, name := filepath.Split(file.Name)
templateChecksumsMap, _ := createTemplateChecksumsMap(checksumFile)
for _, zipTemplateFile := range zipReader.File {
directory, name := filepath.Split(zipTemplateFile.Name)
if name == "" {
continue
}
@ -362,8 +315,7 @@ func (r *Runner) compareAndWriteTemplates(z *zip.Reader) (*templateUpdateResults
}
results.totalCount++
templateDirectory := filepath.Join(r.templatesConfig.TemplatesDirectory, finalPath)
err := os.MkdirAll(templateDirectory, os.ModePerm)
if err != nil {
if err := os.MkdirAll(templateDirectory, os.ModePerm); err != nil {
return nil, fmt.Errorf("failed to create template folder %s : %s", templateDirectory, err)
}
@ -373,28 +325,27 @@ func (r *Runner) compareAndWriteTemplates(z *zip.Reader) (*templateUpdateResults
if _, statErr := os.Stat(templatePath); os.IsNotExist(statErr) {
isAddition = true
}
f, err := os.OpenFile(templatePath, os.O_TRUNC|os.O_CREATE|os.O_WRONLY, 0777)
templateFile, err := os.OpenFile(templatePath, os.O_TRUNC|os.O_CREATE|os.O_WRONLY, 0777)
if err != nil {
f.Close()
templateFile.Close()
return nil, fmt.Errorf("could not create uncompressed file: %s", err)
}
reader, err := file.Open()
zipTemplateFileReader, err := zipTemplateFile.Open()
if err != nil {
f.Close()
templateFile.Close()
return nil, fmt.Errorf("could not open archive to extract file: %s", err)
}
hasher := md5.New()
// Save file and also read into hasher for md5
_, err = io.Copy(f, io.TeeReader(reader, hasher))
if err != nil {
f.Close()
if _, err := io.Copy(templateFile, io.TeeReader(zipTemplateFileReader, hasher)); err != nil {
templateFile.Close()
return nil, fmt.Errorf("could not write template file: %s", err)
}
f.Close()
templateFile.Close()
oldChecksum, checksumOK := previousChecksum[templatePath]
oldChecksum, checksumOK := templateChecksumsMap[templatePath]
checksum := hex.EncodeToString(hasher.Sum(nil))
if isAddition {
@ -405,57 +356,58 @@ func (r *Runner) compareAndWriteTemplates(z *zip.Reader) (*templateUpdateResults
results.checksums[templatePath] = checksum
}
// If we don't find a previous file in new download and it hasn't been
// changed on the disk, delete it.
for k, v := range previousChecksum {
_, ok := results.checksums[k]
if !ok && v[0] == v[1] {
os.Remove(k)
results.deletions = append(results.deletions, strings.TrimPrefix(strings.TrimPrefix(k, r.templatesConfig.TemplatesDirectory), string(os.PathSeparator)))
// If we don't find the previous file in the newly downloaded list,
// and it hasn't been changed on the disk, delete it.
for templatePath, templateChecksums := range templateChecksumsMap {
_, ok := results.checksums[templatePath]
if !ok && templateChecksums[0] == templateChecksums[1] {
os.Remove(templatePath)
results.deletions = append(results.deletions, strings.TrimPrefix(strings.TrimPrefix(templatePath, r.templatesConfig.TemplatesDirectory), string(os.PathSeparator)))
}
}
return results, nil
}
// readPreviousTemplatesChecksum reads the previous checksum file from the disk.
//
// It reads two checksums, the first checksum is what we expect and the second is
// the actual checksum of the file on disk currently.
func readPreviousTemplatesChecksum(file string) (map[string][2]string, error) {
f, err := os.Open(file)
// createTemplateChecksumsMap reads the previous checksum file from the disk.
// Creates a map of template paths and their previous and currently calculated checksums as values.
func createTemplateChecksumsMap(checksumsFilePath string) (map[string][2]string, error) {
checksumFile, err := os.Open(checksumsFilePath)
if err != nil {
return nil, err
}
defer f.Close()
scanner := bufio.NewScanner(f)
defer checksumFile.Close()
scanner := bufio.NewScanner(checksumFile)
checksum := make(map[string][2]string)
templatePathChecksumsMap := make(map[string][2]string)
for scanner.Scan() {
text := scanner.Text()
if text == "" {
continue
}
parts := strings.Split(text, ",")
if len(parts) < 2 {
continue
}
values := [2]string{parts[1]}
templatePath := parts[0]
expectedTemplateChecksum := parts[1]
f, err := os.Open(parts[0])
templateFile, err := os.Open(templatePath)
if err != nil {
return nil, err
}
hasher := md5.New()
if _, err := io.Copy(hasher, f); err != nil {
if _, err := io.Copy(hasher, templateFile); err != nil {
return nil, err
}
f.Close()
templateFile.Close()
values := [2]string{expectedTemplateChecksum}
values[1] = hex.EncodeToString(hasher.Sum(nil))
checksum[parts[0]] = values
templatePathChecksumsMap[templatePath] = values
}
return checksum, nil
return templatePathChecksumsMap, nil
}
// writeTemplatesChecksum writes the nuclei-templates checksum data to disk.
@ -502,60 +454,29 @@ func (r *Runner) printUpdateChangelog(results *templateUpdateResults, version st
table.Render()
}
// fetchLatestVersionsFromGithub fetches latest versions of nuclei repos from github
func (r *Runner) fetchLatestVersionsFromGithub() {
nucleiLatest, err := r.githubFetchLatestTagRepo("projectdiscovery/nuclei")
// fetchLatestVersionsFromGithub fetches the latest versions of nuclei repos from GitHub
//
// This fetches latest nuclei/templates/ignore from https://version-check.nuclei.sh/versions
// If you want to disable this automatic update check, use -nut flag.
func (r *Runner) fetchLatestVersionsFromGithub(configDir string) {
versions, err := client.GetLatestNucleiTemplatesVersion()
if err != nil {
gologger.Warning().Msgf("Could not fetch latest nuclei release: %s", err)
}
templatesLatest, err := r.githubFetchLatestTagRepo("projectdiscovery/nuclei-templates")
if err != nil {
gologger.Warning().Msgf("Could not fetch latest nuclei-templates release: %s", err)
gologger.Warning().Msgf("Could not fetch latest releases: %s", err)
return
}
if r.templatesConfig != nil {
r.templatesConfig.NucleiLatestVersion = nucleiLatest
r.templatesConfig.NucleiTemplatesLatestVersion = templatesLatest
r.templatesConfig.NucleiLatestVersion = versions.Nuclei
r.templatesConfig.NucleiTemplatesLatestVersion = versions.Templates
// If the fetch has resulted in new version of ignore file, update.
if r.templatesConfig.NucleiIgnoreHash == "" || r.templatesConfig.NucleiIgnoreHash != versions.IgnoreHash {
r.templatesConfig.NucleiIgnoreHash = versions.IgnoreHash
r.checkNucleiIgnoreFileUpdates(configDir)
}
}
}
type githubTagData struct {
Name string
}
// githubFetchLatestTagRepo fetches latest tag from github
// This function was half written by github copilot AI :D.
func (r *Runner) githubFetchLatestTagRepo(repo string) (string, error) {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
url := fmt.Sprintf("https://api.github.com/repos/%s/tags", repo)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return "", err
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return "", err
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return "", err
}
var tags []githubTagData
err = json.Unmarshal(body, &tags)
if err != nil {
return "", err
}
if len(tags) == 0 {
return "", fmt.Errorf("no tags found for %s", repo)
}
return strings.TrimPrefix(tags[0].Name, "v"), nil
}
// updateNucleiVersionToLatest implements nuclei auto-updation using Github Releases.
// updateNucleiVersionToLatest implements nuclei auto-update using GitHub Releases.
func updateNucleiVersionToLatest(verbose bool) error {
if verbose {
log.SetLevel(log.DebugLevel)

2
v2/internal/testutils/integration.go
View File

@ -34,7 +34,7 @@ func RunNucleiAndGetResults(template, url string, debug bool, extra ...string) (
var templateLoaded = regexp.MustCompile(`(?:Templates|Workflows) loaded[^:]*: (\d+)`)
// RunNucleiAndGetResults returns a list of results for a template
// RunNucleiBinaryAndGetLoadedTemplates returns a list of results for a template
func RunNucleiBinaryAndGetLoadedTemplates(nucleiBinary string, args []string) (string, error) {
cmd := exec.Command(nucleiBinary, args...)

25
v2/pkg/catalog/config/config.go
View File

@ -3,7 +3,6 @@ package config
import (
"os"
"path/filepath"
"time"
jsoniter "github.com/json-iterator/go"
"github.com/pkg/errors"
@ -13,12 +12,10 @@ import (
// Config contains the internal nuclei engine configuration
type Config struct {
TemplatesDirectory string `json:"templates-directory,omitempty"`
CurrentVersion string `json:"current-version,omitempty"`
LastChecked time.Time `json:"last-checked,omitempty"`
IgnoreURL string `json:"ignore-url,omitempty"`
TemplatesDirectory string `json:"nuclei-templates-directory,omitempty"`
TemplateVersion string `json:"nuclei-templates-version,omitempty"`
NucleiVersion string `json:"nuclei-version,omitempty"`
LastCheckedIgnore time.Time `json:"last-checked-ignore,omitempty"`
NucleiIgnoreHash string `json:"nuclei-ignore-hash,omitempty"`
NucleiLatestVersion string `json:"nuclei-latest-version"`
NucleiTemplatesLatestVersion string `json:"nuclei-templates-latest-version"`
@ -28,7 +25,7 @@ type Config struct {
const nucleiConfigFilename = ".templates-config.json"
// Version is the current version of nuclei
const Version = `2.5.1`
const Version = `2.5.2-dev`
func getConfigDetails() (string, error) {
homeDir, err := os.UserHomeDir()
@ -55,24 +52,14 @@ func ReadConfiguration() (*Config, error) {
defer file.Close()
config := &Config{}
err = jsoniter.NewDecoder(file).Decode(config)
if err != nil {
if err := jsoniter.NewDecoder(file).Decode(config); err != nil {
return nil, err
}
return config, nil
}
// WriteConfiguration writes the updated nuclei configuration to disk
func WriteConfiguration(config *Config, checked, checkedIgnore bool) error {
if config.IgnoreURL == "" {
config.IgnoreURL = "https://raw.githubusercontent.com/projectdiscovery/nuclei-templates/master/.nuclei-ignore"
}
if checked {
config.LastChecked = time.Now()
}
if checkedIgnore {
config.LastCheckedIgnore = time.Now()
}
func WriteConfiguration(config *Config) error {
config.NucleiVersion = Version
templatesConfigFile, err := getConfigDetails()

2
v2/pkg/catalog/loader/filter/tag_filter.go
View File

@ -20,7 +20,7 @@ type TagFilter struct {
var ErrExcluded = errors.New("the template was excluded")
// Match filters templates based on user provided tags, authors, extraTags and severity.
// If the template contains tags specified in the deny list, it will not be matched
// If the template contains tags specified in the deny-list, it will not be matched
// unless it is explicitly specified by user using the includeTags (matchAllows field).
// Matching rule: (tag1 OR tag2...) AND (author1 OR author2...) AND (severity1 OR severity2...) AND (extraTags1 OR extraTags2...)
// Returns true if the template matches the filter criteria, false otherwise.

4
v2/pkg/catalog/loader/loader.go
View File

@ -108,7 +108,7 @@ func (store *Store) ValidateTemplates(templatesList, workflowsList []string) err
func areWorkflowsValid(store *Store, filteredWorkflowPaths map[string]struct{}) bool {
return areWorkflowOrTemplatesValid(store, filteredWorkflowPaths, true, func(templatePath string, tagFilter *filter.TagFilter) (bool, error) {
return parsers.LoadWorkflow(templatePath, store.tagFilter)
return parsers.LoadWorkflow(templatePath)
})
}
@ -183,7 +183,7 @@ func (store *Store) LoadWorkflows(workflowsList []string) []*templates.Template
loadedWorkflows := make([]*templates.Template, 0, len(workflowPathMap))
for workflowPath := range workflowPathMap {
loaded, err := parsers.LoadWorkflow(workflowPath, store.tagFilter)
loaded, err := parsers.LoadWorkflow(workflowPath)
if err != nil {
gologger.Warning().Msgf("Could not load workflow %s: %s\n", workflowPath, err)
}

4
v2/pkg/model/model.go
View File

@ -59,12 +59,12 @@ type Info struct {
// - critical
SeverityHolder severity.Holder `json:"severity,omitempty" yaml:"severity,omitempty"`
// description: |
// AdditionalFields regarding metadata of the template.
// Metadata of the template.
//
// examples:
// - value: >
// map[string]string{"customField1":"customValue1"}
AdditionalFields map[string]string `json:"additional-fields,omitempty" yaml:"additional-fields,omitempty" jsonschema:"title=additional metadata for the template,description=Additional metadata fields for the template"`
Metadata map[string]string `json:"metadata,omitempty" yaml:"metadata,omitempty" jsonschema:"title=additional metadata for the template,description=Additional metadata fields for the template"`
// description: |
// Classification contains classification information about the template.

6
v2/pkg/model/model_test.go
View File

@ -80,7 +80,7 @@ func TestUnmarshal(t *testing.T) {
assert.Equal(t, info.Tags.ToSlice(), tags)
assert.Equal(t, info.SeverityHolder.Severity, severity.Critical)
assert.Equal(t, info.Reference.ToSlice(), references)
assert.Equal(t, info.AdditionalFields, dynamicKeysMap)
assert.Equal(t, info.Metadata, dynamicKeysMap)
return info
}
@ -90,7 +90,7 @@ func TestUnmarshal(t *testing.T) {
tags: ` + strings.Join(tags, ", ") + `
severity: critical
reference: ` + strings.Join(references, ", ") + `
additional-fields:
metadata:
` + dynamicKey1 + `: ` + dynamicKeysMap[dynamicKey1] + `
` + dynamicKey2 + `: ` + dynamicKeysMap[dynamicKey2] + `
`
@ -106,7 +106,7 @@ func TestUnmarshal(t *testing.T) {
reference:
- ` + references[0] + ` # comments are not unmarshalled
- ` + references[1] + `
additional-fields:
metadata:
` + dynamicKey1 + `: ` + dynamicKeysMap[dynamicKey1] + `
` + dynamicKey2 + `: ` + dynamicKeysMap[dynamicKey2] + `
`

8
v2/pkg/operators/common/dsl/dsl.go
View File

@ -114,18 +114,14 @@ var functions = map[string]govaluate.ExpressionFunction{
},
"sha256": func(args ...interface{}) (interface{}, error) {
h := sha256.New()
_, err := h.Write([]byte(types.ToString(args[0])))
if err != nil {
if _, err := h.Write([]byte(types.ToString(args[0]))); err != nil {
return nil, err
}
return hex.EncodeToString(h.Sum(nil)), nil
},
"sha1": func(args ...interface{}) (interface{}, error) {
h := sha1.New()
_, err := h.Write([]byte(types.ToString(args[0])))
if err != nil {
if _, err := h.Write([]byte(types.ToString(args[0]))); err != nil {
return nil, err
}
return hex.EncodeToString(h.Sum(nil)), nil

2
v2/pkg/operators/extractors/compile.go
View File

@ -8,7 +8,7 @@ import (
"github.com/itchyny/gojq"
)
// CompileExtractors performs the initial setup operation on a extractor
// CompileExtractors performs the initial setup operation on an extractor
func (e *Extractor) CompileExtractors() error {
var ok bool
// Set up the extractor type

4
v2/pkg/operators/extractors/extract.go
View File

@ -84,9 +84,7 @@ func (e *Extractor) ExtractJSON(corpus string) map[string]struct{} {
var jsonObj interface{}
err := json.Unmarshal([]byte(corpus), &jsonObj)
if err != nil {
if err := json.Unmarshal([]byte(corpus), &jsonObj); err != nil {
return results
}

4
v2/pkg/operators/extractors/extractors.go
View File

@ -50,7 +50,7 @@ type Extractor struct {
// description: |
// kval contains the key-value pairs present in the HTTP response header.
// kval extractor can be used to extract HTTP response header and cookie key-value pairs.
// kval extractor inputs are case insensitive, and does not support dash (-) in input which can replaced with underscores (_)
// kval extractor inputs are case-insensitive, and does not support dash (-) in input which can replaced with underscores (_)
// For example, Content-Type should be replaced with content_type
//
// A list of supported parts is available in docs for request types.
@ -121,7 +121,7 @@ const (
JSONExtractor
)
// ExtractorTypes is an table for conversion of extractor type from string.
// ExtractorTypes is a table for conversion of extractor type from string.
var ExtractorTypes = map[string]ExtractorType{
"regex": RegexExtractor,
"kval": KValExtractor,

1
v2/pkg/operators/matchers/compile.go
View File

@ -6,6 +6,7 @@ import (
"regexp"
"github.com/Knetic/govaluate"
"github.com/projectdiscovery/nuclei/v2/pkg/operators/common/dsl"
)

8
v2/pkg/operators/matchers/match.go
View File

@ -48,7 +48,7 @@ func (m *Matcher) MatchWords(corpus string) bool {
if m.condition == ANDCondition {
return false
}
// Continue with the flow since its an OR Condition.
// Continue with the flow since it's an OR Condition.
continue
}
@ -76,7 +76,7 @@ func (m *Matcher) MatchRegex(corpus string) bool {
if m.condition == ANDCondition {
return false
}
// Continue with the flow since its an OR Condition.
// Continue with the flow since it's an OR Condition.
continue
}
@ -105,7 +105,7 @@ func (m *Matcher) MatchBinary(corpus string) bool {
if m.condition == ANDCondition {
return false
}
// Continue with the flow since its an OR Condition.
// Continue with the flow since it's an OR Condition.
continue
}
@ -141,7 +141,7 @@ func (m *Matcher) MatchDSL(data map[string]interface{}) bool {
if m.condition == ANDCondition {
return false
}
// Continue with the flow since its an OR Condition.
// Continue with the flow since it's an OR Condition.
continue
}

6
v2/pkg/operators/matchers/matchers.go
View File

@ -17,7 +17,7 @@ type Matcher struct {
// - "regex"
// - "binary"
// - "dsl"
Type string `yaml:"type" jsonschema:"title=type of matcher,description=Type of the matcher,enum=status,enum=size,enum=word,enum=regex,enum=dsl"`
Type string `yaml:"type" jsonschema:"title=type of matcher,description=Type of the matcher,enum=status,enum=size,enum=word,enum=regex,enum=binary,enum=dsl"`
// description: |
// Condition is the optional condition between two matcher variables. By default,
// the condition is assumed to be OR.
@ -131,7 +131,7 @@ const (
DSLMatcher
)
// MatcherTypes is an table for conversion of matcher type from string.
// MatcherTypes is a table for conversion of matcher type from string.
var MatcherTypes = map[string]MatcherType{
"status": StatusMatcher,
"size": SizeMatcher,
@ -151,7 +151,7 @@ const (
ORCondition
)
// ConditionTypes is an table for conversion of condition type from string.
// ConditionTypes is a table for conversion of condition type from string.
var ConditionTypes = map[string]ConditionType{
"and": ANDCondition,
"or": ORCondition,

7
v2/pkg/operators/operators.go
View File

@ -14,7 +14,7 @@ type Operators struct {
// whether the request was successful by doing pattern matching
// on request/responses.
//
// Multiple matchers can be combined together with `matcher-condition` flag
// Multiple matchers can be combined with `matcher-condition` flag
// which accepts either `and` or `or` as argument.
Matchers []*matchers.Matcher `yaml:"matchers,omitempty" jsonschema:"title=matchers to run on response,description=Detection mechanism to identify whether the request was successful by doing pattern matching"`
// description: |
@ -102,7 +102,7 @@ func (r *Result) Merge(result *Result) {
// MatchFunc performs matching operation for a matcher on model and returns true or false.
type MatchFunc func(data map[string]interface{}, matcher *matchers.Matcher) bool
// ExtractFunc performs extracting operation for a extractor on model and returns true or false.
// ExtractFunc performs extracting operation for an extractor on model and returns true or false.
type ExtractFunc func(data map[string]interface{}, matcher *extractors.Extractor) map[string]struct{}
// Execute executes the operators on data and returns a result structure
@ -147,7 +147,7 @@ func (r *Operators) Execute(data map[string]interface{}, match MatchFunc, extrac
return nil, false
}
} else {
// If the matcher has matched, and its an OR
// If the matcher has matched, and it's an OR
// write the first output then move to next matcher.
if matcherCondition == matchers.ORCondition && matcher.Name != "" {
result.Matches[matcher.Name] = struct{}{}
@ -161,6 +161,7 @@ func (r *Operators) Execute(data map[string]interface{}, match MatchFunc, extrac
if len(result.DynamicValues) > 0 {
return result, true
}
// Don't print if we have matchers and they have not matched, regardless of extractor
if len(r.Matchers) > 0 && !matches {
return nil, false

5
v2/pkg/output/file_output_writer.go
View File

@ -20,11 +20,10 @@ func newFileOutputWriter(file string) (*fileWriter, error) {
// WriteString writes an output to the underlying file
func (w *fileWriter) Write(data []byte) error {
_, err := w.file.Write(data)
if err != nil {
if _, err := w.file.Write(data); err != nil {
return err
}
_, err = w.file.Write([]byte("\n"))
_, err := w.file.Write([]byte("\n"))
return err
}

4
v2/pkg/output/output.go
View File

@ -78,9 +78,9 @@ type ResultEvent struct {
Matched string `json:"matched,omitempty"`
// ExtractedResults contains the extraction result from the inputs.
ExtractedResults []string `json:"extracted_results,omitempty"`
// Request is the optional dumped request for the match.
// Request is the optional, dumped request for the match.
Request string `json:"request,omitempty"`
// Response is the optional dumped response for the match.
// Response is the optional, dumped response for the match.
Response string `json:"response,omitempty"`
// Metadata contains any optional metadata for the event
Metadata map[string]interface{} `json:"meta,omitempty"`

5
v2/pkg/parsers/parser.go
View File

@ -39,7 +39,7 @@ func LoadTemplate(templatePath string, tagFilter *filter.TagFilter, extraTags []
}
// LoadWorkflow returns true if the workflow is valid and matches the filtering criteria.
func LoadWorkflow(templatePath string, tagFilter *filter.TagFilter) (bool, error) {
func LoadWorkflow(templatePath string) (bool, error) {
template, templateParseError := ParseTemplate(templatePath)
if templateParseError != nil {
return false, templateParseError
@ -123,8 +123,7 @@ func ParseTemplate(templatePath string) (*templates.Template, error) {
}
template := &templates.Template{}
err = yaml.UnmarshalStrict(data, template)
if err != nil {
if err := yaml.UnmarshalStrict(data, template); err != nil {
errString := err.Error()
if !fieldErrorRegexp.MatchString(errString) {
stats.Increment(SyntaxErrorStats)

9
v2/pkg/projectfile/httputil.go
View File

@ -18,8 +18,7 @@ func hash(v interface{}) (string, error) {
sh := sha256.New()
_, err = io.WriteString(sh, string(data))
if err != nil {
if _, err = io.WriteString(sh, string(data)); err != nil {
return "", err
}
return hex.EncodeToString(sh.Sum(nil)), nil
@ -28,8 +27,7 @@ func hash(v interface{}) (string, error) {
func marshal(data interface{}) ([]byte, error) {
var b bytes.Buffer
enc := gob.NewEncoder(&b)
err := enc.Encode(data)
if err != nil {
if err := enc.Encode(data); err != nil {
return nil, err
}
@ -38,8 +36,7 @@ func marshal(data interface{}) ([]byte, error) {
func unmarshal(data []byte, obj interface{}) error {
dec := gob.NewDecoder(bytes.NewBuffer(data))
err := dec.Decode(obj)
if err != nil {
if err := dec.Decode(obj); err != nil {
return err
}

3
v2/pkg/projectfile/project.go
View File

@ -44,8 +44,7 @@ func (pf *ProjectFile) Get(req []byte) (*http.Response, error) {
var httprecord HTTPRecord
httprecord.Response = newInternalResponse()
err = unmarshal(data, &httprecord)
if err != nil {
if err := unmarshal(data, &httprecord); err != nil {
return nil, err
}

3
v2/pkg/protocols/common/executer/executer.go
View File

@ -24,8 +24,7 @@ func NewExecuter(requests []protocols.Request, options *protocols.ExecuterOption
// Compile compiles the execution generators preparing any requests possible.
func (e *Executer) Compile() error {
for _, request := range e.requests {
err := request.Compile(e.options)
if err != nil {
if err := request.Compile(e.options); err != nil {
return err
}
}

4
v2/pkg/protocols/common/generators/generators.go
View File

@ -12,7 +12,7 @@ type Generator struct {
type Type int
const (
// Sniper replaces each variables with values at a time.
// Sniper replaces each variable with values at a time.
Sniper Type = iota + 1
// PitchFork replaces variables with positional value from multiple wordlists
PitchFork
@ -20,7 +20,7 @@ const (
ClusterBomb
)
// StringToType is an table for conversion of attack type from string.
// StringToType is a table for conversion of attack type from string.
var StringToType = map[string]Type{
"sniper": Sniper,
"pitchfork": PitchFork,

2
v2/pkg/protocols/common/generators/maps.go
View File

@ -16,7 +16,7 @@ func MergeMaps(m1, m2 map[string]interface{}) map[string]interface{} {
return m
}
// ExpandMapValues converts values from flat string to strings slice
// ExpandMapValues converts values from flat string to string slice
func ExpandMapValues(m map[string]string) map[string][]string {
m1 := make(map[string][]string, len(m))
for k, v := range m {

6
v2/pkg/protocols/common/interactsh/interactsh.go
View File

@ -12,6 +12,7 @@ import (
"github.com/karlseguin/ccache"
"github.com/pkg/errors"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/interactsh/pkg/client"
"github.com/projectdiscovery/interactsh/pkg/server"
@ -120,7 +121,7 @@ func (c *Client) firstTimeInitializeClient() error {
item := c.requests.Get(interaction.UniqueID)
if item == nil {
// If we don't have any request for this ID, add it to temporary
// lru cache so we can correlate when we get an add request.
// lru cache, so we can correlate when we get an add request.
gotItem := c.interactions.Get(interaction.UniqueID)
if gotItem == nil {
c.interactions.Set(interaction.UniqueID, []*server.Interaction{interaction}, defaultInteractionDuration)
@ -181,6 +182,9 @@ func (c *Client) URL() string {
gologger.Error().Msgf("Could not initialize interactsh client: %s", err)
}
})
if c.interactsh == nil {
return ""
}
atomic.CompareAndSwapUint32(&c.generated, 0, 1)
return c.interactsh.URL()
}

5
v2/pkg/protocols/dns/dns.go
View File

@ -6,6 +6,7 @@ import (
"github.com/miekg/dns"
"github.com/pkg/errors"
"github.com/projectdiscovery/nuclei/v2/pkg/operators"
"github.com/projectdiscovery/nuclei/v2/pkg/protocols"
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/replacer"
@ -18,7 +19,7 @@ type Request struct {
// Operators for the current request go here.
operators.Operators `yaml:",inline"`
// ID is the the optional id of the request
// ID is the optional id of the request
ID string `yaml:"id,omitempty" jsonschema:"title=id of the dns request,description=ID is the optional ID of the DNS Request"`
// description: |
@ -158,7 +159,7 @@ func questionTypeToInt(questionType string) uint16 {
return question
}
// classToInt converts a dns class name to it's internal representation
// classToInt converts a dns class name to its internal representation
func classToInt(class string) uint16 {
class = strings.TrimSpace(strings.ToUpper(class))
result := dns.ClassINET

2
v2/pkg/protocols/dns/dnsclientpool/clientpool.go
View File

@ -25,7 +25,7 @@ var defaultResolvers = []string{
// Init initializes the clientpool implementation
func Init(options *types.Options) error {
// Don't create clients if already created in past.
// Don't create clients if already created in the past.
if normalClient != nil {
return nil
}

2
v2/pkg/protocols/dns/operators.go
View File

@ -43,7 +43,7 @@ func (r *Request) Match(data map[string]interface{}, matcher *matchers.Matcher)
return false
}
// Extract performs extracting operation for a extractor on model and returns true or false.
// Extract performs extracting operation for an extractor on model and returns true or false.
func (r *Request) Extract(data map[string]interface{}, extractor *extractors.Extractor) map[string]struct{} {
part := extractor.Part
switch part {

5
v2/pkg/protocols/dns/request.go
View File

@ -12,7 +12,7 @@ import (
var _ protocols.Request = &Request{}
// ExecuteWithResults executes the protocol requests and returns results instead of writing them.
func (r *Request) ExecuteWithResults(input string, metadata, previous output.InternalEvent, callback protocols.OutputEventCallback) error {
func (r *Request) ExecuteWithResults(input string, metadata /*TODO review unused parameter*/, previous output.InternalEvent, callback protocols.OutputEventCallback) error {
// Parse the URL and return domain if URL.
var domain string
if isURL(input) {
@ -71,8 +71,7 @@ func (r *Request) ExecuteWithResults(input string, metadata, previous output.Int
// isURL tests a string to determine if it is a well-structured url or not.
func isURL(toTest string) bool {
_, err := url.ParseRequestURI(toTest)
if err != nil {
if _, err := url.ParseRequestURI(toTest); err != nil {
return false
}
u, err := url.Parse(toTest)

5
v2/pkg/protocols/file/file.go
View File

@ -4,6 +4,7 @@ import (
"strings"
"github.com/pkg/errors"
"github.com/projectdiscovery/nuclei/v2/pkg/operators"
"github.com/projectdiscovery/nuclei/v2/pkg/protocols"
)
@ -26,7 +27,7 @@ type Request struct {
// - value: '[]string{".avi", ".mov", ".mp3"}'
ExtensionDenylist []string `yaml:"denylist,omitempty" jsonschema:"title=extensions to deny match,description=List of file extensions to deny during matching"`
// ID is the the optional id of the request
// ID is the optional id of the request
ID string `yaml:"id,omitempty" jsonschema:"title=id of the request,description=ID is the optional ID for the request"`
// description: |
@ -68,7 +69,7 @@ func (r *Request) Compile(options *protocols.ExecuterOptions) error {
}
r.CompiledOperators = compiled
}
// By default use 5mb as max size to read.
// By default, use 5 MB as max size to read.
if r.MaxSize == 0 {
r.MaxSize = 5 * 1024 * 1024
}

6
v2/pkg/protocols/file/find.go
View File

@ -18,8 +18,7 @@ func (r *Request) getInputPaths(target string, callback func(string)) error {
// Template input includes a wildcard
if strings.Contains(target, "*") && !r.NoRecursive {
err := r.findGlobPathMatches(target, processed, callback)
if err != nil {
if err := r.findGlobPathMatches(target, processed, callback); err != nil {
return errors.Wrap(err, "could not find glob matches")
}
return nil
@ -38,8 +37,7 @@ func (r *Request) getInputPaths(target string, callback func(string)) error {
}
// Recursively walk down the Templates directory and run all
// the template file checks
err = r.findDirectoryMatches(target, processed, callback)
if err != nil {
if err := r.findDirectoryMatches(target, processed, callback); err != nil {
return errors.Wrap(err, "could not find directory matches")
}
return nil

2
v2/pkg/protocols/file/operators.go
View File

@ -41,7 +41,7 @@ func (r *Request) Match(data map[string]interface{}, matcher *matchers.Matcher)
return false
}
// Extract performs extracting operation for a extractor on model and returns true or false.
// Extract performs extracting operation for an extractor on model and returns true or false.
func (r *Request) Extract(data map[string]interface{}, extractor *extractors.Extractor) map[string]struct{} {
partString := extractor.Part
switch partString {

2
v2/pkg/protocols/file/request.go
View File

@ -15,7 +15,7 @@ import (
var _ protocols.Request = &Request{}
// ExecuteWithResults executes the protocol requests and returns results instead of writing them.
func (r *Request) ExecuteWithResults(input string, metadata, previous output.InternalEvent, callback protocols.OutputEventCallback) error {
func (r *Request) ExecuteWithResults(input string, metadata /*TODO review unused parameter*/, previous output.InternalEvent, callback protocols.OutputEventCallback) error {
wg := sizedwaitgroup.New(r.options.Options.BulkSize)
err := r.getInputPaths(input, func(data string) {

2
v2/pkg/protocols/headless/engine/action.go
View File

@ -46,7 +46,7 @@ const (
ActionWaitEvent
// ActionKeyboard performs a keyboard action event on a page.
ActionKeyboard
// Action debug slows down headless and adds a sleep to each page.
// ActionDebug debug slows down headless and adds a sleep to each page.
ActionDebug
// ActionSleep executes a sleep for a specified duration
ActionSleep

5
v2/pkg/protocols/headless/engine/engine.go
View File

@ -11,9 +11,10 @@ import (
"github.com/go-rod/rod"
"github.com/go-rod/rod/lib/launcher"
"github.com/pkg/errors"
ps "github.com/shirou/gopsutil/v3/process"
"github.com/projectdiscovery/nuclei/v2/pkg/types"
"github.com/projectdiscovery/stringsutil"
ps "github.com/shirou/gopsutil/v3/process"
)
// Browser is a browser structure for nuclei headless module
@ -103,7 +104,7 @@ func (b *Browser) killChromeProcesses() {
processes, _ := ps.Processes()
for _, process := range processes {
// skip non chrome processes
// skip non-chrome processes
if !isChromeProcess(process) {
continue
}

10
v2/pkg/protocols/headless/engine/page.go
View File

@ -37,18 +37,18 @@ func (i *Instance) Run(baseURL *url.URL, actions []*Action, timeout time.Duratio
}
createdPage.router = router
err = page.SetViewport(&proto.EmulationSetDeviceMetricsOverride{Viewport: &proto.PageViewport{
if err := page.SetViewport(&proto.EmulationSetDeviceMetricsOverride{Viewport: &proto.PageViewport{
Scale: 1,
Width: float64(1920),
Height: float64(1080),
}})
if err != nil {
}}); err != nil {
return nil, nil, err
}
_, err = page.SetExtraHeaders([]string{"Accept-Language", "en, en-GB, en-us;"})
if err != nil {
if _, err := page.SetExtraHeaders([]string{"Accept-Language", "en, en-GB, en-us;"}); err != nil {
return nil, nil, err
}
go router.Run()
data, err := createdPage.ExecuteActions(baseURL, actions)
if err != nil {

37
v2/pkg/protocols/headless/engine/page_actions.go
View File

@ -84,7 +84,7 @@ type requestRule struct {
}
// ActionAddHeader executes a AddHeader action.
func (p *Page) ActionAddHeader(act *Action, out map[string]string) error {
func (p *Page) ActionAddHeader(act *Action, out map[string]string /*TODO review unused parameter*/) error {
in := act.GetArg("part")
args := make(map[string]string)
@ -100,7 +100,7 @@ func (p *Page) ActionAddHeader(act *Action, out map[string]string) error {
}
// ActionSetHeader executes a SetHeader action.
func (p *Page) ActionSetHeader(act *Action, out map[string]string) error {
func (p *Page) ActionSetHeader(act *Action, out map[string]string /*TODO review unused parameter*/) error {
in := act.GetArg("part")
args := make(map[string]string)
@ -116,7 +116,7 @@ func (p *Page) ActionSetHeader(act *Action, out map[string]string) error {
}
// ActionDeleteHeader executes a DeleteHeader action.
func (p *Page) ActionDeleteHeader(act *Action, out map[string]string) error {
func (p *Page) ActionDeleteHeader(act *Action, out map[string]string /*TODO review unused parameter*/) error {
in := act.GetArg("part")
args := make(map[string]string)
@ -131,7 +131,7 @@ func (p *Page) ActionDeleteHeader(act *Action, out map[string]string) error {
}
// ActionSetBody executes a SetBody action.
func (p *Page) ActionSetBody(act *Action, out map[string]string) error {
func (p *Page) ActionSetBody(act *Action, out map[string]string /*TODO review unused parameter*/) error {
in := act.GetArg("part")
args := make(map[string]string)
@ -146,7 +146,7 @@ func (p *Page) ActionSetBody(act *Action, out map[string]string) error {
}
// ActionSetMethod executes an SetMethod action.
func (p *Page) ActionSetMethod(act *Action, out map[string]string) error {
func (p *Page) ActionSetMethod(act *Action, out map[string]string /*TODO review unused parameter*/) error {
in := act.GetArg("part")
args := make(map[string]string)
@ -161,7 +161,7 @@ func (p *Page) ActionSetMethod(act *Action, out map[string]string) error {
}
// NavigateURL executes an ActionLoadURL actions loading a URL for the page.
func (p *Page) NavigateURL(action *Action, out map[string]string, parsed *url.URL) error {
func (p *Page) NavigateURL(action *Action, out map[string]string, parsed *url.URL /*TODO review unused parameter*/) error {
URL := action.GetArg("url")
if URL == "" {
return errors.New("invalid arguments provided")
@ -176,8 +176,7 @@ func (p *Page) NavigateURL(action *Action, out map[string]string, parsed *url.UR
values["BaseURL"] = parsedString
final := fasttemplate.ExecuteStringStd(URL, "{{", "}}", values)
err := p.page.Navigate(final)
if err != nil {
if err := p.page.Navigate(final); err != nil {
return errors.Wrap(err, "could not navigate")
}
return nil
@ -205,7 +204,7 @@ func (p *Page) RunScript(action *Action, out map[string]string) error {
}
// ClickElement executes click actions for an element.
func (p *Page) ClickElement(act *Action, out map[string]string) error {
func (p *Page) ClickElement(act *Action, out map[string]string /*TODO review unused parameter*/) error {
element, err := p.pageElementBy(act.Data)
if err != nil {
return errors.Wrap(err, "could not get element")
@ -220,12 +219,12 @@ func (p *Page) ClickElement(act *Action, out map[string]string) error {
}
// KeyboardAction executes a keyboard action on the page.
func (p *Page) KeyboardAction(act *Action, out map[string]string) error {
func (p *Page) KeyboardAction(act *Action, out map[string]string /*TODO review unused parameter*/) error {
return p.page.Keyboard.Press([]rune(act.GetArg("keys"))...)
}
// RightClickElement executes right click actions for an element.
func (p *Page) RightClickElement(act *Action, out map[string]string) error {
func (p *Page) RightClickElement(act *Action, out map[string]string /*TODO review unused parameter*/) error {
element, err := p.pageElementBy(act.Data)
if err != nil {
return errors.Wrap(err, "could not get element")
@ -266,7 +265,7 @@ func (p *Page) Screenshot(act *Action, out map[string]string) error {
}
// InputElement executes input element actions for an element.
func (p *Page) InputElement(act *Action, out map[string]string) error {
func (p *Page) InputElement(act *Action, out map[string]string /*TODO review unused parameter*/) error {
value := act.GetArg("value")
if value == "" {
return errors.New("invalid arguments provided")
@ -285,7 +284,7 @@ func (p *Page) InputElement(act *Action, out map[string]string) error {
}
// TimeInputElement executes time input on an element
func (p *Page) TimeInputElement(act *Action, out map[string]string) error {
func (p *Page) TimeInputElement(act *Action, out map[string]string /*TODO review unused parameter*/) error {
value := act.GetArg("value")
if value == "" {
return errors.New("invalid arguments provided")
@ -308,7 +307,7 @@ func (p *Page) TimeInputElement(act *Action, out map[string]string) error {
}
// SelectInputElement executes select input statement action on a element
func (p *Page) SelectInputElement(act *Action, out map[string]string) error {
func (p *Page) SelectInputElement(act *Action, out map[string]string /*TODO review unused parameter*/) error {
value := act.GetArg("value")
if value == "" {
return errors.New("invalid arguments provided")
@ -333,7 +332,7 @@ func (p *Page) SelectInputElement(act *Action, out map[string]string) error {
}
// WaitLoad waits for the page to load
func (p *Page) WaitLoad(act *Action, out map[string]string) error {
func (p *Page) WaitLoad(act *Action, out map[string]string /*TODO review unused parameter*/) error {
p.page.Timeout(1 * time.Second).WaitNavigation(proto.PageLifecycleEventNameDOMContentLoaded)()
// Wait for the window.onload event and also wait for the network requests
@ -363,7 +362,7 @@ func (p *Page) GetResource(act *Action, out map[string]string) error {
}
// FilesInput acts with a file input element on page
func (p *Page) FilesInput(act *Action, out map[string]string) error {
func (p *Page) FilesInput(act *Action, out map[string]string /*TODO review unused parameter*/) error {
element, err := p.pageElementBy(act.Data)
if err != nil {
return errors.Wrap(err, "could not get element")
@ -423,7 +422,7 @@ func (p *protoEvent) ProtoEvent() string {
}
// WaitEvent waits for an event to happen on the page.
func (p *Page) WaitEvent(act *Action, out map[string]string) error {
func (p *Page) WaitEvent(act *Action, out map[string]string /*TODO review unused parameter*/) error {
event := act.GetArg("event")
if event == "" {
return errors.New("event not recognized")
@ -480,14 +479,14 @@ func (p *Page) pageElementBy(data map[string]string) (*rod.Element, error) {
}
// DebugAction enables debug action on a page.
func (p *Page) DebugAction(act *Action, out map[string]string) error {
func (p *Page) DebugAction(act *Action, out map[string]string /*TODO review unused parameter*/) error {
p.instance.browser.engine.SlowMotion(5 * time.Second)
p.instance.browser.engine.Trace(true)
return nil
}
// SleepAction sleeps on the page for a specified duration
func (p *Page) SleepAction(act *Action, out map[string]string) error {
func (p *Page) SleepAction(act *Action, out map[string]string /*TODO review unused parameter*/) error {
seconds := act.Data["duration"]
if seconds == "" {
seconds = "5"

3
v2/pkg/protocols/headless/headless.go
View File

@ -2,6 +2,7 @@ package headless
import (
"github.com/pkg/errors"
"github.com/projectdiscovery/nuclei/v2/pkg/operators"
"github.com/projectdiscovery/nuclei/v2/pkg/protocols"
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/headless/engine"
@ -9,7 +10,7 @@ import (
// Request contains a Headless protocol request to be made from a template
type Request struct {
// ID is the the optional id of the request
// ID is the optional id of the request
ID string `yaml:"id,omitempty" jsonschema:"title=id of the request,description=Optional ID of the headless request"`
// description: |

2
v2/pkg/protocols/headless/operators.go
View File

@ -39,7 +39,7 @@ func (r *Request) Match(data map[string]interface{}, matcher *matchers.Matcher)
return false
}
// Extract performs extracting operation for a extractor on model and returns true or false.
// Extract performs extracting operation for an extractor on model and returns true or false.
func (r *Request) Extract(data map[string]interface{}, extractor *extractors.Extractor) map[string]struct{} {
partString := extractor.Part
switch partString {

2
v2/pkg/protocols/headless/request.go
View File

@ -14,7 +14,7 @@ import (
var _ protocols.Request = &Request{}
// ExecuteWithResults executes the protocol requests and returns results instead of writing them.
func (r *Request) ExecuteWithResults(input string, metadata, previous output.InternalEvent, callback protocols.OutputEventCallback) error {
func (r *Request) ExecuteWithResults(input string, metadata, previous output.InternalEvent /*TODO review unused parameter*/, callback protocols.OutputEventCallback) error {
instance, err := r.options.Browser.NewInstance()
if err != nil {
r.options.Output.Request(r.options.TemplateID, input, "headless", err)

5
v2/pkg/protocols/http/build_request.go
View File

@ -15,6 +15,7 @@ import (
"github.com/corpix/uarand"
"github.com/pkg/errors"
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/expressions"
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/generators"
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/http/race"
@ -27,7 +28,7 @@ var (
urlWithPortRegex = regexp.MustCompile(`{{BaseURL}}:(\d+)`)
)
// generatedRequest is a single wrapped generated request for a template request
// generatedRequest is a single generated request wrapped for a template request
type generatedRequest struct {
original *Request
rawRequest *raw.Request
@ -235,7 +236,7 @@ func (r *requestGenerator) fillRequest(req *http.Request, values map[string]inte
}
setHeader(req, "User-Agent", uarand.GetRandom())
// Only set these headers on non raw requests
// Only set these headers on non-raw requests
if len(r.request.Raw) == 0 {
setHeader(req, "Accept", "*/*")
setHeader(req, "Accept-Language", "en")

6
v2/pkg/protocols/http/http.go
View File

@ -4,6 +4,7 @@ import (
"strings"
"github.com/pkg/errors"
"github.com/projectdiscovery/nuclei/v2/pkg/operators"
"github.com/projectdiscovery/nuclei/v2/pkg/protocols"
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/generators"
@ -31,7 +32,7 @@ type Request struct {
// value: |
// []string{"GET /etc/passwd HTTP/1.1\nHost:\nContent-Length: 4", "POST /.%0d./.%0d./.%0d./.%0d./bin/sh HTTP/1.1\nHost: {{Hostname}}\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:71.0) Gecko/20100101 Firefox/71.0\nContent-Length: 1\nConnection: close\n\necho\necho\ncat /etc/passwd 2>&1"}
Raw []string `yaml:"raw,omitempty" jsonschema:"http requests in raw format,description=HTTP Requests in Raw Format"`
// ID is the the optional id of the request
// ID is the optional id of the request
ID string `yaml:"id,omitempty" jsonschema:"title=id for the http request,description=ID for the HTTP Request"`
// description: |
// Name is the optional name of the request.
@ -61,7 +62,8 @@ type Request struct {
// - "OPTIONS"
// - "TRACE"
// - "PATCH"
Method string `yaml:"method,omitempty" jsonschema:"title=method is the http request method,description=Method is the HTTP Request Method,enum=GET,enum=HEAD,enum=POST,enum=PUT,enum=DELETE,enum=CONNECT,enum=OPTIONS,enum=TRACE,enum=PATCH"`
// - "PURGE"
Method string `yaml:"method,omitempty" jsonschema:"title=method is the http request method,description=Method is the HTTP Request Method,enum=GET,enum=HEAD,enum=POST,enum=PUT,enum=DELETE,enum=CONNECT,enum=OPTIONS,enum=TRACE,enum=PATCH,enum=PURGE"`
// description: |
// Body is an optional parameter which contains HTTP Request body.
// examples:

6
v2/pkg/protocols/http/httpclientpool/clientpool.go
View File

@ -36,7 +36,7 @@ var (
// Init initializes the clientpool implementation
func Init(options *types.Options) error {
// Don't create clients if already created in past.
// Don't create clients if already created in the past.
if normalClient != nil {
return nil
}
@ -51,7 +51,7 @@ func Init(options *types.Options) error {
return nil
}
// // Configuration contains the custom configuration options for a connection
// ConnectionConfiguration contains the custom configuration options for a connection
type ConnectionConfiguration struct {
// DisableKeepAlive of the connection
DisableKeepAlive bool
@ -89,7 +89,7 @@ func (c *Configuration) Hash() string {
return hash
}
// HasCustomOptions checks whether the configuration requires custom settings
// HasStandardOptions checks whether the configuration requires custom settings
func (c *Configuration) HasStandardOptions() bool {
return c.Threads == 0 && c.MaxRedirects == 0 && !c.FollowRedirects && !c.CookieReuse && c.Connection == nil
}

4
v2/pkg/protocols/http/operators.go
View File

@ -44,7 +44,7 @@ func (r *Request) Match(data map[string]interface{}, matcher *matchers.Matcher)
return false
}
// Extract performs extracting operation for a extractor on model and returns true or false.
// Extract performs extracting operation for an extractor on model and returns true or false.
func (r *Request) Extract(data map[string]interface{}, extractor *extractors.Extractor) map[string]struct{} {
item, ok := getMatchPart(extractor.Part, data)
if !ok {
@ -85,7 +85,7 @@ func getMatchPart(part string, data output.InternalEvent) (string, bool) {
return itemStr, true
}
// responseToDSLMap converts a HTTP response to a map for use in DSL matching
// responseToDSLMap converts an HTTP response to a map for use in DSL matching
func (r *Request) responseToDSLMap(resp *http.Response, host, matched, rawReq, rawResp, body, headers string, duration time.Duration, extra map[string]interface{}) map[string]interface{} {
data := make(map[string]interface{}, len(extra)+8+len(resp.Header)+len(resp.Cookies()))
for k, v := range extra {

12
v2/pkg/protocols/http/request.go
View File

@ -13,6 +13,9 @@ import (
"time"
"github.com/pkg/errors"
"github.com/remeh/sizedwaitgroup"
"go.uber.org/multierr"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v2/pkg/output"
"github.com/projectdiscovery/nuclei/v2/pkg/protocols"
@ -22,8 +25,6 @@ import (
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/http/httpclientpool"
"github.com/projectdiscovery/rawhttp"
"github.com/projectdiscovery/stringsutil"
"github.com/remeh/sizedwaitgroup"
"go.uber.org/multierr"
)
const defaultMaxWorkers = 150
@ -323,7 +324,7 @@ func (r *Request) executeRequest(reqURL string, request *generatedRequest, previ
}
}
if err != nil {
// rawhttp doesn't supports draining response bodies.
// rawhttp doesn't support draining response bodies.
if resp != nil && resp.Body != nil && request.rawRequest == nil {
_, _ = io.CopyN(ioutil.Discard, resp.Body, drainReqSize)
resp.Body.Close()
@ -411,7 +412,7 @@ func (r *Request) executeRequest(reqURL string, request *generatedRequest, previ
redirectedResponse = bytes.ReplaceAll(redirectedResponse, dataOrig, data)
// Decode gbk response content-types
if contentType := resp.Header.Get("Content-Type"); contentType != "" && (strings.Contains(contentType, "gbk") || strings.Contains(contentType, "gb2312")) {
if contentType := strings.ToLower(resp.Header.Get("Content-Type")); contentType != "" && (strings.Contains(contentType, "gbk") || strings.Contains(contentType, "gb2312")) {
dumpedResponse, err = decodegbk(dumpedResponse)
if err != nil {
return errors.Wrap(err, "could not gbk decode")
@ -430,8 +431,7 @@ func (r *Request) executeRequest(reqURL string, request *generatedRequest, previ
// if nuclei-project is enabled store the response if not previously done
if r.options.ProjectFile != nil && !fromcache {
err := r.options.ProjectFile.Set(dumpedRequest, resp, data)
if err != nil {
if err := r.options.ProjectFile.Set(dumpedRequest, resp, data); err != nil {
return errors.Wrap(err, "could not store in project file")
}
}

3
v2/pkg/protocols/network/network.go
View File

@ -5,6 +5,7 @@ import (
"strings"
"github.com/pkg/errors"
"github.com/projectdiscovery/fastdialer/fastdialer"
"github.com/projectdiscovery/nuclei/v2/pkg/operators"
"github.com/projectdiscovery/nuclei/v2/pkg/protocols"
@ -15,7 +16,7 @@ import (
// Request contains a Network protocol request to be made from a template
type Request struct {
// ID is the the optional id of the request
// ID is the optional id of the request
ID string `yaml:"id,omitempty" jsonschema:"title=id of the request,description=ID of the network request"`
// description: |

6
v2/pkg/protocols/network/networkclientpool/clientpool.go
View File

@ -11,8 +11,8 @@ var (
)
// Init initializes the clientpool implementation
func Init(options *types.Options) error {
// Don't create clients if already created in past.
func Init(options *types.Options /*TODO review unused parameter*/) error {
// Don't create clients if already created in the past.
if normalClient != nil {
return nil
}
@ -29,6 +29,6 @@ func (c *Configuration) Hash() string {
}
// Get creates or gets a client for the protocol based on custom configuration
func Get(options *types.Options, configuration *Configuration) (*fastdialer.Dialer, error) {
func Get(options *types.Options, configuration *Configuration /*TODO review unused parameters*/) (*fastdialer.Dialer, error) {
return normalClient, nil
}

2
v2/pkg/protocols/network/operators.go
View File

@ -39,7 +39,7 @@ func (r *Request) Match(data map[string]interface{}, matcher *matchers.Matcher)
return false
}
// Extract performs extracting operation for a extractor on model and returns true or false.
// Extract performs extracting operation for an extractor on model and returns true or false.
func (r *Request) Extract(data map[string]interface{}, extractor *extractors.Extractor) map[string]struct{} {
partString := extractor.Part
switch partString {

8
v2/pkg/protocols/network/request.go
View File

@ -21,7 +21,7 @@ import (
var _ protocols.Request = &Request{}
// ExecuteWithResults executes the protocol requests and returns results instead of writing them.
func (r *Request) ExecuteWithResults(input string, metadata, previous output.InternalEvent, callback protocols.OutputEventCallback) error {
func (r *Request) ExecuteWithResults(input string, metadata /*TODO review unused parameter*/, previous output.InternalEvent, callback protocols.OutputEventCallback) error {
address, err := getAddress(input)
if err != nil {
r.options.Output.Request(r.options.TemplateID, input, "network", err)
@ -38,8 +38,7 @@ func (r *Request) ExecuteWithResults(input string, metadata, previous output.Int
actualAddress = net.JoinHostPort(actualAddress, kv.port)
}
err = r.executeAddress(actualAddress, address, input, kv.tls, previous, callback)
if err != nil {
if err := r.executeAddress(actualAddress, address, input, kv.tls, previous, callback); err != nil {
gologger.Verbose().Label("ERR").Msgf("Could not make network request for %s: %s\n", actualAddress, err)
continue
}
@ -138,8 +137,7 @@ func (r *Request) executeRequestWithPayloads(actualAddress, address, input strin
}
reqBuilder.Write(finalData)
_, err = conn.Write(finalData)
if err != nil {
if _, err := conn.Write(finalData); err != nil {
r.options.Output.Request(r.options.TemplateID, address, "network", err)
r.options.Progress.IncrementFailedRequestsBy(1)
return errors.Wrap(err, "could not write request to server")

6
v2/pkg/protocols/offlinehttp/find.go
View File

@ -17,8 +17,7 @@ func (r *Request) getInputPaths(target string, callback func(string)) error {
// Template input includes a wildcard
if strings.Contains(target, "*") {
err := r.findGlobPathMatches(target, processed, callback)
if err != nil {
if err := r.findGlobPathMatches(target, processed, callback); err != nil {
return errors.Wrap(err, "could not find glob matches")
}
return nil
@ -35,8 +34,7 @@ func (r *Request) getInputPaths(target string, callback func(string)) error {
// Recursively walk down the Templates directory and run all
// the template file checks
err = r.findDirectoryMatches(target, processed, callback)
if err != nil {
if err := r.findDirectoryMatches(target, processed, callback); err != nil {
return errors.Wrap(err, "could not find directory matches")
}
return nil

4
v2/pkg/protocols/offlinehttp/operators.go
View File

@ -40,7 +40,7 @@ func (r *Request) Match(data map[string]interface{}, matcher *matchers.Matcher)
return false
}
// Extract performs extracting operation for a extractor on model and returns true or false.
// Extract performs extracting operation for an extractor on model and returns true or false.
func (r *Request) Extract(data map[string]interface{}, extractor *extractors.Extractor) map[string]struct{} {
item, ok := getMatchPart(extractor.Part, data)
if !ok {
@ -77,7 +77,7 @@ func getMatchPart(part string, data output.InternalEvent) (string, bool) {
return itemStr, true
}
// responseToDSLMap converts a HTTP response to a map for use in DSL matching
// responseToDSLMap converts an HTTP response to a map for use in DSL matching
func (r *Request) responseToDSLMap(resp *http.Response, host, matched, rawReq, rawResp, body, headers string, duration time.Duration, extra map[string]interface{}) map[string]interface{} {
data := make(map[string]interface{}, len(extra)+8+len(resp.Header)+len(resp.Cookies()))
for k, v := range extra {

2
v2/pkg/protocols/offlinehttp/request.go
View File

@ -20,7 +20,7 @@ var _ protocols.Request = &Request{}
const maxSize = 5 * 1024 * 1024
// ExecuteWithResults executes the protocol requests and returns results instead of writing them.
func (r *Request) ExecuteWithResults(input string, metadata, previous output.InternalEvent, callback protocols.OutputEventCallback) error {
func (r *Request) ExecuteWithResults(input string, metadata /*TODO review unused parameter*/, previous output.InternalEvent, callback protocols.OutputEventCallback) error {
wg := sizedwaitgroup.New(r.options.Options.BulkSize)
err := r.getInputPaths(input, func(data string) {

5
v2/pkg/protocols/protocols.go
View File

@ -1,6 +1,8 @@
package protocols
import (
"go.uber.org/ratelimit"
"github.com/projectdiscovery/nuclei/v2/pkg/catalog"
"github.com/projectdiscovery/nuclei/v2/pkg/model"
"github.com/projectdiscovery/nuclei/v2/pkg/operators"
@ -14,7 +16,6 @@ import (
"github.com/projectdiscovery/nuclei/v2/pkg/protocols/headless/engine"
"github.com/projectdiscovery/nuclei/v2/pkg/reporting"
"github.com/projectdiscovery/nuclei/v2/pkg/types"
"go.uber.org/ratelimit"
)
// Executer is an interface implemented any protocol based request executer.
@ -75,7 +76,7 @@ type Request interface {
GetID() string
// Match performs matching operation for a matcher on model and returns true or false.
Match(data map[string]interface{}, matcher *matchers.Matcher) bool
// Extract performs extracting operation for a extractor on model and returns true or false.
// Extract performs extracting operation for an extractor on model and returns true or false.
Extract(data map[string]interface{}, matcher *extractors.Extractor) map[string]struct{}
// ExecuteWithResults executes the protocol requests and returns results instead of writing them.
ExecuteWithResults(input string, dynamicValues, previous output.InternalEvent, callback OutputEventCallback) error

7
v2/pkg/reporting/dedupe/dedupe.go
View File

@ -11,10 +11,11 @@ import (
"reflect"
"unsafe"
"github.com/projectdiscovery/nuclei/v2/pkg/output"
"github.com/projectdiscovery/nuclei/v2/pkg/types"
"github.com/syndtr/goleveldb/leveldb"
"github.com/syndtr/goleveldb/leveldb/errors"
"github.com/projectdiscovery/nuclei/v2/pkg/output"
"github.com/projectdiscovery/nuclei/v2/pkg/types"
)
// Storage is a duplicate detecting storage for nuclei scan events.
@ -93,7 +94,7 @@ func (s *Storage) Index(result *output.ResultEvent) (bool, error) {
exists, err := s.storage.Has(hash, nil)
if err != nil {
// if we have an error, return with it but mark it as true
// since we don't want to loose an issue considering it a dupe.
// since we don't want to lose an issue considering it a dupe.
return true, err
}
if !exists {

5
v2/pkg/reporting/exporters/disk/disk.go
View File

@ -16,7 +16,7 @@ type Exporter struct {
options *Options
}
// Options contains the configuration options for github issue tracker client
// Options contains the configuration options for GitHub issue tracker client
type Options struct {
// Directory is the directory to export found results to
Directory string `yaml:"directory"`
@ -55,8 +55,7 @@ func (i *Exporter) Export(event *output.ResultEvent) error {
dataBuilder.WriteString(description)
data := dataBuilder.Bytes()
err := ioutil.WriteFile(filepath.Join(i.directory, finalFilename), data, 0644)
return err
return ioutil.WriteFile(filepath.Join(i.directory, finalFilename), data, 0644)
}
// Close closes the exporter after operation

2
v2/pkg/reporting/format/format.go
View File

@ -182,7 +182,7 @@ func ToMarkdownTableString(templateInfo *model.Info) string {
}
toMarkDownTable(fields)
toMarkDownTable(utils.NewInsertionOrderedStringMap(templateInfo.AdditionalFields))
toMarkDownTable(utils.NewInsertionOrderedStringMap(templateInfo.Metadata))
return builder.String()
}

2
v2/pkg/reporting/format/format_test.go
View File

@ -19,7 +19,7 @@ func TestToMarkdownTableString(t *testing.T) {
SeverityHolder: severity.Holder{Severity: severity.High},
Tags: stringslice.StringSlice{Value: []string{"cve", "misc"}},
Reference: stringslice.StringSlice{Value: "reference1"},
AdditionalFields: map[string]string{
Metadata: map[string]string{
"customDynamicKey1": "customDynamicValue1",
"customDynamicKey2": "customDynamicValue2",
},

3
v2/pkg/reporting/trackers/github/github.go
View File

@ -8,11 +8,12 @@ import (
"github.com/google/go-github/github"
"github.com/pkg/errors"
"github.com/projectdiscovery/nuclei/v2/pkg/output"
"github.com/projectdiscovery/nuclei/v2/pkg/reporting/format"
)
// Integration is a client for a issue tracker integration
// Integration is a client for an issue tracker integration
type Integration struct {
client *github.Client
options *Options

5
v2/pkg/reporting/trackers/gitlab/gitlab.go
View File

@ -1,12 +1,13 @@
package gitlab
import (
"github.com/xanzy/go-gitlab"
"github.com/projectdiscovery/nuclei/v2/pkg/output"
"github.com/projectdiscovery/nuclei/v2/pkg/reporting/format"
"github.com/xanzy/go-gitlab"
)
// Integration is a client for a issue tracker integration
// Integration is a client for an issue tracker integration
type Integration struct {
client *gitlab.Client
userID int

6
v2/pkg/templates/compile.go
View File

@ -53,8 +53,7 @@ func Parse(filePath string, preprocessor Preprocessor, options protocols.Execute
data = preprocessor.Process(data)
}
err = yaml.Unmarshal(data, template)
if err != nil {
if err := yaml.Unmarshal(data, template); err != nil {
return nil, err
}
@ -135,8 +134,7 @@ func Parse(filePath string, preprocessor Preprocessor, options protocols.Execute
template.Executer = executer.NewExecuter(requests, &options)
}
if template.Executer != nil {
err := template.Executer.Compile()
if err != nil {
if err := template.Executer.Compile(); err != nil {
return nil, errors.Wrap(err, "could not compile request")
}
template.TotalRequests += template.Executer.Requests()

49
v2/pkg/templates/templates_doc.go
View File

@ -126,7 +126,7 @@ func init() {
MODELInfoDoc.Fields[3].Description = "Description of the template.\n\nYou can go in-depth here on what the template actually does."
MODELInfoDoc.Fields[3].Comments[encoder.LineComment] = "Description of the template."
MODELInfoDoc.Fields[3].AddExample("", "Bower is a package manager which stores packages informations in bower.json file")
MODELInfoDoc.Fields[3].AddExample("", "Bower is a package manager which stores packages information in bower.json file")
MODELInfoDoc.Fields[3].AddExample("", "Subversion ALM for the enterprise before 8.8.2 allows reflected XSS at multiple locations")
MODELInfoDoc.Fields[4].Name = "reference"
@ -148,11 +148,11 @@ func init() {
"high",
"critical",
}
MODELInfoDoc.Fields[6].Name = "additional-fields"
MODELInfoDoc.Fields[6].Name = "metadata"
MODELInfoDoc.Fields[6].Type = "map[string]string"
MODELInfoDoc.Fields[6].Note = ""
MODELInfoDoc.Fields[6].Description = "AdditionalFields regarding metadata of the template."
MODELInfoDoc.Fields[6].Comments[encoder.LineComment] = "AdditionalFields regarding metadata of the template."
MODELInfoDoc.Fields[6].Description = "Metadata of the template."
MODELInfoDoc.Fields[6].Comments[encoder.LineComment] = "Metadata of the template."
MODELInfoDoc.Fields[6].AddExample("", map[string]string{"customField1": "customValue1"})
MODELInfoDoc.Fields[7].Name = "classification"
@ -274,7 +274,7 @@ func init() {
HTTPRequestDoc.Fields[0].Name = "matchers"
HTTPRequestDoc.Fields[0].Type = "[]matchers.Matcher"
HTTPRequestDoc.Fields[0].Note = ""
HTTPRequestDoc.Fields[0].Description = "Matchers contains the detection mechanism for the request to identify\nwhether the request was successful by doing pattern matching\non request/responses.\n\nMultiple matchers can be combined together with `matcher-condition` flag\nwhich accepts either `and` or `or` as argument."
HTTPRequestDoc.Fields[0].Description = "Matchers contains the detection mechanism for the request to identify\nwhether the request was successful by doing pattern matching\non request/responses.\n\nMultiple matchers can be combined with `matcher-condition` flag\nwhich accepts either `and` or `or` as argument."
HTTPRequestDoc.Fields[0].Comments[encoder.LineComment] = "Matchers contains the detection mechanism for the request to identify"
HTTPRequestDoc.Fields[1].Name = "extractors"
HTTPRequestDoc.Fields[1].Type = "[]extractors.Extractor"
@ -307,8 +307,8 @@ func init() {
HTTPRequestDoc.Fields[5].Name = "id"
HTTPRequestDoc.Fields[5].Type = "string"
HTTPRequestDoc.Fields[5].Note = ""
HTTPRequestDoc.Fields[5].Description = "ID is the the optional id of the request"
HTTPRequestDoc.Fields[5].Comments[encoder.LineComment] = " ID is the the optional id of the request"
HTTPRequestDoc.Fields[5].Description = "ID is the optional id of the request"
HTTPRequestDoc.Fields[5].Comments[encoder.LineComment] = " ID is the optional id of the request"
HTTPRequestDoc.Fields[6].Name = "name"
HTTPRequestDoc.Fields[6].Type = "string"
HTTPRequestDoc.Fields[6].Note = ""
@ -339,6 +339,7 @@ func init() {
"OPTIONS",
"TRACE",
"PATCH",
"PURGE",
}
HTTPRequestDoc.Fields[9].Name = "body"
HTTPRequestDoc.Fields[9].Type = "string"
@ -628,7 +629,7 @@ func init() {
EXTRACTORSExtractorDoc.Fields[4].Name = "kval"
EXTRACTORSExtractorDoc.Fields[4].Type = "[]string"
EXTRACTORSExtractorDoc.Fields[4].Note = ""
EXTRACTORSExtractorDoc.Fields[4].Description = "description: |\n kval contains the key-value pairs present in the HTTP response header.\n kval extractor can be used to extract HTTP response header and cookie key-value pairs.\n kval extractor inputs are case insensitive, and does not support dash (-) in input which can replaced with underscores (_)\n For example, Content-Type should be replaced with content_type\n\n A list of supported parts is available in docs for request types.\n examples:\n - name: Extract Server Header From HTTP Response\n value: >\n []string{\"server\"}\n - name: Extracting value of PHPSESSID Cookie\n value: >\n []string{\"phpsessid\"}\n - name: Extracting value of Content-Type Cookie\n value: >\n []string{\"content_type\"}"
EXTRACTORSExtractorDoc.Fields[4].Description = "description: |\n kval contains the key-value pairs present in the HTTP response header.\n kval extractor can be used to extract HTTP response header and cookie key-value pairs.\n kval extractor inputs are case-insensitive, and does not support dash (-) in input which can replaced with underscores (_)\n For example, Content-Type should be replaced with content_type\n\n A list of supported parts is available in docs for request types.\n examples:\n - name: Extract Server Header From HTTP Response\n value: >\n []string{\"server\"}\n - name: Extracting value of PHPSESSID Cookie\n value: >\n []string{\"phpsessid\"}\n - name: Extracting value of Content-Type Cookie\n value: >\n []string{\"content_type\"}"
EXTRACTORSExtractorDoc.Fields[4].Comments[encoder.LineComment] = " description: |"
EXTRACTORSExtractorDoc.Fields[5].Name = "json"
EXTRACTORSExtractorDoc.Fields[5].Type = "[]string"
@ -683,7 +684,7 @@ func init() {
DNSRequestDoc.Fields[0].Name = "matchers"
DNSRequestDoc.Fields[0].Type = "[]matchers.Matcher"
DNSRequestDoc.Fields[0].Note = ""
DNSRequestDoc.Fields[0].Description = "Matchers contains the detection mechanism for the request to identify\nwhether the request was successful by doing pattern matching\non request/responses.\n\nMultiple matchers can be combined together with `matcher-condition` flag\nwhich accepts either `and` or `or` as argument."
DNSRequestDoc.Fields[0].Description = "Matchers contains the detection mechanism for the request to identify\nwhether the request was successful by doing pattern matching\non request/responses.\n\nMultiple matchers can be combined with `matcher-condition` flag\nwhich accepts either `and` or `or` as argument."
DNSRequestDoc.Fields[0].Comments[encoder.LineComment] = "Matchers contains the detection mechanism for the request to identify"
DNSRequestDoc.Fields[1].Name = "extractors"
DNSRequestDoc.Fields[1].Type = "[]extractors.Extractor"
@ -702,8 +703,8 @@ func init() {
DNSRequestDoc.Fields[3].Name = "id"
DNSRequestDoc.Fields[3].Type = "string"
DNSRequestDoc.Fields[3].Note = ""
DNSRequestDoc.Fields[3].Description = "ID is the the optional id of the request"
DNSRequestDoc.Fields[3].Comments[encoder.LineComment] = " ID is the the optional id of the request"
DNSRequestDoc.Fields[3].Description = "ID is the optional id of the request"
DNSRequestDoc.Fields[3].Comments[encoder.LineComment] = " ID is the optional id of the request"
DNSRequestDoc.Fields[4].Name = "name"
DNSRequestDoc.Fields[4].Type = "string"
DNSRequestDoc.Fields[4].Note = ""
@ -768,7 +769,7 @@ func init() {
FILERequestDoc.Fields[0].Name = "matchers"
FILERequestDoc.Fields[0].Type = "[]matchers.Matcher"
FILERequestDoc.Fields[0].Note = ""
FILERequestDoc.Fields[0].Description = "Matchers contains the detection mechanism for the request to identify\nwhether the request was successful by doing pattern matching\non request/responses.\n\nMultiple matchers can be combined together with `matcher-condition` flag\nwhich accepts either `and` or `or` as argument."
FILERequestDoc.Fields[0].Description = "Matchers contains the detection mechanism for the request to identify\nwhether the request was successful by doing pattern matching\non request/responses.\n\nMultiple matchers can be combined with `matcher-condition` flag\nwhich accepts either `and` or `or` as argument."
FILERequestDoc.Fields[0].Comments[encoder.LineComment] = "Matchers contains the detection mechanism for the request to identify"
FILERequestDoc.Fields[1].Name = "extractors"
FILERequestDoc.Fields[1].Type = "[]extractors.Extractor"
@ -801,8 +802,8 @@ func init() {
FILERequestDoc.Fields[5].Name = "id"
FILERequestDoc.Fields[5].Type = "string"
FILERequestDoc.Fields[5].Note = ""
FILERequestDoc.Fields[5].Description = "ID is the the optional id of the request"
FILERequestDoc.Fields[5].Comments[encoder.LineComment] = " ID is the the optional id of the request"
FILERequestDoc.Fields[5].Description = "ID is the optional id of the request"
FILERequestDoc.Fields[5].Comments[encoder.LineComment] = " ID is the optional id of the request"
FILERequestDoc.Fields[6].Name = "max-size"
FILERequestDoc.Fields[6].Type = "int"
FILERequestDoc.Fields[6].Note = ""
@ -831,8 +832,8 @@ func init() {
NETWORKRequestDoc.Fields[0].Name = "id"
NETWORKRequestDoc.Fields[0].Type = "string"
NETWORKRequestDoc.Fields[0].Note = ""
NETWORKRequestDoc.Fields[0].Description = "ID is the the optional id of the request"
NETWORKRequestDoc.Fields[0].Comments[encoder.LineComment] = " ID is the the optional id of the request"
NETWORKRequestDoc.Fields[0].Description = "ID is the optional id of the request"
NETWORKRequestDoc.Fields[0].Comments[encoder.LineComment] = " ID is the optional id of the request"
NETWORKRequestDoc.Fields[1].Name = "host"
NETWORKRequestDoc.Fields[1].Type = "[]string"
NETWORKRequestDoc.Fields[1].Note = ""
@ -870,7 +871,7 @@ func init() {
NETWORKRequestDoc.Fields[6].Name = "matchers"
NETWORKRequestDoc.Fields[6].Type = "[]matchers.Matcher"
NETWORKRequestDoc.Fields[6].Note = ""
NETWORKRequestDoc.Fields[6].Description = "Matchers contains the detection mechanism for the request to identify\nwhether the request was successful by doing pattern matching\non request/responses.\n\nMultiple matchers can be combined together with `matcher-condition` flag\nwhich accepts either `and` or `or` as argument."
NETWORKRequestDoc.Fields[6].Description = "Matchers contains the detection mechanism for the request to identify\nwhether the request was successful by doing pattern matching\non request/responses.\n\nMultiple matchers can be combined with `matcher-condition` flag\nwhich accepts either `and` or `or` as argument."
NETWORKRequestDoc.Fields[6].Comments[encoder.LineComment] = "Matchers contains the detection mechanism for the request to identify"
NETWORKRequestDoc.Fields[7].Name = "extractors"
NETWORKRequestDoc.Fields[7].Type = "[]extractors.Extractor"
@ -943,8 +944,8 @@ func init() {
HEADLESSRequestDoc.Fields[0].Name = "id"
HEADLESSRequestDoc.Fields[0].Type = "string"
HEADLESSRequestDoc.Fields[0].Note = ""
HEADLESSRequestDoc.Fields[0].Description = "ID is the the optional id of the request"
HEADLESSRequestDoc.Fields[0].Comments[encoder.LineComment] = " ID is the the optional id of the request"
HEADLESSRequestDoc.Fields[0].Description = "ID is the optional id of the request"
HEADLESSRequestDoc.Fields[0].Comments[encoder.LineComment] = " ID is the optional id of the request"
HEADLESSRequestDoc.Fields[1].Name = "steps"
HEADLESSRequestDoc.Fields[1].Type = "[]engine.Action"
HEADLESSRequestDoc.Fields[1].Note = ""
@ -953,7 +954,7 @@ func init() {
HEADLESSRequestDoc.Fields[2].Name = "matchers"
HEADLESSRequestDoc.Fields[2].Type = "[]matchers.Matcher"
HEADLESSRequestDoc.Fields[2].Note = ""
HEADLESSRequestDoc.Fields[2].Description = "Matchers contains the detection mechanism for the request to identify\nwhether the request was successful by doing pattern matching\non request/responses.\n\nMultiple matchers can be combined together with `matcher-condition` flag\nwhich accepts either `and` or `or` as argument."
HEADLESSRequestDoc.Fields[2].Description = "Matchers contains the detection mechanism for the request to identify\nwhether the request was successful by doing pattern matching\non request/responses.\n\nMultiple matchers can be combined with `matcher-condition` flag\nwhich accepts either `and` or `or` as argument."
HEADLESSRequestDoc.Fields[2].Comments[encoder.LineComment] = "Matchers contains the detection mechanism for the request to identify"
HEADLESSRequestDoc.Fields[3].Name = "extractors"
HEADLESSRequestDoc.Fields[3].Type = "[]extractors.Extractor"
@ -1064,8 +1065,8 @@ func init() {
WORKFLOWSWorkflowTemplateDoc.Fields[3].Name = "subtemplates"
WORKFLOWSWorkflowTemplateDoc.Fields[3].Type = "[]workflows.WorkflowTemplate"
WORKFLOWSWorkflowTemplateDoc.Fields[3].Note = ""
WORKFLOWSWorkflowTemplateDoc.Fields[3].Description = "Subtemplates are ran if the `template` field Template matches."
WORKFLOWSWorkflowTemplateDoc.Fields[3].Comments[encoder.LineComment] = "Subtemplates are ran if the `template` field Template matches."
WORKFLOWSWorkflowTemplateDoc.Fields[3].Description = "Subtemplates are run if the `template` field Template matches."
WORKFLOWSWorkflowTemplateDoc.Fields[3].Comments[encoder.LineComment] = "Subtemplates are run if the `template` field Template matches."
WORKFLOWSMatcherDoc.Type = "workflows.Matcher"
WORKFLOWSMatcherDoc.Comments[encoder.LineComment] = ""
@ -1085,8 +1086,8 @@ func init() {
WORKFLOWSMatcherDoc.Fields[1].Name = "subtemplates"
WORKFLOWSMatcherDoc.Fields[1].Type = "[]workflows.WorkflowTemplate"
WORKFLOWSMatcherDoc.Fields[1].Note = ""
WORKFLOWSMatcherDoc.Fields[1].Description = "Subtemplates are ran if the name of matcher matches."
WORKFLOWSMatcherDoc.Fields[1].Comments[encoder.LineComment] = "Subtemplates are ran if the name of matcher matches."
WORKFLOWSMatcherDoc.Fields[1].Description = "Subtemplates are run if the name of matcher matches."
WORKFLOWSMatcherDoc.Fields[1].Comments[encoder.LineComment] = "Subtemplates are run if the name of matcher matches."
}
// GetTemplateDoc returns documentation for the file templates_doc.go.

6
v2/pkg/workflows/execute.go
View File

@ -15,8 +15,7 @@ func (w *Workflow) RunWorkflow(input string) bool {
for _, template := range w.Workflows {
swg.Add()
func(template *WorkflowTemplate) {
err := w.runWorkflowStep(template, input, results, &swg)
if err != nil {
if err := w.runWorkflowStep(template, input, results, &swg); err != nil {
gologger.Warning().Msgf("[%s] Could not execute workflow step: %s\n", template.Template, err)
}
swg.Done()
@ -116,8 +115,7 @@ func (w *Workflow) runWorkflowStep(template *WorkflowTemplate, input string, res
swg.Add()
go func(template *WorkflowTemplate) {
err := w.runWorkflowStep(template, input, results, swg)
if err != nil {
if err := w.runWorkflowStep(template, input, results, swg); err != nil {
gologger.Warning().Msgf("[%s] Could not execute workflow step: %s\n", template.Template, err)
}
swg.Done()

6
v2/pkg/workflows/workflows.go
View File

@ -14,7 +14,7 @@ type Workflow struct {
Options *protocols.ExecuterOptions `yaml:"-"`
}
// WorkflowTemplate is a template to be ran as part of a workflow
// WorkflowTemplate is a template to be run as part of a workflow
type WorkflowTemplate struct {
// description: |
// Template is a single template or directory to execute as part of workflow.
@ -31,7 +31,7 @@ type WorkflowTemplate struct {
// Matchers perform name based matching to run subtemplates for a workflow.
Matchers []*Matcher `yaml:"matchers,omitempty" jsonschema:"title=name based template result matchers,description=Matchers perform name based matching to run subtemplates for a workflow"`
// description: |
// Subtemplates are ran if the `template` field Template matches.
// Subtemplates are run if the `template` field Template matches.
Subtemplates []*WorkflowTemplate `yaml:"subtemplates,omitempty" jsonschema:"title=subtemplate based result matchers,description=Subtemplates are ran if the template field Template matches"`
// Executers perform the actual execution for the workflow template
Executers []*ProtocolExecuterPair `yaml:"-"`
@ -49,6 +49,6 @@ type Matcher struct {
// Name is the name of the item to match.
Name string `yaml:"name,omitempty" jsonschema:"title=name of item to match,description=Name of item to match"`
// description: |
// Subtemplates are ran if the name of matcher matches.
// Subtemplates are run if the name of matcher matches.
Subtemplates []*WorkflowTemplate `yaml:"subtemplates,omitempty" jsonschema:"title=templates to run after match,description=Templates to run after match"`
}