daffainfo
/
nuclei
mirror of https://github.com/daffainfo/nuclei.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Fuzzing layer enhancements + input-types support (#4477) 

* feat: move fuzz package to root directory

* feat: added support for input providers like openapi,postman,etc

* feat: integration of new fuzzing logic in engine

* bugfix: use and instead of or

* fixed lint errors

* go mod tidy

* add new reqresp type + bump utils

* custom http request parser

* use new struct type RequestResponse

* introduce unified input/target provider

* abstract input formats via new inputprovider

* completed input provider refactor

* remove duplicated code

* add sdk method to load targets

* rename component url->path

* add new yaml format + remove duplicated code

* use gopkg.in/yaml.v3 for parsing

* update .gitignore

* refactor/move + docs fuzzing in http protocol

* fuzz: header + query integration test using fuzzplayground

* fix integration test runner in windows

* feat add support for filter in http fuzz

* rewrite header/query integration test with filter

* add replace regex rule

* support kv fuzzing + misc updates

* add path fuzzing example + misc improvements

* fix matchedURL + skip httpx on multi formats

* cookie fuzz integration test

* add json body + params body tests

* feat add multipart/form-data fuzzing support

* add all fuzz body integration test

* misc bug fixes + minor refactor

* add multipart form + body form unit tests

* only run fuzzing templates if -fuzz flag is given

* refactor/move fuzz playground server to pkg

* fix integration test + refactor

* add auth types and strategies

* add file auth provider

* start implementing auth logic in http

* add logic in http protocol

* static auth implemented for http

* default :80,:443 normalization

* feat: dynamic auth init

* feat: dynamic auth using templates

* validate targets count in openapi+swagger

* inputformats: add support to accept variables

* fix workflow integration test

* update lazy cred fetch logic

* fix unit test

* drop postman support

* domain related normalization

* update secrets.yaml file format + misc updates

* add auth prefetch option

* remove old secret files

* add fuzzing+auth related sdk options

* fix/support multiple mode in kv header fuzzing

* rename 'headers' -> 'header' in fuzzing rules

* fix deadlock due to merge conflict resolution

* misc update

* add bool type in parsed value

* add openapi validation+override+ new flags

* misc updates

* remove optional path parameters when unavailable

* fix swagger.yaml file

* misc updates

* update print msg

* multiple openapi validation enchancements + appMode

* add optional params in required_openapi_vars.yaml file

* improve warning/verbose msgs in format

* fix skip-format-validation not working

* use 'params/parameter' instead of 'variable' in openapi

* add retry support for falky tests

* fix nuclei loading ignored templates (#4849)

* fix tag include logic

* fix unit test

* remove quoting in extractor output

* remove quote in debug code command

* feat: issue tracker URLs in JSON + misc fixes (#4855)

* feat: issue tracker URLs in JSON + misc fixes

* misc changes

* feat: status update support for issues

* feat: report metadata generation hook support

* feat: added CLI summary of tickets created

* misc changes

* introduce `disable-unsigned-templates` flag (#4820)

* introduce `disable-unsigned-templates` flag

* minor

* skip instead of exit

* remove duplicate imports

* use stats package + misc enhancements

* force display warning + adjust skipped stats in unsigned count

* include unsigned skipped templates without -dut flag

---------

Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>

* Purge cache on global callback set (#4840)

* purge cache on global callback set

* lint

* purging cache

* purge cache in runner after loading templates

* include internal cache from parsers + add global cache register/purge via config

* remove disable cache purge option

---------

Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>

* misc update

* add application/octet-stream support

* openapi: support path specific params

* misc option + readme update

---------

Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
Co-authored-by: Tarun Koyalwar <45962551+tarunKoyalwar@users.noreply.github.com>
Co-authored-by: Dogan Can Bakir <65292895+dogancanbakir@users.noreply.github.com>
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
dev
Ice3man 2024-03-14 03:08:53 +05:30 committed by GitHub
parent 66b722faaf
commit fa56800fcc
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
137 changed files with 9269 additions and 1103 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
.gitignore vendored
View File

@ -30,6 +30,9 @@ pkg/protocols/headless/engine/.cache
/tsgen
/scrapefuncs
/integration_tests/.cache/
/integration_tests/.nuclei-config/
/*.yaml
/pkg/protocols/headless/engine/.nuclei-config/
**/*-config
**/*-cache
/fuzzplayground
integration_tests/fuzzplayground

2
Makefile
View File

@ -42,6 +42,8 @@ jsupdate:
ts:
$(GOBUILD) $(GOFLAGS) -ldflags '$(LDFLAGS)' -o "tsgen" pkg/js/devtools/tsgen/cmd/tsgen/main.go
./tsgen -dir pkg/js/libs -out pkg/js/generated/ts
fuzzplayground:
$(GOBUILD) $(GOFLAGS) -ldflags '$(LDFLAGS)' -o "fuzzplayground" cmd/tools/fuzzplayground/main.go
memogen:
$(GOBUILD) $(GOFLAGS) -ldflags '$(LDFLAGS)' -o "memogen" cmd/memogen/memogen.go
./memogen -src pkg/js/libs -tpl cmd/memogen/function.tpl

39
README.md
View File

@ -113,12 +113,17 @@ Usage:
Flags:
TARGET:
-u, -target string[] target URLs/hosts to scan
-l, -list string path to file containing a list of target URLs/hosts to scan (one per line)
-eh, -exclude-hosts string[] hosts to exclude to scan from the input list (ip, cidr, hostname)
-resume string resume scan using resume.cfg (clustering will be disabled)
-sa, -scan-all-ips scan all the IP's associated with dns record
-iv, -ip-version string[] IP version to scan of hostname (4,6) - (default 4)
-u, -target string[] target URLs/hosts to scan
-l, -list string path to file containing a list of target URLs/hosts to scan (one per line)
-eh, -exclude-hosts string[] hosts to exclude to scan from the input list (ip, cidr, hostname)
-resume string resume scan using resume.cfg (clustering will be disabled)
-sa, -scan-all-ips scan all the IP's associated with dns record
-iv, -ip-version string[] IP version to scan of hostname (4,6) - (default 4)
TARGET-FORMAT:
-im, -input-mode string mode of input file (list, burp, jsonl, yaml, openapi, swagger) (default "list")
-ro, -required-only use only required fields in input format when generating requests
-sfv, -skip-format-validation skip format validation (like missing vars) when parsing input file
TEMPLATES:
-nt, -new-templates run only new templates added in latest nuclei-templates release
@ -134,6 +139,7 @@ TEMPLATES:
-tl list all available templates
-sign signs the templates with the private key defined in NUCLEI_SIGNATURE_PRIVATE_KEY env variable
-code enable loading code protocol-based templates
-dut, -disable-unsigned-templates disable running unsigned templates or templates with mismatched signature
FILTERING:
-a, -author string[] templates to run based on authors (comma-separated, file)
@ -142,8 +148,8 @@ FILTERING:
-itags, -include-tags string[] tags to be executed even if they are excluded either by default or configuration
-id, -template-id string[] templates to run based on template ids (comma-separated, file, allow-wildcard)
-eid, -exclude-id string[] templates to exclude based on template ids (comma-separated, file)
-it, -include-templates string[] templates to be executed even if they are excluded either by default or configuration
-et, -exclude-templates string[] template or template directory to exclude (comma-separated, file)
-it, -include-templates string[] path to template file or directory to be executed even if they are excluded either by default or configuration
-et, -exclude-templates string[] path to template file or directory to exclude (comma-separated, file)
-em, -exclude-matchers string[] template matchers to exclude in result
-s, -severity value[] templates to run based on severity. Possible values: info, low, medium, high, critical, unknown
-es, -exclude-severity value[] templates to exclude based on severity. Possible values: info, low, medium, high, critical, unknown
@ -215,6 +221,7 @@ INTERACTSH:
FUZZING:
-ft, -fuzzing-type string overrides fuzzing type set in template (replace, prefix, postfix, infix)
-fm, -fuzzing-mode string overrides fuzzing mode set in template (multiple, single)
-fuzz enable loading fuzzing templates
UNCOVER:
-uc, -uncover enable uncover engine
@ -231,6 +238,8 @@ RATE-LIMIT:
-c, -concurrency int maximum number of templates to be executed in parallel (default 25)
-hbs, -headless-bulk-size int maximum number of headless hosts to be analyzed in parallel per template (default 10)
-headc, -headless-concurrency int maximum number of headless templates to be executed in parallel (default 10)
-jsc, -js-concurrency int maximum number of javascript runtimes to be executed in parallel (default 120)
-pc, -payload-concurrency int max payload concurrency for each template (default 25)
OPTIMIZATIONS:
-timeout int time to wait in seconds before timeout (default 10)
@ -292,22 +301,26 @@ CLOUD:
-cup, -cloud-upload upload scan results to pdcp dashboard
-sid, -scan-id string upload scan results to given scan id
AUTHENTICATION:
-sf, -secret-file string[] path to config file containing secrets for nuclei authenticated scan
-ps, -prefetch-secrets prefetch secrets from the secrets file
EXAMPLES:
Run nuclei on single host:
$ nuclei -target example.com
$ nuclei -target example.com
Run nuclei with specific template directories:
$ nuclei -target example.com -t http/cves/ -t ssl
$ nuclei -target example.com -t http/cves/ -t ssl
Run nuclei against a list of hosts:
$ nuclei -list hosts.txt
$ nuclei -list hosts.txt
Run nuclei with a JSON output:
$ nuclei -target example.com -json-export output.json
$ nuclei -target example.com -json-export output.json
Run nuclei with sorted Markdown outputs (with environment variables):
$ MARKDOWN_EXPORT_SORT_MODE=template nuclei -target example.com -markdown-export nuclei_report/
$ MARKDOWN_EXPORT_SORT_MODE=template nuclei -target example.com -markdown-export nuclei_report/
Additional documentation is available at: https://docs.nuclei.sh/getting-started/running
```

39
cmd/integration-test/fuzz.go
View File

@ -12,6 +12,10 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/testutils"
)
const (
targetFile = "fuzz/testData/ginandjuice.proxify.yaml"
)
var fuzzingTestCases = []TestCaseInfo{
{Path: "fuzz/fuzz-mode.yaml", TestCase: &fuzzModeOverride{}},
{Path: "fuzz/fuzz-type.yaml", TestCase: &fuzzTypeOverride{}},
@ -19,6 +23,29 @@ var fuzzingTestCases = []TestCaseInfo{
{Path: "fuzz/fuzz-headless.yaml", TestCase: &HeadlessFuzzingQuery{}},
{Path: "fuzz/fuzz-header-basic.yaml", TestCase: &FuzzHeaderBasic{}},
{Path: "fuzz/fuzz-header-multiple.yaml", TestCase: &FuzzHeaderMultiple{}},
// for fuzzing we should prioritize adding test case related backend
// logic in fuzz playground server instead of adding them here
{Path: "fuzz/fuzz-query-num-replace.yaml", TestCase: &genericFuzzTestCase{expectedResults: 2}},
{Path: "fuzz/fuzz-host-header-injection.yaml", TestCase: &genericFuzzTestCase{expectedResults: 1}},
{Path: "fuzz/fuzz-path-sqli.yaml", TestCase: &genericFuzzTestCase{expectedResults: 1}},
{Path: "fuzz/fuzz-cookie-error-sqli.yaml", TestCase: &genericFuzzTestCase{expectedResults: 1}},
{Path: "fuzz/fuzz-body-json-sqli.yaml", TestCase: &genericFuzzTestCase{expectedResults: 1}},
{Path: "fuzz/fuzz-body-multipart-form-sqli.yaml", TestCase: &genericFuzzTestCase{expectedResults: 1}},
{Path: "fuzz/fuzz-body-params-sqli.yaml", TestCase: &genericFuzzTestCase{expectedResults: 1}},
{Path: "fuzz/fuzz-body-xml-sqli.yaml", TestCase: &genericFuzzTestCase{expectedResults: 1}},
{Path: "fuzz/fuzz-body-generic-sqli.yaml", TestCase: &genericFuzzTestCase{expectedResults: 4}},
}
type genericFuzzTestCase struct {
expectedResults int
}
func (g *genericFuzzTestCase) Execute(filePath string) error {
results, err := testutils.RunNucleiWithArgsAndGetResults(debug, "-t", filePath, "-l", targetFile, "-im", "yaml")
if err != nil {
return err
}
return expectResultsCount(results, g.expectedResults)
}
type httpFuzzQuery struct{}
@ -34,7 +61,7 @@ func (h *httpFuzzQuery) Execute(filePath string) error {
ts := httptest.NewTLSServer(router)
defer ts.Close()
results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL+"/?id=example", debug)
results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL+"/?id=example", debug, "-fuzz")
if err != nil {
return err
}
@ -53,7 +80,7 @@ func (h *fuzzModeOverride) Execute(filePath string) error {
})
ts := httptest.NewTLSServer(router)
defer ts.Close()
results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL+"/?id=example&name=nuclei", debug, "-fuzzing-mode", "single", "-jsonl")
results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL+"/?id=example&name=nuclei", debug, "-fuzzing-mode", "single", "-jsonl", "-fuzz")
if err != nil {
return err
}
@ -98,7 +125,7 @@ func (h *fuzzTypeOverride) Execute(filePath string) error {
})
ts := httptest.NewTLSServer(router)
defer ts.Close()
results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL+"?id=example", debug, "-fuzzing-type", "replace", "-jsonl")
results, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL+"?id=example", debug, "-fuzzing-type", "replace", "-jsonl", "-fuzz")
if err != nil {
return err
}
@ -143,7 +170,7 @@ func (h *HeadlessFuzzingQuery) Execute(filePath string) error {
ts := httptest.NewTLSServer(router)
defer ts.Close()
got, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL+"?url=https://scanme.sh", debug, "-headless")
got, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL+"?url=https://scanme.sh", debug, "-headless", "-fuzz")
if err != nil {
return err
}
@ -164,7 +191,7 @@ func (h *FuzzHeaderBasic) Execute(filePath string) error {
ts := httptest.NewTLSServer(router)
defer ts.Close()
got, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
got, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug, "-fuzz")
if err != nil {
return err
}
@ -192,7 +219,7 @@ func (h *FuzzHeaderMultiple) Execute(filePath string) error {
ts := httptest.NewTLSServer(router)
defer ts.Close()
got, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug)
got, err := testutils.RunNucleiTemplateAndGetResults(filePath, ts.URL, debug, "-fuzz")
if err != nil {
return err
}

20
cmd/integration-test/integration-test.go
View File

@ -9,7 +9,9 @@ import (
"github.com/logrusorgru/aurora"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v3/pkg/testutils"
"github.com/projectdiscovery/nuclei/v3/pkg/testutils/fuzzplayground"
sliceutil "github.com/projectdiscovery/utils/slice"
)
@ -53,6 +55,10 @@ var (
"flow": flowTestcases,
"javascript": jsTestcases,
}
// flakyTests are run with a retry count of 3
flakyTests = map[string]bool{
"protocols/http/self-contained-file-input.yaml": true,
}
// For debug purposes
runProtocol = ""
@ -78,6 +84,18 @@ func main() {
os.Exit(1)
}
// start fuzz playground server
defer fuzzplayground.Cleanup()
server := fuzzplayground.GetPlaygroundServer()
defer server.Close()
go func() {
if err := server.Start("localhost:8082"); err != nil {
if !strings.Contains(err.Error(), "Server closed") {
gologger.Fatal().Msgf("Could not start server: %s\n", err)
}
}
}()
customTestsList := normalizeSplit(customTests)
failedTestTemplatePaths := runTests(customTestsList)
@ -150,6 +168,8 @@ func runTests(customTemplatePaths []string) []string {
var err error
if proto == "interactsh" || strings.Contains(testCaseInfo.Path, "interactsh") {
failedTemplatePath, err = executeWithRetry(testCaseInfo.TestCase, testCaseInfo.Path, interactshRetryCount)
} else if flakyTests[testCaseInfo.Path] {
failedTemplatePath, err = executeWithRetry(testCaseInfo.TestCase, testCaseInfo.Path, interactshRetryCount)
} else {
failedTemplatePath, err = execute(testCaseInfo.TestCase, testCaseInfo.Path)
}

6
cmd/integration-test/library.go
View File

@ -19,11 +19,10 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/disk"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/loader"
"github.com/projectdiscovery/nuclei/v3/pkg/core"
"github.com/projectdiscovery/nuclei/v3/pkg/core/inputs"
"github.com/projectdiscovery/nuclei/v3/pkg/input/provider"
"github.com/projectdiscovery/nuclei/v3/pkg/output"
"github.com/projectdiscovery/nuclei/v3/pkg/parsers"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/hosterrorscache"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/interactsh"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolinit"
@ -126,8 +125,7 @@ func executeNucleiAsLibrary(templatePath, templateURL string) ([]string, error)
}
store.Load()
input := &inputs.SimpleInputProvider{Inputs: []*contextargs.MetaInput{{Input: templateURL}}}
_ = engine.Execute(store.Templates(), input)
_ = engine.Execute(store.Templates(), provider.NewSimpleInputProviderWithUrls(templateURL))
engine.WorkPool().Wait() // Wait for the scan to finish
return results, nil

2
cmd/integration-test/workflow.go
View File

@ -137,7 +137,7 @@ type workflowSharedCookies struct{}
// Execute executes a test case and returns an error if occurred
func (h *workflowSharedCookies) Execute(filePath string) error {
handleFunc := func(name string, w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
handleFunc := func(name string, w http.ResponseWriter, _ *http.Request, _ httprouter.Params) {
cookie := &http.Cookie{Name: name, Value: name}
http.SetCookie(w, cookie)
}

28
cmd/nuclei/main.go
View File

@ -22,6 +22,7 @@ import (
"github.com/projectdiscovery/interactsh/pkg/client"
"github.com/projectdiscovery/nuclei/v3/internal/runner"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
"github.com/projectdiscovery/nuclei/v3/pkg/input/provider"
"github.com/projectdiscovery/nuclei/v3/pkg/installer"
"github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
"github.com/projectdiscovery/nuclei/v3/pkg/operators/common/dsl"
@ -46,6 +47,9 @@ var (
)
func main() {
// enables CLI specific configs mostly interactive behavior
config.CurrentAppMode = config.AppModeCLI
if err := runner.ConfigureOptions(); err != nil {
gologger.Fatal().Msgf("Could not initialize options: %s\n", err)
}
@ -201,6 +205,12 @@ on extensive configurability, massive extensibility and ease of use.`)
flagSet.StringSliceVarP(&options.IPVersion, "ip-version", "iv", nil, "IP version to scan of hostname (4,6) - (default 4)", goflags.CommaSeparatedStringSliceOptions),
)
flagSet.CreateGroup("target-format", "Target-Format",
flagSet.StringVarP(&options.InputFileMode, "input-mode", "im", "list", fmt.Sprintf("mode of input file (%v)", provider.SupportedInputFormats())),
flagSet.BoolVarP(&options.FormatUseRequiredOnly, "required-only", "ro", false, "use only required fields in input format when generating requests"),
flagSet.BoolVarP(&options.SkipFormatValidation, "skip-format-validation", "sfv", false, "skip format validation (like missing vars) when parsing input file"),
)
flagSet.CreateGroup("templates", "Templates",
flagSet.BoolVarP(&options.NewTemplates, "new-templates", "nt", false, "run only new templates added in latest nuclei-templates release"),
flagSet.StringSliceVarP(&options.NewTemplatesWithVersion, "new-templates-version", "ntv", nil, "run new templates added in specific version", goflags.CommaSeparatedStringSliceOptions),
@ -226,8 +236,8 @@ on extensive configurability, massive extensibility and ease of use.`)
flagSet.StringSliceVarP(&options.IncludeTags, "include-tags", "itags", nil, "tags to be executed even if they are excluded either by default or configuration", goflags.FileNormalizedStringSliceOptions), // TODO show default deny list
flagSet.StringSliceVarP(&options.IncludeIds, "template-id", "id", nil, "templates to run based on template ids (comma-separated, file, allow-wildcard)", goflags.FileNormalizedStringSliceOptions),
flagSet.StringSliceVarP(&options.ExcludeIds, "exclude-id", "eid", nil, "templates to exclude based on template ids (comma-separated, file)", goflags.FileNormalizedStringSliceOptions),
flagSet.StringSliceVarP(&options.IncludeTemplates, "include-templates", "it", nil, "templates to be executed even if they are excluded either by default or configuration", goflags.FileCommaSeparatedStringSliceOptions),
flagSet.StringSliceVarP(&options.ExcludedTemplates, "exclude-templates", "et", nil, "template or template directory to exclude (comma-separated, file)", goflags.FileCommaSeparatedStringSliceOptions),
flagSet.StringSliceVarP(&options.IncludeTemplates, "include-templates", "it", nil, "path to template file or directory to be executed even if they are excluded either by default or configuration", goflags.FileCommaSeparatedStringSliceOptions),
flagSet.StringSliceVarP(&options.ExcludedTemplates, "exclude-templates", "et", nil, "path to template file or directory to exclude (comma-separated, file)", goflags.FileCommaSeparatedStringSliceOptions),
flagSet.StringSliceVarP(&options.ExcludeMatchers, "exclude-matchers", "em", nil, "template matchers to exclude in result", goflags.FileCommaSeparatedStringSliceOptions),
flagSet.VarP(&options.Severities, "severity", "s", fmt.Sprintf("templates to run based on severity. Possible values: %s", severity.GetSupportedSeverities().String())),
flagSet.VarP(&options.ExcludeSeverities, "exclude-severity", "es", fmt.Sprintf("templates to exclude based on severity. Possible values: %s", severity.GetSupportedSeverities().String())),
@ -303,6 +313,7 @@ on extensive configurability, massive extensibility and ease of use.`)
flagSet.CreateGroup("fuzzing", "Fuzzing",
flagSet.StringVarP(&options.FuzzingType, "fuzzing-type", "ft", "", "overrides fuzzing type set in template (replace, prefix, postfix, infix)"),
flagSet.StringVarP(&options.FuzzingMode, "fuzzing-mode", "fm", "", "overrides fuzzing mode set in template (multiple, single)"),
flagSet.BoolVar(&options.FuzzTemplates, "fuzz", false, "enable loading fuzzing templates"),
)
flagSet.CreateGroup("uncover", "Uncover",
@ -394,6 +405,11 @@ on extensive configurability, massive extensibility and ease of use.`)
flagSet.StringVarP(&options.ScanID, "scan-id", "sid", "", "upload scan results to given scan id"),
)
flagSet.CreateGroup("Authentication", "Authentication",
flagSet.StringSliceVarP(&options.SecretsFile, "secret-file", "sf", nil, "path to config file containing secrets for nuclei authenticated scan", goflags.CommaSeparatedStringSliceOptions),
flagSet.BoolVarP(&options.PreFetchSecrets, "prefetch-secrets", "ps", false, "prefetch secrets from the secrets file"),
)
flagSet.SetCustomHelpText(`EXAMPLES:
Run nuclei on single host:
$ nuclei -target example.com
@ -468,6 +484,14 @@ Additional documentation is available at: https://docs.nuclei.sh/getting-started
config.DefaultConfig.SetTemplatesDir(options.NewTemplatesDirectory)
}
if len(options.SecretsFile) > 0 {
for _, secretFile := range options.SecretsFile {
if !fileutil.FileExists(secretFile) {
gologger.Fatal().Msgf("given secrets file '%s' does not exist", options.SecretsFile)
}
}
}
cleanupOldResumeFiles()
return flagSet
}

101
cmd/tools/fuzzplayground/main.go
View File

@ -1,94 +1,27 @@
package main
import (
"fmt"
"io"
"os/exec"
"strconv"
"strings"
"flag"
"github.com/labstack/echo/v4"
"github.com/labstack/echo/v4/middleware"
"github.com/projectdiscovery/retryablehttp-go"
_ "github.com/mattn/go-sqlite3"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v3/pkg/testutils/fuzzplayground"
)
var (
addr string
)
func main() {
e := echo.New()
e.Use(middleware.Recover())
e.Use(middleware.Logger())
flag.StringVar(&addr, "addr", "localhost:8082", "playground server address")
flag.Parse()
e.GET("/", indexHandler)
e.GET("/info", infoHandler)
e.GET("/redirect", redirectHandler)
e.GET("/request", requestHandler)
e.GET("/email", emailHandler)
e.GET("/permissions", permissionsHandler)
if err := e.Start("localhost:8082"); err != nil {
panic(err)
defer fuzzplayground.Cleanup()
server := fuzzplayground.GetPlaygroundServer()
defer server.Close()
// Start the server
if err := server.Start(addr); err != nil {
gologger.Fatal().Msgf("Could not start server: %s\n", err)
}
}
var bodyTemplate = `<html>
<head>
<title>Fuzz Playground</title>
</head>
<body>
%s
</body>
</html>`
func indexHandler(ctx echo.Context) error {
return ctx.HTML(200, fmt.Sprintf(bodyTemplate, `<h1>Fuzzing Playground</h1><hr>
<ul>
<li><a href="/info?name=test&another=value&random=data">Info Page XSS</a></li>
<li><a href="/redirect?redirect_url=/info?name=redirected_from_url">Redirect Page OpenRedirect</a></li>
<li><a href="/request?url=https://example.com">Request Page SSRF</a></li>
<li><a href="/email?text=important_user">Email Page SSTI</a></li>
<li><a href="/permissions?cmd=whoami">Permissions Page CMDI</a></li>
</ul>
`))
}
func infoHandler(ctx echo.Context) error {
return ctx.HTML(200, fmt.Sprintf(bodyTemplate, fmt.Sprintf("Name of user: %s%s%s", ctx.QueryParam("name"), ctx.QueryParam("another"), ctx.QueryParam("random"))))
}
func redirectHandler(ctx echo.Context) error {
url := ctx.QueryParam("redirect_url")
return ctx.Redirect(302, url)
}
func requestHandler(ctx echo.Context) error {
url := ctx.QueryParam("url")
data, err := retryablehttp.DefaultClient().Get(url)
if err != nil {
return ctx.HTML(500, err.Error())
}
defer data.Body.Close()
body, _ := io.ReadAll(data.Body)
return ctx.HTML(200, fmt.Sprintf(bodyTemplate, string(body)))
}
func emailHandler(ctx echo.Context) error {
text := ctx.QueryParam("text")
if strings.Contains(text, "{{") {
trimmed := strings.SplitN(strings.Trim(text[strings.Index(text, "{"):], "{}"), "*", 2)
if len(trimmed) < 2 {
return ctx.HTML(500, "invalid template")
}
first, _ := strconv.Atoi(trimmed[0])
second, _ := strconv.Atoi(trimmed[1])
text = strconv.Itoa(first * second)
}
return ctx.HTML(200, fmt.Sprintf(bodyTemplate, fmt.Sprintf("Text: %s", text)))
}
func permissionsHandler(ctx echo.Context) error {
command := ctx.QueryParam("cmd")
fields := strings.Fields(command)
cmd := exec.Command(fields[0], fields[1:]...)
data, _ := cmd.CombinedOutput()
return ctx.HTML(200, fmt.Sprintf(bodyTemplate, string(data)))
}

33
go.mod
View File

@ -42,7 +42,6 @@ require (
golang.org/x/oauth2 v0.11.0
golang.org/x/text v0.14.0
gopkg.in/yaml.v2 v2.4.0
moul.io/http2curl v1.0.0
)
require (
@ -60,18 +59,22 @@ require (
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.72
github.com/aws/aws-sdk-go-v2/service/s3 v1.37.0
github.com/charmbracelet/glamour v0.6.0
github.com/clbanning/mxj/v2 v2.7.0
github.com/denisenkom/go-mssqldb v0.12.3
github.com/ditashi/jsbeautifier-go v0.0.0-20141206144643-2520a8026a9c
github.com/docker/go-units v0.5.0
github.com/dop251/goja v0.0.0-20230828202809-3dbe69dd2b8e
github.com/fatih/structs v1.1.0
github.com/getkin/kin-openapi v0.123.0
github.com/go-git/go-git/v5 v5.11.0
github.com/go-ldap/ldap/v3 v3.4.5
github.com/go-pg/pg v8.0.7+incompatible
github.com/go-sql-driver/mysql v1.7.1
github.com/h2non/filetype v1.1.3
github.com/labstack/echo/v4 v4.10.2
github.com/leslie-qiwa/flat v0.0.0-20230424180412-f9d1cf014baa
github.com/lib/pq v1.10.1
github.com/mattn/go-sqlite3 v1.14.22
github.com/mholt/archiver v3.1.1+incompatible
github.com/ory/dockertest/v3 v3.10.0
github.com/praetorian-inc/fingerprintx v1.1.9
@ -95,11 +98,13 @@ require (
github.com/projectdiscovery/wappalyzergo v0.0.112
github.com/redis/go-redis/v9 v9.1.0
github.com/sashabaranov/go-openai v1.15.3
github.com/seh-msft/burpxml v1.0.1
github.com/stretchr/testify v1.9.0
github.com/zmap/zgrab2 v0.1.8-0.20230806160807-97ba87c0e706
golang.org/x/term v0.17.0
gopkg.in/src-d/go-git.v4 v4.13.1
gopkg.in/yaml.v3 v3.0.1
moul.io/http2curl v1.0.0
)
require (
@ -145,6 +150,8 @@ require (
github.com/gin-gonic/gin v1.9.1 // indirect
github.com/go-asn1-ber/asn1-ber v1.5.4 // indirect
github.com/go-fed/httpsig v1.1.0 // indirect
github.com/go-openapi/jsonpointer v0.20.2 // indirect
github.com/go-openapi/swag v0.22.9 // indirect
github.com/go-sourcemap/sourcemap v2.1.3+incompatible // indirect
github.com/goccy/go-json v0.10.2 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
@ -158,30 +165,35 @@ require (
github.com/hashicorp/go-version v1.6.0 // indirect
github.com/hashicorp/golang-lru/v2 v2.0.6 // indirect
github.com/hbakhtiyor/strsim v0.0.0-20190107154042-4d2bbb273edf // indirect
github.com/invopop/yaml v0.2.0 // indirect
github.com/jcmturner/aescts/v2 v2.0.0 // indirect
github.com/jcmturner/dnsutils/v2 v2.0.0 // indirect
github.com/jcmturner/gofork v1.7.6 // indirect
github.com/jcmturner/rpc/v2 v2.0.3 // indirect
github.com/jinzhu/inflection v1.0.0 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/kataras/jwt v0.1.10 // indirect
github.com/klauspost/compress v1.17.4 // indirect
github.com/klauspost/pgzip v1.2.5 // indirect
github.com/klauspost/compress v1.17.6 // indirect
github.com/klauspost/pgzip v1.2.6 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
github.com/lucasb-eyer/go-colorful v1.2.0 // indirect
github.com/mackerelio/go-osstat v0.2.4 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
github.com/mholt/archiver/v3 v3.5.1 // indirect
github.com/minio/selfupdate v0.6.1-0.20230907112617-f11e74f84ca7 // indirect
github.com/mitchellh/go-homedir v1.1.0 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/moby/term v0.5.0 // indirect
github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826 // indirect
github.com/muesli/reflow v0.3.0 // indirect
github.com/muesli/termenv v0.15.1 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.0.2 // indirect
github.com/opencontainers/runc v1.1.12 // indirect
github.com/pelletier/go-toml/v2 v2.0.8 // indirect
github.com/pierrec/lz4/v4 v4.1.2 // indirect
github.com/perimeterx/marshmallow v1.1.5 // indirect
github.com/pierrec/lz4/v4 v4.1.21 // indirect
github.com/pjbgf/sha1cd v0.3.0 // indirect
github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 // indirect
github.com/projectdiscovery/asnmap v1.1.0 // indirect
@ -195,7 +207,6 @@ require (
github.com/shoenig/go-m1cpu v0.1.6 // indirect
github.com/sirupsen/logrus v1.9.3 // indirect
github.com/skeema/knownhosts v1.2.1 // indirect
github.com/smartystreets/assertions v1.0.0 // indirect
github.com/tidwall/btree v1.7.0 // indirect
github.com/tidwall/buntdb v1.3.0 // indirect
github.com/tidwall/gjson v1.17.0 // indirect
@ -258,8 +269,8 @@ require (
github.com/libdns/libdns v0.2.1 // indirect
github.com/lor00x/goldap v0.0.0-20180618054307-a546dffdd1a3 // indirect
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect
github.com/mattn/go-isatty v0.0.19 // indirect
github.com/mattn/go-runewidth v0.0.14 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mattn/go-runewidth v0.0.15 // indirect
github.com/mholt/acmez v1.2.0 // indirect
github.com/microcosm-cc/bluemonday v1.0.26 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
@ -268,7 +279,7 @@ require (
github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c // indirect
github.com/projectdiscovery/blackrock v0.0.1 // indirect
github.com/projectdiscovery/networkpolicy v0.0.8
github.com/rivo/uniseg v0.4.4 // indirect
github.com/rivo/uniseg v0.4.6 // indirect
github.com/saintfish/chardet v0.0.0-20230101081208-5e3ef4b5456d // indirect
github.com/tklauser/go-sysconf v0.3.11 // indirect
github.com/tklauser/numcpus v0.6.0 // indirect
@ -286,20 +297,20 @@ require (
go.uber.org/zap v1.25.0 // indirect
goftp.io/server/v2 v2.0.1 // indirect
golang.org/x/crypto v0.19.0 // indirect
golang.org/x/exp v0.0.0-20240112132812-db7319d0e0e3
golang.org/x/exp v0.0.0-20240119083558-1b970713d09a
golang.org/x/mod v0.14.0 // indirect
golang.org/x/sys v0.17.0 // indirect
golang.org/x/time v0.3.0 // indirect
golang.org/x/tools v0.17.0
google.golang.org/appengine v1.6.7 // indirect
google.golang.org/protobuf v1.31.0 // indirect
google.golang.org/protobuf v1.32.0 // indirect
gopkg.in/alecthomas/kingpin.v2 v2.2.6 // indirect
gopkg.in/corvus-ch/zbase32.v1 v1.0.0 // indirect
)
require (
github.com/Microsoft/go-winio v0.6.1 // indirect
github.com/ProtonMail/go-crypto v0.0.0-20230828082145-3c4c8a2d2371 // indirect
github.com/ProtonMail/go-crypto v1.1.0-alpha.0-proton // indirect
github.com/alecthomas/chroma v0.10.0
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.5 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.35 // indirect

74
go.sum
View File

@ -79,8 +79,8 @@ github.com/Nvveen/Gotty v0.0.0-20120604004816-cd527374f1e5/go.mod h1:lmUJ/7eu/Q8
github.com/OneOfOne/xxhash v1.2.2 h1:KMrpdQIwFcEqXDklaen+P1axHaj9BSKzvpUUfnHldSE=
github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
github.com/ProtonMail/go-crypto v0.0.0-20230217124315-7d5c6f04bbb8/go.mod h1:I0gYDMZ6Z5GRU7l58bNFSkPTFN6Yl12dsUlAZ8xy98g=
github.com/ProtonMail/go-crypto v0.0.0-20230828082145-3c4c8a2d2371 h1:kkhsdkhsCvIsutKu5zLMgWtgh9YxGCNAw8Ad8hjwfYg=
github.com/ProtonMail/go-crypto v0.0.0-20230828082145-3c4c8a2d2371/go.mod h1:EjAoLdwvbIOoOQr3ihjnSoLZRtE8azugULFRteWMNc0=
github.com/ProtonMail/go-crypto v1.1.0-alpha.0-proton h1:P5Wd8eQ6zAzT4HpJI67FDKnTSf3xiJGQFqY1agDJPy4=
github.com/ProtonMail/go-crypto v1.1.0-alpha.0-proton/go.mod h1:rA3QumHc/FZ8pAHreoekgiAbzpNsfQAosU5td4SnOrE=
github.com/PuerkitoBio/goquery v1.8.1 h1:uQxhNlArOIdbrH1tr0UXwdVFgDcZDrZVdcpygAcwmWM=
github.com/PuerkitoBio/goquery v1.8.1/go.mod h1:Q8ICL1kNUJ2sXGoAhPGUdYDJvgQgHzJsnnd3H7Ho5jQ=
github.com/RumbleDiscovery/rumble-tools v0.0.0-20201105153123-f2adbb3244d2/go.mod h1:jD2+mU+E2SZUuAOHZvZj4xP4frlOo+N/YrXDvASFhkE=
@ -200,7 +200,6 @@ github.com/bsm/ginkgo/v2 v2.9.5/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbA
github.com/bsm/gomega v1.26.0 h1:LhQm+AFcgV2M0WyKroMASzAzCAJVpAxQXv4SaI9a69Y=
github.com/bsm/gomega v1.26.0/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0=
github.com/bwesterb/go-ristretto v1.2.0/go.mod h1:fUIoIZaG73pV5biE2Blr2xEzDoMj7NFEuV9ekS419A0=
github.com/bwesterb/go-ristretto v1.2.3/go.mod h1:fUIoIZaG73pV5biE2Blr2xEzDoMj7NFEuV9ekS419A0=
github.com/bytedance/sonic v1.5.0/go.mod h1:ED5hyg4y6t3/9Ku1R6dU/4KyJ48DZ4jPhfY1O2AihPM=
github.com/bytedance/sonic v1.9.1 h1:6iJ6NqdoxCDr6mbY8h18oSO+cShGSMRGCEo7F2h0x8s=
github.com/bytedance/sonic v1.9.1/go.mod h1:i736AoUSYt75HyZLoJW9ERYxcy6eaN6h4BZXU064P/U=
@ -230,12 +229,13 @@ github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5P
github.com/chzyer/readline v1.5.0/go.mod h1:x22KAscuvRqlLoK9CsoYsmxoXZMMFVyOl86cAH8qUic=
github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
github.com/chzyer/test v0.0.0-20210722231415-061457976a23/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
github.com/clbanning/mxj/v2 v2.7.0 h1:WA/La7UGCanFe5NpHF0Q3DNtnCsVoxbPKuyBNHWRyME=
github.com/clbanning/mxj/v2 v2.7.0/go.mod h1:hNiWqW14h+kc+MdF9C6/YoRfjEJoR3ou6tn/Qo+ve2s=
github.com/clbanning/x2j v0.0.0-20191024224557-825249438eec/go.mod h1:jMjuTZXRI4dUb/I5gc9Hdhagfvm9+RyrPryS/auMzxE=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/cloudflare/cfssl v1.6.4 h1:NMOvfrEjFfC63K3SGXgAnFdsgkmiq4kATme5BfcqrO8=
github.com/cloudflare/cfssl v1.6.4/go.mod h1:8b3CQMxfWPAeom3zBnGJ6sd+G1NkL5TXqmDXacb+1J0=
github.com/cloudflare/circl v1.1.0/go.mod h1:prBCrKB9DV4poKZY1l9zBXg2QJY7mvgRvtMxxK7fi4I=
github.com/cloudflare/circl v1.3.3/go.mod h1:5XYMA4rFBvNIrhs50XuiBJ15vF2pZn4nnUKZrLbUZFA=
github.com/cloudflare/circl v1.3.7 h1:qlCDlTPz2n9fu58M0Nh1J/JzcFpfgkFHHX3O35r5vcU=
github.com/cloudflare/circl v1.3.7/go.mod h1:sRTcRWXGLrKw6yIGJ+l7amYJFfAXbZG0kBSc8r4zxgA=
github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc=
@ -330,6 +330,8 @@ github.com/gabriel-vasile/mimetype v1.4.2 h1:w5qFW6JKBz9Y393Y4q372O9A7cUSequkh1Q
github.com/gabriel-vasile/mimetype v1.4.2/go.mod h1:zApsH/mKG4w07erKIaJPFiX0Tsq9BFQgN3qGY5GnNgA=
github.com/geoffgarside/ber v1.1.0 h1:qTmFG4jJbwiSzSXoNJeHcOprVzZ8Ulde2Rrrifu5U9w=
github.com/geoffgarside/ber v1.1.0/go.mod h1:jVPKeCbj6MvQZhwLYsGwaGI52oUorHoHKNecGT85ZCc=
github.com/getkin/kin-openapi v0.123.0 h1:zIik0mRwFNLyvtXK274Q6ut+dPh6nlxBp0x7mNrPhs8=
github.com/getkin/kin-openapi v0.123.0/go.mod h1:wb1aSZA/iWmorQP9KTAS/phLj/t17B5jT7+fS8ed9NM=
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
github.com/gin-contrib/sse v0.1.0 h1:Y/yl/+YNO8GZSjAhjMsSuLt29uWRFHdHYUb5lYOV9qE=
github.com/gin-contrib/sse v0.1.0/go.mod h1:RHrZQHXnP2xjPF+u1gW/2HnVO7nvIa9PG3Gm+fLHvGI=
@ -369,6 +371,10 @@ github.com/go-logr/logr v1.2.4 h1:g01GSCwiDw2xSZfjJ2/T9M+S6pFdcNtFYsp+Y43HYDQ=
github.com/go-logr/logr v1.2.4/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-ole/go-ole v1.2.6 h1:/Fpf6oFPoeFik9ty7siob0G6Ke8QvQEuVcuChpwXzpY=
github.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0=
github.com/go-openapi/jsonpointer v0.20.2 h1:mQc3nmndL8ZBzStEo3JYF8wzmeWffDH4VbXz58sAx6Q=
github.com/go-openapi/jsonpointer v0.20.2/go.mod h1:bHen+N0u1KEO3YlmqOjTT9Adn1RfD91Ar825/PuiRVs=
github.com/go-openapi/swag v0.22.9 h1:XX2DssF+mQKM2DHsbgZK74y/zj4mo9I99+89xUmuZCE=
github.com/go-openapi/swag v0.22.9/go.mod h1:3/OXnFfnMAwBD099SwYRk7GD3xOrr1iL7d/XNLXVVwE=
github.com/go-pg/pg v8.0.7+incompatible h1:ty/sXL1OZLo+47KK9N8llRcmbA9tZasqbQ/OO4ld53g=
github.com/go-pg/pg v8.0.7+incompatible/go.mod h1:a2oXow+aFOrvwcKs3eIA0lNFmMilrxK2sOkB5NWe0vA=
github.com/go-playground/assert/v2 v2.2.0 h1:JvknZsQTYeFEAhQwI4qEt9cyV5ONwRHC+lYKSsYSR8s=
@ -389,6 +395,8 @@ github.com/go-sql-driver/mysql v1.7.1/go.mod h1:OXbVy3sEdcQ2Doequ6Z5BW6fXNQTmx+9
github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572 h1:tfuBGBXKqDEevZMzYi5KSi8KkcZtzBcTgAUUtapy0OI=
github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572/go.mod h1:9Pwr4B2jHnOSGXyyzV8ROjYa2ojvAY6HCGYYfMoC3Ls=
github.com/go-test/deep v1.0.8 h1:TDsG77qcSprGbC6vTN8OuXp5g+J+b5Pcguhf7Zt61VM=
github.com/go-test/deep v1.0.8/go.mod h1:5C2ZWiW0ErCdrYzpqxLbTX7MG14M9iiw8DgHncVwcsE=
github.com/goburrow/cache v0.1.4 h1:As4KzO3hgmzPlnaMniZU9+VmoNYseUhuELbxy9mRBfw=
github.com/goburrow/cache v0.1.4/go.mod h1:cDFesZDnIlrHoNlMYqqMpCRawuXulgx+y7mXU8HZ+/c=
github.com/gobwas/httphead v0.1.0 h1:exrUm0f4YX0L7EBwZHuCF4GDp8aJfVeBrlLQrs6NqWU=
@ -559,10 +567,13 @@ github.com/iancoleman/orderedmap v0.0.0-20190318233801-ac98e3ecb4b0 h1:i462o439Z
github.com/iancoleman/orderedmap v0.0.0-20190318233801-ac98e3ecb4b0/go.mod h1:N0Wam8K1arqPXNWjMo21EXnBPOPp36vB07FNRdD2geA=
github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
github.com/ianlancetaylor/demangle v0.0.0-20220319035150-800ac71e25c2/go.mod h1:aYm2/VgdVmcIU8iMfdMvDMsRAQjcfZSKFby6HOFvi/w=
github.com/imdario/mergo v0.3.12/go.mod h1:jmQim1M+e3UYxmgPu/WyfjB3N3VflVyUjjjwH0dnCYA=
github.com/imdario/mergo v0.3.16 h1:wwQJbIsHYGMUyLSPrEq1CT16AhnhNJQ51+4fdHUnCl4=
github.com/imdario/mergo v0.3.16/go.mod h1:WBLT9ZmE3lPoWsEzCh9LPo3TiwVN+ZKEjmz+hD27ysY=
github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
github.com/influxdata/influxdb1-client v0.0.0-20191209144304-8bf82d3c094d/go.mod h1:qj24IKcXYK6Iy9ceXlo3Tc+vtHo9lIhSX5JddghvEPo=
github.com/invopop/yaml v0.2.0 h1:7zky/qH+O0DwAyoobXUqvVBwgBFRxKoQ/3FjcVpjTMY=
github.com/invopop/yaml v0.2.0/go.mod h1:2XuRLgs/ouIrW3XNzuNj7J3Nvu/Dig5MXvbCEdiBN3Q=
github.com/itchyny/gojq v0.12.13 h1:IxyYlHYIlspQHHTE0f3cJF0NKDMfajxViuhBLnHd/QU=
github.com/itchyny/gojq v0.12.13/go.mod h1:JzwzAqenfhrPUuwbmEz3nu3JQmFLlQTQMUcOdnu/Sf4=
github.com/itchyny/timefmt-go v0.1.5 h1:G0INE2la8S6ru/ZI5JecgyzbbJNs5lG1RcBqa7Jm6GE=
@ -593,6 +604,8 @@ github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHW
github.com/jmespath/go-jmespath/internal/testify v1.5.1 h1:shLQSRRSCCPj3f2gpwzGwWFoC7ycTf1rcQZHOlsJ6N8=
github.com/jmespath/go-jmespath/internal/testify v1.5.1/go.mod h1:L3OGu8Wl2/fWfCI6z80xFu9LTZmf1ZRjMHUOPmWr69U=
github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY=
github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y=
github.com/jpillora/backoff v1.0.0/go.mod h1:J/6gKK9jxlEcS3zixgDgUAsiuZ7yrSoa/FX5e0EB2j4=
github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
github.com/json-iterator/go v1.1.7/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
@ -618,15 +631,16 @@ github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/klauspost/compress v1.4.1/go.mod h1:RyIbtBH6LamlWaDj8nUwkbUhJ87Yi3uG0guNDohfE1A=
github.com/klauspost/compress v1.11.4/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs=
github.com/klauspost/compress v1.17.4 h1:Ej5ixsIri7BrIjBkRZLTo6ghwrEtHFk7ijlczPW4fZ4=
github.com/klauspost/compress v1.17.4/go.mod h1:/dCuZOvVtNoHsyb+cuJD3itjs3NbnF6KH9zAO4BDxPM=
github.com/klauspost/compress v1.17.6 h1:60eq2E/jlfwQXtvZEeBUYADs+BwKBWURIY+Gj2eRGjI=
github.com/klauspost/compress v1.17.6/go.mod h1:/dCuZOvVtNoHsyb+cuJD3itjs3NbnF6KH9zAO4BDxPM=
github.com/klauspost/cpuid v1.2.0/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgoMS4s3ek=
github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
github.com/klauspost/cpuid/v2 v2.0.12/go.mod h1:g2LTdtYhdyuGPqyWyv7qRAmj1WBqxuObKfj5c0PQa7c=
github.com/klauspost/cpuid/v2 v2.2.5 h1:0E5MSMDEoAulmXNFquVs//DdoomxaoTY1kUhbc/qbZg=
github.com/klauspost/cpuid/v2 v2.2.5/go.mod h1:Lcz8mBdAVJIBVzewtcLocK12l3Y+JytZYpaMropDUws=
github.com/klauspost/pgzip v1.2.5 h1:qnWYvvKqedOF2ulHpMG72XQol4ILEJ8k2wwRl/Km8oE=
github.com/klauspost/pgzip v1.2.5/go.mod h1:Ch1tH69qFZu15pkjo5kYi6mth2Zzwzt50oCQKQE9RUs=
github.com/klauspost/pgzip v1.2.6 h1:8RXeL5crjEUFnR2/Sn6GJNWtSQ3Dk8pq4CL3jvdDyjU=
github.com/klauspost/pgzip v1.2.6/go.mod h1:Ch1tH69qFZu15pkjo5kYi6mth2Zzwzt50oCQKQE9RUs=
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/konsorten/go-windows-terminal-sequences v1.0.3/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
@ -648,6 +662,8 @@ github.com/labstack/gommon v0.4.0 h1:y7cvthEAEbU0yHOf4axH8ZG2NH8knB9iNSoTO8dyIk8
github.com/labstack/gommon v0.4.0/go.mod h1:uW6kP17uPlLJsD3ijUYn3/M5bAxtlZhMI6m3MFxTMTM=
github.com/leodido/go-urn v1.2.4 h1:XlAE/cm/ms7TE/VMVoduSpNBoyc2dOxHs5MZSwAN63Q=
github.com/leodido/go-urn v1.2.4/go.mod h1:7ZrI8mTSeBSHl/UaRyKQW1qZeMgak41ANeCNaVckg+4=
github.com/leslie-qiwa/flat v0.0.0-20230424180412-f9d1cf014baa h1:KQKuQDgA3DZX6C396lt3WDYB9Um1gLITLbvficVbqXk=
github.com/leslie-qiwa/flat v0.0.0-20230424180412-f9d1cf014baa/go.mod h1:HbwNE4XGwjgtUELkvQaAOjWrpianHYZdQVNqSdYW3UM=
github.com/lib/pq v1.10.1 h1:6VXZrLU0jHBYyAqrSPa+MgPfnSvTPuMgK+k0o5kVFWo=
github.com/lib/pq v1.10.1/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o=
github.com/libdns/libdns v0.2.1 h1:Wu59T7wSHRgtA0cfxC+n1c/e+O3upJGWytknkmFEDis=
@ -665,6 +681,8 @@ github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0/go.mod h1:zJYVVT2
github.com/lyft/protoc-gen-validate v0.0.13/go.mod h1:XbGvPuh87YZc5TdIa2/I4pLk0QoUACkjt2znoq26NVQ=
github.com/mackerelio/go-osstat v0.2.4 h1:qxGbdPkFo65PXOb/F/nhDKpF2nGmGaCFDLXoZjJTtUs=
github.com/mackerelio/go-osstat v0.2.4/go.mod h1:Zy+qzGdZs3A9cuIqmgbJvwbmLQH9dJvtio5ZjJTbdlQ=
github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0=
github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc=
github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU=
github.com/mattn/go-colorable v0.1.11/go.mod h1:u5H1YNBxpqRaxsYJYSkiCWKzEfiAb1Gb520KVy5xxl4=
github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA=
@ -673,13 +691,16 @@ github.com/mattn/go-isatty v0.0.3/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNx
github.com/mattn/go-isatty v0.0.4/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4=
github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94=
github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM=
github.com/mattn/go-isatty v0.0.19 h1:JITubQf0MOLdlGRuRq+jtsDlekdYPia9ZFsB8h/APPA=
github.com/mattn/go-isatty v0.0.19/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
github.com/mattn/go-runewidth v0.0.2/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzpuz5H//U1FU=
github.com/mattn/go-runewidth v0.0.9/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI=
github.com/mattn/go-runewidth v0.0.12/go.mod h1:RAqKPSqVFrSLVXbA8x7dzmKdmGzieGRCM46jaSJTDAk=
github.com/mattn/go-runewidth v0.0.14 h1:+xnbZSEeDbOIg5/mE6JF0w6n9duR1l3/WmbinWVwUuU=
github.com/mattn/go-runewidth v0.0.14/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
github.com/mattn/go-runewidth v0.0.15 h1:UNAjwbU9l54TA3KzvqLGxwWjHmMgBUVhBiTjelZgg3U=
github.com/mattn/go-runewidth v0.0.15/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
github.com/mattn/go-sqlite3 v1.14.22 h1:2gZY6PC6kBnID23Tichd1K+Z0oS6nE/XwU+Vz/5o4kU=
github.com/mattn/go-sqlite3 v1.14.22/go.mod h1:Uh1q+B4BYcTPb+yiD3kU8Ct7aC0hY9fxUwlHK0RXw+Y=
github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
github.com/matttproud/golang_protobuf_extensions v1.0.4/go.mod h1:BSXmuO+STAnVfrANrmjBb36TMTDstsz7MSK+HVaYKv4=
github.com/mholt/acmez v1.2.0 h1:1hhLxSgY5FvH5HCnGUuwbKY2VQVo8IU7rxXKSnZ7F30=
@ -720,6 +741,8 @@ github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3Rllmb
github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M=
github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
github.com/modocache/gover v0.0.0-20171022184752-b58185e213c5/go.mod h1:caMODM3PzxT8aQXRPkAt8xlV/e7d7w8GM5g0fa5F0D8=
github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826 h1:RWengNIwukTxcDr9M+97sNutRR1RKhG96O6jWumTTnw=
github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826/go.mod h1:TaXosZuwdSHYgviHp1DAtfrULt5eUgsSMsZf+YrPgl8=
github.com/mreiferson/go-httpclient v0.0.0-20160630210159-31f0106b4474/go.mod h1:OQA4XLvDbMgS8P0CevmM4m9Q3Jq4phKUzcocxuGJ5m8=
github.com/mreiferson/go-httpclient v0.0.0-20201222173833-5e475fde3a4d/go.mod h1:OQA4XLvDbMgS8P0CevmM4m9Q3Jq4phKUzcocxuGJ5m8=
github.com/muesli/reflow v0.3.0 h1:IFsN6K9NfGtjeggFP+68I4chLZV2yIKsXJFNZ+eWh6s=
@ -779,12 +802,15 @@ github.com/pelletier/go-buffruneio v0.2.0/go.mod h1:JkE26KsDizTr40EUHkXVtNPvgGtb
github.com/pelletier/go-toml/v2 v2.0.8 h1:0ctb6s9mE31h0/lhu+J6OPmVeDxJn+kYnJc2jZR9tGQ=
github.com/pelletier/go-toml/v2 v2.0.8/go.mod h1:vuYfssBdrU2XDZ9bYydBu6t+6a6PYNcZljzZR9VXg+4=
github.com/performancecopilot/speed v3.0.0+incompatible/go.mod h1:/CLtqpZ5gBg1M9iaPbIdPPGyKcA8hKdoy6hAWba7Yac=
github.com/perimeterx/marshmallow v1.1.5 h1:a2LALqQ1BlHM8PZblsDdidgv1mWi1DgC2UmX50IvK2s=
github.com/perimeterx/marshmallow v1.1.5/go.mod h1:dsXbUu8CRzfYP5a87xpp0xq9S3u0Vchtcl8we9tYaXw=
github.com/pierrec/lz4 v1.0.2-0.20190131084431-473cd7ce01a1/go.mod h1:3/3N9NVKO0jef7pBehbT1qWhCMrIgbYNnFAZCqQ5LRc=
github.com/pierrec/lz4 v2.0.5+incompatible/go.mod h1:pdkljMzZIN41W+lC3N2tnIh5sFi+IEE17M5jbnwPHcY=
github.com/pierrec/lz4 v2.6.1+incompatible h1:9UY3+iC23yxF0UfGaYrGplQ+79Rg+h/q9FV9ix19jjM=
github.com/pierrec/lz4 v2.6.1+incompatible/go.mod h1:pdkljMzZIN41W+lC3N2tnIh5sFi+IEE17M5jbnwPHcY=
github.com/pierrec/lz4/v4 v4.1.2 h1:qvY3YFXRQE/XB8MlLzJH7mSzBs74eA2gg52YTk6jUPM=
github.com/pierrec/lz4/v4 v4.1.2/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=
github.com/pierrec/lz4/v4 v4.1.21 h1:yOVMLb6qSIDP67pl/5F7RepeKYu/VmTyEXvuMI5d9mQ=
github.com/pierrec/lz4/v4 v4.1.21/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=
github.com/pjbgf/sha1cd v0.3.0 h1:4D5XXmUUBUl/xQ6IjCkEAbqXskkq/4O7LmGn0AqMDs4=
github.com/pjbgf/sha1cd v0.3.0/go.mod h1:nZ1rrWOcGJ5uZgEEVL1VUM9iRQiZvWdbZjkKyFzPPsI=
github.com/pkg/browser v0.0.0-20180916011732-0a3d74bf9ce4/go.mod h1:4OwLy04Bl9Ef3GJJCoec+30X3LQs/0/m4HFRt/2LUSA=
@ -915,13 +941,13 @@ github.com/remeh/sizedwaitgroup v1.0.0 h1:VNGGFwNo/R5+MJBf6yrsr110p0m4/OX4S3DCy7
github.com/remeh/sizedwaitgroup v1.0.0/go.mod h1:3j2R4OIe/SeS6YDhICBy22RWjJC5eNCJ1V+9+NVNYlo=
github.com/rivo/uniseg v0.1.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
github.com/rivo/uniseg v0.4.4 h1:8TfxU8dW6PdqD27gjM8MVNuicgxIjxpm4K7x4jp8sis=
github.com/rivo/uniseg v0.4.4/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
github.com/rivo/uniseg v0.4.6 h1:Sovz9sDSwbOz9tgUy8JpT+KgCkPYJEN/oYzlJiYTNLg=
github.com/rivo/uniseg v0.4.6/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg=
github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/rogpeppe/go-internal v1.6.1/go.mod h1:xXDCJY+GAPziupqXw64V24skbSoqbTEfhy4qGm1nDQc=
github.com/rogpeppe/go-internal v1.11.0 h1:cWPaGQEPrBb5/AsnsZesgZZ9yb1OQ+GOISoDNXVBh4M=
github.com/rogpeppe/go-internal v1.11.0/go.mod h1:ddIwULY96R17DhadqLgMfk9H9tvdUzkipdSkR5nkCZA=
github.com/rogpeppe/go-internal v1.12.0 h1:exVL4IDcn6na9z1rAb56Vxr+CgyK3nn3O+epU5NdKM8=
github.com/rogpeppe/go-internal v1.12.0/go.mod h1:E+RYuTGaKKdloAfM02xzb0FW3Paa99yedzYV+kq4uf4=
github.com/rs/xid v1.5.0 h1:mKX4bl4iPYJtEIxp6CYiUuLQ/8DYMoz0PUdtGgMFRVc=
github.com/rs/xid v1.5.0/go.mod h1:trrq9SKmegXys3aeAKXMUTdJsYXVwGY3RLcfgqegfbg=
github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
@ -935,6 +961,8 @@ github.com/sashabaranov/go-openai v1.15.3/go.mod h1:lj5b/K+zjTSFxVLijLSTDZuP7adO
github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529/go.mod h1:DxrIzT+xaE7yg65j358z/aeFdxmN0P9QXhEzd20vsDc=
github.com/segmentio/ksuid v1.0.4 h1:sBo2BdShXjmcugAMwjugoGUdUV0pcxY5mW4xKRn3v4c=
github.com/segmentio/ksuid v1.0.4/go.mod h1:/XUiZBD3kVx5SmUOl55voK5yeAbBNNIed+2O73XgrPE=
github.com/seh-msft/burpxml v1.0.1 h1:5G3QPSzvfA1WcX7LkxmKBmK2RnNyGviGWnJPumE0nwg=
github.com/seh-msft/burpxml v1.0.1/go.mod h1:lTViCHPtGGS0scK0B4krm6Ld1kVZLWzQccwUomRc58I=
github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo=
github.com/sergi/go-diff v1.2.0 h1:XU+rvMAioB0UC3q1MFrIQy4Vo5/4VsRDQQXHsEya6xQ=
github.com/sergi/go-diff v1.2.0/go.mod h1:STckp+ISIX8hZLjrqAeVduY0gWCT9IjLuqbuNXdaHfM=
@ -955,9 +983,8 @@ github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ
github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
github.com/skeema/knownhosts v1.2.1 h1:SHWdIUa82uGZz+F+47k8SY4QhhI291cXCpopT1lK2AQ=
github.com/skeema/knownhosts v1.2.1/go.mod h1:xYbVRSPxqBZFrdmDyMmsOs+uX1UZC3nTN3ThzgDxUwo=
github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d h1:zE9ykElWQ6/NYmHa3jpm/yHnI4xSofP+UP6SpjHcSeM=
github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
github.com/smartystreets/assertions v1.0.0 h1:UVQPSSmc3qtTi+zPPkCXvZX9VvW/xT/NsRvKfwY81a8=
github.com/smartystreets/assertions v1.0.0/go.mod h1:kHHU4qYBaI3q23Pp3VPrmWhuIUrLW/7eUrw0BU5VaoM=
github.com/smartystreets/goconvey v0.0.0-20190330032615-68dc04aab96a/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
github.com/smartystreets/goconvey v1.6.4 h1:fv0U8FUIMPNf1L9lnHLvLhgicrIVChEkdzIKYqbNC9s=
github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
@ -1162,7 +1189,6 @@ golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5y
golang.org/x/crypto v0.0.0-20211209193657-4570a0811e8b/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
golang.org/x/crypto v0.0.0-20220525230936-793ad666bf5e/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
golang.org/x/crypto v0.3.1-0.20221117191849-2c476679df9a/go.mod h1:hebNnKkNXi2UzZN1eVRvBB7co0a+JxK6XbPiWVs/3J4=
golang.org/x/crypto v0.6.0/go.mod h1:OFC/31mSvZgRz0V1QTNCzfAI1aIRzbiufJtkMIlEp58=
golang.org/x/crypto v0.7.0/go.mod h1:pYwdfH91IfpZVANVyUOhSIPZaFoJGxTFbZhFTx+dXZU=
golang.org/x/crypto v0.11.0/go.mod h1:xgJhtzW8F9jGdVFWZESrid1U1bjeNy4zgy5cRr/CIio=
@ -1179,8 +1205,8 @@ golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u0
golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/exp v0.0.0-20200207192155-f17229e696bd/go.mod h1:J/WKrq2StrnmMY6+EHIKF9dgMWnmCNThgcyBT1FY9mM=
golang.org/x/exp v0.0.0-20200224162631-6cc2880d07d6/go.mod h1:3jZMyOhIsHpP37uCMkUooju7aAi5cS1Q23tOzKc+0MU=
golang.org/x/exp v0.0.0-20240112132812-db7319d0e0e3 h1:hNQpMuAJe5CtcUqCXaWga3FHu+kQvCqcsoVaQgSV60o=
golang.org/x/exp v0.0.0-20240112132812-db7319d0e0e3/go.mod h1:idGWGoKP1toJGkd5/ig9ZLuPcZBC3ewk7SzmH0uou08=
golang.org/x/exp v0.0.0-20240119083558-1b970713d09a h1:Q8/wZp0KX97QFTc2ywcOE0YRjZPVIx+MXInMzdvQqcA=
golang.org/x/exp v0.0.0-20240119083558-1b970713d09a/go.mod h1:idGWGoKP1toJGkd5/ig9ZLuPcZBC3ewk7SzmH0uou08=
golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
@ -1254,7 +1280,6 @@ golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd/go.mod h1:CfG3xpIq0wQ8r1q4Su
golang.org/x/net v0.0.0-20220225172249-27dd8689420f/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.0.0-20221002022538-bcab6841153b/go.mod h1:YDH+HFinaLZZlnHAfSS6ZXJJ9M9t4Dl22yv3iI2vPwk=
golang.org/x/net v0.2.0/go.mod h1:KqCZLdyyvdV855qA2rE3GC2aiw5xGR5TEjj8smXukLY=
golang.org/x/net v0.5.0/go.mod h1:DivGGAXEgPSlEBzxGzZI+ZLohi+xUj054jfeKui00ws=
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
@ -1371,7 +1396,6 @@ golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXR
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210220032956-6a3ed077a48d/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.2.0/go.mod h1:TVmDHMZPmdnySmBfhjOoOdhjzdE1h4u1VwSiw2l1Nuc=
golang.org/x/term v0.4.0/go.mod h1:9P2UbLfCdcvo3p/nzKvsmas4TnlujnuoV9hGgYzW1lQ=
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
golang.org/x/term v0.6.0/go.mod h1:m6U89DPEgQRMq3DNkDClhWw02AUbt2daBVO4cn4Hv9U=
@ -1390,7 +1414,6 @@ golang.org/x/text v0.3.4/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.3.8/go.mod h1:E6s5w1FMmriuDzIBO73fBruAKo1PCIq6d2Q6DHfQ8WQ=
golang.org/x/text v0.4.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.6.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.8.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
@ -1550,8 +1573,8 @@ google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQ
google.golang.org/protobuf v1.28.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
google.golang.org/protobuf v1.28.1/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
google.golang.org/protobuf v1.30.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
google.golang.org/protobuf v1.31.0 h1:g0LDEJHgrBl9N9r17Ru3sqWhkIx2NB67okBHPwC7hs8=
google.golang.org/protobuf v1.31.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
google.golang.org/protobuf v1.32.0 h1:pPC6BG5ex8PDFnkbrGU3EixyhKcQ2aDuBS36lqK/C7I=
google.golang.org/protobuf v1.32.0/go.mod h1:c6P6GXX6sHbq/GpV6MGZEdwhWPcYBgnhAHhKbcUYpos=
gopkg.in/alecthomas/kingpin.v2 v2.2.6 h1:jMFz6MfLP0/4fUyZle81rXUoxOBFi19VUFKVDOQfozc=
gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
@ -1591,6 +1614,7 @@ gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.0/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gotest.tools/v3 v3.3.0 h1:MfDY1b1/0xN1CyMlQDac0ziEy9zJQd9CXBRRDHw2jJo=

39
integration_tests/fuzz/fuzz-body-generic-sqli.yaml Normal file
View File

@ -0,0 +1,39 @@
id: fuzz-body-generic
info:
name: fuzzing error sqli payloads in http req body
author: pdteam
severity: info
description: |
This template attempts to find SQL injection vulnerabilities by fuzzing http body
It automatically handles and parses json,xml,multipart form and x-www-form-urlencoded data
and performs fuzzing on the value of every key
http:
- filters:
- type: dsl
dsl:
- method != "GET"
- method != "HEAD"
- contains(path, "/user") # for scope of integration test
condition: and
payloads:
injection:
- "'"
- "\""
- ";"
fuzzing:
- part: body
type: postfix
mode: single
fuzz:
- '{{injection}}'
stop-at-first-match: true
matchers:
- type: word
words:
- "unrecognized token:"
- "null"

40
integration_tests/fuzz/fuzz-body-json-sqli.yaml Normal file
View File

@ -0,0 +1,40 @@
id: json-body-error-sqli
info:
name: fuzzing error sqli payloads in json body
author: pdteam
severity: info
description: |
This template attempts to find SQL injection vulnerabilities by fuzzing http body of json type.
This is achieved by performing [ruleType](example: postfix) on value of json key
Note: this is example template, and payloads/matchers need to be modified appropriately.
http:
- filters:
- type: dsl
dsl:
- method != "GET"
- method != "HEAD"
- contains(content_type, "application/json")
- contains(path, "/user") # for scope of integration test
condition: and
payloads:
injection:
- "'"
- "\""
- ";"
fuzzing:
- part: body
type: postfix
mode: single
fuzz:
- '{{injection}}'
stop-at-first-match: true
matchers:
- type: word
words:
- "unrecognized token:"
- "null"

41
integration_tests/fuzz/fuzz-body-multipart-form-sqli.yaml Normal file
View File

@ -0,0 +1,41 @@
id: body-multipart-error-sqli
info:
name: fuzzing error sqli payloads in body of multipart form data
author: pdteam
severity: info
description: |
This template attempts to find SQL injection vulnerabilities by fuzzing http body of multipart form data (file upload, etc.)
This is achieved by performing [ruleType](example: postfix) on value of body form key
Note: this is example template, and payloads/matchers need to be modified appropriately.
http:
- filters:
- type: dsl
dsl:
- method != "GET"
- method != "HEAD"
- contains(content_type, "multipart/form-data")
- contains(path, "/user") # for scope of integration test
condition: and
payloads:
injection:
- "'"
- "\""
- ";"
fuzzing:
- part: body
type: postfix
mode: single
fuzz:
- '{{injection}}'
stop-at-first-match: true
matchers:
- type: word
words:
- "unrecognized token:"
- "null"
- "SELECTs to the left and right of UNION do not have the same number of result columns"

41
integration_tests/fuzz/fuzz-body-params-sqli.yaml Normal file
View File

@ -0,0 +1,41 @@
id: body-params-error-sqli
info:
name: fuzzing error sqli payloads in body with params
author: pdteam
severity: info
description: |
This template attempts to find SQL injection vulnerabilities by fuzzing http body of x-www-form-urlencoded data
This is achieved by performing [ruleType](example: postfix) on value of body key
Note: this is example template, and payloads/matchers need to be modified appropriately.
http:
- filters:
- type: dsl
dsl:
- method != "GET"
- method != "HEAD"
- contains(content_type, "application/x-www-form-urlencoded")
- contains(path, "/user") # for scope of integration test
condition: and
payloads:
injection:
- "'"
- "\""
- ";"
fuzzing:
- part: body
type: postfix
mode: single
fuzz:
- '{{injection}}'
stop-at-first-match: true
matchers:
- type: word
words:
- "unrecognized token:"
- "null"
- "SELECTs to the left and right of UNION do not have the same number of result columns"

40
integration_tests/fuzz/fuzz-body-xml-sqli.yaml Normal file
View File

@ -0,0 +1,40 @@
id: xml-body-error-sqli
info:
name: fuzzing error sqli payloads in xml body
author: pdteam
severity: info
description: |
This template attempts to find SQL injection vulnerabilities by fuzzing http body of xml type.
This is achieved by performing [ruleType](example: postfix) on value of xml key
Note: this is example template, and payloads/matchers need to be modified appropriately.
http:
- filters:
- type: dsl
dsl:
- method != "GET"
- method != "HEAD"
- contains(content_type, "application/xml")
- contains(path, "/user") # for scope of integration test
condition: and
payloads:
injection:
- "'"
- "\""
- ";"
fuzzing:
- part: body
type: postfix
mode: single
fuzz:
- '{{injection}}'
stop-at-first-match: true
matchers:
- type: word
words:
- "unrecognized token:"
- "null"

59
integration_tests/fuzz/fuzz-cookie-error-sqli.yaml Normal file
View File

@ -0,0 +1,59 @@
id: cookie-fuzzing-error-sqli
info:
name: fuzzing error sqli payloads in cookie
author: pdteam
severity: info
description: |
This template attempts to find SQL injection vulnerabilities by fuzzing http cookies with SQL injection payloads.
Note: this is example template, and payloads/matchers need to be modified appropriately.
http:
- filters:
- type: dsl
dsl:
- 'method == "GET"'
- len(cookie) > 0
condition: and
payloads:
sqli:
- "'"
- ''
- '`'
- '``'
- ','
- '"'
- ""
- /
- //
- \
- \\
- ;
- -- or #
- '" OR 1 = 1 -- -'
- ' OR '' = '
- '='
- 'LIKE'
- "'=0--+"
- OR 1=1
- "' OR 'x'='x"
- "' AND id IS NULL; --"
- "'''''''''''''UNION SELECT '2"
- '%00'
fuzzing:
- part: cookie
type: postfix
mode: single
fuzz:
- '{{sqli}}'
stop-at-first-match: true
matchers:
- type: word
words:
- "unrecognized token:"
- "syntax error"
- "null"
- "SELECTs to the left and right of UNION do not have the same number of result columns"

2
integration_tests/fuzz/fuzz-header-basic.yaml
View File

@ -28,7 +28,7 @@ http:
- "'\"><{{first}}"
fuzzing:
- part: headers
- part: header
type: replace
mode: single
keys: ["Origin"]

2
integration_tests/fuzz/fuzz-header-multiple.yaml
View File

@ -25,7 +25,7 @@ http:
- "secret.local"
fuzzing:
- part: headers
- part: header
type: replace
mode: multiple
keys: ["Origin", "X-Forwared-For"]

43
integration_tests/fuzz/fuzz-host-header-injection.yaml Normal file
View File

@ -0,0 +1,43 @@
id: host-header-injection
info:
name: Host Header Injection
author: pdteam
severity: info
description: Host header injection
variables:
domain: "oast.fun"
http:
- filters:
- type: dsl
dsl:
- 'method == "GET"'
- 'contains(path,"/host-header-lab")' # for integration testing only
condition: and
fuzzing:
- part: header
type: replace
mode: single
fuzz:
X-Forwarded-For: "{{domain}}"
X-Forwarded-Host: "{{domain}}"
Forwarded: "{{domain}}"
X-Real-IP: "{{domain}}"
X-Original-URL: "{{domain}}"
X-Rewrite-URL: "{{domain}}"
Host: "{{domain}}"
# " Host": "{{domain}}" # space before host (not supported yet due to lack of unsafe mode)
matchers:
- type: status
status:
- 200
- type: word
part: body
words:
- "Interactsh"
matchers-condition: and

2
integration_tests/fuzz/fuzz-mode.yaml
View File

@ -5,7 +5,7 @@ info:
author: pdteam
severity: info
requests:
http:
- method: GET
path:
- "{{BaseURL}}"

42
integration_tests/fuzz/fuzz-path-sqli.yaml Normal file
View File

@ -0,0 +1,42 @@
id: path-based-sqli
info:
name: Path Based SQLi
author: pdteam
severity: info
description: |
This template attempts to find SQL injection vulnerabilities on path based sqli and replacing numerical values with fuzzing payloads.
ex: /admin/user/55/profile , /user/15/action/update, /posts/15, /blog/100/data, /page/51/ etc these types of paths are filtered and
replaced with sqli path payloads.
Note: this is example template, and payloads/matchers need to be modified appropriately.
http:
- filters:
- type: dsl
dsl:
- 'method == "GET"'
- regex("/(.*?/)([0-9]+)(/.*)?",path)
condition: and
payloads:
pathsqli:
- "'OR1=1"
- '%20OR%20True'
fuzzing:
- part: path
type: replace-regex
mode: single
replace-regex: '/(.*?/)([0-9]+)(/.*)?'
fuzz:
- '/${1}${2}{{pathsqli}}${3}'
matchers:
- type: status
status:
- 200
- type: word
words:
- "admin"
matchers-condition: and

39
integration_tests/fuzz/fuzz-query-num-replace.yaml Normal file
View File

@ -0,0 +1,39 @@
id: fuzz-query-num
info:
name: Fuzz Query Param For IDOR
author: pdteam
severity: info
description: Query Value Fuzzing using Fuzzing Rules
http:
- filters:
- type: dsl
dsl:
- 'len(query) > 0'
# below filter is related to integration testing
- type: word
part: path
words:
- /blog/post
filters-condition: and
payloads:
nums:
- 200
- 201
fuzzing:
- part: query
type: replace
mode: multiple
values:
- "^[0-9]+$" # only if value is number
fuzz:
- '{{nums}}'
matchers:
- type: status
status:
- 200

2
integration_tests/fuzz/fuzz-query.yaml
View File

@ -5,7 +5,7 @@ info:
author: pdteam
severity: info
requests:
http:
- method: GET
path:
- "{{BaseURL}}"

2
integration_tests/fuzz/fuzz-type.yaml
View File

@ -5,7 +5,7 @@ info:
author: pdteam
severity: info
requests:
http:
- method: GET
path:
- "{{BaseURL}}"

535
integration_tests/fuzz/testData/ginandjuice.proxify.yaml Normal file
View File

@ -0,0 +1,535 @@
timestamp: 2024-02-20T19:24:13+05:30
url: http://127.0.0.1:8082/blog/post?postId=3&source=proxify
request:
header:
Accept-Encoding: gzip
Connection: close
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
host: 127.0.0.1:8082
method: GET
path: /blog/post
scheme: https
raw: |+
GET /blog/post?postId=3&source=proxify HTTP/1.1
Host: 127.0.0.1:8082
Accept-Encoding: gzip
Connection: close
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
response:
header:
Content-Encoding: gzip
Content-Type: text/html; charset=utf-8
Date: Tue, 20 Feb 2024 13:54:13 GMT
Set-Cookie: AWSALB=9l3X2bRs5JVQp5cO8o7xLckrdo3FZIQzbS0ga0n4ctfDApb/nn0K6AiSLLAMbG7CCDHqKOj9kQdyj6T2RzBDULszJ+2Oy8KcyuOKhFsCGVTVabnJTkVwhyXLciFt; Expires=Tue, 27 Feb 2024 13:54:13 GMT; Path=/, AWSALBCORS=9l3X2bRs5JVQp5cO8o7xLckrdo3FZIQzbS0ga0n4ctfDApb/nn0K6AiSLLAMbG7CCDHqKOj9kQdyj6T2RzBDULszJ+2Oy8KcyuOKhFsCGVTVabnJTkVwhyXLciFt; Expires=Tue, 27 Feb 2024 13:54:13 GMT; Path=/; SameSite=None; Secure, session=ymwLa8dKmWjLl43BBpQnrp5LkFZraYkp; Secure; HttpOnly; SameSite=None
X-Backend: ea6c1d8c-a8e5-4bef-b8db-879bbb13cf62
X-Frame-Options: SAMEORIGIN
raw: |+
HTTP/1.1 200 OK
Connection: close
Content-Encoding: gzip
Content-Type: text/html; charset=utf-8
Date: Tue, 20 Feb 2024 13:54:13 GMT
Set-Cookie: AWSALB=9l3X2bRs5JVQp5cO8o7xLckrdo3FZIQzbS0ga0n4ctfDApb/nn0K6AiSLLAMbG7CCDHqKOj9kQdyj6T2RzBDULszJ+2Oy8KcyuOKhFsCGVTVabnJTkVwhyXLciFt; Expires=Tue, 27 Feb 2024 13:54:13 GMT; Path=/
Set-Cookie: AWSALBCORS=9l3X2bRs5JVQp5cO8o7xLckrdo3FZIQzbS0ga0n4ctfDApb/nn0K6AiSLLAMbG7CCDHqKOj9kQdyj6T2RzBDULszJ+2Oy8KcyuOKhFsCGVTVabnJTkVwhyXLciFt; Expires=Tue, 27 Feb 2024 13:54:13 GMT; Path=/; SameSite=None; Secure
Set-Cookie: session=ymwLa8dKmWjLl43BBpQnrp5LkFZraYkp; Secure; HttpOnly; SameSite=None
X-Backend: ea6c1d8c-a8e5-4bef-b8db-879bbb13cf62
X-Frame-Options: SAMEORIGIN
---
timestamp: 2024-02-20T19:24:13+05:30
url: http://127.0.0.1:8082/reset-password
request:
header:
Accept-Encoding: gzip
Connection: close
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
host: 127.0.0.1:8082
method: GET
path: /blog/post
scheme: https
raw: |+
POST /reset-password HTTP/1.1
Host: 127.0.0.1:8082
X-Forwarded-For: 127.0.0.1:8082
Accept-Encoding: gzip
Connection: close
Content-Type: application/json
Content-Length: 23
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
{"password":"12345678"}
response:
header:
Content-Encoding: gzip
Content-Type: text/html; charset=utf-8
Date: Tue, 20 Feb 2024 13:54:13 GMT
Set-Cookie: AWSALB=9l3X2bRs5JVQp5cO8o7xLckrdo3FZIQzbS0ga0n4ctfDApb/nn0K6AiSLLAMbG7CCDHqKOj9kQdyj6T2RzBDULszJ+2Oy8KcyuOKhFsCGVTVabnJTkVwhyXLciFt; Expires=Tue, 27 Feb 2024 13:54:13 GMT; Path=/, AWSALBCORS=9l3X2bRs5JVQp5cO8o7xLckrdo3FZIQzbS0ga0n4ctfDApb/nn0K6AiSLLAMbG7CCDHqKOj9kQdyj6T2RzBDULszJ+2Oy8KcyuOKhFsCGVTVabnJTkVwhyXLciFt; Expires=Tue, 27 Feb 2024 13:54:13 GMT; Path=/; SameSite=None; Secure, session=ymwLa8dKmWjLl43BBpQnrp5LkFZraYkp; Secure; HttpOnly; SameSite=None
X-Backend: ea6c1d8c-a8e5-4bef-b8db-879bbb13cf62
X-Frame-Options: SAMEORIGIN
raw: |+
HTTP/1.1 200 OK
Connection: close
Content-Encoding: gzip
Content-Type: text/html; charset=utf-8
Date: Tue, 20 Feb 2024 13:54:13 GMT
Set-Cookie: AWSALB=9l3X2bRs5JVQp5cO8o7xLckrdo3FZIQzbS0ga0n4ctfDApb/nn0K6AiSLLAMbG7CCDHqKOj9kQdyj6T2RzBDULszJ+2Oy8KcyuOKhFsCGVTVabnJTkVwhyXLciFt; Expires=Tue, 27 Feb 2024 13:54:13 GMT; Path=/
Set-Cookie: AWSALBCORS=9l3X2bRs5JVQp5cO8o7xLckrdo3FZIQzbS0ga0n4ctfDApb/nn0K6AiSLLAMbG7CCDHqKOj9kQdyj6T2RzBDULszJ+2Oy8KcyuOKhFsCGVTVabnJTkVwhyXLciFt; Expires=Tue, 27 Feb 2024 13:54:13 GMT; Path=/; SameSite=None; Secure
Set-Cookie: session=ymwLa8dKmWjLl43BBpQnrp5LkFZraYkp; Secure; HttpOnly; SameSite=None
X-Backend: ea6c1d8c-a8e5-4bef-b8db-879bbb13cf62
X-Frame-Options: SAMEORIGIN
---
timestamp: 2024-02-20T19:24:13+06:30
url: http://127.0.0.1:8082/user/55/profile
request:
header:
Accept-Encoding: gzip
Connection: close
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
host: 127.0.0.1:8082
method: GET
path: /blog/post
scheme: https
raw: |+
GET /user/55/profile HTTP/1.1
Host: 127.0.0.1:8082
Accept-Encoding: gzip
Connection: close
Content-Type: application/json
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
response:
header:
Content-Type: application/json; charset=UTF-8
Date: Tue, 27 Feb 2024 18:46:44 GMT
raw: |+
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Date: Tue, 27 Feb 2024 18:46:44 GMT
Content-Length: 47
{"ID":75,"Name":"user","Age":30,"Role":"user"}
---
timestamp: 2024-02-20T23:25:13+06:30
url: http://127.0.0.1:8082/user
request:
header:
Accept-Encoding: gzip
Connection: close
Content-Type: application/json
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
host: 127.0.0.1:8082
method: POST
path: /user
scheme: http
raw: |+
POST /user HTTP/1.1
Host: localhost:8082
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
Accept: */*
Content-Length: 32
Connection: close
Content-Type: application/json
{"id": 7 , "name": "pdteam"}
response:
header:
Content-Type: text/plain; charset=UTF-8
Date: Tue, 27 Feb 2024 18:46:44 GMT
raw: |+
HTTP/1.1 200 OK
Content-Type: text/plain; charset=UTF-8
Date: Wed, 28 Feb 2024 13:58:52 GMT
Content-Length: 25
User updated successfully
---
timestamp: 2024-02-20T23:26:13+06:30
url: http://127.0.0.1:8082/user
request:
header:
Accept-Encoding: gzip
Connection: close
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
host: 127.0.0.1:8082
method: POST
path: /user
scheme: http
raw: |+
POST /user HTTP/1.1
Host: localhost:8082
User-Agent: curl/8.1.2
Accept: */*
Content-Length: 20
Connection: close
Content-Type: application/x-www-form-urlencoded
id=7&name=pdteam
response:
header:
Content-Type: text/plain; charset=UTF-8
Date: Tue, 27 Feb 2024 18:46:44 GMT
raw: |+
HTTP/1.1 200 OK
Content-Type: text/plain; charset=UTF-8
Date: Wed, 28 Feb 2024 13:58:52 GMT
Content-Length: 25
User updated successfully
---
timestamp: 2024-02-20T23:26:13+06:30
url: http://127.0.0.1:8082/user
request:
header:
Accept-Encoding: gzip
Connection: close
Content-Type: multipart/form-data
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
host: 127.0.0.1:8082
method: POST
path: /user
scheme: http
raw: |+
POST /user HTTP/1.1
Host: localhost:8082
User-Agent: curl/8.1.2
Accept: */*
Content-Length: 226
Connection: close
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW
------WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="id"
7
------WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="name"
pdteam
------WebKitFormBoundary7MA4YWxkTrZu0gW--
response:
header:
Content-Type: text/plain; charset=UTF-8
Date: Tue, 27 Feb 2024 18:46:44 GMT
raw: |+
HTTP/1.1 200 OK
Content-Type: text/plain; charset=UTF-8
Date: Wed, 28 Feb 2024 13:58:52 GMT
Content-Length: 25
User updated successfully
---
---
timestamp: 2024-02-20T19:25:13+06:30
url: http://127.0.0.1:8082/blog/posts
request:
header:
Accept-Encoding: gzip
Connection: close
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
host: 127.0.0.1:8082
Cookie: session=ymwLa8dKmWjLl43BBpQnrp5LkFZraYkp; lang=en
method: GET
path: /blog/posts
scheme: http
raw: |+
GET /blog/posts HTTP/1.1
Host: 127.0.0.1:8082
Accept-Encoding: gzip
Cookie: session=ymwLa8dKmWjLl43BBpQnrp5LkFZraYkp; lang=en
Connection: close
Content-Type: application/json
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
response:
header:
Content-Type: application/json; charset=UTF-8
Date: Tue, 27 Feb 2024 18:46:44 GMT
raw: |+
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Date: Wed, 28 Feb 2024 13:58:52 GMT
Content-Length: 218
[{"ID":1,"Title":"The Joy of Programming","Content":"Programming is like painting a canvas with logic.","Lang":"en"},{"ID":2,"Title":"A Journey Through Code","Content":"Every line of code tells a story.","Lang":"en"}]
---
timestamp: 2024-02-20T23:26:13+06:30
url: http://127.0.0.1:8082/user
request:
header:
Accept-Encoding: gzip
Connection: close
Content-Type: application/xml
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
host: 127.0.0.1:8082
method: POST
path: /user
scheme: http
raw: |+
POST /user HTTP/1.1
Host: localhost:8082
User-Agent: curl/8.1.2
Accept: */*
Content-Length: 76
Connection: close
Content-Type: application/xml
<?xml version="1.0"?>
<user>
<id>7</id>
<name>pdteam</name>
</user>
response:
header:
Content-Type: text/plain; charset=UTF-8
Date: Tue, 27 Feb 2024 18:46:44 GMT
raw: |+
HTTP/1.1 200 OK
Content-Type: text/plain; charset=UTF-8
Date: Wed, 28 Feb 2024 13:58:52 GMT
Content-Length: 25
User updated successfully
---
timestamp: 2024-02-20T19:24:13+05:32
url: http://127.0.0.1:8082/host-header-lab
request:
header:
Accept-Encoding: gzip
Authorization: Bearer 3x4mpl3t0k3n
Connection: close
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
host: 127.0.0.1:8082
method: POST
path: /catalog/product
scheme: https
raw: |+
GET /host-header-lab HTTP/1.1
Host: 127.0.0.1:8082
Authorization: Bearer 3x4mpl3t0k3n
Accept-Encoding: gzip
Connection: close
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
response:
header:
Content-Encoding: gzip
Content-Type: text/html; charset=utf-8
Date: Tue, 20 Feb 2024 13:54:13 GMT
Set-Cookie: AWSALB=Rzm8ZSSL/yQq1mG1SdGmWAahqexWvOcjIhlNKm0wBnk4jvY3Hdy3mRc+NKoMRNJ2RW+FNbtRk7DX+itnfzjMvKNpwtLWWAafxeKybc+v351g0MsLycRNQF5fG78y; Expires=Tue, 27 Feb 2024 13:54:13 GMT; Path=/, AWSALBCORS=Rzm8ZSSL/yQq1mG1SdGmWAahqexWvOcjIhlNKm0wBnk4jvY3Hdy3mRc+NKoMRNJ2RW+FNbtRk7DX+itnfzjMvKNpwtLWWAafxeKybc+v351g0MsLycRNQF5fG78y; Expires=Tue, 27 Feb 2024 13:54:13 GMT; Path=/; SameSite=None; Secure, session=fFcCUjmQguQy820Y8xrnRypp3KBWSPk6; Secure; HttpOnly; SameSite=None
X-Backend: 2235790d-f089-4324-8ac0-f64cc96f2460
X-Frame-Options: SAMEORIGIN
body: |
<!DOCTYPE html>
<html>
<head>
<link href=/resources/labheader/css/scanMeHeader.css rel=stylesheet>
<link href=/resources/css/labsScanme.css rel=stylesheet>
<meta name="viewport" content="width=device-width, user-scalable=no">
<script src="/resources/js/react.development.js"></script>
<script src="/resources/js/react-dom.development.js"></script>
<script type="text/javascript" src="/resources/js/angular_1-7-7.js"></script>
<title>Fruit Overlays - Product - Gin &amp; Juice Shop</title>
</head>
<body ng-app>
<div id="scanMeHeader">
<section class="header-description">
<p>
This is a deliberately vulnerable web application designed for testing web&nbsp;vulnerability&nbsp;scanners.
<span class="link" onmouseenter="window.__x1 = 1" onmouseover="window.__x2 = 1" onmousemove="window.__x3 = 1" onmousedown="window.__x4 = 1" onmouseup="if (window.__x1 && window.__x2 && window.__x3 && window.__x4) location = atob('L3Z1bG5lcmFiaWxpdGllcw==')" onmouseleave="delete window.__x1; delete window.__x2; delete window.__x3; delete window.__x4">Put your scanner to the test!</span>
</p>
</section>
<section class='scanMeBanner'>
<div class=container>
<a href='/'>
<div class=scanme-logo></div>
</a>
<div class=title-container>
<nav>
<ul class="navigation-header-links primary-links">
<li>
<a class="button selected" href="/catalog">Products</a>
</li>
<li>
<a class="button" href="/blog">Blog</a>
</li>
<li>
<a class="button" href="/about">Our story</a>
</li>
</ul>
<ul class="navigation-header-links secondary-links">
<li>
<a class="account-icon" href="/my-account"><svg><use href="/resources/images/icon-account.svg#account-icon"></use></svg></a>
<ul>
<li>
<a class="button" href="/my-account">Log in</a>
</li>
<li>
<a class="button" href="/my-account">My account</a>
</li>
</ul>
</li>
<li>
<a class="cart-icon" href="/catalog/cart"><span>0</span><svg><use href="/resources/images/icon-cart.svg#cart-icon"></use></svg></a>
</li>
<li class="nav-toggle"><a class="nav-trigger"><span></span><span></span><span></span></a></li>
</ul>
</nav>
</div>
</div>
</section>
</div>
<div theme="ecommerce-product">
<section class="maincontainer">
<div class="container is-page">
<header class="notification-header">
</header>
<ul class="breadcrumbs">
<li><a href="/">Home</a></li>
<li><a href="/catalog">Products</a></li>
<li>Fruit Overlays</li>
</ul>
<section class="product">
<div>
<img class="product-image" src="/image/scanme/productcatalog/products/10.png">
</div>
<div>
<h3>Fruit Overlays</h3>
<img class="product-image" src="/image/scanme/productcatalog/products/10.png">
<span class="price-rating">
<span class="price">
$92.79
</span>
<img src="/resources/images/rating3.png">
</span>
<span class="description">
<label>Description:</label>
<p>When it comes to hospitality presentation is key, and nothing looks better than a well-dressed drink. We know gin fans like plenty of fruit in their glasses, some a bit more than they really need, but hey ho, each to their own. But what about fruit not inside your glass, but classily arranged over your glass? In comes Fruitus overlayus, the best way to jazz up any party. The possible colour combinations are endless, just picture that! All you need is a nice selection of small fruits, or maybe even use our Fruit Curliwurlier to add a dash of even more drama, and we will do the rest. This one is a real winner at our Christmas and New year’s outings, give it a go and turn some heads.</p>
<p>CONTENTS: 12 cocktail sticks.</p>
<p>HOW TO USE: Let your creative juices flow (Pun intended), and spend some time working on your colour coordination, try not to think too much about it, just do it! Pick up one of the Fruitus overlayus sticks and carefully slide the fruit along until there is a small space on either end of the stick. Balance the stick across the rim of the glass. Ta-Da! Your first fruit overlay. Keep going until you have as many overlays as you need. You can always purchase more at any time with a discount on bulk buys.</p>
</span>
<span class="stock-check">
<form id="stockCheckForm" action="/catalog/product/stock" method="POST">
<input required type="hidden" name="productId" value="3">
<select name="storeId">
<option value="1" >London</option>
<option value="2" >Paris</option>
<option value="3" >Milan</option>
</select>
<button type="submit" class="button">Check stock</button>
</form>
<span id="stockCheckResult"></span>
<script src="/resources/js/xmlStockCheckPayload.js"></script>
<script src="/resources/js/stockCheck.js"></script>
</span>
<span class="cart-button">
<form id=addToCartForm action=/catalog/cart method=POST>
<input required type=hidden name=productId value=3>
<input required type=hidden name=redir value=PRODUCT>
<select class='product-quantity' required name=quantity>
<option value="1" selected>1</option>
<option value="2">2</option>
<option value="3">3</option>
<option value="4">4</option>
<option value="5">5</option>
<option value="6">6</option>
<option value="7">7</option>
<option value="8">8</option>
<option value="9">9</option>
<option value="10">10</option>
<option value="11">11</option>
<option value="12">12</option>
<option value="13">13</option>
<option value="14">14</option>
<option value="15">15</option>
</select>
<button type=submit class=button>Add to cart</button>
</form>
</span>
<span class="view-cart-button">
<a class="button" href="/catalog/cart">View cart</a>
</span>
</div>
</section>
</div>
</section>
<div class="footer-wrapper">
<section class="footer">
<div class="footer-left"></div>
<div class="footer-center">
<h2>Never miss a deal - subscribe now</h2>
<p>Join our worldwide community of gin and juice fanatics, for exclusive news on our latest deals, new releases, collaborations, and more.</p>
<script src='/resources/js/subscribeNow.js'></script>
<div id="subscribe" class="form" data-method="post" data-action="/catalog/subscribe">
<input required type=email name=email placeholder="Email address">
<input required type="hidden" name="csrf" value="ALUrIPu21ygHSGadxsA8u70XnVcY4V4k">
<button class="button" type=submit>Subscribe</button>
</div>
<dialog id="coupon-dialog">
<div class="coupon-wrapper">
<button class="close-button" onclick="closeCouponDialog(event)"></button>
<div class="coupon-info">
<h1>20% off everything</h1>
<div class="coupon-input">
<h3 id="copyable-coupon">Coupon not found</h3>
<button id="copy-coupon-button" class="copy-button" onclick="copyCoupon(event)"></button>
<div id="coupon-copied-tick" class="coupon-copied-tick hidden"></div>
</div>
<p>Apply this coupon to your Shopping Cart before placing your order.</p>
</div>
</div>
</dialog>
<div class="footer-copyright">
<div class="portswigger-logo"></div>
<div>© 2023 PortSwigger Ltd.</div>
</div>
</div>
<div class="footer-right"></div>
</section>
<section class="footer-lower">
<div class="footerNavigation">
<div class="socialLinks">
</div>
<nav>
<ul class="navigation-header-links primary-links">
<li>
<a class="button selected" href="/catalog">Products</a>
</li>
<li>
<a class="button" href="/blog">Blog</a>
</li>
<li>
<a class="button" href="/about">Our story</a>
</li>
</ul>
</nav>
</div>
</section>
</div>
</div>
<script src='/resources/footer/js/scanme.js'></script>
</body>
</html>
raw: |+
HTTP/1.1 200 OK
Connection: close
Content-Encoding: gzip
Content-Type: text/html; charset=utf-8
Date: Tue, 20 Feb 2024 13:54:13 GMT
Set-Cookie: AWSALB=Rzm8ZSSL/yQq1mG1SdGmWAahqexWvOcjIhlNKm0wBnk4jvY3Hdy3mRc+NKoMRNJ2RW+FNbtRk7DX+itnfzjMvKNpwtLWWAafxeKybc+v351g0MsLycRNQF5fG78y; Expires=Tue, 27 Feb 2024 13:54:13 GMT; Path=/
Set-Cookie: AWSALBCORS=Rzm8ZSSL/yQq1mG1SdGmWAahqexWvOcjIhlNKm0wBnk4jvY3Hdy3mRc+NKoMRNJ2RW+FNbtRk7DX+itnfzjMvKNpwtLWWAafxeKybc+v351g0MsLycRNQF5fG78y; Expires=Tue, 27 Feb 2024 13:54:13 GMT; Path=/; SameSite=None; Secure
Set-Cookie: session=fFcCUjmQguQy820Y8xrnRypp3KBWSPk6; Secure; HttpOnly; SameSite=None
X-Backend: 2235790d-f089-4324-8ac0-f64cc96f2460
X-Frame-Options: SAMEORIGIN

2
integration_tests/run.sh
View File

@ -1,7 +1,7 @@
#!/bin/bash
echo "::group::Build nuclei"
rm integration-test nuclei 2>/dev/null
rm integration-test fuzzplayground nuclei 2>/dev/null
cd ../cmd/nuclei
go build -race .
mv nuclei ../../integration_tests/nuclei

8
internal/runner/inputs.go
View File

@ -8,6 +8,7 @@ import (
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/hmap/store/hybrid"
"github.com/projectdiscovery/httpx/common/httpx"
"github.com/projectdiscovery/nuclei/v3/pkg/input/provider"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
"github.com/projectdiscovery/nuclei/v3/pkg/utils"
stringsutil "github.com/projectdiscovery/utils/strings"
@ -19,10 +20,15 @@ const probeBulkSize = 50
// initializeTemplatesHTTPInput initializes the http form of input
// for any loaded http templates if input is in non-standard format.
func (r *Runner) initializeTemplatesHTTPInput() (*hybrid.HybridMap, error) {
hm, err := hybrid.New(hybrid.DefaultDiskOptions)
if err != nil {
return nil, errors.Wrap(err, "could not create temporary input file")
}
if r.inputProvider.InputType() == provider.MultiFormatInputProvider {
// currently http probing for input mode types is not supported
return hm, nil
}
gologger.Info().Msgf("Running httpx on input host")
var bulkSize = probeBulkSize
@ -41,7 +47,7 @@ func (r *Runner) initializeTemplatesHTTPInput() (*hybrid.HybridMap, error) {
// Probe the non-standard URLs and store them in cache
swg := sizedwaitgroup.New(bulkSize)
count := int32(0)
r.hmapInputProvider.Scan(func(value *contextargs.MetaInput) bool {
r.inputProvider.Iterate(func(value *contextargs.MetaInput) bool {
if stringsutil.HasPrefixAny(value.Input, "http://", "https://") {
return true
}

123
internal/runner/lazy.go Normal file
View File

@ -0,0 +1,123 @@
package runner
import (
"fmt"
"github.com/projectdiscovery/nuclei/v3/pkg/authprovider/authx"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/loader"
"github.com/projectdiscovery/nuclei/v3/pkg/output"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/helpers/writer"
"github.com/projectdiscovery/nuclei/v3/pkg/scan"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
errorutil "github.com/projectdiscovery/utils/errors"
)
type AuthLazyFetchOptions struct {
TemplateStore *loader.Store
ExecOpts protocols.ExecutorOptions
OnError func(error)
}
// GetAuthTemlStore create new loader for loading auth templates
func GetAuthTmplStore(opts types.Options, catalog catalog.Catalog, execOpts protocols.ExecutorOptions) (*loader.Store, error) {
tmpls := []string{}
for _, file := range opts.SecretsFile {
data, err := authx.GetTemplatePathsFromSecretFile(file)
if err != nil {
return nil, errorutil.NewWithErr(err).Msgf("failed to get template paths from secrets file")
}
tmpls = append(tmpls, data...)
}
opts.Templates = tmpls
opts.Workflows = nil
opts.RemoteTemplateDomainList = nil
opts.TemplateURLs = nil
opts.WorkflowURLs = nil
opts.ExcludedTemplates = nil
opts.Tags = nil
opts.ExcludeTags = nil
opts.IncludeTemplates = nil
opts.Authors = nil
opts.Severities = nil
opts.ExcludeSeverities = nil
opts.IncludeTags = nil
opts.IncludeIds = nil
opts.ExcludeIds = nil
opts.Protocols = nil
opts.ExcludeProtocols = nil
opts.IncludeConditions = nil
cfg := loader.NewConfig(&opts, catalog, execOpts)
store, err := loader.New(cfg)
if err != nil {
return nil, errorutil.NewWithErr(err).Msgf("failed to initialize dynamic auth templates store")
}
return store, nil
}
// GetLazyAuthFetchCallback returns a lazy fetch callback for auth secrets
func GetLazyAuthFetchCallback(opts *AuthLazyFetchOptions) authx.LazyFetchSecret {
return func(d *authx.Dynamic) error {
tmpls := opts.TemplateStore.LoadTemplates([]string{d.TemplatePath})
if len(tmpls) == 0 {
return fmt.Errorf("no templates found for path: %s", d.TemplatePath)
}
if len(tmpls) > 1 {
return fmt.Errorf("multiple templates found for path: %s", d.TemplatePath)
}
data := map[string]interface{}{}
tmpl := tmpls[0]
// add args to tmpl here
vars := map[string]interface{}{}
ctx := scan.NewScanContext(contextargs.NewWithInput(d.Input))
for _, v := range d.Variables {
vars[v.Key] = v.Value
ctx.Input.Add(v.Key, v.Value)
}
var finalErr error
ctx.OnResult = func(e *output.InternalWrappedEvent) {
if e == nil {
finalErr = fmt.Errorf("no result found for template: %s", d.TemplatePath)
return
}
if !e.HasOperatorResult() {
finalErr = fmt.Errorf("no result found for template: %s", d.TemplatePath)
return
}
// dynamic values
for k, v := range e.OperatorsResult.DynamicValues {
if len(v) > 0 {
data[k] = v[0]
}
}
// named extractors
for k, v := range e.OperatorsResult.Extracts {
if len(v) > 0 {
data[k] = v[0]
}
}
if len(data) == 0 {
if e.OperatorsResult.Matched {
finalErr = fmt.Errorf("match found but no (dynamic/extracted) values found for template: %s", d.TemplatePath)
} else {
finalErr = fmt.Errorf("no match or (dynamic/extracted) values found for template: %s", d.TemplatePath)
}
}
// log result of template in result file/screen
_ = writer.WriteResult(e, opts.ExecOpts.Output, opts.ExecOpts.Progress, opts.ExecOpts.IssuesClient)
}
_, err := tmpl.Executer.ExecuteWithResults(ctx)
if err != nil {
finalErr = err
}
// store extracted result in auth context
d.Extracted = data
if finalErr != nil && opts.OnError != nil {
opts.OnError(finalErr)
}
return finalErr
}
}

9
internal/runner/options.go
View File

@ -289,6 +289,12 @@ func createReportingOptions(options *types.Options) (*reporting.Options, error)
// configureOutput configures the output logging levels to be displayed on the screen
func configureOutput(options *types.Options) {
// disable standard logger (ref: https://github.com/golang/go/issues/19895)
defer logutil.DisableDefaultLogger()
if options.NoColor {
gologger.DefaultLogger.SetFormatter(formatter.NewCLI(true))
}
// If the user desires verbose output, show verbose output
if options.Debug || options.DebugRequests || options.DebugResponse {
gologger.DefaultLogger.SetMaxLevel(levels.LevelDebug)
@ -304,9 +310,6 @@ func configureOutput(options *types.Options) {
if options.Silent {
gologger.DefaultLogger.SetMaxLevel(levels.LevelSilent)
}
// disable standard logger (ref: https://github.com/golang/go/issues/19895)
logutil.DisableDefaultLogger()
}
// loadResolvers loads resolvers from both user-provided flags and file

102
internal/runner/runner.go
View File

@ -14,6 +14,8 @@ import (
"time"
"github.com/projectdiscovery/nuclei/v3/internal/pdcp"
"github.com/projectdiscovery/nuclei/v3/pkg/authprovider"
"github.com/projectdiscovery/nuclei/v3/pkg/input/provider"
"github.com/projectdiscovery/nuclei/v3/pkg/installer"
uncoverlib "github.com/projectdiscovery/uncover"
pdcpauth "github.com/projectdiscovery/utils/auth/pdcp"
@ -33,7 +35,6 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/disk"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/loader"
"github.com/projectdiscovery/nuclei/v3/pkg/core"
"github.com/projectdiscovery/nuclei/v3/pkg/core/inputs/hybrid"
"github.com/projectdiscovery/nuclei/v3/pkg/external/customtemplates"
"github.com/projectdiscovery/nuclei/v3/pkg/input"
"github.com/projectdiscovery/nuclei/v3/pkg/output"
@ -69,22 +70,21 @@ var (
// Runner is a client for running the enumeration process.
type Runner struct {
output output.Writer
interactsh *interactsh.Client
options *types.Options
projectFile *projectfile.ProjectFile
catalog catalog.Catalog
progress progress.Progress
colorizer aurora.Aurora
issuesClient reporting.Client
hmapInputProvider *hybrid.Input
browser *engine.Browser
rateLimiter *ratelimit.Limiter
hostErrors hosterrorscache.CacheInterface
resumeCfg *types.ResumeCfg
pprofServer *http.Server
// pdcp auto-save options
output output.Writer
interactsh *interactsh.Client
options *types.Options
projectFile *projectfile.ProjectFile
catalog catalog.Catalog
progress progress.Progress
colorizer aurora.Aurora
issuesClient reporting.Client
browser *engine.Browser
rateLimiter *ratelimit.Limiter
hostErrors hosterrorscache.CacheInterface
resumeCfg *types.ResumeCfg
pprofServer *http.Server
pdcpUploadErrMsg string
inputProvider provider.InputProvider
//general purpose temporary directory
tmpDir string
}
@ -219,14 +219,12 @@ func New(options *types.Options) (*Runner, error) {
os.Exit(0)
}
// Initialize the input source
hmapInput, err := hybrid.New(&hybrid.Options{
Options: options,
})
// create the input provider and load the inputs
inputProvider, err := provider.NewInputProvider(provider.InputOptions{Options: options})
if err != nil {
return nil, errors.Wrap(err, "could not create input provider")
}
runner.hmapInputProvider = hmapInput
runner.inputProvider = inputProvider
// Create the output file if asked
outputWriter, err := output.NewStandardWriter(options)
@ -344,7 +342,9 @@ func (r *Runner) Close() {
if r.projectFile != nil {
r.projectFile.Close()
}
r.hmapInputProvider.Close()
if r.inputProvider != nil {
r.inputProvider.Close()
}
protocolinit.Close()
if r.pprofServer != nil {
_ = r.pprofServer.Shutdown(context.Background())
@ -433,6 +433,24 @@ func (r *Runner) RunEnumeration() error {
TemporaryDirectory: r.tmpDir,
}
if len(r.options.SecretsFile) > 0 && !r.options.Validate {
authTmplStore, err := GetAuthTmplStore(*r.options, r.catalog, executorOpts)
if err != nil {
return errors.Wrap(err, "failed to load dynamic auth templates")
}
authOpts := &authprovider.AuthProviderOptions{SecretsFiles: r.options.SecretsFile}
authOpts.LazyFetchSecret = GetLazyAuthFetchCallback(&AuthLazyFetchOptions{
TemplateStore: authTmplStore,
ExecOpts: executorOpts,
})
// initialize auth provider
provider, err := authprovider.NewAuthProvider(authOpts)
if err != nil {
return errors.Wrap(err, "could not create auth provider")
}
executorOpts.AuthProvider = provider
}
if r.options.ShouldUseHostError() {
cache := hosterrorscache.New(r.options.MaxHostError, hosterrorscache.DefaultMaxHostsCount, r.options.TrackError)
cache.SetVerbose(r.options.Verbose)
@ -449,9 +467,16 @@ func (r *Runner) RunEnumeration() error {
}
executorOpts.WorkflowLoader = workflowLoader
store, err := loader.New(loader.NewConfig(r.options, r.catalog, executorOpts))
// If using input-file flags, only load http fuzzing based templates.
loaderConfig := loader.NewConfig(r.options, r.catalog, executorOpts)
if !strings.EqualFold(r.options.InputFileMode, "list") || r.options.FuzzTemplates {
// if input type is not list (implicitly enable fuzzing)
r.options.FuzzTemplates = true
loaderConfig.OnlyLoadHTTPFuzzing = true
}
store, err := loader.New(loaderConfig)
if err != nil {
return errors.Wrap(err, "could not load templates from config")
return errors.Wrap(err, "Could not create loader.")
}
if r.options.Validate {
@ -484,7 +509,7 @@ func (r *Runner) RunEnumeration() error {
}
ret := uncover.GetUncoverTargetsFromMetadata(context.TODO(), store.Templates(), r.options.UncoverField, uncoverOpts)
for host := range ret {
r.hmapInputProvider.SetWithExclusions(host)
_ = r.inputProvider.SetWithExclusions(host)
}
}
// list all templates
@ -496,6 +521,14 @@ func (r *Runner) RunEnumeration() error {
// display execution info like version , templates used etc
r.displayExecutionInfo(store)
// prefetch secrets if enabled
if executorOpts.AuthProvider != nil && r.options.PreFetchSecrets {
gologger.Info().Msgf("Pre-fetching secrets from authprovider[s]")
if err := executorOpts.AuthProvider.PreFetchSecrets(); err != nil {
return errors.Wrap(err, "could not pre-fetch secrets")
}
}
// If not explicitly disabled, check if http based protocols
// are used, and if inputs are non-http to pre-perform probing
// of urls and storing them for execution.
@ -541,7 +574,7 @@ func (r *Runner) RunEnumeration() error {
func (r *Runner) isInputNonHTTP() bool {
var nonURLInput bool
r.hmapInputProvider.Scan(func(value *contextargs.MetaInput) bool {
r.inputProvider.Iterate(func(value *contextargs.MetaInput) bool {
if !strings.Contains(value.Input, "://") {
nonURLInput = true
return false
@ -552,13 +585,13 @@ func (r *Runner) isInputNonHTTP() bool {
}
func (r *Runner) executeSmartWorkflowInput(executorOpts protocols.ExecutorOptions, store *loader.Store, engine *core.Engine) (*atomic.Bool, error) {
r.progress.Init(r.hmapInputProvider.Count(), 0, 0)
r.progress.Init(r.inputProvider.Count(), 0, 0)
service, err := automaticscan.New(automaticscan.Options{
ExecuterOpts: executorOpts,
Store: store,
Engine: engine,
Target: r.hmapInputProvider,
Target: r.inputProvider,
})
if err != nil {
return nil, errors.Wrap(err, "could not create automatic scan service")
@ -589,7 +622,12 @@ func (r *Runner) executeTemplatesInput(store *loader.Store, engine *core.Engine)
return nil, errors.New("no templates provided for scan")
}
results := engine.ExecuteScanWithOpts(finalTemplates, r.hmapInputProvider, r.options.DisableClustering)
// pass input provider to engine
// TODO: this should be not necessary after r.hmapInputProvider is removed + refactored
if r.inputProvider == nil {
return nil, errors.New("no input provider found")
}
results := engine.ExecuteScanWithOpts(finalTemplates, r.inputProvider, r.options.DisableClustering)
return results, nil
}
@ -603,6 +641,7 @@ func (r *Runner) displayExecutionInfo(store *loader.Store) {
// only print these stats in verbose mode
stats.DisplayAsWarning(parsers.HeadlessFlagWarningStats)
stats.DisplayAsWarning(parsers.CodeFlagWarningStats)
stats.DisplayAsWarning(parsers.FuzzFlagWarningStats)
stats.DisplayAsWarning(parsers.TemplatesExecutedStats)
}
@ -643,8 +682,9 @@ func (r *Runner) displayExecutionInfo(store *loader.Store) {
}
}
}
if r.hmapInputProvider.Count() > 0 {
gologger.Info().Msgf("Targets loaded for current scan: %d", r.hmapInputProvider.Count())
if r.inputProvider.Count() > 0 {
gologger.Info().Msgf("Targets loaded for current scan: %d", r.inputProvider.Count())
}
}

29
lib/config.go
View File

@ -4,9 +4,11 @@ import (
"context"
"time"
"github.com/projectdiscovery/goflags"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/ratelimit"
"github.com/projectdiscovery/nuclei/v3/pkg/authprovider"
"github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
"github.com/projectdiscovery/nuclei/v3/pkg/output"
"github.com/projectdiscovery/nuclei/v3/pkg/progress"
@ -163,7 +165,7 @@ func EnableHeadlessWithOpts(hopts *HeadlessOpts) NucleiSDKOptions {
if err != nil {
return err
}
e.executerOpts.Browser = browser
e.browserInstance = browser
return nil
}
}
@ -356,3 +358,28 @@ func EnablePassiveMode() NucleiSDKOptions {
return nil
}
}
// WithAuthOptions allows setting a custom authprovider implementation
func WithAuthProvider(provider authprovider.AuthProvider) NucleiSDKOptions {
return func(e *NucleiEngine) error {
e.authprovider = provider
return nil
}
}
// LoadSecretsFromFile allows loading secrets from file
func LoadSecretsFromFile(files []string, prefetch bool) NucleiSDKOptions {
return func(e *NucleiEngine) error {
e.opts.SecretsFile = goflags.StringSlice(files)
e.opts.PreFetchSecrets = prefetch
return nil
}
}
// EnableFuzzTemplates allows enabling template fuzzing
func EnableFuzzTemplates() NucleiSDKOptions {
return func(e *NucleiEngine) error {
e.opts.FuzzTemplates = true
return nil
}
}

12
lib/multi.go
View File

@ -8,11 +8,10 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/loader"
"github.com/projectdiscovery/nuclei/v3/pkg/core"
"github.com/projectdiscovery/nuclei/v3/pkg/core/inputs"
"github.com/projectdiscovery/nuclei/v3/pkg/input/provider"
"github.com/projectdiscovery/nuclei/v3/pkg/output"
"github.com/projectdiscovery/nuclei/v3/pkg/parsers"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/projectdiscovery/ratelimit"
errorutil "github.com/projectdiscovery/utils/errors"
@ -123,14 +122,7 @@ func (e *ThreadSafeNucleiEngine) ExecuteNucleiWithOpts(targets []string, opts ..
}
store.Load()
inputProvider := &inputs.SimpleInputProvider{
Inputs: []*contextargs.MetaInput{},
}
// load targets
for _, target := range targets {
inputProvider.Set(target)
}
inputProvider := provider.NewSimpleInputProviderWithUrls(targets...)
if len(store.Templates()) == 0 && len(store.Workflows()) == 0 {
return ErrNoTemplatesAvailable

30
lib/sdk.go
View File

@ -5,11 +5,12 @@ import (
"bytes"
"io"
"github.com/projectdiscovery/httpx/common/httpx"
"github.com/projectdiscovery/nuclei/v3/pkg/authprovider"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/disk"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/loader"
"github.com/projectdiscovery/nuclei/v3/pkg/core"
"github.com/projectdiscovery/nuclei/v3/pkg/core/inputs"
"github.com/projectdiscovery/nuclei/v3/pkg/input/provider"
providerTypes "github.com/projectdiscovery/nuclei/v3/pkg/input/types"
"github.com/projectdiscovery/nuclei/v3/pkg/output"
"github.com/projectdiscovery/nuclei/v3/pkg/parsers"
"github.com/projectdiscovery/nuclei/v3/pkg/progress"
@ -65,12 +66,13 @@ type NucleiEngine struct {
catalog *disk.DiskCatalog
rateLimiter *ratelimit.Limiter
store *loader.Store
httpxClient *httpx.HTTPX
inputProvider *inputs.SimpleInputProvider
httpxClient providerTypes.InputLivenessProbe
inputProvider provider.InputProvider
engine *core.Engine
mode engineMode
browserInstance *engine.Browser
httpClient *retryablehttp.Client
authprovider authprovider.AuthProvider
// unexported meta options
opts *types.Options
@ -110,7 +112,7 @@ func (e *NucleiEngine) GetTemplates() []*templates.Template {
func (e *NucleiEngine) LoadTargets(targets []string, probeNonHttp bool) {
for _, target := range targets {
if probeNonHttp {
e.inputProvider.SetWithProbe(target, e.httpxClient)
_ = e.inputProvider.SetWithProbe(target, e.httpxClient)
} else {
e.inputProvider.Set(target)
}
@ -122,13 +124,29 @@ func (e *NucleiEngine) LoadTargetsFromReader(reader io.Reader, probeNonHttp bool
buff := bufio.NewScanner(reader)
for buff.Scan() {
if probeNonHttp {
e.inputProvider.SetWithProbe(buff.Text(), e.httpxClient)
_ = e.inputProvider.SetWithProbe(buff.Text(), e.httpxClient)
} else {
e.inputProvider.Set(buff.Text())
}
}
}
// LoadTargetsWithHttpData loads targets that contain http data from file it currently supports
// multiple formats like burp xml,openapi,swagger,proxify json
// Note: this is mutually exclusive with LoadTargets and LoadTargetsFromReader
func (e *NucleiEngine) LoadTargetsWithHttpData(filePath string, filemode string) error {
e.opts.TargetsFilePath = filePath
e.opts.InputFileMode = filemode
httpProvider, err := provider.NewInputProvider(provider.InputOptions{Options: e.opts})
if err != nil {
e.opts.TargetsFilePath = ""
e.opts.InputFileMode = ""
return err
}
e.inputProvider = httpProvider
return nil
}
// GetExecuterOptions returns the nuclei executor options
func (e *NucleiEngine) GetExecuterOptions() *protocols.ExecutorOptions {
return &e.executerOpts

41
lib/sdk_private.go
View File

@ -8,19 +8,20 @@ import (
"time"
"github.com/logrusorgru/aurora"
"github.com/pkg/errors"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/gologger/levels"
"github.com/projectdiscovery/httpx/common/httpx"
"github.com/projectdiscovery/nuclei/v3/internal/runner"
"github.com/projectdiscovery/nuclei/v3/pkg/authprovider"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/disk"
"github.com/projectdiscovery/nuclei/v3/pkg/core"
"github.com/projectdiscovery/nuclei/v3/pkg/core/inputs"
"github.com/projectdiscovery/nuclei/v3/pkg/input/provider"
"github.com/projectdiscovery/nuclei/v3/pkg/installer"
"github.com/projectdiscovery/nuclei/v3/pkg/output"
"github.com/projectdiscovery/nuclei/v3/pkg/progress"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/hosterrorscache"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/interactsh"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolinit"
@ -29,6 +30,7 @@ import (
"github.com/projectdiscovery/nuclei/v3/pkg/reporting"
"github.com/projectdiscovery/nuclei/v3/pkg/testutils"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
nucleiUtils "github.com/projectdiscovery/nuclei/v3/pkg/utils"
"github.com/projectdiscovery/ratelimit"
)
@ -86,9 +88,7 @@ func (e *NucleiEngine) applyRequiredDefaults() {
// and idea is to disable them to avoid false positives
e.opts.ExcludeTags = append(e.opts.ExcludeTags, config.ReadIgnoreFile().Tags...)
e.inputProvider = &inputs.SimpleInputProvider{
Inputs: []*contextargs.MetaInput{},
}
e.inputProvider = provider.NewSimpleInputProvider()
}
// init
@ -158,6 +158,33 @@ func (e *NucleiEngine) init() error {
ResumeCfg: types.NewResumeCfg(),
Browser: e.browserInstance,
}
if len(e.opts.SecretsFile) > 0 {
authTmplStore, err := runner.GetAuthTmplStore(*e.opts, e.catalog, e.executerOpts)
if err != nil {
return errors.Wrap(err, "failed to load dynamic auth templates")
}
authOpts := &authprovider.AuthProviderOptions{SecretsFiles: e.opts.SecretsFile}
authOpts.LazyFetchSecret = runner.GetLazyAuthFetchCallback(&runner.AuthLazyFetchOptions{
TemplateStore: authTmplStore,
ExecOpts: e.executerOpts,
})
// initialize auth provider
provider, err := authprovider.NewAuthProvider(authOpts)
if err != nil {
return errors.Wrap(err, "could not create auth provider")
}
e.executerOpts.AuthProvider = provider
}
if e.authprovider != nil {
e.executerOpts.AuthProvider = e.authprovider
}
// prefetch secrets
if e.executerOpts.AuthProvider != nil && e.opts.PreFetchSecrets {
if err := e.executerOpts.AuthProvider.PreFetchSecrets(); err != nil {
return errors.Wrap(err, "could not prefetch secrets")
}
}
if e.opts.RateLimitMinute > 0 {
e.executerOpts.RateLimiter = ratelimit.New(context.Background(), uint(e.opts.RateLimitMinute), time.Minute)
@ -172,8 +199,10 @@ func (e *NucleiEngine) init() error {
httpxOptions := httpx.DefaultOptions
httpxOptions.Timeout = 5 * time.Second
if e.httpxClient, err = httpx.New(&httpxOptions); err != nil {
if client, err := httpx.New(&httpxOptions); err != nil {
return err
} else {
e.httpxClient = nucleiUtils.GetInputLivenessChecker(client)
}
// Only Happens once regardless how many times this function is called

31
pkg/authprovider/authx/basic_auth.go Normal file
View File

@ -0,0 +1,31 @@
package authx
import (
"net/http"
"github.com/projectdiscovery/retryablehttp-go"
)
var (
_ AuthStrategy = &BasicAuthStrategy{}
)
// BasicAuthStrategy is a strategy for basic auth
type BasicAuthStrategy struct {
Data *Secret
}
// NewBasicAuthStrategy creates a new basic auth strategy
func NewBasicAuthStrategy(data *Secret) *BasicAuthStrategy {
return &BasicAuthStrategy{Data: data}
}
// Apply applies the basic auth strategy to the request
func (s *BasicAuthStrategy) Apply(req *http.Request) {
req.SetBasicAuth(s.Data.Username, s.Data.Password)
}
// ApplyOnRR applies the basic auth strategy to the retryable request
func (s *BasicAuthStrategy) ApplyOnRR(req *retryablehttp.Request) {
req.SetBasicAuth(s.Data.Username, s.Data.Password)
}

31
pkg/authprovider/authx/bearer_auth.go Normal file
View File

@ -0,0 +1,31 @@
package authx
import (
"net/http"
"github.com/projectdiscovery/retryablehttp-go"
)
var (
_ AuthStrategy = &BearerTokenAuthStrategy{}
)
// BearerTokenAuthStrategy is a strategy for bearer token auth
type BearerTokenAuthStrategy struct {
Data *Secret
}
// NewBearerTokenAuthStrategy creates a new bearer token auth strategy
func NewBearerTokenAuthStrategy(data *Secret) *BearerTokenAuthStrategy {
return &BearerTokenAuthStrategy{Data: data}
}
// Apply applies the bearer token auth strategy to the request
func (s *BearerTokenAuthStrategy) Apply(req *http.Request) {
req.Header.Set("Authorization", "Bearer "+s.Data.Token)
}
// ApplyOnRR applies the bearer token auth strategy to the retryable request
func (s *BearerTokenAuthStrategy) ApplyOnRR(req *retryablehttp.Request) {
req.Header.Set("Authorization", "Bearer "+s.Data.Token)
}

43
pkg/authprovider/authx/cookies_auth.go Normal file
View File

@ -0,0 +1,43 @@
package authx
import (
"net/http"
"github.com/projectdiscovery/retryablehttp-go"
)
var (
_ AuthStrategy = &CookiesAuthStrategy{}
)
// CookiesAuthStrategy is a strategy for cookies auth
type CookiesAuthStrategy struct {
Data *Secret
}
// NewCookiesAuthStrategy creates a new cookies auth strategy
func NewCookiesAuthStrategy(data *Secret) *CookiesAuthStrategy {
return &CookiesAuthStrategy{Data: data}
}
// Apply applies the cookies auth strategy to the request
func (s *CookiesAuthStrategy) Apply(req *http.Request) {
for _, cookie := range s.Data.Cookies {
c := &http.Cookie{
Name: cookie.Key,
Value: cookie.Value,
}
req.AddCookie(c)
}
}
// ApplyOnRR applies the cookies auth strategy to the retryable request
func (s *CookiesAuthStrategy) ApplyOnRR(req *retryablehttp.Request) {
for _, cookie := range s.Data.Cookies {
c := &http.Cookie{
Name: cookie.Key,
Value: cookie.Value,
}
req.AddCookie(c)
}
}

167
pkg/authprovider/authx/dynamic.go Normal file
View File

@ -0,0 +1,167 @@
package authx
import (
"encoding/json"
"fmt"
"strings"
"sync"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/replacer"
errorutil "github.com/projectdiscovery/utils/errors"
)
type LazyFetchSecret func(d *Dynamic) error
var (
_ json.Unmarshaler = &Dynamic{}
)
// Dynamic is a struct for dynamic secret or credential
// these are high level secrets that take action to generate the actual secret
// ex: username and password are dynamic secrets, the actual secret is the token obtained
// after authenticating with the username and password
type Dynamic struct {
Secret `yaml:",inline"` // this is a static secret that will be generated after the dynamic secret is resolved
TemplatePath string `json:"template" yaml:"template"`
Variables []KV `json:"variables" yaml:"variables"`
Input string `json:"input" yaml:"input"` // (optional) target for the dynamic secret
Extracted map[string]interface{} `json:"-" yaml:"-"` // extracted values from the dynamic secret
fetchCallback LazyFetchSecret `json:"-" yaml:"-"`
m *sync.Mutex `json:"-" yaml:"-"` // mutex for lazy fetch
fetched bool `json:"-" yaml:"-"` // flag to check if the secret has been fetched
error error `json:"-" yaml:"-"` // error if any
}
func (d *Dynamic) UnmarshalJSON(data []byte) error {
if err := json.Unmarshal(data, &d); err != nil {
return err
}
var s Secret
if err := json.Unmarshal(data, &s); err != nil {
return err
}
d.Secret = s
return nil
}
// Validate validates the dynamic secret
func (d *Dynamic) Validate() error {
d.m = &sync.Mutex{}
if d.TemplatePath == "" {
return errorutil.New(" template-path is required for dynamic secret")
}
if len(d.Variables) == 0 {
return errorutil.New("variables are required for dynamic secret")
}
d.skipCookieParse = true // skip cookie parsing in dynamic secrets during validation
if err := d.Secret.Validate(); err != nil {
return err
}
return nil
}
// SetLazyFetchCallback sets the lazy fetch callback for the dynamic secret
func (d *Dynamic) SetLazyFetchCallback(callback LazyFetchSecret) {
d.fetchCallback = func(d *Dynamic) error {
err := callback(d)
d.fetched = true
if err != nil {
d.error = err
return err
}
if len(d.Extracted) == 0 {
return fmt.Errorf("no extracted values found for dynamic secret")
}
// evaluate headers
for i, header := range d.Headers {
if strings.Contains(header.Value, "{{") {
header.Value = replacer.Replace(header.Value, d.Extracted)
}
if strings.Contains(header.Key, "{{") {
header.Key = replacer.Replace(header.Key, d.Extracted)
}
d.Headers[i] = header
}
// evaluate cookies
for i, cookie := range d.Cookies {
if strings.Contains(cookie.Value, "{{") {
cookie.Value = replacer.Replace(cookie.Value, d.Extracted)
}
if strings.Contains(cookie.Key, "{{") {
cookie.Key = replacer.Replace(cookie.Key, d.Extracted)
}
if strings.Contains(cookie.Raw, "{{") {
cookie.Raw = replacer.Replace(cookie.Raw, d.Extracted)
}
d.Cookies[i] = cookie
}
// evaluate query params
for i, query := range d.Params {
if strings.Contains(query.Value, "{{") {
query.Value = replacer.Replace(query.Value, d.Extracted)
}
if strings.Contains(query.Key, "{{") {
query.Key = replacer.Replace(query.Key, d.Extracted)
}
d.Params[i] = query
}
// check username, password and token
if strings.Contains(d.Username, "{{") {
d.Username = replacer.Replace(d.Username, d.Extracted)
}
if strings.Contains(d.Password, "{{") {
d.Password = replacer.Replace(d.Password, d.Extracted)
}
if strings.Contains(d.Token, "{{") {
d.Token = replacer.Replace(d.Token, d.Extracted)
}
// now attempt to parse the cookies
d.skipCookieParse = false
for i, cookie := range d.Cookies {
if cookie.Raw != "" {
if err := cookie.Parse(); err != nil {
return fmt.Errorf("[%s] invalid raw cookie in cookiesAuth: %s", d.TemplatePath, err)
}
d.Cookies[i] = cookie
}
}
return nil
}
}
// GetStrategy returns the auth strategy for the dynamic secret
func (d *Dynamic) GetStrategy() AuthStrategy {
if !d.fetched {
_ = d.Fetch(true)
}
if d.error != nil {
return nil
}
return d.Secret.GetStrategy()
}
// Fetch fetches the dynamic secret
// if isFatal is true, it will stop the execution if the secret could not be fetched
func (d *Dynamic) Fetch(isFatal bool) error {
d.m.Lock()
defer d.m.Unlock()
if d.fetched {
return nil
}
d.error = d.fetchCallback(d)
if d.error != nil && isFatal {
gologger.Fatal().Msgf("Could not fetch dynamic secret: %s\n", d.error)
}
return d.error
}
// Error returns the error if any
func (d *Dynamic) Error() error {
return d.error
}

253
pkg/authprovider/authx/file.go Normal file
View File

@ -0,0 +1,253 @@
package authx
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"regexp"
"strings"
errorutil "github.com/projectdiscovery/utils/errors"
"github.com/projectdiscovery/utils/generic"
stringsutil "github.com/projectdiscovery/utils/strings"
"gopkg.in/yaml.v3"
)
type AuthType string
const (
BasicAuth AuthType = "BasicAuth"
BearerTokenAuth AuthType = "BearerToken"
HeadersAuth AuthType = "Header"
CookiesAuth AuthType = "Cookie"
QueryAuth AuthType = "Query"
)
// SupportedAuthTypes returns the supported auth types
func SupportedAuthTypes() []string {
return []string{
string(BasicAuth),
string(BearerTokenAuth),
string(HeadersAuth),
string(CookiesAuth),
string(QueryAuth),
}
}
// Authx is a struct for secrets or credentials file
type Authx struct {
ID string `json:"id" yaml:"id"`
Info AuthFileInfo `json:"info" yaml:"info"`
Secrets []Secret `json:"static" yaml:"static"`
Dynamic []Dynamic `json:"dynamic" yaml:"dynamic"`
}
type AuthFileInfo struct {
Name string `json:"name" yaml:"name"`
Author string `json:"author" yaml:"author"`
Severity string `json:"severity" yaml:"severity"`
Description string `json:"description" yaml:"description"`
}
// Secret is a struct for secret or credential
type Secret struct {
Type string `json:"type" yaml:"type"`
Domains []string `json:"domains" yaml:"domains"`
DomainsRegex []string `json:"domains-regex" yaml:"domains-regex"`
Headers []KV `json:"headers" yaml:"headers"`
Cookies []Cookie `json:"cookies" yaml:"cookies"`
Params []KV `json:"params" yaml:"params"`
Username string `json:"username" yaml:"username"` // can be either email or username
Password string `json:"password" yaml:"password"`
Token string `json:"token" yaml:"token"` // Bearer Auth token
skipCookieParse bool `json:"-" yaml:"-"` // temporary flag to skip cookie parsing (used in dynamic secrets)
}
// GetStrategy returns the auth strategy for the secret
func (s *Secret) GetStrategy() AuthStrategy {
switch {
case strings.EqualFold(s.Type, string(BasicAuth)):
return NewBasicAuthStrategy(s)
case strings.EqualFold(s.Type, string(BearerTokenAuth)):
return NewBearerTokenAuthStrategy(s)
case strings.EqualFold(s.Type, string(HeadersAuth)):
return NewHeadersAuthStrategy(s)
case strings.EqualFold(s.Type, string(CookiesAuth)):
return NewCookiesAuthStrategy(s)
case strings.EqualFold(s.Type, string(QueryAuth)):
return NewQueryAuthStrategy(s)
}
return nil
}
func (s *Secret) Validate() error {
if !stringsutil.EqualFoldAny(s.Type, SupportedAuthTypes()...) {
return fmt.Errorf("invalid type: %s", s.Type)
}
if len(s.Domains) == 0 && len(s.DomainsRegex) == 0 {
return fmt.Errorf("domains or domains-regex cannot be empty")
}
if len(s.DomainsRegex) > 0 {
for _, domain := range s.DomainsRegex {
_, err := regexp.Compile(domain)
if err != nil {
return fmt.Errorf("invalid domain regex: %s", domain)
}
}
}
switch {
case strings.EqualFold(s.Type, string(BasicAuth)):
if s.Username == "" {
return fmt.Errorf("username cannot be empty in basic auth")
}
if s.Password == "" {
return fmt.Errorf("password cannot be empty in basic auth")
}
case strings.EqualFold(s.Type, string(BearerTokenAuth)):
if s.Token == "" {
return fmt.Errorf("token cannot be empty in bearer token auth")
}
case strings.EqualFold(s.Type, string(HeadersAuth)):
if len(s.Headers) == 0 {
return fmt.Errorf("headers cannot be empty in headers auth")
}
for _, header := range s.Headers {
if err := header.Validate(); err != nil {
return fmt.Errorf("invalid header in headersAuth: %s", err)
}
}
case strings.EqualFold(s.Type, string(CookiesAuth)):
if len(s.Cookies) == 0 {
return fmt.Errorf("cookies cannot be empty in cookies auth")
}
for _, cookie := range s.Cookies {
if cookie.Raw != "" && !s.skipCookieParse {
if err := cookie.Parse(); err != nil {
return fmt.Errorf("invalid raw cookie in cookiesAuth: %s", err)
}
}
if err := cookie.Validate(); err != nil {
return fmt.Errorf("invalid cookie in cookiesAuth: %s", err)
}
}
case strings.EqualFold(s.Type, string(QueryAuth)):
if len(s.Params) == 0 {
return fmt.Errorf("query cannot be empty in query auth")
}
for _, query := range s.Params {
if err := query.Validate(); err != nil {
return fmt.Errorf("invalid query in queryAuth: %s", err)
}
}
default:
return fmt.Errorf("invalid type: %s", s.Type)
}
return nil
}
type KV struct {
Key string `json:"key" yaml:"key"`
Value string `json:"value" yaml:"value"`
}
func (k *KV) Validate() error {
if k.Key == "" {
return fmt.Errorf("key cannot be empty")
}
if k.Value == "" {
return fmt.Errorf("value cannot be empty")
}
return nil
}
type Cookie struct {
Key string `json:"key" yaml:"key"`
Value string `json:"value" yaml:"value"`
Raw string `json:"raw" yaml:"raw"`
}
func (c *Cookie) Validate() error {
if c.Raw != "" {
return nil
}
if c.Key == "" {
return fmt.Errorf("key cannot be empty")
}
if c.Value == "" {
return fmt.Errorf("value cannot be empty")
}
return nil
}
// Parse parses the cookie
// in raw the cookie is in format of
// Set-Cookie: <cookie-name>=<cookie-value>; Expires=<date>; Path=<path>; Domain=<domain_name>; Secure; HttpOnly
func (c *Cookie) Parse() error {
if c.Raw == "" {
return fmt.Errorf("raw cookie cannot be empty")
}
tmp := strings.TrimPrefix(c.Raw, "Set-Cookie: ")
slice := strings.Split(tmp, ";")
if len(slice) == 0 {
return fmt.Errorf("invalid raw cookie no ; found")
}
// first element is the cookie name and value
cookie := strings.Split(slice[0], "=")
if len(cookie) == 2 {
c.Key = cookie[0]
c.Value = cookie[1]
return nil
}
return fmt.Errorf("invalid raw cookie: %s", c.Raw)
}
// GetAuthDataFromFile reads the auth data from file
func GetAuthDataFromFile(file string) (*Authx, error) {
ext := filepath.Ext(file)
if !generic.EqualsAny(ext, ".yml", ".yaml", ".json") {
return nil, fmt.Errorf("invalid file extension: supported extensions are .yml,.yaml and .json got %s", ext)
}
bin, err := os.ReadFile(file)
if err != nil {
return nil, err
}
if ext == ".yml" || ext == ".yaml" {
return GetAuthDataFromYAML(bin)
}
return GetAuthDataFromJSON(bin)
}
// GetTemplateIDsFromSecretFile reads the template IDs from the secret file
func GetTemplatePathsFromSecretFile(file string) ([]string, error) {
auth, err := GetAuthDataFromFile(file)
if err != nil {
return nil, err
}
var paths []string
for _, dynamic := range auth.Dynamic {
paths = append(paths, dynamic.TemplatePath)
}
return paths, nil
}
// GetAuthDataFromYAML reads the auth data from yaml
func GetAuthDataFromYAML(data []byte) (*Authx, error) {
var auth Authx
err := yaml.Unmarshal(data, &auth)
if err != nil {
return nil, errorutil.NewWithErr(err).Msgf("could not unmarshal yaml")
}
return &auth, nil
}
// GetAuthDataFromJSON reads the auth data from json
func GetAuthDataFromJSON(data []byte) (*Authx, error) {
var auth Authx
err := json.Unmarshal(data, &auth)
if err != nil {
return nil, errorutil.NewWithErr(err).Msgf("could not unmarshal json")
}
return &auth, nil
}

20
pkg/authprovider/authx/file_test.go Normal file
View File

@ -0,0 +1,20 @@
package authx
import (
"testing"
"github.com/stretchr/testify/require"
)
func TestSecretsUnmarshal(t *testing.T) {
loc := "testData/example-auth.yaml"
data, err := GetAuthDataFromFile(loc)
require.Nil(t, err, "could not read secrets file")
require.NotNil(t, data, "could not read secrets file")
for _, s := range data.Secrets {
require.Nil(t, s.Validate(), "could not validate secret")
}
for _, d := range data.Dynamic {
require.Nil(t, d.Validate(), "could not validate dynamic")
}
}

35
pkg/authprovider/authx/headers_auth.go Normal file
View File

@ -0,0 +1,35 @@
package authx
import (
"net/http"
"github.com/projectdiscovery/retryablehttp-go"
)
var (
_ AuthStrategy = &HeadersAuthStrategy{}
)
// HeadersAuthStrategy is a strategy for headers auth
type HeadersAuthStrategy struct {
Data *Secret
}
// NewHeadersAuthStrategy creates a new headers auth strategy
func NewHeadersAuthStrategy(data *Secret) *HeadersAuthStrategy {
return &HeadersAuthStrategy{Data: data}
}
// Apply applies the headers auth strategy to the request
func (s *HeadersAuthStrategy) Apply(req *http.Request) {
for _, header := range s.Data.Headers {
req.Header.Set(header.Key, header.Value)
}
}
// ApplyOnRR applies the headers auth strategy to the retryable request
func (s *HeadersAuthStrategy) ApplyOnRR(req *retryablehttp.Request) {
for _, header := range s.Data.Headers {
req.Header.Set(header.Key, header.Value)
}
}

42
pkg/authprovider/authx/query_auth.go Normal file
View File

@ -0,0 +1,42 @@
package authx
import (
"net/http"
"github.com/projectdiscovery/retryablehttp-go"
urlutil "github.com/projectdiscovery/utils/url"
)
var (
_ AuthStrategy = &QueryAuthStrategy{}
)
// QueryAuthStrategy is a strategy for query auth
type QueryAuthStrategy struct {
Data *Secret
}
// NewQueryAuthStrategy creates a new query auth strategy
func NewQueryAuthStrategy(data *Secret) *QueryAuthStrategy {
return &QueryAuthStrategy{Data: data}
}
// Apply applies the query auth strategy to the request
func (s *QueryAuthStrategy) Apply(req *http.Request) {
q := urlutil.NewOrderedParams()
q.Decode(req.URL.RawQuery)
for _, p := range s.Data.Params {
q.Add(p.Key, p.Value)
}
req.URL.RawQuery = q.Encode()
}
// ApplyOnRR applies the query auth strategy to the retryable request
func (s *QueryAuthStrategy) ApplyOnRR(req *retryablehttp.Request) {
q := urlutil.NewOrderedParams()
q.Decode(req.Request.URL.RawQuery)
for _, p := range s.Data.Params {
q.Add(p.Key, p.Value)
}
req.Request.URL.RawQuery = q.Encode()
}

39
pkg/authprovider/authx/strategy.go Normal file
View File

@ -0,0 +1,39 @@
package authx
import (
"net/http"
"github.com/projectdiscovery/retryablehttp-go"
)
// AuthStrategy is an interface for auth strategies
// basic auth , bearer token, headers, cookies, query
type AuthStrategy interface {
// Apply applies the strategy to the request
Apply(*http.Request)
// ApplyOnRR applies the strategy to the retryable request
ApplyOnRR(*retryablehttp.Request)
}
// DynamicAuthStrategy is an auth strategy for dynamic secrets
// it implements the AuthStrategy interface
type DynamicAuthStrategy struct {
// Dynamic is the dynamic secret to use
Dynamic Dynamic
}
// Apply applies the strategy to the request
func (d *DynamicAuthStrategy) Apply(req *http.Request) {
strategy := d.Dynamic.GetStrategy()
if strategy != nil {
strategy.Apply(req)
}
}
// ApplyOnRR applies the strategy to the retryable request
func (d *DynamicAuthStrategy) ApplyOnRR(req *retryablehttp.Request) {
strategy := d.Dynamic.GetStrategy()
if strategy != nil {
strategy.ApplyOnRR(req)
}
}

70
pkg/authprovider/authx/testData/example-auth.yaml Normal file
View File

@ -0,0 +1,70 @@
id: pd-nuclei-auth-test
info:
name: ProjectDiscovery Test Dev Servers
author: pdteam
description: |
This is a auth file for ProjectDiscovery dev servers.
It contains auth data of all projectdiscovery dev servers.
# Note: this is a dummy example file. none of the secrets here are real.
# static secrets
static:
# for header based auth session
- type: header
domains:
- api.projectdiscovery.io
- cve.projectdiscovery.io
- chaos.projectdiscovery.io
headers:
- key: x-pdcp-key
value: <api-key-here>
# for query based auth session
- type: Query
domains:
- scanme.sh
params:
- key: token
value: 1a2b3c4d5e6f7g8h9i0j
# for cookie based auth session
- type: Cookie
domains:
- scanme.sh
cookies:
- key: PHPSESSID
value: 1a2b3c4d5e6f7g8h9i0j
# for basic auth session
- type: BasicAuth
domains:
- scanme.sh
username: test
password: test
# for authorization bearer token
- type: BearerToken
domains-regex:
- .*scanme.sh
- .*pdtm.sh
token: test
# dynamic secrets (powered by nuclei-templates)
dynamic:
- template: /path/to/wordpress-login.yaml
variables:
- name: username
value: pdteam
- name: password
value: nuclei-v3.2.0
type: Cookie
domains:
- localhost:8080
cookies:
- raw: "{{wp-global-cookie}}"
- raw: "{{wp-admin-cookie}}"
- raw: "{{wp-plugin-cookie}}"

168
pkg/authprovider/file.go Normal file
View File

@ -0,0 +1,168 @@
package authprovider
import (
"net"
"net/url"
"regexp"
"strings"
"github.com/projectdiscovery/nuclei/v3/pkg/authprovider/authx"
errorutil "github.com/projectdiscovery/utils/errors"
urlutil "github.com/projectdiscovery/utils/url"
)
// FileAuthProvider is an auth provider for file based auth
// it accepts a secrets file and returns its provider
type FileAuthProvider struct {
Path string
store *authx.Authx
compiled map[*regexp.Regexp]authx.AuthStrategy
domains map[string]authx.AuthStrategy
}
// NewFileAuthProvider creates a new file based auth provider
func NewFileAuthProvider(path string, callback authx.LazyFetchSecret) (AuthProvider, error) {
store, err := authx.GetAuthDataFromFile(path)
if err != nil {
return nil, err
}
if len(store.Secrets) == 0 && len(store.Dynamic) == 0 {
return nil, ErrNoSecrets
}
if len(store.Dynamic) > 0 && callback == nil {
return nil, errorutil.New("lazy fetch callback is required for dynamic secrets")
}
for _, secret := range store.Secrets {
if err := secret.Validate(); err != nil {
return nil, errorutil.NewWithErr(err).Msgf("invalid secret in file: %s", path)
}
}
for i, dynamic := range store.Dynamic {
if err := dynamic.Validate(); err != nil {
return nil, errorutil.NewWithErr(err).Msgf("invalid dynamic in file: %s", path)
}
dynamic.SetLazyFetchCallback(callback)
store.Dynamic[i] = dynamic
}
f := &FileAuthProvider{Path: path, store: store}
f.init()
return f, nil
}
// init initializes the file auth provider
func (f *FileAuthProvider) init() {
for _, secret := range f.store.Secrets {
if len(secret.DomainsRegex) > 0 {
for _, domain := range secret.DomainsRegex {
if f.compiled == nil {
f.compiled = make(map[*regexp.Regexp]authx.AuthStrategy)
}
compiled, err := regexp.Compile(domain)
if err != nil {
continue
}
f.compiled[compiled] = secret.GetStrategy()
}
}
for _, domain := range secret.Domains {
if f.domains == nil {
f.domains = make(map[string]authx.AuthStrategy)
}
f.domains[strings.TrimSpace(domain)] = secret.GetStrategy()
if strings.HasSuffix(domain, ":80") {
f.domains[strings.TrimSuffix(domain, ":80")] = secret.GetStrategy()
}
if strings.HasSuffix(domain, ":443") {
f.domains[strings.TrimSuffix(domain, ":443")] = secret.GetStrategy()
}
}
}
for _, dynamic := range f.store.Dynamic {
if len(dynamic.DomainsRegex) > 0 {
for _, domain := range dynamic.DomainsRegex {
if f.compiled == nil {
f.compiled = make(map[*regexp.Regexp]authx.AuthStrategy)
}
compiled, err := regexp.Compile(domain)
if err != nil {
continue
}
f.compiled[compiled] = &authx.DynamicAuthStrategy{Dynamic: dynamic}
}
}
for _, domain := range dynamic.Domains {
if f.domains == nil {
f.domains = make(map[string]authx.AuthStrategy)
}
f.domains[strings.TrimSpace(domain)] = &authx.DynamicAuthStrategy{Dynamic: dynamic}
if strings.HasSuffix(domain, ":80") {
f.domains[strings.TrimSuffix(domain, ":80")] = &authx.DynamicAuthStrategy{Dynamic: dynamic}
}
if strings.HasSuffix(domain, ":443") {
f.domains[strings.TrimSuffix(domain, ":443")] = &authx.DynamicAuthStrategy{Dynamic: dynamic}
}
}
}
}
// LookupAddr looks up a given domain/address and returns appropriate auth strategy
func (f *FileAuthProvider) LookupAddr(addr string) authx.AuthStrategy {
if strings.Contains(addr, ":") {
// default normalization for host:port
host, port, err := net.SplitHostPort(addr)
if err == nil && (port == "80" || port == "443") {
addr = host
}
}
for domain, strategy := range f.domains {
if strings.EqualFold(domain, addr) {
return strategy
}
}
for compiled, strategy := range f.compiled {
if compiled.MatchString(addr) {
return strategy
}
}
return nil
}
// LookupURL looks up a given URL and returns appropriate auth strategy
func (f *FileAuthProvider) LookupURL(u *url.URL) authx.AuthStrategy {
return f.LookupAddr(u.Host)
}
// LookupURLX looks up a given URL and returns appropriate auth strategy
func (f *FileAuthProvider) LookupURLX(u *urlutil.URL) authx.AuthStrategy {
return f.LookupAddr(u.Host)
}
// GetTemplatePaths returns the template path for the auth provider
func (f *FileAuthProvider) GetTemplatePaths() []string {
res := []string{}
for _, dynamic := range f.store.Dynamic {
if dynamic.TemplatePath != "" {
res = append(res, dynamic.TemplatePath)
}
}
return res
}
// PreFetchSecrets pre-fetches the secrets from the auth provider
func (f *FileAuthProvider) PreFetchSecrets() error {
for _, s := range f.domains {
if val, ok := s.(*authx.DynamicAuthStrategy); ok {
if err := val.Dynamic.Fetch(false); err != nil {
return err
}
}
}
for _, s := range f.compiled {
if val, ok := s.(*authx.DynamicAuthStrategy); ok {
if err := val.Dynamic.Fetch(false); err != nil {
return err
}
}
}
return nil
}

59
pkg/authprovider/interface.go Normal file
View File

@ -0,0 +1,59 @@
package authprovider
import (
"fmt"
"net/url"
"github.com/projectdiscovery/nuclei/v3/pkg/authprovider/authx"
urlutil "github.com/projectdiscovery/utils/url"
)
var (
ErrNoSecrets = fmt.Errorf("no secrets in given provider")
)
var (
_ AuthProvider = &FileAuthProvider{}
)
// AuthProvider is an interface for auth providers
// It implements a data structure suitable for quick lookup and retrieval
// of auth strategies
type AuthProvider interface {
// LookupAddr looks up a given domain/address and returns appropriate auth strategy
// for it (accepted inputs are scanme.sh or scanme.sh:443)
LookupAddr(string) authx.AuthStrategy
// LookupURL looks up a given URL and returns appropriate auth strategy
// it accepts a valid url struct and returns the auth strategy
LookupURL(*url.URL) authx.AuthStrategy
// LookupURLX looks up a given URL and returns appropriate auth strategy
// it accepts pd url struct (i.e urlutil.URL) and returns the auth strategy
LookupURLX(*urlutil.URL) authx.AuthStrategy
// GetTemplatePaths returns the template path for the auth provider
// that will be used for dynamic secret fetching
GetTemplatePaths() []string
// PreFetchSecrets pre-fetches the secrets from the auth provider
// instead of lazy fetching
PreFetchSecrets() error
}
// AuthProviderOptions contains options for the auth provider
type AuthProviderOptions struct {
// File based auth provider options
SecretsFiles []string
// LazyFetchSecret is a callback for lazy fetching of dynamic secrets
LazyFetchSecret authx.LazyFetchSecret
}
// NewAuthProvider creates a new auth provider from the given options
func NewAuthProvider(options *AuthProviderOptions) (AuthProvider, error) {
var providers []AuthProvider
for _, file := range options.SecretsFiles {
provider, err := NewFileAuthProvider(file, options.LazyFetchSecret)
if err != nil {
return nil, err
}
providers = append(providers, provider)
}
return NewMultiAuthProvider(providers...), nil
}

67
pkg/authprovider/multi.go Normal file
View File

@ -0,0 +1,67 @@
package authprovider
import (
"net/url"
"github.com/projectdiscovery/nuclei/v3/pkg/authprovider/authx"
urlutil "github.com/projectdiscovery/utils/url"
)
// MultiAuthProvider is a convenience wrapper for multiple auth providers
// it returns the first matching auth strategy for a given domain
// if there are multiple auth strategies for a given domain, it returns the first one
type MultiAuthProvider struct {
Providers []AuthProvider
}
// NewMultiAuthProvider creates a new multi auth provider
func NewMultiAuthProvider(providers ...AuthProvider) AuthProvider {
return &MultiAuthProvider{Providers: providers}
}
func (m *MultiAuthProvider) LookupAddr(host string) authx.AuthStrategy {
for _, provider := range m.Providers {
strategy := provider.LookupAddr(host)
if strategy != nil {
return strategy
}
}
return nil
}
func (m *MultiAuthProvider) LookupURL(u *url.URL) authx.AuthStrategy {
for _, provider := range m.Providers {
strategy := provider.LookupURL(u)
if strategy != nil {
return strategy
}
}
return nil
}
func (m *MultiAuthProvider) LookupURLX(u *urlutil.URL) authx.AuthStrategy {
for _, provider := range m.Providers {
strategy := provider.LookupURLX(u)
if strategy != nil {
return strategy
}
}
return nil
}
func (m *MultiAuthProvider) GetTemplatePaths() []string {
var res []string
for _, provider := range m.Providers {
res = append(res, provider.GetTemplatePaths()...)
}
return res
}
func (m *MultiAuthProvider) PreFetchSecrets() error {
for _, provider := range m.Providers {
if err := provider.PreFetchSecrets(); err != nil {
return err
}
}
return nil
}

16
pkg/catalog/config/constants.go
View File

@ -6,6 +6,20 @@ import (
"github.com/Masterminds/semver/v3"
)
type AppMode string
const (
AppModeLibrary AppMode = "library"
AppModeCLI AppMode = "cli"
)
var (
// Global Var to control behaviours specific to cli or library
// maybe this should be moved to utils ??
// this is overwritten in cmd/nuclei/main.go
CurrentAppMode = AppModeLibrary
)
const (
TemplateConfigFileName = ".templates-config.json"
NucleiTemplatesDirName = "nuclei-templates"
@ -17,7 +31,7 @@ const (
CLIConfigFileName = "config.yaml"
ReportingConfigFilename = "reporting-config.yaml"
// Version is the current version of nuclei
Version = `v3.2.0-dev`
Version = `v3.2.0`
// Directory Names of custom templates
CustomS3TemplatesDirName = "s3"
CustomGitHubTemplatesDirName = "github"

66
pkg/catalog/loader/loader.go
View File

@ -13,7 +13,6 @@ import (
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
cfg "github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/loader/filter"
"github.com/projectdiscovery/nuclei/v3/pkg/model/types/severity"
"github.com/projectdiscovery/nuclei/v3/pkg/parsers"
@ -62,6 +61,8 @@ type Config struct {
Catalog catalog.Catalog
ExecutorOptions protocols.ExecutorOptions
OnlyLoadHTTPFuzzing bool
}
// Store is a storage for loaded nuclei templates
@ -111,19 +112,19 @@ func NewConfig(options *types.Options, catalog catalog.Catalog, executerOpts pro
}
// New creates a new template store based on provided configuration
func New(config *Config) (*Store, error) {
func New(cfg *Config) (*Store, error) {
tagFilter, err := filter.New(&filter.Config{
Tags: config.Tags,
ExcludeTags: config.ExcludeTags,
Authors: config.Authors,
Severities: config.Severities,
ExcludeSeverities: config.ExcludeSeverities,
IncludeTags: config.IncludeTags,
IncludeIds: config.IncludeIds,
ExcludeIds: config.ExcludeIds,
Protocols: config.Protocols,
ExcludeProtocols: config.ExcludeProtocols,
IncludeConditions: config.IncludeConditions,
Tags: cfg.Tags,
ExcludeTags: cfg.ExcludeTags,
Authors: cfg.Authors,
Severities: cfg.Severities,
ExcludeSeverities: cfg.ExcludeSeverities,
IncludeTags: cfg.IncludeTags,
IncludeIds: cfg.IncludeIds,
ExcludeIds: cfg.ExcludeIds,
Protocols: cfg.Protocols,
ExcludeProtocols: cfg.ExcludeProtocols,
IncludeConditions: cfg.IncludeConditions,
})
if err != nil {
return nil, err
@ -131,23 +132,23 @@ func New(config *Config) (*Store, error) {
// Create a tag filter based on provided configuration
store := &Store{
config: config,
config: cfg,
tagFilter: tagFilter,
pathFilter: filter.NewPathFilter(&filter.PathFilterConfig{
IncludedTemplates: config.IncludeTemplates,
ExcludedTemplates: config.ExcludeTemplates,
}, config.Catalog),
finalTemplates: config.Templates,
finalWorkflows: config.Workflows,
IncludedTemplates: cfg.IncludeTemplates,
ExcludedTemplates: cfg.ExcludeTemplates,
}, cfg.Catalog),
finalTemplates: cfg.Templates,
finalWorkflows: cfg.Workflows,
}
// Do a check to see if we have URLs in templates flag, if so
// we need to processs them separately and remove them from the initial list
var templatesFinal []string
for _, template := range config.Templates {
for _, template := range cfg.Templates {
// TODO: Add and replace this with urlutil.IsURL() helper
if stringsutil.HasPrefixAny(template, httpPrefix, httpsPrefix) {
config.TemplateURLs = append(config.TemplateURLs, template)
cfg.TemplateURLs = append(cfg.TemplateURLs, template)
} else {
templatesFinal = append(templatesFinal, template)
}
@ -155,7 +156,7 @@ func New(config *Config) (*Store, error) {
// fix editor paths
remoteTemplates := []string{}
for _, v := range config.TemplateURLs {
for _, v := range cfg.TemplateURLs {
if _, err := urlutil.Parse(v); err == nil {
remoteTemplates = append(remoteTemplates, handleTemplatesEditorURLs(v))
} else {
@ -163,12 +164,12 @@ func New(config *Config) (*Store, error) {
templatesFinal = append(templatesFinal, v) // something went wrong, treat it as a file
}
}
config.TemplateURLs = remoteTemplates
cfg.TemplateURLs = remoteTemplates
store.finalTemplates = templatesFinal
urlBasedTemplatesProvided := len(config.TemplateURLs) > 0 || len(config.WorkflowURLs) > 0
urlBasedTemplatesProvided := len(cfg.TemplateURLs) > 0 || len(cfg.WorkflowURLs) > 0
if urlBasedTemplatesProvided {
remoteTemplates, remoteWorkflows, err := getRemoteTemplatesAndWorkflows(config.TemplateURLs, config.WorkflowURLs, config.RemoteTemplateDomainList)
remoteTemplates, remoteWorkflows, err := getRemoteTemplatesAndWorkflows(cfg.TemplateURLs, cfg.WorkflowURLs, cfg.RemoteTemplateDomainList)
if err != nil {
return store, err
}
@ -186,7 +187,7 @@ func New(config *Config) (*Store, error) {
}
// Handle a case with no templates or workflows, where we use base directory
if len(store.finalTemplates) == 0 && len(store.finalWorkflows) == 0 && !urlBasedTemplatesProvided {
store.finalTemplates = []string{cfg.DefaultConfig.TemplatesDirectory}
store.finalTemplates = []string{config.DefaultConfig.TemplatesDirectory}
}
return store, nil
@ -405,13 +406,13 @@ func (store *Store) LoadTemplatesWithTags(templatesList, tags []string) []*templ
if len(parsed.RequestsHeadless) > 0 && !store.config.ExecutorOptions.Options.Headless {
// donot include headless template in final list if headless flag is not set
stats.Increment(parsers.HeadlessFlagWarningStats)
if cfg.DefaultConfig.LogAllEvents {
if config.DefaultConfig.LogAllEvents {
gologger.Print().Msgf("[%v] Headless flag is required for headless template '%s'.\n", aurora.Yellow("WRN").String(), templatePath)
}
} else if len(parsed.RequestsCode) > 0 && !store.config.ExecutorOptions.Options.EnableCodeTemplates {
// donot include 'Code' protocol custom template in final list if code flag is not set
stats.Increment(parsers.CodeFlagWarningStats)
if cfg.DefaultConfig.LogAllEvents {
if config.DefaultConfig.LogAllEvents {
gologger.Print().Msgf("[%v] Code flag is required for code protocol template '%s'.\n", aurora.Yellow("WRN").String(), templatePath)
}
} else if len(parsed.RequestsCode) > 0 && !parsed.Verified && len(parsed.Workflows) == 0 {
@ -419,9 +420,16 @@ func (store *Store) LoadTemplatesWithTags(templatesList, tags []string) []*templ
stats.Increment(parsers.UnsignedCodeWarning)
// these will be skipped so increment skip counter
stats.Increment(parsers.SkippedUnsignedStats)
if cfg.DefaultConfig.LogAllEvents {
if config.DefaultConfig.LogAllEvents {
gologger.Print().Msgf("[%v] Tampered/Unsigned template at %v.\n", aurora.Yellow("WRN").String(), templatePath)
}
} else if parsed.IsFuzzing() && !store.config.ExecutorOptions.Options.FuzzTemplates {
stats.Increment(parsers.FuzzFlagWarningStats)
if config.DefaultConfig.LogAllEvents {
gologger.Print().Msgf("[%v] Fuzz flag is required for fuzzing template '%s'.\n", aurora.Yellow("WRN").String(), templatePath)
}
} else if store.config.OnlyLoadHTTPFuzzing && !parsed.IsFuzzing() {
gologger.Warning().Msgf("Non-Fuzzing template '%s' can only be run on list input mode targets\n", templatePath)
} else {
loadedTemplates = append(loadedTemplates, parsed)
}

16
pkg/core/engine.go
View File

@ -3,7 +3,6 @@ package core
import (
"github.com/projectdiscovery/nuclei/v3/pkg/output"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
)
@ -22,21 +21,6 @@ type Engine struct {
Callback func(*output.ResultEvent) // Executed on results
}
// InputProvider is an input providing interface for the nuclei execution
// engine.
//
// An example InputProvider implementation is provided in form of hybrid
// input provider in pkg/core/inputs/hybrid/hmap.go
type InputProvider interface {
// Count returns the number of items for input provider
Count() int64
// Scan iterates the input and each found item is passed to the
// callback consumer.
Scan(callback func(value *contextargs.MetaInput) bool)
// Set adds item to input provider
Set(value string)
}
// New returns a new Engine instance
func New(options *types.Options) *Engine {
engine := &Engine{

13
pkg/core/execute_options.go
View File

@ -7,6 +7,7 @@ import (
"github.com/remeh/sizedwaitgroup"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v3/pkg/input/provider"
"github.com/projectdiscovery/nuclei/v3/pkg/output"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
"github.com/projectdiscovery/nuclei/v3/pkg/templates"
@ -20,18 +21,18 @@ import (
//
// All the execution logic for the templates/workflows happens in this part
// of the engine.
func (e *Engine) Execute(templates []*templates.Template, target InputProvider) *atomic.Bool {
func (e *Engine) Execute(templates []*templates.Template, target provider.InputProvider) *atomic.Bool {
return e.ExecuteScanWithOpts(templates, target, false)
}
// ExecuteWithResults a list of templates with results
func (e *Engine) ExecuteWithResults(templatesList []*templates.Template, target InputProvider, callback func(*output.ResultEvent)) *atomic.Bool {
func (e *Engine) ExecuteWithResults(templatesList []*templates.Template, target provider.InputProvider, callback func(*output.ResultEvent)) *atomic.Bool {
e.Callback = callback
return e.ExecuteScanWithOpts(templatesList, target, false)
}
// ExecuteScanWithOpts executes scan with given scanStrategy
func (e *Engine) ExecuteScanWithOpts(templatesList []*templates.Template, target InputProvider, noCluster bool) *atomic.Bool {
func (e *Engine) ExecuteScanWithOpts(templatesList []*templates.Template, target provider.InputProvider, noCluster bool) *atomic.Bool {
results := &atomic.Bool{}
selfcontainedWg := &sync.WaitGroup{}
@ -100,7 +101,7 @@ func (e *Engine) ExecuteScanWithOpts(templatesList []*templates.Template, target
}
// executeTemplateSpray executes scan using template spray strategy where targets are iterated over each template
func (e *Engine) executeTemplateSpray(templatesList []*templates.Template, target InputProvider) *atomic.Bool {
func (e *Engine) executeTemplateSpray(templatesList []*templates.Template, target provider.InputProvider) *atomic.Bool {
results := &atomic.Bool{}
// wp is workpool that contains different waitgroups for
@ -131,11 +132,11 @@ func (e *Engine) executeTemplateSpray(templatesList []*templates.Template, targe
}
// executeHostSpray executes scan using host spray strategy where templates are iterated over each target
func (e *Engine) executeHostSpray(templatesList []*templates.Template, target InputProvider) *atomic.Bool {
func (e *Engine) executeHostSpray(templatesList []*templates.Template, target provider.InputProvider) *atomic.Bool {
results := &atomic.Bool{}
wp := sizedwaitgroup.New(e.options.BulkSize + e.options.HeadlessBulkSize)
target.Scan(func(value *contextargs.MetaInput) bool {
target.Iterate(func(value *contextargs.MetaInput) bool {
wp.Add()
go func(targetval *contextargs.MetaInput) {
defer wp.Done()

5
pkg/core/executors.go
View File

@ -5,6 +5,7 @@ import (
"sync/atomic"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v3/pkg/input/provider"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
"github.com/projectdiscovery/nuclei/v3/pkg/scan"
"github.com/projectdiscovery/nuclei/v3/pkg/templates"
@ -44,7 +45,7 @@ func (e *Engine) executeAllSelfContained(alltemplates []*templates.Template, res
}
// executeTemplateWithTarget executes a given template on x targets (with a internal targetpool(i.e concurrency))
func (e *Engine) executeTemplateWithTargets(template *templates.Template, target InputProvider, results *atomic.Bool) {
func (e *Engine) executeTemplateWithTargets(template *templates.Template, target provider.InputProvider, results *atomic.Bool) {
// this is target pool i.e max target to execute
wg := e.workPool.InputPool(template.Type())
@ -75,7 +76,7 @@ func (e *Engine) executeTemplateWithTargets(template *templates.Template, target
currentInfo.Unlock()
}
target.Scan(func(scannedValue *contextargs.MetaInput) bool {
target.Iterate(func(scannedValue *contextargs.MetaInput) bool {
// Best effort to track the host progression
// skips indexes lower than the minimum in-flight at interruption time
var skip bool

39
pkg/core/inputs/inputs.go
View File

@ -1,39 +0,0 @@
package inputs
import (
"github.com/projectdiscovery/httpx/common/httpx"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
"github.com/projectdiscovery/nuclei/v3/pkg/utils"
)
type SimpleInputProvider struct {
Inputs []*contextargs.MetaInput
}
// Count returns the number of items for input provider
func (s *SimpleInputProvider) Count() int64 {
return int64(len(s.Inputs))
}
// Scan calls a callback function till the input provider is exhausted
func (s *SimpleInputProvider) Scan(callback func(value *contextargs.MetaInput) bool) {
for _, v := range s.Inputs {
if !callback(v) {
return
}
}
}
// Set adds item to input provider
func (s *SimpleInputProvider) Set(value string) {
s.Inputs = append(s.Inputs, &contextargs.MetaInput{Input: value})
}
// SetWithProbe adds item to input provider with http probing
func (s *SimpleInputProvider) SetWithProbe(value string, httpxClient *httpx.HTTPX) {
valueToAppend := value
if result := utils.ProbeURL(value, httpxClient); result != "" {
valueToAppend = result
}
s.Inputs = append(s.Inputs, &contextargs.MetaInput{Input: valueToAppend})
}

140
pkg/fuzz/component/body.go Normal file
View File

@ -0,0 +1,140 @@
package component
import (
"bytes"
"context"
"io"
"strconv"
"strings"
"github.com/pkg/errors"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v3/pkg/fuzz/dataformat"
"github.com/projectdiscovery/retryablehttp-go"
readerutil "github.com/projectdiscovery/utils/reader"
)
// Body is a component for a request body
type Body struct {
value *Value
req *retryablehttp.Request
}
var _ Component = &Body{}
// NewBody creates a new body component
func NewBody() *Body {
return &Body{}
}
// Name returns the name of the component
func (b *Body) Name() string {
return RequestBodyComponent
}
// Parse parses the component and returns the
// parsed component
func (b *Body) Parse(req *retryablehttp.Request) (bool, error) {
if req.Body == nil {
return false, nil
}
b.req = req
contentType := req.Header.Get("Content-Type")
data, err := io.ReadAll(req.Body)
if err != nil {
return false, errors.Wrap(err, "could not read body")
}
req.Body = io.NopCloser(bytes.NewReader(data))
dataStr := string(data)
if dataStr == "" {
return false, nil
}
b.value = NewValue(dataStr)
if b.value.Parsed() != nil {
return true, nil
}
switch {
case strings.Contains(contentType, "application/json") && b.value.Parsed() == nil:
return b.parseBody(dataformat.JSONDataFormat, req)
case strings.Contains(contentType, "application/xml") && b.value.Parsed() == nil:
return b.parseBody(dataformat.XMLDataFormat, req)
case strings.Contains(contentType, "multipart/form-data") && b.value.Parsed() == nil:
return b.parseBody(dataformat.MultiPartFormDataFormat, req)
}
parsed, err := b.parseBody(dataformat.FormDataFormat, req)
if err != nil {
gologger.Warning().Msgf("Could not parse body as form data: %s\n", err)
return b.parseBody(dataformat.RawDataFormat, req)
}
return parsed, err
}
// parseBody parses a body with a custom decoder
func (b *Body) parseBody(decoderName string, req *retryablehttp.Request) (bool, error) {
decoder := dataformat.Get(decoderName)
if decoderName == dataformat.MultiPartFormDataFormat {
// set content type to extract boundary
if err := decoder.(*dataformat.MultiPartForm).ParseBoundary(req.Header.Get("Content-Type")); err != nil {
return false, errors.Wrap(err, "could not parse boundary")
}
}
decoded, err := decoder.Decode(b.value.String())
if err != nil {
return false, errors.Wrap(err, "could not decode raw")
}
b.value.SetParsed(decoded, decoder.Name())
return true, nil
}
// Iterate iterates through the component
func (b *Body) Iterate(callback func(key string, value interface{}) error) error {
for key, value := range b.value.Parsed() {
if strings.HasPrefix(key, "#_") {
continue
}
if err := callback(key, value); err != nil {
return err
}
}
return nil
}
// SetValue sets a value in the component
func (b *Body) SetValue(key string, value string) error {
if !b.value.SetParsedValue(key, value) {
return ErrSetValue
}
return nil
}
// Delete deletes a key from the component
func (b *Body) Delete(key string) error {
if !b.value.Delete(key) {
return ErrKeyNotFound
}
return nil
}
// Rebuild returns a new request with the
// component rebuilt
func (b *Body) Rebuild() (*retryablehttp.Request, error) {
encoded, err := b.value.Encode()
if err != nil {
return nil, errors.Wrap(err, "could not encode body")
}
cloned := b.req.Clone(context.Background())
reusableReader, err := readerutil.NewReusableReadCloser(encoded)
if err != nil {
return nil, errors.Wrap(err, "could not create reusable reader")
}
cloned.Body = reusableReader
cloned.ContentLength = int64(len(encoded))
cloned.Header.Set("Content-Length", strconv.Itoa(len(encoded)))
return cloned, nil
}

175
pkg/fuzz/component/body_test.go Normal file
View File

@ -0,0 +1,175 @@
package component
import (
"bytes"
"io"
"mime/multipart"
"net/url"
"strings"
"testing"
"github.com/projectdiscovery/retryablehttp-go"
"github.com/stretchr/testify/require"
)
func TestBodyComponent(t *testing.T) {
req, err := retryablehttp.NewRequest("POST", "https://example.com", strings.NewReader(`{"foo":"bar"}`))
if err != nil {
t.Fatal(err)
}
req.Header.Set("Content-Type", "application/json")
body := New(RequestBodyComponent)
_, err = body.Parse(req)
if err != nil {
t.Fatal(err)
}
var keys []string
var values []string
_ = body.Iterate(func(key string, value interface{}) error {
keys = append(keys, key)
values = append(values, value.(string))
return nil
})
require.Equal(t, []string{"foo"}, keys, "unexpected keys")
require.Equal(t, []string{"bar"}, values, "unexpected values")
_ = body.SetValue("foo", "baz")
rebuilt, err := body.Rebuild()
if err != nil {
t.Fatal(err)
}
newBody, err := io.ReadAll(rebuilt.Body)
if err != nil {
t.Fatal(err)
}
require.Equal(t, `{"foo":"baz"}`, string(newBody), "unexpected body")
}
func TestBodyXMLComponent(t *testing.T) {
var body = "<?xml version=\"1.0\" encoding=\"UTF-8\"?><stockCheck><productId>1</productId><storeId>1</storeId></stockCheck>"
req, err := retryablehttp.NewRequest("POST", "https://example.com", strings.NewReader(body))
if err != nil {
t.Fatal(err)
}
req.Header.Set("Content-Type", "application/xml")
bodyComponent := New(RequestBodyComponent)
parsed, err := bodyComponent.Parse(req)
if err != nil {
t.Fatal(err)
}
require.True(t, parsed, "could not parse body")
_ = bodyComponent.SetValue("stockCheck~productId", "2'6842")
rebuilt, err := bodyComponent.Rebuild()
if err != nil {
t.Fatal(err)
}
newBody, err := io.ReadAll(rebuilt.Body)
if err != nil {
t.Fatal(err)
}
require.Equal(t, "<?xml version=\"1.0\" encoding=\"UTF-8\"?><stockCheck><productId>2'6842</productId><storeId>1</storeId></stockCheck>", string(newBody), "unexpected body")
}
func TestBodyFormComponent(t *testing.T) {
formData := url.Values{}
formData.Set("key1", "value1")
formData.Set("key2", "value2")
req, err := retryablehttp.NewRequest("POST", "https://example.com", strings.NewReader(formData.Encode()))
if err != nil {
t.Fatal(err)
}
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
body := New(RequestBodyComponent)
_, err = body.Parse(req)
if err != nil {
t.Fatal(err)
}
var keys []string
var values []string
_ = body.Iterate(func(key string, value interface{}) error {
keys = append(keys, key)
values = append(values, value.(string))
return nil
})
require.ElementsMatch(t, []string{"key1", "key2"}, keys, "unexpected keys")
require.ElementsMatch(t, []string{"value1", "value2"}, values, "unexpected values")
_ = body.SetValue("key1", "updatedValue1")
rebuilt, err := body.Rebuild()
if err != nil {
t.Fatal(err)
}
newBody, err := io.ReadAll(rebuilt.Body)
if err != nil {
t.Fatal(err)
}
require.Equal(t, "key1=updatedValue1&key2=value2", string(newBody), "unexpected body")
}
func TestMultiPartFormComponent(t *testing.T) {
formData := &bytes.Buffer{}
writer := multipart.NewWriter(formData)
// Hypothetical form fields
_ = writer.WriteField("username", "testuser")
_ = writer.WriteField("password", "testpass")
contentType := writer.FormDataContentType()
_ = writer.Close()
req, err := retryablehttp.NewRequest("POST", "https://example.com", formData)
if err != nil {
t.Fatal(err)
}
req.Header.Set("Content-Type", contentType)
body := New(RequestBodyComponent)
_, err = body.Parse(req)
if err != nil {
t.Fatal(err)
}
var keys []string
var values []string
_ = body.Iterate(func(key string, value interface{}) error {
keys = append(keys, key)
values = append(values, value.(string))
return nil
})
require.ElementsMatch(t, []string{"username", "password"}, keys, "unexpected keys")
require.ElementsMatch(t, []string{"testuser", "testpass"}, values, "unexpected values")
// Update a value in the form
_ = body.SetValue("password", "updatedTestPass")
rebuilt, err := body.Rebuild()
if err != nil {
t.Fatal(err)
}
newBody, err := io.ReadAll(rebuilt.Body)
if err != nil {
t.Fatal(err)
}
// Check if the body contains the updated multipart form data
require.Contains(t, string(newBody), "updatedTestPass", "unexpected body content")
require.Contains(t, string(newBody), "username", "unexpected body content")
require.Contains(t, string(newBody), "testuser", "unexpected body content")
}

95
pkg/fuzz/component/component.go Normal file
View File

@ -0,0 +1,95 @@
package component
import (
"errors"
"strings"
"github.com/leslie-qiwa/flat"
"github.com/projectdiscovery/retryablehttp-go"
)
// ErrSetValue is a error raised when a value cannot be set
var ErrSetValue = errors.New("could not set value")
func IsErrSetValue(err error) bool {
if err == nil {
return false
}
return strings.Contains(err.Error(), "could not set value")
}
// ErrKeyNotFound is a error raised when a key is not found
var ErrKeyNotFound = errors.New("key not found")
// Component is a component for a request
type Component interface {
// Name returns the name of the component
Name() string
// Parse parses the component and returns the
// parsed component
Parse(req *retryablehttp.Request) (bool, error)
// Iterate iterates over all values of a component
// ex in case of query component, it will iterate over each query parameter
// depending on the rule if mode is single
// request is rebuilt for each value in this callback
// and in case of multiple, request will be rebuilt after iteration of all values
Iterate(func(key string, value interface{}) error) error
// SetValue sets a value in the component
// for a key
//
// After calling setValue for mutation, the value must be
// called again so as to reset the body to its original state.
SetValue(key string, value string) error
// Delete deletes a key from the component
// If it is applicable
Delete(key string) error
// Rebuild returns a new request with the
// component rebuilt
Rebuild() (*retryablehttp.Request, error)
}
const (
// RequestBodyComponent is the name of the request body component
RequestBodyComponent = "body"
// RequestQueryComponent is the name of the request query component
RequestQueryComponent = "query"
// RequestPathComponent is the name of the request url component
RequestPathComponent = "path"
// RequestHeaderComponent is the name of the request header component
RequestHeaderComponent = "header"
// RequestCookieComponent is the name of the request cookie component
RequestCookieComponent = "cookie"
)
// Components is a list of all available components
var Components = []string{
RequestBodyComponent,
RequestQueryComponent,
RequestPathComponent,
RequestHeaderComponent,
RequestCookieComponent,
}
// New creates a new component for a componentType
func New(componentType string) Component {
switch componentType {
case "body":
return NewBody()
case "query":
return NewQuery()
case "path":
return NewPath()
case "header":
return NewHeader()
case "cookie":
return NewCookie()
}
return nil
}
var (
flatOpts = &flat.Options{
Safe: true,
Delimiter: "~",
}
)

138
pkg/fuzz/component/cookie.go Normal file
View File

@ -0,0 +1,138 @@
package component
import (
"context"
"net/http"
"github.com/projectdiscovery/retryablehttp-go"
)
// Cookie is a component for a request cookie
type Cookie struct {
value *Value
req *retryablehttp.Request
}
var _ Component = &Cookie{}
// NewCookie creates a new cookie component
func NewCookie() *Cookie {
return &Cookie{}
}
// Name returns the name of the component
func (c *Cookie) Name() string {
return RequestCookieComponent
}
// Parse parses the component and returns the
// parsed component
func (c *Cookie) Parse(req *retryablehttp.Request) (bool, error) {
if len(req.Cookies()) == 0 {
return false, nil
}
c.req = req
c.value = NewValue("")
parsedCookies := make(map[string]interface{})
for _, cookie := range req.Cookies() {
parsedCookies[cookie.Name] = cookie.Value
}
if len(parsedCookies) == 0 {
return false, nil
}
c.value.SetParsed(parsedCookies, "")
return true, nil
}
// Iterate iterates through the component
func (c *Cookie) Iterate(callback func(key string, value interface{}) error) error {
for key, value := range c.value.Parsed() {
// Skip ignored cookies
if _, ok := defaultIgnoredCookieKeys[key]; ok {
continue
}
if err := callback(key, value); err != nil {
return err
}
}
return nil
}
// SetValue sets a value in the component
// for a key
func (c *Cookie) SetValue(key string, value string) error {
if !c.value.SetParsedValue(key, value) {
return ErrSetValue
}
return nil
}
// Delete deletes a key from the component
func (c *Cookie) Delete(key string) error {
if !c.value.Delete(key) {
return ErrKeyNotFound
}
return nil
}
// Rebuild returns a new request with the
// component rebuilt
func (c *Cookie) Rebuild() (*retryablehttp.Request, error) {
cloned := c.req.Clone(context.Background())
cloned.Header.Del("Cookie")
for key, value := range c.value.Parsed() {
cookie := &http.Cookie{
Name: key,
Value: value.(string), // Assume the value is always a string for cookies
}
cloned.AddCookie(cookie)
}
return cloned, nil
}
// A list of cookies that are essential to the request and
// must not be fuzzed.
var defaultIgnoredCookieKeys = map[string]struct{}{
"awsELB": {},
"AWSALB": {},
"AWSALBCORS": {},
"__utma": {},
"__utmb": {},
"__utmc": {},
"__utmt": {},
"__utmz": {},
"_ga": {},
"_gat": {},
"_gid": {},
"_gcl_au": {},
"_fbp": {},
"fr": {},
"__hstc": {},
"hubspotutk": {},
"__hssc": {},
"__hssrc": {},
"mp_mixpanel__c": {},
"JSESSIONID": {},
"NREUM": {},
"_pk_id": {},
"_pk_ref": {},
"_pk_ses": {},
"_pk_cvar": {},
"_pk_hsr": {},
"_hjIncludedInSample": {},
"__cfduid": {},
"cf_use_ob": {},
"cf_ob_info": {},
"intercom-session": {},
"optimizelyEndUserId": {},
"optimizelySegments": {},
"optimizelyBuckets": {},
"optimizelyPendingLogEvents": {},
"YSC": {},
"VISITOR_INFO1_LIVE": {},
"PREF": {},
"GPS": {},
}

57
pkg/fuzz/component/cookie_test.go Normal file
View File

@ -0,0 +1,57 @@
package component
import (
"net/http"
"testing"
"github.com/projectdiscovery/retryablehttp-go"
"github.com/stretchr/testify/require"
)
func TestCookieComponent(t *testing.T) {
req, err := retryablehttp.NewRequest(http.MethodGet, "https://example.com", nil)
if err != nil {
t.Fatal(err)
}
cookie := &http.Cookie{
Name: "session",
Value: "test-session",
}
req.AddCookie(cookie)
cookieComponent := NewCookie() // Assuming you have a function like this for creating a new cookie component
_, err = cookieComponent.Parse(req)
if err != nil {
t.Fatal(err)
}
var cookieNames []string
var cookieValues []string
_ = cookieComponent.Iterate(func(key string, value interface{}) error {
cookieNames = append(cookieNames, key)
switch v := value.(type) {
case string:
cookieValues = append(cookieValues, v)
case []string:
cookieValues = append(cookieValues, v...)
}
return nil
})
require.Equal(t, []string{"session"}, cookieNames, "unexpected cookie names")
require.Equal(t, []string{"test-session"}, cookieValues, "unexpected cookie values")
err = cookieComponent.SetValue("session", "new-session")
if err != nil {
t.Fatal(err)
}
rebuilt, err := cookieComponent.Rebuild()
if err != nil {
t.Fatal(err)
}
// Assuming the Rebuild function will reconstruct the entire request and also set the modified cookies
newCookie, _ := rebuilt.Cookie("session")
require.Equal(t, "new-session", newCookie.Value, "unexpected cookie value")
}

186
pkg/fuzz/component/headers.go Normal file
View File

@ -0,0 +1,186 @@
package component
import (
"context"
"strings"
"github.com/projectdiscovery/retryablehttp-go"
)
// Header is a component for a request header
type Header struct {
value *Value
req *retryablehttp.Request
}
var _ Component = &Header{}
// NewHeader creates a new header component
func NewHeader() *Header {
return &Header{}
}
// Name returns the name of the component
func (q *Header) Name() string {
return RequestHeaderComponent
}
// Parse parses the component and returns the
// parsed component
func (q *Header) Parse(req *retryablehttp.Request) (bool, error) {
q.req = req
q.value = NewValue("")
parsedHeaders := make(map[string]interface{})
for key, value := range req.Header {
if len(value) == 1 {
parsedHeaders[key] = value[0]
continue
}
parsedHeaders[key] = value
}
q.value.SetParsed(parsedHeaders, "")
return true, nil
}
// Iterate iterates through the component
func (q *Header) Iterate(callback func(key string, value interface{}) error) error {
for key, value := range q.value.Parsed() {
// Skip ignored headers
if _, ok := defaultIgnoredHeaderKeys[key]; ok {
continue
}
if err := callback(key, value); err != nil {
return err
}
}
return nil
}
// SetValue sets a value in the component
// for a key
func (q *Header) SetValue(key string, value string) error {
if !q.value.SetParsedValue(key, value) {
return ErrSetValue
}
return nil
}
// Delete deletes a key from the component
func (q *Header) Delete(key string) error {
if !q.value.Delete(key) {
return ErrKeyNotFound
}
return nil
}
// Rebuild returns a new request with the
// component rebuilt
func (q *Header) Rebuild() (*retryablehttp.Request, error) {
cloned := q.req.Clone(context.Background())
for key, value := range q.value.parsed {
if strings.EqualFold(key, "Host") {
cloned.Host = value.(string)
}
switch v := value.(type) {
case []interface{}:
for _, vv := range v {
if cloned.Header[key] == nil {
cloned.Header[key] = make([]string, 0)
}
cloned.Header[key] = append(cloned.Header[key], vv.(string))
}
case string:
cloned.Header[key] = []string{v}
}
}
return cloned, nil
}
// A list of headers that are essential to the request and
// must not be fuzzed.
var defaultIgnoredHeaderKeys = map[string]struct{}{
"Accept-Charset": {},
"Accept-Datetime": {},
"Accept-Encoding": {},
"Accept-Language": {},
"Accept": {},
"Access-Control-Request-Headers": {},
"Access-Control-Request-Method": {},
"Authorization": {},
"Cache-Control": {},
"Connection": {},
"Cookie": {},
"Content-Length": {},
"Content-Type": {},
"Date": {},
"Dnt": {},
"Expect": {},
"Forwarded": {},
"From": {},
"Host": {},
"If-Match": {},
"If-Modified-Since": {},
"If-None-Match": {},
"If-Range": {},
"If-Unmodified-Since": {},
"Max-Forwards": {},
"Pragma": {},
"Priority": {},
"Proxy-Authorization": {},
"Range": {},
"Sec-Ch-Ua": {},
"Sec-Ch-Ua-Mobile": {},
"Sec-Ch-Ua-Platform": {},
"Sec-Fetch-Dest": {},
"Sec-Fetch-Mode": {},
"Sec-Fetch-Site": {},
"Sec-Fetch-User": {},
"TE": {},
"Upgrade": {},
"Via": {},
"Warning": {},
"Upgrade-Insecure-Requests": {},
"X-CSRF-Token": {},
"X-Requested-With": {},
"Strict-Transport-Security": {},
"Content-Security-Policy": {},
"X-Content-Type-Options": {},
"X-Frame-Options": {},
"X-XSS-Protection": {},
"Public-Key-Pins": {},
"Referrer-Policy": {},
"Access-Control-Allow-Origin": {},
"Access-Control-Allow-Credentials": {},
"Access-Control-Expose-Headers": {},
"Access-Control-Max-Age": {},
"Access-Control-Allow-Methods": {},
"Access-Control-Allow-Headers": {},
"Server": {},
"X-Powered-By": {},
"X-AspNet-Version": {},
"X-AspNetMvc-Version": {},
"ETag": {},
"Vary": {},
"Expires": {},
"Last-Modified": {},
"X-Cache": {},
"X-Proxy-ID": {},
"CF-Ray": {}, // Cloudflare
"X-Served-By": {}, // Varnish, etc.
"X-Cache-Hits": {},
"Content-Encoding": {},
"Transfer-Encoding": {},
"Location": {},
"WWW-Authenticate": {},
"Proxy-Authenticate": {},
"X-Access-Token": {},
"X-Refresh-Token": {},
"Link": {},
"X-Content-Duration": {},
"X-UA-Compatible": {},
"X-RateLimit-Limit": {}, // Rate limiting header
"X-RateLimit-Remaining": {}, // Rate limiting header
"X-RateLimit-Reset": {}, // Rate limiting header
}

51
pkg/fuzz/component/headers_test.go Normal file
View File

@ -0,0 +1,51 @@
package component
import (
"net/http"
"testing"
"github.com/projectdiscovery/retryablehttp-go"
"github.com/stretchr/testify/require"
)
func TestHeaderComponent(t *testing.T) {
req, err := retryablehttp.NewRequest(http.MethodGet, "https://example.com", nil)
if err != nil {
t.Fatal(err)
}
req.Header.Set("User-Agent", "test-agent")
header := NewHeader()
_, err = header.Parse(req)
if err != nil {
t.Fatal(err)
}
var keys []string
var values []string
_ = header.Iterate(func(key string, value interface{}) error {
keys = append(keys, key)
switch v := value.(type) {
case string:
values = append(values, v)
case []string:
values = append(values, v...)
}
return nil
})
require.Equal(t, []string{"User-Agent"}, keys, "unexpected keys")
require.Equal(t, []string{"test-agent"}, values, "unexpected values")
err = header.SetValue("User-Agent", "new-agent")
if err != nil {
t.Fatal(err)
}
rebuilt, err := header.Rebuild()
if err != nil {
t.Fatal(err)
}
require.Equal(t, "new-agent", rebuilt.Header.Get("User-Agent"), "unexpected header value")
}

83
pkg/fuzz/component/path.go Normal file
View File

@ -0,0 +1,83 @@
package component
import (
"context"
"github.com/pkg/errors"
"github.com/projectdiscovery/nuclei/v3/pkg/fuzz/dataformat"
"github.com/projectdiscovery/retryablehttp-go"
)
// Path is a component for a request Path
type Path struct {
value *Value
req *retryablehttp.Request
}
var _ Component = &Path{}
// NewPath creates a new URL component
func NewPath() *Path {
return &Path{}
}
// Name returns the name of the component
func (q *Path) Name() string {
return RequestPathComponent
}
// Parse parses the component and returns the
// parsed component
func (q *Path) Parse(req *retryablehttp.Request) (bool, error) {
q.req = req
q.value = NewValue(req.URL.Path)
parsed, err := dataformat.Get(dataformat.RawDataFormat).Decode(q.value.String())
if err != nil {
return false, err
}
q.value.SetParsed(parsed, dataformat.RawDataFormat)
return true, nil
}
// Iterate iterates through the component
func (q *Path) Iterate(callback func(key string, value interface{}) error) error {
for key, value := range q.value.Parsed() {
if err := callback(key, value); err != nil {
return err
}
}
return nil
}
// SetValue sets a value in the component
// for a key
func (q *Path) SetValue(key string, value string) error {
if !q.value.SetParsedValue(key, value) {
return ErrSetValue
}
return nil
}
// Delete deletes a key from the component
func (q *Path) Delete(key string) error {
if !q.value.Delete(key) {
return ErrKeyNotFound
}
return nil
}
// Rebuild returns a new request with the
// component rebuilt
func (q *Path) Rebuild() (*retryablehttp.Request, error) {
encoded, err := q.value.Encode()
if err != nil {
return nil, errors.Wrap(err, "could not encode query")
}
cloned := q.req.Clone(context.Background())
if err := cloned.UpdateRelPath(encoded, true); err != nil {
cloned.URL.RawPath = encoded
}
return cloned, nil
}

46
pkg/fuzz/component/path_test.go Normal file
View File

@ -0,0 +1,46 @@
package component
import (
"net/http"
"testing"
"github.com/projectdiscovery/retryablehttp-go"
"github.com/stretchr/testify/require"
)
func TestURLComponent(t *testing.T) {
req, err := retryablehttp.NewRequest(http.MethodGet, "https://example.com/testpath", nil)
if err != nil {
t.Fatal(err)
}
urlComponent := NewPath()
_, err = urlComponent.Parse(req)
if err != nil {
t.Fatal(err)
}
var keys []string
var values []string
_ = urlComponent.Iterate(func(key string, value interface{}) error {
keys = append(keys, key)
values = append(values, value.(string))
return nil
})
require.Equal(t, []string{"value"}, keys, "unexpected keys")
require.Equal(t, []string{"/testpath"}, values, "unexpected values")
err = urlComponent.SetValue("value", "/newpath")
if err != nil {
t.Fatal(err)
}
rebuilt, err := urlComponent.Rebuild()
if err != nil {
t.Fatal(err)
}
require.Equal(t, "/newpath", rebuilt.URL.Path, "unexpected URL path")
require.Equal(t, "https://example.com/newpath", rebuilt.URL.String(), "unexpected full URL")
}

92
pkg/fuzz/component/query.go Normal file
View File

@ -0,0 +1,92 @@
package component
import (
"context"
"github.com/pkg/errors"
"github.com/projectdiscovery/nuclei/v3/pkg/fuzz/dataformat"
"github.com/projectdiscovery/retryablehttp-go"
urlutil "github.com/projectdiscovery/utils/url"
)
// Query is a component for a request query
type Query struct {
value *Value
req *retryablehttp.Request
}
var _ Component = &Query{}
// NewQuery creates a new query component
func NewQuery() *Query {
return &Query{}
}
// Name returns the name of the component
func (q *Query) Name() string {
return RequestQueryComponent
}
// Parse parses the component and returns the
// parsed component
func (q *Query) Parse(req *retryablehttp.Request) (bool, error) {
if req.URL.Query().IsEmpty() {
return false, nil
}
q.req = req
q.value = NewValue(req.URL.Query().Encode())
parsed, err := dataformat.Get(dataformat.FormDataFormat).Decode(q.value.String())
if err != nil {
return false, err
}
q.value.SetParsed(parsed, dataformat.FormDataFormat)
return true, nil
}
// Iterate iterates through the component
func (q *Query) Iterate(callback func(key string, value interface{}) error) error {
for key, value := range q.value.Parsed() {
if err := callback(key, value); err != nil {
return err
}
}
return nil
}
// SetValue sets a value in the component
// for a key
func (q *Query) SetValue(key string, value string) error {
if !q.value.SetParsedValue(key, value) {
return ErrSetValue
}
return nil
}
// Delete deletes a key from the component
func (q *Query) Delete(key string) error {
if !q.value.Delete(key) {
return ErrKeyNotFound
}
return nil
}
// Rebuild returns a new request with the
// component rebuilt
func (q *Query) Rebuild() (*retryablehttp.Request, error) {
encoded, err := q.value.Encode()
if err != nil {
return nil, errors.Wrap(err, "could not encode query")
}
cloned := q.req.Clone(context.Background())
cloned.URL.RawQuery = encoded
// Clear the query parameters and re-add them
cloned.Params = nil
cloned.Params = urlutil.NewOrderedParams()
cloned.Params.Decode(encoded)
cloned.Update()
return cloned, nil
}

46
pkg/fuzz/component/query_test.go Normal file
View File

@ -0,0 +1,46 @@
package component
import (
"net/http"
"testing"
"github.com/projectdiscovery/retryablehttp-go"
"github.com/stretchr/testify/require"
)
func TestQueryComponent(t *testing.T) {
req, err := retryablehttp.NewRequest(http.MethodGet, "https://example.com?foo=bar", nil)
if err != nil {
t.Fatal(err)
}
query := NewQuery()
_, err = query.Parse(req)
if err != nil {
t.Fatal(err)
}
var keys []string
var values []string
_ = query.Iterate(func(key string, value interface{}) error {
keys = append(keys, key)
values = append(values, value.(string))
return nil
})
require.Equal(t, []string{"foo"}, keys, "unexpected keys")
require.Equal(t, []string{"bar"}, values, "unexpected values")
err = query.SetValue("foo", "baz")
if err != nil {
t.Fatal(err)
}
rebuilt, err := query.Rebuild()
if err != nil {
t.Fatal(err)
}
require.Equal(t, "foo=baz", rebuilt.URL.RawQuery, "unexpected query string")
require.Equal(t, "https://example.com?foo=baz", rebuilt.URL.String(), "unexpected url")
}

120
pkg/fuzz/component/value.go Normal file
View File

@ -0,0 +1,120 @@
package component
import (
"strconv"
"github.com/leslie-qiwa/flat"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v3/pkg/fuzz/dataformat"
)
// Value is a value component containing a single
// parameter for the component
//
// It is a type of container that is used to represent
// all the data values that are used in a request.
type Value struct {
data string
parsed map[string]interface{}
dataFormat string
}
// NewValue returns a new value component
func NewValue(data string) *Value {
if data == "" {
return &Value{}
}
v := &Value{data: data}
// Do any dataformat decoding on the data if needed
decodedDataformat, err := dataformat.Decode(data)
if err == nil && decodedDataformat != nil {
v.SetParsed(decodedDataformat.Data, decodedDataformat.DataFormat)
}
return v
}
// String returns the string representation of the value
func (v *Value) String() string {
return v.data
}
// Parsed returns the parsed value
func (v *Value) Parsed() map[string]interface{} {
return v.parsed
}
// SetParsed sets the parsed value map
func (v *Value) SetParsed(parsed map[string]interface{}, dataFormat string) {
flattened, err := flat.Flatten(parsed, flatOpts)
if err == nil {
v.parsed = flattened
} else {
v.parsed = parsed
}
v.dataFormat = dataFormat
}
// SetParsedValue sets the parsed value for a key
// in the parsed map
func (v *Value) SetParsedValue(key string, value string) bool {
origValue, ok := v.parsed[key]
if !ok {
v.parsed[key] = value
return true
}
// If the value is a list, append to it
// otherwise replace it
switch v := origValue.(type) {
case []interface{}:
origValue = append(v, value)
case string:
origValue = value
case int, int32, int64, float32, float64:
parsed, err := strconv.ParseInt(value, 10, 64)
if err != nil {
return false
}
origValue = parsed
case bool:
parsed, err := strconv.ParseBool(value)
if err != nil {
return false
}
origValue = parsed
case nil:
origValue = value
default:
gologger.Error().Msgf("unknown type %T for value %s", v, v)
}
v.parsed[key] = origValue
return true
}
// Delete removes a key from the parsed value
func (v *Value) Delete(key string) bool {
if _, ok := v.parsed[key]; !ok {
return false
}
delete(v.parsed, key)
return true
}
// Encode encodes the value into a string
// using the dataformat and encoding
func (v *Value) Encode() (string, error) {
toEncodeStr := v.data
nested, err := flat.Unflatten(v.parsed, flatOpts)
if err != nil {
return "", err
}
if v.dataFormat != "" {
dataformatStr, err := dataformat.Encode(nested, v.dataFormat)
if err != nil {
return "", err
}
toEncodeStr = dataformatStr
}
return toEncodeStr, nil
}

39
pkg/fuzz/component/value_test.go Normal file
View File

@ -0,0 +1,39 @@
package component
import (
"testing"
"github.com/leslie-qiwa/flat"
"github.com/stretchr/testify/require"
)
func TestFlatMap_FlattenUnflatten(t *testing.T) {
data := map[string]interface{}{
"foo": "bar",
"bar": map[string]interface{}{
"baz": "foo",
},
"slice": []interface{}{
"foo",
"bar",
},
"with.dot": map[string]interface{}{
"foo": "bar",
},
}
opts := &flat.Options{
Safe: true,
Delimiter: "~",
}
flattened, err := flat.Flatten(data, opts)
if err != nil {
t.Fatal(err)
}
nested, err := flat.Unflatten(flattened, opts)
if err != nil {
t.Fatal(err)
}
require.Equal(t, data, nested, "unexpected data")
}

92
pkg/fuzz/dataformat/dataformat.go Normal file
View File

@ -0,0 +1,92 @@
package dataformat
import (
"errors"
"fmt"
)
// dataformats is a list of dataformats
var dataformats map[string]DataFormat
func init() {
dataformats = make(map[string]DataFormat)
// register the default data formats
RegisterDataFormat(NewJSON())
RegisterDataFormat(NewXML())
RegisterDataFormat(NewRaw())
RegisterDataFormat(NewForm())
RegisterDataFormat(NewMultiPartForm())
}
const (
// JSONDataFormat is the name of the JSON data format
JSONDataFormat = "json"
// XMLDataFormat is the name of the XML data format
XMLDataFormat = "xml"
// RawDataFormat is the name of the Raw data format
RawDataFormat = "raw"
// FormDataFormat is the name of the Form data format
FormDataFormat = "form"
// MultiPartFormDataFormat is the name of the MultiPartForm data format
MultiPartFormDataFormat = "multipart/form-data"
)
// Get returns the dataformat by name
func Get(name string) DataFormat {
return dataformats[name]
}
// RegisterEncoder registers an encoder
func RegisterDataFormat(dataformat DataFormat) {
dataformats[dataformat.Name()] = dataformat
}
// DataFormat is an interface for encoding and decoding
type DataFormat interface {
// IsType returns true if the data is of the type
IsType(data string) bool
// Name returns the name of the encoder
Name() string
// Encode encodes the data into a format
Encode(data map[string]interface{}) (string, error)
// Decode decodes the data from a format
Decode(input string) (map[string]interface{}, error)
}
// Decoded is a decoded data format
type Decoded struct {
// DataFormat is the data format
DataFormat string
// Data is the decoded data
Data map[string]interface{}
}
// Decode decodes the data from a format
func Decode(data string) (*Decoded, error) {
for _, dataformat := range dataformats {
if dataformat.IsType(data) {
decoded, err := dataformat.Decode(data)
if err != nil {
return nil, err
}
value := &Decoded{
DataFormat: dataformat.Name(),
Data: decoded,
}
return value, nil
}
}
return nil, nil
}
// Encode encodes the data into a format
func Encode(data map[string]interface{}, dataformat string) (string, error) {
if dataformat == "" {
return "", errors.New("dataformat is required")
}
if encoder, ok := dataformats[dataformat]; ok {
return encoder.Encode(data)
}
return "", fmt.Errorf("dataformat %s is not supported", dataformat)
}

54
pkg/fuzz/dataformat/dataformat_test.go Normal file
View File

@ -0,0 +1,54 @@
package dataformat
import (
"testing"
)
func TestDataformatDecodeEncode_JSON(t *testing.T) {
obj := `{"foo":"bar"}`
decoded, err := Decode(obj)
if err != nil {
t.Fatal(err)
}
if decoded.DataFormat != "json" {
t.Fatal("unexpected data format")
}
if decoded.Data["foo"] != "bar" {
t.Fatal("unexpected data")
}
encoded, err := Encode(decoded.Data, decoded.DataFormat)
if err != nil {
t.Fatal(err)
}
if encoded != obj {
t.Fatal("unexpected data")
}
}
func TestDataformatDecodeEncode_XML(t *testing.T) {
obj := `<foo attr="baz">bar</foo>`
decoded, err := Decode(obj)
if err != nil {
t.Fatal(err)
}
if decoded.DataFormat != "xml" {
t.Fatal("unexpected data format")
}
if decoded.Data["foo"].(map[string]interface{})["#text"] != "bar" {
t.Fatal("unexpected data")
}
if decoded.Data["foo"].(map[string]interface{})["-attr"] != "baz" {
t.Fatal("unexpected data")
}
encoded, err := Encode(decoded.Data, decoded.DataFormat)
if err != nil {
t.Fatal(err)
}
if encoded != obj {
t.Fatal("unexpected data")
}
}

61
pkg/fuzz/dataformat/form.go Normal file
View File

@ -0,0 +1,61 @@
package dataformat
import (
"net/url"
)
type Form struct{}
var (
_ DataFormat = &Form{}
)
// NewForm returns a new Form encoder
func NewForm() *Form {
return &Form{}
}
// IsType returns true if the data is Form encoded
func (f *Form) IsType(data string) bool {
return false
}
// Encode encodes the data into Form format
func (f *Form) Encode(data map[string]interface{}) (string, error) {
query := url.Values{}
for key, value := range data {
switch v := value.(type) {
case []interface{}:
for _, val := range v {
query.Add(key, val.(string))
}
case string:
query.Set(key, v)
}
}
encoded := query.Encode()
return encoded, nil
}
// Decode decodes the data from Form format
func (f *Form) Decode(data string) (map[string]interface{}, error) {
parsed, err := url.ParseQuery(data)
if err != nil {
return nil, err
}
values := make(map[string]interface{})
for key, value := range parsed {
if len(value) == 1 {
values[key] = value[0]
} else {
values[key] = value
}
}
return values, nil
}
// Name returns the name of the encoder
func (f *Form) Name() string {
return FormDataFormat
}

48
pkg/fuzz/dataformat/json.go Normal file
View File

@ -0,0 +1,48 @@
package dataformat
import (
"strings"
jsoniter "github.com/json-iterator/go"
)
// JSON is a JSON encoder
//
// For now JSON only supports objects as the root data type
// and not arrays
//
// TODO: Support arrays + other JSON oddities by
// adding more attirbutes to the map[string]interface{}
type JSON struct{}
var (
_ DataFormat = &JSON{}
)
// NewJSON returns a new JSON encoder
func NewJSON() *JSON {
return &JSON{}
}
// IsType returns true if the data is JSON encoded
func (j *JSON) IsType(data string) bool {
return strings.HasPrefix(data, "{") && strings.HasSuffix(data, "}")
}
// Encode encodes the data into JSON format
func (j *JSON) Encode(data map[string]interface{}) (string, error) {
encoded, err := jsoniter.Marshal(data)
return string(encoded), err
}
// Decode decodes the data from JSON format
func (j *JSON) Decode(data string) (map[string]interface{}, error) {
var decoded map[string]interface{}
err := jsoniter.Unmarshal([]byte(data), &decoded)
return decoded, err
}
// Name returns the name of the encoder
func (j *JSON) Name() string {
return JSONDataFormat
}

116
pkg/fuzz/dataformat/multipart.go Normal file
View File

@ -0,0 +1,116 @@
package dataformat
import (
"bytes"
"fmt"
"io"
"mime"
"mime/multipart"
)
type MultiPartForm struct {
boundary string
}
var (
_ DataFormat = &MultiPartForm{}
)
// NewMultiPartForm returns a new MultiPartForm encoder
func NewMultiPartForm() *MultiPartForm {
return &MultiPartForm{}
}
// IsType returns true if the data is MultiPartForm encoded
func (m *MultiPartForm) IsType(data string) bool {
// This method should be implemented to detect if the data is multipart form encoded
return false
}
// Encode encodes the data into MultiPartForm format
func (m *MultiPartForm) Encode(data map[string]interface{}) (string, error) {
var b bytes.Buffer
w := multipart.NewWriter(&b)
if err := w.SetBoundary(m.boundary); err != nil {
return "", err
}
for key, value := range data {
var fw io.Writer
var err error
// Add field
if fw, err = w.CreateFormField(key); err != nil {
return "", err
}
if _, err = fw.Write([]byte(value.(string))); err != nil {
return "", err
}
}
w.Close()
return b.String(), nil
}
// ParseBoundary parses the boundary from the content type
func (m *MultiPartForm) ParseBoundary(contentType string) error {
_, params, err := mime.ParseMediaType(contentType)
if err != nil {
return err
}
m.boundary = params["boundary"]
if m.boundary == "" {
return fmt.Errorf("no boundary found in the content type")
}
return nil
}
// Decode decodes the data from MultiPartForm format
func (m *MultiPartForm) Decode(data string) (map[string]interface{}, error) {
// Create a buffer from the string data
b := bytes.NewBufferString(data)
// The boundary parameter should be extracted from the Content-Type header of the HTTP request
// which is not available in this context, so this is a placeholder for demonstration.
// You will need to pass the actual boundary value to this function.
r := multipart.NewReader(b, m.boundary)
form, err := r.ReadForm(32 << 20) // 32MB is the max memory used to parse the form
if err != nil {
return nil, err
}
defer func() {
_ = form.RemoveAll()
}()
result := make(map[string]interface{})
for key, values := range form.Value {
if len(values) > 1 {
result[key] = values
} else {
result[key] = values[0]
}
}
for key, files := range form.File {
fileContents := []interface{}{}
for _, fileHeader := range files {
file, err := fileHeader.Open()
if err != nil {
return nil, err
}
defer file.Close()
buffer := new(bytes.Buffer)
if _, err := buffer.ReadFrom(file); err != nil {
return nil, err
}
fileContents = append(fileContents, buffer.String())
}
result[key] = fileContents
}
return result, nil
}
// Name returns the name of the encoder
func (m *MultiPartForm) Name() string {
return "multipart/form-data"
}

34
pkg/fuzz/dataformat/raw.go Normal file
View File

@ -0,0 +1,34 @@
package dataformat
type Raw struct{}
var (
_ DataFormat = &Raw{}
)
// NewRaw returns a new Raw encoder
func NewRaw() *Raw {
return &Raw{}
}
// IsType returns true if the data is Raw encoded
func (r *Raw) IsType(data string) bool {
return false
}
// Encode encodes the data into Raw format
func (r *Raw) Encode(data map[string]interface{}) (string, error) {
return data["value"].(string), nil
}
// Decode decodes the data from Raw format
func (r *Raw) Decode(data string) (map[string]interface{}, error) {
return map[string]interface{}{
"value": data,
}, nil
}
// Name returns the name of the encoder
func (r *Raw) Name() string {
return RawDataFormat
}

62
pkg/fuzz/dataformat/xml.go Normal file
View File

@ -0,0 +1,62 @@
package dataformat
import (
"fmt"
"regexp"
"strings"
"github.com/clbanning/mxj/v2"
)
// XML is an XML encoder
type XML struct{}
// NewXML returns a new XML encoder
func NewXML() *XML {
return &XML{}
}
// IsType returns true if the data is XML encoded
func (x *XML) IsType(data string) bool {
return strings.HasPrefix(data, "<") && strings.HasSuffix(data, ">")
}
// Encode encodes the data into XML format
func (x *XML) Encode(data map[string]interface{}) (string, error) {
var header string
if value, ok := data["#_xml_header"]; ok && value != nil {
header = value.(string)
delete(data, "#_xml_header")
}
marshalled, err := mxj.Map(data).Xml()
if err != nil {
return "", err
}
if header != "" {
return fmt.Sprintf("<?%s?>%s", header, string(marshalled)), nil
}
return string(marshalled), err
}
var xmlHeader = regexp.MustCompile(`\<\?(.*)\?\>`)
// Decode decodes the data from XML format
func (x *XML) Decode(data string) (map[string]interface{}, error) {
var prefixStr string
prefix := xmlHeader.FindAllStringSubmatch(data, -1)
if len(prefix) > 0 {
prefixStr = prefix[0][1]
}
decoded, err := mxj.NewMapXml([]byte(data))
if err != nil {
return nil, err
}
decoded["#_xml_header"] = prefixStr
return decoded, nil
}
// Name returns the name of the encoder
func (x *XML) Name() string {
return XMLDataFormat
}

0
pkg/protocols/common/fuzz/doc.go → pkg/fuzz/doc.go
View File

275
pkg/fuzz/execute.go Normal file
View File

@ -0,0 +1,275 @@
package fuzz
import (
"fmt"
"io"
"regexp"
"strings"
"github.com/pkg/errors"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v3/pkg/fuzz/component"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
"github.com/projectdiscovery/retryablehttp-go"
errorutil "github.com/projectdiscovery/utils/errors"
urlutil "github.com/projectdiscovery/utils/url"
)
var (
ErrRuleNotApplicable = errorutil.NewWithFmt("rule not applicable : %v")
)
// IsErrRuleNotApplicable checks if an error is due to rule not applicable
func IsErrRuleNotApplicable(err error) bool {
if err == nil {
return false
}
if strings.Contains(err.Error(), "rule not applicable") {
return true
}
return false
}
// ExecuteRuleInput is the input for rule Execute function
type ExecuteRuleInput struct {
// Input is the context args input
Input *contextargs.Context
// Callback is the callback for generated rule requests
Callback func(GeneratedRequest) bool
// InteractURLs contains interact urls for execute call
InteractURLs []string
// Values contains dynamic values for the rule
Values map[string]interface{}
// BaseRequest is the base http request for fuzzing rule
BaseRequest *retryablehttp.Request
}
// GeneratedRequest is a single generated request for rule
type GeneratedRequest struct {
// Request is the http request for rule
Request *retryablehttp.Request
// InteractURLs is the list of interactsh urls
InteractURLs []string
// DynamicValues contains dynamic values map
DynamicValues map[string]interface{}
// Component is the component for the request
Component component.Component
}
// Execute executes a fuzzing rule accepting a callback on which
// generated requests are returned.
//
// Input is not thread safe and should not be shared between concurrent
// goroutines.
func (rule *Rule) Execute(input *ExecuteRuleInput) (err error) {
defer func() {
if r := recover(); r != nil {
err = fmt.Errorf("got panic while executing rule: %v", r)
}
}()
if !rule.isInputURLValid(input.Input) {
return ErrRuleNotApplicable.Msgf("invalid input url: %v", input.Input.MetaInput.Input)
}
if input.BaseRequest == nil && input.Input.MetaInput.ReqResp == nil {
return ErrRuleNotApplicable.Msgf("both base request and reqresp are nil for %v", input.Input.MetaInput.Input)
}
var finalComponentList []component.Component
// match rule part with component name
for _, componentName := range component.Components {
if rule.partType != requestPartType && rule.Part != componentName {
continue
}
component := component.New(componentName)
discovered, err := component.Parse(input.BaseRequest)
if err != nil {
gologger.Verbose().Msgf("Could not parse component %s: %s\n", componentName, err)
continue
}
if !discovered {
continue
}
// check rule applicable on this component
if !rule.checkRuleApplicableOnComponent(component) {
continue
}
finalComponentList = append(finalComponentList, component)
}
if len(finalComponentList) == 0 {
return ErrRuleNotApplicable.Msgf("no component matched on this rule")
}
baseValues := input.Values
if rule.generator == nil {
for _, component := range finalComponentList {
evaluatedValues, interactURLs := rule.options.Variables.EvaluateWithInteractsh(baseValues, rule.options.Interactsh)
input.Values = generators.MergeMaps(evaluatedValues, baseValues, rule.options.Constants)
input.InteractURLs = interactURLs
err := rule.executeRuleValues(input, component)
if err != nil {
return err
}
}
return nil
}
mainLoop:
for _, component := range finalComponentList {
iterator := rule.generator.NewIterator()
for {
values, next := iterator.Value()
if !next {
continue mainLoop
}
evaluatedValues, interactURLs := rule.options.Variables.EvaluateWithInteractsh(generators.MergeMaps(values, baseValues), rule.options.Interactsh)
input.InteractURLs = interactURLs
input.Values = generators.MergeMaps(values, evaluatedValues, baseValues, rule.options.Constants)
if err := rule.executeRuleValues(input, component); err != nil {
if err == io.EOF {
return nil
}
gologger.Warning().Msgf("[%s] Could not execute rule: %s\n", rule.options.TemplateID, err)
return err
}
}
}
return nil
}
// isInputURLValid returns true if url is valid after parsing it
func (rule *Rule) isInputURLValid(input *contextargs.Context) bool {
if input == nil || input.MetaInput == nil || input.MetaInput.Input == "" {
return false
}
_, err := urlutil.Parse(input.MetaInput.Input)
return err == nil
}
// executeRuleValues executes a rule with a set of values
func (rule *Rule) executeRuleValues(input *ExecuteRuleInput, ruleComponent component.Component) error {
// if we are only fuzzing values
if len(rule.Fuzz.Value) > 0 {
for _, value := range rule.Fuzz.Value {
if err := rule.executePartRule(input, ValueOrKeyValue{Value: value}, ruleComponent); err != nil {
if component.IsErrSetValue(err) {
// this are errors due to format restrictions
// ex: fuzzing string value in a json int field
continue
}
return err
}
}
return nil
}
// if we are fuzzing both keys and values
if rule.Fuzz.KV != nil {
var gotErr error
rule.Fuzz.KV.Iterate(func(key, value string) bool {
if err := rule.executePartRule(input, ValueOrKeyValue{Key: key, Value: value}, ruleComponent); err != nil {
if component.IsErrSetValue(err) {
// this are errors due to format restrictions
// ex: fuzzing string value in a json int field
return true
}
gotErr = err
return false
}
return true
})
// if mode is multiple now build and execute it
if rule.modeType == multipleModeType {
req, err := ruleComponent.Rebuild()
if err != nil {
return err
}
if gotErr := rule.execWithInput(input, req, input.InteractURLs, ruleComponent); gotErr != nil {
return gotErr
}
}
return gotErr
}
// something else is wrong
return fmt.Errorf("no fuzz values specified")
}
// Compile compiles a fuzzing rule and initializes it for operation
func (rule *Rule) Compile(generator *generators.PayloadGenerator, options *protocols.ExecutorOptions) error {
// If a payload generator is specified from base request, use it
// for payload values.
if generator != nil {
rule.generator = generator
}
rule.options = options
// Resolve the default enums
if rule.Mode != "" {
if valueType, ok := stringToModeType[rule.Mode]; !ok {
return errors.Errorf("invalid mode value specified: %s", rule.Mode)
} else {
rule.modeType = valueType
}
} else {
rule.modeType = multipleModeType
}
if rule.Part != "" {
if valueType, ok := stringToPartType[rule.Part]; !ok {
return errors.Errorf("invalid part value specified: %s", rule.Part)
} else {
rule.partType = valueType
}
} else {
rule.partType = queryPartType
}
if rule.Type != "" {
if valueType, ok := stringToRuleType[rule.Type]; !ok {
return errors.Errorf("invalid type value specified: %s", rule.Type)
} else {
rule.ruleType = valueType
}
} else {
rule.ruleType = replaceRuleType
}
// Initialize other required regexes and maps
if len(rule.Keys) > 0 {
rule.keysMap = make(map[string]struct{})
}
for _, key := range rule.Keys {
rule.keysMap[strings.ToLower(key)] = struct{}{}
}
for _, value := range rule.ValuesRegex {
compiled, err := regexp.Compile(value)
if err != nil {
return errors.Wrap(err, "could not compile value regex")
}
rule.valuesRegex = append(rule.valuesRegex, compiled)
}
for _, value := range rule.KeysRegex {
compiled, err := regexp.Compile(value)
if err != nil {
return errors.Wrap(err, "could not compile key regex")
}
rule.keysRegex = append(rule.keysRegex, compiled)
}
if rule.ruleType != replaceRegexRuleType {
if rule.ReplaceRegex != "" {
return errors.Errorf("replace-regex is only applicable for replace and replace-regex rule types")
}
} else {
if rule.ReplaceRegex == "" {
return errors.Errorf("replace-regex is required for replace-regex rule type")
}
compiled, err := regexp.Compile(rule.ReplaceRegex)
if err != nil {
return errors.Wrap(err, "could not compile replace regex")
}
rule.replaceRegex = compiled
}
return nil
}

42
pkg/protocols/common/fuzz/fuzz.go → pkg/fuzz/fuzz.go
View File

@ -20,7 +20,7 @@ type Rule struct {
// - "prefix"
// - "postfix"
// - "infix"
Type string `yaml:"type,omitempty" json:"type,omitempty" jsonschema:"title=type of rule,description=Type of fuzzing rule to perform,enum=replace,enum=prefix,enum=postfix,enum=infix"`
Type string `yaml:"type,omitempty" json:"type,omitempty" jsonschema:"title=type of rule,description=Type of fuzzing rule to perform,enum=replace,enum=prefix,enum=postfix,enum=infix,enum=replace-regex"`
ruleType ruleType
// description: |
// Part is the part of request to fuzz.
@ -28,7 +28,7 @@ type Rule struct {
// query fuzzes the query part of url. More parts will be added later.
// values:
// - "query"
Part string `yaml:"part,omitempty" json:"part,omitempty" jsonschema:"title=part of rule,description=Part of request rule to fuzz,enum=query"`
Part string `yaml:"part,omitempty" json:"part,omitempty" jsonschema:"title=part of rule,description=Part of request rule to fuzz,enum=query,enum=header,enum=path,enum=body,enum=cookie,enum=request"`
partType partType
// description: |
// Mode is the mode of fuzzing to perform.
@ -71,10 +71,20 @@ type Rule struct {
// - name: Examples of fuzz
// value: >
// []string{"{{ssrf}}", "{{interactsh-url}}", "example-value"}
Fuzz []string `yaml:"fuzz,omitempty" json:"fuzz,omitempty" jsonschema:"title=payloads of fuzz rule,description=Payloads to perform fuzzing substitutions with"`
options *protocols.ExecutorOptions
generator *generators.PayloadGenerator
// or
// x-header: 1
// x-header: 2
Fuzz SliceOrMapSlice `yaml:"fuzz,omitempty" json:"fuzz,omitempty" jsonschema:"title=payloads of fuzz rule,description=Payloads to perform fuzzing substitutions with"`
// description: |
// replace-regex is regex for regex-replace rule type
// it is only required for replace-regex rule type
// examples:
// - type: replace-regex
// replace-regex: "https?://.*"
ReplaceRegex string `yaml:"replace-regex,omitempty" json:"replace-regex,omitempty" jsonschema:"title=replace regex of rule,description=Regex for regex-replace rule type"`
replaceRegex *regexp.Regexp `yaml:"-" json:"-"`
options *protocols.ExecutorOptions
generator *generators.PayloadGenerator
}
// ruleType is the type of rule enum declaration
@ -85,13 +95,15 @@ const (
prefixRuleType
postfixRuleType
infixRuleType
replaceRegexRuleType
)
var stringToRuleType = map[string]ruleType{
"replace": replaceRuleType,
"prefix": prefixRuleType,
"postfix": postfixRuleType,
"infix": infixRuleType,
"replace": replaceRuleType,
"prefix": prefixRuleType,
"postfix": postfixRuleType,
"infix": infixRuleType,
"replace-regex": replaceRegexRuleType,
}
// partType is the part of rule enum declaration
@ -100,11 +112,19 @@ type partType int
const (
queryPartType partType = iota + 1
headersPartType
pathPartType
bodyPartType
cookiePartType
requestPartType
)
var stringToPartType = map[string]partType{
"query": queryPartType,
"headers": headersPartType,
"header": headersPartType,
"path": pathPartType,
"body": bodyPartType,
"cookie": cookiePartType,
"request": requestPartType, // request means all request parts
}
// modeType is the mode of rule enum declaration

0
pkg/protocols/common/fuzz/fuzz_test.go → pkg/fuzz/fuzz_test.go
View File

210
pkg/fuzz/parts.go Normal file
View File

@ -0,0 +1,210 @@
package fuzz
import (
"io"
"strings"
"github.com/projectdiscovery/nuclei/v3/pkg/fuzz/component"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/expressions"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/projectdiscovery/retryablehttp-go"
)
// executePartRule executes part rules based on type
func (rule *Rule) executePartRule(input *ExecuteRuleInput, payload ValueOrKeyValue, component component.Component) error {
return rule.executePartComponent(input, payload, component)
}
// checkRuleApplicableOnComponent checks if a rule is applicable on given component
func (rule *Rule) checkRuleApplicableOnComponent(component component.Component) bool {
if rule.Part != component.Name() {
return false
}
foundAny := false
_ = component.Iterate(func(key string, value interface{}) error {
if rule.matchKeyOrValue(key, types.ToString(value)) {
foundAny = true
return io.EOF
}
return nil
})
return foundAny
}
// executePartComponent executes this rule on a given component and payload
func (rule *Rule) executePartComponent(input *ExecuteRuleInput, payload ValueOrKeyValue, ruleComponent component.Component) error {
if payload.IsKV() {
// for kv fuzzing
return rule.executePartComponentOnKV(input, payload, ruleComponent)
} else {
// for value only fuzzing
return rule.executePartComponentOnValues(input, payload.Value, ruleComponent)
}
}
// executePartComponentOnValues executes this rule on a given component and payload
// this supports both single and multiple [ruleType] modes
// i.e if component has multiple values, they can be replaced once or all depending on mode
func (rule *Rule) executePartComponentOnValues(input *ExecuteRuleInput, payloadStr string, ruleComponent component.Component) error {
finalErr := ruleComponent.Iterate(func(key string, value interface{}) error {
valueStr := types.ToString(value)
if !rule.matchKeyOrValue(key, valueStr) {
// ignore non-matching keys
return nil
}
var evaluated string
evaluated, input.InteractURLs = rule.executeEvaluate(input, key, valueStr, payloadStr, input.InteractURLs)
if err := ruleComponent.SetValue(key, evaluated); err != nil {
// gologger.Warning().Msgf("could not set value due to format restriction original(%s, %s[%T]) , new(%s,%s[%T])", key, valueStr, value, key, evaluated, evaluated)
return nil
}
if rule.modeType == singleModeType {
req, err := ruleComponent.Rebuild()
if err != nil {
return err
}
if qerr := rule.execWithInput(input, req, input.InteractURLs, ruleComponent); qerr != nil {
return qerr
}
// fmt.Printf("executed with value: %s\n", evaluated)
err = ruleComponent.SetValue(key, valueStr) // change back to previous value for temp
if err != nil {
return err
}
}
return nil
})
if finalErr != nil {
return finalErr
}
// We do not support analyzers with
// multiple payload mode.
if rule.modeType == multipleModeType {
req, err := ruleComponent.Rebuild()
if err != nil {
return err
}
if qerr := rule.execWithInput(input, req, input.InteractURLs, ruleComponent); qerr != nil {
err = qerr
return err
}
}
return nil
}
// executePartComponentOnKV executes this rule on a given component and payload
// currently only supports single mode
func (rule *Rule) executePartComponentOnKV(input *ExecuteRuleInput, payload ValueOrKeyValue, ruleComponent component.Component) error {
var origKey string
var origValue interface{}
// when we have a key-value pair, iterate over only 1 value of the component
// multiple values (aka multiple mode) not supported for this yet
_ = ruleComponent.Iterate(func(key string, value interface{}) error {
if key == payload.Key {
origKey = key
origValue = value
}
return nil
})
// iterate over given kv instead of component ones
return func(key, value string) error {
var evaluated string
evaluated, input.InteractURLs = rule.executeEvaluate(input, key, "", value, input.InteractURLs)
if err := ruleComponent.SetValue(key, evaluated); err != nil {
return err
}
if rule.modeType == singleModeType {
req, err := ruleComponent.Rebuild()
if err != nil {
return err
}
if qerr := rule.execWithInput(input, req, input.InteractURLs, ruleComponent); qerr != nil {
return err
}
// after building change back to original value to avoid repeating it in furthur requests
if origKey != "" {
err = ruleComponent.SetValue(origKey, types.ToString(origValue)) // change back to previous value for temp
if err != nil {
return err
}
} else {
_ = ruleComponent.Delete(key) // change back to previous value for temp
}
}
return nil
}(payload.Key, payload.Value)
}
// execWithInput executes a rule with input via callback
func (rule *Rule) execWithInput(input *ExecuteRuleInput, httpReq *retryablehttp.Request, interactURLs []string, component component.Component) error {
request := GeneratedRequest{
Request: httpReq,
InteractURLs: interactURLs,
DynamicValues: input.Values,
Component: component,
}
if !input.Callback(request) {
return types.ErrNoMoreRequests
}
return nil
}
// executeEvaluate executes evaluation of payload on a key and value and
// returns completed values to be replaced and processed
// for fuzzing.
func (rule *Rule) executeEvaluate(input *ExecuteRuleInput, _, value, payload string, interactshURLs []string) (string, []string) {
// TODO: Handle errors
values := generators.MergeMaps(input.Values, map[string]interface{}{
"value": value,
}, rule.options.Options.Vars.AsMap(), rule.options.Variables.GetAll())
firstpass, _ := expressions.Evaluate(payload, values)
interactData, interactshURLs := rule.options.Interactsh.Replace(firstpass, interactshURLs)
evaluated, _ := expressions.Evaluate(interactData, values)
replaced := rule.executeRuleTypes(input, value, evaluated)
return replaced, interactshURLs
}
// executeRuleTypes executes replacement for a key and value
// ex: prefix, postfix, infix, replace , replace-regex
func (rule *Rule) executeRuleTypes(_ *ExecuteRuleInput, value, replacement string) string {
var builder strings.Builder
if rule.ruleType == prefixRuleType || rule.ruleType == postfixRuleType {
builder.Grow(len(value) + len(replacement))
}
var returnValue string
switch rule.ruleType {
case prefixRuleType:
builder.WriteString(replacement)
builder.WriteString(value)
returnValue = builder.String()
case postfixRuleType:
builder.WriteString(value)
builder.WriteString(replacement)
returnValue = builder.String()
case infixRuleType:
if len(value) <= 1 {
builder.WriteString(value)
builder.WriteString(replacement)
returnValue = builder.String()
} else {
middleIndex := len(value) / 2
builder.WriteString(value[:middleIndex])
builder.WriteString(replacement)
builder.WriteString(value[middleIndex:])
returnValue = builder.String()
}
case replaceRuleType:
returnValue = replacement
case replaceRegexRuleType:
returnValue = rule.replaceRegex.ReplaceAllString(value, replacement)
}
return returnValue
}

2
pkg/fuzz/parts_test.go Normal file
View File

@ -0,0 +1,2 @@
// TODO: Write tests
package fuzz

80
pkg/fuzz/type.go Normal file
View File

@ -0,0 +1,80 @@
package fuzz
import (
"encoding/json"
"fmt"
mapsutil "github.com/projectdiscovery/utils/maps"
"gopkg.in/yaml.v2"
)
var (
_ json.Marshaler = &SliceOrMapSlice{}
_ json.Unmarshaler = &SliceOrMapSlice{}
_ yaml.Marshaler = &SliceOrMapSlice{}
_ yaml.Unmarshaler = &SliceOrMapSlice{}
)
type ValueOrKeyValue struct {
Key string
Value string
}
func (v *ValueOrKeyValue) IsKV() bool {
return v.Key != ""
}
type SliceOrMapSlice struct {
Value []string
KV *mapsutil.OrderedMap[string, string]
}
// UnmarshalJSON implements json.Unmarshaler interface.
func (v *SliceOrMapSlice) UnmarshalJSON(data []byte) error {
// try to unmashal as a string and fallback to map
if err := json.Unmarshal(data, &v.Value); err == nil {
return nil
}
err := json.Unmarshal(data, &v.KV)
if err != nil {
return fmt.Errorf("object can be a key:value or a string")
}
return nil
}
// MarshalJSON implements json.Marshaler interface.
func (v SliceOrMapSlice) MarshalJSON() ([]byte, error) {
if v.KV != nil {
return json.Marshal(v.KV)
}
return json.Marshal(v.Value)
}
// UnmarshalYAML implements yaml.Unmarshaler interface.
func (v *SliceOrMapSlice) UnmarshalYAML(callback func(interface{}) error) error {
// try to unmarshal it as a string and fallback to map
if err := callback(&v.Value); err == nil {
return nil
}
// try with a mapslice
var node yaml.MapSlice
if err := callback(&node); err == nil {
tmpx := mapsutil.NewOrderedMap[string, string]()
// preserve order
for _, v := range node {
tmpx.Set(v.Key.(string), v.Value.(string))
}
v.KV = &tmpx
return nil
}
return fmt.Errorf("object can be a key:value or a string")
}
// MarshalYAML implements yaml.Marshaler interface.
func (v SliceOrMapSlice) MarshalYAML() (any, error) {
if v.KV != nil {
return v.KV, nil
}
return v.Value, nil
}

29
pkg/input/README.md Normal file
View File

@ -0,0 +1,29 @@
## input
input package contains and provides loading, parsing , validating and normalizing of input data
## [transform](./transform.go)
Transform package transforms or normalizes the input data before it is sent to protocol executer this step mainly involves changes like adding default ports (if missing) , validating if input is file or directory or url and adjusting the input accordingly etc.
## Provider
Provider package contains the interface that every input format should implement for providing that input format to nuclei.
Currently Nuclei Supports three input providers:
1. SimpleInputProvider = A No-Op provider that takes a list of urls and implements the provider interface.
2. HttpInputProvider = A provider that supports loading and parsing input formats that contain complete Http Data like Entire Request, Response etc. Supported formats include Burp,openapi,swagger,postman,proxify etc.
3. ListInputProvider = Legacy/Default Provider that handles all list type inputs like urls,domains,ips,cidrs,files etc.
```go
func NewInputProvider(opts InputOptions) (InputProvider, error)
```
This function returns a InputProvider based by appropriately selecting input provider based on the input format (i.e either list or http) and returns the provider that can handle that input format.

90
pkg/input/formats/README.md Normal file
View File

@ -0,0 +1,90 @@
# formats
Formats implements support for passing a number of request source as input providers to nuclei to be tested for fuzzing related issues.
Currently the following formats are implemented -
- Burp Suite XML Request/Response file
- Proxify JSONL output file
- OpenAPI Specification file
- Postman Collection file
- Swagger Specification file
Each implementation implements either the entire or a subset of the features of the specifications. These can be increased further to add support as new things or requirements are identified.
Refer to the specific code for each implementation to understand supported features of the specs.
## OpenAPI Specification File
It is designed to generate HTTP requests based on an OpenAPI 3.0 Schema. Here is how these schema components are processed:
### Servers
The module supports multiple server URLs defined in the `Servers` section of the OpenAPI document. It will send requests to all the server URLs defined in the schema.
### Paths and Operations
The module supports all HTTP methods defined under each path in the `Paths` section. For each operation on a path, HTTP requests are generated and sent to the defined server URL. If the operation cannot generate a valid request, a warning will be logged.
### Parameters
The module recognizes parameters defined in the `query`, `header`, `path`, and `cookie` categories. When generating requests, if the `requiredOnly` flag is true, only the required parameters are included. Otherwise, all parameters, regardless of their required status, are used.
The `generateExampleFromSchema` function is used to generate suitable example data for each parameter from their respective schema definitions.
### RequestBody
The module also comprehends request bodies and supports various media types defined in the `Content` field. Currently, the following content-types are supported:
- `application/json`: The module creates application-specific JSON from the defined example schema.
- `application/xml`: The example schema is converted into xml format using `mxj` library.
- `application/x-www-form-urlencoded`: The example schema is converted into URL-encoded form data.
- `multipart/form-data`: The module supports multipart form-data and differentiates between fields and files using the `binary` format under the property schema.
- `text/plain`: Converts the example schema into string format and send as plain text.
For unsupported media types, no appropriate content type is found for the body. After setting up the body of the request, the module dumps the request and sends it to the defined server URL.
### Example Request Generation
This module converts each operation into one or more example HTTP requests. Each request is dumped into a string format, accompanied by its method, URL, headers, and body. These are send as a callback for further processing.
_Please note: This document does not cover other features of OpenAPI specification like responses, security schemes, links, callbacks, etc. as these are not currently handled by the module._
## Postman Collection file
This module parser Postman Collection JSON files.
### 1. Request Parsing:
Able to parse requests detailed in the Postman package. The parser is capable of interpreting the HTTP method, URL, and Body of each request present in the collection.
### 2. Header Parsing:
All HTTP headers set in the collection's request are parsed and set in the request.
### 3. Auth Type Parsing:
Able to parse and set the `Authentication` options provided in the postman collection in the request headers.
Supported types of authentiction:
1. **API Key**: In header
2. **Basic**: Setting basic auth through username, password.
3. **Bearer Token**: Involves setting bearer auth using tokens.
4. **No Auth**: No authentication is set.
Note: Not all parts of the Postman Collection specification are supported. This parser does not currently support Postman variables or collection level variables and items. It also does not support more authentication types than detailed above.
### Limitations:
* Does not support Postman variables
* Does not support Collection level variables and items
* Limited Authentication types supported
## Swagger Specification file
Swagger specification file is converted from OpenAPI 2.0 format to OpenAPI 3.0 format. After this, the OpenAPI parser from above is used.
## Burp XML / Proxify JSONL
These modules are generic and parse raw requests from these respective tools.

67
pkg/input/formats/burp/burp.go Normal file
View File

@ -0,0 +1,67 @@
package burp
import (
"encoding/base64"
"os"
"strings"
"github.com/pkg/errors"
"github.com/projectdiscovery/nuclei/v3/pkg/input/formats"
"github.com/projectdiscovery/nuclei/v3/pkg/input/types"
"github.com/projectdiscovery/utils/conversion"
"github.com/seh-msft/burpxml"
)
// BurpFormat is a Burp XML File parser
type BurpFormat struct {
opts formats.InputFormatOptions
}
// New creates a new Burp XML File parser
func New() *BurpFormat {
return &BurpFormat{}
}
var _ formats.Format = &BurpFormat{}
// Name returns the name of the format
func (j *BurpFormat) Name() string {
return "burp"
}
func (j *BurpFormat) SetOptions(options formats.InputFormatOptions) {
j.opts = options
}
// Parse parses the input and calls the provided callback
// function for each RawRequest it discovers.
func (j *BurpFormat) Parse(input string, resultsCb formats.ParseReqRespCallback) error {
file, err := os.Open(input)
if err != nil {
return errors.Wrap(err, "could not open data file")
}
defer file.Close()
items, err := burpxml.Parse(file, true)
if err != nil {
return errors.Wrap(err, "could not decode burp xml schema")
}
// Print the parsed data for verification
for _, item := range items.Items {
item := item
binx, err := base64.StdEncoding.DecodeString(item.Request.Raw)
if err != nil {
return errors.Wrap(err, "could not decode base64")
}
if strings.TrimSpace(conversion.String(binx)) == "" {
continue
}
rawRequest, err := types.ParseRawRequestWithURL(conversion.String(binx), item.Url)
if err != nil {
return errors.Wrap(err, "could not parse raw request")
}
resultsCb(rawRequest) // TODO: Handle false and true from callback
}
return nil
}

33
pkg/input/formats/burp/burp_test.go Normal file
View File

@ -0,0 +1,33 @@
package burp
import (
"testing"
"github.com/projectdiscovery/nuclei/v3/pkg/input/types"
"github.com/stretchr/testify/require"
)
func TestBurpParse(t *testing.T) {
format := New()
proxifyInputFile := "../testdata/burp.xml"
var gotMethodsToURLs []string
err := format.Parse(proxifyInputFile, func(request *types.RequestResponse) bool {
gotMethodsToURLs = append(gotMethodsToURLs, request.URL.String())
return false
})
if err != nil {
t.Fatal(err)
}
if len(gotMethodsToURLs) != 2 {
t.Fatalf("invalid number of methods: %d", len(gotMethodsToURLs))
}
var expectedURLs = []string{
"http://localhost:8087/scans",
"http://google.com/",
}
require.ElementsMatch(t, expectedURLs, gotMethodsToURLs, "could not get burp urls")
}

103
pkg/input/formats/formats.go Normal file
View File

@ -0,0 +1,103 @@
package formats
import (
"errors"
"os"
"strings"
"github.com/projectdiscovery/nuclei/v3/pkg/input/types"
fileutil "github.com/projectdiscovery/utils/file"
"gopkg.in/yaml.v3"
)
// ParseReqRespCallback is a callback function for discovered raw requests
type ParseReqRespCallback func(rr *types.RequestResponse) bool
// InputFormatOptions contains options for the input
// this can be variables that can be passed or
// overrides or some other options
type InputFormatOptions struct {
// Variables is list of variables that can be used
// while generating requests in given format
Variables map[string]interface{}
// SkipFormatValidation is used to skip format validation
// while debugging or testing if format is invalid then
// requests are skipped instead of creating invalid requests
SkipFormatValidation bool
// RequiredOnly only uses required fields when generating requests
// instead of all fields
RequiredOnly bool
}
// Format is an interface implemented by all input formats
type Format interface {
// Name returns the name of the format
Name() string
// Parse parses the input and calls the provided callback
// function for each RawRequest it discovers.
Parse(input string, resultsCb ParseReqRespCallback) error
// SetOptions sets the options for the input format
SetOptions(options InputFormatOptions)
}
var (
DefaultVarDumpFileName = "required_openapi_params.yaml"
ErrNoVarsDumpFile = errors.New("no required params file found")
)
// == OpenAPIParamsCfgFile ==
// this file is meant to be used in CLI mode
// to be more interactive and user-friendly when
// running nuclei with openapi format
// OpenAPIParamsCfgFile is the structure of the required vars dump file
type OpenAPIParamsCfgFile struct {
Var []string `yaml:"var"`
OptionalVars []string `yaml:"-"` // this will be written to the file as comments
}
// ReadOpenAPIVarDumpFile reads the required vars dump file
func ReadOpenAPIVarDumpFile() (*OpenAPIParamsCfgFile, error) {
var vars OpenAPIParamsCfgFile
if !fileutil.FileExists(DefaultVarDumpFileName) {
return nil, ErrNoVarsDumpFile
}
bin, err := os.ReadFile(DefaultVarDumpFileName)
if err != nil {
return nil, err
}
err = yaml.Unmarshal(bin, &vars)
if err != nil {
return nil, err
}
filtered := []string{}
for _, v := range vars.Var {
v = strings.TrimSpace(v)
if !strings.HasSuffix(v, "=") {
filtered = append(filtered, v)
}
}
vars.Var = filtered
return &vars, nil
}
// WriteOpenAPIVarDumpFile writes the required vars dump file
func WriteOpenAPIVarDumpFile(vars *OpenAPIParamsCfgFile) error {
f, err := os.OpenFile(DefaultVarDumpFileName, os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0644)
if err != nil {
return err
}
defer f.Close()
bin, err := yaml.Marshal(vars)
if err != nil {
return err
}
_, _ = f.Write(bin)
if len(vars.OptionalVars) > 0 {
_, _ = f.WriteString("\n # Optional parameters\n")
for _, v := range vars.OptionalVars {
_, _ = f.WriteString(" # - " + v + "=\n")
}
}
return f.Sync()
}

74
pkg/input/formats/json/json.go Normal file
View File

@ -0,0 +1,74 @@
package json
import (
"encoding/json"
"io"
"os"
"github.com/pkg/errors"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v3/pkg/input/formats"
"github.com/projectdiscovery/nuclei/v3/pkg/input/types"
)
// JSONFormat is a JSON format parser for nuclei
// input HTTP requests
type JSONFormat struct {
opts formats.InputFormatOptions
}
// New creates a new JSON format parser
func New() *JSONFormat {
return &JSONFormat{}
}
var _ formats.Format = &JSONFormat{}
// proxifyRequest is a request for proxify
type proxifyRequest struct {
URL string `json:"url"`
Request struct {
Header map[string]string `json:"header"`
Body string `json:"body"`
Raw string `json:"raw"`
} `json:"request"`
}
// Name returns the name of the format
func (j *JSONFormat) Name() string {
return "jsonl"
}
func (j *JSONFormat) SetOptions(options formats.InputFormatOptions) {
j.opts = options
}
// Parse parses the input and calls the provided callback
// function for each RawRequest it discovers.
func (j *JSONFormat) Parse(input string, resultsCb formats.ParseReqRespCallback) error {
file, err := os.Open(input)
if err != nil {
return errors.Wrap(err, "could not open json file")
}
defer file.Close()
decoder := json.NewDecoder(file)
for {
var request proxifyRequest
err := decoder.Decode(&request)
if err == io.EOF {
break
}
if err != nil {
return errors.Wrap(err, "could not decode json file")
}
rawRequest, err := types.ParseRawRequestWithURL(request.Request.Raw, request.URL)
if err != nil {
gologger.Warning().Msgf("jsonl: Could not parse raw request %s: %s\n", request.URL, err)
continue
}
resultsCb(rawRequest)
}
return nil
}

57
pkg/input/formats/json/json_test.go Normal file
View File

@ -0,0 +1,57 @@
package json
import (
"testing"
"github.com/projectdiscovery/nuclei/v3/pkg/input/types"
"github.com/stretchr/testify/require"
)
var expectedURLs = []string{
"https://ginandjuice.shop/",
"https://ginandjuice.shop/catalog/product?productId=1",
"https://ginandjuice.shop/resources/js/stockCheck.js",
"https://ginandjuice.shop/resources/js/xmlStockCheckPayload.js",
"https://ginandjuice.shop/resources/js/xmlStockCheckPayload.js",
"https://ginandjuice.shop/resources/js/stockCheck.js",
"https://ginandjuice.shop/catalog/product/stock",
"https://ginandjuice.shop/catalog/cart",
"https://ginandjuice.shop/catalog/product?productId=1",
"https://ginandjuice.shop/catalog/subscribe",
"https://ginandjuice.shop/blog",
"https://ginandjuice.shop/blog/?search=dadad&back=%2Fblog%2F",
"https://ginandjuice.shop/logger",
"https://ginandjuice.shop/blog/",
"https://ginandjuice.shop/blog/post?postId=3",
"https://ginandjuice.shop/about",
"https://ginandjuice.shop/my-account",
"https://ginandjuice.shop/login",
"https://ginandjuice.shop/login",
"https://ginandjuice.shop/login",
"https://ginandjuice.shop/my-account",
"https://ginandjuice.shop/catalog/cart",
"https://ginandjuice.shop/my-account",
"https://ginandjuice.shop/logout",
"https://ginandjuice.shop/",
"https://ginandjuice.shop/catalog",
}
func TestJSONFormatterParse(t *testing.T) {
format := New()
proxifyInputFile := "../testdata/ginandjuice.proxify.json"
var urls []string
err := format.Parse(proxifyInputFile, func(request *types.RequestResponse) bool {
urls = append(urls, request.URL.String())
return false
})
if err != nil {
t.Fatal(err)
}
if len(urls) != len(expectedURLs) {
t.Fatalf("invalid number of urls: %d", len(urls))
}
require.ElementsMatch(t, urls, expectedURLs)
}

290
pkg/input/formats/openapi/examples.go Normal file
View File

@ -0,0 +1,290 @@
package openapi
import (
"fmt"
"github.com/getkin/kin-openapi/openapi3"
"github.com/pkg/errors"
)
// From: https://github.com/danielgtaylor/apisprout/blob/master/example.go
func getSchemaExample(schema *openapi3.Schema) (interface{}, bool) {
if schema.Example != nil {
return schema.Example, true
}
if schema.Default != nil {
return schema.Default, true
}
if schema.Enum != nil && len(schema.Enum) > 0 {
return schema.Enum[0], true
}
return nil, false
}
// stringFormatExample returns an example string based on the given format.
// http://json-schema.org/latest/json-schema-validation.html#rfc.section.7.3
func stringFormatExample(format string) string {
switch format {
case "date":
// https://tools.ietf.org/html/rfc3339
return "2018-07-23"
case "date-time":
// This is the date/time of API Sprout's first commit! :-)
return "2018-07-23T22:58:00-07:00"
case "time":
return "22:58:00-07:00"
case "email":
return "email@example.com"
case "hostname":
// https://tools.ietf.org/html/rfc2606#page-2
return "example.com"
case "ipv4":
// https://tools.ietf.org/html/rfc5737
return "198.51.100.0"
case "ipv6":
// https://tools.ietf.org/html/rfc3849
return "2001:0db8:85a3:0000:0000:8a2e:0370:7334"
case "uri":
return "https://tools.ietf.org/html/rfc3986"
case "uri-template":
// https://tools.ietf.org/html/rfc6570
return "http://example.com/dictionary/{term:1}/{term}"
case "json-pointer":
// https://tools.ietf.org/html/rfc6901
return "#/components/parameters/term"
case "regex":
// https://stackoverflow.com/q/3296050/164268
return "/^1?$|^(11+?)\\1+$/"
case "uuid":
// https://www.ietf.org/rfc/rfc4122.txt
return "f81d4fae-7dec-11d0-a765-00a0c91e6bf6"
case "password":
return "********"
case "binary":
return "sagefuzzertest"
}
return ""
}
// excludeFromMode will exclude a schema if the mode is request and the schema
// is read-only
func excludeFromMode(schema *openapi3.Schema) bool {
if schema == nil {
return true
}
if schema.ReadOnly {
return true
}
return false
}
// isRequired checks whether a key is actually required.
func isRequired(schema *openapi3.Schema, key string) bool {
for _, req := range schema.Required {
if req == key {
return true
}
}
return false
}
type cachedSchema struct {
pending bool
out interface{}
}
var (
// ErrRecursive is when a schema is impossible to represent because it infinitely recurses.
ErrRecursive = errors.New("Recursive schema")
// ErrNoExample is sent when no example was found for an operation.
ErrNoExample = errors.New("No example found")
)
func openAPIExample(schema *openapi3.Schema, cache map[*openapi3.Schema]*cachedSchema) (out interface{}, err error) {
if ex, ok := getSchemaExample(schema); ok {
return ex, nil
}
cached, ok := cache[schema]
if !ok {
cached = &cachedSchema{
pending: true,
}
cache[schema] = cached
} else if cached.pending {
return nil, ErrRecursive
} else {
return cached.out, nil
}
defer func() {
cached.pending = false
cached.out = out
}()
// Handle combining keywords
if len(schema.OneOf) > 0 {
var ex interface{}
var err error
for _, candidate := range schema.OneOf {
ex, err = openAPIExample(candidate.Value, cache)
if err == nil {
break
}
}
return ex, err
}
if len(schema.AnyOf) > 0 {
var ex interface{}
var err error
for _, candidate := range schema.AnyOf {
ex, err = openAPIExample(candidate.Value, cache)
if err == nil {
break
}
}
return ex, err
}
if len(schema.AllOf) > 0 {
example := map[string]interface{}{}
for _, allOf := range schema.AllOf {
candidate, err := openAPIExample(allOf.Value, cache)
if err != nil {
return nil, err
}
value, ok := candidate.(map[string]interface{})
if !ok {
return nil, ErrNoExample
}
for k, v := range value {
example[k] = v
}
}
return example, nil
}
switch {
case schema.Type == "boolean":
return true, nil
case schema.Type == "number", schema.Type == "integer":
value := 0.0
if schema.Min != nil && *schema.Min > value {
value = *schema.Min
if schema.ExclusiveMin {
if schema.Max != nil {
// Make the value half way.
value = (*schema.Min + *schema.Max) / 2.0
} else {
value++
}
}
}
if schema.Max != nil && *schema.Max < value {
value = *schema.Max
if schema.ExclusiveMax {
if schema.Min != nil {
// Make the value half way.
value = (*schema.Min + *schema.Max) / 2.0
} else {
value--
}
}
}
if schema.MultipleOf != nil && int(value)%int(*schema.MultipleOf) != 0 {
value += float64(int(*schema.MultipleOf) - (int(value) % int(*schema.MultipleOf)))
}
if schema.Type == "integer" {
return int(value), nil
}
return value, nil
case schema.Type == "string":
if ex := stringFormatExample(schema.Format); ex != "" {
return ex, nil
}
example := "string"
for schema.MinLength > uint64(len(example)) {
example += example
}
if schema.MaxLength != nil && *schema.MaxLength < uint64(len(example)) {
example = example[:*schema.MaxLength]
}
return example, nil
case schema.Type == "array", schema.Items != nil:
example := []interface{}{}
if schema.Items != nil && schema.Items.Value != nil {
ex, err := openAPIExample(schema.Items.Value, cache)
if err != nil {
return nil, fmt.Errorf("can't get example for array item: %+v", err)
}
example = append(example, ex)
for uint64(len(example)) < schema.MinItems {
example = append(example, ex)
}
}
return example, nil
case schema.Type == "object", len(schema.Properties) > 0:
example := map[string]interface{}{}
for k, v := range schema.Properties {
if excludeFromMode(v.Value) {
continue
}
ex, err := openAPIExample(v.Value, cache)
if err == ErrRecursive {
if isRequired(schema, k) {
return nil, fmt.Errorf("can't get example for '%s': %+v", k, err)
}
} else if err != nil {
return nil, fmt.Errorf("can't get example for '%s': %+v", k, err)
} else {
example[k] = ex
}
}
if schema.AdditionalProperties.Has != nil && schema.AdditionalProperties.Schema != nil {
addl := schema.AdditionalProperties.Schema.Value
if !excludeFromMode(addl) {
ex, err := openAPIExample(addl, cache)
if err == ErrRecursive {
// We just won't add this if it's recursive.
} else if err != nil {
return nil, fmt.Errorf("can't get example for additional properties: %+v", err)
} else {
example["additionalPropertyName"] = ex
}
}
}
return example, nil
}
return nil, ErrNoExample
}
// generateExampleFromSchema creates an example structure from an OpenAPI 3 schema
// object, which is an extended subset of JSON Schema.
//
// https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.1.md#schemaObject
func generateExampleFromSchema(schema *openapi3.Schema) (interface{}, error) {
return openAPIExample(schema, make(map[*openapi3.Schema]*cachedSchema)) // TODO: Use caching
}

461
pkg/input/formats/openapi/generator.go Normal file
View File

@ -0,0 +1,461 @@
package openapi
import (
"bytes"
"encoding/json"
"fmt"
"io"
"mime/multipart"
"net/http"
"net/http/httputil"
"net/url"
"os"
"strings"
"github.com/clbanning/mxj/v2"
"github.com/getkin/kin-openapi/openapi3"
"github.com/pkg/errors"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
"github.com/projectdiscovery/nuclei/v3/pkg/input/formats"
httpTypes "github.com/projectdiscovery/nuclei/v3/pkg/input/types"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
errorutil "github.com/projectdiscovery/utils/errors"
"github.com/projectdiscovery/utils/generic"
mapsutil "github.com/projectdiscovery/utils/maps"
"github.com/valyala/fasttemplate"
)
const (
globalAuth = "globalAuth"
)
// GenerateRequestsFromSchema generates http requests from an OpenAPI 3.0 document object
func GenerateRequestsFromSchema(schema *openapi3.T, opts formats.InputFormatOptions, callback formats.ParseReqRespCallback) error {
if len(schema.Servers) == 0 {
return errors.New("no servers found in openapi schema")
}
// new set of globalParams obtained from security schemes
globalParams := openapi3.NewParameters()
if len(schema.Security) > 0 {
params, err := GetGlobalParamsForSecurityRequirement(schema, &schema.Security)
if err != nil {
return err
}
globalParams = append(globalParams, params...)
}
// validate global param requirements
for _, param := range globalParams {
if val, ok := opts.Variables[param.Value.Name]; ok {
param.Value.Example = val
} else {
// if missing check for validation
if opts.SkipFormatValidation {
gologger.Verbose().Msgf("openapi: skipping all requests due to missing global auth parameter: %s\n", param.Value.Name)
return nil
} else {
// fatal error
gologger.Fatal().Msgf("openapi: missing global auth parameter: %s\n", param.Value.Name)
}
}
}
missingVarMap := make(map[string]struct{})
optionalVarMap := make(map[string]struct{})
missingParamValueCallback := func(param *openapi3.Parameter, opts *generateReqOptions) {
if !param.Required {
optionalVarMap[param.Name] = struct{}{}
return
}
missingVarMap[param.Name] = struct{}{}
}
for _, serverURL := range schema.Servers {
pathURL := serverURL.URL
for path, v := range schema.Paths.Map() {
// a path item can have parameters
ops := v.Operations()
requestPath := path
for method, ov := range ops {
if err := generateRequestsFromOp(&generateReqOptions{
requiredOnly: opts.RequiredOnly,
method: method,
pathURL: pathURL,
requestPath: requestPath,
op: ov,
schema: schema,
globalParams: globalParams,
reqParams: v.Parameters,
opts: opts,
callback: callback,
missingParamValueCallback: missingParamValueCallback,
}); err != nil {
gologger.Warning().Msgf("Could not generate requests from op: %s\n", err)
}
}
}
}
if len(missingVarMap) > 0 && !opts.SkipFormatValidation {
gologger.Error().Msgf("openapi: Found %d missing parameters, use -skip-format-validation flag to skip requests or update missing parameters generated in %s file,you can also specify these vars using -var flag in (key=value) format\n", len(missingVarMap), formats.DefaultVarDumpFileName)
gologger.Verbose().Msgf("openapi: missing params: %+v", mapsutil.GetSortedKeys(missingVarMap))
if config.CurrentAppMode == config.AppModeCLI {
// generate var dump file
vars := &formats.OpenAPIParamsCfgFile{}
for k := range missingVarMap {
vars.Var = append(vars.Var, k+"=")
}
vars.OptionalVars = mapsutil.GetSortedKeys(optionalVarMap)
if err := formats.WriteOpenAPIVarDumpFile(vars); err != nil {
gologger.Error().Msgf("openapi: could not write params file: %s\n", err)
}
// exit with status code 1
os.Exit(1)
}
}
return nil
}
type generateReqOptions struct {
// requiredOnly specifies whether to generate only required fields
requiredOnly bool
// method is the http method to use
method string
// pathURL is the base url to use
pathURL string
// requestPath is the path to use
requestPath string
// schema is the openapi schema to use
schema *openapi3.T
// op is the operation to use
op *openapi3.Operation
// post request generation callback
callback formats.ParseReqRespCallback
// global parameters
globalParams openapi3.Parameters
// requestparams map
reqParams openapi3.Parameters
// global var map
opts formats.InputFormatOptions
// missingVar Callback
missingParamValueCallback func(param *openapi3.Parameter, opts *generateReqOptions)
}
// generateRequestsFromOp generates requests from an operation and some other data
// about an OpenAPI Schema Path and Method object.
//
// It also accepts an optional requiredOnly flag which if specified, only returns the fields
// of the structure that are required. If false, all fields are returned.
func generateRequestsFromOp(opts *generateReqOptions) error {
req, err := http.NewRequest(opts.method, opts.pathURL+opts.requestPath, nil)
if err != nil {
return errors.Wrap(err, "could not make request")
}
reqParams := opts.reqParams
if reqParams == nil {
reqParams = openapi3.NewParameters()
}
// add existing req params
reqParams = append(reqParams, opts.op.Parameters...)
// check for endpoint specific auth
if opts.op.Security != nil {
params, err := GetGlobalParamsForSecurityRequirement(opts.schema, opts.op.Security)
if err != nil {
return err
}
reqParams = append(reqParams, params...)
} else {
reqParams = append(reqParams, opts.globalParams...)
}
query := url.Values{}
for _, parameter := range reqParams {
value := parameter.Value
// paramValue or default value to use
var paramValue interface{}
// accept override from global variables
if val, ok := opts.opts.Variables[value.Name]; ok {
paramValue = val
} else if value.Schema.Value.Default != nil {
paramValue = value.Schema.Value.Default
} else if value.Schema.Value.Example != nil {
paramValue = value.Schema.Value.Example
} else if value.Schema.Value.Enum != nil && len(value.Schema.Value.Enum) > 0 {
paramValue = value.Schema.Value.Enum[0]
} else {
if !opts.opts.SkipFormatValidation {
if opts.missingParamValueCallback != nil {
opts.missingParamValueCallback(value, opts)
}
// skip request if param in path else skip this param only
if value.Required {
// gologger.Verbose().Msgf("skipping request [%s] %s due to missing value (%v)\n", opts.method, opts.requestPath, value.Name)
return nil
} else {
// if it is in path then remove it from path
opts.requestPath = strings.Replace(opts.requestPath, fmt.Sprintf("{%s}", value.Name), "", -1)
if !opts.opts.RequiredOnly {
gologger.Verbose().Msgf("openapi: skipping optional param (%s) in (%v) in request [%s] %s due to missing value (%v)\n", value.Name, value.In, opts.method, opts.requestPath, value.Name)
}
continue
}
}
exampleX, err := generateExampleFromSchema(value.Schema.Value)
if err != nil {
// when failed to generate example
// skip request if param in path else skip this param only
if value.Required {
gologger.Verbose().Msgf("openapi: skipping request [%s] %s due to missing value (%v)\n", opts.method, opts.requestPath, value.Name)
return nil
} else {
// if it is in path then remove it from path
opts.requestPath = strings.Replace(opts.requestPath, fmt.Sprintf("{%s}", value.Name), "", -1)
if !opts.opts.RequiredOnly {
gologger.Verbose().Msgf("openapi: skipping optinal param (%s) in (%v) in request [%s] %s due to missing value (%v)\n", value.Name, value.In, opts.method, opts.requestPath, value.Name)
}
continue
}
}
paramValue = exampleX
}
if opts.requiredOnly && !value.Required {
// remove them from path if any
opts.requestPath = strings.Replace(opts.requestPath, fmt.Sprintf("{%s}", value.Name), "", -1)
continue // Skip this parameter if it is not required and we want only required ones
}
switch value.In {
case "query":
query.Set(value.Name, types.ToString(paramValue))
case "header":
req.Header.Set(value.Name, types.ToString(paramValue))
case "path":
opts.requestPath = fasttemplate.ExecuteStringStd(opts.requestPath, "{", "}", map[string]interface{}{
value.Name: types.ToString(paramValue),
})
case "cookie":
req.AddCookie(&http.Cookie{Name: value.Name, Value: types.ToString(paramValue)})
}
}
req.URL.RawQuery = query.Encode()
req.URL.Path = opts.requestPath
if opts.op.RequestBody != nil {
for content, value := range opts.op.RequestBody.Value.Content {
cloned := req.Clone(req.Context())
example, err := generateExampleFromSchema(value.Schema.Value)
if err != nil {
continue
}
// var body string
switch content {
case "application/json":
if marshalled, err := json.Marshal(example); err == nil {
// body = string(marshalled)
cloned.Body = io.NopCloser(bytes.NewReader(marshalled))
cloned.ContentLength = int64(len(marshalled))
cloned.Header.Set("Content-Type", "application/json")
}
case "application/xml":
exampleVal := mxj.Map(example.(map[string]interface{}))
if marshalled, err := exampleVal.Xml(); err == nil {
// body = string(marshalled)
cloned.Body = io.NopCloser(bytes.NewReader(marshalled))
cloned.ContentLength = int64(len(marshalled))
cloned.Header.Set("Content-Type", "application/xml")
} else {
gologger.Warning().Msgf("openapi: could not encode xml")
}
case "application/x-www-form-urlencoded":
if values, ok := example.(map[string]interface{}); ok {
cloned.Form = url.Values{}
for k, v := range values {
cloned.Form.Set(k, types.ToString(v))
}
encoded := cloned.Form.Encode()
cloned.ContentLength = int64(len(encoded))
// body = encoded
cloned.Body = io.NopCloser(strings.NewReader(encoded))
cloned.Header.Set("Content-Type", "application/x-www-form-urlencoded")
}
case "multipart/form-data":
if values, ok := example.(map[string]interface{}); ok {
buffer := &bytes.Buffer{}
multipartWriter := multipart.NewWriter(buffer)
for k, v := range values {
// This is a file if format is binary, otherwise field
if property, ok := value.Schema.Value.Properties[k]; ok && property.Value.Format == "binary" {
if writer, err := multipartWriter.CreateFormFile(k, k); err == nil {
_, _ = writer.Write([]byte(types.ToString(v)))
}
} else {
_ = multipartWriter.WriteField(k, types.ToString(v))
}
}
multipartWriter.Close()
// body = buffer.String()
cloned.Body = io.NopCloser(buffer)
cloned.ContentLength = int64(len(buffer.Bytes()))
cloned.Header.Set("Content-Type", multipartWriter.FormDataContentType())
}
case "text/plain":
str := types.ToString(example)
// body = str
cloned.Body = io.NopCloser(strings.NewReader(str))
cloned.ContentLength = int64(len(str))
cloned.Header.Set("Content-Type", "text/plain")
case "application/octet-stream":
str := types.ToString(example)
if str == "" {
// use two strings
str = "string1\nstring2"
}
if value.Schema != nil && generic.EqualsAny(value.Schema.Value.Format, "bindary", "byte") {
cloned.Body = io.NopCloser(bytes.NewReader([]byte(str)))
cloned.ContentLength = int64(len(str))
cloned.Header.Set("Content-Type", "application/octet-stream")
} else {
// use string placeholder
cloned.Body = io.NopCloser(strings.NewReader(str))
cloned.ContentLength = int64(len(str))
cloned.Header.Set("Content-Type", "text/plain")
}
default:
gologger.Verbose().Msgf("openapi: no correct content type found for body: %s\n", content)
// LOG: return errors.New("no correct content type found for body")
continue
}
dumped, err := httputil.DumpRequestOut(cloned, true)
if err != nil {
return errors.Wrap(err, "could not dump request")
}
rr, err := httpTypes.ParseRawRequestWithURL(string(dumped), cloned.URL.String())
if err != nil {
return errors.Wrap(err, "could not parse raw request")
}
opts.callback(rr)
continue
}
}
if opts.op.RequestBody != nil {
return nil
}
dumped, err := httputil.DumpRequestOut(req, true)
if err != nil {
return errors.Wrap(err, "could not dump request")
}
rr, err := httpTypes.ParseRawRequestWithURL(string(dumped), req.URL.String())
if err != nil {
return errors.Wrap(err, "could not parse raw request")
}
opts.callback(rr)
return nil
}
// GetGlobalParamsForSecurityRequirement returns the global parameters for a security requirement
func GetGlobalParamsForSecurityRequirement(schema *openapi3.T, requirement *openapi3.SecurityRequirements) ([]*openapi3.ParameterRef, error) {
globalParams := openapi3.NewParameters()
if len(schema.Components.SecuritySchemes) == 0 {
return nil, errorutil.NewWithTag("openapi", "security requirements (%+v) without any security schemes found in openapi file", schema.Security)
}
found := false
// this api is protected for each security scheme pull its corresponding scheme
schemaLabel:
for _, security := range *requirement {
for name := range security {
if scheme, ok := schema.Components.SecuritySchemes[name]; ok {
found = true
param, err := GenerateParameterFromSecurityScheme(scheme)
if err != nil {
return nil, err
}
globalParams = append(globalParams, &openapi3.ParameterRef{Value: param})
continue schemaLabel
}
}
if !found && len(security) > 1 {
// if this is case then both security schemes are required
return nil, errorutil.NewWithTag("openapi", "security requirement (%+v) not found in openapi file", security)
}
}
if !found {
return nil, errorutil.NewWithTag("openapi", "security requirement (%+v) not found in openapi file", requirement)
}
return globalParams, nil
}
// generateExampleFromSchema generates an example from a schema object
func GenerateParameterFromSecurityScheme(scheme *openapi3.SecuritySchemeRef) (*openapi3.Parameter, error) {
if !generic.EqualsAny(scheme.Value.Type, "http", "apiKey") {
return nil, errorutil.NewWithTag("openapi", "unsupported security scheme type (%s) found in openapi file", scheme.Value.Type)
}
if scheme.Value.Type == "http" {
// check scheme
if !generic.EqualsAny(scheme.Value.Scheme, "basic", "bearer") {
return nil, errorutil.NewWithTag("openapi", "unsupported security scheme (%s) found in openapi file", scheme.Value.Scheme)
}
if scheme.Value.Name == "" {
return nil, errorutil.NewWithTag("openapi", "security scheme (%s) name is empty", scheme.Value.Scheme)
}
// create parameters using the scheme
switch scheme.Value.Scheme {
case "basic":
h := openapi3.NewHeaderParameter(scheme.Value.Name)
h.Required = true
h.Description = globalAuth // differentiator for normal variables and global auth
return h, nil
case "bearer":
h := openapi3.NewHeaderParameter(scheme.Value.Name)
h.Required = true
h.Description = globalAuth // differentiator for normal variables and global auth
return h, nil
}
}
if scheme.Value.Type == "apiKey" {
// validate name and in
if scheme.Value.Name == "" {
return nil, errorutil.NewWithTag("openapi", "security scheme (%s) name is empty", scheme.Value.Type)
}
if !generic.EqualsAny(scheme.Value.In, "query", "header", "cookie") {
return nil, errorutil.NewWithTag("openapi", "unsupported security scheme (%s) in (%s) found in openapi file", scheme.Value.Type, scheme.Value.In)
}
// create parameters using the scheme
switch scheme.Value.In {
case "query":
q := openapi3.NewQueryParameter(scheme.Value.Name)
q.Required = true
q.Description = globalAuth // differentiator for normal variables and global auth
return q, nil
case "header":
h := openapi3.NewHeaderParameter(scheme.Value.Name)
h.Required = true
h.Description = globalAuth // differentiator for normal variables and global auth
return h, nil
case "cookie":
c := openapi3.NewCookieParameter(scheme.Value.Name)
c.Required = true
c.Description = globalAuth // differentiator for normal variables and global auth
return c, nil
}
}
return nil, errorutil.NewWithTag("openapi", "unsupported security scheme type (%s) found in openapi file", scheme.Value.Type)
}

39
pkg/input/formats/openapi/openapi.go Normal file
View File

@ -0,0 +1,39 @@
package openapi
import (
"github.com/getkin/kin-openapi/openapi3"
"github.com/pkg/errors"
"github.com/projectdiscovery/nuclei/v3/pkg/input/formats"
)
// OpenAPIFormat is a OpenAPI Schema File parser
type OpenAPIFormat struct {
opts formats.InputFormatOptions
}
// New creates a new OpenAPI format parser
func New() *OpenAPIFormat {
return &OpenAPIFormat{}
}
var _ formats.Format = &OpenAPIFormat{}
// Name returns the name of the format
func (j *OpenAPIFormat) Name() string {
return "openapi"
}
func (j *OpenAPIFormat) SetOptions(options formats.InputFormatOptions) {
j.opts = options
}
// Parse parses the input and calls the provided callback
// function for each RawRequest it discovers.
func (j *OpenAPIFormat) Parse(input string, resultsCb formats.ParseReqRespCallback) error {
loader := openapi3.NewLoader()
schema, err := loader.LoadFromFile(input)
if err != nil {
return errors.Wrap(err, "could not decode openapi 3.0 schema")
}
return GenerateRequestsFromSchema(schema, j.opts, resultsCb)
}

63
pkg/input/formats/openapi/openapi_test.go Normal file
View File

@ -0,0 +1,63 @@
package openapi
import (
"strings"
"testing"
"github.com/projectdiscovery/nuclei/v3/pkg/input/types"
"github.com/stretchr/testify/require"
)
const baseURL = "http://hackthebox:5000"
var methodToURLs = map[string][]string{
"GET": {
"{{baseUrl}}/createdb",
"{{baseUrl}}/",
"{{baseUrl}}/users/v1/John.Doe",
"{{baseUrl}}/users/v1",
"{{baseUrl}}/users/v1/_debug",
"{{baseUrl}}/books/v1",
"{{baseUrl}}/books/v1/bookTitle77",
},
"POST": {
"{{baseUrl}}/users/v1/register",
"{{baseUrl}}/users/v1/login",
"{{baseUrl}}/books/v1",
},
"PUT": {
"{{baseUrl}}/users/v1/name1/email",
"{{baseUrl}}/users/v1/name1/password",
},
"DELETE": {
"{{baseUrl}}/users/v1/name1",
},
}
func TestOpenAPIParser(t *testing.T) {
format := New()
proxifyInputFile := "../testdata/openapi.yaml"
gotMethodsToURLs := make(map[string][]string)
err := format.Parse(proxifyInputFile, func(rr *types.RequestResponse) bool {
gotMethodsToURLs[rr.Request.Method] = append(gotMethodsToURLs[rr.Request.Method],
strings.Replace(rr.URL.String(), baseURL, "{{baseUrl}}", 1))
return false
})
if err != nil {
t.Fatal(err)
}
if len(gotMethodsToURLs) != len(methodToURLs) {
t.Fatalf("invalid number of methods: %d", len(gotMethodsToURLs))
}
for method, urls := range gotMethodsToURLs {
if len(urls) != len(methodToURLs[method]) {
t.Fatalf("invalid number of urls for method %s: %d", method, len(urls))
}
require.ElementsMatch(t, urls, methodToURLs[method], "invalid urls for method %s", method)
}
}

69
pkg/input/formats/swagger/swagger.go Normal file
View File

@ -0,0 +1,69 @@
package swagger
import (
"encoding/json"
"os"
"path"
"github.com/getkin/kin-openapi/openapi2"
"github.com/getkin/kin-openapi/openapi3"
"github.com/pkg/errors"
"github.com/projectdiscovery/nuclei/v3/pkg/input/formats"
"github.com/projectdiscovery/nuclei/v3/pkg/input/formats/openapi"
"gopkg.in/yaml.v2"
"github.com/getkin/kin-openapi/openapi2conv"
)
// SwaggerFormat is a Swagger Schema File parser
type SwaggerFormat struct {
opts formats.InputFormatOptions
}
// New creates a new Swagger format parser
func New() *SwaggerFormat {
return &SwaggerFormat{}
}
var _ formats.Format = &SwaggerFormat{}
// Name returns the name of the format
func (j *SwaggerFormat) Name() string {
return "swagger"
}
func (j *SwaggerFormat) SetOptions(options formats.InputFormatOptions) {
j.opts = options
}
// Parse parses the input and calls the provided callback
// function for each RawRequest it discovers.
func (j *SwaggerFormat) Parse(input string, resultsCb formats.ParseReqRespCallback) error {
file, err := os.Open(input)
if err != nil {
return errors.Wrap(err, "could not open data file")
}
defer file.Close()
schemav2 := &openapi2.T{}
ext := path.Ext(input)
if ext == ".yaml" || ext == ".yml" {
err = yaml.NewDecoder(file).Decode(schemav2)
} else {
err = json.NewDecoder(file).Decode(schemav2)
}
if err != nil {
return errors.Wrap(err, "could not decode openapi 2.0 schema")
}
schema, err := openapi2conv.ToV3(schemav2)
if err != nil {
return errors.Wrap(err, "could not convert openapi 2.0 schema to 3.0")
}
loader := openapi3.NewLoader()
err = loader.ResolveRefsIn(schema, nil)
if err != nil {
return errors.Wrap(err, "could not resolve openapi schema references")
}
return openapi.GenerateRequestsFromSchema(schema, j.opts, resultsCb)
}

34
pkg/input/formats/swagger/swagger_test.go Normal file
View File

@ -0,0 +1,34 @@
package swagger
import (
"testing"
"github.com/projectdiscovery/nuclei/v3/pkg/input/types"
"github.com/stretchr/testify/require"
)
func TestSwaggerAPIParser(t *testing.T) {
format := New()
proxifyInputFile := "../testdata/swagger.yaml"
var gotMethodsToURLs []string
err := format.Parse(proxifyInputFile, func(request *types.RequestResponse) bool {
gotMethodsToURLs = append(gotMethodsToURLs, request.URL.String())
return false
})
if err != nil {
t.Fatal(err)
}
if len(gotMethodsToURLs) != 2 {
t.Fatalf("invalid number of methods: %d", len(gotMethodsToURLs))
}
expectedURLs := []string{
"https://localhost/users",
"https://localhost/users/1?test=asc",
}
require.ElementsMatch(t, gotMethodsToURLs, expectedURLs, "could not get swagger urls")
}

58
pkg/input/formats/testdata/burp.xml vendored Normal file
View File

@ -0,0 +1,58 @@
<?xml version="1.0"?>
<!DOCTYPE items [
<!ELEMENT items (item*)>
<!ATTLIST items burpVersion CDATA "">
<!ATTLIST items exportTime CDATA "">
<!ELEMENT item (time, url, host, port, protocol, method, path, extension, request, status, responselength, mimetype, response, comment)>
<!ELEMENT time (#PCDATA)>
<!ELEMENT url (#PCDATA)>
<!ELEMENT host (#PCDATA)>
<!ATTLIST host ip CDATA "">
<!ELEMENT port (#PCDATA)>
<!ELEMENT protocol (#PCDATA)>
<!ELEMENT method (#PCDATA)>
<!ELEMENT path (#PCDATA)>
<!ELEMENT extension (#PCDATA)>
<!ELEMENT request (#PCDATA)>
<!ATTLIST request base64 (true|false) "false">
<!ELEMENT status (#PCDATA)>
<!ELEMENT responselength (#PCDATA)>
<!ELEMENT mimetype (#PCDATA)>
<!ELEMENT response (#PCDATA)>
<!ATTLIST response base64 (true|false) "false">
<!ELEMENT comment (#PCDATA)>
]>
<items burpVersion="2023.10.1.2" exportTime="Sat Sep 30 20:11:44 IST 2023">
<item>
<time>Sat Sep 30 20:11:32 IST 2023</time>
<url><![CDATA[http://localhost:8087/scans]]></url>
<host ip="127.0.0.1">localhost</host>
<port>8087</port>
<protocol>http</protocol>
<method><![CDATA[POST]]></method>
<path><![CDATA[/scans]]></path>
<extension>null</extension>
<request base64="true"><![CDATA[UE9TVCAvc2NhbnMgSFRUUC8xLjEKSG9zdDogbG9jYWxob3N0OjgwODcKVXNlci1BZ2VudDogR28taHR0cC1jbGllbnQvMS4xCkNvbnRlbnQtTGVuZ3RoOiA5MApDb250ZW50LVR5cGU6IGFwcGxpY2F0aW9uL2pzb24KQWNjZXB0LUVuY29kaW5nOiBnemlwLCBkZWZsYXRlLCBicgpDb25uZWN0aW9uOiBjbG9zZQoKeyJkcnktcnVuIjpmYWxzZSwiaW5zdGFuY2VzIjoxLCJwdWJsaWNfdGVtcGxhdGVzIjpbImRucyJdLCJ0YXJnZXRzIjpbImh0dHBzOi8vc2Nhbm1lLnNoIl19]]></request>
<status>200</status>
<responselength>152</responselength>
<mimetype>JSON</mimetype>
<response base64="true"><![CDATA[SFRUUC8xLjEgMjAwIE9LDQpDb250ZW50LVR5cGU6IGFwcGxpY2F0aW9uL2pzb24NClZhcnk6IE9yaWdpbg0KRGF0ZTogU2F0LCAzMCBTZXAgMjAyMyAxNDo0MTozMiBHTVQNCkNvbnRlbnQtTGVuZ3RoOiAxMQ0KQ29ubmVjdGlvbjogY2xvc2UNCg0KeyJpZCI6IjEifQo=]]></response>
<comment></comment>
</item>
<item>
<time>Sat Sep 30 20:08:54 IST 2023</time>
<url><![CDATA[http://google.com/]]></url>
<host ip="216.58.196.110">google.com</host>
<port>80</port>
<protocol>http</protocol>
<method><![CDATA[GET]]></method>
<path><![CDATA[/]]></path>
<extension>null</extension>
<request base64="true"><![CDATA[R0VUIC8gSFRUUC8xLjENCkhvc3Q6IGdvb2dsZS5jb20NClVzZXItQWdlbnQ6IGN1cmwvOC4xLjINCkFjY2VwdDogKi8qDQpDb25uZWN0aW9uOiBjbG9zZQ0KDQo=]]></request>
<status>301</status>
<responselength>792</responselength>
<mimetype>HTML</mimetype>
<response base64="true"><![CDATA[SFRUUC8xLjEgMzAxIE1vdmVkIFBlcm1hbmVudGx5DQpMb2NhdGlvbjogaHR0cDovL3d3dy5nb29nbGUuY29tLw0KQ29udGVudC1UeXBlOiB0ZXh0L2h0bWw7IGNoYXJzZXQ9VVRGLTgNCkNvbnRlbnQtU2VjdXJpdHktUG9saWN5LVJlcG9ydC1Pbmx5OiBvYmplY3Qtc3JjICdub25lJztiYXNlLXVyaSAnc2VsZic7c2NyaXB0LXNyYyAnbm9uY2Utay1XLW01X282alRxc0t3M1NOUDdZdycgJ3N0cmljdC1keW5hbWljJyAncmVwb3J0LXNhbXBsZScgJ3Vuc2FmZS1ldmFsJyAndW5zYWZlLWlubGluZScgaHR0cHM6IGh0dHA6O3JlcG9ydC11cmkgaHR0cHM6Ly9jc3Aud2l0aGdvb2dsZS5jb20vY3NwL2d3cy9vdGhlci1ocA0KRGF0ZTogU2F0LCAzMCBTZXAgMjAyMyAxNDozODo1NCBHTVQNCkV4cGlyZXM6IE1vbiwgMzAgT2N0IDIwMjMgMTQ6Mzg6NTQgR01UDQpDYWNoZS1Db250cm9sOiBwdWJsaWMsIG1heC1hZ2U9MjU5MjAwMA0KU2VydmVyOiBnd3MNCkNvbnRlbnQtTGVuZ3RoOiAyMTkNClgtWFNTLVByb3RlY3Rpb246IDANClgtRnJhbWUtT3B0aW9uczogU0FNRU9SSUdJTg0KQ29ubmVjdGlvbjogY2xvc2UNCg0KPEhUTUw+PEhFQUQ+PG1ldGEgaHR0cC1lcXVpdj0iY29udGVudC10eXBlIiBjb250ZW50PSJ0ZXh0L2h0bWw7Y2hhcnNldD11dGYtOCI+CjxUSVRMRT4zMDEgTW92ZWQ8L1RJVExFPjwvSEVBRD48Qk9EWT4KPEgxPjMwMSBNb3ZlZDwvSDE+ClRoZSBkb2N1bWVudCBoYXMgbW92ZWQKPEEgSFJFRj0iaHR0cDovL3d3dy5nb29nbGUuY29tLyI+aGVyZTwvQT4uDQo8L0JPRFk+PC9IVE1MPg0K]]></response>
<comment></comment>
</item>
</items>

26
pkg/input/formats/testdata/ginandjuice.proxify.json vendored Normal file
View File

File diff suppressed because one or more lines are too long

269
pkg/input/formats/testdata/ginandjuice.proxify.yaml vendored Normal file
View File

@ -0,0 +1,269 @@
timestamp: 2024-02-20T19:24:13+05:30
url: https://ginandjuice.shop/blog/post?postId=3&source=proxify
request:
header:
Accept-Encoding: gzip
Connection: close
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
host: ginandjuice.shop
method: GET
path: /blog/post
scheme: https
raw: |+
GET /blog/post?postId=3&source=proxify HTTP/1.1
Host: ginandjuice.shop
Accept-Encoding: gzip
Connection: close
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
response:
header:
Content-Encoding: gzip
Content-Type: text/html; charset=utf-8
Date: Tue, 20 Feb 2024 13:54:13 GMT
Set-Cookie: AWSALB=9l3X2bRs5JVQp5cO8o7xLckrdo3FZIQzbS0ga0n4ctfDApb/nn0K6AiSLLAMbG7CCDHqKOj9kQdyj6T2RzBDULszJ+2Oy8KcyuOKhFsCGVTVabnJTkVwhyXLciFt; Expires=Tue, 27 Feb 2024 13:54:13 GMT; Path=/, AWSALBCORS=9l3X2bRs5JVQp5cO8o7xLckrdo3FZIQzbS0ga0n4ctfDApb/nn0K6AiSLLAMbG7CCDHqKOj9kQdyj6T2RzBDULszJ+2Oy8KcyuOKhFsCGVTVabnJTkVwhyXLciFt; Expires=Tue, 27 Feb 2024 13:54:13 GMT; Path=/; SameSite=None; Secure, session=ymwLa8dKmWjLl43BBpQnrp5LkFZraYkp; Secure; HttpOnly; SameSite=None
X-Backend: ea6c1d8c-a8e5-4bef-b8db-879bbb13cf62
X-Frame-Options: SAMEORIGIN
raw: |+
HTTP/1.1 200 OK
Connection: close
Content-Encoding: gzip
Content-Type: text/html; charset=utf-8
Date: Tue, 20 Feb 2024 13:54:13 GMT
Set-Cookie: AWSALB=9l3X2bRs5JVQp5cO8o7xLckrdo3FZIQzbS0ga0n4ctfDApb/nn0K6AiSLLAMbG7CCDHqKOj9kQdyj6T2RzBDULszJ+2Oy8KcyuOKhFsCGVTVabnJTkVwhyXLciFt; Expires=Tue, 27 Feb 2024 13:54:13 GMT; Path=/
Set-Cookie: AWSALBCORS=9l3X2bRs5JVQp5cO8o7xLckrdo3FZIQzbS0ga0n4ctfDApb/nn0K6AiSLLAMbG7CCDHqKOj9kQdyj6T2RzBDULszJ+2Oy8KcyuOKhFsCGVTVabnJTkVwhyXLciFt; Expires=Tue, 27 Feb 2024 13:54:13 GMT; Path=/; SameSite=None; Secure
Set-Cookie: session=ymwLa8dKmWjLl43BBpQnrp5LkFZraYkp; Secure; HttpOnly; SameSite=None
X-Backend: ea6c1d8c-a8e5-4bef-b8db-879bbb13cf62
X-Frame-Options: SAMEORIGIN
---
timestamp: 2024-02-20T19:24:13+05:32
url: https://ginandjuice.shop/users/3
request:
header:
Accept-Encoding: gzip
Authorization: Bearer 3x4mpl3t0k3n
Connection: close
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
host: ginandjuice.shop
method: POST
path: /catalog/product
scheme: https
raw: |+
POST /catalog/product?productId=3 HTTP/1.1
Host: ginandjuice.shop
Authorization: Bearer 3x4mpl3t0k3n
Accept-Encoding: gzip
Connection: close
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
response:
header:
Content-Encoding: gzip
Content-Type: text/html; charset=utf-8
Date: Tue, 20 Feb 2024 13:54:13 GMT
Set-Cookie: AWSALB=Rzm8ZSSL/yQq1mG1SdGmWAahqexWvOcjIhlNKm0wBnk4jvY3Hdy3mRc+NKoMRNJ2RW+FNbtRk7DX+itnfzjMvKNpwtLWWAafxeKybc+v351g0MsLycRNQF5fG78y; Expires=Tue, 27 Feb 2024 13:54:13 GMT; Path=/, AWSALBCORS=Rzm8ZSSL/yQq1mG1SdGmWAahqexWvOcjIhlNKm0wBnk4jvY3Hdy3mRc+NKoMRNJ2RW+FNbtRk7DX+itnfzjMvKNpwtLWWAafxeKybc+v351g0MsLycRNQF5fG78y; Expires=Tue, 27 Feb 2024 13:54:13 GMT; Path=/; SameSite=None; Secure, session=fFcCUjmQguQy820Y8xrnRypp3KBWSPk6; Secure; HttpOnly; SameSite=None
X-Backend: 2235790d-f089-4324-8ac0-f64cc96f2460
X-Frame-Options: SAMEORIGIN
body: |
<!DOCTYPE html>
<html>
<head>
<link href=/resources/labheader/css/scanMeHeader.css rel=stylesheet>
<link href=/resources/css/labsScanme.css rel=stylesheet>
<meta name="viewport" content="width=device-width, user-scalable=no">
<script src="/resources/js/react.development.js"></script>
<script src="/resources/js/react-dom.development.js"></script>
<script type="text/javascript" src="/resources/js/angular_1-7-7.js"></script>
<title>Fruit Overlays - Product - Gin &amp; Juice Shop</title>
</head>
<body ng-app>
<div id="scanMeHeader">
<section class="header-description">
<p>
This is a deliberately vulnerable web application designed for testing web&nbsp;vulnerability&nbsp;scanners.
<span class="link" onmouseenter="window.__x1 = 1" onmouseover="window.__x2 = 1" onmousemove="window.__x3 = 1" onmousedown="window.__x4 = 1" onmouseup="if (window.__x1 && window.__x2 && window.__x3 && window.__x4) location = atob('L3Z1bG5lcmFiaWxpdGllcw==')" onmouseleave="delete window.__x1; delete window.__x2; delete window.__x3; delete window.__x4">Put your scanner to the test!</span>
</p>
</section>
<section class='scanMeBanner'>
<div class=container>
<a href='/'>
<div class=scanme-logo></div>
</a>
<div class=title-container>
<nav>
<ul class="navigation-header-links primary-links">
<li>
<a class="button selected" href="/catalog">Products</a>
</li>
<li>
<a class="button" href="/blog">Blog</a>
</li>
<li>
<a class="button" href="/about">Our story</a>
</li>
</ul>
<ul class="navigation-header-links secondary-links">
<li>
<a class="account-icon" href="/my-account"><svg><use href="/resources/images/icon-account.svg#account-icon"></use></svg></a>
<ul>
<li>
<a class="button" href="/my-account">Log in</a>
</li>
<li>
<a class="button" href="/my-account">My account</a>
</li>
</ul>
</li>
<li>
<a class="cart-icon" href="/catalog/cart"><span>0</span><svg><use href="/resources/images/icon-cart.svg#cart-icon"></use></svg></a>
</li>
<li class="nav-toggle"><a class="nav-trigger"><span></span><span></span><span></span></a></li>
</ul>
</nav>
</div>
</div>
</section>
</div>
<div theme="ecommerce-product">
<section class="maincontainer">
<div class="container is-page">
<header class="notification-header">
</header>
<ul class="breadcrumbs">
<li><a href="/">Home</a></li>
<li><a href="/catalog">Products</a></li>
<li>Fruit Overlays</li>
</ul>
<section class="product">
<div>
<img class="product-image" src="/image/scanme/productcatalog/products/10.png">
</div>
<div>
<h3>Fruit Overlays</h3>
<img class="product-image" src="/image/scanme/productcatalog/products/10.png">
<span class="price-rating">
<span class="price">
$92.79
</span>
<img src="/resources/images/rating3.png">
</span>
<span class="description">
<label>Description:</label>
<p>When it comes to hospitality presentation is key, and nothing looks better than a well-dressed drink. We know gin fans like plenty of fruit in their glasses, some a bit more than they really need, but hey ho, each to their own. But what about fruit not inside your glass, but classily arranged over your glass? In comes Fruitus overlayus, the best way to jazz up any party. The possible colour combinations are endless, just picture that! All you need is a nice selection of small fruits, or maybe even use our Fruit Curliwurlier to add a dash of even more drama, and we will do the rest. This one is a real winner at our Christmas and New year’s outings, give it a go and turn some heads.</p>
<p>CONTENTS: 12 cocktail sticks.</p>
<p>HOW TO USE: Let your creative juices flow (Pun intended), and spend some time working on your colour coordination, try not to think too much about it, just do it! Pick up one of the Fruitus overlayus sticks and carefully slide the fruit along until there is a small space on either end of the stick. Balance the stick across the rim of the glass. Ta-Da! Your first fruit overlay. Keep going until you have as many overlays as you need. You can always purchase more at any time with a discount on bulk buys.</p>
</span>
<span class="stock-check">
<form id="stockCheckForm" action="/catalog/product/stock" method="POST">
<input required type="hidden" name="productId" value="3">
<select name="storeId">
<option value="1" >London</option>
<option value="2" >Paris</option>
<option value="3" >Milan</option>
</select>
<button type="submit" class="button">Check stock</button>
</form>
<span id="stockCheckResult"></span>
<script src="/resources/js/xmlStockCheckPayload.js"></script>
<script src="/resources/js/stockCheck.js"></script>
</span>
<span class="cart-button">
<form id=addToCartForm action=/catalog/cart method=POST>
<input required type=hidden name=productId value=3>
<input required type=hidden name=redir value=PRODUCT>
<select class='product-quantity' required name=quantity>
<option value="1" selected>1</option>
<option value="2">2</option>
<option value="3">3</option>
<option value="4">4</option>
<option value="5">5</option>
<option value="6">6</option>
<option value="7">7</option>
<option value="8">8</option>
<option value="9">9</option>
<option value="10">10</option>
<option value="11">11</option>
<option value="12">12</option>
<option value="13">13</option>
<option value="14">14</option>
<option value="15">15</option>
</select>
<button type=submit class=button>Add to cart</button>
</form>
</span>
<span class="view-cart-button">
<a class="button" href="/catalog/cart">View cart</a>
</span>
</div>
</section>
</div>
</section>
<div class="footer-wrapper">
<section class="footer">
<div class="footer-left"></div>
<div class="footer-center">
<h2>Never miss a deal - subscribe now</h2>
<p>Join our worldwide community of gin and juice fanatics, for exclusive news on our latest deals, new releases, collaborations, and more.</p>
<script src='/resources/js/subscribeNow.js'></script>
<div id="subscribe" class="form" data-method="post" data-action="/catalog/subscribe">
<input required type=email name=email placeholder="Email address">
<input required type="hidden" name="csrf" value="ALUrIPu21ygHSGadxsA8u70XnVcY4V4k">
<button class="button" type=submit>Subscribe</button>
</div>
<dialog id="coupon-dialog">
<div class="coupon-wrapper">
<button class="close-button" onclick="closeCouponDialog(event)"></button>
<div class="coupon-info">
<h1>20% off everything</h1>
<div class="coupon-input">
<h3 id="copyable-coupon">Coupon not found</h3>
<button id="copy-coupon-button" class="copy-button" onclick="copyCoupon(event)"></button>
<div id="coupon-copied-tick" class="coupon-copied-tick hidden"></div>
</div>
<p>Apply this coupon to your Shopping Cart before placing your order.</p>
</div>
</div>
</dialog>
<div class="footer-copyright">
<div class="portswigger-logo"></div>
<div>© 2023 PortSwigger Ltd.</div>
</div>
</div>
<div class="footer-right"></div>
</section>
<section class="footer-lower">
<div class="footerNavigation">
<div class="socialLinks">
</div>
<nav>
<ul class="navigation-header-links primary-links">
<li>
<a class="button selected" href="/catalog">Products</a>
</li>
<li>
<a class="button" href="/blog">Blog</a>
</li>
<li>
<a class="button" href="/about">Our story</a>
</li>
</ul>
</nav>
</div>
</section>
</div>
</div>
<script src='/resources/footer/js/scanme.js'></script>
</body>
</html>
raw: |+
HTTP/1.1 200 OK
Connection: close
Content-Encoding: gzip
Content-Type: text/html; charset=utf-8
Date: Tue, 20 Feb 2024 13:54:13 GMT
Set-Cookie: AWSALB=Rzm8ZSSL/yQq1mG1SdGmWAahqexWvOcjIhlNKm0wBnk4jvY3Hdy3mRc+NKoMRNJ2RW+FNbtRk7DX+itnfzjMvKNpwtLWWAafxeKybc+v351g0MsLycRNQF5fG78y; Expires=Tue, 27 Feb 2024 13:54:13 GMT; Path=/
Set-Cookie: AWSALBCORS=Rzm8ZSSL/yQq1mG1SdGmWAahqexWvOcjIhlNKm0wBnk4jvY3Hdy3mRc+NKoMRNJ2RW+FNbtRk7DX+itnfzjMvKNpwtLWWAafxeKybc+v351g0MsLycRNQF5fG78y; Expires=Tue, 27 Feb 2024 13:54:13 GMT; Path=/; SameSite=None; Secure
Set-Cookie: session=fFcCUjmQguQy820Y8xrnRypp3KBWSPk6; Secure; HttpOnly; SameSite=None
X-Backend: 2235790d-f089-4324-8ac0-f64cc96f2460
X-Frame-Options: SAMEORIGIN

582
pkg/input/formats/testdata/openapi.yaml vendored Normal file
View File

@ -0,0 +1,582 @@
openapi: 3.1.0
info:
title: VAmPI
description: OpenAPI v3 specs for VAmPI
version: '0.1'
servers:
- url: http://hackthebox:5000
components: {}
paths:
/createdb:
get:
tags:
- db-init
summary: Creates and populates the database with dummy data
description: Creates and populates the database with dummy data
operationId: api_views.main.populate_db
responses:
'200':
description: Creates and populates the database with dummy data
content:
application/json:
schema:
type: object
properties:
message:
type: string
example: 'Database populated.'
/:
get:
tags:
- home
summary: VAmPI home
description: >-
VAmPI is a vulnerable on purpose API. It was created in order to
evaluate the efficiency of third party tools in identifying
vulnerabilities in APIs but it can also be used in learning/teaching
purposes.
operationId: api_views.main.basic
responses:
'200':
description: Home - Help
content:
application/json:
schema:
type: object
properties:
message:
type: string
example: 'VAmPI the Vulnerable API'
help:
type: string
example: 'VAmPI is a vulnerable on purpose API. It was created in order to evaluate the efficiency of third party tools in identifying vulnerabilities in APIs but it can also be used in learning/teaching purposes.'
vulnerable:
type: number
example: 1
/users/v1:
get:
tags:
- users
summary: Retrieves all users
description: Displays all users with basic information
operationId: api_views.users.get_all_users
responses:
'200':
description: See basic info about all users
content:
application/json:
schema:
type: array
items:
type: object
properties:
email:
type: string
example: 'mail1@mail.com'
username:
type: string
example: 'name1'
/users/v1/_debug:
get:
tags:
- users
summary: Retrieves all details for all users
description: Displays all details for all users
operationId: api_views.users.debug
responses:
'200':
description: See all details of the users
content:
application/json:
schema:
type: array
items:
type: object
properties:
admin:
type: boolean
example: false
email:
type: string
example: 'mail1@mail.com'
password:
type: string
example: 'pass1'
username:
type: string
example: 'name1'
/users/v1/register:
post:
tags:
- users
summary: Register new user
description: Register new user
operationId: api_views.users.register_user
requestBody:
description: Username of the user
content:
application/json:
schema:
type: object
properties:
username:
type: string
example: 'John.Doe'
password:
type: string
example: 'password123'
email:
type: string
example: 'user@tempmail.com'
required: true
responses:
'200':
description: Sucessfully created user
content:
application/json:
schema:
type: object
properties:
message:
type: string
example: 'Successfully registered. Login to receive an auth token.'
status:
type: string
enum: ['success', 'fail']
example: 'success'
'400':
description: Invalid request
content: {}
/users/v1/login:
post:
tags:
- users
summary: Login to VAmPI
description: Login to VAmPI
operationId: api_views.users.login_user
requestBody:
description: Username of the user
content:
application/json:
schema:
type: object
properties:
username:
type: string
example: 'John.Doe'
password:
type: string
example: 'password123'
required: true
responses:
'200':
description: Sucessfully logged in user
content:
application/json:
schema:
type: object
properties:
auth_token:
type: string
example: 'eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE2NzAxNjA2MTcsImlhdCI6MTY3MDE2MDU1Nywic3ViIjoiSm9obi5Eb2UifQ.n17N4AxTbL4_z65-NR46meoytauPDjImUxrLiUMSTQw'
message:
type: string
example: 'Successfully logged in.'
status:
type: string
enum: ['success', 'fail']
example: 'success'
'400':
description: Invalid request
content:
application/json:
schema:
type: object
properties:
status:
type: string
enum: ['fail']
example: 'fail'
message:
type: string
example: 'Password is not correct for the given username.'
/users/v1/{username}:
get:
tags:
- users
summary: Retrieves user by username
description: Displays user by username
operationId: api_views.users.get_by_username
parameters:
- name: username
in: path
description: retrieve username data
required: true
schema:
type: string
example: 'John.Doe'
responses:
'200':
description: Successfully display user info
content:
application/json:
schema:
type: array
items:
type: object
properties:
username:
type: string
example: 'John.Doe'
email:
type: string
example: 'user@tempmail.com'
'404':
description: User not found
content:
application/json:
schema:
type: object
properties:
status:
type: string
enum: ['fail']
example: 'fail'
message:
type: string
example: 'User not found'
delete:
tags:
- users
summary: Deletes user by username (Only Admins)
description: Deletes user by username (Only Admins)
operationId: api_views.users.delete_user
parameters:
- name: username
in: path
description: Delete username
required: true
schema:
type: string
example: 'name1'
responses:
'200':
description: Sucessfully deleted user
content:
application/json:
schema:
type: object
properties:
message:
type: string
example: 'User deleted.'
status:
type: string
enum: ['success', 'fail']
example: 'success'
'401':
description: User not authorized
content:
application/json:
schema:
type: object
properties:
status:
type: string
example: 'fail'
enum: ['fail']
message:
type: string
example: 'Only Admins may delete users!'
'404':
description: User not found
content:
application/json:
schema:
type: object
properties:
status:
type: string
example: 'fail'
enum: ['fail']
message:
type: string
example: 'User not found!'
/users/v1/{username}/email:
put:
tags:
- users
summary: Update users email
description: Update a single users email
operationId: api_views.users.update_email
parameters:
- name: username
in: path
description: username to update email
required: true
schema:
type: string
example: 'name1'
requestBody:
description: field to update
content:
application/json:
schema:
type: object
properties:
email:
type: string
example: 'mail3@mail.com'
required: true
responses:
'204':
description: Sucessfully updated user email
content: {}
'400':
description: Invalid request
content:
application/json:
schema:
type: object
properties:
status:
type: string
enum: ['fail']
example: 'fail'
message:
type: string
example: 'Please Provide a valid email address.'
'401':
description: User not authorized
content:
application/json:
schema:
type: object
properties:
status:
type: string
enum: ['fail']
example: 'fail'
message:
type: string
example: 'Invalid Token'
/users/v1/{username}/password:
put:
tags:
- users
summary: Update users password
description: Update users password
operationId: api_views.users.update_password
parameters:
- name: username
in: path
description: username to update password
required: true
schema:
type: string
example: 'name1'
requestBody:
description: field to update
content:
application/json:
schema:
type: object
properties:
password:
type: string
example: 'pass4'
required: true
responses:
'204':
description: Sucessfully updated users password
content: {}
'400':
description: Invalid request
content:
application/json:
schema:
type: object
properties:
status:
type: string
enum: ['fail']
example: 'fail'
message:
type: string
example: 'Malformed Data'
'401':
description: User not authorized
content:
application/json:
schema:
type: object
properties:
status:
type: string
enum: ['fail']
example: 'fail'
message:
type: string
example: 'Invalid Token'
/books/v1:
get:
tags:
- books
summary: Retrieves all books
description: Retrieves all books
operationId: api_views.books.get_all_books
responses:
'200':
description: See all books
content:
application/json:
schema:
type: object
properties:
Books:
type: array
items:
type: object
properties:
book_title:
type: string
user:
type: string
example:
Books:
- book_title: 'bookTitle77'
user: 'name1'
- book_title: 'bookTitle85'
user: 'name2'
- book_title: 'bookTitle47'
user: 'admin'
post:
tags:
- books
summary: Add new book
description: Add new book
operationId: api_views.books.add_new_book
requestBody:
description: >-
Add new book with title and secret content only available to the user
who added it.
content:
application/json:
schema:
type: object
properties:
book_title:
type: string
example: 'book99'
secret:
type: string
example: 'pass1secret'
required: true
responses:
'200':
description: Sucessfully added a book
content:
application/json:
schema:
type: object
properties:
message:
type: string
example: 'Book has been added.'
status:
type: string
enum: ['success', 'fail']
example: 'success'
'400':
description: Invalid request
content:
application/json:
schema:
type: object
properties:
status:
type: string
enum: ['fail']
example: 'fail'
message:
type: string
example: 'Book Already exists!'
'401':
description: User not authorized
content:
application/json:
schema:
type: object
properties:
status:
type: string
enum: ['fail']
example: 'fail'
message:
type: string
example: 'Invalid Token'
/books/v1/{book_title}:
get:
tags:
- books
summary: Retrieves book by title along with secret
description: >-
Retrieves book by title along with secret. Only the owner may retrieve
it
operationId: api_views.books.get_by_title
parameters:
- name: book_title
in: path
description: retrieve book data
required: true
schema:
type: string
example: 'bookTitle77'
responses:
'200':
description: Successfully retrieve book info
content:
application/json:
schema:
type: array
items:
type: object
properties:
book_title:
type: string
example: 'bookTitle77'
owner:
type: string
example: 'name1'
secret:
type: string
example: 'secret for bookTitle77'
'401':
description: User not authorized
content:
application/json:
schema:
type: object
properties:
status:
type: string
enum: ['fail']
example: 'fail'
message:
type: string
example: 'Invalid Token'
'404':
description: Book not found
content:
application/json:
schema:
type: object
properties:
status:
type: string
enum: ['fail']
example: 'fail'
message:
type: string
example: 'Book not found!'

159
pkg/input/formats/testdata/postman.json vendored Normal file
View File

@ -0,0 +1,159 @@
{
"info": {
"_postman_id": "20a3fd41-6a86-4e49-8860-f796559d0223",
"name": "advancedsearch",
"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
},
"item": [
{
"name": "List Projects, Assets and Hosts",
"protocolProfileBehavior": {
"disableBodyPruning": true
},
"request": {
"method": "GET",
"header": [
{
"key": "Content-Type",
"name": "Content-Type",
"value": "application/json",
"type": "text"
}
],
"body": {
"mode": "raw",
"raw": "",
"options": {
"raw": {
"language": "json"
}
}
},
"url": {
"raw": "http://127.0.0.1:8000/api/v1/search/",
"protocol": "http",
"host": ["127", "0", "0", "1"],
"port": "8000",
"path": ["api", "v1", "search", ""]
}
},
"response": []
},
{
"name": "List Assets and Hosts",
"protocolProfileBehavior": {
"disableBodyPruning": true
},
"request": {
"method": "GET",
"header": [
{
"key": "Content-Type",
"name": "Content-Type",
"type": "text",
"value": "application/json"
}
],
"body": {
"mode": "raw",
"raw": "",
"options": {
"raw": {
"language": "json"
}
}
},
"url": {
"raw": "http://127.0.0.1:8000/api/v1/search/?projectId=1,2",
"protocol": "http",
"host": ["127", "0", "0", "1"],
"port": "8000",
"path": ["api", "v1", "search", ""],
"query": [
{
"key": "projectId",
"value": "1,2"
}
]
}
},
"response": []
},
{
"name": "List Hosts",
"protocolProfileBehavior": {
"disableBodyPruning": true
},
"request": {
"method": "GET",
"header": [
{
"key": "Content-Type",
"name": "Content-Type",
"type": "text",
"value": "application/json"
}
],
"body": {
"mode": "raw",
"raw": "",
"options": {
"raw": {
"language": "json"
}
}
},
"url": {
"raw": "http://127.0.0.1:8000/api/v1/search/?projectId=1,2&assetId=1,2",
"protocol": "http",
"host": ["127", "0", "0", "1"],
"port": "8000",
"path": ["api", "v1", "search", ""],
"query": [
{
"key": "projectId",
"value": "1,2"
},
{
"key": "assetId",
"value": "1,2"
}
]
}
},
"response": []
},
{
"name": "Search Request",
"request": {
"method": "POST",
"header": [
{
"key": "Content-Type",
"name": "Content-Type",
"value": "application/json",
"type": "text"
}
],
"body": {
"mode": "raw",
"raw": "{\n\t\"query\": \"query\",\n\t\"projectId\": [4,3,4],\n\t\"assetId\": [2,3,4],\n\t\"hostId\": [1,2,3],\n \"limit\": 10,\n \"offset\": 10\n}",
"options": {
"raw": {
"language": "json"
}
}
},
"url": {
"raw": "http://127.0.0.1:8000/api/v1/search/",
"protocol": "http",
"host": ["127", "0", "0", "1"],
"port": "8000",
"path": ["api", "v1", "search", ""]
}
},
"response": []
}
],
"protocolProfileBehavior": {}
}

37
pkg/input/formats/testdata/swagger.yaml vendored Normal file
View File

@ -0,0 +1,37 @@
swagger: "2.0"
info:
title: Sample API
description: API description in Markdown.
version: 1.0.0
host: localhost
basePath: /v1
schemes:
- https
paths:
/users:
get:
summary: Returns a list of users.
description: Optional extended description in Markdown.
produces:
- application/json
responses:
200:
description: OK
/users/{userId}:
get:
summary: Returns a user by ID.
parameters:
- in: path
name: userId
required: true
type: integer
default: 1
description: Parameter description in Markdown.
- in: query
name: test
type: string
enum: [asc, desc]
description: Type of query
responses:
200:
description: OK

Some files were not shown because too many files have changed in this diff Show More