Added part definition information to docs + misc

dev
Ice3man543 2021-11-26 16:23:54 +05:30
parent caaa5c6594
commit c720354be2
12 changed files with 537 additions and 1 deletions

View File

@ -9,6 +9,8 @@ Template is a YAML input file which defines all the requests and
<hr />
<div class="dd">
@ -290,6 +292,8 @@ reference: https://zxsecurity.co.nz/research/argunment-injection-ruby-dragonfly/
severity: high
```
<hr />
<div class="dd">
@ -545,6 +549,8 @@ CWE-22
## severity.Holder
Holder holds a Severity type. Required for un/marshalling purposes
@ -555,6 +561,8 @@ Appears in:
<hr />
<div class="dd">
@ -598,6 +606,8 @@ Appears in:
<hr />
<div class="dd">
@ -724,6 +734,26 @@ path:
method: GET
```
Part Definitions:
- <code>template-id</code> - ID of the template executed
- <code>template-info</code> - Info Block of the template executed
- <code>template-path</code> - Path of the template executed
- <code>host</code> - Host is the input to the template
- <code>matched</code> - Matched is the input which was matched upon
- <code>type</code> - Type is the type of request made
- <code>request</code> - HTTP request made from the client
- <code>response</code> - HTTP response recieved from server
- <code>status_code</code> - Status Code received from the Server
- <code>body</code> - HTTP response body received from server (default)
- <code>content_length</code> - HTTP Response content length
- <code>header,all_headers</code> - HTTP response headers
- <code>duration</code> - HTTP request time duration
- <code>all</code> - HTTP response body + headers
- <code><cookies_from_response></code> - HTTP response cookies in <name>:<value> format
- <code><headers_from_response></code> - HTTP response headers in <name>:<value> format
<hr />
<div class="dd">
@ -1266,6 +1296,8 @@ Appears in:
<hr />
<div class="dd">
@ -1599,6 +1631,8 @@ Appears in:
<hr />
<div class="dd">
@ -1655,6 +1689,8 @@ Appears in:
<hr />
<div class="dd">
@ -1935,6 +1971,8 @@ Appears in:
<hr />
<div class="dd">
@ -1979,6 +2017,8 @@ Appears in:
<hr />
<div class="dd">
@ -2017,6 +2057,8 @@ Appears in:
<hr />
<div class="dd">
@ -2081,6 +2123,24 @@ retries: 2
recursion: true
```
Part Definitions:
- <code>template-id</code> - ID of the template executed
- <code>template-info</code> - Info Block of the template executed
- <code>template-path</code> - Path of the template executed
- <code>host</code> - Host is the input to the template
- <code>matched</code> - Matched is the input which was matched upon
- <code>request</code> - Request contains the DNS request in text format
- <code>type</code> - Type is the type of request made
- <code>rcode</code> - Rcode field returned for the DNS request
- <code>question</code> - Question contains the DNS question field
- <code>extra</code> - Extra contains the DNS response extra field
- <code>answer</code> - Answer contains the DNS response answer field
- <code>ns</code> - NS contains the DNS response NS field
- <code>raw,body,all</code> - Raw contains the raw DNS response (default)
- <code>trace</code> - Trace contains trace data for DNS request if enabled
<hr />
<div class="dd">
@ -2317,6 +2377,8 @@ Appears in:
<hr />
<div class="dd">
@ -2375,6 +2437,17 @@ extensions:
- all
```
Part Definitions:
- <code>template-id</code> - ID of the template executed
- <code>template-info</code> - Info Block of the template executed
- <code>template-path</code> - Path of the template executed
- <code>matched</code> - Matched is the input which was matched upon
- <code>path</code> - Path is the path of file on local filesystem
- <code>type</code> - Type is the type of request made
- <code>raw,body,all,data</code> - Raw contains the raw file contents
<hr />
<div class="dd">
@ -2562,6 +2635,19 @@ matchers:
- zookeeper.version
```
Part Definitions:
- <code>template-id</code> - ID of the template executed
- <code>template-info</code> - Info Block of the template executed
- <code>template-path</code> - Path of the template executed
- <code>host</code> - Host is the input to the template
- <code>matched</code> - Matched is the input which was matched upon
- <code>type</code> - Type is the type of request made
- <code>request</code> - Network request made from the client
- <code>body,all,data</code> - Network response recieved from server (default)
- <code>raw</code> - Full Network protocol data
<hr />
<div class="dd">
@ -2765,6 +2851,8 @@ Appears in:
<hr />
<div class="dd">
@ -2884,6 +2972,8 @@ Appears in:
<hr />
<div class="dd">
@ -2920,6 +3010,18 @@ Appears in:
Part Definitions:
- <code>template-id</code> - ID of the template executed
- <code>template-info</code> - Info Block of the template executed
- <code>template-path</code> - Path of the template executed
- <code>host</code> - Host is the input to the template
- <code>matched</code> - Matched is the input which was matched upon
- <code>type</code> - Type is the type of request made
- <code>req</code> - Headless request made from the client
- <code>resp,body,data</code> - Headless response recieved from client (default)
<hr />
<div class="dd">
@ -3019,6 +3121,8 @@ Appears in:
<hr />
<div class="dd">
@ -3092,6 +3196,8 @@ Appears in:
<hr />
<div class="dd">
@ -3168,6 +3274,15 @@ Appears in:
Part Definitions:
- <code>type</code> - Type is the type of request made
- <code>response</code> - JSON SSL protocol handshake details
- <code>not_after</code> - Timestamp after which the remote cert expires
- <code>host</code> - Host is the input to the template
- <code>matched</code> - Matched is the input which was matched upon
<hr />
<div class="dd">
@ -3249,6 +3364,16 @@ Appears in:
Part Definitions:
- <code>type</code> - Type is the type of request made
- <code>success</code> - Success specifies whether websocket connection was successful
- <code>request</code> - Websocket request made to the server
- <code>response</code> - Websocket response recieved from the server
- <code>host</code> - Host is the input to the template
- <code>matched</code> - Matched is the input which was matched upon
<hr />
<div class="dd">
@ -3388,6 +3513,8 @@ Appears in:
<hr />
<div class="dd">
@ -3459,6 +3586,8 @@ Appears in:
<hr />
<div class="dd">
@ -3542,6 +3671,8 @@ Appears in:
<hr />
<div class="dd">

View File

@ -40,7 +40,7 @@ require (
github.com/projectdiscovery/retryabledns v1.0.13-0.20211109182249-43d38df59660
github.com/projectdiscovery/retryablehttp-go v1.0.2
github.com/projectdiscovery/stringsutil v0.0.0-20210830151154-f567170afdd9
github.com/projectdiscovery/yamldoc-go v1.0.3-0.20211125140040-b396ca47606e
github.com/projectdiscovery/yamldoc-go v1.0.3-0.20211126104922-00d2c6bb43b6
github.com/remeh/sizedwaitgroup v1.0.0
github.com/rs/xid v1.3.0
github.com/segmentio/ksuid v1.0.4

View File

@ -656,6 +656,8 @@ github.com/projectdiscovery/yamldoc-go v1.0.3-0.20211125135235-2e6dd74132d0 h1:m
github.com/projectdiscovery/yamldoc-go v1.0.3-0.20211125135235-2e6dd74132d0/go.mod h1:7uSxfMXaBmzvw8m5EhOEjB6nhz0rK/H9sUjq1ciZu24=
github.com/projectdiscovery/yamldoc-go v1.0.3-0.20211125140040-b396ca47606e h1:0ZxOM0Q0/ESa24L/vq3fxs9YipxfHR4Y3jM/H2ReJ5E=
github.com/projectdiscovery/yamldoc-go v1.0.3-0.20211125140040-b396ca47606e/go.mod h1:7uSxfMXaBmzvw8m5EhOEjB6nhz0rK/H9sUjq1ciZu24=
github.com/projectdiscovery/yamldoc-go v1.0.3-0.20211126104922-00d2c6bb43b6 h1:DvWRQpw7Ib2CRL3ogYm/BWM+X0UGPfz1n9Ix9YKgFM8=
github.com/projectdiscovery/yamldoc-go v1.0.3-0.20211126104922-00d2c6bb43b6/go.mod h1:8OfZj8p/axkUM/TJoS/O9LDjj/S8u17rxRbqluE9CU4=
github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
github.com/prometheus/client_golang v0.9.3-0.20190127221311-3c4408c8b829/go.mod h1:p2iRAGwDERtqlqzRXnrOVns+ignqQo//hLXqYxZYVNs=
github.com/prometheus/client_golang v1.0.0/go.mod h1:db9x61etRT2tGnBNRi70OPL5FsnadC4Ky3P0J6CfImo=

View File

@ -78,6 +78,26 @@ type Request struct {
Resolvers []string `yaml:"resolvers,omitempty" jsonschema:"title=Resolvers,description=Define resolvers to use within the template"`
}
// RequestPartDefinitions contains a mapping of request part definitions and their
// description. Multiple definitions are separated by commas.
// Definitions not having a name (generated on runtime) are prefixed & suffixed by <>.
var RequestPartDefinitions = map[string]string{
"template-id": "ID of the template executed",
"template-info": "Info Block of the template executed",
"template-path": "Path of the template executed",
"host": "Host is the input to the template",
"matched": "Matched is the input which was matched upon",
"request": "Request contains the DNS request in text format",
"type": "Type is the type of request made",
"rcode": "Rcode field returned for the DNS request",
"question": "Question contains the DNS question field",
"extra": "Extra contains the DNS response extra field",
"answer": "Answer contains the DNS response answer field",
"ns": "NS contains the DNS response NS field",
"raw,body,all": "Raw contains the raw DNS response (default)",
"trace": "Trace contains trace data for DNS request if enabled",
}
func (request *Request) GetCompiledOperators() []*operators.Operators {
return []*operators.Operators{request.CompiledOperators}
}

View File

@ -52,6 +52,19 @@ type Request struct {
allExtensions bool
}
// RequestPartDefinitions contains a mapping of request part definitions and their
// description. Multiple definitions are separated by commas.
// Definitions not having a name (generated on runtime) are prefixed & suffixed by <>.
var RequestPartDefinitions = map[string]string{
"template-id": "ID of the template executed",
"template-info": "Info Block of the template executed",
"template-path": "Path of the template executed",
"matched": "Matched is the input which was matched upon",
"path": "Path is the path of file on local filesystem",
"type": "Type is the type of request made",
"raw,body,all,data": "Raw contains the raw file contents",
}
// defaultDenylist is the default list of extensions to be denied
var defaultDenylist = []string{".3g2", ".3gp", ".7z", ".apk", ".arj", ".avi", ".axd", ".bmp", ".css", ".csv", ".deb", ".dll", ".doc", ".drv", ".eot", ".exe", ".flv", ".gif", ".gifv", ".gz", ".h264", ".ico", ".iso", ".jar", ".jpeg", ".jpg", ".lock", ".m4a", ".m4v", ".map", ".mkv", ".mov", ".mp3", ".mp4", ".mpeg", ".mpg", ".msi", ".ogg", ".ogm", ".ogv", ".otf", ".pdf", ".pkg", ".png", ".ppt", ".psd", ".rar", ".rm", ".rpm", ".svg", ".swf", ".sys", ".tar.gz", ".tar", ".tif", ".tiff", ".ttf", ".vob", ".wav", ".webm", ".wmv", ".woff", ".woff2", ".xcf", ".xls", ".xlsx", ".zip"}

View File

@ -25,6 +25,20 @@ type Request struct {
options *protocols.ExecuterOptions
}
// RequestPartDefinitions contains a mapping of request part definitions and their
// description. Multiple definitions are separated by commas.
// Definitions not having a name (generated on runtime) are prefixed & suffixed by <>.
var RequestPartDefinitions = map[string]string{
"template-id": "ID of the template executed",
"template-info": "Info Block of the template executed",
"template-path": "Path of the template executed",
"host": "Host is the input to the template",
"matched": "Matched is the input which was matched upon",
"type": "Type is the type of request made",
"req": "Headless request made from the client",
"resp,body,data": "Headless response recieved from client (default)",
}
// Step is a headless protocol request step.
type Step struct {
// Action is the headless action to execute for the script

View File

@ -167,6 +167,28 @@ type Request struct {
SkipVariablesCheck bool `yaml:"skip-variables-check,omitempty" jsonschema:"title=skip variable checks,description=Skips the check for unresolved variables in request"`
}
// RequestPartDefinitions contains a mapping of request part definitions and their
// description. Multiple definitions are separated by commas.
// Definitions not having a name (generated on runtime) are prefixed & suffixed by <>.
var RequestPartDefinitions = map[string]string{
"template-id": "ID of the template executed",
"template-info": "Info Block of the template executed",
"template-path": "Path of the template executed",
"host": "Host is the input to the template",
"matched": "Matched is the input which was matched upon",
"type": "Type is the type of request made",
"request": "HTTP request made from the client",
"response": "HTTP response recieved from server",
"status_code": "Status Code received from the Server",
"body": "HTTP response body received from server (default)",
"content_length": "HTTP Response content length",
"header,all_headers": "HTTP response headers",
"duration": "HTTP request time duration",
"all": "HTTP response body + headers",
"<cookies_from_response>": "HTTP response cookies in <name>:<value> format",
"<headers_from_response>": "HTTP response headers in <name>:<value> format",
}
// GetID returns the unique ID of the request if any.
func (request *Request) GetID() string {
return request.ID

View File

@ -78,6 +78,21 @@ type Request struct {
dynamicValues map[string]interface{}
}
// RequestPartDefinitions contains a mapping of request part definitions and their
// description. Multiple definitions are separated by commas.
// Definitions not having a name (generated on runtime) are prefixed & suffixed by <>.
var RequestPartDefinitions = map[string]string{
"template-id": "ID of the template executed",
"template-info": "Info Block of the template executed",
"template-path": "Path of the template executed",
"host": "Host is the input to the template",
"matched": "Matched is the input which was matched upon",
"type": "Type is the type of request made",
"request": "Network request made from the client",
"body,all,data": "Network response recieved from server (default)",
"raw": "Full Network protocol data",
}
type addressKV struct {
ip string
port string

View File

@ -13,6 +13,28 @@ type Request struct {
compiledOperators []*operators.Operators
}
// RequestPartDefinitions contains a mapping of request part definitions and their
// description. Multiple definitions are separated by commas.
// Definitions not having a name (generated on runtime) are prefixed & suffixed by <>.
var RequestPartDefinitions = map[string]string{
"template-id": "ID of the template executed",
"template-info": "Info Block of the template executed",
"template-path": "Path of the template executed",
"host": "Host is the input to the template",
"matched": "Matched is the input which was matched upon",
"type": "Type is the type of request made",
"request": "HTTP request made from the client",
"response": "HTTP response recieved from server",
"status_code": "Status Code received from the Server",
"body": "HTTP response body received from server (default)",
"content_length": "HTTP Response content length",
"header,all_headers": "HTTP response headers",
"duration": "HTTP request time duration",
"all": "HTTP response body + headers",
"<cookies_from_response>": "HTTP response cookies in <name>:<value> format",
"<headers_from_response>": "HTTP response headers in <name>:<value> format",
}
// GetID returns the unique ID of the request if any.
func (request *Request) GetID() string {
return ""

View File

@ -145,6 +145,17 @@ func (request *Request) ExecuteWithResults(input string, dynamicValues, previous
return nil
}
// RequestPartDefinitions contains a mapping of request part definitions and their
// description. Multiple definitions are separated by commas.
// Definitions not having a name (generated on runtime) are prefixed & suffixed by <>.
var RequestPartDefinitions = map[string]string{
"type": "Type is the type of request made",
"response": "JSON SSL protocol handshake details",
"not_after": "Timestamp after which the remote cert expires",
"host": "Host is the input to the template",
"matched": "Matched is the input which was matched upon",
}
// getAddress returns the address of the host to make request to
func getAddress(toTest string) (string, error) {
if strings.Contains(toTest, "://") {

View File

@ -357,6 +357,18 @@ func (request *Request) GetCompiledOperators() []*operators.Operators {
return []*operators.Operators{request.CompiledOperators}
}
// RequestPartDefinitions contains a mapping of request part definitions and their
// description. Multiple definitions are separated by commas.
// Definitions not having a name (generated on runtime) are prefixed & suffixed by <>.
var RequestPartDefinitions = map[string]string{
"type": "Type is the type of request made",
"success": "Success specifies whether websocket connection was successful",
"request": "Websocket request made to the server",
"response": "Websocket response recieved from the server",
"host": "Host is the input to the template",
"matched": "Matched is the input which was matched upon",
}
func (request *Request) MakeResultEventItem(wrapped *output.InternalWrappedEvent) *output.ResultEvent {
data := &output.ResultEvent{
TemplateID: types.ToString(request.options.TemplateID),

View File

@ -301,6 +301,72 @@ func init() {
FieldName: "requests",
},
}
HTTPRequestDoc.PartDefinitions = []encoder.KeyValue{
{
Key: "template-id",
Value: "ID of the template executed",
},
{
Key: "template-info",
Value: "Info Block of the template executed",
},
{
Key: "template-path",
Value: "Path of the template executed",
},
{
Key: "host",
Value: "Host is the input to the template",
},
{
Key: "matched",
Value: "Matched is the input which was matched upon",
},
{
Key: "type",
Value: "Type is the type of request made",
},
{
Key: "request",
Value: "HTTP request made from the client",
},
{
Key: "response",
Value: "HTTP response recieved from server",
},
{
Key: "status_code",
Value: "Status Code received from the Server",
},
{
Key: "body",
Value: "HTTP response body received from server (default)",
},
{
Key: "content_length",
Value: "HTTP Response content length",
},
{
Key: "header,all_headers",
Value: "HTTP response headers",
},
{
Key: "duration",
Value: "HTTP request time duration",
},
{
Key: "all",
Value: "HTTP response body + headers",
},
{
Key: "<cookies_from_response>",
Value: "HTTP response cookies in <name>:<value> format",
},
{
Key: "<headers_from_response>",
Value: "HTTP response headers in <name>:<value> format",
},
}
HTTPRequestDoc.Fields = make([]encoder.Doc, 26)
HTTPRequestDoc.Fields[0].Name = "matchers"
HTTPRequestDoc.Fields[0].Type = "[]matchers.Matcher"
@ -827,6 +893,64 @@ func init() {
FieldName: "dns",
},
}
DNSRequestDoc.PartDefinitions = []encoder.KeyValue{
{
Key: "template-id",
Value: "ID of the template executed",
},
{
Key: "template-info",
Value: "Info Block of the template executed",
},
{
Key: "template-path",
Value: "Path of the template executed",
},
{
Key: "host",
Value: "Host is the input to the template",
},
{
Key: "matched",
Value: "Matched is the input which was matched upon",
},
{
Key: "request",
Value: "Request contains the DNS request in text format",
},
{
Key: "type",
Value: "Type is the type of request made",
},
{
Key: "rcode",
Value: "Rcode field returned for the DNS request",
},
{
Key: "question",
Value: "Question contains the DNS question field",
},
{
Key: "extra",
Value: "Extra contains the DNS response extra field",
},
{
Key: "answer",
Value: "Answer contains the DNS response answer field",
},
{
Key: "ns",
Value: "NS contains the DNS response NS field",
},
{
Key: "raw,body,all",
Value: "Raw contains the raw DNS response (default)",
},
{
Key: "trace",
Value: "Trace contains trace data for DNS request if enabled",
},
}
DNSRequestDoc.Fields = make([]encoder.Doc, 12)
DNSRequestDoc.Fields[0].Name = "matchers"
DNSRequestDoc.Fields[0].Type = "[]matchers.Matcher"
@ -945,6 +1069,36 @@ func init() {
FieldName: "file",
},
}
FILERequestDoc.PartDefinitions = []encoder.KeyValue{
{
Key: "template-id",
Value: "ID of the template executed",
},
{
Key: "template-info",
Value: "Info Block of the template executed",
},
{
Key: "template-path",
Value: "Path of the template executed",
},
{
Key: "matched",
Value: "Matched is the input which was matched upon",
},
{
Key: "path",
Value: "Path is the path of file on local filesystem",
},
{
Key: "type",
Value: "Type is the type of request made",
},
{
Key: "raw,body,all,data",
Value: "Raw contains the raw file contents",
},
}
FILERequestDoc.Fields = make([]encoder.Doc, 8)
FILERequestDoc.Fields[0].Name = "matchers"
FILERequestDoc.Fields[0].Type = "[]matchers.Matcher"
@ -1008,6 +1162,44 @@ func init() {
FieldName: "network",
},
}
NETWORKRequestDoc.PartDefinitions = []encoder.KeyValue{
{
Key: "template-id",
Value: "ID of the template executed",
},
{
Key: "template-info",
Value: "Info Block of the template executed",
},
{
Key: "template-path",
Value: "Path of the template executed",
},
{
Key: "host",
Value: "Host is the input to the template",
},
{
Key: "matched",
Value: "Matched is the input which was matched upon",
},
{
Key: "type",
Value: "Type is the type of request made",
},
{
Key: "request",
Value: "Network request made from the client",
},
{
Key: "body,all,data",
Value: "Network response recieved from server (default)",
},
{
Key: "raw",
Value: "Full Network protocol data",
},
}
NETWORKRequestDoc.Fields = make([]encoder.Doc, 10)
NETWORKRequestDoc.Fields[0].Name = "id"
NETWORKRequestDoc.Fields[0].Type = "string"
@ -1142,6 +1334,40 @@ func init() {
FieldName: "headless",
},
}
HEADLESSRequestDoc.PartDefinitions = []encoder.KeyValue{
{
Key: "template-id",
Value: "ID of the template executed",
},
{
Key: "template-info",
Value: "Info Block of the template executed",
},
{
Key: "template-path",
Value: "Path of the template executed",
},
{
Key: "host",
Value: "Host is the input to the template",
},
{
Key: "matched",
Value: "Matched is the input which was matched upon",
},
{
Key: "type",
Value: "Type is the type of request made",
},
{
Key: "req",
Value: "Headless request made from the client",
},
{
Key: "resp,body,data",
Value: "Headless response recieved from client (default)",
},
}
HEADLESSRequestDoc.Fields = make([]encoder.Doc, 5)
HEADLESSRequestDoc.Fields[0].Name = "id"
HEADLESSRequestDoc.Fields[0].Type = "string"
@ -1253,6 +1479,28 @@ func init() {
FieldName: "ssl",
},
}
SSLRequestDoc.PartDefinitions = []encoder.KeyValue{
{
Key: "type",
Value: "Type is the type of request made",
},
{
Key: "response",
Value: "JSON SSL protocol handshake details",
},
{
Key: "not_after",
Value: "Timestamp after which the remote cert expires",
},
{
Key: "host",
Value: "Host is the input to the template",
},
{
Key: "matched",
Value: "Matched is the input which was matched upon",
},
}
SSLRequestDoc.Fields = make([]encoder.Doc, 4)
SSLRequestDoc.Fields[0].Name = "matchers"
SSLRequestDoc.Fields[0].Type = "[]matchers.Matcher"
@ -1288,6 +1536,32 @@ func init() {
FieldName: "websocket",
},
}
WEBSOCKETRequestDoc.PartDefinitions = []encoder.KeyValue{
{
Key: "type",
Value: "Type is the type of request made",
},
{
Key: "success",
Value: "Success specifies whether websocket connection was successful",
},
{
Key: "request",
Value: "Websocket request made to the server",
},
{
Key: "response",
Value: "Websocket response recieved from the server",
},
{
Key: "host",
Value: "Host is the input to the template",
},
{
Key: "matched",
Value: "Matched is the input which was matched upon",
},
}
WEBSOCKETRequestDoc.Fields = make([]encoder.Doc, 8)
WEBSOCKETRequestDoc.Fields[0].Name = "matchers"
WEBSOCKETRequestDoc.Fields[0].Type = "[]matchers.Matcher"