Adding support for code templates (#2930)

* Adding support for code templates

* adding support for python, powershell and echo (test)

* removing debug code

* introducing command + trivial trust store mechanism

* updating tests

* adding basic tests

* removing deprecated oracle

* mod tidy

* adding signature proto with debug prints

* removing debug code

* fixing test

* fixing param order

* improving test conditional build

* disable file+offlinehttp+code with cloud

* adding env vars

* removing debug code

* reorganizing test folders

* adding code template test prototype with dummy priv/pub keys

* bump go to 1.20

* fixing go version

* fixing lint errors

* adding fatal on pub-key test failure

* switching to ecdsa asn1

* removing unused signature

* fixing signature

* adding more tests

* extending core with engine args + powershell win test

* adding unsigned code test

* skip template signing in particular test case

* improving test coverage

* refactoring key names + adding already signed algo

* removing debug code

* fixing syntax

* fixing lint issues

* removing test template

* fixing dns tests path

* output fmt

* adding interact

* fixing lint issues

* adding -sign cli helper

* fixing nil pointer + parse inline keys

* making rsa default

* adding code prot. ref

* moving file to correct loc

* moving test

* Issue 3339 headless fuzz (#3790)

* Basic headless fuzzing

* Remove debug statements

* Add integration tests

* Update template

* Fix recognize payload value in matcher

* Update tempalte

* use req.SetURL()

---------

Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>

* Auto Generate Syntax Docs + JSONSchema [Fri Jun  9 00:23:32 UTC 2023] 🤖

* Add headless header and status matchers (#3794)

* add headless header and status matchers

* rename headers as header

* add integration test for header+status

* fix typo

* add retry to py-interactsh integration test

---------

Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
Co-authored-by: Shubham Rasal <shubham@projectdiscovery.io>
Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Dogan Can Bakir <65292895+dogancanbakir@users.noreply.github.com>
Co-authored-by: Tarun Koyalwar <45962551+tarunKoyalwar@users.noreply.github.com>

README.md

@@ -37,7 +37,7 @@
 
 ---
 
-Nuclei is used to send requests across targets based on a template, leading to zero false positives and providing fast scanning on a large number of hosts. Nuclei offers scanning for a variety of protocols, including TCP, DNS, HTTP, SSL, File, Whois, Websocket, Headless etc. With powerful and flexible templating, Nuclei can be used to model all kinds of security checks.
+Nuclei is used to send requests across targets based on a template, leading to zero false positives and providing fast scanning on a large number of hosts. Nuclei offers scanning for a variety of protocols, including TCP, DNS, HTTP, SSL, File, Whois, Websocket, Headless, Code etc. With powerful and flexible templating, Nuclei can be used to model all kinds of security checks.
 
 We have a [dedicated repository](https://github.com/projectdiscovery/nuclei-templates) that houses various type of vulnerability templates contributed by **more than 300** security researchers and engineers.
 
@@ -143,8 +143,8 @@ FILTERING:
    -em, -exclude-matchers string[]    template matchers to exclude in result
    -s, -severity value[]              templates to run based on severity. Possible values: info, low, medium, high, critical, unknown
    -es, -exclude-severity value[]     templates to exclude based on severity. Possible values: info, low, medium, high, critical, unknown
-   -pt, -type value[]                 templates to run based on protocol type. Possible values: dns, file, http, headless, tcp, workflow, ssl, websocket, whois
+   -pt, -type value[]                 templates to run based on protocol type. Possible values: dns, file, http, headless, tcp, workflow, ssl, websocket, whois, code
-   -ept, -exclude-type value[]        templates to exclude based on protocol type. Possible values: dns, file, http, headless, tcp, workflow, ssl, websocket, whois
+   -ept, -exclude-type value[]        templates to exclude based on protocol type. Possible values: dns, file, http, headless, tcp, workflow, ssl, websocket, whois, code
    -tc, -template-condition string[]  templates to run based on expression condition
 
 OUTPUT:

integration_tests/protocols/code/ecdsa-priv-key.pem

@@ -0,0 +1,5 @@
+-----BEGIN EC PRIVATE KEY-----
+MHcCAQEEIEjTOyV8a3ZbhEM1Ti58cQrZNvKEmig+Yw8NKtePvcZ1oAoGCCqGSM49
+AwEHoUQDQgAErRysbgMYhazyMIfpkpvlrtzzCFhqc6zr0aLhXtmtHcJQ8YVhexSx
+nbnzC//84yryOKkBRHOfH+xwrQvZzPbiRw==
+-----END EC PRIVATE KEY-----

integration_tests/protocols/code/ecdsa-pub-key.pem

@@ -0,0 +1,4 @@
+-----BEGIN PUBLIC KEY-----
+MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAErRysbgMYhazyMIfpkpvlrtzzCFhq
+c6zr0aLhXtmtHcJQ8YVhexSxnbnzC//84yryOKkBRHOfH+xwrQvZzPbiRw==
+-----END PUBLIC KEY-----

integration_tests/protocols/code/ps1-snippet.yaml

@@ -0,0 +1,29 @@
+id: ps1-code-snippet
+
+info:
+  name: ps1-code-snippet
+  author: pdteam
+  severity: info
+  tags: code
+  description: |
+    ps1-code-snippet
+
+code:
+  - engine:
+      - powershell
+      - powershell.exe
+    args:
+      - -ExecutionPolicy
+      - Bypass
+      - -File
+    pattern: "*.ps1"
+    source: |
+      $stdin = [Console]::In
+      $line = $stdin.ReadLine()
+      Write-Host "hello from $line"
+    
+    matchers:
+      - type: word
+        words:
+          - "hello from input"
+# digest: 4a0a00473045022023beecb1c4ef5b3b3a4d936a689d0fa5fea35524d23bbc12001fa0b21ca2500b02210082484d006ee0663ba1c8450ff0d10eb053308137af25cde223406c3423c4e5d1

integration_tests/protocols/code/py-env-var.yaml

@@ -0,0 +1,23 @@
+id: py-code-snippet
+
+info:
+  name: py-code-snippet
+  author: pdteam
+  severity: info
+  tags: code
+  description: |
+    py-code-snippet
+
+code:
+  - engine:
+      - py
+      - python3
+    source: |
+      import sys,os
+      print("hello from " + sys.stdin.read() + " " + os.getenv('baz'))
+    
+    matchers:
+      - type: word
+        words:
+          - "hello from input baz"
+# digest: 4b0a00483046022100f0a31212d9950a839f1f4e92893dede0a0f26d99abeb0676bf9654009b180f16022100c402c89514be47a23c642d229fcb90bc5e686f9c14226ac97587c375f1c790fa

integration_tests/protocols/code/py-file.yaml

@@ -0,0 +1,21 @@
+id: py-file
+
+info:
+  name: py-file
+  author: pdteam
+  severity: info
+  tags: code
+  description: |
+    py-file
+
+code:
+  - engine:
+      - py
+      - python3
+    source: protocols/code/pyfile.py
+    
+    matchers:
+      - type: word
+        words:
+          - "hello from input"
+# digest: 490a00463044022034180b19f4aa1e311500144a637db838cdbeb694a78fdeecdacfacbe2de65edc022017a3c0397555967be9aacee11973c7ba571f42e7ae39b0f2f5e5764b4cf9248d

integration_tests/protocols/code/py-interactsh.yaml

@@ -0,0 +1,28 @@
+id: testcode
+
+info:
+  name: testcode
+  author: testcode
+  severity: info
+  tags: code
+  description: |
+    testcode
+
+variables:
+  i: "{{interactsh-url}}"
+
+code:
+  - engine:
+      - py
+      - python3
+    # Simulate interactsh interaction
+    source: |
+      import os
+      from urllib.request import urlopen
+      urlopen("http://" + os.getenv('i'))
+
+    matchers:
+      - type: word
+        part: interactsh_protocol
+        words:
+          - "http"

integration_tests/protocols/code/py-snippet.yaml

@@ -0,0 +1,23 @@
+id: py-code-snippet
+
+info:
+  name: py-code-snippet
+  author: pdteam
+  severity: info
+  tags: code
+  description: |
+    py-code-snippet
+
+code:
+  - engine:
+      - py
+      - python3
+    source: |
+      import sys
+      print("hello from " + sys.stdin.read())
+    
+    matchers:
+      - type: word
+        words:
+          - "hello from input"
+# digest: 4a0a00473045022023beecb1c4ef5b3b3a4d936a689d0fa5fea35524d23bbc12001fa0b21ca2500b02210082484d006ee0663ba1c8450ff0d10eb053308137af25cde223406c3423c4e5d1

integration_tests/protocols/code/pyfile.py

@@ -0,0 +1,2 @@
+import sys
+print("hello from " + sys.stdin.read())

integration_tests/protocols/code/rsa-priv-key

@@ -0,0 +1,27 @@
+-----BEGIN OPENSSH PRIVATE KEY-----
+b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAABFwAAAAdzc2gtcn
+NhAAAAAwEAAQAAAQEAx8mFIi6bhD/oZZFmFziBuadPZ2mdmI1w+yFLC701SZxDFfHum2Fk
+EJPmArA0uTWf8cebwo1QCJPfGmap6APqerxYfOZzquKcZj3nulwjcn+QBSaYgEomucCELK
+llmlme6mSklOJ7tddgAAWUim13dR/yHsqRb9T2Ew0W2tK9s8E6mQ5B3Q4+4pJikgsyaoEO
+3817NrXDYxM5dmtv2Z3FC9tu+aBOgQDiXPb8rIGszFqYexRgubaQvVpCTSA3K+lojC87r/
+GCzhGnVw9PhekcGDjxZwYy1kASRpCSxe5vER/ST6nY521sYTMlskXhozaFeaXQf1hgJoQt
+N+BWuOv6iwAAA9CaQMkzmkDJMwAAAAdzc2gtcnNhAAABAQDHyYUiLpuEP+hlkWYXOIG5p0
+9naZ2YjXD7IUsLvTVJnEMV8e6bYWQQk+YCsDS5NZ/xx5vCjVAIk98aZqnoA+p6vFh85nOq
+4pxmPee6XCNyf5AFJpiASia5wIQsqWWaWZ7qZKSU4nu112AABZSKbXd1H/IeypFv1PYTDR
+ba0r2zwTqZDkHdDj7ikmKSCzJqgQ7fzXs2tcNjEzl2a2/ZncUL2275oE6BAOJc9vysgazM
+Wph7FGC5tpC9WkJNIDcr6WiMLzuv8YLOEadXD0+F6RwYOPFnBjLWQBJGkJLF7m8RH9JPqd
+jnbWxhMyWyReGjNoV5pdB/WGAmhC034Fa46/qLAAAAAwEAAQAAAQEAhHfL/JQmrZOqRd4Y
+cQEYkr6q2Yif5Ay0gu7aUZhNAtfHa3+UlBYJQSyvb8zhyIQT3z5YurD0Bhv17A1yTtJ54J
+ONoJM00nOa+/fD9D8vibWnCqNCrp17++H4QZy4L5WI1yWQMt/Q4wtBLgKFMRvP/ysFYQEz
+WZE87/jX1JOzEMG8J+RmyvRyLWsm1dERfSl7e2Fub42bn8lSy4fH1rnlsX6M3w01g1YCNa
+GqGGHncVtw/xPt3y/c7LMCMnz1esN4JCSEoLkUoL7yckjGMU90UzxaYO8xKk8uk69RkQ+o
+jRau3nKodMgdHqYQJZ23F3dlVzeBMDoJGEpW5t0+qqaEEQAAAIEApStSDLRjd2K9bBa3u8
+Gkoc3W4A3PeFyVlKQbtGnpE0EBQvn1X6susq4fGZUam6H7aoI32f9pGp/At2e/Xr5TcFSL
+YyIpAuBHjPemhsduH6PNMPk8I6tqeK/5ZgSWKRyrEaTkXS4KT9CHLBzNtJ1O0bH1eCO0xF
+PQlU2+WAX7VZkAAACBAOwbtK/wh3Wu8o4Y6Dwps0pSVQ6pe0RrhdFdkg+dfgsy2ZM73Cs9
+THkEtKvenkQz+gi+eITFnWaN6GTgGBseb3QN2yGs3LnRF8L/H15R6p7dZ+q2R9HoyuSnzK
+U6vrvuZrPxd/Nu3ttUc019bVWEvHSQw+lGCoQd+JCJL6zqjGCvAAAAgQDYnnSkcCp5fkth
+/OzcRW25ZrZOpEZjsaQJm4m9wAsOjmMD7BU+5DKwIveanCp1YjgiJNsIIa/H4UeSbsrRGv
+Y7eVcyJbA2fxf3QlIip3EcufVRmK+pXqmkCq68R0y49q8yrKhpdm7EGWWb4axq3JX2Yi3W
+6xiSJ5uI+Y5CzbHi5QAAABVtYXJjb0BERVNLVE9QLUVMVDA4NVIBAgME
+-----END OPENSSH PRIVATE KEY-----

integration_tests/protocols/code/rsa-pub-key

@@ -0,0 +1 @@
+ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHyYUiLpuEP+hlkWYXOIG5p09naZ2YjXD7IUsLvTVJnEMV8e6bYWQQk+YCsDS5NZ/xx5vCjVAIk98aZqnoA+p6vFh85nOq4pxmPee6XCNyf5AFJpiASia5wIQsqWWaWZ7qZKSU4nu112AABZSKbXd1H/IeypFv1PYTDRba0r2zwTqZDkHdDj7ikmKSCzJqgQ7fzXs2tcNjEzl2a2/ZncUL2275oE6BAOJc9vysgazMWph7FGC5tpC9WkJNIDcr6WiMLzuv8YLOEadXD0+F6RwYOPFnBjLWQBJGkJLF7m8RH9JPqdjnbWxhMyWyReGjNoV5pdB/WGAmhC034Fa46/qL pd@test

integration_tests/protocols/code/rsa-signed.yaml

@@ -0,0 +1,22 @@
+id: rsa-signed-code-snippet
+
+info:
+  name: rsa-signed-code-snippet
+  author: pdteam
+  severity: info
+  tags: code
+  description: |
+    rsa-signed-code-snippet
+
+code:
+  - engine:
+      - py
+      - python3
+    source: |
+      print("rsa signed code")
+    
+    matchers:
+      - type: word
+        words:
+          - "rsa signed code"
+# digest: 34ff81030101095369676e617475726501ff820001030106466f726d6174010c000104426c6f62010a00010452657374010a000000fe0110ff8201077373682d72736101fe0100000d6ef7e5efcb8f8d543805c44ae59bc23d22f5302537dd20efa298aab362552ffc77633dc0898b6718c3e939d1972d906f7e3f8276a8062a3ca587df5cb9a3d4542f335d38d3d8804c83172ddd47d1b58c4dee389377786a7be181021c838c4e90402d19ec445a8cd322fa5ec4606f444f753d7d500b9767001dae17ed4c5640ff4c7655f88742cbd46fe5ba502c8cf3725aff6f5de4e0998821522c847648797897319c4d4c611fda2b22093c85b1f66ba121b9f46bab6f1468ca6a662ed6e9029aff1853d2753e5b17cc4ff776fc5c15b70abae35ab0ca82b58dc0bf5339b98c34e3316393f229ea460cb134787442a958b0c9802a0ea6c79f43d13c934900

integration_tests/protocols/code/unsigned.yaml

@@ -0,0 +1,21 @@
+id: unsigned-code-snippet
+
+info:
+  name: unsigned-code-snippet
+  author: pdteam
+  severity: info
+  tags: code
+  description: |
+    unsigned-code-snippet
+
+code:
+  - engine:
+      - py
+      - python3
+    source: |
+      print("unsigned code")
+    
+    matchers:
+      - type: word
+        words:
+          - "unsigned code"

integration_tests/protocols/headless/file-upload.yaml

@@ -15,7 +15,7 @@ headless:
       args:
         by: xpath
         xpath: /html/body/form/input[1]
-        value: headless/file-upload.yaml
+        value: protocols/headless/file-upload.yaml
     - action: sleep
       args:
         duration: 2

integration_tests/protocols/headless/headless-header-status-test.yaml

@@ -0,0 +1,24 @@
+id: headless-header-status-test
+
+info:
+  name: headless header + status test
+  author: pdteam
+  severity: info
+
+headless:
+  - steps:
+      - args:
+          url: "{{BaseURL}}"
+        action: navigate
+      - action: waitload
+
+    matchers-condition: and
+    matchers:
+      - type: word
+        part: header
+        words:
+          - text/plain
+
+      - type: status
+        status:
+          - 200