Commit Graph

50 Commits (42a59189f39cf62e275a8057643fdf183fe575a9)

Author SHA1 Message Date
Tarun Koyalwar c62dc01f9f
uncover logic refactor to v0.0.4 (#3663)
* uncover logic refactor to v0.0.4

* remove deprecated import: stringsutil
2023-05-09 03:57:56 +05:30
Jonathan Steele 710ac0839c
chore(fix): Ensure proper scanning strategy (#3464)
This will ensure that the host spray scanning strategy will actually
run rather than the system exiting quickly if that strategy is
applied. Once merged, this will resolve projectdiscovery/nuclei#3463.
2023-03-23 21:53:17 +05:30
Mzack9999 0bf8fc027d
Fixing nil pointer reference + use map helper (#3421)
* Fixing nil pointer reference + use map helper

* bump tlsx version to v1.0.6

* increase interactsh polling in integration_test

* fix nil pointer dereference in integration_test

* fix lint error

---------

Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
2023-03-14 21:27:48 +05:30
Mzack9999 68550695f8
Fixing atomic bool check (#3376) 2023-03-06 00:37:40 +05:30
xm1k3 1132456ec6 bump mapcidr version + asn 2023-02-27 10:22:50 +01:00
Notealot aeb5dbd293
feat: add RawStringSlice / fix: reference url case insensitive (#3346)
* feat: add RawStringSlice and refactor

* restored old logic + refactor

* restored files

---------

Co-authored-by: mzack <marco.rivoli.nvh@gmail.com>
2023-02-26 01:17:47 +05:30
Tarun Koyalwar 0b2a3e296a
fix url encoding issues and inconsistencies (#3211)
* fix url encoding issues

* complete requested changes and improvements

* fix missing issue-tracker-config.yaml

* fuzz: deepcopy and use urlutil.Params
2023-01-24 22:04:52 +05:30
Tarun Koyalwar 6ebf5a789e
fix host spray race condition (#3213)
* core: bug fixes

* best practices: uniform comments
2023-01-20 23:49:04 +05:30
Ice3man 67c094444e
Added cloud scan progress tracking using stats (#3180)
* Added cloud scan progress tracking using stats

* Changed log messsage

* Fixed linting error

* Fixed bug in progress calculation logic

* Changed requests to input with cloud flag

* Changed progress name + removed redundant fields
2023-01-13 13:41:05 +05:30
Tarun Koyalwar e899afafdf
skip scanallip if input is ip (#3186)
* skip scanallips for ip input

* uncover,url parsing bug fix

* minor changes:best practices
2023-01-11 22:50:57 +05:30
Mzack9999 2c3453500c
Adding empty comparison for scan strategy (#3100)
* Adding empty edge case to scan strategy

* Auto Generate Syntax Docs + JSONSchema [Thu Dec 29 10:10:55 UTC 2022] 🤖

* adding test file

* removing test file

Co-authored-by: GitHub Action <action@github.com>
2022-12-29 20:12:52 +05:30
Tarun Koyalwar ff17d12ced
adds scan strategy (#3075) 2022-12-28 02:18:00 +05:30
Tarun Koyalwar aee0870617
scanallip handle edge cases (#3080)
* bug fix:remove port during dns resolution

* scanallip fix edge cases

* add scanallips testcases

* workflow fix

* removing pull cmd

* Auto Generate Syntax Docs + JSONSchema [Sat Dec 24 13:29:21 UTC 2022] 🤖

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
Co-authored-by: GitHub Action <action@github.com>
2022-12-24 19:03:23 +05:30
Sandeep Singh 96646c8f53
cloud templates targets sync (#2959)
* Add s3 bucket template provider

- Refactor the custom github template code
- add interface for template provider

* Validate if aws creds are passed if bucket flag

- refactor s3 provider struct to take client
- add function which returns the aws s3 client
- update error messages

* Add aws s3 bucket flags documentation in README.md

- Rename the github_test.go to customTemplate_test.go

* go mod update

* Move template provider code to pkg/external/customtemplates dir

* Added initial data_source sync to cloud

* Misc

* Add pagination to scan output and scan list (#2858)

* Add pagination to scan output and scan list

* Use time based parameters instead of page numbers

* Fix linting errors

* Do not check limits at client, check at server

* Remove unused constant

* Misc update

* Removed unnecessary flags

* Misc

* Misc

* Misc endpoint additions

* Added more routes

* Typo fix

* Misc fixes

* Misc

* Misc fixes to cloud target logic + use int for IDs

* Misc

* Misc fixes

* Misc

* Misc fixes

* readme update

* Add JSON output support for list-scan option (#2876)

* Add JSON output support for list-scan option

* Fix typo in cloud JSON output description

* Following changes

- Update status(finished, running) to be lower-case by default
- Convert status to upper-case in DisplayScanList()

* Update status to be lower-case by default

* Remove additional json flag, instead use existing

* Merge conflict

* Accomodate comment changes and restructure code

Co-authored-by: Jaideep K <jaideep@one2n.in>

* Use integer IDs for scan tasks

* Added get-templates-targets endpoint + JSON + validation

* Added target count list

* misc option / description updates

* Added changes as per code review

* duplicate options + typo updates

* Added tablewriter for tabular data writing by default

* Fixed list scan endpoint

* Review changes

* workflow fix

* Added cloud tags etc based filtering (#3070)

* Added omitempty for filtering request

* go mod tidy

* misc format update

Co-authored-by: shubhamrasal <shubhamdharmarasal@gmail.com>
Co-authored-by: Ice3man <nizamulrana@gmail.com>
Co-authored-by: Jaideep Khandelwal <jdk2588@gmail.com>
Co-authored-by: Siddharth Shashikar <60960197+shashikarsiddharth@users.noreply.github.com>
Co-authored-by: Jaideep K <jaideep@one2n.in>
2022-12-21 22:48:43 +05:30
Mzack9999 2445662fb1
Fix memory leak on hmap with LRU (#3052)
* fix memory leak on hmap with LRU

* fixing lint issues
2022-12-20 15:37:48 +05:30
Tarun Koyalwar 2874a9dff7
fix -iv & minor bugs (#2951)
* fix -iv & minor bugs

* fix ipversion defaults

Co-authored-by: sandeep <sandeep@projectdiscovery.io>
2022-12-05 18:22:04 +05:30
Tarun Koyalwar 3b31799847
Issue 2840 aws signature (#2924)
* docker go version update

* docker fix

* version update

* update chinese readme and typo fixes. (#2862)

* fix aws request signer

* fix reader by upgrading retryablehttp-go

* go mod tidy

Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
Co-authored-by: Xc1Ym <xuedongyuming2233@gmail.com>
2022-12-03 07:10:57 +05:30
Mzack9999 a1ecef6b61
Mocking dns server (#2908)
* mocking dns server

* fix lint errors
2022-11-28 13:52:40 +05:30
Shubham Rasal 6b142d794a
Issue 2254 uncover integration (#2786)
* nuclei -uq 'vuln:CVE-2021-26855' -t cves/2021/

- `nuclei -uq 'vuln:CVE-2021-26855' -t cves/2021/`

* Add automatic template execution using metadata

- Query uncover after the template is loaded.
- Add the received hosts to the input provider from uncover
- Make NormalizeStoreInputValue() function public to add hosts from the
  runner after uncover hosts received.

* run go mod tidy

* Remove unnecessary comments

* Resolve the requested changes

- move uncover code to protocols/common/uncover package
- Use uncover delay to create uncover rate limiter
- Use single ratelimiter object and remove not required ratelimiters
- Create Set() method for input provider interface
- Rename normalizeStoreInputValue to Set() method

* Solved the uncover running twice.

- flag StringSliceVarP adds the default value twice in the variable
- Check if provider keys exists or not
- Add uncover help block to english readme.md

* Add uncover field functionality

- ./nuclei -uq 'vuln:CVE-2021-26855' -t dns -duc -uf host
- ./nuclei -uq 'vuln:CVE-2021-26855' -t dns -duc -uf ip:port

* Update error messages and solve nuclei hang for wrong uncover engine

- Get uncover engine values from uncover package

* Resolve merge conflicts

* misc option update

* Update logging for templates

- remove duplicate env log printing
- Log message for template queries

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2022-11-16 11:12:39 +05:30
Mzack9999 1fbbca66f9
Adding support to scan all v4/v6 IPs (#2709)
* Adding support to scan all v4/v6 IPs

* adding tests

* metainput prototype

* using new signature

* fixing nil pointer

* adding request context with metadata

* removing log instruction

* fixing merge conflicts

* adding clone helpers

* attempting to fix ipv6 square parenthesis wrap

* fixing dialed ip info

* fixing syntax

* fixing output ip selection

* adding integration tests

* disabling test due to gh ipv6 issue

* using ipv4 only due to GH limited networking

* extending metainput marshaling

* fixing hmap key

* adding test for httpx integration

* fixing lint error

* reworking marshaling/id-calculation

* adding ip version validation

* improving handling non url targets

* fixing condition check
2022-11-09 18:48:56 +05:30
vrenzolaverace 2aaf2a2158
Use utils helpers libraries (#2809) (#2810)
* Use utils helpers libraries (#2809)

* Use utils helpers libraries (#2809)
2022-11-07 01:54:23 +05:30
Ice3man 363ffb75db
Added probing for URL + input based on protocol (#2614)
* Added workflow names based condition

* Added conditional filtering to workflow executor

* Replaced names with single name stringslice

* Added probing for URL + input based on protocol

* Remove debug comments

* Fixed typo

* Fixed failing tests

* Fixed workflow matcher condition + tests

* Fixed workflow item name

* Switch to if-else

* Fixed review comment strict

* Increase bulk size

* Added default port for SSL protocol + misc changes

* Fixed failing tests

* Fixed misc changes to executer

* Fixed failing self-contained and offlinehttp tests

* Fixed atomic increment operation

* misc update

* Fixed failing builds

Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2022-10-20 17:23:00 +05:30
Shubham Rasal 897dbe8e4c
Add AS input support (#2712)
* Add AS input support

- Integrate mapcidr asn function to handle ASN number input support
- Check if input is ASN number or not. If yes then query for cidrs
- issue #2706

* Remove \r coz of failing test cases in windows os

* Replace newline char for windows

* remove extra line

* rename goldenfile dir to tests

* fixing folder name

Co-authored-by: mzack <marco.rivoli.nvh@gmail.com>
2022-10-14 00:25:00 +05:30
Shubham Rasal 3ebd1f689b
cidr input support #2665 (#2675)
* Add CIDR input support

- Add expandCIDRInputValue function which accepts the cidr,
  and stores the IPs into hmap. It uses mapcidr to get the expanded IPs
- Add test case to test expandCIDRInputValue and isCIDR function
- Update dsl_test.go which had typo. coz of failing test

* Resolve the requested changes
2022-10-07 14:28:02 +05:30
Mzack9999 781e4e6105
Shared Execution Context Prototype (#2576)
* renaming var

* Introducing shared execution context prototype

* fixing field name

* adding shared values propagation

* adding shared context lock

* add slice values normalization

* adding integration tests

* adding metadata support for dns

* adding multi-protocol context sharing test

* removing debug test files

* moving contextargs around

* adding comments

* refactoring code

- getter/setter for complex types
- using pointers to avoid heap allocations
2022-10-03 15:42:20 +05:30
Ice3man 03c7e89d9e
Added workflow names based condition (#2594)
* Added workflow names based condition

* Added conditional filtering to workflow executor

* Replaced names with single name stringslice

* Fixed workflow matcher condition + tests

* Fixed workflow item name
2022-09-19 16:49:30 +05:30
mzack 37c016a5f0 fixing lint errors 2022-09-19 08:38:52 +02:00
Mike Rheinheimer 9efba05e0c
expose hosterrorscache.Cache as an interface (#2291)
* expose hosterrorscache as an interface, change signature to capture the error reason

* use the hosterrorscache.CacheInterface as struct field so users of Nuclei embedded can provide their own cache implementation

Co-authored-by: Mike Rheinheimer <mrheinheimer@atlassian.com>
2022-07-19 02:05:53 +05:30
Mzack9999 476773ff8c
Replacing hasstdin with helper library (#2191)
* Replacing hasstdin with helper library

* adding timeout reader on stdin

* adding large input read timeout

* reducing stdin timeout + nostdin flag

* go mod update

* readme update

* go mod tidy

Co-authored-by: sandeep <sandeep@projectdiscovery.io>
2022-06-30 17:51:54 +05:30
Sami 6ca4374f91
sonar category: String literals should not be duplicated (#1944)
* sonar category: String literals should not be duplicated

* lint error fix

* better naming conventions for constants

* improved naming conventions and methods
2022-05-12 15:40:14 +05:30
Ice3man ffe4fea237 Performance improvements + wappalyzer as default + misc 2022-03-08 12:43:24 +05:30
Ice3man d9a121344c Merge branch 'dev' of https://github.com/projectdiscovery/nuclei into automatic-workflows 2022-03-07 13:19:37 +05:30
forgedhallpass 72af50155d Typo fixes 2022-02-07 16:41:55 +02:00
Ice3man be75480256 Added initial smart workflow implementation 2022-01-18 20:59:37 +05:30
mzack 1f878192ec Merge branch 'dev' into issue-614-scan-resume 2021-12-23 13:15:47 +01:00
mzack 63b194c10a refactoring stop-resume logic with better tracking 2021-12-16 22:32:03 +01:00
Ice3man543 65a1858f94 fix #1303: speed regression due to missing goroutine in executer
Fixed speed regression introduced in dev due to missing go() statement when executing template input
2021-11-30 00:17:52 +05:30
mzack 2140ffa382 Adding stop-resume support 2021-11-29 14:38:45 +01:00
forgedhallpass 7e22d70ded refactor/documentation: typos and grammatical errors 2021-11-25 18:54:16 +02:00
forgedhallpass 47340f06b0 refactor: uniformly sorted imports 2021-11-25 17:09:20 +02:00
Mzack9999 ca9676f52e
Adding support for clustering within workflow (#1255)
* Adding support for clustering within workflow
2021-11-22 04:49:53 +05:30
Ice3man543 d442c51c1a Changed input to target in core 2021-11-08 16:14:47 +05:30
Ice3man543 8ad3ebcd05 Made code changes as per review comments 2021-11-05 03:01:41 +05:30
Ice3man543 390ca8b3c6 Merge from dev 2021-11-03 18:58:00 +05:30
Ice3man543 bdb415b0c7 Misc changes as per review 2021-11-03 02:23:48 +05:30
Ice3man543 5393cc4cd5 Adjusting packages for more API-type design 2021-10-29 03:19:43 +05:30
Ice3man543 0abc7202b1 Misc fixes with goflags 2021-10-28 17:45:38 +05:30
Ice3man543 d124dbacc7 Moved all important execution stuff to engine 2021-10-28 17:20:07 +05:30
Ice3man543 df78ea72c5 misc 2021-10-27 18:41:39 +05:30
Ice3man543 c16c93fe7c refactor the modules to core 2021-10-27 16:50:36 +05:30