7841621a06
Removed digicert source
...
digicert source is not working anymore, "CT log search has been discontinued"
2020-07-20 21:45:01 +02:00
9d6d79f19d
Report request item error only if status code is 404
2020-07-20 21:41:52 +02:00
af4509c9b9
Unescaped URL on unexpected status code agent message
2020-07-20 21:37:23 +02:00
937ad6a441
Fixed isNotFound check
2020-07-20 21:34:01 +02:00
8e2ce0aa6e
Continue on 404 error while reading item code before matching subdomains
2020-07-20 20:59:35 +02:00
3af1d48d5a
New method to discard a HTTP response on unexpected status
2020-07-20 20:49:50 +02:00
a4576376d7
Return the response on unexpected status code
...
Sometimes we may have to check the status code to act accordingly, ej: GitHub source needs to check for 403 to know when a request rate limit exceeded occurs, invalidate the current token and get the next in the pool.
2020-07-20 19:38:52 +02:00
773b61d1a1
Handle not OK status code as Unexpected status
2020-07-20 19:33:20 +02:00
cdb6104a08
Wrap HTTP request to handle not OK status codes
2020-07-20 17:24:37 +02:00
b0e9587f6c
Merge pull request #273 from vzamanillo/list-sources-needs-key
...
Identify and print which sources need an API key / token when ls is used
2020-07-20 17:34:01 +05:30
15f0cd8452
Do not print keys due to private data leakage
2020-07-20 13:26:35 +02:00
1bbd4d34dd
Identify and print which sources need an API key / token when ls is used
2020-07-20 12:44:08 +02:00
cfd9696c91
Merge pull request #271 from vzamanillo/dnsdb#264
...
Do not run dnsdb source if keys are not specified
2020-07-19 22:37:14 +05:30
d8aa749241
Merge pull request #272 from vzamanillo/readme-list-sources
...
Added "ls" option to "Usage"
2020-07-19 22:33:32 +05:30
c5967c41fe
Added "ls" option to "Usage"
2020-07-19 18:03:14 +02:00
d342973eba
Do not run dnsdb source if keys are not specified
2020-07-19 17:57:00 +02:00
c3266cbd81
Merge pull request #270 from vzamanillo/list-sources#256
...
Added flag to list all available sources
2020-07-19 16:27:56 +05:30
fe4e0abc93
Added flag to list all available sources
2020-07-19 12:45:58 +02:00
0f53e1ae54
version update
2020-07-18 16:50:30 +05:30
3804133c36
readme update
2020-07-18 16:48:26 +05:30
e30c5670c3
Merge pull request #267 from vzamanillo/github-rate-limit
...
Handle GitHub Search API rate limit exceeded
2020-07-18 16:45:13 +05:30
c0b4d45746
Merge pull request #269 from vzamanillo/commoncrawl-urlencoded-fix
...
Fixed encoded characters in the subdomain results for Common Crawl source
2020-07-18 16:42:16 +05:30
798815c519
Fixed code format
2020-07-17 22:59:27 +02:00
0cfb970e60
Round-robin token managment for distributed GitHub token use
2020-07-17 22:55:32 +02:00
6b6e4b865b
Oh, missed QueryUnescape
2020-07-16 11:09:05 +02:00
d8ef589e20
Fixed encoded characters in the subdomain results
...
We use QueryUnescape to unescape the body before finding the subdomains avoding the presence of unwanted characters in the results instead TrimPrefix because is case sensitive, we can receive %2f or %2F.
2020-07-16 11:03:33 +02:00
c8ae7a5bd3
Better regular expression to matching subdomains
...
Avoid results starting with "." for wildcard subdomains in content
2020-07-15 22:13:44 +02:00
f0ccebbd8d
Find subdomains in TextMatches too, normalize content before matching
2020-07-15 22:07:45 +02:00
06b7137439
Simplified regular expression
2020-07-15 20:38:38 +02:00
d2cf41d939
FindAllString to get all matches ignoring submatches
...
We do not need submatches because we are not using it.
2020-07-15 20:27:09 +02:00
eadd2e571a
Fixed request rate limit exceeded message
2020-07-15 15:07:25 +02:00
eda27f6934
Handle GitHub Search request rate limit
...
- Support multiple tokens to randomly pick one to avoid exceeding request rate limit
- Retry after "Retry-After" seconds header value if rate limit abuse is detected.
2020-07-15 13:19:31 +02:00
5e9dec7de8
added github key example
2020-07-14 22:42:47 +05:30
bf664d4fd0
readme update
2020-07-14 17:38:12 +05:30
41442e721b
Update banners.go
2020-07-14 12:24:06 +05:30
b943041a4d
Merge pull request #261 from projectdiscovery/bugfix-release
...
using ld compiler
2020-07-13 19:33:01 +02:00
b9ce80415a
using ld compiler
2020-07-13 19:32:33 +02:00
fd8e57dc12
Update config.yaml
2020-07-12 17:16:01 +05:30
5de674fb3a
Update README.md
2020-07-12 17:13:48 +05:30
46d1a0693c
readme update
2020-07-12 17:05:25 +05:30
c1c8723734
Merge pull request #255 from vzamanillo/github-search
...
Added GitHub search source
2020-07-07 22:32:55 +02:00
f09392260d
Merge pull request #254 from himanshudas/patch-1
...
Update commoncrawl.go
2020-07-07 13:37:53 +05:30
5a01a53da9
Removing old year
...
I've removed one year from last, to have latest results and removed last one to make sure we don't get timedout on most runs.
2020-07-07 13:17:56 +05:30
44c58f4d53
Verbose info for next GitHub search results URL
2020-07-05 20:50:45 +02:00
032ee977dd
Added GitHub search source
...
Search for subdomains in GitHub code providing a Personal access token
2020-07-05 20:04:20 +02:00
4d1bc4a9d6
Update commoncrawl.go
...
Updating commoncrawl to include results from year - 2020
2020-07-01 21:05:30 +05:30
064e7a3fe7
updated docker endpoint
2020-06-08 16:24:42 +05:30
02852e3b1c
Update README.md
2020-06-08 15:53:59 +05:30
a64318d9cb
version update
2020-06-08 15:41:48 +05:30
e0f0e91bb3
Update config.yaml
2020-06-08 15:40:05 +05:30
Víctor Zamanillo
bauthard
Mzack9999
Himanshu Kumar Das