User
|
0f9cf57b79
|
Follow redirects
|
2020-09-21 20:46:29 +03:00 |
Casper Guldbech Nielsen
|
ecec5ce411
|
Increasing length of body requried to be matched
Signed-off-by: Casper Guldbech Nielsen <whopsec@protonmail.com>
|
2020-09-12 21:03:03 +02:00 |
|
Casper Guldbech Nielsen
|
79c47bfca1
|
Updated matcher to dsl to only match files that contains more than "Disallow: /"
Signed-off-by: Casper Guldbech Nielsen <whopsec@protonmail.com>
|
2020-09-12 20:13:00 +02:00 |
bauthard
|
34d5f1c586
|
Update robots.txt.yaml
|
2020-08-31 22:47:14 +05:30 |
|
Casper Guldbech Nielsen
|
c833fd9da0
|
Adding robots.txt as file module
Signed-off-by: Casper Guldbech Nielsen <whopsec@protonmail.com>
|
2020-08-31 17:25:09 +02:00 |