add googlebooks virtual urls to disallow
after we added author links, robots started creating db entries by following links in search pagepull/1/head
parent
a9152bdbe3
commit
ca08c5caef
|
@ -4,6 +4,7 @@ User-agent: *
|
|||
Disallow: /accounts/
|
||||
Disallow: /feedback/
|
||||
Disallow: /socialauth/
|
||||
Disallow: /googlebooks/
|
||||
{% else %}
|
||||
User-agent: *
|
||||
Disallow: /
|
||||
|
|
Loading…
Reference in New Issue