EbookFoundation
/
regluit
mirror of https://github.com/EbookFoundation/regluit.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

https://ahrefs.com/robot/ got stuck in our site.

pull/1/head
eric 2013-12-11 11:52:48 -05:00
parent 2e0e282765
commit 9440f88d67
1 changed files with 3 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
frontend/templates/robots.txt
View File

@ -1,7 +1,9 @@
# Hi Robot, welcome to UnglueIt :-) # Hi Robot, welcome to UnglueIt :-)
{% ifequal request.META.HTTP_HOST "unglue.it" %} {% ifequal request.META.HTTP_HOST "unglue.it" %}
User-agent: * User-agent: *
Disallow: /nothing--please-crawl-us Disallow: /accounts/
Disallow: /feedback/
Disallow: /socialauth/
{% else %} {% else %}
User-agent: * User-agent: *
Disallow: / Disallow: /