5 Commits (e1336243ea0614316018b6f9ee03cd37345dc3f4)

Author	SHA1	Message	Date
eric	4802eea09d	Disallow: /search/	2018-01-12 11:53:10 -05:00
eric	ca08c5caef	add googlebooks virtual urls to disallow ... after we added author links, robots started creating db entries by following links in search page	2016-06-28 14:56:59 -04:00
eric	9440f88d67	https://ahrefs.com/robot/ got stuck in our site.	2013-12-11 11:52:48 -05:00
Ed Summers	3d9eb90174	allow crawling of unglue.it	2012-01-15 23:21:47 +00:00
eric	3eb3653947	robots.txt	2011-11-19 22:15:26 -05:00