4 Commits (c3fd4347a5a3d4464a653635b3b36a31b0fa8e0c)

Author	SHA1	Message	Date
eric	ca08c5caef	add googlebooks virtual urls to disallow ... after we added author links, robots started creating db entries by following links in search page	2016-06-28 14:56:59 -04:00
eric	9440f88d67	https://ahrefs.com/robot/ got stuck in our site.	2013-12-11 11:52:48 -05:00
Ed Summers	3d9eb90174	allow crawling of unglue.it	2012-01-15 23:21:47 +00:00
eric	3eb3653947	robots.txt	2011-11-19 22:15:26 -05:00