Robotcop - robots.txt: It's the Law

Description: Robotcop is an open source module for webservers which helps webmasters prevent spiders from accessing parts of their sites they have marked off limits.

Keywords: robotcop spam spiders crawlers apache module webserver webmaster robots robots.txt privacy uce