A simple library to obey robots.txt files

WWW: https://github.com/chriskite/robotex