]> gitweb.factorcode.org Git - factor.git/commit
a robots.txt parser
authorDoug Coleman <doug.coleman@gmail.com>
Sat, 21 Mar 2009 02:56:48 +0000 (21:56 -0500)
committerDoug Coleman <doug.coleman@gmail.com>
Sat, 21 Mar 2009 02:56:48 +0000 (21:56 -0500)
commitc626cab383a9f8db56a8de2caee4e0dfe222a47d
treed4b5c054a324a9f1c8f59dbfedf138cee68955d8
parent054a2e5b7c02eda05de4c67eeb1b169b06c15b44
a robots.txt parser
extra/robots/authors.txt [new file with mode: 0644]
extra/robots/robots-tests.factor [new file with mode: 0644]
extra/robots/robots.factor [new file with mode: 0644]
extra/robots/robots.txt [new file with mode: 0644]