web/robots.txt
author cavaliet
Thu, 23 Feb 2012 18:52:59 +0100
changeset 73 c8ecf6c0bdd3
parent 0 7f95f8617b0b
permissions -rwxr-xr-x
Add configuration to enable curl request with proxy or any other CURLOPT

# www.robotstxt.org/
# www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449

User-agent: *