A collection of tidbits and links that I refer to periodically:

robots.txt

FAQ: http://www.robotstxt.org/wc/faq.html

Validator: http://tool.motoricerca.info/robots-checker.phtml

sitemap.xml

FAQ: http://rorweb.com/faq.htm

Generator: http://www.rorweb.com/ror-feed-generator.htm