Official Google Webmaster Central Blog: Optimize your crawling & indexing
http://googlewebmastercentral.blogspot.com/2009/08/optimize-your-crawling-indexing.html