sharepointyankee.com
Stay away from my search result pages [insert search engine name here] bot!!
Now, this is a bit overkill, but, there are almost as many different ways a search crawler makes use of a robots.txt file as there are search engines (this may be highly over-exaggerated, but anywa…