Robots Exclusion Protocol for Yahoo, Google and Microsoft
Yahoo announced yesterday that they were releasing detailed documentation on the Robots Exclusion Protocol (REP) and how Yahoo, Google and MSN would handle it. The REP, commonly utilized by webmasters in the form of a robots.txt file or in META tags, has been around since the 90s. By specifying certain robot rules, such as noindex or nofollow, webmasters can in some ways "hide" sections of their website which they do not want included in search results.