teleread.org
The Internet Archive will soon stop honoring robots.txt files
One of the fixtures of the modern web is the robots.txt file—a file intended to notify web-crawling robots what parts of web sites are off-limits to them, so as to avoid reindexing duplicate …