phibetaiota.net
Stephen E. Arnold: Open Source HTTP Crawler (Norconex)
Norconex Offers Open Source HTTP Crawler Most commercial enterprise search vendors offer their own HTTP crawler, and several are open-source. One new entry to the field stands out, though, for its …