In case you spot an odd-bot.

NLI: In November 2007 the National Library of Ireland is undertaking a comprehensive crawl and harvest of the Ireland web domain using the services of the Internet Archive, the first such crawl of this size. For the purpose of this collection, the Ireland web domain includes .ie domain sites. In addition some sites identified by DNS lookup as having an IP address located in Ireland may be included.

This Post Has 2 Comments

  1. I banned them. They were requesting odd urls and somehow redirecting to the same url dozens of times…

  2. It’s actually contracted out to archive.org Donncha, who should really know better.

Leave a Reply

Close Menu