Wikipedia:Spiders and bots policy

From Wikipedia, the free encyclopedia

This Wikipedia page is currently inactive and is retained as a historical archive.
Either the page is no longer relevant or consensus has become unclear. If you want to revive discussion regarding the subject, you should seek broader input via a forum such as the proposals page of the village pump.

[edit] Search engine spiders

A huge portion of our traffic comes from referrals from search engines. Obviously, we want search engines to spider over the site and index us! These spiders are usually well-behaved; they take their time, obey the robots.txt file, and don't stress the server too much.

[edit] Other spiders

Other well-behaved spiders are also welcome.

But from time to time spiders are not friendly -- they don't obey the restrictions in robots.txt and overuse database-stressing dynamic functions, and/or request pages at so great a rate that they're preventing legitimate human users from getting at the database. If a spider is found to be causing that kind of trouble, the IP address will be banned from access to the site. Currently so banned are: