Wikipedia:Spiders and bots policy

From Wikipedia, the free encyclopedia

You have new messages (last change).
This Wikipedia page is currently inactive and is kept primarily for historical interest. If you want to revive discussion regarding the subject, you should ask for broader input, for instance at the village pump.


[edit] Search engine spiders

A huge portion of our traffic comes from referrals from search engines. Obviously, we want search engines to spider over the site and index us! These spiders are usually well-behaved; they take their time, obey the robots.txt file, and don't stress the server too much.

[edit] Other spiders

Other well-behaved spiders are also welcome.

But from time to time spiders are not friendly -- they don't obey the restrictions in robots.txt and overuse database-stressing dynamic functions, and/or request pages at so great a rate that they're preventing legitimate human users from getting at the database. If a spider is found to be causing that kind of trouble, the IP address will be banned from access to the site. Currently so banned are:

  • 144.167.21.15
  • 192.153.22.246
  • 202.69.76.19
  • 212.27.33.00/24
  • 24.130.248.43
  • 194.209.152.200
  • 80.192.68.91
  • 80.2.170.93
  • 209.208.186.2
  • 68.62.88.211
  • 203.175.70.118
  • 65.60.161.156
  • 81.86.203.137
  • 63.164.242.215
  • 66.147.154.3
  • 62.101.126.224
  • 209.125.45.130
  • 172.189.3.245
  • 61.30.127.4
  • 61.30.14.26
  • 137.230.1.11
  • 194.228.168.42

(These don't show up in Special:Ipblocklist, which blocks IPs from saving edits to the wiki.)

[edit] Bots that make edits to the wiki

Well-behaved bots are acceptable under some circumstances. Please see Wikipedia:Bots.

Retrieved from "http://en.wikipedia.org../../../s/p/i/Wikipedia%7ESpiders_and_bots_policy_3fe9.html"