Googlebot

From Wikipedia, the free encyclopedia

A Googlebot is a search bot used by Google. It collects documents from the web to build a searchable index for the Google search engine.

If a webmaster wishes to restrict the information on their site available to a Googlebot, or other well-behaved spider, they can do so by with the appropriate directives in a robots.txt file.

Googlebot has two versions, deepbot and freshbot. Deepbot, the deep crawler, tries to follow every link on the web and download as many pages as it can to the Google indexers. Currently (12 2006), it completes this process about once a month. Freshbot crawls the web looking for fresh content. It visits websites that change frequently, according to how frequently they change. Ideally, freshbot would visit a daily newspaper's website every day and a weekly ezine would get crawled once every 7 days. This collection of information is also known as "The Google Dance".

Googlebot discovers pages by harvesting all of the links on every page it finds. It then follows these links to other web pages. New web pages must be linked to from another known page on the web in order to be crawled and indexed.

A problem which webmasters have often noted with the Googlebot is that it takes up an enormous amount of bandwidth. This can cause websites to exceed their bandwidth limit and be taken down temporarily. This is especially troublesome for mirror sites which host many gigabytes of data.

If, as a webmaster, you register your website on Google Webmaster Tools you can somewhat hint the Googlebot about what pages are to index and what are the priorities of each. You can also configure the bot to crawl your website less frequently. All this is subject to creating a Google Account.

[edit] See also

• mediabot

[edit] External links

• Google's official Googlebot FAQ