[Gay]

http://www.discoveryengine.com/discobot.html

What is Discobot?
Discobot is the experimental web crawler for Discovery Engine.

Why are you crawling my web site?
We are building a large document index for a next-generation search engine.

How can I stop your crawler from visiting my website?
If you would like our crawler to stop visiting your site, please add the following lines to your robots.txt file:

    User-agent: discobot
    Disallow: /

From what IP addresses does your crawler run?
We are currently crawling from the following range of IP addresses:

    38.101.148.97-38.101.148.126

Whom can I contact with a question or concern about your crawler?
Please send all questions and feedback about the crawler to discobot@discoveryengine.com.

When will your new search engine be released?
A private alpha version of Discovery Engine became available in 2010. A beta version will be released in 2011.

What else can you tell me about Discovery Engine?
“We will sell no wine before its time.”

[electra]

A web crawler is a relatively simple automated program, or script, that methodically scans or "crawls" through Internet pages to create an index of the data it's looking for; these programs are usually made to be used only once,

[appdeveloper]

Lovely information and this is really great post.

[danibuggy]

crawling is refereed as spider or google bot. it visit on the web pages.if the web page has good quality , if it has a relevant information related to keyword then google start crawling the website.it constantly send the "spiders" and "bots" for checking the website. 

[SEO]

crawling is refereed as spider or google bot. it visit on the web pages.if the web page has good quality , if it has a relevant information related to keyword then google start crawling the website.it constantly send the "spiders" and "bots" for checking the website. 

 No, it's not only Google. There are many crawlers/spiders as we learned from this topic. 

[SEO]

 

More about the Googlebot:

http://support.google.com/webmasters/bin/answer.py?hl=en&answer=182072