☯☼☯ SEO and Non-SEO (Science-Education-Omnilogy) Forum ☯☼☯



☆ ☆ ☆ № ➊ Omnilogic Forum + More ☆ ☆ ☆

Your ad here just for $2 per day!

- - -

Your ads here ($2/day)!

Author Topic: Crawlers | Web crawlers  (Read 9489 times)

0 Members and 2 Guests are viewing this topic.

Gay

  • Just another internet fan
  • SEO Admin
  • SEO hero member
  • *****
  • Posts: 1981
  • SEO-karma: +333/-0
  • Internet knowledge
    • View Profile
    • SEARCH ENGINE OPTIMIZATION
Discobot
« Reply #15 on: October 21, 2011, 01:51:59 PM »
http://www.discoveryengine.com/discobot.html

What is Discobot?
Discobot is the experimental web crawler for Discovery Engine.

Why are you crawling my web site?
We are building a large document index for a next-generation search engine.

How can I stop your crawler from visiting my website?
If you would like our crawler to stop visiting your site, please add the following lines to your robots.txt file:

    User-agent: discobot
    Disallow: /

From what IP addresses does your crawler run?
We are currently crawling from the following range of IP addresses:

    38.101.148.97-38.101.148.126

Whom can I contact with a question or concern about your crawler?
Please send all questions and feedback about the crawler to discobot@discoveryengine.com.

When will your new search engine be released?
A private alpha version of Discovery Engine became available in 2010. A beta version will be released in 2011.

What else can you tell me about Discovery Engine?
“We will sell no wine before its time.”

electra

  • SEO jr. member
  • **
  • Posts: 72
  • SEO-karma: +14/-0
    • View Profile
    • Email
Re: Crawlers | Web crawlers
« Reply #16 on: December 15, 2011, 05:34:08 PM »
A web crawler is a relatively simple automated program, or script, that methodically scans or "crawls" through Internet pages to create an index of the data it's looking for; these programs are usually made to be used only once,
I have been using HostGator hosting and i recommend HostGator ReviewRead also about WebHostingHub

appdeveloper

  • SEO jr. member
  • **
  • Banned!
  • Posts: 95
  • SEO-karma: +21/-7
    • View Profile
    • Application Development
    • Email
Re: Crawlers | Web crawlers
« Reply #17 on: February 14, 2012, 11:08:48 PM »
Lovely information and this is really great post.

danibuggy

  • SEO newbie
  • *
  • Posts: 6
  • SEO-karma: +1/-0
    • View Profile
Re: Crawlers | Web crawlers
« Reply #18 on: February 16, 2012, 03:27:23 PM »
crawling is refereed as spider or google bot. it visit on the web pages.if the web page has good quality , if it has a relevant information related to keyword then google start crawling the website.it constantly send the "spiders" and "bots" for checking the website. 

SEO

  • SEO master
  • SEO Admin
  • SEO hero member
  • *****
  • Posts: 7306
  • SEO-karma: +723/-1
  • SEO expert
    • View Profile
    • SEO
Not only Google
« Reply #19 on: February 16, 2012, 06:22:48 PM »
crawling is refereed as spider or google bot. it visit on the web pages.if the web page has good quality , if it has a relevant information related to keyword then google start crawling the website.it constantly send the "spiders" and "bots" for checking the website. 

 No, it's not only Google. There are many crawlers/spiders as we learned from this topic.  :)

SEO

  • SEO master
  • SEO Admin
  • SEO hero member
  • *****
  • Posts: 7306
  • SEO-karma: +723/-1
  • SEO expert
    • View Profile
    • SEO
More about the Googlebot
« Reply #20 on: May 23, 2013, 09:40:17 PM »

 

Your ad here just for $1 per day!

- - -

Your ads here ($1/day)!

About the privacy policy
How Google uses data when you use our partners’ sites or apps
Post there to report content which violates or infringes your copyright.