[rachana]

Robots.txt is a text file that indicates whether the pages of a website should be crawled or not.

[ameliadavies]

Robots.txt is used by website owners to give instructions about their sites to robots.
It is also known as Robots Exclusion Protocol.
If "Disallow: /" is mentioned, it tells the robot should not visit that page on the site.

[rachana]

Robots.txt is used to instruct web robots about how to crawl pages in their website. It also indicates whether the pages of their website should be crawled or not.

[jackluter]

Robot.txt is file which is useful for protect your secure information in your website , if have any private pages in your website then secure with robot.txt file

[jackluter]

Robot.txt is file which is useful for protect your secure information in your website , if have any private pages in your website then secure with robot.txt file

[gurunath]

Robots.txt controls the search engines which pages in the website want to crawl and don't want to crawl.

[rachana]

Robots.txt is used to instruct web robots about how to crawl the pages of the website and also whether the pages should be crawled or not.

[durga]

robots.txt is file to  increse the comunication speed between google crwllers and website

[clarac12]

A robots.txt file gives instructions to web robots about the pages the website owner doesn’t wish to be ‘crawled’.  For instance, if you didn’t want your images to be listed by Google and other search engines, you’d block them using your robots.txt file.

[andreausa]

Duplicated (copy-pasted) content deleted. User -- banned! More about the ban: http://www.seo-forum-seo-luntan.com/else-topics/forum-(luntan)-problems/msg35549/#msg35549

[SEO]

Welcome to the Big ban list!

[fayeseom]

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

[fayeseom]

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

[MattHogan]

The search engines will look at your robots.txt file before crawling as the instructions on where they are allowed to visit and save the on search engines