0 Members and 2 Guests are viewing this topic.
A Web crawler is an Internet bot which systematically browses the World Wide Web, typically for the purpose of Web indexing. A Web crawler may also be called a Web spider, an ant, an automatic indexer, or (in the FOAF software context) a Web scutter.
Yahoo!, Google (AdSense), Google (3), Baidu (20), Alexa