[danielnash]

I learnt here about sitemap.xml file. Now I'm little bit confused for what is robots.txt used for?

Please guys clear it.

[mojo]

I'll link you to this page, which should explain it for you.

[MSL]

Yeah! It's a great time for this forum. Mojo's 1-st answer in the SEO (as 'Search Engine Optimization') board! Now, Mojo, be ready for future accuses like "You are a spammer!" from some imbeciles in some other websites, because "SEO=spam". I remember your funny (and right) short example in another website, when we talked about 'fat people hate' and you posted they (the haters) may start to downvote recipes, just because, you know, "food=fat". Well, the 'logic' about the SEO is all the same -- no matter what kind of SEO you're doing and discussing (ethical or not, white or gray/black, etc.) for those ignorant netizens, you're just a:

• 'spammer'
• 'search engine poisoning'

Except this I'm not going to add something more, because another one is answering and I'm sure Daniel Nash is going to receive a good robots.txt answer here.

[SEO]

You're one of our intelligent native English speakers and I'm sure you'll have no problem to understand what is robots.txt. It's a good question. Important point is the new webmasters to remember that it's not 'robot' (singular), it's 'robots' (plural), so your spelling way of the robots.txt is right.

This robots.txt thing is the robots exclusion standard (a.k.a. the robots exclusion protocol). This standard is used by webmasters to make their websites able to communicate with web crawlers and other web robots (a note: not every web robot is a web crawler /a.k.a. 'spider'/). Robots.txt specifies how to inform the web robots about which areas of a certain website should be and/or not be processed/scanned. Robots.txt is often used by SEs (i.e. search engines) to categorize web sites. It's good, but... not all robots cooperate with the standard -- e-mail harvesters, spambots and malware robots may even start with the portions of the website where they have been told to stay out! As far as you mentioned the sitemap.xml, well the robots.txt is different from, but can be used in conjunction with it. So, as you already understand -- it (robots.txt) is effective, but not 100% effective and, I think, that this is one of the first things you should know about it: yes, you may choose to outline rules on the behavior of internet bots by implementing a robots.txt file (and this file is simply text stating the rules governing a bot's behavior)... Any bot interacting with (or 'spidering') any server that does not follow these rules should, in theory, be denied access to, or removed from, the affected website. If the only rule implementation by a server is a posted text file with no associated program/software/app, then adhering to those rules is entirely voluntary -- in reality there is no way to enforce those rules, or even to ensure that a bot's creator or implementer acknowledges, or even reads, the robots.txt files' contents.
 
If you worry much about this, maybe you may learn more about the so called 'spider trap': https://en.wikipedia.org/wiki/Spider_trap

And now the examples:

Code: [Select]

User-agent: *
Allow:

(This tells all robots that they can visit all files because the * specifies all robots.)

Code: [Select]

User-agent: *
Disallow: /

(The opposite -- it tells all robots to stay out of a website.)

Code: [Select]

User-agent: *
Disallow: /cgi-bin/

(It tells all robots not to enter a /one/ directory.)

Code: [Select]

User-agent: *
Disallow: /directory/ourfile.html
(It tells all robots to stay away from a /one/ specific file; all other files in the specified directory will be processed.)

In case you need more examples, just let me know.

[SEO]

Yeah! It's a great time for this forum. Mojo's 1-st answer in the SEO (as 'Search Engine Optimization') board! Now, Mojo, be ready for future accuses like "You are a spammer!" from some imbeciles in some other websites, because "SEO=spam". I remember your funny (and right) short example in another website, when we talked about 'fat people hate' and you posted they (the haters) may start to downvote recipes, just because, you know, "food=fat". Well, the 'logic' about the SEO is all the same -- no matter what kind of SEO you're doing and discussing (ethical or not, white or gray/black, etc.) for those ignorant netizens, you're just a:

• 'spammer'
• 'search engine poisoning'

Except this I'm not going to add something more, because another one is answering and I'm sure Daniel Nash is going to receive a good robots.txt answer here.

Their thinking is a primitive thinking. It sounds as: "All swans are black!", "All Americans love Trump!", "All Chinese are martial artists!", etc. Ignorance, hatred, regress.
I'm very glad there are forums and communities (including this one), where people may chat, link, share, discuss and think, without SEO prejudices.

[mojo]

be ready for future accuses like "You are a spammer!" from some imbeciles in some other websites, because "SEO=spam".

To be honest here, I have no dogs in this tussle. (another way to say no benefit from discussion) My aim was merely to inform on a question. Not to support nor condemn the practices of SEO. I would benefit in no way from providing such a link, now or in the future. Rather it was in the nature of being informative and as a good netizen for a site I seem to have been adopted to.

[MSL]

 Of course. I understand it. And I only wanted to show my irony for those haters and degenerates in voat.co and in another website (empeopled.com), who have no idea what's SEO, what's spam, what's trust, what's humanity and... many others.

[SEO]

be ready for future accuses like "You are a spammer!" from some imbeciles in some other websites, because "SEO=spam".

To be honest here, I have no dogs in this tussle. (another way to say no benefit from discussion) My aim was merely to inform on a question. Not to support nor condemn the practices of SEO. I would benefit in no way from providing such a link, now or in the future. Rather it was in the nature of being informative and as a good netizen for a site I seem to have been adopted to.

And I'm very glad that you taking part in this sub-forum, too. The only disadvantage mentioned already above is that for some foolish guys in other websites (I'll be straight: till now our website got this kind of misunderstanding and hostility in voat.co and in empeopled.com) think that if you're a part of a website which focus about search engines you're a 'spammer', a 'SEO spammer', a 'SEO poisoning' and you're not welcome there, because you will 'spam' them with your 'SEO spam' and you will 'destroy' their super community. (But you're using different nicks, so in your case MSL's helpful and informative joke isn't important. For the others /those, who use the same nicks here and in the other websites/ it's important: every participation in a forum or other site, which focus more or less about SEO makes you a potential victim of spamphobia/spamophobia.)

[mojo]

'SEO spammer'

There's a hidden meaning you are probably aware of in the context of SEO. So many sites have dealt with the unscrupulous on SEO advertising that the initials are linked to spamming as an unspoken but very much agreed on definition. This more than anything is the reason you are receiving such hostility when the initials are brought up. Experience over time has taught the majority of surfers when they hear or see those initials that spam is to follow. Even worse that the spam is liable to be from the unwholesome people who populate the net solely to prey upon others for the money.

So when they arrive and see the title of the forum, their very first impression (and mine the first time I showed up) is that it is a group intent on creating troubles, solely from the name as a first impression. Yes, I know you have a different meaning to the SEO now but I didn't first time I showed up. First time I came here, I saw the title and closed the site without looking further. That first impression is everything and I tell you this as to my initial reaction to seeing the name of the site. It's not what is; it's what is implied it is.

I hope you see the fine line here.

[И.]

This happens only to people who had a bit knowledge of what SEO stands for. Those who never heard of it and those who are more familiar with it are friendly or okay with the SEOs. This topic is a very precious example what the good SEOs like us are dealing with: we discus .txt, which is useful for your website and for the search engines to communicate better and to rank you more accurately. In another topic we may chat how to describe better the pics and images in our sites to get better ranking or how to choose a better CAPTCHA and other anti-spamic ides. All this is the bright side of the SEO and it is harmless for the others. But, if they associate SEO only with the black tactics (which currently are less and less effective, after the Google algorithm updates), then we all may be in a situation like: "You are working in the army? - Yes, I do. - Killers! - No, I'm a cleaner and my buddy is a military doctor. We never kill people. - KILLERS!!! KILLERS DETECTED!"
Yes, I can't see anything wrong with your line. You're frank and all you said makes big sense. 

[MSL]

 I see 100% anytime only fine lines in your posts (maybe because I know you a bit better, but I do hope everybody soon will know and feel the same). In this post I see a good and honest feedback, too.
 I'd like to post something for the others, who can't make difference:
Some SEOers are spammers. But not every spammer is a SEOer. Most of the spammers are people, who are very far away from the true SEO knowledge. For example, imagine a housewife, who never heard of SEO, but she's so proud of her first knitting or cooking or something-else blog, that she's linking it like crazy in all the online websites she can find and she links it again and again, flooding with it non-stop in related or non-related threads, boards, subs, etc. Is this an SEO? No. Not at all. It's a pure spam activity.
 (If there are still individuals, who can't understand what I mean, like the imbeciles I mentioned already -- a last attempt: Some birds are crows, but not every bird is a crow.)

[mojo]

It should be obvious by now that I see the difference here compared to past experiences. Had I not seen it, I'd not be here. I took the time to read a bit and learn from the parts where I can.

MSL knows me a bit better from other places and other than that, the words have to speak for themselves. If those words contain some sort of wisdom, rather than condemnation, they will fall upon fertile ground. If not, they will be discarded as not being worth the effort to answer. It is the way of things.

As has been noted it was in an effort to be straight forward and as clear as possible about the circumstances you find yourself in when mentioning the name of the site. It's not to rub the fur the wrong way say is said.

All of which I suspect you have long ago figured out for yourselves.

[MSL]

 Yes, it's obvious. I wish all the people were like you: to understand the facts without prejudices and fast, but... it's just a dream.

[johnniewalk]

In a simple manner robot.txt is used for to inform the search engine to crawl a particular links or not to crawl a particular link which is disallowed in the  robot.txt

[bcetindra]

 robots.txt is used to websites to communicate with web crawlers and other web robots.....