MoneyOnComputer

[varunroy]

Hello Friends,

What is robots.txt and what is the use of it?

[MOC]

Hi,

It's to tell [BOTS] (individually or collectively) what pages on your website you'd like them to crawl and which ones you'd like them not to crawl for their search engine indexing purposes. Most reputable [BOTS] will follow your specific robots.txt commands.

[RH-Calvin]

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.

[Maria Jonas]

Robots. txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website.