Code and general online PC help forum. Website. Programming. Technical.
4 posts in topic - Login  ssl lock  Register
    1,369 topic views 

What is robots.txt?

Unread postby » varunroy » Thu, 7 May 2020, 9:59 am | #1 of 4 |

Hello Friends,

What is robots.txt and what is the use of it?


Share Permalink | PID: #p6724
User avatar
varunroy
AVZoom: Regular Author
Posts: 48
 
#976 Joined:
Fri, 10 November 2017, 7:04 am
Location: India
Name: Varun88


Login to clear banner


  SSL Secure Login - You will return to topic What is robots.txt? on login.

Re: What is robots.txt?

Unread postby » MOC » Thu, 7 May 2020, 11:27 am | #2 of 4 |

Hi,

It's to tell [BOTS] (individually or collectively) what pages on your website you'd like them to crawl and which ones you'd like them not to crawl for their search engine indexing purposes. Most reputable [BOTS] will follow your specific robots.txt commands.


Reload | PID: #p6726
User avatar
MOC
AVZoom: Admin
100 200 300 400 500 600
Posts: 644
 
#2 Joined:
Tue, 1 May 2012, 4:54 pm
Location: Australia
Thanks: 23
Thanked: 13

Re: What is robots.txt?

Unread postby » RH-Calvin » Tue, 26 May 2020, 10:55 am | #3 of 4 |

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
Cheap VPS | $1 VPS Hosting
Cheap Dedicated Server | Servers starting at $29/MO


Reload | PID: #p6745
User avatar
RH-Calvin
AVZoom: Century Club
100
Posts: 119
 
#951 Joined:
Tue, 20 June 2017, 7:23 am
Location: Forum
Thanked: 1
Name: RHCalvin

Re: What is robots.txt?

Unread postby » Maria Jonas » Sat, 20 March 2021, 5:50 am | #4 of 4  Reply

Robots. txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website.


Reload | PID: #p7003
User avatar
Maria Jonas
AVZoom: Century Club
100
Posts: 131
 
#1070 Joined:
Thu, 17 October 2019, 10:31 am
Thanks: 2
Name: Maria

Ask the questions you need an answer for and answer any questions you can.
    1,369 topic views 

LOGIN or REGISTER @ MoneyOnComputer.com

Return to PROGRAMMING AND TECHNICAL Back

WhoIs Online

Users browsing this forum: No registered users and 10 guests

cron