MoneyOnComputer

Internet Marketing Forums
2   1   1
one pair
Forum ‹ PROGRAMMING AND TECHNICAL
TOPIC › What is robots.txt?
Code and general online PC help forum. Website. Programming. Technical.
4 posts in topic - Login  ssl lock  Register
    3,934 topic views 
Post a reply

What is robots.txt?

Unread postby » varunroy » Thu, 7 May 2020, 9:59 am | #1 of 4 | ∇

Hello Friends,

What is robots.txt and what is the use of it?


• Share Permalink | PID: #p6724
User avatar
varunroy
AVZoom: Regular Author
Posts: 48
 
#976 Joined:
Fri, 10 November 2017, 7:04 am
Location: India
Name: Varun88


Login to clear banner

Top
  SSL Secure Login - You will return to topic What is robots.txt? on login.

Re: What is robots.txt?

Unread postby » MOC » Thu, 7 May 2020, 11:27 am | #2 of 4 | ∇

Hi,

It's to tell [BOTS] (individually or collectively) what pages on your website you'd like them to crawl and which ones you'd like them not to crawl for their search engine indexing purposes. Most reputable [BOTS] will follow your specific robots.txt commands.


• Reload | PID: #p6726
User avatar
MOC
AVZoom: Admin
100 200 300 400 500 600
Posts: 646
 
#2 Joined:
Tue, 1 May 2012, 4:54 pm
Location: Australia
Thanks: 23
Thanked: 13
Top

Re: What is robots.txt?

Unread postby » RH-Calvin » Tue, 26 May 2020, 10:55 am | #3 of 4 | ∇

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
Cheap VPS | $1 VPS Hosting
Cheap Dedicated Server | Servers starting at $29/MO


• Reload | PID: #p6745
User avatar
RH-Calvin
AVZoom: Century Club
100
Posts: 119
 
#951 Joined:
Tue, 20 June 2017, 7:23 am
Location: Forum
Thanked: 1
Name: RHCalvin
Top

Re: What is robots.txt?

Unread postby » Maria Jonas » Sat, 20 March 2021, 5:50 am | #4 of 4 • Reply

Robots. txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website.


• Reload | PID: #p7003
User avatar
Maria Jonas
AVZoom: Century Club
100
Posts: 131
 
#1070 Joined:
Thu, 17 October 2019, 10:31 am
Thanks: 2
Name: Maria
Top
Ask the questions you need an answer for and answer any questions you can.
    3,934 topic views 
Post a reply
TOPIC › What is robots.txt?
• MOC ‹ COMPUTER FORUMS ‹ PROGRAMMING AND TECHNICAL

LOGIN or REGISTER @ MoneyOnComputer.com

Return to PROGRAMMING AND TECHNICAL Back

WhoIs Online

Users browsing this forum: No registered users and 8 guests
Nav Back
MoneyOnComputer
Cash Master MoneyOnComputer Digital Network
Website online 13.507 years / 4,930 days
Powered by phpBB ® © 2012 - 2025
cron