The older, the simpler: robotstxt.org. robots.txt in a nutshell

Post a reply


This question is a means of preventing automated form submissions by spambots.
Smilies
:D :) ;) :( :o :shock: :? 8-) :lol: :x :P :oops: :cry: :evil: :twisted: :roll: :!: :?: :idea: :arrow: :| :mrgreen: :geek: :ugeek:

BBCode is ON
[img] is ON
[url] is ON
Smilies are ON

Topic review
   

If you wish to attach one or more files enter the details below.

Maximum filesize per attachment: 1 MiB.

Expand view Topic review: The older, the simpler: robotstxt.org. robots.txt in a nutshell

The older, the simpler: robotstxt.org. robots.txt in a nutshell

by axew3 » Mon Apr 05, 2021 7:35 pm

I fall on this today, after i know it since his creation: has been a surprise to see that, in all these years, it changed nothing at all!
http://www.robotstxt.org

Top