I fall on this today, after i know it since his creation: has been a surprise to see that, in all these years, it changed nothing at all!
http://www.robotstxt.org
The older, the simpler: robotstxt.org. robots.txt in a nutshell
-
axew3
- w3all User
- Posts: 2883
- Joined: Fri Jan 22, 2016 5:15 pm
- Location: Italy
- Contact: