Just a quick correction. Robots.txt doesn’t block anything, it simply politely asks the search engine to not look here or use what is contained within. A lot of the search engines ignore this and catalogue the contents of the website anyway. And pretty much all of the AI database downloaders ignore it with impunity.
Edit history
Just a quick correction. Robots.txt doesn’t block anything, it simply politely asks the search engine to not look here or use what is contained within. A lot of the search engines ignore this and catalogue the contents of the website anyway.