Google’s AI Is Destroying Search, the Internet, and Your Brain

cross-posted from: slrpnk.net/post/25347642

www.404media.co/googles-ai-is-destroying-search-the-internet-and-your-brain/

I haven’t used Google as a primary search engine in a decade or more. Nothing I do depends on Gemini (I don’t even use Apple Intelligence that’s a Gemini white label).

Personally, I think what Gemini is destroying is a market-based financial model that many came to associate with the Internet, information access and lifestyle.

Most of the websites I visit pre-date Google and some have gone as far as blocking all bots via robots.txt. They survive not via predatory ad networks but by donations and volunteers.

It’s why I like Lemmy; it follows the same model.

I don’t think today’s bots take any notice of robots.txt. It’s a relic from a more civilized time.

I run a few personal sites with some being available to the wider internet, and I have found that they generally do respect it. All the bots which advertise themselves as OpenAI, Anthropic, or Google do obey. However, I do occasionally see a stealth bot appear which uses a normal browser’s user agent and those just do whatever they want.

Meta absolutely does not respect robots.txt. they are scraping what I have at around 200 hits / minute.

I also notice there is some data center somewhere (or multiple) proxying all its requests through residential proxies so I can’t tell who is scraping. They don’t scrape like Meta though. Far slower. These are the stealthy bois using spoofed user agents you mention. Can’t tell who they are, though. How do they get so many residential IPs?

I have no organic traffic (I am literally just running a crawler tarpit. My page has nothing) so its really obvious that Im watching AI scrappers. My tarpit generates random links that all resolve to the same place so its super obvious its not human. Its also super obvious when two distinct IPs follow the same word salad link milliseconds apart.