AI Companies Are Buying Tons of Old Books Because They're Free of AI Slop

I’m waiting for the day when desperate LLM companies start paying people to post real human content, only for those people to just ask ChatGPT to do it.

Big AI mostly switched to synthetic training data anyway. The books they’re digitizing are being used to gather knowledge, not writing styles or logic (mostly).

As in, when you ask ChatGPT how long some book is, it can just go check (if it’s in the database). It’s also useful if you ask about that book or about knowledge contained in that book. It’ll even reference books now (if you demand that in your prompt).

It’s not the same as earlier LLM tech which relied on scanned text to figure out how to respond to any given prompt (from a language standpoint). The “language” part of LLMs is a solved problem now (thanks to the synthetic training). At least for English 🤷

The article quotes a post from ISBNdb saying the issue is model collapse from training on synthetic data.

That’s like saying, “they had some failure modes from the synthetic data, so they should just obviously stop trying forever.”

They’ll just fix the edge cases and move on. Like any programming task.