I read an article today over on The Register about how AI may be entering its ‘model collapse’ phase. The basic idea is that more and more of the content that AI ingests to produce its models is, in fact, AI generated. This means that, at its best, AI’s responses will decrease in quality. This sounds very bad, and very predictable.
What is ‘model collapse’?
All current commercial large language model AIs only work because of the vast amounts of human-produced content they ingest. All this data is used, in extremely simplified terms, to generate statistical models regarding the probability of certain words being associated with other words within a particular context. LLMs actually don’t know or understand a thing: truth is meaningless in this statistical model of the vast human word salad they ingest.
The problem is that these models have an implicit assumption: that the information they ingest is in some way ‘truthful’ or accurate. This generally works on the vast collection of human generated content that is the internet: don’t laugh! Yes, there are tons of inaccuracies an outright lies created by humanity on the internet. But enough of it is accurate to for the models to produce reasonable answers most of the time. Sure, they hallucinate and otherwise generate falsehoods, but the error rate is ‘manageable’.
But what happens when more and more of the supposedly human-produced content is actually AI generated? As more and more student papers, business documents, and even peer-reviewed scientific papers are generated by AI? Each generation of models becomes increasingly ‘wrong’ is what happens. At a certain point the model is based on more AI generated ‘approximations’ of truth and reality than on actual facts.
At its worst, AI will start to produce more and more completely wrong responses as the inaccuracies and falsehoods of previous AI generated responses begin to dominate the model. This would logically progress in a downward spiral or collapse until nothing AI produces is right.
Can AI model collapse be avoided?
There are ideas regarding how to avoid model collapse. So far the ones I’ve read have focused on somehow (magically?) separating previous model data from ‘fresh’ (good) data. But it is not clear to me how you can guarantee that the ingested data is going to be ‘clean’ of prior AI generation.
More and more humans are using AI to produce material they pass off as their own ‘work’ without making it clear that it is AI generated. A lot of humans are intentionally lying about the source of their work product. Human nature is to be ‘lazy’: when a tool exists to ease the labour, we generally use it. And when that stretches into areas like formal science and legal documents it becomes really worrisome.
Some businesses have made use of AI generated ‘work product’ to replace people part of their core business model. Most list ‘use of AI’ as a big workforce cost reduction strategy. That’s really what AI proponents in business are selling: the idea that you can replace expensive people with cheap AI is a feature not a bug. ‘Augmenting’ humans is just the stalking horse they use to sugar coat the reality- the real objective is to reduce use of costly humans wherever possible.
Unless the people building the AI models can accurately distinguish the ‘human’ content from the ‘AI’ content, model collapse seems like an almost certain outcome. Hopefully whole teams of very smart people somewhere are working on this problem instead of focusing on firing humans to replace them with AI. But I don’t really trust human nature on this, not with the current generative AI technologies at least.
Google Search currently adds an AI summary at the top of many searches automatically. It’s called “AI Overview”. I was originally skeptical about its usefulness but after a few experiments it became apparent all it was doing was pulling up a subset of the same links it was going to give me in the search anyway. It seems to me that as long as AI quotes its sources and users check them rather than just blindly accepting them, it makes for a handy shortcut. But then, I always get at least two references for anything that matters and if possible three.
The test is going to be whether, as we continue to use search as we always have, it gets harder to find the correct information, or easier, or stays much the same. There has to be a tipping point beyond which information received through search becomes so unreliable the average user no longer bothers with it but even to imagine such a thing suggests either that people will be willing to give up searching altogether when they hit it or that they will move to an alternative form of search. The former seems very unlikely and I’m not sure what the latter would even be. Can you imagine us all going back to printed directories or books when we want to know the nearest pizza joint or who was the 17th president of the USA? I suspect we’re stuck with AI now, whether we like it or not, unless and until it becomes a liability rather than an asset for the megacorps who are pushing it, at which point it will vanish like summer snow.
That’s generative AIas used in search, though. As for AI being worthless, I think that’s a bit like saying transport is going to be worthless or entertainment. There are a lot of types of AI and some of them don’t need verifiable facts to provide the output the users are looking for. Unfortunately, that applies as much to students and their papers as it does to people using it as a creative tool. So long as the AI can pump out the relevant number of words in coherent sentences and the educational institutions can charge the students to hand it in, I’m not sure either side will be all that interested in whether any of it is “true” or not.
You are right, Bhagpuss, that AI will remain useful until it isn’t. You are also completely correct that AI isn’t just generative large language models.
The problem right now is that basically the only AI game in town, the only technique with hundreds of billions of dollars of investment, is generative LLM AI. And the current methods used by all model ‘owners’ involves basing each new model on the previous model plus whatever it can scrape from the internet. The ‘whatever it can scrape from the internet’ part of things is being rapidly polluted with AI generated material that cannot be distinguished with any reliability from ‘clean’ sources.
As the articles I reference say, model collapse seems inevitable unless something changes, and rather quickly. That collapse is likely the ‘tipping point’ you refer to, the point where the results become so full of falsehoods and made-up garbage that they become effectively useless.
Right now it seems as if the big AI players are afraid to talk about model collapse. This fear makes sense because they are riding high on hundreds of billions of dollars in investment, and talking about a seriously critical problem that currently has no solution might just cause the money train to grind to a halt. The real ‘fix’ might be to come up with AI that actually has some intelligence, that can properly assess ‘right’ from ‘wrong’, ‘good’ data from ‘bad’, ‘truth’ from ‘falsehood’. But today’s generative AI has absolutely no such discrimination.
The real problem is that the vast majority of people won’t even notice the difference caused by model collapse. They will continue to assume that AI is ‘correct’ as it provides them with increasingly wrong but pleasing answers, possibly very subtly wrong solutions, and completely made up references that are hard to prove false because they refer to other AI generated falsehoods.
Generative AI collapsing is a good thing in my mind. People are using as real general artificial intelligence rather than what it is, a large language model.
It will still be useful as a large language model, in more restrictive use cases, and there are all sorts of specialized learned by algorithms that technically are AI that are useful and will remain so.
But the toxic idea that because it sounds intelligent and creative and human, so you can replace intelligent creative humans with it needs to die.
I tend to agree, Chris, although ‘die’ is a bit too strong for my opinion. Generative AI and LLM AI in particular are potentially dangerous in the hands of people who overlook the complete lack of intelligence they embody. But the usefulness of such systems is still pretty impressive.
I think what has caught the imagination (and pocket books) of the researchers and their corporate overlords is the fact that there was a huge surge in capability in LLMs a few years ago. LLMs started being able to do things that the developers had never really expected: write moderately capable computer code, for example. No one really planned on that sort of ’emergent’ behaviour becoming remotely as effective as it has.
And I use generative AI all the time: to find quick answers (that are often subtly wrong), to review my written words for errors or unintentional tonal qualities, to generate quick-and-dirty images based on prompts. I could see it being adapted into a really good sort of ‘helper’ conversational system like a friendly but kind of mentally sketchy companion. It is all pretty useful and I like having another tool at my disposal.
That means I don’t necessarily want such AI to ‘die’. I don’t wish for ‘model collapse’. But I do wish that some people who are far smarter and vastly more educated than I am could get the funding and support the LLM folks receive to figure out something better. Model collapse could be the thing that pushes those smart people and the money they need in that direction.
I don’t think throwing more data, processing power, and electricity at LLMs will suddenly make such generative models become intelligent in a general sense. I think we’ve pretty much seen all the surprising tricks and emergent behaviour LLMs have to offer. Something new like some kind of ‘executive’ intelligence layer is needed, something that can develop a reality-based model of ‘truth’ and ‘accuracy’. The kind of thing a 3 year old learns by sticking their finger in hot water or by seeing how lies hurt people including themselves, only (hopefully) more efficient.