The key is “confidently”
A lot of people in power are they because the have no self doubt. They are confident. Women say the desire men that are confident and know what they want – which you note does not exclude him being an idiot.
Job ads look for “confident, outgoing, goal oriented individuals”, who may or may not have any talent, skills, or knowledge of the job they are being hired for. Fake it till you make it is the mantra
A large portion of the population doesn’t judge what a person says based on content. Many can’t even if they wanted to. They lack the knowledge and critical thinking skills. Instead they judge emotion, mood and tone…
“He seems to know what he’s doing”
Just like CEO’s and political pundits that confident comments about things they know nothing about, LLMs are confidently ignorant, and people, especially confident ignorant people will respond to that.
And each iteration is supposed to be really smart this time, even though it isn’t really any better than before, except perhaps more confident. It is “fake it till you make it” on an industrial scale