Un-employ us all
The story they sell us about AI, the magic beans they are peddling, is that AI will be able to do most things for us and people will simply dictate what should be done. While I know you shouldn't look a gift horse in the mouth, the word Trojan makes me think of things that aren't horses as well. So, I have some questions.
The most important question being: which people? They certainly don't mean all people. After all, we can't afford it and certainly won't be able to afford it when we've all been laid off. Its interesting that even in the world where the creation of food and the other necessities of life would be seemingly effortless, when they talk they always describe it like there will still be an economy. Their idea of an economy, in particular, of course. Yes, I suppose there would need to be a universal basic income so the unemployed masses can afford to pay for the slop you want to sell them. There really is no other choice. After all, what would happen to your bottom line otherwise?
The next question is if people aren't doing anything anymore, how will they know what should be done? I mostly worked in tech (I know) and, more specifically I worked in programming. The thing I noticed is that I don't realize what should be done until I'm trying to do something and I notice that whatever I did before is now making my life harder. I fix it and then remember it, so in future, I take into account likely future needs when I'm working on something. If no one is doing anything, how can they possibly see what should be done?
It is very easy to think something should be done but we have numerous studies of fallacies and pitfalls of human psychology. You're probably familiar with and have maybe even experienced The Gambler's Fallacy and the Sunk Cost Fallacy. Human psychology is manipulable; there are dark patterns in software and games that show that if you set things up right you directly influence human behavior on average. I can't imagine a person living in this world would be able to know what should be done. It seems particularly likely to me (and the Dunning-Kruger effect) that those least suited would feel that they know what should be done and they're frequently the loudest.
Furthermore, how will we know if the AI is right or doing the right thing? People don't do anything but maybe we won't stop learning. We do have a natural drive, after all. Curiosity and interest are what got us here. But if we're not doing things, not overseeing things, how can we know that when the crop fails it's not because AI hallucinated that "electrolytes are what plants crave" is a fact.
The free models we have access to right now are very easily "convinced" of falsehoods and produce true or false statements with equal ease. This hallucination is a feature, not a bug, because it is what allows AI to produce "information" outside of its original learning set. For LLMs in particular, the processing of the training data produces a web of likelihoods of words or concepts (tokens actually) being associated with each other. The added randomness element (temperature) at the point of use allows for it to produce "novel information", where novel information sometimes means outright falsehoods. Not to mention if the original training resulted in the most likely next token being factually incorrect, the LLM would still produce a falsehood (hallucination) even if there wasn't any randomness involved at all. Humans can't evaluate the web to check it for falsehoods, the web isn't really visible as a thing a human can understand and even if it was it is massive. It speaks (writes) like it thinks but it doesn't actually do spatial reasoning. I recently had an AI tell me that cutting a short grain A3 paper in half would produce short grain A4.
The paid models do work better, especially in specific domains and when used by experts, but this structural and conceptual problem still exists in the system itself. The beans they're selling might not even be beans let alone magic.
Finally, if labor is going to be "humanless" why aren't they already thinking of Star Trek style utopian civilizations?
I think we can all guess the answer to this one.