Slop or not, AI undermines trust
There's a ton of debate about the impact of AI-generated text on human communities. Even bosses that have rapidly embraced the technology and are extremely enthusiastic about imposing it on their workforces are complaining about the consequences. Shopify's CEO calls drive-by AI-generated comments that reflect a shallow understanding "slop grenades." Substack integrated an AI-detection tool into their platform to give authors ands users a way to attempt to deal with the deluge of AI-generated prose. A lot of people are putting out a lot of slick, overconfident text, and a lot of other people don't like it.
A common refrain when people express their frustration that it is difficult and a waste of time to engage with AI-generated text is:
Why do you care whether the text you're reading is AI generated? Shouldn't it be about the quality of the ideas in the text?
Because our lives are so thoroughly mediated by screens and written language, it's surprisingly easy to forget that language exists in the first place not to produce written artifacts that can be judged according to an abstract standard of "quality" or even "accuracy." Language developed to coordinate activity, cognition, and form social understanding among human beings. Why would evolution have produced it otherwise? Why else would humans have extended it into other mediums?
I don't care that much about language as an abstract expression of thought. I do care about human beings, and hope that language can serve the ends of cooperation, conviviality and the common good. An essential quality of shared human activity is trust.
In her book How to Be Trustworthy, the philosopher Katherine Hawley linked the very act of speech to the idea of trust. She argues that "asserting involves promising to speak truthfully" - specifically, promising "to speak in ways which in fact match the world, not merely to do one's best on this front." When we make a claim, we aren't just generating words for people to hear - we're making a commitment about the truth of what we're saying, whether we realize this or not.
We rely on this when we ask people we don't know for directions or recommendations. We do this in spite of the fact that we may not have enough information to make a judgement about them, indicating that it's built into human communication on a pretty deep level.
We all have known people who just say things in a way that, even when you don't think it's intentionally deceptive, makes us question their commitment to the truth. You hear your acquaintance's tall tales enough times and you're no longer evaluating his words in an abstract sense of accuracy, but instead making a holistic judgment about everything he's saying. It's not worth the effort to question the details because you don't trust him. You can't rely on him to be committed to the truth. In strong enough cases we have a technical term for this: bullshit.
LLMs have a similar relationship to the truth. They flip their responses easily when questioned. They exhibit both overconfidence and a strong sensitivity to potentially contradictory information. They really need to produce the most plausible-sounding next token, even if the information necessary to produce the actually-correct next token is not present in the model or its active context window. 
Would you trust a colleague who always relied on texting their friend before responding or researching a question themselves?
Even if you believe that what an LLM does is "cognition," the person proffering LLM output as their own is substituting something else for their own cognition. They are not in a position to make any kind of commitment with regard to the truth of the text being offered, because they have offloaded all the work necessary to produce an analysis to a machine, and that machine doesn't seem particularly reliable when confronted with alternative ideas or approaches.
As they have become more capable and more people use them, it has become trivial for an AI agent to use an app's CLI or API to respond "on behalf of" someone who grants it permission to use their credentials to post or add comments. These accounts were, in almost all cases, developed with the implicit expectation that an account's activity represented the activity of a human being. Even where there were widespread exceptions to this before LLMs, such as on GitHub, bots tended to be clearly labeled. AI agents have demolished that assumption and broken a social contract that existed across internet communities and workplaces. 
And it isn't just a question of quality. Even accurate and good-quality AI text undermines this trust. A résumé and a cover letter used to reflect a candidate's own description of their experience and interest in the role. The flaws or personal quirks in the writing communicated, however imperfectly, information about whether someone is capable of doing a job. A perfect-quality résumé generated with AI conveys almost nothing about the candidate as an individual - apart from the fact that they felt it acceptable to offload this work.1
Among peers in a community, writing used to be a marker that someone was in the process of forming an understanding of a topic. Even if that understanding was wrong, there was at least a malleable idea forming in a person's mind that could be corrected with feedback. AI writing skips that process entirely, leaving people with a brittle simulacrum of understanding that will fall apart when you have to apply it in a novel situation or revisit the topic later. You'll ace the homework and then fail the exam.
Plus, the prose isn't just untrustworthy — it's annoying.2 AI-generated text presents things in a jumbled, cliché-ridden, out-of-order way that doesn't make sense to a human reader. As Ossama Chaib has thoroughly documented, it generates bizarre jargon that sounds artificially precise on the surface but is actually incredibly vague. Artifacts of the token-generation process wind up in the output text. Regardless of context, it resembles marketing copy. Even people using AI regularly sometimes dislike it so much that they'll burn more tokens attempting to make it sound more like English.
As Charity Majors, certainly no AI skeptic, described it:
Yes, AI is just software. But it is special in one way: the slick, sycophantic, uncanny valleyness of the way it communicates. Human-like, but not human, which is somehow vastly more alienating than messages that are plainly automated. This is why communication that sounds like AI is so degrading to trust. We all know how easy it is to have the machine spit out some bullshit on our behalf, and we don’t want it done to us.
If I'm in the camp that refuses to use LLMs, I have no interest in what they say. And if I do use them, then I have access to the same tools. In either case, the question is the same: what's the point of me reading the output of your conversation with a LLM?
I once had the alienating experience of discussing my reservations about AI's applications in surveillance with someone face to face, only for him to put bits of the conversation into ChatGPT as we spoke in real time. He said, "see, ChatGPT is saying many of the same things you are - it's mentioning risks to privacy!"
I looked him directly in the eye and responded: "If I wanted to know what ChatGPT has to say, I can prompt it myself."