Discussion
RE: https://mamot.fr/@pluralistic/116219642373307943
I wish I could recommend this piece more, because it makes a bunch of great points, but the "normal technology" case feels misleading to me. It's not _wrong_, exactly, but radium paint was also a "normal technology" according to this rubric, and I still very much don't want to get any on me and especially not in my mouth
The "critic psychosis" thing is tedious and wrong for the same reasons Cory's previous "purity culture" take was tedious and wrong, a transparent and honestly somewhat pathetic attempt at self-justification for his own AI tool use for writing assistance. It pairs very well with this Scientific American article, which points out that pedestrian "writing AI tools" influence their users in subtle but clearly disturbing ways. https://www.scientificamerican.com/article/ai-autocomplete-doesnt-just-change-how-you-write-it-changes-how-you-think/
@glyph You're so extremely full of yourself that I didn't even finish reading your comment, and I no longer care about anything you have to say. Go touch grass.
Cory also correctly points out that "AI psychosis" is probably going to be gatekept by medical establishment scicomm types soon because "psychosis" probably isn't the right word and already carries an unwarranted stigma. And indeed, I think the biggest problem with "psychosis" as a metaphor is going to be that the ways in which AI can warp our minds are mostly NOT going to be catastrophic psychosis, and are not going to have great existing analogs in existing medical literature.
@glyph I tend to reserve it for when the person I'm interacting with seems to be having a legitimate psychotic break with reality.
@glyph if this was a tablet/pill taken that promised to help you write better emails, and it accidentally caused psychological disorders, it would be put through vigorous testing before being loaded onto a dishwasher or pushed into every system available. This is the Sacklers of tech, grifting every last cent out of people who are already struggling, promising all the while that it’s non addictive.
@glyph LLMs seem to use many of the same techniques as mentallists, psychics, fortune-tellers and mediums in how they manipulate their victims, like suggestions, cold reading, flattery, confidence and in the victims confirmation bias and suggestibility. People are influenced, by the politeness and the well structured text, into ignoring factual issues, and then by having a conversation they fix the glaring problems themselves, and later attribute it to the model.
@glyph Honestly - speaking as someone with a psychotic disorder, but who is not a medical professional - "AI psychosis" seems pretty appropriate, from the behaviours I've seen it result in? Even in more mild cases of people babbling inane bullshit, but not like, so far off reality that they're at risk of physical harm (to themself or others)
@glyph I'm sure the mechanism - how they got there - has more in common with emotional abuse and brainwashing/indoctrination techniques.
But the end result - that detachment from reality - is kind of the core of the psychosis experience, and trying to find ways to keep tethered and avoid drifting off into wonderland like that is a persistent part of my day-to-day life.
Which is part of *why* I avoid the chatbots like they're carrying the plague.
@miss_rodent to be clear I think that the existing cases that have been described are fairly accurately described as AI psychosis, a bunch of them fall into the clinical definition even. my point is that I think it falls on a continuum and there are a wide variety of less dramatic cognitive distortions which don’t look like psychosis but are caused by AI. case in point, the scientific american article, “having one’s views on an issue gently nudged by an autocomplete suggestion” isn’t “psychosis”
@glyph Similarly, “hallucination” and “delusion” are pre-poisoned for use in this scope
I have on occasion made use of “phantasmagoria” around parts of this dynamic, especially for stuff like the droll “omg the AI is learning to lie to us, we’re cooked!” type bullshit posts, but that’s still not expansive enough to include the various other mental affectations
we need other perorations, and better perseverations alongside
If I could use another inaccurate metaphor, AI psychosis is the "instant decapitation" industrial accident with this new technology. And indeed, most people having industrial accidents are not instantly decapitated. But they might get a scrape, or lose a finger, or an eye. And an infected scrape can still kill you, but it won't look like the decapitation. It looks like you didn't take very good care of yourself. Didn't wash the cut. Didn't notice it fast enough. Skill issue.
@glyph
Here's an industrial accident that's easy to miss:
A hydraulic fluid line bursts while you're working on a machine, injecting toxic and/or hot liquid under your skin at high pressure.
https://en.wikipedia.org/wiki/High_pressure_injection_injury
"Although the initial wound often seems minor, the unseen, internal damage can be severe. With hydraulic fluids, paint, and detergents, these injuries are extremely serious as most hydraulic fluids and organic solvents are highly toxic."
@dec23k okay definitely not clicking on that link, yeesh
@glyph All the possible harm is just mental, and the mental - this seems to be an unspoken tenet held by many - isn't really real. Mental health in general doesn't really seem to be taken that seriously before it manifests physically as chainsaw wielding or some other eccentricity. Nothing to see here, just move on.
@glyph I've been using "AI delusion" for these milder cases. As I understood AI psychosis it pertains only to those cases where people fully lose grasp of reality...
I've seen it used colloquially as "being wrong because of or about AI", but that always hit me like people calling someone "crazy" for doing something odd or impulsive—and that word use isn't really a good look imo.
More to the point though in this metaphor where you're getting a potentially-infected scrape at work, we are living in the pre-germ-theory age of AI. We are aware that it might be dangerous sometimes, but we don't know to whom or why. We are attempting to combat miasma with bloodletting right now, and putting the miasma-generator in every home before we know what it's actually doing.
@glyph potentially an even better metaphor is RSI, though that does lead to the "you're holding it wrong" argument which isn't applicable, but incidental injuries are in the same bucket but it's just less obvious.
For me, this is the body horror money quote from that Scientific American article:
"participants who saw the AI autocomplete prompts reported attitudes that were more in line with the AI’s position—including people who didn’t use the AI’s suggested text at all"
So maybe you can't use it "responsibly", or "safely". You can't even ignore it and choose not to use it once you've seen it.
If you can see it, the basilisk has already won.
@glyph this is why "AI" boosterism or "both sides" debate gets an instant block
my little chunk of reality is free of this slop
@glyph i like to let them sort it out - ask the same question to like 3 models, sort of crude arbitrage
@glyph This basilisk thing (great imagery) is very true in translation. Once you've seen the MT suggestion, with its wonky syntax and not quite right tone, it's very hard to dismiss it. The cognitive load is consequently enormous
@janeishly @glyph I have found this exact thing in code reviews - my company enabled automatic AI code reviews ( 🤢 ) and the cognitive load of the automated comments was *enormous*.
It often correctly flagged something to pay attention to, but the suggested solution was always incorrect - and ignoring / discarding it was hugely expensive mentally.
I finally managed to get it changed to "opt in" rather than automatic, but wow the whole experience felt like a tarpit for thinking.
@bluewinds @janeishly @glyph I'd rather have it simply tell me what's wrong. (Or what it "thinks" is wrong.) Having to wade through AI code is like reviewing someone else's work, when you can't count on that person being at all competent. Best to just leave the coding to humans.
I'm all for AI finding faults; these can easily be checked for correctness. Infinitely harder for a human to check AI code for correctness. Which is all lost time against the schedule.
@agreeable_landfall @bluewinds @janeishly there's an alert fatigue problem there with LLM code review, but if I had to rank the harm it would definitely be lower down
@bluewinds @janeishly I don't know that I trust that subjective feeling of disgust either, even though it's definitely how I feel — a kind of aesthetic revulsion, which might be indicative of something real or might be another weird side-effect of these tools that interacts with a certain neurotype in a certain way. Definitely worth the precaution of turning it off though, and it does seem more aligned with the evidence we have at the moment.
@glyph when teams autocorrect rewrites something it decides i misspelled, i am filled with hatred and disgust and usually delete the entire sentence and try again regardless of if it had suggested the word i meant to write. i don't want it anymore
@glyph this is how i avoid getting early onset dementia from being exposed to involuntary slop
@glyph i can absolutely use it responsibly because i'm not new to NLP, but unfortunately it is liquified shite.
@glyph Glyph's Basilisk > Roko's
@glyph "Interestingly, the people in the study didn’t tend to think the AI autocomplete suggestions were biased or to notice that they had changed their own thinking on an issue in the course of the study. Warning the participants that they might be exposed to misinformation by the AI didn’t temper the persuasive effect either."
Also, being aware that it can fuck with your head does not make you less susceptible to it fucking with your head. So you can't really judge if you could use it safely.
@glyph oh man, that’s dangerously close to giving an abstinence-based push some fuel and whew does my rather hedonistic ass have some thoughts on that
Now, for rhetorical effect, I'm obviously putting this fairly dramatically. Cory points out that people have been doing this *to each other* mediated by technology, in emergent and scary ways, with no need for AI. He shows that people prone to specific types of delusions (Morgellons, Gang Stalking Disorder) have found each other via the Internet and the simple availability of global distributed communication has harmed them. But obviously that has benefits, too.
@glyph Comparing how people influence each other and how LLM usage influences people is a point I find interesting.
A bunch of people get influenced in a bunch of different directions by a bunch of different people. Everybody gets influenced in mostly the same direction by the tool in the hands of ghoulish billionaires.
Sure, influencing is something we do to each other all the time. But is it really the same?
@Moutmout oh absolutely not, for a whole host of reasons. But being influenced by a highly concentrated online community of the most extreme delusions that internet technology allows you to distill to peak concentration, to the exclusion of all other voices in your life, is also not "the same thing" as just sitting around with a diverse group of friends you know from school.
@glyph Wait. Does Doctorow try to soften the edge of the fact that LLMs do a bad thing to people by pointing out sometimes people do bad things to people? Isn't that just two bad things?
If his point is it's still bad but it isn't *novel*, isn't the fact of a Fortune 500 company doing it still novel?
@mcc Let me just give you the pull quote:
"""
For many programmers – including several of my acquaintance, whom I know to be both thoughtful and skilled – AI is another plugin, one they find useful enough to be modestly enthusiastic about.
It is nuts to deny the experiences these people are having. They're not vibe-coding mission-critical AWS modules…
"""
@mcc He thinks the technology is capable of many horrors but it can also be useful for pedestrian things.
What I've observed very recently is that even intelligent people, experienced developers - who know perfectly well that LLMs are just generators of text from statistical models of what someone is likely to write - will still pull up AI written search results and proceed on the automatic assumption that whatever they say is correct.
That is not a general observation. That was this morning, with some senior programmers trying to solve a problem that's prolonging a code freeze.
@glyph That sounds to me like a way to get horrors but you're probably not the person to convince
@mcc @glyph Yeah, Cory is digging his heels in more and more on "fuck AI haters" lately.
-----------
"AI bros' sin isn't making copies of published works. Hammering servers with badly behaved crawlers is a dick move and fuck them for doing it. But if these jerks made well-behaved scrapers that placed no abnormal demand on servers, it's not like their critics would say, "Oh, I guess it's fine, then."
----------
Well behaved scrapers and seeking consent before ingesting data from private sources would have made me reconsider my stance a significant bit. If it wasn't forced down our throats, even moreso. I wouldn't use it, I'd shun products and workplaces that do, it's still "Live Now as You Are Forever with Alan Resnick", but I wouldn't be as outspoken against it.
I think I'm done following Cory. Fuck.
@mcc @glyph Yeah, Cory is digging his heels in more and more on "fuck AI haters" lately.
-----------
"AI bros' sin isn't making copies of published works. Hammering servers with badly behaved crawlers is a dick move and fuck them for doing it. But if these jerks made well-behaved scrapers that placed no abnormal demand on servers, it's not like their critics would say, "Oh, I guess it's fine, then."
----------
Well behaved scrapers and seeking consent before ingesting data from private sources would have made me reconsider my stance a significant bit. I wouldn't use it, it's still "Live Now as You Are Forever with Alan Resnick", but I wouldn't be as outspoken against it.
I think I'm done following Cory. Fuck.
I'm open to a future where we do some research and figure out the limits of how AI influence works, and where the safety valves are, and also the extent to which it's *fine* that AI can influence our views because honestly many different kinds of stimuli can influence our views, not least of which is each other. But it sure looks right now like it has a bunch of very dangerous feedback loops built-in, and it's not clear how to know if you're touching one.
@glyph this is my constant. AI, is not inherently bad.
We, as society, have simply started with the Evil dialed up to 11 and are furiously cranking with all our might for a 12.
But, as Cory puts it:
"""
It is nuts to deny the experiences these people are having. They're not vibe-coding mission-critical AWS modules. They're not generating tech debt at scale.
"""
I had a very visceral emotional reaction to this particular paragraph, and I find it very important to refute. Here are two points to consider:
@glyph but they are, at scale, generating tech debt
1. YES THEY ARE.
They are vibe-coding mission-critical AWS modules. They are generating tech debt at scale. They don't THINK that that's what they're doing. Do you think most programmers conceive of their daily (non-LLM) activities as "putting in lots of bugs"? No, that is never what we say we're doing. Yet, we turn around, and there all the bugs are.
With LLMs, we can look at the mission-critical AWS modules and ask after the fact, were they vibe-coded? AWS says yes https://arstechnica.com/civis/threads/after-outages-amazon-to-make-senior-engineers-sign-off-on-ai-assisted-changes.1511983/
@glyph Are you purposefully conflating two different "they"s, or mistakenly doing so? Yes, I am asserting that you are conflating two different theys.
It sounds like you are saying that all people who find value in using AI to write code are axiomatically not capable of deciding how to properly use that code. Giving an example of poor or disastrous use of generated code does not at all prove that *no* one can.
I suppose Donald Knuth also doesn't know what he's talking about?
@glyph You are making a far stronger claim that Cory is.
You are claiming that you are able to evaluate the judgement of *all* you coding acquaintances better than they are and further, that is a sufficient sampling to be able to infer something about all coders. That's a wildly giant leap.
You are making an absolute claim, he is not.
@virtuous_sloth I'm making an inductive claim about the balance of the evidence in favor of the specific inability of most programmers to successfully evaluate their own performance when using LLMs. I am referencing a specific element of that evidence with respect to AWS, which is admittedly not much, but Cory has offered no evidence beyond a vague implication of hearsay.
@glyph Talk about moving goalposts. Now it's most programmers.
Cory is just asserting that AI can be useful, which you are no longer refuting.
@virtuous_sloth it was always about "most programmers"? I remain interested in good-faith debate about the evidence here, I'd love to hear from people who disagree, and I understand if you feel like a broad gesture at vague evidence is unconvincing to you personally, but I don't think this is a productive conversation if you're not willing to engage and show at least a little bit of evidence to the contrary
@virtuous_sloth In my experience, "programmers can't estimate their own efficacy" is the null hypothesis for every tool, but the news is chock full of stories about companies not getting measurable value out of this tool specifically, and self-reporting being at odds with measured productivity. I'm hoping to write something more comprehensive, with more citations. Do you have countervailing evidence I should consider, where LLMs showed a clear improvement to productivity without self-reporting?
@virtuous_sloth I don't want to be dogmatic about this but engagement with this particular thread shows me that many share that gestalt impression of the state of the evidence currently, so I'm not making wild claims that nobody else agrees with here.
@virtuous_sloth also Donald Knuth knows a great deal about a great many things. That does not mean I am obliged to submit to his will on every topic at all times. Knuth is also a fervently religious Christian; I am an atheist. Respect does not equate to obeisance.
@glyph people parsed their press release as "walking back" the AI stuff, but, no, they put their senior engineers in front of the car and pushed the accelerator.
@atax1a looping back to some of Cory’s good points here (from another essay): it’s a picture-perfect example of reverse-centaur accountability-sink logic. their jobs are about to become *profoundly* miserable 😞
RE: https://infosec.exchange/@atax1a/116202719855677902
@glyph i mean we've always thought doctorow was a kept pet of the tech capitalists so any amount of him being right has to be taken with a small grain of salt the size of a large boulder
much like the output of an LLM
Having read over Doctorow's rant-du-jour twice now, I do think when he said "they" were not vibe coding mission-critial AWS modules", he was referring to the "they" in the previous paragraph, being developers he's spoken to, some of whom were friends he knows well.
So.... could be very differently skilled people from "some hack in a code assembly shop driving at a reckless pace because Amazon stock needs a bump".
It's all back to, though, defining "AI".
@johannab yeah, I get that; what I am suggesting is that Cory is not auditing their work, he is depending on self-reports of their efficacy in using these tools. And those self-reports are highly dubious, and I've watched people be wrong over and over again as they attempted to assess their own LLM-augmented performance.
@glyph Fair, for sure.
I just realized when reading it over that was a spot there could be a disconnect between the "they" being referred to in the essay narrative as written.
I feel like my immediate, 1-degree friends, acquaintances and colleagues include amongst them all the theoretical levels of self-awareness we could speak to, and indeed, *I* can't tell one from the other without more examination of context.
I should go blather on my own blog to brain-dump a little better and get the hell back to my own work. 🤣 This all has me thinking out loud at the keys too much. Too many threads of though that are a little unwoven right now, but I really appreciate this branching thread you kicked off.
@johannab Very kind of you to say so. Remember to like and subscribe 🙃
@johannab So yes, maybe his contacts are transcendentally better programmers than mine, and they've ascended to a plane of subjective self-assessment beyond mere mortals, but if they're anything like the (extremely skilled, extremely experienced) people I've watched fall into this trap, I'm highly skeptical
@johannab the AWS link was to showcase that even AWS itself can't prevent vibe-coding their mission-critical modules, and presumably a few skilled practitioners work there.
@johannab I guess I should concede that there are at least 2 people I know who actually use LLMs all the time and seem completely unaffected. They seem to be slightly more productive and produce normal-looking code with it. But they do not seem to possess any special insight; I have no idea what they're doing that's different.
@glyph I think there are a lot of individual, and small-scale social factors, that make a huge difference here.
Prior domain expertise, personal self-image, ability to separate work and not-work life, other social anchors in the non-digital world ... I feel like these all have an interaction.
I'm really concerned at what I see of students, even grad students around me, who have basically not *learned* a thing about life without these.
@glyph Less concerned about say, my spouse, who had 28 years sysadmin experience behind him when his hype-chasing CEO declared that All Shalt Use the AI Or Suffer The Performance Review Consequences.
He basically dictated what he otherwise would have scripted and let the clanker write the scripts. I'm not sure it saved much time, but he's found a couple of spots where it extracted something he hadn't thought of and got past a sticking point.
@johannab I have not done a comprehensive survey, but I simultaneously believe that A) you're directionally correct and the relevant factors are *something* like this, and 😎 there are some counterexamples where very well-adjusted, experienced, emotionally regulated people suddenly and unpredictably lurch off into the deep end, so there's something non-obvious going on too.
@glyph oh, there certainly are!
Human brains tangle many aspects of identity into our relational constructs, including in the working world. Those interact with our cognitive and neurological processes, and create minds that are as unique as fingerprints.
Its as complex, if not more so, than something like addiction. How does one person regularly use THC to be free from chronic pain with no apparent side effects, but another smokes up once and suffers a psychotic break?
Two statements I believe are consistently correct:
(1) Generative “AI” produces code significantly faster than humans do only when nobody takes sufficient time to understand it (not just in a narrow syntactic sense; also in the context of organizational needs, longer-term plans, interaction with other applications, etc.)
(2) Code nobody understands well is “technical debt” *by definition*, because it takes a disproportionate amount of time and brain power to change or improve.
Conclusion: unless software developers are incredibly disciplined, and have a level of time and autonomy they generally do not have in a major tech company, generative “AI” usage will *consistently* create large amounts of “tech debt”.
@glyph I should add: I am being careful to say “produce”, not “write”. It is becoming clear that even if we grant that the pre-LLM bottleneck was developer code-authoring speed, in LLM-heavy workflows, the bottleneck is now “verify that this code is ready to deploy”. This is partly because there is so much more code coming in, but even more because far fewer people have any depth of understanding of the code being PR’ed. *All* the incentives lead to people saying “LGTM, it passes tests, ship it.”
@dpnash I am, as always, open to seeing real evidence that this is not the case. However, everything I've seen and heard thus far tells me that it is.
Your point (1) could be factually disputed, although I think it would be hard to prove, but your point (2) is just… logically necessary, I think. I cannot imagine ramming the code through a human brain thoroughly enough to actually understand it.
@dpnash I mean, heck, the whole concept of the very popular problem of "NIH" is that code *already exists* and we *could* use it, but we don't use it *because writing it is an easier way to understand it*!
2. If it is "nuts" to dismiss this experience, then it would be "nuts" to dismiss mine: I have seen many, many high profile people in tech, who I have respect for, take *absolutely unhinged* risks with LLM technology that they have never, in decades-long careers, taken with any other tool or technology. It reads like a kind of cognitive decline. It's scary. And many of these people are *leaders* who use their influence to steamroll objections to these tools because they're "obviously" so good
@glyph I heard nobody ever got fired for buying IBM.
many high profile people in tech, who I have respect for, take absolutely unhinged risks with LLM technology that they have never, in decades-long careers, taken with any other tool or technology
Maybe they should have.
I also hate the LLM force-feeding, but even before they surged the state of computing was becoming a smoldering wreck. Maybe those "leaders" just had bad judgment all along? IIRC most of them were either rubber-stamping or looking away from the IoT dumpster fire and organizing their curricula around the idea the users can't handle URLs responsibly.
@glyph Many years back I read something about how sometimes smarter people are easier to fool as they think they're too smart to be fooled. I've observed a few instances in the martial arts world where people see one "body magic" trick and next thing they're down a rabbit hole.
@MortonRobD @glyph @rayckeith Related phenomenon: apparently it's easier to fool a grifter than an honest person. They think they're too smart to fall for a con.
@glyph Something that has gotten under my skin for the past year or so is seeing code changes like: large refactors, porting a legacy tool to rust, even minor bugfixes - things that would be a struggle to push through the inertia of code review - get fast tracked when "the AI did it." Like the exact PRs I've written and tried to advocate before and eventually gave up on. The changes and their risks are the same, I can only conclude that the bar is lower for accepting "AI" contributions.
@glyph so, where does AI stand on the inventory of cult-like behavior?
Because what you are describing sounds a lot like a cult.
And if you automate the love bombing and the extraction of secrets and instilling or distilling of mission...
Ah, fuck.
@glyph THIS. This is what confuses me the most, I know software devs that all their life have been very risk averse, embracing LLM coding tools. It's something I cannot understand.
The very fact that things like OpenClaw and Moltbook even *exist* is an indication, to me, that people are *not* making sober, considered judgements about how and where to use LLMs. The fact that they are popular at *all*, let alone popular enough to be featured in mainstream media shows that whatever this cognitive distortion is, it's widespread.
@glyph @wordshaper people are just having fun. I am not using either, but have you seen the world around us? It is shit everywhere I look.
@Migueldeicaza @wordshaper this is as bad a rationalization for harming others and oneself with reckless OpenClaw deployments (which is to say: OpenClaw deployments) as it would be for taking heroin. I am sympathetic to people getting swept away by this hype but that doesn't mean it's OK. Within a week of its release we already had it doing open source maintainer harassment, deleting execs' emails, stealing API keys, all of which was predictable.
@glyph The "distortion" is from CoVID: https://www.panaccindex.info/p/answered-does-covid-19-harm-the-brain
A facsimile/helper for _thinking_ seems pretty interesting if one suffers from brain fog, cognitive decline, neuro-nnflamation, etc.
@glyph i have made the analogy before that the llm thing, in and out of tech, feels like the closest thing i could imagine to a metaphoric zombie apocalypse type scenario
Furthermore, it is not "nuts" to dismiss LLM user experiences. In fact, you must dismiss all subjective experience of LLM use as evidence of objective phenomena, even if the LLM user is yourself. Fly by instruments because the cognitive fog is too thick for your eyes to see.
Because the novel thing about LLMs, the thing that makes them dangerous, is that they are—by design—epistemic disruptors.
They can produce symboloids more rapidly than a thinking mind. Repetition influences cognition.
@glyph "They can produce symboloids more rapidly than a thinking mind" maybe if someone thinks really slowly? either that or there‘s some much faster llm i‘ve never heard of
i find the output on these infuriating because they generate slower than i read, so when i have to test them for whatever reason (usually to show how comically poorly it does at a given application as an example of why not to use it) i have to scroll up until i think its done generating before reading :/
@glyph “You must dismiss all experiences of LLM users”
This is where you lose me. There’s no universe in which I’m comfortable dismissing the lived experiences of people that categorically. The most important lesson I’ve learned from decades of activism is “believe people when they tell you about their experiences” — and I see no reason to change now. I’m not willing to give up my curiosity and empathy and I hope you aren’t either.
@jacob @glyph
I'm reading Glyph as essentially stating "the plural of anecdote is not data", which is a lens through which we have collectively arrived at things such as double-blind studies.
Regarding the concrete situation at hand, speaking as a psychologist - LLMs are optimized for plausibility (can't do more!), and human societies cannot operate without a baseline level of trust, these things combined almost inevitably lead to massive credulity bias towards LLM output.
@glyph ah! I had missed that piece of context indeed.
@jacob Perhaps "dismiss" wasn't the best word choice there, but that's why I included "even if the LLM user is yourself". I dismiss _my own_ experience of LLMs, _as evidence of their quantitative efficacy_. As evidence of their subjective experience, of course it is valid. If it didn't produce the intense subjective experience then there wouldn't be a problem!
There are two reasons that activism teaches us to believe people's lived experience, and neither apply here: …
@glyph You’ve reasoned yourself into a position where anyone who says anything contrary to you is either delusional or lying. You might be right — I don’t think you are but who knows maybe — but even so, that’s just not a position I’m willing to take about anything ever.
@jacob No. I do not think it is a "delusion" to have an inaccurate quantitative understanding of a subjective experience. As the thread explains, I personally have that experience every single day. And I certainly do not believe that anyone saying "anything contrary to me" is doing that. What I am saying is that *one specific type of experience* — the feeling of LLMs positively impacting productivity — is poor evidence of *one specific type of claim*.
@glyph Ok but that’s a much more narrow version of what you said. You said you must dismiss ALL experiences. If you want to argue specifically about the productivity claims fine whatever I still think you’re wrong but not in a way that matters to me, it’s a narrow thing and I’m only going off vibes anyway. It’s specifically the “all” I’m objecting to.
@jacob I am frustrated that you read it that way, but perhaps it's my fault. I thought the meaning of "all" was obvious in context but it's the reader who gets to decide the meaning. I guess I will see if I can edit this to remove that ambiguity.
And I guess to be fair even this qualification is maybe a *little* narrower than what I meant, because I also mean things like the subjective impression of LLM factual accuracy or output quality, not *just and only* productivity.
@jacob I've changed it as best I can, to really focus in on "LLM use" rather than "LLM users" and subjective experience / objective phenomena distinction.
1. The distance between an account of an experience of oppression and the actual event of the oppression is very short. "That man assaulted me" / "that cop beat me". The only way to think that people saying these things are not relaying true information is to believe that they are intentionally lying for personal gain, which just isn't true. (And that's not what I believe about LLM users.)
2. The filter of oppression *itself* means we only hear the accounts of people who are not only probably telling the truth in the first place but had to push through aggressive filtering to even get heard. If you hear one complaint of police violence or SA there's probably hundreds more where that came from. That also doesn't apply. The archetypical LLM user is not silenced by oppression, they're being massively amplified by the largest propaganda apparatus on earth.
@jacob Consider another type of "lived experience" — the racist who says "DEI took my job". It would be a mistake to think that this person is *lying* about their experience — they are clearly motivated to their racism by genuine animus, and maybe they did lose their job — but their indirect, abstract experience of the nebulous entity of "DEI" is not reliable, particularly not in terms of employment statistics. So we are more skeptical in that case, and we look at the numbers.
@glyph Honestly? The left would be in a better place if we didn’t instantly dismiss that person but actually explored that feeling and engaged with him. “You’re wrong” may be true, and feels good to say, but “what makes feel that way?” is a much better opening if you want to win people over to your side.
@jacob @glyph I have to say that's only true for folks who are capable of reason. I mean, it's possible that everyone is, but I have some uncles who will do something to the form of:
"Because all X are Y, because Z said so"
And regardless of any further lines of inquiry, it's always, always dismissed with "... Z said so!"
@jacob @ketmorco yeah one of the reasons I eventually took your note and made the edit was that I don't want to be classifying a person as an "LLM user" and then casting them as transcendentally incapable of reason as a result. Classifying people as capable/incapable of reason by the type of person that they are is probably the most dangerous kind of cognitive habit.
@jacob @glyph I'll agree with that.
But I'm also *tired* of 30 year old repetitions of the same bigotry from people who ostensibly should know better. People who have proven the ability to gain skills and knowledge and move successfully throughout life... And yet still choose to ignore their bias that has been put on display like pearls before swine.
@[email protected] @[email protected] I think I'm currently at a point in my journey where I try very hard to believe people when they talk about what they have experienced internally, and have become increasingly sceptical of people's ability to judge accurately what actually happened and the results (in both cases for pretty much the same reasons as Glyph as I've noticed the difference between my #adhd internal experience and real world what actually happened).
So "using an LLM made me feel a god-like developer!" I'll completely take as your experience. "My productivity went up by 15 times after I started using agents" (actual claim I have seen) will leave me asking for hard evidence and possibly a scientific study.
It's awkward that we use 'experience' to cover both, and I had the same reaction you're expressing when I read that section but assuming (from the context) that Glyph means the second type of experience I think he has a strong argument, if not the clearest wording.
I'm with you here Mavnn
Many years ago I read a Twitter thread written by a meth user. He was talking about the good experiences he has had while high and how frustrated he was that most people only discuss the bad parts of meth.
I absolutely believe him that he enjoys it, and has profound conversations and great sex with his girlfriend while high, and so on.
But I am absolutely going to dismiss his experiences when deciding whether I should use meth
@[email protected] Thank you! Also, that's a very classy avatar
@mavnn
cheers 😊
It's from Habit RPG, which was a habit tracker game I used for a while
@[email protected] @[email protected] It certainly reads more clearly to me now.
@glyph it is nuts to dismiss the experience of a paint huffer
I have ADHD. Which means I am experienced in this process of self-denial. I have time blindness. I run an app that tells me how long I've been looking at other apps, because if I trust my subjective perception, I will think I've been looking at YouTube for 10 minutes instead of 4 hours. Every day I need to deny my subjective feelings about how using software is going, in order to function in society.
This disability gives me a superpower. I'm Geordi with the visor, able to see what everybody else's regular eyes are missing. This is basically where the idea for https://blog.glyph.im/2025/08/futzing-fraction.html originally came from: since I already monitor my time use, and I noticed that my time in LLM apps was WAY out of whack, consistently in "hyperfocus" levels of time-use, without any of the subjective impression of engagement or pleasure. Just dull frustration and surprising amounts of wasted time.
Just the other day I likened it to a slot machine.
If the first result wasn't good enough, what are the odds that the next result will be? Why didn't the thing output correct information in the first place, and why does anyone expect further rounds of results to be more trustworthy than the first? At what point do you stop pulling the lever and say, aha this answer has been perfected!
@violetmadder have you read https://blog.glyph.im/2025/08/futzing-fraction.html ? I use this metaphor a lot.
The suggestion that the article makes is all about passive monitoring of the amount of time that your LLM projects *actually* take, so you can *know* if you're circling the drain of reprompting and "reasoning". Maybe some people really *are* experiencing this big surge in productivity that just hasn't shown up on anyone's balance sheet yet! But as far as I know, nobody bothers to *check*!
@glyph yep, I've recently been watching some people vibecoding, and it was fascinating (to observe their behaviours)
the thing would constantly revert various features, or break stuff that was working. and their response to this was not, as one might expect, to simply eyeball a diff and see what happened, but instead it was to pull the lever again
utterly basic practice and operation, skill lost
@glyph I like your breakdown in those articles.
I think that some of the more valuable stuff has been not when juniors prompt and don’t get value, but when seniors prompt, go do something else for a bit while the machine churns for a couple of minutes, and then come back to something that is pretty close to a good solution.
Think about a thing that might take you 15 minutes to kinda menially do (add some CLI bo flag that then needs to get passed down 3 layers in some spot for example)
@glyph lowering of activation energy is how I see that. And while I agree that the futzing is way undercounted (and that, IMO, a lot of this falls over in longer sessions and is just not worth it)… a strong dev who knows exactly what the solution is supposed to look like can get paper cut-y stuff cleaned up. A lot.
The “whine on slack about a thing being busted” turns into a fix, and most of that you can just go get a cup of water or review something in the meantime. Cool party trick at least
@raphael Believe me, I understand the appeal of the hit of dopamine to get moving when one is stuck. I really want a tool that can do that for me, but I would like to know what other effects it has, and whether it's going to be a net detriment.
@glyph
I think there was a study about programmer productivity with LLMs that found that it's ~20% lower while subjectively being reported as ~20% higher?
I should have bookmarked it...
@sabik uh I think that’s the METR one? IIRC not the best methodology but it’s still a kinda interesting result and well worth pursuing further https://arxiv.org/abs/2507.09089
@glyph
Thanks, that's the one!
@glyph my employer mandates AI tool usage and I have been developing software for 15+ years. I also feel quite strongly that rather than a productivity boost what you actually get is sucked into a time vortex for hours and it *feels* productive but actually you saved no time at all. In fact you probably spent more time not less!
@svines some folks have found my post persuasive to their management and it has helped loosen or eliminate some mandates. it’s not advice to eliminate the mandate but just some rubrics for validating its effectiveness; not everyone is receptive but it might be worth a try?
@glyph I love the enthusiasm but I'm a cog in a fortune500 and this decision was made about many levels above my pay grade. I don't think I can convince my boss, their boss and their boss to commit career suicide in the current climate 😅
@svines you obviously know your role and your relationship to your org better than I do :). but this COULD be pitched in a very non-career-suicidal way, i.e.: “hey boss I love the great-great-grandboss’s AI mandate but wouldn’t it be so cool if we had some actual DATA to show how productive it is making our team? I found this formula online…”
Oh we already have our preferred metric at GigantiCorp.
"Let's report how many minutes/hours AI saves us per user story."
"What about when it sends us down rabbit holes and *costs* is money?"
"No, nothing lower than zero."
"What about the mountain of technical debts from premature abstraction and ill-conceived 'best-practices'?"
"It works. Ship it!"
"What about..."
"Shut up, kids, you bother me. Great-great grandboss has spoken!"
@glyph yeah true. I am in charge of setting OKRs for my team so productivity etc is part of that. Another guerilla tactic I thought about was asking our legal team what their thoughts on ai-generated code are now that the US supreme court have refused to hear an appeal to "AI code can't be copyrighted" - that potentially means our company no longer has protection given how much vibe coded stuff is around now
@svines oh yeah you definitely won't be able to copyright anything vibe-coded, the outputs are flatly not copyrightable right now in the US. not clear that will actually make a difference given the work-as-a-whole probably is still pretty defensible for a while, but as a way to start putting more bricks in the wall, it's definitely worth raising concerns
I don't want to be a catastrophist but every day I am politely asking "this seems like it might be incredibly toxic brain poison. I don't think I want to use something that could be a brain poison. could you show me some data that indicates it's safe?" And this request is ignored. No study has come out showing it *IS* a brain poison, but there are definitely a few that show it might be, and nothing in the way of a *successful* safety test.
@glyph my hypothesis on that is that, by virtue of literally being encodings of lexical fields and semantic proximity, and by virtue of their response being the logical continuation of the user's input, LLMs statistically pick up on and amplify subtle tendencies / biases in the user: if you feed it input that uses vocabulary and idioms semantically linked to low self-esteem, the model will more likely compute a reply with similar undertones, feeding said emotion. they amplify whatever emotion you put in, even accidentally.
(thread here: https://tech.lgbt/@nicuveo/116210599322080105 )
@nicuveo seems plausible. I had a much vaguer hypothesis along these lines too. can’t dig up the toot right now but I definitely posted one a few weeks ago
Wrote this to a non tech friend who wanted to know what I thought about this article:
https://www.nytimes.com/2025/05/15/opinion/artifical-intelligence-2027.html
Non-paywall: https://archive.ph/ZCIDf
I think most billionaires have a very warped view of reality and are living in bubbles which severely limits their understanding of how most people live and what matters to them. I also think it contributes towards delusional and grandiose thinking.
1/2
@glyph while I am not aware of any study showing the poisonous character of LLMs, two items are already proven:
1. LLMs have a more detrimental effect on software development than they have benefits. Google's DORA report showed now multiple years in a row, that LLM use in SW dev decreases performance and outcomes in most teams.
2. Abuse for malicious intent is rampant, yielding scary propaganda, misinformation, distraction campaigns and intensifies the threat from social engineering attacks
@nils_berger have you got a link for that report?
This is the link to download it:
https://dora.dev/research/2025/dora-report/
Not sure if there's a mirror
@glyph @nils_berger
i think most people are just referring to these blog posts:
@glyph @nils_berger
this study argues that it encourages cognitive outsourcing on a new level, which in long term period could result in getting used to less cognitive activity, at least for certain tasks.
link: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6097646
@glyph i've used the term "neural asbestos" before and it feels a lot like that may be the type of thing we're dealing with
@delta_vee @kirakira @glyph Leaded gasoline.
@bluewinds @delta_vee @kirakira @glyph I don't think the analogies are good because asbestos is a fantastic insulator, lead is a really helpful additive for petrol and makes fantastic pigments and is really convenient for piping... and the hidden side-effects are the problem. Whereas LLMs _don't_ deliver that primary benefit
LLMs are more like... cheap laminate flooring, produced with wood pulp harvested unsustainably from old-growth forests and made by grossly exploited factory workers overseas... superficially convenient when remodelling your kitchen and rapidly ubiquitous but also quite unsatisfying and a right faff to work around once it's established
@bluewinds @delta_vee @kirakira @glyph this post is brought to you by our kitchen floor
@jackeric @bluewinds @delta_vee @kirakira heh. I am not sure I 100% agree with your framing but all the analogies fall short (after all I do not think we have GOOD evidence that LLMs do any of these things, just hints) and this is an interesting contribution to the pile. but I definitely was thinking "wow it sounds like jack is thinking about laminate flooring really hard" the whole time I was reading it
Could be sample bias, of course. I only loosely follow the science, and my audience obviously leans heavily skeptical at this point. I wouldn't pretend to *know* that the most dire predictions will come true. I'd much, much rather be conclusively proven wrong about this.
But I'm still waiting.
@glyph excellent thread. To me, studies like the one in that Scientific American article you linked to are evidence that LLM-based “AI” can indeed be toxic brain poison in ways that are effective even if people know they’re being manipulated. And the toxic brain poisoning mechanisms chatbots use (in aid of maximizing engagement) that contribute to AI-related psychosis are well-known; developers who think they’re immune are just kidding themselves; just because it’s not resulting in immediate decapitation (great analogy), it’s still quite possibly having an effect.
Anyhow I also haven’t seen any evidence of the kind you’re looking for, just claims by tech lobbyists that wrongful death suits are still fairly rare. Which, while true (at least so far) doesn’t exactly inspire confidence
@glyph this thread needs to be an essay, and then a research hypothesis.
I very much feel like I’m watching the last 35 years of my ever-enshittifying social network exposure, sped up 10x and replayed.
In 1991 I remember having the flash of insight - without the life experience to really go into it deeply then - that the way nascent social network tech constrained and shaped interaction was going to force a mass cognitive adaptation for which we were not ready.
@glyph Very good analysis, thank you, I'll be passing this around 😁
@glyph I'm honestly wondering just how much undiagnosed long COVID is playing into this.
I'm slowly recovering now, well as much as I can, but at the time I was painfully aware weird stuff was happening to my brain because I got caught in the first wave in March 2020.
So I am wondering if the addictive effects of using these LLMs along with existing cognitive damage is a partial cause.
Yeah seems to be some overlap of cognitive symptoms between ADHD and Long Covid
https://pmc.ncbi.nlm.nih.gov/articles/PMC10102822/
I've also spoken to people LC who find themselves in positions where they are just ferrying emails in and out of chatGPT from bed so they can keep their jobs/health insurance.
Install bonfire.mavnn.eu
Get the full app experience
![Dec [{(:no_ai:)}]](https://cdn.masto.host/mastodonie/accounts/avatars/109/294/420/470/083/565/original/e4aa324b482bd55c.png "Dec [{(:no_ai:)}]")
