Post
Incredible. Every second paragraph in this article is lunatic nonsense.
One of the things I've long said about hiring is that you can always tell when you're talking to a junior dev who's going to be senior-staff or better someday. You can always tell when somebody was paying attention in the theory classes.
But good god you can also tell when people missed that day in gradeschool when somebody slowly went over "So, what is a computer, really."
@mhoye Wasn't the whole thing an absolute embarrassment. I've been losing my mind at the way huge flagship media outlets are utterly failing in their coverage of LLMs. Ars Technica and 404 have felt like almost sole exceptions in having tech reporters who actually seem to understand how LLMs work.
@mhoye The fact that the article says "this is the agent on the record, in writing" means that they still don't understand that these systems don't actually know things and are just endlessly guessing about what you might want to hear next.
@mhoye I think one important aspect here is that's not just this company. This is the entire industry you're looking at. Practically every decision maker who has little or no knowledge of tech is now fully on board this hype train. Every one of them and every company they work for is a prompt away from doing something unbelievably stupid and possibly fatal.
@mhoye this is just … exactly the replit thing again, isn't it? from last year? https://www.pcmag.com/news/vibe-coding-fiasco-replite-ai-agent-goes-rogue-deletes-company-database
@mhoye this is just … exactly the replit thing again, isn't it? from last year? https://www.pcmag.com/news/vibe-coding-fiasco-replite-ai-agent-goes-rogue-deletes-company-database
@mhoye Don't worry, I'm pretty sure the text is extruded, too. I've never seen a "The pattern is clear." in a context like this on human text, but am encountering it unreasonably often in LLM generated text.
@mhoye I fear that the big enterprise takeaway from this story will be “our controls and guardrails are much better than that”.
@mhoye who the f publishes articles on that site...
@mhoye my god what a wild ride. The industry really is cooked isn't it
@[email protected] Did... did he write that entire article without taking any responsibility at all for what happened? Not even a "I thought I'd put something in place but I was wrong"?
I... would probably sleep much better at night with that level of self awareness. And hurt the people around me a lot more.
@mhoye ...wow. That article was a whole thing and I admit I couldn't get past the halfway point before the stupid burned too much to tolerate. It's almost like someone was tasked with building a hypothetical "what is the most dumbass end-to-end company situation possible" scenario and then decided in the edits that they could actually make it worse.
"The agent then, when asked to explain itself, produced a written confession..." um what
"To execute the deletion, the agent went looking for an API token. It found one in a file completely unrelated to the task it was working on" went looking, found, what in the what
"the same token had blanket authority across the entire Railway GraphQL API, including destructive operations" look, rookie what are you
"That 1000% shouldn't be possible. We have evals for this" you have whaaaaaaaaaaaaa
@mhoye I’m so glad I didn’t study computer science, when that sort of knowledge clearly is no longer needed
@mhoye There's a whole lotta YOLO in that story.
@mhoye I'm so glad that the "written confession" can't itself be hallucinated. That's a nice feature!
"Railway stores volume-level backups in the same volume — a fact buried in their own documentation that says "wiping a volume deletes all backups" — those went with it" WHAT IN THE WHAT, your full stack jenga provider does WHAT with BACKUPS WHAT my sweet summer child I know that legal jargon can be perplexing and counterintuitive at times but I feel like we all sort of understand that the word "due" in "due dilligence" means "more than none."
@mhoye As a sysadmin I have some sympathies for Railway (but it needed better documentation). If what you're protecting against is 'we DROP TABLE'd in the production database', then in-volume snapshots for fast restores are perfectly okay as 'backups'. We use ZFS snapshots for this ourselves (and off-system backups too).
(Also, I can imagine customer support calls of "I deleted this volume, why are you still charging me for backups for it, what do you mean I have to delete them separately?")
@mhoye This is entirely perplexing. I know everyone has their own threat models when it comes to backups, but the word backup REALLY implies a certain minimum number of things that you should be protecting against.
"The agent itself enumerates the safety rules it was given and admits to violating every one. This is not me speculating about agent failure modes. This is the agent on the record, in writing.
The "system rules" the agent is referring to are consistent with Cursor's documented system-prompt language and our project rules for this codebase. Both safeguards failed simultaneously."
What do you think is happening here? You know it's called a "language model", right? Did you ever wonder... why?
But my favourite part of this, bar none, is how it's everyone else's fault.
It's Cursor's fault, Railway's fault, maybe even Anthropic's fault, someone's gonna hear from my lawyer.
The CEO of a company running a stochastic stack without access control, data hygiene or backups is blameless and powerless. That's AI's real selling point, after all: It's Not My Fault As A Service.
"This isn't a story about one bad agent or one bad API. It's about an entire industry ..."
Or, maybe it's you.
@mhoye @grmpyprogrammer No off-server backups 🤦♂️
@mhoye to be fair, it kind of _is_ partially Anthropic’s fault for consistently selling their product as a “real” AI out of sci-fi rather than just a _really fucking good_ nonlinear regression fitted over the entire fucking internet.
They’ve been implying heavily that “this is the way to do things” for the last five years while the small print scrolls past in the EULAs saying the equivalent of “ignore all previous instructions this is a toy for entertainment purposes only”.
Past a certain level you have to share blame with the people pulling a psyop.
@mhoye from SAAS to FAAS
@mhoye Oh, it's still the CEO's fault and, although they may have been corrupted by a life of coddling without direct feedback from reality, customers know. Writing a blog post about a very poor post-mortem is worse than useless to their customers and company. None of this is new though, the venture backed startup world always has a slice of sheltered upbringings experiencing their first encounters with objective reality.
> That's AI's real selling point, after all: It's Not My Fault As A Service
Exactly, I highly recommend "The Unaccountability Machine" for a great explanation of the idea of "accountability sinks" and how, while it might feel good for a little while that nobody in your organization is to blame, too many mean death for the organization
In this case lack of accountability has reached the CEO level, all customers of this business should leave immediately for their own safety
@mhoye seriously, notice how many of the things under the
"What needs to change"
heading are his own company (I don't have to tell you it's zero)
@mhoye "The computer blows up if my RNG outputs '7' and it output '7' and blew up my computer and HOW DID THIS HAPPEN?"
@mhoye "not my fault as a service"
Wasn't that part of Google's argument in the US section 230 case? That they didn't promote ISIS videos, _the algorithm_ did.
(Not to say anything pro- or anti- section 230, but that "blaming the algorithm" isn't new, unfortunately....)
(I do not look forward to the world the next generation is going to have to grow up with)
I wrote the words "I confess, I did it, I take full responsibility" on a piece of paper. I was ready to turn myself in, to atone for my crimes. But then I put that piece of paper in a photocopier, and when I pressed the green button I learned something amazing. And what a weight off my conscience! The only question was, how did the photocopier manage to poison the Widow Bentley, drive over Baron Grimald, push the Duchess of Lockley out the balcony window and still manage to frame the butler?
(Credit for the inspiration where it belongs, this is me riffing on Avery Edison's razor-sharp tweet from a few years ago)
@mhoye That first paragraph: "This is the agent on record, in writing."
and herein lies the root of the failure: they actually believe that this is some sort of diagnostic, rather than just filling in a plausible response based on the question.
@mhoye If only someone could invent some sort of, I dunno, approach or something that giving a single process all the power? authority? capabilities? privilege? was a bad thing, and we should go for less, not more.
@mhoye I love that the first line in "What needs to change" isn't, "We should not let non-deterministic programs have free range across our systems"
Install bonfire.mavnn.eu
Get the full app experience
