I Stopped Chasing New Model Releases. The Hallucinations Didn't Stop.

Claude Fable 5.1 just shipped, and Anthropic says it’s already live on every platform.[19] Half a year ago I’d probably have started hunting for benchmarks, building tables, getting ready to ride the news into an article. Seeing it now, I wasn’t that excited. My first reaction was “oh, another update,” and I swiped past.

There have been a lot of releases and updates this year, coming fast, but the intelligence of these models seems to have hit some threshold. Each new model tops some leaderboard and claims to be the strongest ever, and yet in actual work it feels about the same as the one before — and on some evals it even looks like it regressed.

First, I’m genuinely lucky that the company I work at lets me use the current frontier models (OpenAI and Anthropic) for free, at nearly unlimited volume. I used to sit and wait for release announcements, then ride the news, use the model to scrape all kinds of benchmark write-ups, and stitch them into an article. This past half year I basically never stopped, using the newest model every day to get work done — and the enthusiasm slowly drained, while a stack of questions I couldn’t articulate piled up. This piece is me laying those out one at a time, and, along the way, having AI dig up a fair amount of material to see whether it’s my imagination or whether things really are moving fast.

Four Opus generations in half a year, and I didn’t feel any better at my work#

Opus 4.6 shipped on February 5, 2026; Opus 5 shipped on July 24.[1][2] In between came 4.7 and 4.8 — 4.8 landed on May 28, roughly six weeks after 4.7.[15][16]

Which means: in half a year the flagship Opus model shipped four generations, at a cadence of about once every six weeks, and across all four the price never changed — $5 in, $25 out, per million tokens.[1][2][16]

Figure 1: across four generations the price didn’t move, and the difference I could feel kept shrinking

A lot of the comments I scroll through lean toward the 4.6 version. That’s probably a tiny sample, and I have no large-scale preference data to back it up. I’ve also seen people explain it — that the newest model “doesn’t talk like a person” is supposedly a form of overfitting — but I couldn’t find any evidence to support that, and forcing the framing on just makes a judgment look more solid than it is.

So I can’t explain this one either. I’ll only keep the personal impression: the releases come more and more often, and my “wow” comes less and less.

Then why do I reach for the newest version every time?#

If the new models really are useless, why do I keep switching?

Thinking it through, I’d mushed “the model is stronger” into a single blob. It’s really at least four things: capability ceiling, factual reliability, speaking style, and cost and speed.

• The capability ceiling really is rising. I’ll grant that. Anthropic said Opus 5 was the strongest at the time on several coding and knowledge-work evals.[2]
• Cost barely moved — all four generations were 5/25.[1][2] Speed is sometimes faster, sometimes not.
• Output style is a matter of taste, and it’s where “I prefer 4.6” comes from — purely subjective.
• And factual reliability — whether it will confidently make things up — that’s the part that, in my experience, barely budged.

Figure 2: the capability-ceiling curve climbs, while reliability and my real output stay pinned near the floor

Even the model vendors say the leaderboards aren’t worth that much. Anthropic’s engineering blog put it plainly: at the top of these coding leaderboards the ranks often differ by only a few percentage points, and they even went and studied the “infrastructure noise” inside the evals.[10]

The leaderboards get more absurd than that. A third party mapped it out: in June 2026 the top of SWE-bench Verified reported 95%, but 99 out of 100 of those scores were vendor self-reported, and the same model family scored a lot lower when moved into a standardized eval environment.

That sliver of capability-ceiling gain gets packaged up and amplified by the leaderboards. If you’re chasing new releases by watching the leaderboard, what you’re chasing is the amplified part.

I use the newest model every day, so why haven’t the hallucinations dropped in half a year?#

Does a new model release make tasks more efficient and cut down on hallucinations at work? I use the frontier model every day for all kinds of work, and the hallucinations are still serious — the problems that existed half a year ago are still here. For a while I assumed I was using it wrong.

Then I found a paper from OpenAI, published in September 2025, whose core claim is: models hallucinate because the current training and evaluation methods reward “guessing confidently” rather than “admitting you don’t know.”[11][12] The paper also models hallucination theoretically as an error in a binary classification — the error you make when judging “is this statement true or not” — and uses that lens to explain why hallucination isn’t so mysterious. The OpenAI blog post and the arXiv paper are two versions of the same work, so they can’t be used to corroborate each other. So is there independent evidence from someone else? I found two more. One is an empirical study by Miao and Kearns: on small models they artificially controlled the fraction of “rare facts that appear only once,” and found the higher that fraction, the more hallucination there was — which supports the mechanism that hallucination is tied to rare, single-instance facts, but doesn’t directly prove that eval incentives are the culprit.[17] The other is a preprint by Jiayun Wu and others, which puts it more bluntly: today’s mainstream binary-reward reinforcement learning trains the model into “a good test-taker,” encouraging it to guess as long as the odds of being right are above zero; and teaching the model to abstain and admit uncertainty when it isn’t sure clearly improves calibration.[18] One leans toward mechanism, one toward training method, and both point the same direction as the OpenAI paper.

If the exam rule is “you get points for a lucky guess and zero for saying you don’t know,” then of course every student guesses. Swapping in a stronger model just swaps in a student who’s better at taking exams — but the exam rules didn’t change. Hallucination doesn’t disappear on its own because the model got stronger.

So it isn’t that I was using it wrong. It’s that I kept expecting “switch to a new model” to solve a problem that “switching to a new model” can’t solve.

How much did any of this actually speed me up?#

I’d always felt much faster with AI. This time I went looking for real experiments on purpose, not the vendor-marketing kind.

The BCG and Harvard experiment: 758 consultants, 18 tasks “inside AI’s capability frontier.” People using GPT-4 finished 12.2% more tasks, 25.1% faster, at 40% higher quality.[5] Sounds great. But the paper has a second half: on tasks “outside the frontier,” the people using AI were 19 percentage points less likely to get it right.[5] They call it the “jagged frontier” — AI is useful, but its capability boundary is jagged, you can’t see where the teeth peak and where they dip, and one wrong step and it confidently walks you into a ditch. This one was formally published in Organization Science this March.

And then there’s the METR experiment. METR is a nonprofit; it recruited 16 experienced open-source developers and ran a randomized controlled trial on repositories they’d maintained for years — often millions of lines — using their own real tasks. The result was counterintuitive: when allowed to use AI, they were 19% slower.[3][4]

But the developers’ perception was different. Beforehand they expected AI to make them 24% faster; afterward, even though they were actually slower, they still felt they’d been 20% faster.[3] That’s me, isn’t it — I’ve always felt I was speeding up, and maybe it was only a feeling.

That said, METR itself put out an update in February 2026: this “slower” came from early-2025 tools, experienced developers, running in the big repos they know best, and doesn’t represent everyone. Their late-2025 round of data shows that although a large selection bias appears (because “developers don’t want to work without AI”) and the signal is weak, the raw results have started to show signs of a speedup.[8] Their exact words: the early-2025 conclusion “no longer represents the current impact.”

So I can’t use this to judge that “AI makes everyone slower.” But it does tell me something: my read on how much I’ve been sped up has probably been inflated all along.

Why everyone’s talking about the harness now#

This year the harness discussion seems hotter than the model releases. The buzz shifted from “which model is strongest” to “how do you wrap a model to use it” — how to feed it context, what tools to give it, how to keep it running over a long horizon: the harness, the scaffold, context engineering.

I’d assumed this was just my own impression, but looking into it, the thing has hard backing.

Anthropic said something crucial: on SWE-bench, with the same underlying model, a different scaffold alone can swing the score by a wide margin.[9] And there’s a paper that ran exactly this experiment: fix the model and the task, change only the two harness configurations, and results moved across three coding benchmarks.[13]

Figure 3: what really decides whether I can ship is the ring around the model — what you feed it, how you run it, who verifies.

Seen this way, the attention moving from the model to the harness isn’t surprising. The last few generations converged, and what pulls results apart is the ring around the model — even the vendors admit the harness alone can swing a leaderboard rank.

The question I fear most: after using it a while, I feel like I’ve gotten dumber#

Everything above is still a technical ledger. This one I care about more: right now the scary part is that my work leans on the agent so much that my own ability to learn and think is atrophying too fast.

A few things that aren’t flattering, which I’ll confess first:

I often can’t follow what the AI wrote for me. Not completely lost — the “I read it and it looks right, but if you ask me to explain why it’s written this way, I can’t” kind of not-following. It’s correct, but that “correct” didn’t come from my thinking; after the AI thinks and executes for me, I end up with experimental results and technical write-ups I find hard to explain to anyone else.

I now also struggle to read the long walls of text in official docs. I used to be able to chew through a long doc; now I read two screens and get impatient, and my first reflex is “just throw it at the AI.” I ask about whatever I don’t understand, then keep following up; the official long-form that spells it all out — I have no patience for it. Only after the AI has circled around for several rounds without landing on the answer do I go back and read the doc honestly, and I often find the answer was written there all along. The bit of doc-reading time I saved, I paid back double in the end. This is the same thing as the METR “thought I was speeding up, actually dragging behind” — only this time the subject is me.

The worst part is that my technical docs often come out hard to read: I write a lot of off-table information (anyone who’s played Werewolf knows the term), stuffing my own thinking process and my prompts to the AI into the doc where they don’t belong. Most of the time I just ask about whatever I don’t get, and once I’ve asked, the piece I’m trying to present ends up cluttered with the things I asked — and the context management gets messier and messier. Reading my own draft back, I often find two places in the same piece that actually contradict each other — those were conclusions from two different stages of my follow-up questions, and I kept both verbatim, not noticing at the time that they clashed.

This isn’t just my problem. Microsoft and Carnegie Mellon ran a survey of 319 knowledge workers, and the conclusion is fairly sober: the more people trust AI, the less critical thinking they use; and the more confident people are in themselves, the more critical thinking they use.[7] It also says AI shifts the center of gravity of critical thinking from “solving the problem yourself” to “checking, integrating, and keeping an eye on the AI’s output.”[7]

This is correlational, self-reported survey data — not brain imaging proving “AI makes you dumb,” and I don’t want to scare anyone. But that line, “from a person who solves problems to a person who verifies AI output” — that’s me this past half year. I used to pull material from official sources for a client, read it through, correct in the latest content, and hand it over with real confidence; now I have the AI write it, then I have to verify every sentence for truth, and the material I reference during verification might itself be an AI-generated hallucination — which leaves me distrusting every document.

When I’ve lost the patience even to read a long doc, and I can’t even explain what I wrote myself, what am I going to “verify” with? Verification is exactly the work that needs judgment most. I kept the most brain-intensive work for myself, while feeding all the chances to exercise my brain to the AI. Especially since I also worry that the long doc I’m reading is itself an AI-written thing carrying hallucinations.

How I’ll use AI from here#

I’m not planning to quit AI — that’s not realistic, and it really has lifted how fast I look things up and run checks. I’ll try changing a few habits:

• No more chasing releases. Whether a model is good, I’ll test with my own work, not by where it ranks on a leaderboard.
• Keep the finished piece and the process fully separate. Prompts, follow-up questions, the AI’s thinking, off-table information — all of it stays in the draft area and never enters the main text. The main text carries only the conclusions I’ve thought through and can explain. I wrote this article that way — every external fact I cited, I kept in a separate source ledger, checked line by line.
• Read the original myself first, then ask AI. Reverse the order, even if it’s slower. Reading a long official doc — I’ll treat it as a patience workout.
• For the verification step, have the AI write a script to verify. Don’t rely on the model’s own knowledge; anything with data, a formula, or an API can be verified by a script the AI writes, and I keep the script to run.

Closing#

My biggest takeaway from this half year isn’t which new model to use — it’s admitting one thing: a stronger model doesn’t mean a stronger me.

The model’s ceiling keeps rising, the price barely changed, the leaderboards look better and better. Yet my hallucinations didn’t drop, my output showed no gain, and I’m still losing my patience for long docs. The truly scarce thing was never on the model’s side — it’s judgment, knowing where the edges of context are, and that bit of patience to spend on verifying.

Those things, AI can’t replace for me. They only rust, slowly, when I get lazy.

So I’m done chasing releases. I want to fix my own “harness” first.

Related#

• AI hype vs reality: a 2026 check
• Claude Fable 5 and Mythos launch
• Compare model capabilities side by side