The Agents Are Here. The Autonomy Isn't.

What actually changed

Cast your mind back to 2023 and AutoGPT, which went viral on a beautifully simple premise: give a language model a goal, let it loop, and watch it run your business. It didn’t work. There was no planning, no recovery from error, no way for the thing to check its own output. The whole demo economy of that year collapsed under one question: how does the agent know it’s right?

What eventually fixed agents was not a smarter model. It was verification. The systems that broke through — coding agents above all — were trained inside environments where correctness could be checked automatically. Tests pass or they don’t. Code compiles or it doesn’t. Wrap a model in a loop of act, observe, correct; give it a small set of well-designed tools; appoint a compiler as referee. Suddenly the thing works — not because it became wiser, but because something outside it could finally tell it when it was wrong.

The best single measure of how far this has got comes from METR, a research institute asking an unusually practical question: how long a task, measured in human working time, can an agent finish with 50 percent reliability? In January 2026 the answer for the best model was 320 minutes. Just over five hours of human work, done in one go, half the time. For models released since 2024, that horizon has been doubling roughly every three months.

But hold both halves of that sentence together. Five hours. Half the time. A coin flip is not a colleague. METR is admirably candid about the fine print, too: the error bars run about twofold in either direction, the horizon is 40 to 100 times shorter for anything involving a screen and a mouse, and the metric says nothing whatever about the 99-percent-plus reliability that production systems actually need.

A click is not proof

I learned the verification lesson the plain way, which is to say by getting it wrong.

I built an agent that runs a Substack publication on its own. It writes short posts, comments on other people’s work, replies to whoever replies to it, and does all of this on a schedule, on my own server, without asking me anything.

Early on it taught me something no paper would have persuaded me of: a click is not proof. The agent would press publish. The button would depress. The log would faithfully record a success. And the post would not be there. Somewhere between the browser and Substack’s servers the action had quietly evaporated, and my agent’s private bookkeeping had become a work of short fiction — which is much worse than an error, because it never tells you.

The fix wasn’t cleverness. It was demoting my own records. After every action the agent now goes back and asks Substack whether the content is really there, and only then writes it down as done. Reality is the source of truth; our accounts of reality are not. The same principle, at a different scale, is why a restart halfway through a run no longer publishes the same post twice. (I’ve written up the whole build — the failures included — as a case study.)

The gap between the benchmark and Monday

Here is the most important pair of numbers in this story. On SWE-bench Verified, the standard exam for AI software engineering, frontier models score around 90 percent. In telemetry from roughly 400,000 real coding sessions, the verified success rate — for professional developers, the best-case users — was 34 percent.

Ninety on the exam. Thirty-four on the job.

The difference isn’t fraud. It’s everything a benchmark strips away: real tasks are underspecified, real codebases carry years of undocumented context, real work has no answer key. The benchmarks are also leakier than anyone would like — one study found models can identify the buggy file from the issue description alone, which is a polite way of saying they have memorised the answers.

The pattern recurs wherever you look closely. Deep-research agents produce reports that look immaculate: measured against expert checklists, their presentation scores 85 to 90 percent, while the completeness of what they actually found scores about 40. The report reads finished; it isn’t. Treat one as a list of leads to check, never a set of findings.

And customer service has the best-measured ceiling of all, because one benchmark does something the others don’t: it runs the same task repeatedly and reports how often the agent gets it right every single time. On retail tasks, agents succeed about 69 percent of the time on one attempt. Ask for eight consecutive successes and the rate collapses to around 25 percent.

That is the whole disappointment of enterprise AI in one line. A system that works two times out of three is a demo. A process is something that works every time.

The productivity numbers are jagged, and that is the finding

Ask whether agents make people more productive and the research answers: it depends who you are and what you’re doing, with an almost cruel symmetry.

Novices on narrow tasks gain a great deal. The landmark study of 5,000 customer-support agents found 14 percent more cases resolved per hour on average — and 34 percent among the least experienced, with roughly nothing for the best. Experts on open-ended work are another story. In a randomised trial, seasoned open-source developers using AI tools were 19 percent slower, while believing they had been 20 percent faster. (The researchers have since flagged problems with how the sample was drawn and treat the figure as a lower bound, so don’t carve it in stone. “Confidently wrong about your own speedup” survives intact.)

Zoom out to a whole economy and it flattens further. Danish payroll records covering about 25,000 workers show precisely nothing — no measurable effect on earnings or hours two years after adoption, with any effect above 2 percent statistically excluded. Meanwhile a Stanford analysis of US payroll data found the one place something has moved: employment among 22-to-25-year-olds in AI-exposed occupations is running about 19 percent below trend. Not layoffs. Hiring that quietly never happened. The bottom rung is thinning while everything above it looks normal.

There is even a name for one mechanism eating the gains: workslop, meaning AI-generated material that looks like finished work and isn’t. Two-fifths of surveyed office workers had received some in the past month, and each instance cost about two hours to sort out. The productivity gain is real for the sender; the bill goes to the recipient.

Why they fail: a mechanism, not a mystery

For a while, “agents fall apart on long tasks” was folklore. Now it has a mechanism. Research published this year showed that long-horizon failures are mostly failures of execution, not reasoning — models fail even when handed both the knowledge and a finished plan. The culprit has a name: self-conditioning. An agent that can see its own earlier mistakes in its context becomes more likely to make new ones. Making the model bigger doesn’t fix it. (There is a cheerful corollary, rarely quoted: because errors compound multiplicatively, small gains in per-step accuracy buy large gains in how far an agent can go.)

Two further limits look more stubborn. Agents cannot reliably correct themselves without an outside signal — a second opinion from the same model makes reasoning worse, which is precisely why every working system leans on a verifier, whether that’s a test suite, a schema or a human. And they lack tacit knowledge: the unwritten context, the negative results nobody published, the “we tried that in 2019” that exists only in someone’s head. In that coding telemetry, experts succeeded in 34 percent of sessions and novices in 15. Same model, different context. The missing input isn’t intelligence.

One limit isn’t technical at all. When Air Canada’s chatbot invented a refund policy, the airline argued in court that the bot was “a separate entity responsible for its own actions.” The tribunal’s reply deserves framing: it makes no difference whether the information comes from a static page or a chatbot. Your agent’s words are your words.

And looming over everything: prompt injection is unsolved. Researchers at Google DeepMind took twelve published defences and broke them with success rates above 90 percent. The practical wisdom fits in one rule, Simon Willison’s “lethal trifecta”: never let a single agent combine private data, untrusted content, and the ability to send anything out. Pick any two.

The one-sentence operating manual

Strip away the noise and the evidence supports a single principle: an agent pays off exactly where verifying its work costs less than doing the work.

Coding agents thrive because the test suite is a free, instant, incorruptible referee. Research agents disappoint because checking every citation costs almost as much as doing the research yourself. Everything else is a corollary. Prefer tasks you can undo in ten minutes; every famous agent catastrophe has been a failure of irreversibility, not of intelligence. Cut work into chunks of two hours of human effort or less, with a checkpoint between them, because that is where today’s reliability actually lives. Ask vendors for the every-time success rate rather than the best-of-one. And price the worst plausible outcome, not the median one.

My own favourite humiliation belongs to the irreversibility rule. I had built a dry-run mode that blocked all model calls, so I ran it with total confidence — and two likes went out on the live account, because the switch stopped the models but not the browser. Nothing was harmed. Nothing was recoverable, either. Now there is a separate check in front of every single action that touches the outside world, and the test suite has to prove it can catch the failure it claims to catch.

Firms that have internalised all this are getting real returns in thoroughly unglamorous places: test-writing, migrations, dependency upgrades, first drafts a human will rewrite anyway, first-line support with a hard escalation path. Firms that haven’t are becoming statistics — by one analysis, only 16 percent of corporate “agent” deployments are agents at all. The rest are ordinary workflows wearing this season’s word.

The view from Poland

Poland makes a sharp case study in what actually blocks adoption, because it isn’t the technology.

On Eurostat’s numbers, 8.4 percent of Polish companies use AI against an EU average of 20 percent — third from the bottom. Read that as national backwardness and you misread it: among large Polish firms, 44 percent already report using AI agents, a thoroughly Western-tier figure. The gap is a small-business gap and a skills gap.

The telling statistic is a different one. Eighty percent of Polish employees have no formal permission to use generative AI at work. They are, of course, using it anyway. The tools arrived before the rules, the workforce moved before management did, and an enormous amount of AI-assisted work is happening in the dark — unmeasured, ungoverned and officially nonexistent.

Which is, in miniature, the story everywhere. The agents learned to work faster than institutions learned to work with them. The models will keep climbing their steep, well-measured curve; the bottleneck has moved to us — to how precisely we can say what we want, how cheaply we can check what we got, and how honestly we count it afterwards. As Thoughtworks’ CTO Rachel Laycock put it this spring: “The inflection point we’re at isn’t about the technology. It’s about technique.”

My own agent taught me the same thing in a less quotable form. The version before the one that works had 71,598 lines of Python, 2,817 tests and 42 schema migrations, and it produced two articles before collapsing under its own machinery. The one that runs every day has 6,526 lines. The autonomy will come, or it won’t. The work of learning to check has already arrived.