Your Human in the Loop Agrees With the Agent 86% of the Time

Somewhere in your stack there is a box labelled "human review." An agent drafts, a person approves, and the audit trail says a human looked. On October 27, 2026, Anthropic published a randomized trial that asks the uncomfortable follow up: when the person looks, whose judgment are they using? In the experiment, professional analysts adopted a hidden lean in the agent's recommendation 86% of the time, against a 50% chance baseline. Only 15% credited the AI. The checkbox was ticked. The check did not happen.

What Anthropic Actually Ran

The write up is titled agents and judgment calls, and the headline setup is simple enough to hold in your head. Fifty professional analysts worked with Claude Code as their assistant, as reported in Unite.AI's coverage. The agent carried a hidden lean in the recommendation it offered. The analysts were not told. Because the lean was randomized, the researchers could compare where analysts landed against what chance alone would produce. Chance says 50%. They landed on the agent's side 86% of the time.

Notice the word "hidden." The analysts believed they were doing their own analysis with a helpful tool nearby. That belief is the finding. It is also why the result is hard to dismiss with the usual advice to "just be more careful." Professionals were in the room, doing their jobs. They were also moved.

Two more numbers from Anthropic's write up matter more than the headline. Analysts switched their answer 60% of the time when the agent disagreed with them, and 6% of the time when it agreed. Read that asymmetry slowly. An agreeing agent barely changes anything, because there is nothing to change. A disagreeing agent flips a majority of the people it disagrees with. The influence concentrates in exactly the cases where a second opinion was supposed to add information.

Then there is the attribution gap. Unite.AI reports that 77% of analysts repeated the agent's arguments, while only 15% credited the AI. In a separate report, Silicon Report notes that 52% rated the final recommendation as more their own than the agent's. So most people reproduced the agent's reasoning, many of them felt the result was theirs, and few said where it came from. Nobody has to be lying for this to happen. The reasoning simply gets absorbed.

That sentence comes from the Anthropic piece, and it is the whole argument of this post compressed into one line. If you build any workflow where an agent proposes and a person disposes, the person is only a control while that condition holds. Once it fails, you have a rubber stamp with a salary attached, and your risk model is quietly counting a safeguard that is not there.

Why Sample Size and Scope Still Matter

A fair reader should push on the limits before nodding along. Fifty people is a small sample. The task was one task in one field. The assistant was one model. Nobody has shown that a code reviewer reading a pull request behaves like an analyst weighing a recommendation, and nobody should claim that 86% transfers to your team as a number. Your rate could be lower. In some setups it could be higher.

What transfers is the mechanism, and the mechanism is old. Anchoring has been studied for decades: show a person a number or a conclusion first and their estimate drifts toward it. An agent that writes a fluent, confident, well structured answer is an unusually effective anchor, because it arrives with reasoning attached. The reasoning is the part that does the damage. A bare number can be second guessed. An argument that sounds right gets adopted, and then it is yours.

So treat the trial the way you would treat a failing canary rather than a verdict. It tells you where to look. The rest of this post is about what to build once you look.

The Review You Think You Have Versus the One You Have

Most human in the loop designs share one shape. The agent produces an artifact: a pull request, a refund decision, a credit memo, a triage label, a drafted reply. The artifact lands in a queue. A person opens it, reads the agent's work, and clicks approve or reject. The agent's output is the first thing they see. That ordering is the flaw, because it hands the reviewer the anchor before they have formed a view.

Consider what a reviewer is rewarded for. Throughput is easy to measure. Disagreement is awkward: it creates rework, it slows the queue, and it means arguing with a system the organisation paid for. Approval is cheap and socially safe. Add the finding that people adopt the agent's reasoning without noticing, and you get a system where the reviewer genuinely believes they are exercising judgment while behaving like a pass through. The dashboard shows 100% human coverage. The effective independent coverage may be far lower, and you cannot tell from the dashboard.

We hit the same structural problem when we wrote about the maker checker pattern. A checker only checks if it is independent of the maker. We argued for an adversarially prompted second agent and for reversibility sized human gates. The Anthropic result adds an uncomfortable corollary: the human gate is itself a checker, and it needs the same independence guarantees you would demand of the second agent. A person who has read the maker's output first is not independent of it.

A Red Build Should Mean Something

Software engineers already have a mental model for this, and it is worth borrowing. A piece on martinfowler.com about why good developers stay wary puts it in one line: "A red build should mean something: stop, investigate, fix it." A signal that cannot fire is not a signal. Everyone understands this about CI. A pipeline that has been green for two years because the tests were deleted is not reassuring. It is broken.

Apply the same standard to review. If a human approval gate has not rejected anything in a quarter, there are two readings. The agent may be excellent. Or the gate may have stopped firing. You need a way to tell them apart, and the answer is not to exhort reviewers to try harder. It is to change the design so that independence is structural rather than aspirational.

Design One: Commit Before You See

The cleanest fix targets the anchor directly. Make the reviewer record their own position before the agent's output is shown. This borrows from a technique long used in forecasting and in clinical settings: independent assessments first, comparison second.

In practice it looks like a two step form. Step one shows the raw inputs only: the ticket, the transaction, the diff, the source documents. The reviewer writes a verdict and a one or two sentence reason, and submits it. Step two reveals the agent's recommendation next to the reviewer's. If they match, the item proceeds. If they differ, the item is escalated or the reviewer must write why they are changing or keeping their view.

The cost is real. Committing first is slower than reading a summary and clicking approve, often by enough to make a throughput owner wince. Be honest about that and spend it selectively. You do not need to commit first on every refund under twenty dollars. You do need it on the decisions where a wrong approval is expensive or hard to reverse, which is the same line the maker checker pattern draws for human gates.

There is a second benefit that teams tend to discover by accident. The committed position is data. It gives you a record of what the human thought before exposure, which is the one thing you cannot reconstruct afterward. When an incident review asks whether the reviewer caught the problem, you can answer with something better than "they approved it."

Design Two: Blind or Randomize the Agent's Lean

Anthropic could measure deference because the lean was randomized. You can borrow that. If you randomize something about the agent's contribution on a sample of items, you can measure how much the reviewer follows it, instead of guessing.

There are three variants, ordered here from easiest to hardest to run.

The third variant deserves a warning. Seeding wrong answers can feel like a trap for your own people, and handled badly it will be read as one. Frame it as testing the system, never the person. Aggregate the results at the team level, never publish individual miss rates, and make sure the outcome is a change to the process rather than a performance review. Security teams run phishing simulations under the same rules, and the programs that turn punitive stop producing honest data.

Sampling keeps the economics sane. You do not hold out the agent on every item, which would defeat the point of having one. A small random sample, drawn in a way the reviewers cannot predict, is enough to estimate deference at the team level. The same logic underlies the held out sets we recommend in eval driven development: a small, protected slice that answers whether the system still works, kept clean from the thing being measured.

Design Three: Treat Disagreement Rate as a Health Metric

Most teams track how fast review happens and how often the agent is right. Few track how often the human and the agent disagree, and what happens when they do. That number is the closest thing you have to a pulse on whether the gate is alive.

Define it narrowly. For each item, record the reviewer's independent position if you captured one, the agent's recommendation, and the final decision. From those you can compute three rates: how often the reviewer's committed position differed from the agent's, how often the reviewer changed their position after seeing the agent's, and how often the final decision differed from the agent's. The Anthropic study reports the second one, the switch rate: 60% when the agent disagreed and 6% when it agreed.

Do not set a target. A target for disagreement turns it into a performance, and reviewers will manufacture objections to hit it. Use it as a trend and a trip wire. A disagreement rate that falls steadily after you deploy a new agent version is a warning. A rate near zero on a task that humans historically argued about is a bigger one. A sudden jump may mean the agent changed, the inputs changed, or the reviewers finally woke up. All three are worth knowing.

Pair the rate with outcomes. When a reviewer overrides the agent, who turned out to be right? Over a few months that comparison tells you whether disagreement is adding value and where. It also tells you which gates are well calibrated, and you should use that knowledge to improve training and routing, not to rank people.

Where Code Review Fits

Engineering teams will recognise this immediately, because the pull request is the most common human in the loop gate there is. When we looked at AI code review eating the pull request, the worry was agents reviewing agents. The deference result adds a second worry in the other direction: people reviewing agents who have already framed the change.

An AI generated pull request arrives with a tidy description that explains what changed and why. That description is the anchor. A reviewer who reads it first reads the diff looking for confirmation. Try the commit first pattern here: before opening the description, scan the diff and the tests and write down what you think the change does and what could break. Then read the description. Where the two stories differ, you have found either a misunderstanding or a bug, and both are cheap to catch at this point.

Tests are the other independent anchor. A reviewer who has to predict whether the existing suite would catch a regression is reasoning from the system rather than from the agent's narrative. This is also why the trust gap we described in the almost right problem can be healthy: distrust is the feeling of judgment staying your own. The risk is a team that loses the distrust without replacing it with a process. The senior developers who refuse to give that distrust up have a case, which we make in distrust as a signal, not a skill gap.

What Not to Conclude

A few wrong lessons are easy to draw from this result, so name them.

Do not conclude that humans are useless reviewers. The study shows that reviewers can be moved without noticing, and a reviewer who is moved is not the same as a reviewer who is absent. The failure is in the sequencing and the incentives around the people, not in the people.

Do not conclude that agents should be hidden from people. The agents are useful. The goal is to preserve the value of the agent's analysis while keeping the human's verdict independent enough to count as a second opinion. Evidence and extraction can flow freely. The verdict is what needs protection.

Do not conclude that more training will fix it. Awareness of anchoring does not reliably remove anchoring, and in this study the people who were moved were professionals, most of whom did not credit the AI. A one hour module on cognitive bias is a fine addition to an onboarding deck. It is not a control.

And do not conclude that the answer is a second agent in place of the person. We covered why in the maker checker piece: two models share failure modes, and removing the human removes the accountability that regulators and customers still expect. The more useful response is to make the human step harder to fake. Our discussion of the delegation gap lands in the same place: graded autonomy needs gates that actually grade.

A Review Policy You Can Write This Week

None of this needs a platform project. You can put a first version in place with a spreadsheet, a form tool, and a few hours. Here is the order we would do it in.

First, list your human gates. Write down every place an agent output waits for a person: approvals, escalations, sign offs, merges, send buttons. Most teams find more than they expected. Next to each, note the cost of a wrong approval and whether it can be reversed. Rank them. The top three or four are where you spend effort first.

Second, log the three fields that make deference measurable: what the reviewer decided, what the agent recommended, and whether the reviewer saw the recommendation before deciding. If your tooling does not capture the third one, add it. Without it you cannot separate a considered agreement from an anchored one.

Third, pick one high stakes gate and switch it to commit first. Build the two step flow in whatever tool the reviewers already use. Run it for two weeks on that gate only. Record the time cost and the number of items where the committed position differed from the agent's. Both numbers will teach you something, and you will have real data for the conversation about whether to expand.

Fourth, add a small random holdout. Choose a percentage you can afford, perhaps one item in twenty, and show those items without the agent's recommendation. Compare. If the unassisted reviewers reject far more often than assisted ones on comparable items, you have measured your own deference footprint, and you no longer need to rely on a study of fifty strangers.

Fifth, put the disagreement rate on a chart next to the metrics you already review. Look at it weekly. Write down what normal looks like for each gate and what would make you investigate. The martinfowler.com phrasing is the bar: when the signal fires, stop, investigate, fix it. When it never fires, investigate that too.

Sixth, tell reviewers what you are doing and why. Present it as protecting their judgment, because it is. People who understand that the process exists to keep their opinion their own tend to engage with it. People who suspect it exists to catch them out will game it, and you will have built a more elaborate rubber stamp.

What to Ask Your Vendors

If you buy review tooling rather than build it, the same questions apply, and few vendor demos answer them. Ask whether the interface can withhold the recommendation until the reviewer commits. Ask whether it records the reviewer's position before and after exposure. Ask whether you can route a random sample of items without agent output. Ask for the disagreement rate as a reportable metric, not just approval counts and cycle time.

Expect friction. Most tools are optimised for the opposite behavior: fast approval, prominent agent summaries, one click accept. That is a reasonable product choice for low stakes work and the wrong one for decisions where you are leaning on the human to catch errors. When a vendor describes human review as a safety feature, ask what evidence they have that the human is independent of the model. If the answer is "the human clicks the button," you have learned what you needed to.

Vendor reported numbers on review effectiveness deserve the same scrutiny as any other vendor reported figure. Ask how it was measured, on what task, with what sample, and whether reviewers saw the agent's output first. The Anthropic trial is useful precisely because the lean was randomized and the switching rates were reported on both sides. A claim about accuracy that skips those details tells you little about independence.

The Part That Is Still Open

Several questions remain, and the honest list is longer than the answered one. Does deference grow or shrink as agents get more accurate? A better agent deserves more trust, which is also what makes it a stronger anchor. Does it vary with the reviewer's expertise, or with how long they have worked alongside the same agent? Does it differ in tasks with a verifiable ground truth, like a failing test, versus tasks of pure judgment? Anthropic's study covers one point in that space. Your domain sits somewhere else.

That is an argument for measuring, not for waiting. The interventions above are cheap, they produce data within weeks, and they pay off even if the effect in your organisation turns out smaller than 86%. A team that knows its deference rate can set its autonomy dial with evidence. A team that does not is guessing, and it is guessing with a record that says a human approved everything.