Price per token is the wrong number. Measure cost per task.

Google announced Gemini 4 Carbon on October 28, 2026 with an introductory price of $2 per million input tokens and $10 per million output tokens, rising to $4 and $20. On a price list, that sits above most of its rivals. A day later the pricing page was the least interesting number in the story. Artificial Analysis ran the model through its Intelligence Index and found that Carbon completed the work for less money than Claude Opus 5.5 at max effort, despite the higher sticker. If you pick models by comparing price lists, you will pick wrong, and you will not find out until the invoice arrives.

What Artificial Analysis Actually Found

Start with the facts, because the headline makes it easy to misremember them. Google's own launch post for Gemini 4 Carbon gives the pricing: $2 and $10 per million tokens as an introductory rate, rising to $4 and $20. The same post says broad availability is not live yet. So the number every buyer will eventually pay is the higher one, and for now the model is something you can read about more easily than you can deploy.

Then there is the independent measurement. In its write up of Gemini 4 Carbon, Artificial Analysis reports a cost per Intelligence Index task of $7.76 for Carbon and $10.04 for Claude Opus 5.5 at its max setting. It also reports that Carbon spends about 123k output tokens per task where Opus 5.5 (max) spends about 173k, which the article frames as 29% fewer. And it places Carbon at roughly 1.2 times the cost of GPT-6 Astra, which comes in at $6.25.

Read that last comparison twice, because it is the one that keeps the story honest. Carbon is not the cheapest model on the chart. It is cheaper than one expensive rival and about 1.2 times the cost of another. A per token price list could not have told you either fact. It would have ranked Carbon above Opus on price, when per task Carbon wins, and it would have said nothing about where GPT-6 Astra lands.

Notice also what the benchmark does and does not claim. It measures cost across the tasks in one index, run by one evaluator, under one harness. That is a useful prior. It is not your workload. Hold both thoughts at once: the direction of the result is a warning about price lists in general, and the size of the result tells you nothing certain about your own bill.

The Arithmetic Behind the Inversion

Spend on a task is price times tokens, summed over every call the task needed. A price list gives you the first factor and hides the second. When two models differ by 29% in output tokens, the higher volume model has to be priced at least 1.41 times lower per output token just to break even on output cost alone, since 1 divided by 0.71 is about 1.41. That is a derived figure, not a reported one, and it ignores input tokens, but it shows how little room a price advantage has once volume diverges.

Output tokens are also the expensive side of the bill. In the launch pricing above, output costs five times as much as input per million tokens, at $10 against $2, and the same ratio holds after the increase to $20 against $4. A model that thinks out loud for longer pays for it at the higher rate. A model that reaches the answer in fewer tokens collects a discount that never appears on any pricing page.

Three effects stack on top of raw output volume, and none of them is visible in a price list.

The first is reasoning effort. Models with a max setting can spend a large share of their output on thinking before they answer. Two labs can both advertise a flagship at similar accuracy and spend very different amounts of thought getting there. The setting is a dial you control, which means the right comparison is between models at the effort level you would actually run, not at the headline configuration in the launch blog.

The second is context growth. In an agent loop, each step carries the history of the previous ones. Input tokens are the cheap side, but they are billed again on every turn, so a loop that takes twice as many steps does not cost twice as much on input. It costs more than that, because the average context is longer too.

The third is failure. A run that does not solve the task still costs money. If one model fails a task and the harness retries it, the successful result carries the price of the failed attempt. A cheap model that solves most tasks on the first try and needs a second attempt for the rest has a different effective price than its list suggests, and a different one again if some tasks never get solved at all.

Agent Loops Multiply the Gap

A single call has a simple cost. An agent run is a sequence of calls: plan, read a file, run a test, read the failure, patch, run again. Efficiency differences that look small in one call compound over a dozen. We covered the underlying machinery in our piece on the agent harness: the loop controller, the tool layer and the context management around the model decide how many turns a task takes, and that number matters as much as the model's price.

Vendors have started to say this out loud. GitHub's October 28 changelog entry on routing and orchestrating models with GitHub Copilot Fusion makes a cost claim in task terms rather than token terms. GitHub reports that Fusion uses about half the average API cost per task and about half the tokens compared with Claude Code running Opus 5.5 Max. This is vendor reported, produced by the company selling the product, and you should treat it as a claim to test rather than a result to cite in a budget meeting. What deserves attention is the framing. A vendor selling an orchestration layer chose to advertise cost per task and token count, because that is where an orchestration layer can show a difference.

Mistral made the same move on the open weights side. Its Mistral Large 4 announcement says the model matches GLM-5.3 at 48% on the AA Coding Agent Index while using 47% fewer tokens per task. Again this is reported by the vendor in its own launch post. The shape of the claim matters more than the number: equal score, fewer tokens. When two models tie on accuracy, token efficiency is the tiebreaker, and a tiebreaker worth nearly half the volume is not small.

In the same week, Artificial Analysis, GitHub and Mistral each reported cost or tokens per task alongside, or instead of, a price list.

We made the budget side of this argument earlier in why token pricing breaks enterprise AI coding budgets. That piece was about forecasting: consumption, not seats, drives the bill. This one is about selection. If consumption drives the bill, then the model with lower consumption per finished task wins even when its rate card loses.

Define the Metric Before You Measure It

The metric is cost per solved task. In words: take every dollar spent on a set of tasks, including the dollars spent on failures and retries, and divide by the number of tasks that passed your check. Two details do the work.

First, the numerator includes failures. A model that solves 8 of 10 tasks has paid for 10 tasks of work and delivered 8. If you compute average cost over successes only, you reward models that give up cheaply. If you compute it over all attempts but divide by attempts rather than solves, you hide the failure rate inside the average. Dividing total spend by solved tasks is the only version that punishes both waste and failure.

Second, "solved" has to mean something your product would accept. An agent that reports success is not a solved task. A test suite that passes, a schema validator that accepts the output, a rubric scored by a calibrated judge: these are solved tasks. If you cannot write the check, you are not ready to price the task, and the first week of work is the check itself. We wrote about that discipline in eval driven development, and everything in this post assumes you have a harness like it.

What to Log on Every Run

A cost per task number is only as good as the record behind it. For every run in your eval harness, capture the following, and capture it at the call level so you can roll it up later instead of guessing what happened.

• Input tokens, output tokens, and any separately billed reasoning or cached tokens, per call. Providers report these in the response usage field; store the raw values rather than a computed dollar figure, so you can reprice later.
• The model identifier, the effort or reasoning setting, and the price table version in force when the run happened. Without the last item you cannot explain why last month's number differs from this month's.
• The number of calls and tool invocations per task. This is the loop count, and it is often the best single predictor of cost.
• Retries, split by cause: a rate limit retry is infrastructure noise, a self correction after a failed test is model behavior. Mixing them corrupts the comparison.
• Wall clock time from task start to final verdict. A cheaper model that takes four times as long may still lose, because engineers wait for it.
• The pass or fail verdict from your check, plus the failure category when it fails. Category matters because a model that fails by timing out and a model that fails by producing wrong output call for different fixes.

Store each run as one row keyed by task, model, setting and attempt number. Keep calls in a child table. If you already trace your agents, you have most of this; the trace is the raw material, and the roll up is a query. Our piece on agent observability with OpenTelemetry covers how to get the spans in place so the cost fields are not a retrofit.

The Roll Up, Step by Step

Once the rows exist, the calculation is short. The steps below give you a number per model per setting, plus the context needed to trust it.

Step one: price every call from stored token counts and the price table for that date. Sum calls into a cost per attempt. Keep input, output and cached costs as separate columns, because output usually dominates and you will want to see it.

Step two: group attempts by task and model. A task is solved if any attempt within your retry policy passes. The cost of that task is the sum of all its attempts, including the failed ones. A task that never passes still contributes its cost to the numerator.

Step three: divide total cost by solved tasks. Report the pass rate next to it, always. A model with a lower cost per solved task and a much lower pass rate may not be what you want, because the unsolved tasks still need a human or a second model, and that cost belongs to the comparison too.

Step four: report a spread, not just a mean. Agent runs are noisy. Report the median and a high percentile, such as the 90th, for cost per solved task and for wall time. A model whose mean cost is low but whose tail is long will blow budgets one stuck run at a time, which is exactly what a mean hides.

Step five: run enough tasks and enough repeats to tell signal from noise. A handful of tasks gives you an anecdote. Run each task several times per model and look at how much the same model disagrees with itself before you believe a difference between two models. If the spread within one model is as wide as the gap between models, you have not measured anything yet.

Choosing Tasks That Represent Your Work

Public indexes are averages over tasks someone else chose. The Artificial Analysis result is a good reason to doubt a price list. It is a poor reason to believe Carbon will be cheaper for you by the same margin. On their index the gap between $7.76 and $10.04 works out to about 23% (my arithmetic, not their claim); on your tasks it could be larger, smaller or reversed.

Build your task set from real work. Pull a few dozen tasks from the last quarter: closed tickets with a known good resolution, support threads with an approved reply, pull requests with tests. Stratify them. Include the boring majority, because that is where volume and therefore spend sits, and include the hard tail, because that is where models diverge and where retries pile up.

Remove anything that leaks the answer. An agent that can read the merged diff will look brilliant and cost almost nothing. Run in a clean checkout at the commit before the fix, with the same tools the production agent would have.

Match the production configuration, not the demo one. Use the same system prompt, the same tool set, the same context limits and the same retry policy. A model tested with generous limits will look cheaper per task than it will behave once your guardrails cut it off mid run.

Routing Changes the Question

Once you can see cost per solved task by task type, a second finding usually appears: no single model wins everywhere. The cheapest model for classification is rarely the cheapest for multi file refactors. That is the economic case for routing, and we walked through the architecture in routing to the smallest model that clears the bar. The metric in this post is what tells the router where the bar is and what clearing it costs.

The vendor claims above point the same way. GitHub's Fusion announcement, which we unpack in the Copilot Fusion margin fight, is about routing and orchestrating several models. If it holds up in your tests, the saving comes from sending each step to a model suited to it, which is a statement about cost per task across a mix of models, and you cannot verify it with a per token comparison of any single model.

Add one rule. Whatever router you adopt, evaluate it as a whole. The unit under test is the system: router plus models plus harness. Measure its cost per solved task against the best single model on the same task set. If the router cannot beat that, you have added a moving part for nothing.

Where the Metric Misleads

Cost per task is better than price per token. It is not complete. Four failure modes are worth naming so you do not trade one false certainty for another.

Price changes invalidate it. Carbon's introductory rate is half its stated future rate. Any cost per task you compute at $2 and $10 will be wrong the day the price steps to $4 and $20. This is why the logs store token counts and a price table version rather than dollars. Recompute under both prices now and you will know how exposed you are before the increase lands.

Availability matters as much as price. The launch post says broad availability is not live yet. A model you cannot get capacity on has an unbounded cost per task. Rate limits, regional access and quota tiers are part of the real price, and a harness that measures only dollars will miss them.

Quality below your check goes unseen. A pass rate counts tasks that cleared your bar. Two models can both pass while one writes code your team would not merge. If your check is only the tests, add a second pass for the things tests do not cover: style, diff size, security findings. A cheaper model that produces larger diffs shifts cost to your reviewers.

Wall time is a cost with a different owner. If a run takes twenty minutes, someone is waiting or a pipeline slot is held. Track it as its own column and decide up front how many dollars a minute is worth in your setting. Do not fold it into a single blended score nobody can interpret.

Treat Vendor Numbers as Hypotheses

Three of the four sources in this post are vendors describing their own products. Google gave the pricing and availability for Carbon. GitHub gave the Fusion savings. Mistral gave the Large 4 token reduction, and the Large 4 license decides whether that saving reaches you at all. Only the Artificial Analysis numbers come from a party that does not sell the model, and even those are one evaluator's harness.

That does not make vendor claims false. It makes them the starting point for a test. A claim like "about half the tokens per task" is falsifiable on your shortlist in a day. Write it down as a hypothesis, run your task set, and record whether it held. Over a few launches you will learn which vendors' efficiency claims survive contact with your workload, and that knowledge is worth more than any one benchmark.

A Rerun Protocol for Your Shortlist

New model launches arrive often. If every launch triggers a debate, you will spend more on the debate than on the models. Turn it into a procedure with a fixed cost.

Keep a shortlist of a few models, covering at least two vendors and one open weights option. Keep a frozen task set and a frozen check. When a new model or a new price shows up, add the model to the shortlist run, execute the full set with repeats, and publish the table internally: cost per solved task at the median and the 90th percentile, pass rate, wall time, loop count. One page.

Set decision thresholds in advance. For instance: switch the default model only if cost per solved task improves by a margin larger than the spread within one model, with a pass rate no worse than the incumbent. Pick the margin before you see the data. That rule protects you from switching on noise and from staying put out of habit.

Rerun on price changes too. A vendor stepping from introductory to list pricing is an event, even if no model changes. Because you stored token counts, repricing the existing runs takes minutes with no new inference spend. Do that first; only rerun the full set if the ranking flips.

Finally, separate the harness from the model in your experiments. Change one at a time. A prompt change that cuts loop count is a model independent saving that applies to every row on the shortlist, and it will often beat anything a model swap gives you. Cheaper tokens are a one time discount. Fewer steps compound.

Cutting Cost Per Task Without Changing Models

The metric also points to levers that never show up on a price list. Once you track loop count and retries, a few interventions become visible.

• Trim the context you resend. Large tool outputs that the model read once and never needs again are billed on every later turn. Summarize or drop them.
• Cap reasoning effort per step type. Planning may deserve the max setting; reading a file does not.
• Fail fast on stuck runs. If a run repeats the same tool call three times, stop it and escalate rather than paying for step forty.
• Use caching where the provider supports it, and verify the cached token counts appear in your logs, since a cache that never hits saves nothing.
• Send easy tasks to a cheaper model and keep the frontier model for the hard tail, then confirm with the same metric that the mix beats either alone.

Each of these shows up as a movement in a column you already log. That is the practical advantage of the metric: it turns cost reduction from a negotiation with a vendor into an engineering loop you can close yourself.

What to Do This Week

You do not need a platform. You need a table and a few days. Here is a sequence that fits in one working week.

Monday: pick a few dozen tasks from last quarter's real work and write the pass check for each. If you cannot write a check for a task, drop it. Tuesday: add usage logging to the harness so every call stores token counts, model, setting, attempt number and verdict. Wednesday: run your current default model on the whole set, several times over, and compute cost per solved task at the median and the 90th percentile.

Thursday: run two challengers. If Gemini 4 Carbon is not available to you yet, note that and include whichever model your team is actually considering. Compute the same table at both the introductory and the list price where a price step is pending. Friday: read the table, apply the threshold you set before you started, and write down the decision and the reasons. Put the task set under version control so the next launch costs you an hour, not a week.

The first run will probably surprise you in one of two ways. Either a model you dismissed on price turns out cheaper per solved task, as Carbon did on the Artificial Analysis index, or the model you trusted turns out to burn most of its budget on a handful of stuck runs. Both outcomes are worth more than another comparison of rate cards.