Only 40% of the Stats AI Assistants Cite Trace Back to a Source

On 2026-10-28 Trakkr published Zombie Stats, a trace of 1,285 statistics that five AI assistants quoted with a source attached. Only 40% led to a study that said the same thing. By the next day it was on the Hacker News front page. Suspicion is cheap. What the study adds is a number, and the number lands on a problem every team shipping retrieval augmented generation already owns: a link next to an answer proves the model retrieved something. It does not prove the thing it retrieved says what the answer claims.

What Trakkr Actually Measured

Trakkr asked five AI assistants questions that invite a statistic, then traced 1,285 statistics from the answers and checked whether the cited page contained the figure and whether the trail led to a study saying the same thing.

Four outcomes came out. In 40% of cases (95% range 34% to 46%) the trail ended at a study that matched the figure and the claim. In 8% the number had changed somewhere between the study and the answer. In 40% the trail could not be checked. In 11% the link led nowhere.

Notice what is not in that list: a clean category called "fabricated." The headline is not that assistants invent numbers from nothing. Some do, and the 11% that led nowhere is the closest thing to it, but the dominant failure is quieter. A number exists, a page exists, the two sit next to each other, and the relationship between them is weaker than the citation format implies.

That 82% is the number I would put on a slide for an engineering team. If you only checked that the figure appears on the cited page, you would pass four answers in five. If you checked that the page traces to a study saying the same thing, you would pass two in five. Same answers, same links, a different verdict, depending entirely on what your evaluation asks. The gap is not noise. It is the distance between "the page repeats the number" and "the number is true and attributed correctly."

Read the Limits Before You Quote It

Trakkr is explicit about what the study cannot tell you, and you should carry those limits into any conversation where you use the number. It used the assistants' APIs, not their consumer apps, and the consumer products add their own search layers, prompts and ranking. It ran on a single day, so it is a snapshot of a moving system. And it made one ask per question, so it cannot separate a stable behavior from a lucky or unlucky sample. The same sourcing failure shows up when people let a model draft the prose, which is why readers distrust it, as we found in readers can smell LLM prose.

Two consequences follow. First, do not write "60% of AI citations are wrong." The study does not say that. The 40% that were uncheckable are, by definition, not shown to be wrong, and some portion of them are probably fine. Second, do not treat the 40% as a quality score for any one product. The study compares a pooled set of five, and your own pipeline will land somewhere else, better or worse, for reasons you can only find by measuring it.

Citation Presence Is Not Citation Correctness

Most RAG teams track retrieval quality, and they track it well. Recall at k, hit rate, mean reciprocal rank, the share of questions where the gold document appears in the top results. These metrics answer one question: did the right material reach the model? They are necessary. We argued in why naive RAG is dead that retrieval is where most pipelines are weakest, and nothing in this post changes that.

What retrieval metrics cannot see is what happens after the documents arrive. The model reads a passage that says "roughly a third of respondents" and writes "35%." It reads a vendor report and states the finding as an industry fact. It reads two passages that disagree and picks the one that fits the sentence it was already writing. In every one of those cases the retrieval hit rate is perfect, the citation is present and resolves, and the answer is wrong or unsupported.

I think of it as two independent scores that teams collapse into one. Retrieval asks whether the evidence was available. Faithfulness asks whether the answer stayed inside the evidence. A pipeline can score high on the first and low on the second, and the dashboard that only shows retrieval will report good news right up to the day a customer pastes your tool's invented percentage into a board deck.

A citation is a claim about provenance. Treat it like any other claim your system makes, and test it.

Four Ways a Citation Fails

Trakkr's four outcome buckets are a good starting taxonomy, but for engineering purposes I find it more useful to split by where in the pipeline the failure is introduced, because that decides where you fix it.

Failures one through three are mechanical. You can detect them with a fetch, a parser and a comparison, no judgment required. Failure four needs more machinery, and it is the one most worth building for, because it is the one a human reviewer will miss. A reviewer who clicks the link, sees the number, and moves on has done exactly what the 82% describes.

Why Secondary Sources Poison the Well

Most statistics that circulate widely were produced once, by one study, and then copied. Each copy is a company blog or a listicle with a link to the previous copy. After a few generations the figure is still there, the link still resolves, and the study at the end of the chain may say something slightly different, or may be gone. Trakkr's name for the result is a zombie stat. My own gloss: a number that keeps walking after the evidence behind it stopped moving.

Retrieval systems are structurally biased toward these pages. A company blog is optimized to be found: a clear heading, the number in the first paragraph, a clean sentence a model can lift. The original study is a PDF, or a methods section, or a table on page nine. A retriever ranking by semantic similarity to "what percentage of developers use X" will prefer the blog every time, and a model asked to be concise will prefer the blog's sentence to the study's hedge. If you want the longer argument about what happens to the web when content is produced for retrieval, our piece on context engineering covers the selection side of it.

None of this means secondary sources are always wrong. A good trade publication that read the study and summarized it faithfully is useful. The problem is that your system cannot tell the difference between that and a blog that read another blog. So it should do what a careful analyst does: mark the source type, and make the answer's confidence depend on it.

Five Assistants, Five Answers

One more result from the study deserves its own section. Across 71 questions, there were 28 on which the five assistants returned four or five different figures. Same question, same day, same kind of system, and four or five different answers.

There are two readings and they are not exclusive. One: the underlying literature genuinely varies, with different surveys, populations and years producing different numbers, and the assistants each picked a defensible one. Two: each assistant grabbed whichever page its retrieval surfaced first and reported its number with the same confidence. The study's design cannot fully separate these. Either way, a user who asked once and got one confident figure had no way to know they were sampling from a spread.

For a builder this is a gift, because disagreement is measurable inside your own system. Ask the same question several ways, or with several retrieval configurations, and compare the figures. If the answers scatter, the system should not state one number with confidence. It should say the sources differ and show the range with each source. That is a product decision as much as an engineering one, and it costs you the satisfying one line answer. It also keeps you out of the next zombie stat study.

Hallucination Rates Are Not Provenance Rates

It is tempting to read the Trakkr result as a model quality problem that the next release will fix. Model quality matters, but it is the wrong lever for this particular failure, and comparing across models shows why. In its writeup of Gemini 4 Carbon, Artificial Analysis reports AA-Omniscience hallucination rates of 11% for Carbon, 50% for Claude Opus 5.5 (max) and 4% for Gemini 4 Argon. That is a spread of more than twelve times between the best and worst figure on the list.

I am not citing this to rank models. I am citing it because it measures something different from what Trakkr measured, and the difference is the point. A hallucination rate on a knowledge benchmark tells you how often a model asserts something false when answering from its own parameters. Provenance asks a different question: when the model has a source in front of it and points at that source, does the source back it up? A model with a low hallucination rate can still attach a link to a claim the link does not support, and a model with a higher one can be perfectly faithful when you constrain it to quote.

The practical consequence: do not buy your way out with a model upgrade and stop measuring. Choosing among models is a real decision and the benchmark spread above is large enough that it matters. But the same logic we laid out in our post on contaminated coding benchmarks applies: a public number predicts your result only as far as your task resembles the benchmark. Your task is "answer from these documents and cite them correctly." Measure that.

What a Provenance Check Looks Like

A provenance check is a function that takes an answer, its claims, and its citations, and returns a verdict for each claim. It does not need to be sophisticated to be useful. Three layers cover the four failures above, ordered by cost.

Layer one is mechanical. Fetch every cited URL. Record the status, the final URL after redirects, and the extracted main text. Then extract every number from the answer sentence and check whether each appears in the fetched text, allowing for formatting differences like "35 percent" versus "35%". This layer is deterministic, cheap, and catches failures one and two. It runs in milliseconds per citation and needs no model.

Layer two is semantic. For each claim that survived layer one, find the passage on the page that contains the figure, and ask a verifier whether that passage supports the claim as stated, including the population, the year and the direction of the effect. This catches failure three, where the number is right and the claim around it is not, such as a figure about respondents reported as a figure about companies. The verifier can be a model, and the usual rules from our work on maker checker loops apply: use an independent checker with an adversarial prompt, not the same call that wrote the answer.

Layer three is about origin. Classify the cited page: is it the study itself, a news outlet reporting the study, or a company blog? If it is not the study, look for the study it links to, and run layers one and two on that page too. This is the expensive layer and you do not need to run it on every answer. Run it on every statistic in your offline eval set and on a sample in production, and run it always for claims you will display as bold, headline numbers.

The "Did the Source Say This" Eval

Here is the eval I would build first, because it is small, runs on data you already have, and answers the question Trakkr's study raised. It scores each cited claim on a short ladder, and the ladder matters more than any single threshold.

1. Resolves. The cited URL returns content. A failure here is the Trakkr "led nowhere" bucket.
2. Contains the figure. Every number in the claim appears on the page, after normalizing units, percent signs and thousands separators.
3. Supports the claim. A verifier, shown only the claim and the surrounding passage, judges the claim supported, contradicted or unsupported. Population, time period and direction must all match.
4. Primary or secondary. The page is classified as the original study, a reporting outlet, or a company or aggregator blog. Secondary pages are flagged, not failed.
5. Traces to origin. For secondary pages, the link chain is followed up to a fixed depth and rungs two and three are repeated on the page at the end. If the chain ends at no study, the claim is marked uncheckable.

Rungs one through three map to your existing pass or fail tooling. Rungs four and five produce the number that mirrors Trakkr's: the share of claims that trace to a matching study. Report that as its own line. Do not merge it into a single "citation accuracy" figure, because a team that sees one blended number will tune it by making answers cite fewer statistics, which is a way of passing the eval by saying less.

Building the Dataset Without a Big Labeling Budget

You do not need thousands of examples. Start with the answers your system has already produced. Pull a few hundred production answers that contain a statistic and a citation, weighted toward the questions users ask most often and the questions where the cited domain is a company blog. Run the three layers over them. Then have a person read a sample of the verdicts, not the answers, and mark the ones they disagree with. That tells you how far to trust each layer and where the verifier needs a better prompt.

Keep a small human labeled core, even if it is only fifty claims, and never let your verifier tune against it. The same discipline applies to any LLM based judge, and it is the same reason eval driven development insists on calibrating the judge against people: a verifier you have not checked is one more component making confident claims about sources.

Add adversarial cases on purpose. Take a real passage and edit the figure by a small amount. Swap "2025" for "2024". Change "of respondents" to "of companies". Replace the cited URL with a different but related page. If your layers do not flag these seeded errors, they will not flag the real ones. Seeded errors are also the cheapest regression suite you will ever write, since you know the right answer by construction.

Flag Secondary Sources, Do Not Ban Them

The instinct after reading the 82% versus 40% gap is to blocklist company blogs. Resist it. Blogs often contain the best synthesis and the only readable summary of a dense study. What you want is a label and a policy, both visible.

The label is a source type stored with every retrieved chunk at indexing time: primary research, reporting outlet, vendor content, aggregator, unknown. Classifying domains by hand for your top few hundred sources covers most of the traffic, and a model can handle the long tail with a human spot check. The policy is how the label changes the answer. A statistic whose best support is a vendor page should be phrased as a vendor claim: "according to a vendor report" rather than "research shows." That is the same convention we apply in this post, and the reason the Trakkr study and the Artificial Analysis writeup are each named where their numbers appear.

The same applies to your own content. If your company publishes numbers, you are a node in someone's retrieval index. Link each statistic to its origin, state the sample and the date next to the figure, and label vendor reported numbers as such. You are protecting your readers, and you are also protecting your reputation from being the page at the end of someone else's broken chain.

Score Faithfulness Apart From Retrieval

Put the two metrics side by side in the same report and give each its own owner. Retrieval gets recall at k and hit rate. The generation stage gets claim level support rate, the share of cited claims judged supported, and a drift rate for figures that changed. The provenance layer gets a trace rate, the share of statistics that reach a primary source, plus a secondary source share so you can watch it creep.

Splitting them makes diagnosis possible. If retrieval is high and support is low, the model is wandering away from evidence it was given, and the fix is in the prompt, the decoding or the model. If retrieval is low and support is low, you have a search problem and no amount of prompting will help. If both are high and trace rate is low, your index is full of pages that repeat each other, and the fix is in what you ingest. A single blended score makes all three look the same.

Slice every metric. By question type, since numeric questions fail differently from definitional ones. By source type, since vendor content will behave differently from primary research. By domain, since a handful of sites tend to account for most citations. The slice where your support rate is lowest is usually where your best fix is, and it is almost never the slice your headline average suggests.

Design Changes That Beat Better Checking

Detection is the floor. Some changes in how you build the answer remove whole failure classes before they reach the checker.

• Never let the model write a URL. Give each retrieved chunk an identifier, let the model cite identifiers, and resolve them to URLs in code. This removes most of the "led nowhere" bucket by construction.
• Require a verbatim quote for every statistic. The model returns the exact sentence it relied on, and your code checks that the sentence is a substring of the retrieved text. A number without a supporting quote is not displayed as a fact.
• Prefer extraction to paraphrase for numbers. Ask the model to copy the figure and its qualifiers (who, when, how many people) from the passage into structured fields, then compose the sentence from the fields.
• Retrieve for the origin. When a retrieved page cites a study, fetch the study too, and rank it above the page that mentions it.
• Surface disagreement. When two retrieved sources give different figures, show both with their sources. The model should not adjudicate silently.
• Abstain deliberately. "I found a figure on a vendor page but could not trace it to a study" is a legitimate answer, and better than a confident number.

Each of these costs something. Quotes lengthen answers. Abstaining disappoints the user who wanted a clean number. Origin retrieval doubles the fetch cost on those queries. Pay for them selectively: where a wrong number does real damage, such as anything entering a financial, medical, legal or procurement decision, and relax them for casual lookups.

Where This Approach Breaks

A provenance check is a filter, not a truth machine, and it has holes you should know about before you rely on it.

Paywalled and dynamic pages. Trakkr's 40% uncheckable bucket shows that some pages cannot be verified automatically. If your fetcher cannot see the page, you cannot verify the claim, and the honest verdict is "unverified," not "supported." Track that rate separately. When it is high for a domain, either arrange access or stop citing that domain for numbers.

The study may be wrong. Matching a study is not the same as being true. A faithful citation of a weak survey is a provenance success and an epistemic problem. That is a judgment about source quality, and no fetch script resolves it. Keep the human in the loop for sources that anchor high stakes claims.

The verifier can fail. A model judging "does this passage support that claim" is itself fallible, tends to be lenient on near matches, and can be fooled by a passage that mentions the right entities in the wrong relationship. This is why you calibrate it against labeled examples and why the seeded errors matter.

Your own measurement shares Trakkr's limits. One run on one day with one ask per question cannot tell you how stable your system is. Run each eval question several times, and rerun the suite on a schedule, since sources change and links rot. A trace that passed in October can fail in January for reasons that have nothing to do with your code.

What to Do This Week

You can get real signal in about five working days with one engineer. This is the order I would do it in.

1. Day one: sample. Export 200 production answers that contain a statistic and a link. Stratify so that vendor domains and your most common question types are represented.
2. Day two: layer one. Write the fetch and number match. Run it. Record the share that fails to resolve and the share where the figure is absent from the page. Those two numbers are your first real findings, and they cost almost nothing.
3. Day three: layer two. Add the verifier for claim support. Hand label forty examples first so you know how often it agrees with a person.
4. Day four: source types. Classify your top fifty cited domains by hand as primary, reporting, vendor or aggregator. Compute the share of your citations that land on vendor or aggregator pages.
5. Day five: report. Publish one page with retrieval hit rate and the provenance ladder side by side. Pick the one failure class that is biggest and open a ticket for the design change that removes it, most likely identifier based citations or required quotes.

Then put the check in CI so the number cannot drift without anyone noticing, and sample live traffic weekly. A provenance rate you measured once is an anecdote. One you track is a property of your system.

Trakkr measured five assistants once and found that about three in five of what looked like sourced statistics could not be shown to trace to a matching study: 8% changed, 40% could not be checked, 11% led nowhere. That is an uncomfortable number to find in someone else's product and a much more uncomfortable one to find in your own. Better to find it on a Tuesday, in a script you wrote, than in a customer's slide deck.