Attribution is a ludic fallacy

There is a slide in most media reviews that splits every sale across channels, down to the decimal. Paid search, 34%. Retargeting, 22%. Email, 11%. It adds to a hundred and it looks like measurement. It is closer to a set of casino odds, and the market it describes does not run like a casino.

Nassim Taleb has a name for this mistake1. He calls it the ludic fallacyTaleb's term, from The Black Swan: treating real-world uncertainty as if it were a game with fixed, knowable rules and computable odds, like a casino. Real markets have neither.: taking the neat, bounded odds of a game and applying them to the open world, where the rules are not fixed and the next event may be one the model never contained. A roulette wheel has knowable odds. A market has customers who change their minds, competitors who move, and a hundred causes that never leave a trace. Attribution takes the second and treats it as if it had the fixed odds of the first.

What the model measures, and what it claims#

Last-click attribution records the final touch before a sale. It then presents that record as the cause, which is a different and much larger claim.

Start with what an attribution model actually records. Last-click, and most of its more sophisticated cousins, log the touchpoints on the path to a sale and hand out credit by a rule. That record is real. The trouble is the sentence people say next to it: that these are the things that caused the sale. Recording the last few clicks before someone buys is not the same as explaining why they bought. The model saw a click. It did not see the two years of preference that made the click happen, because preference does not click.

The demand with no click#

The customer who was always going to buy you, and searched your name to do it, looks identical in the data to a customer performance created. Attribution cannot tell them apart.

The customer who was always going to buy and the one performance created look identical in the report.

Here is the case the model handles worst. Someone has wanted your product for months. When they are ready they search your brand name, click the ad at the top, and buy. Last-click hands that sale to paid search. On the slide it becomes evidence that paid search drives revenue. What paid search did was stand in the doorway and collect a customer who was walking in anyway. The demand was built somewhere the model cannot see, so it does the only thing it can: hand the credit to the nearest click.

Taleb’s word for the rest is silent evidenceTaleb's term for the failures, near-misses and non-events that never make it into the data, so we reason from the survivors and the visible cases only.. The demand your brand created that never resolved into a trackable click leaves no row in the table. Read the table and you would conclude that demand does not exist, for the same reason a survey of lottery winners would conclude that buying tickets makes you rich.

Recording the last few clicks before someone buys is not the same as explaining why they bought.

Why the false precision wins anyway#

In a budget meeting, a number to the decimal beats an honest “we cannot attribute this exactly”.

None of this would matter if the model lost the argument. It wins it. In a budget meeting, the person pointing at the dashboard, paid search drove 34% of revenue, beats the person who says the brand work probably built a chunk of that demand but cannot pin it to a click. Every time. A decimal reads as rigour. An honest “we cannot measure this precisely” reads as an excuse. So the room backs the number it can show, and cuts the brand work first.

Where this argument is weakest#

Incrementality testing gives up the decimal and asks a cruder question: run the media, hold a group out, measure the gap. It admits uncertainty, which is why it is closer to true.

There is a more honest tool, and it is less impressive on a slide. Incrementality testingA test that measures how many conversions your media actually caused: run it in some regions and hold it out in others, or use ghost ads, and measure the gap against the customers who would have converted anyway. gives up the per-channel decimal and asks a blunter question. Run the media in some places, hold it out in others, and measure the difference. The answer comes back as a range, not a point, and the range is often uncomfortable. That discomfort is the tool working. It admits the uncertainty that attribution hides, and that is why it lands closer to the truth. I made this case in full in the first essay. The accounts I have seen rarely ran the test.

The question to ask#

Attribution can only credit what leaves a click. Demand built by brand work leaves none, so the spend drifts to whatever the report can count. One question exposes it: what does the model do with a sale that had no click on the way in?

Attribution is one of the main reasons budgets drift. A sale with a click behind it gets credited; a sale that arrived because someone already knew the name gets filed under direct and forgotten. Feed a budget decision a tool that can only see half the work, and the money follows the visible half. Nobody in the room chose to cut brand spend. They let the number decide, and felt rigorous doing it.

Attribution to the decimal has one weak point, and it is worth finding early: the sale that had no click on the way in. If the answer is that it files it under “direct” or “organic” and moves on, the model is pricing an open market like a casino, and the confidence it sells is paid for by the brand work it cannot see.