A price is a probability

By now the core idea of this series is familiar: a market price is a probability. A contract trading at 73¢ is the crowd saying 73%.¹ But a probability is a slippery kind of claim. If I tell you there's a 73% chance of rain and the sky stays dry, was I wrong? Not really — unlikely things are supposed to happen 27% of the time. You cannot grade a single probabilistic call on a single outcome.

So how do you tell a real forecaster from a lucky one — or a calibrated expert from a confident fraud? You stop staring at any one call and look at all of them. Two tools do the work: calibration and the Brier score. They're how you hold a number accountable.

The only way to grade a probability

A 73% forecast is not a promise that the thing will happen. It's a claim about frequency: of all the times I say 73%, the event should happen about 73 times in 100. That reframing is the whole trick. One call tells you nothing; a track record tells you everything. So gather every market that closed near 73¢ and count how many resolved YES. Then do it for every price level. What you get is a calibration curve.

The calibration curve

Plot what the market said on one axis and what actually happened on the other. A perfect forecaster lands on the 45° line: when it says 30%, the event happens 30% of the time; when it says 90%, 90%. Real markets hug that line remarkably well — far better than pundits, who are reliably overconfident and pay nothing for it.⁶

Miscalibration is just distance from the line. A forecaster whose "90%" calls only happen 70% of the time is overconfident — their curve sags below the diagonal at the high end. The picture tells you not only that someone is off, but exactly how.

One number — the Brier score

A curve is a diagnosis; sometimes you want a single grade. The Brier score is the standard one. Write each outcome as \(o_t = 1\) if the event happened and \(o_t = 0\) if it didn't, and let \(p_t\) be the price as a probability. The score is the average squared miss:²

It runs from 0 (you said 100% and were right, every time) to 1 (you said 100% and were wrong, every time). The number that matters is 0.25 — what you score by saying "50%" to everything, the forecast of someone who knows nothing. Beat 0.25 and you're adding information; liquid markets beat it comfortably.

The squaring is the soul of it. Say 80% on something that happens and you eat \((0.8-1)^2 = 0.04\). Say 80% on something that doesn't, and you eat \((0.8-0)^2 = 0.64\) — sixteen times the penalty. Confidence is cheap only when you're right; the Brier score makes you pay for bluster.

Why the market stays honest

Why are markets so well-calibrated? The same reason everything in this series comes back to: money. If a market reliably said 70% for things that happen half the time, that gap is free money — you sell at 70¢, collect more often than you pay out, and your selling drags the price back toward the truth. Miscalibration is a profit opportunity, and a liquid market competes it away. A pundit can be overconfident for a whole career; a trader is overconfident only until they're broke.

What calibration won't tell you

Calibration is necessary, not sufficient. A forecaster who just parrots the base rate — "8% of startups become unicorns, so 8% for every startup" — is perfectly calibrated and perfectly useless. Good forecasting needs calibration and sharpness: confident, differentiated calls that still land on the line. The Brier score actually splits into exactly those two pieces.³

Markets have one well-known wrinkle, too: the favorite–longshot bias. Longshots tend to be priced a touch too high and heavy favorites a touch too low — buy a basket of 5¢ lottery-ticket contracts and you'll slightly underperform their price.⁴ It's small in deep markets, but it's real — a reminder that "the price is a probability" is an excellent approximation, not a law of physics. And it holds for the sports contracts that make up most of the volume just as much as for elections: a calibrated number is a forecast even when people are trading it for the thrill.

So "a price is a probability" was never a slogan. It's a claim you can test — and the test has a verdict. But there's a second, bolder claim hiding inside the first, and it's the one that sounds like marketing: that a liquid market doesn't just produce honest probabilities, it produces better ones than the credentialed experts.⁵ A crowd of anonymous traders out-predicting analysts with decades of training? It has the shape of a too-good story. The rest of this piece pays that debt — it puts the actual research on the table, and then, just as carefully, draws the lines around it. Because the honest version is more interesting than the slogan: markets do tend to win, the margin is real and repeatable and modest, and it comes apart under specific, knowable conditions. Knowing where the edge lives is worth more than cheering for it.

First, the case against the expert

Start with the credentialed forecaster, because that's the benchmark everyone assumes is strong. It isn't — and we know that from one of the most patient experiments in the history of social science.

Over roughly two decades, the psychologist Philip Tetlock collected something like 82,000 probabilistic forecasts from 284 experts — political scientists, economists, intelligence professionals, the people TV bookers call when something happens. He then waited for the questions to resolve and graded them, exactly the way the calibration machinery above says you must. The finding, published in Expert Political Judgment in 2005, is now famous: the average expert was barely better than chance at assigning probabilities to political and economic outcomes, and on many questions was beaten by simple statistical rules — even by a naïve baseline that just extrapolated the recent past.⁶ Tetlock's most-quoted line is that the median expert was "roughly as accurate as a dart-throwing chimpanzee." The barb is unfair to the best of them, but the distribution is the point.

Two patterns inside the data matter more than the punchline. First, confidence and accuracy were essentially unrelated — the experts who were most certain, and most in demand on television, were not the most accurate; if anything, fame correlated with overconfidence. (This is the sag below the diagonal from Fig 1, made flesh: a career of "90%" calls that land 70% of the time, with no Brier score keeping the bill.) Second, cognitive style mattered more than credentials: Tetlock's "foxes," who held many small models and updated readily, beat his "hedgehogs," who explained everything through one big idea. The lesson isn't that expertise is worthless. It's that a single expert, however decorated, is a high-variance instrument that nobody is grading — and an ungraded forecaster drifts toward confident and wrong.

Then it got weirder

The natural objection: fine, pundits are overconfident, but that's a soft domain. Put real professionals on a hard question with real stakes and the gap closes. It was tested. It didn't.

After 9/11, the U.S. intelligence community funded a tournament to find out what actually improves forecasting. Several university teams competed to predict geopolitical events — coups, elections, currency moves, conflicts. Tetlock and Barbara Mellers ran one of them, the Good Judgment Project, and it won so decisively that the others were dropped.⁷ The detail that should stop you: a few hundred of GJP's best volunteers — a retired pharmacist, a homemaker, a software engineer doing this as a hobby — were measurably more accurate than professional intelligence analysts working the same questions with access to classified information. The civilians, on open-source data, reportedly beat the analysts by a margin variously reported around 30%.⁷

How? Two ingredients, and you'll recognize both because they're exactly what a market supplies for free. The superforecasters were relentless updaters — they moved their estimates in small increments as news arrived instead of planting a flag and defending it. And the project aggregated across many forecasters, then sharpened the result, so no single person's blind spot survived contact with the group. Skill plus aggregation plus updating. Hold that thought.

Now point it at markets

The Good Judgment work is about people forecasting. A market is the same idea with the aggregation and the updating welded into the price — and with money deciding whose view counts. So does the welded-together version actually beat the alternatives? The cleanest long-run evidence comes from elections, because elections resolve unambiguously and we have decades of them.

The Iowa Electronic Markets — a real-money exchange the University of Iowa has run for research since 1988 — is the canonical dataset. Across a large set of U.S. elections, on most days of most campaigns, the IEM's closing prices were closer to the eventual result than the contemporaneous opinion polls.⁸ In the studied presidential races, the market beat the poll the clear majority of the time, and — this is the part pollsters dislike — it was especially better far from election day, when polls are noisiest and least predictive. The market wasn't running a secret model. It was letting people with skin in the game price the polls, discount the noise, and re-quote continuously.

And the prices weren't just directionally right; they were well-calibrated as probabilities — they passed exactly the test we built at the top of this piece. When this kind of market says 70%, the thing happens about 70% of the time; its reliability curve hugs the diagonal and its Brier score sits well left of the no-skill line. So the two claims fuse: liquid markets are honest and they clear the bar that experts mostly fail. Their probabilities mean what they say, and what they say is sharper.

Why the market wins

None of this is luck, and none of it is magic. Four structural properties do the work — the same four that made the superforecasters good, except a market enforces them automatically instead of relying on unusually disciplined humans. They're also, not coincidentally, the four reasons the calibration above holds up: each one is a mechanism that drags the price toward the diagonal.

It aggregates dispersed knowledge. This is the deepest reason, and it's old. In 1945 Friedrich Hayek argued that the knowledge a society needs is never concentrated in one mind — it's scattered as "dispersed bits of incomplete and frequently contradictory knowledge" across thousands of people, and a price is the device that pulls those fragments into one public number.⁹ A prediction market is that machine aimed at the future: every trader stakes their private sliver, and the price collapses all of it into a single estimate that no individual participant could have produced alone. I unpacked this fully in What is a prediction market? — but it's worth restating here because it's the whole reason the crowd can beat the credential. The expert knows one big thing. The market knows a little of everyone's thing.

Skin in the game filters cheap talk. A poll answer is free, and a wrong one costs nothing. A trade is a position, and being wrong costs money. That single difference is a filter: it discounts the confident-but-uninformed and rewards the people who've actually done the work, because they're the ones willing to bet size. A pundit can be wrong on air for a decade and keep the gig. A trader who's wrong goes broke and stops moving the price. The market doesn't ask who has credentials; it asks who's willing to pay to be heard, and then makes them pay if they're wrong — the same self-correcting pressure that keeps the price honest in the first place.

It updates continuously. A poll is a snapshot taken every few days; a committee memo is a snapshot taken once. A market re-prices the instant the world changes, because the first trader to understand a piece of news has a profit waiting if they act before everyone else. Forecasting accuracy is mostly about not being stale, and a market is the only forecaster that's never stale by construction.

No single point of failure. Every individual forecaster has correlated blind spots — a worldview, a bad week, an ego invested in last month's call. Averaging across many independent-ish errors cancels a lot of them; the aggregate is more robust than any term in it. This is just the mathematics of aggregation, and it's why "the crowd" so often beats "the star." The market has no ego to defend and nothing to be embarrassed about. It only has a price.

Where markets don't win

Here's the part the slogan leaves out, and it's the part I'd actually argue about. A market is not an oracle. It's a specific mechanism with specific failure modes, and the same evidence that vindicates it also fences it in. If you only remember the cheerleading, you'll trust a price exactly when you shouldn't.

Thin liquidity breaks everything. Every result above carries the same qualifier — when the market is liquid enough to function. The edge is a property of depth, not of the format. On a market with three traders and ten dollars of volume, the "price" is one person's hunch wearing a probability costume; a single careless order swings it. The mechanism that aggregates dispersed knowledge needs dispersed knowledge actually showing up. A thin market is not a wise crowd. It's a loud individual.

Markets can be confidently, expensively wrong. Calibration is a property of the long run, not a guarantee on any one event — the same caveat that came with the reliability curve above. Speculative bubbles are markets being wrong in public for extended stretches; prediction markets are not immune to the same herding, thin-float distortions, and the occasional whale with an agenda. "Well-calibrated over thousands of contracts" and "right about the one you care about tonight" are different promises, and only the first is supported.

The longshot bias is the standing tax. It's the wrinkle we already met — low-probability events priced a little too high, heavy favorites a little too low. Small in deep markets, and it doesn't sink the broader claim, but it's a permanent reminder that "the price is the true probability" is an excellent approximation, not a law of physics.

It needs a clean question. A market can only price what it can unambiguously resolve. Vague, long-horizon, or definitionally slippery questions ("will this be a good decade for democracy") don't get sharp prices, because there's no clean payout to anchor the trade. The forecasting edge lives where the question is crisp and the resolution is honest — which is its own deep problem, and the reason resolution gets its own essay in this series.

And about 2016

You're thinking it, so let's deal with it. "Didn't the markets blow 2016?" It's the standard rebuttal, and it's mostly a misremembering. On the night, the leading election markets had the eventual winner at something like one chance in six to one in five. That's not a market saying it wouldn't happen — that's a market saying it was unlikely-but-live, roughly the odds of rolling a particular number on a die. Plenty of polls and pundits were far more lopsided and far more certain. When the underdog won, the people who'd said "95% the other way" took the loss; the market, which had said "more like 80/20," was closer to honestly uncertain the whole time.

So 2016 is a genuine caution, just not the one people quote. It's not evidence that markets are dumb. It's the calibration lesson from the first half of this piece, paid out in public: a probability is not a promise — an 83% favorite loses 17% of the time, and that's the system working, not failing. If you grade markets the way you should grade any forecaster — over many calls, by calibration, not by cherry-picking the upset — they hold up. The cherry-pick is the error, and a market is the forecaster least likely to commit it, because it never claimed certainty in the first place.

Markets, polls, models, experts — the honest ranking

Put it together without the cheerleading. Across the better studies, the ranking is fairly stable: a liquid market ≥ a well-built aggregate (poll average, superforecaster pool, ensemble model) > a single good model > a lone expert > a confident pundit ≈ chance. But the gaps between the top three are small and conditional, and the headline result of the last twenty years isn't "markets beat experts." It's subtler and more useful: aggregation beats individuals, and incentives beat opinions. The market wins because it is the purest available form of both at once — not because there's something mystical in a price.

Liquid market

Aggregates + incentivizes + updates

Usually the most accurate when there's real depth and a clean question. Well-calibrated as probabilities. Fails on thin liquidity.

Aggregated humans

Poll average · superforecaster pool

Close behind, sometimes level. The Good Judgment Project shows a well-run, updated, aggregated pool is genuinely hard to beat.

A single good model

One ego-free statistical rule

Beats most humans on stable questions — Tetlock's simple extrapolations embarrassed the experts — but blind to anything outside its inputs.

A lone expert

High variance · ungraded · often loud

Barely beats chance on average; confidence doesn't track accuracy. The benchmark everyone overrates — and the one the others beat.

A benchmark, not an oracle

If markets aren't magic, what's the right way to use them? Not as a crystal ball — as a benchmark. The best forecast of a hard question usually combines the three: a market to aggregate and stay live, models to enforce discipline and catch what the crowd is fading, expert judgment to frame the question and read the regime the data can't see. They have uncorrelated weaknesses, which is exactly why blending them works; this is the lesson of every serious forecasting tournament, where the winning entry is almost always an ensemble, not a hero.

What the market gives you is the number to beat. If your model disagrees with a deep, liquid market, that's not automatically your edge — it's a flag that says show your work. Maybe you know something the crowd doesn't and there's money on the floor; that's the whole game, and the people who do the work get paid for exactly this. Or maybe the market is pricing something you missed, and the disagreement is your error wearing the costume of an insight. The market doesn't tell you which. It just sets an honest, money-backed bar — the calibrated, gradeable number we started with — and dares you to clear it.

So return to where we began. A price is a probability: not a slogan, but a claim you can test, and the market passes that test better than almost any expert, pundit, or poll — because it's the only forecaster that goes broke when it's wrong. The real claim is narrower and far better supported than "the crowd is always right," which it isn't, and the longshots and the thin markets and the occasional public bubble prove it. It's this: a liquid market, on a clean question, is the most honest and least improvable forecast we currently know how to produce — and when it's beatable, beating it is real work, not an opinion. Twenty years of data, from dart-throwing experts to hobbyist superforecasters to the Iowa markets, all point at the same unglamorous mechanism: aggregate widely, weight by skin in the game, update forever. Next time a number flashes on a screen — 73% — you know what it means, you know exactly how you'd check it, and you know it's probably sharper than the expert on the other channel. That is what separates a market from an opinion: the number is accountable. That's not magic. It's better than magic, because you can check it.