Before an election, pollsters ask people how they will vote and forecast the result. In a country of 30 million voters, how many random people do you need to ask for the forecast to land within ±3 points of the true result 95% of the time? Lock in your estimate, then check it in a game, on a simulation of typical mistakes and with the explanations.
Before an election, pollsters ask people how they will vote and forecast the result. For the forecast to land within ±3 points 95% of the time, 1,068 random people are enough, and it hardly matters how many voters the country has: 50 thousand, 30 million or 244 million, as in the US. What matters is that the sample is truly random. Millions of answers from "convenient" people miss by more than a thousand random ones.
Asking everyone is impossible, so pollsters ask a sample and judge the whole country by it. The question is how big that sample has to be.
The country has 30 million voters. Two candidates run, A and B. We do not know what share candidate A will get. That is what we want to forecast.
We pick people at random from the list of all voters: everyone is equally likely to be asked. Everyone asked says honestly how they will vote.
A's share in the poll must be within ±3 percentage points of the true share, and this must hold in 95 polls out of 100. That is the standard most published polls work to.
How many people do you need to ask?
"At least one percent of the population, so hundreds of thousands"
And also: "the US needs more than a small country". In fact 1,068 people are enough for a country of 30 million and for the US alike. A poll's precision depends on how many people are asked, not on what share of the population they are.
A country of 30 million voters, an error of ±3 points, 95 polls out of 100. Lock in your estimate before reading on.
A breakdown of your estimate will appear here.
The computer has made up an election result. Ask random voters in batches and watch the margin of error shrink. Then switch the country: a village, a town, a country of 30 million or the US. For the big ones the margin is the same at the same sample size. When you are ready, reveal the true result.
Now we know the country completely. Voters fall into three age groups: the young, 18–34 (28%), vote 64% for A, people aged 35–59 (44%) 52%, and those 60+ (28%) 40%. Altogether A gets exactly 52.0%. Pick a polling method. The computer runs hundreds of polls of different sizes, from 100 to a million people, and shows where they converge.
The bar shows the group's share among those asked, the tick its share among all voters.
The precision of a random sample depends only on its size. A biased sample also carries a bias, and it does not shrink however many people you ask.
To check whether soup has enough salt, you taste one spoonful. The spoon is the same for a five-litre pot and a twenty-litre cauldron. Only one thing matters: the soup must be stirred well.
A poll works the same way. Each person asked is a spoonful of well-stirred soup: the chance of hitting an A voter equals A's share in the country, whether it has 50 thousand voters or 244 million. So precision depends on how many people are asked, not on the size of the country.
A's share in the poll scatters around the true share with a spread of √(p(1 − p)/n). The spread falls like 1/√n: to halve the error you must ask four times as many people.
| Asked | Margin, 95% |
|---|---|
| 100 | ± 9,8 |
| 400 | ± 4,9 |
| 1,068 | ± 3,0 |
| 4,000 | ± 1,5 |
| 10,000 | ± 1,0 |
Margin in percentage points for a share near 50%. Beyond a few thousand, each extra point of precision gets very expensive, which is why most polls stop at 1,000–2,000 people.
The country's size enters the formula only through the correction √((N − n)/(N − 1)). It matters when we ask a large share of the voters. In a village of 1,000, 517 are enough for ±3 points: once we have asked half the village, we know a lot about the other half.
| Voters, N | Needed for ±3% |
|---|---|
| 1,000 | 517 |
| 10,000 | 965 |
| 50,000 | 1,045 |
| 1 million | 1,066 |
| 30 million | 1,068 |
| 8 billion | 1,068 |
A poll's error has two parts: random noise and bias. The noise does fall like 1/√n. Bias appears when some people are more likely to get into the sample than others and their views differ. It does not depend on the sample size.
A landline poll in our country gives 45.4% instead of 52.0%. With 1,068 people the noise is ±3 points, so there is still some chance of a lucky hit. With a million the noise is ±0.1 points, and the forecast confidently shows the wrong number, to a tenth of a point.
A pollster calls a random person, who does not pick up. The easy thing is to call the next one. After 1,068 interviews the report looks fine: a full sample, no refusals. But really it is a sample of people who pick up the phone, and those are more often older. Our forecast drops to 48.9%, and the wrong candidate comes out ahead.
Substitution does not remove the bias, it hides it. The share of people who did not answer, and who they were, was the only sign of trouble. After substitution it is gone, and there is nothing left to correct the bias with.
The ±3 applies to one candidate's share. The lead of A over B comes from the same interviews: when A's share is too high by chance, B's is too low by the same amount. So the lead's margin is almost twice as large: ±6 points with 1,068 people.
A headline saying "A leads B by 2 points" from a poll of a thousand really means "we do not know who is ahead".
The spread of the share in a poll of n people, and the margin at 95% confidence.
Solve for n. The worst case is p = 0.5, a margin of 3 points.
The correction for the population size N. It matters only when the sample is a noticeable share of N.
Where a biased poll converges: w is a group's share, s the chance its member gets into the sample, a the share of A in the group. The error is bias plus noise.
Meng's identity (2018): ρ is the correlation between getting into the sample and the answer, f = n/N. Hence the effective sample size.
The second row gives the sample size for a share near 50%, the worst case. The third shows that N only slightly reduces the sample needed. The fourth computes where a biased poll converges; b in the error formula is the bias. The last, Xiao-Li Meng's identity, says the same in the language of correlation: even a small ρ, multiplied by a huge √(N/n), gives a big error.
How many people you need for any precision and any country, why the quality of a sample matters more than its size, and how real pollsters fight bias.
A thousand people are enough for the whole country. But to know the result in each of 50 states with the same precision, each state needs its own thousand, about 50 thousand in all. That is why exit polls in big countries ask tens of thousands: what costs money is not the country but the detail.
The design effect accounts for the fact that polls are rarely a simple random sample. An exit poll first picks polling stations, then people at them. Neighbours vote alike, so 20 interviews at one station carry less information than 20 from all over the country. A design effect of 2 means the sample must be doubled.
A share near 50% gives the largest spread, p(1 − p) = 0.25. If a candidate polls at 10%, 385 people give the same margin. So planning "for 50%" is a safe margin.
In 2018 the statistician Xiao-Li Meng showed that any poll's error splits into three factors: the correlation ρ between whether a person got into the sample and their answer; the spread of views in the country; and √((N − n)/n). For a random sample ρ is zero on average. For a biased one even ρ = 0.005, an almost invisible link, is multiplied by a huge √(N/n).
Hence his big data paradox: once the sample is not random, the size of the country matters again, and the bigger the country, the worse. Millions of answers with a small correlation can be worth a few hundred random people.
Real pollsters know a perfectly random sample does not exist: some people do not pick up, some refuse. So they weight the answers. If the sample has half as many young people as the country, each young person's answer counts twice. The shares of age groups, education levels and regions are known from the census.
Weighting by age fully removes the landline, street and substitution biases in our country, because there the chance of being sampled depends only on age. But it does not help when the supporters of one candidate are the ones who refuse to answer: they can be any age. And the fix has a price: if the sample holds only a few dozen young people, their answers get a big weight and the forecast scatters more.
When weighting cannot save them, everyone misses together. In Britain's 2015 election the Conservatives beat Labour by 6.6 points, while the final polls showed almost a tie, YouGov among them with 10,307 interviews. The British Polling Council's inquiry named unrepresentative samples, which weighting did not fix, as the main cause.
In the US the exit poll for ABC, CBS, CNN and NBC is run by Edison Research. In 2024 it chose 279 polling places as a probability sample. Interviewers at the exits approached every nth voter, about 75 interviews per place. Early and mail voters were reached by phone, text and email.
Sampling by polling place gives a larger error than a simple random sample: by the exit poll's own table it is ±15 points for 100 voters instead of ±10. This is the design effect from the calculator at work.
In 2004 the exit poll overstated John Kerry: across 1,460 precincts the margin shifted 6.5 points his way on average. The Edison/Mitofsky report put it down to Kerry voters being more willing to take part. Had 56% of Kerry voters and 50% of Bush voters agreed, that would explain the whole error. It is our "shy voters" trap, the other way round.
In 1936 the Literary Digest mailed 10 million ballots and got about 2.4 million back. Its forecast: Alf Landon would win. Franklin Roosevelt took 60.8% and 523 of 531 electoral votes. The magazine drew its addresses from telephone books, club rosters, directories and similar lists.
George Gallup did not just name the right winner. By asking 3,000 people from the same kinds of lists, he forecast in advance that the Digest would give Roosevelt 44%. The magazine showed 43%. That is the account in Freedman, Pisani and Purves's textbook "Statistics".
Gallup's final poll had Thomas Dewey on 49.5% and Harry Truman on 44.5%. It came out the other way: Truman 49.5%, Dewey 45.1%. The early edition of the Chicago Daily Tribune ran a headline declaring Dewey the winner. Polling had stopped well before election day, and interviewers chose whom to ask within their quotas.
Xiao-Li Meng worked it out for the 2016 US election: answers from 1% of the voters, about 2.3 million people, with a correlation between taking part and the vote of just −0.005 have the same error as a truly random sample of 400. The sample size shrinks by 99.98%.
According to the Pew Research Center, 36% of people answered a typical telephone survey in 1997, 9% in 2012 and only 6% in 2018. Even so, Pew found weighted phone surveys still give accurate results on most questions.
In the second round of Ukraine's presidential election, the National Exit Poll (the Democratic Initiatives Foundation, KIIS and the Razumkov Centre) asked 14,561 people at 300 polling stations using secret ballots. It gave Volodymyr Zelensky 73.0%, against the official 73.22%. Almost 30 million voters were on the lists.
Share your own explanation, an objection or a variant of the puzzle. You need a GitHub account to comment.
I keep adding new puzzles, new ways to explain them and write-ups of the discussions they start. Leave your email and I'll write when something new is out. No spam, one-click unsubscribe.
The sign-up form will appear here soon.