Lesson 6 of 9 · about 18 minutes
Uncertainty: Samples, Margins of Error and Significance
You'll be able to
- Explain in plain words why a sample of 600 can describe a county
- Calculate a poll's margin of error and the margin on a lead
- Say what a confidence interval means without overstating it
- Explain what 'statistically significant' means and does not mean
- List the errors a margin of error does not cover
You can taste a pot of soup with one spoon. You don't need to drink the pot, as long as you stirred it first. That is sampling in one sentence, and the stirring is the part everyone skips.
Why a sample works
If you pick people at random, so every person in the county has a known chance of being picked, then a few hundred answers will usually land close to what you would get by asking everyone. "Usually" and "close" can be measured. That measurement is the margin of error.
The surprising part is that the size of the population barely matters once it is much bigger than the sample. A random sample of 600 describes a county of 40,000 about as precisely as a state of 2 million. The soup spoon doesn't care how big the pot is.
The catch is "random." A sample of people who chose to answer a newspaper's web poll is not a random sample. It is a sample of people who read that site and felt strongly. No formula fixes that.
The margin of error, calculated
For a simple random sample, the 95 percent margin of error on a percentage is:
MOE = 1.96 x square root of ( p x (1 minus p) / n )
p is the result as a decimal, n is the sample size. The margin is largest at p = 0.5, so pollsters usually report that worst case.
| Sample size | MOE at 50%, 95% confidence |
|---|---|
| 100 | plus or minus 9.8 points |
| 400 | plus or minus 4.9 points |
| 600 | plus or minus 4.0 points |
| 1,000 | plus or minus 3.1 points |
| 1,500 | plus or minus 2.5 points |
| 2,500 | plus or minus 2.0 points |
Quadrupling the sample halves the margin. That is why polls stop at 1,000 or so: going from 1,000 to 4,000 costs four times as much for about 1.5 points of precision.
Real polls are weighted to match the population by age, sex, education and so on. Weighting widens the margin (statisticians call this the design effect), so a well-run poll's reported margin is often a bit larger than the table.
The margin on a lead
Example: the county commission poll
An invented poll of 600 likely voters in Kessler County: Candidate A 52 percent, Candidate B 45 percent, margin of error plus or minus 4 points.
- A's margin: 1.96 x square root of (0.52 x 0.48 / 600) = 0.040, so 48 to 56 percent.
- B's range, with the same 4-point margin: 41 to 49 percent.
- Margin on the lead: 1.96 x square root of ((0.52 + 0.45 minus 0.07 squared) / 600) = 0.079, or about 7.9 points.
The 7-point lead has a margin of nearly 8 points. As Pew Research Center explains, the margin on the gap between two candidates in one poll is about twice the stated margin. Honest copy: "A leads B 52 to 45 percent, within the poll's margin of error for the gap between them."
Subgroups
Polls often break results out by age or region. Each subgroup is a smaller sample. If 150 of the 600 respondents are under 35, their margin is about plus or minus 8 points. A finding that "young voters favor B by 6" from 150 people is mush.
What the margin does not cover
The margin of error measures one thing: the luck of the draw in random sampling. Pew and AAPOR both list what it leaves out:
- Coverage: people the poll could never reach.
- Nonresponse: people reached who refused, if they differ from people who answered.
- Wording and order: "a tax increase for schools" and "an investment in our children" poll differently.
- Modeling: guesses about who will actually vote.
And for opt-in online panels, AAPOR's position is blunt: it is currently impossible to produce statistically valid margins of sampling error from nonprobability opt-in samples. Some pollsters report a "credibility interval" instead, which depends on the pollster's statistical model being right. AAPOR says the public cannot rely on it the way it relies on a margin of sampling error.
Confidence intervals in plain words
A 95 percent confidence interval comes from a method that, used over and over on new samples, catches the true value about 95 times in 100. For any single poll, you don't know if yours is one of the 95 or one of the 5.
Safe phrasing: "The poll estimates support at 52 percent, with a margin of error of 4 points." Unsafe: "There is a 95 percent chance support is between 48 and 56 percent." Statisticians argue about that second sentence, and you don't need to join them.
Watch the confidence level. Polls usually use 95 percent. The Census Bureau publishes ACS margins at 90 percent, which makes them narrower for the same data. To compare, multiply an ACS margin by about 1.19 (1.96 divided by 1.645).
Statistical significance
"Statistically significant" means a result would be unlikely if there were really no effect, unlikely enough to cross a threshold the researchers chose, often p below 0.05. That is all it means.
In 2016 the American Statistical Association issued a formal statement on p-values. Among its six principles: a p-value does not measure the probability that a hypothesis is true; decisions should not rest only on whether a p-value passes a threshold; and a p-value does not measure the size or importance of an effect.
Example: significance is not size
| Study | Groups | Result | p-value | 95% interval for the difference |
|---|---|---|---|---|
| Big records study | 1 million each | ER visits 10.0% vs 9.9% | 0.02 | 0.02 to 0.18 points |
| Small pilot | 20 each | Recovery 50% vs 25% | 0.10 | minus 4 to plus 54 points |
The big study is "significant" and the effect is a tenth of a point. The pilot is "not significant" and the effect could be huge or nothing. Neither headline word tells the reader what they need. The interval does.
So report effect sizes with their ranges. "Not significant" does not mean "no effect." It often means "too small a study to tell."
Common mix-ups
- Claim: A lead larger than the margin of error is a sure thing. What the evidence shows: The lead needs about twice the stated margin, and nonsampling errors can be bigger than either.
- Claim: Significant means important. What the evidence shows: The ASA statement says a p-value does not measure the size or importance of an effect.
- Claim: A bigger population needs a much bigger sample. What the evidence shows: Once the population is far larger than the sample, precision depends almost entirely on sample size.
Reporting angle
- For any poll, ask for the full topline and crosstabs, field dates, sample source (probability or opt-in), weighting and the sponsor.
- For any study in a press release, find the paper and look for the effect size and confidence interval, not just the word "significant."
- When two survey estimates differ, run the test (lesson 1 shows the Census Bureau version) before writing that one is higher.
Try it
- Calculate the 95 percent margin of error for a poll of 800 at 50 percent. (Expect about 3.5 points.)
- Find a recent published poll. Note sample size, method, margin and the leading gap. Is the gap larger than twice the margin?
- Find a news story that uses "significant" about a study. Find the study and write one sentence giving the effect size in plain terms.
- Rewrite: "There's a 95 percent chance the levy will pass, since it leads by 3 points with a 3-point margin of error."
Check yourself
1. A poll of 600 likely voters has Candidate A at 52 percent and B at 45 percent, margin of error plus or minus 4 points. What is the honest description?
The margin on the gap between two candidates in the same poll is roughly twice the stated margin, about 7.9 points here. A 7-point lead sits inside it.
2. About what is the 95 percent margin of error for a simple random sample of 1,000 at a 50 percent result?
1.96 x the square root of (0.5 x 0.5 / 1,000) = 0.031, or about 3.1 points.
3. Which error does a poll's margin of error cover?
The margin reflects sampling variability only. Pew Research Center and AAPOR both stress that coverage, nonresponse, wording and modeling errors are outside it.
4. According to the American Statistical Association's 2016 statement, what does a p-value NOT tell you?
The ASA statement says a p-value does not measure the probability that the studied hypothesis is true, or that the data were produced by chance alone.
5. A study of 2 million records finds ER visit rates of 10.0 percent versus 9.9 percent, p = 0.02. What is the best summary?
With huge samples, tiny differences reach significance. The ASA statement says a p-value does not measure the size or importance of an effect.
6. AAPOR's position on margins of error for opt-in online polls is that...
AAPOR says it is currently impossible to produce statistically valid margins of sampling error from nonprobability opt-in samples.
Terms in this lesson
- Sample
- The people or records actually measured, chosen to stand in for a larger population.
- Confidence interval
- A range built from sample data by a method that captures the true value a stated share of the time, such as 95 percent.
- Statistically significant
- A result unlikely enough under a no-difference assumption that it passes a chosen threshold, often p below 0.05. It says nothing about size or importance.
- p-value
- The probability of data at least this extreme if a specified model, usually 'no effect,' were true.
- Probability sample
- A sample where every member of the population has a known chance of being chosen, the basis for a valid margin of error.
Sources
- Pew Research Center, Understanding the margin of error in election polls (2016) checked 2026-10
- AAPOR, Understanding a credibility interval and how it differs from the margin of sampling error checked 2026-10
- American Statistical Association, Statement on Statistical Significance and P-Values (2016) checked 2026-10
- U.S. Census Bureau, Using ACS Estimates and Margins of Error (training slides) checked 2026-10