NC.M3.S-IC.4 · S-IC.5Print or save as PDF
Simulation, Margin of Error, and Comparing Groups
A sample is only a piece of a population, so the number you get changes every time you sample. Simulation lets you see how much it changes. You will use it two ways. First, to estimate a population mean or proportion and find a margin of error. Second, to decide whether a difference between two groups in an experiment is real or could just be luck of the draw.
Source: Authored for NCMath (not from the PDF prototype) — needs teacher review
The big idea: repeat it many times
One sample is one guess
Ask 100 students a yes/no question and 56 say yes. The sample proportion is 0.56. Ask a different 100 students and you might get 0.52 or 0.61.
A statistic is a number computed from a sample, like a sample proportion or a sample mean. Its value changes from sample to sample. Simulation shows how much.
A simulation uses random numbers to imitate taking a sample (or running an experiment) again and again. You record the statistic each time, then look at the spread of all the results.
- Decide what to imitate. For an estimate, imitate sampling from a population that looks like your sample. For two groups, imitate the case where the group makes no difference.
- Do one random trial and record the statistic.
- Repeat many times. This guide uses 1000 trials.
- Graph all the results in a dot plot or histogram. Look at the center and the spread.
| Standard | Question | What you simulate | What you read off |
|---|---|---|---|
| S-IC.4 | How well does a sample estimate the population? | Many samples of the same size | Spread of the sample statistics, which gives the margin of error |
| S-IC.5 | Is a difference between two groups real? | Many shuffles of the group labels | Where the observed difference falls in the shuffled results |
Estimating a proportion and its margin of error (S-IC.4)
Simulate many samples and look at the spread
Margin of error Use the SD of the simulated statistics, not the SD of the original data. — margin of error ≈ 2 × (SD of the simulated sample statistics)
The estimate — estimate = sample statistic ± margin of error
A poll of 100 students finds that 56 prefer a later start time, so the sample proportion is 0.56. We want to estimate the proportion of all students.
To simulate, act as if the whole population has a proportion of 0.56. Draw 1000 random samples of size 100 from it and record each sample proportion. The graph shows all 1000 results. Each bar counts how many samples gave that proportion.
The 1000 results are centered at about 0.559 and have a standard deviation (SD) of about 0.050. The dashed line marks the observed 0.56.
The thick green bar runs from 2 SD below the center to 2 SD above it. In this simulation, 952 of the 1000 samples (about 95%) landed inside it. The few orange bars outside are the unusual samples.
Example
Use the simulation above to estimate the proportion of all students who prefer a later start time.
SD of the 1000 simulated proportions ≈ 0.050. Margin of error ≈ 2 × 0.050 = 0.10. Estimate: 0.56 ± 0.10, so from 0.46 to 0.66. We estimate that between 46% and 66% of all students prefer a later start.
SD of the 1000 simulated proportions ≈ 0.050. Margin of error ≈ 2 × 0.050 = 0.10. Estimate: 0.56 ± 0.10, so from 0.46 to 0.66. We estimate that between 46% and 66% of all students prefer a later start.
Trap: Using the SD of the original yes/no answers, or just using the sample proportion with no margin at all. The margin of error comes from the spread of the simulated sample proportions.
Estimating a mean works the same way
Nothing changes when the statistic is a sample mean. Simulate many samples, compute the mean of each, and find the SD of those simulated means.
Example
A sample of 40 cereal boxes has a mean weight of 12.1 oz. A simulation of 1000 samples of size 40 gives sample means with an SD of 0.15 oz. Estimate the population mean weight.
Margin of error ≈ 2 × 0.15 = 0.30 oz. Estimate: 12.1 ± 0.3 oz, so from 11.8 oz to 12.4 oz. We estimate that the mean weight of all boxes is between 11.8 oz and 12.4 oz.
Margin of error ≈ 2 × 0.15 = 0.30 oz. Estimate: 12.1 ± 0.3 oz, so from 11.8 oz to 12.4 oz. We estimate that the mean weight of all boxes is between 11.8 oz and 12.4 oz.
Bigger samples give smaller margins of error
Samples of 25
Same population (proportion 0.56), but now each sample has only 25 people. A sample of 25 with 14 yes answers also gives 0.56.
The 1000 simulated proportions are centered at about 0.558, but they are spread wide. The SD is about 0.095.
Margin of error ≈ 2 × 0.095 ≈ 0.19, so the estimate is 0.56 ± 0.19, from 0.37 to 0.75.
Samples of 100
Both graphs use the same scale, so you can compare them directly. With samples of 100 the results bunch up tightly around the center.
The SD is about 0.050, so the margin of error is about 0.10. The estimate is 0.56 ± 0.10, from 0.46 to 0.66.
| Sample size n | SD of simulated proportions | Margin of error (2 SD) | Estimate |
|---|---|---|---|
| 25 | 0.095 | 0.19 | 0.56 ± 0.19 |
| 100 | 0.050 | 0.10 | 0.56 ± 0.10 |
Example
How did the margin of error change when the sample size went from 25 to 100?
The sample size was multiplied by 4 (25 × 4 = 100). The margin of error went from 0.19 to 0.10, about half. A bigger sample gives a smaller margin of error, so a more precise estimate.
The sample size was multiplied by 4 (25 × 4 = 100). The margin of error went from 0.19 to 0.10, about half. A bigger sample gives a smaller margin of error, so a more precise estimate.
Trap: Thinking that doubling the sample size cuts the margin of error in half. To cut it in half you need about 4 times as many people. Doubling the sample only shrinks the margin of error by about 30%.
Is the difference between two groups real? (S-IC.5)
An experiment with two groups
Ten plants were randomly given fertilizer and ten were not. After a month the growth in centimeters was measured. The fertilized plants grew a mean of 16.7 cm and the control plants grew a mean of 14.4 cm.
The observed difference in means is 16.7 − 14.4 = 2.3 cm. The question: did the fertilizer cause the extra growth, or could random assignment alone produce a gap like that?
Shuffle the labels (re-randomization)
Suppose the fertilizer did nothing. Then each plant's growth would be the same whichever group it landed in, and the labels "fertilizer" and "control" would be just random tags. Shuffling the tags imitates running the experiment again under that assumption.
- Write all 20 growth values in one pool.
- Shuffle the pool and deal 10 values to a fake "fertilizer" group and 10 to a fake "control" group.
- Record the difference in the two group means.
- Repeat 1000 times.
- Graph the 1000 differences and mark the observed difference on the graph.
Where does the observed difference fall?
Decision rule Compare the observed difference with the shuffled differences. — beyond about 2 SD, or rarely reached by shuffling → evidence the groups really differ
The 1000 shuffled differences are centered at about 0, which makes sense: shuffling destroys any real group effect. Their SD is about 0.92, so the middle 95% runs from about −1.8 to 1.8 (the green bar).
The observed difference of 2.3 is far out in the tail, about 2.5 SD from the center. Only 19 of the 1000 shuffles gave a difference as far from 0 as 2.3 in either direction (orange bars and beyond).
A difference in the tails, beyond about 2 SD or rarely reached by shuffling, is hard to explain by chance alone. That is evidence of a real difference.
Example
Use the simulation to decide whether the fertilizer made a difference in plant growth.
SD of the shuffled differences ≈ 0.92, so 2 SD ≈ 1.84. Observed difference = 2.3, which is more than 1.84, so it is beyond 2 SD. Only 19 of 1000 shuffles (1.9%) were that extreme. Conclusion: there is evidence that the fertilizer increased plant growth.
SD of the shuffled differences ≈ 0.92, so 2 SD ≈ 1.84. Observed difference = 2.3, which is more than 1.84, so it is beyond 2 SD. Only 19 of 1000 shuffles (1.9%) were that extreme. Conclusion: there is evidence that the fertilizer increased plant growth.
When the difference is not in the tail
If the observed difference lands in the middle of the shuffled results (within about 2 SD, or reached by many of the shuffles), then random assignment alone could easily have produced it.
In that case the data do not give evidence of a real difference. This is not the same as showing the groups are equal.
Trap: Writing "this proves the fertilizer works" or "there is no difference." A simulation gives evidence, not proof. Say "there is evidence that…" for a result in the tail and "there is not enough evidence that…" for a result in the middle.
Memory hook: Estimating: simulate samples, margin of error ≈ 2 SD, bigger n means smaller margin. Comparing: shuffle the labels, see where the observed difference lands, tail means evidence.
Practice
1. A sample of 36 pizza deliveries has a mean delivery time of 28 minutes. A simulation of 1000 samples of size 36 gives sample means with a standard deviation of 1.5 minutes. Give an estimate of the population mean delivery time with its margin of error, as an interval.
2. In a poll of 200 students, 130 said they have a part-time job, so the sample proportion is 0.65. A simulation of 1000 samples of size 200 gives sample proportions with a standard deviation of 0.034. Estimate the proportion of all students with a part-time job, with a margin of error.
3. A simulation with samples of size 50 gives a margin of error of about 0.14. About what margin of error would you expect from a simulation with samples of size 200? Explain.
4. A poll of 100 voters finds 54% support a new bus line. A simulation of 1000 samples of size 100 gives sample proportions with a standard deviation of 0.05. A reporter writes: "A majority of all voters support the bus line." Does the estimate back up that claim? Explain.
5. In an experiment, 15 students were randomly assigned to be tutored and 15 were not. The tutored group's mean test score was 6 points higher than the other group's. To see if that could be chance, the group labels were shuffled 1000 times and the difference in means recorded each time. The table shows how the shuffled results compared. What do you conclude?
| Shuffled difference | Number of the 1000 shuffles |
|---|---|
| As far from 0 as 6 points (either direction) | 12 |
| Closer to 0 than 6 points | 988 |
6. Ten students studied with flashcards (group A) and ten studied with a summary sheet (group B), then took a memory quiz scored out of 20. Group A averaged 13.5 and group B averaged 13.1, an observed difference of 0.4. The graph shows the differences from 1000 shuffles of the group labels. The shuffled differences have a mean of 0.00 and a standard deviation of 0.69. Is the 0.4 evidence that the study method matters?
7. In the plant experiment, a student writes: "The observed difference of 2.3 cm is beyond 2 standard deviations of the shuffled differences, so this proves the fertilizer works." (a) What do the shuffles assume? (b) Rewrite the student's conclusion with correct wording.
8. Twelve students were randomly split into two groups of six. Group 1 listened to music while studying and group 2 did not. On the quiz, group 1's mean was 1.9 points higher. After 1000 shuffles of the group labels, the shuffled differences had a mean of 0 and a standard deviation of 1.0. A student concludes: "Music has no effect on quiz scores." Is that conclusion justified? What can you say?