Draft — content is not yet teacher-reviewed
NCMath
NC Math 3
Strands
Number and Quantity1 ✓Algebra13 ✓Functions13 ✓Geometry9 ✓Statistics and Probability5 ✓
NC.M3.S-IC.1 · S-IC.3 · S-IC.6Print or save as PDF
Sampling, Study Design, and Reading Reports
Most of the time we cannot ask everyone. We study a sample and use it to say something about a whole population. This guide shows why a random sample lets you do that, how the three kinds of study (sample surveys, observational studies and experiments) use randomization differently, and what each one lets you conclude. Then you will practice evaluating a report: who collected the data, how, and how the graph shows it.
Source: Authored for NCMath (not from the PDF prototype) — needs teacher review
From a random sample to a population (S-IC.1)
Population, sample, parameter, statistic
The population is the whole group you want to know about. The sample is the part of it you actually measure.
A number that describes the whole population is a parameter. A number that describes a sample is a statistic. You usually cannot get the parameter, so you use the statistic to estimate it. This is called making an inference.
The sample is called random when chance, not a person's choice, decides who is in it. In a simple random sample, every member of the population and every group of the same size has an equal chance of being picked.
TermMeaningExample
PopulationEveryone or everything you want to know aboutAll 2,000 students at a school
SampleThe part of the population you measure25 students picked by random number
ParameterA number about the whole population40% of all 2,000 students ride the bus
StatisticA number about the sample9 of the 25 sampled students ride the bus: 9 ÷ 25 = 36%
Example
A school has 2,000 students. A random sample of 25 is asked how they get to school, and 9 ride the bus. Name the population, the sample, and the statistic.
Population: all 2,000 students. Sample: the 25 students picked. Statistic: the sample proportion 9 ÷ 25 = 0.36 = 36%. The parameter is the percent of all 2,000 students who ride the bus. It is not known from this sample alone, but the 36% is the estimate of it.
Trap: Calling the sample's percent "the percent for the school". A statistic is only an estimate of the parameter, and a different sample would give a different statistic.
Why the sample must be random
If a person chooses who is in the sample, the choice can lean toward one kind of member without anyone meaning to. Then the sample's statistic is pushed in one direction, and every new sample is pushed the same way. That push is bias.
Random selection gives every member a fair chance, so the sample has no built-in lean. A random sample is more likely to look like the population, but it is never guaranteed to match it.
The inference only goes as far as the population the sample was drawn from. A random sample of students at one school says something about that school, not about every school in the state.
20406080100true value 40%
Sampling variability: different samples, different statistics
In this simulation, the population is 2,000 students and exactly 40% (800 students) ride the bus. We took 24 different random samples of 25 students. Each dot is one sample, placed at the percent of that sample who ride the bus. The dashed line marks the true value, 40%.
The dots are not all at 40%. They run from 24% to 52%, and only 4 of the 24 samples hit exactly 40%. All of the samples were taken the same fair way, so this spread is just chance. It is called sampling variability.
The dots pile up around 40%, with fewer further away. That is why one random sample is a reasonable estimate of the parameter. It is also why you should not expect it to be exactly right.
Each dot is one random sample of 25 students. Horizontal axis: percent of the sample who ride the bus.
20406080100true value 40%
A non-random sample misses in one direction
Now suppose the 25 students are always picked from the 300 students standing at the bus loop (210 of them, or 70%, ride the bus). We again took 24 samples, with the same dashed line at the true 40%.
The spread from sample to sample is still there, 44% to 84%. But the whole group of dots sits to the right of 40%. All 24 samples are above the true value. Taking more samples this way would not fix it, because the method itself leans.
Same population and same size of sample, but the students were picked at the bus loop instead of at random.
Trap: Thinking a big sample cures bias. Sampling variability shrinks when you ask more people, but a biased method stays biased however many people you ask.
Three kinds of study (S-IC.3)
Survey, observational study, experiment
A sample survey asks a sample questions to estimate something about a population, such as a percent who favor a plan. The purpose is to describe the population.
An observational study records what happens to subjects without changing anything. The researcher does not choose who gets which condition. The purpose is to look for patterns and associations.
An experiment applies a treatment to subjects on purpose and compares the results with a group that did not get it. The purpose is to find out whether the treatment causes a difference.
There are two different jobs for chance. Random selection means chance decides who is in the sample. It lets you generalize to the population. Random assignment means chance decides who gets which treatment. It lets you make cause-and-effect conclusions.
DesignRandom how?Can conclude
Sample surveyRandom selection of who is askedDescribes the population (an estimate). No cause and effect.
Observational studyRandom selection of subjects, when possible. Nothing is assigned.An association (a pattern) only. No cause and effect.
ExperimentRandom assignment of subjects to treatments (and random selection too, if possible)Cause and effect, when the groups were assigned at random. It applies to the population only as far as the subjects were randomly chosen from it.
Why only an experiment can show cause and effect
In an observational study, the groups form themselves. People who take vitamin D may also exercise more, eat better or see a doctor more often. These other factors, called confounding variables, are mixed in with the thing you are looking at. You cannot tell which one caused the difference.
In an experiment, chance puts each subject in a group. The groups are alike on average in every way, including the ways nobody measured. If the results are different afterward, the treatment is the reasonable explanation.
A control group (no treatment, or a placebo, which is a fake treatment) gives you something to compare with.
Example
Researchers find that students who sleep 8 hours score higher on tests. Another team takes 60 volunteer students, uses a random number generator to put 30 on an 8-hour schedule and 30 on a 6-hour schedule for a week, then gives all 60 the same test. Which is which, and which can show that sleep causes higher scores?
The first is an observational study: the researchers only recorded sleep and scores, and the students chose their own sleep. It can show an association, but not that sleep caused the scores (students who sleep more might also study more). The second is an experiment: the researchers assigned the sleep schedules, and chance decided who got which. If the scores are different by more than chance, the sleep schedule is the reasonable cause. Because the 60 were volunteers and not a random sample of all students, the result applies most safely to students like them.
Trap: Mixing up the two kinds of randomness. A random sample (selection) lets you generalize to a population. Random assignment lets you claim cause and effect. A study can have either one, both or neither.
Evaluating reports (S-IC.6)
Three questions to ask about any report, then a verdict
  1. Source: who collected and paid for the data? Is the group trying to sell something or win an argument? Is the original data available, or only a headline?
  2. Design: was it a survey, an observational study or an experiment? How were the subjects chosen, and were groups assigned at random? How big was the sample?
  3. Display: does the graph or table show the data fairly? Check the axes, the intervals and the shapes.
  4. Then decide what the report really supports. A claim of cause and effect needs a randomized experiment. A claim about a whole population needs a random sample of that population.
Spotting bias in how the sample was collected
A fix for the first three is the same: choose the sample by random selection from a list that includes the whole population. A fix for the fourth is to word the question neutrally.
BiasWhat went wrongExample
Voluntary responsePeople choose themselves to answer. Those with strong opinions respond most.A radio call-in poll about a new stadium
ConvenienceThe sample is whoever is easiest to reach.Surveying only the first 30 students into the cafeteria
UndercoverageSome part of the population has no chance of being chosen.A survey made only by landline phone, which leaves out people who have no landline
Leading questionThe wording pushes toward one answer."Don't you agree the wasteful stadium should be stopped?"
Brand ABrand B48495051525352%50%
Misleading graphs: a truncated axis
Two brands report 52% and 50% satisfied customers. In this graph the vertical axis starts at 48, not 0, so bar A is 4 units tall and bar B is only 2. A looks twice as tall as B, but the numbers differ by just 2 percentage points.
Always read the axis numbers. A bar's height should be in proportion to the value, which only works when the axis starts at 0.
Truncated axis: the same data as the next graph. Look at the numbers on the left.
Trap: Judging by how the bars look. Here 52 ÷ 50 = 1.04, so A is 4% larger, not 100% larger.
Brand ABrand B20406052%50%
The same data, drawn fairly
With the axis starting at 0, the bars are in proportion to the values and the small difference looks small.
If a small change really matters, a report can say so in words and still show the whole axis.
Axis starts at 0: bar heights are in proportion to 52 and 50.
Other misleading displays
Unequal intervals: a histogram or a number line whose steps are not the same size, such as 0–10, 10–20, 20–50, 50–100, bends the picture. Equal steps on the axis should be equal amounts.
3D pictographs: a picture whose height, width and depth all grow makes a doubled value look much bigger. If a glass icon is twice as tall and twice as wide to show a value that is twice as big, its area is 2 × 2 = 4 times as big, and a 3D version suggests even more.
Others to watch for: a graph with no axis labels or units, a cropped time period, and a headline that claims more than the data show.
Example
A web page for a vitamin company says, "Survey: 9 of 10 customers feel more energetic!" The survey was an option on the company's own site, and a 3D picture of a glass is drawn about twice as large for the vitamin as for the competitor. Critique the report.
Source: the company that sells the product, so it has a reason to show good news. Design: an online survey that customers chose to answer is voluntary response, and it only includes customers, not people who stopped buying. It is a survey or an observational study, not an experiment, so even a real 90% could not show the vitamin caused the energy. Display: a doubled 3D icon exaggerates the difference. The report also gives no sample size.
Memory hook: Random selection lets you generalize. Random assignment lets you say "causes". Before you trust a report, ask: who, how, and how shown?
Practice
1. A school has 1,200 students. The principal chooses 100 of them using a random number generator and asks each how they get to school. 37 say they walk. (a) Name the population and the sample. (b) What is the statistic, and what does it estimate? (c) If she chose 100 different students, would she expect the same statistic?
2. Classify each study as a sample survey, an observational study or an experiment. (a) A polling group phones 800 randomly chosen registered voters and asks which candidate they prefer. (b) A researcher looks at medical records of 2,000 adults and compares the blood pressure of those who chose to take a daily vitamin with those who did not. (c) A doctor randomly divides 60 patients into two groups of 30. One group takes a new pill and the other takes a placebo, and she compares their blood pressure after a month.
3. Name the main source of bias in each plan. (a) A radio station asks listeners to call in and say whether they support a new stadium. (b) A student surveys the first 30 classmates who walk into the cafeteria about lunch quality. (c) A town surveys residents about internet use by calling only landline numbers from an old phone book. (d) A survey asks, "Don't you agree that the wasteful, overpriced stadium should be stopped?"
4. A teacher wants to test whether a new tutoring app improves quiz scores. Eighty students volunteer. (a) Describe how to use randomization. (b) If the app group scores clearly higher, can she say the app caused the improvement? (c) Can she say it works for all students in the state?
5. A newspaper reports that students who eat breakfast have a higher mean GPA than students who skip it. Researchers looked at school records for 300 students that show whether each one eats breakfast and what their GPA is. The principal says, "Let's require breakfast to raise GPAs." (a) What kind of study is this? (b) Is the principal's causal conclusion justified? (c) Give a reason the data could show this pattern even if breakfast has no effect.
6. A school has 2,000 students and 40% ride the bus. Five different random samples of 25 students gave these numbers of bus riders: 7, 11, 9, 13, 10. (a) Find each sample proportion. (b) Find the range of the five proportions. (c) Does the spread mean someone sampled incorrectly?
Brand ABrand B59606162636463%60%
7. A magazine compares two phone brands. The graph shows Brand A at 63% satisfied and Brand B at 60% satisfied, and Brand A's bar looks 4 times as tall. (a) What is misleading about the graph? (b) How many times as tall should Brand A's bar be if the axis starts at 0? (c) How could the graph be fixed?
8. A website for a snack company says: "Taste test: 8 out of 10 people prefer our chips!" It explains that the 10 people were the first 10 customers to walk up to the company's booth at a fair. The report shows a picture of a chip bag for each brand, and the bag for the company's brand is drawn twice as tall and twice as wide as the competitor's. (a) Identify the source, the design and the display. (b) Name the biggest problem with each. (c) Can the report say 80% of all people prefer the chips?
a devpromptu thing