Most of the time we cannot ask everyone. We study a sample and use it to say something about a whole population. This guide shows why a random sample lets you do that, how the three kinds of study (sample surveys, observational studies and experiments) use randomization differently, and what each one lets you conclude. Then you will practice evaluating a report: who collected the data, how, and how the graph shows it.
From a random sample to a population (S-IC.1)
Population, sample, parameter, statistic
The population is the whole group you want to know about. The sample is the part of it you actually measure.
A number that describes the whole population is a parameter. A number that describes a sample is a statistic. You usually cannot get the parameter, so you use the statistic to estimate it. This is called making an inference.
The sample is called random when chance, not a person's choice, decides who is in it. In a simple random sample, every member of the population and every group of the same size has an equal chance of being picked.
Term
Meaning
Example
Population
Everyone or everything you want to know about
All 2,000 students at a school
Sample
The part of the population you measure
25 students picked by random number
Parameter
A number about the whole population
40% of all 2,000 students ride the bus
Statistic
A number about the sample
9 of the 25 sampled students ride the bus: 9 ÷ 25 = 36%
Example
A school has 2,000 students. A random sample of 25 is asked how they get to school, and 9 ride the bus. Name the population, the sample, and the statistic. Population: all 2,000 students. Sample: the 25 students picked. Statistic: the sample proportion 9 ÷ 25 = 0.36 = 36%. The parameter is the percent of all 2,000 students who ride the bus. It is not known from this sample alone, but the 36% is the estimate of it.
Trap: Calling the sample's percent "the percent for the school". A statistic is only an estimate of the parameter, and a different sample would give a different statistic.
Why the sample must be random
If a person chooses who is in the sample, the choice can lean toward one kind of member without anyone meaning to. Then the sample's statistic is pushed in one direction, and every new sample is pushed the same way. That push is bias.
Random selection gives every member a fair chance, so the sample has no built-in lean. A random sample is more likely to look like the population, but it is never guaranteed to match it.
The inference only goes as far as the population the sample was drawn from. A random sample of students at one school says something about that school, not about every school in the state.
Sampling variability: different samples, different statistics
In this simulation, the population is 2,000 students and exactly 40% (800 students) ride the bus. We took 24 different random samples of 25 students. Each dot is one sample, placed at the percent of that sample who ride the bus. The dashed line marks the true value, 40%.
The dots are not all at 40%. They run from 24% to 52%, and only 4 of the 24 samples hit exactly 40%. All of the samples were taken the same fair way, so this spread is just chance. It is called sampling variability.
The dots pile up around 40%, with fewer further away. That is why one random sample is a reasonable estimate of the parameter. It is also why you should not expect it to be exactly right.
Each dot is one random sample of 25 students. Horizontal axis: percent of the sample who ride the bus.
A non-random sample misses in one direction
Now suppose the 25 students are always picked from the 300 students standing at the bus loop (210 of them, or 70%, ride the bus). We again took 24 samples, with the same dashed line at the true 40%.
The spread from sample to sample is still there, 44% to 84%. But the whole group of dots sits to the right of 40%. All 24 samples are above the true value. Taking more samples this way would not fix it, because the method itself leans.
Same population and same size of sample, but the students were picked at the bus loop instead of at random.
Trap: Thinking a big sample cures bias. Sampling variability shrinks when you ask more people, but a biased method stays biased however many people you ask.
Three kinds of study (S-IC.3)
Survey, observational study, experiment
A sample survey asks a sample questions to estimate something about a population, such as a percent who favor a plan. The purpose is to describe the population.
An observational study records what happens to subjects without changing anything. The researcher does not choose who gets which condition. The purpose is to look for patterns and associations.
An experiment applies a treatment to subjects on purpose and compares the results with a group that did not get it. The purpose is to find out whether the treatment causes a difference.
There are two different jobs for chance. Random selection means chance decides who is in the sample. It lets you generalize to the population. Random assignment means chance decides who gets which treatment. It lets you make cause-and-effect conclusions.
Design
Random how?
Can conclude
Sample survey
Random selection of who is asked
Describes the population (an estimate). No cause and effect.
Observational study
Random selection of subjects, when possible. Nothing is assigned.
An association (a pattern) only. No cause and effect.
Experiment
Random assignment of subjects to treatments (and random selection too, if possible)
Cause and effect, when the groups were assigned at random. It applies to the population only as far as the subjects were randomly chosen from it.
Why only an experiment can show cause and effect
In an observational study, the groups form themselves. People who take vitamin D may also exercise more, eat better or see a doctor more often. These other factors, called confounding variables, are mixed in with the thing you are looking at. You cannot tell which one caused the difference.
In an experiment, chance puts each subject in a group. The groups are alike on average in every way, including the ways nobody measured. If the results are different afterward, the treatment is the reasonable explanation.
A control group (no treatment, or a placebo, which is a fake treatment) gives you something to compare with.
Example
Researchers find that students who sleep 8 hours score higher on tests. Another team takes 60 volunteer students, uses a random number generator to put 30 on an 8-hour schedule and 30 on a 6-hour schedule for a week, then gives all 60 the same test. Which is which, and which can show that sleep causes higher scores? The first is an observational study: the researchers only recorded sleep and scores, and the students chose their own sleep. It can show an association, but not that sleep caused the scores (students who sleep more might also study more). The second is an experiment: the researchers assigned the sleep schedules, and chance decided who got which. If the scores are different by more than chance, the sleep schedule is the reasonable cause. Because the 60 were volunteers and not a random sample of all students, the result applies most safely to students like them.
Trap: Mixing up the two kinds of randomness. A random sample (selection) lets you generalize to a population. Random assignment lets you claim cause and effect. A study can have either one, both or neither.
Evaluating reports (S-IC.6)
Three questions to ask about any report, then a verdict
Source: who collected and paid for the data? Is the group trying to sell something or win an argument? Is the original data available, or only a headline?
Design: was it a survey, an observational study or an experiment? How were the subjects chosen, and were groups assigned at random? How big was the sample?
Display: does the graph or table show the data fairly? Check the axes, the intervals and the shapes.
Then decide what the report really supports. A claim of cause and effect needs a randomized experiment. A claim about a whole population needs a random sample of that population.
Spotting bias in how the sample was collected
A fix for the first three is the same: choose the sample by random selection from a list that includes the whole population. A fix for the fourth is to word the question neutrally.
Bias
What went wrong
Example
Voluntary response
People choose themselves to answer. Those with strong opinions respond most.
A radio call-in poll about a new stadium
Convenience
The sample is whoever is easiest to reach.
Surveying only the first 30 students into the cafeteria
Undercoverage
Some part of the population has no chance of being chosen.
A survey made only by landline phone, which leaves out people who have no landline
Leading question
The wording pushes toward one answer.
"Don't you agree the wasteful stadium should be stopped?"
Misleading graphs: a truncated axis
Two brands report 52% and 50% satisfied customers. In this graph the vertical axis starts at 48, not 0, so bar A is 4 units tall and bar B is only 2. A looks twice as tall as B, but the numbers differ by just 2 percentage points.
Always read the axis numbers. A bar's height should be in proportion to the value, which only works when the axis starts at 0.
Truncated axis: the same data as the next graph. Look at the numbers on the left.
Trap: Judging by how the bars look. Here 52 ÷ 50 = 1.04, so A is 4% larger, not 100% larger.
The same data, drawn fairly
With the axis starting at 0, the bars are in proportion to the values and the small difference looks small.
If a small change really matters, a report can say so in words and still show the whole axis.
Axis starts at 0: bar heights are in proportion to 52 and 50.
Other misleading displays
Unequal intervals: a histogram or a number line whose steps are not the same size, such as 0–10, 10–20, 20–50, 50–100, bends the picture. Equal steps on the axis should be equal amounts.
3D pictographs: a picture whose height, width and depth all grow makes a doubled value look much bigger. If a glass icon is twice as tall and twice as wide to show a value that is twice as big, its area is 2 × 2 = 4 times as big, and a 3D version suggests even more.
Others to watch for: a graph with no axis labels or units, a cropped time period, and a headline that claims more than the data show.
Example
A web page for a vitamin company says, "Survey: 9 of 10 customers feel more energetic!" The survey was an option on the company's own site, and a 3D picture of a glass is drawn about twice as large for the vitamin as for the competitor. Critique the report. Source: the company that sells the product, so it has a reason to show good news. Design: an online survey that customers chose to answer is voluntary response, and it only includes customers, not people who stopped buying. It is a survey or an observational study, not an experiment, so even a real 90% could not show the vitamin caused the energy. Display: a doubled 3D icon exaggerates the difference. The report also gives no sample size.
Memory hook: Random selection lets you generalize. Random assignment lets you say "causes". Before you trust a report, ask: who, how, and how shown?
NC Math 3 · NC.M3.S-IC.1 · S-IC.3 · S-IC.6
Sampling, Study Design, and Reading Reports
Practice
Name ____________________ Date ____________
1. A school has 1,200 students. The principal chooses 100 of them using a random number generator and asks each how they get to school. 37 say they walk. (a) Name the population and the sample. (b) What is the statistic, and what does it estimate? (c) If she chose 100 different students, would she expect the same statistic?
2. Classify each study as a sample survey, an observational study or an experiment. (a) A polling group phones 800 randomly chosen registered voters and asks which candidate they prefer. (b) A researcher looks at medical records of 2,000 adults and compares the blood pressure of those who chose to take a daily vitamin with those who did not. (c) A doctor randomly divides 60 patients into two groups of 30. One group takes a new pill and the other takes a placebo, and she compares their blood pressure after a month.
3. Name the main source of bias in each plan. (a) A radio station asks listeners to call in and say whether they support a new stadium. (b) A student surveys the first 30 classmates who walk into the cafeteria about lunch quality. (c) A town surveys residents about internet use by calling only landline numbers from an old phone book. (d) A survey asks, "Don't you agree that the wasteful, overpriced stadium should be stopped?"
4. A teacher wants to test whether a new tutoring app improves quiz scores. Eighty students volunteer. (a) Describe how to use randomization. (b) If the app group scores clearly higher, can she say the app caused the improvement? (c) Can she say it works for all students in the state?
5. A newspaper reports that students who eat breakfast have a higher mean GPA than students who skip it. Researchers looked at school records for 300 students that show whether each one eats breakfast and what their GPA is. The principal says, "Let's require breakfast to raise GPAs." (a) What kind of study is this? (b) Is the principal's causal conclusion justified? (c) Give a reason the data could show this pattern even if breakfast has no effect.
6. A school has 2,000 students and 40% ride the bus. Five different random samples of 25 students gave these numbers of bus riders: 7, 11, 9, 13, 10. (a) Find each sample proportion. (b) Find the range of the five proportions. (c) Does the spread mean someone sampled incorrectly?
7. A magazine compares two phone brands. The graph shows Brand A at 63% satisfied and Brand B at 60% satisfied, and Brand A's bar looks 4 times as tall. (a) What is misleading about the graph? (b) How many times as tall should Brand A's bar be if the axis starts at 0? (c) How could the graph be fixed?
8. A website for a snack company says: "Taste test: 8 out of 10 people prefer our chips!" It explains that the 10 people were the first 10 customers to walk up to the company's booth at a fair. The report shows a picture of a chip bag for each brand, and the bag for the company's brand is drawn twice as tall and twice as wide as the competitor's. (a) Identify the source, the design and the display. (b) Name the biggest problem with each. (c) Can the report say 80% of all people prefer the chips?
NC Math 3 · NC.M3.S-IC.1 · S-IC.3 · S-IC.6
Sampling, Study Design, and Reading Reports
Practice answer key
Name ____________________ Date ____________
1. (a) Population: all 1,200 students. Sample: the 100 chosen students. (b) The statistic is 37 ÷ 100 = 0.37 = 37%. It estimates the parameter, the percent of all 1,200 students who walk. (c) Not necessarily: a different random sample would give a statistic that is close but probably not the same.
The population is everyone the principal wants to know about, which is all 1,200 students. The sample is only the 100 who were asked. A number about the sample is a statistic, here the proportion 37 ÷ 100 = 0.37. The parameter is the same kind of number for all 1,200 students, and the principal does not know it. The statistic is her estimate of it. Because the sample was chosen by chance, a second sample of 100 would include different students, and sampling variability means its percent would likely come out a bit different, such as 33% or 41%.
(a) The pollsters select a random sample and only ask questions. Nothing is assigned. That is a sample survey. (b) The researcher only looks at records. The adults chose for themselves whether to take the vitamin, so no treatment was assigned. That is an observational study. (c) The doctor assigns the treatments (pill or placebo) and chance decides who gets which. That is an experiment. The key question is: did the researchers impose a treatment? Only in (c) did they.
(a) Listeners choose to call, and people with strong opinions are the most likely to call. That is voluntary response. (b) The first 30 students in the door are just the easiest ones to reach, and they may eat early for a reason. That is a convenience sample. (c) Residents with no landline, or not in the old phone book, have no chance to be picked, which is undercoverage. They may also use the internet differently from the people who are called. (d) The words "wasteful" and "overpriced" push the answer, so it is a leading question.
4. (a) Randomly assign 40 of the volunteers to use the app and 40 to not use it, then compare scores. (b) Yes, because assignment was random. (c) No: the 80 were volunteers, not a random sample of all students, so the result only applies safely to students like them.
Question
Needs
Does she have it?
Did the app cause the change?
Random assignment
Yes
Does it work for all students in the state?
Random selection from all students
No (volunteers)
(a) She should not let students pick their group, because the more motivated might choose the app. Instead chance assigns them: for example, numbers drawn at random put 40 in each group. (b) With random assignment, the two groups are alike on average in motivation, past grades and everything else, so if the app group does much better than chance alone would explain, the app is the reasonable cause. (c) Generalizing needs random selection from the population. These students volunteered, so they may differ from other students. Cause and effect comes from random assignment, and generalizing comes from random selection, and she has only the first.
5. (a) Observational study. (b) No. (c) Other factors could explain it, such as students who eat breakfast having a more regular routine or sleep schedule.
(a) Nobody assigned breakfast or no breakfast. The students already had their own habits and the researchers only looked at records, so this is an observational study. (b) An observational study can only show an association. Cause and effect needs random assignment. (c) A confounding variable can be mixed in with breakfast: for example, students with a steady sleep schedule, a calm home or a parent who checks homework may be the ones who eat breakfast and also do well in school. Then breakfast could look helpful even if it does nothing. To test the principal's claim, you would randomly assign some students to eat breakfast and others not to, and compare.
(a) Divide each count by 25: 7 ÷ 25 = 0.28, 11 ÷ 25 = 0.44, 9 ÷ 25 = 0.36, 13 ÷ 25 = 0.52, 10 ÷ 25 = 0.40. (b) The largest is 52% and the smallest is 28%, so the range is 24 percentage points. (c) Every sample was chosen at random, so nobody made a mistake. Different samples of the same population give different statistics. They tend to be near the true 40%, and here they run from 28% to 52%, much like the dots in the simulation above. Only one of the five equals 40% exactly.
7. (a) The vertical axis starts at 59, not 0, so the bar heights are not in proportion to the values. (b) 63 ÷ 60 = 1.05 times as tall. (c) Start the axis at 0 so the bar heights match the values.
The axis labels run from 59 to 64, which means each bar is only drawn from 59 up. Bar A is 63 − 59 = 4 units tall and Bar B is 60 − 59 = 1 unit tall, so A looks 4 ÷ 1 = 4 times as tall. But the values are 63 and 60, and 63 ÷ 60 = 1.05, so A is only 5% larger, a difference of 3 percentage points. A bar chart compares bar heights, so it should start at 0. The fixed graph shows the two bars almost the same height.
8. (a) Source: the snack company's own website. Design: a taste test of the first 10 people at its booth. Display: a pictograph with the company's bag drawn twice as tall and wide. (b) Source: the company has a reason to show good news. Design: a tiny convenience sample, not random. Display: doubling both dimensions makes the area 4 times as large. (c) No.
The source is the company that sells the chips, so you cannot assume the report is neutral, and it does not tell you who ran the test. The design is a convenience sample: the first 10 people to walk up to the booth are not chosen at random, and people at the booth may already like the brand. Ten people is also a very small sample, so a different 10 could give a very different result. The display is a pictograph in which the bag is twice as tall and twice as wide, so its area is 2 × 2 = 4 times as big. The eye reads area, so the picture makes the company's bag look 4 times as big as the other one. A pictograph should change only one dimension, so the area matches the value. Because the sample was not random and the group was tiny, the 80% describes only those 10 people. The report cannot say that 80% of all people prefer the chips.
NC Math 3 · NC.M3.S-IC.1 · S-IC.3 · S-IC.6
Sampling, Study Design, and Reading Reports
Quiz
Name ____________________ Date ____________
1. A forester measures the trunk diameters of a simple random sample of 36 of the 2,400 oak trees in a state forest. The mean diameter of the sample is 14.2 inches. (a) Name the population. Is 14.2 a parameter or a statistic? (b) A second forester takes her own simple random sample of 36 oaks from the same forest and gets a mean of 15.1 inches. Did one of them make a mistake? (c) Can the first forester use the 14.2 to describe oak trees in other states?
2. A district has 6,000 students. To estimate the percent who would ride a late bus, a researcher takes 30 random samples of 20 students and 30 random samples of 200 students. Each dot is the percent who said yes in one sample. The two dot plots use the same scale, but they are not labeled by sample size. (a) Which plot used the samples of 200 students? How can you tell? (b) About what percent of all 6,000 students would ride the late bus? (c) Suppose the samples of 200 had been taken only from students waiting in the band room, instead of at random. Would plot Y still center on the true percent? Explain.
3. For each study, say whether it used random selection, random assignment, both or neither. Then say what it can conclude. Study 1: A hospital randomly picks 300 of last year's patient records and compares the recovery times of patients whose doctors chose physical therapy with those whose doctors did not. Study 2: A shelter has 24 volunteer dogs. A coin flip decides which 12 dogs get food A and which 12 get food B for two months, and then the dogs' weights are compared.
4. A gardener wants to know whether a new plant food makes tomato plants grow taller. She puts the plant food on the 10 plants along the sunny south fence and none on the 10 plants along the shady north fence. At the end of the summer the fed plants are 8 cm taller on average. (a) Why can't she conclude that the plant food caused the difference? (b) Name a confounding variable. (c) Describe a better design.
5. Name the main source of bias in each plan, and say how to fix it. (a) A TV show asks viewers to text YES or NO about building a new park, and reports the percent of texts that say YES. (b) A researcher polls a city's residents, using a list of homeowners from the county tax office. (c) A survey asks, "Given how dangerous texting while driving is, don't you agree it should be banned?"
6. A gym newsletter prints the graph below, where the bar heights are the numbers of members in each age group. Its headline says, "Most of our members are between 30 and 50!" (a) What about the graph makes the headline look true? (b) What fraction of the members are actually between 30 and 50? (c) Which age group has the most members per year of age, and how could the graph be redrawn fairly?
NC Math 3 · NC.M3.S-IC.1 · S-IC.3 · S-IC.6
Sampling, Study Design, and Reading Reports
Quiz answer key
Name ____________________ Date ____________
1. (a) Population: all 2,400 oak trees in the forest. 14.2 is a statistic (a sample mean). (b) No: this is ordinary sampling variability. (c) No: the sample was drawn only from this forest.
(a) The population is every tree the forester wants to know about, all 2,400 oaks in the forest. A number computed from the sample, like the sample mean 14.2, is a statistic. The parameter would be the mean diameter of all 2,400 oaks, which nobody measured. (b) Both samples were chosen by chance, so neither forester did anything wrong. Different random samples contain different trees, so their means differ: here by 15.1 − 14.2 = 0.9 inch. This is sampling variability. Both numbers are estimates of the same parameter. (c) An inference only reaches as far as the population the sample was drawn from. Trees in other states had no chance of being picked, so the result does not describe them.
2. (a) Plot Y: its dots are packed much closer together. (b) About 35%. (c) No. A biased method leans one way no matter how many students are asked.
(a) Bigger random samples vary less from sample to sample, so their statistics bunch together. Plot Y's dots run from about 28% to 42%, while plot X's run from 10% to 65%. So Y is the samples of 200 and X is the samples of 20. (b) Both plots center at about 35%, and a random sample is not leaning either way, so about 35% of all students would ride the late bus. (c) The band room is not a random selection of the students. If band members like the late bus more or less than other students, every sample would lean the same way, and the dots would center away from the true percent. A larger sample makes the dots closer together but does not fix a leaning method.
3. Study 1: random selection only. It can describe last year's patients at that hospital, but it cannot show that physical therapy caused the difference in recovery. Study 2: random assignment only. It can show that the food caused a difference in weight for dogs like these, but it cannot be generalized to all dogs.
Study
Random selection?
Random assignment?
Can conclude
1: patient records
Yes
No
Association; describes last year's patients; no cause and effect
2: shelter dogs
No
Yes
Cause and effect for dogs like these; not all dogs
Study 1 used chance to pick the records, so what it finds can be generalized to last year's patients at that hospital. But the researchers did not assign anything: doctors chose who got therapy, and doctors may choose the patients with worse injuries, or the ones who will follow the plan. Those other factors are mixed in, so only an association can be claimed. Study 2 used chance to decide who gets which food, so the two groups are alike on average in everything else, and a difference in weight can reasonably be blamed on the food. But the dogs were volunteers, not a random sample of all dogs, so the result is safest for dogs like these.
4. (a) She did not assign the food at random: the fed plants were also the sunnier plants. (b) Amount of sunlight. (c) Randomly assign which 10 of the 20 plants get the food, treat all of them the same otherwise, and compare.
(a) She did apply a treatment, but she chose which plants got it, and her choice put the food and the sunny spot together. If the fed plants grew taller, it could be the food, the sun, or both, and there is no way to separate them. (b) Sunlight is the confounding variable: it differs between the groups and could affect height. (c) Chance should decide the groups, for example by drawing 10 names out of a hat. Then sunny and shady plants end up spread about evenly across both groups, so a clear difference in height afterward can reasonably be blamed on the plant food.
5. (a) Voluntary response; choose people at random instead of letting them choose themselves. (b) Undercoverage (renters are left out); choose at random from a list of all residents. (c) Leading question; ask it in neutral words.
(a) Viewers pick themselves, and people with strong opinions are the most likely to bother texting. That is voluntary response. A fix is to choose the people by random selection from a list that includes everyone. (b) Residents who rent do not own property, so they are not on the tax list and have no chance of being picked. That is undercoverage. A fix is to select at random from a list that includes all residents. (c) The words "how dangerous" and "don't you agree" push toward YES. That is a leading question. A fix is neutral wording, such as "Do you think texting while driving should be banned?"
6. (a) The 30–50 bar is five times as wide as each of the three younger groups' bars, so it takes up the most room. (b) 40 of 152 members, about 26%, which is not most. (c) Ages 22–26, with 9 members per year of age. Redraw with equal-width age groups or members per year.
(a) The age groups are not the same size: the three younger groups are 4 years wide, while 30–50 (and 50–70) are 20 years wide. The eye reads the whole area of a bar, so the wide bar seems to hold most of the members even though its height, 40, is only a little taller than the others. (b) The counts are 28 + 36 + 32 + 40 + 16 = 152 members, and 40 ÷ 152 ≈ 0.26, so only about 26% are between 30 and 50. (c) Divide each count by the width of its group: 28 ÷ 4 = 7, 36 ÷ 4 = 9, 32 ÷ 4 = 8, 40 ÷ 20 = 2 and 16 ÷ 20 = 0.8 members per year of age. The most crowded ages are 22–26. A fair graph uses equal-width age groups, or plots members per year of age, as in the answer graph.