6 Types of Sampling Bias: How to Avoid Sampling Bias

Sampling bias is the phenomenon that occurs when a research study design fails to collect a representative sample of a target population. This typically occurs because the selection criteria for respondents failed to capture a wide enough sampling frame to represent all viewpoints.

The cause of sampling bias almost always owes to one of two conditions.

• Poor methodology: In most cases, non-representative samples pop up when researchers set improper parameters for survey research. The most accurate and repeatable sampling method is simple random sampling where a large number of respondents are chosen at random. When researchers stray from random sampling (also called probability sampling), they risk injecting their own selection bias into recruiting respondents.
• Poor execution: Sometimes data researchers craft scientifically sound sampling methods, but their work is undermined when field workers cut corners. By reverting to convenience sampling (where the only people studied are those who are easy to reach) or giving up on reaching non-responders, a field worker can jeopardize the careful methodology set up by data scientists.

To visualize an example of sampling bias, imagine reading survey research that reported the most beloved food in America was steamed turnips. You find that result odd, as no one you know has ever expressed an affinity for steamed turnips. When you investigate the researchers' sampling methods, you discover they chose a very small sample size with strong opinions about turnip cuisine. Specifically, you've found an oversampling of people who work in the turnip industry, which makes the entire research sample dubious. As such, the survey research did not accurately capture Americans' culinary preferences, and follow-up research will be required.

Consider six of the most common types of sampling bias, which can dampen the external validity of study data.

1. 1. Self-selection bias: Also known as non-response bias, this source of bias plagues studies that rely on voluntary responses. When samples show self-selection bias, they over-represent the subset of the population that feels some sort of inclination to weigh in on an issue. For example, in presidential election polls, self-selection bias has been shown to over-represent people who trust institutions (like polling firms) and closely follow the news.
2. 2. Observer bias: This type of sampling bias often pops up when researchers weave their own opinions into questionnaires, which can impede people from answering neutrally. Surveys weighed down by loaded questions often surface in the partisan polling industry.
3. 3. Survivorship bias: Survivorship bias overweights respondents who have "survived" the selection criteria but don’t necessarily represent an overall population. For instance, surveying high school graduates about the efficacy of public schools may cut out the needed perspective of dropouts who did not earn a diploma.
4. 4. Undercoverage bias: Also called exclusion bias, this occurs when a population of interest is under-surveyed by researchers. This can sometimes trace back to the practice of convenience sampling, where researchers and field workers only survey individuals who are easy to reach.
5. 5. Healthy user bias: This type of bias—most commonly found in health studies—oversamples healthy members of a population.
6. 6. Berkson's fallacy: The opposite of a healthy user bias, Berkson's fallacy occurs when surveyors only study those who are very ill, such as hospital patients. In this situation, healthy people end up under-represented.

The best way to avoid sampling bias is to stick to probability-based sampling methods. These include simple random sampling, systematic sampling, cluster sampling, and stratified sampling. In these methodologies, respondents are only chosen through processes of random selection—even if they are sometimes sorted into demographic groups along the way.

The opposite form of sampling is called non-probability sampling, and it injects forms of bias into data collection. This bias can come down to convenience—as in the case of convenience sampling—or it can involve predetermined ideas about a population of interest, as is the case in quota sampling. Non-probability sampling does play a role in data collection and analysis. Among other things, it's typically more affordable than true random sampling, but its downside is its vulnerability to bias.

Get the MasterClass Annual Membership for exclusive access to video lessons taught by science luminaries, including Terence Tao, Bill Nye, Neil deGrasse Tyson, Chris Hadfield, Jane Goodall, and more.