Participation bias

Participation bias or non-response bias is a phenomenon in which the results of studies, polls, etc. become non-representative because the participants disproportionately possess certain traits which affect the outcome. These traits mean the sample is systematically different from the target population, potentially resulting in biased estimates.^[1]

For instance, a study found that those who refused to answer a survey on AIDS tended to be "older, attend church more often, are less likely to believe in the confidentiality of surveys, and have lower sexual self disclosure."^[2] It may occur due to several factors as outlined in Deming (1990).^[3]

Non-response bias can be a problem in longitudinal research due to attrition during the study.^[4]

Example

If one selects a sample of 1000 managers in a field and polls them about their workload, the managers with a high workload may not answer the survey because they do not have enough time to answer it, and/or those with a low workload may decline to respond for fear that their supervisors or colleagues will perceive them as surplus employees (either immediately, if the survey is non-anonymous, or in the future, should their anonymity be compromised). Therefore, non-response bias may make the measured value for the workload too low, too high, or, if the effects of the above biases happen to offset each other, "right for the wrong reasons." For a simple example of this effect, consider a survey that includes, "Agree or disagree: I have enough time in my day to complete a survey."

Prior to the 1936 U.S. presidential election, The Literary Digest mailed out 10 million questionnaires, of which 2.38 million were returned. Based on these returns, they predicted that Republican candidiate Alf Landon would be elected president with 57.08% of the popular vote and 370 of 531 electoral votes: in the election, Landon received only 37.54% of the popular vote and eight electoral votes in an unprecedented landslide loss to Democratic incumbent President Franklin D. Roosevelt. Subsequent research published in 1976 and 1988 concluded that non-response bias was the primary source of this error, although their sampling frame was also quite different from the vast majority of voters.^[1]

Non-responders have been shown to be associated with younger patients, poorer communities and those who are less satisfied and subsequently could be a source of bias.^[5]

Test

There are different ways to test for non-response bias. A common technique involves comparing the first and fourth quartiles of responses for differences in demographics and key constructs.^[6] In e-mail surveys some values are already known from all potential participants (e.g. age, branch of the firm, ...) and can be compared to the values that prevail in the subgroup of those who answered. If there is no significant difference this is an indicator that there might be no non-response bias.

In e-mail surveys those who didn't answer can also systematically be phoned and a small number of survey questions can be asked. If their answers don't differ significantly from those who answered the survey, there might be no non-response bias. This technique is sometimes called non-response follow-up.

Response rate

A common assumption is that response rates are linked to non-response bias, with lower response rates incurring a greater likelihood for results to be influenced by non-response bias. Academic research has disputed substantial linkages between response rate and non-response bias. A meta-analysis of 30 methodological studies on non-response bias by Robert M. Groves found that the coefficient of determination for variance in non-response bias by response rate was only 0.11, making it a weak predictor of non-response bias. Another meta-analysis of 44 methodological studies similarly found that methods that resulted in higher response rates, such as prior notification and incentives, would not necessarily reduce non-response bias and could sometimes even show increased non-response bias.^[7]^[8]^[9]

Some academic journals, particularly in the medical space, require minimum response rates to publish survey research as a means of mitigating non-response bias. For instance, JAMA has implemented a requirement of a 60% response rate for survey research, which has been compared to other heuristics like 0.05 p-values for statistical significance in social science research.^[10]

The pursuit of higher response rates can be counterproductive due to its questionable relationship with non-response bias and potentially unnecessary costs spent on methods of boosting response rates that could be better applied elsewhere. It may also result in gatekeeping of surveys that may be valid on their merits, but fail to satisfy a heuristic requirement on response rates.^[8]^[11]

Related terminology

Self-selection bias is a type of bias in which individuals voluntarily select themselves into a group, thereby potentially biasing the response of that group.
Response bias is not the opposite of non-response bias, but instead relates to a possible tendency of respondents to give inaccurate or untruthful answers for various reasons.

References

^ ^a ^b Fowler, Floyd (2009). Survey Research Methods (4th ed.) - SAGE Research Methods. doi:10.4135/9781452230184. ISBN 9781412958417.
^ "Participation Bias in AIDS-Related Telephone Surveys: Results From the National AIDS Behavioral Survey (NABS) Non-Response Study".
^ Deming, W. Edwards. Sample design in business research. Vol. 23. John Wiley & Sons, 1990.
^ Ann, Bowling (2014-07-01). Research methods in health : Investigating health and health services. Milton Keynes. ISBN 9780335262755. OCLC 887254158.{{cite book}}: CS1 maint: location missing publisher (link)
^ Imam, Mohamed A.; et al. (2014). "Loss to follow-up after total hip replacement: a source of bias in patient reported outcome measures and registry datasets?". HIP International. 24 (5): 465–472. doi:10.5301/hipint.5000141. PMID 24970318. S2CID 37276808.
^ Armstrong, J.S.; Overton, T. (1977). "Estimating Nonresponse Bias in Mail Surveys". Journal of Marketing Research. 14 (3): 396–402. CiteSeerX 10.1.1.36.7783. doi:10.2307/3150783. JSTOR 3150783.
^ Groves, Robert (2006). "NONRESPONSE RATES AND NONRESPONSE BIAS IN HOUSEHOLD SURVEYS". Public Opinion Quarterly. 70 (5): 646–675.
^ ^a ^b Davern, Michael (2013). "Nonresponse Rates are a Problematic Indicator of Nonresponse Bias in Survey Research". Health Services Research. 48 (3): 905–912.
^ Groves, Robert; Peytchava, Emilia (2008). "The Impact of Nonresponse Rates on Nonresponse Bias: A Meta-Analysis". Public Opinion Quarterly. 72 (2): 167–189.
^ Livingston, Edward; Wislar, Joseph (2012). "Minimum Response Rates for Survey Research". JAMA Surgery. 147 (2): 110.
^ Meterko, Mark; Restuccia, Joseph; Stolzmann, Kelly; Mohr, David; Brennan, Caitlin; Glasgow, Justin; Kaboli, Peter (2015). "Response Rates, Nonresponse Bias, and Data Quality: Results from a National Survey of Senior Healthcare Leaders". Public Opinion Quarterly. 79 (1): 130–144.

v t e Biases
Cognitive biases	Acquiescence Ambiguity Affinity Anchoring Attentional Attribution Actor–observer Correspondence Authority Automation Availability Mean world Belief Blind spot Choice-supportive Commitment Confirmation Selective perception Compassion fade Congruence Cultural Declinism Distinction Dunning–Kruger Egocentric Curse of knowledge Emotional Extrinsic incentives Fading affect Framing Frequency Frog pond effect Halo effect Hindsight Horn effect Hostile attribution Impact Implicit In-group Illusion of transparency Mean world syndrome Mere-exposure effect Narrative Negativity Normalcy Omission Optimism Out-group homogeneity Outcome Overton window Precision Present Pro-innovation Proximity Response Restraint Self-serving Social comparison Social influence bias Spotlight Status quo Substitution Time-saving Trait ascription Turkey illusion von Restorff effect Zero-risk In animals
Statistical biases	Estimator Forecast Healthy user Information Psychological Lead time Length time Non-response Observer Omitted-variable Participation Recall Sampling Selection Self-selection Social desirability Spectrum Survivorship Systematic error Systemic Verification Wet
Other biases	Academic Basking in reflected glory Déformation professionnelle Funding FUTON Inductive Infrastructure Inherent In education Liking gap Media False balance Vietnam War Norway South Asia Sweden United States Arab–Israeli conflict Ukraine Net Political bias Publication Reporting White hat
Bias reduction	Cognitive bias mitigation Debiasing Heuristics in judgment and decision-making
Lists: General Memory