Survey response rates and nonresponse bias, explained

A survey response rate is the share of eligible sampled cases that complete an interview. It tells you how much of the sample took part, not whether the people who took part resemble those who did not. That second question is nonresponse bias, and it has to be assessed estimate by estimate, with evidence beyond the rate itself.

Participation has fallen far. Pew Research Center reported a response rate of 36% in 1997 for its standard US telephone survey and 6% in 2018 for its typical telephone surveys. Yet its estimates stayed close to high-response benchmarks on most measures and were far off on a few. In Pew’s reading, a low rate signals a higher risk of error but is an unreliable measure of accuracy.

How to calculate a survey response rate

The reference standard is the American Association for Public Opinion Research’s Standard Definitions, tenth edition. It defines a response rate as complete interviews divided by eligible units in the sample, and builds every rate from each case’s final disposition:

• I: complete interview
• P: partial interview
• R: refusal or break-off
• NC: non-contact with an eligible case
• O: other eligible non-interview, such as a language barrier
• UH, UR, UO: unknown eligibility, because household status (UH), respondent eligibility (UR) or something else (UO) was never established
• e: the estimated share of unknown cases that are eligible

Ineligible cases, such as disconnected or business numbers in a random-digit-dial sample, are left out of every denominator.

Two choices move the result. One is what counts as a partial, which must be defined before fieldwork; AAPOR’s examples treat 50–80% of applicable questions answered as a partial and more as a complete. The other is e, whose basis must be stated and which, AAPOR insists, must not be chosen to boost the rate.

A worked example

Suppose a random-digit-dial study ends with 5,550 sampled numbers: 300 completes, 20 partials, 400 refusals and break-offs, 900 non-contacts, 80 other non-interviews, 2,550 ineligible and 1,300 unresolved. Of the 4,250 resolved numbers, 40% were eligible, so e = 0.4. Estimating e from resolved numbers is the approach the CDC’s Behavioral Risk Factor Surveillance System (BRFSS) describes for its telephone response rates.

• RR1 = 300 ÷ 3,000 = 10.0%
• RR3 = 300 ÷ (1,700 + 0.4 × 1,300) = 300 ÷ 2,220 = 13.5%
• RR4 = 320 ÷ 2,220 = 14.4%
• RR6 = 320 ÷ 1,700 = 18.8%
• CON1 = 800 ÷ 3,000 = 26.7%
• COOP1 = 300 ÷ 800 = 37.5%

The same fieldwork supports a “response rate” from 10% to 19%, which is why AAPOR deems a bare “the response rate is X” unacceptable. Contact and cooperation rates locate the loss: about three-quarters of potentially eligible cases were never reached, but more than a third of those reached took part.

Count cases, not call attempts

Every rate counts sampled cases, each with one final disposition. AAPOR’s rules collapse attempt-level outcomes, such as callbacks and broken appointments, into the most informative final code for the case. If the example study made 15,000 call attempts, it achieved 2 completes per 100 dials: a measure of productivity and cost, not participation, so report it separately.

What is a good survey response rate?

No threshold makes a response rate good. AAPOR’s council cautions that no single number reflects a survey’s total quality, and AAPOR’s guidance on response rates notes that in some cases the least biased results have come from surveys with less than optimal response rates. Published figures still help set expectations, if read with their definitions.

These figures are not comparable: populations, samples, protocols and definitions differ. The online average comes from a meta-analysis by Wu, Zhao and Fils-Aime. Across 114 experimental comparisons, Daikeler, Bošnjak and Lozar Manfreda found web response rates 12 percentage points lower than other modes.

One familiar threshold is a trigger, not a target: the US Office of Management and Budget’s 2006 standards for federal statistical surveys call for a nonresponse bias analysis when the unit response rate is below 80%. A good response rate is one reported under a named definition, achieved with a protocol suited to the population, and backed by evidence about bias in the key estimates.

What is nonresponse bias?

Nonresponse bias is the error that arises when people who respond differ from those who do not on what a survey measures. The OMB standards give the bias of a respondent mean as:

Bias = (nonrespondents ÷ sample size) × (respondent mean − nonrespondent mean)

That is, the nonresponse rate multiplied by the difference between respondents and nonrespondents. A high nonresponse rate makes room for bias; the difference decides whether it appears, and it varies from one variable to the next.

Setting sampling variation aside, suppose 10% of a sample responds, and an attribute is present among 30% of respondents and 40% of nonrespondents. The full-sample figure is 39% (0.1 × 30 + 0.9 × 40), so the respondent estimate of 30% is nine points low: 0.9 × (30 − 40) = −9. If both groups were at 30%, the same response rate would produce no bias, while a 70% response rate with a 30-point gap would produce the same nine-point bias.

Why the rate is a weak guide to bias

Groves and Peytcheva’s 2008 meta-analysis drew on 59 studies designed to estimate nonresponse bias, to explore when nonresponse rates relate to bias and when they do not. Reviewing that literature, methodologists at Statistics Netherlands concluded that response rates alone are not good indicators of nonresponse bias.

Experiments agree. In Keeter and colleagues’ 2000 study, a five-day telephone survey reached a 36.0% response rate and an eight-week rigorous version 60.6%, yet across 91 comparisons the average difference was about 2 percentage points and none exceeded 9.

Pew’s 2017 benchmarking study shows the other side. Its telephone estimates on 13 demographic, lifestyle and health questions differed from high-response benchmarks by 2.7 percentage points on average in 2016. But on working with neighbours, the telephone survey showed 46% against 8% in the Current Population Survey. Pew concluded that response rate is an unreliable indicator of bias.

How to check for nonresponse bias

Bias is specific to each estimate, so start from the estimates the study most needs and use whatever evidence exists about the people who did not respond.

• Compare respondents with nonrespondents on frame data. Variables known for every case, such as region or customer tenure, show where participation differs; the OMB standards suggest subgroup comparisons and a multivariate model of response.
• Compare with external benchmarks. Census figures or large government surveys can test shared measures, though wording, reference period, coverage and mode also produce gaps, and benchmarks have errors of their own.
• Analyse level of effort. Compare early respondents with those who needed many calls or refusal conversion, as Keeter and colleagues did in 2006 with their hardest-to-reach respondents. Late respondents may still differ from people who never respond.
• Follow up a subsample of nonrespondents. A second phase with more intensive methods, a design AAPOR’s Standard Definitions covers, gives direct evidence about people the main protocol missed.
• Summarise representativeness with an R-indicator. Schouten, Cobben and Bethlehem proposed R = 1 − 2S, where S is the standard deviation of response propensities estimated from auxiliary variables; R equals 1 when propensities are equal. It is only as informative as those variables.

What weighting can and can’t fix

Weighting can make a sample match known totals for age, education or region. That shows the adjustment worked as specified, not that outcome estimates are unbiased. Groves (2006) notes that post-survey adjustment for nonresponse requires auxiliary variables, so its value depends on the design and on the quality of those variables.

Weighting helps only when the variables explain both who responds and the outcome. Pew found that for volunteering, standard weighting falls short and leaves estimates biased; weighting volunteers down would reduce that bias without necessarily improving other estimates.

Inspect the weights too. A few very large weights make estimates unstable, and trimming trades instability for some reintroduced imbalance, so document the rule and test alternatives. A margin of error does not include residual nonresponse bias.

How to increase phone survey response rates

Telephone recruitment now starts against call screening. In a July 2020 Pew survey, 19% of US adults said they generally answered cellphone calls from unknown numbers; 67% let such calls go unanswered but checked any voicemail. Pew has also reported spam-flagging tools wrongly flagging survey calls. Test each lever below against your current protocol, and check whether it changes who responds, not only how many.

Send advance notice

A meta-analysis by de Leeuw and colleagues found that advance letters raised telephone response rates (RR1) from 58% to 66% on average, and cooperation rates (COOP1) from 64% to 75%. Letters need an address, which list and address-based samples often have and random mobile samples usually lack. Whether a text message works as well needs testing.

Spread calls across days and times

In a large national telephone survey of adult men, Weeks, Kulka and Pierson found first calls much more likely to be answered and to yield an interview on weekday evenings and weekends than on weekday daytimes. That is landline-era evidence, so check your own call records. AAPOR’s best practices recommend multiple contacts at different times of day, within the calling hours each market permits; our guide to polling in Australia and New Zealand sets out the Australian research-call rules.

Allow enough attempts and time

Effort raises response. In Pew’s 2012 study, a high-effort survey with a longer field period, monetary incentives, letters to people who had declined and interviewers experienced at persuading reluctant respondents achieved 22% against the standard survey’s 9%. Yet 28 of 40 comparisons differed by two percentage points or less. Extra attempts are worth most when they bring in people whose views are otherwise missing.

Offer an incentive

Analysing 39 incentive experiments in telephone and face-to-face surveys, Singer and colleagues concluded that incentives raised response rates in both modes, though the effects were relatively modest once other variables were controlled. Cash worked better than gifts, a promised incentive still beat none, and incentives did least where response without one was already high.

Identify the caller and get the introduction right

In a randomised Gallup experiment, Callegaro, McCutcheon and Ludwig found initial evidence that a caller ID name works like a “condensed survey research organization business card”. Interviewers with low refusal rates were judged to speak with more pitch variation, more loudly, faster and more clearly (Oksenberg, Coleman and Cannell). Across 1,380 recorded invitations, agreement was highest when interviewers were moderately disfluent, neither robotic nor so disfluent as to seem incompetent (Conrad and colleagues).

For AI voice interviewers, voice and pacing are design choices worth testing. Because break-offs count as refusals, questions written for the ear protect the response rate too.

Add a second mode

AAPOR lists higher response rates and lower nonresponse error among the reasons for multi-mode designs. But mode also changes how people answer: see CATI vs CAWI vs CAPI, and the IVR mode experiment for how completion and break-off rates differed. Record each case’s mode so the effects can be separated.

Checklist for reporting response rates

• Name the rate, such as “AAPOR RR3”, not “the response rate”.
• Publish the final disposition of every sampled case.
• State how e was estimated, and why.
• Define complete, partial and break-off before fieldwork.
• Report contact, cooperation and refusal rates alongside.
• Report effort separately: attempts per case, field period, calling windows, advance contact and incentives.
• Say which key estimates were checked for bias, and against what.
• Describe the weighting variables and any trimming, and note that the margin of error excludes nonresponse bias.

catilyst runs telephone interviews with AI voice interviewers in parallel, following the researcher’s questionnaire and routing, with calling windows and retry rules set per study. That changes how calls are made, not whether the achieved sample is representative, so the checks above still apply. See CATI software or the market research use case.