CATI vs CAWI vs CAPI: how survey mode shapes the data

Survey mode is often chosen on budget and timetable, as if it were a neutral container for the questionnaire. It is not. Whether a question is asked by a telephone interviewer, read on a screen or asked in person affects who can be reached, who takes part and how they answer.

None of the three common modes wins outright. Reviewing the mode-comparison literature, Edith de Leeuw found that when equivalent questionnaires were compared, no mode was superior on all criteria. A randomised experiment by Kreuter, Presser and Tourangeau reached the same conclusion for telephone interviewing, interactive voice response (IVR) and the web. The practical question is which errors a study can least afford.

Three modes, defined

CATI, computer-assisted telephone interviewing, is an interview by phone in which software controls the script: the interviewer reads each question and records the answer, while the program applies routing and stores the data. Our guide to CATI as a method covers it in depth.

CAWI, computer-assisted web interviewing, is self-completion in a browser, on a computer or phone. The respondent reads, sets the pace and answers without anyone listening.

CAPI, computer-assisted personal interviewing, puts an interviewer face to face with the respondent, running the questionnaire on a laptop or tablet. The Australian Bureau of Statistics notes that show cards can help personal interviews yield more detailed responses. For sensitive sections the interviewer can hand over the device for private self-completion, a combination de Leeuw describes as popular for sensitive topics.

Coverage: who can be reached

A telephone study needs numbers, and the ABS notes that people without a phone, or who are rarely available, are excluded. The web has no general-population equivalent of a number to dial. The AAPOR task force on transitions from telephone surveys notes that email recruitment is limited to list-based populations such as students or employees, web panels, studies that collect an address in a screener, and later waves of longitudinal surveys. Otherwise first contact needs another mode, such as a letter.

Face-to-face fieldwork can use area or address samples that need neither; de Leeuw describes early dual-frame designs pairing the cost savings of telephone interviewing with the coverage of area sampling.

Nonresponse: who takes part

Web surveys carry a persistent participation deficit. A meta-analysis by Daikeler, Bošnjak and Lozar Manfreda of 114 experimental comparisons found web response rates 12 percentage points lower than other modes, almost unchanged from the 11-point gap an earlier meta-analysis had found. The ABS notes that personal interviews can achieve better response rates than mail or telephone surveys.

In the Kreuter experiment, 1,501 University of Maryland graduates were randomly assigned after a telephone screener to finish by CATI, IVR or web: 94.7% completed with an interviewer, against 61.1% by IVR and 56.8% online. Part of that gap was the handover, since CATI respondents stayed on the line. Once started, the web lost only 5 of 368 to breakoff and CATI 9 of 329, while IVR lost 90 of 410, as our analysis of IVR surveys charts. A higher response rate is not, by itself, a smaller bias; that depends on who is missing.

Measurement: what respondents say

An interviewer on the line

The most pronounced mode differences involve sensitive topics. In de Leeuw’s review, interviewer modes produced more socially desirable and less consistent answers than self-completion, but more detailed responses to open questions. Face-to-face and telephone differed little.

Among Maryland graduates whose university records showed a D or F grade, 33.0% failed to report one to a CATI interviewer, against 28.3% by IVR and 19.9% on the web, and the web had the lowest rate of failing to report on every undesirable item.

Pew Research Center’s 2015 mode study randomly assigned 3,003 panel members to answer by phone or web. Across 60 questions the mean difference was 5.5 percentage points, with most of the largest gaps where social desirability could play a role. By phone, 62% were “very satisfied” with their family life, against 44% on the web. Pew cautioned that for most differences it could not say which mode was more accurate, while noting that research with known true values has generally favoured self-administration.

Interviewers still earn their place. The AAPOR task force lists what they add: noticing misunderstanding, clarifying, probing inadequate answers and motivating respondents to finish. As de Leeuw frames it, the more control respondents have, the more they disclose, but the fewer chances there are to persuade, motivate or explain.

Heard versus seen

On the telephone, respondents rely entirely on what they hear, holding the question and its options in working memory. The AAPOR report notes that this encourages top-of-the-head answers and is especially demanding for older and less educated respondents. De Leeuw links aural presentation of long lists to recency effects, where the last options heard are chosen more often, and visual presentation to primacy effects, though she describes the evidence as mixed.

Rating scales show a more persistent pattern: ordinal opinion items draw more extreme positive answers in interviewer modes, especially by telephone. In one study cited by AAPOR, phone respondents were about twice as likely as mail respondents to choose the most positive satisfaction rating. Proposed explanations range from response order to reluctance to give an interviewer a negative evaluation, and tests have been inconclusive. For satisfaction tracking in market research, the same scale can produce a rosier top box by phone. Telephone items have to be written for listening.

Cost and speed

The ABS describes personal interviews as very expensive, while telephone interviewing reduces costs, needing fewer interviewers and no travel, and produces more timely results. Of organisations the AAPOR task force surveyed about moving telephone surveys to self-administered or mixed modes, 13 of 19 answering said costs fell.

Mixed-mode designs, and why switches break trackers

De Leeuw separates two kinds of mix. Using a second mode only for contact, screening or reminders is a win–win, because the data are still collected in one mode. Collecting the data itself in several modes raises the question of whether answers can be combined or compared over time. Her one well-documented exception is a more private mode for a subset of sensitive questions.

When respondents choose their own mode, self-selection and mode effects are confounded and hard to correct. When a repeated survey changes mode, AAPOR warns, estimates can shift because of who responds and how they answer rather than true change, creating a break in series. Pew named the same issue in its own shift to the web: how to track trends when the mode of interview has changed.

The defences cost money: a bridge survey in the old mode measures the effect of the switch but, as AAPOR notes, means running two surveys. For panels, de Leeuw suggests interviewing a random subsample in the previous wave’s mode so mode and time effects can be separated. Election polls face this problem in public, as we discuss for Australia and New Zealand.

Where AI CATI fits

AI CATI is CATI in which the interviewer is an AI voice. Software still controls the script; the respondent still hears the questions and answers aloud; the interviewer can still repeat an item, offer an approved definition or ask a scripted follow-up. What is missing is a person listening.

AAPOR describes modes as differing on three dimensions: interviewer- or self-administered, aural or visual, and computerised or not. Moving a telephone study to the web changes the first two at once. AI CATI keeps the aural channel and conversational give-and-take, but removes the human audience. It aims to combine an interviewer’s responsiveness, the clarifying, probing and motivating AAPOR describes, with the lower social-desirability pressure that made the Maryland web answers more accurate. That is a design goal, not a demonstrated result: we are not aware of a published record-check experiment comparing an AI telephone interviewer with CATI and the web.

Adjacent evidence is suggestive. In a health-screening study by Lucas, Gratch, King and Morency, participants who believed a virtual human interviewer was automated, rather than operated by a person, reported lower fear of self-disclosure and less impression management. That was a laboratory study with a virtual human, not a telephone survey, but it isolates the mechanism AI CATI relies on.

Some things should not change. AI CATI inherits the telephone frame, the problem of getting calls answered and the limits of aural lists. Whether it softens telephone’s extreme positive ratings is an open question: if reluctance to criticise an interviewer drives the effect, it may shrink; if aural processing does, it should persist. A randomised split against an existing mode can tell, and moving a CATI tracker to AI interviewing deserves a bridge of its own; evaluating AI voice interviewers sets out how to design one.

catilyst is built for that kind of design: AI voice interviewers calling in parallel, following the researcher’s questionnaire and routing with approved probes, screening against quotas during the interview and returning structured responses with CSV export. See the CATI software overview, how it works, or its use in academic research.

The modes side by side

Choosing deliberately

• Start with the frame. Which mode reaches the population you need to describe?
• Find the sensitive items. Decide whether they need a private mode.
• Write for the channel. Lists and scales behave differently heard and seen.
• Protect the series. Bridge before a tracker switches mode.