IVR surveys vs AI CATI: why conversation beats the keypad

An IVR survey, short for interactive voice response, is a telephone questionnaire with no interviewer. A recorded or synthesised voice reads each question, and the respondent answers by pressing keys on the phone keypad or by giving a short reply that a speech recogniser matches to a fixed list of options. IVR surveys run after contact-centre calls (“press 1 to 5”) and as outbound polls, sometimes called robo-polls.

IVR made telephone surveys cheap by taking the conversation out. Without an interviewer there are no wages, rosters or call centres, and every respondent hears the same recording. The price is everything an interviewer did: keeping a respondent on the line, explaining a question, following up a vague answer and coping with the ordinary disorder of a phone call.

AI CATI changes that bargain. It is computer-assisted telephone interviewing in which the interviewer is an AI voice. Software still controls the questionnaire, routing and data capture, but the respondent talks to something that listens and replies. A study keeps IVR’s scale and consistency without giving up the conversation, and, as a classic experiment shows, without putting a person back on the line.

The trade-off a classic mode experiment exposed

The clearest evidence on what an interviewer adds, and what one costs, comes from an experiment by Kreuter, Presser and Tourangeau. After a telephone screener, 1,501 University of Maryland graduates were randomly assigned to finish the questionnaire with a CATI interviewer, by IVR or on the web. Because the university held records of grades, honours and donations, the researchers could check many of the answers against the truth.

The interviewer kept people in the study. Nearly 95% of graduates assigned to human CATI completed the interview, against 61% for IVR and 57% for the web. Part of that gap was the handover: CATI respondents simply stayed on the line, while the others had to be transferred or go online. The difference persisted once people had started, though. Of the 410 graduates who began the IVR questionnaire, 90 (22%) hung up before the end, against 9 of 329 (2.7%) with an interviewer. IVR breakoffs did not rise after the sensitive questions, which suggests respondents were tiring of the format rather than balking at the subject matter.

Without an interviewer, answers were more accurate. Among graduates whose records showed at least one D or F grade, 33% failed to report it to a CATI interviewer, against 28% in IVR and 20% on the web. The same ordering held for a grade point average below 2.5, and the web had the lowest rate on every undesirable item. In the authors’ pooled model, web respondents were significantly less likely than CATI respondents to misreport in the socially desirable direction; the IVR difference was smaller and only marginally significant. For desirable facts, such as honours and donations, the differences between modes were small and inconsistent.

Read together, the two charts describe a trade-off. A person on the line kept graduates in the interview, and the absence of one produced truer answers. No mode in the experiment managed both, and the authors concluded that none was best on every source of error. That is the gap AI CATI is designed to close: a responsive conversation that keeps people talking, with no human listener whose opinion a respondent might want to manage.

The experiment did not include an AI interviewer, so where AI CATI lands on the second chart is something to measure rather than assume. The early signs are encouraging. In a separate IVR experiment, Couper, Singer and Tourangeau found few discernible differences between a recorded human voice and more or less human-sounding synthetic voices, and concluded that self-administration by IVR yielded better data on sensitive items. The pressure, in other words, seems to come from a person listening rather than from a voice that sounds like one. An AI interviewer also responds, which IVR does not, and that is the part worth testing.

Six things a conversation can do that a keypad cannot

1. Keep people on the line

Breakoffs are IVR’s best-documented weakness, and the Maryland experiment is not an outlier. A review by Tourangeau, Steiger and Wilson found breakoff rates as high as 31% across the IVR studies it examined, and Couper and colleagues cite a telephone self-interviewing study by Gribble and colleagues in which 24% of respondents broke off, against 2% of those interviewed by a person. Breakoffs are seldom random. They cluster at awkward items and among particular respondents, which turns a usability problem into a nonresponse problem.

Most of those exits follow a moment of friction: a misheard instruction, a key pressed in error, a question that does not fit, or a respondent wondering how much longer it will take. A recording cannot notice any of them. A conversational interviewer can, and can answer. The opening matters too. A recorded introduction is easy to mistake for a sales robocall, while an interviewer who can say who is calling, why and for how long has a chance to earn the interview. Telephone researchers have long tied the sound of the interviewer to cooperation, as in Oksenberg, Coleman and Cannell’s study of interviewers’ voices and refusal rates.

2. Ask the question the research needs

In an IVR survey the keypad shapes the questionnaire. An eleven-point 0–10 scale, the standard for Net Promoter Score, needs one more key than the phone has. A list longer than a handful of options has to be held in memory and matched to digits. Ages, postcodes and dollar amounts become multi-key entries ending with the hash key. “Other, please specify” is effectively unavailable.

Spoken answers remove those constraints. A respondent can say “eight”, name a brand from a long list without waiting for it to be read out as option seven, give a postcode, or describe an answer that fits none of the categories. The questionnaire can be written for the research question rather than for the twelve buttons on a phone.

3. Follow up a vague answer

IVR can only record open-ended answers as audio for transcription and coding later, and nobody can ask what a respondent meant. “It was fine” stays “it was fine”. An AI interviewer can use the probes a study approves, such as asking what would have made the experience better, and code the answer while the respondent is still on the line.

In a 1,800-participant experiment, Barari and colleagues found that chatbots probing open answers produced responses that were more detailed and informative, at a slight cost to respondent experience. They also found that respondents’ tendency to agree inflated some codes, a reminder that probes need careful wording. That study used text on the web, but the capability is the same one a telephone study needs: the follow-up question that explains a score. It is the heart of the customer experience use case.

4. Explain what a question means

Respondents often need to know where a question’s boundary lies: whether a partner who works away counts as part of the household, or whether “last week” includes today. IVR can only repeat the question. In Schober and Conrad’s experiment with professional telephone interviewers, allowing clarification improved accuracy when respondents’ circumstances mapped ambiguously onto survey concepts, at the cost of longer interviews. An AI interviewer can offer the scripted definitions a study approves, whenever they are needed, without improvising new ones.

5. Handle a real phone call

Phone calls are untidy. People correct themselves (“seven, no, six”), ask for a question to be repeated, interrupt, say they are driving, or ask who is calling and how their number was obtained. An IVR system has no response to most of this beyond a timeout. A conversational interviewer can accept the correction, repeat the item, answer with an approved script, or record that the respondent asked to be called back.

Speaking is also simply easier. On a smartphone, every keypad answer means taking the phone from the ear, finding the dialpad and returning, so each item is a small interruption. Talking keeps the respondent in the conversation, and may suit people who find small keys or on-screen dialpads hard to use.

6. Return richer, more usable data

An IVR dataset is a column of digits and a folder of audio still waiting to be coded. An AI interview can return structured answers alongside the respondent’s own words, with each code traceable to the transcript or audio behind it. Screening and eligibility questions can be asked and confirmed conversationally, and respondents can be screened against full quota cells during the interview rather than after it. The result is a dataset that explains its own numbers.

IVR and AI CATI side by side

Where IVR still has a case

For a single closed question at the end of a service call, IVR may be all that is needed. And where a study’s most sensitive items are its core measures, test how they behave in an AI interview before switching, or keep those few items self-completed. For anything longer, the conversation is the point.

What does not change

An AI voice does not escape the rules for automated calls. In the United States, the Telephone Consumer Protection Act restricts calls made with an “artificial or prerecorded voice”, and in February 2024 the Federal Communications Commission confirmed that AI-generated voices fall within that phrase. Such calls require the prior express consent of the called party unless an emergency purpose or exemption applies. In 2016, Pew Research Center observed that US pollsters dialling cellphones typically used live interviewers, while IVR pollsters combined landline samples with online ones. In Australia, the 2017 industry standard for telemarketing and research calls permits research calls between 9 am and 8.30 pm on weekdays and between 9 am and 5 pm on weekends. Consent, disclosure and calling rules need to be confirmed for each market before fieldwork.

Nor does a conversation guarantee a good record. An AI interviewer’s coding is a measurement step that needs its own audit; evaluating AI voice interviewers sets out how to test one against human CATI.

Moving a study from IVR to AI CATI

• Rewrite keypad items as questions. Replace “press 1 for…” with the question you meant to ask, and restore the scales and lists the keypad forced you to cut.
• Add the probes you were missing. Decide which answers deserve a follow-up, and write the approved wording.
• Write the clarification rules. Agree the definitions the interviewer may offer, and keep them with the questionnaire.
• Decide how to treat sensitive items. Test them in the new mode, or keep them self-completed.
• Bridge the series. If the IVR survey is a tracker, run both modes in parallel for a wave, so that a change in the numbers can be separated from the change in method.

catilyst is built for exactly this move: AI voice interviews that follow your questionnaire and return structured data, from questionnaire to analysis. Pollsters weighing IVR against live interviewing can start with political polling.