AI voice interviewers for surveys: what the evidence shows

An AI voice interviewer is a system that conducts a spoken survey interview by telephone. It asks the questions in a researcher’s questionnaire, recognises what the respondent says, codes each answer to the question’s categories and follows the routing to the next item. In effect, it takes the interviewer’s seat in computer-assisted telephone interviewing (CATI).

“AI survey” is a loose label, covering tools that draft questionnaires, code open-ended answers, run chatbot surveys or lead qualitative interviews. This note is about the interview itself, and mainly about quantitative telephone interviewing, where every answer becomes a data point in an estimate.

A fluent conversation can still produce a poor record: a qualified answer put in the wrong category, a pause mistaken for the end of a turn, an explanation that changes what the question means. A demonstration shows how a system speaks; judging how it interviews takes a study.

AI interviewers, IVR, chatbots and AI-moderated interviews

These four are often discussed together, but they differ in channel, freedom and output, so evidence about one does not transfer automatically to another.

IVR (interactive voice response) plays recorded or synthesised questions and takes keypad presses or short fixed replies, with no conversation; our comparison of IVR surveys and AI CATI covers the trade-offs. A preprint position paper by Tirumala and colleagues, from the team behind one of the telephone pilots below, argues that AI interviewers already exceed IVR’s capabilities but flags transcription errors and uneven follow-up quality.

Conversational surveys are text dialogues. In a field study of about 600 people, Xiao and colleagues found that a chatbot asking open-ended questions and probing answers drew more engaged and more informative responses than a conventional online survey.

AI-moderated interviews are qualitative: the system follows a topic guide and writes its own follow-up questions. That freedom is what a quantitative interview has to constrain. AAPOR’s 2026 report on responsible AI in survey research grades the risk by the agency an AI interviewer is given: relatively low for switching modality, moderate for translating or paraphrasing, and high when it can generate questions, alter wording or influence survey logic.

What the research shows so far

The evidence is early. Only two of the studies below involve telephone interviews; both are preprints, and both involved the systems’ developers.

Together, these studies show that AI interviewers can administer questionnaires and elicit usable answers. None randomly assigns telephone respondents to AI or human interviewers on the same questionnaire, the comparison needed before replacing human CATI. And the text studies cannot test speech recognition, interruptions or timing, where telephone interviewing adds new error.

In the SSRS pilot, completion improved between waves, but the software, questionnaire and calling schedule all changed. The panellists had chosen telephone interviewing, had not opted out of automated surveys and were incentivised, so the results do not describe a fresh telephone sample.

Follow-up probing has been uneven. The Peru authors report that when an answer was brief or vague, the AI usually accepted it and moved on. In Wuttke and colleagues’ study, the AI missed follow-ups it had handled in internal tests, which the authors take as a reminder that minor prompt changes can have unintended effects.

Where errors enter an AI interview

In conventional CATI, a person hears and interprets the answer and the software applies the routing. An AI interviewer does every step, and each can go wrong:

• Recognition. A missed “not” or a misheard number changes the answer.
• Turn-taking. A pause mid-answer is taken as the end of it.
• Interpretation. A qualified answer is forced into an option it does not fit.
• Routing. An earlier error sends the interview down the wrong path.

Follow one answer through the system

Take a hypothetical question about employment in the previous seven days. A respondent says, “I normally work, but I was on unpaid leave all last week.” Hearing only the first clause is a recognition or turn-taking error, and applying the wrong reference period is an interpretation error. Asking a question meant for a different employment category then carries the error through the interview.

An audit should compare the audio, transcript, code and route, because a tidy transcript does not show that the code was justified. Test corrections, refusals, uncertain answers and clarification requests, with the expected action written down first. AAPOR’s report adds that rules such as not mapping “most of the time” to “often” need rigorous testing to show they hold.

Recognition is not equal across speakers

In a 2020 evaluation of speech recognition systems from Amazon, Apple, Google, IBM and Microsoft, Koenecke and colleagues found average word error rates of 35% for Black speakers and 19% for white speakers. Those figures describe the systems tested then, not any current product. They are still a reason to measure recognition and coding error by subgroup with the deployed system, as AAPOR suggests for different voices and accents.

Sensitive questions and social desirability

It is well established that respondents give fewer socially desirable answers when no interviewer is involved. In a randomised experiment checked against university records, Kreuter, Presser and Tourangeau found that the web produced more, and more accurate, reporting of sensitive information than CATI, with IVR in between. Our note on IVR surveys charts the results.

Where an AI interviewer falls on that scale is untested: it has no human listener, but it responds like an interviewer. In the SSRS pilot’s small follow-up, most respondents said they were at least as comfortable discussing topics such as personal finances, crime and violence with the AI as with a human. That is a stated preference, not a measure of accuracy.

AI CATI aims to combine an interviewer’s responsiveness with less social-desirability pressure. Treat that as a design goal, and compare core sensitive measures between modes before relying on them.

How to evaluate an AI interviewer before you field

Whether you are comparing AI survey tools or piloting one, start with the decision. Replacing interviewers on a tracker needs evidence that estimates stay continuous; adding an AI callback option needs evidence about who it brings in.

Randomise, and analyse every assigned case

For a replacement decision, randomly assign sampled cases to human or AI administration before contact, keeping the questionnaire, sample source, calling windows, attempt limits and incentives comparable. A change of interviewer is a change of mode in miniature, with the risks to a series set out in our comparison of CATI, CAWI and CAPI.

Analyse all assigned cases. If the AI introduction deters a particular group, comparing completed interviews alone compares different people and mixes participation with measurement, and weighting cannot be assumed to repair it. Agreement with human CATI does not establish accuracy; use external records where they exist.

Set equivalence bounds in advance

Agree beforehand which differences would change the decision, such as a key prevalence estimate or coding accuracy on an eligibility item. AAPOR’s report likewise says thresholds should be set before deployment. As Lakens explains in his primer on equivalence testing, a nonsignificant difference does not show equivalence: set bounds at the smallest difference that matters, and plan enough precision to test them.

Sample calls for human review

Review a random sample of interviews as well as flagged calls, since flags cannot reveal failures the system has not learned to detect. Have reviewers code independently against the instrument and report denominators: five routing failures in 100 opportunities says more than five failures somewhere in an archive. AAPOR cautions against evaluation that is “LLMs all the way down”.

What to measure

Disclosure, consent and regulation

An AI voice does not escape the rules for automated calls. In February 2024 the US Federal Communications Commission confirmed that the Telephone Consumer Protection Act’s restrictions on “artificial or prerecorded voice” calls cover AI-generated voices. Such calls need prior express consent absent an emergency purpose or exemption, and must identify the entity responsible for the call.

In Australia, the 2017 industry standard for telemarketing and research calls allows research calls from 9 am to 8.30 pm on weekdays and 9 am to 5 pm on weekends, and none on national public holidays. Calls must identify the organisation and purpose. A call made solely with a synthetic voice need not give a caller’s given name, but must offer a way to request contact details.

AAPOR’s report calls for new norms for telling respondents how their data are processed, stored and used. Its minimum disclosure for AI interviewing is the task the AI performed, whether there was human oversight or validation, and the number of human respondents.

Document the version you evaluated

A finding applies to a particular configuration: a change in recognition, voice, instructions or logic can change respondent behaviour with the questionnaire unchanged. Record the model, settings, prompts, validation and oversight against fieldwork dates, and note it if a provider cannot name the model version. Decide, too, what recordings are kept, who can access them and when they are deleted.

Checklist before fielding an AI interviewer

• Decision defined. Replacement, addition or technical rehearsal.
• Agency limited. Wording and routing fixed; probes and clarifications approved.
• Written for the ear. Questions follow good phone survey question practice.
• Randomised. Cases assigned before contact, and every assigned case analysed.
• Bounds agreed. Equivalence bounds and sample size set in advance.
• Test cases written. An expected action for corrections, refusals, qualified answers and clarification requests.
• Subgroups checked. Recognition and coding accuracy across the voices the study expects to reach.
• Review sampled. Random and flagged calls, coded independently, with denominators.
• Rules confirmed. Consent, disclosure, calling hours and data handling for each market.
• Version recorded. The configuration evaluated, linked to fieldwork dates.

For teams planning that kind of pilot, catilyst conducts telephone interviews with AI voice interviewers calling in parallel. They follow the researcher’s questionnaire and routing, with approved probes and scripted clarification, screen against quotas during the interview, keep to set calling windows and retry rules, and return structured responses with CSV export. See how it works, the CATI software overview, or its use in academic research.