A telephone respondent meets a questionnaire as speech. Each question is heard once, in sequence, at the pace of the person reading it, with no show-card and nothing to look back at. The Australian Bureau of Statistics’ guide to data collection methods notes that it is often difficult to obtain answers to complex questions by telephone, whereas self-administered surveys can include explanatory notes and personal interviews can use show-cards.
Good phone survey questions are therefore written for listening memory. The principles below apply whether the interviewer is a person working from a CATI script or an AI voice, and they are why a web questionnaire cannot simply be read aloud (see CATI, CAWI and CAPI compared).
Respondents cannot see the options
On a screen, a respondent can scan a list of twelve brands and pick one. On the phone, the same list has to be held in memory while it is read. Keep read-out lists short, make the options words or short phrases rather than sentences, and consider asking a long list as an open question that the interviewer codes against a precoded frame.
Order matters too. Holbrook, Krosnick, Moore and Tourangeau analysed 548 response-order experiments in Gallup telephone surveys, all using two-option categorical questions. Significant order effects appeared in a minority of questions; when they did appear, they were predominantly recency effects, favouring the option read last. They were more common when questions were harder to comprehend, when the options were complete sentences, when the wording encouraged respondents to wait for every choice (“Which of the following…”), after many earlier questions, and among respondents with less education.
The authors’ first recommendation was that survey organisations should routinely rotate the order of response choices. Pew Research Center’s guidance on writing survey questions adds a caveat: randomisation does not eliminate order effects, but spreads them randomly across respondents. Ordinal scales are generally kept in order, because the order helps respondents place themselves, though it can be reversed for some respondents.
Branch bipolar scales instead of reading them out
A seven-point scale from “extremely dissatisfied” to “extremely satisfied” is easy to display and tiring to hear. The telephone alternative is to branch: ask about direction first, then extremity. Krosnick and Berent reported eight experiments, using telephone, face-to-face and self-administered questionnaires, in which fully labelled branching measures were more reliable than partially labelled non-branching ones. The difference appeared to reflect both the verbal labels and the branching.
Malhotra, Krosnick and Thomas then compared ways of building the branches, using four national surveys conducted face to face, by telephone and online. Their results support three rules: after a three-option question about direction, respondents choosing either end should be offered three levels of extremity; respondents choosing a precisely labelled midpoint should not be asked which way they lean; and seven-point bipolar scales measure more accurately than three-, five- or nine-point versions.
“Overall, are you satisfied, dissatisfied, or neither satisfied nor dissatisfied with the service?” If satisfied: “Would you say you are extremely, moderately or slightly satisfied?”
Branching is designed for bipolar constructs. Unipolar measures, such as how often or how much, still need verbal labels short enough to remember.
Read the options the same way every time
On paper, every respondent sees the same options. On the phone, consistency depends on the script. Specify whether options are always read or only when needed, whether “don’t know” is offered or accepted only when volunteered, and what happens when a respondent asks to hear the question again: the full question and every option, rather than a paraphrase. A respondent who hears “satisfied or dissatisfied” has answered a different question from one who also heard the neutral option. Where options rotate or a scale is reversed, store the order each respondent heard.
One idea, one period, defined terms
Pew’s example of a double-barrelled question asks about confidence in a president to handle “domestic and foreign policy”. Such questions, Pew notes, are difficult to answer and produce responses that are difficult to interpret. A listener cannot reread the stem, so split them.
Put reference periods and definitions first, before the respondent starts to answer. “Thinking only about the past four weeks, how many times have you seen a GP?” works better by ear than a question that ends with “…in the past four weeks, not counting telehealth calls”. Write approved definitions for likely queries into the script now, not during fieldwork.
Ask about the thing, not about agreement
Agree/disagree batteries are quick to write and quick to read. Saris, Revilla, Krosnick and Shaeffer point to more than five decades of research documenting acquiescence bias in these items. In experiments combining random assignment with a multitrait-multimethod design in representative surveys, they found that agree/disagree questions produced data of much lower quality than comparable questions with item-specific response options.
Pew reports that less educated and less informed respondents are more inclined to agree with statements, that this is more pronounced when an interviewer is present, and recommends offering a choice between alternative statements instead. So rather than “Do you agree or disagree that the website was easy to use?”, ask “How easy or difficult was the website to use: very easy, somewhat easy, somewhat difficult or very difficult?” Whether an AI voice elicits the same deference as a person remains an empirical question; item-specific wording sidesteps it.
Open questions need approved probes
Closed lists shape answers. In Pew’s example from the 2008 US presidential election, 58% of respondents chose the economy when it was offered as one of five options, but only 35% volunteered it when the question was open-ended. Among those asked the open version, 43% gave a response that was not on the closed list. Open questions capture what the list-maker did not anticipate, provided thin answers are followed up.
That follow-up is where standardisation is won or lost. Write the permitted probes for each open item (“What else?”, “Could you tell me a little more about that?”), state how many may be used and when to stop, and avoid probes that suggest content. Capture the answer verbatim alongside any code. A keypad IVR survey can record an open answer as audio, but it cannot ask the second question.
In a web experiment with 1,800 participants, Barari and colleagues found that text chatbots probing for elaboration produced open responses that were more detailed and informative, at a slight cost to respondent experience, while respondent acquiescence slightly inflated false-positive codes. That was text rather than voice, but it shows why probes deserve the same scrutiny as questions.
The introduction is the first question
The ABS lists the difficulty of convincing respondents of the authority behind a survey and the confidentiality of results among the problems of telephone interviewing. Write the introduction to answer those doubts directly: who is calling, on whose behalf where that can be disclosed, why, how long it will take, that participation is voluntary, and how answers will be used.
In Australia, the 2017 industry standard for telemarketing and research calls requires research callers to give, as soon as the call starts, the caller’s given name (unless the call uses only a recorded or synthetic voice), the name of the calling organisation and the purpose of the call. Confirm the rules in each market. State the interview length honestly, and if the interviewer is an AI voice, say so plainly at the start.
What changes when the interviewer is an AI voice
A trained interviewer fills gaps in a script from briefings and experience. An AI interviewer does exactly what the questionnaire allows. That is a strength for standardisation and a demand on the specification: every definition it may offer, every probe it may use and every boundary it must respect has to be written down, including how to confirm an unclear answer (“I heard six. Is that right?”).
The evidence on clarification is candid about the trade-off. In Schober and Conrad’s experiment with professional telephone interviewers, allowing clarification improved accuracy when respondents’ circumstances mapped ambiguously onto survey concepts, at the cost of longer interviews. Keypad IVR can only repeat the question. A conversational interviewer, human or AI, can give the approved definition at the moment it is needed, the approach set out in our account of standardised clarification in CATI.
Delivery is part of the instrument too. In Holbrook and colleagues’ recordings, interviewers’ pausing patterns helped explain why some question types were more prone to recency effects. Listen to test calls with that in mind, and evaluate recognition and coding against the intended instrument, as described in evaluating AI voice interviewers.
A checklist for phone survey questions
- Short lists. Read-out lists are short, options are words or phrases, and long lists are asked open and coded.
- Rotation. Unordered options are rotated or randomised and the order heard is stored; ordinal scales stay in order or are reversed for a random half.
- Branching. Bipolar scales ask direction, then three levels of extremity; a precisely labelled midpoint is not followed by a lean question.
- Consistent reading. The script says which options are always read, how “don’t know” is handled and what a repeat contains.
- One idea, terms first. No double-barrelled items; reference periods and definitions come before the question.
- Item-specific wording. Agree/disagree batteries are rewritten as questions about the thing being measured.
- Approved probes. Each open item has permitted probes, a limit and verbatim capture.
- Scripted clarification. Definitions are written in advance, with a rule for questions they do not cover.
- A complete introduction. Who is calling, why, how long, voluntary participation and each market’s requirements.
- Tested by ear. The whole questionnaire has been heard aloud before fieldwork.
catilyst is built around this kind of specification. Its AI voice interviewers follow the researcher’s questionnaire and routing, with approved follow-up probes and scripted clarification, screen against quotas during the interview within set calling windows, and return structured responses with CSV export. See how it works, the CATI software overview, or its use in market research and customer experience studies.
Writing a telephone questionnaire?
Share your draft questionnaire with the Voxworks team to talk through scales, probes and clarification rules before fieldwork begins.