Expanded and non-conforming answers in standardized survey interviews

2.1 Standardization of interaction in survey interviews

To minimize deviations from the script and thus reduce measurement error, survey questions are designed to result in so-called paradigmatic question-answer (QA) sequences, which are unexpanded sequences consisting of a question and unelaborated answer and sometimes an acknowledgment of the answer (Maynard and Schaeffer 2013), after which the interviewer can move on to the next question. What contributes to preventing QA sequences from expanding – for example, by a repair sequence – is when question recipients provide type-conforming answers. In the case of a yes/no-question, a version of yes or no is type-conforming (Raymond 2003). For wh-questions, type-conforming answers correspond to the projected answer type (Schegloff and Lerner 2009). For example, “who” projects a person and “how many” projects a number. Subsequently, Koole and Verberg (2017) have extended the concept of type-conformity to answers that repeat one of the options presented in an alternative question, like “cash or credit?”

In excerpt 1, we can see three QA sequences that are paradigmatic because all answers are type-conforming. The “how old” question (line 2) receives an “age” answer, “what education” (line 5) is answered with a school type (“MTS”), and the yes/no-interrogative (line 7) receives a yes (For transcription conventions, please see Appendix A.)^[1]

2.2 Pre-testing in survey research

To ensure paradigmatic QA sequences, survey designers pre-test questions, so interviewers can effortlessly read them out as worded, and respondents can answer by choosing one of the options provided or implied (Presser et al. 2004). Essentially, pre-testing methods are evaluative in nature and are used to determine whether questions require revision (Yan et al. 2012). Several different pre-testing methods are available, such as expert reviews (Forsyth and Lessler 1991), cognitive interviewing (Willis 2005), interviewer debriefing (Blair and Presser 1992), behavior coding (Fowler 2011), and statistical procedures (Biemer 2004; Yan et al. 2012). These methods differ with respect to reliability, validity, and the types of problems they detect (Presser et al. 2004). A method that evaluates the effect of question-wording on the flow of the interview is behavior coding. This method quantifies deviations from QA sequences that are considered “paradigmatic.” Behavior coding provides an overview of the questions that are not read as scripted, the times respondents ask for clarification of a question, and many more possible deviations.

2.3 Deviations in the answering process

When asked a question, people sometimes do more than answer. A fairly elaborate overview of survey interviewers’ and respondents’ interactional behavior is presented in Ongena and Dijkstra (2007). In their model, respondents’ interactional behaviors are discussed in light of respondents understanding the question and cognitive problems with respect to retrieval, judgment and formatting. The options respondents have in explicit requests for clarification, and actions that clearly constitute a side-track in the conversation (i.e., comments and digressions) will not be further discussed in this article. However, what Ongena and Dijkstra (2007) call “implicit requests” partly overlaps with deviations described in this article. For example, rather than providing just an answer, respondents can indicate that their answer has a degree of imprecision or uncertainty. Schaeffer et al. (1993) describe these markers and how interviewers accept responses with low levels of uncertainty, while probing those presented as more uncertain. In addition to the answer’s level of uncertainty, interviewers, especially in Computer assisted telephone interviews (CATI) surveys, can be motivated by time constraints to accept answers with uncertainty or imprecision markers to retain a fast pace of interviewing (Holbrook et al. 2003).

Another example of a deviation in the answering process is not providing a type-conforming answer. According to Ongena and Dijkstra (2010), responses that are not type-conforming are the most important cause for survey interviewers to deviate from their script. In Conversation Analysis (CA), type-conforming responses are defined as “responses that conform to the constraints embodied in the grammatical form” of the question (Raymond 2003: 946). Depending on the question, a type-conforming response can be a yes or no, a number, or a category such as a brand name (Koole and Verberg 2017; Schegloff and Lerner 2009).

Non-type-conforming answers can take the form of “reportings” (Drew 1984), designed to leave it to the recipient to gather the upshot. Such reportings allow respondents to defer judgment, letting the interviewer assign an answer category (Moore 2004; Schaeffer and Maynard 2002).

Finally, responses can be prefaced before or elaborated after the type-conforming answer desired by survey researchers. According to Raymond (2000), a preface to a type-conforming answer attempts to modify the terms of the question and, consequently, the action performed by the following type-conforming response. In other words, the respondent adjusts the question before answering it (Clayman 1993, 2001]; Raymond 2000). Elaborations placed after type-conforming answers, however, accomplish a “modification of the stance taken by that type-conforming token” (Raymond 2000: 207).

2.4 Survey data validity

It is unclear how deviations in the question-answering process relate to the validity of survey interview data. Validity refers to the agreement of observed variables (responses given in an interview) with latent variables (the variable or construct of interest). Due to measurement error – for instance, because of respondents’ misunderstanding of questions – observed variables are not always the same as latent variables, and systematic differences lower validity, whereas coincidental differences lower reliability (Alwin 2010). Findings on the relation between deviations and data quality are complex. For example, interruptions may decrease the reliability of responses for questions followed by a definition but have no effects for questions preceded by a definition (Schaeffer and Dykema 2011). Such complications necessitate further development of interaction analysis as a method to improve question-wording. In this article, we have used CA to study deviations in the question-answering process in the context of the interaction in which they occur, thus reformulating existing interaction codes into more meaningful categories.

3.1 Data

For the qualitative analysis in this article, we used three corpora of telephone survey interviews carried out by leading Dutch survey centers, two of which were collected by Hanneke Houtkoop-Steenstra. For the quantitative study, we have used a fourth dataset. For all data, informed consent was obtained, and ethics approval was granted prior to conducting and recording the interviews. Table 1 provides an overview of the surveys included.

Computer-Assisted Telephone Interviews (CATI) in survey 1 used standardized interviewing techniques (see Houtkoop-Steenstra 2000: 15). Houtkoop-Steenstra’s second corpus was a nationwide survey recorded for a quantitative analysis of how different introductions affect response rates (see Houtkoop-Steenstra and Van den Bergh 2000). The third corpus is a collection of interviews carried out in 2004. These interviews were not designed for other research purposes than the survey itself and aimed to study respondents’ satisfaction with a magazine subscription.

For the quantitative analysis (Section 5), we turned to a fourth corpus, collected in 2004 as part of a question-wording experiment and since reused for several quantitative projects involving behavior coding (Ongena and Dijkstra 2010). As most questions were derived from existing, frequently used questionnaires, the questionnaire can be considered representative of the average survey. However, the interviewer training emphasized handling non-conforming answers. Half of the interviewers were instructed to always probe with a full list of response alternatives, whereas the other half were allowed to probe with only a selection of alternatives based on the respondent’s answer. The interviewers were given examples of non-conforming answers and how to probe adequately. The 610 completed survey interviews resulted in a dataset of 25,670 QA sequences with 140,117 utterances.

We will display excerpts of the interviews transcribed according to the Conversation Analysis conventions developed by Gail Jefferson (2004) (see Appendix A for an overview of the symbols used). The original data are in Dutch and each fragment is accompanied by an English translation.

3.2 Methods of analysis

For the qualitative research, we used the methodology of Conversation Analysis (CA) (Sidnell and Stivers 2013; see Houtkoop-Steenstra 2000 for a detailed account of CA research of survey interaction). The goal of Conversation Analysis is “the description and explication of the competencies that ordinary speakers use and rely on in participating in intelligible, socially organized interaction” (Heritage and Atkinson 1984: 1). We used CA to explore what respondents accomplish by deviating from or expanding on type-conforming answers and how these turns are understood and treated by the interviewers. The examples in this article are selected from our findings to illustrate patterns found throughout our data; online supplement 2 contains more excerpts for each finding.

For the quantitative study, the transcribed recordings of all 610 interviews of the CATI Survey 4 dataset were coded employing Dijkstra’s (2002) coding scheme, using the Sequence Viewer Program (www.sequenceviewer.nl). Dijkstra’s (2002) coding scheme involves a multivariate coding scheme. In the scheme, every utterance in the conversation is evaluated on a set of variables, each with different values. The combination of values for an utterance then yields a code string that is a meaningful description of that utterance. For instance, the code string RA0M means that the respondent (R) gives an answer (A) that is directly related to the question (0) but does not match the pre-specified response options (M), whereas the code string IQ0A means that the interviewer (I) asks a question (Q) that is directly related to the question as given in the questionnaire (0), and is posed adequately (A). The reliability of the initial coding was assessed by coding a random sample of 10 % of all QA sequences (Cohen’s Kappa = 0.81). For this study, the original coding was reviewed by a research assistant. More details on this review procedure are given in Section 5.

In what follows, we first explore respondents’ expanded and non-conforming responses using Conversation Analysis, clarifying what is accomplished by such responses. The answers fall on a continuum, ranging from minimally expanded answers, to more drastically expanded ones that still contain a type-conforming response, to turns that omit a type-conforming response. Then, our quantitative analysis will show how likely interviewers and respondents are to accept or initiate repair on each of these types of expanded and non-conforming responses.

4.1 Serial extras

We have called the first type of turn expansions discussed here “serial extras.” These extras signal the answer’s relation to previous answers and thus its position in a series. In survey interviews, questions often come in coherent series, with questions about related topics grouped together. When such related questions occur, the respondents sometimes display sensitivity to the position of their answer within the list.

In the following example, we come across a related block of questions with one parent question. The respondent is asked whether they read certain magazines (lines 1–2), projecting a list of related questions. Although the overarching question (“which […] magazines do you read”) is a wh-question, the subsequent list of magazines makes acceptance or rejection relevant. Yes and no are the most straightforward way of accomplishing this task, making those two options type-conforming responses.

In this excerpt, an exciting pattern unfolds. The respondent signals similarity between consecutive items by responding first with a simple “no” (arrows “a”) and next with “also not” (arrows “b”).

In lines 18 and 24, the respondent replies “no” to whether they read a particular type of magazine. Their response consists of a type-conforming answer and nothing else. In both cases, the next question receives another negative reply, but this time the respondent expands it (“no also not” in line 21, and “also not” in line 26), signaling sensitivity to the sequential position of this particular item. With “also,” the respondent displays awareness that the answer is part of a list and that they have given the same answer more than once. Because they thus treat their answer as part of a series, we refer to this a “serial extra.” The serial extra avoids the impression that they are answering on automatic pilot and instead marks each answer as genuine.

The serial extra does not influence the way this sequence runs off. The interviewer proceeds by asking the next question, implicitly accepting the answer. This reaction is no different from those in unelaborated, paradigmatic instances: a short pause and the subsequent delivery of the next item with a drop-rise question contour. In other words, the paradigmatic QA sequence is left intact.

Since the interviewer’s behavior does not change due to the serial extras, we can assert that the standardization of the interview is not endangered by this type of turn expansion. Our quantitative study reported below enhances the reliability of this finding.

4.2 Uncertainty markers

Another type of answer expansion is the imprecision marker or uncertainty marker (see also Schaeffer et al. 1993). Like the serial extras discussed above, uncertainty markers are often small and appear to have little impact on the QA sequence. Respondents sometimes design their answers as guesses or approximations by adding items like “I think” or “just about” to their answers.

The following excerpt takes place after the respondent has indicated which newspapers they sometimes read. The respondent now needs to answer how many of the latest six issues they have read, so a type-conforming answer consists of a number between zero and six.

In lines 3 and 7, the respondent marks a slight lack of precision in answering by saying “about.” The answers are designed to convey that they make something of a guess at how many newspapers they have read. Paradoxically, this imprecise design carefully calibrates the quality of information provided to the interviewer’s needs in conducting the survey.

Despite the display of imprecision, the answers are treated by both participants as final. Following the answer, the interviewer takes a moment to type, and the respondent could, but does not, use this gap for repair, letting the answer stand. Similarly, the interviewer treats the level of certainty as acceptable for this question. After the response, they can be heard to type information into their computer (see Komter 2002 on how typing after an answer signifies acceptance), and in lines 5 and 11 they accept the prior turn by moving on to the next question. These uncertainty markers do not cause the interviewer to deviate from the script. While the respondent’s answers may be less precise than what the survey designers had in mind, the research instrument itself remains standardized.

4.3 Prefaces before and elaborations after type-conforming answers: working with the terms of the question

Just like in the previous sections, the turn expansions discussed in this section occur in addition to a type-conforming answer. The respondent chooses one of the answer categories and adds something before or after the answer. Although these expansions are not as small as serial extras or uncertainty markers, it will become clear that they do not necessarily require the interviewer to stray from the script. In fact, some of them are designed by the respondent to make their answer even more precise than the type-conforming answer alone.

4.4 Responses that do not contain a type-conforming answer

Now, we look at sequences in which respondents produce an answer to the question without supplying a type-conforming response in the first turn after the question. Interviewers accept some of these non-conforming answers, while others are probed, forcing the interviewer to go off-script. These non-conforming responses are most problematic for standardization, as they do not contain a type-conforming component that can be entered as an answer. Instead, interviewers need to interfere one way or another: either they interpret the response and enter an answer in the computer without expanding the sequence, or they expand the QA sequence by initiating repair.

Some non-conforming answers take the shape of reportings: descriptions of behavior or attitudes. By describing, respondents avoid committing to one of the answer options because they either are not sure which applies to their situation or resist the choices offered. We start by discussing a case where the respondent employs such a reporting to defer judgment, allowing the interviewer to assign one of the answer categories.

4.5 Summary of qualitative analysis

Our qualitative analysis shows that departures from standardization vary, and different expanded and non-conforming responses have a different impact on the interaction. We have shown type-conforming answers accompanied by serial extras, uncertainty marks, and prefaces before and elaborations after type-conforming answers, as well as responses without a type-conforming answer. We also found that type-conforming responses with expansions are more compatible with the paradigmatic QA sequence than non-conforming responses. Table 2 lists the category, excerpt number, short examples, (interactional) purpose of the action, and whether the interviewer accepts the answer.

5.1 Results

The first column of Table 3 shows the frequency of occurrence of the six types of answer attempts. Non-conforming answers are the most frequent; as a collapsed category, they occur in 16 percent (7.1 + 8.9) of all QA sequences (see second column) and comprise 63.6 percent (28.6 + 35.0) of all expansion-utterances. The next most frequent are uncertainty markers, post-elaborations, and prefaces (about 3 percent each), whereas serial extras are least frequent (1 percent). In 19,717 QA sequences (75 %), none of the turn expansions occur.

Exploratory analysis shows three different major outcomes of expansions: interviewer accepts (by means of a short acknowledgment or asking the next question); interviewer does not accept (by means of a probe, repeating the question or the response alternatives); or respondent self-repairs (i.e., restates his answer).

The type of turn expansion and subsequent outcome are significantly related (χ² (10) = 1899.28, p < 0.01). After a serial extra, the interviewer is much more likely to accept the answer (90.7 %) than after an uncertainty marker (68.5 %), an answer with post-elaboration (56.8 %), a prefaced answer (73.7 %), a non-conforming answer that does not require probing (25.9 %), or a non-conforming answer that does require probing (10.8 %). The analysis shows that, except for non-conforming answers, the interviewer accepts expansions in more than half of the cases. Since interviewers are trained to probe after non-conforming answers that require probing, it is no surprise that in 76.2 % of the cases, these are followed by interviewer probing. However, these types of turn expansions are also often self-repaired (13.0 %), leaving just 10.8 % as accepted. Serial extras are least likely to be followed by interviewer probing (6.3 %) or respondent repair (3 %), and similar results are found for prefaced type-conforming answers.