Do we know what we are asking? Individual and group cognitive interviews¹

1 Introduction

There is a relatively long tradition of using questionnaires in social psychology in spite of persistent criticism that questionnaire scales and items are unclear, vague and open to various interpretations (see e.g. Furnham & Steele, 1993 for a critique of questionnaires measuring locus of control, Gutek, Murphy, & Douma, 2004 for a critique of the Sexual Experiences Questionnaire (SEQ) and Gray and Durrheim, 2013 for a critique of the Right-Wing Authoritarianism scale). Questionnaires that have not been standardized but created ad hoc have serious limitations. The paper deals with two instruments for measuring prejudice against stigmatized minorities: the SCM (Stereotype Content Model, Fiske et al., 2002) and INTERMIN (Interventions for Reducing Prejudice against Stigmatized Minorities, for more details, see Lášticová & Findor, 2016, this issue). These will be used to measure the effectiveness of direct and indirect contact interventions designed to reduce ethnic prejudice.^[2]

Since there are no Slovak versions of the standardized instruments used to measure prejudice, which would allow for international comparison and testing of the impact of this kind of intervention, our aim was firstly to create a comprehensive questionnaire (INTERMIN) covering the areas most relevant to contact research (see Lášticová & Findor, 2016, this issue), and secondly to adapt the Stereotype Content Model for use in Slovakia (SCM, Fiske et al., 2002). The first versions of these two instruments were than verified in a quantitative pilot study, which also included qualitative cognitive interviews about the instruments. Standard statistical analyses of scale reliability and factor structure were conducted in order to eliminate items that did not fit the scales well. The second version of the instrument was tested consecutively in a new (second) round of cognitive interviews held to test items that had been changed, added and were generally problematic.

Two different methods were therefore used to improve the reliability and validity of the instruments for measuring prejudice—quantitative and qualitative—although this is not common practice in social psychology. Many researchers advocate the use of these two methods on the basis of several reasons. For instance, Ryan et al. (2012, p. 415) argue that despite quantitative methods being commonly used to analyse the relationship between the questionnaire items and the scales, “the process of testing survey question with cognitive interviews, as part of questionnaire design and refinement, can lead to better informed judgements about the potential quality of survey evidence”. Bradburn (2006) explains that many comprehension problems occur because of lexical or structural ambiguity. He gives some simple examples: the word “table” can mean a piece of furniture for placing things on or data arranged in rows and columns, i.e. lexical ambiguity means that one word can have different meanings. Structural ambiguity occurs in sentences like “Flying planes can be dangerous”, where “flying” can be read as a verb or adjective and thus the meaning of the whole sentence alters. This can create significant problems when creating items for questionnaires. When designing a question, the researcher’s aim is to “get respondents to understand the question in the same way that the researcher does”, which is quite problematic “because of the many subtleties and ambiguities of language” (Tourengeau & Bradburn, 2010, p. 318). Of course, these problems can also emerge when trying to translate and/or transfer a questionnaire into another language and/or culture. When they occur as part of an attempt at something complicated, e.g. translating a group of synonyms that have specific connotations, they can be more difficult to spot and to avoid. The aim of the paper is to demonstrate the usefulness of cognitive interviews for psychologists or the instruments they routinely use so they can be sure they really measure what they ought to measure and/or are culturally universal (like the SCM).

Cognitive interviewing can be characterised as a method for pre-testing survey questions or questionnaires^[3]. It focuses on questions that may be vague or not easy to understand or leave too much scope for unintended subjective interpretation. Beatty and Willis (2007, p. 287) define cognitive interviewing “as the administration of draft survey questions while collecting additional verbal information about the survey responses, which is used to evaluate the quality of the response or to help to determine whether the question is generating the information that its authors intend”. Blair and Brick (2010) warn that while this method has been widely used since the late 1980s, there is still no generally accepted procedure for using cognitive techniques and interpreting the data obtained. Equally there is no agreement on the optimal sample size, the number of interviews, interviewers or rounds of interviewing (Beatty & Willis, 2007). Furthermore, Presser et al. (2004) stress that there is no consensus on best practices regarding techniques, procedures and evaluating the results. For example, we have yet to resolve the questions of which of the two basic techniques—thinking-aloud interviewing and verbal probing—is better to use in what circumstances. The first technique involves the participants actively verbalizing their thoughts whilst answering the survey questions (e.g. “How Roma are viewed by Slovaks in general? Please tell me everything that comes into your mind”). When using the second technique the interviewer asks specific probing questions either after each survey question or at the end, once the entire questionnaire has been completed, in order to obtain additional information to help in understanding and/or interpreting the survey questions (e.g. “What does the term ‘in general’ mean to you?”). The advantages and disadvantages of these alternative techniques can be found in full in Willis (1999), and Willis and Artino (2013). Since in this research we use verbal probing we only cover the pros and cons of using this technique.

If it is to be performed appropriately and without bias, verbal probing demands some skills or training, to avoid, for instance, devoting too much attention to or speculating over questions which can be quite easily understood without probing. This technique allows the researcher to focus on more problematic cognitive processes such as understanding the particular wording/phrasing of more complex questions. It can also help in cross-cultural comparisons of questionnaire data in ascertaining whether the items are understood or interpreted in the same way in different cultures. At the same time it is a flexible procedure, so any problems and/or difficulties that emerge during the interview can be addressed and explained.

When designing surveys the perceived seriousness and extent of the problems associated with cognitive interviews depend mainly on the aims of the cognitive interviewing. For someone seeking to create the ideal questionnaire, it may become a never-ending process of constantly refining the wording of questions. However, abandoning the requirement to achieve the ideal and settling for having an understanding of the pros and cons of the alternative phrasing of the items can lead to relative satisfaction with the improvements to the questionnaire. In both cases, systematic and rigorous cognitive interviewing may significantly enhance the design of questionnaire items. Our aim is closer to the second, i.e. to learn about the ambiguities of the questionnaire items and either improve the wording or replace or eliminate the “troubling” item. In another words, our aim was to obtain useful information in order to improve the questionnaire items to ensure they were unambiguous, i.e. respondents would know what we are asking. Our aim is to enhance the validity of the questionnaires rather than produce “ideal” items.

We pre-tested two questionnaires. The first is based on questions that feature most frequently in studies using and testing the Stereotype Content Model (SCM) (Cuddy, Fiske, & Glick, 2007; Fiske et al., 2002; Cuddy et al., 2009; Fiske & North, 2014). These questions were translated into Slovak from the original English. The SCM measures two basic dimensions of social perception: warmth and competence. These two universal dimensions of social judgments relate closely to stereotypes and/or prejudices towards different outgroups or minority groups (Cuddy, Fiske, & Glick, 2008). Different groups can be placed in one of four possible quadrants specified/defined by these bipolar scales. However, according to the SCM, we frequently find two mixed clusters of ambivalent stereotypes on many outgroups: high warmth together with low competence or high competence combined with low warmth (Fiske et al., 2002). Stereotypically high status groups are viewed as competent, while low warmth groups are perceived as competitive (Cuddy et al., 2009). The Slovak version of the SCM also included scales to measure the emotions of contempt, admiration, pity and envy in addition to protective vs. aversive behaviour towards the minority group and frequency of different types of direct contact with the minority group in accordance with the BIAS map proposed by Cuddy, Fiske, and Glick (2007).

It should be noted that there is no definitive version of a questionnaire for investigating the SCM. However, there is a clearly defined pool of items employed in the various versions and the four-item and two-item versions are most used to measure each of the two basic dimensions of social perception: warmth and competence. Translating the items from the English versions into Slovak caused us substantial problems as the number of synonyms and their connotations differ between languages. Other difficulties resulted from cultural transferability, as different social practices prevail in different societies and if the questions are translated literally, they can have very different meanings from the original. The aim of course is to find a translation that captures the spirit of the original rather than a literal one, but when the goal is to use a specific measure, e. g. the SCM questionnaire, finding a good translation – one that is precise and meaningful in the other country’s context) is often far from easy and a slow process.

The second questionnaire – INTERMIN – was compiled by our team, based on the items that are most frequently used in contact research. It has subscales for social anxiety, social trust, behavioural intentions and contact in general because these are important in intergroup relationships (Turner, West, & Christie, 2013). We adapted the questionnaire for schools and plan to use it later on a large sample. It is designed for use in contact research on outgroups, but at the time of writing we are mainly interested in the Roma minority, which is heavily stigmatized in Slovakia, and we pre-tested it (along with the SCM questionnaire) by asking specifically about the Roma outgroup. The first draft discussed in this article consisted of 26 items plus demographic questions about the participants (for more details, see Lášticová & Findor, 2016, this issue).

2 Cognitive interview research sample

We conducted two rounds of cognitive interviews^[4] with both the questionnaires in order 1) to learn how respondents understand or interpret the meanings of the items, and 2) to pre-test whether the translation of the items from the English instruments used in contact research is suitable for use in Slovakia. The first round (see Table 1) was administered to university students, while the second round (using improved versions of the questionnaires) was carried out with high school students, as they are the target group of planned experiments based on the contact paradigm. The university student sample was a convenience sample; we were attempting to gain access to bigger groups of students from different educational establishments as well as of different specializations and from different years of study. The gender proportion was skewed as more women attend social sciences courses.

The cognitive interviews conducted differ slightly from the standard type, as many of them were held not with one respondent, but with groups of students. The main reason for this was that group interviews lead to more heated discussion and argument. On the other hand the disadvantages of group cognitive interviews are that many participants tend to adopt the dominant view and that opinions may become polarized in a way that would not arise in individual cognitive interviews.

We also tested a simplified version in which the students first completed the questionnaire pilot and then wrote comments on each question after they had been asked probing questions formulated by the research team (N=100, Trnava, university, 1^st year, pedagogy; and N=50, Bratislava, university, psychology, 1^st year ). We used this data as an additional source of information.

In the second round of cognitive interviews (see Table 2), the research sample was much more specific. It consisted of 24 participants, half of whom were from Prešov, Slovakia’s third biggest city, located in the east, which has a relatively numerous Roma population (the proportion of Roma residents is 10.06%) and half were based in Bratislava, the capital, where the proportion of Roma residents is 2.12% (Mušinka & Matlovičová, 2015). In each of the cities, six participants were involved in pre-testing and discussing the SCM and six in pre-testing and discussing the INTERMIN questionnaire. Women and men were equally represented in these samples.

In the group cognitive interviews we used verbal probing once the participants had completed the questionnaire. In some cognitive interviews two researchers were involved –one asked the probe questions while the other took written notes on each item discussed. In some cases only one researcher was involved. Other interviews were recorded and then analyzed. We did not transcribe the interviews verbatim (as is standard in qualitative interviewing for example) as we were not interested in the linguistic or discursive nuances; instead we simply analyzed the information they provided.

3 Results

We will not describe the two questionnaires in full but will present categories of typical problems (concerning one or both questionnaires) that triggered relatively extensive and passionate discussion among the participants and researchers. The problems can be grouped around six issues. The first two deal with the strategies the participants used to answer our questions – what representations of the category Roma they used and whom were they thinking of when completing the questionnaires (see 3.1), and what “optics” were used for the opinions they provided and whose viewpoint did they give in their answers (see 3.2). The next four problems are around nuances in the way the questions were formulated and generally have to do with how the participants understood our questions—they concern the assumption that distinct items were logically interconnected (see 3.3); the period of time and locality referred to in our questions (see 3.4); translation and transferability of meanings from one language to another (see 3.5); and double negation (3.6).

Our results are arranged such that in the first instance we introduce the original item, then the comments of the participants on the wording of the item (whether it was clear or vague, how it was understood, what was disruptive, etc.) and finally we consider whether, how, why and to what extent the original wording was changed.

4 Second round of cognitive interviews

After modifying the two questionnaires on the basis of the results of the statistical analyses and the first round of individual and group cognitive interviews, a second round of cognitive interviews was held with high school students. This showed that the modifications to the original questions made in light of the comments and suggestions from the first round of individual and group cognitive interviews had improved the way our participants understood the instruments. The instructions were now clear to participants and the wording/phrasing of questions was not vague. The items concerning the various types of contact with Roma were now comprehensible and respondents could answer them without problems. Nevertheless, some other problems emerged, but these seem easy to improve. For example in the SCM questionnaire in relation to the question “How much does special treatment (e.g. occupation selection privilege) given to Roma make things more difficult for other people” it was suggested that we replace our example in parenthesis with a more specific example, like housing, education, meal allowances, or welfare privileges. Another kind of problem that several of the participants in the second round of cognitive interviews had with the SCM questionnaire concerned the scales to measure emotions. They consist of pairs of items which have quite similar meanings, so distinguishing between them can be tricky. Specifically, the participants have to consider the feelings of the majority towards Roma regarding contempt (contempt, disgust), admiration (admire, proud), pity (pity, sympathy) and envy (envious, jealous). Surprisingly, we subsequently realized that it had also been mentioned in the first round of cognitive interviews, but only marginally, so we had paid insufficient attention to it. After the second round of cognitive interviews it was suddenly clear that quite a lot of the participants had had problems differentiating between feelings of envy/jealousy and contempt/disgust. In order not to confuse participants by putting them in the position of having to consider the degree of similarity our research team decided that instead of placing the two relatively similar questions one after another it would be better to mix them up. So, for example the item “To what extent do people tend to feel envy toward Roma” would no longer be followed by “To what extent do people tend to feel jealousy toward Roma” but by “To what extent do people tend to feel contempt toward Roma”.

The second tested draft of the INTERMIN questionnaire proved even less problematic, as we quite often asked whether a different (usually more specific) wording of the question would be better and generally the participants thought both variations were perfectly understandable. Also we have to stress that when we shortened the lists of items used in line with the results of the statistical analysis of the scale reliability a lot of the earlier problems regarding excessively similar synonyms and perceived interconnectedness disappeared.

For an overview of the six problematic areas, the original versions of the questions/items, and the changes following the first and second round of the cognitive interviews, see Table 3.

5 Discussion and conclusions

To conclude, according to our findings the participants in the cognitive interviews generally expressed difficulties in understanding and/or interpreting questions that seemed clear to us. However, in the context of the cognitive interviews we have to take into consideration the fact that we were also actively problematizing the measure and sometimes explicitly asking about existing problems and difficulties – therefore the participants may have focused on finding issues and so perhaps also reported minor ones.

With this in mind, we learned (or rather, we already knew this in theory, but our experiences during the cognitive interviews made this a much more urgent concern) that firstly, even seemingly very banal, easy and/or straightforward questions or items can be understood /interpreted quite differently by different participants. This concerns the content and clarity of the questions (c.f. Willis & Artino, 2013). Obviously, the solutions to this include formulating the items more clearly so they are easier to understand, not using foreign or complicated words, stating clearly the time frame or locality being asked about, paying immense attention whilst translating from one language to another, not using double negation, giving repeated information on unusual scales, etc.

Secondly, we have learned that sometimes different participants will adopt very different strategies when answering the same questions. For example, the questions may be answered from different viewpoints or using very different representations of the outgroup being investigated. In the SCM questionnaire, we wanted participants to answer them from a specific viewpoint (that of the majority in Slovakia), and so we solved this by reformulating and clarifying the question, despite it not simply being a banal issue of unclear wording. In order to deal with sometimes very different representations of the outgroup, we came up with the solution of including an explicit “category representation question” at the end of the questionnaire to improve the researchers’ understanding, e.g. in this case asking participants how exactly they were constructing the category of the Roma person or persons they were thinking of when answering the question.

Thirdly, the logic underpinning the construction of the questionnaire can become problematized if items that the researchers intended to be part of a subscale, e.g. for anxiety, are understood by the participants to measure something else or are simply not considered as part of the subscale. As the main part of this article was about how to mitigate the problems summarized in the first and second points, we did not dwell on the problems with scales reliability in any depth. We just mentioned them briefly when discussing changes to the second drafts of the questionnaires, but again, we would like to stress that a thorough statistical analysis of scale reliability and factor structure can reveal internal problems with the consistency of subscales. In other words, it can clearly show which items participants do not consider fit into the scale (even where researchers think they fit perfectly) and badly affect coherence. Based on this analysis we were able to discard several problematic items in the two questionnaires, and the new versions are more consistent. However, this method does not suffice alone (without being combined with cognitive interviews). A scale reliability analysis shows which items are weak, which cause problems for the (sub)scales and which can/should be removed. Statistics cannot be used to establish exactly why some of these items are weak and how they should be improved where we want to measure what we are interested in, but cognitive interviews can be used to achieve this.

In general, using cognitive interviews helped us to refine the questionnaire design and thus improve the quality of the questionnaire. We are fully aware that questionnaires are still not ideal, as all instruments of this kind contain language subtleties and ambiguities and although it is possible to enhance the validity of the questionnaires, producing “ideal” items is an unrealistic expectation.

To conclude, we think that if researchers use a questionnaire without having conducted cognitive interviews, they have only a very small degree of control over what they are “really asking” – what the participants understand the questions to mean and what they are answering. With cognitive interviews, the researchers can see where the main problems are and try to improve the questionnaire. However, at least for some areas of research, other data gathering methods seem to provide a more fine-tuned option, e.g. the qualitative interview, where researchers can explain a question to the participants when it is clear they do not understand the question or have interpreted it differently and the researchers can ask deeper questions if they do not understand. However, if we want to measure the effect of different interventions e.g. in dealing with bias against a stigmatized outgroup, we need to compare data about post-intervention attitudes with the control group data and for this we need to use quantitative measures. Thus, in our project we have created new versions (modified for Slovakia, for the Roma minority and for schools) of the two questionnaires measuring contact, attitudes, emotions, social anxiety, social distance and behavioural intentions. The cognitive interviews fulfilled their assigned function enabling us to improve these measures to the extent that we now at least partially understand what we are “really” asking and what our participants are answering about.