Minority genders in quantitative survey research: a data-driven approach to clear, inclusive, and accurate gender questions

2.1 Including an “other” option

One of the principal existing studies to investigate the inclusion of further genders beyond male and female in gender questions is that of Medeiros et al. (2020). This study’s main aim was to test whether including a third gender option “other” would be confusing or undesirable to nationally representative samples of the population in Canada, the US, and Sweden. They found that participants did not differ in survey satisfaction rates based on the inclusion of an “other” option. They posited that the addition of non-binary options at the very least provides data on a vulnerable population without posing any adverse reactions to the survey itself. This was also found in an exploration of a non-binary sex question for the census in Scotland (National Records of Scotland 2021), where the question included the option “in another way” (as opposed to identifying as a man or a woman), which did not negatively affect response rates.

One major drawback of the approach taken by Medeiros et al. (2020) is that by framing the third gender option as “other” compared to the binary options of “male” and “female”, it frames those minority gender identities as essentially and most importantly in opposition to binary genders, grouping all other genders as one and the same. This reinforces the situation in which non-binary individuals must always contend with the domination and centring of the notion of a gender binary (cf. Taylor et al. 2019; but also Dembroff 2020). Furthermore, the phrasing “other” quite literally others participants with minority genders (English 2022). Finally, the singular “other” option has a particularly high risk of erasing minority genders outside of Western contexts: minority genders such as hijra (see e.g. Goel 2016), two-spirit people (Robinson 2020), and travestis (Jarrín 2016) have different histories, contexts, and meanings that are more specific than “anything outside the binary”.

2.2 The two-step approach

Bauer et al. (2017) focused on Canadian LGBT+ people’s own responses to questions about gender and trans status for health-based population surveys. They explored how different sets of questions were evaluated within gender-diverse groups, whilst separating questions on sex assigned at birth from questions on gender identity. They found that non-responses were rare for such questions (only one out of 311 survey respondents) and that comprehension was good. They recommended a specific two-step question with two options for sex assigned at birth – “male” or “female” – and four options for gender identity – “male”, “female”, “indigenous or cultural minority identity (e.g. two-spirit)”, or “something else (e.g. gender fluid, non-binary)”.

Still, trans populations in Bauer et al.’s (2017) study run the risk of misclassification, as a number of trans respondents did answer that their gender identity is in line with their sex assigned at birth. This suggests that asking for participants’ sex assigned at birth is not an effective way to determine whether a participant is trans. For example, some people who have medically transitioned do not see themselves as having a differentiation between their sex and their gender (Bauer et al. 2017: 15). This may also be the case for many trans people who have not undertaken medical steps to transition. In these cases it may be better to ask directly about trans status and leave questions about sex assigned at birth to more specialist healthcare settings. Furthermore, the term “sex assigned at birth” may simply not be a familiar term to all participants.

2.3 Open questions

To address the issues with potential erasure and misclassification discussed above, Cameron and Stinson (2019) proposed leaving the categorization up to participants themselves: they argued that the best way to ask participants about their genders is to ask an open question using a text box into which participants type their answers. The open text box option offers complete autonomy and control for participants to name their gender identity, rather than having to pick an option that is “close enough”. Whilst such text box data may seem difficult to treat, Lindqvist et al. (2021) found that most responses to an open gender question are easy to code (99 % of cases in their sample of 794 US participants in the general population were entries like “male”, “female”, “f”, “m”, “woman”, and “man”).

The open question may also have its drawbacks however, as Bauer et al. (2017) argued: the approach leaves space open for ambiguity in a way that may not always be easy to resolve when it comes to queer data. Whilst combining the answers “male”, “man”, “Man”, and “just a man” may be low-stakes, it becomes more difficult to decide whether the answer “trans man” should be categorized within that same category, or whether this participant specifically wanted to stress the trans aspect of his gendered experiences and wanted to be classified separately. Guyan (2022: 129) pointed out that what may be perceived as an “error” in data cleaning, may in fact be a conscious subversion of the rules and expectations of data collection practices. For example, if one of the participant responses to a gender question is the entry “gay”, this could be read as an error (with the participant misinterpreting the question as a sexuality rather than a gender question). However the participant may have intentionally given this response to say their gendered experiences are best captured as being read as gay and this matching their internal sense of gender.

The open question proposal has not yet been tested evaluatively by means of a demographic survey, although English (forthcoming) found that in focus groups many queer people report that they would find open questions the most desirable.

2.4 De-essentializing and essentializing minority genders

All efforts to categorize and ask about gender as a demographic measure encounter the problem that they essentialize the gender identities involved in the question. This runs counter to a shift within queer linguistics where researchers are moving away from treating LGBTQ+ speakers as essentialized or as a homogeneous group (see e.g. Jones 2021). Rather, current research tends to shed light on the complex nature of queer identities and how they are negotiated in different contexts (Barrett 2017; Borba 2019; Jones 2018). This movement speaks to a tension in the collection of queer data that Guyan (2022: 127) has described: researchers have to balance being precise about the nuances, complexities, and porous boundaries of different identities and experiences with the sometimes strategic use of essentializing terms or identities to pursue policy goals and mobilize action. The latter, coined “strategic essentialism” by Spivak (1988), constitutes a temporary presentation of a group as having a shared essence to pursue specific goals (e.g. in this case making visible and normalizing genders outside a male-female binary by introducing a “non-binary” option), rather than a universal conceptualization of this group as having a shared essence (e.g. many second-wave feminist approaches establishing women as different from men based on a shared cultural essence; e.g. Bucholtz 2014: 31). As Guyan (2022: 128) pointed out, strategic essentialism has its pitfalls in that it erases differences within marginalized groups – especially for those within the groups which are already most likely to be erased.

In the case of survey research, the explicit mention and labelling of a minority gender category like “non-binary” carves out space for participants for whom this label fits well, countering some of the invisibility that non-binary participants may otherwise face. At the same time, it may erase the diversity of minority genders beyond this label, which open questions would be less likely to do. To be more fully inclusive of participants of minority genders, survey research needs to take both interests into account.

2.5 Research questions

Whilst it has been shown that improving gender questions by adding response options beyond the gender binary does not hinder participant comprehension (Bauer et al. 2017; Medeiros et al. 2020), alternative questions and response options have rarely been tested, and even less so in comparison to each other. The current paper compares a range of five gender questions, including those proposed in these previous studies, by asking the following research questions:

1. How well do different gender questions represent gender diversity among LGBTQ+ participants? (insight into diversity)

2. Which questions are LGBTQ+ participants most likely to refuse? Which ones are they most likely to answer? (refusal rates)

3. How are the different gender questions evaluated in terms of their accuracy, inclusivity, and clarity? (evaluation)

For the first two questions we focused on data from all LGBTQ+ participants, in order to map gender diversity across questions, as well as refusal rates for the wider community. This was to ensure no gender diversity was missed for participants who do not use the label “trans”, as well as to assess if cisgender participants refused to answer any questions they perceived as less inclusive. For the issue of evaluation, we centred non-cisgender perspectives. This was to ensure the appropriate weight was given to their experiences taking into consideration the frequent marginalization of their knowledge (see Fricker and Jenkins 2017 on epistemic injustice harming trans individuals as knowers). It is also for this reason that no cisgender heterosexual participants were invited to the survey; note that Medeiros et al. (2020) found that for general populations the inclusion of a non-binary option in gender questions was not a problem. It is of course possible that this depends on the elaborateness of the question, and may differ if the range of responses is more complex than that proposed by Medeiros et al. (2020), but this was beyond the scope of the current paper.

3.1 Survey questions

The survey used was built in Qualtrics and approved by the ethics committees at Utrecht University and the University of Edinburgh. Participants were asked to fill out five gender questions with different response options as they would if they were presented with one in a different research survey. The questions that were tested, together with the response options provided, are given in Table 1.

For each of the multiple-choice questions, an option was added where participants could indicate that they “would refuse to answer this question and leave a survey that included it”. This was then coded in the analysis as a refusal. It was also possible to simply not select any option, which was coded as a non-response. For the open question, participants were instructed to not write anything in the text box if they wanted to indicate they would refuse to answer this question. The five questions were presented in randomized order. After each question participants were asked to evaluate the relevant question by indicating on three 5-point Likert scales the degree to which they found the question clear, inclusive, and accurate to themselves.

Finally, participants were asked some demographic questions, including about trans status (“cisgender”, “transgender”, “neither”, or “unsure”), sexuality, country of residence, age, university education, and whether they considered themselves a person of colour.

3.2 Distribution and sample

The survey was distributed through our own personal networks and a range of LGBTQ+ networks in the United States, the United Kingdom, and the Netherlands, as well as through social media. It was advertised to anyone with an LGBTQ+ identity, rather than just trans and non-binary people, in order to include any participants who feel they do not quite fit the gender binary but may not use the trans and/or non-binary labels.

As is shown in Table 2, a total of 682 participants filled out the survey, 233 from the Netherlands, 224 from the UK, 172 from the US, and 47 from other countries; three participants did not indicate their location. Of the sample, 242 participants were cisgender, 271 were transgender, 95 were neither, and 68 were unsure; three participants did not indicate their trans status.

The sample was less balanced for university education and race. Only 60 participants were not attending and had never attended university, and only 47 participants considered themselves a person of colour, with 595 participants indicating they did not, 30 participants indicating the term “person of colour” did not adequately describe them, 6 participants selecting “other”, and 4 participants not answering the question. Young people made up the largest share of the participants, as can be seen in the age distribution given in Figure 1.

Figure 1:

Age distribution of the sample.

4.1 Overview

The results of the survey questions showed that the inclusion of more than two gender options greatly increased accuracy to the diversity of gender identities within LGBTQ+ populations and reduced the number of participants refusing to partake in a survey or giving non-responses. Larger numbers of response options also substantially improved participants’ evaluations of the questions. The highest rated and least refused gender question was the six-option multiple-choice question.

4.2 Participant diversity and question refusal

4.2.1 Diversity

The sample of LGBTQ+ participants in our study showed a much wider range than just two gender labels. Increasing the number of response options allowed for this population’s gender diversity to be more accurately represented. This is summarized in Figure 2, which shows the answers given across the multiple-choice questions with two, three, and six response options available. As can be seen in the figure, the two-option multiple-choice question was met with a high number of refusals and non-responses (38 %), whilst a large group of participants indicated “other” in the three-option question (37 %). In the six-option question, further diversity came to light: 34 % of respondents selected “non-binary” as their gender, 0.4 % selected a culturally specific minority gender, and 9 % of respondents made use of the write-in option “other, please specify”. Generally speaking, these respondents were the same ones that refused to answer the binary question. However, the number of people indicating one of the two binary genders went down from 62 % in the binary question to 54 % in the three-option question and to 50 % in the six-option question, which implies that some participants who may not have refused a two-option question actually did have a gender identity that was better represented by one of the other response options offered. This suggests that studies which work with binary gender questions do not just lose participants, but also gather data which is inaccurate to participants’ genders.

Figure 2:

Responses to gender questions with an increasing number of multiple-choice questions.

When participants were given the option to answer the open question with no preset responses, we see even more diversity. Figure 3 shows the range of repeated answers after removing all spaces, punctuation, and capitalization. Three things stand out in the figure. First, a considerable part of the sample gave a unique answer which no other participants gave (16 %). Second, many participants gave a combination of multiple gender markers (e.g. “non-binary/male”). Both highlight the possible complexity and nuance of participants’ identities beyond one-word labels. Third, when given the open question, many participants included information about their trans status (e.g. “trans man”, “cis female”). This suggests that participants were unsure about which types of gendered information the researchers were interested in. This also comes to the fore in participants’ evaluations of the open question, which was ranked relatively low for clarity (see Figure 7).

Figure 3:

Recurring answers to the open gender question, with no preset responses. The total number of unique answers is shown to the right of the graph, as well as the number of empty responses.

4.2.2 Question refusal

As can be seen in Figure 4, a considerable number of participants refused to answer some of the gender questions, depending on which response options were given. This was especially true when only two options were given (38 % refusals and non-responses), or when participants were asked about their sex assigned at birth in the two-step question (22 % refusals and non-responses). The open question received a relatively large number of non-responses, despite leaving the classification of one’s gender up to the participants themselves entirely (11 %, which is higher than the more limited three-option question, which was refused or not responded to by 10 % of participants). The six-option question was refused or not responded to by only 3 % of participants, whilst 4 % selected the option “prefer not to say”. Although the latter option meant that no information about a participant’s gender was given, it meant participants were still willing to continue a survey or experiment which included this question (which was not the case for the other refusals). No respondents refused all questions.

Figure 4:

Response refusals across different question types.

4.3 Question evaluations

To investigate how the different gender questions were rated we focused on non-cisgender participants’ evaluations (i.e. those who indicated being transgender, neither cis nor trans, or those who were unsure). Participants rated these questions for three parameters: accuracy, inclusivity, and clarity. We tested the differences by means of Kruskall-Wallis tests and tested comparisons between groups by means of Benjamini-Hochberg adjusted Dunn’s tests.

The ranked evaluations for accuracy are shown in Figure 5, created using the likert() function in the HH package in RStudio (Heiberger and Robbins 2014). As can be seen here, the most highly rated question type for accuracy is the six-option multiple-choice question, followed by the open question. These were both rated very positively. Then, the two-step question was perceived somewhat more neutrally with some high and some low evaluations. The three-option multiple-choice question was rated quite negatively, although not as negatively as the binary multiple-choice question, which was rated mostly negatively. The differences in ratings were highly significant (χ² = 999.41, p < 0.001), with Benjamini-Hochberg adjusted Dunn’s tests showing all comparisons being significant. Most comparisons were significant at the level of p < 0.001, with the exception of the one between the six-option question and the open question, which was significant at the level of p = 0.017 (below the Holm-Bonferroni corrected α = 0.025).

Figure 5:

Non-cisgender participants’ Likert scale ratings for accuracy evaluations (ranked).

For the inclusivity evaluations, the ranked options are identical to the accuracy rankings, with the six-option question being rated most positively and the binary question most negatively. Here, the differences in ratings were also highly significant (χ² = 1,232, p < 0.001), with Benjamini-Hochberg adjusted Dunn’s tests showing all comparisons being significant at the level of p < 0.001, other than the comparison between the six-option question and the open question, which was significant at the level of p = 0.036 (below the Holm-Bonferroni corrected α = 0.05). The ranked evaluations are shown in Figure 6.

Figure 6:

Non-cisgender participants’ Likert scale ratings for inclusivity evaluations (ranked).

As can be seen in Figure 7, patterns for the clarity-based evaluations differed in two ways from the inclusivity- and accuracy-based ones: the open question was rated lower than the two-step question, and the ratings for the two-option and three-option questions were much less overwhelmingly negative than those for inclusivity and accuracy. Still, ratings all differed significantly between the question types (χ² = 318.08, p < 0.001), with Benjamini-Hochberg adjusted Dunn’s tests showing all comparisons being significant at the level of p < 0.001.

Figure 7:

Non-cisgender participants’ Likert scale ratings for clarity evaluations (ranked).