Assessing the Importance of Sample Choice and Selectivity for Sex Segregation in College Majors: A Replication of Ochsenfeld (2016)

2.1 Brief Summary of the Ochsenfeld (2016) Study

In 2016, Ochsenfeld published a study that examines the factors explaining the persisting gender-based segregation in college majors. Specifically, the study investigates the explanatory power of preferences (i.e. vocational interests) and constraints (e.g., approval of the choice of major by significant others, the relative strength in mathematics, anticipated working hours) for gendered major choices.

The study employs conditional logit models (McFadden 1974), grounded in the theoretical concept that major choices are essentially a match between individual-level factors (e.g., vocational interests, job values, relative math grades) and college major characteristics (e.g., mathematical content, wage prospects, overwork norms). Thus, the analysis considers both individual variables reflecting the preferences and constraints of college entrees as well as structural variables representing college major attributes of 23 different fields.

Ochsenfeld (2016) obtains the individual-level data from the NEPS SC5 sample of college students. The sample size of the individual data comprises 9,109 cases. Structural data regarding the 23 majors come from the datasets NEPS SC5, Micro Census 2007/2008/2009, and the HIS graduate panel studies 1997/2001.

The core of the analysis is a decomposition analysis based on conditional logit models and designed to assess the explanatory power of each examined mechanism for gender-based segregation. This decomposition reveals that the investigated preferences and constraints can account for approximately 46 % of the gender gap in major choices. 37 % of the overall gap is explained by gender differences in vocational interests. To a lesser extent, factors such as peer approval of major choice (4 %) and relative strength in mathematics (5 %) also contribute to the gender gap in college major choices. Job values, anticipated discrimination, and parental approval of the major choice appear to have minimal to no influence on gender differences.

These highly relevant findings suggest that women and men choose different majors not primarily due to environmental or social factors but as a result of internalizing gendered vocational interests. However, the implementation of this study faces methodological problems that could not be addressed due to data limitations at the time of the Ochsenfeld (2016) study. These problems could potentially introduce biases, emphasizing the need for a replication.

2.2 Potential Sources of Bias

In this study, we refer to the terminology of biases introduced by Morgan & Winship (2007) and Elwert & Winship (2014) to sociology. Typically, every research design faces the challenges of overcoming common confounding bias, overcontrol bias, and endogenous selection bias (e.g., Elwert & Winship 2014; Morgan & Winship 2007). While common confounding bias refers to the omission of important variables that influence both the outcome and the explanatory variable, overcontrol bias refers to the inclusion of mediators, i.e., important variables that explain a relationship between an outcome and an explanatory variable. In this replication study, however, we concentrate on endogenous selection bias, i.e., conditioning on collider variables. Collider bias emerges when a model conditions on “… a variable that is itself caused by two other variables, one that is (or is associated with) the treatment and another that is (or is associated with) the outcome.” (Elwert & Winship 2014: 31).

In the following, we show in stylized directed acyclic graphs (DAGs) how collider bias might affect conclusions from previous work. In doing so, we briefly show how (i) measurement error and (ii) sample selection might introduce bias in the estimates presented by Ochsenfeld (2016).

2.2.1 Post Hoc Rationalizations

A possible form of bias displays Figure 1 in a stylized DAG, which does not take unconfounding conditions between the variables into account because we want to focus on collider bias issues and not on common confounding bias. Additionally, we want to stay close to the original study, which also used stylized DAGs. Figure 1 shows that vocational interests are measured at a time when the choice of a major has already taken place—a circumstance that may introduce collider bias (Elwert & Winship 2014) and that was already mentioned by the original study (Ochsenfeld 2016: 129). According to theory, gender (X) (i.e., gender-specific socialization and internalization processes over the early life course) leads to the formation of gender-specific vocational interests (P₀). These interests, which have been developed prior to the actual decision, cause the choice of a specific major (J) that matches these interests at time t₀ (prior to the choice of a major). The choice (J) subsequently leads to the observed male- or female-dominated major (Y).

However, as depicted in Figure 1, the NEPS-SC5 data employed by the Ochsenfeld (2016) study measure vocational interests not at t₀ but at the same time as the outcome (t₁). Thus, previous findings may suffer from collider bias (i.e., endogenous selection) if the major choice and pretransition interests influence respondents’ answers to preference questions at the time of the survey (t₁). In other words, prior research used the vocational interests measured in t₁ as if they were measured in t₀.

Figure 1: Stylized DAG depicting collider bias due to post hoc rationalizations.

Note: For simplification and demonstration, we do not assume (unobserved and observed) common causes in this DAG. Therefore, we omitted the unconfounding conditions.

The data used in this replication do not suffer from endogenous selection bias that emerges due to the conditioning of a collider variable, which opens the noncausal pathway P_t0 → P_t1 ← J. For instance, if an individual studies pedagogy who did not have a particularly strong social interest before, he or she may nevertheless report a high social interest after starting the study. This can be explained by post hoc rationalization or an actual shift in interests as a result of studying. However, both of these factors arose after the student began the studies and therefore do not represent a causal explanation for the student's major choice. The new data set measures vocational interests (i.e., preferences) prior to the transition (P_t0). Therefore, in our replication, we do not condition on a collider variable, and post hoc rationalizations of respondents cannot bias our results.

2.2.2 Sample Choice

Another form of bias that was not addressed in the original study could emerge from conditioning on a postoutcome collider. Because the Ochsenfeld (2016) study relies on a data set that incorporates only university students, a conditioning on the outcome problem might occur (Elwert & Winship 2014). Figure 2 depicts the central problem in a stylized DAG.

In this stylized DAG, survey participation (S) constitutes a collider variable along the noncausal pathway M → S ← Y (M, match of individuals’ preferences and major characteristics; Y, university enrollment and major choice; S, survey participation). This form of postoutcome collider bias is likely to occur because in student data, we only observe individuals enrolled in university and who have a (certain) match between their preferences and major characteristics. However, certain matches between individuals’ preferences and major characteristics may divert individuals from studying for a bachelor’s degree and may lead to the receipt of vocational training (see Allmendinger 1989 for a classification of the German education system). If this is the case, sample selection (i.e., conditioning on studying) could introduce postoutcome collider bias and affect the validity of prior findings.

Figure 2: Stylized DAG depicting postoutcome collider bias due to sample selection.

Note: For simplification and demonstration, we do not assume (unobserved and observed) common causes in this DAG. Therefore, we omitted the unconfounding conditions.

Using prospective panel data that incorporate transitions from high school to different postsecondary educational states (i.e., NEPS-SC4 data) allows us to implement an inverse probability weighting (IPW) approach that accounts for sample selection. Breen and Ermisch (2021) have recently demonstrated that in many cases, IPW may reduce postoutcome collider bias.

3.1 Data Sets, Models and Variables

The main difference between the replication and the Ochsenfeld (2016) study involves the use of prospective panel data from the NEPS starting cohort 4 (NEPS, SC4, version 10.0.0)^[1], which includes pretransition information, whereas the original study relied on NEPS starting cohort 5 data (NEPS, SC5, version 4.0.0), which include only university students (see the “Data overview” chapter in the appendix for a brief overview of the data). The NEPS SC4 panel started with 9th graders from all educational tracks in Germany. Consequently, these data survey individuals both during their school years and during the transition into higher education, vocational education and the job market. Because both data sets are provided by the same institute (Leibniz Institute for Educational Trajectories), all employed individual-level survey items used in our replication are the same as in the original study. To further enhance comparability across our study and the replicated study, our data preparation used the data preparation do-files of the original study (Ochsenfeld 2015). This constitutes an ideal setting to address the outlined collider bias problems with a new data source.^[2]

A key innovation of the original study is the accurate test of the theoretical considerations by means of conditional logit models (McFadden 1974). The key theoretical argument is that major choices reflect a match between individual-level (e.g., vocational interests) and major characteristics (e.g., math intensity). The conditional logit model enables scholars to test this theoretical process. In our replication, we specify the following model thereby using the Ochsenfeld (2016) specification:

The term p_ik indicates the probability for an individual to choose a major k. J represents the number of alternatives (i.e., 23 college majors)^[3], and X_ij is a vector of the independent variables. The major characteristics can be considered predictors in the model. However, characteristics that do not vary between the 23 majors cannot be included. Thus, we need information about the values or costs individuals assign to each major. Since we do not have any direct measurements available for this, we interact the individual characteristics with the major characteristics (Shauman 2006). This ensures, for example, that math-intensive fields of study have a higher value or lower cost for individuals with relative math strength than less math-intensive fields.

The conditional logit considers the individual value for each major, not only for the chosen one (McFadden 1974). Moreover, for the conditional logit models, we need to translate the gender differences in major choice into a structural characteristic (Ochsenfeld 2016). Therefore, we created a variable that reflects the proportion of female students in each major (prop.female). This structural variable is interacted with individuals’ gender (female x prop.female).

The key part of the analyses is the decomposition based on the conditional logit models. The decomposition is a mediation analysis with a baseline model to which we add the theoretically identified mechanisms and observe how much the coefficient female x prop.female diminishes. The modeling strategy in our replication is exactly the same as in the original study by Ochsenfeld (2016) and differs only in the employed individual-level data (NEPS-SC4 vs. NESP-SC5).

Our replication sets up all variables according to the original Ochsenfeld (2016) study (for an overview of all operationalizations, see Table A2 in the appendix). The crucial variable of vocational interests is measured by RIASEC scores, which are intended to approximate individuals’ preferences for different (job) tasks. Vocational interests are classified into six dimensions: realistic, investigative, artistic, social, entrepreneurial, and conventional. It is important to note that all individual-level characteristics stem from the new prospective NEPS- SC4 data and that they are the same constructs as in the original study (see Table A2 in the Appendix).

As outlined above, the theoretical and methodological test requires operationalization of the structural characteristics of majors. To ensure comparability, we use the same data sources as in the original study to create the structural variables.^[4] Our data sources include NEPS-SC5 (version 4.0.0), German Micro Census (2011, 2012, 2013), and the HIS graduate panel (1997, 2001). The operationalization of the structural variables is the same as in the analyses of Ochsenfeld (2016) (for an overview, please refer to the chapter “Operationalization of the constructs used” in the appendix for more details).

Importantly, to retrieve major characteristics, we aggregated (group means) individual data from respondents who studied one of the 23 majors. To keep measurement error low, analogous to Ochsenfeld (2016), we only use majors for which the computation of group means is based on at least 100 observations.

3.2 Analytical Approach of Replication Study

The analytical strategy for our replication is straightforward. As mentioned earlier, we constructed each employed variable as in the Ochsenfeld (2016) study. The key difference is the application of prospective panel data. These data entail the same preference measure as the original study. However, preferences were measured prior to transitions into university education. To elaborate on whether the importance of preferences depends on the timing of measurement, we first present our baseline replication results, which we display next to findings from Ochsenfeld (2016).

Second, to investigate whether the importance of preferences and constraints in explaining gender differences in college majors depends on sample selection, we show the results from a model that takes such sample selection into account. A logistic regression model estimates the selection into treatment (i.e., treatment model). To this end, we use the population of all individuals with a standard university entrance qualification^[5] in the data set (N=2,700). The dependent variable of the treatment model is a binary indicator for studying or not after completing high school. To retrieve the propensity score, we regress on the binary outcome gender (dummy indicator), migration background (dummy indicator and a dummy for missing value), parental socioeconomic status (dummy indicator for at least one parent with university education and a dummy variable for missing cases), the six RIASEC scores (composite index with a range from zero to one for each of the six interest dimensions: realistic, investigative, artistic, social, entrepreneurial, and conventional), two metric job value indicators (one approximates a preference for high earnings, while the other proxies a preference for a good work-life balance), and two central school performance measures (metric math and German grades). The results from the matching equation are in Table A3.1 in the appendix. After predicting the conditional probability of studying, we generate the inverse probability weights as P(T=1|covariates) if T = 1 (individual starts studying) and 1-P(T=1|covariates) if T = 0 (individual does not start studying) (Hernán & Robins 2020). The mean of the derived inverse probability weights is 1.99 with a standard deviation of 1.35. The minimum of the weights is 1.05, and the maximum is 11.69. The conditional logit models (i.e., outcome models) are weighted with the inverse of the conditional probability of studying for each individual. The weighting of each individual in the outcome models creates a pseudopopulation that considers individual heterogeneity across the treatment and control groups (Hernán & Robins 2020). Therefore, sample selection based on the included variables in the treatment equation no longer biases the estimation.

Third, to elaborate on whether the importance of preferences and constraints in explaining gender differences in college majors depends on the selectivity of a sample, we use the propensity score to showcase how the gender gap and the explanatory power of the Ochsenfeld (2016) specification vary across its distribution. We provide a model that splits the sample into above- and below-median sample inclusion probability and a model that splits the sample into four different subsamples corresponding to four quantiles of the propensity score distribution. In so doing, we are able to speculate the extent to which certain skewed samples have the potential to distort substantial conclusions (i.e., the predictive power of sociological theories).

4.1 The Role of Post Hoc Rationalizations and Sample Choice for the Validity of Prior Findings

Table 1 shows the main results of our replication. The first column includes the original Ochsenfeld (2016) results. The second column shows our replication results with the new vocational preference measure. The third column shows our results additionally accounting for potential sample selection using IPW. The first row of Table 1 indicates that our point estimates are very close to those of the Ochsenfeld (2016) study, already hinting toward strong validity of the earlier findings. When turning to the second row of Table 1, our results indicate that the replication models also explain a substantial part of the gender differences in major choices (52.8 percent in column 2 and 53.4 percent in column 3, respectively).^[6] The explanatory power of the Ochsenfeld (2016) specification even increases slightly when using panel data that account for the two potential sources of collider bias associated with the use of cross-sectional university student data.

Regarding the preference measures employed, our replication results (columns 2 and 3 of Table 1), which use an interest measure that cannot be affected by actual major choices, indicate that the explanatory power of these measures is hardly affected. Gender differences in vocational interests are therefore the major driver of sex segregation in the choice of major in Germany. The constraint part of Table 1 also indicates that our replication findings are more or less in line with the original study. However, one substantial difference emerges. The last row of Table 1 shows negative values for the match between female gender and majors with high levels of anticipated discrimination. This is somewhat surprising because this result indicates that if women choose majors with high levels of anticipated labor market discrimination (i.e., male-dominated fields), sex segregation increases. One possible explanation could be that the data sets employed to derive the discrimination variable (HIS data from the late 1990s and early 2000s) are outdated. Unfortunately, more recent starting cohorts of the HIS graduate panel no longer include the corresponding variable. Moreover, measuring discrimination as agreement with the statement that the “right gender” is important for finding a job is not optimal. In occupations with a high proportion of women, the female gender could be considered the “right” gender. This would make the measurement ambiguous. Thus, future analyses could search for more up-to-date data and a cleaner measurement of discrimination against women.

4.2 Assessing the Importance of Sample Choice

Table 2 again shows the original Ochsenfeld (2016) results in the first column. The second column shows results based on a sample of individuals with a study propensity below the median, while the third column shows results based on a sample comprising only individuals with a study propensity equal to or above the median. In so doing, we want to showcase how sample selection may distort substantial conclusions on sex segregation in higher education.

In contrast to our main results (provided in Table 1), Table 2 indicates pronounced differences in the point estimates between the original results presented in column 1 and the replication results in columns 2 and 3 that rely on a sample with a low and high study propensity. Regarding sex segregation, our results indicate a more pronounced gender gap among individuals with a low study propensity, while the gender gap is less pronounced among individuals with a high study propensity.

The influence of potential sample selection on substantial conclusions becomes even more evident when turning to the explanatory parts of Table 2. As row two indicates, the explanatory power of the Ochsenfeld (2016) specification also varies by the study propensity. While the original specification accounts for only 32.6 percent of the variation between the genders among individuals with a low study propensity, it accounts for 67.0 percent of sex differences among individuals with a high study propensity. While this result might also have substantial implications, it reveals the importance of sample selection, which has the potential to substantially change our understanding of sex segregation in major choices.

Regarding preferences, our replication results indicate that among a sample of individuals with a high study propensity, the explanatory power of preferences increases, while it remains roughly the same among a sample of individuals with a low study propensity (48.8 percent in column 3 and 36.9 percent in column 2, respectively).

The role of constraints changes substantially. Across the two samples, point estimates differ substantially, and the explanatory power also differs. The first pronounced difference emerges when turning to the role of approval by friends. While approval of friends substantially (10.9 percent) reduces the gender gap in a sample of individuals with a high study propensity, friends’ approval increases sex segregation in higher education among a sample with individuals with a low study propensity. In addition, in a sample of individuals with a high study propensity, the relative math grade appears to be almost three times as important as in a sample with individuals with a low study propensity (percentage explained in column 3 8.3 in comparison to 2.9 in column 2). Again, the discrimination interaction is negative, which, as mentioned earlier, is a somewhat counterintuitive finding. Nevertheless, the negative influence is more pronounced in the sample with individuals with a high study propensity (see last row of columns 2 and 3).

Figure 3 additionally shows that the results tend to differ even more when we further differentiate the likelihood of sample inclusion. To this end, Figure 3 shows how the results of the Ochsenfeld (2016) speciation differ when we split the sample into four equal parts representing the quartiles of the sample inclusion probability. Note that these samples become rather small. Thus, the results presented in Figure 3 represent stylized findings that should be interpreted with caution. However, they enable us to illustrate how potential sample selectivity may distort earlier findings.

Figure 3 shows that the gender gap (black solid line) in major choice is lowest among a highly selective sample of individuals with a very high study propensity (above the 75th percentile), whereas it is highest for the group of individuals with a very low study propensity (under the 25th percentile). Furthermore, Figure 3 shows that the explanatory power (black dashed line) of the specification also tends to differ over the study propensity. While the contribution of constraints (gray dashed line) to sex segregation also varies by study propensity, the explanatory power of preferences (gray solid line) does not change substantially across potential sample selectivity. Overall, we find evidence that previous findings of sex segregation could be biased by neglecting the heterogeneity of groups with a high/low study propensity. This finding may stimulate future work in the field of educational sociology.

Figure 3. Gender gap and explanatory power of the Ochsenfeld (2016) specification across sample inclusion probability. Data: NEPS-SC4, NEPS-SC5, Micro Census, HIS graduate panel studies.