3.1 Criticism 1: Esser and Seuring’s treatment definition differs from previous studies

A first limitation of ES’s analysis concerns the scope of institutional variation covered in their analysis. Their analysis exploits institutional variation across 13 of the 16 German federal states (Bundesländer).^[1] Notably, all of these states have between-school tracked secondary education systems that start to track in grade 5, when students are about 10 years of age. Hence, all of the 13 education systems included in ES’s analysis fall into the most strongly or early-tracking systems by international standards. Their key institutional variable accordingly is neither the timing of tracking nor the number of school tracks – the two features that have received the greatest attention in (cross-national) research – but the strictness of tracking, as captured by a qualitative grouping of the German federal states into three groups based on whether track recommendations at the end of primary schools are binding (or can be overridden by parents) and the extent to which the rules for these recommendations are standardized. As such, ES’s analysis covers only a small portion of the international variation in secondary education systems and cannot provide direct evidence on the performance of comprehensive education with no tracking in secondary education.

While this point may seem obvious, we believe that it does not receive sufficient attention in ES’s contribution – particularly since the authors frame their work as a rather comprehensive and wholesale refutation of the previous literature and the standard position, including work that takes a broad cross-national perspective.

3.2 Criticism 2: Esser and Seuring’s measures of classroom composition are problematic

Given their emphasis on classroom homogeneity as the key variable mediating the effect of strict tracking on student achievement, direct measures of classroom composition are central to ES’s analysis. ES characterize classroom composition in terms of both socio-economic status (SES), proxied by the maximum parental ISEI-88 score, and student ability (ABL), measured using the NEPS-MAT test, a short test of reasoning capabilities designed for the NEPS. For both dimensions, a first measure captures the classroom average, labelled NSES/NABL (with the prefix “N” for “niveau”), and a second measure captures homogeneity as the additive inverse of the within-classroom standard deviation, labelled HSES/HABL. We plot their distributions in Fig. A3 in the online appendix.

While these classroom characteristics are conceptually straightforward, there are two major issues with their implementation in ES. First, ES compute one unique value of each measure per classroom, using all students with complete information. This means that a student’s own SES or ABL score is used in calculating the respective classroom averages assigned to her. This can lead to serious bias by creating spurious correlations between individual characteristics and classroom composition (Angrist 2014). It is generally preferable to construct peer averages using a “leave-i-out” approach, where a student’s own value in question is not included in the calculation of the classroom average (Angrist 2014).

Second, the number of students with complete information is very (almost certainly too) small for many classrooms. We illustrate this in Fig. A1 in the online appendix, which plots the number of students with valid information on SES and ABL for each classroom: in more than half of the classrooms the composition measures are calculated using fewer than 10 students (average class size in Germany is 22). Not only does this create problems of measurement error, failure to implement a “leave-i-out” approach will also be particularly consequential for small classrooms. Both issues become very obvious when one considers the non-negligible number of classroom where there is only one student with valid information. In these cases, the classroom averages, NSES and NABL, are identical to the individual-level measures, SES and ABL. Even more problematically, ES set the within-classroom standard deviations of SES and ABL to zero for these classrooms, so the homogeneity measures (HSES and HABL) are maximal by construction.^[2]

Even though these are serious issues and, at the very least, classrooms with only one student observation should be excluded from an analysis focusing on the role of classroom composition, we stick to ES’s variable and sample definitions in our replications below to avoid differences due to different samples. We have, however, re-run the main regressions with improved “leave-i-out” measures of the classroom averages and excluding classrooms with only one student observation from the sample (as it is impossible to construct reasonable estimates of classroom homogeneity in these cases and ES’s assumption of maximal homogeneity is clearly untenable). These additional regressions, reported in Tab. B1 to B3 in the online appendix, provide even less support for ES’s claims than the regressions that we present in the main article (see in particular Section 3.3).^[3]

3.3 Criticism 3: Classroom homogeneity shows no unambiguous relationship with student performance and does not mediate the effect of strict tracking

We begin our reanalysis of ES with the results pertaining to the hypothesized mediation of the effect of (strict) tracking through classroom homogeneity, which are presented in Tab. 4 of their paper. Our reanalysis, presented in Tab. 1, deviates from ES only by adding several model specifications that are missing from their analysis, despite being crucial for testing their claims.^[4] Reassuringly, all models presented in ES are exactly reproduced in our reanalysis.

As discussed above, ES emphasize the importance of controlling for potential selection effects (i. e., differences in student composition between states with different strictness of tracking), yet curiously their models only include student-level ability and SES when they also include the corresponding classroom-level aggregates. We agree that good selectivity controls are important and therefore include individual-level SES and ABL in all specifications (save for baseline Model 1). This means that we report a key specification missing from ES’s analysis: Model 2 is a controlled baseline model that includes individual ABL and SES as selection controls but does not yet include any of the compositional measures, thus allowing the reader to inspect the effect of tracking before moving to the mediation analysis. Comparison of Models 1 and 2 shows that the inclusion of ABL and SES alone noticeably attenuates the strictness of the tracking coefficients, leaving the contrast between the least strict (T1, the reference category) and the medium strict (T2) states significant at a 10 % level only. ES misattribute this attenuation to the classroom composition measures because they add the classroom- and student-level measures jointly in one step.

We now turn to the potential mediation of the (remaining) effect of tracking through classroom composition. Somewhat unconventionally, ES’s assessment of the mediation hypothesis is based exclusively on “interactive” specifications where the effects of homogeneity (i. e., HABL/HSES) are allowed to vary by the corresponding classroom average (i. e., NABL/NSES). We follow a more standard approach and enter the composition measures sequentially and additively before including interactions. ES’s central claim is that the positive effect of strict tracking operates through the creation of more homogeneous classrooms. As the most straightforward test of this claim, Model 3 in Tab. 1 only adds the measure for homogeneity in terms of ability, HABL, to the controlled baseline model. This key specification is missing from ES. Strikingly, the addition of HABL leaves the coefficients on the strictness of tracking indicators virtually unchanged (if anything, the effect of T2 increases), providing strong evidence against mediation of the effect of strictness of tracking through HABL. The inclusion of further compositional measures in the remaining columns does lead to some attenuation of the strictness of tracking coefficients, but this is driven by the classroom averages (NABL and NSES). The homogeneity measures (HABL and HSES) emphasized by ES play essentially no role in this attenuation.^[5]

More than that, the results in Tab. 1 do not even show a clear-cut positive relation between performance and classroom homogeneity. While the coefficient on HABL is positive in Model 3, Model 5 shows that when HABL and NABL are added jointly, the coefficient on homogeneity drops to zero, indicating that the positive association between HABL and performance in Model 3 is spurious and comes from more homogeneous classroom typically being classrooms with higher average ability levels. Model 6 adds the interaction between NABL and HABL (as noted above, ES only show such interactive specifications and do not consider the kinds of additive specifications that we report in columns 3–5, 7, and 9). While the coefficient on HABL is indeed positively signed and statistically significant in Model 6, the presence of the interaction term implies that it no longer corresponds to a general or average “effect”. As is well-known (see, for example, Kam and Franzese 2007), it now has to be interpreted as the effect of HABL for classrooms with NABL = 0, that is, for classrooms with minimal average ability (because, following ES, the classroom composition measures are rescaled to range from 0 to 1). The strongly negative interaction term implies that the predicted effect of homogeneity starts to turn negative in the middle ranges of the classroom ability distribution (more precisely, when NABL is around .59 because .96/1.62 ≈ .59), which is consistent with the near-zero average effect from Model 5. ES do not make these important qualifications when discussing the corresponding estimates, nurturing the misleading impression that their results are consistent with the general achievement-enhancing effect of homogeneity suggested by the MoAbiT.^[6]

Altogether, these results – at most – support a performance-enhancing effect of homogeneity for a subset of (low-ability) classrooms, with essentially no evidence that this effect can account for the relationship between strict tracking and performance. Importantly, even the conditional positive effect of homogeneity for low-ability classrooms becomes highly questionable once we address the problems with ES’s classroom composition measures noted above (see Section 3.2): Tab. B1 in the online appendix shows that not only the average effect of HABL (in Models 5 and 9) but also its main effect and interaction with NABL (in Models 6 and 10) are essentially zero when the classroom composition measures are calculated correctly (i. e., when NABL is constructed using the “leave-i-out” approach and single-student classrooms are excluded instead of adopting ES’s assumption that these classrooms are maximally homogeneous).

3.4 Criticism 4: The relationship between strict tracking and student performance is largely accounted for by initial performance

While the analysis of the previous section casts serious doubt on the mediating role of classroom homogeneity, it does not explain why seventh-graders tend to perform better in states with stricter tracking. In this section, we show that students in strictly tracking states perform better already at the start of secondary school (grade 5) and that this initial advantage explains the largest portion of their performance advantage in grade 7. Hence, the achievement advantage of students in strictly tracking systems appears to reflect selection and/or anticipation effects rather than the positive effects of strict tracking on competence development in secondary school emphasized by ES.

As student bodies differ substantially between German federal states, the key challenge for any study aiming to identify causal effects of schooling policy using between-state policy variation is to convincingly control for these differences. ES rely on a rather small control set that consists only of indicators for gender, migration background and early childcare attendance, next to the two mentioned continuous measures of family background (SES) and ability (ABL). The latter is based on a very short general cognition (reasoning) test administered at the beginning of grade 5.^[7] The outcome variable, ACH, measures grade 7 achievement by averaging two elaborate domain-specific competence tests in mathematics and reading. Importantly, similar competence tests in mathematics and reading were also administered at the beginning of grade 5.^[8] This makes it possible to control for achievement differences at baseline – when there has been only minimal exposure to tracked secondary education – in the regressions for grade 7 achievement. Somewhat surprisingly, ES do not make use of this information in their paper.

Tab. 2 presents this natural robustness check.^[9] Models 2, 3, and 4 repeat key specifications from Tab. 1 (Models 2, 5, and 10, respectively) with grade 5 achievement, Gr. 5 ACH, added as an individual-level control for initial differences in student achievement. For reference, in Model 1 we repeat the controlled baseline model without grade 5 achievement from above (i. e., Model 2 in Tab. 1). Note that we now z-standardize the individual-level measures ABL and SES to ease comparisons with Gr. 5 ACH.^[10] Comparing Models 1 and 2, we see that grade 5 achievement turns out highly predictive of grade 7 achievement – in fact, much more so than ES’s ability control, ABL. More importantly, controlling for grade 5 achievement leads to a drastic attenuation of the tracking coefficients: the difference in grade 7 achievement between the most and least strictly tracking states drops to .08 standard deviations (SD) – merely one third of the difference before controlling for initial achievement (see Model 1) – and the difference between the medium and least strictly tracking group declines to a negligible and statistically insignificant .03 SDs. This indicates that the largest portion of the relationship between tracking and grade 7 performance is attributable to students in strictly tracking states being stronger performers already at the start of secondary school.

This conclusion is reinforced by a direct (placebo) test for pre-treatment differences between more and less strictly tracking states in Models 5 to 7. Specifically, we now use grade 5 rather than grade 7 achievement as the dependent variable. If strict tracking enhances student performance by creating homogeneous learning environments, as stipulated by the MoAbiT, we should see performance advantages emerge over the course of secondary education because of continued exposure to the supposedly beneficial homogeneous learning environments. However, Models 5 to 7 show that the positive relationship between tracking and achievement already exists in grade 5 when exposure to tracked programs has been minimal, and that it is about as strong as in grade 7 (in terms of SDs of the outcome variable).^[11]

Notes: Shown are coefficients and, in parentheses, standard errors from multilevel mixed-effects linear regressions with random intercepts at the class level. The dependent variable is the average of grade 7 (Models 1–4) or grade 5 (Models 5–7) math and reading test scores, standardized to mean zero and unit standard deviation. T2 = Medium strictness of tracking; T3 = High strictness of tracking; Gr. 5 ACH = Grade 5 achievement; ABL = cognitive abilities; SES = socio-economic status; NABL = classroom average cognitive abilities; HABL = classroom homogeneity cognitive abilities; NSES = classroom average socio-economic status; HSES = classroom homogeneity socio-economic status; ECEC = early-childhood education and care attendance in months. Grade 5 scores are missing for three students, explaining the difference in the number of observations compared to Tab. 1. Stars indicate significance levels: * p < 0.10, ** p < 0.05, *** p < 0.01. These findings seem to leave two (non-mutually exclusive) possibilities. The first is that the association between strictness of tracking and student performance is spurious in the sense of being attributable to unobserved factors that affect student achievement both in grade 5 and in grade 7. A second possibility is that tracking does exert a causal effect on student performance but that this effect occurs already prior to the onset of actual tracking. In particular, strict tracking might create performance incentives and induce greater educational investments among primary school students, their teachers, and/or their parents. In a recent study based on NEPS data, Bach and Fischer (2020) provide evidence supporting this interpretation. Clearly, such anticipation effects are very different from the homogeneity mechanism emphasized by ES.

3.5 Criticism 5: Evidence for interaction effects between homogeneity and strict tracking is weak at best

In this section, we revisit ES’s moderation hypothesis that predicts stronger effects of classroom homogeneity on achievement in more strictly tracking systems. In a nutshell, ES’s theoretical argument for such a moderation effect is that strict tracking facilitates the optimal tailoring of curricula, teacher education, and other factors to the different ability levels of students.

Empirically, such moderation should manifest in a positive interaction between the strictness of tracking and classroom homogeneity. Unfortunately, the results presented in Tab. 5 in ES make it extremely difficult to tell whether such an interaction exists, as the authors only present specifications that include three-way interactions between tracking and the two measures of classroom composition. In their discussion of the results on p. 293, ES point to the (positive and statistically significant) two-way interactions between strict tracking and classroom composition in terms of ability (i. e., T3*NABL and T3*HABL). However, the immediate theoretical significance of these two-way interactions is very limited, because due to the included three-way interaction (T3*NABL*HABL), they again refer to specific conditional effects: the coefficient on T3*HABL tells us how the “effect” of ability homogeneity for classrooms with minimal average ability differs between the most and least strictly tracking states. By the same token, T3*NABL captures how the effect of average ability for classrooms with minimal homogeneity differs between the two systems. These highly specific effects for classrooms located at the very extremes of the classroom composition distributions tell us little about a possible general moderating effect of tracking.

To see whether ES’s moderation hypothesis is actually supported by the data, we estimate simpler specifications that do not include three-way interactions. Tab. 3 presents two blocks of models, each comprising four specifications. Models 1 to 4 use the same of set of controls as ES, so results can be directly compared. Models 5 to 8 include grade 5 achievement as an additional control, which, as shown above, is important to control for selectivity. The first model in each sequence is a baseline specification that does not include any interactions between strictness of tracking and the classroom composition measures. We then add the interactions between tracking and ability composition (second) or SES composition (third), while the fourth model includes all two-way interactions.

When using the same set of controls as ES, we find some evidence for the predicted moderation effect: both Model 2 and 4 show (marginally) statistically significant positive interactions between strict tracking (T3) and the two measures of classroom ability composition (NABL and HABL) with meaningful effect sizes. However, the second set of models shows that these effects are not fully robust to controlling for grade 5 achievement, which the previous section has shown to be key in controlling for selectivity. The relevant interaction terms, T3*NABL and T3*HABL, remain positive, but their magnitude declines substantially and only one estimate – the T3*NABL interaction in Model 6 – continues to reach statistical significance. Hence, we conclude that the evidence for ES’s moderation hypothesis, especially for the key theoretical variable HABL, is suggestive at best – certainly not strong enough to justify the bold wording used by ES, who conclude that “the results correspond almost perfectly to the theoretical model” (“Die Befunde entsprechen nahezu lückenlos dem theoretischen Modell”; Esser & Seuring 2020: 295, authors’ translation).