Noise or News? Learning about the Content of Test-Based School Achievement Measures

1 Introduction

Proponents of school choice argue that the structure of the public educational system – where education is mainly provided by government with substantial monopoly power and largely no competition – leaves parents with limited choice among schools. They suggest that this may result in a disconnect between school quality and parents’ preferences. A growing literature in economics suggests that education reforms that expand school choice for students (e.g. open enrollment systems, magnet and charter schools, private school vouchers, and expanded public school choice for students in poorly performing schools) could improve educational outcomes such as test scores, educational attainment, expenditures, efficiency, etc. They argue that expanding school choice will improve access to higher quality schools, especially for relatively disadvantaged children. Moreover, expanding options for students will induce competition in education market that will put pressure on poorly performing schools to either improve their performance or to shrink as families “vote with their feet” (Friedman 1955; Borland and Howsen 1992; Becker 1995; Dee 1998; Hoxby 2003; Belfield and Levin 2003, Misra, Grimes, and Rogers 2012).^[1]

One suggested approach to make these policies more effective is to increase parents’ access to information about school quality and performance so they can benefit from the opportunity to attend sought-after schools (e.g. Lai, Sadoulet, and de Janvry 2009, Allen and Burgess 2013).^[2] A growing body of evidence suggests that test-based information about school performance affects parents’ school choice decisions and could consequently have real impacts on students’ educational outcomes (e.g. Hastings, Van Weelden, and Weinstein 2007; Hastings and Weinstein 2008; and others). A more recent study by Friesen et al. (2012) examines the effect of the public release of information about school achievement on parental choice behavior. Using data that is very similar to this study, they find that the public release of test-based information about school achievement substantially affects the mobility decisions of students in the Lower Mainland of British Columbia, Canada.^[3] Their results suggest that some parents respond to information soon after it is publicly disseminated, and continue to respond to subsequent releases in the following years. However, it remains an open question and a concern whether test-based measures of school achievement, that are one of the main factors that affect student mobility, actually provide parents with meaningful information about school quality and performance, or whether they are too imprecise to do so. This is especially important in jurisdictions, such as the one studied by Friesen et al. (2012) and this paper, where publicly disseminated test-based information about schools gets updated annually and is found to be one of the important factors that affects parents’ school choice decisions and education policy.

The imprecision in test-based school achievement measures also has important implications for jurisdictions with school accountability systems. As Kane and Staiger (2001, 2002b) also point out, to the extent these test-based school achievement measures are used in school accountability systems to evaluate schools’ performance, high degree of imprecision in these measures can wreak havoc in school accountability systems and subject schools to sanctions or rewards based on factors that are beyond their control.^[4] Moreover, to the extent these measures are used to identify best strategies of school reform, the imprecision in these measures could mislead policy makers and educational authorities in their evaluation of the merits of different education practices.

Kane and Staiger (2001, 2002a) suggest two potential sources of transitory variation in school average test scores. The first is sampling variation, which is a purely statistical phenomenon. Different cohorts of students attending a school in different years can be considered as random samples drawn from a local supplying population. Therefore, there will be year-to-year variation in a school’s test score solely arising from random year-to-year variation in the composition of its student body (i.e. sampling variation). The magnitude of sampling variation depends on school size: it is decreasing in the number of students who write the test. Since the enrollment is usually quite small in an average elementary school (41 in grade 4 and 50 in grade 7 in our sample) the variation induced by sampling variation could be quite serious. The second source of transitory variation is idiosyncratic factors that generate non-persistent differences in schools’ mean test scores; for example, a school-wide illness at the time of the exam, the presence of a few troublemaking students in a class, teacher absenteeism due to personal issues, etc. Following Kane and Staiger (2001, 2002a), we refer to this second source of transitory variance as “other transitory factors” to distinguish it from sampling variation.

There is evidence that suggests mobility decisions could be costly for students and their parents and might not contribute much to students’ academic performance. Cullen, Jacob, and Levitt (2005) find little or no evidence that students systematically achieve higher school quality (measured by value-added of schools to student outcomes) by choosing a non-neighborhood school. Their results also suggest that the main effect of Chicago’s open-enrollment program has been to induce segregation. Kane and Staiger (2001) find little evidence that schools with significant improvement in their test scores over time improved on any measures of student engagement. Hanushek, Kain, and Rivkin (2004) find that mobility generates negative externalities for students that entail a substantial cost for movers and non-movers alike. Cullen, Jacob, and Levitt (2006) exploit randomized lotteries that determine high-school admission and find little evidence that lottery winners (who attend what appear to be substantially better high schools; e.g. schools with higher achievement levels (and gains), higher graduation rates, and lower poverty rates) receive any benefits across a wide variety of traditional academic measures (e.g. standardized test scores, graduation, attendance rates, course-taking patterns, and credit accumulation). They also find some disadvantages associated with winning a lottery and therefore attending schools with higher achieving peers, such as lower class ranks throughout high school and higher chances of dropping out.

Wolf et al. (2007) study the federally sponsored voucher program in Washington, DC, and find that using a voucher to attend a private school had no impact on average reading or math scores. Mayer et al. (2002) also find similar results using experimental voucher evidence from New York City. Using data from 150 municipalities in Chile, Hsieh and Urquiola (2006) find that communities where private schools grew by more after the introduction of a comprehensive school voucher program did not experience a faster increase in their average test score, and in fact had worse average repetition and grade-for-average measures. Consistent with the findings of Cullen, Jacob, and Levitt (2005) and Fiske and Ladd’s (2001), they also find evidence that similar voucher programs led to increased segregation. Booker et al. (2007) study the impact of charter school attendance in Texas and find that students attending a charter school experience a poor test score growth in their initial year and it takes about 3 years for them to recover from this initial disruption. Bifulco and Ladd (2006) examine the performance of charter schools in North Carolina and find that students in charter schools make considerably smaller achievement gains than they would have in public schools. Their results attribute 30% of the negative effect of charter schools to high student turnover rates.^[5] A related literature examines the effect of information about school-average achievement on real estate prices (Black 1999; Gibbons and Machin 2003; Kane et al. 2003, Figlio and Lucas 2004; Kane and Staiger 2006, Fiva and Kirkebøen 2010; Ries and Somerville 2010; and others). These studies generally find a positive relationship between information about school performance and housing prices.

We investigate the content of publicly disseminated information about schools’ average test scores to determine whether they provide parents with meaningful information about persistent inter-school differences in achievement and school quality. Since there is evidence that suggests test-based school achievement measures are one of the factors that shape parents’ mobility decisions, and since the studies reviewed above suggest that these decisions could be costly with negative, weak, or no academic benefits to students, it is important to investigate the degree of imprecision in these test-based measures that could contribute to parents’ mobility decisions. If these decisions are heavily based on noisy measures of school achievement, they could impose a substantial cost on parents and make them and their child considerably worse off. Moreover, making costly location decisions to obtain access to (currently) high-achieving schools could generate a lot of costly churning in the real estate market, and again making families worse off.

This is particularly important since Friesen et al. (2012) find that “parents in low income neighborhoods are most likely to alter their school choice decisions in response to new information [that is based on test-based school achievement measures that we are also examining in this study].” These are potentially families that are less likely to obtain information about school quality through other sources (for example due to their more limited access to well-informed networks) and are therefore more likely to put a large weight on these test-based measures to inform their school choice decisions. Moreover, given their socioeconomic background, mobility decisions are potentially more costly for these families and will more negatively affect them if they are heavily shaped by highly transitory information about school-average achievement.

Finally, if test-based measures of school achievement are strongly influenced by transitory factors, to the extent they are consulted to identify best practices in education, they will only confuse or mislead policy makers and educational authorities, which could be also quite costly. It should be emphasized that the use of test-based accountability systems is also one of the important motivations behind examining the degree of imprecision in test-based school achievement measures. However, since in British Columbia (BC), and in Canada in general, there are currently no test-based school accountability systems, our main motivation for this study is the effect of test-based information on parents’ school choice decisions. This is particularly important in the jurisdiction under study, British Columbia, where school choice has been one of the priorities of the provincial government and the commitment to it is mentioned as part of the BC education plan.^[6]

We extend the work of Kane and Staiger (2002a) by laying out a more rigorous statistical model that shows more clearly the underlying model and the assumptions under which one can separately measure the effect of sampling variation and other transitory factors on the cross-sectional variance of school mean test scores. We find that sampling variation and other transitory factors are a significant source of observed variation in schools’ average test scores even among substantially large schools, and pooling student test scores across grades and years to dampen the volatility does not resolve this issue.

2 Previous Literature

Not much is known about the statistical properties of school-level measures of test-based achievement. Kane and Staiger (2001) find that test-based elementary school rankings in North Carolina resemble a lottery and argue that small within-school sample size that exacerbates the sampling variation is the main cause. Using methods developed by McClellan and Staiger (1999) for the analysis of hospital performance measures, they estimate that for an elementary school of average size in North Carolina, 28% and 10% of the variance in grade 5 reading scores is due to sampling variation and other transitory factors, respectively. They also decompose the variance of school-level test scores and find that gain scores tend to have “less signal variation and more variation due to non-persistent factors” than test score levels. Therefore, they caution against the use of gain scores in evaluating schools’ or teachers’ performance.

Using data from North Carolina, Kane and Staiger (2002a) examine the effect of sampling variation and other transitory factors on between-school variance in the average test scores, gain scores, and changes in the average test scores between 2 consecutive years. They estimate that for an average school in their sample, sampling variation is responsible for 14–15% of the variation in fourth-grade mean math and reading test scores. They also find that the effect of sampling variation on between-school variance in average gain scores (i.e. changes in student’s test scores between two consecutive grades in a given school) is twice as large as that of average test score levels. For schools of different size, they find that transitory factors explain 10–20% of the between-school variance in combined reading and math scores in grade 4, 28–58% of the variance in gains between third- and fourth-grade scores, and 73–80% of the variance in annual change in fourth-grade scores.

Mizala, Romoguera, and Urquiola (2007) find evidence that test-based rankings mostly reflect socioeconomic status. They also argue that the more highly correlated are test-based achievement and socioeconomic status, the lower is the year-to-year volatility in rankings based on test-based school achievement measures. Finally, using evidence from Chile’s P-900 program, Chay, McEwan, and Urquiola (2005) show that noise and mean-reverting shocks in schools’ test scores complicate evaluation of policies to improve school’s quality. They find that for a median-sized school, 33% and 21% of the variance in language and math scores, respectively, are due to transitory factors. They argue that such noise in mean test scores might limit the ability to identify “good” schools from “bad” schools.

3 A Simple Model

It is often costly to determine school quality, which makes school choice decisions difficult for parents. Although a parent seeking information about a school’s quality might choose to meet the principal, speak with neighbors, and visit the facilities, idiosyncrasies in the fit between child and school make true quality difficult to observe. In the absence of perfect information, parents must therefore form beliefs about school quality and base their school choice decisions on those beliefs. Recent studies suggest that signals received by parents in form of school-level test-based achievement measures play an important role in shaping these decisions.

We present a simplified model of school choice that accounts for uncertainty about school quality and focuses attention on the effects of noisy information based on school average test scores.^[7] Assume parent i’s utility (U_is) depends on the quality (q_s) of their child’s school s,

where q_s represents an index of school characteristics that determines parents’ utility, such as teacher quality, peer quality, school’s state of technology, and class size. εis∼N0,σε2 represents how utility of individual i from school s differs from the tastes of the average individual. We assume that parents don’t observe q_s. Given the characteristics of a school that are observed prior to any test-based information, parents hold a prior about the quality of school that is normally distributed with mean Xs′β and precision hqi (i.e. qs∼NXs′β,hqi−1. This suggests that parents use Xs′β, their expectation of the distribution, as their guess for the quality of school, where Xs′β is a combination of directly observable school characteristics Xs such as neighborhood income and the demographic composition of the student body, and β is the vector of weights that parents assign to these characteristics. The precision of the prior is known to parents and is the same across schools in parents’ choice set. However, it is allowed to vary across parents to reflect the idea that some (e.g. new immigrants) may have less precise beliefs about school quality than others (e.g. a native born individual who has lived in the area for many years).

If observed school characteristics are the only source of information about school quality for parents, a parent chooses school s if

where c_is measures the direct (e.g. tuition) and indirect (e.g. commuting distance) costs of attending school s. Given the assumptions made before, this implies that the average parent chooses school s if

Now suppose that parents also receive a noisy signal S_s regarding school’s quality,

where ηs∼N0,hηis−1. We have in mind that the signal is based on standardized test scores aggregated to the school level. The noise component of the signal η_s has zero mean, implying that test scores provide unbiased information about school quality. The true precision of test scores as a signal of school quality, hηis, varies across schools and parents. This reflects the idea that the precision of average test scores depends on the extent to which they are influenced by sampling variation and other transitory factors, and therefore varies across schools. The precision of average test scores is also allowed to vary across parents reflecting the possibility that different parents might have different perceptions regarding the precision of average test scores as a signal of school quality, which is influenced by factors such as parents’ socioeconomic background, access to well-informed social networks, etc. We assume parents do not observe the true precision of average test scores, hηis, since parents cannot perfectly measure the extent to which these test scores are influenced by transitory factors. Moreover, for a given parent, the precision of test scores as a signal of school quality is the same across different schools, since parents cannot measure how the effect of transitory factors on average test scores varies across schools. Therefore, for parent i,ηs∼N0,hηi−1,∀s∈Σ.

Parents are assumed to assimilate new information via Bayesian learning.^[8] After observing the test-based signal, parents update their expectations about each school’s quality using Bayes’ rule. Since the true signal precision, hηis, is not known to parents, they update their expectation using what is known to them regarding the precision of signal, hηi. Therefore, the normal learning model indicates that the expected quality of school s for the representative parent after receiving the signal is

Equation [5] suggests that parents’ revised expectation of school quality is a precision-weighted average of the signal and their prior expectation; the more optimistic a parent’s perception of precision of test scores as a signal of school quality relative to prior information, the greater the weight that parent would place on test scores. Prior information regarding school quality, Xs′β, will continue to dominate the beliefs about school quality for parents who believe test scores are not a good signal of school quality (parents with low hηi).

Defining θ=hηihηi+hqi−1 as the weight that parents assign to test scores, and rearranging eq. [5], we can write the representative parent’s expected utility as

We can define Ss∗=Ss−Xs′β to be the test score “shock,” which represents the new information the signal Ss provides to parents beyond what they already knew. Parents will now choose to enroll their child in school s if

Since parents cannot distinguish between schools in terms of precision of their average test scores as a signal of school quality, the weight they assign to test scores, θ, is the same for different schools. This lack of knowledge about the true precision of test-based signals, and the difference in precision of test scores across schools might hinder parents’ ability to make the optimal decision. For instance, given the true precision of test-based signals, if signals received for school s and k are not equally precise and informative (i.e. θs≠θk), we could have

where θs=hηishηis+hqi−1 and θk=hηikhηik+hqi−1. This suggests that the optimal decision is to choose school k over school s, while due to lack of knowledge regarding the true precision of test-based signals, as eq. [7] suggests, it might seem optimal to choose school s over school k. Even if the signals received regarding the quality of school s and k are equally precise, but parents’ perceived precision of the signal is larger than the true precision of the signal (i.e. θs=θk<θ), we could get a similar result. Therefore, the larger the effect of transitory factors on test-based school achievement measures, the higher the probability that parents school choice decision will be misled by this information and they become worse off. It is important not only to know the degree of imprecision in test-based signals of school quality, but also to be able to compare this degree of imprecision across schools in the choice set. For instance, if small schools tend to experience higher volatility in their average test scores due to transitory factors, and if parents are not well-informed about this, it could lead to schools choice decisions that are not optimal to make.^[9]

4 Institutional Background and Data

5 Volatility in School Mean Test Scores

We begin with a descriptive analysis of variability in school-average test scores. Then, using a simple statistical model built on the work of Kane and Staiger (2002a), we decompose the variation in school-average test scores into two different components: the variation that is due to sampling variation and the variation that is due to other nonpersistent factors.

Figures 1 and 2 portray the distribution of regression-adjusted average grade 4 and grade 7 reading and numeracy test scores by grade enrollment between 1999 and 2006.^[17] It is immediately apparent that there is more variation in average test scores among small schools than big schools, while there does not seem to be a significant difference in average performance by school size. Moreover, Figures 3 and 4 suggest that small schools also experience more year-to-year fluctuation in their grade 4 and grade 7 average test scores compared to big schools.^[18] The most likely cause is sampling variation, since its magnitude is a decreasing function of school size. We return to this hypothesis below.

Figure 1:

Grade 4 Adjusted Reading and Numeracy Average Test Scores by enrollment level (1999–2006).

Figure 2:

Grade 7 Adjusted Reading and Numeracy Average Test Scores by enrollment level (1999–2006).

Figure 3:

Grade 4 Adjusted Reading and Numeracy Gains by enrollment level (1999–2006).

Figure 4:

Grade 7 Adjusted Reading and Numeracy Gains by enrollment level (1999–2006).

Tables 2 and 3 further illustrate transitory year-to-year variation in grade 4 and grade 7 average test scores. If mean test scores are strongly influenced by transitory factors, a ranking based on these scores will be similar to a ranking based on a pure lottery. The upper panel of each table uses FSA reading scores and the lower panel uses FSA numeracy scores to compare school rankings under different scenarios.

Column 1 illustrates the case where school-average test scores are completely stable over time. Under this scenario, 20% of schools will always appear in the top 20% of the distribution of school-average test scores, and 80% of schools will never appear in the top 20%.^[19] In contrast, column 2 illustrates the case where schools are assigned to different percentiles based on a pure lottery: all schools have an independent 20% probability of appearing in the top 20% of the distribution of school-average test scores in each year. Under this scenario, we expect nearly 17% of schools to never appear in the top 20%, only 0.1% appear in the top 20% 6 times out of 8, and none to appear in the top 20% in all 8 years.

Column 3 ranks schools based on their actual school-average FSA scores. The data lie somewhere between the two extremes of complete stability and a lottery. Looking at grade 4 results, 46% of schools never appear in the top 20% of average FSA reading scores and 2.2% appear in the top 20% for all 8 years; these numbers are 39% and 6% for grade 7, respectively. The results are almost identical for numeracy scores. However, one should bear in mind that part of the stability in ranking based on simple FSA levels may reflect the unchanging characteristics of the student populations feeding the schools and not necessarily due to persistent educational practices in the schools. Controlling for students’ observable characteristics, columns 4 results more closely resemble a lottery. Columns 5 and 6 also control for students’ observable characteristics, but restrict the sample to schools with enrollment less than 30 and 20 students, respectively. The results suggest that among smaller schools, the volatility in schools-average test scores is even more similar to a lottery. The most likely cause is sampling variation, whose magnitude decreases with sample size.

Finally, as column 8 illustrates, school gain scores (the change in school-average test score from 1 year to the next) are even more volatile than score levels, and consequently rankings based on gain scores are even more similar to a lottery.^[20] This is potentially partly due to the fact that gain scores are based on differences in test scores over 2 consecutive years, which magnifies the effect of nonpersistent factors on variation in schools’ test scores and removes any permanent components.

5.1 Sampling Variation

We can think of successive cohorts of students entering a school over time as random samples drawn from a local population feeding that school. Therefore, even if the feeding population does not change, school-average test scores will vary from year to year due to random variation in the sample of students, or sampling variation. Given the relatively small number of students in an average school (41 in grade 4 and 50 in grade 7 in our data), and given the fact that only 10–15% of the variation in students’ test scores in our data is between schools, sampling variation can substantially change the position of a school in the distribution of school average test scores.

There are two factors that determine the magnitude of sampling variation: the number of students who write the test in the school, and the heterogeneity of individual test scores in the feeding population. Conditional on all other factors that could influence a school’s average test score over time, smaller schools will experience more volatility in their mean test scores due to sampling variation than bigger schools.

In what follows, we use a simple model and apply basic sampling theory to estimate the expected amount of variance in a school’s average test score due to sampling variation.^[21] Consider a simple model of test score determination:

Here, Yijt denotes student i’s test score at school j in year t, λj is the school-level time-invariant component of a student’s score, δjt is the school-level time-varying component (decomposed below into a persistent component and a transitory component), θi is student-level time-invariant component (representing test-taking ability or smartness, socioeconomic characteristics, and other factors), and Uijt is a student-level transitory shock.

Assumption 1: Transitory shocks are idiosyncratic: Uijt∼iid0,σu2.

Assumption 2: Different cohorts of students entering a school in a given year are random samples drawn from a population feeding that school, so that θi∼iidμθ,σθ2 for the feeding population.^[22]

Given eq. [9], the average test score for a given school j in year t has the form

where Njt is enrollment level at school j at time t. The cross-sectional variance in average test scores over all schools in a given year is

Defining εijt=θi+Uijt, we can rewrite eq. [9] as

An unbiased estimator of the within-school variance of individual test scores at a given school is

As eq. [13] suggests, the source of variation in student test scores within a given school is either due to differences in students’ time-invariant characteristics (test-taking ability, socioeconomic background, etc.), Sθ2, or due to idiosyncratic shocks affecting a student performance in the exam (all factors that could cause “random variation in health, motivation, mental efficiency, concentration, forgetfulness, carelessness, subjectivity or impulsiveness in response and luck in random guessing”^[23]), SU2.

Applying sampling theory, the expected variance in school-average test scores due to sampling variation is

Given Assumptions 1 and 2, Sθ2 and SU2 are unbiased estimates of σθ2 and σu2. Therefore, the higher is the variance induced by student-level transitory shocks (σu2), and the higher are differences in students’ time-invariant characteristics (σθ2) in the population feeding a given school, the larger will be the variance induced by year-to-year variation in the composition of students (i.e. sampling variation). On the other hand, the larger number of students within a given school in a given year (Njt) lowers the effect of sampling variation. This is driven by the fact that averaging over a larger number of students dampens the noise created by sampling variation.

Averaging eq. [14] over all schools gives the expected variance in mean test scores due to sampling variation for an average school:

Note this is an unbiased estimator of the last component in eq. [11].

The average within-school variance of FSA reading and numeracy scores is 0.88 and 0.83, respectively, for both grade 4 and grade 7. Since the overall variance of individual FSA reading and numeracy scores is normalized to one for each grade, this implies that within the average school, heterogeneity in students’ test scores is nearly as large as in the overall population. In other words, on average, the test scores of two students drawn randomly from a given school are likely to differ nearly as much as two students drawn from the population of BC students at large.^[24]

The estimated expected variance in mean FSA reading and numeracy scores for an average school due to sampling variation, from eq. [15], is 0.028 (reading) and 0.025 (numeracy) for grade 4, and 0.026 (reading) and 0.024 (numeracy) for grade 7.^[25] If we focus on schools of average size by looking only at the two middle quartiles of enrollment by grade, the overall cross-sectional variance of school-average FSA reading and numeracy scores is 0.13 and 0.17, respectively, for grade 4; and 0.13 and 0.19, respectively, for grade 7. It follows that for these average-sized schools, 21.5% and 14.7% of the total variation in fourth-grade mean FSA reading and numeracy scores, respectively, is due to sampling variation. For grade 7, the figures are 20% and 12.6%, respectively.^[26]

In order to gain a better understanding of the importance of sampling variation, we calculate the 95% confidence interval for an average school’s mean test score using the estimated expected variance due to sampling variation:

where YˉRe is the overall mean of school-average FSA reading scores. The confidence interval extends from roughly the 17th to 84th percentile of the distribution of school-average FSA reading scores for grade 4, and from the 19th to 80th percentile for grade 7. This implies that for an average grade 4 school in our sample, we cannot rule out with 95% probability that its average score the following year is as large as the 84th percentile, or as small as the 17th percentile, due to variability induced by sampling variation alone.

A more conservative measure is based on the 50% confidence interval for an average school’s mean test score given sampling variation, which measures the degree of variability induced by sampling variation with the probability of a coin toss. This confidence interval extends from the 36th to 63rd percentile of the distribution of school-average reading scores for both grade 4 and grade 7. This remains a very wide interval.

5.2 Other Transitory Factors

Sampling variation is only one of the transitory factors that can affect school-average test scores. There are other transitory shocks that generate nonpersistent changes in schools’ mean test scores in addition to sampling variation. Specifically, the school-level time-varying component of student test scores (i.e. δjt in eq. [9]) will also induce nonpersistent variation in school-average test scores. Real-world examples include the existence of a few troublemaking students in a class, a dog barking in the playground or construction noise on the day of the exam, a school-wide illness, or special chemistry between a teacher and a cohort of students, etc.

Following Kane and Staiger (2002a) and given the framework developed in Section 4.1, we apply an indirect method to estimate the nonpersistent variation in test scores attributed to these other transitory factors.^[27] First, we estimate the total variation in mean test scores due to all transitory factors by measuring the degree of persistence in change in test scores between two consecutive years. Then we back out the portion due to sampling variation. The remaining component is attributable to other transitory factors.

We first decompose the school-level time-varying effect on student scores in eq. [9] into two components:

[16]δjt=νjt+τjt

where νjt is the persistent component which is assumed to depend on last year value in addition to a new innovation each year (νjt=νjt−1+ρjt), and τjt is the transitory component. The persistent component reflects differences across schools in teacher quality, curriculum, facilities, etc. that persist over time, and the transitory component reflects differences driven by idiosyncratic shocks at the school-level discussed before.

Assumption 3: ρjt∼iid0,σρ2 and τjt∼iid(0,στ2).^[28]

The average test score for a given school j in year t will therefore have the form

[17]yˉjt=λj+νjt+τjt+θˉit+Uˉijt

[18]Δyˉjt≡yˉjt−yˉjt−1=ρjt+τjt−τjt−1+θˉit−θˉit−1+Uˉijt−Uˉijt−1

The correlation between test score gains this year and last year can be written as

[19]CorrΔyˉjt,Δyˉjt−1=CovΔyˉjt,Δyˉjt−1VarΔyˉjt×VarΔyˉjt−1=−Ej(στ2)+σθ2+σu2Ej1NjtVarΔyˉjt

The numerator is the negative of the total variance of all transitory factors and the denominator is the total variance of gain scores (Δyˉjt). Since we can calculate the left-hand side of eq. [19] directly, as well as VarΔyˉjt, we can recover the total variance of all transitory factors from their product. Subtracting from this our earlier estimate of the magnitude of sampling variation (from eq. [15]), we can back out the variance of all other transitory factors, Ej(στ2).

Tables 4 and 5 summarize the results for different quartiles of grade 4 and grade 7 enrollment for both FSA reading and numeracy scores. The analysis is done separately for different years, while the bottom panel of each table presents the results of the variance decomposition for all years combined. As expected, the estimated sampling variance is bigger for small schools than bigger schools and the magnitudes are stable over time. The estimated variance of other transitory factors however varies over time, which is due to the random nature of the events that generate them.

Table 4:

Decomposition of variance in schools’ mean FSA scores – grade 4.

School enrollment quartile	Average size	Total variance	Sampling variance	Other nonpersistent variance	Total proportion nonpersistent	Total variance	Sampling variance	Other nonpersistent variance	Total proportion nonpersistent
		Grade 4 FSA Reading Scores				Grade 4 FSA Numeracy Scores
Year = 2000
Bottom quartile	21	0.156	0.051	0.042	0.600	0.175	0.045	0.054	0.571
Two middle quartiles	41	0.100	0.022	0.020	0.435	0.126	0.022	0.026	0.380
Top quartile	66	0.081	0.013	0.014	0.343	0.105	0.013	0.010	0.225
Year = 2001
Bottom quartile	20	0.202	0.049	0.036	0.424	0.267	0.045	0.048	0.349
Two middle quartiles	40	0.151	0.022	0.026	0.326	0.187	0.021	0.037	0.317
Top quartile	68	0.109	0.013	0.012	0.234	0.137	0.012	0.014	0.197
Year = 2002
Bottom quartile	20	0.221	0.049	0.033	0.374	0.276	0.043	0.043	0.313
Two middle quartiles	39	0.137	0.023	0.020	0.318	0.166	0.022	0.026	0.295
Top quartile	65	0.104	0.014	0.004	0.182	0.129	0.013	0.004	0.140
Year = 2003
Bottom quartile	20	0.178	0.051	0.037	0.499	0.234	0.042	0.051	0.402
Two middle quartiles	40	0.131	0.023	0.020	0.333	0.167	0.022	0.021	0.261
Top quartile	67	0.115	0.013	0.016	0.259	0.156	0.013	0.017	0.199
Year = 2004
Bottom quartile	19	0.226	0.052	0.069	0.540	0.336	0.043	0.123	0.495
Two middle quartiles	39	0.144	0.023	0.024	0.332	0.209	0.021	0.035	0.271
Top quartile	67	0.115	0.013	0.008	0.191	0.146	0.013	0.021	0.239
Year = 2005
Bottom quartile	19	0.235	0.052	0.051	0.440	0.244	0.043	0.040	0.345
Two middle quartiles	39	0.140	0.023	0.025	0.344	0.175	0.022	0.030	0.300
Top quartile	66	0.099	0.013	0.005	0.193	0.133	0.013	0.019	0.246
All years (2000–2005)
Bottom quartile	20	0.203	0.050	0.046	0.475	0.253	0.044	0.058	0.406
Two middle quartiles	40	0.132	0.023	0.022	0.339	0.171	0.021	0.029	0.299
Top quartile	66	0.104	0.013	0.010	0.230	0.134	0.013	0.015	0.213

Tables 5:

Decomposition of variance in schools’ mean FSA scores – grade 7.

School enrollment quartile	Average size	Total variance	Sampling variance	Other nonpersistent variance	Total proportion nonpersistent	Total variance	Sampling variance	Other nonpersistent variance	Total proportion nonpersistent
		Grade 7 FSA Reading Scores				Grade 7 FSA Numeracy Scores
Year = 2000
Bottom quartile	20	0.142	0.050	0.010	0.418	0.183	0.047	0.027	0.401
Two middle quartiles	42	0.109	0.022	0.028	0.456	0.130	0.021	0.013	0.267
Top quartile	97	0.064	0.010	0.014	0.371	0.108	0.011	0.009	0.184
Year = 2001
Bottom quartile	20	0.182	0.051	0.016	0.368	0.252	0.042	0.042	0.335
Two middle quartiles	42	0.125	0.022	0.003	0.206	0.186	0.022	0.021	0.232
Top quartile	97	0.119	0.011	0.012	0.194	0.179	0.011	0.014	0.141
Year = 2002
Bottom quartile	20	0.188	0.048	0.004	0.276	0.250	0.042	0.032	0.295
Two middle quartiles	44	0.137	0.021	0.015	0.268	0.189	0.021	0.030	0.266
Top quartile	98	0.114	0.011	0.010	0.179	0.170	0.011	0.006	0.098
Year = 2003
Bottom quartile	20	0.235	0.047	0.053	0.425	0.270	0.044	0.054	0.363
Two middle quartiles	43	0.150	0.022	0.011	0.219	0.198	0.020	0.021	0.206
Top quartile	99	0.116	0.011	0.016	0.232	0.214	0.010	0.035	0.212
Year = 2004
Bottom quartile	20	0.270	0.048	0.058	0.393	0.327	0.043	0.048	0.280
Two middle quartiles	43	0.141	0.021	0.021	0.298	0.218	0.020	0.029	0.225
Top quartile	99	0.119	0.011	0.015	0.217	0.194	0.010	0.025	0.182
Year = 2005
Bottom quartile	19	0.232	0.052	0.047	0.427	0.278	0.048	0.065	0.405
Two middle quartiles	43	0.144	0.022	0.023	0.309	0.214	0.020	0.018	0.177
Top quartile	98	0.104	0.011	0.014	0.246	0.169	0.011	0.013	0.142
All years (2000–2005)
Bottom quartile	20	0.210	0.050	0.031	0.385	0.262	0.045	0.044	0.341
Two middle quartiles	42	0.135	0.022	0.017	0.289	0.190	0.021	0.022	0.226
Top quartile	97	0.105	0.011	0.014	0.232	0.169	0.011	0.018	0.167

Overall, the results suggest that school-average FSA reading and numeracy scores are not very reliable measures of persistent differences in school-average achievement, particularly among small schools. Looking at the bottom panel of each table, nearly 47% of the variance of school-average grade 4 FSA reading scores and 40% of FSA numeracy scores among schools in the smallest quartile of enrollment size is due to sampling variation and other nonpersistent factors. For grade 7 the estimates are 38% and 34%, respectively. As expected, transitory factors account for a smaller share of the cross-sectional variance among larger schools. For the two middle quartiles, 33% and 29% of the cross-sectional variance in grade 4 and grade 7 average FSA reading scores is due to nonpersistent factors respectively. The corresponding numbers are 30% and 23% for numeracy scores. For schools in the largest quartile, 23% of the variance across schools in grade 4 and grade 7 average FSA reading scores is due to transient factors, while for numeracy scores these numbers are 21% and 17% for grade 4 and grade 7, respectively. It also worth mentioning that almost across the board, the amount of imprecision in average FSA numeracy scores due to nonpersistent factors is smaller than average FSA reading scores.^[29]

One interesting question that could arise from these results is whether schools could become large enough so their test scores provide reasonable information about their performance? We cannot extrapolate outside our data range to predict the school size at which test-based school achievement measures will provide a reliable signal of school quality. This is because as schools become larger, the cross-sectional variance across schools will also change. For instance, as our Tables 4 and 5 suggest, total variance in test scores across schools becomes smaller across the board as we move to higher enrollment quartiles. This highlights the fact raised by Kane and Staiger (2001) that “it is not the absolute amount of imprecision that matters, but the amount of imprecision relative to the underlying signal that dictates the reliability of any particular measure.”^[30] In addition, it is not obvious how the variation induced by other non-persistent factors is going to change, given their random nature, as schools become larger.

However, despite the aforementioned limitations, we can examine the effect of transitory factors for larger schools by pooling student-level observations across years for each school. We can also pool these observations across grades to create even larger schools. Doing so will also provide some insights into the extent to which pooling test scores across grades and years will reduce the imprecision in these measures. In order to be able to implement the variance decomposition results similar to those reported in Tables 4 and 5, we need at least 3 years of observation for each school. This is because in order to calculate the contribution of other non-persistent factors we need to estimate the correlation between the changes in test scores in two consecutive years (as described in eq. [19]).Therefore, given that we have 8 years of data for each school, we use the most recent 6 years and pool observations for each school grade over three 2-year time periods: 2001–2002, 2003–2004, and 2005–2006. Additionally, we also pool observations for each school both across grades and across these 2-year time periods. This results in even larger school sizes.

The results of variance decomposition for these larger schools are reported in Table 6. First of all, pooling observations across grades and years gives us substantially larger schools. The four largest quartiles have average school sizes of 131, 161, 197, and 269. Consistent with results reported in Tables 4 and 5, the contribution of sampling variation becomes smaller as we move to higher quartiles. However, more careful inspection of the relationship between school size and sampling variance suggests that sampling variance decreases at a decreasing rate as school size increases. Figures 5 and 6 illustrate the relationship between school size and the magnitude of sampling variance for reading and numeracy scores, respectively, using all the estimates reported in Tables 4–6. While the sampling variance drops quickly for smaller schools as the school size increases, it does not change significantly at higher levels of school size. Figures 7 and 8 suggest a similar relationship between school size and the proportion of total cross-sectional variance across schools due to sampling variation. As expected, the magnitude of other transitory factors does not follow the same pattern given their random nature. For example, looking at the bottom panel of Table 6, it decreases as we move from the bottom to the two-middle quartiles, but then increases as we move to the top quartile.

Tables 6:

Decomposition of variance for larger schools.

Pooling across years – grade 4
School enrollment quartile	Average size	Total variance	Sampling variance	Other nonpersistent variance	Total proportion nonpersistent	Total variance	Sampling variance	Other nonpersistent variance	Total proportion nonpersistent
		FSA Reading Scores				FSA Numeracy Scores
Bottom quartile	39	0.136	0.025	0.034	0.439	0.202	0.021	0.036	0.288
Two middle quartiles	78	0.109	0.011	0.014	0.241	0.155	0.011	0.016	0.175
Top quartile	131	0.104	0.006	0.015	0.218	0.138	0.006	0.022	0.210
pooling across years – grade 7
School enrollment quartile	Average size	Total variance	Sampling variance	Other nonpersistent variance	Total proportion nonpersistent	Total variance	Sampling variance	Other nonpersistent variance	Total proportion nonpersistent
		FSA Reading Scores				FSA Numeracy Scores
Bottom quartile	40	0.196	0.023	0.021	0.227	0.231	0.021	0.023	0.194
Two middle quartiles	85	0.124	0.010	0.005	0.132	0.188	0.010	0.013	0.126
Top quartile	197	0.102	0.005	0.005	0.103	0.174	0.005	0.013	0.106
Pooling across grades and years
School enrollment quartile	Average size	Total variance	Sampling variance	Other nonpersistent variance	Total proportion nonpersistent	Total variance	Sampling variance	Other nonpersistent variance	Total proportion nonpersistent
		FSA Reading Scores				FSA Numeracy Scores
Bottom quartile	81	0.125	0.011	0.019	0.247	0.168	0.010	0.020	0.186
Two middle quartiles	161	0.098	0.005	0.002	0.086	0.136	0.005	0.004	0.072
Top quartile	269	0.107	0.003	0.009	0.121	0.150	0.003	0.012	0.107

^[31]

Nearly 22 (21)% of variance in school-average FSA reading (numeracy) scores among schools in the fourth largest quartile in terms of average school size (with an average school size of 131) is due to sampling variation and other transitory factors. For the third largest quartile (with an average school size of 161), these estimates are 8.6% and 7.2%. However, as we move to the second largest quartile (with an average school size of 197), the contribution of non-persistent factors becomes larger, 10.3% and 10.6% for reading and numeracy scores, respectively. Finally, as we move to the largest quartile (with an average school size of 269), while the contribution of non-persistent factors does not changes for numeracy scores, it becomes even larger for reading scores at 12.1%. We observe the same patterns across other quartiles in the table.

These results highlight two important issues. First, due to the decreasing effect of larger school size on reduction in sampling variation (illustrated in Figures 5–8), changes in cross-sectional variance across school as we move across quartiles, and random changes in the contribution of other transitory factors that are not a monotonic function of changes in school size, it is not obvious whether the effect of total transitory factors becomes smaller as the schools become substantially larger. Second, total transitory factors can still have a considerable impact on test-based school achievement measures for substantially large schools. For instance, comparing our results in the bottom panel of Table 5 with our results in bottom panel of Table 6, while the average school size has increased by 1,245% (from 20% to 269%), the contribution of transitory factors only drops by 68% (from 38% to 12%). The 12% variation in test scores among the largest schools in our sample, driven by total transitory factors, is still considerably large. To put it into perspective, if we create a 95% confidence interval for the average school in this quartile using the estimated total expected variance induced by transitory factors, it will extend from 25th to 75th percentile of the average test score distribution.^[31]

Figure 5:

Relationship between school size and sampling variance (reading scores), Fractional polynomial fitted plot.

Figure 6:

Relationship between school size and proportion of total cross-sectional variance due to sampling variance (reading scores), Fractional polynomial fitted plot.

Figure 7:

Relationship between school size and sampling variance (numeracy scores), Fractional polynomial fitted plot.

Figure 8:

Relationship between school size and proportion of total cross-sectional variance due to sampling variance (numeracy scores), Fractional polynomial fitted plot.

As it was mentioned before, we also measure the proportion of cross-sectional variance in the Fraser Institute school-level scores that is due to all transitory factors (sampling variation and other transitory factors). Friesen et al. (2012) find that public release of these scores has a substantial impact on parents’ school choice decisions. Moreover, since the Fraser Institute scores are constructed combining 10 different indicators (see footnote 12), it would be interesting to investigate the extent to which these more sophisticated measures of school performance are influenced by transitory factors. Results are summarized in Table 7and suggest that these scores are also strongly influenced by transitory factors. Between 18% and 22.4% of the cross-sectional variance in the Fraser Institute scores are due to transitory factors.

Table 7:

The Fraser Institute annual scores variance decomposition.

Year	Total variance	Total variation due to nonpersistent factors	Total proportion nonpersistent
2001	3.010	0.676	0.224
2002	2.987	0.665	0.222
2003	3.182	0.750	0.235
2004	3.244	0.587	0.181

^[33]

6 Conclusion

Sampling variation and one-time mean-reverting shocks are a significant source of observed variation in schools’ average test scores. This is of critical importance because there is growing evidence that suggests providing information about school-level achievement has real effects on parents’ school choice decisions. More importantly, the results of Friesen et al. (2012), which exploits a very similar sample of schools and students, suggest that some parents’ school choice decisions respond to publicly disseminated information about school average test scores soon after it becomes available, and continue to respond to subsequent releases in the following years. This raises even more concerns regarding the noisiness of test-based school achievement measures that are publicly disseminated to parents and are perceived by some as a signal of school performance.

Since school choice decisions are inherently costly, and since there is considerable amount of evidence that suggests these mobility decisions do not necessarily provide academic benefits to students, to the extent they are shaped by imprecise or noisy measures of school performance, they could potentially impose a net cost on parents and students and make them significantly worse off. Moreover, if it becomes apparent to parents that these measures are very noisy and fluctuate significantly from 1 year to the next, they might stop paying attention to new information about school-level achievement, undermining the effectiveness of school choice policies that partly hinges upon the ability to distinguish high-achieving schools from low-achieving schools. In addition, to the extent these noisy test-based school achievement measures are used as one of the factors to identify best educational practices, they could generate a lot of confusion over advantages and disadvantages of different strategies of school reform, and could mislead policy makers and educational authorities into decisions that are costly. Finally, to the extent noisy and imprecise test-based school achievement measures are used in jurisdictions with school accountability systems to evaluate schools’ performance, policy makers and educational authorities could be misled into assigning rewards or sanctions to schools based on factors that are outside their control and therefore undermine the effectiveness of these accountability systems. Chay, McEwan, and Urquiola (2005) provide strong evidence that suggests policy evaluations based on such noisy measures of school effectiveness have the potential to be misleading.

Nonpersistent factors are found to induce greater year-to-year variation in average test scores of small schools than bigger schools. However, even bigger schools still experience a significant volatility in their average test scores due to transitory factors. This suggests that combining test scores from more than one grade, or from more than 1 year, to increase the sample size and dampen the transitory volatility, will not resolve the concerns regarding the reliability of these test-based school achievement measures.^[32] Our results suggest that due to the decreasing effect of larger school size on reduction in sampling variation, and changes in between-school variance and the effect of other transitory factors, total transitory factors still have a considerable impact on test-based school achievement measures even after pooling test scores across grades and years, and could therefore move a school significantly in the distribution of scores. Therefore, it is not clear whether consulting several years of results, for example, as suggested by the Fraser Institute, will provide meaningful information to parents. As the previous discussion of results in Tables 2 and 3 suggests, the changes in the position of schools, particularly small schools, in the distribution of test scores over several years, is not far from a lottery process.^[33] To the extent these test-based measures are used by parents to inform their school choice decisions, this is likely to only add more confusion to parents’ decision-making regarding school choice, and might mislead them in this process into making costly decisions with no academic benefits for their children.

Our results also suggest that significant changes in average test scores of larger schools should be treated as a stronger signal about persistent changes in performance compared to small schools. This is something that is not directly reflected in the constructed scores by the Fraser Institute, and again could mislead or confuse some parents. This also highlights the importance of communicating the statistical properties of these measures, especially when they are publicly disseminated. To the extent these measures are used to identify best practices in education, failure to do so might send confusing signals to parents and schools and could make it difficult to identify the areas of academic performance in which improvement can be made. These results should also warn educational authorities against naïve policies or interventions that attach monetary/nonmonetary rewards or sanctions to schools based on noisy measures of school performance. As a growing literature attests, designing meaningful measures of school effectiveness continues to be a challenge (Ladd and Walsh 2002; Hægeland et al. 2004; Mizala, Romoguera, and Urquiola 2007).^[34]