Band 3, Heft 2

Value-added models are being implemented in many states in an attempt to measure the contributions of individual teachers and schools toward students' learning. Scores from these models are increasingly used for high-stakes purposes such as selecting teachers to receive merit pay increases, determining which teachers are to be fired, closing schools, and allocating money to programs. The statistician W. Edwards Deming wrote extensively about improving quality in education and the damage caused by performance rankings. In this paper, Deming's writings are related to uses of value-added models in the United States. Some of the models used for value-added assessment are reviewed and assessed. A simulation study based on Deming's red bead experiment shows the effects of strategies teachers and schools might adopt to raise their value-added scores without changing their teaching

Background: Millennium Development Goal 4 calls for a reduction in the under-5 mortality rate (U5MR) by two-thirds between 1990 and 2015. Due to issues with data quality and quantity, there is uncertainty about how much progress has been made in many countries, and conflicting estimates of the overall levels and rate of decline have been reported. Consequently, there is a clear and urgent need to validate estimates, but consensus guidelines for model validation are lacking. Methods: We propose a validation method, based on the question: If current U5MR estimation methodology had been used in the past, using all available data available at that time, how accurate would the U5MR estimates and uncertainty intervals have been?Findings: Validation of the estimates and uncertainty intervals reported in 2010 by the Institute for Health Metrics and Evaluation showed that estimates of both the past absolute levels and the rate of decline in under-5 mortality between 1990 and 2000 would have changed significantly between 2004 and 2010 for many countries, with the inclusion of additional data. For high-mortality countries in particular, over 40% of the updated estimates for the U5MR in 2000 fell outside previously constructed 95% uncertainty intervals, and the absolute change in the estimated rate of decline was more than 1% for about three out of five countries.Interpretation: In light of evaluating Millennium Development Goal 4 in 2015, we hope that extensive model validation exercises will become standard practice for validating the accuracy of under-5 estimates and uncertainty intervals. Ultimately the goal of including appropriate validation methods is to improve the reliability of both estimates and uncertainty intervals, which will allow for a more fully informed assessment of both the absolute levels of, and rates of change in, the under-5 mortality rate.

Maternal mortality is widely regarded as a key indicator of population health and of social and economic development. Its levels and trends are monitored closely by the United Nations and others, inspired in part by the UN's Millennium Development Goals (MDGs), which call for a three-fourths reduction in the maternal mortality ratio between 1990 and 2015. Unfortunately, the empirical basis for such monitoring remains quite weak, requiring the use of statistical models to obtain estimates for most countries.In this paper we describe a new method for estimating global levels and trends in maternal mortality. For countries lacking adequate data for direct calculation of estimates, we employed a parametric model that separates maternal deaths related to HIV/AIDS from all others. For maternal deaths unrelated to HIV/AIDS, the model consists of a hierarchical linear regression with three predictors and variable intercepts for both countries and regions. The uncertainty of estimates was assessed by simulating the estimation process, accounting for variability both in the data and in other model inputs.The method was used to obtain the most recent set of UN estimates, published in September 2010. Here, we provide a concise description and explanation of the approach, including a new analysis of the components of variability reflected in the uncertainty intervals.Final estimates provide evidence of a more rapid decline in the global maternal mortality ratio than suggested by previous work, including another study published in April 2010. We compare findings from the two recent studies and discuss topics for further research to help resolve differences.

Both research and practice of value-added models (VAM) have been growing in recent years due to the widespread effort to quantify teacher effectiveness. The existing VAM literature has not yet tested the sensitivity of value-added estimates to the rules that define which students contribute to each teacher’s value-added estimate. Student-teacher linkages are often a complex network due to various transfers, students taking multiple courses in the same subject, and students receiving special education or other “pull-out” services. Complex linkages are often considered as among the main threats to the validity of VAM. In this paper we conducted a case study to examine the sensitivity of VAM to the alternative link definitions. We examined three popular VAM approaches and applied alternative rules for linking students to teachers with each method. We found no overall sensitivity of estimated teacher effects to the linking rules. Even though more teachers had value-added estimates under more inclusive rules, for a teacher with estimates under all three rules, the correlation among pairs of estimates created using different linking rules was always above 0.95 and generally above 0.98 for each VAM approach. The value-added estimates of a small number of teachers were affected by highly different link definitions and these tended to be teachers with small numbers of students. Restricting the minimal sample size for calculating individual teachers’ value-added largely reduced the sensitivity in link definition.

Graphs and tables are an effective means of communication. However, relatively little experimental work exists examining differences between various formats in how well people understand provided information. We conducted two web-based experiments with a large, diverse sample to explore the effects of display format on respondents’ comprehension. We found that comprehension depended on task. Graphs were better for estimating differences; however, tables were better when estimating equality and sums. We found 3D display formats reduced comprehension of pie charts but not of bar charts. Although pie charts never assisted comprehension, they often did not significantly impair comprehension either. Comprehension based on a 3-way table was as good as that for clustered bar charts but was worse for divided bar charts. Information can be conveyed graphically even with 3-way tables, but the choice of display format needs to be sensitive to the task at hand.