OECD PISA - An Example of Stochastic Illiteracy?

Abstract

PISA stands for “Programme for International Student Assessment” performed, by the OECD and consisting of a cyclic evaluation of the new generations' basic competences. The first evaluation took place in 2000 and the results were published on 4 December 2001. According to the official statements an overall of 32 countries participated in this first study. The publication of the results in Germany caused some great excitement, as Germany the “country of arts and science” was ranked at the lower edge not so far from Brazil. Politicians, teachers, scientists and any other people expressed their opinions, looked for those responsible, for the bad state and demanded immediate actions for improving the education in Germany.

Bavaria, the German state which pretends to have the best educational system, at least in Germany, claimed that the Turkish and Jugoslavian students in Germany had caused the bad grade. Other being more cautious demanded better teachers, another educational system and better pupils. The nationwide discussion arouse my curiosity and thus, I tried to get some details on PISA and found them e.g. on the homepage of the OECD, the renowned Max-Planck-Institut Berlin (http://www.mpib-berlin.mpg.de/pisa) and other national institutions in the participating countries.

The study has an ambitious aim and is essentially a statistical one. Therefore, a thorough evaluation of the quality of the study and its results should stand at the beginning of any discussion. This paper looks at PISA exclusively from a statistical point of view and makes an attempt to evaluate its quality. It is an attempt, because only very limited information on the applied statistical procedures were available despite of the fact that hundred of pages describing the study are made available through internet.