Are small schools better?

small schoolCan you get fooled by random events? You bet you can. Here’s an interesting example, recounted by Daniel Kahneman in his book Thinking Fast and Slow.

A survey of average student test scores at over 1600 schools focused in on the top 50. One item stood out. Of the 50 best schools, 6 were small. This was surprising since there were very few small schools in the study. In fact, the small schools were overrepresented among the best by a factor of 4. This started the small school movement, supported by the major foundations and the US Department of Education.

Yet, if you looked at the bottom 50 among the same schools, you would see small schools overrepresented there as well. If the study had focused on fixing bad schools rather than emulating good schools, they might have seen this data and concluded that bigger is better. What’s going on here?

There’s a simple explanation that shows that this anomaly is not only possible, but to be expected. It’s all explained by the Central Limit Theorem, which says that the more items you average, the smaller the variation from one average to the next. If you look at the test scores of individual students, you will see the full range, including both some very poor and very good students. These individual differences swamp any educational differences. You won’t see this spread in the average test scores for a school, since there will usually be a mix of very good and very poor students, who average each other out. In fact, a very large school will closely match the general population, and so will have average test scores very close to those of the general population. In other words, the big schools will all have about the same average test scores. On the other hand, a small school may by chance have a few very good or very poor students who swing the average, so the average scores at the small schools will show more variability than those at the large schools.

Now when we look at the very tails of the distributions, we find small schools, not because they’re better or worse, but simply because their average scores are more variable. So which is better? We really won’t know until someone does an analysis that includes all the schools.