How Statistics Makes Success Look Like Failure

Kevin Drum has a couple of posts looking at the most recent results of the National Assessment of Educational Progress, the "Nation's Report Card."

By the way, did you know that last month the Department of Education released the most recent data for the NAEP?  Yeah, neither did I until I saw these posts.  The NY Times does not have one single story about this year's test scores, which were released on June 20.  Amazing.  More on this below.

Anyway, Kevin makes a great point about how statistics can make it look like we're making very little progress when, in fact, we're doing a lot better than we used to:

[W]hat's really important—and what I forgot to mention—is that you have to disaggregate test scores to really understand what's going on. Let me give you an example from a different area: English literacy. Suppose you have 100 residents of the country, 90 of them native and 10 of them recent immigrants. Among the natives, English literacy runs at 90 percent. Among the immigrants it's 70 percent. This means that a total of 88 residents (81 + 7) are literate in English.

Now suppose the composition changes: we have 80 natives and 20 recent immigrants. And both groups are doing well! Native literacy has improved to 91 percent and immigrant has improved to 75 percent. That sounds great, doesn't it? But what if you lump everyone together? Then it turns out that 88 residents are literate (73 + 15). It looks like there's been no progress at all. Literacy is up among all groups, but the rising share of immigrants has pushed down the average.

This is exactly what's going on with the NAEP.  Here is a table from Kevin's post (note that a 10 point increase is approximately 1 grade level):

The age 17 level shows the phenomenon most clearly.  All races are seeing their scores improve, and we have seen astounding improvements among black and hispanic students.  But they are still behind whites, and as a their numbers grow relative to whites, they bring down the average despite their dramatic improvements.

Bob Somersby points out that at least one major news outlet talked about the scores when they came out:

On NBC Nightly News, Brian Williams described “a grim report card on our nation’s high school seniors,” who were said to be “leaving high school no better in reading or math than students in the 1970s.”

This is just about as false a picture as one can paint about how American students are doing.  Now I'm not one to engage in conspiracy theories about why all we hear is bad things about our public schools, but....

Public schools are doing an outstanding job.  Too bad you'll never hear about it from the MSM.