This is an abstract. Our full report is here.
On Oct. 18, the Chicago Tribune published a huge article about the ISAT test, which the state of Illinois uses to assess third- through eighth-grade students in math and reading. The gist of the article was that kids today need to get fewer questions right than they did in the past, so of course the scores are going up.
The paper quoted an “expert” who has developed a Web site:
Children who pass state exams are considered proficient, but what it means to be proficient is called into question when passing requirements go down, [psychometrician John] Wick said.
“How much ‘excellence’ is shown when an 8th grader, in math, scores 40 percent correct on the test and is declared ‘proficient?'” Wick wrote in his study. The state lowered the passing bar on the eighth-grade math test in 2006, part of a revamp of ISAT exams.
As has happened so many times when people who are good reporters but bad statisticians try to report on a school issue, the Tribune has missed some of the most critical pieces of information in their mammoth story, which was quite good on other counts.
Their story hangs its hat on the fact that on many of the ISAT tests, students don’t have to answer as many questions right as they have in the past. Authors write: “Compared with 2006, fewer correct answers are required to pass 11 of 12 Illinois Standards Achievement Tests in reading and math, state data show, raising questions about how much students really have to know. … It took 36 of 76 points to pass the fifth-grade math exam in 2006. That dropped to 35 points in 2008; to 33 in 2009 and to 32 this year.”
And here’s the kicker, which fell on flat ears at the Tribune: The scale shifts from year to year, based on the number of kids who get the linking items right. Although I don’t work for the Illinois State Board of Education, I do work for the Maryland State Department of Education, and I’m sure Illinois’s procedures are not too different from ours. Every year, a test is developed that has questions on it that have so many from the easy set, so many medium, and so many hard, for each area of the state curriculum that the board wants to test.
After the test is given, the performance of students on the linking items is compared with their performance on those same exact items in previous years. If they perform exactly the same on every one of the linking items, then kids will have to answer exactly the same number of questions of exactly the same level of difficulty correctly in order to “pass” (to be counted as proficient or “at” grade level).
But that’s a lot of variables. In reality, the performance on the linking items is never the same from one year to the next. Teachers can look on the state board’s Web site to get a preview of test questions. They can then train students to answer those types of questions, and even though the numbers will be different, the state board probably provides a good sampling of questions on the practice tests.
Second, questions are chosen for tests based on their performance during a field test, which may have taken place several years ago, when the performance on linking items was also different. The numbers on the horizontal axis for these will have to be adjusted, meaning the point on the horizontal axis that constitutes the state’s definition of “at grade level” will have more or fewer students getting the question right.
The number of correct answers required, therefore, to be “at” grade level, which is what all NCLB tests are required, by law, to measure, will change a little from one year to the next. What is a little suspicious about the Illinois data (and Texas, by the way, which was reported by the Tribune) is that the number keeps going down. But it doesn’t merit any kind of investigation, and I just wanted to make sure everyone understood that the Tribune article contained a lot of nice quotes from educators but was short on facts.
I particularly liked their ending, which quoted David Chiszar, director of research and assessment for Naperville Unit School District 203: “You’d have to overtly say, ‘I want to make the test easier. The state would have to be monkeying around with the data behind the scenes and lying to us.”
It could happen, and if the Tribune would like to show a real statistician the test curve from ISBE’s technical report and get his or her comments, let’s have at it. But short of seeing the curves of the test and for the items that constituted the test in 2010 and 2006, the number of questions right will not be comparable. We’re talking about apples and oranges here, potentially. And that’s just scare tactics in reporting, which the Tribune has certainly chosen to engage in with this story.
