The New York Daily News somehow got a hold of actual questions from the state’s fifth-grade reading test and published them, here. One question is as follows:
From the article “What are wind tunnels” by David Hitt: The statement that “NASA also works with others that need to use wind tunnels” most strongly suggests that:
A) Many different groups are developing space shuttles
B) NASA hopes to buy vehicles made by other agencies
C) NASA has the largest wind tunnels in existence
D) Many companies do not have their own wind tunnels
I will not publish the others, but you can read some in the original article as well as a follow-up blog post in the Washington Post, here.
The test, Book 1 of Form C of fifth-grade reading, could not possibly have been obtained legally. Someone hijacked a copy or took illegal photographs. The News isn’t exactly the New York Times or even the New York Post, so ethics considerations are irrelevant here.
The only point before us is, the items were published and now can’t be used on future tests. This action will cripple the state of New York in that standardized tests need to repeat items in different years in order to compare the results—and hopefully see improvement—from one year to the next.
However, the decision to publish or not publish information that comes into my hands is a question I deal with every day. I just consider the interests of the state in administering a test that has a slight chance of statistically showing improvement higher than the interests of the public in knowing the contents of any given test question. Furthermore many states release items from time to time, and those provide ample opportunity for comment.
This item in particular is poor
I read the passage quoted in the item (I won’t reprint the passage because of copyright issues, but the link has been provided).
The correct answer is (D), I assume, because (A), (B), and (C) have nothing to do either with the extracted sentence or with fact. (D) isn’t technically correct, because, in my opinion, it’s a pretty far leap to go, as kids would read the passage, from the real goal of NASA letting other companies use its wind tunnels—achieving greater safety in air travel or, by extension, to serve the public good, which is what the government is supposed to do—all the way over to “many companies don’t have their own wind tunnels.”
The word “many” is a relative term, of course, and if you consider “many” to be “thousands of,” say, the option is the only one that bears any resemblance to fact. However, if you consider “many” to mean “millions of,” well, there aren’t even a million companies, I think. But most kids probably read the choice as “the majority of.” Of 10 companies I can probably recall off the top of my head, nine of them don’t have wind tunnels, probably. But the question’s not about what is true; it’s about drawing inferences from a reading passage.
I can infer option (D), perhaps, based on the word “need,” but the passage doesn’t exactly set it up as a main point for why I would read the passage in the first place. This question is clearly testing kids’ ability to draw inferences from technical articles, but the only reason (D) is the correct answer here is that the question puts in the qualifier “most strongly.” If (A), (B), and (C) aren’t even correct on fact, (D) is “most strongly” suggested by the statement, although it is most definitely not the “strongest” suggestion from that extracted sentence. NASA’s goal of helping the public or of increasing air travel safety are suggested much more strongly than the number of companies that have wind tunnels of their own.
Another reason this is a bad question is that we infer ideas from sentences, yes, but most inference is drawn in context. In context, the two paragraphs in which the sentence can be found talk about air safety, including these sentences: “Wind tunnels help NASA test ideas for ways to make aircraft better and safer. Engineers can test new materials or shapes for airplane parts.” I certainly hope the majority of our companies do something besides making airplane parts, so perhaps option (D) should read, “Some companies do not have their own wind tunnels.” But that’s a minor technicality of the wording. What I initially infer is that NASA is doing what every responsible government agency does: using its resources to help people and companies that can’t help themselves in some way. That’s not even one of the options.
It’s a generally good design principle for multiple-choice questions that kids should know the answer before they read the choices. This question is also of poor quality in that the answer I thought I would find after deeply reading the passage isn’t on the list.
So, what we have here is a question that purports to measure deeper reading skills, critical thinking, and things like that, but instead has a “correct” answer only when the sentence is taken out of context and all the options are read and evaluated by the student.
In other words, even if New York’s teachers had taught their students properly to make inferences from reading, this question throws all that instruction out the window. Now, not all questions on tests are good or even count in the final score, so I don’t know anything about New York’s test map. This may be a field test question and not an operational question. If it’s a field test question, it won’t count toward kids’ scores or teacher evaluations.
However, it still appeared on an actual test, and kids don’t have any clue what the test blueprint is. As far as a New York fifth grader is concerned, this question demands an answer, which isn’t really there. This type of question frustrates kids, whose teachers may have taught them how to read in context properly but who will get no credit for this question because the “strongest” suggestion from the sentence isn’t a choice.
So, they sometimes raise their hands, although their teachers are prohibited from giving them any assistance in answering specific questions. But they most often just sit there, frustrated, and guess. Be very careful in making broad inferences, such as how well kids are learning to make inferences, from tests that contain questions like this.
Other comments about the tests
I’ve read a great deal of commentary about the tests in New York, which are aligned for the first time this year to the Common Core. Some commentary has come from parents and teachers in the schools, who say kids could be seen crying, unable to finish the tests, and even physically sick over them.
Other “testing experts” have said the exams were above grade level. This, however, was expected when New York aligned the exams to the Common Core, which generally pushes reading and math topics and learning standards into lower grade levels than they would have been in the curriculum previously used in states.
I’m not saying it was a good idea to have kids take a test for which teachers haven’t had enough time to prepare them, but charges that the exams were too difficult or “above grade level” are misdirected. Scores are going to decrease, but statisticians should be able to account for that, given new statewide means, medians, standard deviations, and so on.
Whether or not they do that is still an open question, of course. However, I hope the tests will be equated with exams given in previous years, despite the unfortunate publication of one form from this year’s test. Equating in future years to observe improvement will be more difficult than it would have been if the questions were kept confidential.
