Saturday, April 17, 2021

Opinion: Stats go too far in medicine & education


Robert M Wachter writes in today’s New York Times that although we need to hold professionals, including teachers and doctors, accountable for their work, numbers mislead us into thinking that we have met our goals when, in fact, we’ve accomplished nothing of lasting value.

“Health care and education, have become increasingly subjected to metrics and measurements,” the professor and interim chairman of the department of medicine at the University of California, San Francisco, and the author of The Digital Doctor: Hope, Hype, and Harm at the Dawn of Medicine’s Computer Age, writes. But, “the focus on numbers has gone too far. We’re hitting the targets, but missing the point.”

Education is experiencing its own version of measurement fatigue. Educators complain that the focus on student test performance comes at the expense of learning. Art, music and physical education have withered, because, really, why bother if they’re not on the test?

He goes on to quote Avedis Donabedian, a now-deceased professor at the University of Michigan’s School of Public Health who developed what is known as Donabedian’s triad for measuring the quality of medical care. The triad holds that quality is measured by looking at

  • Outcomes (how the patients fared)
  • Processes (what was done)
  • Structures (how the work was organized)

Shortly before he died, Dr Donabedian was asked about his view of quality. “The secret of quality is love,” this “hard-nosed scientist” said, Dr Wachter wrote.

The same principle could be applied to education. We hear continuous messages from educators that test scores (the outcomes in Donabedian’s triad) need to be used in conjunction with other estimates of student understanding.

My sister just received individual student reports (ISRs) for the PARCC tests her kids took in third and fourth grade. The Illinois school district sent out the ISRs right along with their report cards, which were, of course—since PARCC results took nearly seven months to be distributed to schools and parents—much more current.

Side by side, these two reports showed both of them to be very good students in math and slightly better than average students in English. The PARCC results reinforced the report cards, but the report cards showed much more “love,” I guess we can call it, since written notes were far more personal than PARCC’s statistics-laden language.

My sister, to be honest, couldn’t understand what the language used on the PARCC ISRs was intended to mean, let alone how she should use the ISRs to help her kids in math or English. And my sister has a college degree from a major university, much more than the majority of parents have who will try to make heads or tails out of these PARCC ISRs this winter and next.

It’s just another example of measurements from standardized tests that have gone too far, of corporate giants like Pearson, which produces the PARCC ISRs, creating something useless for the majority of people it should be helping, people whom the state of Illinois is paying for it to help, actually. Thank goodness the school and my nephews’ teachers have the good sense to spread a little love in the form of a handwritten report card for 10-year-olds!

One commenter on the article writes this:

Two vital measurements have been too long ignored. First, does the provider have a complete and adequate understanding of all the patient’s past and present problems? It is impossible to provide the best care when blinded to the full set of moving parts. Second, and most important and difficult, has the provider negotiated goals for the management of all active problems with the patient? If you aim at nothing, you will hit it.

Unless and until these metrics are tested, we will have providers drowning in a sea of satisfaction surveys, computerized lists and the like (and leaving the job) just as the swelling ranks of the aging population crest on the steps of our offices and hospitals.

I echo this in education with regard to “personalized” instruction now being touted by so many for-profit education chains: Does the teacher have a complete and adequate understanding of all the student’s past and present learning goals and achievements? It is impossible to provide “personalized instruction” when blinded to the “whole” student, a set of moving variables that aren’t accounted for on standardized tests.

Second, and most important and difficult, has the teacher negotiated goals for the management of the “whole” student? If you aim at something that is not, by any definition, personalized instruction, you will hit it, just as lots of schools came quite close to achieving “100 percent proficiency” in math and reading under the old No Child Left Behind law.

Paul Katula
Paul Katula is the executive editor of the Voxitatis Research Foundation, which publishes this blog. For more information, see the About page.

Recent Posts

Md. to administer tests in math, English

Students in Md. will still have to take standardized tests this spring in math and English language arts, following action of the state board.

A week of historic cold and snow

Perseverance lands on Mars

Summer vacation, summer job, or summer school?

Biden is sworn in as 46th president

Florida balances optimism after the riots