Are Hospital Ratings a Mess, a Message, or Both?

Last Friday, I criticized a report in the public policy journal Health Affairs arguing that consumers' relatively restrained use of hospital ratings and data (such as America's Best Hospitals) can be blamed on confusion: The ratings and rankings measure different aspects of care, cannot be compared, and often contradict each other. I wrote that consumers can wade through the information, just as families sift through facts and numbers in America's Best Colleges and other college guides of more than 1,500 pages and cobble together custom lists online by sorting and clicking. I charged the study authors with underestimating consumers and condescending to their capabilities to do the same with hospital data.

My feeling is that if the public hasn't embraced hospital ratings with quite the enthusiasm it has for devouring college information, the reason has less to do with confusion and muddled data and more to do with timing, negative vibes, and the assumption that the exercise doesn't matter. Or, to be more specific, a hospital stay generally isn't predictable, the prospect is not thrilling, and the choice of destination is thought—mistakenly, let's note—to be beyond the control of the person going there. None of these applies to selecting a college, meaning that printouts and ripped-out magazine pages of college facts and figures are far likelier to end up on the refrigerator door for quick reference. And it's not as if the college guides and the facts and numbers packed into them are easy reading. But families sifting through colleges are motivated.

Among the comments that came in was one from a longtime student of health policy, Michael Millenson, that challenged my assertion (and my reaction). And below is a sharp and thoughtful response from Michael Rothberg, lead author of the Health Affairs study and an assistant professor at Tufts University School of Medicine, who accepted my invitation to continue the conversation.

*********************************************************************

Are we really being condescending? Is motivation enough to let consumers make sense of the conflicting data? Let your readers try it for themselves. Let's say I am that patient you mention with a history of heart attack. "Hospital Compare, here I come," you write, because this site assesses hospitals' faithfulness in adhering to certain "process measures" in treating such patients. I note that of two hospitals near me, one has a higher rate of providing aspirin within 24 hours, but the other has a higher rate of beta-blocker use. Which should I choose?

Or let's say one is a lot better on both measures, but a friend points out that HealthGrades, another site you mention, has mortality statistics, and, as you point out, process measures don't always correlate with mortality. Shouldn't I care more whether I survive the hospitalization than whether I get an aspirin when I arrive? But uh-oh, when I look at the same mortality ratings on my state Web site, Massachusetts Healthcare Quality and Cost Information, I find that the order of the hospitals is reversed—the raters can't seem to agree on which hospital will give me a better chance of surviving. Don't they know how many people died at each hospital? Actually, it isn't the mortality rate I should worry about, but the risk-adjusted mortality rate, because the best hospitals will get a lot of traffic from difficult cases directed to them by U . S . News. These patients will not do as well, on average, precisely because they are complicated. Unfortunately, there is no agreed-on method for risk adjustment, so HealthGrades uses one methodology and Hospital Compare uses another, producing disparate results.

It may be too much for us to ask that consumer-oriented data promoted by both the government and private enterprises be "accessible, interpretable and consistent," but we don't think it is "almost delusional," as you wrote. Nor do I agree with your assertion that the different sites are aimed at different types of consumers. Do you really believe that the U . S . News rankings are used only by the most difficult cases? The implicit assumption is that a hospital that can handle the really hard cases will do even better with the routine ones.

I agree, as you say, that individual patients have individual needs, but there is no indication that one type of reporting is particularly good for a particular illness or severity of patient. Hospital rating systems do report on different things, but none are things that most consumers are familiar with, so they don't know how to value them. It isn't like choosing a hotel, where I might care about the health club, while someone else values a nice view. Few patients, even well-educated ones, have a way to compare the value of an electronic medical record or a culture of safety with performance on government-collected measures like giving an aspirin within 24 hours. While mortality is an attractive candidate measure—after all, everyone has a concept of what it means to be dead—current statistical methods do not allow us to reliably differentiate among hospitals based on mortality.

The truth is that public reporting has tremendous potential to improve hospital quality and safety, whether or not consumers actually use it. If we hope to make it useful to consumers, even highly educated ones, we still have a lot of work to do. We need to find out what consumers want to know, and provide that information in an easily understood way. If we are to provide technical information about infection rates, risk-adjusted mortality and process measures, then we need to educate consumers about how to interpret and prioritize the information provided. If we don't do that, then we are probably wasting our time and will find, as the Kaiser Family Foundation recently did, that consumer use of public reporting is actually declining.

*********************************************************************************