U.S. News Responds to White et al. (Health Affairs, 2014)

Despite paper's assertion, the hospital rankings are not "driven by reputation."


I have great respect for Health Affairs, a monthly journal of health care and health-policy research. So I was disappointed that the journal's peer review process did not identify and correct problems in a recent paper by Chapin White, James D. Reschovsky and Amelia M. Bond. Health Affairs has told me it plans to publish a version of the following letter to the editor in an upcoming issue.

The work of Chapin White and coauthors (Feb. 2014) misconstrues a key citation and consequently mischaracterizes the hospital rankings published by U.S. News & World Report. In addition to perpetuating a myth that the rankings are 'largely based on reputation,' their error undermines their conclusion that 'high-priced hospitals ... have good reputations.' The authors used as a variable whether a hospital was among the 50 U.S. News Best Hospitals in cancer, cardiology, gynecology or orthopedics. They cited a 2010 study (Sehgal) to justify their use of that binary variable as a marker of reputation. It is not. While Sehgal concluded that reputation contributed to 'the relative standings of the top 50 hospitals' — in other words, which hospital is No. 1 versus No. 2 — he did not find reputation to be a significant factor in determining which hospitals are or are not among the top 50.

In fact, U.S. News uses objective, primarily Medicare-derived quality measures such as mortality, volume and patient safety indicators to calculate 67.5% of each hospital's rank-determining score in the four specialties analyzed. Reputation accounts for only 32.5% and will soon be reduced to 27.5%. Many hospitals with negligible or no reputation are ranked among the top 50. U.S. News publishes hospitals' reputation scores at usnews.com/hospitals. It is unclear why the authors didn't use these available data in their analysis.

Moreover, methodology changes introduced by U.S. News in 2011 attenuated the influence of reputation. To our knowledge, Sehgal's finding has not been reproduced in the current era of the U.S. News methodology.

Additional limitations of the present paper are of concern. First, the analysis compared Medicare-derived outcomes to claims data from a non-Medicare population, which may be an inappropriate comparison. Second, U.S. News seeks to measure hospitals' quality of care for the sickest, most complicated patients. The acuity of the study population may differ.

Health Affairs' space constraints precluded raising another question. Chapin and his coauthors wrote that "insurers may face resistance if they attempt to steer patients away from high-priced hospitals because these facilities have good reputations."

This is an intriguing hypothesis that could be tested in part by studying the relationship between hospitals' prices and hospitals' reputations among patients. But U.S. News doesn't survey patients. We survey specialized physicians.

Perhaps the authors made an unstated (but reasonable) assumption that hospitals' reputations among patients are based in part on the rankings. If the assumption is true, it would be consistent with their study design and broader conclusions. Did they mean to assert that whether or not reputation influences the rankings, the rankings influence reputation?

I e-mailed that question today to Chapin White, who is a senior policy researcher at RAND, and have invited him to respond for this column.