Hospital Leaders, U.S. News Editors Hold Summit on Best Hospitals Rankings

Discussion focused on ways U.S. News could refine its Best Hospitals methodology.

By + More

"These rankings are not static." The affirmation by U.S. News & World Report editor and chief content officer Brian Kelly underscored the point of a summit on hospital rankings that U.S. News and Mount Sinai Medical Center co-hosted at Mount Sinai on September 27. The summit convened medical thought leaders from around the country for a four-hour discussion about the methodology behind the U.S. News Best Hospitals rankings. At the summit, hospital presidents and clinical leaders shared ideas, suggestions, and critiques of the U.S. News rankings. There seemed to be a consensus that the U.S. News methodology is as sound as any in existence. Multiple panelists and participants referred to its "face validity"; several called it the "gold standard."

"No methodology is perfect," said Kenneth Davis, president and CEO of Mount Sinai Medical Center, who delivered the summit's opening and concluding remarks. "This [U.S. News] methodology is certainly superior." And he saluted U.S. News for being "open to feedback."

The program began with a conversation between Kelly and U.S. News health rankings editor Avery Comarow, who has overseen Best Hospitals since its debut in 1990. Comarow noted that the last such gathering of hospital leaders, in 2006, led U.S. News to conclude that it should publish a separate ranking of Best Children's Hospitals, which was initiated the following year and has appeared annually since. Comarow added that U.S. News receives a steady stream of suggestions from hospital leaders. "You might be surprised at how many of [the suggestions] get incorporated in some form or another."

The event's first panel addressed the question of the usefulness of hospital rankings to consumers. Panelists remarked on the limited quality of relevant data in the public domain. Data on patient safety, for example, are highly susceptible to coding practices that vary considerably from one hospital to the next, said Bradly Narr, who chairs the surgical and procedural committee at the Mayo Clinic in Rochester, Minn. Patient safety indicators account for 5 percent of each hospital's score in the U.S. News methodology. They carry more weight in other hospital scorecards.

Panelists grappled with what they described as hospitals' obligation to be transparent about their performance. Some spoke of data "dashboards" that their hospitals make available online to the public, but these data presentations are voluntary and rarely comparable from one hospital to the next.

They also discussed whether patient satisfaction should be factored into the rankings. The federal government collects survey data on that measure, as the Hospital Consumers Assessment of Healthcare Providers and Systems (HCAHPS) database, and displays it on Hospital Compare. Patient satisfaction ratings from the HCAHPS surveys are displayed for most hospitals on as well but are not factored into the rankings. Evidence is lacking that a hospital's clinical abilities are reflected by surveys showing a high percentage of satisfied patients. Still, said panelist Timothy Gardner, medical director of the Center for Heart and Vascular Health at Christiana Care in Newark, Del., U.S. News should attempt to examine how well hospitals have served in the eyes of their patients.

Integrating clinical research into the rankings was put forward by several panel members. Panelists suggested that U.S. News could use NIH-funded clinical trials as a measure of how innovative a hospital is in translating cutting-edge research into therapies.

The second panel focused on the role of reputation in the U.S. News rankings. In most specialties, each hospital's reputation accounts for 32.5 percent of the hospital's score. U.S. News determines reputation through a national survey of board-certified specialists.

Somewhat surprisingly, given that the use of reputation has long been a focus of critics of the methodology, panelists and attendees generally agreed that reputation is a reasonable approach to assessing a hospital's ability to master the process of delivering topnotch care. "I think it's a pretty good proxy," said Steven Corwin, CEO of New York-Presbyterian Hospital. Most chief medical officers concur, according to a survey conducted prior to the event by U.S. News; 86 percent of survey respondents felt that reputation deserves a role in the rankings. The majority, however, would like it to get less weight—60 percent named a figure ranging from 10 to 25 percent of the score.

Panelists noted unanticipated consequences. "The reputation score is very, very important ... from a fundraising perspective," said Sharon O'Keefe, president of the University of Chicago Medical Center. Other panelists observed that some hospitals try to boost their reputation scores using advertising and other means unrelated to the quality of patient care.

They also suggested that the rankings may be interpreted by the public as an indicator of how well a hospital treats ordinary patients, while U.S. News tries to make it clear to surveyed physicians, and to the public in describing the project, that they are meant to show how well hospitals perform for complex care. The public may assume—perhaps incorrectly—that "if you're good at the hard things, you're going to be good at everything else you do," O'Keefe said.

Summit participants also questioned whether a national survey of all board-certified physicians represented the most qualified pool of people from which to survey. Davis suggested oversampling doctors who've earned certain distinctions, such as membership in the invitation-only Institute of Medicine; O'Keefe proposed that chief medical officers, chief safety officers, and other hospital staff might be more qualified than a pure cross-section of specialists would be to judge the efforts of their peer institutions.

The third panel, subtitled "Separating Spin from Substance," gave panelists an opportunity to critique what Corwin called "distortions," or the temptation on the part of hospitals to take actions that may give them a boost in the rankings but might not deliver a corresponding boost to patient care.

Several panelists criticized the rankings because they are rankings. Brent James, chief quality officer of Intermountain Healthcare in Salt Lake City, described what he called "false precision" in the rank-ordering of hospitals. Philip Ozuah, executive vice president and chief operating officer at Montefiore Medical Center in the Bronx, N.Y., said that while the U.S. News methodology does well at identifying the best of the best, "as you go farther down the list, it becomes less precise. People look at this as though it has the precision of a rectal temperature. My board [of trustees] certainly does." ("And sometimes," he said, "it feels that way." His expression suggested he was not entirely joking.)

One issue, panelists said, is that many hospitals' scores cluster tightly, making any small advantage or disadvantage important. In that context, Ozuah said, dichotomous variables can have an outsize impact. Vinita Bahl, director of clinical information and decision support services at the University of Michigan Hospitals and Health Centers in Ann Arbor, argued that it doesn't take much to dramatically change a hospital's position in the rankings.

Several panelists expressed misgivings about U.S. News's use of one particular dichotomous variable: Nurse Magnet accreditation. O'Keefe and Bahl each said their respective institutions had determined that their rankings would improve in several specialties if the hospital became recognized as a Nurse Magnet. The University of Chicago's decision to pursue Nurse Magnet accreditation was influenced by the effect it would have on the hospital's rankings, O'Keefe said. The University of Michigan, on the other hand, has not applied for accreditation. "We get penalized for that," Bahl said.

A recurring theme of the day was that data-supported rankings and other forms of public reporting can spur hospitals to focus on tracking the metrics on which they're being measured—sometimes for worse, often for better. Meri Armour, president and CEO of Le Bonheur Children's Hospital in Memphis, Tenn., said that when U.S. News introduced Best Children's Hospitals, she knew that Le Bonheur, a modest-size children's hospital in a poor, flyover city, wouldn't be ranked. The hospital nevertheless benefited from participating in the annual clinical survey that U.S. News asks children's hospitals to complete. "It was a way of measuring ourselves against the leaders," she said. "It gave us benchmarks." The improvements brought about by that yearly effort ultimately pushed Le Bonheur into the rankings. It's ranked in five specialties this year.

Toward the end of the summit, Comarow stated that U.S. News is intent on broadening the mission of Best Hospitals to evaluate hospitals in certain areas of routine care such as common heart and orthopedic procedures as well as labor and delivery. Doing so, he said, will require that U.S. News develop a whole new methodology to complement—though not replace—the existing methodology. He asked the hospital leaders take an active role in advising U.S. News on data sets to use and how to responsibly use them.

In response to Comarow's comments, several panelists said they applauded U.S. News's intent to expand. Peter Slavin, president of Massachusetts General Hospital in Boston, said that an important aspect of evaluating hospitals on how well they do procedures is determining whether they do too many of them. Bahl suggested that U.S. News establish expert panels similar to those it has used to generate measures and improvements for the Best Children's Hospitals methodology.

In closing, Mount Sinai's Davis restated several recommendations for U.S. News, including factoring in the average socioeconomic status of each hospital's patients and refining the way statistical weight is assigned to several advanced technologies that can count toward a hospital's rankings. Davis argued that patients in similar health but from different socioeconomic backgrounds will experience different outcomes even when they receive equivalent care. Factoring in socioeconomic factors, he said, would improve upon U.S. News's existing risk-adjustment methods, which current account for, among other factors, how sick each patient was upon admission to the hospital.