U.S. News Hospital Rankings to Double Role of Patient Safety, Cut Back Reputation

Patient safety measures will account for 10% of each hospital’s score, up from 5%


U.S. News will make several changes to the Best Hospitals rankings methodology this year. Most significantly, in 12 specialties, the weight assigned to patient safety will double to 10 percent of each hospital's overall score. The weight given to hospital reputation will drop from 32.5 percent to 27.5 percent.

This shift reflects the ongoing evolution of the Best Hospitals rankings from their 1990 origin as survey-driven lists toward increasingly objective assessments. This movement is made possible by the growing availability of publicly reported, rigorously studied hospital quality metrics.

No metric is flawless, however, and our decision to increase the importance of specific patient safety measures may have its critics. They will receive a fair hearing. Accepting thoughtful and constructive criticism is part of our DNA at U.S. News.

Our decision stems from a multi-year evaluation conducted last fall by RTI International, U.S. News's longstanding contractor, of the Patient Safety Indicators compiled and published by the federal Agency for Healthcare Research & Quality (AHRQ). The agency has defined 17 PSIs that quantify various incidents and errors that put hospital patients at risk, from falls to surgical incisions that reopen after patients leave the OR. U.S. News used six of the PSIs in the currently posted 2013-14 rankings to create a Patient Safety Score, which accounts for 5 percent of a hospital's overall score.

RTI's recent analysis led to a recommendation, which U.S. News has accepted, to include two of the 11 excluded PSIs. The two new entries are AHRQ PSI 03 and PSI 08, which respectively capture incidence of skin breakdown (decubitus ulcer) and postoperative hip fracture. We had previously felt that these indicators were too unreliable to be included in the methodology. But RTI took another look because starting last year, all PSI data that we used in the rankings included so-called present-on-admission data that can identify patients who were at elevated risk of adverse events when they were admitted to the hospital. (For the six PSIs already used in Best Hospitals, statistical adjustments to account for present-on-admission indicators were incorporated into the methodology last year.)

Besides PSI 03 and 08, RTI evaluated PSI 12 (postoperative pulmonary embolism or deep vein thrombosis) as a possible candidate. Based on RTI's analysis and recent peer-reviewed research that highlighted PSI 12's limited reliability as a quality measure, U.S. News decided not to incorporate it this year. (In an upcoming column in this space, we will discuss the problems with PSI 12.)

As described in this recent column, we are also considering giving credit to hospitals that regularly conduct patient safety culture surveys, which measure how well the organizational culture supports efforts to improve patient safety.

The addition of the two new PSIs and the improved reliability of all PSIs thanks to the availability of present-on-admission data both factored into our decision to increase the weighting of patient safety from 5 percent to 10 percent.

Another consideration was our decision to calculate each hospital's Patient Safety Score, which is based on PSI data, in a more nuanced manner. Until now, a hospital was placed in one of three tiers – high, moderate or low. Hospitals with scores between the 25th and 75th percentiles were treated the same, with all receiving a "moderate" Patient Safety Score. In the next rankings, we will calculate Patient Safety Score as a continuous variable, based on each hospital's composite score in the eight PSIs.

This change may cause some shifting in the rankings by boosting the rankings of hospitals with high-moderate PSI scores while diminishing those with low-moderate scores. We feel that this is appropriate and serves the interests of health care consumers. U.S. News further expects that continuous scoring will reduce year-to-year volatility in the rankings going forward.

Even as we downgrade the weight given to Reputation Score, we are taking a significant step to improve how we capture a complete picture of physicians' opinions of the hospitals they know. In the past, we have annually mailed and e-mailed Best Hospitals surveys to 200 board-certified specialists in each of the 16 adult specialties that U.S. News evaluates. We will do the same this year, but in addition we will survey approximately 50,000 other board-certified physicians through a collaboration with Doximity, a professional network for doctors.

That survey began yesterday, in fact, when Doximity began to notify survey-eligible members when they log on that they are invited to participate. Survey-eligible nonmembers who join Doximity during the survey period will also be invited to participate. Not all Doximity members are eligible, because U.S. News surveys only physicians who are board-certified in a specialty or subspecialty within one of the 16 ranked specialties.

Survey responses from Doximity members and non-members will be statistically weighted to ensure proportional representation of the national population of survey-eligible physicians.

Readers of this column are invited to post comments and questions below. To the extent possible, our editorial team will respond to them here or in future Second Opinion columns.