We frequently hear two criticisms about the annual physician survey, probably the most-discussed element of the Best Hospitals rankings:
It’s tempting to let the two viewpoints cancel each other out, but a rational explanation is called for. Regarding the first criticism: We consider doctors’ assessments of hospitals to be meaningful – a sort of peer review that is informed by experience and professional judgment. The survey, which generates each hospital’s Reputation Score and accounts for a little less than a third of its overall Hospital Score in most specialties, is carefully designed to produce a statistically valid measure of opinions of the entire population of about 200,000 U.S. physicians who meet our criteria for survey eligibility. This past year’s survey is described in detail in the current Best Hospitals methodology report.
As for the second criticism, we are looking into expanding the survey sample beyond its current 3,200 physicians per year, or 200 in each of the 16 specialties in which U.S. News ranks hospitals. (This excludes the 1,500 pediatric specialists surveyed annually for Best Children’s Hospitals.) Increasing the sample size has the potential to improve the statistical precision of the results.
Early in 2014, therefore, in addition to the conventional survey, U.S. News and its Best Hospitals data contractor, RTI International, plan to conduct a novel survey in collaboration with Doximity Inc., a San Mateo, Calif.-based company that has built a HIPAA-compliant professional network open to all U.S. physicians. The new survey’s sample will be drawn from the membership of Doximity’s network. More than 30 percent of physicians already are Doximity members, with more joining every day. Membership, which is free, requires each physician to have his or her identity verified by Doximity. (Membership also enables a physician to update his or her profile on usnews.com. Any physician can join by locating his or her profile in the U.S. News Doctor Finder and clicking on “Claim/Edit Profile.”)
U.S. News expects to incorporate data from the Doximity member survey into the Reputation Scores used in the 2014-15 rankings. How Doximity members’ nominations will be incorporated into the complete survey will be determined by U.S. News and RTI in the spring of 2014, following an analysis of results to examine the data for potential statistical biases. It will be spelled out in the annual methodology report published along with the rankings in July 2014.
With slight modifications, a conventional survey of 3,200 additional physicians will also be conducted in early 2014, and its results incorporated into the 2014-15 rankings. The overall role of Reputation Score in the rankings will not be increased.
In preparation for the new survey, Doximity, RTI and U.S. News recently completed an online pilot survey of approximately 200 Doximity members. We want to acknowledge and thank those physicians who took the time to participate. Their contribution has helped strengthen the design of the survey we will field early next year.
The coming year’s experiment is part of the ongoing effort to refine and improve the Best Hospitals methodology. Analysis of the results of the Doximity member survey should help us continue to improve the reputational component of the Best Hospitals methodology in subsequent years.