Can it be that the number of "likes" on a hospital's Facebook page has something to do with the quality of the hospital? Could be, say researchers at the Healthcare Innovation and Techology Lab, a New York think tank.
They state in a recently published analysis that they found a "strong negative, statistically significant relationship" between the number of "likes" on the Facebook pages of 40 New York-area hospitals and the hospitals' 30-day heart attack death rates. The higher the number of likes, in other words, the greater the chance that a hospital had a lower mortality rate. "These findings have implications for researchers and hospitals looking for a quick and widely available measure" of hospital quality, the team stated in their study, which was published online in the "American Journal of Medical Quality."
If that is true, however, it can't be determined from the study. Apart from the oft-stated and sensible principle that correlation does not equal causation, what is missing is evidence. Interested parties like the Best Hospitals team at U.S. News would want to see the data from the 40 hospitals represented graphically, such as on a scatterplot, which would show where each one falls on the strength of the relationship between its "likes" and its heart attack mortality. That information is absent even in the form of a table. The use of a single quality indicator based on mortality is contentious. Nor is the work free from selectivity and sloppiness. There are many more hospitals in the defined universe — those within 25 miles of New York — than were examined. Major New York hospitals with Facebook pages, such as Bellevue Hospital Center, Montefiore Medical Center, and St. Francis Hospital, are missing. The list of hospital names is riddled with errors. (A call to the Healthcare Innovation and Technology Lab was not returned.)
But there is a larger concern: No one knows what is behind a "like." The act of clicking on Facebook's "like" button can be impulsive, carefully considered, or somewhere in between. On a hospital's page, it might indicate appreciation for a mother's life saved, gratitude for a billing representative's patience explaining a statement, or surprised pleasure at the great barbecue in the cafeteria buffet. Which clicks relate to hospital quality? The study does not raise or address the question. To me, there is too much missing from the study for the conclusions to meet an elementary sniff test.
Wanting a sniff test of my sniff test, however, I turned to Joe Murphy at RTI International, a large Triangle, N.C.-based consulting organization that is the principle contractor for the annual Best Hospitals and Best Children's Hospitals rankings. Among his other responsibilities at RTI, he is director of the Program on Digital Technology & Society in the survey research division, which is investigating the use of social media in research. He's excited by the possibilities. But the claims of this study, he says, are over the top.
The possibilities, Murphy says, arise from the mind-boggling amount of available information about people, their behaviors and their opinions conveyed by social media like Facebook and Twitter. "Compared to conducting a traditional survey, social media are immediate and easy to mine, and this could lead to improvements in the quality, timeliness, efficiency and relevance of research," he told me in an email.
"But researchers (and readers) should resist the temptation to accept findings that do not seem plausible on their face," Murphy continued. "Just because a statistical test shows a relationship between social media and 'hard data' doesn't mean one can be used as a substitute for the other. In the example of linking Facebook data to the rate of heart attack survival in a hospital, there is just not enough known about why, how and when people click 'like' to suggest that we should expect it to be a valid proxy."
In their New York hospital study, the authors write that their analysis implies that "any hospital can start a Facebook page, but those with higher levels of quality and patient satisfaction are more likely to attract 'likes' to their page, all other things being equal."
In the real world, it is rare that all other things are equal. It is hard to imagine that they are in this case.