By Gerrit Van Wyk
Parachutes are optional for jumping from a plane.
A double-blind study was published, in which subjects were randomized to receive a parachute or not and then jump from a plane. The authors found 100% in both groups survived the jump and concluded parachutes are optional from jumping from a plane. As a footnote they mentioned the plane was parked on the tarmac. The experiment was performed immaculately and written perfectly in a scientific format, but to interpret it, the context is vital, and that is missing from many if not most academic papers in health care.
The number of doctors I came across who can read and interpret research is quite small. We assume as experts and professionals they can, but they can’t, which has immense implications for health care practice. Patients, managers, and planners rely on doctors and researchers to provide them with information, assuming the information is accurate and useful in practice. The technological and pharmaceutical industry steps in to the gap by funding research and disseminating the research to doctors, which, in the ways it is presented, ensures doctors know parachutes are optional when jumping from an airplane.
A movement started based on the idea doctors should only make decisions based on published scientific evidence, and the best knowledge comes from randomized controlled studies with positive outcomes confirmed by statistical tests, such as the parachute study. Small studies can be pooled in a meta-analysis, which is the best evidence an intervention works, and based on that, clinical guidelines which will improve quality and efficiency can be designed and used for treatment and making policy decisions. This is a deceptively simple formula, but what happens when you release it in the soup of complex human society?
To begin with, most published studies are methodologically and statistically flawed, as Ioannides pointed out. Patients drop out of studies or tell researchers what they want to hear, researchers don’t always stick rigorously to the method, endpoints are often changed in the middle of the study, outcomes are selected arbitrarily, and studies are often stopped before completed. Funders have a vested interest in outcomes, tenure track and promotion depend on the volume of publications not on their importance, most research is performed on a select group of patients in first world countries, which cannot be extrapolated to other populations, and research is performed and massaged based on the likelihood of getting published.
As far as meta-analyses are concerned, most medical journals only publish studies with positive outcomes, creating a file drawer problem; the bulk of study outcomes remains outside the journal domain. Getting published is not a simple matter. Submitted papers are modified and rewritten until it reflects the bias of the editor and journal, hence there is a long delay before results enter the public domain, by which time it may no longer be relevant. Many studies are milked for multiple publications to increase academic credibility, publications are biased towards English speaking institutions, a small number of researchers with big reputations set the agenda for everyone, and popular subjects are researched rather than more meaningful ones. The data in all meta-analyses are therefore incomplete and likely biased.
Designing proper guidelines is time consuming, expensive, and requires a lot of resources. Most guidelines are completed in too short a time without enough resources. The expert panel is often prominent researchers biased by their own interests, and sponsors often have a vested interest in the outcome. There is no evidence grading systems work, and there is significant variation between expert users of those systems. Most guidelines are based on local evidence, practices, and beliefs, hence different ones for the same condition often contradict each other, guidelines are based on the statistically average patient representing the mean of a population, and we all differ from that measure in one way or another. The statistical probability someone will benefit from an intervention like the average patient doesn’t mean everyone suffering from the condition will benefit from it. Policy makers, managers, regulating authorities, and the legal system often use guidelines as a tool of coercion, but doctors are forced to adapt them to the challenges of local conditions to make it work in practice.
In short, a wonderfully simplistic idea, as can be expected, falls short in our complex human social world. I am not arguing science and data have no role in health care, I am arguing instead we should use it pragmatically with common sense, which includes admitting to the problems and difficulties of the data we work with. That to me means does the information available to me solve the problem, and if it doesn’t, how do I collect useful data or find better information that will? If we use information available to us without thinking about it, we’ll find out jumping from an airplane without a parachute, based on the best available evidence, can have painful consequences.