Focus On The Body Of Evidence…Not Just Individual Studies

JAMA published an important article last week. This is an important article because it reminds us to review the totality of the evidence and not just an individual study. There are several reasons to focus on the whole body of evidence and not just individual studies:
So as not to be misled by an individual publication. Individual studies are subject to publication bias- the disproportionate publication of positive studies to the exclusion of negative studies. Individual studies are also just one of a distribution of findings.
Evaluate the consistency of the effect. Is the magnitude of the effect similar across studies? If so this is reassuring. If not you will have to figure out why and what it means.
Individual studies can be flawed/biased and a body of evidence is less likely to be flawed (though it could be).
– Multiple small underpowered studies can become more useful when combined (ie metaanlysis). If you only read the one underpowered study you might conclude that the intervention is useless.

A colleague of mine has recently changed his prescribing habits for diuretics in the treatment of hypertension. He based this decision on the recent publication of a single study comparing hydrochlorothiazide to chlorthalidone. When I examined the totality of the evidence (2 meta-analyses I come to a different conclusion than he did (I will be posting a blog on this in the near future).

So what does this mean as you try to answer a clinical question? What about when you are just reading through your weekly journal?
1) Use synthesized evidence first to answer questions. These are higher up in the Haynes’ hierarchy (the figure below) and included evidence based textbooks like Dynamed. Second line would be synopses of systematic reviews (called syntheses in the Haynes’ hierarchy). Next would be a systematic review (synopsis).
Haynes' 6S Hierarchy

2) After reading an individual study look for systematic reviews on the same intervention. Authors will sometimes refer to them in the intro and discussion sections of their paper. Look to see how the current paper compares to the rest of the evidence so you can put the new findings in perspective.

The limitation of this approach is time (although reading an individual study actually takes more time than reading a synthesis) and access to resources. There is no reason for a practicing clinician not to have access to an evidence based resource like Dynamed (or UpToDate- though I hesitantly put this here for my own reasons). Costs of these resources is reasonable and is money better spent than journal subscriptions.

Emotions Over Evidence

I believe in following the evidence as much as possible when practicing medicine. “We” have decided in this country that teaching EBM skills is important and these concepts are part of every medical school’s curriculum. Why don’t we teach evidence-based living? So much of what we do and believe in is emotionally based. There is lots of evidence around us but our emotions make us ignore it. The gun control debate we are having in this country really underscores this. There are no randomized controlled trials of gun control. There never will be. What we have are observational studies. Like all evidence it is filtered through each individual’s prism and gets bent in a variety of directions.

There is no convincing either side of the gun debate. I like guns. I have several guns. My emotions tell me they are not the problem (None of my guns has ever attacked me. Thus my belief that guns dont kill). I BELIEVE there is evidence supporting my point of view. But so does the gun control community. Just like in many facets of medicine there is a limited body of evidence that both sides (pro-gun vs anti-gun) look at differently because of their emotional investment in the subject. When you have a tragedy like Connecticut emotions further overwhelm the evidence. There were many laws broken that day. More laws wouldn’t have prevented that tragedy and won’t prevent a tragedy like that in the future. Why can’t they see this evidence?

When a women is diagnosed with early stage  breast cancer because of a screening mammogram she believes it saved her life. So too will her doctor. There is an emotional attachment to the interpretation of this event. But what about the evidence? Most breast cancer detected by screening will never hurt you (N Engl J Med 2012;  367:1998-2005). But try to recommend against mammographic screening and emotions take over. Evidence be damned!

It’s very difficult to disconnect your emotions from evidence. Especially if that evidence goes against what you have been taught….what you see in your limited world….what you believe! I guess my point of this rant is to recognize that you have to take a dispassionate look at evidence…in all aspects of life. The only filter it should go through is the validity filter- is the evidence likely unbiased. Leave your beliefs and emotions out if it. You might be surprised by what you see.