Tarnished Gold Chapter 3: Prove It

This chapter dealt with the issues of what constitutes evidence. Instead of focusing on their views I will focus on my views of evidence.


A common criticism of EBM is that it very strict in what it considers  acceptable evidence and it doesn’t consider clinical experience and pathophysiological rationale as important. Early EBM did focus too much on the RCT and Cochrane systematic reviews but this has changed. The current EBM paradigm focuses on multifactorial “evidence” including the patient’s clinical state and circumstances, clinical experience, and the best available evidence. Sometimes this will be a systematic review but often it will just be patient experience (what worked or didn’t work for them in the past) or pathophysiology. The early EBM paradigm cautioned us that we can be misled by our unsystematic observations and the pathophysiological rationale. For the latter, it’s because our understanding of pathophysiology changes and diseases are complex and multifactorial and interventions we study tend to be unifactorial. Nonetheless, clinical experience is evidence and is very important and no EBMer will say otherwise. Understanding pathophysiology is important and no EBMer will say otherwise. The key is to understand the limitations of any evidence source.

Evidence supports a belief and doesn’t have to be true. In clinical medicine we can never know the truth. We can only try to estimate the truth with a study because we can’t study every person with a given disease. We have to infer a lot. We generalize from a sample in a study to a whole population and back down to an individual patient. The authors of Tarnished Gold have a real problem with this paradigm but it’s what we do in clinical medicine. Bench research works differently. Rats can all be genetically and phenotypically the same. Bacteria can all be clones of each other. Bench scientists can study a whole population of something and declare an effect. We can’t do this in clinical medicine because we are all so heterogeneous and have free will.

EBM no longer worships only the RCT and the Cochrane review. Patient inputs are viewed as very important and slowly becoming equally important. Qualitative studies are gaining importance. Clinical experience will always be prominent in deciding what should be done from what could be done.

Useful diagram to teach basic EBM concepts

Dr. La Rochelle published an article in BMJ EBM this month with a very useful figure in it (see below). It is useful because it can help our learners (and ourselves) remember the relationship between the type of evidence and its believability/trustworthiness.


Lets work through this figure. The upright triangle should be familiar to EBM aficionados as it is the typical hierarchy triangle of study designs, with lower quality evidence at the bottom and highest quality at the top (assuming, of course, that the studies were conducted properly). The “Risk of Bias” arrow next to this upright triangle reflects the quality statement I just made.  Case reports and case series, because they have no comparator group and aren’t systematically selected are at very high risk of bias. A large RCT or systematic review of RCTs is at the lowest risk of bias.

The inverted triangle on the left reflects possible study effects, with the width of the corresponding area of the triangle (as well as the “Frequency of Potential Clinically relevant observable effect arrow) representing the prevalence of that effect. Thus, very dramatic, treatment altering effects are rare (bottom of triangle, very narrow). Conversely, small effects are fairly common (top of triangle, widest part).

One way to use this diagram in teaching is to consider the study design you would choose (or look for) based on the anticipated magnitude of effect. Thus, if you are trying to detect a small effect you will need a large study that is methodologically sound. Remember bias is a systematic error in a study that makes the findings of the study depart from the truth. Small effects seen in studies lower down the upright pyramid are potentially biased (ie not true). If you anticipate very large effects then observational studies or small RCTs might be just fine.

An alternative way to use this diagram with learners is to temper the findings of a study. If a small effect is seen in a small, lower quality study they should be taught to question that finding as likely departing from the truth. Don’t change clinical practice based on it, but await another study. A very large effect, even in a lower quality study, is likely true but maybe not as dramatic as it seems (ie reduce the effect by 20-30%).

I applaud Dr. La Rochelle for developing a figure which explains these relationships so well.

What’s The Evidence For That? A new series I am starting

One thing I have noticed is that current residents don’t seem to know the evidence supporting a lot of the treatments they use, especially if the studies were done before they started med school.  I also don’t think they really want to read articles from the past when there is so many new things that excite them more. So I came up with a new series of data summaries I am going to make for teaching purposes during rounds to remind the residents and students that there is evidence behind some of what we do.  I designed them to be one pagers and answer what I feel are the key questions on the particular topic I am covering. I also try to follow the Hayne’s 6 S Hierarchy and focus on evidence higher up the pyramid (that isn’t UpToDate or Dynamed). I want to hit the less sexy topics that we encounter a lot on the inpatient medicine service like COPD exacerbation, hepatic encephalopathy, etc.

So here’s the first one I made: What’s The Evidence For That: Steroids for COPD Exacerbation. (Steroids for AECOPD) This took about 1.5-2 hours to make… mostly because I had to figure out how to make the template in Publisher do what I wanted it to do (and it fought me all the way).

Feel free to copy it and use it in your clinical teaching. Let me know if it is useful and how it could be made better. If you make any share them with me.

As I make more of these I will publish them here. I also plan to make Touchcasts of them and will post that here when I do.