What is EBM?

With all the discussion of EBM in crisis and EBM on trial it strikes me that maybe these other folks have a different definition or concept of EBM than I do. I think to have any discussion needs to come from a common ground of just what is EBM.

Evidence based medicine is the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients.

This is the original definition of EBM published in 1996. It urged us to strive to use best available evidence in making clinical decisions. It also cautioned not to be a slave to evidence as evidence was often not applicable to individual patients. This definition served us well until the patient-centered paradigm of care became popular and the definition of EBM evolved to its current form:

This definition is more explicit about the order of importance of the individual elements of the components of EBM: patient preferences and actions is foremost, followed by the clinical state and circumstances and the research evidence. All this is tempered or tied together by our clinical expertise. The evidence tells us what could be done while the rest tells us what should be done.

The other way to look at EBM is that it is just a set of skills:

  1. asking an answerable clinical question
  2. finding the best available evidence
  3. critically appraising the evidence
  4. applying the evidence to individual patients
  5. appraising how well you did on each step and, I think, appraising the impact on a patient

So from this background I find it difficult to lay blame on EBM for many of the problems with the evidence. I blogged on this previously and will refute their claims at EvidenceLive2015 in April.

Will EBM be found guilty or not guilty?

Carl Heneghan recently wrote a blog for BMJ blogs entitled Evidence based medicine on trial focusing mostly on the problem with the evidence part of EBM. While I mostly concur with his list  of the problems (distortion of the research agenda, very poor quality research, and lack of transparency for published evidence) I wonder who is at fault. “EBM” seems to get the blame as if there is an entity that is EBM and it controls all research. EBM is but a set of skills: question asking, searching, critical appraisal, and application to individual patients. It is nothing more. So why are people being so critical and place so much blame on a set of skills? There will be several sessions at EvidenceLive 2015 (one of which I will be speaking at in defense of EBM) on real vs rubbish EBM.

I want to focus on the distortion of the research agenda. Professor Heneghan rightly points out that the research agenda is driven by industry. Is that good or bad? I think its both but mostly good. The only other major funders of research  are governmental agencies like the NIH. Profit drives innovation. It is very expensive to bring a drug to market. The government could not afford to bring the current drugs we have and need to market. One failed drug alone would deplete the coffers. Failure is the biggest driver of cost. Fewer than 1 in 10 drugs tested makes it to market. Would we tolerate that poor of a success rate at such a big cost by the government? No.

…adjusting that estimate for current failure rates results in an estimate of $4 billion in research dollars spent for every drug that is approved.

I agree that industry seems at times to make a drug then find a “disease” for it. I think the example Professor Heneghan gives is spot on. I don’t believe in adult ADHD but we have drugs for it. Do we need them? No and this video demonstrates why:  Drug free treatment of ADHD. Who is really at fault are the doctors who prescribe the drugs that Professor Heneghan feels aren’t necessary. Not the companies for making them.

On a serious note…what about all the devices we use regularly like stents, defibrillators, etc? Would government have independently brought these to market? Likely not. We had balloon angioplasty (without stenting) that worked just fine albeit short term only. It would have been “good enough for government work” as the saying goes. What about advancements in imaging modalities? Again likely not. The old CT scanners worked just fine. Industry is largely responsible for innovation and improvement in all walks of life. Yes for a profit but profit is not a bad thing. Those who say otherwise please return your iPhones.

2014 in review

The WordPress.com stats helper monkeys prepared a 2014 annual report for this blog.

Here’s an excerpt:

The concert hall at the Sydney Opera House holds 2,700 people. This blog was viewed about 8,400 times in 2014. If it were a concert at Sydney Opera House, it would take about 3 sold-out performances for that many people to see it.

Click here to see the complete report.

Publication Bias is Common in High Impact Journal Systematic Reviews

A very interesting study was published earlier this month in the Journal of Clinical Epidemiology assessing publication bias reporting in systematic reviews published in high impact factor journals.  Publication bias refers to the phenomenon that statistically significant positive results are more likely to be published than negative results. They also tend to be published more quickly and in more prominent journals. The issue of publication bias is an important one because the goal of a systematic review is to systematically search for and find all studies on a topic (both published and unpublished) so that an unbiased estimate of effect can be determined from including all studies (both positive and negative). If only positive studies, or a preponderance of positive studies, are published and only these are included in the review then a biased estimate of effect will result.

Onishi and Furukawa’s study is the first study to examine the frequency of significant publication bias in systematic reviews published in high impact factor general medical journals. They identified 116 systematic reviews published in the top 10 general medical journals in 2011 and 2012: NEJM, Lancet, JAMA, Annals of Internal Medicine, PLOS Medicine, BMJ, Archives of Internal Medicine, CMAJ, BMC Medicine, and Mayo Clinic Proceedings. They assessed each of the systematic reviews that did not report an assessment of publication bias for publication bias using Egger test of funnel plot asymmetry, contour-enhanced funnel plots, and tunnel effects. RESULTS: The included systematic reviews were of moderate quality as shown in the graph below. About a third of “systematic reviews” didn’t even perform a comprehensive literature search while 20% didn’t  assess study quality. Finally, 31% of systematic reviews didn’t assess for publication bias. How can you call your review a systematic review when you don’t perform a comprehensive literature search and you don’t determine if you missed studies?

Quality of included reviews

From J Clin Epi 2014;67:1320

Of the 36 reviews that did not report an assessment of publication bias, 7 (19.4%) had significant publication bias. Saying this another way, if a systematic review didn’t report an assessment of publication bias there was about a 20% chance publication bias was present. The authors then assessed what impact publication bias had on the estimated pooled results and found that the estimated pooled result was OVERESTIMATED by a median of 50.9% because of publication bias. This makes sense as mostly positive studies are published and negative studies aren’t. Thus, you would expect the estimates to be overly optimistic.

The figure below reports the results for individual journals. JAMA had significant publication bias in 50% of the reviews that didn’t assess publication bias while the Annals had 25% and BMJ 10%. It is concerning that these high impact journals publish “systematic reviews” that are of moderate quality and have a significant number of reviews that don’t report any assessment of publication bias.

Results by journal

From J Clin Epi 2014;67:1320

Bottom Line: Always critically appraise systematic reviews published in high impact journals. Don’t trust that an editor, even of a prestigious journal, did their job….they likely didn’t.

Mix up your EBM teaching

Actually you should mix up all your teaching. Keep it fresh and interesting to your learners. I am getting a Master in Educational Technology and as part of one of my classes I had to design a WebQuest activity. You can learn more about WebQuests from WebQuest.org.

A WebQuest is an inquiry-oriented lesson format in which most or all the information that learners work with comes from the web.

I developed a WebQuest on determining pretest probability. You can look at my WebQuest here. It is geared toward 3rd and 4th year medical students but could be used with residents also. Feel free to use it.

probabilityheader2

While using the Web to teach is nothing new it is good to change up how you teach. Instead of lecturing on determining pretest probability have your learners do this WebQuest on their own time and flip that part of your lesson on diagnostic testing. It can also be done during class time and many of the WebQuests on WebQuest.org are designed for in-class use. While I had to actually develop/code the website there are online WebQuest creators (just look on the WebQuest.org site for the Creating WebQuests tab).

Why do clinicians continue medications with questionable benefit in advanced dementia?

A recent study in JAMA Internal Medicine estimated the prevalence of medications with questionable benefit being used by nursing home residents with advanced dementia. This is an important question because significant healthcare resources are utilized in the last 6 months of life. Furthermore, if there is no benefit then the only possible outcomes can be excess cost with or without harm. As the authors note most patients at this stage just want comfort care and maximization of quality of life.

The researchers studied medication use deemed of questionable benefit by nursing home residents with advanced dementia using a nationwide long-term care pharmacy database.  A panel of geriatricians and palliative medicine physicians defined a list of medications that are of questionable benefit when the patient’s goal of care is comfort and included cholinesterase inhibitors, memantine hydrochloride, antiplatelets agents (except aspirin), lipid-lowering agents, sex hormones, hormone antagonists, leukotriene inhibitors, cytotoxic chemotherapy, and immunomodulators.

53.9% nursing home residents with advanced dementia were prescribed at least 1 questionably beneficial medication during the 90-day observation period with cholinesterase inhibitors (36.4%), memantine hydrochloride (25.2%), and lipid-lowering agents (22.4%) being most commonly prescribed. Patients residing in facilities with a high prevalence of feeding tubes were more likely to be prescribed these questionably beneficial medications.

Of only those residents who used at least 1 questionably beneficial medication, the mean (SD) 90-day drug expenditure was higher ($2317 [$1357]; IQR, $1377-$2968 compared to $1815 for all residents), of which 35.2% was attributable to medications of questionable benefit (mean [SD], $816 [$553]; IQR, $404-$1188).

Picture of elderly patient

I think this study demonstrates excess medication usage and excess costs in a population in which costs are already high and for which this added cost is of little benefit. So why are these medications continued in this population? Are physicians unaware of the lack of benefit of these medications in this population? Are they aware but worried that stopping them will make the patient worse? I suspect a little of both is the correct answer.

A major challenge for EBM is getting the E out there. Numerous resources are available but a major step in accessing a resource is recognizing a knowledge deficit. How do you know you don’t know something? Pushing evidence (for example by email) is useful for general knowledge but isn’t useful to answer specific questions. Most of the push email services require enrollment to get the emails and many clinicians probably don’t even know this exists. Maintenance of certification could be a useful tool to improve knowledge if it was designed properly but can’t cover everything for all clinicians.

So we have a dilemma. Studies like this show areas for improvements in knowledge and practice but there are no great practical ways to improve either in a nursing home setting. Clinical reminders still have to be acted upon. Payers could refuse to pay for certain services but docs will likely continue to order them with the patient picking up the bill. Protocols could be put in place but they have to be followed and agreed upon by clinicians. They all will have anecdotal evidence of grandpa getting worse when his cholinesterase inhibitor was stopped. And they will say “What’s the harm in continuing it?”.

Affect Heuristic,COI, or Lack of Knowledge? Why Do Cardiologists Overestimate Benefits of PCI in Stable Angina?

A recent study in JAMA Internal Medicine by Goff and colleagues made me wonder if the Cardiologists studied are uninformed of the limited benefits of stenting (PCI) for chronic stable angina, do they have too strong of a conflict of interest due to economic gain, or is the affect heuristic playing a big part? Probably  a mixture of them all. The COURAGE Trial taught us that PCI was better than medical therapy at reducing anginal symptoms but wasn’t any better for reducing MI and death.

Goff and colleagues reviewed 40 recordings of actual encounters of Cardiologists with patients being considered for cardiac catheterization and PCI. I am unsure if these were video recordings or audio recordings. As best I can tell these were all private practice Cardiologists.  Cardiologists either implicitly or explicitly overstated the benefits of angiography and PCI. They presented medical therapy as being inferior to angiography and PCI (a statement that defies the findings of the COURAGE trial). In fact, in only 2 of the encounters did they state PCI would not reduce the risk of death or MI. These Cardiologists also didn’t use good communications styles that encouraged patient participation in the decision making process.

Why might these Cardiologists do this? They could be uninformed of the limited benefits of PCI in stable angina, but I doubt it. COURAGE was a landmark publication in one of the world’s most prominent medical journals. I find it hard to believe that Cardiologists wouldn’t be aware of the results of this trial.

They certainly have a financial stake in their recommendations.  The image below shows that a diagnostic cath is reimbursed at approximately $9,000 while a PCI with DES is reimbursed at approximately $15,000. That has to have an impact on decision making. I don’t accuse these Cardiologists of doing a procedure only for money but subconsciously this is playing a role. Recommending medical therapy only gets you an office visit reimbursement (maybe $200 or so).

What about the affect heuristic? My colleague Bob Centor writes about this often in his blog. A heuristic is a quick little rule we use to make decisions. The affect heuristic is a particular rule we use that is based on our emotions about a topic. Do I like it? Do I hate it? How strongly do I feel about it? The affect heuristic leads to the answer to an easy question (How do I feel about something?) serving as the answer to a much harder question (What do I think about something?) Its not hard to imagine (and data in the Goff paper supports this) a Cardiologist feeling that PCI is beneficial and should be done. They are emotionally tied to angiography and PCI….they have seen patients “saved” because of this procedure.

So what can be done? The solution is harder than determining the problem (as is often the case). The easiest solution is for insurance companies to stop reimbursing for the procedure in stable angina unless patients have failed optimal medical therapy but this is draconian. I also worry that patients will then receive bills for unreimbursed  catheterization charges. I think using the technology that was used in this study combined with feedback could be useful but logistically impossible. I have always wondered why we don’t use secret shopper fake patients to evaluate physician skills and knowledge (of course the answer is a logistic one) instead of the MOC system. Just publishing a study doesn’t work if physicians don’t read or if they don’t use that study to answer a clinical question. Patient decision aids (like this excellent example) could be very useful but the physician would have to use the tool and many don’t even know they exist.

Some would argue EBM has failed again. A well done study was published and it hasn’t made a difference. The principles of EBM have not failed and in fact, if they were used, could limit the inappropriate use of PCI in stable angina patients. What has failed is the desire to learn and use these skills by the older Cardiologists in this study. Like many physicians, they rely on outdated knowledge and emotions or beliefs. As stated by Bob Centor in a post about the affect heuristicDecision making bodies have biases. Until they understand their biases, we will have the problem of unfortunate, unnecessary and potential dangerous unintended consequences“. In this case the Cardiologists are the decision making bodies and the unintended consequences are the MIs, strokes, renal injury, and death that can and do occur from cardiac catheterization.

Hopefully you are now aware of what the affect heuristic is and how it impacts decision making. Acknowledge it and separate your feelings about a topic from the data. Your patients will benefit.