There has been an interesting discussion on how to critically appraise a study on the Evidence-Based Health listserv over the last day. It is interesting to see different opinions on the role of critical appraisal.
One important thing to remember as pointed out by Ben Djulbegovic is that critical appraisal relies on the quality of reporting as these 2 studies showed: http://www.ncbi.nlm.nih.gov/pubmed/22424985 and http://www.ncbi.nlm.nih.gov/pmc/articles/PMC313900/ . The implications are important but difficult for the busy clinician to deal with.
There are 3 questions you should ask yourself as you read a clinical study:
- Are the findings TRUE?
- Are the findings FREE OF THE INFLUENCE OF BIAS?
- Are the findings IMPORTANT?
The most difficult question for a clinician to answer initially is if the findings are TRUE. This question gets at issues of fraud in a study. Thankfully major fraud(ie totally fabricated data) is a rare occurrence. Totally fraudulent data usually gets exposed over time. Worry about fraudulent data when the findings seem too good to be true (ie not consistent with clinical experience). Usually other researchers in the area will try to replicate the findings and can’t. There are other elements of truth that are more subtle and occur more frequently. For example, did the authors go on a data dredging expedition to find something positive to report? This would most commonly occur with post hoc subgroup analyses. These should always be considered hypothesis generating and not definitive. Here’s a great example of a false subgroup finding:
The Second International Study of Infarct Survival (ISIS-2) investigators reported an apparent subgroup effect: patients presenting with myocardial infarction born under the zodiac signs of Gemini or Libra did not experience the same reduction in vascular mortality attributable to aspirin that patients with other zodiac signs had.
Classical critical appraisal, using tools like the Users’ Guides, is done to DETECT BIASES in the design and conduct of studies. If any are detected then you have to decide the degree of influence that the bias(es) has had on the study results. This is difficult, if not impossible, to determine with certainty but there are studies that estimate the influence of various biases (for example, lack of concealed allocation in a RCT) on study outcomes. Remember, most biases lead to overestimate of effects. There are 2 options if you detect biases in a study: 1) reduce the “benefit” seen in the study by the amounts demonstrated in the following table and then decide if the findings are still important enough to apply in patient care, or 2) discard the study and look for one that is not biased.
This table is synthesized from findings reported in the Cochrane Handbook (http://handbook.cochrane.org/chapter_8/8_assessing_risk_of_bias_in_included_studies.htm)
BIAS EXAGGERATION OF EFFECT OF BENEFIT
Lack of randomization 25% (-2 to 45%)
Lack of allocation concealment 18% (5 to 29%)
Lack of blinding 9% (NR)
Finally, if you believe the findings are true and are free of significant bias, you have to decide if they are CLINICALLY IMPORTANT. This requires clinical judgment and understanding the patient’s baseline risk of the bad outcome the intervention is trying to impact. Some people like to calculate NNTs to make this decision. Don’t just look at the relative risk reduction and be impressed because you can be misled by this measure as I discuss in this video: https://youtu.be/7K30MGvOs5s
I have always suspected that one reason that physicians don’t critically appraise articles is that the criteria for critical appraisal are not readily available in a convenient, easy to use package. No more. I, with the help of some undergraduate computer science students, have created a critical appraisal app for Android devices. Its in the Google playstore and will be listed in the Amazon app store. Hopefully will develop an iOS version if this version is successful.
I tried to take critical appraisal to the next step by “scoring” each study and giving an estimate of the bias in the study. I then make a recommendation of whether or not the user should trust the study or reject it and look for another study. I think one of the shortcomings of the Users’ Guides series is that no direction is given to the user about what to do with the article after you critically appraise it. EBM Rater will give a suggestion about the trustworthiness of the study.
EBM Rater contains criteria to critically appraise all the major study designs including noninferiority studies. It even contains criteria to evaluate surrogate endpoints, composite endpoints, and subgroup effects.
Finally, it contains standard EBM calculators like NNT, NNH, and posttest probability. I added 2 unique calculators that I have not seen in any other app: patients specific NNT and NNH. Many of our patients are sicker or healthier that the patients included in a study. NNTs and NNHs are typically calculated with data from a study so the NNT and NNH is for the study patients. With my calculator you can figure out your individual patient’s NNT or NNH.
I hope you will give it a try and give me some feedback.
Just about every internal medicine residency program has a journal club. One could argue about the evidence behind this activity but it seems to serve its purpose if nothing else than to make housestaff read some journal articles (and not just UpToDate!). I think it does serve a purpose of encouraging critical appraisal/thinking about research publications. Doctors will always have to read new research studies. It takes time for studies to be incorporated into secondary publications like Dynamed and UpToDate. Furthermore, not everything makes it into these evidence-based resources. Also research (published in every journal) is full of biases that lead to departure of the findings from the truth. Critical appraisal is the only way to detect them.
This is not our flier but one I found on the internet that I thought was interesting
Since 1999 or so I have been intimately involved in the journal club at UAB. At times I have run it completely but now I serve more as a guide and EBM expert for one of the chief residents who puts it all together. I think it has gotten greater buy-in from the housestaff coming from the CMR instead of me.
So I thought I would cover some of what we have done at UAB. Not that we are the world’s beacon for journal club but we have tried alot of stuff over the years. Some of it failed….some of it successful.
Time of day: we have done everything from 8am, noon, to at night at a faculty member’s house. What has gotten the best turnout is 8am before their day gets started.
Article Selection: This has been a debatable topic since day 1. We have done several things:
1) Latest articles in major journals
2) Rotating subspecialty articles (one month cardiology, one month GI, etc)
3) Article chosen by resident based on problems they saw during patient care
4) Article chosen by me to prove an EBM principle
5) Now we seem to be focusing on articles written by UAB faculty so that they can come as an expert guest.
6) We are considering using classics in medicine articles that are the foundation of what we do (eg first article on ACE inhibitors in CHF) because current residents are unlikely to ever read these articles.
Format: We seem to vary this almost yearly:
1) Faculty reviews article and asks questions of the housestaff about what various things mean
2) Teams of residents argue for or against using a drug, etc against another team of residents
3) Each individual reads the article and comes to JC not knowing what they could potentially be asked
4) A handout with Users Guides questions and a few other questions on design or applying the information is given out ahead of time but is only discussed by those willing to answer
5) Same handout given but with individual residents assigned specific questions to answer (this was the first time we could show that the residents actually read the paper ahead of time)
6) Groups of residents work on questions outside of JC on their own time (usually 3rd yr resident assigned to coordinate the group meeting) with the expectation to teach the other groups at JC. (this worked pretty good actually)
7) Last year we went to a flipped learning format where I put alot of material on edmodo.com that the residents were to do ahead of time (if they needed to) with assigned questions to be answered by individual residents. They felt like this was too much work to go thru all the material online.
8) This year we are to perhaps our most successful format (from resident satisfaction standpoint) where a handout of questions is answered in JC as a group project. A faculty expert gives a very short didactic talk at 2 points during JC on a very specific EBM topic related to the article (eg what is a likelihood ratio). The only expectation is that the article is read prior to JC. We still use somewhat of a flipped format where I reference a short video or 2 to watch about topics in the chosen article but its much less time intensive than last year.
I think overall what has been successful for us is when JC has the following elements:
1) Group work. Engaged learning is always desirable.
2) Clinical and EBM faculty expert present. Seems to give the article a little more value.
3) Case-based. We always solve a real world problem. I always tell the CMR making up JC to make sure the residents walk away with something they can use clinically.
4) Flipped light– giving the residents some information, but not too much, that they can review about EBM principles leads to many of them actually watching the videos or reading background papers. They come much more prepared and have a good basic knowledge that we can then build upon.
Physicians mainly read the abstract of a journal article (JAMA 1999;281:1129). I must admit I am guilty of this also. Furthermore, I would bet that the most often read section of the entire article is the conclusions of the abstract. We are such a soundbite society.
I had always thought the literature showed how bad abstracts were…that they were often misleading compared to the body of the article. But I was wrong. A recent study published in BMJ EBM found that 53.3% are abstracts had a discrepancy compared to information in the body of the article. That sounds bad doesn’t it? But only 1 of them was clinically significant. Thus most of the discrepancies were not important enough to potentially cause patient harm or alter a clinical decision.
This is good news as effectively practicing EBM requires information at the point of care. Doctors don’t have time to read an entire article at the point of care for every question they have but they do have time to read an abstract. It’s good to know that structured abstracts (at least from the major journals that were reviewed in this study) can be relied upon for information. I especially like reading abstracts in evidence based journals like BMJ EBM or ACP Journal Club as even their titles give the clinical information you need.