EBM Rater is finally available

I have always suspected that one reason that physicians don’t critically appraise articles is that the criteria for critical appraisal are not readily available in a convenient, easy to use package. No more. I, with the help of some undergraduate computer science students, have created a critical appraisal app for Android devices. Its in the Google playstore and will be listed in the Amazon app store. Hopefully will develop an iOS version if this version is successful.

 

screen shot

I tried to take critical appraisal to the next step by “scoring” each study and giving an estimate of the bias in the study. I then make a recommendation of whether or not the user should trust the study or reject it and look for another study. I think one of the shortcomings of the Users’ Guides series is that no direction is given to the user about what to do with the article after you critically appraise it. EBM Rater will give a suggestion about the trustworthiness of the study.

EBM Rater contains criteria to critically appraise all the major study designs including noninferiority studies. It even contains criteria to evaluate surrogate endpoints, composite endpoints, and subgroup effects.
screen shot

Finally, it contains standard EBM calculators like NNT, NNH, and posttest probability. I added 2 unique calculators that I have not seen in any other app: patients specific NNT and NNH. Many of our patients are sicker or healthier that the patients included in a study.  NNTs and NNHs are typically calculated with data from a study so the NNT and NNH is for the study patients. With my calculator you can figure out your individual patient’s NNT or NNH.

screen shot

I hope you will give it a try and give me some feedback.

Allocation Concealment Is Often Confused With Blinding

During journal clubs on randomized controlled trials there is often confusion about allocation concealment. It is often confused with blinding. In a sense it is blinding but not in the traditional sense of blinding. One way to think of allocation concealment is blinding of the randomization schedule or scheme. Allocation concealment hides the randomization or allocation sequence (what’s coming next) from patients and those who would enroll patients in a study. Blinding occurs after randomization and keeps patients, providers, researchers, etc from knowing which arm of the study the patient is in (i.e. what treatment they are getting).

Why is allocation concealment important in a randomized controlled trial? Inadequate or unclear allocation concealment can lead to an overestimation (by up to 40%!) of treatment effect (JAMA 1995;273:408). First, consider why we randomize in the first place. We randomize to try to equally distribute confounding and prognostic factors between arms of a study so we can try to isolate the effect of the intervention. Consider a physician who wants to enroll a patient in a study and wants to make sure her patient receives the therapy she deems likely most effective. What if she figured out the randomization scheme and knows what therapy the next patient will be assigned to? Hopefully you can see that this physician could undermine the benefits of randomization if she preferentially funnels sicker (or healthier) patients into one arm of the study. There could be an imbalance in baseline characteristics. It could also lead to patients who are enrolled in the study being fundamentally different or not representative of the patient population.

From The Lancet

From The Lancet

You will have to use your judgment to decide how likely it is that someone could figure out the randomization scheme. You can feel more comfortable that allocation concealment was adequate if the following were used in the RCT:
sequentially numbered, opaque, sealed envelopes: these are not able to be seen through even if held up to a light. They are sealed so that you can’t peek into them and see what the assignment is. As each patient is enrolled you use the next numbered envelope.
pharmacy controlled: enrolling physician calls the pharmacy and they enroll the patient and assign therapy.
centralized randomization: probably the most commonly used. The enrolling physician calls a central research site and the central site assigns the patient to therapy.

Proper randomization is crucial to a therapy study and concealed allocation is crucial to randomization. I hope this post helps readers of RCTs better understand what concealed allocation is and learn how to detect whether it was done adequately or not. Keep in mind if allocation concealment is unclear or done poorly the effect you see in the study needs to be tempered and possible cut by 40%.

Journal Club- The UAB Experience

Just about every internal medicine residency program has a journal club.  One could argue about the evidence behind this activity but it seems to serve its purpose if nothing else than to make housestaff read some journal articles (and not just UpToDate!). I think it does serve a purpose of encouraging critical appraisal/thinking about research publications. Doctors will always have to read new research studies. It takes time for studies to be incorporated into secondary publications like Dynamed and UpToDate. Furthermore, not everything makes it into these evidence-based resources.  Also research (published in every journal) is full of biases that lead to departure of the findings from the truth. Critical appraisal is the only way to detect them.

journal club

This is not our flier but one I found on the internet that I thought was interesting

Since 1999 or so I have been intimately involved in the journal club at UAB. At times I have run it completely but now I serve more as a guide and EBM expert for one of the chief residents who puts it all together. I think it has gotten greater buy-in from the housestaff coming from the CMR instead of me.

So I thought I would cover some of what we have done at UAB. Not that we are the world’s beacon for journal club but we have tried alot of stuff over the years. Some of it failed….some of it successful.

Time of day: we have done everything from 8am, noon, to at night at a faculty member’s house. What has gotten the best turnout is 8am before their day gets started.

Article Selection: This has been a debatable topic since day 1. We have done several things:
1) Latest articles in major journals
2) Rotating subspecialty articles (one month cardiology, one month GI, etc)
3) Article chosen by resident based on problems they saw during patient care
4) Article chosen by me to prove an EBM principle
5) Now we seem to be focusing on articles written by UAB faculty so that they can come as an expert guest.
6) We are considering using classics in medicine articles that are the foundation of what we do (eg first article on ACE inhibitors in CHF) because current residents are unlikely to ever read these articles.

Format: We seem to vary this almost yearly:
1) Faculty reviews article and asks questions of the housestaff about what various things mean
2) Teams of residents argue for or against using a drug, etc against another team of residents
3) Each individual reads the article and comes to JC not knowing what they could potentially be asked
4) A handout with Users Guides questions and a few other questions on design or applying the information is given out ahead of time but is only discussed by those willing to answer
5) Same handout given but with individual residents assigned specific questions to answer (this was the first time we could show that the residents actually read the paper ahead of time)
6) Groups of residents work on questions outside of JC on their own time (usually 3rd yr resident assigned to coordinate the group meeting) with the expectation to teach the other groups at JC. (this worked pretty good actually)
7) Last year we went to a flipped learning format where I put alot of material on edmodo.com that the residents were to do ahead of time (if they needed to) with assigned questions to be answered by individual residents. They felt like this was too much work to go thru all the material online.
8) This year we are to perhaps our most successful format (from resident satisfaction standpoint) where a handout of questions is answered in JC as a group project. A faculty expert gives a very short didactic talk at 2 points during JC on a very specific EBM topic related to the article (eg what is a likelihood ratio). The only expectation is that the article is read prior to JC. We still use somewhat of a flipped format where I reference a short video or 2 to watch about topics in the chosen article but its much less time intensive than last year.

I think overall what has been successful for us is when JC has the following elements:
1) Group work. Engaged learning is always desirable.
2) Clinical and EBM faculty expert present. Seems to give the article a little more value.
3) Case-based. We always solve a real world problem. I always tell the CMR making up JC to make sure the residents walk away with something they can use clinically.
4) Flipped light– giving the residents some information, but not too much, that they can review about EBM principles leads to many of them actually watching the videos or reading background papers. They come much more prepared and have a good basic knowledge that we can then build upon.

Jazzing Up Journal Club

I am preparing to write something about what I think works well for journal club. We’ve tried lots of different things here at UAB. I would like to hear from you about what you do for your journal club. What works well in your journal club? What doesn’t work so well?

We need a dialogue here so please enter some comments. I’ll collate what I get and then give my own thoughts about Jazzing Up Journal Club.

Journal Club- Basic Stats: Answers

Here are my answers to the journal club questions. I have also added links to some of my youtube videos to answer questions

1) The authors designed the study to have a “power of more than 80%“. What does this mean?
Power is the probability of the study finding a difference given that one truly exists. So this study was designed with at least an 80% chance of finding a difference between treatment and control groups (given that one truly exists). This video explains power in a little more depth.
2) What was the planned type 1 error rate in this study? Type 1 error is also called the alpha error. They planned on a 5% type 1 error rate. This video explains type 1 error in a little more detail
3) What is a type 2 error and how is it related to power? Type 2 error is also called beta error. It is related to power in that power is 1 (or 100%) minus the beta error. So if power is 80% the type 2 error rate is 20%. This video  explains type 2 error in more detail.
4) What are the determinants of sample size in this study? How does varying the estimates of these components affect sample size? Sample size is determined by a variety of factors: power, type 1 and 2 error rates, estimated difference between study groups and variability in the data (though this last one has less of an effect). See this video explaining these factors and their effect on sample size.
5) The authors use a variety of statistical tests (chi-square, Fisher’s exact, t-tests, etc) to analyze the data. In general, what do statistical tests do?
Statistical tests look at the data and calculate a test statistic (e.g. t statistic for a t test). The test statistic is then used to determine the p-value assosicated with the data.

Review Table 2 and answer the following questions:1) The primary outcome occurred in 1.92/100 person-yrs in the control group compared to 1.83/100 person-yrs in the intervention group. The p-value associated with this comparison is 0.51. What does this p-value mean? Can p-values be used to detect bias in the study? The simple interpretation is that the difference is not statistically significant because the p-value is > 0.05. Another interpretation would be that the difference seen between the groups or one more extreme is due 51% likely due to chance. P-values cannont detect bias (systematic errors) in a study. Critical appraisal detects bias.
2) The hazard ratio comparing the intervention group to the control group for the primary outcome is 0.95 with a 95% confidence interval of 0.83-1.09. What does this confidence interval tell you about the effect? Can confidence intervals be used to detect bias in the study? It tells you a couple of things: 1. that the difference is not statistically significant as the CI included the point of no difference…1.0 and 2. that the benefit could be up to 17% reduction in cardiovascular events or 9% increase. This video explains how to interpret hazard ratios and this video confidence intervals.

Finally the extra credit: These 4 things can explain study findings: truth, chance, bias, confounding

I hope this was somewhat helpful. I will have another journal club next month on another EBM topic.

Journal Club- Basic Stats: Cardiovascular Effects of Intensive Lifestyle Intervention in Type 2 Diabetes

I decided to start a new feature that hopefully you and I will find useful. It will only be useful if you work thru the questions and have a dialogue thru the comments section.

My plan is to post a different article about once a month and have questions for you to answer. 1 week later (or so) I will then either post a video review of my answers or just write about the answers. I plan to focus mostly on the basics but at times I will cover advanced topics also as “extra credit”. I am mostly going to parallel the journal club curriculum we are using this year at UAB. I welcome comments to make this better or articles you want to read.

journal club

58 yo M with DM-2, hyperlipidemia and HTN presents to you for a follow-up visit. He takes metformin 1000mg BID, lisinopril 20mg daily, and pravastatin 40mg nightly. His most recent HgA1C was 6.9% and LDL was 88 mg/dl. In the office his blood pressure is 128/67 mm Hg and BMI is 32. You counsel him to lose weight and he responds “My blood pressure, cholesteol and A1C are good. How is losing weight going to help my heart?” What do you tell him?

Article: The Look AHEAD Research Group. Cardiovascular Effects of Intensive Lifestyle Intervention in Type 2 Diabetes. NEJM 2013;369:145-54.

After reading the statistical analysis section (pgs 147-148) of the article answer the following questions:
1) The authors designed the study to have a “power of more than 80%“. What does this mean?
2) What was the planned type 1 error rate in this study?
3) What is a type 2 error and how is it related to power?
4) What are the determinants of sample size in this study? How does varying the estimates of these components affect sample size?
5) The authors use a variety of statistical tests (chi-square, Fisher’s exact, t-tests, etc) to analyze the data. In general, what do statistical tests do?

Review Table 2 and answer the following questions:1) The primary outcome occurred in 1.92/100 person-yrs in the control group compared to 1.83/100 person-yrs in the intervention group. The p-value associated with this comparison is 0.51. What does this p-value mean? Can p-values be used to detect bias in the study?
2) The hazard ratio comparing the intervention group to the control group for the primary outcome is 0.95 with a 95% confidence interval of 0.83-1.09. What does this confidence interval tell you about the effect? Can confidence intervals be used to detect bias in the study?

Extra Credit:
List the 4 things that can explain study findings