Critical appraisal skills are essential to informed decision-making

Rahul Mhaskar¹, Patricia Emmanuel², Shobha Mishra³, Sangita Patel³, Eknath Naik⁴, Ambuj Kumar^5
1 Center for Evidence-based Medicine and Health Outcomes Research; and Clinical and Translational Science Institute, University of South Florida, College of Medicine, Tampa, Florida, USA
² Clinical and Translational Science Institute and Department of Internal Medicine, University of South Florida, College of Medicine, Tampa, Florida, USA
³ Department of Preventive and Social Medicine, Government Medical College, Vadodara, India
⁴ Department of Internal Medicine, University of South Florida, College of Medicine, Tampa, Florida, USA
⁵ Center for Evidence-based Medicine and Health Outcomes Research; Department of Health Outcomes and Behavior, Moffitt Cancer Center; Clinical and Translational Science Institute, University of South Florida; and Department of Internal Medicine, University of South Florida, College of Medicine, Tampa, Florida, USA

Correspondence Address:
Ambuj Kumar
Division and Center for Evidence Based Medicine and Outcomes Research, Department of Internal Medicine, College of Medicine, University of South Florida, Clinical and Translational Science Institute, 12901 Bruce B Downs Blvd, MDC 3122, Tampa, FL 33612, USA

Source of Support: Fogarty International Center/USNIH: Grant #1D43TW006793-01A2-AITRP., Conflict of Interest: None

Check

14

DOI: 10.4103/0253-7184.62770

Abstract

Whenever a trial is conducted, there are three possible explanations for the results: a) findings are correct (truth), b) represents random variation (chance) or c) they are influenced by systematic error (bias). Random error is deviation from the 'truth' and happens due to play of chance (e.g. trials with small sample, etc.). Systematic distortion of the estimated intervention effect away from the 'truth' can also be caused by inadequacies in the design, conduct or analysis of a trial. Several studies have shown that bias can obscure up to 60% of the real effect of a healthcare intervention. A mounting body of empirical evidence shows that 'biased results from poorly designed and reported trials can mislead decision making in healthcare at all levels'. Poorly conducted and reported RCTs seriously compromise the integrity of the research process especially when biased results receive false credibility. Therefore, critical appraisal of the quality of clinical research is central to informed decision-making in healthcare. Critical appraisal is the process of carefully and systematically examining research evidence to judge its trustworthiness, its value and relevance in a particular context. It allows clinicians to use research evidence reliably and efficiently. Critical appraisal is intended to enhance the healthcare professional's skill to determine whether the research evidence is true (free of bias) and relevant to their patients.

Keywords: Critical appraisal, randomized controlled trials, decision-making

Introduction

The Need for Critical Appraisal

1. Are the results valid? Do the findings of this study represent the truth? That is, do the results provide an unbiased estimate of the treatment effect, or have they been clouded by bias leading to false conclusions?
2. How precise are the results? If the results are unbiased, they need further examination in terms of precision. The precision would be better in larger studies compared with smaller studies.
3. Are the results applicable to my patient? What are the patient populations, disease and treatments (including comparators) under investigation? What are the benefits and risks associated with the treatment? Do the benefits outweigh the harms?

1. Did the study address a clearly focused issue?
   
   Is there a clearly stated aim and research question? Ideally, a well-designed RCT should follow the acronym of PICOTS, which stands for patient, intervention, control, outcome, timing and setting, to formulate the research question. Is there a sound and understandable explanation of the population being studied (inclusion and exclusion criteria)? Is there an overview of which interventions are being compared? Are the outcomes being measured clearly stated with a rationale as to why these outcomes were selected for the study?



2. Was the assignment of participants to treatments randomized?
   
   Randomization aims to balance the groups for known and unknown prognostic factors by allocating patients to two groups by chance alone. The aim is to minimize the probabilities of treatment differences attributed to chance and to maximize the attribution to treatment effects. Therefore, it is important that the intervention and control groups are similar in all aspects apart from receiving the treatment being tested. Otherwise, we cannot be sure that any difference in outcome at the end is not due to pre-existing disparity. If one group has a significantly different average age or social class make-up, this might be an explanation of why that group did better or worse. The best method to create two groups that are similar in all important respects is by deciding entirely by chance, into which group a patient will be assigned. ^[5] This is achieved by random assignment of patients to the groups. In true randomization, all patients have the same chance as each other of being placed into any of the groups. ^[4,5] Allocation concealment ensures that those assessing eligibility and assigning subjects to groups do not have knowledge of the allocation sequence. In order to ensure adequate allocation concealment, centralized computer randomization is ideal and is often used in multicenter RCTs. The successful randomization process will result in similar groups. It is important to note that randomization will not always produce groups balanced for known prognostic factors. A large sample size will potentially reduce the likelihood of placing individuals with better prognoses in one group. ^[4],[6]



3. Were all the participants who entered the study properly accounted for at its conclusion? Description of dropouts and withdrawals: How do dropouts threaten validity?
   
   Participants in a RCT are considered lost to follow-up when their status or outcomes are not known at the end of the study. Dropouts might happen due to several natural reasons (e.g. participant relocation). It is assumed that such dropouts will not necessarily be a substantial number. However, when the dropout rate is large (e.g. >10%), it is a reason for concern. Often the reason that patients are lost to follow up relates to a systematic difference in their prognosis compared with patients who continue with a study until the end (e.g., patients lost to follow-up may have more adverse events or worse outcomes than those who continue). Therefore, the loss of many participants may threaten internal validity of the trial. Furthermore, if loss to follow-up and drop outs is different between two study groups, it may result in missing data that can disrupt the balance in groups created by randomization.
   
   The abovementioned questions should be adequate for screening purposes of the manuscript to help make a decision whether to continue assessing the article further. That is, if the answers to the first three questions are negative, it is not worth evaluating the findings, as the results from this study would not be considered reliable to be used in decision-making.

1. What are the results?
   
   The readers of the paper should be attentive as to whether authors of the study adequately describe the data collection that was employed in the study. These should be clearly described and justified. All outcome measures should be referenced and their validity reviewed. If data was self-reported by patients, it would need to be verified in some way for maximum credibility.
   
   Were the methods of analysis appropriate, clearly described and justified? Analysis should relate to the original aims and research questions. Choice of statistical analyses should be explained with a clear rationale. Any unconventional tests should be justified with references to validation studies.
   
   An important issue to consider in the analysis of a RCT is whether the analysis was performed according to the intention to treat (ITT) principle. According to this principle, all patients are analyzed in the same treatment arm regardless of whether they received the treatment or not. Intention to treat (ITT) analysis is important because it preserves the benefit of randomization. If certain patients in either arm dropped out, were non-compliant or had adverse events are not considered in the analysis, it is similar to allowing patients to select their treatment and the purpose of randomization is rendered futile.
   
   Alternatively, a 'per-protocol' or 'on treatment' analysis includes only patients who have sufficiently complied with the trial's protocol. Compliance covers exposure to treatment, availability of measurements and absence of major protocol violations and is necessary to measure the treatment-related harm (TRH) outcomes. Adhering to ITT analysis for assessment of TRH leads to biased estimates. ^[7],[8] In summary, analysis of benefits associated with treatments should be performed using the ITT principle and associated risks according to per protocol.



2. What are the key findings?
   
   The findings should answer the primary research question(s). Each outcome measure should be analyzed and its results presented with comparisons between the groups. Other issues of analysis include the magnitude of the effect, which can be measured by relative and absolute risk differences, odds ratios and numbers needed to treat. The significance of any differences between the groups should be discussed, with p-values given to indicate statistical significance (<0.05 being the common threshold for significance). The confidence intervals should be presented to demonstrate the degree of precision of the results. Was the sub-group analyses preplanned or derived post hoc (derived from a fishing expedition after data collection)? For example, in the RCT mentioned in the clinical scenario, tacrolimus in children aged 6 to 17 years suffering from atopic dermatitis had a statistically significant difference compared with vehicle group for the outcome of Physician's Global Evaluation of clinical response. ^[1] It is important to note that the study was adequately powered to address the primary question only. It would be incorrect if the investigators decided to address the issue of efficacy according to age groups (e.g. subgroup of 6 to 8 years old versus 9 to 17) or any other subgroups if it was not decided a priori. That is, subgroup analysis could be justified for hypothesis generation but not hypothesis testing. The validity of subgroup-treatment effects can only be tested by reproducing the results in future trials. Rarely are trials powered to detect subgroup effects. There is often a high false negative rate for tests of subgroup-treatment effect interaction when a true interaction exists. ^[9]
   
   The authors should include the descriptive analysis of the data (mean, median, standard deviation, frequencies etc.) and not just the results of statistical tests used. Most often, results are presented as dichotomous outcomes (yes or no). Consider a study in which 25% (0.25) of the control group died and 20% (0.20) of the treatment group died. The results can be expressed in many ways as shown in [Table 1]. ^[10]



3. Are the results applicable to my patient?
   
   How will the results help me work with my patients/clients? Can the results be applied to the local population of my practice and clients? How similar is the study sample to your own clients? Are there any key differences that you would need to consider for your own practice?
   
   In the case of our five-year-old with atopic dermatitis, even if the trial was performed flawlessly with positive results, if the population studied was not similar then the evidence is not applicable. An example of this would be if the study was performed in patients with severe atopic disease who had failed all other regimens but our patient was naοve to any therapy. If the population was a different age range or ethnicity, it could also impact relevance.
   
   Other important considerations are whether you have the necessary skills or resources to deliver the intervention. Were all the important outcomes considered? Has the research covered the most important outcomes for your patients? If key outcomes were overlooked, do you need further evidence before changing your practice? Are the benefits of the intervention worth the harms and the costs? If the study does not answer this question, you will need to use your own judgment, taking into account your patients, all the stakeholders, yourself and your colleagues.

Acknowledgement

References

1.	Boguniewicz M, Fiedler VC, Raimer S, Lawrence ID, Leung DY, Hanifin JM. A randomized, vehicle-controlled trial of tacrolimus ointment for treatment of atopic dermatitis in children. J Allergy Clin Immunol 1998;102:637-44.
2.	Moher D, Schulz KF, Altman D; CONSORT Group. The CONSORT Statement: Revised recommendations for improving the quality of reports of parallel-group randomized trials 2001. Explore (NY) 2005;1:40-5.
3.	Burls A. What is critical appraisal? 2009.
4.	Guyatt GH, Sackett DL, Cook DJ. Users' guides to the medical literature. II. How to use an article about therapy or prevention. A. Are the results of the study valid? Evidence-Based Medicine Working Group. JAMA 1993;270:2598-601.
5.	Jόni P, Altman DG, Egger M. Systematic reviews in health care: Assessing the quality of controlled clinical trials. BMJ 2001;323:42-6.
6.	Guyatt GH, Sackett DL, Cook DJ. Users' guides to the medical literature. II. How to use an article about therapy or prevention. B. What were the results and will they help me in caring for my patients? Evidence-Based Medicine Working Group. JAMA 1994;271:59-63.
7.	Armitage. Exclusions, losses to follow-up and withdrawals in clinical trials. Clinical trials, Shapiro SH. New York: Marcel Dekker; 1983.
8.	Gail MH. Eligibility exclusions, losses to follow-up, removal of randomized patients, and uncounted events in cancer clinical trials. Cancer Treat Rep 1985;69:1107-13.
9.	Rothwell PM. Treating individuals 2. Subgroup analysis in randomised controlled trials: importance, indications, and interpretation. Lancet 2005;365:176-86.
10.	Oxford, T.U.o. Critical appraisal sheet for RCT; 2009.
11.	Egger M, Altman DG, Schulz KG. The CONSORT Statement: Explanation and Elaboration. Ann Intern Med 2002;136:926-7.
12.	Ioannidis JP, Evans SJ, Gψtzsche PC, O'Neill RT, Altman DG, Schulz K, et al. Better reporting of harms in randomized trials: an extension of the CONSORT statement. Ann Intern Med 2004;141:781-8.
13.	Sackett DL. Evidence-based medicine. How to practice and teach EBM. New York: Churchill Livingstone Inc; 2000. P. 136.

This article has been cited by 1 What are ambulance personnel experiences of sexual harassment and sexual assault in the workplace? A rapid evidence review Isobel Abbott, Gregory Adam Whitley International Emergency Nursing. 2023; 71: 101376 [Pubmed] | [DOI] 2 Scholarship in anesthesiology: the role of critical appraisal, literature review, quality improvement, journal club, and presentation skills K.A. Fedoruk, Y.K. Chan, C.E. Williams International Journal of Obstetric Anesthesia. 2023; : 103639 [Pubmed] | [DOI] 3 From critical appraisal to risk of bias assessment: clarifying the terminology for study evaluation in JBI systematic reviews Jennifer C. Stone, Timothy Hugh Barker, Edoardo Aromataris, Merel Ritskes-Hoitinga, Kim Sears, Miloslav Klugar, Jo Leonardi-Bee, Zachary Munn JBI Evidence Synthesis. 2023; 21(3): 472 [Pubmed] | [DOI] 4 Patient-reported outcomes for medication-related quality of life: A scoping review Kadesha A. James, Lauren Cadel, Sander L. Hitzig, Sara J.T. Guilcher Research in Social and Administrative Pharmacy. 2022; [Pubmed] | [DOI] 5 Paediatric family activated rapid response interventions; qualitative systematic review Shannon Cresham Fox, Nicola Taylor, Takawira C. Marufu, Elizabeth Hendron, Joseph C. Manning Intensive and Critical Care Nursing. 2022; : 103363 [Pubmed] | [DOI] 6 The Barriers and Facilitators to the Use of Lifestyle Apps: A Systematic Review of Qualitative Studies Habiba Shabir, Matthew D’Costa, Zain Mohiaddin, Zaeem Moti, Hamza Rashid, Daria Sadowska, Benyamin Alam, Benita Cox European Journal of Investigation in Health, Psychology and Education. 2022; 12(2): 144 [Pubmed] | [DOI] 7 The efficacy of the HEART score in prehospital settings Charles Jones, Bryan Lightowler Journal of Paramedic Practice. 2022; 14(5): 198 [Pubmed] | [DOI] 8 Critically appraising for antiracism Ramona Naicker Education for Information. 2022; : 1 [Pubmed] | [DOI] 9 The Role of Diet in Influencing the Diversity of Gut Microbiome Related to Lupus Disease Activities: A Systematic Review Prisly Z. Putri, Laniyati Hamijoyo, Edhyana Sahiratmadja, Todd R. Callaway International Journal of Microbiology. 2022; 2022: 1 [Pubmed] | [DOI] 10 The Experiences of Informal Carers during the COVID-19 Pandemic: A Qualitative Systematic Review Cara Bailey, Ping Guo, John MacArtney, Anne Finucane, Susan Swan, Richard Meade, Ellie Wagstaff International Journal of Environmental Research and Public Health. 2022; 19(20): 13455 [Pubmed] | [DOI] 11 Over half of clinical practice guidelines use non-systematic methods to inform recommendations: A methods study Carole Lunny, Cynthia Ramasubbu, Lorri Puil, Tracy Liu, Savannah Gerrish, Douglas M. Salzwedel, Barbara Mintzes, James M. Wright, Tim Mathes PLOS ONE. 2021; 16(4): e0250356 [Pubmed] | [DOI] 12 Twelve tips for students who wish to write and publish Rajaei K Sharma, Harriet L Ogle Medical Teacher. 2021; : 1 [Pubmed] | [DOI] 13 Methodological review to develop a list of bias items used to assess reviews incorporating network meta-analysis: protocol and rationale Carole Lunny, Andrea C Tricco, Areti-Angeliki Veroniki, Sofia Dias, Brian Hutton, Georgia Salanti, James M Wright, Ian White, Penny Whiting BMJ Open. 2021; 11(6): e045987 [Pubmed] | [DOI] 14 Factors that facilitate or hinder whole system integrated care for obesity and mental health: a scoping review protocol Fani Liapi, Angel Chater, Gurch Randhawa, Yannis Pappas BMJ Open. 2021; 11(8): e050527 [Pubmed] | [DOI] 15 Preliminary image findings of lower limb stress fractures to aid ultrasonographic diagnoses: A systematic review and narrative synthesis Madeleine Schaper, James Harcus Ultrasound. 2021; 29(4): 208 [Pubmed] | [DOI] 16 Evidence Based Medicine: A Review of Basic Concepts, Learning Theories, Teaching Approaches and Assessment Methods Özlem Serpil ÇAKMAKKAYA Tip Egitimi Dünyasi. 2021; : 122 [Pubmed] | [DOI] 17 Reply to “Comments on ‘Critically Low Confidence in the Results Produced by Spine Surgery Systematic Reviews: An AMSTAR-2 Evaluation From 4 Spine Journals’ by Dettori et al” Joseph R. Dettori, Andrea C. Skelly, Erika D. Brodt Global Spine Journal. 2020; 10(8): 1080 [Pubmed] | [DOI] 18 Public Health Lessons Learned From Biases in Coronavirus Mortality Overestimation Ronald B. Brown Disaster Medicine and Public Health Preparedness. 2020; 14(3): 364 [Pubmed] | [DOI] 19 An integrative literature review exploring the clinical management of delirium in patients with advanced cancer Hayley Lawley, Alistair Hewison Journal of Clinical Nursing. 2017; 26(23-24): 4172 [Pubmed] | [DOI]

	Previous article Next article