Evidence-Based Practice: APPRAISE - Critical Appraisal
This guide is designed to assist health care professionals to become effective and efficient users of the medical and nursing literature.
Evidence-Based Practice: Navigation
EBP Home | Pre-Reqs: Before you Dive in ... Learn About the Medical Literature | Introduction to EBP in 6 Steps | AJN's EBP Series | ASK - PICO | Levels of Evidence and Study Designs | SEARCH EBP Tools & Pyramid | EBP Tools Table | Literature Review | Getting to Fulltext | APPRAISE-Critical Appraisal | DISSEMINATE - Writing for Publication
Other EBP Guides, Tutorials, Terms | RN Residency Teaching Activities | How to Identify Peer-Reviewed Journals
How are resources evaluated?
Appraisal is the third step in the Evidence Based Practice process. It requires that the evidence found be evaluated for its validity and clinical usefulness.
Definition:
- Critical Appraisal is the systematic evaluation of clinical research papers in order to establish the validity of the methodology, results, and applicability to patient care. (from CEBM)
Ask 3 Questions:
- Is there a high risk of bias in the methodology?
- Are the results valid?
- Can the results be applied to my patient?
Caveat Evidence:
Helpful Books
Rough Guide to Spotting Bad Evidence
From Compound Chem: "This graphic looks at the different factors that can contribute towards ‘bad’ science – it was inspired by the research I carried out for [a different project,] where many articles linked the compound to causing breast cancer, referencing scientific research which drew questionable conclusions from their results." Read more ...
Sketchy EBM Video: How I Read a Paper
Evaluating a Study - Parachute Example
Yeh RW, Valsdottir LR, Yeh MW, Shen C, Kramer DB, Strom JB, Secemsky EA, Healy JL, Domeier RM, Kazi DS, Nallamothu BK; PARACHUTE Investigators.
BMJ. 2018 Dec 13;363:k5094. doi: 10.1136/bmj.k5094. Erratum in: BMJ. 2018 Dec 18;363:k5343.
- PMID: 30545967 PubMed Citation with Abstract Link to Article at BMJ
Sample questions for evaluating a study:
- Has the study's aim been clearly stated?
- Does the sample accurately reflect the population?
- Has the sampling method and size been described and justified?
- Have exclusions been stated?
- Is the control group easily identified?
- Is the loss to follow-up detailed?
- Can the results be replicated?
- Are there confounding factors?
- Are the conclusions logical?
- Can the results be extrapolated to other populations?
- Does it make sense?
How to Read a Paper by Trisha Greenhalgh
- Is my practice evidence-based?
BMJ 313 (7063): 957-8, 19 Oct 1996.
- The Medline database
BMJ 315 (7101): 180-3, 19 Jul 1997.
- Getting your bearings (deciding what the paper is about).
BMJ 315 (7102): 243-6, 26 Jul 1997.
- Assessing the methodological quality of published papers. [see comments].
BMJ 315 (7103): 305-8, 2 Aug 1997.
- Statistics for the non-statistician. I.Different types of data need different statistical tests.
BMJ 315 (7104): 364-6, 9 Aug 1997. [ erratum BMJ 13 Sep 1997, 315 (7109):675].
- Statistics for the non-statistician. II. "Significant" relations and their pitfalls.
BMJ 315 (7105): 422-5, 16 Aug 1997.
- Papers that report drug trials.
BMJ 315 (7106): 480-3, 23 Aug 1997.
- Papers that report diagnostic or screening tests.
BMJ 315 (7107): 540-3, 30 Aug 1997.erratum in BMJ, 11 Oct 1997, 315 (7113):942 ; erratum in BMJ, 17 Jan 1998; 316(7126):225.
- Papers that tell you what things cost (economic analyses).
BMJ 315 (7108): 596-9, 6 Sep 1997.
- Papers that summarise other papers (systematic reviews and meta-analyses).
BMJ 315 (7109): 672-5, 13 Sep 1997.
- Papers that go beyond numbers (qualitative research).
BMJ 315 (7110): 180-3, 20 Sep 1997.
3 Ways to Read a Scientific Paper - Fast
Reading a Scientific Paper - In Brief
- Read the Abstract
- Skim the Introduction
- Skip the Methods for now
- Read Results
- Read Introduction
- Read Discussion
- Read Methods - If the methods has a high risk of bias, find a different article.
"Quick & Dirty" Review of a Medical Study
- Identify the Working Hypothesis
- Identify the Type of Study
- State the Conditional Prediction
- If [working hypothesis] is true, then in this type of study [prediction].
- Look at the results to see if it bears out the prediction.
- Assess the validity of the author's conclusions.
Keshav, S. How to Read a Paper. [The 3-Pass Method] ACM SIGCOMM Computer Communication Review. 2007 July; 37(3): 83-84. Available at: http://ccr.sigcomm.org/online/files/p83-keshavA.pdf Accessed 9 March 2022.
-
- Abstract: Researchers spend a great deal of time reading research papers. However, this skill is rarely taught, leading to much wasted effort. This article outlines a practical and efficient three-pass method for reading research papers.
Summary of Evidence Tables
Critical Appraisal Tools and Worksheets
Protocols and Checklists for SR/MAs
The following protocols and checklists can help you ascertain if a systematic review or metanalysis was done according to best practices.
Critical Appraisal of Practice Guidelines
While many groups develop practice guidelines, they do not necessarily reach the same conclusions and recommendations for practice. The following journal article and document provide guidance for the critical appraisal of practice guidelines.
-
Critical Appraisal of Clinical Guidelines [Article]Netsch, Debra S.; Kluesner, Jean A. (2009). Critical Appraisal of Clinical Guidelines. Journal of Wound, Ostomy & Continence Nursing, 37(5),470-473. doi: 10.1097/WON.0b013e3181edee71
-
AGREE II InstrumentAppraisal of Guidelines for Research & Evaluation II:
The product of an international collaboration, the AGREE II Instrument is a generic tool designed to help both guideline developers and users assess the methodological quality of guidelines; provide a methodological strategy for the development of guidelines; and inform what information and how information ought to be reported in guidelines.
Even RCTs Can Be Biased
In recent years, the drawbacks to randomized studies have become increasingly recognized.
- Patients who are enrolled in such studies typically represent a subset of the general population, compromising “generalizability.”
- Many patients (and their referring physicians) may not wish to have their treatment randomized, particularly when the outcome difference might be existential.
- As the inclusion rate drops, so does the reliability.
- Commercial bias is always a consideration.
- Investigators are more likely to find in favor of a commercial device if it is provided free of charge for their study. They may feel indebted to the company, and may depend on the company for supplies after the trial.
Swanson E. Levels of evidence in cosmetic surgery: Analysis and recommendations using a new CLEAR classification. Plast Reconstr Surg Glob Open. 2013;1:e66