TDNet Discover

Evidence-Based Practice: APPRAISE - Critical Appraisal

This guide is designed to assist health care professionals to become effective and efficient users of the medical and nursing literature.

Evidence-Based Practice: Navigation

EBP Home   |    Pre-Reqs: Before you Dive in ... Learn About the Medical Literature    |     Introduction to EBP in 6 Steps   |     AJN's EBP Series     |   ASK - PICO    |   Levels of Evidence and Study Designs    |    SEARCH EBP Tools & Pyramid    |    EBP Tools Table    |   Literature Review    |    Getting to Fulltext   |    APPRAISE-Critical Appraisal    |    DISSEMINATE - Writing for Publication   

Other EBP Guides, Tutorials, Terms     |   RN Residency Teaching Activities        |   How to Identify Peer-Reviewed Journals

How are resources evaluated?

Appraisal is the third step in the Evidence Based Practice process.  It requires that the evidence found be evaluated for its validity and clinical usefulness. 

Definition:

  • Critical Appraisal is the systematic evaluation of clinical research papers in order to establish the validity of the methodology, results, and applicability to patient care.  (from CEBM)

Ask 3 Questions:

  1. Is there a high risk of bias in the methodology?
  2. Are the results valid?
  3. Can the results be applied to my patient?

Caveat Evidence:

Helpful Books

JAMA | Users' Guides to the Medical Literature (at JAMA)

How to Read Articles that Use Machine Learning (2019): JAMA

eBook on Ebsco (c2008)

 

Rough Guide to Spotting Bad Evidence

From Compound Chem:  "This graphic looks at the different factors that can contribute towards ‘bad’ science – it was inspired by the research I carried out for [a different project,] where many articles linked the compound to causing breast cancer, referencing scientific research which drew questionable conclusions from their results."  Read more ...

Sketchy EBM Video: How I Read a Paper

Evaluating a Study - Parachute Example

Parachute use to prevent death and major trauma when jumping from aircraft: randomized controlled trial.

Yeh RW, Valsdottir LR, Yeh MW, Shen C, Kramer DB, Strom JB, Secemsky EA, Healy JL, Domeier RM, Kazi DS, Nallamothu BK; PARACHUTE Investigators.

BMJ. 2018 Dec 13;363:k5094. doi: 10.1136/bmj.k5094. Erratum in: BMJ. 2018 Dec 18;363:k5343.

PMID:   30545967     PubMed Citation with Abstract    Link to Article at BMJ

Sample questions for evaluating a study:

  • Has the study's aim been clearly stated?
  • Does the sample accurately reflect the population?
  • Has the sampling method and size been described and justified?
  • Have exclusions been stated?
  • Is the control group easily identified?
  • Is the loss to follow-up detailed?
  • Can the results be replicated?
  • Are there confounding factors?
  • Are the conclusions logical?
  • Can the results be extrapolated to other populations?
  • Does it make sense?

 

How to Read a Paper by Trisha Greenhalgh

3 Ways to Read a Scientific Paper - Fast

Reading a Scientific Paper - In Brief

  1. Read the Abstract
  2. Skim the Introduction
  3. Skip the Methods for now
  4. Read Results
  5. Read Introduction
  6. Read Discussion
  7. Read Methods - If the methods has a high risk of bias, find a different article.  

"Quick & Dirty" Review of a Medical Study

  1. Identify the Working Hypothesis
  2. Identify the Type of Study
  3. State the Conditional Prediction
    • If [working hypothesis] is true, then in this type of study [prediction].
  4. Look at the results to see if it bears out the prediction.
  5. Assess the validity of the author's conclusions.

Keshav, S.  How to Read a Paper. [The 3-Pass Method] ACM SIGCOMM Computer Communication Review. 2007 July; 37(3): 83-84.  Available at: http://ccr.sigcomm.org/online/files/p83-keshavA.pdf  Accessed 9 March 2022.

    • Abstract: Researchers spend a great deal of time reading research papers. However, this skill is rarely taught, leading to much wasted effort. This article outlines a practical and efficient three-pass method for reading research papers.

Summary of Evidence Tables

     *  Building an Evidence Table ( Rutgers University Libraries)
     *  Examples of Evidence Tables (NICE Guidelines)
 
     *  Johns Hopkins Nursing EBP Individual Evidence Summary Tool
 
     *  EBP Rapid Review Workbench (Google Sheets)
            > Template for documenting the work of rapid reviews with tabs for PICO / Search Terms, Database Search strategies, Summary Table, Synthesis Table.  Created by Aida Smith (Aida.Smith@ascension.org.  Used by permission. 

Critical Appraisal Tools and Worksheets

     *  CASP Checklist
 
     *  Critical Appraisal Tools and Worksheets - CEBM
 
     *  Critical Appraisal and other EBM Worksheets - Dartmouth College
 
     *  Critical Appraisal Worksheets at Duke
 
     *  Calculate Results - Duke University
 
     *  Worksheet for Rapid Review of Journal Articles [Google Docs]

Protocols and Checklists for SR/MAs

The following protocols and checklists can help you ascertain if a systematic review or metanalysis was done according to best practices.

     *  PRISMA-S (updated 2021)
          -- Long checklist and flow diagram that explains how a SR winowed down to a usable set of similar studies.
 
     *  MOOSE Guidelines (for observational studies) 
          -- Meta-analysis of Observational Studies in Epidemiology checklist contains specifications for reporting of meta-analyses of observational studies in epidemiology. Editors will expect you to follow and cite this checklist.  It refers to the Newcastle-Ottawa Scale for assessing the quality of non-randomized studies, a method of rating each observational study in your meta-analysis.
 
     * Cochrane Handbook 
          -- The Cochrane Handbook isn't set down to be a standard, but it has become the de facto standard for planning and carrying out a systematic review. Chapter 6, Searching for Studies, is most helpful in planning your review.

Critical Appraisal of Practice Guidelines

While many groups develop practice guidelines, they do not necessarily reach the same conclusions and recommendations for practice. The following journal article and document provide guidance for the critical appraisal of practice guidelines.

Even RCTs Can Be Biased

In recent years, the drawbacks to randomized studies have become increasingly recognized. 

  • Patients who are enrolled in such studies typically represent a subset of the general population, compromising “generalizability.”
  • Many patients (and their referring physicians) may not wish to have their treatment randomized, particularly when the outcome difference might be existential.
  • As the inclusion rate drops, so does the reliability.
  • Commercial bias is always a consideration.
  • Investigators are more likely to find in favor of a commercial device if it is provided free of charge for their study. They may feel indebted to the company, and may depend on the company for supplies after the trial.

 

Swanson E. Levels of evidence in cosmetic surgery: Analysis and recommendations using a new CLEAR classification. Plast Reconstr Surg Glob Open. 2013;1:e66

Catalog of Bias in Healthcare - CEBM

How to Interpret Graphs in Research Studies and Systematic Reviews

Loading...

Confirm

Alert

Place a hold

Log in