Journal Club 

Proprietary Big Datasets vs. External Validation: Guess Who Wins This Matchup?

By Nirav Patel, MD

The use of standardized tools to assess risk has long been a hallmark of clinical medicine. With the advent of large datasets, the potential for more accurate and robust tools exists, especially for disease states that need timely intervention but also require synthesis of complex clinical features. Sepsis remains a challenging clinical conundrum and a disease state for which rapid intervention is necessary to change patient outcomes (not to mention, specific attention in the form of quality measures, such as the Centers for Medicare and Medicaid Services Severe Sepsis and Septic Shock Early Management Bundle [SEP-1] metric).

Epic, one of the largest electronic health record (EHR) vendors, developed the Epic Sepsis Model (ESM), which integrates a large number of variables typically entered into the EHR to identify patients who are developing sepsis. Wong and collaborators recently published the first and only external validation of the ESM using over 38,000 hospitalizations at the University of Michigan prior to the COVID-19 pandemic. Their findings appear in JAMA Internal Medicine. The area under the receiver operating characteristic curve was 0.63, with a sensitivity of 33%, specificity of 83%, positive predictive value of 12%, and negative predictive value of 95%.

The accompanying editorial notes some of the challenges of these models, including the need for external validation, the critical importance of changes in clinical setting and patient populations to model performance, as well as the value of open-access models, which allow for independent analysis, among other structural recommendations. As such tools evolve, hopefully designers will consider the editorialists’ proposed framework. Until then, the matchup will likely continue to be skewed against such proprietary models.

(Wong et al. JAMA Intern Med. Published online: June 21, 2021.)