Implementation and evaluation of a Sepsis Best Practice Advisory alert utilizing predictive analytics to improve patient outcomes: An implementation study

Sepsis is among the leading causes of death in the United States of America (USA) with more than 700,000 hospitalizations and 200,000 deaths attributed to sepsis annually.^[1] Globally, in 2017, approximately 48.9 million cases and 11 million sepsis-related deaths were reported, underscoring the severe worldwide impact of sepsis.^[2] Cited global rates for mortality due to sepsis or septic shock vary,^{[2, 3]} depending on the methods used for data collection, populations included (adults and/or children) and global region. Rudd and colleagues reported sepsis rates in 2017 to be 19.7% of all deaths, this included all ages, infection, injury and non-communicable diseases.^[2] Bauer and others conducted a meta-analysis and report 30-day mortality rates for septic shock as 34.7%; and 90-day mortality as 38.5%.^[3] These rates are for adults and include Australia and New Zealand, Europe and the USA. Underdeveloped regions were excluded because rates varied greatly. The study was done over a ten-year period (2009-2019).^[3] Moreover, Markwart et al.^[4] in their systematic review and meta-analysis found that the mortality rates of hospital acquired sepsis mortality rates exceeded 40%. Despite varying rates of sepsis associated deaths reported, experts agree sepsis is a major health concern globally.^{[2, 3, 4]} In addition to loss of life, sepsis costs the USA healthcare system more than $24 billion dollars annually.^[2] Early identification and treatment have been shown to decrease mortality.^{[5, 6, 7]} In some states, hospitals are required to implement evidence-based protocols and treatment guidelines to identify adult patients at risk for sepsis.^[8]

A comprehensive review of the literature revealed the area under the curve (AUC) or the receiver operating characteristic (ROC) curve as leading indicators of accurate and effective performance of early sepsis detection.^{[6, 9, 10, 11]} A meta-analysis of seven studies, found Sepsis Predictive models out-performed traditional methods for sepsis detection.^[11] In Islam’s analysis, each study utilized the same gold standard to define sepsis with a different machine learning tool which resulted in sepsis detection three to four hours earlier than traditional methods. The authors calculated the sensitivity, specificity, and ROC value with a 95% confidence interval (CI) for predicting sepsis for patients in intensive care units (ICU). The ROC curve plots the true positive rate against the false positive rate (at various threshold settings). The use of multiple models with no specific best model defined was a limitation of this meta-analysis identified by the authors. In addition, variables were included but not weighted by importance. Of the seven studies, only three used external validation, which was also identified as a study limitation. Goh et al. ^[12] developed a sepsis predictive model that evaluated clinical notes also using the AUC, sensitivity (ability to measure correct assignment as positive) and specificity (ability to measure correct assignment as false) to determine the algorithms accuracy. The authors concluded that clinical text was beneficial for early predictions and that the algorithm is a tool that should complement, not replace, clinical judgement.

False positives in sepsis alert systems are costly to healthcare organizations; unacceptable levels of false positives can lead to inefficient use of resources including additional follow-up diagnostic testing and treatment, and lack of clinician trust.^{[12, 13]} Wong et al. showed the proprietary Sepsis Predictive Model had an AUC of 0.63 (95% CI, 0.62-0.64) and the authors noted that this was a lower AUC than that reported by the vendor.^[13] There were notable challenges in this Predictive Model including a failure to prompt administration of antibiotics (7%) when compared to usual practice. The authors reported a lack of identifying all patients with sepsis, while at the same time triggering many false alerts. The importance of selecting and setting an appropriate threshold for alerts was stressed. Furthermore, over-use of alerts causes a high burden of alarm fatigue, and false positive alerts can also result in added costs of diagnostics and treatment.^[14] In the conclusions of these studies, an interdisciplinary team was recommended to monitor the effects of the implementation of alert systems.^{[13, 15]} The literature emphasizes that it is crucial for the clinical healthcare team to have high-performing sepsis predictive tools for early detection and intervention.

Although such tools can ensure early identification and treatment for sepsis, it is vital that these tools identify risk without over alerting and causing alarm fatigue. The approach by this healthcare system was to independently evaluate the alert system by establishing an interprofessional team to operationalize and closely monitor protocol development. The purpose of this study was to address the critical health issue of sepsis and ensure an alert system that was optimal. This initiative was proposed through the development of an interprofessional healthcare team to closely monitor and evaluate the effectiveness of alerts customized for our in-patient population, using the proprietary electronic health record (EHR) called the Epic® Sepsis Predictive Model, Version 1. Additionally, a quantitative prospective survey of healthcare professionals (end users) was conducted to examine perspectives of processes and outcomes. A full report of the feedback survey is not included in this manuscript.

Permission to conduct this evidence-based initiative, using an implementation science approach that included a retrospective review and analysis, was provided by the organization’s Institutional Review Board (IRB). Because no experimental interactions with individuals were required, a waiver of consent was provided by the IRB. No IRB oversight was required for this retrospective analysis. None of the researchers have any conflict of interest.

The team developed for this initiative was composed of clinical nurses, practitioners, quality improvement experts, informatics nurses, clinical administrators, and researchers. Scientists from the vendor who developed the proprietary model were also consulted on an ad hoc basis. This team met weekly for four years throughout the development of the program, although events, such as COVID-19 surges caused some gaps in weekly meetings. This interprofessional team reviewed the data, developed protocols, changed policies, oversaw clinician education, and implementation of the sepsis protocols. More specifically, the team evaluated the effect of the model with the incorporation of a Sepsis Best Practice Advisory (BPA) to alert the healthcare team and identify patients at risk for sepsis, preventing progression, without over alerting. The team reviewed the impact of the implemented interprofessional practice changes at each interval of the rollout phases and reported for approvals through the Quality and Patient Safety Service Line. These modifications included raising the Sepsis BPA threshold score, extending the lockout periods, and introducing additional BPAs for practitioners to optimize the BPA protocols.

The first step in this initiative was to evaluate the performance of this newly implemented system-wide Sepsis Predictive Model decision support tool protocol utilizing a customized BPA alert. The AUC was considered the best standard, with calculations of specificity and sensitivity, to evaluate the effectiveness of the Sepsis Predictive Model. This included assessing the model’s ability to correctly identify all individuals at risk for sepsis (sensitivity, also known as the true positive rate) who received a clinical intervention to mitigate the risk of sepsis. At the same time, the team aimed to reduce alerts after a clinical intervention was already initiated to reduce alert fatigue. The team also intended to reduce the incorrectly triggered alerts, or improve the negative predictive value (NPV), also called the specificity which measures the ability of a tool to correctly identify the negative findings.^[16]

The proprietary tool utilized by the healthcare system to assess sepsis risk used a set of approximately 80 data points, identified in the EHR, that were weighted according to its contribution to the risk of sepsis. The data points included demographics, vital signs, laboratory results, and documented hospital problem list amongst others. A score was generated for individual patients at various timepoints; the score was based on these contributing data elements and reflected the risk of sepsis.^[17]

During a baseline review of the Sepsis Predictive Model’s performance, it was identified that certain events were deemed important such as timing of lab draws, blood cultures, and the first interventions along with patient indicators such as changes in vital signs. Date and time stamping for these collective events was not easily available at the start of the project. To address this need, a customized one-page Sepsis Score Report was created and integrated into the EHR. The summary provides visualization of orders completed with trends of vital signs and laboratory results. This summary also includes a time-stamped order of events, enabling healthcare professionals to visualize the measures in sequence and better assess the clinical progression of suspected sepsis. A link is also provided on the report for ease of viewing a practitioner review of the BPA and breakdown of factors contributing to the score.

After the baseline review, three major phases were conducted: Pilot I, Pilot II and the System-Wide Rollout of the Sepsis Predictive Model. To foster decision making, the team reviewed a confusion matrix report generated by the EHR vendor. As the team reviewed and considered different thresholds to trigger alerts, such as 4, 5, or 6, the confusion matrix output flexed the various metrics such as sensitivity and specificity that correlated with the specified or chosen threshold. The initial setting to trigger alerts was set at 4 for the Pilot I phase; a subsequent Pilot II phase was performed using an alert threshold level of 5. The Concordance Statistic (C-Stat) was used by the vendor to determine the accuracy of the sepsis alerts. C-Stat is comparable to the AUC for binary variables; it plots the false positive rate and the true positive rate.^[18] The threshold score, number of patients with alerts and the average number of alerts per day for each threshold point were tracked utilizing a data visualization dashboard (created using Power BI). Sepsis mortality was also monitored on an ongoing basis.

Two protocol evaluations were conducted by the team post system-wide implementation. One was a detailed chart review in a single community hospital within the system. The second was a system evaluation of the performance of the tool approximately one year after the system-wide rollout. Mortality rates were also reviewed. The two pilot studies, system-wide rollout, and two evaluation studies are briefly described below:

Pilot I (August 31 – October 31, 2020) included five medical centers within the system, and twelve units (including medical, surgical, progressive care, step-down). Sepsis Risk Screening was initiated (see Table 1, Sepsis Risk Screening). Once the sepsis score threshold was reached and the BPA was triggered, if one or more risk factors were present, the nurse was expected to notify the practitioner. The prompt provided relevant information, including previous orders, test results, the calculated sepsis score, and the ability to place orders necessary to meet protocol guidelines. Nurses were responsible for notifying the practitioner, and if needed, they could initiate a Rapid Response Team, a team composed of health care professionals to quickly evaluate the patient. While the BPA offered essential prompts and information for nurses to begin the protocol, it didn’t alert the practitioners, who are key to managing sepsis.

Table 1 provides signs and symptoms to alert nurses and healthcare professionals that sepsis should be considered.

Pilot II (June 1 – August 31, 2021) included establishment of a Practitioner BPA and optimization of the nurse BPA screening and nursing initiation of lab testing per protocol. Enhancement of the existing data set for practitioners was implemented. The additional BPA for practitioners, once acknowledged, silenced the BPA for nurses (see Figure 1). This notification provided satisfaction to the practitioner team, allowing for individualized clinical assessment, and it satisfied the nurses as it accomplished automatic notification of practitioners with a reduction in the nurse BPA alerts.

Figure 1.

Best Practice Advisory workflow

Figure 1 shows the workflow for assessments and communication between healthcare team members related to sepsis BPA alerts. BPA = Best Practice Advisory; min = Minutes; RN = Registered nurse; CBC = Complete blood count; CMP = Comprehensive metabolic panel

Additionally, a summary of the BPA history of actions was added to the Sepsis Score Report in the EHR to allow all team members to see the actions taken to address the BPA and by whom. This eliminated a duplication of efforts and ease of quality review and provided interprofessional communication. Notably, without Pilot I, the additions for workflow efficiency may not have occurred. An inclusive team for pilot testing, education, and planning was an integral part of the success of the initiative.

The system-wide roll out (September 15 – November 11, 2021) included a policy update that reflected changes applied. An educational rollout plan was implemented. Educational programs were held prior to the roll out for all healthcare professionals. Data were reviewed after the policy update and rollout.

The post system implementation analysis of one community medical center (February 2022) conducted over one month, was completed to further validate the value of the Sepsis BPA. This included an in-depth manual chart review of 703 admissions in which a BPA was triggered. Several variables were triggered such as demographic data, the number of times the sepsis alert was triggered, the responses of the healthcare team members, and outcomes such as final diagnosis of sepsis and/or infections. Additional recommendations after this review included re-educating staff and adjusting BPA settings to suppress alerts for six hours instead of three hours after a lactate was drawn. This added suppression further reduced unnecessary alerts for already identified sepsis patients once transferred to an inpatient unit from the emergency department.

The one-year post system wide evaluation over a 3-month period (November 1, 2021 – January 31, 2022) was conducted to evaluate the specificity and sensitivity of the sepsis BPA alerts customized for our patient population which included changes in the C-Stat pre and post BPA customizations implemented by the interdisciplinary team. The frequency of interventions for those patients in which a BPA alert was triggered with and without a final diagnosis of sepsis, and with and without a final diagnosis of infection was also evaluated. This analyses helped to determine the accuracy of the customized alert system to identify those patients at high risk for sepsis by determining the number of patients that had a BPA trigger and had an intervention.

For this study, an intervention in response to sepsis risk was defined as any of the following: fluids, lactate, antibiotics, radiology, blood cultures, or the sepsis order set used. An increase in fluids and administration of antibiotics are evidence-based treatments for sepsis: lactate levels and the use of the sepsis order set, which includes these evidence-based interventions, are recommended to assist with sepsis diagnosis and treatment, and cultures and radiology are imperative for source identification.^{[19, 20, 21]} These interventions were selected due to their salience related to the care of a potentially septic patient and ease of data extraction from the EHR. Interventions are routinely documented by healthcare professionals in the EHR and were extracted using a customized report for the purpose of this initiative.

Inclusion and Exclusion Criteria for the review of records were established. Inclusion for data in the retrospective reports and/or manual review of EHR included adults admitted to an inpatient acute care facility within the system. Patients with observation status and extended recovery were also included. To facilitate accurate data collection, records with at least 75% of data variables used for analysis needed to be documented on the EHR. Records of individuals younger than 18 years of age and individuals cared for in critical care, emergency department, pediatrics, or in the mother baby or labor and delivery units were excluded.

The changes made from Pilot 1 (baseline) through the system-wide roll out included an increase in the threshold from 4 to 5 resulting in a 52% reduction of alerts with a score of 5, and a 54% reduction of alerts at a score above 5. Detailed data associated with timepoints, actions, number of patients, number of alerts, and reductions realized are available in Table 2, Pilot 1 and 2 Comparisons to System-Wide Rollout. Table 2 provides key initiatives at every phase of the study, as well as number of patients included in the analysis and metrics related to scores and alerts. A 54% reduction in the average number of alerts per patient/per calendar day was realized after the system-wide rollout.

The one-year post-implementation system-wide analysis included adult discharges over a three-month period (n = 22,521). The C-Stat was 0.763, with a Positive Predictive Value (PPV) of 0.281 as opposed to the prior C-Stat of 0.762, PPV of 0.304. PPV reflects the number of true positives (TP) reported out of all the positives reported; PPV = (TP/(TP+false positives)).^[16]

The BPA was triggered in 18% of cases (4,037 patients), and 3% of patients (734) had a final diagnosis of sepsis. Intervention within 24 hours post-BPA included at least one of the following actions – drawing blood cultures, increasing or adding intravenous fluids, starting or changing antibiotics, and ordering radiology tests.

The proportion of patients who had a BPA alert, had an infection, and had an intervention (99.5%) was significantly greater than the proportion of patients who had a BPA alert, had an infection, and did not have an intervention (92.03%, p-value < .001) (see Figure 2). Figure 2 shows the proportion of patients who had a BPA fire, had an infection, and had an intervention (3,881/3,899, 99.5%) was significantly greater than the proportion of patients who had a BPA fire, had an infection, and did not have an intervention (127/138, 92.03%, p-value < .001). [Intervention within 24 hrs. post-BPA-Cultures drawn; Fluids increased or added; Antibiotic started or changed; Radiology testing.]

Figure 2.

BPA alerts with final diagnosis of infection

BPA = Best Practice Advisory; DX = Diagnosis, hrs. = Hours

The proportion of patients who had a BPA alert had sepsis and had an intervention (18.83%) was significantly greater than the proportion of patients who had a BPA alert had sepsis, and did not have an intervention (4.35%, p-value < .001) (see Figure 3).

Figure 3.

BPA alerts with final diagnosis of sepsis

BPA = Best Practice Advisory; DX = Diagnosis

Figure 3 shows the proportion of patients who had a BPA fire, had sepsis, and had an intervention (734/3,899, 18.83%) was significantly greater than the proportion of patients who had a BPA fire, had sepsis, and did not have an intervention (6/138, 4.35%, p-value < .001).

A decrease in percent mortality for patients diagnosed with sepsis, not present on admission (NPOA), was observed consistently over a four-year period, from 59% to 41% (see Figure 4). Figure 4 shows the percent of mortality for patients with severe sepsis, or septic shock, not present on admission.

Figure 4.

Percent mortality of patients with sepsis 2021-2024

NPOA = Not present on admission

While the AUC, or the C-Stat, showed a slight improvement from 0.762 to 0.763, it is intriguing that a slight decrease in the PPV from 0.304 to 0.281 was also observed. Implementing a clinical team, as identified as best practice to facilitate recognition and treatment of infection before a patient becomes septic may play a role.^[15] A patient recognized early to be at risk for sepsis and who received an intervention, may not progress to sepsis. The early intervention could inadvertently be analyzed as a false positive, thus creating a lower value for the PPV, as observed in this study.

Cognizant of the unexpected finding of a decrease in the PPV further data review took place. The interdisciplinary team evaluated and compared patients who had BPA alert, had an intervention, had an infection (thus placing them at a high risk of sepsis), to those who did not have an infection. Similarly, we evaluated patients who had a BPA alert, had an intervention, and had a diagnosis of sepsis compared to those who did not have a sepsis diagnosis. These findings, coupled with a reduced number of sepsis cases and sepsis mortality, support previously reported research underscoring the importance of early intervention to prevent sepsis, severe sepsis, and septic shock.^{[5, 22]} The significant value of interventions after an alert strengthens the expectation that the BPA alert is triggering for patients truly at risk for sepsis rather than a false trigger.

For example, false alerts were observed in this project for patients with an increased heart rate due to cardiac diagnosis or increased temperature in post-operative patients. Concurrent with the added BPA alert for practitioners in Pilot II, a comprehensive interdisciplinary review of the BPA threshold for alerting was completed. The review revealed over-alerting in patients not at risk for sepsis and a need to increase actions taken on BPA. Therefore, the threshold was increased from 4 to 5 and re-education was complete. The impact of the interprofessional team to closely monitor and customize the Sepsis BPA alerts and make changes to the protocols to optimize the use of the predictive analytic tool was supported by the measures and positive outcomes.

Although the study demonstrated improvements in the generation of meaningful alerts, several limitations were identified. Practitioner resistance to adopting alerts generated by predictive analytics was initially observed, highlighting the need for continued education around the role of such tools in complementing—not replacing—clinical judgment. Additionally, the patient populations within operative settings, labor and delivery, mother-baby units, emergency departments, and intensive care units were not included, which limits the generalizability of findings across diverse care environments. While interventions following alerts were closely monitored, it remains uncertain whether the avoidance of sepsis can be directly attributed to these actions alone. It is also critical to acknowledge that any modifications to the EHR system or predictive model necessitate a comprehensive reassessment to maintain alert accuracy. Ongoing evaluation using key performance metrics, such as C-Stat, and PPV, and NPV is essential to uphold best practices and ensure sustained clinical impact.

Implementing clinical decision tools such as the Sepsis Predictive Model, should be closely monitored and enhanced with the development of interprofessional teams to ensure integration into workflows and optimize effectiveness. These tools can alert the healthcare team to a potential risk and support the clinical team in decision-making. The value of regularly scheduled interprofessional meetings, close collaboration with EHR experts, iterative reviews of performance metrics and patient outcomes, and the establishment of clear intervention protocols has proven effective in enhancing early alert systems for hospitalized individuals at risk for sepsis. Customizing alert systems to align with clinical workflows and support frontline evaluations contributes to more timely and accurate patient care decisions. To maintain and build upon these improvements, continuous evaluation and refinement must remain a priority in subsequent initiatives—ensuring sustained efficacy and responsiveness in sepsis detection and treatment strategies.

This approach significantly enhanced the early recognition and treatment of patients who were at risk for developing sepsis, while also strengthening reporting capabilities. Through iterative collaboration and refinement, early interventions were optimized, and unnecessary alerts were minimized, resulting in improved patient outcomes and more efficient use of clinical resources such as predictive analytics.

Laura L. Reilly: Conception and design, acquisition of data, analysis and interpretation, drafting, critical revision; Natalie Peleg: Analysis and interpretation, drafting, critical; Maria Stratton: Conception and design, acquisition of data, analysis and interpretation, drafting, critical revision; Mildred Ortu Kowalski: Conception and design, analysis and interpretation, drafting, critical revision; Jyothi Jagadeesh: Acquisition of data, analysis and interpretation, drafting, critical revision; Michelle T. Martins: Conception and design, analysis and interpretation, drafting, critical revision; Stephanie Chiu: Acquisition of data, analysis and interpretation, critical revision; Cristen Mackwell: Acquisition of data, analysis and interpretation, drafting; Jeanne Giaquinto, Janet Pagulayan: Acquisition of data, drafting; Florise Altino-Pierre: Analysis and interpretation, drafting; Michael Robes, Danielle L. Wolf: Acquisition of data, analysis and interpretation, drafting, critical revision.

This research was not funded.

The authors declare they have no conflicts of interest.

There were no experimental interactions with individuals, therefore a waiver of consent was provided by the IRB.

The Publication Ethics Committee of the Association for Health Sciences and Education. The journal’s policies adhere to the Core Practices established by the Committee on Publication Ethics (COPE).

Because no active recruitment was done for this retrospective analysis, the study was not registered.

Not commissioned; externally double-blind peer reviewed.

The data that support the findings of this study are available from the corresponding author upon reasonable request.

The raw data from this study are not publicly available due to institutional confidentiality requirements.