Surgery: OR Crisis Checklists

Partners: Agency for Healthcare Research and Quality, Stanford University

Study Results: Implementation Science, 2018

Previous research has shown that cognitive aids, such as checklists or manuals, help surgical teams complete more of the life-saving steps needed to manage a crisis versus relying on memory alone. However, these tools are not regularly used in operating rooms (ORs) across the country. In this implementation study, the researchers sought to better understand what helps and hinders successful use of OR crisis checklists and manuals at surgical facilities in the United States.

The study team surveyed clinicians who downloaded either the  Ariadne Labs’ Operating Room Crisis Checklists or Stanford’s Emergency Manual: Cognitive Aids for Perioperative Critical Events. The 368 U.S.-based respondents came from hospitals and other surgical centers across geographic regions and ranging in size from one OR to more than 30. Most were anesthesiologists, and 54 percent had more than 20 years of professional experience.

Using a five-point Likert scale, respondents reported how true they believed the statement “at my facility, the tool is used regularly during applicable clinical events” to be. They provided information on how the tool was introduced and used at their facilities, including how many steps from an eight-step implementation process were completed. The steps listed were —presenting the tool at meetings with the surgical team, —customizing the tool, pilot testing it, —training people how to use the tool, providing ongoing or routine training on its effective use, monitoring its use and expanding use beyond the OR. Another item asked respondents whether the tool was used outside of critical events in the OR, such as to prepare for a complex case or to debrief after a critical event. The survey also asked whether the facility had previously engaged in other quality improvement efforts, such as simulation training, communication and teamwork training or emergency drills.

What did we learn?

• There are several factors associated with successful implementation. The results show that facilities were more likely to regularly use the checklists or manuals for crisis events when they:
  • Completed more implementation process steps.
  • Incorporated the tools into work outside of the OR in more ways.
  • Had more experience with other quality improvement initiatives.
  • Had support from department and hospital leadership.
  • Had dedicated time to train staff on how to use the tools.
• Respondents also identified barriers to implementation. These barriers included:
  • When clinical providers were resistant to the tool.
  • When there wasn’t a clear champion of the work.
  • When clinicians found the content and design of the tool unsatisfactory.

CONCLUSION: This work aligns with previous research findings that certain factors, like leadership support and following a clear and comprehensive plan, are important to successful implementation. The study team highlighted a number of key recommendations based on these findings, such as ensuring dedicated training time, using a multi-step implementation plan and using OR crisis checklists and manuals outside of emergency OR events.

Partners: South Carolina Hospital Association

Study Results: Annals of Surgery, 2017

While the success of the program to implement the WHO Surgical Safety Checklist in eight pilot sites showed the efficacy of using a team-based surgical safety checklist to improve patient outcomes, there have been few studies on how to translate research trial results into effective population-based implementation. Ariadne Labs, in collaboration with the South Carolina Hospital Association, addressed this gap. A 12-step, voluntary implementation program was developed with the goal of creating an operating room communication culture that improves overall surgical care and safety before, during, and after an operation.

The study team compared postoperative mortality rates after inpatient surgery between 14 hospitals in South Carolina who completed the Safe Surgery 2015 program by December 2013 and those that had not completed it. The program was built on principles taken from quality improvement initiatives that have successfully introduced team-based tools into complex clinical environments. There was a particular emphasis on multidisciplinary engagement, team alignment, and creating a culture of patient safety. Statewide all-payer discharge claims from 2008 to 2013 were matched up with state vital statistics to determine the program’s impact on postoperative mortality.

What did we learn?

• Before implementation, surgical mortality looked similar across the hospitals in the state. Before the launch of the program, the rate of death in the 30 days after surgery was the same across hospitals (3.38 percent in 2010 for completing hospitals versus 3.5 percent among other hospitals).
• After implementation in the completing hospitals, deaths after surgery dropped significantly. After the program was implemented, 30-day postoperative deaths in the 14 participating hospitals dropped by 22 percent compared to other hospitals.

CONCLUSION: Although all of the hospitals in the state had similar 30-day postoperative mortality rates before the program began, those hospitals completing the voluntary, checklist-based surgical quality improvement program were able to significantly lower mortality rates over the three years of the program. The study shows that a large-scale implementation of the surgical safety checklist is feasible to introduce at the population level and has a meaningful impact on patient outcomes.

Partners: South Carolina Hospital Association

Study Results: Medical Care Research and Review, 2015; Journal of American College of Surgeons, 2016; Annals of Surgery, 2017.

While the work of Ariadne Labs and others studying patient surgical safety has shown the effectiveness of checklists in both regular operating room practice and crisis scenarios, the implementation of a new tool can face challenges from the culture at a healthcare institution. An operating room climate may not be open to innovation. There may be deeply ingrained hierarchies in the surgical team that undermine the team-based approach to a new intervention. Individual clinicians may not be interested in changing the way they practice. These challenges can turn a promising intervention into little more than a box-ticking exercise. The study team sought to learn more about effective implementation of surgical checklists by exploring surgical team member perceptions of readiness, teamwork, adherence to, and consequence of safe surgical practice.

As part of the Safe Surgery 2015 South Carolina program, surgical team members at 38 hospitals completed a survey at the start of the program to assess perceptions of perioperative safety. The survey touched upon four domains:

• The organization’s readiness to implement the initiative, based on how much experience it has in implementing similar initiatives and how much staff across disciplines believe that the initiative is important for patient safety.
• How team members interact, communicate with and respect one another
• How much surgical teams follow established surgical safety practices
• Perception of the surgical team members of the impact the initiative will have on patient outcomes

Of the 38 hospitals that completed the survey at baseline, 13 completed the full implementation program. Once they were done, these hospitals answered the survey again. The post-intervention survey omitted the readiness items and instead added measures of implementation process and effectiveness as well as items to the dimension related to perceptions of the surgical team.

What did we learn?

• In the pre-program surveys, results varied widely across hospitals. For all domains of the survey, there was great variation in the average percentage of negative and neutral responses among hospitals.
• There was room for improvement in all hospitals. In each dimension that was measured, a substantial proportion of respondents reported negative or neutral perceptions around the different aspects of culture. In addition, 15 percent of surgical team members overall and up to 57 percent in individual hospitals indicated they would not feel safe being treated in their own operating room. Even among hospitals with the most positive responses, there was clearly room for improvement to overall culture.
• Staff reported an improvement in teamwork after implementing the program. In the hospitals that completed the program, survey respondents reported perceived improved coordination, communication, and respect for members of the surgical team. They felt more comfortable speaking up if they had a safety concern and that the team was better at adhering to safety practices.
• Staff also reported improvement in whether they would feel safe as patients in their ORs, but there is still work to be done. There was improvement in whether OR personnel would feel safe being treated as patients at their respective hospitals after implementation of the SSC, from 41.7percent in the completing hospitals pre-program to 49percent post-program. Despite the increase, the overall low response suggests the need for more work to improve perioperative safety.
• Staff felt crises were averted. Nearly three-quarters (73.6 percent) of post-program survey respondents said that using checklists had allowed their surgical team to avoid problems or complications.
• There were notable differences between how surgeons and other OR personnel believed the the SSC was performed in the post-program survey. Surgeons reported a higher completion rate at all three critical pause points in the SSC (before induction of anesthesia, before the first incision, and before the patient leaves the OR) compared with other OR personnel. Yet, they reported that using the SSC helped promote efficiency and avert problems or complications in the OR less frequently than nurses and surgical technicians. Conversely, nurses reported less often that their surgical teams always stopped at all three critical points, but they were among the most likely to agree or strongly agree that the SSC had improved efficiency and safety.

CONCLUSIONS: Our findings expand on previously published research which showed the positive effect of using an SSC on the perioperative culture of safety, teamwork, and communication. This study highlights the importance of assessing hospital culture when considering introducing a quality improvement program and identifying areas that may need attention to ensure success of the initiative. It also reinforces that the SSC fosters communication within the surgical team, leading to better teamwork and ultimately better outcomes for patients. Findings from this study suggest that implementation of the SSC affected health care professionals differently, depending on their role. Whichever the source, the need for more perspective sharing across disciplines is clear. Effectively implementing an SSC and using it in a way that improves teamwork is critical to actualizing potential improvements in operative outcomes.

Partners: Stanford University School of Medicine

Study Results: Journal of the American Medical Association, 2015

Cesarean section (C-section) is the most commonly performed operation in the world. Rates of cesarean childbirth delivery vary widely globally, from 2 percent to more than 50 percent of live births. Since the mid-1900s, the international health care community has seen the ideal C-section rate to be between 10 percent and 15 percent for optimal maternal and neonatal outcomes.

The study team gathered and correlated national C-section, maternal and neonatal mortality rates in a single year (2012) for all 194 World Health Organization member countries.

What did we learn?

• An increase in C-section rates reduces maternal and neonatal mortality, but only to a certain point. As the country-level C-section rate increases up to 19 percent, maternal and neonatal mortality rates decline. However, C-section delivery rates above 19 percent showed no further improvement in maternal and neonatal mortality rates.
• Some countries do not perform enough C-sections. There are many countries in which C-sections are not performed enough, meaning women do not have adequate access to safe and timely emergency obstetric care when they need it.
• Some countries perform too many C-sections. Conversely, there are many countries that perform more C-sections than will yield health benefits, possibly exposing women unnecessarily to the risk of complications.

CONCLUSION: This is the first study to offer a comprehensive analysis of C-section rates for all WHO countries in a single year. The findings present a compelling argument for improving surgical capacity in countries where access to care is limited to make C-sections available to women in instances of obstetric emergencies. Furthermore, the 19 percent rate found to be associated with health benefits to mother and child is a higher rate than recommended by the World Health Organization, suggesting that countries and policy makers should revisit the policies that set benchmarks for C-section rates on country-wide level. Though the findings do not pertain to individual patients or facilities, the study can provide some guidance on resource allocation and particular goals if governments are trying to strengthen their health care systems.

Partners: Hospital Nicolae Testemitanu, Chisinau, Moldova, and the University of Auckland

Study Results: Annals of Surgery, 2013; Journal of the American Medical Association, 2015

As work continues to spread the WHO Surgical Safety Checklist around the globe, attention has been given to low- and middle-income countries that face barriers generally not seen in higher income settings – lack of access to technology, such as pulse oximetry; lower levels of training for clinical staff; and less experience with quality improvement initiatives.

The study team set out to better understand implementation in a resource-limited setting by designing a program to introduce the WHO Surgical Safety Checklist in a hospital in Chisinau, Moldova, a lower-middle-income country in Eastern Europe. Prior to the study, pulse oximeters were not available in the hospital, preventing surgical teams from effectively monitoring a patient for hypoxemia, abnormally low levels of oxygen in the blood. Hypoxemia can be caused by complications of anesthesia and presents a significant risk of harm to the patient if not properly managed. The anesthesia and surgery leadership of the hospital recognized that the lack of pulse oximetry prior to the study was a barrier to implementing the Checklist and improving quality of care for their patients. The provision of pulse oximeters was an important element in the design of the program.

The study team sought to explore whether the steps on the checklist were completed, if there was a reduction in major complications, and if there was a decline in the rate of hypoxemia, abnormally low levels of oxygen in the blood. Furthermore, the team sought to learn if this program could be sustained outside of the research context by following up with the hospital leadership and surgical team two years after the end of the study.

What did we learn?

• Safety practices increased significantly. Adherence to all safety processes increased significantly, from 0 percent pre-intervention to 66.9 percent afterward.
• Complications decreased significantly. Rates of overall complications decreased from 21.5 percent to 8.8 percent, representing a 59 percent decrease. Surgical site infections dropped by 62 percent, from 17.7 percent to 6.7 percent. Other complications decreased by 55 percent, from 2.6 percent to 1.5 percent.
• Hypoxemia declined. The number of hypoxemic episodes lasting 2 minutes or longer per 100 hours of oximetry decreased by 44 percent from 11.5 to 6.4.
• The improvements were sustainable and improved over time. Two years after the initial implementation, the checklist was still in use, and there were continued improvements in process measures and reductions in complications 30-days postoperatively. Complications decreased by over an additional 30 percent from the post-intervention rate, and surgical site infections decreased by over an additional 40 percent. The rate of hypoxemic events continued to decrease as well. The average rate of completion of items on the checklist increased from 88 percent at the end of the study to 92 percent two years later.

CONCLUSION: Not only did the initial implementation show improvements in adherence to safety practices and overall complication rates, but there was continued improvement after the study ended without oversight from a research team. This highlights the importance of local leadership buy-in to the initiative and local champions among the surgical team who are engaged and committed to quality improvement.

Partners: CRICO / Risk Management Foundation

Study Results: Abstract presented at the American College of Surgeons 100th Annual Clinical Congress, 2014

Studies have shown that surgical team performance impacts patient safety, and health care facilities are eager to find ways to improve behaviors that may negatively affect how a surgical team provides care to their patients. Three hundred sixty degree evaluations are a performance appraisal process in which employees receive confidential and anonymous feedback about their their strengths and areas for improvement from their colleagues. The study team set out to learn whether 360-degree evaluations would be feasible and acceptable for surgeons.

Nearly 400 surgeons across a range of specialties in eight academically affiliated hospitals underwent 360-degree evaluations. Reviewers were members of the surgical team, and surgeons completed a self-assessment as well. Six months after the evaluations, a follow up survey was completed by surgeons, department heads, and reviewers to capture their perceptions of the feedback process and its impact.

What did we learn?

• Feedback was accurate. The majority of surgeons (85 percent) felt that the feedback they received from their colleagues was accurate, and the majority of department heads (80 percent) felt the feedback was reflective of the performance of their surgeons.
• Feedback catalyzed behavior change. Sixty percent of the surgeons said that they used the insights and observations provided by their colleagues to make changes to their behavior and practice. Some reviewers acknowledged behavior change in their colleagues after the evaluations were completed.
• The process was valuable. The majority of reviewers (70 percent) found the 360-degree evaluation process to be valuable, and 82 percent reported willingness to participate in future reviews.

Partners: Agency for Healthcare Research and Quality

Study Results: The New England Journal of Medicine, 2013

Operating room crises, such as hemorrhage or cardiac arrest, are rare, but the surgical team’s response to these events can mean the difference between life and death for a patient. Failure to manage these situations has been recognized as the largest source of variation in surgical mortality among hospitals. Other high-risk industries, such as aviation, have long used checklists as an acceptable tool to aid performance during rare, unpredictable, and critical events, and the use of surgical safety checklists as part of standard care is associated with reduced morbidity and mortality.

In this randomized controlled trial, teams from both academic and community hospitals were presented with a series of crisis scenarios in a high-fidelity simulated operating room. The teams, consisting of anesthesia staff, OR nurses, surgical technologists, and surgeons, were assessed on whether they followed evidence-based best practices to address these dangerous events. In half of the scenarios, the teams were given a set of checklists designed to address OR crisis events. In the other half, the teams worked from memory, the usual standard of care. Participants shared their thoughts on using the checklists through a post-simulation survey.

What did we learn?

• Fewer critical steps were missed when the checklists were used. During the scenarios in which the checklists were available, clinicians completed 93 percent of life-saving steps. Without the checklists they completed 77 percent.
• Participants liked using the checklists. The post-simulation surveys revealed that participants thought the checklists were easy to use and helped them feel better prepared. They reported that if they encountered these emergency scenarios in real life, they would use the checklists.
• Participants would want the checklists used if they were the patient. Perhaps most striking, almost all of the respondents (97percent) said that if they were having an operation during which an emergency occurred, they would want their surgical team to use the checklists.

CONCLUSION: In a setting where time-sensitive and appropriate care is essential, this study has shown that the use of checklists is acceptable to surgical teams and has the potential to meaningfully affect clinical practice and surgical outcomes.

Partners: World Health Organization (WHO)

Study Results: The New England Journal of Medicine, 2009

Implementation of the WHO Surgical Safety Checklist as part of the Safe Surgery Saves Lives program was associated with reductions in death and complication rates in eight hospitals around the globe.

The study team conducted a prospective study at eight hospitals participating in the WHO’s Safe Surgery Saves Lives program: Toronto, Canada; New Delhi, India; Amman, Jordan; Auckland, New Zealand; Manila, Philippines; Ifakara, Tanzania; London, England; and Seattle, Washington, United States. These sites were chosen to represent the diversity of clinical and socioeconomic surgical settings around the world.

The intervention was two-part and consisted of the checklist and an implementation program. The Checklist was introduced to the local surgical staff through lectures, written materials, and direct guidance. The primary investigators were involved with training local teams as well through a recorded video, teleconferences, and in-person visits to each site. Data collectors followed patients postoperatively for 30 days or until discharge from the hospital, whichever came first. The study team enrolled 3,733 patients during the baseline period and 3,955 patients after implementation of the checklist.

What did we find?

• Complications decreased 36 percent from 11 percent in all sites at baseline to 7 percent after introduction of the checklist.
• Rates of complication fell 30 percent among high-income sites, and 41 percent among lower-income sites.
• Deaths decreased from 1.5 percent to 0.8 percent, representing a 47 percent decrease.

CONCLUSIONS: The reduction in the rates of death and complications suggests that the checklist program can improve the safety of surgical patients in diverse clinical and socioeconomic environments. Though the study clearly resulted in better surgical outcomes, the exact mechanism that led to these improvements was likely due to multiple factors. The study team concluded that, applied at a global scale, the WHO Surgical Safety Checklist could prevent large numbers of deaths and complications, and that additional research would help identify the mechanism by which these improvements could be brought to diverse settings.

Partners: World Health Organization (WHO)

Study Results: Bulletin of the World Health Organization, 2016; The Lancet, 2008.

The amount and type of surgical care available worldwide has been difficult to document given the breadth of surgery and the varying levels of health infrastructure across countries. Using data collected from 192 WHO member states, the study team, including founding members of the Ariadne Labs Safe Surgery Program, devised a modelling strategy to estimate the number of major operations that occur around the world in 2004.  The team described the distribution of operations across countries based on per capita expenditure on health and then assessed the importance of surgical care in global public health policy. Using various data sources, a model was created to estimate the rate of major surgery in countries for which data were not available. This process was repeated again with data available for 2012.

What did we find?

• The volume of surgery worldwide is large and growing. The study team estimated that 266.2 to 359.5 million operations were performed in 2012. This represents an increase of 38 percent from the team’s 2004 estimate of 234.2 million.
• The more a country spends on health care, the more surgical procedures they perform. In 2012, countries in the lowest health expenditure category (US$100 or less per capita) only had an estimated average of 666 surgical procedures per 100,000 population, while countries in the highest health expenditure category (US$1000+ per capita) had 11,168.
• The distribution of surgical procedures does not align with the location of the world’s population. Countries with the lowest health expenditure account for over a third of the world’s population and yet only 6.3 percent of all surgical procedures in 2012. Meanwhile, countries with the highest health expenditure make up just under 18 percent of the world’s population but a staggering 60 percent of all operations.

CONCLUSIONS: The number of surgeries has continues to grow, particularly in very-low and low-expenditure countries. Given the volume of surgery worldwide and the risks from major surgical procedures, surgical safety deserves attention from the global public health community. Despite the increase of procedures in the lowest health expenditure countries, the disparity between surgical procedures and population size suggests that the significant gap in addressing disease burden in these settings persists eight years after the initial study. The 2008 Lancet study concluded that public health efforts and surveillance related to surgical safety should be established, laying the groundwork for the WHO Surgical Safety Checklist. The 2012 update to this work found that surgical surveillance continues to be weak and poorly standardized and urged the WHO member states to prioritize surgical safety and quality of care and limits the precision of these estimates.