Educational Article
Control chart methods for monitoring surgical performance: A case study from gastro-oesophageal surgery

https://doi.org/10.1016/j.ejso.2010.10.008Get rights and content

Abstract

Graphical methods are becoming increasingly used to monitor adverse outcomes from surgical interventions. However, uptake of such methods has largely been in the area of cardiothoracic surgery or in transplants with relatively little impact made in surgical oncology. A number of the more commonly used graphical methods including the Cumulative Mortality plot, Variable Life-Adjusted Display, Cumulative Sum (CUSUM) and funnel plots will be described. Accounting for heterogeneity in case-mix will be discussed and how ignoring case-mix can have considerable consequences. All methods will be illustrated using data from the Scottish Audit of Gastro-Oesophageal Cancer services (SAGOCS) data set.

Introduction

Continuous inspection schemes to monitor the performance of surgeons, surgical centres, or new and complex surgical procedures are important methods to examine and assist in explaining sequences of poor surgical outcomes.1, 2, 3, 4, 5, 6 The rapid detection of deterioration in surgical performance and implementing new procedures is crucial in safeguarding the lives of patients.7, 8 Furthermore, not only healthcare providers must be able to identify and explain sequences of poor outcome but they should identify opportunities for continuous improvement using some form of objective monitoring and quality control.7 Traditional surgical approaches to this focussed on individual and unit level audits, and in more recent years on the accumulation of speciality databases nationally or regionally. Evidence that these efforts have been effective in improving surgical outcomes is weak. Two obvious problems have been the lack of an infrastructure for implementing change and the length of the feedback loop. When surgeons do identify problems with their practice, they often do not have the power to alter systems of practice to eliminate the problems, whilst it often takes a long time to accumulate enough cases to demonstrate conclusively that a problem has in fact occurred. The concept of continuous performance monitoring, or statistical process control, offers the possibility of dealing with the feedback loop length problem by making changes in success rates visible at an early stage and on a continuous basis.

The surgical oncology literature is relatively sparse regarding continuous outcomes monitoring compared to other disciplines such as cardiothoracic surgery,3, 9, 10, 11, 12, 13, 14, 15 kidney transplants16, 17 and obstetrics.18, 19, 20 Monitoring has been used successfully in oncology21 providing feedback to surgeons on their learning curves, typically in laparoscopic colon and rectal surgery22, 23, 24, 25 and in examining sentinel lymph node biopsy in breast cancer surgery.26, 27

This article illustrates the use of some of the current methods to monitor the outcomes from surgical oncology in a group of patients who have undergone gastro-oesophageal cancer surgery. This type of continuous surveillance has obvious potential as part of a quality-improvement system. Such systems, which are regarded as an indispensable standard tool in many industrial settings, are beginning to be reported in surgery. The use of a real example allows us to explore some of the problems which need to be overcome to develop and test a quality-improvement system for surgery based on this kind of statistical tool.

Section snippets

Patients and methods

The subjects of this study included all patients who underwent resection or attempted resection of the stomach or oesophagus in the Scottish Audit of Gastro-Oesophageal Cancer services (SAGOCS) between 1997 and 1999.28 The database was specifically designed to collect information on all aspects of treatment of gastric and oesophageal cancer. Age, sex, centre (hospital), date of operation, tumour stage and ASA grade were recorded for each patient. In this study the primary outcome variable was

Results

Between 1997 and 1999 there were data on 1300 operations from 41 centres recorded on the SAGOCS database, with an overall mortality rate of 12.9%. Fourteen centres were excluded as they contributed less than ten patients, leaving 1257 patients from 27 centres. Characteristics of the centres are presented in Table 1. The median mortality in the 27 centres was 12.8% (range 3.8–33.3%). Three quarters of all patients had ASA grade 1 or 2. The predicted mortality rate using the logistic regression

Discussion

Monitoring surgical outcomes is a key component in clinical governance. The ability to provide feedback on performance levels is important to improve healthcare and saves lives. We have illustrated the cumulative mortality plot, variable life-adjusted display, CUSUM and funnel plot as four such tools. No one single tool should ideally be used; instead a combination of methods should be used to monitor surgical outcomes.

Conclusions

Whatever the choice of continuous inspection scheme adopted, the ability to promptly detect underlying trends in mortality is limited. No single method will provide an instantaneous warning,4 without significantly increasing the number of false alarms. However, monitoring schemes like the ones described in this article can provide some insight into identifying changes in mortality rate providing the chart is appropriately constructed and assumptions on acceptable mortality rates are not

Conflict of interest

We have no conflicts of interest to declare.

Acknowledgements

We would like to thank the Scottish Audit of Gastric and Oesophageal Cancer and the Clinical Research and Audit Group of the Scottish Executive for supplying and permitting us to use the SAGOCS data set.

References (50)

  • D.B. Toll et al.

    Validation, updating and impact of clinical prediction rules: a review

    J Clin Epidemiol

    (2008)
  • D.A. Cook et al.

    Review of the application of risk-adjusted charts to analyse mortality outcomes in critical care

    Crit Care Resusc

    (2008)
  • G.L. Grunkmeier et al.

    Cumulative sum techniques for assessing surgical results

    Ann Thorac Surg

    (2003)
  • D.J. Spiegelhalter et al.

    Risk-adjusted sequential probability ratio tests: applications to Bristol, Shipman and adult cardiac surgery

    Int J Qual Health Care

    (2003)
  • T. Treasure et al.

    Surgical performance measurement

    Health Care Manag Sci

    (2002)
  • W. Woodall

    The use of control charts in health-care and public health surveillance

    J Qual Technol

    (2006)
  • J. Thor et al.

    Application of statistical process control in healthcare improvement: systematic review

    Qual Saf Health Care

    (2007)
  • A.A. Klein et al.

    Perception and reporting of cardiac surgical performance

    Semin Cardiothorac Vasc Anesth

    (2008)
  • A.A. Albert et al.

    On-line variable life-adjusted displays with internal and external risk-adjusted mortalities. A valuable method for benchmarking and early detection of unfavourable trends in cardiac surgery

    Eur J Cardiothorac Surg

    (2004)
  • J. Poloniecki et al.

    Retrospective cohort study of false alarm rates associated with a series of heart operations: the case for hospital mortality monitoring groups

    BMJ

    (2004)
  • C.B. Beiles et al.

    Cumulative sum control charts for assessing performance in arterial surgery

    ANZ J Surg

    (2004)
  • D. Collett et al.

    The UK scheme for mandatory continuous monitoring of early transplant outcome in all kidney transplant centers

    Transplantation

    (2009)
  • T. Sibanda et al.

    The CUSUM chart method as a tool for continuous monitoring of clinical outcomes using routinely collected data

    BMC Med Res Meth

    (2007)
  • S. Lane et al.

    Monitoring obstetricians’ performance with statistical process charts

    BJOG

    (2007)
  • T.A. Bowles et al.

    Time to CUSUM: simplified reporting of outcomes in colorectal surgery

    ANZ J Surg

    (2007)
  • Cited by (26)

    • Current methods for quality control and monitoring of medical care

      2014, Revista de Senologia y Patologia Mamaria
    • Monitoring methods in the surgical arena: Assumptions and potential pitfalls

      2013, European Journal of Surgical Oncology
      Citation Excerpt :

      This tendency penalizes smaller hospitals for treating larger proportions of disadvantaged patients. For the funnel plot, we imported the data of the original 27 centers and the original proportions of “sick” and “healthy” patients from Table 1 of the paper by Collins et al.8. We assumed that success rates for all 18 centers treating less than 25% “sick” patients are similar, and the other 9 centers will have a different rate of success as displayed in Table 2B.

    • Development and implementation of prospective outcome monitoring in a regional burns service using cumulative sum (CUSUM) techniques

      2013, Burns
      Citation Excerpt :

      This is the first study demonstrating the implementation of prospective outcome monitoring using CUSUM in healthcare since its use by cardiac surgeons [4,5]. Several studies [6] have discussed CUSUM in retrospective outcome assessment, but the authors believe that CUSUM has the potential to revolutionise the way healthcare professionals appraise their results and appreciate the power of learning from both positive and negative outcomes. All burns ITU admissions from 2003 to 2010 were assessed.

    • Healthcare 4.0 for the Improvement of the Surgical Monitoring Business Process

      2023, Communications in Computer and Information Science
    View all citing articles on Scopus
    View full text