Educational ArticleControl chart methods for monitoring surgical performance: A case study from gastro-oesophageal surgery
Introduction
Continuous inspection schemes to monitor the performance of surgeons, surgical centres, or new and complex surgical procedures are important methods to examine and assist in explaining sequences of poor surgical outcomes.1, 2, 3, 4, 5, 6 The rapid detection of deterioration in surgical performance and implementing new procedures is crucial in safeguarding the lives of patients.7, 8 Furthermore, not only healthcare providers must be able to identify and explain sequences of poor outcome but they should identify opportunities for continuous improvement using some form of objective monitoring and quality control.7 Traditional surgical approaches to this focussed on individual and unit level audits, and in more recent years on the accumulation of speciality databases nationally or regionally. Evidence that these efforts have been effective in improving surgical outcomes is weak. Two obvious problems have been the lack of an infrastructure for implementing change and the length of the feedback loop. When surgeons do identify problems with their practice, they often do not have the power to alter systems of practice to eliminate the problems, whilst it often takes a long time to accumulate enough cases to demonstrate conclusively that a problem has in fact occurred. The concept of continuous performance monitoring, or statistical process control, offers the possibility of dealing with the feedback loop length problem by making changes in success rates visible at an early stage and on a continuous basis.
The surgical oncology literature is relatively sparse regarding continuous outcomes monitoring compared to other disciplines such as cardiothoracic surgery,3, 9, 10, 11, 12, 13, 14, 15 kidney transplants16, 17 and obstetrics.18, 19, 20 Monitoring has been used successfully in oncology21 providing feedback to surgeons on their learning curves, typically in laparoscopic colon and rectal surgery22, 23, 24, 25 and in examining sentinel lymph node biopsy in breast cancer surgery.26, 27
This article illustrates the use of some of the current methods to monitor the outcomes from surgical oncology in a group of patients who have undergone gastro-oesophageal cancer surgery. This type of continuous surveillance has obvious potential as part of a quality-improvement system. Such systems, which are regarded as an indispensable standard tool in many industrial settings, are beginning to be reported in surgery. The use of a real example allows us to explore some of the problems which need to be overcome to develop and test a quality-improvement system for surgery based on this kind of statistical tool.
Section snippets
Patients and methods
The subjects of this study included all patients who underwent resection or attempted resection of the stomach or oesophagus in the Scottish Audit of Gastro-Oesophageal Cancer services (SAGOCS) between 1997 and 1999.28 The database was specifically designed to collect information on all aspects of treatment of gastric and oesophageal cancer. Age, sex, centre (hospital), date of operation, tumour stage and ASA grade were recorded for each patient. In this study the primary outcome variable was
Results
Between 1997 and 1999 there were data on 1300 operations from 41 centres recorded on the SAGOCS database, with an overall mortality rate of 12.9%. Fourteen centres were excluded as they contributed less than ten patients, leaving 1257 patients from 27 centres. Characteristics of the centres are presented in Table 1. The median mortality in the 27 centres was 12.8% (range 3.8–33.3%). Three quarters of all patients had ASA grade 1 or 2. The predicted mortality rate using the logistic regression
Discussion
Monitoring surgical outcomes is a key component in clinical governance. The ability to provide feedback on performance levels is important to improve healthcare and saves lives. We have illustrated the cumulative mortality plot, variable life-adjusted display, CUSUM and funnel plot as four such tools. No one single tool should ideally be used; instead a combination of methods should be used to monitor surgical outcomes.
Conclusions
Whatever the choice of continuous inspection scheme adopted, the ability to promptly detect underlying trends in mortality is limited. No single method will provide an instantaneous warning,4 without significantly increasing the number of false alarms. However, monitoring schemes like the ones described in this article can provide some insight into identifying changes in mortality rate providing the chart is appropriately constructed and assumptions on acceptable mortality rates are not
Conflict of interest
We have no conflicts of interest to declare.
Acknowledgements
We would like to thank the Scottish Audit of Gastric and Oesophageal Cancer and the Clinical Research and Audit Group of the Scottish Executive for supplying and permitting us to use the SAGOCS data set.
References (50)
- et al.
The influence of the surgeon on outcome
Best Pract Res Clin Anaesthesiol
(1999) Monitoring surgical performance
J Thorac Cardiovasc Surg
(2004)- et al.
Bristol, Shipman, and clinical governance: Shewart’s forgotten lessons
Lancet
(2001) - et al.
Direct comparison of risk-adjusted and non-risk-adjusted CUSUM analyses of coronary artery bypass surgery outcomes
J Thorac Cardiovasc Surg
(2006) - et al.
Monitoring the results of cardiac surgery by variable life-adjusted display
Lancet
(1997) - et al.
Control chart methods for monitoring cardiac surgical performance and their interpretation
J Thorac Cardiovasc Surg
(2004) - et al.
Transplant center quality assessment using a continuously updatable, risk-adjusted technique (CUSUM)
Am J Transplant
(2006) - et al.
Prospective evaluation of a continuous monitoring and quality improvement system for reducing adverse neonatal outcomes
Am J Obstet Gynecol
(2009) - et al.
Long-term survival after non-small cell lung cancer surgery: development and validation of a prognostic model with a preoperative and postoperative mode
J Thorac Cardiovasc Surg
(2006) - et al.
ASCOT: a comprehensive clinical database for gastro-oesophageal cancer surgery
Eur J Surg Oncol
(2001)
Validation, updating and impact of clinical prediction rules: a review
J Clin Epidemiol
Review of the application of risk-adjusted charts to analyse mortality outcomes in critical care
Crit Care Resusc
Cumulative sum techniques for assessing surgical results
Ann Thorac Surg
Risk-adjusted sequential probability ratio tests: applications to Bristol, Shipman and adult cardiac surgery
Int J Qual Health Care
Surgical performance measurement
Health Care Manag Sci
The use of control charts in health-care and public health surveillance
J Qual Technol
Application of statistical process control in healthcare improvement: systematic review
Qual Saf Health Care
Perception and reporting of cardiac surgical performance
Semin Cardiothorac Vasc Anesth
On-line variable life-adjusted displays with internal and external risk-adjusted mortalities. A valuable method for benchmarking and early detection of unfavourable trends in cardiac surgery
Eur J Cardiothorac Surg
Retrospective cohort study of false alarm rates associated with a series of heart operations: the case for hospital mortality monitoring groups
BMJ
Cumulative sum control charts for assessing performance in arterial surgery
ANZ J Surg
The UK scheme for mandatory continuous monitoring of early transplant outcome in all kidney transplant centers
Transplantation
The CUSUM chart method as a tool for continuous monitoring of clinical outcomes using routinely collected data
BMC Med Res Meth
Monitoring obstetricians’ performance with statistical process charts
BJOG
Time to CUSUM: simplified reporting of outcomes in colorectal surgery
ANZ J Surg
Cited by (26)
Current methods for quality control and monitoring of medical care
2014, Revista de Senologia y Patologia MamariaMonitoring methods in the surgical arena: Assumptions and potential pitfalls
2013, European Journal of Surgical OncologyCitation Excerpt :This tendency penalizes smaller hospitals for treating larger proportions of disadvantaged patients. For the funnel plot, we imported the data of the original 27 centers and the original proportions of “sick” and “healthy” patients from Table 1 of the paper by Collins et al.8. We assumed that success rates for all 18 centers treating less than 25% “sick” patients are similar, and the other 9 centers will have a different rate of success as displayed in Table 2B.
Development and implementation of prospective outcome monitoring in a regional burns service using cumulative sum (CUSUM) techniques
2013, BurnsCitation Excerpt :This is the first study demonstrating the implementation of prospective outcome monitoring using CUSUM in healthcare since its use by cardiac surgeons [4,5]. Several studies [6] have discussed CUSUM in retrospective outcome assessment, but the authors believe that CUSUM has the potential to revolutionise the way healthcare professionals appraise their results and appreciate the power of learning from both positive and negative outcomes. All burns ITU admissions from 2003 to 2010 were assessed.
Healthcare 4.0 for the Improvement of the Surgical Monitoring Business Process
2023, Communications in Computer and Information Science