Consensus-based validation of key quality indicators in pancreatic cancer surgery in Catalonia: A modified Delphi study

Abstract

Introduction

Pancreatic cancer surgery is highly complex and associated with significant perioperative risk, high mortality and morbidity, low quality of life, and increased healthcare costs. Establishing quality indicators to monitor these procedures may improve care quality and outcomes. This study aims to define a set of quality indicators to evaluate pancreatic cancer surgery in Catalonia.

Methods

A preliminary list of quality indicators was developed based on a literature review. Subsequently, a modified Delphi study was undertaken, involving 11 expert pancreatic surgeons, one from each center where these complex surgeries are authorized in Catalonia, to achieve consensus on quality indicators for pancreatic cancer surgery.

Results

The literature review yielded 21 potential quality indicators. After 4 rounds of voting, the expert panel accepted 17 indicators and rejected 4. The final set of quality indicators encompasses critical aspects including positive tumor resection margin, lymph nodes retrieved, postoperative complications, hospital readmission, reoperation, 90-day mortality and one-year survival.

Conclusion

The outcome of this study is a set of agreed quality indicators to measure, describe and monitor the quality and outcomes of pancreatic cancer surgery. These quality indicators can be used as a foundation for benchmarking studies and systematic evaluations of quality of care and enhanced surgical performance in pancreatic cancer.

Introduction

La cirugía del cáncer de páncreas es altamente compleja y se asocia a un riesgo perioperatorio significativo, elevada mortalidad y morbilidad, baja calidad de vida e incremento de los costes sanitarios. Establecer indicadores de calidad para monitorizar estos procedimientos puede contribuir a mejorar la calidad asistencial y los resultados. Este estudio tiene como objetivo definir un conjunto de indicadores de calidad para evaluar la cirugía del cáncer de páncreas en Cataluña.

Métodos

Se elaboró una lista preliminar de indicadores de calidad a partir de una revisión de la literatura. Posteriormente, se llevó a cabo un estudio Delphi modificado con la participación de 11 cirujanos expertos en páncreas, uno de cada uno de los centros autorizados para realizar estas cirugías complejas en Cataluña, con el fin de alcanzar un consenso sobre los indicadores de calidad en la cirugía del cáncer de páncreas.

Resultados

La revisión de la literatura identificó 21 indicadores de calidad potenciales. Tras cuatro rondas de votación, el panel de expertos aceptó 17 indicadores y rechazó 4. El conjunto final de indicadores de calidad abarca aspectos clave como el margen de resección tumoral positivo, el número de ganglios linfáticos extraídos, las complicaciones postoperatorias, el reingreso hospitalario, la reintervención, la mortalidad y la supervivencia.

Conclusión

El resultado de este estudio es un conjunto consensuado de indicadores de calidad para medir, describir y monitorizar la calidad y los resultados de la cirugía del cáncer de páncreas. Estos indicadores pueden servir como base para estudios comparativos que permitan una evaluación sistemática de la calidad asistencial y la mejora del rendimiento quirúrgico en el cáncer de páncreas.

Introduction

Pancreatic cancer surgery is highly complex, involving significant perioperative risks and requiring multidisciplinary expertise. The associated high mortality and morbidity can increase healthcare costs and negatively impact patients’ quality of life.¹ These surgeries are generally categorized into 2 distinct procedures: pancreaticoduodenectomy and left-side pancreatic resection. Each procedure has different technical implications, complications and levels of complexity.² Pancreaticoduodenectomy involves the resection of the pancreatic head as well as the duodenum, distal bile duct and the gastric antrum (in certain cases), which necessitates a reconstructive phase to restore gastrointestinal continuity. Left-sided pancreatic resection involves the resection of the pancreatic body and/or tail.³ Depending on the extent of the disease, resection may be performed at different levels of the gland. Both procedures are highly standardized in terms of the definition of radicality (resection margins, number of lymph nodes retrieved, and relationship with the superior mesenteric vessels), surgical technique, and the classification of potential complications. Index complications for pancreatic surgery include clinically relevant postoperative pancreatic fistula (CR-POPF),⁴ delayed gastric empty (DGE)⁵ and postoperative pancreatic hemorrhage (PPH),⁶ all of which are clearly defined in the literature following the definitions of the International Study Group of Pancreatic Surgery (ISGPS).

Considering the highly complex nature of these procedures, various countries have centralized the performance of pancreatic cancer surgery in specialized reference centers as a way to improve clinical outcomes.⁷ In Catalonia (Spain), a centralization strategy for high-complexity procedures was implemented in 2012. This strategy designates reference centers for a given procedure based on a minimum annual volume of procedures performed (high-volume expertise), along with certain qualitative factors. This strategy has led to a significant improvement in 30-day postoperative mortality in tumors such as those of the esophagus, pancreas, liver, and rectum.⁸ As of the latest update in 2019, reference hospitals for pancreatic cancer surgery must perform at least 11 surgical procedures (pancreatectomy) per year; currently, 11 tertiary hospitals in Catalonia’s Public Healthcare System (CatSalut) meet this threshold.⁹ The regulations governing the centralization strategy stipulate that the Catalonian Cancer Plan monitor the quality of procedures and provide feedback to the centers. In this context, the mandatory audits of the first years (2012-2015) were replaced in 2019 with a prospective registry, the Oncology Tertiary Care Registry.

Systematic analyses of results using benchmark indicators^10–14 or textbook outcomes^15–17 have driven improvements in surgical performance. Benchmarking studies compare an institution’s performance against established standards, while textbook outcomes allow for the comprehensive evaluation of surgical processes and outcomes.^10–17 Thus, the selection of key quality indicators is essential to monitor procedural success and enhance the standard of care. Various international experiences have used the Delphi method to select and prioritize quality indicators across a range of healthcare areas.^18–21 This technique is a structured, systematic method used to gather expert opinions through iterative rounds of questionnaires, and feedback is provided after each round to refine responses and build consensus. The modified Delphi technique starts with a set of pre-selected items (rather than open-ended questions) to improve initial response rates and ground the study in existing research.²²

This study aims to define a set of quality indicators to evaluate pancreatic cancer surgery in Catalonia.

Methods

A modified Delphi method^22–24 was used to devise a list of key quality indicators for pancreatic cancer surgery in populations with the following characteristics: age ≥18 years; no metastatic disease; undergoing pancreatic surgery; and histological diagnosis of ductal adenocarcinoma or another malignant process (excluding neuroendocrine tumors).

Literature review

The first step in the study was to formulate a preliminary list of quality indicators for pancreatic cancer surgery, starting from the list of quality indicators developed for an earlier study by our group.²⁵ We then built on this through an additional search undertaken in the MEDLINE/ PubMed database (from 2013 to 2023) using MeSH terms referring to relevant key words: cancer (eg, carcinoma, neoplasm, hepatopancreaticobiliary), surgery (eg, resection, procedures, hepatopancreatic, operative) and quality indicators (eg, quality measure, textbook outcome, benchmark). The search was restricted to studies in English. For the complete search strategy, see Supplementary Table 1.

Table 1.

Reference hospitals for pancreatic cancer surgery in the Catalonian Public Healthcare System^¥

Hospitals (n = 11)
Hospital Universitari Parc Taulí
Hospital del Mar de Barcelona
Hospital Clínic de Barcelona
Hospital Universitari de Bellvitge
Hospital de la Santa Creu i Sant Pau
Hospital Universitari Germans Trias i Pujol
Hospital Universitari Doctor Josep Trueta
Hospital Universitari Mútua de Terrassa
Hospital Universitari de Tarragona Joan XXIII
Hospital Universitari Vall d’ Hebron
Hospital Universitari Arnau de Vilanova

¥

The reference center hospitals within the Catalonian Public Healthcare System are included in the centralization strategy for high-complexity procedures.⁹

The reference lists of all the articles retrieved were reviewed to identify additional studies for potential inclusion. Publications by prominent authors in the field, as well as citations made on key papers, were also searched and reviewed.

Expert panel

The panel was made up of the 11 section chiefs and expert reference surgeons in pancreatic cancer surgery, one from each center where these complex surgeries are authorized in Catalonia (Table 1). Expert pancreatic surgeons were defined as senior surgeons practicing in accredited high-volume referral centers, with a sustained annual case volume of ≥ 20 major pancreatic resections, cumulative experience > 5 years in pancreatic surgery, and documented academic or scientific involvement in the field of pancreatic cancer. This definition is consistent with international accreditation frameworks for complex pancreatic surgery, which link expertise to both surgeon-level and center-level volume. Current European guidelines recommend concentration of pancreatic resections in referral centers with structured quality programs, and expert status is typically attributed to senior surgeons with stable annual volumes and recognized academic contribution. These elements are common to the criteria adopted by the European-African Hepato-Pancreato-Biliary Association (E-AHPBA),²⁶ the European Reference Networks ERN²⁷ and The European Society for Medical Oncology (ESMO).² All experts were invited to participate through an email sent from the Catalonian Cancer Plan on 23 February 2024. Each participant was required to disclose any conflicts of interest before taking part in the study. As each round of consultations drew to a close, reminders were sent to encourage those who had not yet responded to submit their contributions.

Modified Delphi method

A modified Delphi method was employed to achieve consensus on a closed set of key quality indicators. The process is characterized by anonymity, iteration with controlled feedback, statistical group response, and expert input.^22–24

To reach a final set of quality indicators, 4 rounds of voting were conducted in the spring and summer of 2024, on 8 March, 3 April, 2 May, and 30 July. Experts were asked to rate their agreement with each proposed indicator on a three-point scale: agree, neither agree nor disagree, or disagree. We calculated the percentage of experts who rated each quality indicator as "agree", "neither agree nor disagree", or "disagree”. Criteria for including each indicator in the final set were determined a priori, with consensus defined as ≥ 80% agreement.²⁸ Quality indicators that did not reach consensus in a given round were refined and adjusted based on expert feedback and then subjected to another vote in the subsequent round. After the first 3 rounds, a detailed report summarizing the results was sent to the experts. The report included a table compiling the voting results, a response to expert comments, and a graphical representation of the feedback. After the final round, the consensus document including the set of quality indicators for pancreatic cancer surgery was sent to all experts.

Results

The literature search identified 417 records published from 2013 to 2023. After removing duplicates and screening titles and abstracts, 388 records were excluded. Following full-text review, an additional 13 studies were excluded due to the absence of data on quality indicators for pancreatic cancer surgery. Three additional studies were subsequently included, for a total of 19 relevant publications that were used to establish a preliminary list of 21 quality indicators for pancreatic cancer surgery. These proposed indicators were subsequently included in the modified Delphi rounds (see Supplementary Table 2). The definitions of the terms included in the different quality indicators are presented in Supplementary Table 3.

Table 2.

Number of quality indicators subjected to vote and achieving consensus in each modified Delphi round.

Round	Quality indicators submitted for voting (n)	Quality indicators with consensus (n)	Quality indicators without consensus - progression to next round (n)	Quality indicators without consensus - excluded (n)
First	21	15	5	1
Second	5	1	4^β	0
Third	3	0	2	1
Fourth	3	1	_	2

β

One indicator was not assessed in the third round and was amended for inclusion in the fourth round.

Table 3.

Description and formulas for the final set of consensus-based key quality indicators for pancreatic cancer surgery.

No.	Domain	Description	Formula
1	Positive tumor resection margin (R1)	Proportion of cases with R1 margins^¥	(Number of cases with R1 margins / total population) × 100
2	Lymph nodes retrieved	Proportion of cases with 15 or fewer lymph nodes retrieved	(Number of cases with 15 or fewer lymph nodes retrieved / total population) × 100
3	Postoperative complication	Proportion of cases with at least one postoperative complication classified as Clavien-Dindo grade ≥ III^¥ within 90 days after surgery	(Number of cases with at least one postoperative complication of Clavien-Dindo grade ≥ III within 90 days after surgery / total population) × 100
4	Postoperative complication	Proportion of cases with at least one postoperative complication classified as Clavien-Dindo grade IV^¥ within 90 days after surgery	(Number of cases with at least one postoperative complication of Clavien-Dindo grade IV within 90 days after surgery / total population) × 100
5	Postoperative complication	Proportion of patients with CCI^¥ >P75 within 90 days after surgery	(Number of cases with a CCI > P75 within 90 days after surgery / total population) × 100
6	Postoperative complication	Proportion of cases with surgical site infection^¥, including superficial or deep incisional, or organ/space infection within 90 days after surgery	(Number of cases with superficial or deep incisional, or organ/space surgical site infection within 90 days after surgery / total population) × 100
7	Postoperative complication	Proportion of cases with organ/space surgical site infection^¥ within 90 days after surgery	(Number of cases with organ/space surgical site infection within 90 days after surgery / total population) × 100
8	Postoperative complication	Failure to rescue rate ^¥ within 90 days after surgery	(Number of deaths among cases with Clavien-Dindo > II within 90 days after surgery / number of cases with Clavien-Dindo > II) × 100
9	Postoperative complication	Proportion of cases with a CR-POPF grade B or C ^¥ within 90 days after surgery	(Number of cases with CR-POPF [grade B or C] within 90 days after surgery / total population) × 100
10	Postoperative complication	Proportion of cases with a CR-POPF grade B ^¥ within 90 days after surgery	(Number of cases with CR-POPF [grade B] within 90 days after surgery / total population) × 100
11	Postoperative complication	Proportion of cases with a CR-POPF grade C ^¥ within 90 days after surgery	(Number of cases with CR-POPF [grade C] within 90 days after surgery / total population) × 100
12	Postoperative complication	Proportion of cases with biliary fistula^¥ within 90 days after surgery	(Number of cases with postoperative biliary fistula within 90 days after surgery / total population) × 100
13	Postoperative complication	Proportion of cases with severe PPH (grade B or C)^¥ within 90 days after surgery	(Number of cases with severe PPH [grade B or C] within 90 days after surgery / total population) × 100
14	Hospital readmission	Proportion of cases with a hospital readmission within 90 days after surgery	(Number of cases readmitted to the hospital within 90 days after surgery / total population) × 100
15	Reoperation	Proportion of cases with a reoperation^¥ within 90 days after surgery due to complications	(Number of cases requiring reoperation within 90 days after surgery / total population) × 100
16	Mortality	Proportion of deaths within 90 days after surgery	(Number of deaths within 90 days after surgery / population 1^a or population 2^b) × 100
17	Survival	Proportion of cases alive one year after surgery procedure	(Number of patients alive at one year after surgery/ total number of patient-years of follow-up) × 100

CCI: Comprehensive Complication Index

¥ Term definitions are collected in Supplementary Material Table S2.

a. Patients having undergone pancreaticoduodenectomy, without metastasis (no M1), ≥18 years, excluding R2 affected margins

b. Patients having undergone pancreaticoduodenectomy for ductal adenocarcinoma, without metastasis (no M1), ≥18 years, excluding R2 affected margins

The study involves 11 expert pancreatic surgeons, each representing one reference hospital in Catalonia authorized to perform pancreatic cancer surgery. The participation rate was consistently 100% (11/11 experts) in all 4 rounds. The expert panel ultimately accepted 17 of the proposed quality indicators and rejected 4. Table 2 describes the number of quality indicators voted on and those that achieved or did not achieve consensus in each round of voting. Supplementary Tables 4–7 show the rates of agreement, disagreement, or neither agreement nor disagreement for each quality indicator and round of consultation. Table 3 presents the final set of consensus-based quality indicators, along with their definitions and formula, for pancreatic cancer surgery in Catalonia. The 4 quality indicators ultimately rejected were: proportion of cases with 19 or fewer lymph nodes retrieved, proportion of cases with at least one postoperative complication classified as Clavien-Dindo grade I-V within 90 days after surgery; proportion of cases with at least one postoperative complication classified as Clavien-Dindo grade I-II within 90 days after surgery, and proportion of cases with a length of hospital stay over the 75^th percentile.

Discussion

This four-round modified Delphi study established a set of 17 quality indicators for pancreatic cancer surgery. These will enable monitoring of the performance and outcomes in the 11 reference centers where this complex procedure is authorized within the Catalonian Health System. The indicators encompass critical aspects of oncological outcomes (positive tumor resection margin and lymph nodes retrieved), postoperative complications, hospital readmission, reoperation, 90-day mortality, and one-year survival. Over 80% of the surgeons at authorized centers agreed on each quality indicator.

Pancreatic surgery requires a specific approach to quality assessment, as many of its index complications are unique to this type of surgery. Consequently, it is essential to define and reach consensus not only on general postoperative complications—typically graded using the Clavien–Dindo classification—but also on specific, potentially life-threatening complications, such as CR-POPF and PPH, which directly reflect the technical quality of the surgery. In addition, quality indicators related to the oncological adequacy of the resection—such as margin status and lymph node yield—must also be considered when evaluating the overall quality of pancreatic surgical interventions.

Some studies have proposed quality indicators specific to hepatobiliary and pancreatic surgery, such as CR-POPF rates, postoperative biliary fistula and PPH,^11,29–31 while others have detailed indicators tailored to oncology surgery, such as resection margin status or number of lymph nodes resected.^{11,29,30,32–37} Different authors have also suggested some cross-cutting quality indicators applicable to various surgical procedures, including metrics such as postoperative complications, surgical site infection, hospital readmission, reoperation mortality, and survival.^{11,15,29–32,34–36,38–45}

Although one study in Spain has already identified key quality indicators for pancreatic surgery,²⁹ these do not enjoy the explicit consensus of all the reference centers from the different autonomous communities in the country. The decentralized nature of the Spanish healthcare system makes this endeavor particularly challenging. Our study took place within the framework of the Catalonian Health System and included all reference hospitals in Catalonia where pancreatic cancer surgery is authorized.

There is no widely accepted gold standard in this field at the European level. Specific quality indicators have been selected in several regions and countries, such as the Netherlands,^{31,32,34,35,39,41} and some have been proposed through international multicenter studies, for instance the papers by Sánchez-Velázquez, et al¹¹ on pancreaticoduodenectomy or Woodhouse et al⁴⁶ on hepatectomy, pancreatectomy, complex biliary surgery and cholecystectomy.

However, there have been European efforts aimed at establishing tumor-specific quality indicators for oncological care, including pancreatic cancer. For example, the Joint Action Innovative Partnership for Action Against Cancer (iPAAC), aimed to create respective sets of quality indicators for the treatment and care of colorectal and pancreatic cancer. These were intended to support continuous quality improvement efforts at national, regional, and comprehensive cancer care network (CCCN) levels across EU Member States.⁴⁷ At the time of writing, an updated set of quality indicators for pancreatic cancer in the CCCN was under development within the framework of Joint Action European Network of Comprehensive Cancer Centers (EUnetCCC).⁴⁸ Another notable European initiative is the consensus recommendations for improving pancreatic cancer care, discussed in the Bratislava Statement.⁴⁹ Building on this foundation, an analysis of different quality metrics for pancreatic surgery could help establish reference values, identify unmet needs, and support decision-making for the organization, management, and delivery of pancreatic cancer care.

The continuous improvement of healthcare quality is a fundamental priority for any health system, and complex procedures like pancreatic cancer surgery require precise evaluation to ensure optimal outcomes and efficient resource use.^29,30 As a second step, the quality indicators established could be used as a basis for a comprehensive benchmarking study.^11–13,29 In fact, one such study has already been launched in Catalonia to evaluate the outcomes of pancreatic cancer surgery by procedure in authorized centers, using the consensus-based quality indicators identified in the present study. Quality indicators provide objective, measurable criteria that enable the systematic evaluation of surgical outcomes and healthcare performance across different centers. Hence, the definition of these indicators could promote the identification of best practices and areas for improvement; standardize, simplify, and accelerate the evaluation process; and enable comparisons between centers and the provision of feedback to improve outcomes.

The main advantage of this study is the full participation by all reference centers authorized to perform pancreatic cancer surgery in Catalonia. However, the study does have some limitations. First, one indicator was unintentionally omitted after the second round but was reintroduced in the last; nevertheless, all initially proposed indicators were finally assessed and reached consensus. Secondly, the way the information and survey were framed could introduce a risk of information bias. Although the survey was not piloted, the formulation of the questions was simple and clear, with neutral language to mitigate the risk of bias. Moreover, the survey followed a standardized format to ensure consistency and comparability of responses. Finally, the list of indicators was not open for further additions.

Conclusions

This study identified 17 quality indicators for pancreatic cancer surgery, encompassing critical aspects such as positive tumor resection margin, lymph nodes retrieved, postoperative complications, hospital readmission, reoperation, 90-day mortality, and one-year survival. The quality indicators were identified based on scientific evidence and agreed upon by pancreatic surgeons from all reference centers in Catalonia. The set of indicators can be used to measure, describe, and monitor the outcomes and quality of surgeries performed in this setting.

The consensus achieved in this study can also be used in benchmark studies or textbook outcomes to evaluate the quality of care for pancreatic surgery, comparing different types of procedures (such as pancreaticoduodenectomy or left-sided pancreatic resection), patient cohorts across various centers, and against established reference standards, which would support the improvement of surgical performance.

Conflicts of Interest

Authors declare that they have no conflict of interest.

Funding Statement

The authors declare they have not received financial support.

Informed Consent Statement

Since this was a modified Delphi study, informed consent was not required.

Uncited references

[50,51,52,53,54,55,56,57].

Acknowledgment

We thank Meggan Harris for her editorial assistance in the preparation of this paper.