Systematic reviews (SR) and meta-analyses (MA) are syntheses of existing studies. Using a precise and strict methodology that provides scientific quality, they allow a comprehensive view of the different results from addressing a specific clinical question. They support in clinical decision making and can guide future research. However, SR-MRs also have biases and limitations. This methodological letter analyses the quality of SR-MRs along with their deficient areas and points for improvement in the field of surgery.
The intrinsic quality of SR-MAs is crucial to the validity and reliability of their conclusions and will depend on each of the stages in the development of an SR being correctly conducted. Mei et al.,1 assessed the registration and reporting quality of SRs of randomised clinical trials (RCT) on surgical intervention. They found that 37% were registered, and that this was associated with higher data reporting quality based on the PRISMA guideline, which shows the importance of registration in the transparency and reproducibility of SR-MAs.
Tools such as ROBIS and AMSTAR-2 are available to facilitate the assessment of SR-MAs.2 In a review of 150 MAs focusing on surgical interventions, the level of compliance with PRISMA was observed as moderate and adherence to AMSTAR as poor, with only 48% of the elements adequately described.3 This finding underlines the importance of improving the presentation of results, as even a methodologically sound SR-MA can be negatively affected by poor presentation of results.
Analysis of publication bias and heterogeneity are critical elements in assessing the quality of an MA. An analysis of 200 randomly selected SRs of health interventions revealed that only 10% formally assessed publication bias.4 Implementing strategies to identify and mitigate publication bias is crucial. Tools such as ROB-ME are now available to specifically identify publication bias due to missing evidence,5 specifically analysing the risk of bias in the outcome of a meta-analysis arising from missing studies and missing results in the included studies. Platforms such as https://metabias.io/ allow analysis of p-hacking,6 which occurs when data or statistical analyses are selected in research until significant results are obtained.
The heterogeneity of SR-MAs in surgery often reflects the inherent difficulties of RCTs of surgical interventions, where factors such as differences in surgical techniques, surgeons’ experience, pre- and post-operative management, as well as individual preferences, can make it difficult to interpret results and generalise conclusions. The ROLARR trial, for example, included centres at different stages of the learning curve in robotic surgery. When analysing the effect of learning on postoperative outcomes, the results obtained differed from those of the primary analysis of the RCT.7 These differences in outcomes, if not analysed, may transfer over to SR-MAs in surgery.
A challenge in conducting SRs of surgical interventions is the incomplete recording of intervention-related data. In 204 SRs involving surgical RCTs, only 40.2% provided an adequate description of the interventions.8 Guidelines are recommended to address these deficiencies, such as TIDieR,9 un checklist of the information that should be included in studies about the intervention.
In addition, the outcome variable in surgery may vary between studies, making it difficult to extract data and make comparisons. In an MA on postoperative complications in liver surgery, missing and variable data were limitations in performing meta-regression.10 Using guideline-based definitions to homogenise outcome variables allows better comparison of results between studies and facilitates the development and interpretation of SR-MAs.
Research in surgery has focused on clinical experience and observational studies, which limits the availability of quality evidence for inclusion in SR-MAs. Dixon et al. reported that 35% of published MAs in surgery included 10 studies or fewer.11 This may make interpretation of heterogeneity difficult because heterogeneity tests have low power when there are few studies in an MA and may not detect genuine discrepancies.12 On the other hand, including a significant number of studies may result in including very heterogeneous study populations with differences in populations and interventions. It is important to establish inclusion/exclusion criteria for studies that accurately answer the scientific question.
One limitation of an SR-MA is the impossibility of comparing more than two alternative treatments at the same time. In clinical situations where there are multiple treatment options or where there are no RCTs comparing two or more interventions it is recommended to use a network meta-analysis,13 a statistical technique that uses direct and indirect comparisons from a network of studies to examine the effects of various treatments more thoroughly.
In summary, adopting rigorous approaches, using appropriate methodological tools and specifically addressing heterogeneity are essential aspects to improve the quality of SR-MS in surgery. This will help a more reliable and relevant synthesis of evidence for clinical decision making.
