Umbrella reviews

Introduction

Umbrella reviews (UR), which are also called overviews, are evidence synthesis designs that compile and synthesize the results of multiple systematic reviews (SR). URs are especially suitable for fields in which there is a lot of scattered and inconclusive evidence, coming from updated and quality SRs that respond to a number of questions.¹ For clinicians, URs are useful for drawing conclusions from SRs, and better contextualising their results. URs lead to a more general perspective, exploring trends and patterns, and facilitating decision-making.

As a type of literature review, they follow the same general structure as the SR, already discussed in a previous methodological note² but have specificities. This methodological letter presents the main aspects for developing a UR, focusing on its differences with respect to the SR, and providing examples of its application in the field of surgery.

Umbrella reviews objectives and inclusion criteria

URs are able to respond to a wide range of research questions, as shown in the examples in Table 1. The structure of the questions will follow the same pattern as in an SR, but the studies to be included in the synthesis project will be SRs. Therefore, an important aspect will be to identify and adequately define the minimum criteria that an SR must meet to be included in the UR. There are more or less strict criteria, but advisable requisites for inclusion are that an SR must at least: 1) describe a clear clinical question; 2) translate this question into explicit inclusion and exclusion criteria; 3) describe a reproducible bibliographic search in at least two sources of information; and 4) perform a risk of bias assessment of the included studies. Similarly, structured criteria have been proposed to make an informed decision on the inclusion of SRs in the UR, based on the overlap between existing SRs and the information they provide on the outcomes of interest.³

Table 1.

Examples of the main types of UR.

Objective	Examples in surgery
Assess the impact of different interventions for a specific health problem or population	Efficacy of robotic surgery compared with non-robot-assisted surgery (laparotomy or laparoscopy)^a
Assess the impact and safety of an intervention for different health problems or populations	Efficacy of the HARMONIC ultrasound cautery system in different applications^b
Assess the diagnostic accuracy of one or several different tests for the same health problem	Diagnostic accuracy of C-reactive protein for the identification of post-surgical complications^c
Assess the prevalence or prognosis of a health problem associated with one or more prognostic factors	Identification of prognostic factors for prosthetic infection^d

UR: Umbrella Reviews.

a

Romero-Tamarit et al., 2020. Ministerio de Sanidad. Informes de Evaluación de Tecnologías Sanitarias.

b

Kloosterman et al., 2023. https://doi.org/10.1186/s13643-018-0695-8.

c

Yang et al., 2022. https://doi.org/10.1016/j.ijsu.2022.106788.

d

Sabater-Martos et al., 2023. https://doi.org/10.1016/j.recot.2023.04.010.

Identification of relevant studies

There are many resources that index SR, although to carry out an UR it would be enough to search in MEDLINE (through PubMed) and Epistemonikos. According to the results of one study, this strategy, complemented by reviewing references from relevant studies, provides the best performance.⁴

Assessment of bias risk

In a UR, once the SRs to be included have been identified and selected, their risk of bias and validity must be evaluated. There are, among others, two complementary tools called AMSTAR and ROBIS.^5,6 AMSTAR measures methodological quality (e.g., how good the methods applied in a UR are) while ROBIS measures the risk of bias (e.g., how the limitations of the included SRs can lead to unreliable estimators). The application of both tools requires familiarity with the SR methodology, so it is advisable to carry out an initial pilot test among the authors to agree on criteria and improve their application and interpretation.

Assessment of overlapping

A specific phenomenon of the UR is the overlap between the included SRs. This overlap refers to the inclusion of primary studies in two (or more) SRs, and is considered as the volume of overlapping studies with respect to the total volume of SRs and primary studies. If the degree of overlap in a UR is high, then the results of the narrative or quantitative synthesis of the UR may be biased, due to the artificially high weight that overlaping studies will receive. There are tools to graphically show the nature and extent of overlap in a UR, such as the GROOVE tool⁷ or citation matrices.⁸ These matrices are cross tables whose columns show the SRs included in the UR, and rows show the individual studies that they include (Table 2). From the citation matrix, the degree of overlap can be manually calculated using the percentage of overlap and the corrected covered area (CCA). The CCA is a measure of the area of the matrix, once the first appearance of each primary study is eliminated; Therefore, the greater the overlap, the greater the ACC. Generally, a CCA between 0–5 is considered slight overlap, 6–10 moderate, 11–15 high, and greater than 15 very high.

Table 2.

Example of the citation matrix for calculation of overlap.

	Columns
Rows	RS	RS 1 (4 primary studies)	RS 2 (7 primary studies)	RS 3 (5 primary studies)
	Primary study
1	Primary study 1		x
2	Primary study 2			x
3	Primary study 3	x	x
4	Primary study 4			x
5	Primary study 5	x	x	x
6	Primary study 6		x
7	Primary study 7		x
8	Primary study 8	x
9	Primary study 9	x	x	x
10	Primary study 10		x	x

Overlap calculation formulas	% overlap: 3/10 = 30%
N: sum of all articles in all SRs	CCA: (16−10)/(10 × 3−10) = 0.2
r: number of rows or primary studies
c: number of columns or RS
% overlap: Number of RSs with 2 or more shared primary studies/r
Corrected covered area: (N – r)/(rc – r)

SR: Systematic Reviews.

Umbrella reviews reporting

For the publication of a UR results, the PRIOR guide is available, which consists of a checklist with 27 elements (and 19 sub-elements) that cover all the stages of a UR, as well as the justification and examples for each element.⁹ The use of reporting guidelines improves the quality and transparency in the presentation of the UR, and facilitates the evaluation of its reliability and applicability.

Funding

None.

Conflicts of interest

None.