Systematic reviews (SR) and meta-analyses (MA) are scientific investigations in which the unit of analysis is the original primary studies.1 They are now are an essential tool for synthesising available scientific information, increasing the validity of the conclusions of individual studies and identifying areas of uncertainty, and they play an important role in clinical decision making in the context of evidence-based clinical practice.2 However, for an SR/MA to be valid, it must meet certain quality criteria, which may be compromised when the original studies that make up the SR/MA have methodological deficits or biases in how they were conducted, or if these defects are present in the design of the SR/MA itself. Therefore, it is a priority to detect poorly conducted and/or poorly reported SRs, as this can lead to inappropriate use of evidence.3 The main factors affecting the quality of SR/MAs are:
- 1
Factors related to the original studies. Factors related to the original studies. All SRs/MAs must include an analysis of the main biases that may be present in the studies they include. Multiple tools have been developed to assess for bias, the most frequently used at present being the Cochrane Collaboration's tool to assess risk of bias.4 It has been found in different studies that bias is frequently present in original studies, and this impacts the final result of the review. However, it has also been observed that allocation of the presence or absence of bias by the investigators conducting the SR/MA of the primary studies is frequently discrepant, even in the studies carried out by the Cochrane Collaboration.5
- 2
Factors related to the conduct of an SR/MA. The biases that can occur in this type of study can also occur during the conduct of the review itself. All the stages in the construction of an SR/MA, from the formulation of the research question to be analysed to an exaggeration of the conclusions, can be affected by subjective decisions made by the investigators. Hence the importance of appropriate critical reading of the article to detect inherent biases in the research work.6 Some measures can minimise these biases, such as a documentalist being present in the task of the bibliographic search, or the search, the screening of articles, the extraction of data from them, and the analysis of risk of bias being performed by several people in the research team, resolving discrepancies by consensus or by another member of the team. On the other hand, the publication or availability of the study protocol makes it possible to check the similarity between what was planned and what was published.
Publication bias is another issue in conducting an SR/MA. The very essence of the SR/MA is the exhaustive analysis of all published and unpublished research conducted on the research question. This bias occurs especially when comparison of two interventions does not demonstrate the expectations of the researchers or sponsors of the research paper, or the differences between the comparisons are not significant, which may affect the final publication of the article. Therefore, the SR/MA should show that publication bias has been analysed with any of the tools available for that purpose.7
Tools for the comprehensive assessment of an SR/MABearing in mind the importance of evidence-based decision-making and its impact on patients, multiple tools and instruments have been designed and validated, generally by means of a checklist, to assess the methodological quality of published research papers. In relation to SR/MA, one of the first instruments used in Spain for critical reading were the checklists of the Spanish Critical Reading Skills Programme (CASPe). Other tools have subsequently been designed for this purpose. Of these, the most evaluated and validated is the AMSTAR-2 (Table 1).8 The AMSTAR-2 scale consists of a checklist of 16 items or domains whose verification by an evaluator or reader provides information on the methodological quality of the SR/MA, if it has been conducted with integrity and rigour. The answer to the different items assessed is simple, "yes" if the available information denotes a positive result for the question, "no" if is not adhered to or the information provided in the research work is insufficient to answer, and "partial yes" if the information is not complete. The critical items (those that most strongly affect the validity of the study) are considered to be items 2, 4, 7, 9, 11, 13, and 15. Depending on the response to the items and the impact of the critical items, the confidence in the results and conclusions of the review is rated as high / medium / low, or critically low. This tool can also be used as a guide for the construction and publication of a SR. The ease of use of this scale together with the time needed to complete it (between 15 and 32 min) make the AMSTAR-2 a useful tool to analyse the quality and therefore, the confidence in an SR.8
Domains of the AMSTAR-2 tool to analyse the methodological quality of an SR/MA.
| 1. Did the research questions and inclusion criteria for the review include the components of PICO? |
| 2. Did the report of the review contain an explicit statement that the review methods were established prior to the conduct of the review and did the report justify any significant deviations from the protocol? |
| 3. Did the review authors explain their selection of the study designs for inclusion in the review? |
| 4. Did the review authors use a comprehensive literature search strategy? |
| 5. Did the review authors perform study selection in duplicate? |
| 6. Did the review authors perform data extraction in duplicate? |
| 7. Did the review authors provide a list of excluded studies and justify the exclusions? |
| 8. Did the review authors describe the included studies in adequate detail? |
| 9. Did the review authors use a satisfactory technique for assessing the risk of bias in individual studies that were included in the review? |
| 10. Did the review authors report on the sources of funding for the studies included in the review? |
| 11. If meta-analysis was performed, did the review authors use appropriate methods for statistical combination of results? |
| 12. If meta-analysis was performed, did the review authors assess the potential impact of risk of bias in individual studies on the results of the meta-analysis or other evidence synthesis? |
| 13. Did the review authors account for risk of bias in primary studies when interpreting/discussing the results of the review? |
| 14. Did the review authors provide a satisfactory explanation for, and discussion of, any heterogeneity observed in the results of the review? |
| 15. If they performed quantitative synthesis did the review authors carry out an adequate investigation of publication bias (small study bias) and discuss its likely impact on the results of the review? |
| Did the review authors report any potential sources of conflict of interest, including any funding they received for conducting the review? |
A characteristic of research is that the authors of a research paper must clearly and completely present what was done, how it was done, and what the findings were, and this defines the transparency of a study. This is essential so that other researchers can reproduce the work, confirm the results and thus the conclusions (reproducibility). For this purpose, different tools have been designed to improve the transparency of scientific studies. The PRISMA statement is the most widely used in the field of SR/MA.9 The PRISMA guideline provides minimum recommendations to improve the transparency of SR presentation. It is a 27-item checklist to help researchers present their review papers. It is not specifically a tool that measures the methodological quality of an SR. Adequate transparency in the research and its presentation enables reproducibility. This enhances credibility and trust in a research study. Another problem that SRs/MAs have shown is that despite the PRISMA Declaration, the presentation of results is often insufficient, and the studies conducted to reproduce the work have shown that in many cases this cannot be done appropriately. Guidelines have been made to improve what is known as the "reproducibility crisis", based on the principle of openness of the entire scientific process, including improving the transparency of study design and the availability of data (open data).10
Therefore, although the SR/MA continues to be one of the main tools for adapting scientific evidence and transferring it to patient care, they are often not adequately conducted and/or reported. Although we have instruments to facilitate the presentation, and to analyse the quality of the SR/MA, these are often ignored so that the work can be published rapidly, encouraged an academic and professional system of incentives that is often perverse. If we want our decisions to significantly improve clinically, we must start with transparent research that allows reproducibility, thus increasing confidence in scientific articles and favouring better evidence-based clinical practice.
FundingThe authors have not received any funding for the preparation of this manuscript.
