Effect size in meta-analysis

Effect size indices

The effect size is the statistical value that reports the magnitude at which a phenomenon occurs or exists in a given population, providing information on the practical or clinical significance of a result.^1,2 In meta-analyses, effect size enables the results of different studies to be expressed in a common metric and presented as a global index. To this end, the effect size of an outcome of interest and its variance are calculated for each of the individual studies. The numerous effect size indices depend on the level of measurement of the outcome variables, the type of analysis, and the objectives of the study.³ Some of the most commonly used indices in meta-analysis are shown in Table 1.

Table 1.

Effect size indices and examples of their use.

Index	Example of use	Measurement of result	Statistic for calculation
Standardized mean difference	Magnitude of the difference between the means of 2 groups on a pain scale (1–10 points) after a surgical procedure.	Quantitative	n, means and standard deviation post-test in each group
Standardized mean change	Magnitude of change pre-test to post-test on a pain scale (1–10 points) in a single group after a surgical procedure	Quantitative	n, means and standard deviation pre-test and post-test
Standardized mean difference	Magnitude of the difference between the pre-test and post-test changes of 2 groups on a pain scale (1–10 points) after a surgical procedure	Quantitative	n, means and standard deviations pre-test and post-test in each group
Relative risk	Magnitude of the difference between mortality rates in 2 groups	Dichotomous	Total n and n with the event of interest in each group
Odds ratio	Ratio between the mortality advantages in 2 groups	Dichotomous	Total n and n with the event of interest in each group
Pearson correlation coefficient	Magnitude of the correlation between the pain scale score (1–10 points) and the time since the surgical intervention	Quantitative	n and correlation

For the comparison of groups in quantitative variables, the d family indices are most appropriate, as they are based on the difference between the means and allow for 2 groups to be compared with a quantitative variable (e.g., a 10-point pain scale). These can be used with both experimental and quasi-experimental designs by selecting the most appropriate index.^4,5

To compare 2 groups regarding a dichotomous variable, such as survival/mortality rate, indices based on hazard ratios are used. Hazard ratios refer to the probability that an event of interest (favorable or unfavorable) will occur in the presence or absence of a specific factor. The most common of these indices are the risk ratio (RR) and the advantage ratio or odds ratio (OR), along with their logarithmic transformations.⁶

Regarding effect size indices to evaluate the degree of association between variables, the most widely known is the Pearson correlation (r). In a meta-analysis of correlations, it is possible to use 2 different analytical strategies. In the first, the Pearson correlations would be combined directly, while in the second the Fisher z transformation would be applied first.⁷

When the study does not report the complete set of necessary statistics (a fairly common situation in practice), there are formulas to calculate effect sizes from partial data.³

Statistical models: Fixed effect vs. Random effects

The main purpose of meta-analysis is usually to obtain an average or overall effect size. When conducting a meta-analysis, one of the decisions that researchers must make is the selection of the most appropriate statistical model, with important implications for both the analysis and interpretation of the results. From the fixed-effect model, it is assumed that all studies estimate a single true effect size in the population and that the differences between the effects observed in the studies are due to sampling errors. From the random-effect model, it is assumed that the different studies estimate parametric effect sizes belonging to different populations.⁸

The fixed-effect model is appropriate when the studies have been carried out following identical procedures and the participants belong to the same defined population (e.g., patients from the same hospital). On the other hand, the random-effect model is more appropriate when the studies are heterogeneous. This last situation is the most common in practice, especially in the context of bibliographic reviews, so the random-effect model is the appropriate choice in most cases.

Interpretation of effect size

To interpret the magnitude of the effect, an effect can be classified as low, moderate or high in magnitude according to value ranges.⁹ Despite the existence of these mathematical criteria, researchers must take into account the context and the nature of the phenomenon studied by explaining the meaning of the effect in the real world.

Another factor to consider when interpreting the effect size is the accuracy of the estimate, which is determined by the confidence interval.³ The range of this interval indicates the value range of the average effect size in a universe of comparable populations at a 95% confidence level.¹⁰ The confidence interval also provides information about the statistical significance of the effect size. When this does not include the null value (e.g., 0 in the case of the standardized mean), the effect is statistically significant.

The confidence interval should not be confused with the prediction interval, which provides information about the dispersion of the population effects, indicating the minimum and maximum values between which 95% of the population effect sizes would be found.¹⁰ This heterogeneity of effect sizes also has important implications for the interpretation of the results. If there is little dispersion between the true effects of the studies, the overall effect size will be a good estimator of the average effect of each of the different populations. In contrast, if the dispersion is high, the meta-analysis should focus on the factors responsible for this variability.