Previous publications demonstrated that the presentation of treatment benefits in terms of relative risk reduction (RRR) rather than in terms of absolute risk reduction (ARR) or number of patients to treat (NNT) might favor the perception of outcome effectiveness. The objective was to perform a cognitive evaluation to assess how the manner in which risks and benefits of screening methods and treatments are presented can affect medical care decision-taking in a sample of cardiologists.
MethodsFour-hundred and six Latin American cardiologists answered a questionnaire reporting the results of clinical trials presented as RRR, ARR or NNT, associated or not to biased graphs.
ResultsCardiologists’ decision-taking was different when comparing treatment benefits presented as RRR (62.2%) vs. ARR (40.4%) (p=0.000000), and as RRR vs. NNT (44.4%) (p=0.000000). However, their decision-taking was similar when information was presented as NNT or ARR (p=0.073). The inclusion of biased graphs was misinterpreted as an actual data difference (RRR: 61.6% vs. ARR: 14.0%, p=0.000000).
ConclusionsThis study demonstrated that Latin American cardiologists could misinterpret statistical data when information of clinical trials is presented in terms of RRR. We emphasize the need to enhance cardiologists’ training in quantitative techniques, to improve medical care decision-making.
Publicaciones previas demostraron que la presentación de los beneficios terapéuticos en términos de reducción del riesgo relativo (RRR) en lugar de en términos de reducción del riesgo absoluto (RRA) o número necesario a tratar (NNT) favorecería la percepción de que los resultados son más efectivos. El objetivo fue realizar una evaluación en una muestra de cardiólogos para determinar cómo afecta a la decisión médica la manera como se presentan los riesgos y beneficios de métodos diagnósticos y terapéuticos.
MétodosCuatrocientos seis cardiólogos latinoamericanos respondieron a un cuestionario que contenía los resultados de ensayos clínicos como RRR, RRA y/o NNT, asociados o no a gráficos sesgados.
ResultadosLa toma de decisión de los cardiólogos fue diferente al comparar los beneficios terapéuticos presentados como RRR (62.2%) vs. RRA (40.4%) (p=0.000000), y como RRR vs. NNT (44.4%) (p=0.000000); sin embargo, fue similar cuando la información se presentó como NNT o RRA (p=0.073). La inclusión de gráficos sesgados fue malinterpretada como una diferencia real (RRR: 61.6% vs. RRA: 14.0%, p=0.000000).
ConclusionesSe demostró que los cardiólogos latinoamericanos podrían confundir resultados estadísticos cuando la información de los ensayos clínicos se presenta como RRR. Hacemos hincapié en la necesidad de entrenar a los cardiólogos en las técnicas cuantitativas para mejorar la toma de decisiones.
Misinterpretation of numerical data can be induced by the form in which the risks and benefits of clinical trials are reported. These different interpretations influence not only the opinion of medical students and patients, but also the physician care decision-taking process.1–3 Previous research demonstrated that presenting treatment benefits in terms of relative risk reduction (RRR) rather than in terms of absolute risk reduction (ARR) or number needed to treat (NNT) might favor the perception of outcome effectiveness.4 Clinical experience and medical training do not seem to be sufficient to protect physicians against the tendency to interpret data erroneously. A recent meta-analysis by Covey5 has shown that these framing effects are heterogeneous and seem to be moderated by key differences between the methodologies used in each study. Several authors have speculated that differences in the presentation of outcomes may lead to discrepancies in decision-making and treatments, but previous studies have reported a wide variety of results. Therefore, considering the importance of the problem in cardiological medical practice, a cognitive evaluation study was performed in a group of Latin American cardiologists to assess how the manner in which risks and benefits of screening methods and treatments are presented can affect medical care decision-making.
MethodsThis study was performed in a non-randomized sample of 406 Spanish speaking cardiologists included in the Interamerican Society of Cardiology register, proportionally representing the regional distribution of Cardiology specialists. They underwent via e-mail an anonymous and voluntary assessment on the interpretation of results of several medical investigations. The questionnaire was in Spanish and consisted of nine questions with multiple choice answers. The questions presented the summary of clinical results from different controlled trials, showing the data as relative risk reduction (RRR), absolute risk reduction (ARR) or number needed to treat (NNT), with or without associated graphs. Questions were randomly ordered and were associated as follows:
Group 1: a question expressed with RRR and NNT, the same question with RRR, the same question with ARR and the same question with NNT.
Group 2: a second question expressed with AAR, the same question with RRR and the same question with NNT.
Group 3: finally, a third question presented with RRR and the same question with ARR, together with corresponding graphic representations. In this case, the relative proportion of the graphs was biased to favor the choice of RRR. The abridged questions are shown in Table 1. To compare with other studies, test questions were formulated according to three previous studies on the same subject.4,6,7 The wording of these questions intended to overcome previous study problems, mainly through the addition of examples containing NNT, the combination of RRR+NNT and supporting graphs. At the end of each example, the physician was asked to consider these results as sufficient evidence favoring the proposed treatment. The multiple choice answer to each question could be answered on some occasions with “yes, no, or I do not have sufficient information”, and in others with “very useful, moderately or slightly useful, not useful, or I do not have sufficient information”.
Table 1.Exact wording of questions.4,6,7
Absolute risk reduction (ARR) Relative risk reduction (RRR) Number needed to treat (NNT) NNT+RRR G1. In a controlled clinical trial, 1900 men receiving a drug to lower their cholesterol level were compared with 1906 men receiving placebo. After 7 years of follow-up, the rate of cardiovascular death was 2% for the placebo group and 1.6% for the group receiving the drug, representing an absolute risk reduction of 0.4% (statistically significant). G1. A controlled clinical trial of almost 4.000 men with hypercholesterolemia, treated with a drug or placebo revealed a relative risk reduction of 20% in cardiovascular death rate over 7 years. This difference in favor of the drug was statistically significant. G1. In a controlled clinical trial, 3806 patients were divided into two groups to compare a drug vs. placebo. The rate of cardiovascular death evidenced a statistically significant difference favoring the drug, with 250 patients who needed to be treated to prevent one death over 7 years. G1. Approximately 4000 persons participated in a controlled clinical study to evaluate a drug vs. placebo in hypercholesterolemia. Results over 7 years favored drug treatment, with a relative decrease in the rate of cardiovascular death of 20% (statistically significant), and 250 patients who needed to be treated to avoid one event. G2. A medical intervention resulted in 1.4% absolute reduction in the 5-year incidence of fatal and non-fatal myocardial infarction (2.5% vs. 3.9%) (the study was a controlled clinical trial and the difference was statistically significant). G2. A medical intervention reported 36% relative risk reduction of fatal and non-fatal myocardial infarction over 5 years (it was a controlled clinical trial and the difference was statistically significant). G2. A certain medical intervention requires treating 72 persons during 5 years to prevent one fatal or non-fatal myocardial infarction (the study was a controlled clinical trial and the difference was statistically significant). G3. The use of a certain non-invasive diagnostic method for population screening provided the following survival results G3. The implementation of a certain non-invasive diagnostic method to screen a population gave the following mortality results in late follow-up
The analysis of results was performed comparing the rate of favorable choice for RRR presentations vs. ARR and NNT. The study hypothesis postulated that respondent cardiologists would choose more easily RRR rather than ARR or NNT as evidence favoring a treatment. Moreover, questions from groups 1 and 2 were conjointly analyzed. Thus, the total number of answers was 812 (406 respondents with 2 questions each) in the case of ARR, 812 (406 respondents with 2 questions each) in the case of RRR and 1218 (406 respondents with 3 questions each) in the case of NNT and RRR+NNT (2 questions in group 1 and one question in group 2).
Groups were compared using χ2. The odds ratio logarithm (log-OR) with its 95% confidence interval (CI95%) was calculated to compare the study results with those previously published with the same format.
ResultsFig. 1 shows the participating countries and the proportion of respondent cardiologists is compared to the total number of cardiologists in each country, except for Brazil, as the proportion of Spanish speaking physicians is negligible in this country. The same graph shows the proportional distribution of cardiologists in the region.
In the conjoint analysis of group 1 and 2 questions, 62.2% (505/812) of the respondents accepted the proposals as valid when the data were presented as RRR, whereas only 40.4% (328/812) accepted it when the results were presented as ARR. On the other hand, 44.4% (541/1218) cardiologists validated the proposals when the data were expressed as NNT (p=0.000000). The comparison between answers when the propositions were made with RRR and ARR was log-OR=0.89, CI95% 0.683–1.092, p=0.000000, and the comparison of NNT vs. ARR was log-OR=0.17, CI95% −0.020 to 0.351, p=0.073, indicating that the respondent cardiologists managed the statistical evidence in the same manner both with ARR as with its equivalent NNT. Finally, the comparison between RRR and NNT was log-OR=0.72, CI95% 0.536–0.908, p=0.000000. In group 3, where questions were presented with biased bar charts, 61.6% (250/406) cardiologists accepted as recommendable the proposed screening method when the data were presented in the first place as RRR; however, when the same problem was presented as ARR, only 14.0% (57/406) cardiologists accepted the method as useful (log-OR=2.28, CI95% 1.926–2.643, p=0.000000) (Fig. 2).
Discussion
Studies on cognitive psychology demonstrate that the possible interpretation of quantitative evidence is deeply affected by the way in which it is presented.8,9 Thus, the interpretation of numerical information may differ depending on the data presentation format. For example, a 25% RRR can be generated both from the difference in event rates between two groups of 40% and 30% as well as from the difference between two groups of 3% and 4%; which in the first case represents an absolute difference of 10% and in the second case only 1%. If the data were only presented as RRR, the tendency would be to accept these differences as having clinical relevance, neglecting or ignoring the real event rate reduction. Although medical journals require in general the publication of the real event rates in terms of ARR and/or NNT, there is no uniform policy in the presentation of clinical study results, either among researchers or in the pharmaceutical industry marketing.6,10 The “illusion of benefit” produced when results are presented as RRR has strong implications in the acceptance and adoption of innovations. In our work, although the respondents must already be trained to identify different measurement presentations of effect magnitude, faults were found in the interpretations of the results. When the benefits were presented in terms of RRR, more than 60% cardiologists accepted as valid evidence which should have been rejected or at least questioned. This percentage was significantly reduced both in the presentation of data as ARR or NNT. Graphics intentionally biased, generated even more significant confusion in the study. The magnification of relative proportions in the bar graphs was easily confused with real data difference.
It is essential to train students and physicians in the quantitative analysis of statistical techniques in order to correct these issues and improve decision-taking. The results of other studies as well as ours, demonstrate that the current learning process of these methods is apparently inadequate and unsatisfactory.11–13 The meta-analysis performed by Covey5 concluded that physicians as well as students and public in general may be equally influenced by the manner in which results are presented; in other words, medical training and clinical experience are not enough to immunize subjects against the tendency to interpret data differently when it is presented in terms of RRR, ARR or NNT. This difficulty to acquire the adequate knowledge in mathematics and statistics to interpret health decision making has been thoroughly studied under the name of numeracy.14,15 The importance of this concept is based on the fact that the training and comprehension of statistical tools might not be within the reach of many health-care decision takers, and particularly the patients who might wish to make their own health decisions.16
Fourteen former studies analyzing the effects of presenting treatment benefits or therapeutic risks in different formats to physicians belonging to several fields of medicine, have reported a wide dispersion in the results.4–6,17–27 Grouped together, the weighted log-OR was 1.19 when comparing RRR vs. ARR and 1.08 in the comparison of RRR vs. NNT, which would indicate a higher tendency to accept a result as valid when it is presented in terms of relative risk. Although these overall findings differ slightly from those observed in our work, there were minimal differences when the individual results of Lacy18 and Ward23 for RRR vs. ARR and of Bucher,20 Nikolajevic22 and Nexoe25,26 for RRR vs. NNT were considered. Specifically, in a study performed by cardiologists in Spain, Brotons et al.24 reported that presenting clinical trials results in the form of RRR, as compared with presenting results in terms of ARR or NNT, led to overestimation of the efficacy of interventions, without influencing the likelihood of prescribing a given drug therapy.
ConclusionsThe assessment of a Latin-American sample of cardiologists showed a different decision-taking in the comparison of the therapeutic benefits observed in a series of clinical trials when the presentation was done in terms of relative risks instead of absolute risks. There was a tendency to favor outcomes when the data was presented as RRR. The inclusion of biased graphics was also interpreted as real data difference. This study demonstrated the risk of statistical data misinterpretation and the need to improve cardiologists’ training in quantitative techniques, in order to ameliorate the care decision-making process.
FundingNo endorsement of any kind received to conduct this study/article.
Conflict of interestThe authors declare no conflict of interest.