This study aimed to validate the IberScore cardiovascular risk model in a population attended in the primary care setting.
MethodsA cohort of patients with no history of cardiovascular disease visited in a primary care center during the years 2008 and/or 2009 and followed up until 2018 was selected.
Cardiovascular risk was calculated with the IberScore formula for all the subjects of the cohort and the model was calibrated, graphically represented by risk deciles the proportion of expected events and proportion of observed events at 10-year follow-up, stratified by sex. The area under the ROC curve was calculated to assess the discrimination of the model.
ResultsA total of 10,085 patients visited during the years 2008 and/or 2009 were included in the study. Men showed a mean 10-year risk of suffering a fatal or non-fatal cardiovascular events according to IberScore of 17.07% (SD 20.13), with a mean estimated vascular age of more than 4 years higher than the biological age; while women had a mean 10-year risk of 7.91% (SD 9.03), with an estimated vascular age of more than 2 years above the biological age.
The area under the ROC curve showed a discrimination index of the model of 0.86 (95% CI 0.84–0.88) in men and 0.82 (95% CI 0.79–0.85) in women.
ConclusionIberScore model discriminates well in the population attended in primary care but the model overestimates the risk.
Validar el modelo de riesgo cardiovascular IberScore en una población atendida en atención primaria.
MétodosSe seleccionó una cohorte de pacientes sin antecedentes de enfermedad cardiovascular visitados en un centro de atención primaria durante los años 2008 y/o 2009 y seguidos hasta el año 2018.
El riesgo cardiovascular se calculó con la fórmula IberScore para todos los sujetos de la cohorte y se calibró el modelo, representado gráficamente por la proporción de eventos esperados y la proporción de eventos observados a los 10años de seguimiento por deciles de riesgo, estratificando por sexo. Se calculó el área bajo la curva ROC para evaluar la discriminación del modelo.
ResultadosSe incluyeron en el estudio un total de 10.085 pacientes visitados durante los años 2008 y/o 2009. Los hombres presentaron un riesgo medio a los 10años de sufrir un evento cardiovascular fatal o no fatal según IberScore del 17,07% (DE: 20,13), con una edad vascular media estimada superior en más de 4años a su edad biológica. Las mujeres tenían un riesgo medio a 10años del 7,91% (DE: 9,03), con una edad vascular estimada de más de 2años por encima de su edad biológica.
El área bajo la curva ROC mostró un índice de discriminación del modelo de 0,86 (IC95%: 0,84-0,88) en hombres y de 0,82 (IC95%: 0,79-0,85) en mujeres.
ConclusiónEl modelo IberScore discrimina bien en la población atendida en atención primaria pero sobreestima el riesgo.
The European guidelines for the management of vascular risk factors and cardiovascular prevention recommend the use of the European SCORE model (the most recent, SCORE2 and SCORE2 OP) to estimate the risk and establish preventive priorities (modification of lifestyles, initiation of pharmacological treatment)1–3; although only from the age of 40 it is possible to estimate the risk with these scales. Identifying people at high cardiovascular risk is challenging, specially at a younger age. The estimated ten-year risk with these scales in young individuals is low or moderate even in the presence of several cardiovascular risk factors. This is relevant, as it has been shown how CVR factors are significantly involved in the prevalence and progression of subclinical arteriosclerosis in individuals under 50 years of age.4 In this regard, the latest European Guidelines for Cardiovascular Disease Prevention already recommend including and assessing the calculation of lifetime risk or vascular age.1 For subjects under 40 years of age, several models have been developed that estimate lifetime cardiovascular risk (CVR), such as the QRISK-LTR equation, which estimates lifetime CVR between 30 and 95 years of age5 and the “IBERLIFERISK” model,6 recently recalibrated,7 developed in a large sample of the Spanish working population with a mean age of 35 years, which estimates the lifetime risk from 18 to 75 years of age. However, as acknowledged by the authors, there is no evidence regarding the present threshold for considering a patient at high risk according to the lifetime risk.6 Also, the evidence of the benefit of applying these lifetime models in the long term is limited and highly variable depending on the model applied. The percentage of patients who were reclassified as high-risk patients in the long-term calculation was 1.61% (95% CI 1.55–1.66) for the QRISK-LTR,5 while for the model developed by the American College of Cardiology/American Heart Association (ACC/AHA) for subjects over 50 years was 27.1% (95% CI 27.11–27.70).8
Taking into account the above-mentioned points, our group has also developed a model to estimate the 10-year risk in a large sample of the working population (almost one million workers included with a mean age of 34 years), which improves risk prediction for fatal and non-fatal events compared to the European SCORE model.9 The aim of this paper is to validate this model in a population attended in the primary care setting.
MethodsThe ICARIA10,11 study evaluates the cardiovascular health of the Spanish working population by analyzing a cohort of workers who underwent an occupational health examination at the Ibermutuamur Prevention Society (now Ibermutua) during the years 2004–2018. The methodology of the ICARIA study has already been described previously.10,11
The IberScore Project9,11 included in the ICARIA study,10,11 systematically evaluates the cardiovascular risk factors of the study cohort and aims to develop and validate a mathematical model for cardiovascular risk prediction for fatal and non-fatal events at 10 years for the working population.
In the IberScore equation, coronary disease is classified as a non-fatal cardiovascular event, codes 410–414 according to the International Classification of Diseases, 9th revision, clinical modification (ICD-9-CM) in 2002, heart failure (ICD-9-CM 428), cerebrovascular diseases (ICD-9-CM 431–438) and peripheral vascular disease (ICD-9-CM 440–444). Events that were not of atherosclerotic origin, corresponding according to the ICD-9-CM to codes 432.1, 437.2, 437.3, 437.7, 442, 443.0 and 443.1, were not considered a cardiovascular event. Fatal cardiovascular events include, in addition to non-fatal events codes, different cardiovascular conditions (coded according to the ICD-10-CM as I20–I26, I46, I50, I60–I89, I95, I97–I99), hypertension (I10–I13, I15) and arrhythmias (I44, I45, I47–I49).
The IberScore model computes an estimated risk exposure index based on exposure to risk factors as a weighted average of standardized abnormal values stratifying by age and sex. The model assumes that there is an association between cardiovascular disease and cardiovascular aging, a latent variable whose values may differ from the patient's chronological age. Cardiovascular age might be higher than chronological age when the cardiovascular risk was greater than the baseline risk expected for the patient's age and sex. Exposure to cardiovascular risk factors might influence said deviation. Following these assumptions, the model estimates a cardiovascular biological age from the patient's chronological age modified by the exposure index (to the cardiovascular risk factors), independently by sex. The model assumes that there is a correlation between the estimated cardiovascular biological age and the cardiovascular aging (or cardiovascular health status).9 In the internal validation of the equation carried out on a sample of 366,008 workers, the equation showed good discrimination (AUC=0.8) and correct calibration.9
The present study aims to assess an external validation of the IberScore equation in a cohort of patients with no history of cardiovascular disease visited in a primary care center during the years 2008 and/or 2009 and followed up until 2018. Baseline data were recorded during clinical visits made to medical and/or nursing professionals during the study period. The blood samples were processed in the center's reference laboratory, while the rest of the measurements were carried out in the primary care center's consultations.
Non-fatal cardiovascular events included those that appeared in the patient's medical history or reported by the patients themselves, providing the corresponding hospital discharge report to verify the diagnosis after the date of inclusion in the study. Mortality data were obtained from the Spanish National Institute of Statistics.
Statistical analysisThe missing values in the main variables that integrate the IberScore equation were iteratively and stochastically imputed using the distribution of the observed data to reflect the uncertainty of the real values. For this, a data augmentation algorithm is applied in an iterative way of the Monte Carlo method of Markov chains,12 combining the effect of a total of 20 imputations following Rubin's rules.13
Once the cardiovascular risk was calculated with the IberScore formula for all the subjects of the cohort and imputation techniques were used for replacing the missing values, descriptive analysis of the data was carried out and then the model was calibrated, graphically represented by risk deciles the proportion of expected events and proportion of observed events at 10-year follow-up, stratified by sex. The percentage of observed events was calculated using the non-parametric estimator of the accumulated risk incidence of Nelson-Aalen,14 which assumes in the calculation the subjects with censored data by taking into account the presence of competitive events (deaths from any other cause). The area under the ROC curve was calculated to assess the discrimination of the model.
Statistical analysis was performed using the STATA/MP 14.2 program.
ResultsCharacteristics of the study populationA total of 10,085 patients visited during the years 2008 and/or 2009 were included in the study. 55.3% of the subjects were women and the overall mean age was 41.6 (SD 12.96) years. 0.18% of the participants had a diagnosis of type 1 diabetes mellitus and 3.3% of type 2, 9.3% of hypertension and 9.4% of dyslipidemia (Table 1). 300 (2.98%) cardiovascular events (fatal and non-fatal) and 207 (2.05%) competitive events were recorded. In Table 1 it is shown the original mean values within the total number of valid cases, and the imputed mean values for each variable. The two variables with more missing values were glycaemia and cHDL.
Clinical characteristics of the study sample.
| Menn=4511 (44.7%) | Womenn=5574 (55.3%) | Totaln=10,085 | |
|---|---|---|---|
| Age, mean (SD) | 40.95 (14.65) | 42.06 (13.07) | 41.56 (12.96) |
| Diabetes mellitus 1 or 2, n (%) | 203 (4.50%) | 149 (2.67%) | 352 (3.49%) |
| Hypertension, n (%) | 630 (13.97%) | 696 (12.49%) | 1.326 (13.15%) |
| Dyslipidaemia, n (%) | 541 (11.59%) | 654 (11.73%) | 1.195 (11.85%) |
| Smoker (active), n (%) | 865 (36.84%) | 843 (27.10%) | 1.708 (16.94%) |
| BMI [n] | [n=1691] | [n=2319] | [n=4010] |
| Mean (SD) | 27.36 (4.55) | 25.97 (5.30) | 26.55 (5.04) |
| Imputed, mean (SD) | 26.90 (3.06) | 25.57 (3.72) | 26.16 (3.50) |
| Total cholesterol [n] | [n=2152] | [n=3129] | [n=5281] |
| Mean (SD) | 199.02 (40.87) | 199.28 (39.84) | 199.17 (40.26) |
| Imputed, mean (SD) | 193.37 (30.58) | 192.63 (33.13) | 192.96 (32.01) |
| HDL [n] | [n=1528] | [n=2124] | [n=3652] |
| Mean (SD) | 47.92 (11.15) | 59.06 (13.99) | 54.40 (14.00) |
| Imputed, mean (DE) | 47.08 (7.21) | 58.45 (9.34) | 53.36 (10.17) |
| Glycemia [n] | [n=864] | [n=1283] | [n=2147] |
| Mean (SD) | 96.45 (29.16) | 89.43 (21.15) | 92.25 (24.92) |
| Imputed, mean (DE) | 90.37 (16.46) | 86.95 (12.58) | 88.48 (14.54) |
| Non-fatal CV events, n (%) | 170 (3.77%) | 79 (1.42%) | 249 (2.47%) |
| Fatal CV events, n (%) | 28 (0.62%) | 23 (0.41%) | 51 (0.51%) |
| Deaths from other causes n (%) | 116 (2.57%) | 91 (1.63%) | 207 (2.05%) |
SD: standard deviation; BMI: body mass index; HDL: high density lipoprotein; CV: cardiovascular.
The total follow-up of cardiovascular events was 45,562.67 and 57,941.77 person-years, for men and women, respectively.
Men showed a mean risk of suffering a fatal or non-fatal cardiovascular events according to IberScore of 17.07% (SD 20.13), with a mean estimated vascular age of more than 4 years higher than the biological age; while women had a mean risk of 7.91% (SD 9.03), with an estimated vascular age of more than 2 years above the biological age.
The area under the ROC curve showed a discrimination index of the model of 0.86 (95% CI 0.84–0.88) in men (Fig. 1) and 0.82 (95% CI 0.79–0.85) in women (Fig. 2). The calibration is presented plotting predicted versus observed risks, taking into account death from causes other than cardiovascular causes as competing events and stratified by risk deciles for men (Fig. 3) and for women (Fig. 4).
Discussion
The present study shows the results of the external validation of the IberScore equation in a sample of the population visited in the primary care setting, concluding that the aforementioned model discriminates well, with ROC and Harrel's C indices greater than 0.80 for men and women (values greater than 0.7 indicates correct discrimination). However, this model overestimates the risk of events, more markedly in women. Therefore, it would be necessary to recalibrate the IberScore equation to enable its use in primary care.
This overestimation was expected, since significant differences were found in the characteristics of the referral population (mostly young, mean age 35 years, and “healthy” active workers: workers on sick leave were excluded from occupational medical examinations) and the external population (attended in primary care, mean age 41.5 years, which includes subjects from age above working age range). In this regard, the effects of the risk factors or their distributions will be different. Other reasons could be that there were some potential predictors are not included or to overfitting of the model in the derivation sample, all of which can contribute to an inadequate calibration.
Regarding the high percentage of missing values, although multiple imputation has been applied, it may happen that the distributions, after imputation, do not correctly reflect the real distributions in the population. Although in general, the imputed distributions will resemble those of the non-missing data, the model could not be validated for the population group without missing values.
On the other hand, it is common for a model derived in one population, even having a good discrimination, do not calibrate well in an external population. This was the case of the original Framingham tables, which overestimated the real risk in the Spanish population by more than 2.5 times15; after calibrating the equation, replacing the prevalence of risk factors and the incidence rates of coronary events for those of Spain,16 the equation was validated for the Spanish population.17 This is also the case with the tables of the European SCORE model; the Third European Task Force recommended adapting the model taking into account the level of risk and mortality rates in each country. In that case the calibration was carried out using the Spanish mortality rates and the risk factors of the MONICA-Catalonia cohort, assuming, among other aspects, that the prevalence of risk factors in that cohort could be extrapolated to all of Spain.18
Regarding strengths, the validation test is carried out in a representative sample of the population attended in primary care as it has been used in previous studies9 and with a high rate of long-term follow-up. The IberScore model was derived from a sample of more than 600,000 workers, and it was validated in a sample of more than 360,000 workers, other than those included in the referral cohort. Both cohorts had a 10-year follow-up, showing greater precision in the estimates compared to the SCORE model.19
To overcome the overestimation of the model, the easiest way to do it would be to recalibrate the IberScore model. If still the recalibration do not fully work, the model parameters will have to be re-estimated in a new population; that is, to derive a new model for primary care but using the same specifications of the IberScore model.
In conclusion, the IberScore model discriminates well in the population attended in primary care but the model overestimates the risk.
FundingThe author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: this work was supported by Instituto de Salud Carlos III and Fondo Europeo de Desarrollo Regional-FEDER (grant number PI18/01809), in 2018.
Conflict of interestsThe author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.
This study is part of a larger research project in cardiovascular risk, the ICARIA (Ibermutua Cardiovascular Risk Assessment) Project, which has been active since 2004. A broad team of general practitioners, specialists and nurses contributed to the collection of demographic and clinical data. The authors wish to thank all the members of the ICARIA Study Group.