Elsevier

Health Policy

Volume 77, Issue 3, August 2006, Pages 352-367
Health Policy

“Yes”, “No” or “Yes, but”? Multinomial modelling of NICE decision-making

https://doi.org/10.1016/j.healthpol.2005.08.008Get rights and content

Abstract

The National Institute for Health and Clinical Excellence (NICE) issues mandatory guidance on health technologies to the UK NHS, based on clinical evidence, cost-effectiveness and other considerations. However, the exact factors considered, their relative importance and tradeoffs between them are not made explicit. Previous research modelled NICE decisions as a binary choice (accept/reject) dependent on cost-effectiveness, amongst other variables. This paper proposes and tests an alternative model of decision-making that may better represent the “yes, but…” nature of many NICE decisions. Decisions were categorised as “recommended for routine use”, “recommended for restricted use” or “not recommended”. The NICE appraisal process was modelled as a single decision between the three categories. Multinomial logistic regression techniques were used to evaluate the impact of: quantity/quality of clinical evidence; cost-effectiveness; decision date; existence of alternative treatments; budget impact; technology type. Results suggest that interventions supported by more randomised trials are more likely to be recommended and endorsed for routine use. Higher cost-effectiveness ratios increased the likelihood of interventions being rejected rather than recommended for restricted use but did not significantly affect the decision between routine and restricted use. Pharmaceuticals, interventions appraised early in the NICE programme and those with more systematic reviews were also less likely to be rejected, while patient group submissions made a recommendation for routine rather than restricted use more likely. The presence of factors affecting the decision between routine and restricted use but not that between routine use and rejection suggests that modelling these three outcomes reflects NICE decision-making more closely than binary-choice analyses.

Introduction

The National Institute for Health and Clinical Excellence (NICE) was established in 1999 to appraise the clinical benefits and the costs of health care interventions and to make recommendations [1]. Since it is mandatory for health professionals to take NICE guidance “fully into account when exercising their clinical judgement” [2], NICE decisions have important implications for NHS resource allocation [3], patients’ access to new technologies [4] and pharmaceutical industry sales and profits [4]. By January 2005, NICE had published 87 guidance documents.1

NICE statements suggest that various factors are taken into account in their decision-making process including: the broad clinical priorities of health ministers; the degree of patients’ clinical need; the broad balance of benefits and costs; the effective use of available resources; any guidance from health ministers on the resources likely to be available and on such matters as they see fit [5]. Thus, although cost-effectiveness evidence is widely considered to be central to NICE recommendations, it is not the only factor influencing decisions. However, precisely which factors are considered, their relative importance and the tradeoffs that NICE is prepared to make between them is not explicitly stated.

Devlin and Parkin investigated these questions using a binary choice model to estimate NICE's threshold cost-effectiveness ratio (CER) and the tradeoffs between cost-effectiveness and other variables based on NICE decisions up to May 2002 [6]. Six explanatory variables were considered: cost per quality-adjusted life-year (QALY) gained (CQG) or cost per life-year gained (CLYG); uncertainty surrounding the CER; net NHS costs; burden of disease; availability of alternative treatments; specific factors mentioned by NICE. This analysis suggested that NICE has a probability-based “threshold” in the region of £ 30,000–45,000 per QALY. In addition to cost-effectiveness, uncertainty around the CER and burden of disease appeared to contribute to NICE decisions. However, given the small number of decisions reporting CQG before May 2002, the cost-effectiveness threshold was estimated by pooling observations on CLYG and CQG, even though different types of CER may not be treated in the same way in practice.

This paper adds to this literature by proposing and testing an alternative model of NICE decision-making. While previous research characterised all NICE decisions as either “yes” or “no” [6], NICE guidance is (as Raftery points out [3]) frequently “yes, but…”, in that a technology may be recommended, but only for certain types of patients. Such restrictions on NICE endorsement narrow the group of patients to whom the treatment can be given beyond the limits of the product's license.

Distinguishing between NICE recommendations for restricted rather than routine use and elucidating the factors determining which type of recommendation is made is clearly relevant to all stakeholders. From the perspective of patient groups, a recommendation for restricted rather than routine use limits members’ access to new treatments. From the Government's perspective, the distinction between routine use, restricted use and rejection has budgetary and planning implications. For the pharmaceutical industry, restricted recommendations may limit sales and profits.

In addition to the importance to stakeholders, the distinction between restricted and routine use has implications for the modelling of NICE decision-making. Whilst Devlin and Parkin considered recommendations for restricted and routine use to be identical, the weight placed on various types of evidence or the criteria applied to the evidence may differ between these two types of recommendation; if this is the case, differentiating between routine and restricted use may yield a better model of decision-making.

The aim of this paper is to gain additional insight into the determinants of NICE decisions by extending the analysis in three ways. First, we propose and test a new model of HTA decision-making, capturing the differences between the three types of NICE recommendation noted above. Secondly, we investigate the role of non-economic variables, such as the quantity and quality of clinical effectiveness evidence, the type of technology being evaluated and the presence/absence of a patient group submission. Thirdly, given the availability of more observations, we are able to test additional hypotheses, including the possibility that there have been changes over time in the information NICE uses or the criteria it employs in making its decisions.

Section snippets

Model of NICE decision-making

A model of the decision-making process was developed to identify the determinants of the three possible decision outcomes (recommended for routine use, recommended for restricted use and not recommended).

This model characterises the decision-making process as a simple, single-step decision between the three outcomes, where each outcome is considered to be qualitatively different. No ranking is assumed between the outcomes: to do so would require an assumption regarding their relative

Data extraction

A team of analysts extracted the variables shown in Table 1 from the full guidance documents and, where necessary, the HTA reports and appeal decision documents for each appraisal. All 73 NICE appraisals published before 31 December 2003 were initially included in the analysis, including seven reviews. Appraisals considering several different treatments were subdivided if variables such as the outcome, level of evidence or CER differed between treatments. In these cases, the separate

Univariate analysis

Of the 73 appraisals published within the timeframe of this study, 72 were included in the analysis, including seven reviews. Appraisal No. 6 (proton pump inhibitors for dyspepsia) [8] was excluded since the HTA report contained insufficient information on clinical evidence. Fifteen appraisal topics were subdivided into 2–4 separate recommendations, each concentrating on a specific product or type of intervention. A total of 94 recommendations were considered in the analysis, of which 20 (21%)

Discussion

The univariate and logistic regression analyses suggest that several of the factors considered play an important role in NICE decision-making. The number of RCTs supporting an intervention was one of the most important determinants of favourable NICE appraisals. This variable was particularly influential in the decision between recommending an intervention for routine use and recommending against any use. As this variable was statistically significant in almost all of the analyses, we can be

Acknowledgements

The authors would like to thank David Parkin and Aran Ratcliffe for advice on the methodology and statistical issues, as well as Juliet Warner and Simon Howard for proofreading the document and Manpreet Sidhu and Jim Swift for assisting with data extraction.

References (21)

  • A.R. Jadad et al.

    Assessing the quality of reports of randomized clinical trials: is blinding necessary?

    Controlled Clinical Trials

    (1996)
  • National Institute for Clinical Excellence. Framework Document. London:...
  • National Institute for Clinical Excellence. A guide to NICE. London: NICE, April...
  • J. Raftery

    NICE: faster access to modern treatments? Analysis of guidance on health technologies

    British Medical Journal

    (2001)
  • Howard S, Harrison L. NICE guidance implementation tracking: data sources, methodology and results. A report...
  • National Institute for Clinical Excellence. Guide to the technology appraisal process. London: National Health Service,...
  • N. Devlin et al.

    Does NICE have a cost-effectiveness threshold and what other factors influence its decisions? A binary choice analysis

    Health Economics

    (2004)
  • Stata Corporation. Stata Reference Manual, release 8. College Station, TX: Stata Corporation,...
  • National Institute for Clinical Excellence. Guidance on the use of proton pump inhibitors in the treatment of...
  • National Institute for Clinical Excellence. Guidance on the use of computerised cognitive behavioural therapy for...
There are more references available in the full text version of this article.

Cited by (110)

  • Are Recently Evaluated Drugs More Likely to Receive Positive Reimbursement Recommendations in South Korea? 11-year Experience of the South Korean Positive List System

    2020, Clinical Therapeutics
    Citation Excerpt :

    Because economic evaluation is required for only drugs whose comparative clinical benefit is improved and priced higher than alternatives, 77% of the submissions did not conduct economic evaluations, and thus the impact of ICER on our analysis would not be significant. The findings regarding the significance of the budget impact on reimbursement recommendations are mixed; some studies suggested that the budget impact was not a significant predictor,4,5,22,23 and some found the contrary, with a higher budget impact entailing lower odds of receiving positive recommendations.7 Our analysis suggested that drugs with high budget impact were significantly associated with positive recommendations, which seems counterintuitive to our finding that cheaper drugs are more likely to receive positive recommendations.

  • Ranking the Criteria Used in the Appraisal of Drugs for Reimbursement: A Stated Preferences Elicitation With Health Technology Assessment Stakeholders Across Jurisdictional Contexts

    2020, Value in Health
    Citation Excerpt :

    Cost-effectiveness was a significant influence in all countries except Wales.28 A few other factors found to have significant impact on appraisal were disease type,16,21,23 process factors,24 submissions from patient groups,26 and the recommendations made elsewhere.51 Our experimental results further corroborate revealed preference results in terms of the non-influence of budget impact considerations.15

View all citing articles on Scopus
View full text