- Research article
- Open Access
Description of the use of multicriteria to support pricing and reimbursement decisions by European health technology assessment bodies
BMC Health Services Research volume 21, Article number: 814 (2021)
Heterogeneity in drug access throughout Europe may be influenced by differences in drug-assessment strategies. The EUnetHTA’s assessment core model (EUnetHTA-core) and the EVIDEM’s multicriteria framework are reference methodologies in this context, the latter including a wider compromise between non-contextual and contextual criteria. Compliance of 37 European Health Technology Assessment bodies (HTAb) with EUnetHTA-core has been reported, but the use of EVIDEM by this HTAb is still unknown.
To describe the uptake and use of multicriteria approaches to evaluate drug value by European HTAb using EVIDEM as reference framework, a multicriteria framework was obtained based on EVIDEM model. The criteria used for drug appraisal by HTAb was extracted from the EUnetHTA report, and completed through search of websites, publications and HTAb reports. Use of EVIDEM assessment model in 37 European HTAb has been described semi-quantitatively and summarized using an alignment heatmap.
Aligned, medium or misaligned profiles were seen for 24,3%, 51,4% and 24,3% of HTAb when matching to EVIDEM dimensions and criteria was considered. HTAb with explicit responsibilities in providing specific advice on reimbursement showed more aligned profiles on contextual and non-contextual dimensions.
EUnetHTA’s core model is limited in assessing medicines while EVIDEM’s framework provides contextual dimension used by some HTAb in Europe that can be escalated to other agencies. Most of the 37 European HTAb have room to broaden their contextual assessment tools, especially when social and medical perception of need requires to be explicit to support payer’s decision on reimbursement.
One of the major cost drivers in the European healthcare systems is the pharmaceutical ‘innovation’; even considered more relevant than demographics . At the same time, it is also recognized as one of the main contributors to the improvement of the population health status .
According to the most recent study from the Organization for Economic Co-operation and Development (OECD) , pharmaceutical expenditure accounts for a percentage that range between 11.4% (UK) and 19.1% (Spain) of total healthcare expenditure across the five largest European drug markets (France, Germany, Italy, Spain, and the UK). Specifically, the oncological and hematological drugs are leading the budget impact related to pharmaceutical innovation. The impact is driven by the expansion of multiple new indications normally based on a molecular definition that restricts the population to be treated and the drug ends up being designated as orphan-like medicines . As estimated by a recent study [5, 6], the healthcare expenditures on cancer in the European Union member states represented roughly 6% of total healthcare expenditures. The steady increase of oncology costs is aligned with the disease increasing incidence, the progressive reduction of mortality as well as high prices, in contrast with the less robust evidence data on outcomes .
A recent study  estimated that 40% of the new orphan drugs authorized in Europe are related to rare neoplastic disorders, and compare to non-oncologic indications, the authorization is received at more advance stages of the clinical development and recognizing a higher potential clinical benefit. From 2009 to 2013, only 35% the 68 oncology indications approved by the European Medicines Agency (EMA) showed a significant prolongation of survival and only 10% showed an improvement in quality of life at the time of market approval. The magnitude of the benefit on overall survival ranged from 1.0 to 5.8 months (median 2.7 months). In the subsequent post marketing period (3,3 years later) there was evidence for extension of life in 7% of the previous authorizations and reported benefit on quality of life in 11% of the cases .
Occasionally, when the drug can cover clinical unmet needs with poor prognosis, the regulators trend to accept less and poorer evidence and include especial approvals, such as conditional approval related to further of adequate risk benefit rate in real world, after commercialization, or approval under exceptional circumstances when this may not be achieved, in order to ensure an earlier access to market. As described recently  the potential benefit of patients’ early access to new medicines in areas of high unmet medical need, and based on initial data only, have relevant implications in terms of medical and economic costs (opportunity costs of using alternative more efficient treatments available for patients). Several initiatives have been developed in Europe to address these challenges of funding premium priced products related to clear medical unmet needs but with limited evidence . New access management models of these drugs have been promoted across Europe recently, especially for advance therapies, orphan drugs and medicines for cancer, and including innovative access schemes as value-based pricing, conditional reimbursement schemes or risk sharing approaches . Despite the smooth increase of these new access schemes, the number of outcome-based solutions is still very limited being the lack of a systematic and harmonized value assessment methodology one of the main limitations .
Beyond the general awareness among healthcare authorities to ensure “value for money”, or the link between price and social or clinical value of the pharmaceutical innovation , the reimbursement process and value assessment of drugs is still an open debate in Europe . Several methods have been developed to assess the value of drugs and set meaningful prices affordable to health-care systems . These methods are normally based on the clinical benefits of the drugs and partially on value-based pricing (e.g. cost-effectiveness analysis). However, there is neither a consensus nor a European harmonization related to drug-pricing systems and, based on a comparative international policy analysis, value-based approaches to determine the prices of innovative products are diverse : including the implicit clinical value of the quality-adjusted life-years (QALYS), mainly used in UK, Sweden or Australia, or the value classification based on innovation scales (used in France, Italy, Germany, Austria, Canada or Japan) . Normally new drugs classified as an innovative medicine are reimbursed at a higher price than the current therapeutic alternatives; although the amount, type and methodology to set the premium is normally veiled by the healthcare authorities .
In Europe, the European Network for Health Technology Assessment (EUnetHTA) was set up in 2006 and includes all EU Member States to provide strategic guidance and policy orientation on the assessment of health technologies (including drugs), by developing policy papers and discussing areas of potential collaboration. During the last decade the network has focused the efforts on the development of common methodologies, piloting and producing joint early dialogues and Health Technology Assessment (HTA) joint assessment reports, as well as developing and maintaining common tools . One of the most relevant tools developed by the network is the HTA Core Model for Rapid Relative Effectiveness Assessment (REA) . The Model is a methodological framework for the collaborative production and sharing of HTA information that defines the content elements to be considered in an HTA and it enables standardized assessment reporting across Europe. Because of the objective of the framework is to share commonly required elements of information, only information that is considered both important and transferable is collected. The model brings a standardized framework that allows a common comparison of the drivers that lead pricing and reimbursement decisions among different European authorities.
HTA Network approach is focused on technical aspects while methods to support alignment of decisions with the compassionate impetus of healthcare systems is lacking . In many countries, healthcare authorities are including a broader approach to assess the pharmaceutical products (especially in therapeutic areas like oncology and rare diseases) . EVIDEM  (Evidence and Value Impact on Decision Making) was developed based on an analysis of the foundations of healthcare systems, the reasoning underlying decisions and fair processes, and has become a reference for multicriteria decision approaches in this setting. It includes the concept of reflective multicriteria assuming decision-makers are guided by a generic interpretative frame rooted in the baseline values of the healthcare systems, drawing on several domains of knowledge including healthcare ethics, evidenced-based medicine, health economics or health technology assessment approaches. A multicriteria analysis provides an effective approach to increase the legitimacy of decisions. Beyond a tool, reflective multicriteria pioneered by EVIDEM is geared to transform the vision of the value of healthcare interventions and how they might contribute to relevant, equitable and sustainable healthcare systems. EVIDEM can be used to compare various healthcare interventions and prioritize its implementation using a performance matrix underpinned in the several dimensions and criteria defined by the framework .
EVIDEM criteria overlap with EUnetHTA-core except for 4 non-contextual and 3 social criteria, which are absent or partially included in the EUnetHTA framework. Inversely, 2 EUnetHTA criteria are absent in the EVIDEM framework (Table 1).
Although multicriteria EVIDEM approach is now applied by several healthcare authorities , especially when the social and medical perception of need requires a more holistic assessment framework to support the payer’s decision, a formal and systematic comparison of EUnetHTA’s and EVIDEM’s methodological frameworks and whether European health technology assessment bodies (HTAb) are aligned with the EVIDEM methodology standards is lacking . Since EUnetHTA and EVIDEM frameworks differ in a substantial number of criteria, it is of interest to know the extent of compliance with EVIDEM framework of HTAs as an additional way to explore potential reasons of assessment discrepancies. Despite the compliance of 37 European HTAb with using the supportive criteria for decision making proposed in the EUnetHTA-core framework has been previously reported , whether these HTAb do also comply with the wider EVIDEM multicriteria is unknown.
Thus, the main aim of this study is to describe the uptake and use of multicriteria approaches to appraise drug value by 37 European HTAb, using EUnetHTA and EVIDEM as reference frameworks.
A quantitative validation of the degree of alignment with the EUnetHTA’s standard framework of 37 European HTAb from 28 countries was done, based on a previous qualitative analysis conducted by the European Commission  and an additional thorough search of websites, publications and reports of HTAb. The criteria used for appraisal by the different HTAb were identified and classified, and the matching with the criteria described in the EVIDEM methodological framework were described semi-quantitatively using a heatmap of alignment.
The items reported included those criteria in the HTA Core Model, namely: Relative Effectiveness Assessment (REA) of pharmaceuticals, EUnetHTA methodological guidelines  and procedure descriptions [26, 27]. Also, criteria related to the types of technologies assessed, the administrative level (national, regional, institutional) and the formal background (legislation, formal agreement, internal guideline) of certain methodological requirements were also used.
An updated version of EVIDEM framework (v.10) was analyzed in order to assess how the dimensions and criteria included in the EUnetHTA methodological framework fitted within the EVIDEM’s methodological framework.
The EVIDEM framework includes 13 non-contextual dimensions and 6 contextual dimensions (Table 1). The non-contextual dimensions (EVIDEM core-model) include normative aspects combined with the description of the technical knowledge available. Contextual dimensions tailor the framework to the context of decision-making.
An HTAb heatmap was developed, where heatmap categories were generated for each EVIDEM’s dimension using as a source the mentioned criteria in the EUnetHTA’s report , webs and reports available from the different HTAb analyzed (supplementary file). The contribution (weight) of each mentioned criterion to the final heatmap’s score by dimension was equal and proportioned to the number of criteria by dimension described in Table 1. Only when the mentioned criteria were not fully aligned with the EVIDEM’s criteria, the mention was weighted by 50% of contribution:
Descriptive statistics (mean, standard deviation, percentiles) were used to summarize the data and 95% confidence interval for each dimension and HTAb Figs. 1 and 2, and conditional formatting was used to automatically color code each cell using Microsoft Excel (Windows Office 365) so that graded colors were used with green coding for highest alignment (100) and red for lowest alignment (0). Values outside the interquartile range were used to assess alignment with the EVIDEM’s model . HTAbs with and average heat score above the 75th percentile were considered “Aligned” with the EVIDEM model, and those below 25th percentile were considered “Misaligned”. The rest were classified as “Medium” in terms of EVIDEM model’s alignment.
Most of the non-contextual criteria of EVIDEM are overlapped with the core model of EUnetHTA, except for the type of prevention benefits, non-medical comparative cost consequences, systematic use of expert consensus and use of clinical guidelines to define state-of-the-art, which are not or partially included on the EUnetHTA’s framework (Table 1). Regarding contextual criteria, the assessment of the system capacity and appropriate use of intervention is the most aligned criteria between both frameworks, followed by the political/historical/cultural context assessment, the mandate and scope of the healthcare system, the special population priorities and equity on access criteria. Other social criteria (stakeholders management, conflict of interest assessment or environmental impact assessment) are not reflected in the EUnetHTA’s framework. A systematic general description of the assessed technology and the request of clarification of the assessment process (guidelines and legislation) are key aspects considered by the EUnetHTA analysis that are not explicitly included in the EVIDEM framework.
Most of the non-contextual dimensions (such as disease severity, size of affected population, unmet needs, comparative effectiveness, comparative safety/tolerability or type of therapeutic benefit) show consistently high rates among the HTAb (mean above 85% and standard deviation below 16%); other non-contextual dimensions (type of preventive benefit, comparative non-medical costs, expert consensus) and relevant contextual dimensions (such as population priorities, common goal, environmental impact, system capacity or political/historical/cultural context) are systematically rated low (Table 2).
All HTAb address consistently the health problem and current use of technology, technical characteristics, clinical effectiveness and safety criteria, which are included in the EUnetHTA core model. Choices on comparator, methodology of comparison, endpoints and methods of evidence search and synthesis, are consistently aligned. On the contrary, non-clinical domains, assessment approaches, methodology, modelling algorithms and data are consistently dis-aligned (Table 3).
None of the local HTAb had high heat scores with regards to the use of contextual criteria (Table 2). Considering alignment to EVIDEM-driven assessment framework, three patterns of HTAs emerged: “Aligned”, “Medium” and “Misaligned” (Table 4).
Nine agencies in Bulgary, Hungary, Italy, Malta, Slovakia, Spain, Sweden and UK showed an “Aligned” profile (average heat score above the 75th percentile) with a consistent alignment on non-contextual dimensions and significantly high alignment scores on political/historical/cultural context, system capacity and appropriate use of the intervention.
Most HTAb (19/37; 51%) showed a “Medium” alignment profile. Alignment rates for non-contextual criteria were mainly high (e.g. patient perceived health and quality of evidence dimensions) in these HTAb, and also other contextual dimensions (such as the mandate and scope of the healthcare system, system capacity and appropriate use of the intervention) were rated high. On the contrary, population priorities and access dimension systematically rated below 50%, except for AEMPS.
In 9/37 (24%) HTAb the profile was considered “Misaligned”, with low scores on alignment (average score below 25th percentile) in dimensions such as patients perceived health methods, cost-consequence analysis (cost of intervention and other medical costs) and quality of the evidence. Considering the non-contextual perspective, the German G-BA and the NIPH in Norway show high scores focused and limited to the technical comparison of alternatives (effectiveness, safety and quality of evidence assessment). From the contextual perspective, all the HTAb of this group rated low on the mandate and scope of the healthcare system, population priorities on access, system capacity, appropriate use of the interventions and political/historical/cultural context.
HTAb with explicit responsibilities in providing specific advice on pricing and reimbursement (normally regional agencies in countries with more than one HTAb in place, such as Belgian KCE, German IQWIG, Irish HIQA, Italian UCSC, Portuguese INFARMA, Slovakian UHIF, Spanish SESCS or Swedish SBU) showed higher and similar scores on contextual and non-contextual dimensions.
Discussion and conclusions
The alignment between EVIDEM and EUnetHTA methodological frameworks is consistently high, especially when assessing domains related to health problem description, current use of the technology, technical characteristics, clinical effectiveness, and safety. However, other non-contextual dimensions of the EVIDEM framework and the EUnetHTA core model are consistently misaligned.
The main EUnetHTA core model criteria, such as clinical effectiveness, safety conditions, health problem description and current use of technology; are consistently addressed by all HTAb. As previously reported  the institutions go only partially beyond these criteria and it is normally dependent on the topic of assessment. For those European HTAb directly advising on price and reimbursement decisions, the reported criteria used to support their decisions show a more balanced alignment between both methodological approaches. That conclusion could explain why in many cases, the subnational HTAb in those countries with multiple agencies, are the ones showing a balanced profile among contextual and non-contextual dimensions.
EVIDEM provides a generic interpretive frame (MCDA – Multi-Criteria Decision Analysis – reflective grid) that can be used to elicit individual values and facilitate deliberations through a common structure that includes interpretive scores (quantitative criteria), qualitative impacts (qualitative criteria) as well as narrative comments (all criteria) . EVIDEM framework was designed to minimize the limitations of the deliberation process by ensuring that: generic assessment criteria (either quantitative or qualitative) are included; evidence relevant to each criterion is made available through an efficient synthesis methodology; and face validity is checked at each step of the process (weights, scores and corresponding narratives, aggregated measures). EVIDEM framework is sufficiently flexible to be adapted to the local assessment context, although it also requires consistency in the identification of a set of criteria, scoring scale and weights when assessing a broad range of competing interventions in a specific local context [29, 30].
A holistic approach is required to consistently assess the social and medical needs to support payer’s decision on prices and reimbursement conditions of certain drugs, such as disruptive innovations or orphan drugs, broadening the need of using EVIDEM-like contextual assessment tools by European HTAb.
Availability of data and materials
All data generated or analysed during this study are included in this published article and its supplementary information files (supplementary file.xls).
European Medicines Agency
European Network for Health Technology Assessment
Evidence and Value Impact on Decision Making
Health Technology Assessment
Health Technology Assessment Bodies
Multi-Criteria Decision Analysis
Organization for Economic Co-operation and Development
Relative Effectiveness Assessment
Newhouse JP. Medical care costs: how much welfare loss? J Econ Perspect. 1992;6(3):3–21. https://doi.org/10.1257/jep.6.3.3.
Hogerzeil H, Liberman J, Wirtz V, Kishore S, Selvaraj S, Kiddell-Monroe R, et al. Promotion of access to essential medicines for non-communicable diseases: Practical implications of the unpolitical declaration. Lancet. 2013;381(9867):680–9.
OECD. Pharmaceutical spending (indicator); 2018. https://doi.org/10.1787/998febf6-en. Accessed 5 July 2018
Godman B, Bucsics A, Vella Bonanno P, Oortwijn W, Rothe CC, Ferrario A, et al. Barriers for access to new medicines: searching for the balance between rising costs and limited budgets. Front Public Health. 2018;6:328. https://doi.org/10.3389/fpubh.2018.00328.
Kawalec P, Sagan A, Pilc A. The correlation between HTA recommendations and reimbursement status of orphan drugs in Europe. Orphan J Rare Dis. 2016;11(1):122. https://doi.org/10.1186/s13023-016-0501-4.
Jönsson B, Hofmarcher T, Lindgren P, Wilking N. The cost and burden of cancer in the European Union 1995–2014. Eur J Cancer. 2016;66:162–70. https://doi.org/10.1016/j.ejca.2016.06.022.
Cohen D. Cancer drugs: high price, uncertain value. BMJ. 2017;359:j4543.
Davis C, Naci H, Gurpinar E, Poplavska E, Pinto A, Aggarwal A. Availability of evidence of benefits on overall survival and quality of life of cancer drugs approved by European medicines agency: retrospective cohort study of drug approvals 2009-13. BMJ. 2017;359:j4530.
Pontes C. et al,. Time to review authorisation and funding for new cancer medicines in Europe? Inferences from the case of Olaratumab. Appl Health Econ Health Policy 2019. https://doi.org/10.1007/s40258-019-00527-x.
Godman B, Malmstrom RE, Diogene E, Gray A, Jayathissa S, Timoney A, et al. Are new models needed to optimize the utilization of new medicines to sustain healthcare systems? Expert Rev Clin Pharmacol. 2015;8(1):77–94. https://doi.org/10.1586/17512433.2015.990380.
Pauwels K, Huys I, Vogler S, Casteels M, Simoens S. Managed entry agreements for oncology drugs: lessons from the European experience to inform the future. Front Pharmacol. 2017;8:171.
Piatkiewicz TJ, Traulsen JM, Holm-Larsen T. Risk-sharing agreements in the EU: a systematic review of major trends. Pharmacoecon Open. 2018;2(2):109–23. https://doi.org/10.1007/s41669-017-0044-1.
Tuffaha H, Aitken J, Chambers S, Scuffham P. A framework to prioritise health research proposals for funding: integrating value for money. Appl Health Econ Health Policy. 2019;17(6):761–70.
Uyl-de Groot CA, Löwenberg B. Sustainability and affordability of cancer drugs: a novel pricing model. Nat Rev Clin Oncol. 2018;15(7):405–6. https://doi.org/10.1038/s41571-018-0027-x.
Chandra A, Shafrin J, Dhawan R. Utility of cancer value frameworks for patients, payers, and physicians. JAMA. 2016;315(19):2069–70. https://doi.org/10.1001/jama.2016.4915.
Wilsdon T, Attridge J, Fiz E. Assessing the value of biopharmaceitical innovation in key therapy areas in middle-income countries; 2013.
Hidalgo-Vega A. Evaluación, financiación y regulación de los medicamentos innovadores en los países desarrollados; 2018.
Kristensen FB. Mapping of HTA methodologies in EU and Norway. In: European Commission, Science & Policy; 2017.
Kristensen F, Lampe K, Wild C, Cerbo M, Goettsch W, Becla L. The HTA core model®—10 years of developing an international framework to share multidimensional value assessment. Value Health. 2017;20(2):244–50.
Goetghebeur MM, et al. Can reflective multicriteria be the new paradigm for healthcare decision-making? The EVIDEM journey. Cost Eff Resour Alloc. 2018;16(Suppl 1):54.
Friedmann C, Levy P, Hensel P, Hiligsmann M. Using multi-criteria decision analysis to appraise orphan drugs: a systematic review. Expert Rev Pharmacoecon Outcomes Res. 2017;18(2):135–46. https://doi.org/10.1080/14737167.2018.1414603.
Goetghebeur M, Wagner M, Khoury H, Levitt R, Erickson L, Rindress D. Evidence and Value: Impact on DEcisionMaking - The EVIDEM framework and potential applications. BMC Health Serv Res. 2008;8:270.
Guarga L, Badia X, Obach M, Fontanet M, Prat A, Vallano A, et al. Implementing reflective multicriteria decision analysis (MCDA) to assess orphan drugs value in the Catalan Health Service (CatSalut). Orphan J Rare Dis. 2019;14(1):1–9.
Radaelli G, Lettieri E, Masella C, Merlino L, Strada A, Tringali M. Implementation of EUNetHTA core model in Lombardia: the VTS framework. Int J Technol Assess Health Care. 2014;30(1):105–12. https://doi.org/10.1017/S0266462313000639.
EUnetHTA. Methods for health economic evaluations-A guideline based on current practices in Europe. www.eunethta.eu/wp-content/uploads/2018/03/Methods_for_health_economic_evaluations.pdf. Accessed 21 Apr 2020.
Guyatt G, Oxman A, Akl E, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1. Introduction - GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64(4):383–94.
Jaeschke R, Guyatt G, Dellinger P, Schünemann H, Levy M, Kunz R, et al. Use of GRADE grid to reach decisions on clinical practice guidelines when consensus is elusive. BMJ. 2008;337(7665):327–30.
Vicent B, Wiitala W, Luginbill K, Molling D, Hofer T, Ryan A, et al. Template matching for benchmarking hospital performance in the veterans affairs healthcare system. Medicine. 2019;98(20):1–9.
Youngkong S, Tromp N, Chitama D. The EVIDEM framework and its usefulness for priority setting across a broad range of health interventions. Cost Effect Res Alloc. 2011;9(1):8. https://doi.org/10.1186/1478-7547-9-8.
Thokala P, Devlin N, Marsh K, Baltussen R, Boysen M, Kalo Z, et al. Multiple criteria decision analysis for health care decision making—An introduction: report 1 of the ISPOR MCDA emerging good practices task force. Value Health. 2016;19:1–13.
No funding was received for the conduct of the study.
Ethics approval and consent to participate
Consent for publication
David Elvira is a full-time employee of Sanofi. Caridad Pontes and Mercè Obach are public employees with no other conflicts of interest.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
Elvira, D., Obach, M. & Pontes, C. Description of the use of multicriteria to support pricing and reimbursement decisions by European health technology assessment bodies. BMC Health Serv Res 21, 814 (2021). https://doi.org/10.1186/s12913-021-06784-8
- Health technology assessment
- Multicriteria assessment methods
- Reimbursement systems