Skip to main content
  • Research article
  • Open access
  • Published:

Mapping of multiple criteria for priority setting of health interventions: an aid for decision makers



In rationing decisions in health, many criteria like costs, effectiveness, equity and feasibility concerns play a role. These criteria stem from different disciplines that all aim to inform health care rationing decisions, but a single underlying concept that incorporates all criteria does not yet exist. Therefore, we aim to develop a conceptual mapping of criteria, based on the World Health Organization’s Health Systems Performance and Health Systems Building Blocks frameworks. This map can be an aid to decision makers to identify the relevant criteria for priority setting in their specific context.


We made an inventory of all possible criteria for priority setting on the basis of literature review. We categorized the criteria according to both health system frameworks that spell out a country’s health system goals and input. We reason that the criteria that decision makers use in priority setting exercises are a direct manifestation of this.


Our map includes thirty-one criteria that are distributed among five categories that reflect the goals of a health system (i.e. to improve level of health, fair distribution of health, responsiveness, social & financial risk protection and efficiency) and leadership/governance one category that reflects feasibiliy based on the health system building blocks (i.e. service delivery, health care workforce , information, medical products, vaccines & technologies, financing and).


This conceptual mapping of criteria, based on well-established health system frameworks, will further develop the field of priority setting by assisting decision makers in the identification of multiple criteria for selection of health interventions.

Peer Review reports


Concerns on the costs, effectiveness and cost-effectiveness of health interventions have dominated the debate on health rationing in a wide range of countries since long [13]. More recently, the explicit use of a number of equity-related criteria have been put forward, like severity of disease, socio-economic status, or gender, reflecting the increased attention for distribution of health in a population, as summarized by Johri and Norheim [4]. Still other criteria, like ease of implementation [5] or political acceptability [6] are presently finding their way in the prioritization of health interventions. The recognition that not a single but multiple criteria should explicitly be considered has led to the development of multi-criteria decision analysis (MCDA). This method sets programme priorities by referring to a comprehensive set of explicit criteria and guides decision makers in understanding the trade-offs between values that may be conflicting. For example while mobile clinics for HIV testing may be costly and therefore inefficient, they may deserve priority because they reach out to remote areas and therefore contribute to equity in service delivery. A core component in any MCDA is the identification of criteria that decision-makers consider important in their specific contexts. As a next step, MCDA scores the performance of health interventions on these criteria [7, 8].

At the same time, surprisingly little work has been done to develop a meaningful conceptual mapping of criteria that can help to identify the relevant set of criteria. A recent report that advices the UK’s National Institute for Health and Clinical Excellence (NICE) for the use of MCDA reviewed the literature and one of the findings was that most applications have a fixed set of criteria and lack explanation of the rationale behind the selected criteria and the categories used. In addition, it was concluded that the applications give neither flexibility nor assistance to decision makers to select an unique set of criteria in their decision context [8]. More specifically, in 1999, Musgrove presented the ‘nine criteria for public spending on health care’ in a spider-web like diagram, however, without classification of criteria [9]. Baltussen and Niessen presented in 2006 the ‘cloud of criteria’, suggesting that criteria cannot be systematically categorized [7]. Another framework, introduced by EVIDEM in 2010, does not explain an underlying rationale for the choice of categories that are used [10, 11]. Furthermore, various reviews simply list priority setting criteria [4, 8, 11, 12]. Only the list of criteria reported in the review of Golan et al., categorizes criteria according to the principles of allocative justice for rationing health care, i.e. need, maximizing and egalitarian principles, but is therefore limited in scope [13]. Our paper aims to develop a conceptual mapping of a comprehensive set of criteria, including efficiency, equity and feasibility concerns.

Categorization of criteria is important for two main reasons. Firstly, for decision makers, the grouping of a large and diffuse set of criteria into categories may ease their interpretation and facilitate decision-making. Second, such a categorization may be an aid to well-define criteria, and to avoid overlap and double-counting of criteria. Criteria, especially those related to health distribution (i.e. equity) concerns are often difficult to define and a proper mapping sets boundaries to facilitate this.


Conceptual mapping of criteria

This paper introduces a conceptual mapping of criteria based on an integration of two well-established health systems frameworks, i.e. the World Health Organization’s (WHO) Health Systems Performance framework[14] and Health Systems Building Blocks framework[15, 16] (Figure 1). These frameworks spell out a country’s health system goals and input – we reason that the criteria that decision makers use in priority setting exercises are a direct manifestation of this. More specifically, the Health System Performance framework is a generally accepted concept to reflect the goals of a health system. Here, in our interpretation, the framework reflects criteria that indicate the goals of interventions in health, i.e. to improve the level and distribution of health, to improve responsiveness, to offer financial protection and to make efficient use of resources. This can be loosely defined as ‘what should a health system do’. The Health System Building Blocks framework is a generally accepted concept to reflect the required components (or inputs) for an effective health system. Here, in our interpretation, the framework reflects criteria that relate to the feasibility of interventions, or loosely defined as ‘what can a health system do’. These criteria relate to the building blocks: ‘service delivery’, ‘health workforce’, ‘information’, ‘medical products, vaccines & technologies’, ‘financing’, and ‘leadership/governance’. Together, the two frameworks offer a comprehensive framework for classifying priority setting criteria. We employ both WHO frameworks because they are worldwide used by decision makers at country level and are credible conceptualizations of health systems [15, 17].

Figure 1
figure 1

The building blocks and goals of the health system.

We carried out two steps to develop our conceptual mapping of criteria. In a first step, we made an inventory of all possible criteria for priority setting on the basis of literature reviews [4, 8, 1113]. In a second step, we categorized these criteria according to the health systems goals and building blocks, based on their definitions (Table 1). For example, following the health system performance framework, we distinguished the objective to improve health in two categories, i.e. to improve the ‘level of health’ and the ‘distribution of health’. We defined and classified criteria in these categories in order to avoid overlap between criteria.

Table 1 Definitions of categories used in the criteria map (based on the health system goals and building blocks)


The conceptual mapping of criteria for priority setting is provided in Figure 2. The definitions of the categories are similar to those in the original WHO health system frameworks and are presented in Table 1. Our literature review resulted in the identification of a large set of criteria, which are often similar in concept but different in the ways they are described. In the Additional file 1, Overview of criteria considered for inclusion in criteria map, we list all criteria we considered for inclusion in our framework, and present a rationale for their inclusion or exclusion. The included thirty-one criteria are all single not-overlapping arguments to prioritize health interventions and are defined in Table 2. Here, we will give the rationale used for a selection of the criteria considered.

Figure 2
figure 2

Mapping of priority setting criteria.

Table 2 Definitions of criteria for priority setting included in the criteria map

On the right panel of the map we distinguish five categories of criteria related to intervention’s goals. The first category is ‘health level’, and includes criteria ‘effectiveness on individual level’, ‘effectiveness on population level’ ‘patient reported health status’ and ‘safety’. Whereas reviews include ‘quality of evidence on effectiveness’, we excluded this from our map, as we consider quality of evidence to be relevant to all criteria, e.g. how costly or complex an intervention is. Rather, we propose to capture quality of evidence in uncertainty analysis. The second category is ‘health distribution’, and included criteria that relate to the core concept of ‘equal life time health’, which means that all people - independent of their background, their disease status or the availability of treatment - should have a fair chance to live a full healthy life [18]. This concept encompasses both horizontal and vertical equity. We define horizontal equity as the provision of equal treatment for people with equal health needs. Horizontal equity is non-discriminative towards certain groups in society to give them equal access to care as other groups with the same needs. We define vertical equity as the provision of unequal but equitable treatment for people with unequal health needs – this implies giving priority to certain groups in society, on the basis of their background (‘socioeconomic status’, ‘area of living’, ‘sex and gender’, ‘age’, ‘ethnicity’, ‘sexual orientation’), disease status (‘severity of disease’) or the ‘availability of treatment’ [19]. ‘Responsibility for health’ is included as interventions that focus on people that have bad luck may deserve more priority. The precise identification and definition of equity-related criteria is topic of a recent collaboration between different experts (ethicists, public health experts, economist, etc.) from academic institutes and the WHO [4, 19]. Therefore we only provisionally list and define these criteria here.

The third category is ‘responsiveness’, and includes ‘patient perceived quality of care’. Interventions should be responsive to people’s expectations in regard to non-health matters and reflect the importance of people’s dignity and autonomy, and the confidentiality of information. Although this seems a general, and therefore system-wide concern, some interventions do better than other interventions in satisfying perceived quality of care. We include ‘burden of disease’ to represent the wish of society to target high burden diseases. The fourth category is ‘social and financial risk protection’ and includes ‘catastrophic health expenditure’, ‘economic productivity and care for others’ and ‘rare diseases’. Regarding the latter, interventions targeting rare diseases may deserve priority because they may be very costly (as intervention (especially drug) development costs are only shared by a small number of patients) and could push people into poverty. The fifth category is ‘improved efficiency’ and reflects the economies of scale that can be obtained when reaching large number of people. Therefore we have included the criterion ‘size of target group’. We decided to exclude ‘cost-effectiveness’ as a criterion on itself, as theoretically costs (as a feasibility constraint) and effectiveness (as a goal) are both implicitly included in the mapping as individual criteria. However, regarding the importance of the criterion of cost-effectiveness in decision-making, decision-makers may nevertheless consider it as a separate criterion.

On the left panel of the map we distinguish one category of criteria related to the feasibility of implementation of interventions. This category is divided in six subcategories, based on the building blocks, i.e. ‘service delivery’, ‘health workforce’, ‘information’, ‘medical products, vaccine and technologies’, ‘financing’, and ‘leadership/governance’. In contrast to the criteria related to intervention goals (discussed above), little work has been done on these criteria (except on the criteria ‘cost’) and the criteria we put forward are first propositions. The first four subcategories relate to the current capacity of the health system and criteria reflect the requirements the for implementation of an intervention. The fifth subcategory on financing encompasses ‘unit costs’, ‘budget impact’ and ‘financing party’. The ‘unit costs’ are the total costs per patient from a health systems perspective whereas 'budget impact' incorporates the scale of an intervention. The ‘financing party’ criterion captures who is paying for a health intervention and reflects notions on its financial sustainability. The sixth subcategory represents the feasibility in terms of leadership/governance and includes ‘congruency previous priority setting’, ‘cultural acceptability’, ‘political acceptability’, ‘stakeholder acceptability’, and ‘legal barriers’.


Our map should not be regarded as a top-down expert advice on a fixed set of criteria that should always be considered in all prioritization decisions, but rather as an aid to decision-makers in their selection of relevant criteria. We see two broad applications of priority setting, and therefore of our mapping of criteria. First, it can inform decision makers who work in a specific context, e.g. on the reimbursement decision of a single intervention. These decisions are taken in the presence of a known budget and are likely limited by factors such as the currently available physical infrastructure, human resources or political consideration, at least in the short- to medium-term. This is labeled ‘context-specific priority setting’. The second application is to guide decisions on a wide range of interventions, to provide general information on their relative rank order to arrive a more informed debate on resource allocation priorities. Because priority setting in this application is not meant to provide a solution to a specific resource allocation question, it need not be highly contextualized in terms of e.g. physical infrastructure and/or human resources constraints. This is labeled ‘generalized priority setting’ [20]. The set of criteria for ‘context-specific priority setting’ is likely to be much more specific than those for ‘generalized priority setting’, but stem from the same conceptual mapping of criteria as presented above. That our mapping of criteria should not be considered as a fixed set of criteria is especially clear when setting priorities in a specific disease area. For example, an important criterion in HIV/AIDS health rationing decisions is whether the intervention reduces stigma. To include these disease-specific criteria in a generic list would increase its total number of criteria and make it unmanageable.

As our mapping is based on the WHO health systems frameworks, it has a certain credibility among decision makers. However, the choice of a different underlying framework might lead to another mapping of criteria.

In our framework, we included criteria related to the health system inputs and health system goals, but no criteria related to intermediate outcome measures as access and utilization. These measures are instrumental to reach health system goals, and are as such no goals in themselves. However, decision-makers can use them to monitor and evaluate progress towards the realization of the health system goals.

Another important step in MCDA is to define indicators to operationalize the criteria. For example, the severity of disease that an intervention targets can be measured in terms of health state valuations [21] and health gains in terms of disability adjusted life years averted. The operationalization of criteria would complement our mapping of criteria, and would allow the construction of a performance matrix that systematically demonstrates the performance of an intervention on all criteria [7, 8]. Such a matrix can consequently be the basis for rationing decisions on (a set of) health interventions. Such decisions should eventually also account for non-quantifiable criteria related to e.g. complicated ethical judgments. These criteria are not reflected in our framework, and further research should be carried out on how these can best be accounted for, e.g. through a process of elaboration [8, 20, 22].

We consider the presented mapping of priority setting criteria as preliminary only and not as a final map. We welcome discussions to further develop it, to improve the use of MCDA for setting priorities in health.


This conceptual mapping of criteria, based on well-established health system frameworks, will further develop the field of priority setting by assisting decision makers in the identification of multiple criteria for selection of health interventions.



Evidence and Value: Impact on DEcisionMaking


Multi Criteria Decision Analysis


National Institute for Health and Clinical Excellence


World Health Organization.


  1. Bank W: World Development Report: Investing in Health. 1993, New York: Oxford University Press

    Google Scholar 

  2. Ham C: Priority setting in health care: learning from international experience. Health policy (Amsterdam, Netherlands). 1997, 42 (1): 49-66. 10.1016/S0168-8510(97)00054-7.

    Article  CAS  Google Scholar 

  3. WHO-CHOICE: Making choices in health: WHO guide to cost-effectiveness analysis. 2003, Geneva: World Health Organization

    Google Scholar 

  4. Johri M, Norheim O: Can cost-effectiveness analysis integrate concerns for equity? systematic review. Int J Technol Assess Health Care. 2012, 28: 2.

    Article  Google Scholar 

  5. Gericke CA, Kurowski C, Ranson MK, Mills A: Intervention complexity–a conceptual framework to inform priority-setting in health. Bull World Health Org. 2005, 83 (4): 285-293.

    PubMed  PubMed Central  Google Scholar 

  6. Goddard M, Hauck K, Smith PC: Priority setting in health - a political economy perspective. Health Econ Policy Law. 2006, 1 (Pt 1): 79-90.

    PubMed  Google Scholar 

  7. Baltussen R, Niessen L: Priority setting of health interventions: the need for multi-criteria decision analysis. Cost Eff Resour Alloc. 2006, 4: 14-10.1186/1478-7547-4-14.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Devlin N, Sussex J: Incorporating multiple criteria in HTA: methods and processes. 2011, London: Office of Health Economics,

    Google Scholar 

  9. Musgrove P: Public spending on health care: how are different criteria related?. Health policy (Amsterdam, Netherlands). 1999, 47 (3): 207-223. 10.1016/S0168-8510(99)00024-X.

    Article  CAS  Google Scholar 

  10. The EVIDEM (Evidence and Value: Impact on DEcisionMaking) framework.

  11. Guindo L, Wagner M, Baltussen R, Rindress D, Van Til J, Kind P, Goetghebeur M: From efficacy to equity: literature review of decision criteria for resource allocation and healthcare decisionmaking. Cost Eff Resour Alloc. 2012, 10: 9-10.1186/1478-7547-10-9.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Cleary SM, Mooney GH, McIntyre DE: Claims on health care: a decision-making framework for equity, with application to treatment for HIV/AIDS in South Africa. Health Policy Plan. 2011, 26 (6): 464-470. 10.1093/heapol/czq081.

    Article  PubMed  Google Scholar 

  13. Golan O, Hansen P, Kaplan G, Tal O: Health technology prioritization: Which criteria for prioritizing new technologies and what are their relative weights?. Health policy (Amsterdam, Netherlands). 2011, 102 (2–3): 126-135.

    Article  Google Scholar 

  14. World Health Organization: The world health report 2000 - Health systems: improving performance. 2000, Geneva: World Health Organization

    Google Scholar 

  15. World Health Organization: Everybody’s Business: Strengthening Health Systems to Improve Health Outcomes: WHO’s Framework for Action. 2007, Geneva: World Health Organization

    Google Scholar 

  16. World Health Organization: The world health report 2008 - Primary health care: now more than ever. 2008, Geneva: World Health Organization

    Google Scholar 

  17. Frenk J: Bridging the divide: global lessons from evidence-based health policy in Mexico. Lancet. 2006, 368 (9539): 954-961. 10.1016/S0140-6736(06)69376-8.

    Article  PubMed  Google Scholar 

  18. Daniels N: Just Health. 2008, Cambridge: Cambridge University Press

    Google Scholar 

  19. World Health Organization: Setting equitable priorities in health and health care: from theory to practice. Brocher Symposium 2011. 2011, Geneva,

    Google Scholar 

  20. Baltussen R, Youngkong S, Paolucci F, Niessen L: Multi-criteria decision analysis to prioritize health interventions: Capitalizing on first experiences. Health policy (Amsterdam, Netherlands). 2010, 96 (3): 262-264. 10.1016/j.healthpol.2010.01.009.

    Article  Google Scholar 

  21. Murray C: Quantifying the burden of disease: the technical basis for disability-adjusted life years. Bull World Health Org. 1994, 72 (3): 429-45.

    CAS  PubMed  PubMed Central  Google Scholar 

  22. Youngkong S, Baltussen R, Tantivess S, Koolman X, Teerawattananon Y: Criteria for priority setting of HIV/AIDS interventions in Thailand: a discrete choice experiment. BMC Heal Serv Res. 2010, 10: 197-10.1186/1472-6963-10-197.

    Article  Google Scholar 

Pre-publication history

Download references


We thank the Radboud University Nijmegen Medical Center for their financial support for this study. The sponsor employs both authors. The funding agreement ensured the authors’ independence in designing the study, interpreting the data, writing, and publishing the report. We thank the De ArgumentenFabriek ( for inspiring us to map the criteria according to their proposed format.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Noor Tromp.

Additional information

Competing interests

The authors declare that they have no competing interests.

Authors’ contributions

NT contributed to the design of the study, led the data collection and selection of criteria for inclusion in the map and drafted and finished the article. RB contributed to the design of the study and the selection of criteria for inclusion in the map and assisted in writing the article.

Electronic supplementary material

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Authors’ original file for figure 2

Rights and permissions

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article

Tromp, N., Baltussen, R. Mapping of multiple criteria for priority setting of health interventions: an aid for decision makers. BMC Health Serv Res 12, 454 (2012).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: