‘Real-world’ health care priority setting using explicit decision criteria: a systematic review of the literature
BMC Health Services Research volume 15, Article number: 164 (2015)
Health care decision making requires making resource allocation decisions among programs, services, and technologies that all compete for a finite resource pool. Methods of priority setting that use explicitly defined criteria can aid health care decision makers in arriving at funding decisions in a transparent and systematic way. The purpose of this paper is to review the published literature and examine the use of criteria-based methods in ‘real-world’ health care allocation decisions.
A systematic review of the published literature was conducted to find examples of ‘real-world’ priority setting exercises that used explicit criteria to guide decision-making.
We found thirty-three examples in the peer-reviewed and grey literature, using a variety of methods and criteria. Program effectiveness, equity, affordability, cost-effectiveness, and the number of beneficiaries emerged as the most frequently-used decision criteria. The relative importance of criteria in the ‘real-world’ trials differed from the frequency in preference elicitation exercises. Neither the decision-making method used, nor the relative economic strength of the country in which the exercise took place, appeared to have a strong effect on the type of criteria chosen.
Health care decisions are made based on criteria related both to the health need of the population and the organizational context of the decision. Following issues related to effectiveness and affordability, ethical issues such as equity and accessibility are commonly identified as important criteria in health care resource allocation decisions.
Health care decision making requires the balancing of the demand for programs, services, and technologies that improve human health, and the need for fiscal restraint and the reality of a finite resource pool. All health care decision making involves making choices between many attractive alternatives, and saying ‘no’ to some things that might be desirable and valuable.
Because health care resources are, in many jurisdictions, provided through public subsidy, health care decision makers have an ethical obligation to allocate those resources in a way that is fair, transparent, and accountable . Further, health care decisions should be made according to the available evidence relating to a number of possible decision making criteria, including effectiveness, cost-effectiveness, equity considerations, feasibility, affordability etc. of the proposed program, service, or technology. But allocation decisions are also influenced by a number of factors other than medical and health economic evidence – pragmatic issues of organizational structure and political realities are legitimate and important components of these decisions [2-6] and must be considered as well.
Two popular proposed methods for guiding the setting of health care resource allocation priorities are Programme Budgeting and Marginal Analysis (PBMA) and Multi-Criteria Decision Analysis (MCDA). PBMA involves the listing of all relevant activities and their resource requirements, the evaluation of the effectiveness of these activities according to a set of explicit criteria, and the application of that evaluation to the available budget . The PBMA process can be summarized by asking five questions :
What resources are available in total?
In what ways are these resources currently spent?
What are the main candidates for more resources and what would be their level of effectiveness?
Are there any areas of care which could be provided to the same level of effectiveness but with fewer resources to fund candidates from (3) (i.e., addressing technical efficiency)?
Are there areas of care which, despite being effective, should receive fewer resources because a proposal from (3) is more effective (per dollar spent) (i.e., addressing allocative efficiency)?
PBMA exercises are commonly conducted by an advisory panel of expert stakeholders, and should be accompanied by an evaluation of the outputs to ensure allegiance with the priorities and needs of the organization .
MCDA involves the numerical quantification of the merit of competing options, according to explicit decision criteria . The primary aim of MCDA is to develop models of decision maker objectives and their value trade-offs so that options under consideration can be compared with each other in a consistent and transparent manner . A key principle is that decisions between different options (for example different interventions) should be consistent with stakeholders’ objectives. MCDA is transparent in that it shows that decisions are the logical implications of those objectives. In MCDA, objectives are deemed to be within the discretion of the decision-makers. That is, they are not predetermined by some underlying theory from economics or ethics (such as utilitarianism).
MCDA then typically consists of two overarching stages [10,11]. First, problem structuring involves generating a set of alternatives and a set of criteria against which the alternatives are to be evaluated and compared. In order to structure the problem, the first questions to ask are ‘what priority setting objectives do decision-makers wish to pursue?’ And, ‘what locally relevant criteria do decision-makers use when deciding between alternative interventions?’ Objectives are the principles that determine priority setting policies (e.g. improving population health) whereas criteria are the standards that alternative interventions are judged by (e.g. health outcomes from different treatments). Second, model building entails constructing some form of model which represents decision-makers’ objectives and their value judgements. There are then two key considerations to be addressed in this type of model [10,11]: the methods used to describe decision-makers preferences and elicit importance weights for decision-making criteria; and, the type of aggregation model used to combine criteria scores (see Peacock 2009 for more detail).
Similar to PBMA, MCDA is conducted with the input of decision makers and relevant stakeholders in the ultimate resource allocation decision . In addition, MCDA can be employed within the broader PBMA approach as the mechanism for benefit measurement to inform allocation or re-allocation recommendations. As such, these approaches in our view are best viewed as complementary.
In a time when organizations are adopting explicit criteria-based decision methods and deciding which method is the best fit for their organization [12,13], it is important to examine the criteria that have been used in previous decisions. While reviews of the priority-setting literature have been conducted [2,14], these studies have focused on hypothetical exercises and stated preferences rather than cases where decision-making bodies have had to make decisions under the actual constraints of budgetary and political realities.
The purpose of this paper is to summarize the available literature on health care decision-making where explicit criteria-based methods like PBMA or MCDA were used (i.e., a set list of factors were weighed against each other according to some underlying framework), in order to examine the criteria used by decision makers in ‘real world’, rather than hypothetical, settings. The added value of this paper is that our review, based on significant experience in this field over the past 12 years, focused on ‘real-world’ priority setting using explicit decision criteria. These examples may be interpreted as a better predictor of future health care priority setting than decisions made in the abstract.
A systematic review of the literature was conducted, based on English language search terms used in a previously published review of this literature . Relevant subject headings were abstracted from the articles included in that review: “Decision Making, organizational”; “Health Care rationing”; “Health Priorities”; “Budgets”; and “Community Health Planning”.
To broaden the scope of our search to ensure inclusion of exercises using popular decision-making methods, the terms “Program (me) Budget”, “Marginal Analysis”, and “Multi-Criteria Decision Analysis” as well as their respective acronyms (“PBMA”, “MCDA”) were also added to the search terms.
The relevant search terms were used to search the MEDLINE, ECONLit, and PAIS databases. The search terms were also entered into the Google™ search engine to investigate the presence of ‘grey literature’ (i.e., non peer-reviewed publications). Retrieval was limited to documents published between January 1st, 2000 and July 31st, 2013, to reflect the relevant time period since the previous literature review.
Articles were identified as potentially eligible based on a review of their abstracts. Potentially eligible items underwent closer examination based on exclusion criteria. Items were excluded if they did not meet the following description: a description of a priority setting exercise in which a funding decision was made based on a set of explicitly-defined criteria. Articles published in languages other than English were not included in this review.
The reference lists of all items (i.e., excluded and included) were manually searched for potentially-eligible items. The reference-identified items underwent the same scrutiny and application of exclusion criteria. The search strategy is diagrammed in Figure 1. One reviewer (IC) conducted the review and data abstraction from all studies. Two other reviewers (SJP and CM) verified the accuracy of the search process and the application of the exclusion criteria. Disagreements about eligibility were resolved through consensus. Because studies included in this exercise were descriptive rather than quantitative, and because of the nature of the research question, no formal assessment of the quality or bias of included items was undertaken. Because this research did not involve human participants, no Research Ethics approval was sought or gained.
A total of sixty-five articles were identified as potentially eligible from the indexed databases. An additional fifty-two were identified from the manual reference search and 25 from web search, for a total of 142 potentially eligible items. Manual search from the grey literature did not yield any additional unique items. After applying the exclusion criteria, thirty-five articles were identified as eligible. Two peer-reviewed items were duplicates of items from the grey literature and were deemed redundant, bringing the total number of included items to thirty-three.
Characteristics of the included studies, including the setting, method of decision-making used, and the result of the exercise (e.g., a change in policy, a list of priorities, etc.), are presented in Table 1. Most exercises were conducted in North American or European or European-descended (“Western”) countries, primarily Canada (10) and the United Kingdom (7). There was equal representation of studies using an MCDA framework (15) as using PBMA (15). Some studies did not explicitly state the decision-making method, or used a synonymous term (e.g., “Decision Science”), and were classified as PBMA or MCDA based on the characteristics of the method used (e.g., generic vs. algorithmic methods of criteria weighting and decision ranking, inclusion of budgetary information, etc.).
The most common outcome of MCDA exercises was a ranked list of alternatives, rather than an explicit funding decision of the type that was more common among exercises that used PBMA. Allocation of points, whereby an agreed-upon number of points can be assigned to different categories, was a common method of assigning weights to decision criteria. Many MCDA exercises used Discrete Choice experiments to elicit criteria weights. A number of studies did not describe any weighting method, and may have simply considered all criteria equally important.
Choice of decision-making criteria
The decision-making criteria used in each included item were identified and extracted. Where possible, criteria with identical/similar definitions across different studies were collapsed into a single criterion (by IC, decisions reviewed by SJP and CM), to make direct comparison between exercises more possible. A total of seventy-two unique criteria were identified among the included items. These criteria are listed in Additional file 1: Appendix A. The most common criteria were:
The effectiveness of the program (21 items)
Budgetary impact/Affordability (16 items)
Reducing inequalities between groups/”Equity” (14 items)
Number of people likely to benefit from program/intervention (13 items)
Ability to access the program/intervention (13 items)
Cost-effectiveness or other health economics evidence (12 items)
Quality of the available evidence (10 items)
It is important to note that, because of variations in the way in which criteria were described, it may be possible that some criteria are ‘redundant’, insofar as some may simply be more specific iterations of others. For example, the criteria of “poverty reduction” and “age/risk of target group” could both be collapsed into the “Equity” criterion. A conservative approach was taken to combining criteria in this way, preferring to list criteria individually in cases where there was ambiguity about whether or not terms were truly synonymous. Similarly, the classification of a given criterion into a domain was prone to subjectivity – criteria were not always well-described in the text of the exercise. We again used a conservative approach for this classification process, and resolved ambiguities through consensus.
It should also be noted that many studies listed broad criteria that contained a number of sub-criteria within them (e.g., one study listed an “effectiveness” criterion that included “number of patients”, “individual benefit”, “magnitude of benefit”, “duration of benefit”, “personal networks”, “collective benefits”, “population impact”, and “social capital” as included sub-criteria). In cases where criteria weights were not provided for each sub-criterion, the broad criteria were preferred.
Domains of decision-making criteria
Criteria used on priority setting exercises were classified into ten descriptive ‘domains’, using a classification system described by Tanios and colleagues :
Intervention outcomes and Benefits
Type of Health Service
Disease Impact (burden)
Environmental Impact of the Intervention
Quality/Uncertainty of Evidence
A full list of all criteria included in each domain is provided in Additional file 1: Appendix A. A proportional breakdown of each domain is provided in Figure 2. Economic Impact and Intervention Outcomes/Benefits were the two most frequently-cited domains, followed by Overall Context, Disease Impact (burden), and Priorities (fairness). Only two of the included items included a Miscellaneous category (i.e., a category labeled “miscellaneous” in the report/manuscript itself).
Domain by type of exercise
It is possible that the types of criteria used may differ due to the differences in the setting and methodology between PBMA and MCDA exercises. The frequency of criteria use within each domain was calculated for each type of approach. Results are presented in Figure 3. Overall, the types of criteria used were similar across the exercise types. Differences between the proportion of domain in each exercise were not significant (two-tailed t-test; α = 0.05).
Domain by country of origin
It is similarly possible that the size of the national economy affects the types of criteria used in health care decisions. Exercises in our study were grouped based on membership in the G7 group of countries (Canada, France, Germany, Italy, Japan, the United Kingdom, and the United States). The frequency of criteria use within each domain was calculated for exercises conducted in G7 and non-G7 countries. Once again, the frequency of criteria was largely similar between the two groups. Exercises performed in non-G7 countries were more likely to consider criteria in the Disease Impact (Burden) domain than those in G7 countries (two-tailed t test, p = 0.002); conversely, exercises in G7 countries were more likely to consider criteria in the Overall Context domain (p = 0.006) – other differences were not statistically significant. Results are presented in Figure 4.
Our search yielded 33 distinct ‘real-world’ priority setting exercises conducted using explicit decision-making criteria. Decisions were made largely along common categories of criteria that included the likely impact of a program/intervention, the ability of a program/intervention to address inequalities, and the political/organizational realities of the entities that are making funding decisions.
Summary of main findings
Our review found that while health-specific decision making criteria are of primary importance in priority setting exercises, criteria relating to organizational and political considerations (i.e., criteria that fall primarily into the “Implementation Complexity” and “Overall Context” domains) are also important elements in decision making. Health care delivery must be guided by the available biomedical evidence, but as it is still a human endeavour, political realities must be considered and weighed. Decision makers should be forthcoming about the need to balance the goal of improving human health with the pragmatic constraints all health systems are under.
It is worth noting that fewer than half of the items included in this review explicitly included cost-effectiveness evidence in their decision-making process. While it is certainly true that costs and outcomes are considered in other criteria (i.e., Affordability, Effectiveness of program), incremental cost-effectiveness is not a component of these analyses. Health economic evidence is uniquely suited to provide valuable information when making decisions about scarce resources, but was not explicitly considered in the majority of the identified exercises. This finding is in line with a growing body of research suggesting that while decision-makers would like to use health economic evidence, they may not be able to do so [16,17]. Health economists and policy makers must continue to work together to determine the best way of making health economic findings more policy-relevant and usable.
It is valuable to note that, after “effectiveness of the program” and “affordability”, issues related to equity and fairness were foremost in the decision-making process. This suggests that, at least where decision criteria are made explicit, decision makers are concerned with addressing systemic inequalities and providing health care on a ‘by-need’ basis, underlining the important role of ethics in health care decision making [4,18].
We found that, broadly speaking, the studies in this review fell equally into two categories: MCDA exercises and PBMA exercises. It is important to note that a number of studies did not explicitly categorize themselves as either PBMA or MCDA, for example in one case using the term “Decision Science” to describe the methodology. In these cases, we classified all studies in this review as either PBMA or MCDA based on the methods described in the publication. It is conceivable, therefore, that other reviewers may classify them differently. Our discussion of the contrast between “MCDA exercises” and “PBMA exercises” should be read with this caveat in mind.
The primary outcome of the MCDA items included in this review was the prioritization of alternatives. It is not clear from the manuscripts themselves whether the priorities identified through the MCDA process became official policy, rather than simply a set of recommendations. By way of contrast, the PBMA studies often refer explicitly to investment/disinvestment actions taken as a result of the exercise. It may be valuable to policy-makers to know, when deciding what priority setting approach they prefer, how the approach translates into actual policy decisions.
Given that the resource constraints in developing economies are greater than those of countries with larger budgets, the use of criteria-guided priority setting is of particular importance in countries with developing economies [19,20]. MCDA was used in a number of countries with developing economies, whereas PBMA was used exclusively in countries with developed economies (especially Canada and the UK). The preference for MCDA appears to be due in part to the effort of investigators associated with Dr. Rob Baltussen, who is listed as an author on four of the seven relevant MCDA exercises. In parallel, application of PBMA has tended to be associated with a handful of applied researchers working in the UK, Australia and Canada. While the socio-geographic disparity seen in our review is likely to be a simple reflection of this authorship ‘clustering’ effect, it may be worthwhile to investigate whether there are barriers or relevant factors to the use of other decision-making techniques in these countries; whether there is a true difference in the suitability of PBMA or MCDA in particular political/financial resource environments.
We looked for differences in the type of criteria used in PBMA vs. MCDA exercises, as well as in G7 countries vs. the rest of the world. Care should be taken, however, in extrapolating from these results. The small number of available studies and the lack of a consistent definition of criteria mean that such comparisons are inherently difficult to make. Researchers may wish to investigate the extent to which decision-makers in wealthy countries face different pressures when allocating health resources than those faced by decision-makers in less wealthy countries. Our findings suggest that both PBMA and MCDA can incorporate the criteria that are most important to the local context, and that the decision-making method should be chosen independently of the chosen criteria, based on the feasibility and applicability of a given method to that context.
Regardless of the method of priority setting used, the use of explicit criteria is valuable for all levels of health care decision making. Because no health care system can fund all possible alternatives, decision makers have an ethical obligation to act as good stewards of finite health care resources, and should be accountable to the communities they serve.
Our search criteria were based on a previously-published review of PBMA exercises. As a result, it is possible that the terms are biased toward one decision-making method at the expense of others. We intentionally used MeSH terms and included an additional term that specifies MCDA in order to counteract any potential bias toward PBMA. We also used manual searches of the reference section of all papers included in the review, in order to ensure that as wide a ‘net’ was used in the search. It is worth noting that while the majority of the items included in this review were found using generic search terms (e.g., “Decision Making”, “Health Priorities”, etc.), we did supplement our search to use PBMA and MCDA-specific terms as well. It is possible, therefore, that our review is biased toward these methods. Regardless, we believe that this exercise represents a fair encapsulation of the status of the literature.
We chose to classify our criteria according to a previously-published conceptual framework ; however, our classification was subject to our own interpretation – criteria were not listed in consistent language, requiring us to make our own decisions. It is possible that our interpretations were biased, and that other raters would classify a given criterion differently. It is additionally possible that an interested reader would classify or combine the listed criteria differently (e.g., “efficiency” and “appropriate use of resources” may be seen as identical criteria, though this review counts them separately). The conservative approach we used – preferring to list criteria verbatim to reduce rater bias – leaves a great deal of subjectivity up to the reader. Accordingly, we have listed the criteria used in this analysis, as well as the way in which the criteria were assigned to domains, in Additional file 1: Appendix A.
The conceptual framework we used also makes these evaluations vulnerable to the bias of the raters. An alternative framework has been described by Tromp and Baltussen  that classifies criteria according to two broad categories – ‘health system goals’ and ‘health system building blocks’ – as well as a number of subcategories. There is a great deal of overlap between the Tanios et al. framework and the one proposed by Tromp and Baltussen, and there is a level of arbitrariness in choosing one rather than the other. Perhaps importantly, the Tromp and Baltussen framework specifically does not consider “Quality of Evidence” as an independent criterion, rather treating it as a component that underlies all criteria. Given that some type of evidence quality criterion was present thirteen times in the studies in this review, we feel that the Tanios framework accurately encapsulates the terms that health care decision makers have used to describe their work.
It is worth noting that health care decisions are often made on an ad hoc basis rather than using a specific framework like MCDA or PBMA – our review includes all published studies, which likely overrepresents the frequency with which these two approaches are used. It is unlikely that our search comprises an ‘exhaustive’ review of all criteria-based priority setting exercises in health care. As many entities within the UK’s National Health Service (NHS) have adopted PBMA as part of its decision-making process, more exercises are likely either in progress or completed without publicly-available documentation. Similarly, successes of MCDA as a priority setting method in developing countries will likely yield further use of the process in future decision-making. While not a comprehensive review of all such exercises, this review does provide insight into the types of criteria that decision-making bodies consider important when allocating scarce resources.
The authors of this review are reliable advocates of PBMA as a priority setting exercise, and are named as authors on several of the studies included in this review. One advantage of using PBMA is that budgetary analysis is an intrinsic component of the priority setting process, which allows for the important process of disinvestment – the reduction of funding to programs that do not deliver acceptable value for money. As health systems and regions in various parts of the world face pressure to control, or in some cases reduce the size of their budgets, disinvestment becomes an increasingly important component of health care priority setting. As described in Table 1, several studies were able to find ways to disinvest as a method of reallocating scarce funds – all of these studies used PBMA.
A review conducted by Guindo et al.  explored the criteria used in resource allocation decision-making; however, the Guindo et al. review included focus groups and other activities not explicitly tied to a particular resource allocation decision, rather than those decisions made in a ‘real-world’ setting. As in our review, the criteria of equity, effectiveness, organizational requirements, and availability of cost-effectiveness literature were commonly cited as important. An important difference between the two studies, however, is that “stakeholder interests & pressures” was a highly influential criterion in the Guindo review, and not in ours. This may suggest that, though they identify it as important, decision-makers are less likely to include stakeholder input in actual decision-making exercises tied to explicit funding decisions. Of course, the overlap of papers was not great and thus there could also be other reasons for this difference including the setting and/ or individuals involved in the given work. Further investigation should be conducted into the use of stakeholder input into decision-making processes, to see how, and at what level, such preferences are incorporated into health care decision making.
This review contributes to the scientific literature in two important ways: first, it includes much of the ‘grey’ literature that is not present in previously-published reviews of this field. Because a great deal of health care decision-making is done outside the context of peer-reviewed journals, it is critical for researchers and decision-makers alike to be able to draw from as broad a set of examples as possible when trying to decide what sort of criteria are relevant to their particular context of interest. Secondly, by focusing on health care resource allocation decisions that have been made, as opposed to hypothetical exercises, our review reflects a set of decision-making criteria that is more ‘pragmatic’ than the more ‘aspirational’ criteria that may be generated in the abstract.
This review points to the criteria that are most important to health care decision makers in actual policy-setting environments, and builds on previous reviews that included hypothetical decision criteria. It is important to recognize that each priority-setting environment has its own unique challenges, and the criteria used will reflect this heterogeneity. However, this review does suggest some convergence in those criteria that are most frequently used in a ‘real-world’ setting.
Multi-Criteria Decision Analysis
Programme Budgeting and Marginal Analysis
Peacock S, Ruta D, Mitton C, Donaldson C, Bate A, Murtagh M. Using economics to set pragmatic and ethical priorities. BMJ. 2006;332(7539):482–5.
Guindo LA, Wagner M, Baltussen R, Rindress D, van Til J, Kind P, et al. From efficacy to equity: Literature review of decision criteria for resource allocation and healthcare decisionmaking. Cost Effective Resour Alloc. 2012;10(1):9.
Smith N, Mitton C, Peacock S, Cornelissen E, MacLeod S. Identifying research priorities for health care priority setting: a collaborative effort between managers and researchers. BMC Health Serv Res. 2009;9:165.
Kapiriri L, Norheim OF, Martin DK. Fairness and accountability for reasonableness. Do the views of priority setting decision makers differ across health systems and levels of decision making? Soc Sci Med. 2009;68(4):766–73.
Mitton C, Donaldson C. Health care priority setting: principles, practice and challenges. Cost Effective Resour Alloc. 2004;2(1):3.
Patten S, Mitton C, Donaldson C. From the trenches: views from decision-makers on health services priority setting. Health Serv Manage Res. 2005;18(2):100–8.
Peacock SJ, Richardson JR, Carter R, Edwards D. Priority setting in health care using multi-attribute utility theory and programme budgeting and marginal analysis (PBMA). Soc Sci Med. 2007;64(4):897–910.
Mitton C, Donaldson C. Tools of the trade: a comparative analysis of approaches to priority setting in healthcare. Health Serv Manage Res. 2003;16(2):96–105.
Mitton C, Donaldson C. Priority setting toolkit: a guide to the use of economics in healthcare decision making. London, UK: BMJ Books; 2004.
Baltussen R, Niessen L. Priority setting of health interventions: the need for multi-criteria decision analysis. Cost Effective Resour Alloc. 2006;4:14.
Peacock S, Mitton C, Bate A, McCoy B, Donaldson C. Overcoming barriers to priority setting using interdisciplinary methods. Health Policy. 2009;92(2–3):124–32.
Devlin NJ, Sussex J. Incorporating multiple criteria in HTA: methods and processes. Whitehall, London, UK: Office of Health Economics; 2011.
Thokala P, Duenas A. Multiple criteria decision analysis for health technology assessment. Value Health. 2012;15(8):1172–81.
Tanios N, Wagner M, Tony M, Baltussen R, Til Jv, Rindress D, et al. Which criteria are considered in healthcare decisions? Insights from an international survey of policy and clinical decision makers. Int J Technol Assess Health Care. 2013; e-pub ahead of print.
Mitton C, Donaldson C. Twenty-five years of programme budgeting and marginal analysis in the health sector, 1974–1999. J Health Serv Res Policy. 2001;6(4):239–48.
Mitton C, Patten S, Donaldson C, Waldner H. Priority-setting in health authorities: moving beyond the barriers. The Calgary experience. Healthc Q. 2005;8(3):49–55.
Ruta D, Mitton C, Bate A, Donaldson C. Programme budgeting and marginal analysis: bridging the divide between doctors and managers. BMJ. 2005;330(7506):1501–3.
Gibson J, Mitton C, Martin D, Donaldson C, Singer P. Ethics and economics: does programme budgeting and marginal analysis contribute to fair priority setting? J Health Serv Res Policy. 2006;11(1):32–7.
Youngkong S, Kapiriri L, Baltussen R. Setting priorities for health interventions in developing countries: a review of empirical studies. Trop Med Int Health. 2009;14(8):930–9.
Kapiriri L, Martin DK. Successful priority setting in low and middle income countries: a framework for evaluation. Health Care Anal. 2010;18(2):129–47.
Tromp N, Baltussen R. Mapping of multiple criteria for priority setting of health interventions: an aid for decision makers. BMC Health Serv Res. 2012;12:454.
Astley J, Wake-Dyster W. Evidence-based priority setting. Aust Health Rev. 2001;24(2):32–9.
Baltussen R, ten Asbroek AH, Koolman X, Shrestha N, Bhattarai P, Niessen LW. Priority setting using multiple criteria: should a lung health programme be implemented in Nepal? Health Policy Plan. 2007;22(3):178–85.
Baughan S, Fergunson B: Road testing Programme Budgeting and Marginal Analysis (PBMA) in three English Regions: Hull Teaching Primary Care Trust Diabetes pilot project. In. Edited by England PH. Yorkshire, UK; 2008.
Bohmer P, Pain C, Watt A, Abernethy P, Sceats J. Maximising health gain within available resources in the New Zealand public health system. Health Policy. 2001;55(1):37–50.
Defechereux T, Paolucci F, Mirelman A, Youngkong S, Botten G, Hagen TP, et al. Health care priority setting in Norway a multicriteria decision analysis. BMC Health Serv Res. 2012;12:39.
Dionne F, Mitton C, Smith N, Donaldson C. Evaluation of the impact of program budgeting and marginal analysis in Vancouver Island Health Authority. J Health Serv Res Policy. 2009;14(4):234–42.
Goetghebeur MM, Wagner M, Khoury H, Rindress D, Gregoire JP, Deal C. Combining multicriteria decision analysis, ethics and health technology assessment: applying the EVIDEM decision-making framework to growth hormone for Turner syndrome patients. Cost Effective Resour Alloc. 2010;8:4.
Grocott R. Applying Programme Budgeting Marginal Analysis in the health sector: 12 years of experience. Expert Rev Pharmacoecon Outcomes Res. 2009;9(2):181–7.
Halma L, Mitton C, Donaldson C, West B. Case study on priority setting in rural Southern Alberta: keeping the house from blowing in. Can J Rural Med. 2004;9(1):26–36.
Honore PA, Fos PJ, Smith T, Riley M, Kramarz K. Decision science: a scientific approach to enhance public health budgeting. J Public Health Manag Pract. 2010;16(2):98–103.
Iqbal Z, Pryce A, Afza M. Rationalizing rationing in health care: experience of two primary care trusts. J Public Health (Oxf). 2006;28(2):125–32.
Jehu-Appiah C, Baltussen R, Acquah C, Aikins M, d’Almeida SA, Bosu WK, et al. Balancing equity and efficiency in health priorities in Ghana: the use of multicriteria decision analysis. Value Health. 2008;11(7):1081–7.
Johnson-Masotti AP, Eva K. A decision-making framework for the prioritization of health technologies. In: Beach CM, Chaykowsi RP, Shortt S, St-Hillaire F, Sweetman A, editors. Health services restructuring in Canada: new evidence and new directions. Kingson, Ontario, Canada: Queen’s University Policy Studies; 2006.
Kemp L, Fordham R: Road Testing Programme Budgeting and Marginal Analysis (PBMA) in three English Regions: Norfolk Mental Health PBMA Pilot Project. In. Edited by England PH. Norfolk, UK: University of East Anglia Norwich; 2007.
Kreng VB, Yang CT. The equality of resource allocation in health care under the National Health Insurance System in Taiwan. Health Policy. 2011;100(2–3):203–10.
Lee CW, Kwak NK. Strategic enterprise resource planning in a health-care system using a multicriteria decision-making model. J Med Syst. 2011;35(2):265–75.
Makundi E, Kapiriri L, Norheim OF. Combining evidence and values in priority setting: testing the balance sheet method in a low-income country. BMC Health Serv Res. 2007;7:152.
Marsh K, Dolan P, Kempster J, Lugon M. Prioritizing investments in public health: a multi-criteria decision analysis. J Public Health (Oxf) 2012.
Miot J, Wagner M, Khoury H, Rindress D, Goetghebeur MM. Field testing of a multicriteria decision analysis (MCDA) framework for coverage of a screening test for cervical cancer in South Africa. Cost Effective Resour Alloc. 2012;10(1):2.
Mitton C, Patten S, Waldner H, Donaldson C. Priority setting in health authorities: a novel approach to a historical activity. Soc Sci Med. 2003;57(9):1653–63.
Mitton C, Mackenzie J, Cranston L, Teng F. Priority setting in the Provincial Health Services Authority: case study for the 2005/06 planning cycle. Healthcare Policy. 2006;2(1):91–106.
Mitton C, Dionne F, Damji R, Campbell D, Bryan S. Difficult decisions in times of constraint: criteria based resource allocation in the Vancouver Coastal Health Authority. BMC Health Serv Res. 2011;11:169.
Mitton C, Levy A, Gorsky D, Macneil C, Dionne F, Marrie T. Allocating limited resources in a time of fiscal constraints: a priority setting case study from Dalhousie university faculty of medicine. Acad Med. 2013;88(7):939–45.
Nuti S, Vainieri M, Bonini A. Disinvestment for re-allocation: a process to identify priorities in healthcare. Health Policy. 2010;95(2–3):137–43.
Poulin P, Austen L, Kortbeek JB, Lafreniere R. New technologies and surgical innovation: five years of a local health technology assessment program in a surgical department. Surg Innov. 2012;19(2):187–99.
Poulin P, Austen L, Scott CM, Waddell CD, Dixon E, Poulin M, et al. Multi-criteria development and incorporation into decision tools for health technology adoption. J Health Organ Manag. 2013;27(2):246–65.
Robson A, Bate A, Donaldson C: Road Testing PBMA in three English regions: Newcastle CAMHS PBMA pilot project. In. Edited by England H. Newcastle, UK; 2008.
Urquhart B, Mitton C, Peacock S. Introducing priority setting and resource allocation in home and community care programs. J Health Serv Res Policy. 2008;13 Suppl 1:41–5.
Wilson EC, Rees J, Fordham RJ. Developing a prioritisation framework in an English Primary Care Trust. Cost Effective Resour Alloc. 2006;4:3.
Wilson E, Sussex J, Macleod C, Fordham R. Prioritizing health technologies in a Primary Care Trust. J Health Serv Res Policy. 2007;12(2):80–5.
Youngkong S, Teerawattananon Y, Tantivess S, Baltussen R. Multi-criteria decision analysis for setting priorities on HIV/AIDS interventions in Thailand. Health Res Policy Syst. 2012;10:6.
Youngkong S, Baltussen R, Tantivess S, Mohara A, Teerawattananon Y. Multicriteria decision analysis for including health interventions in the universal health coverage benefit package in Thailand. Value Health. 2012;15(6):961–70.
The authors wish to acknowledge the support of their colleagues at ARCC and C2E2.
The Canadian Centre for Applied Research in Cancer Control (ARCC) is funded by the Canadian Cancer Society Research Institute grant #019789.
The authors declare that they have no competing interests.
IC was responsible for the manual searching of the scientific literature and made the primary argument for inclusion/exclusion of studies. IC performed the analysis of the data and was primarily responsible for the preparation of the manuscript. SJP conceived of the study, was responsible for screening all eligible studies and determining their eligibility. SJP approved all analyses, provided material input to the interpretation of the findings, and was responsible for the preparation of the manuscript. CM was responsible for screening all eligible studies and determining their eligibility. The search strategy used in this study was based on work previously conducted by CM and colleagues. CM approved all analyses, provided material input to the interpretation of the findings, and was responsible for the preparation of the manuscript. All authors read and approved the final manuscript.
SJP and CM are both longtime health services researchers specializing in criteria-guided decision making, particularly PBMA.
About this article
Cite this article
Cromwell, I., Peacock, S.J. & Mitton, C. ‘Real-world’ health care priority setting using explicit decision criteria: a systematic review of the literature. BMC Health Serv Res 15, 164 (2015). https://doi.org/10.1186/s12913-015-0814-3