Skip to main content
  • Research article
  • Open access
  • Published:

‘Real-world’ health care priority setting using explicit decision criteria: a systematic review of the literature

Abstract

Background

Health care decision making requires making resource allocation decisions among programs, services, and technologies that all compete for a finite resource pool. Methods of priority setting that use explicitly defined criteria can aid health care decision makers in arriving at funding decisions in a transparent and systematic way. The purpose of this paper is to review the published literature and examine the use of criteria-based methods in ‘real-world’ health care allocation decisions.

Methods

A systematic review of the published literature was conducted to find examples of ‘real-world’ priority setting exercises that used explicit criteria to guide decision-making.

Results

We found thirty-three examples in the peer-reviewed and grey literature, using a variety of methods and criteria. Program effectiveness, equity, affordability, cost-effectiveness, and the number of beneficiaries emerged as the most frequently-used decision criteria. The relative importance of criteria in the ‘real-world’ trials differed from the frequency in preference elicitation exercises. Neither the decision-making method used, nor the relative economic strength of the country in which the exercise took place, appeared to have a strong effect on the type of criteria chosen.

Conclusions

Health care decisions are made based on criteria related both to the health need of the population and the organizational context of the decision. Following issues related to effectiveness and affordability, ethical issues such as equity and accessibility are commonly identified as important criteria in health care resource allocation decisions.

Peer Review reports

Background

Health care decision making requires the balancing of the demand for programs, services, and technologies that improve human health, and the need for fiscal restraint and the reality of a finite resource pool. All health care decision making involves making choices between many attractive alternatives, and saying ‘no’ to some things that might be desirable and valuable.

Because health care resources are, in many jurisdictions, provided through public subsidy, health care decision makers have an ethical obligation to allocate those resources in a way that is fair, transparent, and accountable [1]. Further, health care decisions should be made according to the available evidence relating to a number of possible decision making criteria, including effectiveness, cost-effectiveness, equity considerations, feasibility, affordability etc. of the proposed program, service, or technology. But allocation decisions are also influenced by a number of factors other than medical and health economic evidence – pragmatic issues of organizational structure and political realities are legitimate and important components of these decisions [2-6] and must be considered as well.

Two popular proposed methods for guiding the setting of health care resource allocation priorities are Programme Budgeting and Marginal Analysis (PBMA) and Multi-Criteria Decision Analysis (MCDA). PBMA involves the listing of all relevant activities and their resource requirements, the evaluation of the effectiveness of these activities according to a set of explicit criteria, and the application of that evaluation to the available budget [7]. The PBMA process can be summarized by asking five questions [8]:

  1. 1.

    What resources are available in total?

  2. 2.

    In what ways are these resources currently spent?

  3. 3.

    What are the main candidates for more resources and what would be their level of effectiveness?

  4. 4.

    Are there any areas of care which could be provided to the same level of effectiveness but with fewer resources to fund candidates from (3) (i.e., addressing technical efficiency)?

  5. 5.

    Are there areas of care which, despite being effective, should receive fewer resources because a proposal from (3) is more effective (per dollar spent) (i.e., addressing allocative efficiency)?

PBMA exercises are commonly conducted by an advisory panel of expert stakeholders, and should be accompanied by an evaluation of the outputs to ensure allegiance with the priorities and needs of the organization [9].

MCDA involves the numerical quantification of the merit of competing options, according to explicit decision criteria [10]. The primary aim of MCDA is to develop models of decision maker objectives and their value trade-offs so that options under consideration can be compared with each other in a consistent and transparent manner [11]. A key principle is that decisions between different options (for example different interventions) should be consistent with stakeholders’ objectives. MCDA is transparent in that it shows that decisions are the logical implications of those objectives. In MCDA, objectives are deemed to be within the discretion of the decision-makers. That is, they are not predetermined by some underlying theory from economics or ethics (such as utilitarianism).

MCDA then typically consists of two overarching stages [10,11]. First, problem structuring involves generating a set of alternatives and a set of criteria against which the alternatives are to be evaluated and compared. In order to structure the problem, the first questions to ask are ‘what priority setting objectives do decision-makers wish to pursue?’ And, ‘what locally relevant criteria do decision-makers use when deciding between alternative interventions?’ Objectives are the principles that determine priority setting policies (e.g. improving population health) whereas criteria are the standards that alternative interventions are judged by (e.g. health outcomes from different treatments). Second, model building entails constructing some form of model which represents decision-makers’ objectives and their value judgements. There are then two key considerations to be addressed in this type of model [10,11]: the methods used to describe decision-makers preferences and elicit importance weights for decision-making criteria; and, the type of aggregation model used to combine criteria scores (see Peacock 2009 for more detail).

Similar to PBMA, MCDA is conducted with the input of decision makers and relevant stakeholders in the ultimate resource allocation decision [10]. In addition, MCDA can be employed within the broader PBMA approach as the mechanism for benefit measurement to inform allocation or re-allocation recommendations. As such, these approaches in our view are best viewed as complementary.

In a time when organizations are adopting explicit criteria-based decision methods and deciding which method is the best fit for their organization [12,13], it is important to examine the criteria that have been used in previous decisions. While reviews of the priority-setting literature have been conducted [2,14], these studies have focused on hypothetical exercises and stated preferences rather than cases where decision-making bodies have had to make decisions under the actual constraints of budgetary and political realities.

The purpose of this paper is to summarize the available literature on health care decision-making where explicit criteria-based methods like PBMA or MCDA were used (i.e., a set list of factors were weighed against each other according to some underlying framework), in order to examine the criteria used by decision makers in ‘real world’, rather than hypothetical, settings. The added value of this paper is that our review, based on significant experience in this field over the past 12 years, focused on ‘real-world’ priority setting using explicit decision criteria. These examples may be interpreted as a better predictor of future health care priority setting than decisions made in the abstract.

Methods

A systematic review of the literature was conducted, based on English language search terms used in a previously published review of this literature [15]. Relevant subject headings were abstracted from the articles included in that review: “Decision Making, organizational”; “Health Care rationing”; “Health Priorities”; “Budgets”; and “Community Health Planning”.

To broaden the scope of our search to ensure inclusion of exercises using popular decision-making methods, the terms “Program (me) Budget”, “Marginal Analysis”, and “Multi-Criteria Decision Analysis” as well as their respective acronyms (“PBMA”, “MCDA”) were also added to the search terms.

The relevant search terms were used to search the MEDLINE, ECONLit, and PAIS databases. The search terms were also entered into the Google™ search engine to investigate the presence of ‘grey literature’ (i.e., non peer-reviewed publications). Retrieval was limited to documents published between January 1st, 2000 and July 31st, 2013, to reflect the relevant time period since the previous literature review.

Articles were identified as potentially eligible based on a review of their abstracts. Potentially eligible items underwent closer examination based on exclusion criteria. Items were excluded if they did not meet the following description: a description of a priority setting exercise in which a funding decision was made based on a set of explicitly-defined criteria. Articles published in languages other than English were not included in this review.

The reference lists of all items (i.e., excluded and included) were manually searched for potentially-eligible items. The reference-identified items underwent the same scrutiny and application of exclusion criteria. The search strategy is diagrammed in Figure 1. One reviewer (IC) conducted the review and data abstraction from all studies. Two other reviewers (SJP and CM) verified the accuracy of the search process and the application of the exclusion criteria. Disagreements about eligibility were resolved through consensus. Because studies included in this exercise were descriptive rather than quantitative, and because of the nature of the research question, no formal assessment of the quality or bias of included items was undertaken. Because this research did not involve human participants, no Research Ethics approval was sought or gained.

Figure 1
figure 1

Search strategy and results.

Results

A total of sixty-five articles were identified as potentially eligible from the indexed databases. An additional fifty-two were identified from the manual reference search and 25 from web search, for a total of 142 potentially eligible items. Manual search from the grey literature did not yield any additional unique items. After applying the exclusion criteria, thirty-five articles were identified as eligible. Two peer-reviewed items were duplicates of items from the grey literature and were deemed redundant, bringing the total number of included items to thirty-three.

Study characteristics

Characteristics of the included studies, including the setting, method of decision-making used, and the result of the exercise (e.g., a change in policy, a list of priorities, etc.), are presented in Table 1. Most exercises were conducted in North American or European or European-descended (“Western”) countries, primarily Canada (10) and the United Kingdom (7). There was equal representation of studies using an MCDA framework (15) as using PBMA (15). Some studies did not explicitly state the decision-making method, or used a synonymous term (e.g., “Decision Science”), and were classified as PBMA or MCDA based on the characteristics of the method used (e.g., generic vs. algorithmic methods of criteria weighting and decision ranking, inclusion of budgetary information, etc.).

Table 1 Summary of included items

The most common outcome of MCDA exercises was a ranked list of alternatives, rather than an explicit funding decision of the type that was more common among exercises that used PBMA. Allocation of points, whereby an agreed-upon number of points can be assigned to different categories, was a common method of assigning weights to decision criteria. Many MCDA exercises used Discrete Choice experiments to elicit criteria weights. A number of studies did not describe any weighting method, and may have simply considered all criteria equally important.

Choice of decision-making criteria

The decision-making criteria used in each included item were identified and extracted. Where possible, criteria with identical/similar definitions across different studies were collapsed into a single criterion (by IC, decisions reviewed by SJP and CM), to make direct comparison between exercises more possible. A total of seventy-two unique criteria were identified among the included items. These criteria are listed in Additional file 1: Appendix A. The most common criteria were:

  • The effectiveness of the program (21 items)

  • Budgetary impact/Affordability (16 items)

  • Reducing inequalities between groups/”Equity” (14 items)

  • Number of people likely to benefit from program/intervention (13 items)

  • Ability to access the program/intervention (13 items)

  • Cost-effectiveness or other health economics evidence (12 items)

  • Quality of the available evidence (10 items)

It is important to note that, because of variations in the way in which criteria were described, it may be possible that some criteria are ‘redundant’, insofar as some may simply be more specific iterations of others. For example, the criteria of “poverty reduction” and “age/risk of target group” could both be collapsed into the “Equity” criterion. A conservative approach was taken to combining criteria in this way, preferring to list criteria individually in cases where there was ambiguity about whether or not terms were truly synonymous. Similarly, the classification of a given criterion into a domain was prone to subjectivity – criteria were not always well-described in the text of the exercise. We again used a conservative approach for this classification process, and resolved ambiguities through consensus.

It should also be noted that many studies listed broad criteria that contained a number of sub-criteria within them (e.g., one study listed an “effectiveness” criterion that included “number of patients”, “individual benefit”, “magnitude of benefit”, “duration of benefit”, “personal networks”, “collective benefits”, “population impact”, and “social capital” as included sub-criteria). In cases where criteria weights were not provided for each sub-criterion, the broad criteria were preferred.

Domains of decision-making criteria

Criteria used on priority setting exercises were classified into ten descriptive ‘domains’, using a classification system described by Tanios and colleagues [14]:

  1. a.

    Intervention outcomes and Benefits

  2. b.

    Type of Health Service

  3. c.

    Disease Impact (burden)

  4. d.

    Therapeutic Context

  5. e.

    Economic Impact

  6. f.

    Environmental Impact of the Intervention

  7. g.

    Quality/Uncertainty of Evidence

  8. h.

    Implementation Complexity

  9. i.

    Priorities (fairness)

  10. j.

    Overall Context

A full list of all criteria included in each domain is provided in Additional file 1: Appendix A. A proportional breakdown of each domain is provided in Figure 2. Economic Impact and Intervention Outcomes/Benefits were the two most frequently-cited domains, followed by Overall Context, Disease Impact (burden), and Priorities (fairness). Only two of the included items included a Miscellaneous category (i.e., a category labeled “miscellaneous” in the report/manuscript itself).

Figure 2
figure 2

Decision criteria by domain.

Domain by type of exercise

It is possible that the types of criteria used may differ due to the differences in the setting and methodology between PBMA and MCDA exercises. The frequency of criteria use within each domain was calculated for each type of approach. Results are presented in Figure 3. Overall, the types of criteria used were similar across the exercise types. Differences between the proportion of domain in each exercise were not significant (two-tailed t-test; α = 0.05).

Figure 3
figure 3

Criteria domain frequency by priority-setting method (PBMA vs. MCDA).

Domain by country of origin

It is similarly possible that the size of the national economy affects the types of criteria used in health care decisions. Exercises in our study were grouped based on membership in the G7 group of countries (Canada, France, Germany, Italy, Japan, the United Kingdom, and the United States). The frequency of criteria use within each domain was calculated for exercises conducted in G7 and non-G7 countries. Once again, the frequency of criteria was largely similar between the two groups. Exercises performed in non-G7 countries were more likely to consider criteria in the Disease Impact (Burden) domain than those in G7 countries (two-tailed t test, p = 0.002); conversely, exercises in G7 countries were more likely to consider criteria in the Overall Context domain (p = 0.006) – other differences were not statistically significant. Results are presented in Figure 4.

Figure 4
figure 4

Criteria domain frequency by size of national economy (G7 vs. non-G7).

Discussion

Our search yielded 33 distinct ‘real-world’ priority setting exercises conducted using explicit decision-making criteria. Decisions were made largely along common categories of criteria that included the likely impact of a program/intervention, the ability of a program/intervention to address inequalities, and the political/organizational realities of the entities that are making funding decisions.

Summary of main findings

Our review found that while health-specific decision making criteria are of primary importance in priority setting exercises, criteria relating to organizational and political considerations (i.e., criteria that fall primarily into the “Implementation Complexity” and “Overall Context” domains) are also important elements in decision making. Health care delivery must be guided by the available biomedical evidence, but as it is still a human endeavour, political realities must be considered and weighed. Decision makers should be forthcoming about the need to balance the goal of improving human health with the pragmatic constraints all health systems are under.

It is worth noting that fewer than half of the items included in this review explicitly included cost-effectiveness evidence in their decision-making process. While it is certainly true that costs and outcomes are considered in other criteria (i.e., Affordability, Effectiveness of program), incremental cost-effectiveness is not a component of these analyses. Health economic evidence is uniquely suited to provide valuable information when making decisions about scarce resources, but was not explicitly considered in the majority of the identified exercises. This finding is in line with a growing body of research suggesting that while decision-makers would like to use health economic evidence, they may not be able to do so [16,17]. Health economists and policy makers must continue to work together to determine the best way of making health economic findings more policy-relevant and usable.

It is valuable to note that, after “effectiveness of the program” and “affordability”, issues related to equity and fairness were foremost in the decision-making process. This suggests that, at least where decision criteria are made explicit, decision makers are concerned with addressing systemic inequalities and providing health care on a ‘by-need’ basis, underlining the important role of ethics in health care decision making [4,18].

We found that, broadly speaking, the studies in this review fell equally into two categories: MCDA exercises and PBMA exercises. It is important to note that a number of studies did not explicitly categorize themselves as either PBMA or MCDA, for example in one case using the term “Decision Science” to describe the methodology. In these cases, we classified all studies in this review as either PBMA or MCDA based on the methods described in the publication. It is conceivable, therefore, that other reviewers may classify them differently. Our discussion of the contrast between “MCDA exercises” and “PBMA exercises” should be read with this caveat in mind.

The primary outcome of the MCDA items included in this review was the prioritization of alternatives. It is not clear from the manuscripts themselves whether the priorities identified through the MCDA process became official policy, rather than simply a set of recommendations. By way of contrast, the PBMA studies often refer explicitly to investment/disinvestment actions taken as a result of the exercise. It may be valuable to policy-makers to know, when deciding what priority setting approach they prefer, how the approach translates into actual policy decisions.

Given that the resource constraints in developing economies are greater than those of countries with larger budgets, the use of criteria-guided priority setting is of particular importance in countries with developing economies [19,20]. MCDA was used in a number of countries with developing economies, whereas PBMA was used exclusively in countries with developed economies (especially Canada and the UK). The preference for MCDA appears to be due in part to the effort of investigators associated with Dr. Rob Baltussen, who is listed as an author on four of the seven relevant MCDA exercises. In parallel, application of PBMA has tended to be associated with a handful of applied researchers working in the UK, Australia and Canada. While the socio-geographic disparity seen in our review is likely to be a simple reflection of this authorship ‘clustering’ effect, it may be worthwhile to investigate whether there are barriers or relevant factors to the use of other decision-making techniques in these countries; whether there is a true difference in the suitability of PBMA or MCDA in particular political/financial resource environments.

We looked for differences in the type of criteria used in PBMA vs. MCDA exercises, as well as in G7 countries vs. the rest of the world. Care should be taken, however, in extrapolating from these results. The small number of available studies and the lack of a consistent definition of criteria mean that such comparisons are inherently difficult to make. Researchers may wish to investigate the extent to which decision-makers in wealthy countries face different pressures when allocating health resources than those faced by decision-makers in less wealthy countries. Our findings suggest that both PBMA and MCDA can incorporate the criteria that are most important to the local context, and that the decision-making method should be chosen independently of the chosen criteria, based on the feasibility and applicability of a given method to that context.

Regardless of the method of priority setting used, the use of explicit criteria is valuable for all levels of health care decision making. Because no health care system can fund all possible alternatives, decision makers have an ethical obligation to act as good stewards of finite health care resources, and should be accountable to the communities they serve.

Limitations

Our search criteria were based on a previously-published review of PBMA exercises. As a result, it is possible that the terms are biased toward one decision-making method at the expense of others. We intentionally used MeSH terms and included an additional term that specifies MCDA in order to counteract any potential bias toward PBMA. We also used manual searches of the reference section of all papers included in the review, in order to ensure that as wide a ‘net’ was used in the search. It is worth noting that while the majority of the items included in this review were found using generic search terms (e.g., “Decision Making”, “Health Priorities”, etc.), we did supplement our search to use PBMA and MCDA-specific terms as well. It is possible, therefore, that our review is biased toward these methods. Regardless, we believe that this exercise represents a fair encapsulation of the status of the literature.

We chose to classify our criteria according to a previously-published conceptual framework [14]; however, our classification was subject to our own interpretation – criteria were not listed in consistent language, requiring us to make our own decisions. It is possible that our interpretations were biased, and that other raters would classify a given criterion differently. It is additionally possible that an interested reader would classify or combine the listed criteria differently (e.g., “efficiency” and “appropriate use of resources” may be seen as identical criteria, though this review counts them separately). The conservative approach we used – preferring to list criteria verbatim to reduce rater bias – leaves a great deal of subjectivity up to the reader. Accordingly, we have listed the criteria used in this analysis, as well as the way in which the criteria were assigned to domains, in Additional file 1: Appendix A.

The conceptual framework we used also makes these evaluations vulnerable to the bias of the raters. An alternative framework has been described by Tromp and Baltussen [21] that classifies criteria according to two broad categories – ‘health system goals’ and ‘health system building blocks’ – as well as a number of subcategories. There is a great deal of overlap between the Tanios et al. framework and the one proposed by Tromp and Baltussen, and there is a level of arbitrariness in choosing one rather than the other. Perhaps importantly, the Tromp and Baltussen framework specifically does not consider “Quality of Evidence” as an independent criterion, rather treating it as a component that underlies all criteria. Given that some type of evidence quality criterion was present thirteen times in the studies in this review, we feel that the Tanios framework accurately encapsulates the terms that health care decision makers have used to describe their work.

It is worth noting that health care decisions are often made on an ad hoc basis rather than using a specific framework like MCDA or PBMA – our review includes all published studies, which likely overrepresents the frequency with which these two approaches are used. It is unlikely that our search comprises an ‘exhaustive’ review of all criteria-based priority setting exercises in health care. As many entities within the UK’s National Health Service (NHS) have adopted PBMA as part of its decision-making process, more exercises are likely either in progress or completed without publicly-available documentation. Similarly, successes of MCDA as a priority setting method in developing countries will likely yield further use of the process in future decision-making. While not a comprehensive review of all such exercises, this review does provide insight into the types of criteria that decision-making bodies consider important when allocating scarce resources.

Further discussion

The authors of this review are reliable advocates of PBMA as a priority setting exercise, and are named as authors on several of the studies included in this review. One advantage of using PBMA is that budgetary analysis is an intrinsic component of the priority setting process, which allows for the important process of disinvestment – the reduction of funding to programs that do not deliver acceptable value for money. As health systems and regions in various parts of the world face pressure to control, or in some cases reduce the size of their budgets, disinvestment becomes an increasingly important component of health care priority setting. As described in Table 1, several studies were able to find ways to disinvest as a method of reallocating scarce funds – all of these studies used PBMA.

A review conducted by Guindo et al. [2] explored the criteria used in resource allocation decision-making; however, the Guindo et al. review included focus groups and other activities not explicitly tied to a particular resource allocation decision, rather than those decisions made in a ‘real-world’ setting. As in our review, the criteria of equity, effectiveness, organizational requirements, and availability of cost-effectiveness literature were commonly cited as important. An important difference between the two studies, however, is that “stakeholder interests & pressures” was a highly influential criterion in the Guindo review, and not in ours. This may suggest that, though they identify it as important, decision-makers are less likely to include stakeholder input in actual decision-making exercises tied to explicit funding decisions. Of course, the overlap of papers was not great and thus there could also be other reasons for this difference including the setting and/ or individuals involved in the given work. Further investigation should be conducted into the use of stakeholder input into decision-making processes, to see how, and at what level, such preferences are incorporated into health care decision making.

This review contributes to the scientific literature in two important ways: first, it includes much of the ‘grey’ literature that is not present in previously-published reviews of this field. Because a great deal of health care decision-making is done outside the context of peer-reviewed journals, it is critical for researchers and decision-makers alike to be able to draw from as broad a set of examples as possible when trying to decide what sort of criteria are relevant to their particular context of interest. Secondly, by focusing on health care resource allocation decisions that have been made, as opposed to hypothetical exercises, our review reflects a set of decision-making criteria that is more ‘pragmatic’ than the more ‘aspirational’ criteria that may be generated in the abstract.

Conclusions

This review points to the criteria that are most important to health care decision makers in actual policy-setting environments, and builds on previous reviews that included hypothetical decision criteria. It is important to recognize that each priority-setting environment has its own unique challenges, and the criteria used will reflect this heterogeneity. However, this review does suggest some convergence in those criteria that are most frequently used in a ‘real-world’ setting.

Abbreviations

MCDA:

Multi-Criteria Decision Analysis

PBMA:

Programme Budgeting and Marginal Analysis

References

  1. Peacock S, Ruta D, Mitton C, Donaldson C, Bate A, Murtagh M. Using economics to set pragmatic and ethical priorities. BMJ. 2006;332(7539):482–5.

    Article  PubMed  PubMed Central  Google Scholar 

  2. Guindo LA, Wagner M, Baltussen R, Rindress D, van Til J, Kind P, et al. From efficacy to equity: Literature review of decision criteria for resource allocation and healthcare decisionmaking. Cost Effective Resour Alloc. 2012;10(1):9.

    Article  Google Scholar 

  3. Smith N, Mitton C, Peacock S, Cornelissen E, MacLeod S. Identifying research priorities for health care priority setting: a collaborative effort between managers and researchers. BMC Health Serv Res. 2009;9:165.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Kapiriri L, Norheim OF, Martin DK. Fairness and accountability for reasonableness. Do the views of priority setting decision makers differ across health systems and levels of decision making? Soc Sci Med. 2009;68(4):766–73.

    Article  PubMed  Google Scholar 

  5. Mitton C, Donaldson C. Health care priority setting: principles, practice and challenges. Cost Effective Resour Alloc. 2004;2(1):3.

    Article  Google Scholar 

  6. Patten S, Mitton C, Donaldson C. From the trenches: views from decision-makers on health services priority setting. Health Serv Manage Res. 2005;18(2):100–8.

    Article  PubMed  Google Scholar 

  7. Peacock SJ, Richardson JR, Carter R, Edwards D. Priority setting in health care using multi-attribute utility theory and programme budgeting and marginal analysis (PBMA). Soc Sci Med. 2007;64(4):897–910.

    Article  PubMed  Google Scholar 

  8. Mitton C, Donaldson C. Tools of the trade: a comparative analysis of approaches to priority setting in healthcare. Health Serv Manage Res. 2003;16(2):96–105.

    Article  CAS  PubMed  Google Scholar 

  9. Mitton C, Donaldson C. Priority setting toolkit: a guide to the use of economics in healthcare decision making. London, UK: BMJ Books; 2004.

    Google Scholar 

  10. Baltussen R, Niessen L. Priority setting of health interventions: the need for multi-criteria decision analysis. Cost Effective Resour Alloc. 2006;4:14.

    Article  Google Scholar 

  11. Peacock S, Mitton C, Bate A, McCoy B, Donaldson C. Overcoming barriers to priority setting using interdisciplinary methods. Health Policy. 2009;92(2–3):124–32.

    Article  PubMed  Google Scholar 

  12. Devlin NJ, Sussex J. Incorporating multiple criteria in HTA: methods and processes. Whitehall, London, UK: Office of Health Economics; 2011.

    Google Scholar 

  13. Thokala P, Duenas A. Multiple criteria decision analysis for health technology assessment. Value Health. 2012;15(8):1172–81.

    Article  PubMed  Google Scholar 

  14. Tanios N, Wagner M, Tony M, Baltussen R, Til Jv, Rindress D, et al. Which criteria are considered in healthcare decisions? Insights from an international survey of policy and clinical decision makers. Int J Technol Assess Health Care. 2013; e-pub ahead of print.

  15. Mitton C, Donaldson C. Twenty-five years of programme budgeting and marginal analysis in the health sector, 1974–1999. J Health Serv Res Policy. 2001;6(4):239–48.

    Article  CAS  PubMed  Google Scholar 

  16. Mitton C, Patten S, Donaldson C, Waldner H. Priority-setting in health authorities: moving beyond the barriers. The Calgary experience. Healthc Q. 2005;8(3):49–55.

    Article  PubMed  Google Scholar 

  17. Ruta D, Mitton C, Bate A, Donaldson C. Programme budgeting and marginal analysis: bridging the divide between doctors and managers. BMJ. 2005;330(7506):1501–3.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Gibson J, Mitton C, Martin D, Donaldson C, Singer P. Ethics and economics: does programme budgeting and marginal analysis contribute to fair priority setting? J Health Serv Res Policy. 2006;11(1):32–7.

    Article  PubMed  Google Scholar 

  19. Youngkong S, Kapiriri L, Baltussen R. Setting priorities for health interventions in developing countries: a review of empirical studies. Trop Med Int Health. 2009;14(8):930–9.

    Article  PubMed  Google Scholar 

  20. Kapiriri L, Martin DK. Successful priority setting in low and middle income countries: a framework for evaluation. Health Care Anal. 2010;18(2):129–47.

    Article  PubMed  Google Scholar 

  21. Tromp N, Baltussen R. Mapping of multiple criteria for priority setting of health interventions: an aid for decision makers. BMC Health Serv Res. 2012;12:454.

    Article  PubMed  PubMed Central  Google Scholar 

  22. Astley J, Wake-Dyster W. Evidence-based priority setting. Aust Health Rev. 2001;24(2):32–9.

    Article  CAS  PubMed  Google Scholar 

  23. Baltussen R, ten Asbroek AH, Koolman X, Shrestha N, Bhattarai P, Niessen LW. Priority setting using multiple criteria: should a lung health programme be implemented in Nepal? Health Policy Plan. 2007;22(3):178–85.

    Article  CAS  PubMed  Google Scholar 

  24. Baughan S, Fergunson B: Road testing Programme Budgeting and Marginal Analysis (PBMA) in three English Regions: Hull Teaching Primary Care Trust Diabetes pilot project. In. Edited by England PH. Yorkshire, UK; 2008.

  25. Bohmer P, Pain C, Watt A, Abernethy P, Sceats J. Maximising health gain within available resources in the New Zealand public health system. Health Policy. 2001;55(1):37–50.

    Article  CAS  PubMed  Google Scholar 

  26. Defechereux T, Paolucci F, Mirelman A, Youngkong S, Botten G, Hagen TP, et al. Health care priority setting in Norway a multicriteria decision analysis. BMC Health Serv Res. 2012;12:39.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Dionne F, Mitton C, Smith N, Donaldson C. Evaluation of the impact of program budgeting and marginal analysis in Vancouver Island Health Authority. J Health Serv Res Policy. 2009;14(4):234–42.

    Article  PubMed  Google Scholar 

  28. Goetghebeur MM, Wagner M, Khoury H, Rindress D, Gregoire JP, Deal C. Combining multicriteria decision analysis, ethics and health technology assessment: applying the EVIDEM decision-making framework to growth hormone for Turner syndrome patients. Cost Effective Resour Alloc. 2010;8:4.

    Article  Google Scholar 

  29. Grocott R. Applying Programme Budgeting Marginal Analysis in the health sector: 12 years of experience. Expert Rev Pharmacoecon Outcomes Res. 2009;9(2):181–7.

    Article  PubMed  Google Scholar 

  30. Halma L, Mitton C, Donaldson C, West B. Case study on priority setting in rural Southern Alberta: keeping the house from blowing in. Can J Rural Med. 2004;9(1):26–36.

    PubMed  Google Scholar 

  31. Honore PA, Fos PJ, Smith T, Riley M, Kramarz K. Decision science: a scientific approach to enhance public health budgeting. J Public Health Manag Pract. 2010;16(2):98–103.

    Article  PubMed  Google Scholar 

  32. Iqbal Z, Pryce A, Afza M. Rationalizing rationing in health care: experience of two primary care trusts. J Public Health (Oxf). 2006;28(2):125–32.

    Article  Google Scholar 

  33. Jehu-Appiah C, Baltussen R, Acquah C, Aikins M, d’Almeida SA, Bosu WK, et al. Balancing equity and efficiency in health priorities in Ghana: the use of multicriteria decision analysis. Value Health. 2008;11(7):1081–7.

    Article  PubMed  Google Scholar 

  34. Johnson-Masotti AP, Eva K. A decision-making framework for the prioritization of health technologies. In: Beach CM, Chaykowsi RP, Shortt S, St-Hillaire F, Sweetman A, editors. Health services restructuring in Canada: new evidence and new directions. Kingson, Ontario, Canada: Queen’s University Policy Studies; 2006.

    Google Scholar 

  35. Kemp L, Fordham R: Road Testing Programme Budgeting and Marginal Analysis (PBMA) in three English Regions: Norfolk Mental Health PBMA Pilot Project. In. Edited by England PH. Norfolk, UK: University of East Anglia Norwich; 2007.

  36. Kreng VB, Yang CT. The equality of resource allocation in health care under the National Health Insurance System in Taiwan. Health Policy. 2011;100(2–3):203–10.

    Article  PubMed  Google Scholar 

  37. Lee CW, Kwak NK. Strategic enterprise resource planning in a health-care system using a multicriteria decision-making model. J Med Syst. 2011;35(2):265–75.

    Article  PubMed  Google Scholar 

  38. Makundi E, Kapiriri L, Norheim OF. Combining evidence and values in priority setting: testing the balance sheet method in a low-income country. BMC Health Serv Res. 2007;7:152.

    Article  PubMed  PubMed Central  Google Scholar 

  39. Marsh K, Dolan P, Kempster J, Lugon M. Prioritizing investments in public health: a multi-criteria decision analysis. J Public Health (Oxf) 2012.

  40. Miot J, Wagner M, Khoury H, Rindress D, Goetghebeur MM. Field testing of a multicriteria decision analysis (MCDA) framework for coverage of a screening test for cervical cancer in South Africa. Cost Effective Resour Alloc. 2012;10(1):2.

    Article  Google Scholar 

  41. Mitton C, Patten S, Waldner H, Donaldson C. Priority setting in health authorities: a novel approach to a historical activity. Soc Sci Med. 2003;57(9):1653–63.

    Article  PubMed  Google Scholar 

  42. Mitton C, Mackenzie J, Cranston L, Teng F. Priority setting in the Provincial Health Services Authority: case study for the 2005/06 planning cycle. Healthcare Policy. 2006;2(1):91–106.

    PubMed  PubMed Central  Google Scholar 

  43. Mitton C, Dionne F, Damji R, Campbell D, Bryan S. Difficult decisions in times of constraint: criteria based resource allocation in the Vancouver Coastal Health Authority. BMC Health Serv Res. 2011;11:169.

    Article  PubMed  PubMed Central  Google Scholar 

  44. Mitton C, Levy A, Gorsky D, Macneil C, Dionne F, Marrie T. Allocating limited resources in a time of fiscal constraints: a priority setting case study from Dalhousie university faculty of medicine. Acad Med. 2013;88(7):939–45.

    Article  PubMed  Google Scholar 

  45. Nuti S, Vainieri M, Bonini A. Disinvestment for re-allocation: a process to identify priorities in healthcare. Health Policy. 2010;95(2–3):137–43.

    Article  PubMed  Google Scholar 

  46. Poulin P, Austen L, Kortbeek JB, Lafreniere R. New technologies and surgical innovation: five years of a local health technology assessment program in a surgical department. Surg Innov. 2012;19(2):187–99.

    Article  PubMed  Google Scholar 

  47. Poulin P, Austen L, Scott CM, Waddell CD, Dixon E, Poulin M, et al. Multi-criteria development and incorporation into decision tools for health technology adoption. J Health Organ Manag. 2013;27(2):246–65.

    Article  PubMed  Google Scholar 

  48. Robson A, Bate A, Donaldson C: Road Testing PBMA in three English regions: Newcastle CAMHS PBMA pilot project. In. Edited by England H. Newcastle, UK; 2008.

  49. Urquhart B, Mitton C, Peacock S. Introducing priority setting and resource allocation in home and community care programs. J Health Serv Res Policy. 2008;13 Suppl 1:41–5.

    Article  PubMed  Google Scholar 

  50. Wilson EC, Rees J, Fordham RJ. Developing a prioritisation framework in an English Primary Care Trust. Cost Effective Resour Alloc. 2006;4:3.

    Article  Google Scholar 

  51. Wilson E, Sussex J, Macleod C, Fordham R. Prioritizing health technologies in a Primary Care Trust. J Health Serv Res Policy. 2007;12(2):80–5.

    Article  PubMed  Google Scholar 

  52. Youngkong S, Teerawattananon Y, Tantivess S, Baltussen R. Multi-criteria decision analysis for setting priorities on HIV/AIDS interventions in Thailand. Health Res Policy Syst. 2012;10:6.

    Article  PubMed  PubMed Central  Google Scholar 

  53. Youngkong S, Baltussen R, Tantivess S, Mohara A, Teerawattananon Y. Multicriteria decision analysis for including health interventions in the universal health coverage benefit package in Thailand. Value Health. 2012;15(6):961–70.

    Article  PubMed  Google Scholar 

Download references

Acknowledgments

The authors wish to acknowledge the support of their colleagues at ARCC and C2E2.

The Canadian Centre for Applied Research in Cancer Control (ARCC) is funded by the Canadian Cancer Society Research Institute grant #019789.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Ian Cromwell.

Additional information

Competing interests

The authors declare that they have no competing interests.

Authors’ contributions

IC was responsible for the manual searching of the scientific literature and made the primary argument for inclusion/exclusion of studies. IC performed the analysis of the data and was primarily responsible for the preparation of the manuscript. SJP conceived of the study, was responsible for screening all eligible studies and determining their eligibility. SJP approved all analyses, provided material input to the interpretation of the findings, and was responsible for the preparation of the manuscript. CM was responsible for screening all eligible studies and determining their eligibility. The search strategy used in this study was based on work previously conducted by CM and colleagues. CM approved all analyses, provided material input to the interpretation of the findings, and was responsible for the preparation of the manuscript. All authors read and approved the final manuscript.

Authors’ information

SJP and CM are both longtime health services researchers specializing in criteria-guided decision making, particularly PBMA.

Additional file

Rights and permissions

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cromwell, I., Peacock, S.J. & Mitton, C. ‘Real-world’ health care priority setting using explicit decision criteria: a systematic review of the literature. BMC Health Serv Res 15, 164 (2015). https://doi.org/10.1186/s12913-015-0814-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12913-015-0814-3

Keywords