- Research article
- Open Access
Prioritization approaches in the development of health practice guidelines: a systematic review
BMC Health Services Research volume 19, Article number: 692 (2019)
Given the considerable efforts and resources required to develop practice guidelines, developers need to prioritize what topics and questions to address. This study aims to identify and describe prioritization approaches in the development of clinical, public health, or health systems guidelines.
We searched Medline and CINAHL electronic databases in addition to Google Scholar. We included papers describing prioritization approaches in sufficient detail allowing for reproducibility. We synthesized findings in a semi-quantitative way. We followed an iterative process to develop a common framework of prioritization criteria that captures all of the criteria reported by each included study.
Our search captured 33,339 unique citations out of which we identified 10 papers reporting prioritization approaches for guideline development. All of the identified approaches focused on prioritizing guideline topics but none on prioritizing recommendation questions or outcomes. The two most frequently reported steps of the development process for these approaches were reviewing the grey literature (9 out of 10, 90%) and engaging various stakeholders (9 out of 10, 90%). We derived a common framework of 20 prioritization criteria that can be used when prioritizing guideline topics. The most frequently reported criteria were the health burden of disease which was included in all of the approaches, practice variation (8 out of 10, 80%), and impact on health outcomes (7 out of 10, 70%). Two of the identified approaches stood out as being comprehensive and detailed.
We described 10 prioritization approaches in the development of health practice guidelines. There is a need to assess the effectiveness, efficiency and transparency of the identified approaches and to develop standardized and validated priority setting tools.
The development of high-quality guidelines is a rigorous and complex process that requires an average of two to three years per guideline . Due to the rapid accumulation of new evidence, guideline development should be followed by revisions and update as necessary . Given the considerable efforts and resources required to develop and update guidelines, developers need to prioritize what topics and questions to address . Therefore, priority setting is a key aspect of developing health practice guidelines [4, 5].
Priority setting should incorporate the values of various stakeholders while responding to a fundamental challenge faced by all health systems, the allocation of finite resources [6, 7]. Prioritization of guideline topics will direct efforts and funds towards the most important health needs, and will ensure that guidelines are focused and of a proper scope. This represents a step toward enhancing the delivery of evidence-informed care and improving health outcomes.
In addition to prioritizing topics, the guideline development process entails the prioritization of questions and outcomes [5, 8]. Similarly, the adaptation of guidelines may require prioritizing which questions addressed in the original guidelines will be adapted . Also, updating guidelines requires prioritization of which guidelines, guideline sections, or recommendations need to be updated .
Some investigators have provided general guidance on the prioritization of topics in guideline development by highlighting essential criteria or describing guiding principles . Others have developed detailed tools and approaches for prioritizing topics in guideline development. For example, Schünemann et al. recommended nine steps for priority setting as part of their guideline development checklist .
While recognizing the need to outline the various prioritization approaches and highlight common themes, the aim of this study was to identify and describe prioritization approaches in the development of clinical, public health, or health systems guidelines.
Our study design consisted of a systematic review of the health literature to identify prioritization approaches in the development of health practice guidelines. We followed a detailed methodology that we describe in the protocol included in Additional file 1. The project’s team included expertise in the fields of guideline development and priority setting.
▪ Paper type: We included papers of all types except for editorials, commentaries, correspondences, letters to editors, news, and abstracts. We excluded reviews but assessed all of the addressed approaches for potential eligibility.
▪ Scope: We included papers describing a prioritization approach in the de novo development, update or adaptation of health practice guidelines addressing clinical, public health, or health systems topics. The description of the approach should be thorough enough to allow for reproducibility (at least one section dedicated to that description). We excluded papers describing prioritization exercises conducted during guideline development without providing a detailed description of the process used to develop the prioritization approach. We also excluded papers describing individual prioritization items or criteria. In addition, we excluded papers where the focus of the prioritization approach was different from the guideline development process (e.g., prioritization of quality indicators derived from clinical guidelines).
▪ Setting: We included eligible papers irrespective of whether the setting was low-, middle- or high-income countries, or primary, secondary or tertiary healthcare facilities.
We searched Medline and CINAHL electronic databases from their respective dates of inception until July 2019. We developed the search strategy with the help of an information specialist. The search combined various terms for health prioritization and included both medical subject headings (MeSH terms) and free-text words. We did not restrict the search to specific languages or dates. The detailed search strategy is provided in Additional file 2. We complemented the electronic databases search with the manual search of Google Scholar. We also screened the reference lists of included and other relevant papers and reviews to retrieve additional studies.
Teams of two reviewers screened in duplicate and independently all titles and abstracts of identified citations for potential eligibility. We retrieved the full texts for citations judged as potentially eligible by at least one of the two reviewers. Then, teams of two reviewers screened the full texts in duplicate and independently for potential eligibility. They resolved disagreements by discussion or with the help of a third reviewer (EAA) when consensus could not be reached. We used a standardized and pilot-tested screening form. We also conducted two rounds of calibration exercises before the screening process.
Two reviewers (AEH and RZM) abstracted data from eligible studies in duplicate and independently. They used a standardized and pilot-tested data abstraction form. Disagreements were resolved by discussion or with the help of a third reviewer (EAA). We conducted a calibration exercise to enhance the validity of the process.
We collected the following data from each included paper:
▪ General characteristics of the approaches for prioritizing guideline topics: authors; location; year of publication; lead entity; target audience; field (e.g., clinical, public health, or health systems); focus of prioritization (e.g., guideline topic, recommendation questions, or outcomes); and type of guideline development (update, adaptation or de novo development);
▪ Steps of the development process for the approach; we used the abstracted data to come up with a common categorization of the steps (e.g., literature review, consensus building, ranking of proposed prioritization criteria, pilot testing, primary research and stakeholder involvement);
▪ Aspects proposed to be addressed when prioritizing guideline topics; we used the abstracted data to come up with a common categorization of the aspects (e.g., guideline development steps at which prioritization should happen, steps for generating an initial list of topics, prioritization criteria, types of stakeholders to involve and method for involvement).
Due to the nature of data, we synthesized the findings in a semi-quantitative way. We used the abstracted data to come up with common categorizations of relevant concepts (e.g., prioritization aspects, generation of initial list of topics), using an iterative process of review and refinement. As part of this process, we analyzed the content of each study at least twice; once when drafting the initial categories, and after producing an advanced draft. We reported the results in both narrative and tabular formats.
In addition, we followed an iterative process of drafting and revision to create a common framework of prioritization criteria that captures all of the criteria reported by each included study. Subsequently, we attempted to match the reported criteria to those of the common framework (see Additional file 3). The iterative process included drafting an initial list of criteria by one of the researchers (AEH) based on an initial review of the criteria reported in the different included papers. Another researcher (EAA) verified the resulting list to improve the clarity and relevance of the proposed criteria and to evaluate the need for potentially merging, adding, or modifying criteria. Then, multiple meetings were held to refine the list of criteria through discussion and consensus. A third reviewer (RF) verified independently the proposed criteria against the ones reported by the included studies. This represented an opportunity to review the criteria, suggest refinements, avoid redundancy and propose new criteria. This was followed by a meeting to resolve disagreements through consensus and finalize the criteria list.
Figure 1 shows the study flow diagram which summarizes the selection process. Out of the 33,339 citations identified through the electronic databases search, 10 papers met our inclusion criteria. We excluded 898 articles based at the full text screening stage for the following reasons: not paper type of interest (n = 49), not describing a prioritization approach (n = 324), not about practice guidelines (n = 525). We provide a detailed tabular description of each of the 10 included prioritization approaches in Additional file 4.
The general characteristics of the 10 distinct approaches for prioritizing guideline topics described in the papers are reported in Table 1. Most of the papers (7 out of 10, 70%) focused on guidelines for clinical practice [12,13,14, 16,17,18, 20]; one paper proposed a prioritization approach that is applicable to the clinical, public health and health systems fields ; and the two remaining papers proposed prioritization approaches respectively for World Health Organization (WHO) healthcare recommendations  and for public health guidelines . All of the identified prioritization approaches focused on prioritizing guideline topics; none on prioritizing guideline recommendation questions or outcomes. None of the approaches were specific to the update or adaptation of guidelines; all focused on the de novo development of practice guidelines.
Table 2 shows steps of the development process of each of the 10 included approaches for prioritizing guideline topics. The two steps most frequently reported to be used in the development process were: reviewing the grey literature (e.g., databases of guideline developing organizations) (9 out of 10, 90%) [8, 12,13,14,15,16,17,18, 20] and engaging various stakeholders (9 out of 10, 90%) [8, 12,13,14,15,16,17, 19, 20]. Patient and public involvement was reported to be used in the development of only one prioritization approach . Conducting primary research was reported in the development of four out of the 10 approaches (40%) [12, 16, 17, 20]. The primary aim of conducting this type of research was to rate the importance of the suggested prioritization criteria and to assess the participants’ views regarding existing prioritization approaches in their respective organizations. Two studies followed all of the steps in the development process and were thus the most comprehensive and detailed [17, 20].
Aspects of prioritization
Table 3 shows the aspects proposed to be addressed when prioritizing guideline topics. Only one study highlighted the need to conduct prioritization during the various steps of guideline development, such as prioritizing the target audience, scope of guideline, questions of potential interest, effort of synthesizing evidence, recommendations, and recommendations for research . Six studies (60%) included steps to generate an initial list of topics [8, 14, 16,17,18, 20]. Table 4 represents the steps proposed for generating an initial list of topics when prioritizing guideline topics. All of the studies incorporated the use of prioritization criteria as an aspect of the prioritization approach. Most of the studies (9 out of 10, 90%) included the involvement of stakeholders as one aspect of prioritization [8, 12,13,14,15,16,17,18, 20]. Table 5 shows the proposed types of stakeholders to involve in prioritizing guideline topics and the method for their involvement. Three studies covered the highest number of aspects of prioritization, that is four out of the five aspects [8, 18, 20].
We identified 118 prioritization criteria; 68% of the criteria (80 out of 118) were either defined or categorized under specific domains. 8% (9 out of 118) were supplied with data sources. The studies included a mean of 12 criteria (range 5–41). We derived from the 118 criteria a common framework of guideline prioritization criteria and of the domains they fall under. The framework is composed of 20 prioritization criteria clustered in six domains (Table 6) including: (1) disease-related factors; (2) interest; (3) practice; (4) guideline development; (5) potential impact of the intervention; and (6) implementation considerations. The most frequently reported criteria were related to the health burden of disease which was included in all of the prioritization approaches, practice variation (8 out of 10, 80%) [8, 13, 14, 16,17,18,19,20] and impact on health outcomes (7 out of 10, 70%) [12,13,14, 16,17,18, 20]. Urgency was included in only one of the approaches , while very few approaches reported criteria on an interest at the national level (2 out of 10, 20%) [17, 20] and on the potential impact of the intervention on equity/access (2 out of 10, 20%) [17, 18].
Summary of findings
Our study aimed to identify and describe prioritization approaches that have been suggested in the development of health practice guidelines. We identified 10 prioritization approaches (seven for clinical practice, one for public health, one for WHO healthcare recommendations, and one for all three fields). There were variabilities in the steps followed to develop the approaches, in the aspects proposed to be addressed when prioritizing guideline topics, and in the prioritization criteria.
Stakeholder involvement and the use of prioritization criteria represented key aspects of most of the prioritization approaches. There is a global movement calling to increase the engagement of diverse stakeholders (consumers; health service providers; policy makers; and researchers) in developing research agendas and determining research priorities [21, 22]. The net benefit of this involvement needs to be further examined in developing prioritization approaches, as very few studies considered this aspect [23, 24].
We developed a common framework of prioritization criteria that captures all of the 118 criteria reported by the included studies. In the field of guideline development, recent documents on when and how to develop practice guidelines reported only examples of deciding which guidelines should be developed (e.g., WHO) .
A recent systematic review of the literature addressed prioritization but was limited to the update of health decision-making tools, one of which was guidelines . Consistent with our findings, this systematic review found that the studies proposing an overall development strategy of guidelines did not provide a detailed description of the prioritization criteria used .
Strengths and limitations
This study has several strengths. First, it responds to calls by researchers and health professionals globally emphasizing the importance of setting priorities in guideline development [26, 27]. To our knowledge, this is the first systematic review to describe prioritization approaches in the development of health practice guidelines. Another strength of the present study is that we used a rigorous and transparent process in its conduct (comprehensive search strategy, duplicate and independent selection, and duplicate and independent data abstraction) . Finally, we followed an iterative process of drafting and revision to create a common framework of prioritization criteria that captures all of criteria reported by each of the ten included study. This represents a step towards standardizing the terminology for prioritization and enhancing the clarity of the criteria for decision-making.
One potential limitation of the study is that we did not search the grey literature and therefore we could have missed on potentially relevant information. However, we believe that our search was comprehensive enough and did not miss any study included by García et al. (2017)  that would have been eligible for our review.
We described 10 approaches in the development of guidelines. The review findings can assist clinicians, funders, policymakers, and other stakeholders seeking to develop health practice guidelines in prioritizing topics to be addressed. It might be challenging to provide specific guidance on which approach to use given the variability in the processes followed to develop the approaches. However, guideline developers can choose the prioritization approach and criteria that best fit their needs.
The wide variability in the identified prioritization approaches necessitates that researchers develop standardized and validated priority setting tools in the development of health practice guidelines. There is also a need to develop methods for prioritization of questions and outcomes for guidelines projects. Researchers are encouraged to provide guidance on the conduct and reporting of studies on prioritization approaches.
Further rigorous methodological research is required to assess the effectiveness, efficiency and transparency of the identified approaches. This kind of evaluation research would lead to a better understanding of potential facilitators and barriers to prioritization. Furthermore, and because all of the included approaches were developed by researchers from middle- and high-income countries, future studies can focus on the effectiveness of the suggested approaches in low-income countries. It is also essential to evaluate the impact of those approaches on resource allocation and on clinical outcomes.
Availability of data and materials
- MeSH terms:
Medical subject headings
World Health Organization
Gopinathan U, Hoffman SJ. Institutionalising an evidence-informed approach to guideline development: progress and challenges at the World Health Organization. BMJ Global Health. 2018;3(5):e000716.
Kredo T, Bernhardsson S, Machingaidze S, Young T, Louw Q, Ochodo E, et al. Guide to clinical practice guidelines: the current state of play. Int J Qual Health Care. 2016;28(1):122–8.
Richter Sundberg L, Garvare R, Nyström ME. Reaching beyond the review of research evidence: a qualitative study of decision making during the development of clinical practice guidelines for disease prevention in healthcare. BMC Health Serv Res. 2017;17(1):344.
National Institute for Health and Care Excellence (NICE). Selecting and prioritising guideline and quality standard topics. 2014. Available from: Retrieved from: https://www.nice.org.uk/about/what-we-do/our-programmes/nice-guidance/nice-guidelines/selecting-and-prioritising-guideline-and-quality-standard-topics.
World Health Organization (WHO). WHO handbook for guideline development. 2nd ed. Geneva: World Health Organization; 2014.
Aidem JM. Stakeholder views on criteria and processes for priority setting in Norway: a qualitative study. Health Policy. 2017;121(6):683–90.
Boivin A, Lehoux P, Lacombe R, Burgers J, Grol R. Involving patients in setting priorities for healthcare improvement: a cluster randomized trial. Implement Sci. 2014;9(1):24.
Schunemann HJ, Wiercioch W, Etxeandia I, Falavigna M, Santesso N, Mustafa R, et al. Guidelines 2.0: systematic development of a comprehensive checklist for a successful guideline enterprise. CMAJ. 2014;186(3):E123–42.
Schunemann HJ, Wiercioch W, Brozek J, Etxeandia-Ikobaltzeta I, Mustafa RA, Manja V, et al. GRADE evidence to decision (EtD) frameworks for adoption, adaptation, and de novo development of trustworthy recommendations: GRADE-ADOLOPMENT. J Clin Epidemiol. 2017;81:101–10.
Becker M, Jaschinski T, Eikermann M, Mathes T, Buhn S, Koppert W, et al. A systematic decision-making process on the need for updating clinical practice guidelines proved to be feasible in a pilot study. J Clin Epidemiol. 2018;96:101–9.
Eccles MP, Grimshaw JM, Shekelle P, Schunemann HJ, Woolf S. Developing clinical practice guidelines: target audiences, identifying topics for guidelines, guideline group composition and functioning and conflicts of interest. Implement Sci. 2012;7:60.
Battista RN, Hodge MJ. Setting priorities and selecting topics for clinical practice guidelines. Can Med Assoc J. 1995;153(9):1233.
Field MJ. Setting priorities for clinical practice guidelines. Washington (DC): National Academies Press (US); 1995.
McClarey M, Duff L. Identifying priorities for national clinical guidelines. Nurs Stand (through 2013). 1999;13(41):37.
Oxman AD, Schünemann HJ, Fretheim A. Improving the use of research evidence in guideline development: 2. Priority setting. Health Res Policy Syst. 2006;4(1):14.
Ketola E, Toropainen E, Kaila M, Luoto R, Mäkelä M. Prioritizing guideline topics: development and evaluation of a practical tool. J Eval Clin Pract. 2007;13(4):627–31.
Reveiz L, Tellez DR, Castillo JS, Mosquera PA, Torres M, Cuervo LG, et al. Prioritization strategies in clinical practice guidelines development: a pilot study. Health Res Policy Syst. 2010;8(1):7.
Atkins D, Perez-Padilla R, MacNee W, Buist AS, Cruz AA. Priority setting in guideline development: article 2 in integrating and coordinating efforts in COPD guideline development. An official ATS/ERS workshop report. Proc Am Thorac Soc. 2012;9(5):225–8.
Reddy B, Kelly M, Thokala P, Walters S, Duenas A. Prioritising public health guidance topics in the National Institute for health and care excellence using the analytic hierarchy process. Public Health. 2014;128(10):896–903.
Mounesan L, Sayarifard A, Haghjou L, Ghadirian L, Rajabi F, Nedjat S. A manual for prioritizing the topics of clinical practice guidelines for family physicians. Int J Prev Med. 2016;7:64.
Barnieh L, Jun M, Laupacis A, Manns B, Hemmelgarn B. Determining research priorities through partnership with patients: an overview. Semin Dial. 2015;28:141-6.
Davison KM, D’Andreamatteo C, Mitchell S, Vanderkooy P. The development of a national nutrition and mental health research agenda with comparison of priorities among diverse stakeholders. Public Health Nutr. 2017;20(4):712–25.
Hunter DJ, Kieslich K, Littlejohns P, Staniszewska S, Tumilty E, Weale A, et al. Public involvement in health priority setting: future challenges for policy, research and society. J Health Organ Manag. 2016;30(5):796–808.
Wilson P, Mathie E, Keenan J, McNeilly E, Goodman C, Howe A, et al. Health services and delivery research. ReseArch with patient and public invOlvement: a RealisT evaluation - the RAPPORT study. Southampton: NIHR Journals Library; 2015.
García LM, Pardo-Hernandez H, Superchi C, de Guzman EN, Ballesteros M, Roteta NI, et al. Methodological systematic review identifies major limitations in prioritization processes for updating. J Clin Epidemiol. 2017;86:11–24.
Rosenfeld RM, Shiffman RN, Robertson P. Clinical practice guideline development manual: a quality-driven approach for translating evidence into action. Otolaryngol Head Neck Surg. 2013;148(1_suppl):S1–S55.
Shekelle PG, Woolf SH, Eccles M, Grimshaw J. Developing guidelines. BMJ. 1999;318(7183):593–6.
Higgins J. GS, Cochrane handbook for systematic reviews of interventions. Cochrane Collaboration; 2011.
We would like to thank Ms. Lama Hishi for her contribution to the search strategy and title and abstract screening and Dr. Ahmed Ali for his contribution to the title and abstract screening.
This study was supported by the Alliance for Health Policy and Systems Research, WHO, Geneva. The funder was not involved in the study design, data collection, analysis, and interpretation, or in writing the manuscript.
Ethics approval and consent to participate
Consent for publication
EAA is an author of one of the included papers. He was not involved in the study selection and data extraction processes.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Study protocol. The study protocol detailed the methodology of the systematic review.
Search strategy. The search combined various terms for health prioritization and included both medical subject headings (MeSH terms) and free-text words.
Framework of prioritization criteria in the development of health practice guidelines. The common framework of prioritization criteria captured all of the criteria reported by each included study.
Detailed findings of the included papers on the development processes of the prioritization approaches and the aspects to be addressed when prioritizing guideline topics. This represents a detailed tabular description of each of the 10 included prioritization approaches.
About this article
Cite this article
El-Harakeh, A., Morsi, R.Z., Fadlallah, R. et al. Prioritization approaches in the development of health practice guidelines: a systematic review. BMC Health Serv Res 19, 692 (2019). https://doi.org/10.1186/s12913-019-4567-2
- Priority setting
- Prioritization approaches
- Health priorities
- Research prioritization
- Guideline development