Reaching beyond the review of research evidence: a qualitative study of decision making during the development of clinical practice guidelines for disease prevention in healthcare

Development of clinical practice guidelines

Development of high-quality guidelines involves technical, cognitive, and interpersonal processes [17]. Considerable efforts have been invested to strengthen technical aspects of guideline development (e.g. systematic collection and assessment of evidence) [18–21]. However, procedures for increasing guideline quality and making accurate use of evidence depend on human information processing, judgement, and sense-making, both individually and in groups. The core understanding of identical evidence can vary between groups [22], but this topic, compared to the technical aspects, has been sparsely explored empirically [23].

Guideline development generally includes identifying and refining the subject area, assembling panels, identifying and assessing evidence, translating evidence into clinical practice guidelines, and finally reviewing and updating the guidelines [16, 24]. Guideline development groups (GDGs) have often involved scientists and health professionals tasked with following explicit guideline development procedures [25, 26]. During the last decades, the emphasis on cost-effectiveness, ethics, and equity aspects to inform judgments and formulations of guideline recommendations has evolved [27–30].

Decisions about guideline recommendations have usually been made by a group of health profession representatives so as to assemble a wide range of knowledge, experiences, and opinions. Considerations made when recruiting individuals to such groups have involved variation in geographic representation, disciplines, stakeholder interests, gender, and different schools of thought [25, 31, 32].

Guideline development processes have typically been informed by manuals or models that share common features [8]. Recurring activities are to assemble a GDG, involve patients, identify clinical questions, perform systematic searches for and appraisals of research evidence, draft and finalize recommendations, and implement and continually update the guideline process [8]. Several studies have shown discrepancies between manual contents and the actual practice of guideline development [33–35] and manuals have been criticized for not being specific enough [36].

Guideline development groups and decision making

The functioning of GDGs has been shown to influence how information is shared and processed and ultimately affects the guideline quality [37, 38]. In a study of 15 clinical guidelines on diabetes recommendations were found to differ significantly, even though they were based on similar evidence [39]. To avoid unwanted variation, guideline producers have aimed to instruct and direct the GDGs. However, how this steering has been operationalized in decision making models (e.g. through criteria, cues or vignettes) has varied [25].

For many patient populations and conditions evidence is limited [40]. To aid decisions on resource allocation, guidelines also need to involve value-based judgements. Consensus decision methods have often been used to define levels of agreement on controversial subjects [41], in particular the Delphi method, the nominal group technique and the consensus development conference [25].

The traditional Delphi method is characterized by iterations of individual decisions and group feedback to reach consensus, although participants do not meet with each other face-to-face and therefore can present and react to ideas unbiased by other participants [42, 43]. The Nominal Group Technique starts with individual generation of ideas that are discussed in a face-to-face group meeting and then statistically aggregated to form group judgements [44–46]. The consensus development conference involves one open and one closed chaired session. In the open session, evidence on a topic is presented by experts to a decision making group. In the closed session, the decision making group considers the issue in light of the data presented and aims to reach consensus [25].

Members of guideline groups tend to be biased in favour of procedures or theories linked to their field of knowledge or experience [47–49]. Higher levels of agreement more likely occurred if evidence was presented as a literature review or if the GDG consisted of a single profession (e.g. GPs) and the evidence was in accordance with group members’ beliefs [49]. Further, members’ willingness to reach consensus and their status in the group can outweigh evidence or other formal decision criteria when formulating guideline recommendations [50]. The chair of the group has the potential to influence processes and norms in a direction where information is distributed and shared [50, 51].

Group size and structure have been found to be relevant for the validity and reliability of guideline recommendations [52]. Between 6 and 15 multidisciplinary members has been suggested as the ideal group size for guideline development. Too few members might provide insufficient knowledge while too many might hinder effective group functioning [24, 50].

The Swedish guideline development model

In Sweden, the National Board for Health and Welfare (NBHW) is responsible for the development of national guidelines. Guideline development is led by an executive board based at NBHW. The formal purpose of guidelines is to support and guide resource allocation within health and medical care and the social services system. Primary target audience is decision-makers within health care, while professionals, patients, and the public are secondary target groups.

NBHW guideline policies and instructions for guideline development state that rigorous assessment and use of scientific evidence should be the basis for guideline recommendations. Evidence is ranked by using the GRADE system (i.e. a formal method to categorise the quality of evidence and strength of recommendations in clinical practice guidelines) [18]. Guideline development is also affected by the Swedish Health Care Act that states that human dignity, patients’ needs, solidarity, and cost efficiency should inform all decisions involving resource prioritizations in health care, referred to as the ethical platform [53].

Guideline development at NBHW consists of four phases and shares the characteristics of several other guideline frameworks [8]. The development process of the disease-prevention guidelines is illustrated in Fig. 1 and has been described in more detail in a previous study [15]. The first phase aims to define the guideline area (e.g. disease prevention). Within the guideline area, relevant health conditions (e.g. lifestyle habits such as smoking) are connected to remedial measures (e.g. motivational interviewing) in condition-intervention pairs. In the second phase, scientific experts perform a systematic literature search for studies on each condition-intervention pair. Evidence is assessed and graded using GRADE evidence assessment tool [18]. Further, health economists calculate cost effectiveness of the condition-intervention pairs. In the third phase, a group consisting of clinical experts (hereafter referred to as the Prioritization group, and group participants referred to as members) is assembled. The members rank the condition-intervention pairs based on quality of research evidence supporting the effect, the curative/preventive effect size, severity of the condition and cost-effectiveness.. The members also consider ethical implications according to the ethical platform. Ranking is done in two steps. In the first step, assessments are made individually and handed in anonymously. In the second step, all individual assessments are presented and discussed in the Prioritization group setting. The condition-intervention pairs are ranked on a scale from 1 (strongly recommended) to 10 (not recommended). The Prioritization group meetings are led by a chair and 1–3 representatives from NBHW national guideline department. During the fourth phase, the director general of NBHW decides on guideline recommendations based on the Prioritization group’s input. A preliminary guideline is distributed to stakeholders and target groups (e.g. decision-makers in the county councils) that provide feedback on applicability and potential organizational consequences. Based on analysis of this feedback, the final guidelines, including indicators for measurement of guideline adherence and patient-related outcomes, are produced. In the final fifth phase, the outcomes of guidelines are monitored and after 3–5 years the guidelines are up-dated.

Up until todayNBHW has developed 15 guidelines using the process outlined above.

Our focus is on the judgement and decision making process of the Prioritization group in the third phase of the NBHW guideline development model, i.e. prioritizing and deciding on guideline recommendations. Accordingly, the aim of this study was to investigate the bases for decisions and the decision making process of the Prioritization group during development of clinical guidelines with a disease preventive scope in Sweden.

Key participants of the studied guideline development process

A qualitative, longitudinal, case-study approach was used to investigate decision making. In phase three (Fig. 1), the Prioritization group had to rank approximately 230 condition-intervention pairs related to area of disease prevention divided into four lifestyle areas: tobacco use, hazardous use of alcohol, insufficient physical activity and unhealthy eating habits. The Prioritization group members were selected by the NBHW based on their clinical expertise in relation to the preventive guideline scope. The number of persons that were contacted by the two representatives from the NBHW guideline executive board and the Chair via phone and email cannot be determined and the number of persons who refused or were unable to attend is not known.The experts included in the process explicitly declared no competing interests. The Prioritization group consisted of 25 members (16 women and 9 men) who represented different geographical areas, disciplines (e.g. psychiatry, obstetrics) and professional groups (e.g. physiotherapists, nurses). The Prioritization group participated in meetings to make judgments and reach consensus decisions for the selected 230 condition-intervention pairs (e.g. motivational interviewing for at-risk alcohol use). The meetings had a pre-set agenda that guided which clusters of condition-intervention pairs that were to be covered at each meeting. To finalize all condition-intervention pairs, the Prioritization group met during 18 days (at nine occasions, distributed over three years). Each meeting was led by a chair and two representatives from the NBHW guideline executive board (hereafter referred to as executive board members). Before each meeting the Prioritization group members individually rated each condition-intervention pair. Ratings were based on the criteria advised by the NBHW model for prioritization (i.e. condition seriousness, curative/preventive effect size, evidence supporting the effect, cost-effectiveness and ethical implications). The meeting procedure included presenting individual covert ratings for each such pair, followed by group discussions and open voting. This procedure was by the NBHW described as a modified Delphi-method since face-to-face group interactions were the key aspect of the studied process, an approach that differs from the traditional Delphi-methodology [25].

Data collection

To answer the first research question an inductive approach was undertaken and data were collected from questionnaires, non-participant observations of meetings, and documents. For the second research question non-participant observations were the main data source. The third research question used data from observations and questionnaires. Two researchers (LRS and MN) conducted non-participant observations of 18 days of group meetings (Ʃ104 hours) using a semi-structured protocol indicating time and pre-set meeting agenda. Observational data contained aspects of verbal content (e.g. members’ verbal argumentation or the NBHW written agenda for the meeting) and the process (e.g. types of activities and procedures) and type of actor that made input (member or executive board members). Three questionnaires (Q1, Q2, Q3) were developed guided by the research questions and used to collect data on members’ experiences of the process. The questionnaires were distributed in connection to meetings, Q1 in the beginning (3^nd meeting, Sept 2009), Q2 in the middle (7^th meeting, March 2010), and Q3 at the end of the development process (9^th meeting, May 2011).

The questionnaires were anonymous and all three (Q1–Q3) contained a section with close-ended questions concerning socio-demographics (i.e. gender, education, occupation and employer). The reason for repeating these questions was to make analysis of the roles of these factors in each phase of the guideline development possible.In addition the questionnaires contained open-ended questions concerning experiences from the decision making process. Questionnaire number 1 (Q1) focused on members’ anticipation and experience of the initial engagement in the guideline development process (example of question in Q1: How did you prepare for participating in the prioritization group?). Questionnaire number 2 (Q2) focused on Prioritization group members’ experience of the NBHW model for prioritization and what decision criteria they based their decisions on (example of question in Q3: “Describe the principles for taking decisions in the prioritization group?”). Questionnaire number 3 (Q3) addressed Prioritization group members’ perception and experience of the decision making process in retrospect (example of question in Q3:“To what degree have you been able to communicate your opinions and knowledge in the group?”).

Data analysis

Data analysis focused on identifying decision criteria used, variation in decision criteria over time, and types of dilemmas encountered (guided by research questions I–III). With these restrictions we used two approaches to qualitative content analysis; conventional and summative [54]. Conventional content analysis allows categories to emerge from data inductively and is used to capture the unique characteristics of the studied case and to allow for new understandings [54]. The conventional approach was used to analyse data from documents, questionnaires and observations, responding to research question I and III. The summative content analysis allows for the quantification of words or specific content in data sources and aims to illuminate the usage of a word or a group of words by analysing the frequency and the latent content of data [54]. The summative approach was used to analyse data from observations and to answer research question II.

The conventional analysis was conducted in several steps. First, the text in observation notes (n = 482 pages), questionnaires (n = 62 responses) and documents (n = 89 pages) was read through several times to get a sense of the material as a whole. In a second step, the text was divided into coherent text units and further analysed using software NVivo9. The text units contained information on how (e.g. discussions, behaviours, and statements) and on what basis (e.g. criteria or arguments) group decisions were made. In a third step text units with coherent statements (i.e. sentences and sections of text with coherent content) were coded with a condensed label intended to capture the essence of the unit’s meaning. Codes with a common meaning were then grouped into categories. In the fourth step, categories with a common meaning were merged into themes. The basis for and labelling of categories and themes were discussed among and decided on by two researchers (LRS and MN). Seven decision criteria were identified during the analysis. To assess the usage and development of various decision criteria during the two year prioritization process, we used summative content analysis [54] of the observational data. Synonymous expressions for the seven main decision criteria found in the conventional content analysis were identified. As an example of this, synonyms for the key term Research evidence were found in the terms scientific base, scientific investigations and scientific arguments. Key terms and their corresponding synonyms were then sought in the text through computer-assisted searches. Number of occurrences from each key terms and alternative term were summarized. Finally the occurrences of the terms were organized chronologically, indicating if they were expressed in the beginning (meetings 1–3), the middle (meetings 4–6), or the end (meetings 7–9) of the process in Phase 3.

The conventional and summative analysis was conducted by one researcher (LRS) and interpretations continuously discussed with another researcher (MN). LRS and MN were both aware of the content of the formal guideline development model.

Decision criteria used by the Prioritization group

Group-decisions and judgments were found to be based on seven main criteria. The first four criteria are in accordance with the NBHW decision making model: 1) Research evidence, 2) Severity of the condition, 3) Cost-effectiveness of the intervention, and 4) Ethical considerations. The other three criteria emerged over time as an additional basis for decisions: 5) Needs of vulnerable groups, 6) Clinical knowledge and experience and 7) Potential consequences of guideline recommendations.There were also additional aspects that influenced the decision process: 8) Additional factors – Gender, social status, and interpersonal skills.

The decision making process over time

The summative analysis indicated that the usage of different types of decision criteria in observed Prioritization group discussions varied in frequency over time. Figure 2 provides an overview of how the use of criteria evolved over time.

Terms related to 2) Severity of the condition and 3) Cost-effectiveness showed relatively stable patterns in occurrence over time. Arguments related to criterion 4) Ethical considerations were most frequent in the beginning and middle sections of the process and declined at the end. The added criterion 5) Needs of vulnerable groups was frequently used after it was established during the third meeting. Discussions on 7) Potential guideline consequences of the proposed recommendations increased in the later part of the process.

Emerging dilemmas and related strategies

Several dilemmas emerged during the process, closely related to the variation of the discussion of different criteria over time. Three categories of dilemmas were identified, further described below.

The formal decision model versus informal criteria and group influences

Information sharing processes may have been hindered as a consequence of several group-related circumstances (i.e. group size, group composition, consensus decision model) in the studied case. The studied Prioritization group (25 members) was larger than the ideal group size of 6–15 members [23], making it more difficult to involve all participants in discussions. The Prioritization group members’ gender, social status, and interpersonal skills were perceived to affect their relative contribution in group discussions. This finding corresponds well with several other studies [22, 45, 49, 50, 52]. The consensus decision model used was described as a “modified Delphi” by the NBHW but might also be viewed as an Estimate-Talk-Estimate process [55, 56]. This decision-making model made face-to face group-discussions highly influential for final decisions.

The observed deviation from the NBHW formal guideline development model is also in line with previous findings [33–35]. Most guideline manuals are generic outlines, intended for a wide range of guideline topics, target users, and populations [25, 31, 32]. The NBHW model is similar to other national guideline models in this sense [25, 32], and reflects a current trend of balancing rigour and pragmatism during guideline development [57]. Adhering to multiple criteria makes it harder for GDGs to reach consensus and the need for guidance by guideline development bodies, chairpersons, and explicit consensus-development methods could be expected to increase [24, 50], also exemplified in this study.

Accepted methodological standards for guideline development have been pushed to continuously higher levels of rigor [2, 4], with consequences in terms of costs, time, and usability of the guidelines [57]. As such, the tasks and demands for future guideline development groups will probably become even more complex, raising the question of the type of comptences and support guideline development groups will need.

Research evidence versus clinical expertise and experience

Clinical expertise is emphasized in definitions of EBP, stating that EBP is about “integrating individual clinical expertise and the best external evidence” ([58] p. 312). In reality, EBP has been focused on generating, assessing, and making use of research evidence rather than on giving proportional attention to clinical experience and patient preferences [59]. Contemporary guideline development models (e.g. NBHW, NICE, WHO) underline the importance of stakeholder involvement in the process. One of the purposes of stakeholder involvement is to underpin guideline recommendations with the views and experiences of guideline target users (e.g. health professionals) [60].

In the studied case it was found that over time, research evidence out-triumphed arguments based on clinical knowledge and experience. The criteria Research evidence and Clinical knowledge and experience were contrasted in the Prioritization group’s discussions, but research evidence was almost always given a higher priority. The precedence of research evidence was underlined by the NBHW executive board and also supported by the NBHW instructions for guideline development.

The purpose of involving clinical experts in the guideline development process was rather vague and their expected contribution remained slightly unclear in the studied case. Diverse approaches to stakeholder involvement among guideline developers have also been seen in previous research. Most guideline developers involve stakeholders (e.g. patients or health professionals), in different stages of the process [61], mainly with the purpose of engaging future users of the guidelines and increase their quality with a broadened knowledge base [32]. Still, among leading guideline development agencies, such as NICE and WHO, clinical experts tend to play a minor role (e.g. comment on drafts, promote implementation) and they are not part of the voting or decisionmaking on the final guideline recommendations [62]. The ambivalence regarding the roles of clinical experts in previous research is also reflected within this case, where health professionals were involved based on their clinical expertise but where other actors tried to limit the influence of their clinical experience and knowledge in the decision process.

This tension between evidence and clinical experience and expertise is not often discussed in the guideline literature. How and to what extent clinical experience can be systematically identified and applied in guideline development remains unclear and is an essential area for further investigation.

The prioritization process and its relation to implementation

Some of the key determinants of guideline adherence are settled already during guideline development (e.g. strength of recommendations, trialability, applicability, adoptability) [13, 15, 63]. Guideline manuals have also shown shortcomings in terms of information that could help prepare for guideline implementation, despite known organizational and system-level barriers [36]. Previous studies have investigated the possibilities to improve guideline implementability during guideline development [10, 12, 14].

The observed deviation from the pre-set NBHW decision-model is interesting from this implementation perspective as several of the emergent decision-criteria were connected to specific target users, target populations, postulated barriers, or guideline contexts that are linked to implementation determinant framework [63]. In the studied case, the Prioritization group added valuable perspectives to the NBHW decision model that potentially would aid implementation, such as accommodation (e.g. anticipated changes in workflow and professional roles), applicability (e.g. strategies to individualize recommendations based on clinical expertise), communicability (e.g. anticipated responses from health professionals and patients), and implementation (barriers and strategies).

The Prioritization group struggled to find a balance between recommending major changes to clinical practices based on best available research evidence and suggesting more feasible changes based on a combination of evidence, good practice, stated values of health care professionals, and existing disease-prevention health services. Adaption of a general pre-set decision model to fit the specific topic of the guideline and to the target users is inevitable and might also be beneficial. Adapting models to fit current settings and melding values, evidence, and costs into guideline recommendations might arguably increase the possibilities to successfully implement recommendations in health care organisations.

It is well known that clinical experience and research evidence sometimes are in conflict with each other [57, 59]. The results of our study are linked to recent debates on the role of methodological rigor in guideline development, where arguments for a balance between rigor and pragmatism, that invites stakeholders to negotiate criteria and adapt guidelines accordingly, have been put forward [57]. In this case our results show that when stakeholders are involved in guideline development in cases where the evidence is scarce a trade-off between validity and efficiency might occur. The Prioritization group struggled with a number of dilemmas related to the characteristics of research and implications for guidelines and target users. It took time and effort to solve the dilemmas, and a modified decision model was developed during the process. It could be questioned whether the implicit dilemmas in many of the discussions concerned differences in sources of input, i.e. research evidence, clinical experience, and patient preferences. Could this “rocky journey” have been foreseen, and could guideline models and manuals provide guidance on how to solve issues related to characteristics of evidence/research and adaptation to local contexts in the same spirit of transparency as for the methodological procedures? It could be argued that if the common dilemmas faced by guideline developers are not addressed, the use of guideline development models will not reach its full potential. In order to understand and improve guideline development processes further studies of decision making with a similar focus may provide guidance.

Methodological considerations

This study examines only a part of the entire guideline development and implementation process – the process of prioritization when deciding on guideline recommendations. Thus, the reader should bear this in mind when thinking about the recommendations and suggestions made. Examining guideline development by observing it over time, documenting members’ experiences, and collecting documents did allow us to investigate the prolonged process of reasoning, judgment, and decision-making, and also permitted validation of findings from different sources. However, the methods used in this study have some limitations. Trustworthiness of studies is always important to consider. The observations are restricted by the ability of the observers to follow and document the process. Observations could have been improved by using video recordings, but the large number of members and the characteristics of the premises made this very difficult. In this study two researches were involved in observations (4 of 9 meetings) and analyses in order to reduce bias and shortcomings. Furthermore, it was presumed that group decisions were based on arguments explicitly articulated during meetings and/or indicated by members in questionnaires. The increase (or decrease) of an articulated argument over time might not entirely reflect its role in the final decision and should be interpreted with caution. It was though considered to be an acceptable indicator of how decision criteria were used over time.