Implementing a cirrhosis order set in a tertiary healthcare system: a theory-informed formative evaluation
BMC Health Services Research volume 23, Article number: 636 (2023)
Standardized order sets are a means of increasing adherence to clinical practice guidelines and improving the quality of patient care. Implementation of novel quality improvement initiatives like order sets can be challenging. Before the COVID-19 pandemic, we conducted a formative evaluation to understand healthcare providers’ perspectives on implementing clinical changes and the individual, collective and organizational contextual factors that might impact implementation at eight hospital sites in Alberta, Canada.
We utilized concepts from the Consolidated Framework for Implementation Research (CFIR) and Normalisation Process Theory (NPT) to understand the context, past implementation experiences, and perceptions of the cirrhosis order set. Eight focus groups were held with healthcare professionals caring for patients with cirrhosis. Data were coded deductively using relevant constructs of NPT and CFIR. A total of 54 healthcare professionals, including physicians, nurses, nurse practitioners, social workers and pharmacists and a physiotherapist, participated in the focus groups.
Key findings revealed that participants recognized the value of the cirrhosis order set and its potential to improve the quality of care. Participants highlighted potential implementation challenges, including multiple competing quality improvement initiatives, feelings of burnout, lack of communication between healthcare provider groups, and a lack of dedicated resources to support implementation.
Implementing a complex improvement initiative across clinician groups and acute care sites presents challenges. This work yielded insights into the significant influence of past implementation of similar interventions and highlighted the importance of communication between clinician groups and resources to support implementation. However, by using multiple theoretical lenses to illuminate what and how contextual and social processes will influence uptake, we can better anticipate challenges during the implementation process.
Standardized order sets are based on clinical practice guidelines that guide healthcare providers in prescribing appropriate treatments and tests for patients admitted to the hospital . Typically, standardized order sets offer direction for treating patients with conditions like cirrhosis, septic shock, or heart failure. These order sets affect the practices of physicians, nurses, and other healthcare providers throughout the patient’s hospital stay . By using order sets, healthcare providers can improve adherence to guidelines, reduce in-hospital mortality, increase the quality of care, decrease variability in care practice, and reduce medication errors and length of stay [1, 3,4,5,6,7,8,9,10]. With the increased use of electronic medical records (EMRs), order sets embedded within these technologies have been identified as promising for improving patient outcomes and benefiting the healthcare team [3, 9]. While there are a few studies that have implemented a cirrhosis order set (see Table 1) at a single-center [11,12,13], there are no studies that have taken a broad approach, implementing a cirrhosis order set at multiple hospital sites within a large healthcare organization . Further, we need to expand our lens to consider the pre-implementation conditions, particularly in the context of pragmatic trials intended to test the effectiveness of an intervention in a real-world clinical setting .
As with any complex intervention in healthcare, implementation, scale, spread and sustaining change is challenging. Healthcare systems are emergent structures in complex adaptive systems [25, 26]. They are not systematically derived and cannot be conceived as complicated machines; rather, they are characterized by culture, communication networks, and local contextual realities which influence the daily interactions involved in the work required to provide high-quality patient care [27, 28]. Implementing innovations like order sets is entangled with contextual factors like organizational resources, culture, communication networks, organizational policies, leadership and physical environment [29, 30]. Further, these factors may be influenced by the social processes within an organization and individuals’ readiness and approach to change, all of which may fluctuate over time [31, 32]. There is a need to understand implementation context(s), including perceptions of the proposed intervention, social factors, capabilities and relationships of individuals, and organizational culture and capacity before the introduction of an intervention [33,34,35] to understand how context might impact the implementation process and identify strategies to overcome perceived challenges [36, 37].
We conducted a real-world formative evaluation of a complex intervention across multiple hospitals in an extensive health system. Formative evaluations not only present an opportunity for the research team to engage with stakeholders at individual study sites, but they also enable an understanding of how factors within complex contexts may potentially or actually influence the implementation of the intervention [38, 39]. Further, they can provide a clear linkage between pre-implementation context, implementation strategies and outcomes, emphasizing mechanisms of action and enhancing understanding of contextual factors impacting the implementation . A thorough understanding of pre-implementation conditions is crucial to ensuring a systematic, measured approach to implementation and is also necessary to sustain and spread the intervention in other clinical settings [41, 42].
This study aimed to explore healthcare provider perspectives on (i) implementing standardized order sets, and (ii) the contextual and social factors at the individual, collective and organizational levels which will contribute to implementation effectiveness, with these insights used to inform our implementation strategy.
A qualitative descriptive design  was adopted in this study. This study was part of a larger pragmatic trial (Clinical Trials Info) that received ethics approval from the University of Alberta Health Research Ethics Board (Pro00094054). Our results are reported according to the COREQ framework .
Guiding theory & framework
This formative evaluation was guided by Normalization Process Theory (NPT) and the Consolidated Framework for Implementation Research (CFIR) to provide a nuanced understanding of both the contextual factors and social processes that affect implementation  (SEE Additional file 1). CFIR is a framework intended to comprehensively assess factors that may influence intervention implementation and effectiveness and has been used extensively in pre- and post-implementation evaluations [29, 45]. While CFIR comprises 39 constructs, divided into five domains, including (i) characteristics of the intervention, (ii) inner setting, (iii) outer setting, (iv) characteristics of individuals, and (v) implementation process , for this pre-implementation study, we focused on understanding clinician’s perspectives on the intervention and how their work setting may influence uptake. This corresponded to the domains of intervention characteristics, inner and outer settings, and characteristics of individuals. To expand upon the process components of CFIR, we used NPT. This theory focuses on the work of individuals and groups to implement an innovation, including how they incorporate novel interventions into their everyday work, what work is involved, and what structural supports and cognitive processes are required for implementation [32, 46]. For this pre-implementation study, the domains of coherence and cognitive participation in NPT were relevant, with the others (i.e., collective action and reflexive monitoring) helpful in understanding social processes after implementation.
We purposively recruited healthcare professionals, including physicians, nurses, social workers, dieticians, and physiotherapists who provided care to patients hospitalized with cirrhosis across the eight hospitals in Alberta, Canada involved in the Cirrhosis Care Alberta (CCAB) project. Participants were recruited between October 2019 and February 2020. They were chosen for their clinical expertise and ability to provide insight into the current care for hospitalized patients with cirrhosis, including the context in which care is provided and their experiences implementing other similar quality improvement initiatives.
Potential participants were identified via project site leads who were asked to share an email invitation with any clinical or administrative staff with experience caring for patients admitted to the hospital with cirrhosis. Consideration was given to participant demographics to ensure a diverse group of healthcare professionals, including the type of professional (i.e., physician, nurse, etc.), length of time in professional role, and job type (ie, administrator, manager, front-line healthcare professional).
The research team was made up of seven researchers: AH, a Registered Nurse with expertise in qualitative research; EJ, a research assistant; TL an anthropologist with implementation and qualitative research expertise; DS, a Registered Nurse with expertise in qualitative and implementation science research, MC, a Nurse Practitioner with expertise in cirrhosis care, DCS, a primary care physician with expertise in qualitative and implementation science research, and PT, a hepatologist with expertise in cirrhosis and development of a clinical quality improvement initiative.
We selected focus groups as our data collection method in recognition of their strengths in gathering diverse viewpoints and crucial group interaction data efficiently [47, 48]. One in-person focus group was conducted for each of the eight sites, with the groups lasting 30–120 min in length. Immediately before the focus groups, participants were provided with a 15-min overview of the CCAB by the principal study investigator (PT) that detailed the order set and other aspects of the intervention. At the outset of the focus groups, the focus group lead (AH) reviewed the study details and obtained informed consent from all participants. Sociodemographic information, including age, sex, professional role, and length of time in the professional role, was collected from participants.
Focus groups were guided by a semi-structured interview guide based on relevant NPT and CFIR constructs (see Additional file 1 for the interview guide). A research assistant (EJ) recorded detailed field notes, including contextual insights and group interactions . Focus groups were recorded via digital recorder and transcribed verbatim.
Data collection and analysis were conducted concurrently to enable the refinement of our interview guide. We used a deductive content analysis approach in recognition of the explanatory nature of NPT and CFIR and their potential to produce actionable findings that could influence implementation . First, our research team members (AH, EJ, DCS) developed our NPT and CFIR coding framework based on previous work by DS, TL, and DCS . Transcripts were read and independently coded by AH and EJ, with team meetings held throughout the analysis process to review our framework and resolve discrepancies in coding. Lastly, we examined the coded data, including how it could be interpreted using NPT and CFIR . Table 2 presents NPT and CFIR characteristics that were relevant to our analysis.
We used several strategies to enhance the rigor of our data collection and analysis processes .
A detailed audit trail and reflexive notes were maintained throughout the research process, and we involved several research team members in the collection and analysis phases to ensure agreement in coding and cross-coding (investigator triangulation) . Data were managed using Quirkos .
Fifty-four healthcare professionals from 8 hospital sites participated in the focus groups, with focus groups ranging in size from 3–12 participants. All participants had experience providing care to patients admitted to the hospital with cirrhosis. Table 3 provides characteristics of study sites and demographic characteristics of participating healthcare providers.
The perspectives of healthcare providers from across study sites, in conjunction with CFIR and NPT focus of our inquiry, resulted in three questions that structure our findings: (i) What is the cirrhosis order set, and why is it important? (ii) Who will be impacted, and how will it affect work? and (iii) How will the cirrhosis order set be supported? These questions and the supporting data acknowledge the complex interplay of contextual factors and social processes in implementing a novel innovation in the healthcare environment. Table 4 shows representative quotes of constructs from CFIR and NPT.
What is the cirrhosis order set? Why is it important?
Participants across all study sites were familiar with using standardized order sets to guide acute care management of chronic diseases as they had experienced the implementation of paper-based order sets within the last 10 years. They generally recognized the inherent value of order sets and their potential to make significant “improvements in patient care”. They associated the use of order sets with improved clinical outcomes like “reduced length of stay and admissions,” with data serving as a compelling reason for their use.
Others noted that order sets presented an opportunity for increased clinical efficiency and as a means of “simplifying” care. Further, they saw order sets as a means of improving patient safety. They shared:
“…[order sets] simplify a lot of things, so it’s less likely that you’re going to miss something. If it’s handwritten, it can be misread…it just helps to standardise care” (Pharmacist, H5).
Their past experiences framed participants’ understanding of the cirrhosis order set with similar innovations. Specifically, they were concerned with the “oversaturation of order sets” in a clinical environment crowded with order sets for stroke, diabetes, heart failure (HF), and chronic obstructive pulmonary disease (COPD). One participant shared:
“...the order sets are very long, way too long...it takes a lot of time and you can’t expect front-line nurses to complete that” (Nurse, H1).
Despite these concerns, most participants felt that implementing the cirrhosis order set made sense, with clear benefits evident to patients and providers. Several participants commented on the “expansive and inclusive” admission and discharge portions of the order set, noting that they would ensure “holistic care.” They valued the guidance contained within the order set on the “dosing of drugs” and “ordering of lab tests,” with one participant sharing: “Nurses love order sets because we follow them and it’s nice to know that there’s nothing that’s missed”.
Further, they recognized that a cirrhosis order set would help a vulnerable patient population that experienced frequent and prolonged hospitalizations. Though they acknowledged that patients with cirrhosis, were seen less frequently than other chronic disease populations like COPD and HF, they “…believe that this is an important part of the population they see”. One nurse reflected on this:
“They’re [patients with cirrhosis] coming in with ascites and we’re draining a litre or 2 and they go back home and don’t change their diet...Then they’re coming back within weeks for the same thing” (Nurse, H2).
They believed an order set, with guidance on a standardized approach to caring for patients with cirrhosis, was one means of improving the quality of care, and decreasing hospital readmissions.
Who will be impacted and how will it affect work?
Improving the quality of care for patients with cirrhosis was identified as a priority for participants across all study sites. They recognized that they “have a lot of cirrhotic patients” and that implementing a cirrhosis order set was an important and necessary undertaking. The order set was perceived as an “efficient” means of enhancing care that would ensure care “wasn’t being missed”. Many participants were “enthusiastic” and “very keen to learn” about implementation of the order set, with one physician sharing:
“I’m actually excited about this for all the reasons-you’ve got a clear order set. It seems to be really intuitive”. (Physician, H3)
Others, however, acknowledged that the implementation of other order sets colored their attitudes toward the cirrhosis order set. They reported feeling “burnt out” and “frustrated” by the constant cycles of change. One participant reflected on this:
“…there’s regular policy changes, and there’s new initiatives, you’re just changing constantly. It’s just hard for staff to keep up or something’s falling off” (Nurse, H3)
Further, they shared concerns with challenges in communication between clinician groups and noted this as a barrier to uniform uptake of past order sets. One unit manager reflected on the challenges they encountered with communication with a past implementation:
“Uptake varies especially with physicians and residents. A lot of the residents don’t even know about it [the order sets], so that does hinder their use…and if the interdisciplinary team is not involved in filling out their portion, then half of it is not completed…there’s a lot of gaps in terms of the entire team being actively engaged in the order sets” (Unit Manager, H5).
Despite these concerns, participants believed they could integrate the cirrhosis order set into their daily practice and that it would become a “routine” part of care for patients with cirrhosis.
“It sounds like it [the cirrhosis order set] is embedded in the EMR and you can print your discharge summary and patient education. As a nurse, the more things I have to find and remember, the much less chance that it happens, especially if the patient has multiple conditions. This [the cirrhosis order set] is much easier to do. The more that’s handed to you, the better”. (Nurse, H8).
Healthcare providers also discussed the need for collaborative practice with a “coordinated rollout” across health disciplines, noting that focused implementation within sole provider groups resulted in confusion and disjointed patient care.
“Sometimes the nurses get something and the physicians don’t know about it and that doesn’t go well. Or, the physicians are really excited about a tool and the nurses don’t know what it is, so it doesn’t work well either” (Physician, H5).
Though all study sites were part of the same larger provincial health organization, it was evident that there were key differences in perceptions of interventions (like order sets) to improve the quality of care and contextual factors that might impact implementation. Participants recognized the potential impacts of order sets on their existing work processes. This, in combination with the patient load and complexity, influenced the overall willingness of the site to engage in such improvement initiatives. One participant shared:
“I think we’ve not been the most cooperative site from the physician side of things…we’ve not been supportive of order sets…we’re saying we can’t fill out six different order sets…that is just killing us” (Physician, H3).
This view was in contrast to other sites, which reported implementation of similar interventions having “a concerted effort”, with staff being “very keen to learn”.
How will the cirrhosis order set be supported?
Despite the number of order sets and extensive implementation campaigns across all study sites, no policies or regulations mandating the use of order sets, with the onus for use placed on individual healthcare providers. Participants saw this as potentially hindering implementation, noting the importance of policy and institutional culture in the uptake of order sets. One participant reflected on institutional culture and ongoing monitoring of paper-based order set use:
“The uptake [of order sets] is still poor within our physician group especially for the COPD and heart failure order sets. For our stroke order set, we have different monitoring with more rigorous stats, and the uptake has been greater” (Physician, H4).
Participants also expressed worries about “change fatigue” and the overabundance of order sets and quality improvement initiatives. One nurse who worked on a general medicine unit that instituted order sets for congestive heart failure and chronic obstructive pulmonary disease worried about the impact of adding another order set:
“I think part of our struggle, from an oversight view, is how do we do everything and how do we do it well, right. There are pressures related to the new government, I don’t want this [use of the order set] to become an onerous task for staff” (Nurse, H7).
In reflecting on the cirrhosis order set and past order set implementation, participants highlighted the importance of having buy-in from organizational leadership and leadership from each healthcare provider group. Across the larger provincial health organization, the cirrhosis order set has been identified as a priority initiative for implementation, with upper leadership endorsing it and encouraging sites to incorporate it into their clinical care for patients with cirrhosis. Similarly, the implementation of this initiative was supported by medical leadership at all hospital sites, with participants expressing that there were gaps in care for patients with cirrhosis. Though most sites had strong front-line leadership support, leaders at several sites shared concerns that leadership was not unified amongst provider groups. That is, there was strong support for use of the cirrhosis order set amongst nursing leadership and front-line staff, but not always the same support from physician leadership. One nurse leader reflected on this:
“Physician leadership really helps. You can tell who has good structure within their department because things flow, there’s communication, they [physicians] hear about it at their division meetings and they have their marching orders [to use it]” (Nurse Manager, H1).
Resources, including education, time and personnel to support implementation, were identified as being crucial to the success of the cirrhosis order set. Participants felt that adding personnel and funding to support implementation would decrease “frustration” amongst staff and the perception of “doing more with the same or what feels like less”. Furthermore, they reflected on how aspects of the cirrhosis order set, like enhanced discharge education for patients, would be challenging for staff to implement given time constraints in an already busy clinical environment. One nurse shared:
“It’s the capacity of the nurses and changing to a bit more proactive discharge planning and education. It was a struggle with the COPD pathway, and we’re continually working on it. So that [the cirrhosis order set] will bring some fresh problems [when it is introduced]” (Nurse, H3).
Using CFIR and NPT, we gathered the insights of healthcare providers caring for patients with cirrhosis on their perceptions of our standardized order set and past implementation experiences. Our findings highlight the complexity of introducing a cirrhosis order set across a large provincial healthcare organization, illuminating potential implementation challenges related to organizational capacity and resources, involvement of different clinician groups and disruption of routine clinical care. Conducting a pre-implementation study allowed us to improve our initial implementation strategy  in a way that considers the intricate relationship between contextual factors and the implementation process while also being open to the unique needs of each hospital site.
A core focus of our formative evaluation was gathering clinician perspectives on the intervention itself: a standardized order set for hospitalized patients with cirrhosis. Though a standardized order set specific to caring for patients with cirrhosis is new in Alberta, participants reflected on their recent experiences with paper-based order sets for other chronic diseases like heart failure and COPD [55, 56] to frame their understanding of the cirrhosis order set. Despite the novel use of an EMR to deliver our intervention, healthcare providers still drew upon these past experiences to formulate their opinions on the cirrhosis order set. This tendency to draw upon past experiences was confirmed by previous studies highlighting the influence of ‘parallel’ initiatives and their potential contributions to implementation failure or a lack of engagement in the change process [57, 58]. Given the strong influence of these past initiatives, it is crucial to recognize the importance of the sensemaking work healthcare providers must do to understand the new order set and how it can be integrated into their current care practices for patients with cirrhosis.
Participants recognized that implementing the cirrhosis order set would require significant time and resources in an environment already plagued with capacity issues and “change fatigue.” They expressed concerns that though executive leadership identified the order set as a priority intervention across the larger healthcare organization, implementation would be challenging without additional resources like personnel and funding. Key to overcoming these contextual challenges will be a focus on fostering communities of engaged healthcare providers at each study site, identifying site champions , and creating awareness of the positive impacts of the order set [32, 60].
Increasingly, healthcare environments are being conceptualized as complex adaptive systems- dynamic networks of numerous human and non-human components interacting in a non-linear, somewhat unpredictable fashion [27, 28, 61]. This conceptualization acknowledges the inherent complexity of patients with chronic diseases like cirrhosis, the dynamic interactions between healthcare providers with differing priorities, and a healthcare system with internal and external pressures . We illuminated the perceived complexities of implementing a cirrhosis order set across multiple hospital sites with different groups of healthcare providers; they recognized existing challenges in communication between physicians and nurses as a significant factor in past implementation of order sets. While healthcare providers endorse a shared goal: providing quality care to patients with cirrhosis, their approach to understanding and incorporating innovations, like standardized order sets, is not uniform. Therefore, our approach to implementation across clinician groups and study sites must be mindful of these communication networks (or lack thereof) and the social processes we can use to create a commitment to using the cirrhosis order set . This complexity also precludes the identification of a singular implementation strategy that will be effective across all of our study sites. Instead of a one-size-fits-all approach, we need to work with our study sites to create tailored implementation plans that consider each site’s strengths and weaknesses and the diverse needs of the healthcare professionals.
This study builds upon the work of Schroeder et al. who proposed the integration of CFIR and NPT to better understand the influence of implementation process and contextual determinants on the uptake of an intervention within a complex adaptive system . In our study, this integrated approach allowed us to look beyond how healthcare providers viewed the cirrhosis order set and their clinical contexts to understand what processes may be required to facilitate successful implementation. A review by Kirk et al. found that most studies using CFIR did so during or after the implementation process, with only 8% using the framework to guide pre-implementation study . Further, most CFIR pre-implementation studies focused on systematically understanding the context and identifying barriers and facilitators that may be encountered [63,64,65,66,67,68,69]. While this approach can certainly be useful in understanding the contextual factors that may positively or negatively influence implementation, we suggest that this approach with a singular framework like CFIR minimizes the complexity in our healthcare systems. Further, it dichotomizes contextual determinants as either barriers or facilitators to implementation and assumes equal importance of all determinants. By using NPT, we gained insights into potentially important social and implementation processes to understand how they may impact the uptake of the cirrhosis order set across different hospital sites.
Currently, little research is available that evaluates the implementation of standardized order sets in the acute care setting; we could locate no literature addressing the pre-implementation contexts and social processes that may impact uptake. A recent review  found 14 studies that evaluated the impact of standardized order sets in acute care, but these studies only focused on clinical and cost-effectiveness outcomes such as length of stay, mortality, readmission rates and prescribing errors. They did not examine the implementation processes and experiences of healthcare providers. While these clinical and cost-effectiveness outcomes are important, they do not convey the story of implementation and the work that healthcare providers must do to routinize these practices into their daily work; processes are key to explaining the degree to which clinical outcomes are achieved. A comprehensive examination of the pre-and post-implementation processes and experiences of healthcare providers supports developing strategies to improve implementation and the quality of care for patients.
Despite efforts to recruit a robust and diverse group of healthcare providers at each study site, focus groups did not have any physician or allied health representation at several study sites. While this is generally representative of the staffing mix in acute care settings, the additional participation of physicians whose practice will be particularly influenced by this initiative may have yielded additional insights into challenges we may face in implementation. Further, due to scheduling challenges, we could only conduct one focus group per study site with a mixture of healthcare providers. Though there was the potential for dominant voices in these groups related to participant status within the healthcare system (i.e., physicians vs. allied health professionals), researchers ensured questions were directed at the non-dominant voices, which encouraged lively discussion and participation by all attendees [70, 71].
It should also be noted that this study was conducted before implementing the cirrhosis order set. Though participants were provided with an introduction and overview of the cirrhosis order set at the outset of each session, they had not actually used the order set in practice. Therefore, influencing contextual factors and social processes presented in this study were solely focused on the pre-implementation phase and different factors may manifest during implementation. Future work will compare anticipated barriers and facilitators to those reported post-implementation. This study was conducted before the COVID-19 pandemic in March 2020. Though informal communication (beyond the scope of this study) with healthcare providers at each site has confirmed that our findings remain relevant, before formally commencing implementation at each site, we must consider how our implementation strategies must be altered to best suit the current practice contexts. We would like to acknowledge that while we used CFIR and NPT as guides for our study, we did not delve into all their constructs. As a pre-implementation inquiry focused on gathering the perspectives of the front-line healthcare providers using the cirrhosis order set in their daily practice, exploring the CFIR domains of inner and outer settings with organizational leaders was beyond the scope of this study. The characteristics of individuals were not considered in-depth as an individual’s self-efficacy, stage and motivation to engage in change are not static but rather fluctuate over time in relation to context and other influencing social processes . We limited our exploration of NPT to two constructs (coherence and cognitive participation), as the remaining constructs (collection action and reflective monitoring) help us understand the process only after implementation.
This formative evaluation study is the first to use an integrated theoretical approach (NPT and CFIR) to explore healthcare providers’ perspectives before introducing a standardized order set across multiple acute care sites. This work yielded insights into the significant influence of past implementation of similar interventions and highlighted the importance of communication between clinician groups and resources to support implementation. Implementing a complex intervention that impacts practice across clinician groups and acute care sites presents many challenges. However, by using multiple theoretical lenses to illuminate what and how contextual and social processes will influence uptake, we can better anticipate challenges during the implementation process.
Availability of data and materials
The dataset used in during the current study is available from the corresponding author upon reasonable request.
Wells C, Loshak H. CADTH rapid response reports. In: Standardized hospital order sets in acute care: a review of clinical evidence, cost-effectiveness, and guidelines. Ottawa: Canadian Agency for Drugs and Technologies in Health Copyright © 2019 Canadian Agency for Drugs and Technologies in Health; 2019.
Meleskie J, Eby D. Adaptation and implementation of standardized order sets in a network of multi-hospital corporations in rural Ontario. Healthc Q. 2009;12(1):78–83.
Kim WR, Brown RS Jr, Terrault NA, El-Serag H. Burden of liver disease in the United States: summary of a workshop. Hepatology. 2002;36(1):227–42.
Bates DW, Leape LL, Cullen DJ, Laird N, Petersen LA, Teich JM, Burdick E, Hickey M, Kleefield S, Shea B, et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA. 1998;280(15):1311–6.
Garg AX, Adhikari NKJ, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J, Sam J, Haynes RB. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005;293(10):1223–38.
Ballard D, Ogola G, Fleming N, Stauffer B, Leonard B, Khetan R. Impact of a standardized heart failure order set on mortality, readmission, and quality and costs of care. Int J Qual Health Care. 2010;22(6):437–44.
Haynes K, Linkin DR, Fishman NO, Bilker WB, Strom BL, Pifer EA, Hennessy S. Effectiveness of an information technology intervention to improve prophylactic antibacterial use in the postoperative period. J Am Med Inform Assoc. 2011;18(2):164–8.
Schnipper JL, Liang CL, Ndumele CD, Pendergrass ML. Effects of a computerized order set on the inpatient management of hyperglycemia: a cluster-randomized controlled trial. Endocr Pract. 2010;16(2):209–18.
Bright TJ, Wong A, Dhurjati R, Bristow E, Bastian L, Coeytaux RR, Samsa G, Hasselblad V, Williams JW, Musty MD, et al. Effect of clinical decision-support systems. Ann Intern Med. 2012;157(1):29–43.
Brown KE, Johnson KJ, DeRonne BM, Parenti CM, Rice KL. Order set to improve the care of patients hospitalized for an exacerbation of chronic obstructive pulmonary disease. Ann Am Thorac Soc. 2016;13(6):811–5.
Bhavsar-Burke I, Guardiola JJ, Hamade N, Gerke S, Burke SM, Merrill K, Mulvey KD, Orman ES, Fayad N. Use of a cirrhosis admission order set improves adherence to quality metrics and may decrease hospital length of stay. Am J Gastroenterol. 2023;118(1):114–20.
Louissaint J, Grzyb K, Bashaw L, Mohammad RA, Parikh ND, Tapper EB. An electronic decision support intervention reduces readmissions for patients with cirrhosis. Am J Gastroenterol. 2022;117(3):491–4.
Tapper E, Finkelstein D, Mittleman M, Piatkowski G, Chang M, Lai M. A quality improvement initiative reduces 30-day rate of readmission for patients with cirrhosis. Clin Gastroenterol Hepatol. 2016;14(5):753–9.
Ali SA, Kloseck M, Lee K, Walsh KE, MacDermid JC, Fitzsimmons D. Evaluating the design and reporting of pragmatic trials in osteoarthritis research. Rheumatology (Oxford). 2018;57(1):59–63.
Tapper EB, Parikh ND. Mortality due to cirrhosis and liver cancer in the United States, 1999–2016: observational study. BMJ. 2018;362:k2817.
McPherson S, Lucey MR, Moriarty KJ. Decompensated alcohol related liver disease: acute management. BMJ. 2016;352:i124.
Durante A, St Louis T, Meek J, Navarro V, N Sofair A. The mortality burden of chronic liver disease may be substantially underestimated in the United States, vol. 72. 2008.
Carbonneau M, Davyduke T, Congly SE, Ma MM, Newnham K, Den Heyer V, Tandon P, Abraldes JG. Impact of specialized multidisciplinary care on cirrhosis outcomes and acute care utilization. Can Liver J. 2021;4(1):38–50.
Kim D, Cholankeril G, Li AA, Kim W, Tighe SP, Hameed B, Kwo PY, Harrison SA, Younossi ZM, Ahmed A. Trends in hospitalizations for chronic liver disease-related liver failure in the United States, 2005–2014. Liver Int. 2019;39(9):1661–71.
Stepanova M, De Avila L, Afendy M, Younossi I, Pham H, Cable R, Younossi ZM. Direct and indirect economic burden of chronic liver disease in the United States. Clin Gastroenterol Hepatol. 2017;15(5):759-766.e755.
Neff GW, Duncan CW, Schiff ER. The current economic burden of cirrhosis. Gastroenterol Hepatol (N Y). 2011;7(10):661–71.
Bajaj JS, Reddy KR, Tandon P, Wong F, Kamath PS, Garcia-Tsao G, Maliakkal B, Biggins SW, Thuluvath PJ, Fallon MB, et al. The 3-month readmission rate remains unacceptably high in a large North American cohort of patients with cirrhosis. Hepatology. 2016;64(1):200–8.
Asrani SK, Kouznetsova M, Ogola G, Taylor T, Masica A, Pope B, Trotter J, Kamath P, Kanwal F. Increasing health care burden of chronic liver disease compared with other chronic diseases, 2004–2013. Gastroenterology. 2018;155(3):719-729.e714.
Carbonneau M, Eboreime EA, Hyde A, Campbell-Scherer D, Faris P, Gramlich L, Tsuyuki RT, Congly SE, Shaheen AA, Sadler M, et al. The cirrhosis care Alberta (CCAB) protocol: implementing an evidence-based best practice order set for the management of liver cirrhosis - a hybrid type I effectiveness-implementation trial. BMC Health Serv Res. 2020;20(1):558.
Plsek PE, Greenhalgh T. Complexity science: the challenge of complexity in health care. BMJ. 2001;323(7313):625–8.
Sweeney K, Griffiths F (eds). Complexity and Healthcare: an introduction. Oxon: Radcliffe Medical Press Ltd.; 2002.
Ratnapalan S, Lang D. Health care organizations as complex adaptive systems. Health Care Manag. 2020;39(1):18–23.
Gomersall T. Complex adaptive systems: a new approach for understanding health practices. Health Psychol Rev. 2018;12(4):405–18.
Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4(1):50.
Nilsen P, Bernhardsson S. Context matters in implementation science: a scoping review of determinant frameworks that describe contextual determinants for implementation outcomes. BMC Health Serv Res. 2019;19(1):189.
Schroeder D, Luig T, Finch TL, Beesoon S, Campbell-Scherer DL. Understanding implementation context and social processes through integrating Normalization Process Theory (NPT) and the Consolidated Framework for Implementation Research (CFIR). Implement Sci Commun. 2022;3(1):13.
May CR, Cummings A, Girling M, Bracher M, Mair FS, May CM, Murray E, Myall M, Rapley T, Finch T. Using normalization process theory in feasibility studies and process evaluations of complex healthcare interventions: a systematic review. Implement Sci. 2018;13(1):80.
Coles E, Anderson J, Maxwell M, Harris FM, Gray NM, Milner G, MacGillivray S. The influence of contextual factors on healthcare quality improvement initiatives: a realist review. Syst Rev. 2020;9(1):94.
Daivadanam M, Ingram M, Sidney Annerstedt K, Parker G, Bobrow K, Dolovich L, Gould G, Riddell M, Vedanthan R, Webster J, et al. The role of context in implementation research for non-communicable diseases: answering the ‘how-to’ dilemma. PLoS One. 2019;14(4):e0214454.
Murray E, Treweek S, Pope C, MacFarlane A, Ballini L, Dowrick C, Finch T, Kennedy A, Mair F, O’Donnell C, et al. Normalisation process theory: a framework for developing, evaluating and implementing complex interventions. BMC Med. 2010;8(1):63.
Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, Proctor EK, Kirchner JE. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10(1):21.
Waltz TJ, Powell BJ, Fernández ME, Abadie B, Damschroder LJ. Choosing implementation strategies to address contextual barriers: diversity in recommendations and future directions. Implement Sci. 2019;14(1):42.
Elwy AR, Wasan AD, Gillman AG, Johnston KL, Dodds N, McFarland C, Greco CM. Using formative evaluation methods to improve clinical implementation efforts: description and an example. Psychiatry Res. 2020;283:112532.
Stetler CB, Legro MW, Wallace CM, Bowman C, Guihan M, Hagedorn H, Kimmel B, Sharp ND, Smith JL. The role of formative evaluation in implementation research and the QUERI experience. J Gen Intern Med. 2006;21 Suppl 2(Suppl 2):S1-8.
Huynh AK, Hamilton AB, Farmer MM, Bean-Mayberry B, Stirman SW, Moin T, Finley EP. A pragmatic approach to guide implementation evaluation research: strategy mapping for complex interventions. Front Public Health. 2018;6:134.
Kowalski CP, Veeser M, Heisler M. Formative evaluation and adaptation of pre-and early implementation of diabetes shared medical appointments to maximize sustainability and adoption. BMC Fam Pract. 2018;19(1):109.
Davis M, Beidas RS. Refining contextual inquiry to maximize generalizability and accelerate the implementation process. Implement Res Pract. 2021;2:2633489521994941.
Bradshaw C, Atkinson S, Doody O. Employing a qualitative description approach in health care research. Glob Qual Nurs Res. 2017;4:2333393617742282.
Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349–57.
Kirk MA, Kelley C, Yankey N, Birken SA, Abadie B, Damschroder L. A systematic review of the use of the Consolidated Framework for Implementation Research. Implement Sci. 2016;11:72.
May C, Finch T. Implementing, embedding, and integrating practices: an outline of normalization process theory. Sociology. 2009;43(3):535–54.
Duggleby W. What about focus group interaction data? Qual Health Res. 2005;15(6):832–40.
Stewart DW, Shamdasani PN. Focus groups: theory and practice. 3rd ed. Thousand Oaks: SAGE; 2014.
Phillippi J, Lauderdale J. A guide to field notes for qualitative research: context and conversation. Qual Health Res. 2017;28(3):381–8.
Elo S, Kyngäs H. The qualitative content analysis process. J Adv Nurs. 2008;62(1):107–15.
Schroeder D, Luig T, Beesoon S, Robert J, Campbell-Scherer D, Brindle M. What work is required to implement and sustain the National Surgical Quality Improvement Program (NSQIP)? A qualitative study of NSQIP implementation in Alberta, Canada. BMJ Open. 2021;11(9):e044720.
Elo S, Kääriäinen M, Kanste O, Pölkki T, Utriainen K, Kyngäs H. Qualitative content analysis: a focus on trustworthiness. SAGE Open. 2014;4(1):2158244014522633.
Archibald MM. Investigator triangulation: a collaborative strategy with potential for mixed methods research. J Mixed Methods Res. 2015;10(3):228–50.
[Computer Software] [https://quirkos.com]. Accessed 13 Oct 2022.
Pendharkar SR, Ospina MB, Southern DA, Hirani N, Graham J, Faris P, Bhutani M, Leigh R, Mody CH, Stickland MK. Effectiveness of a standardized electronic admission order set for acute exacerbation of chronic obstructive pulmonary disease. BMC Pulm Med. 2018;18(1):93.
Miller RJH, Bell A, Aggarwal S, Eisner J, Howlett JG. Computerized electronic order set: use and outcomes for heart failure following hospitalization. CJC Open. 2020;2(6):497–505.
McMillan K, Perron A. Nurses amidst change: the concept of change fatigue offers an alternative perspective on organizational change. Policy Polit Nurs Pract. 2013;14(1):26–32.
Michas M, Deuchar L, Leigh R, Bhutani M, Rowe BH, Stickland MK, Ospina MB, Damant R, Mayers I, Hall J, et al. Factors influencing the implementation and uptake of a discharge care bundle for patients with acute exacerbation of chronic obstructive pulmonary disease: a qualitative focus group study. Implement Sci Commun. 2020;1(1):3.
Miech EJ, Rattray NA, Flanagan ME, Damschroder L, Schmid AA, Damush TM. Inside help: an integrative review of champions in healthcare-related implementation. SAGE Open Med. 2018;6:2050312118773261.
Perry CK, Damschroder LJ, Hemler JR, Woodson TT, Ono SS, Cohen DJ. Specifying and comparing implementation strategies across seven large implementation interventions: a practical application of theory. Implement Sci. 2019;14(1):32.
Rickles D, Hawe P, Shiell A. A simple guide to chaos and complexity. J Epidemiol Community Health. 2007;61(11):933–7.
Martin CM. Complex adaptive systems approaches in health care—a slow but real emergence? J Eval Clin Pract. 2018;24(1):266–8.
Yuan S, Wang F, Li X, Jia M, Tian M. Facilitators and barriers to implement the family doctor contracting services in China: findings from a qualitative study. BMJ Open. 2019;9(10):e032444.
Guyatt S, Ferguson M, Beckmann M, Wilkinson SA. Using the Consolidated Framework for Implementation Research to design and implement a perinatal education program in a large maternity hospital. BMC Health Serv Res. 2021;21(1):1077.
Ellis J, Band R, Kinsella K, Cheetham-Blake T, James E, Ewings S, Rogers A. Optimising and profiling pre-implementation contexts to create and implement a public health network intervention for tackling loneliness. Implement Sci. 2020;15(1):35.
Warner G, Lawson B, Sampalli T, Burge F, Gibson R, Wood S. Applying the consolidated framework for implementation research to identify barriers affecting implementation of an online frailty tool into primary health care: a qualitative study. BMC Health Serv Res. 2018;18(1):395.
Robins LS, Jackson JE, Green BB, Korngiebel D, Force RW, Baldwin LM. Barriers and facilitators to evidence-based blood pressure control in community practice. J Am Board Fam Med. 2013;26(5):539–57.
English M. Designing a theory-informed, contextually appropriate intervention strategy to improve delivery of paediatric services in Kenyan hospitals. Implement Sci. 2013;8(1):39.
King DK, Shoup JA, Raebel MA, Anderson CB, Wagner NM, Ritzwoller DP, Bender BG. Planning for implementation success using RE-AIM and CFIR frameworks: a qualitative study. Front Public Health. 2020;8:59.
Clavering EK, McLaughlin J. Crossing multidisciplinary divides: exploring professional hierarchies and boundaries in focus groups. Qual Health Res. 2007;17(3):400–10.
Tausch AP, Menold N. Methodological aspects of focus groups in health research: results of qualitative interviews with focus group moderators. Glob Qual Nurs Res. 2016;3:2333393616630466.
We are grateful to the healthcare professionals who shared their perspectives as part of this study.
This study was funded through Alberta Innovates Health Solutions (Grant number: RES0043017). Additional funding support from Alberta Health Services and the Alberta SPOR unit Pragmatic Clinical Trials Platform (RES0048995). AH is supported by a fellowship from Mitacs and Lupin Pharma. EJ is supported by a TRIANGLE (Training a new generation of researchers in Gastroenterology and Liver). This study was also supported by a financial contribution from the Government of Alberta via the Physician Learning Program. The views expressed herein do not necessarily represent the official policy of the Government of Alberta.
Ethics approval and consent to participate
Ethics approval for this study has been obtained from the Health Research Ethics Board of the University of Alberta (ID Pro00094054). Permission was also obtained from the AHS Health System Access and the Northern Alberta Clinical Trials and Research Centre to carry out the study at various study sites. Written and signed informed consent to participate was obtained from all study participants prior to qualitative interviews. All methods were carried out in accordance with relevant guidelines and regulations.
Consent for publication
The authors declare no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Hyde, A.M., Johnson, E., Luig, T. et al. Implementing a cirrhosis order set in a tertiary healthcare system: a theory-informed formative evaluation. BMC Health Serv Res 23, 636 (2023). https://doi.org/10.1186/s12913-023-09632-z