Skip to main content
  • Research article
  • Open access
  • Published:

Assessing the implementation processes of a large-scale, multi-year quality improvement initiative: survey of health care providers



Beginning in 2012, Lean was introduced to improve health care quality and promote patient-centredness throughout the province of Saskatchewan, Canada with the aim of producing coordinated, system-wide change. Significant investments have been made in training and implementation, although limited evaluation of the outcomes have been reported. In order to better understand the complex influences that make innovations such as Lean “workable” in practice, Normalization Process Theory guided this study. The objectives of the study were to: a) evaluate the implementation processes associated with Lean implementation in the Saskatchewan health care system from the perspectives of health care professionals; and b) identify demographic, training and role variables associated with normalization of Lean.


Licensed health care professionals were invited through their professional associations to complete a cross-sectional, modified, online version of the NoMAD questionnaire in March, 2016. Analysis was based on 1032 completed surveys. Descriptive and univariate analyses were conducted. Multivariate multinomial regressions were used to quantify the associations between five NoMAD items representing the four Normalization Process Theory constructs (coherence, cognitive participation, collective action and reflexive monitoring).


More than 75% of respondents indicated that neither sufficient training nor resources (collective action) had been made available to them for the implementation of Lean. Compared to other providers, nurses were more likely to report that Lean increased their workload. Significant differences in responses were evident between: leaders vs. direct care providers; nurses vs. other health professionals; and providers who reported increased workload as a result of Lean vs. those who did not. There were no associations between responses to normalization construct proxy items and: completion of introductory Lean training; participation in Lean activities; age group; years of professional experience; or employment status (full-time or part-time). Lean leader training was positively associated with proxy items reflecting coherence, cognitive participation and reflexive monitoring.


From the perspectives of the cross-section of health care professionals responding to this survey, major gaps remain in embedding Lean into healthcare. Strategies that address the challenges faced by nurses and direct care providers, in particular, are needed if intended goals are to be achieved.

Peer Review reports


The province-wide introduction of Lean as a strategy to improve health care quality and promote patient-centredness was initially launched by the Government of Saskatchewan of Canada in 2012 [1]. Lean refers to a set of operating philosophies, tools and activities that help to create maximum value for patients by reducing sources of waste in a process [2, 3].This large scale transformation was aimed at effecting coordinated, system-wide change affecting multiple organizations and care providers [4]. In keeping with recommendations that active implementation strategies are essential to producing a coherent and multilevel approach to health care transformation [5,6,7,8], significant investments in implementation were directed towards training initiatives, particularly at the senior leadership level. Initial implementation of Lean was delivered through a paid consultant and later transitioned to local leadership. More than 1000 projects [1] have been undertaken throughout the publicly-funded health regions in this province of 1.1 million people and the government re-confirmed its commitment to support the ongoing use of Lean in its 2016 strategic plan [1].

Because Lean is a “high-touch, high-maintenance enterprise” [9, 10], the costs of implementation are high. Estimates of the investment by the provincial government in Lean since 2010 are in excess of $44 million dollars [11], contributing to the politicization of this quality improvement strategy [12] in a setting where public tax revenues fund the health care system. The Provincial Auditor has noted the absence of evaluation on the outcomes of Lean implementation [13], although the government reported that the initial outcomes of Lean implementation have been promising. Evidence supporting the use of Lean in health care, however, is weak, with a recent systematic review [14] concluding that Lean interventions: were not associated with patient satisfaction or health outcomes; were negatively associated with financial costs and worker satisfaction; and had potential but inconsistent benefits for safety and patient flow.

Achieving the stated goals of Lean implementation to improve health care quality and promote patient-centredness is contingent upon health care professionals working individually and collectively to embed this new strategy into their organizational and professional contexts [15]. Given the lack of consensus about how to measure the success of complex interventions in general [15, 16], Normalization Process Theory (NPT) [17] offers an approach to understanding the complex influences that make innovations such as the introduction of Lean “workable” in practice settings [18, 19]. The overall aim of this study was to examine the extent to which Lean has become embedded within health care after four years of implementation using an online survey of licensed health care providers. The objectives of the study were to: a) evaluate the implementation processes associated with Lean implementation in the Saskatchewan health care system from the perspectives of health care professionals; and b) identify demographic, training and role variables associated with normalization of Lean.

Although there is a substantial literature relating to implementation of complex interventions in service organizations, understanding the ways in which large-scale transformations are embedded and integrated into evolving and diverse practice settings remains a challenge.8 Normalization Process Theory (NPT) [17] is one approach to better understanding the work required at individual and collective levels to implement complex interventions. NPT addresses the generative processes that underpin: a) implementation (bringing practices into action); b) embedding (when practices are routinely embedded into the everyday work of individuals and groups); and c) integration (when practices are reproduced and sustained within the social matrices of an organization). Four core constructs have been identified that represent the different kinds of work people and organizations do to implement a new practice such as Lean; each construct is comprised of four distinct components (Additional file 1: Table S1).

According to the NPT model, the organizing structures and social norms of practice environments influence the way in which new practices are accommodated and directly affect the organizing factors of a practice, while the group process and conventions affect the way in which a practice is produced and reproduced in actual patterns of interpersonal behavior [17]. Organizing structure and social norms, together with group processes and conventions, affect perceptions of coherence (the meaningful qualities of a practices) of an intervention to those involved in implementation. Coherence affects and is affected by cognitive participation (the enrolment and engagement of individuals and groups), which has a reciprocal relationship with the practices already existing (collective action). Reflexive monitoring refers to the ways a practice is understood and assessed by those involved. This understanding and assessment serves to modify both organizing structures and social norms as well as group processes and conventions.

A substantial body of evidence supports NPT as a means of explaining processes associated with implementation of complex interventions ranging from telecare and e-health [20, 21] to maternity services [22] to management of depression [23, 24]. Recently, the NoMAD instrument was developed to operationalize and measure the core constructs of NPT with the intent that this tool could be customized for a wide range of purposes and across different settings [15].


Ethics and consent

A cross-sectional, online survey design was used. The questionnaire was administered during the month of March, 2016 for a period of 30 days. Ethical approval (BEH 13–294) was obtained from the University of Saskatchewan. Consent to participate was implied by completion of the survey.


The NoMAD tool is a 23 item instrument using a 5-point Likert scale developed to assess healthcare professionals’ perspectives on the processes related to implement complex interventions in practice [15]. Surveys are amongst the methods that can be appropriately employed to study the process of improvement initiatives [25]. Given that the development and testing of tools designed to evaluate the processes involved in quality improvement initiatives such as Lean remains an emerging focus of research [25], the decision to use the NoMAD was made on the basis of this conceptual fit of this tool with the overall objectives of this study. Descriptive analysis and psychometric testing of the NoMAD was recently undertaken by the scale authors using 831 completed questionnaires, with confirmatory factor analysis demonstrating the model achieved an acceptable fit (personal communication, T. Finch, September 2017). Construct validity of the four NPT constructs was supported with internal consistencies ranging from 0.65 (reflexive monitoring) to 0.81 (cognitive participation.

The NoMAD Survey core constructs and components are detailed in Additional file 1: Table S1. Adaptations to the survey were made by the team to reflect the local context and nature of intervention. Because the original item “I will continue to support [the intervention]” implied that providers, in fact, were supportive, and this had not been established, the following revision was used: “I support the use of Lean in health care”. Another adaptation was made to the original item “I have confidence in other people’s ability to use [the intervention]”. Given that we were uncertain about the extent to which Lean practices were actually being implemented in practice by the range of practitioners we planned to survey, this item was modified to read: “I am confident in the skills of people leading the use of Lean”. Finally, the original item worded “I am aware of reports about the effects of [the intervention]” was revised to ask about the outcomes, rather than the effects, of Lean. The remaining 20 items were used directly from the original tool.

At the beginning of the questionnaire, definitions of Lean principles and activities were included to orient and ensure respondents were able to distinguish the intervention from others that may have been occurring in their workplaces. To ensure the definitions reflected the nature of Lean as implemented within the province, the following definitions were identified in consultation with the provincial Health Quality Council (personal communication, T. Verrall, February 2016), the agency responsible for coordinating Lean implementation in the province during the period 2013–15. Lean principles were defined as: a dedication to continuous improvement; focus on eliminating waste; improving the flow of patient; providers and supplies; and ensuring all processes add value to the customer. Problems are identified and addressed by front line members as Lean suggests that the people doing the work are best suited to solving the problem. An extensive list of Lean activities included activities such as Daily Visual Management, Wall Walks, and Rapid Process Improvement Workshops (RPIW).

Respondents were initially asked to identify their level of familiarity with these Lean principles and activities on a 1–10 Likert-type scale and to identify the types of Lean training received. Respondents identified the types of Lean activities they had participated in as health care providers, the duration to which they had been exposed to Lean and the extent to which implementation of Lean had affected their perceived workload. Key demographic and professional characteristics were included based upon our previous work with Lean and health care professionals [26, 27].

Population and sample

Health care providers registered with professional licensing bodies throughout the province were invited to participate in the online survey. Application to conduct the survey with members was approved by the following licensing bodies: the Saskatchewan Registered Nurses Association (n = 10,000 with 4800 opting into receiving any surveys); the Saskatchewan Association of Licensed Practical Nurses (n = 3400); the Registered Psychiatric Nurses of Saskatchewan (n = 879); the Saskatchewan College of Physiotherapists (n = 600); the Saskatchewan College of Pharmacy Professionals (n = 264 hospital-based pharmacists); the Saskatchewan Medical Association (n = 2144); and the Saskatchewan Dietitians Association (n = 355). Several other professional associations with fewer members did not have the resources to participate in the survey. Given that it was not possible to ascertain a priori which professionals may have been using Lean in their practices and thus target the sample only to those providers involved with Lean, the decision was made to survey all members, with the exception of pharmacists who could be identified as practicing in a hospital setting and thus were exposed to Lean implementation. The medical association provided a link to the survey in their bi-weekly newsletters, while the remaining licensing bodies sent a personal email to members containing a link to the survey.


In collaboration with the Social Science Research Laboratory at the University of Saskatchewan, the online questionnaire was pilot-tested with 10 professionals representing the disciplines included in the survey. Minor editorial revisions were made to the questionnaire, which was hosted by Qualtrics™ and was available for 30 days. Reminder emails and announcements were sent 2 weeks following the initial invitations.

Statistical analysis

Statistical analyses were conducted using SAS/STAT® software, Version 9.4 of the SAS System for Windows. Frequencies were computed for all variables and presented as observed/total and percentage. Responses to the main survey items were collapsed for clarity of interpretation into three categories: Agree (strongly agree or agree), Neutral (neither agree nor disagree), and Disagree (strongly disagree or disagree). We summarized responses of Nurses (who comprise the largest occupational category of health care providers) and compared them to the responses of other Health Professionals using chi-square tests to examine differences between the two groups.

In order to identify demographic, training and role variables associated with normalization of Lean, five items from the survey were selected by the research team on the basis of conceptual importance as outcome variables to represent each of the NPT constructs [14]. These items were: “I can see the potential value of Lean for my work” (Coherence item reflecting internalization); “I support the use of Lean in health care” (Cognitive Participation item reflecting activation); “Sufficient training is provided to enable health care providers to implement Lean in health care” (Collective Action item reflecting skillset workability); “Sufficient resources are available to support the implementation of Lean in health care” (Collective Action item reflecting contextual integration); and “The people I work with believe that Lean is worthwhile” (Reflexive Monitoring item reflecting communal appraisal).

Univariate analyses were conducted to determine simple associations. Multinomial logistic regression was used to investigate the relationships between the multivariable relationships between the five variables selected as measures of Lean normalization and variables representing respondents’ professional and personal characteristics, their Lean-related training, and the extent of experience with Lean events. Descriptive analyses were performed to ensure all assumptions underlying the use of multinomial regression were met. The strength of associations was measured based on the Odds Ratio (O-R). The sensitivity of estimated O-R was assessed using bootstrap analysis and 95% percentile confidence intervals (CI) of the estimates were computed from 5000 bootstrap sample. We also reported model-based adjusted p-value for each estimate; adjustment was made following Benjamini and Hochberg [28].


A total of 1378 health care providers completed the survey. Response rate by professional group is reported in Additional file 2: Table S2. Because there were only 35 physician responses, the decision was made to exclude their responses from the analysis. Participants’ profession was not disclosed in 308 surveys, of which 299 had at least 25% missing values to the NoMAD items, leaving nine surveys to enter the study. These nine subjects, along with three more with similar level of missing responses, were excluded from the analysis. On the basis of similarities in role, the various classifications of nurses were combined into one category (Nursing), while the remaining providers were aggregated into a second category labelled health professionals (HP). Responses were retained for: 734 nurses (395 Registered Nurses; 314 Licensed Practical Nurses; 25 Registered Psychiatric Nurses) and 298 Health Professionals (44 Registered Dietitians; 67 Physiotherapists; 56 Occupational Therapists; 131 Pharmacists). Details of the sample selection process has been summarized in a flow-diagram in Fig. 1.

Fig. 1
figure 1

Flow Diagram of Survey Responses

Reflecting the composition of the healthcare workforce in this province [29], most respondents were female and direct care providers. Nurse respondents were older, less likely to work full-time and more likely to work in acute care settings than HPs. There were no significant differences between nurses and HPs in terms of years of experience, primary role or geographic location, with respondents from settings outside of the large urban centers being well-represented (44.1%). A full description of respondent characteristics is located in Table 1.

Table 1 Characteristics of Respondents

In terms of Lean familiarity, integration and the impact on workload (Table 2), wide variability was reported within both groups. While approximately two-thirds of respondents reported no, little or moderate familiarity with Lean principles and activities, 32.6% self-rated their familiarity as substantial or complete. Lean was not currently a part of their work for more than one-third of respondents, and 32% of nurses and 26% of HPS believed that Lean was likely to become part of their normal work in the future. Significantly different (p < 0.0001) impacts on workload were reported by nurses and HPs, with 75% of nurses and 42% of HPs reporting that Lean had increased their workload.

Table 2 Lean Familiarity, Integration and Impact on Workload

Participation in Lean training and activities is reported in Table 3. No significant differences in Lean training were noted between nurses and HPs. While the majority of respondents had attended an introductory session (Kaizen Basics), 17.5% reported they had received no training at all in Lean principles or activities. A separate cross-tabulation (not shown) to examine possible collinearity between reporting a primary work role as a senior or front-line leader and completion of Lean leadership training indicated that close to 70% of leaders had not received this specialized training. The majority of respondents (91%) had participated in at least one Lean activity. The Lean activity most frequently reported by respondents was the Visibility Wall/Wall Walk (71–73%).

Table 3 Participation in Lean Training and Activities

Additional file 3 Table S3 displays responses to the NoMAD construct items. Because further psychometric testing is still required on subscale and cumulative scores, items are reported individually under the constructs and not tallied. Significant differences (p < 0.05 or greater) in responses were noted between nurses and HPs on 13 of the 20 items, with nurses displaying less propensity towards normalization in all cases.

Results for the multivariate multinomial regression models for each of the five selected outcome variables are reported in Tables 4, 5, 6, 7 and 8. Professional characteristics were significantly associated with agreement with the NPT items used as measures of Lean normalization. Having a leadership role, compared to a direct care role, was one of the strongest predictor variables for the cognitive participation, coherence, and reflexive monitoring models. Respondents in leadership roles were significantly more likely to agree than direct care providers with the following survey items: I support the use of Lean in health care (O.R. = 10.29, 95% C.I. 5.65–24.7), I can see the potential value of Lean for my work (O.R = 7.83, 95% C.I. 4.29–17.8) and that the people they worked with believed Lean was worthwhile (O.R. = 4.33, 95% C.I. 2.24–9.74). There were no statistically significant differences between those in a leadership role and those in direct care in terms of responses to the sufficient training or sufficient resources (collective action) items.

Table 4 Multivariate Multinomial Model: Adjusted Associations between Coherence Item and Demographic, Personal and Practice Variables
Table 5 Multivariate Multinomial Model: Adjusted Associations between Cognitive Participation Item and Demographic, Personal and Practice Variables
Table 6 Multivariate Multinomial Model: Adjusted Associations between Collective Action (Training) Item and Demographic, Personal and Practice Variables
Table 7 Multivariate Multinomial Model: Adjusted Associations between Collective Action (Resources) Item and Demographic, Personal and Practice Variables
Table 8 Multivariate Multinomial Model: Adjusted Associations between Reflexive Monitoring Item and Demographic, Personal and Practice Variables

Professional designation as an HP, in contrast to being a nurse, was also associated with agreement responses to items representing cognitive participation, coherence, reflexive monitoring and collective action (training). HPs were significantly more likely than nurses to support the use of Lean in health care (O.R. =4.11, 95% C.I. 2.70–7.20); to see the potential value of Lean for their work (O.R. =2.57, 95% C.I. 1.73–4.12); to indicate that the people they worked with believe that Lean is worthwhile (O.R. =2.06, 95% C.I. 1.21–3.96); and to believe sufficient training had been provided (O.R. =1.57, 95% CI 1.02–2.45). No differences were identified between HPs and nurses in terms of the sufficiency of resources.

In terms of personal and contextual characteristics, increased workload was strongly associated with weaker normalization across all five models. The associations were particularly negative between somewhat increased workload (O.R. =0.19, 95% C.I. 0.11–0.29) or substantially increased workload and support for the use of Lean (O.R. = 0.05, 95% C.I 0.02–0.09). There were no significant associations between age group, years of professional experience, or employment status and any of the NPT outcome items. Weak but significant associations were noted between several normalization outcomes and: a) practice location (urban vs. rural/remote); and b) employment in long-term care.

Lean Leader training, which focused on Lean tools and methods, was positively associated with all normalization outcomes, except for the provision of sufficient resources. Compared to those without Lean Leader training, respondents with leadership training were more likely to agree that: the people I work with believe Lean is worthwhile (O.R. =5.21, 95% CI = 2.03–13.40); I can see the potential value of Lean for my work (O.R. =3.72, 95% CI = 1.53–18.70); and I support the use of Lean (O.R. = 3.45, 95% CI = 1.24–14.90). Compared to those with any type of training, respondents with no training were much less likely to support the use of Lean in health care (O.R = 0.28, 95% CI = 0.10–0.77), but did not differ from those with training in terms of agreement with the other items. Neither completion of Kaizen Basics training nor participation in Lean activities were associated with any of the outcome items.


More than four years after the ambitious and costly launch of Lean as a province-wide strategy to improve health care quality and promote patient-centredness within healthcare in Saskatchewan, the findings of our survey reveal that major gaps remain in embedding the principles and activities of Lean into everyday health care practice, particularly among nurses and direct care providers. Assessing implementation processes of initiatives such as Lean that are ongoing can serve to target issues that require additional support and attention.

Significant differences in responses were evident between: leaders vs. direct care providers; nurses vs. other health professionals; and providers who reported increased workload as a result of Lean vs. those who did not. Respondents in leadership positions were much more likely to view the implementation and benefits of Lean in a positive manner than were direct care providers. More than 75% of respondents indicated that neither sufficient training nor resources (collective action) had been made available to them for the implementation of Lean. Compared to other health care providers, nurses were more likely to report that Lean increased their workload.

The importance of strong leadership in effecting the cultural changes needed for major health reform has been well-recognized [30]. A wide gulf between the perspectives of leaders and direct care providers was apparent, however, with respect to the constructs of coherence, cognitive participation and reflexive monitoring. Based on the premise that senior leaders “are central to ensuring that Lean will pervade the organization’s management system and [to] provide an example of Lean principles to other personnel” [2], major investments had been made by the Ministry of Health to ensure that those in health leadership roles were “committed to learning and applying Lean principles” [2]. These investments at the senior leadership level included a rigorous certification program that included didactic and experiential training, including leadership of Lean events, and an opportunity to visit organizations in the U.S. that have long experience with the use of Lean in manufacturing and health care [31]. Given that only 30% of leaders had Lean Leader training, the basis for the strong support shown by leaders for Lean implementation might be better explained by factors other than training alone. A subsequent paper will compare the narrative responses on the survey between leaders and nurses to shed further light on this finding.

Adequate training and resources are foundational to any attempt to transform health care [32], but the majority of respondents believed these had not been sufficient (Additional file 2: Table S2). Only one quarter of all respondents believed that training had been sufficient and 18% felt that resources had been sufficient to implement Lean. The nature and type of Lean education and participation in Lean activities had interesting implications for the extent of normalization. The majority of respondents reported attending some form of Lean training, but only the variable of having no Lean training at all was associated with (lack of) support for Lean. Kaizen Basics training, which was originally an eight hour introductory program (later scaled back), was intended to provide staff with “a broad overview of Lean principles and methods, so they understand the changes taking place and have a sense of what to expect when they are invited to participate in an improvement event” [33]. This training was attended by three quarters of respondents, but we found no associations between attendance at these sessions and the normalization outcomes examined in this study.

Participation in Lean activities was not associated with the items representing normalization of the Lean approach, in spite of the appeal of providing “hands-on training” as a pedagogical strategy. Our findings contrast with those reported in a recent study examining the introduction of Lean in primary care that noted that the time and intensity of exposure to redesign activities was a positive influence on acceptance by providers [34].

The anticipated benefits of the implementation of Lean are contingent upon widespread dissemination of the philosophy and processes throughout the entire system, resulting in a major cultural shift that focuses on continuous performance improvement [32]. Critical factors such as perceived lack of ownership and subcultural diversity have been noted to derail attempts at transforming culture when direct care providers are not engaged [12]. As recently noted by Bohmer [35], “delivery of care is ultimately governed by structure and process at the ward, clinic or practice level”. Lean implementation in health care can be perceived to compromise professional autonomy [36] and can challenge established professional hierarchy [12], thereby promoting resistance from key stakeholders required to enact the transformation within the system. McIntosh [12] noted that the Saskatchewan government failed to appreciate that key factors such as nurses (and physicians) had the independent ability to push back against this initiative in response to top-down implementation and that Lean “cannot be implemented in the top-down, directive manner displayed to date but must accommodate other powerful decision-makers in these sectors” [12].

Although nurses constitute the largest proportion of licensed health care providers in health care system and their contributions are well-recognized to be pivotal in ensuring patient safety and high-quality care [37], the perspectives of nurse respondents suggest that there are fundamental differences in the way in which Lean impacts the work of nurses compared to other health professionals, despite similarities in Lean training and participation in activities. Nurses play many critical roles in health care delivery, but globally share concerns about understaffing and inadequate training and support [38]. Nurses have been found to encounter an average of 8.4 work system failures in an 8-h shift, a fact compounded by frequent interruptions of their work [39, 40]. Nurses, compared to physicians, have been found to have less work autonomy, fewer professional development opportunities and fewer options for career change [41]; nurses are considered the health professionals most exposed to work strain that compromises their physical and psychological well-being [41, 42]. The implications of Lean for nurses’ work, particularly for bedside nurses, requires further thoughtful consideration. A 2014 survey conducted by the Saskatchewan Union of Nurses [43] reported a statistically significant negative effect of Lean on nurse engagement, usefulness, patient care, time for patient care, workplace issues, availability of supplies, workload, stress and patient care. Our results lend support to the notion of a potential misalignment between the principles and activities of Lean as it had been implemented and the work of clinical nursing, which we are currently investigating in related projects.

Our findings also point to the importance of context in the implementation of Lean. Greater agreement with normalization constructs was noted for non-urban respondents, whose practice involves different demands than larger urban settings and highlights again the importance of context in introducing system-wide change. Context is a critical consideration in the implementation of large-scale interventions [44,45,46,47], with “dynamic elements of context play[ing] a powerful role in shaping participants’ capacity and potential” [46] Complex interventions that cannot be integrated smoothly within organizational contexts and do not prove to be workable alongside other tasks and duties are unlikely to be normalized [48]. Conversely, adaptations to organizational contexts and tasks and duties may act to facilitate the normalization of interventions such as Lean.

Our results highlight the importance of attending to increased job demands that may result from new interventions. Job demands are aspects of work associated with physical, emotional and cognitive efforts and can translate into job stressors and burn-out if sufficient resources are not available [48]. Perceived increased workload was the only variable to be a significant predictor of negative responses across all five of the models. The implementation of Lean was reported to have different implications for the workload of individual respondents, pointing to the need for leaders to carefully evaluate how interventions may impact upon the workload of their subordinates.

The use of Lean as a large-scale quality improvement strategy has generated controversy on multiple fronts, but evidence on outcomes of Lean implementation is still emerging. Batalden et al. [49] have questioned the wisdom and appropriateness of implementing quality improvement systems such as Lean which are underpinned by “goods-dominated” logic systems in health care. Improvement strategies that recognize the dynamic environment of health care provision, where goods and services are consumed and produced simultaneously, may be more easily integrated into health care practice.


While our cross-sectional survey findings include the responses of over 1000 health care professionals with a wide range of professional roles, diverse practice settings and geographic locations, we acknowledge the potential for non-response bias resulting from the lower than desired response rates. Because reasons for nonresponse are not known to researchers, one strategy to assess the effect of non-response bias is to analyze the known demographic or organizational characteristics of the population [50]. The demographic profile of respondents in this survey reflects some of the key aspects of the Canadian healthcare workforce. Eighty per cent of all health care providers in Canada are female with an average age of 43 years [29], although the proportion of nurses was over-represented in this survey, given that registered nurses, licensed practical nurses and registered psychiatric nurses account for slightly more than one-third of all Canadian health care workers [29]. Within the province of Saskatchewan, the 2011 census indicates that one-third of the population resides in a rural location [51], suggesting that the views of rural health care professionals (41.4%) may have also been over-represented.

Sampling bias was mitigated by having professional associations distribute the invitation to participate to all members, with the exception of Registered Nurses whose members could choose to opt out of surveys. According to Dillman and colleagues [52], our sample sizes of 734 nurses out of 10,000 possible respondents and 298 HPs out of 1219 possible respondents (excluding physicians) achieved the completed sample size necessary for a ± 5% margin of error in both groups. Strategies to maximize response rates, such as having the associations send reminders [49], were employed.

The decision to recruit professionals through their licensing bodies for this online survey was made consciously to maintain independence of the survey from those with vested interests in the outcomes of evaluation, such as employers, government or unions. Because exposure to Lean had been widespread throughout Saskatchewan, it was not possible to identify a priori individuals who were actually engaged in Lean implementation versus those who were not in order to better tailor the sampling frame. Potential explanations for the low response rate are myriad - mode of administration, time of year, survey fatigue, email or work overload, and perceived lack of relevance are just a few possibilities. Surveys, as a research strategy, are recognized to be challenged by declining response rates throughout developed countries [53]. In spite of these issues, the survey method allowed us to hear the voices of a broad cross-section of health providers across the province in a manner other research designs would not have.

Over 300 respondents chose not to identify their profession, but went on to complete the remainder of the survey. Their responses were not included in this analysis, but subsequent analysis is planned to compare responses between those who did and did not list their professions. Although Lean reforms were putatively meant as a vehicle to empower both patients and workers [12], the highly polarized nature of the debate surrounding Lean in health care, an atmosphere of mistrust in health care and fear of reprisal [54] from employers may have contributed to the reluctance to identify profession on the survey. Hearing the voices and perspectives of health care providers is essential to authentic and sustainable transformation of health care [55]. Creating opportunities to engage direct providers in reform would support achievement of this large-scale transformation.

Low rates of participation on the part of physicians and subsequent exclusion of their perspectives on this topic also limits the conclusions we can draw from this cross-sectional survey. Physicians’ lack of participation was unfortunate, but not surprising, in a professional group recognized to have low survey response rates [56] and when the regulations of their professional association did not allow for the individual email contacts that were sent to the other groups of health care providers. Physician leadership and engagement has been noted as a strong lever for driving interdisciplinary work forward [57] and are critical to the successful implementation of health care reforms [58], making their viewpoints central to assessment of implementation of complex interventions such as Lean. Further work is needed to identify strategies to hear the perspectives of physicians.

The outcomes for this study were selected to reflect the extent to which Lean had been embedded in Saskatchewan health care. While a number of survey instruments were evaluated for use in this study, the NoMAD was selected on the basis of its strong conceptual underpinnings and relevance to our objectives. Currently undergoing psychometric assessment (T. Finch, personal communication, September 2016), the NoMAD requires additional evaluation to guide overall and subscale scoring and interpretation. Because each of the items was considered to have face validity and reflected a key aspect of normalization important for assessment in this study and because scoring directions for the NoMAD are not yet available, the decision was made to conduct the analyses using individual items.


Normalization Process Theory offered a feasible and highly applicable model through which to assess implementation processes of the province-wide adoption of Lean. Use of a modified NoMAD survey allowed us to capture key elements of implementation that were of primary interest. The findings of this project provide a window into the perspectives of health care professionals who are in the midst of undergoing large scale transformation of a health care system and can inform future research, practice and policy related to implementation processes.

The implementation of Lean in Saskatchewan health care represents an ambitious, high-investment, multi-pronged attempt at a large-scale transformation “aimed at coordinated, system-wide change affecting multiple organizations and care providers” [4]. Recognizing that successful transformation is the result of constant small-scale changes to structures and process over long periods of time, Bohmer [35] suggests that the actual model of quality improvement (Lean or alternatives) is less important to achieving the goals of health reform than internalized repetitive and consistent process. The findings of our survey highlight that, while substantial progress has been made in the province-wide effort to implement, embed and integrate Lean into the Saskatchewan health care system over the past few years, ongoing support and innovative implementation strategies will be required if large-scale quality improvement initiatives are to become a routine part of practice for health care providers across the province and produce the intended improvements in quality and patient-centredness.



Confidence Interval


Normalization Process Theory


Odds Ratio


United States


  1. Government of Saskatchewan. Continuous Quality Improvement. Accessed 26 Mar 2018.

  2. Black J, Miller D. The Toyota way to healthcare excellence: increase efficiency and improve quality with lean. Chicago: Health Administration Press; 2008.

    Google Scholar 

  3. Lawal A, Rotter T, Kinsman L, Sari N, Harrison L, Jeffery C, et al. Lean management in health care: definition, concepts, methodology and effects reported (systematic review protocol). System Rev. 2014;3:103.

    Google Scholar 

  4. Best A, Greenhalgh T, Lewis S, Saul JE, Carroll S, Bitz J. Large-system transformation in healthcare a realist review. Milbank Q. 2012;90:421–56.

    PubMed  PubMed Central  Google Scholar 

  5. Ferlie EB, Shortell SM. Improving the quality of health care in the United Kingdom and the United States: a framework for change. Milbank Q. 2001;79:281–315.

    CAS  PubMed  PubMed Central  Google Scholar 

  6. Greenhalgh T, Robert G, MacFarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82:581–629.

    PubMed  PubMed Central  Google Scholar 

  7. Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients’ care. Lancet. 2003;362:1225–30.

    PubMed  Google Scholar 

  8. Nystrom ME, Strehlenert H, Hansson J, Hasson H. Strategies to facilitate implementation and sustainability of large system transformations: a case study of a national program for improving quality of care for elderly people. BMC Health Serv Res. 2014;14:401.

    PubMed  PubMed Central  Google Scholar 

  9. Mann D. The missing link: lean leadership. Front Health Serv Manag. 2009;26:15–26.

    Google Scholar 

  10. Mann D. Creating a lean culture: tools to sustain lean conversions. New York: Productivity Press; 2005.

    Google Scholar 

  11. Sari N, Rotter T, Goodridge D, Harrison L, Kinsman L. An economic evaluation of a system wide Lean approach: cost esstimation of the implementation of Lean in the Saskatchewan healthcare system for 2012–2014. BMC Health Serv Res. 2017;17:523.

    PubMed  PubMed Central  Google Scholar 

  12. McIntosh T. Rolling-out lean in the Saskatchewan health care system: politics derailing policy. Health Ref Obs. 2016;4(1):Article 3.

    Google Scholar 

  13. Ferguson J. Report of the provincial auditor, 2015 report, volume 2. 2015. Accessed 26 Mar 2018.

  14. Moraros J, Lemstra M, Nwankwo C. Lean interventions in healthcare: do they actually work? A systematic literature review. Int J Qual Health Care. 2016;28:150–65.

    PubMed  PubMed Central  Google Scholar 

  15. May C, Rapley T, Mair FS, Treweek S, Murray E, Ballini L et al. Normalization process theory on-line users manual, toolkit and NoMAD instrument. 2015. Accessed 23 Sept, 2016.

    Google Scholar 

  16. Finch T. Can we measure health care implementation processes using normalization process theory (NPT): development of the NOMAD survey tool. Accessed 26 Mar 2018.

  17. May CR, Mair F, Finch T, MacFarlane A, Dowrick C, Treweek S, et al. Development of a theory of implementation and integration: normalization process theory. Imp Sci. 2009;4:29.

    Google Scholar 

  18. Finch TL, Mair FS, O’Donnell C, Murray E, May CR. From theory to ‘measurement’ in complex interventions: methodological lessons from the development of an e-health normalisation instrument. BMC Med Res Meth. 2012;12:69.

    Google Scholar 

  19. May CR, Sibley A, Hunt K. The nursing work of hospital-based clinical practice guidelines implementation: an explanatory systematic review using normalisation process theory. Int J Nurs Stud. 2014;51:289–99.

    PubMed  Google Scholar 

  20. Morrison D, Mair FS. Telehealth in practice: using normalisation process theory to bridge the translational gap. Prim Care Respir J. 2012;20:351–2.

    Google Scholar 

  21. Mair FS, Hiscock J, Beaton SC. Understanding factors that inhibit or promote the utilization of telecare in chronic lung disease. Chronic Illn. 2008;4:110–7.

    PubMed  Google Scholar 

  22. Forster D, Newton M, McLachlan H, Willis K. Exploring implementation and sustainability of models of care: can theory help? BMC Pub Health. 2011;11(Suppl 5):S8.

    Google Scholar 

  23. Gask L, Rogers A, Campbell S, Sheaff R. Beyond the limits of clinical governance: the case of mental health in primary care. BMC Health Serv Res. 2008;8:63.

    PubMed  PubMed Central  Google Scholar 

  24. Gunn J, Palmer V, Dowrick C, Herrman H, Griffiths F, Kokanovic R, Blashki G, et al. Embedding effective depression care: using theory for primary care organisational and systems change. Impl Sci. 2010;5:1–15.

    Google Scholar 

  25. Portela MC, Pronovost PJ, Woodcock T, Carter P, Dixon-Woods M. How to study improvement interventions: a brief overview of possible study types. BMJ Qual Saf. 2015;24:325–36.

    PubMed  PubMed Central  Google Scholar 

  26. Goodridge D, Westhorp G, Rotter T, Dobson R, Bath B. Lean and leadership practices: development of an initial realist program theory. BMC Health Serv Res. 2015;15:362.

    PubMed  PubMed Central  Google Scholar 

  27. Rotter T, Kinsman L, Bath B, Goodridge D, Harrison L, Dobson R, Sari N, Jeffery C, Adegboyega L, Kutz M, Bourassa C, Westhorp G. A first phase evaluation of Saskatchewan's Lean health care transformation: final report. Accessed 26 Mar 2018.

  28. Benjamini Y, Hochberg Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J Royal Stat Soc. 1995;57:289–300.

    Google Scholar 

  29. Canadian Institute for Health Information. Canada’ health care providers: provincial profiles. 2013. Accessed 23 Aug, 2017.

    Google Scholar 

  30. Scott T, Mannion R, Davies HTO, Marshall MN. Implementing culture change in health care: theory and practice. Int J Qual Health Care. 2003;13:111–8.

    Google Scholar 

  31. Gerlach L, Johnson S, Kavas S, Latta L for the Saskatchewan Institute for Health Leadership. Saskatchewan health system: staff engagement in the lean transformation. 2013. Accessed 23 Aug, 2017.

    Google Scholar 

  32. Mackenzie J, Hall W. Lean in Canadian health care: doing less while achieving more. Conference Board of Canada. 2014. Accessed 15 Oct, 2016.

    Google Scholar 

  33. Health Quality Council. Saskatchewan health care management system, 2013. 21 Oct, 2016.

    Google Scholar 

  34. Hung D, Gray C, Martinez M, Schmittdiel J, Harrison MI. Acceptance of lean redesigns in primary care: a contextual analysis. Health Care Manag Rev. 2015;42(3) [epub ahead of print]

  35. Bohmer RMJ. The hard work of health care transformation. NEJM. 2016;375:709–11.

    PubMed  Google Scholar 

  36. Poksinska B. The current state of lean implementation in health care: literature review. Qual Manag Health Care. 2010;19:319–29.

    PubMed  Google Scholar 

  37. Neill D. Nursing workload and the changing health care environment: a review of the literature. Admin Issues J: Educ Prac and Res. 2016;1:132-143.

  38. The All-Party Parliamentary Group on Global Health. Triple impact: how developing nursing will improve health, promote gender equality and support economic growth,2016. Accessed 26 Oct, 2016.

    Google Scholar 

  39. Tucker AL, Spear SJ. Operational failures and interruptions in hospital nursing. Health Serv Res. 2006;41:643–62.

    PubMed  PubMed Central  Google Scholar 

  40. Al T. The impact of operational failures on hospital nurses and their patients. J Oper Manag. 2004;22:151–69.

    Google Scholar 

  41. Aiken LH, Sermeus W, Van den Heede K, Sloane DM, Busse R, McKee M, et al. Patient safety, satisfaction and quality of hospital care: cross sectional surveys of nurses and patients in 12 countries in Europe and the United States. BMJ. 2013;344:e1717.

    Google Scholar 

  42. Janssen PPM, De Jonge J, Bakker AB. Specific determinants of intrinsic work motivation, burnout and turnover intentions: a study among nurses. J Adv Nurs. 1999;29:1360–9.

    CAS  PubMed  Google Scholar 

  43. Saskatchewan Union of Nurses. Lean: a safe and effective tool for registered nursing? 2014. Accessed 22 Aug, 2017.

    Google Scholar 

  44. Squires JE, Estabrooks CA, Scott S, Cummings G, Hayduk L, Kang SH, et al. The influence of organizational context on the use of research by nurses in Canadian pediatric hospitals. BMC Health Serv Res. 2013;13:351.

    PubMed  PubMed Central  Google Scholar 

  45. May CR, Johnson M, Finch T. Implementation, context and complexity. Implem Sci. 2016;11:141.

    Google Scholar 

  46. Dixon-Woods M, Martin GP. Does quality improvement improve quality? Future Hosp Board. 2016;3:191–4.

    Google Scholar 

  47. MacFarlane A, O’Reilly-de BM. Using a theory-driven conceptual framework in qualitative health research. Qual Health Res. 2012;22:607–18.

    PubMed  Google Scholar 

  48. Montgomery A, Spanu F, Baban A, Panagopoulou E. Job demands, burnout and engagement among nurses: a multi-level analysis of ORCAB data investigating the moderating effect of teamwork. Burnout Res. 2015;2:71–9.

    Google Scholar 

  49. Batalden M, Batalden P, Margolis P, Armstrong G, Opipari-Arrigan L, Hartung H. Coproduction of healthcare service. BMJ Qual Saf. 2016;25:509–17.

    PubMed  Google Scholar 

  50. Lewis EF, Hardy M, Snaith B. Estimating the effect of nonresponse bias in a survey of hospital organizations. Eval Health Prof. 2013;36:330–51.

    PubMed  Google Scholar 

  51. Statistics Canada. Population, urban and rural, by province and territory, 2017. Accessed 22 Aug, 2017.

    Google Scholar 

  52. Dillman DA, Smyth JD, Christian LM. Internet, mail and mixed mode surveys: the tailored design method. 3rd ed. Hoboken: John Wiley & Sons, Inc; 2008.

    Google Scholar 

  53. Groves RM, Peytcheva E. The impact of nonresponse rates on nonresponse bias. Pub Opin Quar. 2008;72:167–89.

    Google Scholar 

  54. Castel E, Ginsburg L, Zaheer S, Tamim H. Understanding nurses’ and physicians’ fear of repercussions for reporting errors: clinician characteristics, organization demographics, or leadership factors? BMC Health Serv Res. 2015;15:326.

    PubMed  PubMed Central  Google Scholar 

  55. Canadian Foundation for Healthcare Improvement. Ingredients for successful patient engagement: the power of relationships. n.d. Accessed 3 Dec, 2016.

  56. Cunningham CT, Quan H, Hemmelgarn B, Noseworthy T, Beck CA, Dixon E, et al. Exploring physician response rates to web-based surveys. BMC Med Res Methodol. 2015;5:32.

    Google Scholar 

  57. O’Reilley PPO, Lee SH, O’Sullivan M, Cullen W, Kennedy C, MacFarlane A. Assessing the facilitators and barriers of interdisciplinary team working in primary care using normalization process theory: an integrative review. PLoS One. 2017;12:e0177026.

    Google Scholar 

  58. Densi J-L, Baker GR, Black C, Langley A, Lawless B, Leblanc D et al. Exploring the dynamics of physician engagement and leadership for health system improvement: prospects for Canadian healthcare systems. 2013. Accessed 14 Nov, 2016.

    Google Scholar 

Download references


We gratefully acknowledge the assistance of Dr. Josh Lawson, Associate Professor in the Canadian Centre for Health and Safety in Agriculture, University of Saskatchewan for providing expert statistical consultation as well as Dr. Tracy Finch, Senior Lecturer in Psychology of Health Care in the Institute of Health and Society at Newcastle University for helpful guidance in using the NoMAD survey. This study was funded by a grant from the College of Medicine, University of Saskatchewan.


This project was funded by the College of Medicine, University of Saskatchewan.

Availability of data and materials

The datasets generated or analyzed during the current study are not publicly available due to them containing information that could compromise research participant privacy/consent.

Author information

Authors and Affiliations



DG conceptualized the study. DG, EH, TR, RD, GG and SU were involved in adapting the NOMAD instrument. DG and EH worked with licensing bodies to gain approval for administration to professional members. DG oversaw the administration of the survey. MR, DG and JL developed the plan for statistical analysis. MR conducted the statistical analyses. DG prepared the initial draft of the manuscript. All authors contributed to the interpretation of the data, reviewed and approved the final paper.

Corresponding author

Correspondence to Donna Goodridge.

Ethics declarations

Ethics approval and consent to participate

Approval to conduct this study was grant by the University of Saskatchewan Behavioral Research Ethics Board (13–294). Consent was implied by responding to the questionnaire.

Consent for publication

Not applicable.

Competing interests

The authors declare they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional files

Additional file 1:

Table S1. Normalization Process Theory Core Construct. (DOCX 13 kb)

Additional file 2:

Table S2. Responses by Professional Group. (DOCX 12 kb)

Additional file 3:

Table S3. Revised NoMAD Survey Responses by Profession. (DOCX 14 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Goodridge, D., Rana, M., Harrison, E.L. et al. Assessing the implementation processes of a large-scale, multi-year quality improvement initiative: survey of health care providers. BMC Health Serv Res 18, 237 (2018).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: