Lean Enterprise Transformation in VA: a national evaluation framework and study protocol

Abstract

Background

The goal of Lean Enterprise Transformation (LET) is to go beyond simply using Lean tools and instead embed Lean principles and practices in the system so that it becomes a fundamental, collective mindset of the entire enterprise. The Veterans Engineering Resource Center (VERC) launched the Veterans Affairs (VA) LET pilot program to improve quality, safety, and the Veteran’s experience. A national evaluation will examine the pilot program sites’ implementation processes, outcomes and impacts, and abilities to improve LET adoption and sustainment. This paper describes the evaluation design for the VA LET national evaluation and describes development of a conceptual framework to evaluate LET specifically in healthcare settings.

Methods

A targeted literature review of Lean evaluation frameworks was performed to inform the development of the conceptual framework. Key domains were identified by a multidisciplinary expert group and then validated with key stakeholders. The national evaluation design will examine LET implementation using qualitative, survey, and quantitative methods at ten VA facilities. Qualitative data include site visits, interviews, and field observation notes. Survey data include an employee engagement survey to be administered to front-line staff at all pilot sites. Quantitative data include site-level quality improvement metrics collected by the Veterans Services Support Center. Qualitative, quantitative, and mixed-methods analyses will be conducted to examine implementation of LET strategic initiatives and variations in implementation success across sites.

Discussion

This national evaluation of a large-scale LET implementation effort will provide insights helpful to other systems interested in embarking on a Lean journey. Additionally, we created a multi-faceted conceptual framework to capture the specific features of a Lean healthcare organization. This framework will guide this evaluation and may be useful as an assessment tool for other organizations interested in implementing Lean principles at an enterprise level.

Background

Finding ways to improve quality and efficiency while controlling costs is crucial in American health care. Lean is one promising management approach to achieving these aims. “Lean thinking,” a quality improvement philosophy, is a bundle of concepts, methods and tools originated by the Toyota Motor Company and has been successfully employed in a variety of manufacturing industries [1]. Because of these successes, the healthcare sector has been trying to implement Lean in settings including laboratories [2, 3], intensive care units [4], emergency departments [5, 6], operating rooms [7], and pharmacies [8, 9]; and a few even have tried to implement Lean hospital-wide [10].

Most Lean implementation efforts in healthcare organizations result in modestly positive short-term improvements in targeted areas and have not successfully created transformational change that encompasses the organization’s culture and administrative structures [11,12,13,14]. Recognizing that organization-wide change is difficult and requires going beyond the application of Lean tools to foster a culture of continuous learning and improvement, organizations are increasingly turning to Lean Enterprise Transformation (LET) [15, 16]. The goal of LET is to embed Lean principles, strategies, practices and behaviors in the system so that once there is a critical mass of Lean behavior in the enterprise, it becomes a “way of life” and a fundamental, collective mindset of the entire enterprise.

In 2014, the Veterans Health Administration (VHA) faced a crisis following an Office of the Inspector General report describing inappropriate scheduling practices by staff, Veterans experiencing significant delays in receiving care, and alleged deaths related to the delays [17]. The subsequent rebuilding of the Department of Veterans Affairs (VA) healthcare system as a high-performance network included implementing Lean management strategies to improve quality, safety, and the Veteran’s experience [18, 19]. Initial deployment of Lean management at individual VA medical centers began in 2011, however, in 2015, the VHA National Leadership Council formally adopted Lean management as a national quality improvement strategy [20]. This VA LET program consists of a centralized deployment strategy that includes sensei (coaching) services for each pilot site along with programmatic and implementation support and guidance. To our knowledge, this innovative program is the largest, multi-site Lean enterprise transformation effort to date.

We designed an evaluation of the VA LET pilot program that uses principles based in the community-based participatory research (CBPR) paradigm which recommends involvement of stakeholders in research tasks and procedures [21]. This partnership between researchers and operational partners facilitates the integration of knowledge and produces findings that have “real world” value [22]. While CBPR represents our approach to research, we needed a comprehensive conceptual framework to guide our evaluation design. Previously developed conceptual frameworks do not focus on the organizational transformation process or include other concepts unique to Lean, such as respect for people, that would be critical for studying effective LET implementation. The conceptual framework described in this evaluation protocol provides a standardized approach to evaluating the VA LET pilot sites. Additionally, we propose a mixed-methods evaluation design that seeks to: (1) assess the implementation of the strategic initiatives, and variations in implementation success across sites, in order to inform successful implementation of future initiatives; (2) understand the specific contextual characteristics and deployment mechanisms that facilitate success of LET programs; (3) identify provider and staff-level barriers and facilitators to adoption of LET (e.g., knowledge, attitudes, experience); and (4) to determine if and how LET programs result in the transformation of key aspects of an organization, such as culture.

Methods/design

Setting

The VHA, comprised of 170 hospitals and 1063 outpatient sites of care, provides inpatient, outpatient, mental health, rehabilitation, and long-term care services to more than nine million enrolled Veterans. VHA, the nation’s largest integrated healthcare system, employs more than 306,000 full time health care professionals and support staff [23].

In 2011 the VA Veterans Engineering Resource Center (VERC) developed the VA LET program and deployment plan that consists of several components considered critical to the successful implementation of Lean. These include conducting annual Transformational Plan of Care (TPOC) events, developing educational and communication plans, identifying and developing value stream areas (VSAs), engaging in continuous daily improvement activities, and receiving expert-level “sensei” facilitation support at the executive and mid-organizational levels [24]. The VERC, the VA Quality Enhancement Research Initiative (QUERI), and investigators at the VA Palo Alto and Boston Healthcare Systems, formed the LET Partnered Evaluation Center to assess the pilot program sites’ implementation processes, outcomes and impacts and abilities to improve LET adoption and sustainment.

This partnered evaluation includes ten VA medical centers that as early adopters of Lean contracted with VERC to deploy the LET framework and receive sensei facilitation. These pilot sites are geographically dispersed across the US and range in VA medical center complexity designation from least (level 3c) to most complex (level 1a). This rating, obtained from VA administrative data sources, is comprised of factors including patient population characteristics, clinical services complexity, research funding, and level of medical training provided (Table 1).

Table 1 Characteristics of VA medical centers participating in the LET evaluation

The VERC instructed each site to select one clinical and one non-clinical value stream for initial focus from the following designated priority areas: inpatient medicine, inpatient surgery, emergency medicine, primary care, outpatient specialty care and outpatient mental health, as well as logistics and human resources.

Conceptual framework design

The conceptual framework was developed in two stages: an information gathering process and an interactive process. First, to identify key domain areas, we reviewed the literature for existing frameworks and conceptual models that evaluated Lean in health care and focused on successful implementation and organizational transformation. This included searching keywords including Lean, implementation, model and/or framework, and “snowballing” based on sentinel papers that emerged (including both peer-reviewed literature and grey literature such as government reports). Additional domain areas were solicited from key informants that included our partners involved in implementing Lean and partners responsible for funding the Lean program.

Next, literature review results and key informant input were evaluated using an informed, interactive process. We convened a multidisciplinary academic research group (n = 12) with expertise in Lean, quality improvement, organizational theory, management, and program evaluation. Initial key domains and constructs were selected during a half-day in-person meeting, which was then followed by a series of biweekly virtual meetings for refinement. Agreement on relevant domains and constructs was reached through discussion and consensus. The final model was presented and discussed with key stakeholders, including healthcare providers, VHA policymakers and national experts outside of the VHA for input and modification.

Evaluation design

We propose a three-part mixed-methods design to evaluate LET implementation at the pilot sites. Interview teams will first conduct semi-structured interviews and site visits; findings will be qualitatively analyzed to identify themes and patterns of successful LET implementation. The results will also be used to develop a Lean implementation survey, the second component of the evaluation. The third component includes an analysis of relevant quality improvement metrics created and collected by the VHA to assess changes in clinical and non-clinical processes and outcomes at each VA facility. Below, we present the details of each design component.

Recruitment and sampling

Qualitative Site Visit/Interviews

The pilot sites will be invited to participate in the evaluation per an introductory email from the VERC leadership and the QUERI director. For each site, we will identify a site contact to assist with identifying, inviting, and scheduling the key informant interviews and coordinating the site visit activities. Prior to any site visit, the site contact will provide information about the site’s history with process improvement and the LET program (i.e., when they first started working with VERC to implement LET, current and planned value streams, date of last TPOC event, etc.). Interview participants will include the medical center director and other executive leadership (e.g., Chief of Staff), the director of systems redesign/process improvement, the Lean senseis (coaches), the two value stream process owners - who will typically be the Director/Service Chief of the departments in which the value stream will occur (e.g., Inpatient Medicine, Human Resources), middle managers and front-line supervisors working on one value stream, and front-line staff working on the other value stream.

Staff survey

All ten pilot sites will be invited to participate in a web-based survey. With approval from the medical center director, the survey will be administered to all employees below the level of executive leadership. Following the Dillman procedure [25], the medical center director will send to all staff, at one week intervals, a series of email notifications: a pre-survey email informational notice, an email containing the link to the online survey, and two follow-up reminder emails.

Quantitative Analysis

The VHA has a comprehensive database of metrics to support performance improvement. One such tool is a sophisticated matrix of over 25 measures and measure sets known as Strategic Analytics for Improvement and Learning (SAIL) [26, 27]. It includes all Veteran-relevant metrics reported to the Centers for Medicare and Medicaid Services and the National Committee for Quality Assurance, as well as assessments of factors thought to enable quality and safety improvement (e.g., employee morale, nursing turnover, leadership vacancies, and selected utilization metrics). The SAIL metrics related to the targeted value stream areas will be the focus of the quantitative stage of the evaluation.

Data collection

Qualitative Site Visit/Interviews

Three rounds of interviews will be conducted at approximately six-month intervals. The first round will serve as a baseline assessment of the sites’ Lean status. They will be conducted during a two-day on-site visit by at least two evaluation members and the subsequent rounds will be telephone-based interviews. During the in-person visit, the site-visit team will tour the facility, paying special attention to visual management tools such as huddle boards, recognition for Lean participation and accomplishments, and other visual displays related to Lean efforts. They will note whether the visual displays contain relevant and up-to-date measures, and reflect alignment between the individual work area’s focus of efforts and the medical center’s strategic goals. The visit will also include attending huddle meetings and project report-out meetings. The site-visit team will document their observations using a Field Observations and Impressions Summary form (Additional file 1). Using the structured form, the team will describe their observations as they relate to framework domain areas and summarize their overall impressions of the site. Photographs of the visual management displays will also be collected. For consistency and to evaluate change over time, the Round 2 interviews will target the same individuals in the Round 1 value stream areas (when applicable) and at least one member of the Round 1 site-visit team will conduct Round 2 interviews. Round 3 interviews will focus on obtaining a summative assessment from three to five key informants (two executive leaders, two value stream process owners and the system redesign director) regarding the site’s Lean journey to date.

Interview guide questions, based on our conceptual framework’s domains and constructs, will elicit key informants’ current perceptions and knowledge about the LET deployment efforts at their facility. Guide development will be iterative and reviewed by Lean experts and operational partners to allow for input and modification. The guides will include questions specific to the interview round as well as questions revisited each time to document progress in the site’s Lean journey. For example, only Round 1 interviews will include questions to assess the site’s history with process improvement and to better understand the cultural context of the site. The Round 2 interview guide will incorporate questions about the effects of management changes and ongoing commitment to Lean. The final interview round (Round 3) will assess the trajectory of Lean efforts, especially to determine whether prior accomplishments have been sustained, whether Lean has been spread to additional parts of the organization, and the lessons learned. Unique interview guides will be created for each interview type (leadership, value stream owners, middle managers, front-line staff, senseis, and system redesign). All interviews will be conducted using the interview guide appropriate for the individual or group (middle manager and front-line staff) being interviewed. Trained members of the evaluation team will conduct the interviews designed to take approximately one hour to conduct. The interviews will be audiotaped and subsequently transcribed verbatim with the participant’s permission.

A codebook will be developed using a deductive open-coding approach with codes largely generated from the conceptual framework [28]. Trained coders will code interview transcripts and site visit field notes in Atlas.ti. Coders will meet weekly, and with the larger evaluation team bi-weekly, to ensure a shared interpretation of the codes and to discuss potential inductive generated codes. Agreement between coders will be monitored by quarterly Kappa testing, with a target kappa coefficient of 0.80. Secondary coding will occur for 30% of all interviews for quality assurance. Differences will be resolved by consensus.

Staff survey

The survey items will be generated to operationalize each domain from the conceptual framework. Items will be written as close-ended, declarative statements from the staff member’s perspective, describing staff behaviors, feelings, perceptions and beliefs about Lean implementation at their site, as well as their observations of their own manager’s behavior. Next, a panel consisting of Lean content experts, front-line staff, and a survey development expert will review the initial item pool and rate each item for validity and clarity to create a content validity index (CVI) assessing fit of each item to its conceptual domain [29]. Items having a CVI of 0.7 or greater will be retained. Other survey items will be eliminated or modified based on the expert panel review. The resulting survey will be piloted by administering it to staff at eight VHA medical centers that are not a part of the evaluation but are involved in process improvement efforts. Using the pilot data, we will conduct a serially exploratory factor analysis with at least three items per factor, and reliable fit statistics [30]. We will then examine these items using a confirmatory factor analysis, looking for appropriate fit statistics, loadings, and clinical relevance [31, 32].

Quantitative Analysis

We will collect SAIL data from 2010 to 2017 from the Veterans Services Support Center, which provides informatics tools, data reports, and analytics services to support improvements in Veterans’ healthcare [33]. The SAIL measures organized by the targeted value stream areas are listed in Additional file 2.

Data analysis

Qualitative Site visit/Interviews

Following Rounds 1 and 2 interviews, the sites transcripts will be coded and each site will be scored on the conceptual framework constructs using a 0–4 scale to indicate the extent to which each construct exists at the site at the time of the interviews (Table 2). This method of coding and scoring is based on a rating system used previously to evaluate organizational transformation [34, 35]. An extensive training process will ensure all team members have consistent understanding of the construct definition and examples.

Table 2 Rating scale to score the Lean transformation constructs

Members from the evaluation interview teams will be paired and assigned a set of constructs to rate for all the sites. This will ensure consistent rating scores across all sites and timeframes. For each construct, the raters will be provided all coded narrative passages related to their assigned constructs from the transcripts. Raters will independently review the evidence and rate their assigned constructs. Next, the pairs will meet to review their scores and if there is a discrepancy they will review the data to reach a consensus score. Raters may indicate if there is insufficient data provided from the narrative passages to reliably rate the construct. Raters will also summarize their assessments to support their rating and note exemplar quotes that best illustrate the construct and associated rating. After construct ratings are complete, construct scores will be summed to create an overall score for each site, for each round. Difference and percent change from Round 1 to Round 2 for each construct will be calculated.

The rating scores will be used to compare each site’s progress and degree of Lean implementation compared to the other pilot sites. Based on the total summed rating score, sites will be grouped into high, medium and low Lean maturity sites. Lean maturity describes the stage in an organization’s Lean journey. This ordering will facilitate identification of differences between the groups, providing insight into facilitators and barriers to successful implementation. We will use Round 3 interview data to generate summative assessments of the sites’ Lean journey thus far and to compile a list the significant lessons learned by the leaders of the LET implementation process.

Staff Survey

The survey data will be analyzed to compare sites on staff engagement and Lean spread within sites. We will test whether the survey measures are associated with the sites’ level of implementation as measured by the qualitative data.

First, a confirmatory factor analysis will be conducted and internal scale reliabilities will be examined. We will then examine basic descriptive statistics, associations among different measures at the individual respondent level, and differences in scales and key individual items associated with different respondent demographics (such as profession/occupation, level of supervisory authority, gender and tenure). Because individual staff can be associated with the value stream area that they have worked on we can assess the distribution and extent of staff involvement in Lean, and whether staff that have participated in a particular value stream report more use of Lean tools and techniques and are more engaged in their medical center than those working in other value streams or those who have not been involved in value stream Lean efforts. In addition, we will test whether involvement in a value stream that is a designated priority for a given medical center is even more strongly associated with the survey measures. We will also assess whether participation in continuous daily improvement activities separate from value stream involvement is associated with increased engagement. Finally, we will analyze scales and key individual items using random effects models, where employee demographic characteristics are entered as fixed effects and value streams and medical centers as random effects.

Quantitative Analysis

Sites will be longitudinally evaluated and compared on improvements in SAIL measures in the respective value streams relevant to the areas they selected for Lean improvement efforts. We will measure the change in site-specific SAIL measures using an interrupted time series design. This is a robust technique to evaluate the immediate and sustained effect of a change in healthcare delivery, such as a policy change or quality improvement intervention. All data captured from 2010 to six months after LET implementation will be used for calculating trends prior to LET implementation (pre-implementation period). This approach allows for a six month implementation period (e.g., washout period). Data from six months after LET implementation through the end of 2017, will be used to calculate trends in SAIL measures after LET implementation (post-implementation period). Wald tests will be used to calculate the significance between pre- and post-SAIL trends. Estimates will be provided for each individual SAIL measure as well as in aggregate for each site. Two-sided p-values less than 0.05 will be considered statistically significant. Statistical analyses will be conducted using R [36].

Discussion

Based on the review of the literature, the evaluation team selected the Organizational Transformation Model (OTM) as a starting point for the development of a conceptual framework model [37]. The OTM is a model for moving organizations from short-term, isolated performance improvements to sustained, reliable, organization-wide transformation of healthcare delivery. The model identifies five elements critical for successful transformation: (1) Impetus to transform; (2) Leadership commitment to quality; (3) Improvement initiatives that actively engage staff in meaningful problem solving; (4) Alignment to achieve consistency of organization goals with resource allocation and actions at all levels of the organization; and (5) Integration to bridge traditional intra-organizational boundaries among departments and staff roles. The OTM is most often used in process improvement environments [12, 34, 38, 39]. To capture the specific features of a Lean organization, we integrated the OTM with concepts derived from our literature review, input from our operational partners that included additional Lean specific concepts such as just culture, use of metrics for decision-making, capability development, and a focus on the customer, in this case, specifically the Veteran.

Our final conceptual model for evaluating Lean transformation in healthcare settings includes ten domains and 24 constructs (Table 3).

Table 3 Conceptual framework domains and constructs to evaluate VA’s Lean Enterprise Transformation

The first domain, Impetus to Transform, requires leadership to identify and continuously use an incident, goal or passion, meaningful to the entire organization, to create a sense of urgency among the staff to participate in the Lean journey. The Leadership Commitment to Quality domain focuses on the abilities of the leaders to create and implement a long-term vision for Lean within the organization and recognize the multi-faceted complexity of the transformational change process. Also included in this domain is the expectation that senior leaders engage middle management to both participate and support their staff in participating in Lean efforts while also actively participating themselves the Lean activities (e.g., they do regular gemba walks, attend huddles, or rapid process improvement workshops.) The Organizational Culture domain incorporates several aspects of culture including the concept of a just culture in which mistakes and errors and poor performance on metrics are seen as opportunities for improvement rather than to blame individuals. Respect for all individuals supports a psychologically safe environment in which staff may experiment and propose innovative ideas without fear of repercussions from management or colleagues. For example, individual’s opinions are sought after and valued by their fellow staff, their managers and the leadership. This domain also acknowledges the need for individual ownership and accountability for their commitment to meeting the organization’s Lean objectives. One of the central requirements of a Lean organization is to develop a culture as a learning organization such that barriers to quality improvement are systematically identified and resolved. The last construct within this domain is that of spread; successfully improved processes and procedures are spread from one area to other appropriate areas of the organization and similarly lessons learned can be shared to increase the rate of a successful transformation. The fourth domain incorporates one of the central tenants of Lean management- making informed decisions based on the results from valid and meaningful metrics. Further, the data should be up to date, readily available and widely available especially to those working on related projects. The Integration across Boundaries domain encompasses the idea of staff from different work areas (clinical and administrative) and disciplines (e.g., nurses and physicians) working together effectively on improvement efforts, which helps to break down silos and build a stronger, more respectful organizational culture. Our conceptual framework includes two domains related to Alignment. These domains incorporate several ideas such as leadership and management should use incentives and reward structures as well as provide resources such as protected time to encourage Lean involvement by staff. Additionally, the organization ensures that selected metrics for value stream areas and continuous daily improvement efforts are in line with the organization’s True North or strategic goals. Communication is another domain that addresses the need for staff to receive regular and consistent information from higher levels of the organization regarding Lean expectations and goals. It is also important to encourage staff input regarding Lean activities and that managers and leaders authentically consider and acknowledge it. Equally valuable is to broadcast and celebrate Lean successes to generate awareness and enthusiasm. The Capability Development domain addresses the need to have and implement a clear educational plan that allows individuals at all levels of the organization to obtain training about Lean concepts and the use of Lean tools. Training can include both traditional classroom time as well as just-in-time training, for example at the start of a value stream project. To enforce training, staff must have the opportunity to use their new skills and receive coaching from supervisors and colleagues already Lean skilled. Patient Engagement is the final domain. The voice of the customer (i.e., patient in the healthcare setting) is the concept of engaging patients in the Lean transformation by getting their input via methods such as surveys and interviews, as well as having them participate on Lean related committees and activities such as rapid process improvement workshops.

This evaluation will use a newly developed conceptual framework and a mixed-methods approach to characterize and describe lessons learned from a widespread effort to implement LET in the VHA. Staff commentary coupled with detailed data analysis will support a more thorough understanding of outcomes and impacts, including cultural change, at the enterprise level—across multiple healthcare facilities implementing LET. This approach will fill an evidence gap critical to healthcare leaders and quality improvement staff within healthcare organizations planning to adopt Lean. The lessons learned by the LET pilot sites may allow more effective implementation of Lean practices and help transform the organizations’ culture to one of ongoing process improvement that benefits the organization, the staff and the patients they serve.