Complexity and involvement as implementation challenges: results from a process analysis

Background The study objective was to analyse the implementation challenges experienced in carrying out the IMPROVE programme. This programme was designed to implement checklist-related improvement initiatives based on the national perioperative guidelines using a stepped-wedge trial design. A process analysis was carried out to investigate the involvement in the implementation activities. Methods An involvement rating measure was developed to express the extent to which the implementation programme was carried out in the hospitals. This measure reflects the number of IMPROVE-implementation activities executed and the estimated participation in these activities in all nine participating hospitals. These data were compared with prospectively collected field notes. Results Considerable variation between the hospitals was found with involvement ratings ranging from 0 to 6 (mean per measurement = 1.83 on a scale of 0–11). Major implementation challenges were respectively the study design (fixed design, time planning, long duration, repeated measurements, and data availability); the selection process of hospitals, departments and key contact person(s) (inadequately covering the entire perioperative team and stand-alone surgeons); the implementation programme (programme size and scope, tailoring, multicentre, lack of mandate, co-interventions by the Inspectorate, local intervention initiatives, intervention fatigue); and competitive events such as hospital mergers or the introduction of new IT systems, all reducing involvement. Conclusions The process analysis approach helped to explain the limited and delayed execution of the IMPROVE-implementation programme. This turned out to be very heterogeneous between hospitals, with variation in the number and content of implementation activities carried out. The identified implementation challenges reflect a high complexity with regard to the implementation programme, study design and setting. The involvement of the target professionals was put under pressure by many factors. We mostly encountered challenges, but at the same time we provide solutions for addressing them. A less complex implementation programme, a less fixed study design, a better thought-out selection of contact persons, as well as more commitment of the hospital management and surgeons would likely have contributed to better implementation results. Trial registration Dutch Trial Registry: NTR3568, retrospectively registered on 2 August 2012. Supplementary Information The online version contains supplementary material available at 10.1186/s12913-021-07090-z.


Rationale
Informal or formal frameworks, models, concepts, and/or theories used to explain the problem, any reasons or assumptions that were used to develop the intervention(s), and reasons why the intervention(s) was expected to work

Specific aims
Purpose of the project and of this report

Methods
What did you do?

Context
Contextual elements considered important at the outset of introducing the intervention(s)

Intervention(s)
a. Description of the intervention(s) in sufficient detail that others could reproduce it b. Specifics of the team involved in the work

Study of the Intervention(s)
a. Approach chosen for assessing the impact of the intervention(s) b. Approach used to establish whether the observed outcomes were due to the intervention(s)

Measures
a. Measures chosen for studying processes and outcomes of the intervention(s), including rationale for choosing them, their operational definitions, and their validity and reliability b. Description of the approach to the ongoing assessment of contextual elements that contributed to the success, failure, efficiency, and cost c. Methods employed for assessing completeness and accuracy of data 11. Analysis a. Qualitative and quantitative methods used to draw inferences from the data b. Methods for understanding variation within the data, including the effects of time as a variable

Ethical Considerations
Ethical aspects of implementing and studying the intervention(s) and how they were addressed, including, but not limited to, formal ethics review and potential conflict(s) of interest

Results
What did you find?

Results
a. Initial steps of the intervention(s) and their evolution over time (e.g., time-line diagram, flow chart, or table), including modifications made to the intervention during the project b. Details of the process measures and outcome c. Contextual elements that interacted with the intervention(s) d. Observed associations between outcomes, interventions, and relevant contextual elements e. Unintended consequences such as unexpected benefits, problems, failures, or costs associated with the intervention(s). f. Details about missing data

Discussion
What does it mean?
14. Summary a. Key findings, including relevance to the rationale and specific aims b. Particular strengths of the project

Interpretation
a. Nature of the association between the intervention(s) and the outcomes b. Comparison of results with findings from other publications c. Impact of the project on people and systems d. Reasons for any differences between observed and anticipated outcomes, including the influence of context e. Costs and strategic trade-offs, including opportunity costs

Limitations
a. Limits to the generalizability of the work b. Factors that might have limited internal validity such as confounding, bias, or imprecision in the design, methods, measurement, or analysis c. Efforts made to minimize and adjust for limitations

Conclusions
a. Usefulness of the work b. Sustainability c. Potential for spread to other contexts d. Implications for practice and for further study in the field e. Suggested next steps

Funding
Sources of funding that supported this work. Role, if any, of the funding organization in the design, implementation, interpretation, and reporting

Assumptions
Reasons for choosing the activities and tools used to bring about changes in healthcare services at the system level.

Context
Physical and sociocultural makeup of the local environment (for example, external environmental factors, organizational dynamics, collaboration, resources, leadership, and the like), and the interpretation of these factors ("sense-making") by the healthcare delivery professionals, patients, and caregivers that can affect the effectiveness and generalizability of intervention(s).

Ethical aspects
The value of system-level initiatives relative to their potential for harm, burden, and cost to the stakeholders. Potential harms particularly associated with efforts to improve the quality, safety, and value of healthcare services include opportunity costs, invasion of privacy, and staff distress resulting from disclosure of poor performance.

Generalizability
The likelihood that the intervention(s) in a particular report would produce similar results in other settings, situations, or environments (also referred to as external validity).

Healthcare improvement
Any systematic effort intended to raise the quality, safety, and value of healthcare services, usually done at the system level. We encourage the use of this phrase rather than "quality improvement," which often refers to more narrowly defined approaches.

Inferences
The meaning of findings or data, as interpreted by the stakeholders in healthcare servicesimprovers, healthcare delivery professionals, and/or patients and families

Initiative
A broad term that can refer to organization-wide programs, narrowly focused projects, or the details of specific interventions (for example, planning, execution, and assessment)

Internal validity
Demonstrable, credible evidence for efficacy (meaningful impact or change) resulting from introduction of a specific intervention into a particular healthcare system.

Intervention(s)
The specific activities and tools introduced into a healthcare system with the aim of changing its performance for the better. Complete description of an intervention includes its inputs, internal activities, and outputs (in the form of a logic model, for example), and the mechanism(s) by which these components are expected to produce changes in a system's performance.
Loss of the ability to perform other tasks or meet other responsibilities resulting from the diversion of resources needed to introduce, test, or sustain a particular improvement initiative

Problem
Meaningful disruption, failure, inadequacy, distress, confusion or other dysfunction in a healthcare service delivery system that adversely affects patients, staff, or the system as a whole, or that prevents care from reaching its full potential

Process
The routines and other activities through which healthcare services are delivered

Rationale
Explanation of why particular intervention(s) were chosen and why it was expected to work, be sustainable, and be replicable elsewhere.

Systems
The interrelated structures, people, processes, and activities that together create healthcare services for and with individual patients and populations. For example, systems exist from the personal selfcare system of a patient, to the individual provider-patient dyad system, to the microsystem, to the macrosystem, and all the way to the market/social/insurance system. These levels are nested within each other.

Theory or theories
Any "reason-giving" account that asserts causal relationships between variables (causal theory) or that makes sense of an otherwise obscure process or situation (explanatory theory). Theories come in many forms, and serve different purposes in the phases of improvement work. It is important to be explicit and well-founded about any informal and formal theory (or theories) that are used.