Using a cluster-randomized controlled trial design, specialist mental health clinics were invited to implement recommended practices for the treatment of psychosis. Each clinic chose two of four predefined practices they intended to implement and was randomized to implementation support for one practice (intervention arm) and not for the other (control arm). We prospectively measured health professionals’ perceptions of the implementation effort and the clinics’ degree of implementation every sixth month during an 18-month period. This was measured for both the practice they received implementation support for and for the control in each clinic.
Sample and randomization
This study was conducted within specialized mental health care in six Norwegian public health trusts (local health authorities) serving 38% of the country’s population in both rural and urban areas. Thirty-nine specialist mental health care clinics, i.e., 21 community mental health centres and 13 mental health hospital departments (e.g., forensic wards or specialist units for the treatment of psychosis) participated. They were invited in 2016 by the research group to take part in a study to improve compliance with the national clinical guidelines for the treatment of psychosis. Four evidence-based practices described in the guideline were chosen by the project group after a survey among the included mental health clinics. These were practices or programmes for antipsychotic medication management (MED), physical health care (PHYS), family psychoeducation (FAM), and illness management and recovery (IMR). Each clinic decided two of these four practices for implementation. Among the 39 clinics invited, 17 chose antipsychotic medication, 26 chose physical health care, 14 chose family psychoeducation and 21 chose illness management and recovery. Four unique fidelity scale were developed and employed by audit teams to score the degree of compliance to the core components of the practices the clinics had chosen. (The scales are further described under “Measures”).
By using pairwise randomization, we balanced the number of sites assigned to the intervention or control arms within each practice and ensured that each clinic was represented in both the intervention group and the control group. In the US National Evidence-Based Practice Project [25], the mean EBP fidelity increased from 2.28 (SD 0.95) at baseline to 3.76 (SD 0.78) at 12 months (See: Ruud et al. [26]). We assumed initially a similar mean increase in fidelity over 18 months for the experimental practices and no increase for control practices. Based on a two-tailed significance level of 5 and 90% power, we estimated that the overall hypothesis would be adequately powered with a minimum of eight sites in each arm for each practice.
To represent each clinic, a purposive sample of care providers, selected by their manager as particularly important for the clinical practice of psychosis treatment, constituted the sample of Implementation Process Assessment Tool (IPAT) responders. A sample of 10–12 care providers per clinic were suggested from the project, but the managers decided how many of their staff meet the inclusion criteria, resulting in five to 32 being invited per clinic. We decided a cut-off of three or more responders per clinical unit as an inclusion criterion for this study. The cut-off was set based on the logic of balancing generalizability to the unit and representation of a variety of clinical units. At the time of the first data collection (baseline), the clinics were recently informed about which practice they would receive implementation support for to enable preparations, such as deciding who should attend the clinical training.
The intervention (implementation support)
The intervention bundle was designed to affect care providers’ readiness and engagement and included tools and measures described in the literature [25, 27] and more innovative measures. It consisted of clinical training, implementation facilitation, and assessment and feedback every sixth month on level of compliance to recommended care and health professionals’ anonymous report on how they perceived the implementation effort. The clinical training included workshops with lectures and meeting colleagues from across the country, lasting one day for medication and physical health practices and 2 × 2 days for IMR and family support practices. For the IMR practice and the family support practice, the clinicians were offered additional supervision by a clinical expert on the practice. For IMR, this was offered every week for the first six months and every second week for the next six months. For the family support it was offered every other week for six months and then monthly for the next six months. A website with further educational material and tools was made available for the four practices. The implementation facilitation included support and training to enhance positive perceptions of the implementation effort i.e., planning and using tools and methods such as flow charts, process measuring and team activities to improve shared understanding and accountability. The clinics were offered one meeting with trained implementation facilitators every second week for the first six months and then once a month for a year. The implementation facilitators were trained by implementation experts in the project to recognize and interpret care providers’ perceptions of the factors facilitating and hindering implementation, measured by the IPAT questionnaire, and to suggest tools and methods to improve these perceptions and thereby make implementation teams successful. They had relevant clinical experience and training but were not clinical experts on any of the four practices to be implemented. The intervention included assessments and feedback on the clinic’s degree of compliance with the practice they had chosen and received support for, measured by the fidelity scale. Additionally, feedback on implementation factors, i.e., clinicians’ perceptions of the implementation effort as measured by the IPAT questionnaire, was provided. Fidelity assessments and the IPAT were delivered every six months over the 18-month intervention period. The feedback on IPAT scores was provided to the manager, the implementation team and the implementation facilitator at each clinic shortly after each measurement together with a short report to help interpret and suggest interventions to improve low scores. Please see Fig. 1. This intervention bundle to support implementation was in an earlier paper from this trial shown to have a positive impact on the fidelity scores. We found the difference between experimental and control sites in mean increase in fidelity score (within a range 1–5) over 18 months was 0.86 with 95% CI (0.21–1.50), p = 0.009, with corresponding effect size 0.89 (95% CI 0.43–1.35) [26].
Measures
The implementation was assessed using a specific fidelity scale for each of the four evidence practices. The four fidelity scales are the Physical Health Care Fidelity Scale [28], the Antipsychotic Medication Management Fidelity Scale [29], the Family Psychoeducation Fidelity Scale [30], and the Illness Management and Recovery Fidelity Scale [31]. Each fidelity scale contains 13–17 items for rating core components of the practice and detailed criteria for rating each item from 1 (lack of adherence to the evidence-based model) to 5 (full adherence to the model). The total fidelity score is calculated as the unweighted mean score of the items in the fidelity scale. The fidelity scales and their psychometric properties are further described in earlier papers [28,29,30,31]. An assessment team of two clinicians trained to use the fidelity scales conducted the scoring. At each clinic, they interviewed clinicians and managers, reviewed documents (e.g., procedures and data on performance) and conducted audits of ten randomly selected patient records for patients receiving psychosis treatment. The 13–17 items of the fidelity scales were rated on a scale from 1 (not implemented) to 5 (perfectly implemented). The unit of analysis for fidelity score is the clinic, defined by the mean of the items.
Health professionals’ perceptions of the implementation factors were measured by the IPAT [32]. The IPAT questionnaire is theoretically based on constructs defined in the Consolidated Framework for Implementation Research [33], including readiness for change [34] and stages of change [14, 15, 35]. In the theoretical model for the IPAT construct, outer- and inner-setting factors, intervention characteristics and the characteristics of individuals are perceived and interpreted at the individual and team levels. The implementers react to their perceptions by proceeding (or resisting and reversing) on the stages of change continuum from insight to acceptance and sustainment. This progression, in turn, affects how the implementers perceive the implementation effort [32]. The 27 items measure progression in the stages of change, individual activities and perceived support, collective readiness and support, and individual perception of the intervention. Examples of items are “I have discussed with colleagues how this new practice will work in our unit”,” I believe the patient will benefit from the improvement” and “We who work here agree that we have a potential for improvement in [new practice]”. The questionnaire was distributed to a purposive sample of health professionals central to the provision of care in question. The Likert scale from 1 (not agree/not true) to 6 (= agree/correct) for each of the 27 items revealed variation between respondents and between clinics. The internal consistency for the total scale and for each of the four underlying constructs was found to be high (Cronbach’s alpha >.83) [32]. The IPAT’s unit of analysis is the clinical unit and is represented by the mean of the responders from the clinic.
Fidelity and IPAT were scored for the practice in which implementation support was provided and for the control practice in each clinic at baseline, 6 months, 12 months and 18 months.
Analyses
Descriptive methods (mean, percent and standard deviation) and confidence intervals were used to describe the IPAT responders and the IPAT and fidelity scores. The associations between care provider perceptions (IPAT scores) and fidelity scores at each time point were explored by linear mixed effects models for fidelity scores depending on the IPAT adjusted by group (support/no support) and the random intercept for practice and unit. These models were estimated both with and without the interaction between IPAT and group, i.e., enabling or disabling various slopes for the groups. The effect of support on the IPAT was assessed using the linear mixed effects model for the IPAT depending on group, time and the interaction between group and time adjusted by random intercept for cases, practice and unit with simple contrasts in the time domain. The interaction term in these models estimates the change of differences between groups from baseline. For all analyses the significant level was set to 0.05. Since the primary analysis considered only the effect of support on the IPAT, no adjustment for multiple testing was necessary.
Statistical analyses were conducted using SPSS 26.0 (IBM Corp., Armonk, NY) and R 4.0 [36], and MATLAB 2020b (MathWorks Inc., Natick, MA) was used for graphics.