Study design and sites
The COMPASS study design and selection of hospitals are described in detail elsewhere [10, 11]. Briefly, hospitals were randomized to either the intervention (COMPASS-TC) or control (usual care) arm in Phase 1. In Phase 2, usual care hospitals crossed over to implement the intervention, and the original intervention hospitals attempted to sustain COMPASS-TC with limited study support.
In this paper we analyzed implementation of COMPASS-TC in the 20 hospitals randomized to the intervention arm in Phase 1, including 2751 patients enrolled July 2016 through March 2018 [12]. We evaluated patient and hospital characteristics associated with successful implementation. Our reporting follows the Standards for Reporting Implementation Studies (StaRI) guidelines for transparent and accurate reporting of implementation studies [13] and adheres to CONSORT guidelines.
Context: hospital engagement
As required by the Patient-Centered Outcomes Research Institute (PCORI), hospitals used their existing infrastructure, budget, and staffing to deliver the intervention. The study paid hospitals $50 per enrolled patient (plus $105 per patient who returned to the COMPASS follow-up clinic and received a care plan within 18 days of discharge) but did not pay for staff time to deliver the intervention. Hospitals also received $14,000 prior to implementation to offset costs of training and building infrastructure. These payments were not meant to cover costs. In line with pragmatic trial design as well as PCORI funding guidelines, the financial assistance to hospitals was minimal [14]. The study did not provide financial assistance for intervention costs including compensation for personnel that were delivering the intervention and equipment and materials costs associated with delivery of the intervention. COMPASS-TC was integrated into patient care without additional personnel resources provided to hospitals [10].
Study sample
This implementation analysis included all data from patients enrolled in the Phase 1 intervention arm (n = 2751 events among n = 2689 patients). Patients were eligible for enrollment if they were aged 18 years or older, spoke English or Spanish, were diagnosed with ischemic stroke, hemorrhagic stroke (excluding subdural or aneurysmal hemorrhage), or TIA, and were discharged directly home [10].
COMPASS-TC as a billable TCM intervention
The COMPASS-TC intervention was designed to be consistent with CMS TCM reimbursement requirements [15,16,17], which include requiring that the patient:
-
Transition from an inpatient setting (e.g., acute care hospital) to home;
-
Be of moderate- or high-complexity medical decision-making;
-
Receive communication (direct contact, telephone, or electronic) within two business days of discharge or two or more documented attempts within two business days; and
-
Have a face-to-face clinical visit within seven calendar days (for CPT Code 99496) or within 14 calendar days (for CPT Code 994965) of discharge from the inpatient setting [17].
The foundational component of the COMPASS-TC intervention was post-acute care coordination and management by a registered nurse (RN), post-acute care coordinator (PAC) and an advanced practice provider (APP), defined as a nurse practitioner (NP), physician assistant (PA), or physician. The PAC and/or APP contacted the patient in-person before hospital discharge to home, by phone two days post-discharge (or two attempts), and saw the patient in-person at the COMPASS-TC clinic visit within 14 days of discharge. During the pre-discharge visit, the PAC/APP described COMPASS-TC to the patient and family. The hospital-based team utilized an electronic TC planning tool developed by the study team to systematically evaluate patients [15] during both telephone and face-to-face follow-up. During the two-day call, the PAC/APP asked if any cognitive or physical deficits became apparent at home, completed medication review and reconciliation, referred to home health or outpatient care if indicated, provided patient education stroke symptoms, and reminded the patient of the upcoming clinic visit. Hospitals had flexibility in their COMPASS clinic setting (neurology clinic, hospital-based non-specialty clinic, primary care provider (PCP) clinic). During the clinic visit, the PAC/APP performed a standardized assessment of the patient’s functional status, medical and neurological care needs, social determinants of health, and caregiver’s capacity for assisting the patient during recovery [15, 16]. The results of the comprehensive assessment were captured electronically and generated an individualized electronic care plan (eCare Plan) that was shared with the patient, caregiver, PCP, and home health and outpatient therapy where applicable. The eCare Plan identified areas of patient/caregiver need and directed the PAC/APP in appropriate referrals to relevant community-based services (e.g., caregiver support services, medication management). Each hospital, in partnership with the study team, assembled a community resource directory of available services in their county to populate the eCare Plan, and a network of service providers in their area (community resource network) to support these referrals.
Implementation strategies
An implementation strategy is defined as an activity that facilitates adoption, implementation, and sustainability [18]. We utilized a number of implementation strategies [19] prior to hospital implementation and during active implementation of COMPASS-TC (Additional file 2: Table S1). After hospitals were randomized, training for intervention hospitals consisted of: two-day intensive training “boot camp” (to explain the care model), 6-h site visit (to tailor implementation, identify available community resources, and build community resource networks (CRN)), bi-monthly peer problem-solving calls, monthly data feedback on performance, and one-on-one same-day consulting as requested. Training and ongoing consultation was provided to all intervention hospitals by the study’s Director of Implementation who had both clinical and administrative experience, and a team of multidisciplinary providers with experience in stroke care. All training materials were approved by stakeholders, including patients and caregivers. Educational and training modules were made available online, and monthly educational webinars were provided on a variety of stroke/TIA topics relevant to clinical practice.
RE-AIM framework & measures
To assess implementation, we used the RE-AIM framework, the most widely used framework in implementation science for evaluation [20]. RE-AIM evaluates the Reach, Effectiveness, Adoption, Implementation, and Maintenance of health promoting interventions in real-world, complex settings, attending to both individual and organizational levels of impact, to address questions of translation into practice and generalizability. This framework focuses on features of the settings and participants, features of the implementers, and the frequency and intensity of intervention activities [13] and thus is suitable for process evaluations.
Reach assesses who received the intervention [21] and is a patient-level measure of participation [12]. Randomization occurred at the hospital level. The study met criteria for a waiver of consent and HIPAA Authorization; therefore all patients were enrolled automatically and given opportunities to decline participation in the outcomes survey or to withdraw from the research study entirely [22]. Hospital staff screened, identified, and enrolled eligible patients and initiated the intervention. All patients in the study were theoretically eligible for TCM at discharge as they all transitioned from the hospital directly home with a diagnosis of stroke or TIA, which meet the criteria for moderate- or high-complexity medical decision making. Thus, successfully “reached” patients were those whose care met all TCM billing requirements, including receipt of a call two business days post-discharge (or two attempts) and attendance at a clinic visit (which includes an eCare Plan) 14 calendar days post-discharge. Reach was calculated as the proportion of reached patients out of the total enrolled by hospital staff.
Effectiveness assesses what effect the intervention had on important outcomes [21]. The primary outcome for the COMPASS Study was physical function measured with the Stroke Impact Scale-16 (SIS-16) [23]. The Effectiveness metric was defined as within-hospital mean differences in SIS-16 between patients receiving a 14-day clinic visit versus not. A comparison of COMPASS-TC versus usual care has been presented elsewhere [9].
Adoption assesses where the intervention was implemented [21]. The Adoption metrics were: the number and proportion of hospitals that initiated the intervention, number and characteristics of intervention agents (clinical team delivering the intervention: PAC, PAC back-up, APP, and APP back-up) that were trained, time it took to launch after training, and number of intervention agents that delivered the intervention.
Implementation assesses how each component of the intervention was delivered [21] as intended, including time needed for the implementation [12] of the clinic visit, which is the core of the COMPASS-TC model. The Implementation metrics were defined as:
-
(1)
Fidelity to each of the components of the intervention measured with quality measures that were created as part of the COMPASS Study to provide real-time feedback to hospitals (Appendix for details):
-
Proportion of patients identified and enrolled within 2 days,
-
Proportion of patients who had a two-day call or documentation of two attempts (i.e., delivered per protocol),
-
Proportion of patient scheduled/offered a clinic visit within 14 days, even if patient did not attend (i.e. delivered per protocol),
-
Proportion of all completed visits occurring within 14 days, and
-
Proportion of all completed visits during which an eCare Plan generated.
-
(2)
Clinic visit duration (minutes)
-
(3)
Number of days from discharge to clinic visit among participants who attended a visit at any time.
Maintenance assesses how long the intervention was sustained. The maintenance metric was defined as the absolute number, proportion, and characteristics of hospitals that continued to enroll patients and deliver components of COMPASS-TC for a minimum of 6 months after the end of Phase 1.
Data collection
Process data, such as dates of completion and reasons for not completing a study-related task, were used to compute quality measures. These were collected during the study by hospital staff using forms embedded in a web-based application.
Hospital characteristics were obtained from public data files (e.g., Rural-Urban Community Area (RUCA) code website [24], Joint Commission website [25], NC Stroke Care Collaborative hospital characteristics, Medicare Provider of Services files) [26], and baseline hospital surveys [10, 27], which ascertained staff turnover, dates of training, and clinic locations. Additional surveys administered approximately 6 months into the study captured clinic visit duration, organizational readiness, and partnership synergy. Organizational readiness was measured by the Organizational Readiness to Implement Change (ORIC) Scale [28], which measures change commitment and change efficacy. ORIC is a validated instrument that was administered to the intervention agents (clinical team delivering the intervention: PAC, PAC back-up, APP, and APP back-up). The Partnership Synergy Scale measured the level of engagement between the clinical team and their CRN, as a proxy for how much the community network assisted PACs with linking patients to community-based social services [29]. This is a validated instrument and was administered to intervention agents, site principal investigator, hospital leadership, and community network members (e.g., engaged community-based pharmacists, social service providers, rehabilitation providers supporting the PAC in linking patients to needed services and resources outside the hospital).
Patient demographic and clinical characteristics were obtained from electronic health records and recorded by hospital staff. Patient addresses were geocoded using ArcMap 10.5.1 and the World Geocoding Service. Three (0.1%) patients did not have home addresses and were not geocoded. Shortest distance to the COMPASS clinic visit location was computed using Open Streetmap and ArcGIS Network Analyst.
Patient outcomes were obtained at 90 days by trained (blinded) interviewers administering a phone survey that included self-reported post-stroke physical function, measured with the Stroke Impact Scale-16 (SIS-16) [23]. The SIS-16 was selected as the primary patient outcome for the COMPASS study because of its strong psychometric properties, including validation for proxy, phone, and mail administration [30, 31]. It is a patient-centric measure, developed in with input from patients, caregivers, and providers [23]. It was designed to capture significant residual deficits in mild and moderate stroke, and is superior to other measures in capturing residual deficits that matter to patients. Even in those with the mildest strokes (NIH Stroke Severity Score 0 to 5), only 10% report full function on the SIS-16. The 16 items measure ADLS, IADLS, and physical activities on a scale of 0–100, with higher scores indicating higher function. Patient outcomes were collected by interviewers at the Carolina Survey Research Laboratory. Interviewers were trained on study-specific protocols that incorporated patient feedback obtained during pilot testing. Interviewers were blinded to randomization arm and administered all interviews using standardized computer-assisted telephone interviewing software and scripts [10]. Interviewers were monitored biweekly by their supervisors for quality control.
Hospital audits Hospitals participated in two unannounced case ascertainment audits, each covering a 2 month period, to evaluate the proportion of eligible cases that hospitals correctly identified and enrolled.
Consent Institutional review board (IRB) approval was received through Wake Forest University Health Sciences (central IRB), or through local hospital IRBs. At 90 days, patients provided informed consent [22] for collection of outcomes. Additionally, participants consented to collection of process measures, including the ORIC and Partnership Synergy Scale via email survey.
Statistical analyses
Descriptive statistics were used to summarize each RE-AIM domain. Characteristics of enrolled patients who were reached versus not were compared using Fisher’s exact and Wilcoxon rank sum tests. Generalized linear mixed models (GLMMs) were used to evaluate patient characteristics associated with clinic visit attendance (a primary component of patient Reach) and were adjusted for clinic setting, distance to the clinic, and organizational readiness. Analyses were performed conditional on successful implementation of the 14-day visit (i.e., a visit being scheduled or offered to the patient).
Associations between hospital characteristics and both Reach and Implementation quality measures were estimated with GLMMs, adjusted for patient characteristics including age, race, gender, diagnosis, stroke severity, history of stroke or TIA, presence of at least one cardiovascular comorbidity (i.e., cardiovascular disease, heart failure, atrial fibrillation, or diabetes), and distance to the COMPASS-TC follow-up clinic.
Linear mixed models were used to estimate overall and hospital-specific mean differences in SIS-16 [23] between patients who received a visit and eCare plan within 14 days compared with those who did not receive a visit (Effectiveness). These models were adjusted for age, race, stroke severity, diagnosis, and a log-transformed patient-specific propensity score. Propensity scores were constructed to account for differences in patient characteristics of those who did and did not attend the clinic visit. Propensity scores were estimated with conditional logistic regression and incorporated information such as medical history and comorbidity [9].
Finally, p-values for associations between hospital characteristics and Maintenance were obtained using Fisher’s Exact and Wilcoxon Rank Sum tests.
Missing Data Some covariate data were incompletely ascertained (e.g., 2% missing stroke severity). Multiple imputation with chained equations [32] was used to construct 100 complete datasets that were analyzed with GLMMs as described above and estimates were combined using standard techniques [33].
