Implementation of neurological group-based telerehabilitation within existing healthcare during the COVID-19 pandemic: a mixed methods evaluation

Abstract

Background

There is a need to evaluate if and how telerehabilitation approaches might co-exist within healthcare in the long-term. Our aim was to implement and evaluate a multidisciplinary group-based telerehabilitation approach for people engaging in neurological rehabilitation.

Methods

NeuroRehabilitation OnLine (NROL) was adapted and implemented within an existing healthcare system as a programme of repeating six-week blocks. A robust evaluation was undertaken simultaneously using a convergent parallel design underpinned by implementation frameworks. This included service data, and patient and staff interviews. Implementation success was conceptualised using the outcomes of appropriateness, acceptability and sustainability.

Results

Eight NROL blocks delivered 265 sessions with 1347 patient contacts, and NROL continues as part of standard practice. The approach was appropriate for varied demographics and had positive patient opinions and outcomes for many. Staff perceived NROL provided a compatible means to increase therapy and help meet targets, despite needing to mitigate some challenges when fitting the approach within the existing system. NROL was considered acceptable due to good attendance (68%), low drop-out (12%), and a good safety record (one non-injury fall). It was accepted as a new way of working across rehabilitation disciplines as an ‘extra layer of therapy’. NROL had perceived advantages in terms of patient and staff resource (e.g. saving time, energy and travel). NROL provided staffing efficiencies (ratio 0.6) compared to one-to-one delivery. Technology difficulties and reluctance were surmountable with dedicated technology assistance. Leadership commitment was considered key to enable the efforts needed for implementation and sustained use.

Conclusion

Pragmatic implementation of group-based telerehabilitation was possible as an adjunct to neurological rehabilitation within an existing healthcare system. The compelling advantages reported of having NROL as part of rehabilitation supports the continued use of this telerehabilitation approach. This project provides an exemplar of how evaluation can be run concurrently with implementation, applying a data driven rather than anecdotal approach to implementation.

Background

The ambition to harness the potential of technology to transform healthcare has been expressed in policy for years [1, 2]. Although telerehabilitation has existed in some countries for several years [3, 4] overall adoption has been slow [5, 6], and the pandemic provided the impetus for a dramatic increase in uptake of telehealth [7]. Globally rehabilitation services faced substantial disruptions with variable provision of in-person therapy, group therapies suspended, and diminished workforce capacity [8,9,10,11]. In many healthcare services, telerehabilitation was readily instigated into routine practice to provide timely care to the substantial and increasing number of service users requiring rehabilitation [8]. This hastened approach to implementation was warranted within the context of the pandemic, but now there is a need to robustly evaluate if and how telerehabilitation approaches might co-exist within healthcare systems in the long-term [12, 13].

Telerehabilitation involves the provision of one or more rehabilitation disciplines delivered remotely via telecommunication devices. It can be used as an alternative delivery method for providing conventional in-person rehabilitation. Within neurological rehabilitation it has been used for delivery of upper and lower limb training, mobility training, post-hospital discharge support programs and communication therapies [14,15,16,17]. Cochrane reviews have shown low to moderate quality evidence that telerehabilitation is as effective as in-person rehabilitation for Stroke and Multiple Sclerosis (MS) patients [16, 17]. Positive findings are also emerging from data collected during the rapid expanded use of telerehabilitation during the pandemic [13, 18]. Looking forward, telerehabilitation may contribute to a hybrid approach for delivering rehabilitation services [13, 19,20,21]. Some challenges are likely to exist when integrating and sustaining telerehabilitation within healthcare practice such as the requirement for dedicated resources and support, adequate environment, and training [22, 23]. Nevertheless, telerehabilitation has the potential to facilitate timely and equitable rehabilitation by mitigating some barriers to access and provision, but it is important to ensure that patient and service outcomes are comprehensively evaluated [13, 21].

A promising multi-disciplinary group-based telerehabilitation approach called Neuro-Rehabilitation OnLine (N-ROL) was developed in London in response to the first lockdown in the UK [24]. They delivered a standalone 15-week programme to predominantly stroke patients. Their programme consisted of physical and talking therapy groups set within a holistic working approach, incorporating psychoeducation and peer-support elements (for full TIDieR description see Beare and colleagues) [24]. Results were positive with participating patients improving significantly on patient outcome measures. The aim of the current project was to adapt and implement a tailored version of NROL to see how it could be delivered when embedded within an existing healthcare system. Robust evaluation was undertaken alongside implementation to determine the appropriateness, acceptability, and to provide direction for sustained use.

Methods

Implementation context and setting

This project is a collaboration between a healthcare organisation (East Lancashire Hospitals NHS Trust, ELHT) and researchers (University of Central Lancashire, UCLan), as part of an established clinical academic partnership. It is registered with ELHT as a service evaluation and for the qualitative study approval has been given by UCLan Health Ethics Review Panel (HEALTH 0155).

The healthcare setting is a neurological rehabilitation service consisting of two teams, the Neurorehabilitation team and Stroke therapy team. The teams provide multi-disciplinary rehabilitation across hospital and community settings to adults who have a sudden onset or progressive/intermittent neurological condition. The service typically offers in-person individual and group therapy. During the evaluation period (January 2021—April 2022) in-person group therapies were suspended and individual in-person inputs were intermittently curtailed due to pandemic restrictions. From January 2021, patients undertaking outpatient or community neurological rehabilitation could be referred by a treating therapy staff member to participate in NROL as an adjunct to their existing rehabilitation. In addition to service eligibility, inclusion to NROL required patients to be English speaking or have access to a translator and have willingness to participate in online group therapy with or without carer support. Patients were excluded from NROL if they lacked access to appropriate computer hardware or connectivity.

NROL

NROL was adapted for purpose from the standalone version of N-ROL described by Beare and colleagues [24]. In brief, this updated version of NROL delivers synchronous (in real-time) group-based neurological rehabilitation to outpatient and community patients over six-week recurring blocks via the online platform Microsoft Teams. It consists of talking (Cognitive education, Cognitive processing, Living Well, Fatigue, Dysarthria, Dysphasia and Bookclub) and physical (Balance & Mobility and Upper Limb) targeted therapy groups and community groups (NROL entry, exit, follow-up and Café NROL). Groups are run by a multidisciplinary team comprising therapy staff (allied health professional and psychology), assistant practitioners and patient volunteers. NROL, and its groups, are described in more detail elsewhere [25,26,27,28,29,30,31]. To facilitate access to NROL patients are provided with set-up and ongoing technical assistance by a dedicated NROL technology staff member. Technology assistance is also available for staff and patients during all NROL sessions. Administrative support is provided by a service administrator within the scope of their role. Oversight for NROL is provided by project leads and a coordinator.

Design

A convergent parallel design was used throughout delivery of NROL including data collection and analysis at a service level, and patient and staff interviews. Both quantitative and qualitative data were collected, analysed separately, then merged and interpreted together. Implementation success was conceptualised using selected Proctor outcomes [32]. These were chosen to fit the evaluation purpose and agreed with decision-makers (commissioners and managers) as those that were relevant for our evaluation, informed by the early stage of implementation and resources available. The outcomes were appropriateness (perceived fit, relevance, or compatibility), acceptability (attendance, safety and perception among implementation stakeholders) and sustainability (key considerations for NROL to continue as a routine part of care delivery) (Fig. 1).

Fig. 1

Mixed methods convergent parallel design

Quantitative data collection and analysis

Service data included details of NROL delivery (number of blocks, groups, and sessions), patient and staff attendance, technology assistance provided, and patient outcome.

Three patient-reported outcome measures were used on entry and exit to NROL. The Stroke Self-Efficacy Questionnaire (SSEQ, max 39) provided a self-report measure of self-efficacy [33, 34]. Modification of SSEQ wording was required to be inclusive of ‘stroke and other neurological conditions’. The EQ-5D-5L provided measures of health-related quality of life [35]. It consists of 2 parts: the EQ Visual Analogue Scale (EQ-VAS, max 100) rates individual’s perceived overall current health and the EQ descriptive system profiles individual’s health state. The descriptive system question responses were transformed into an EQ Index score (EQ-Index, max 1.000) using the EuroQoL Group’s crosswalk methodology with a United Kingdom population value set [36]. The Patient Specific Functional Scale (PSFS, max 10) was used to generate patients’ average rating on performing activities of function that are important to them and which were impacted by their neurological condition [37, 38].

All patients referred to NROL were characterised using routinely collected demographic and clinical data and analysed. Reasons for withdrawal or drop-out from NROL were recorded. Intention to treat analysis was used for patients who dropped out of NROL after participating in at least one targeted therapy session, or when patients withdrew from a specific group.

Descriptive statistics were used to summarise service level data. For patient outcome data, because some outcomes were not normally distributed and had outliers, we report non-parametric tests for all to allow a more conservative test of significance. Wilcoxon signed rank tests were used to test for change from entry to exit. Mean differences are provided for comparison with minimal clinically important differences (MCIDs), defined as SSEQ ≥ 4 points, EQ-VAS ≥ 10 points, EQ-Index ≥ 0.080 and PSFS ≥ 3 points using published data or 10% of maximum total score [38,39,40]. The proportion of patients who improved or remained stable (change score (exit – entry) ≥ 0), and those who met or exceeded the MCID, are reported. Subgroup analyses were conducted to help understand the appropriateness of NROL to different patient cohorts given the varied demographics, which included patients with conditions with differing expected disability trajectories and time periods since diagnosis. Chi-Square Tests, or where appropriate Fisher’s Exact Test 2 × 2 comparisons, were used to determine any association between the direction of change in patient-reported outcome measure score (improved or stable (change score ≥ 0) vs. declined (change score < 0)) and condition category (sudden onset vs. progressive/intermittent) or chronicity (subacute vs. chronic). Classification of conditions was guided by the NHS Long Term Conditions National Service Framework [41], and chronicity defined as subacute if 1 week to six months after diagnosis and chronic if over six months [42].

Analyses were performed using Microsoft Excel 2008 and IBM SPSS Statistics version 28.0. Statistical significance was set at α < 0.05 and all tests were two-tailed. For multiple comparisons we used the Benjamini–Hochberg procedure, with the false discovery rate fixed to 5% [43].

Qualitative data collection and analysis

Patients and healthcare staff were invited to participate in a one-off interview if they had used or been involved with NROL. A purposive approach ensured a broad mix of participants in terms of gender, age, ethnicity, diagnosis (patients) and profession and job role (staff).

The Consolidated Framework for Implementation Research (CFIR) [44] was used in the development of the interview guide for the study, based on a previous interview guide used to evaluate the implementation of stroke rehabilitation interventions [45,46,47] (see detail in the supplementary file: interview guide). The CFIR was chosen as it is one of the most commonly used determinant frameworks and provides a menu of constructs that have been associated with effective implementation [44, 48]. Domains include characteristics of the individuals, intervention, and inner setting. Interviews were conducted over Microsoft Teams and lasted approximately 30 and 60 min. Participants were aware that the interviewer was not part of the clinical team, and an honest perspective was wanted to learn lessons for implementation, and that criticisms were welcomed. Interviews were recorded, anonymized, and transcribed.

Researcher characteristics and reflexivity

This project was part of a clinical academic partnership, with LC & SA being both experienced researchers and physiotherapists in neurological rehabilitation. JR is a psychologist who works in the Neurorehabilitation team and was involved in design and delivery of NROL content. Two further researchers (NW & PB) were involved in the analysis and interpretation of the data and were not from a clinical background and independent from the clinical team to help reduce any associated bias from those associated with implementing the NROL intervention. NW is a researcher with a social psychology background and PB is experienced in health services research.

Data analysis

Interview transcripts were imported into NVivo 12 for analysis. Framework analysis was undertaken using the CFIR to code data deductively, with additional free codes developed where needed. To establish a shared understanding and interpretation of the coding framework, all researchers started by coding the same two transcripts. The coded transcripts were compared and any variance in interpretation of data and application of codes was discussed to arrive at a mutual decision. Subsequently the remaining transcripts were coded separately by two researchers independently (with one from a clinical background and one from an external research perspective). The relevant CFIR domains to this evaluation (characteristics of NROL, the individuals (staff and patients) and the inner setting (Neurorehabilitation and Stroke teams) relating to the implementation outcomes are reported.

Mixed methods integration

A joint display, defined as a way to “integrate the data together through a visual means to draw out new insights beyond the information gained from the separate quantitative and qualitative results.” [49] provided the structure to integrate quantitative and qualitative analyses to explore the implementation outcomes of appropriateness, acceptability and sustainability. This is a suggested method to integrate and represent mixed methods analyses [50]. The juxtaposing of quantitative results and qualitative findings side-by-side facilitated insights in relation to the implementation outcomes. These were discussed and agreed by the research team and the clinical staff involved in NROL.

Results

NROL participation

A total of 137 patients were referred within the evaluation period (January 2021 – April 2022). To understand NROL fit with the overall service, a small-scale review was made over the period from January 2021 to August 2021 between patients referred to NROL and those who were not. This indicated NROL referrals were received for about 10% of patients receiving outpatient or community neurological rehabilitation services. Evaluation of selected demographics (age, sex, index multiple deprivation) indicated patients referred to NROL were younger (Mdn = 54y, IQR = 43.5–65) than those not referred (Mdn = 67y, IQR = 54–77), p < 0.001).

All NROL referrals were accepted; 109 patients participated in NROL, 28 patients did not start for a variety of reasons (Fig. 2). Participating patient characteristics are presented in Table 1 and are comparable with those who were referred to NROL but did not start (all p > 0.05).

Fig. 2

NROL participation profile

Table 1 Patient demographics

Service data

NROL delivery

Eight blocks of NROL were undertaken in the evaluation period. The median number of patients participating in a block was 21 (range = 12–26). 13 different groups were offered across a range of therapies, with the median number of groups per block being 8 (range = 6–9). Groups offered and session content was iteratively modified to meet the requirements of NROL patient cohorts. In total, 265 NROL sessions were delivered with a median of 4 patients per session (range = 1–14). Summary data including individual group sessions delivered are presented in Table 2. One adverse event was recorded: a non-injury fall in a physical group. NROL staff followed standard operating procedure guidelines for managing and documenting this event, no first aid or follow-up was required. NROL continues to be delivered successfully beyond the evaluation period and has become part of standard care.

Table 2 Service data summary over 8 blocks of NROL

Patient attendance

Most patients participated in 1 NROL block (55%), but some participated in 2 (36%), 3 (7%) or 4 (2%). There were 1347 patient contacts in total. Overall NROL attendance rate was 68% (excluding the Café NROL group which was optional to attend). Attendance rate at targeted therapy groups (talking and physical) ranged from 93 to 60%. Sixty-eight patients (62%) chose to participate in Café NROL. Summary data including individual group attendance are presented in Table 2.

Staff attendance

There were 727 staff contacts (including therapy staff, assistant practitioner and technology support) during delivery of the 1347 patient contacts. This included staff attendance for observation/training purposes. The median staff:patient ratio across all sessions was 0.6 (range = 0.1 – 4.0). All groups were delivered with a median staff:patient ratio less than or equal to 1.0, except the Cognitive processing group (Mdn = 1.3). The talking groups had the highest staff:patient ratio (Mdn = 0.8, range = 0.1 – 4.0) and the community groups the lowest (Mdn = 0.4, range = 0.2 – 3.0). The occurrence of staff:patient ratios with values up to 4.0, as indicated by the reported ranges, reflects the fact that sessions were required to have a minimum number of staff attending regardless of the number of patients present and that many sessions included staff for experiential learning. Summary data for individual group contacts and staff:patient ratios are presented in Table 2. There were 84 student and 34 patient-volunteer contacts.

Technology assistance

Total technology assistance for set-up and any further individual support (excluding in-session technology assistance which is captured in staff:patient ratio) took a median of 11 min (IQR = 6–22, max 250 min) and 1 contact (IQR = 1–2, max 9) per patient. The main technology areas addressed included: assisting installation of Teams software, advising on use of the Teams App (rather than the Browser due to limited functionality), aiding use of microphone and camera functions, enabling patients to see other group members (rather than only the person speaking), and building a patient’s overall skill and confidence to enable successful use of Teams and its access through email invites. Technology assistance was offered to carers if required.

Patient outcome data

Complete data sets were available for SSEQ, EQ-5D-5L and PSFS for 84, 96 and 78 patients, respectively. Variation in number is largely due to phased introduction of these measures during pragmatic implementation.

At a group level, statistically significant improvements were seen over time in all patient outcome measures, with all p-values significant using the Benjamini–Hochberg procedure with our false discovery rate (SSEQ: Mdn entry = 19 (IQR = 14–23.75), Mdn exit = 21 (IQR = 14–25), p = 0.027 (M difference = 1.50 (SD = 6.37)); EQ-Index: Mdn entry = 0.547 (IQR = 0.231–0.663), Mdn exit = 0.558 (IQR = 0.418–0.691), p = 0.001 (M difference = 0.083 (SD = 0.228); EQ-VAS score: Mdn entry = 50 (IQR = 40–70), Mdn exit = 60 (IQR = 45 – 75), p = 0.022 (M difference = 3.80 (SD = 21.52)); PSFS: Mdn entry = 3.5 (IQR = 2.0–4.5), Mdn exit = 5.5 (IQR = 3.50–7.13), p < 0.001 (M difference = 1.98 (SD = 2.09)). Large overlaps in interquartile ranges (and standard deviations) are noted and only EQ-Index mean difference met the MCID.

At an individual level, the proportions of patients who improved or remained stable were between 62%—86% (SSEQ: 62%; EQ-Index: 67%; EQ-VAS: 68%; PSFS: 86%), with 36%—49% (SSEQ: 38%; EQ-Index: 41%; EQ-VAS: 49%; PSFS: 36%) of patients meeting or exceeding the MCID.

Subgroup analyses

Subgroup analyses indicated no association between the direction of change in SSEQ, EQ-VAS or EQ-Index, or PSFS scores (improved or stable vs. declined) and condition (sudden vs. progressive/intermittent) or chronicity (subacute vs. chronic) (all p > 0.05).

Qualitative data

Seventeen staff and 13 patient interviews were conducted between March- July 2021. Demographics are detailed in Table 3.

Table 3 Interviewed patient and staff demographics

Constructs influencing implementation of NROL

Constructs that were reported as key to the implementation of NROL are summarised in Tables 4, 5 and 6 according to the relevant CFIR domains, together with exemplar quotes. Further detail is available in a Supplementary file: interview analysis. Participants are identified by their participant code. For the ‘characteristics of the individuals’ and ‘characteristics of NROL’ domains, both staff and patients highlighted issues related to the implementation outcomes. For the ‘inner setting’, only staff quotes are given as patients were generally unaware of the constructs related to the service itself.

Table 4 Key characteristics of individuals and exemplar quotes

Table 5 Key characteristics of NROL and exemplar quotes

Table 6 Key characteristics of inner setting and exemplar quotes

Mixed methods integration

A joint display (Fig. 3) integrates the analyses to explore the implementation outcomes of appropriateness, acceptability, and sustainability.

Fig. 3

Joint display merging mixed methods data

Discussion

The current project provides an exemplar of concurrent implementation and evaluation of a telerehabilitation approach. Pragmatic implementation of the group-based telerehabilitation, NROL, was possible, well-attended, and demonstrated appropriateness to complement existing therapy for patients receiving community-based neurological rehabilitation. It was accepted as a new way of working across rehabilitation disciplines as an ‘extra layer of therapy’. NROL had perceived advantages in terms of patient and staff resource (e.g. saving time, energy and travel). NROL provided staffing efficiencies (ratio 0.6) compared to one-to-one delivery. Technology difficulties and reluctance were surmountable with dedicated technology assistance. Leadership commitment was considered key to enable the efforts needed for implementation and sustained use. NROL continues as part of standard practice. Evaluation was undertaken in a timely way to consider fit within existing healthcare from the outset.

NROL was compatible with usual service provision. It was appropriate for mixed demographics, including patients with a variety of neurological diagnoses (sudden, intermittent and progressive conditions) and chronicity. This breadth reflects the reality of neurological rehabilitation services but also creates challenges for group content. However, findings indicate delivering groups to patients with varying conditions and chronicity was suitable given there were no associations between these characteristics and direction of change in patient outcome scores. Additionally, benefits were identified by staff including enabling collaborative working across teams and caseloads. Relative to the wider caseload NROL patients were likely younger. A factor could be gate-keeping from therapists, as there was recognition NROL is not suitable for all patients which inherently requires decision-making. NROL was provided to patients from diverse socioeconomic backgrounds, and most lived with a family member or carer. Of note, we did not have any care home residents participating in NROL despite a likely need. Equity considerations warrant further investigation to ensure that embracing technology does not increase inequalities or create new ones.

Patient outcomes indicated improvements for many alongside emotive examples of how NROL benefitted patients’ lives. Echoing the results of Beare and colleagues [24], we found statistical improvement in self-efficacy. Additionally, we found that patients rating of their ‘ability to perform functional activities that were important to them’ increased by a similar magnitude to those reported following multidisciplinary in-person home-based rehabilitation with older adults [40]. Even with the relatively short time-period of the NROL programme, and the wider context of the on-going pandemic, health related quality of life also increased. Despite the caveat that group level improvements did not exceed referenced minimal clinically important differences, many individuals had scores that remained stable or improved for their self-efficacy, quality of life and activity performance. Several of these improvements were to a potentially clinically meaningful degree, which is notable given the inclusion of patients with deteriorating conditions. Overall patient outcome data align with the interview data which imply patients felt NROL helped from a physical, and mental health perspective and supported them to gain confidence and motivation outside of NROL. However, it was emphasised by both patients and staff that NROL must sit alongside in-person therapy, rather than replacing it. It is known that current therapy dose falls short of what is required in clinical guidelines and there is a recognised need to optimise opportunity to rehabilitation [51,52,53]. We demonstrated that telerehabilitation delivery methods could offer a compatible way to increase therapy and help meet guideline targets, but it should supplement rather than substitute therapy provision [12].

Telerehabilitation provided the opportunity to develop a new skill set for staff, with technology support being given to staff as well as to patients to enable their successful use of the online platform. Staff confidence with technology use increased over time. There has been suggestion that specific training is required for staff to deliver telerehabilitation [23, 54]. We found experiential training with technology support to be sufficient, which confers with Lawford and colleagues that first-hand experience of this method of delivery is key [55]. The staff:patient ratios reported reflect not only the staff delivery and technology assistance during sessions but also within-session staff training. The data captures a relatively early snapshot of delivering NROL, with the numbers of qualified staff per patient reducing over time. Telerehabilitation enables extra staff to attend as a way of sharing learning with newer team members and students, and potentially shadowing a virtual session is easier than being an extra person in an in-person session. This is important for sustainability as it grows skills that will increasingly form part of prospective service delivery and as future patients are more comfortable with technology.

As a concept, telerehabilitation has many advantages which were conveyed in our evaluation in terms of making the best use of the limited resources of both patients and staff. Telerehabilitation aligns with many current policies and priorities [1, 2, 56]. Strategic drivers include improved patient choice, managing workforce challenges and effective use of resources, exploiting the potential of digital technologies, and reducing carbon emissions. These drivers will continue to be relevant long after COVID-19 has receded thus to enable advantages to come to fruition, there needs to be investment. The implementation of NROL did take resources, time and effort which required buy-in and commitment from leaders (clinicians, management and clinical academics). These resources needed for implementation are not unexpected [6, 57] but current healthcare systems are often not set-up to facilitate this upfront effort. Going beyond initial implementation, the sustainability of telerehabilitation approaches are likely to need some service redesign [58]. For example, we identified that incorporating technology support roles within staffing allocations will be required. It can be a challenge to enable long-term sustainability as this requires buy-in from the stakeholders who have authority for system change (commissioners and managers). Arguably, influencing key stakeholders using data to demonstrate need and benefits of telerehabilitation enhances the likelihood of service redesign to embrace this new way of working.

The impact of COVID-19 meant that we could not wait for stronger evidence of effectiveness before implementing telerehabilitation [13]. This was not an effectiveness trial, rather it explored an alternative delivery method for neurological rehabilitation to help inform future practice. For NROL, the content is similar to in-person rehabilitation and existing effectiveness evidence shows telerehabilitation is not inferior to conventional delivery [16, 59]. As suggested in guidance on complex intervention research, the implementation and evaluation phases are not always sequential [60] and in this case are concurrent. We asked broad questions about NROL, assessing its value from multiple perspectives. We adapted NROL to fit within a healthcare system and evaluated this telerehabilitation approach in a rapidly changing context. The use of traditional randomised controlled trials was not fit for this purpose. A pragmatic, pluralist and timely approach to evaluation is required [60], with the use of implementation frameworks advocated [13]. This has many parallels with action research. We used the Consolidated Framework for Implementation Research [44] in our qualitative data collection and analysis and selected Proctor’s implementation outcomes to structure our evaluation findings. A challenge was that there is no clear consensus on how to define or use Proctor’s implementation outcomes. For example, what is deemed as acceptable and according to who? Historically, quantitative methods e.g. [61] or qualitative methods have been used [62] to deem what is acceptable. We used a mixed methods approach to ensure a comprehensive understanding of appropriateness, acceptability and sustainability from multiple perspectives, with triangulation from different stakeholders and methods. Going forwards, more discussion of how implementation outcomes should be defined and evaluated will help progress the field in generating broader evidence that is timely and fit for purpose.

Limitations

This was a pragmatic and constantly changing implementation and evaluation of a telerehabilitation programme in a changing context. Data capture was difficult, both from being able to extract from electronic systems within healthcare and consistent outcome measure use. Specific examples include it was not possible to extract comprehensive data on ethnicity, and it was also difficult to capture what was not done (e.g. who isn’t referred). Effort was made to capture data from outcome measures in current use (to improve resource efficiency and assist long-sustainability) but support was needed to ensure adequate data collection. NROL was delivered to patients with varied diagnoses, which has challenges with validity of outcome measures for different populations. With respect to the interviews, there is potential positive and social desirability bias, and the timing of the interviews was relatively early in the implementation so opinions may have changed over time. Although the conclusions regarding NROL advantages were primarily drawn from this interview data, a previous study has demonstrated quantitative service and travel efficiencies [4]. However, it is acknowledged that further evaluation combining qualitative and quantitative data would be beneficial, as would incorporating a cost analysis.

Conclusion

Pragmatic implementation of group-based telerehabilitation was possible within an existing healthcare system. Providing NROL as an adjunct to neurological rehabilitation has compelling advantages of in terms of leveraging patient and staff resource, and staffing efficiencies, supporting the continued use of this telerehabilitation approach. Additionally, NROL was seen as an opportunity to develop new professional skills and evolve existing roles. This project provides an exemplar of how evaluation can be run concurrently with implementation. A data driven rather than anecdotal approach is needed to harness opportunities afforded from the COVID-19 pandemic, rather than reverting to ‘normal’. Continuous evaluation is required to understand more, including on equity, and future delivery should explore how telerehabilitation can be upscaled and fit within a changing context.