Design of the Whole Systems Demonstrator trial

The WSD was designed as a pragmatic trial to assess the impact of telehealth when introduced as part of the routine delivery of care. Unlike explanatory trials that compare highly-specified interventions in controlled situations on tightly defined cohorts of patients, pragmatic trials often use broad eligibility criteria to enrol patients, test flexible interventions as practiced by typical (not expert) practitioners, and make comparisons with usual care [25]. Therefore, an important part of the evaluation design was to leave aspects of the design of the telehealth interventions to local teams. The protocol for the trial and evaluation has already been published [14], along with a detailed description of the interventions (Web Appendix [21], and Web Appendix 2 [20]). We summarise the main features below.

The Department of Health selected the WSD sites in 2006 through a competitive process because (a) they were considered the most likely to succeed in scaling up remote care as part of a whole system redesign, and (b) they were considered representative of the range of local health and social care systems in England [22]. Cornwall, Kent and Newham in East London were chosen. In order to recruit suitably large numbers of patients, and because individual randomisation was unlikely to be acceptable to stakeholders, cluster randomisation was used [14]. Thus, all general practices in the three demonstration sites were invited to participate by site teams, and the general practices that accepted the invitation were randomised according to a centrally-administered minimisation algorithm to provide patients for either telehealth or usual care.

Patient inclusion criteria specified only age 18 years or over, plus a diagnosis of diabetes, chronic obstructive pulmonary disease or heart failure. Patients with these diagnoses were sourced from the existing lists of patients registered at participating general practices, using routine primary and secondary care data and referral letters from clinicians. Exclusion criteria related to the patient not understanding the instructions for the equipment (which were provided in English), or living in a home unsuitable for telehealth (for example, with inadequate telephone line connection). Eligibility was not conferred on the basis of a formal clinical assessment of disease severity, and patients with additional co-morbidities (in addition to the targeted long-term conditions) were not excluded from participation.

Once potentially eligible patients had been identified, they were written to at home and asked for their permission to share data with the research team so that their eligibility could be confirmed. Thus, all participants signed a consent form to agree to their details being examined to establish their eligibility for the trial. Once eligibility had been established, they were visited at home, where they signed a further consent form to agree to participate in the trial. Treatment allocations for patients followed those of the general practices at which the patients were registered. Though patients were not told about these allocations until after consent had been given, the long period of recruitment and cluster-randomised method meant it was not always possible to blind those recruiting patients.

The target sample size for the trial was 3,000 patients, based on the numbers needed to detect a 17.5% change in the primary outcome (hospitalisation proportion) from the levels assumed for usual care (with an intra-cluster correlation of 0.001 and an assumed cluster size of 25) [19]. Patient recruitment began in May 2008 and was scheduled to end in September 2009, though a small number of patients were recruited later.

The current study was conducted on the subset of WSD patients who had diabetes as their ‘index condition’. In the WSD trial, these index conditions were assigned in order to limit the number of disease-specific questionnaires given to patients; where a patient had more than one of the targeted long-term conditions, one condition was selected using randomisation [20]. We further restricted the sample to patients with type 2 (rather than type 1) diabetes by applying codes to the diagnoses recorded in general practice data (codes given in Additional file 1).

Interventions

The trial was not designed to investigate the effect of individual service configurations or technologies [14]. Rather, it sought to understand the effects of ‘telehealth’, as a class of technologies that was added to standard support and treatment, compared with standard care alone. Consistent with the pragmatic design, sites were intentionally left to design aspects of their own telehealth services, such that there was some scope to tailor the services to local needs and resources. However, in all sites, telehealth equipment included a base unit (freestanding or a television set top box) and a glucometer for diabetes, weighing scales for heart failure and pulse oximeters for chronic obstructive pulmonary disease. In addition, most intervention patients received blood pressure monitors regardless of their long-term condition. Patients with multiple conditions could receive several peripherals. The peripherals communicated with the base unit either wirelessly or by cable.

The telehealth equipment reminded participants to take physiological measurements at the same time each day for up to five days per week, with the frequency adjusted according to participants’ individual clinical histories. For example, participants with diabetes and well-controlled blood glucose were typically asked to take readings less frequently than participants whose blood glucose was poorly controlled. In addition to the telemonitoring aspect of the intervention, symptom questions (e.g., ‘how are you feeling today compared with yesterday?’) and educational messages were sent to participants, either via the telehealth base unit or via a set top box connected to a television, depending on the study site.

Readings and answers to symptom questions were sent to monitoring centres using store-and-forward technology and secure servers. These centres were staffed with trained support workers and either specialist nurses or community matrons, depending on the site. Incoming biometric readings were automatically transformed into a traffic light classification of clinical risk, using thresholds (set initially on the basis of clinical guidelines or by a clinician responsible for the patient’s care) and algorithms that differed between sites. Urgent readings (‘red flags’) were reviewed and responded to daily, while other readings could also be reviewed to identify deteriorations in health. The scope of nurses to respond to the information from patients varied according to their role, with community matrons tending to have greater scope than monitoring centre nurses to initiate clinical interventions, such as a home visit or medicine titration within certain limits. However, all nurses could evaluate the patient, offer advice on disease management and refer patients to other healthcare professionals, such as general practitioners. Monitoring nurses were also able to transmit health related questions, messages, or videos to educate patients on their conditions, using the telehealth base unit or set top box.

The Department of Health provided funding to the demonstration sites for the telehealth interventions, as well as project manager support for the implementation of telehealth. Telehealth patients were not charged for using the telehealth services (including calls to the monitoring teams), but they were expected to have a telephone line and electricity.

Control patients were not provided with any specific support as part of their participation in the trial, though a subset participated in nested questionnaire studies [20, 21]. They received usual care for the study sites, which excluded telehealth but for some patients included self-monitoring of blood glucose as agreed with their clinician. Control patients were offered telehealth at the end of the 12-month trial period if they were still eligible at that point.

Collection of data on HbA1c

The original trial protocol specified that HbA1c would be collected at baseline, 3-month follow-up, and 12-month follow-up. During the course of the trial, however, practical issues with the complex, community-based recruitment procedure were identified that meant that the protocol was revised so that bespoke clinical measures were no longer collected. Instead, an analysis of the electronic medical record was planned. In England, these electronic records are used to calculate general practice entitlements to performance-related pay under the Quality Outcomes Framework (QOF). At the time of the trial, QOF criteria included the maintenance of codes for the diagnosis of diabetes, together with regular recording of HbA1c [26]. Although it was recognised that these data would have limitations (discussed below), it was expected that they would be more complete for diabetes than for heart failure or chronic obstructive pulmonary disease.

All general practices participating in the trial were asked to share event-level data from the electronic medical record for their registered adult patients [27]. Data were effectively anonymised (‘pseudonymised’) before they were transferred to the evaluation team, with patient identifiers removed and a unique patient identifier (the ‘NHS number’) encrypted. The encrypted NHS number was used to link the general practice data to trial data sets [17].

The use of routine data meant that it was necessary to exclude a small number of participants. First, we excluded patients who were not linked to the routine data because NHS numbers were not available for encryption. A second exclusion arose because our study focused on the period from two years before a patient was recruited in the trial to 12 months following that recruitment (or death, if sooner) – this was known as the ‘observation period’. We were concerned that, although all patients were registered at a participating general practice at the time of trial recruitment, they might have been registered at another general practice at some other point during the observation period. If this other general practice did not provide data for the evaluation (for example, because it did not participate in the trial), then we would have incomplete information about diagnosis codes and HbA1c. As a result, we restricted the analysis to patients who were always registered with one of the observed general practices during the observation period.

Baseline characteristics

Baseline variables were derived using the general practice data (see codes in Additional file 1). Consistent with other strands of the evaluation [19], for the purpose of calculating variables, the trial start date was taken as the date of telehealth installation for intervention patients, and as the date of the initial project team visit for controls.

In cluster-randomised trials, selection bias is theoretically possible, either through systematic differences between practices in the control and intervention groups, or through similar differences at the individual level [28]. Following previous work [19], we present standardised differences as a summary measure of differences between groups at baseline; these were defined as the differences in sample means (or proportions), divided by the pooled standard deviation [29]. For continuous variables, we also present the variance ratio, defined as the variance in the intervention group divided by the variance among controls [30].

Analysis of HbA1c frequency

We did not expect telehealth to alter the frequency with which HbA1c was recorded in general practice (as indicated by previous work [27]). To confirm this, we compared numbers of HbA1c readings during the 12-month trial period between treatment groups using chi-squared tests. We also tested whether readings were made uniformly across the trial period, using the Kolmogorov-Smirnov test and pooled data across both treatment groups.

Comparison using repeated measures models

Analysis of HbA1c levels was conducted on an intention-to-treat basis, using all HbA1c readings recorded within the observation period. Analysis was conducted using a repeated measures linear regression model, which included adjustment factors (fixed effects) for:

As HbA1c readings for the same individual at different time points will tend to be correlated, the model included a first-order autoregressive correlation structure. As patients at the same general practice will tend to have more similar HbA1c readings than patients at different general practices, the model included random effects at the general practice (cluster) level. Finally, the model included random effects for the time of the readings because these were not made on a fixed schedule.

Comparison of average HbA1c before and after recruitment

In order to test the robustness of our results to the models used, we conducted a series of secondary analyses. These were done on a ‘completer’ basis and thus were restricted to the subset of study participants who had at least one HbA1c recorded during the 12-month trial period. Analyses were conducted at the patient level, summarising the HbA1c readings recorded for each individual during the trial period using the mean. Differences between the treatment groups were then assessed using mixed linear regression [31]. The model included random effects for the general practice. As well as doing unadjusted analysis, we also adjusted for a set of baseline characteristics in the regression. These were: last-recorded HbA1c prior to recruitment; age; sex; last-recorded blood pressure, cholesterol quartile and BMI quartile prior to recruitment; site (Cornwall, Kent or Newham); smoking status; co-morbid ischemic heart disease; and diabetes management (insulin or non-insulin). We conducted a range of sensitivity analyses with other model specifications. These are described in Additional file 2.

Proportions of patients below thresholds

At the time of this trial, general practices in England had a specific goal to maintain glycosylated haemoglobin (HbA1c) below 7.5% (58 mmol/mol) for patients with diabetes. Therefore, we compared the proportion of intervention and control patients under this threshold, again averaging multiple readings for the same individual. Between-group differences were assessed using mixed logistic regression, with random effects for the general practice. We also used graphical methods to compare the distributions of HbA1c between intervention and control groups (again, averaged over the trial period). This was done using the HbA1c cut-off points adopted in a previous study, which used electronic data from general practices in the UK to examine the relationship between HbA1c and survival [32].

Ethical approval

The Liverpool Research Ethics Committee (reference 08/H1005/4) approved the study in all of the participating sites.

Strengths and weaknesses

The complexity of the recruitment processes and the size of the trial meant that we relied on routine electronic data for HbA1c, rather than bespoke data collection. Routine data offer promising opportunities for research [34], and this study offers one model for how routinely-recorded clinical information can be used in analyses of randomised controlled trials. One challenge was that the timing of the available HbA1c measurements was contingent on decisions made by patients and professionals, rather than, as in a conventional trial, specified by the trial protocol. If, for example, HbA1c tends to be taken in response to concerns about patient health, we may have disproportionately sampled HbA1c when it was higher. In assessing the potential bias from our data-collection approach, we were reassured that the intervention itself did not appear to lead to changes in the frequency of measurement. Further, results from our primary and secondary analyses were similar. As the secondary analysis averaged multiple readings recorded for each individual, they ensured that all individuals contributed equal weight to the analysis, regardless of the number of readings made for each one The similarity of the results of the analyses suggests that they were not biased towards patients who attended general practice most often. We note that our findings reflect the average HbA1c recorded over a 12-month period. We did not assess whether short-term effects on HbA1c (such as at three months) were larger than those at 12 months. This could have occurred, if, for example, patients adhered to the monitoring schedule more closely at the start than at the end of the trial.

Selection bias is recognised as a risk in cluster randomised trials, particularly when the individuals recruiting patients have foreknowledge about treatment allocations, as might sometimes have been the case in this trial. Reassuringly, we did not find large baseline differences between intervention and control groups in the primary study of all 3,230 patients [19]. In the current study, with smaller samples, larger differences were more likely to have occurred by chance. The most notable difference concerned the size of the treatment groups (300 patients for the intervention group vs. 213 patients for the control group). Furthermore, although general practices in the intervention and control arms typically provided similar numbers of patients for the current study (median 3 patients in each arm), some intervention practices supplied up to 38 patients, compared with up to 16 for control practices. This pattern was also not systematic across the wider trial, and indeed the primary study found differences in cluster sizes in the opposite direction [19].

To limit the impact of selection bias, we conducted analysis on an intention-to-treat basis and adjusted for important prognostic variables including age, smoking status, co-morbid ischemic heart disease, and diabetes management. Adjustment tended to make estimated effect sizes larger. The use of routine data meant that some covariates of interest were not available, in particular time since diagnosis, prior self-monitoring activity and micro-vascular complications. We therefore had to rely on randomisation to balance these characteristics between intervention and control groups. Although cluster randomisation has limitations, it reduced the scope for contamination between treatment groups as no general practice provided care for both telehealth and control participants. Individual randomisation would not have been acceptable to the stakeholders involved in this study [14].

Approximately 80% of the individuals invited to participate in the WSD trial refused to do so [19]. Such high rates of patient refusal are not unusual for telehealth trials [35], and by themselves suggest that telehealth might not always meet the perceived needs of patients and that many patients may be reluctant to change from their current mode of care [23, 36]. A qualitative study explored the reasons given by patients for not participating in the WSD trial. These related to the requirements for technical competence and operation of the telehealth equipment; threats to identity, independence and self-care; and expectations and experiences of disruption to services [23]. One implication of the high refusal rate is that the included patients were probably not fully representative of the wider population with type 2 diabetes. Compared to cross-sectional studies of the diabetes population, the current study included a relatively high proportion of insulin-treated patients (49.7%) [3]. Also, baseline levels of HbA1c were higher in the current study than in IDEATel [11] (mean 8.5% vs. 7.4%, or 69 vs. 57 mmol/mol). Some comorbidity arose from the other two conditions targeted by WSD, namely heart failure (affecting 13.0% of intervention patients) and chronic obstructive pulmonary disease (affecting 16.3%). Overall, it seems likely that we studied a population with relatively less well-controlled diabetes. A formal, empirical assessment of the generalisability of the WSD trial is planned in relation to its primary endpoint of hospitalisation. This assessment will use observational data sets to compare the baseline characteristics and outcomes of WSD control patients with the baseline characteristics and outcomes of patients receiving usual care outside of the trial [37].

Usual care already consisted of a blend of different (non-telehealth) approaches to improving glycaemic control. Information was not available on rates of self-monitoring in the control group, but it would not have been universal [4]. Telehealth was implemented in the context of wider, whole-systems redesigns on-going in the sites. This redesign might have affected care delivered to both intervention and control patients, though a companion study found that the extent of integration achieved during the evaluation period was quite limited [22].

Although the WSD sites were diverse, they were selected based on their history of innovations in these areas of care. Furthermore, there was dedicated funding and project management for telehealth in this trial. Telehealth might have different effects in other settings, and important factors might include: which patients are recruited; the available staff and how they undertake their roles; other on-going approaches to managing long-term conditions; the local context, including financial incentives; integration between care teams; and the details of the telehealth and health information technology and monitoring systems. In general, the recruitment and monitoring protocols used in randomised controlled trials can affect outcomes for both intervention and control groups, for example through Hawthorne effects [38].

Comparison with existing studies

Our findings are in line with previous meta-analyses of the effect of telehealth versus usual care. One of these meta-analyses examined 12 randomised controlled trials and found reductions in HbA1c of around 0.22%, or 2.4 mmol/mol (95% CI, 0.08 to 0.35%, or 0.9 to 3.8 mmol/mol) [10]. Another examined 15 trials and found smaller impacts (-0.10% or 1.1 mmol/mol) that did not reach statistical significance [39]. In comparison to these meta-analyses, larger effect sizes where found in a review of telehealth interventions using mobile phones for patients [40]. These interventions were associated with reductions in HbA1c of 0.8%, or 8.7 mmol/mol (95% CI, 0.5 to 1.1%, or 5.5 to 12.0 mmol/mol). Overall, the evidence suggests that interventions such as telehealth are, at best, associated with modest reductions in HbA1c compared with usual care [41].

A number of differences exist between WSD and the previous IDEATel trial, which found larger impacts on HbA1c [12]. For example, the telehealth services developed by the WSD sites were largely based in primary care [14], while IDEATel contained an element of case management under diabetologist supervision. A particular feature of WSD is that we left the development of telehealth services largely to the participating sites. Although the resulting plurality might be problematic for the purposes of replicating the specific interventions, in some ways it is the merit of a pragmatic trial as it means that our results reflect decisions made by local teams, thus increasing their generalisability to routine clinical practice.