Application of the development stages of a cluster randomized trial to a framework for evaluating complex health interventions

Abstract

Introduction

Trials of complex health interventions often pose difficult methodologic challenges. The objective of this paper is to assess the extent to which the various development steps of a cluster randomized trial to optimize antibiotic use in nursing homes are represented in a recently published framework for the design and evaluation of complex health interventions. In so doing, the utility of the framework for health services researchers is evaluated.

Methods

Using the five phases of the framework (theoretical, identification of components of the intervention, definition of trial and intervention design, methodological issues for main trial, promoting effective implementation), corresponding stages in the development of the cluster randomized trial using diagnostic and treatment algorithms to optimize the use of antibiotics in nursing homes are identified and described.

Results

Synthesis of evidence needed to construct the algorithms, survey and qualitative research used to define components of the algorithms, a pilot study to assess the feasibility of delivering the algorithms, methodological issues in the main trial including choice of design, allocation concealment, outcomes, sample size calculation, and analysis are adequately represented using the stages of the framework.

Conclusions

The framework is a useful resource for researchers planning a randomized clinical trial of a complex intervention.

Background

Antibiotic use in nursing home residents is an important public health issue given that 50% to 75% of residents are exposed to at least one course of antibiotics annually [1–4]. Such overuse can potentially lead to the development of multi-drug antibiotic resistant bacteria, drug-related adverse effects, and harmful drug interactions [5–14]. The appropriate use of antibiotics in this setting is challenging. The diagnosis of infection in residents of nursing homes is often difficult, since symptoms may be vague and signs difficult to elicit [14]. The evaluation of residents is often made by nursing staff who have varying levels of expertise and knowledge about infectious diseases. Frequently, treatment decisions are made by busy clinicians who rely on nursing assessment as their primary source of information. Interventions to optimize antibiotic use in this setting therefore need to directly involve both nurses and physicians and have the potential for strong "buy-in" from these healthcare professionals. Such interventions are aimed at changing behavior and by necessity will involve multiple components.

Recently, Campbell and colleagues published a framework for the design and evaluation of complex health interventions [15]. Derived from a document drafted by members of the MRC Health Services and Public Health Research Board, the paper is based on the premise that a different construct is required for trials that evaluate complex interventions. Examples include interventions to improve delivery of health services, to change health professionals' behavior, or to promote health through strategies aimed at individuals, schools, healthcare facilities, or communities. It is apparent that such trials pose very different challenges when compared to clinical trials designed, for example, to assess the efficacy of pharmacologic therapy [16]. Amongst these are the development, identification, documentation, and reproduction of the interventions. Although such challenges are well known [17–19], few published strategies exist for researchers about to embark on a clinical trial of these interventions. Campbell et al. propose that the development and evaluation of such complex interventions be considered in a series of phases, analogous to the sequential phases of drug development [15].

The design and evaluation framework proposed by Campbell et al. appears to be well suited to interventions for optimizing antibiotic use in long-term care facilities. The objective of this paper is to evaluate the extent to which the various steps in the development of a cluster randomized trial to optimize antibiotic use in nursing homes are represented in the framework for the design and evaluation of complex health interventions [15]. Often it is not possible to evaluate complex health interventions, such as strategies to improve antibiotic use in long-term care facilities, by randomizing individuals. Cluster randomized trials randomize groups of individuals, healthcare professionals, or institutions. Applying the development steps of this trial to the framework proposed by Campbell and colleagues allows for a formal evaluation of the framework's utility.

Methods

The five phases of the framework of Campbell et al [15] include 1) a pre-clinical or theoretical phase, 2) Phase I, identification of the components of the intervention, 3) Phase II, definition of the trial and intervention design, 4) Phase III, methodological issues for the main trial, and 5) Phase IV, promoting effective implementation of the intervention. Using the framework, the corresponding stages of the nursing home trial development are identified and described.

Results

Preclinical or theoretical phase

Overview

The purpose of this phase is to determine if the intervention has the potential to have the desired effect. This step can take into account previous studies demonstrating empirical evidence for the intervention as well as the theoretical basis for the intervention. Review of the theoretical basis for an intervention may lead to an altered hypothesis and improved specification of "potentially active ingredients" [15].

The first step in the development of the algorithms was to systematically review the literature for data that could lead to strategies for reducing antibiotic use. Lower respiratory infections, urinary infections, and skin and soft tissue infections account for the majority of bacterial infections in residents of long-term care facilities [1, 2, 4, 14]. No data to support reducing antibiotics for lower respiratory or skin infections was located. In contrast, data to support reducing antibiotic use for urinary indications was found. Asymptomatic bacteriuria, the presence of bacteria in the urine in the absence of urinary symptoms, occurs in up to 50% of older institutionalized women and 35% of institutionalized older men [20]. There are four randomized controlled trials that demonstrate no benefit in treating asymptomatic bacteriuria in residents of long-term care facilities [21–24]. Despite the evidence, one third of all prescriptions for urinary indications in residents of nursing homes are for asymptomatic bacteriuria [1]. Accordingly, the algorithms indicate that urine should not be cultured and antibiotics should not be prescribed in the absence of clinical features of urinary infection (Figures 1 and 2). The algorithms also consider the increased risk of urinary infection with indwelling urinary catheters and with specific urinary symptoms [25–28].

Figure 1

Diagnostic algorithm. This algorithm guides physicians and nurses in the ordering of urine cultures for nursing home residents with suspected infections.

Figure 2

Treatment algorithm. This algorithm allows physicians and nurses to optimize antibiotic use in residents with suspected infections.

Phase I: defining components of the intervention

Overview

This phase, often inter-related to the previous phase, emphasizes modeling or simulation studies to improve understanding of the components of an intervention. Use of qualitative research and descriptive studies may help define components of the intervention. Furthermore, in this phase potential barriers in trials that seek to change behavior may be elicited [15].

In order to assess the potential effectiveness of the two algorithms, we applied them to data from a study assessing antibiotic use in Ontario chronic care facilities [1]. In this modeling exercise, we found that the algorithms could reduce antibiotic use for urinary indications by 80%. Previously, we had conducted focus groups among physicians and nurses who provide care to nursing home residents. Our objective was to explore the perceptions, attitudes, and opinions of those involved in the process of prescribing antibiotics for asymptomatic bacteriuria [29]. Twenty-two physicians and 16 nurses, each in two focus groups, participated. Both physicians and nurses emphasized that there was a wide range of indicators, more subtle than the classic symptoms of urinary tract infection, that influence the ordering of cultures and the prescribing of antibiotics. Because residents in long-term care facilities frequently have cognitive impairment and are unable to articulate their symptoms, health care providers rely instead on signs of more subtle functional and behavioural changes. Misconceptions about the definition of symptoms of urinary tract infection were also discussed. Although the nurses participating in the focus groups had learned that foul-smelling urine alone was not sufficient reason to order a culture, a number of physicians still felt that this was one of the main reasons why cultures are ordered. Other key findings included the influence nurses have over physician prescribing through their description of the clinical situation, often conveyed over the telephone. Physicians attributed their use of antibiotics to treat bacteriuria in residents with no urinary symptoms to relatively vague non-urinary symptoms (e.g. lethargy, change in functional status). To these physicians, this meant that such residents were not in fact "asymptomatic", despite the fact that absence of urinary symptoms alone was used to define asymptomatic bacteriuria in the clinical trials [21–24]. This was an important barrier to evidence-based care that would not have been discovered without using a qualitative approach.

To assess the factors leading to the ordering of urine cultures, we conducted a prospective survey in four nursing homes. Nursing staff were asked to complete a brief questionnaire every time a request for a urine culture was made. They were asked to specify the signs and symptoms present at the time the culture was ordered and to specify who initiated the request for the culture. Physicians, after they received the results of the urine culture, were asked to indicate the clinical reasons for their therapeutic decision. Forty-eight percent of the 545 urine cultures obtained over six months were ordered by nursing staff and 52% by physicians. The urine culture was suggested by nursing staff 56% of the time. For 67% of urine culture orders, there were no urinary symptoms present at the time the culture was requested. One in every three urine cultures ordered resulted in an antibiotic prescription. Of these prescriptions, one third were for asymptomatic bacteriuria.

Phase II: defining trial and intervention design

Overview

In this phase, the information gathered in phase I is used to develop the optimum intervention and study design. The feasibility of delivering the intervention is tested and the acceptability to healthcare providers and patients is assessed [15].

The diagnostic and treatment algorithms were tested and further refined in a three-month pilot study conducted in four nursing homes. Administrators and directors of nursing of the homes felt that the algorithms should be introduced to the nursing staff by each facility's infection control practitioner. To ensure that the infection control practitioners themselves understood the algorithms, they were asked to decide whether urine cultures should be obtained and antibiotics ordered for six case-scenarios using the algorithms. Using this "train the trainer" approach, the practitioners then conducted brief seminars with registered nursing staff to introduce the algorithms in their facilities. This approach was unsuccessful however. Semi-structured interviews with staff revealed that in two of the homes there were nurses who were completely unfamiliar with the algorithms two-thirds of the way into the pilot study. This led to the development of a standardized 30-minute presentation of six case-scenarios facilitated by one of two study investigators. Participation was active, staff were asked to decide whether to order antibiotics and urine cultures and to justify their answers using the algorithms. In addition, a video-tape replicating the seminar material was made and distributed to the nursing homes. The use of logs to document appropriate adherence to the algorithms was also piloted during the three-month pilot. Office visits to introduce the algorithms to nursing home physicians were conducted using the six case-scenarios presented to nursing staff. Feedback about the algorithms from both physicians and nurses was positive, both groups felt that the algorithms were user friendly and feasible in the long-term care setting.

Additionally, the initial poor adherence to the algorithms indicated that regularly scheduled on-site visits should be another component of the intervention, serving as an important reminder or cue to the participants. Meetings with the local champion for the study (usually the director of nursing) to discuss barriers or obstacles to completing the study were therefore held during the pilot. Feedback about adherence to the protocol was given. Such follow up visits were acceptable to staff and administrators as they were not overly time consuming.

Phase III: methodological issues for the main trial

Overview

Campbell and colleagues state that this step addresses issues normally posed by randomized trials, such as sample size, inclusion and exclusion criteria, methods of randomization, as well as the challenges of complex interventions [15].

The main antibiotic trial uses a randomized matched pair design where one of 11 pairs of nursing homes (matched by size and case-mix), has been randomized to the clinical algorithms and the other to usual management of presumptive urinary tract infections. Only free-standing, community-based nursing homes are eligible i.e. those representative of the majority of nursing homes in Canada and the United States. To be eligible, nursing homes must also: 1) have 100 or more residents (since the required sample size of antibiotics courses will be achieved sooner); 2) have no stated policy for diagnosis or treatment of urinary tract infections; 3) agree to refrain from introducing new management strategies for antibiotic utilization or urinary tract infection during the study. The primary outcome of this trial is the total number of antibiotic courses prescribed. Other outcomes include 1) the proportion of antibiotic courses prescribed for urinary indications, 2) rates of urine cultures ordered, 3) hospitalization rates for urinary tract infections, and 4) mortality rates. The last two outcomes are being measured to ensure that the algorithms can safely be applied. Within a nursing home, randomization of individual healthcare providers to the algorithm likely would introduce bias due to contamination. Therefore, for the quantitative component of this study, the nursing home will serve as the unit of allocation and analysis.

To conceal allocation, one nursing home in each pair was arbitrarily assigned an even number (and the other odd) by a hospital administrative assistant not associated with the study or any of the investigators and who was blinded to the nursing homes' identity. A statistician who is not associated with the study and who was unaware of the nursing home pairs, generated a list of 11 random numbers using a random numbers table, one for each pair of nursing homes. The corresponding odd or even numbered nursing home in each pair was assigned the intervention.

An absolute reduction in 20% of all antibiotic use, which corresponds to a reduction in the proportion of antibiotic prescriptions for urinary indications from 30% to 10%, was considered the minimal clinically important effect worth detecting. To detect this difference, for an alpha of 0.05 and 80% power, adjusting for the effect of within cluster dependency, 4513 prescriptions need to be collected in total. This means that 20 or 10 pairs of nursing homes will need to be followed for 10 months. Given the potential for withdrawals of nursing homes, 22 homes will be recruited.

A paired t-test will be used to analyse the within-pair differences between the proportions of antibiotics prescribed for urinary indications in matched pairs of nursing homes. This way, the fact that the denominator of the proportions is actually an outcome is taken into consideration. Differences in rates of overall antibiotic use (antibiotic courses per 1000 resident days) will be compared using a paired t-test. Rates of antibiotic use for urinary indications (antibiotic courses per 1000 resident days and defined daily dosages/1000 resident days), rates of urine cultures obtained (urine cultures per 1000 resident days), rates of hospitalization (per 1000 resident days), and overall mortality rates will be compared using paired t-tests and Wilcoxon signed rank test. Logistic regression analysis that adjusts for the effect of clustering will be performed to adjust for potentially important co-variates such as nursing home size, proportion of residents bed/wheelchair bound, and pharmacy automatic stop dates.

Phase IV: promoting effective stability of the intervention, assessing for the existence of adverse effects

Qualitative studies to assess the attitudes and perceptions of both nurses and physicians will help us understand the key elements to maintaining sustainability should the intervention prove to be successful in reducing antibiotic use. We plan to conduct focus group studies for nursing staff and semi-structured interviews amongst physicians and directors of nursing care. If the intervention is not successful, this qualitative approach may help explain the barriers leading to failure of the intervention.

Although the trial itself is meant to examine the effectiveness of the intervention, including adherence and adverse effects, if the results demonstrate that the intervention can reduce overall antibiotic use and is safe, one possibility is to conduct a cohort study of the algorithms in nursing homes over a large geographic area will be made.

Discussion

The application of the trial to improve antibiotic use in nursing home residents to the framework demonstrates that the framework is a useful resource for health services researchers. Not surprisingly, the stages of the framework have been used in previous studies of complex interventions, even before the framework was published. One recent example is the Rapid Early Action for Coronary Treatment (REACT), a multi-component, community-based educational intervention of community development, public education, provider education, and patient education (31). The description of the trial described in this paper however allows for a focused assessment of the components of this framework.

The pre-clinical phase of the antibiotic trial involved a summary of the evidence needed to construct the algorithms. This phase should include a systematic review of the literature with respect to the intervention [32], a point omitted in the pre-clinical phase description of the design paper but discussed in the Medical Research Council paper http://www.mrc.ac.uk/index/publications/publications-electronic_publications.htm described in the text [15]. It was also evident that the preclinical and first phase cannot be clearly demarcated, given that it was the modeling study in phase I which actually determined if the intervention had the potential to have the desired effect, the stated aim of the preclinical phase [15].

The use of qualitative research to help define the intervention was extremely valuable. Focus group studies helped to accurately identify and anticipate barriers to the intervention, even prior to the pilot study. Determining physicians and nurses attitudes, beliefs, and behaviors toward antibiotic use for urinary infections highlighted the importance of nonspecific, non-urinary symptoms and the key role of nurses in the process of antibiotic treatment of asymptomatic bacteriuria in institutionalized older adults. As specified in the framework, these findings illustrate the utility of using qualitative methods early in the development of complex interventions. Focus groups and semi-structured interviews will also be conducted at the end of the main trial. We anticipate that such qualitative data will help us identify strategies to sustain the algorithms should the algorithms prove to be effective in reducing antibiotic use [33, 34].

Results of the questionnaires administered to nurses and physicians used in developing the algorithms provided convincing evidence about the need to target both ordering of urine cultures and antibiotic prescribing in order to reduce overall antibiotic use. The particular importance of introducing the algorithms to nurses in order to culturing of urine was emphasized. This confirmed the qualitative findings. It is notable that although the phases of the framework are described as being sequential [15], components of the frameworks might vary temporally, as exemplified by the qualitative and questionnaire data described in phase I being collected prior to the actual development of the algorithms.

The pilot study was instrumental in refining and introducing the algorithms. One of the most valuable aspects of the pilot study was finding the optimal way to introduce the algorithms to nurses and physicians. Perhaps more emphasis could have been placed on strategies to introduce the complex health interventions in the design framework [15].

An important methodological issue for randomized trials of complex health interventions is allocation concealment. This was a key consideration in the design and implementation of the antibiotic trial. Empiric evidence exists to suggest that failure to conceal allocation can result in larger biased estimates of effect than failure to blind [35]. When a limited number of clusters are randomized, such as in this trial, there is extra concern [36]. Although it is often impossible to blind patients, practitioners, and researchers to the intervention, it is always possible to conceal allocation [35]. Such issues are well described in the Medical Research Council paper described in the text [15].

Sample size calculation is an important challenge in cluster-randomized trials. Cluster randomization reduces power and the sample size generally needs to be substantially increased. Further elaboration on this topic in the Phase III of the framework would be of benefit to researchers.

Conclusion

The application of a randomized trial to optimize antibiotic use in nursing homes to the framework reported by Campbell and colleagues demonstrated that the framework is a useful resource for researchers embarking on a clinical trial of a complex health intervention.