From concept to content: assessing the implementation fidelity of a chronic care model for frail, older people who live at home
BMC Health Services Research volume 15, Article number: 18 (2015)
Implementation fidelity, the degree to which a care program is implemented as intended, can influence program impact. Since results of trials that aim to implement comprehensive care programs for frail, older people have been conflicting, assessing implementation fidelity alongside these trials is essential to differentiate between flaws inherent to the program and implementation issues. This study demonstrates how a theory-based assessment of fidelity can increase insight in the implementation process of a complex intervention in primary elderly care.
The Geriatric Care Model was implemented among 35 primary care practices in the Netherlands. During home visits, practice nurses conducted a comprehensive geriatric assessment and wrote a tailored care plan. Multidisciplinary team consultations were organized with the aim to enhance the coordination between professionals caring for a single patient with complex needs. To assess fidelity, we identified 5 key intervention components and formulated corresponding research questions using Carroll’s framework for fidelity. Adherence (coverage, frequency, duration, content) was assessed per intervention component during and at the end of the intervention period. Two moderating factors (participant responsiveness and facilitation strategies) were assessed at the end of the intervention.
Adherence to the geriatric assessments and care plans was high, but decreased over time. Adherence to multidisciplinary consultations was initially poor, but increased over time. We found that individual differences in adherence between practice nurses and primary care physicians were moderate, while differences in participant responsiveness (satisfaction, involvement) were more distinct. Nurses deviated from protocol due to contextual factors and personal work routines.
Adherence to the Geriatric Care Model was high for most of the essential intervention components. Study limitations include the limited number of assessed moderating factors. We argue that a longitudinal investigation of adherence per intervention component is essential for a complete understanding of the implementation process, but that such investigations may be complicated by practical and methodological challenges.
The Netherlands National Trial Register (NTR). Trial number: 2160.
In the last decades, gerontological and geriatric researchers have been increasingly incorporating the concept of frailty into their work. In general, people are considered frail when they have entered a life stage defined by multiple health problems and loss of reserves on several domains of functioning [1–3]. As populations are aging and life expectancy is rising, the group of frail, older people is increasing. In Europe, the prevalence of frailty is estimated to range from 4.1% among middle-aged people to 17.0% among people of 65 and over . Since frailty is associated with multimorbidity, disability and loss of functioning, frail, older people often have an increased need for chronic care . Changes in health policy and the increasing desire to ‘age in place’ have caused the sites on which this care is delivered to relocate from the institutional setting to the home environment [5,6], making frail, older people one of the fastest growing populations in primary care [7,8]. As a result, pressure on primary and community care systems has been increasing, and elderly care is facing a number of challenges: frail older people’s health and care needs are not always identified at a timely stage, care is fragmented and coordination between care professionals is often inadequate, and older people experience a lack of involvement in their own care process . These challenges necessitate a reorganization of the way in which primary elderly care is organized and provided.
With the aim to achieve this reorganization, researchers, policy makers and health care professionals have been looking to design and implement models of care delivery that relieve the pressure on primary care systems whilst upholding quality standards for elderly care. One of these models is the Geriatric Care Model, a framework for integrated care delivery based on the Chronic Care Model (CCM) . The CCM aims to guide quality improvement and disease management activities in care for patients with a chronic disease. The model envisions productive interactions between a prepared, proactive practice team and an informed, activated client, and is based on the premise that the delivery of chronic care services takes place in a primary care environment that consists of three domains: the community domain, the health care system domain, and the provider organization domain. In this environment, the foundation of optimal care delivery consists of six essential elements, or ‘pillars’, of which four refer to the content of the care process (i.e. self-management support, delivery system design, decision support, and clinical information systems) and two to its context (i.e. community resources and policies and health care organization). Since the adaptability of the CCM framework allows it to be tailored to practice, in Europe, the framework has been used to guide disease management for a diverse range of patients with a chronic illness in several health care environments and settings [11–13]. However, its usefulness in integrated primary or community care settings that cater to patients with complex health problems, such as frail older people, has not been as thoroughly explored [13,14].
Although there are indications that the CCM has the potential to improve quality of care delivery, its effect on patient outcomes is uncertain. Some studies report results on outcomes such as length of hospital stay, while others find no results at all [15–17]. The inconsistencies might be due to the fact that CCM-based interventions are often complex: they involve multiple health and social care disciplines and combine elements of existing interventions. The aggregation of several interacting and potentially effective components and variations in delivery across sites complicate the relationship between intervention and impact, which challenges the interpretation of research outcomes. Insight in the true nature and delivery of CCM-based models in primary care is necessary in order to help researchers understand what factors cause such interventions to fail or succeed, what components work, and for whom [18–21].
One way to gain insight in the way in which an intervention is carried out is by assessing implementation fidelity, defined as the degree to which interventions are implemented as planned by the developers . Implementation fidelity moderates the relationship between an intervention’s content and its intended outcomes. Previous studies have demonstrated that the fidelity with which a program is implemented influences its level of success [22–26]. When implementing health and care interventions, measuring implementation fidelity helps researchers understand whether a lack of success is due to inappropriate service delivery or program inadequacies, which prevents them from drawing inaccurate conclusions about program effectiveness (a so-called ‘type III error’) [27–29]. In addition, insight into the fidelity of an intervention facilitates further improvement of intervention outcomes and dissemination of research findings into practice. However, despite repeated recommendations, fidelity assessments have not yet been fully adopted as common practice in primary elderly care research [19,20,30,31].
When implementing and testing the Geriatric Care Model in primary care, we therefore aimed to assess implementation fidelity, using the framework proposed by Carroll et al. . Carroll sees adherence, defined as “the degree to which implementers adhere to the intervention as intended by the intervention designers”, as the main element of fidelity, and distinguishes four adherence subcategories: content, coverage, frequency and duration. Content refers to the ‘active ingredients’ of an intervention, i.e. the essential services that an intervention aims to deliver to intervention participants. Coverage, frequency and duration quantify how much of the intervention’s content was delivered to the target group, how often and for how long. According to Carroll’s framework, all four concepts need to be evaluated to get an understanding of fidelity: the more subcategories of adherence investigated, the more ‘complete’ the assessment of fidelity. In addition, Carroll suggests that a sound assessment of implementation fidelity includes an investigation of factors that may modify fidelity. Identification of such factors contributes to our comprehension of what causes fidelity to be either high or low, and could help us overcome barriers to adequate implementation in the future. Four potential moderators are intervention complexity, availability of facilitation strategies, quality of delivery and participant responsiveness ; Hasson has proposed adding a fifth and a sixth moderator, i.e. recruitment and context .
In this paper, we present the results of a study that assessed the implementation fidelity of the Geriatric Care Model. We describe key components of the intervention program, and give insight in the degree to which professionals adhered to the delivery of these components. In order to provide an explanation for our findings, we describe moderating factors and other factors that may have influenced implementation. We hope to contribute to the existing knowledge about the implementation of the CCM in geriatric settings, and to the assessment of fidelity of complex interventions in primary elderly care.
The geriatric care model: objective and content
The Geriatric Care Model aims to restructure the way in which care for frail, older people in primary care is delivered. Through early identification of health and care needs, improved client autonomy and enhanced coordination between care professionals, it envisions to improve older people’s quality of care and, subsequently, their quality of life. The geriatric care model was integrated into routine practice by primary care practices, supported by practice nurses and geriatric expert teams. Geriatric teams managed practice nurses, who proactively visited older people at home and carried out a geriatric assessment. Assessment outcomes were reviewed by the nurse and a primary care physician. The Geriatric Care Model was implemented at a patient level, care professional level and organizational level by means of the older Adults: care in Transition (ACT) study . The intervention and its intended three-level implementation is described below.
Implementation at a patient level: geriatric assessment and care plan
Every six months, frail older people received an assessment of health and care and a tailored care plan. This procedure involved two home visits. During the first visit, practice nurses conducted a comprehensive geriatric assessment using the web-based Community Health Assessment version 9.1 of the Resident Assessment Instrument (RAI-CHA) . RAI-CHA aims to facilitate the identification of possible health and care needs, helps users standardize their routines and works as a reminder system for follow-up. After each assessment, practice nurses wrote a care plan in consultation with a primary care physician. Care plans were based on Client Analysis Protocols (CAPs) generated by the RAI-CHA instrument. RAI-CHA items trigger CAPs in several domains (e.g. physical wellbeing, social functioning, living and safety) and help users identify possible targets for care. Approximately two weeks after the first visit, practice nurses discussed the care plan with the older person in a follow up-visit. During this second visit, nurses provided information on guideline-concordant management and treatment options, and stimulated older people to get involved in the decision making process. If necessary, an evaluation consultation was scheduled after 3 months.
Implementation at a care professional level: quality management and education
Practice nurses were organized in care teams managed by geriatric expert teams. A geriatric expert team consisted of a geriatric nurse and an elderly care physician. In the Netherlands, an elderly care physician is specialized in the health issues of older people with complex or chronic disorders. The geriatric expert team guided and managed quality processes through one-on-one coaching and regular care team meetings. The care team meetings provided a platform for feedback, peer supervision and knowledge exchange between practice nurses. Training and education of practice nurses took place before as well as alongside the intervention. Before the intervention, nurses participated in a 3-day motivational interviewing course and received a one-day RAI-CHA training. Alongside the intervention, nurses received an extra ‘training on the job’ motivational interviewing training, and regularly received education about geriatric themes during care team gatherings.
Implementation at an organizational level: multidisciplinary team consultations and community networks
Multidisciplinary team consultations (MTCs) were organized with the aim to enhance the coordination between professionals caring for a single patient with complex needs. MTCs were attended by a practice nurse, a primary care physician, the geriatric expert team, a pharmacist and, if present, caregivers involved in the older person’s treatment or care. If relevant and feasible, older people’s partner or family members were invited to join the meeting. Older people were informed of MTC outcomes by the practice nurse. To build community networks, the geriatric expert teams organized meetings with local service providers and organizations. The aim of such meetings was to target the fragmentation of care delivery on a community level by facilitating coordination between professionals and encouraging optimal use of local resources.
The ACT study
The ACT study introduced the Geriatric Care Model to 1147 frail, older patients of 35 primary care practices in two regions of the Netherlands, using a stepped-wedge Randomised Controlled Trial (RCT) design . Frailty was established by a primary care physician using a composite definition of frailty (experiencing one or more limitations in either physical, psychological and/or social areas)  and a polypharmacy criterion (5 or more drugs prescribed in the last 3 months); frail older people who scored 3 or higher on the Program on Research for Integrating Services for the Maintenance of Autonomy case-finding tool for disability (PRISMA-7)  were eligible for trial entry. As per stepped-wedge RCT protocol, practices in both regions were randomly allocated to four groups (i.e. group 1, consisting of 10 practices; group 2, consisting of 9 practices; group 3 and 4, both consisting of 8 practices), that started working according to the Geriatric Care Model at different times during the trial period (at 0, 6, 12 and 18 months, respectively, see Figure 1). Practice nurses joined the care team (one care team per region) when the primary care practice they worked for started with the intervention. The ACT study received approval by the medical ethics committee of the VU University medical centre.
Procedures and outcomes
To operationalize implementation fidelity, we first identified ‘active’ components of the Geriatric Care Model (i.e. components of which optimal implementation is theorized as essential in order for the intervention to be successful). On a patient level, we considered the geriatric assessment and the care plans essential; on a care professional level, we considered the care team meetings essential; on organizational level, we considered the MTCs and community network meetings essential. Subsequently, we determined criteria for planned delivery of the essential intervention components. Depending on group allocation, it was planned that older people receive a geriatric assessment at least four (group 1), three (group 2), two (group 3) or one (group 4) times throughout the intervention period, with a maximum of 6 months between the starting time of the intervention and the first assessment and between two subsequent assessments. All geriatric assessments are carried out in the older person’s home using the web-based version of the RAI-instrument, and assessment questions are administered in the order in which they appear. Furthermore, it was planned that each assessment is followed by a care plan, which is discussed with the client in a follow-up home visit and left with the client. Care team meetings are organized regularly by geriatric expert teams, and are attended by all practice nurses. Each primary care practice organizes at least 2 MTCs every 6 months of the intervention, and MTC’s are attended by a practice nurse, a primary care physician, the geriatric expert team, a pharmacist and, if indicated, caregivers involved in the older person’s treatment or care. Community network meetings are organised throughout the intervention period.
Based on the above criteria for delivery, we formulated research questions for each subcategory of adherence based on Carroll’s framework for fidelity, as well as for the moderating factors ‘availability of facilitation strategies’ and ‘participant responsiveness’. In addition, we formulated research questions to investigate other factors that could have moderated adherence (see Table 1). Quantitative data were collected using three measurement protocols (Table 1 and Figure 1). Protocol 1 measured (1) coverage of assessment delivery, care plan delivery and MTC delivery, and (2) frequency of assessment delivery, care plan delivery, and MTC delivery. Protocol 2 measured (1) frequency of care team meetings and community network meetings, and (2) duration of assessment delivery, care plan delivery, and MTC delivery. Protocol 3 measured the moderating factors ‘availability of facilitation strategies’ and ‘participant responsiveness’. Qualitative data were collected alongside the intervention to assess the intervention’s content and other factors that could have moderated adherence.
Since the starting time of the intervention varied per study participant and primary care practice, we defined ‘start of the intervention’ on a participant and care professional level as the date an older person was invited for a first assessment, and ‘start of the intervention’ on a organisational level as the date the first patient of a practice was invited for a first assessment. The total intervention period was determined as follows: starting time plus 24 months (group 1); starting time plus 18 months (group 2), starting time plus 12 months (group 3); starting time plus 6 months (group 4).
Coverage of geriatric assessment delivery was measured per 6-month interval, and calculated by dividing the number of older people who received at least one assessment by the total number of older people intended to receive at least one assessment. Coverage of care plan delivery was measured per 6-month interval, and calculated by dividing the total number of older people who received a care plan after each assessment by the total number of assessments. Coverage of MTC delivery was measured per 6-month interval, and calculated by dividing the number of primary care practices that achieved the intended amount of MTCs by the total number of primary care practices that participated in the intervention. One MTC corresponded with one patient discussed. Since the number of care team meetings and community network meetings was not specified before the start of the intervention, we did not include these in the assessment of coverage.
Frequency of geriatric assessment delivery was measured per 6-month interval, and calculated by dividing the total number of assessment deliveries by the number of intended assessment deliveries. Frequency of care plan delivery was measured per 6-month interval, and calculated as the total number of care plan deliveries divided by the number of intended care plan deliveries. Frequency of MTC delivery was measured per 6-month interval, and calculated as the total number of MTCs that were organized by primary care practices divided by the intended number of MTC’s per primary care practice. Since no intended 6-month delivery was formulated at the start of the intervention, frequency of care team meetings and community network meetings was calculated as the total number of care team and network meetings that took place during the intervention period.
To determine duration of the intervention components, practice nurses (N = 14) used time tracking. Duration of assessment delivery was calculated as the average amount of time (in minutes) it took nurses to carry out one assessment, including pre-assessment and post-assessment activities. Duration of care plan delivery was calculated as the average amount of time (in minutes) it took for a practice nurse to write a care plan, discuss it with a primary care physician, and discuss it during a second follow-up home visit. Duration of MTCs was calculated as the average amount of time (in minutes) it took to discuss one person in an MTC. The duration of care team meetings and network meetings was not included in the assessment.
Moderating factors measurements
To assess the moderating factor availability of facilitation strategies, we determined how many educational sessions were organized by the geriatric expert team. In addition, we determined how many practice nurses attended the pre-intervention training sessions. To assess participant responsiveness, practice nurses and primary care physicians were asked to rate their satisfaction and involvement with the intervention on a scale of 1 to 10 (1 representing minimal satisfaction/involvement, 10 representing maximum satisfaction/involvement).
Qualitative data collection
To investigate content and find factors that moderated adherence, we used qualitative data collection. Semi-structured interviews with practice nurses (N = 15), geriatric expert team members (N = 3) and primary care physicians (N = 10) were conducted alongside the intervention. We included the following topics (per intervention component): order in which protocol activities are carried out; work practices, deviations from protocol, component duration, perceived quality and usefulness of the protocol, barriers and facilitators to carrying out the protocol as intended, self-perceived competence to carry out the protocol as intended, points of improvement. We took a cyclic approach: when a new topic emerged from an interview, it was added to the topic list of the next interview. All interviews were audio recorded and transcribed. To enhance the quality of the qualitative data, respondents were asked to approve a summary of the interview (member check). In addition, the researcher who carried out the interviews (MM) kept a research log in which she reflected on methodological decisions and her own role in the research process.
Quantitative data were analyzed with descriptive statistics using SPSS Statistics 20. Study drop out was accounted for on a patient and primary care practice level. Since adherence was calculated per 6-month interval, older people who had dropped out were excluded from analysis for the whole 6-month interval in which the drop-out occurred. For instance, a drop out at 16 months implied exclusion from analysis from 12 months onwards. One primary care practice in group 4 was excluded from analysis due to drop out prior to its designated starting time. We assessed coverage of assessment and care plan delivery on a practice nurse level to allow for an investigation of inter-nurse variance. We used Spearman’s rank correlation coefficient to investigate the correlation between participant responsiveness rates of practice nurses and family physicians and their average coverage rates of the key intervention components (geriatric assessment, care plan and MTC), using the criteria for the interpretation of the effect size proposed by Ferguson (weak, moderate, strong) .
Qualitative data were analyzed using a framework approach, a method of analysis in which some of the research questions are predetermined. Two researchers (MM and KL) analyzed the interviews separately and discussed their results to achieve intersubjective agreement. First, both researchers familiarized with the data. Second, they applied labels to the data and arranged the data according to broader themes.
Adherence to assessments and care plans
Coverage of geriatric assessment delivery in group 1, 2 and 3 was high in the first 6 months of the intervention (ranging from 83.2% to 91.3%), but declined in the following months. In group 4, first 6-month coverage was the lowest (79.8%). Table 2 shows coverage of geriatric assessment delivery per tranche and per 6-month interval. Average coverage rates between practice nurses varied: 13 nurses reached an average of 70-100%, and 4 nurses reached an average of 40-70%. Frequency of geriatric assessment delivery was overall the highest in group 1, with a peak in the first 6 months (103.0%). In all four groups, frequency never fell below 72% (see Table 3). It took practice nurses an average of 76 minutes to carry out the assessment. The average time spent on pre-assessment and post-assessment activities (e.g. administration) was 43 minutes. All nurses regularly or always used a paper version of the assessment instrument instead of the online version, and four nurses said that they did not always ask the RAI-CHA questions as intended. Two nurses indicated that when an older person visited their office for something not related to the intervention, assessments were sometimes carried out there instead of in the clients’ home.
In group 1, coverage of care plan delivery stayed between 99% and 94% in the first 18 months of the interventions but dropped to 78% in the last 6 months. In group 2 and 3, coverage stayed between 89% and 99% throughout the intervention period; group 4 showed the lowest coverage of care plan delivery (74.8%) (Table 2). Besides one nurse in group 4 whose coverage remained 0%, we found little variation between nurses: only 3 nurses reached an average of 60-80%, while the remaining nurses achieved averages up to 100%. In group 1-3, frequency of care plan delivery did not fall below 90% until the last 6 months of the intervention; in group 3, it never fell below 90%. Frequency of care plan delivery was lowest in group 4 (74.5%) (see Table 3). Nurses spent an average of 37 minutes writing a participant’s care plan and a little over half an hour discussing the care plan during the follow-up visit. Care plans and follow-up visits were not always carried out as intended. Some care plans did not include the intended information (i.e. they were incomplete), and some care plans did not get written at all. In addition, follow-up visits did not always take place. Seven nurses did not always leave the care plans with their clients.
Adherence to MTCs, care team meetings and community network meetings
In groups 1 to 3, MTC coverage started low but improved over time, with coverage in group 1 and group 3 reaching 50% in the last 6 months of the intervention (Table 2); in group 4, none of the primary care practices organized the MTCs as intended. Average coverage varied between individual practices: 13 achieved a coverage between 50-100%, 5 achieved a coverage of 0-50%, and 17 practices never carried out an MTC as intended. Group 1 contained almost 40% of the practices with a 50-100% coverage rate, whereas group 4 contained 40% of practices with a 0% coverage rate. Frequency of MTC delivery varied between and across the four groups. Generally, frequency increased over time. Practices in group 1 organized three times as much MTCs as practices in group 2 and 3. In group 4, only one MTC was organized during the designated intervention period. Preparing and carrying out MTCs took practice nurses almost 40 minutes per participant. MTCs were always attended by the primary care physician, the attending practice nurse, both members of the geriatric expert team and a pharmacist. Almost half of the MTCs were joined by other care professionals (such as a home care nurse or a physiotherapist) or the client’s partner or family members. Most MTCs involved discussing clients with complex care needs (17 meetings), severe pain (7 meetings), cognitive or mood problems (5 meetings), poor support networks (3 meetings) and rejection of care behaviour (2 meetings). In addition, some meetings were specifically aimed at coordinating care delivery (6 meetings) or revise medication (2 meetings).
Over the course of the intervention, the geriatric expert teams in the two regions organized a total of 45 team meetings. During the meetings, nurses discussed complex cases, consulted the geriatric expert team and received project updates. In addition, the geriatric expert teams organized 10 community network meetings. Network meetings were attended by professionals participating in the ACT study, local government representatives and care organizations active in the community.
Moderating factors: availability of facilitation strategies and participant responsiveness
Throughout the intervention period, geriatric expert team members organised 18 educational sessions, 9 of which featured short seminars by care professionals of regional care organizations aimed at educating practice nurses about available community resources. Session and seminar topics ranged from urinary incontinence and pain to nutrition and dementia. Of the 21 practice nurses, 20 nurses completed the 3-day training program ‘motivational interviewing’, and 17 nurses attended the RAI-CHA assessment training. All practice nurses said that they highly valued the motivational interview training, but did not always evaluate the RAI-CHA training as sufficiently educational or useful. Most nurses did not feel sufficiently confident about their assessment skills when they started with the intervention, and indicated that they would have preferred a more intensive assessment training schedule.
On average, practice nurses rated their involvement in the ACT study with a 7.1 (range 2-10) and their satisfaction with their work with a 6.5 (range 3-8). Associations between nurses’ satisfaction and involvement and their achieved coverage rates were weak and not significant. Primary care physicians rated their involvement with a 6.7 (range 2-9) and their satisfaction with a 6.3 (range 3-9). Association between physician involvement and satisfaction and their achieved MTC coverage rates were weak and moderate, respectively.
Other moderating factors
Results of qualitative investigation showed several factors that may have moderated adherence of the various intervention components.
Adherence to geriatric assessment delivery was influenced by factors at a participant, practice nurse and resource level. First, older people included in the ACT-study were not always available for an assessment due to long-term hospital stays, out-of-town holidays and episodes of illness or weakness. Second, nurses’ preference to carry out the assessment on paper instead of online was the result of limited network services, and since nurses generally preferred a natural conversation with a client over a structured interview, they often did not ask assessment questions in their intended order. Finally, the time it took to carry out an assessment was impacted by nurses’ own competencies (four nurses mentioned that their competencies increased over the course of the intervention, which they said reduced overall assessment time) and by the extent to which older people wished to share stories and experiences.
Adherence to care plan delivery was mostly influenced by factors at a patient and a practice nurse level. Some nurses indicated that whether they wrote a care plan and scheduled a follow-up visit depended on the amount of time pressure they experienced and on the assessment outcomes (i.e. they abstained from writing a care plan and following up when an assessment had yielded no health or care needs). One nurse in group 4 never adopted the care plan method, because she felt that the method was not tailored to her needs (for instance, it could not be integrated into a digital patient information system) and had developed her own way of communicating assessment outcomes with clients and colleagues. Nurses also mentioned several reasons why they did not always leave the care plan with their client as intended: one nurse believed the plan was redundant in the presence of existing medical documentation, others said that older people refused to accept the plan because it contained sensitive information, because it was of little or no value to them, or because they had no health or care needs.
Adherence to MTC delivery was mostly influenced by factors at a participant and primary care physician level. Results show that an MTC was cancelled twice due to the older person passing away unexpectedly or to a resolving of the issues that initially urged the MTC. Four primary care physicians indicated that they believed MTCs were redundant, too time consuming or inefficient. Three practice nurses mentioned that physicians diverted their efforts to organize an MTC, and geriatric team members experienced that it took a while before physicians got used to the idea of attending an MTC.
Finally, factors on a practice nurse and geriatric team level played a role in the adherence to care team meetings and community network meetings. Care team meetings were valued by nurses, who mentioned that they appreciated the opportunity for peer-to-peer practical information exchange about intervention procedures and tasks, such as working with the RAI-assessment instrument and care plans. We found that organizing community network meetings was challenging for geriatric teams for several reasons. One team member mentioned the challenge of identifying and locating network partners, another team member was unsure about her own role in the networking process. Moreover, it was pointed out that networking was hindered by the decentralization and fragmentation of target organisations and providers.
Using Carroll’s framework, we assessed implementation fidelity of the geriatric care model, a chronic care model for frail, older people who live at home.
Overall, we found that the adherence to geriatric assessment and care plan delivery was high: especially in the first 6 months after the start of the intervention, a large share of participants received the geriatric assessments and care plans as planned and as often as planned. However, over time, adherence of these two components decreased. This gradual decrease may be partly explained by ‘delayed delivery’ of the intervention, i.e. the number of months between two subsequent geriatric assessments exceeded the intended six. While practice nurses eventually delivered such ‘delayed assessments’, assessments increasingly fell beyond the intended time frame. As a consequence, we classified them as ‘not delivered as intended’. Delayed delivery likely occurred as a consequence of early start-up issues that impeded initial implementation.
In group 4, first 6-month coverage of geriatric assessment delivery and care plan delivery was lower than in the other three groups. In the case of assessment delivery, the cause of this difference remains unclear – we found no obvious inter-group variations in adherence or moderating factors on a professional, patient or practice level. In the case of care plan delivery, the difference could be explained by the fact that one nurse in group 4 never adopted the care plan method, a deviation that may be associated with a lack of involvement in the intervention: the same nurse never attended a care team meeting, and infrequently responded to emails or phone calls.
As opposed to the geriatric assessments and care plans, overall first 6-month adherence to MTC delivery was low but increased over time. Our data suggest that the longer a practice was exposed to the intervention, the higher the overall coverage. This finding may be explained by the fact that in the Netherlands, structural multidisciplinary encounters are a relatively new phenomenon in primary care, and at the time of the intervention few examples existed as to how to integrate MTCs into daily practice. In addition, organising and carrying out an MTC is complex, a component characteristic known to be associated with lower fidelity .
Care team meetings were organized as intended and highly valued by the participants. Since most of the time nurses performed their intervention activities alone, they appreciated the interactive nature of the meetings, and the opportunity to give and receive direct peer feedback. Community network meetings were more challenging to implement. Geriatric expert team members experienced barriers in their efforts to build and sustain professional networks. These challenges may be partly explained by Rogers’ ‘Diffusion of Innovation’ theory, that suggests that individuals who are more interpersonally connected within an organisational system are more likely to adopt an innovation . Expert team members were motivated to set up networks because they were in the core group of Geriatric Care Model implementers, whereas potential network partners were on the group periphery and thus less likely to change their routines. In addition, implementation of network meetings may have been challenging because geriatric expert team members had limited previous networking experience. Finally, the decentralisation issues that urged the meetings may also have hindered the meetings’ actualisation in practice.
Our results show that factors at a participant, care professional, organizational and resource level may have moderated adherence to the various intervention components. We found that there were large variations in the extent to which nurses and physicians felt satisfied and involved with the intervention. These variations were most likely determined by individual factors: previous research suggests that nurse job satisfaction is often influenced by work attitude, job stress and nurse-physician interaction, among others [38,39], and that physician job satisfaction can be influenced by factors such as personality, workload and recognition [40,41]. Involvement and satisfaction of both nurses and physicians was only weakly or moderately associated with coverage of geriatric assessment delivery and care plan delivery, which suggests that there was little influence of participant responsiveness on nurses’ or physicians’ individual achievements. Furthermore, the pre-intervention RAI-CHA training program did not always succeed in giving practice nurses the knowledge and skills to implement the assessment as intended, which may have affected the adherence subcategories duration and content. Future implementors of the Geriatric Care Model should secure that training programs are tailored to the learning needs of the users.
Measuring CCM implementation fidelity
Research papers that report outcomes of the implementation of CCM-based interventions in randomised controlled trials do not always include a thorough investigation of the degree of implementation. However, researchers who do undertake such an investigation typically opt to develop their own conceptual and methodological framework due to a lack of standardized methods. For instance, when Pearson et al. assessed CCM implementation of 42 organisations, they distinguished between fidelity and intensity, defining fidelity in terms of alignment of observed change activities with the various CCM elements and intensity in terms of quantity and depth . Haggstrom et al. used surveys based on qualitative interviews to assess the implementation of the CCM among community health centers , while Hroscikoski et al. used a qualitative, comparative case study design to evaluate CCM implementation in a large health care organisation . The differences in operationalization and methodology hinders between-program comparisons of implementation parameters, limiting our understanding of what aspects of a program’s content and context contribute to its potential to achieve a high degree of implementation. We therefore recommend researchers who aim to assess the implementation of a CCM-based program to chose an existing framework, such as Carroll’s, and adjust it according to the program’s particularities.
Several study aspects may have limited our insight in the implementation fidelity of the Geriatric Care Model. First, there were limitations to our investigation of adherence. Time constraints prevented practice nurses from consistently tracking the duration of their activities, which restricted the collection of longitudinal data on intervention duration. The intended delivery of care team meetings and community network meetings was unspecified, which caused us to measure adherence to these components only at the end of the intervention period. Second, due to the complexity of the intervention, we investigated only two of the six factors that moderate fidelity proposed by Hasson et al. . Investigation of each of the moderating factors was subject to limitations. Participant responsiveness was measured only at the end of the intervention and using a single scale. Collecting more data on responsiveness, for instance through qualitative interviews, could have deepened our understanding of the way responsiveness influenced adherence. A longitudinal investigation of participant responsiveness could have given us insight in the way in which responsiveness developed throughout the intervention; in addition, it could have enhanced our understanding of the influence of responsiveness on adherence over time. Our understanding of the moderating influence of facilitation strategies would have benefitted from an investigation of attendance at the education sessions; however, data collection was impaired due to incomplete records and attendance sheets.
Nonadherence in real world settings
It is recognized that interventions cannot always be fully implemented as intended in the real world [20,45]. This is even more true for multicomponent interventions, whose complex nature increases the scope for variations in delivery . Some authors argue therefore that, in order to accommodate successful implementation in non-experimental settings, it is necessary to tailor program content to local conditions. Others share this opinion, but state that low fidelity of ‘active’ intervention components (i.e. key program elements considered essential to intervention success) can result in interventions not achieving the intended effects . Although the professionals who carried out the Geriatric Care Model performed their activities within the framework of a protocol, they were simultaneously encouraged to adjust their activities to the local primary care context. By making these adjustments, professionals may have deviated from the initial intervention protocol. Since this protocol served as a template for the fidelity assessment, deliberate deviations may have consequentially been interpreted as nonadherence (i.e. such activities may have been characterized as ‘not implemented as intended’). Hypothetically, we are therefore confronted with a paradox: while it is likely that poor adherence or nonadherence contributed to a lower overall estimation of implementation fidelity, it could at the same time have improved actual implementation of the geriatric care model in the real world. This paradox eventually complicates the use of outcomes of fidelity assessment to explain findings of effectiveness evaluation.
This study is the first to investigate implementation fidelity of the Geriatric Care Model, a comprehensive care model for frail, older people in a primary care setting based on the Chronic Care Model. We found that, despite variations in nurses’ level of participant responsiveness, the adherence to geriatric assessment delivery and care plan delivery was high, and that adherence to multidisciplinary team meetings was initially low but increased over time. In addition, we found that the level of adherence varied between professionals, which most likely can be attributed to professional’s individual characteristics and circumstances.
We demonstrated that a longitudinal investigation of adherence can contribute to our understanding of the ways in which adherence develops over time, and therefore recommend that researchers who aim to investigate fidelity of complex interventions measure all essential intervention components and moderating factors at multiple times during the intervention period. Researchers who carry out such an investigation should keep in mind that factors on a professional level, such as time restraints and lack of involvement, could complicate data collection and subsequently influence results. In addition, researchers should be aware of the fact that the real life setting increases the likelihood that professionals use nonadherence as a means to tailor the intervention to local conditions, which may challenge their ability to relate outcomes of fidelity assessments to conclusions about program feasibility or effectiveness.
Fried LP, Tangen CM, Walston J, Newman AB, Hirsch C, Gottdiener J, et al. Frailty in older adults: evidence for a phenotype. J GerontolA Biol Sci Med Sci. 2001;56:M146–56.
Rockwood K, Song X, MacKnight C, Bergman H, Hogan DB, McDowell I, et al. A global clinical measure of fitness and frailty in elderly people. CMAJ. 2005;173:489–95.
Schuurmans H, Steverink N, Lindenberg S, Frieswijk N, Slaets JPJ. Old or frail: what tells us more? J GerontolA Biol Sci Med Sci. 2004;59:M962–5.
Santos-Eggimann B, Cuenoud P, Spagnoli J, Junod J. Prevalence of frailty in middle-aged and older community-dwelling Europeans living in 10 countries. J GerontolA Biol Sci Med Sci. 2009;64:675–81.
Gillsjo C, Schwartz-Barcott D, von Post I. Home: the place the older adult cannot imagine living without. BMC Geriatr. 2011;11:10.
Wiles JL, Leibing A, Guberman N, Reeve J, Allen RES. The meaning of “aging in place” to older people. Gerontologist. 2012;52:357–66.
De Lepeleire J, Iliffe S, Mann E, Degryse JM. Frailty: an emerging concept for general practice. Br J Gen Pract. 2009;59:e177–82.
Lacas A, Rockwood K. Frailty in primary care: a review of its conceptualization and implications for practice. BMC Med. 2012;10:4.
European Commission. Synthesis report on the public consultation on the European Innovation Partnership on Active and Healthy Ageing. 2011. European Commission - DG Health and Consumers and DG Information Society and Media. Ref Type: Report
Bodenheimer T, Wagner EH, Grumbach K. Improving primary care for patients with chronic illness. JAMA. 2002;288:1775–9.
Thota AB, Sipe TA, Byard GJ, Zometa CS, Hahn RA, Knight-Eily LR, et al. Collaborative care to improve the management of depressive disorders: a community guide systematic review and meta-analysis. Am J Prev Med. 2012;42:525–38.
Drewes HW, Steuten LMG, Lemmens LC, Baan CA, Boshuizen HC, Elissen AMJ, et al. The effectiveness of chronic care management for heart failure: meta-regression analyses to explain the heterogeneity in outcomes. Health Serv Res. 2012;47:1926–59.
Sing D, Ham C. Improving Care for People with Long-Term Conditions: A Review of UK and International Frameworks. Birmingham: Health Services Management Centre, University of Birmingham, and NHS Institute for Innovation and Improvement. Ref Type: Report; 2006.
Spoorenberg SLW, Uittenbroek RJ, Middel B, Kremer BPH, Reijneveld SA, Wynia K. Embrace, a model for integrated elderly care: study protocol of a randomized controlled trial on the effectiveness regarding patient outcomes, service use, costs, and quality of care. BMC Geriatr. 2013;13:62.
Pasricha A, Deinstadt RTM, Moher D, Killoran A, Rourke SB, Kendall CE. Chronic care model decision support and clinical information systems interventions for people living with HIV: a systematic review. J Gen Intern Med. 2013;28:127–35.
Adams SG, Smith PK, Allan PF, Anzueto A, Pugh JA, Cornell JE. Systematic review of the chronic care model in chronic obstructive pulmonary disease prevention and management. Arch Intern Med. 2007;167:551–61.
Coleman K, Austin BT, Brach C, Wagner EH. Evidence on the chronic care model in the new millennium. Health Aff (Millwood). 2009;28:75–85.
Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82:581–629.
Campbell M, Fitzpatrick R, Haines A, Kinmonth AL, Sandercock P, Spiegelhalter D, et al. Framework for design and evaluation of complex interventions to improve health. BMJ. 2000;321:694–6.
Carroll C, Patterson M, Wood S, Booth A, Rick J, Balain S. A conceptual framework for implementation fidelity. Implement Sci. 2007;2:40.
Hulscher MEJL, Laurant MGH, Grol RPTM. Process evaluation on quality improvement interventions. Qual Saf Health Care. 2003;12:40–6.
Dusenbury L, Brannigan R, Falco M, Hansen WB. A review of research on fidelity of implementation: implications for drug abuse prevention in school settings. Health Educ Res. 2003;18:237–56.
Dane AV, Schneider BH. Program integrity in primary and early secondary prevention: are implementation effects out of control? Clin Psychol Rev. 1998;18:23–45.
Durlak JA, DuPre EP. Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation. Am J Community Psychol. 2008;41:327–50.
Elliott DS, Mihalic S. Issues in disseminating and replicating effective prevention programs. Prev Sci. 2004;5:47–53.
Mihalic S. The importance of implementation fidelity. Emotional Behav Disord Youth. 2004;4:99–105.
Dobson D, Cook TJ. Avoiding type III error in program evaluation: results from a field experiment. Eval Program Plann. 1980;3:269–76.
Grant A, Treweek S, Dreischulte T, Foy R, Guthrie B. Process evaluations for cluster-randomised trials of complex interventions: a proposed framework for design and reporting. Trials. 2013;14:15.
Oakley A, Strange V, Bonell C, Allen E, Stephenson J. Process evaluation in randomised controlled trials of complex interventions. BMJ. 2006;332:413–6.
Boutron I, Moher D, Altman DG, Schulz KF, Ravaud P. Extending the CONSORT statement to randomized trials of nonpharmacologic treatment: explanation and elaboration. Ann Intern Med. 2008;148:295–309.
de Vos AJBM, Bakker TJ, de Vreede PL, van Wijngaarden JDH, Steyerberg EW, Mackenbach JP, et al. The prevention and reactivation care program: intervention fidelity matters. BMC Health Serv Res. 2013;13:29.
Hasson H. Systematic evaluation of implementation fidelity of complex interventions in health and social care. Implement Sci. 2010;5:67.
Muntinga M, Hoogendijk E, van Leeuwen K, van Hout H, Twisk J, van der Horst H, et al. Implementing the chronic care model for frail older adults in the Netherlands: study protocol of ACT (frail older Adults: Care in Transition). BMC Geriatr. 2012;12:19.
Hirdes JP, Fries BE, Morris JN, Steel K, Mor V, Frijters D, et al. Integrated health information systems based on the RAI/MDS series of instruments. Healthc Manage Forum. 1999;12:30–40.
Raiche M, Hebert R, Dubois MF. PRISMA-7: a case-finding tool to identify older adults with moderate to severe disabilities. Arch Gerontol Geriatr. 2008;47:9–18.
Ferguson CJ. An effect size primer: a guide for clinicians and researchers. Prof Psychol: Res Pract. 2009;40:532.
Rogers E. Diffusion of Innovation. 5th edition. 5th ed. New York: Free Press; 2003.
Zangaro GA, Soeken KL. A meta−analysis of studies of nurses’ job satisfaction. Res Nurs Health. 2007;30:445–58.
Kovner C, Brewer C, Wu Y, Cheng Y, Suzuki M. Factors associated with work satisfaction of registered nurses. J Nurs Scholarsh. 2006;38:71–9.
Judge TA, Heller D, Mount MK. Five-factor model of personality and job satisfaction: a meta-analysis. J Appl Psychol. 2002;87:530.
Van Ham I, Verhoeven AAH, Groenier KH, Groothoff JW, De Haan J. Job satisfaction among general practitioners: a systematic literature review. Eur J Gen Pract. 2006;12:174–80.
Pearson ML, Wu S, Schaefer J, Bonomi AE, Shortell SM, Mendel PJ, et al. Assessing the implementation of the chronic care model in quality improvement collaboratives. Health Serv Res. 2005;40:978–96.
Haggstrom DA, Taplin SH, Monahan P, Clauser S. Chronic care model implementation for cancer screening and follow-up in community health centers. J Health Care Poor Underserved. 2012;23:49–66.
Hroscikoski MC, Solberg LI, Sperl-Hillen JM, Harper PG, McGrail MP, Crabtree BF. Challenges of change: a qualitative study of chronic care model implementation. Ann Fam Med. 2006;4:317–26.
Blakely CH, Mayer JP, Gottschalk RG, Schmitt N, Davidson WS, Roitman DB, et al. The fidelity-adaptation debate: implications for the implementation of public sector social programs. Am J Community Psychol. 1987;15:253–68.
We would like to acknowledge the organizations and professionals involved in ‘Ouderennnet VUmc en partners’ and all health care professionals who contributed to the ACT study. We owe many thanks all practice nurses, primary care physicians and members of the geriatric expert teams who facilitated data collection for this research project. The ACT-study is funded by The Netherlands Organization for Health Research and Development (ZonMw): Dutch National Care for the Elderly Program grant number 311080201.
The authors declare that they have no competing interests.
MM, KvL and AJ drafted the manuscript, and MM wrote the manuscript. AJ and GN supervised the writing process. All authors critically revised the manuscript for important intellectual content. All authors approved the final manuscript.
About this article
Cite this article
Muntinga, M.E., Van Leeuwen, K.M., Schellevis, F.G. et al. From concept to content: assessing the implementation fidelity of a chronic care model for frail, older people who live at home. BMC Health Serv Res 15, 18 (2015). https://doi.org/10.1186/s12913-014-0662-6