Sustainability and scalability of a volunteer-based primary care intervention (Health TAPESTRY): a mixed-methods analysis
BMC Health Services Research volume 17, Article number: 514 (2017)
Chronic diseases are a significant public health concern, particularly in older adults. To address the delivery of health care services to optimally meet the needs of older adults with multiple chronic diseases, Health TAPESTRY (Teams Advancing Patient Experience: Strengthening Quality) uses a novel approach that involves patient home visits by trained volunteers to collect and transmit relevant health information using e-health technology to inform appropriate care from an inter-professional healthcare team. Health TAPESTRY was implemented, pilot tested, and evaluated in a randomized controlled trial (analysis underway). Knowledge translation (KT) interventions such as Health TAPESTRY should involve an investigation of their sustainability and scalability determinants to inform further implementation. However, this is seldom considered in research or considered early enough, so the objectives of this study were to assess the sustainability and scalability potential of Health TAPESTRY from the perspective of the team who developed and pilot-tested it.
Our objectives were addressed using a sequential mixed-methods approach involving the administration of a validated, sustainability survey developed by the National Health Service (NHS) to all members of the Health TAPESTRY team who were actively involved in the development, implementation and pilot evaluation of the intervention (Phase 1: n = 38). Mean sustainability scores were calculated to identify the best potential for improvement across sustainability factors. Phase 2 was a qualitative study of interviews with purposively selected Health TAPESTRY team members to gain a more in-depth understanding of the factors that influence the sustainability and scalability Health TAPESTRY. Two independent reviewers coded transcribed interviews and completed a multi-step thematic analysis. Outcomes were participant perceptions of the determinants influencing the sustainability and scalability of Health TAPESTRY.
Twenty Health TAPESTRY team members (53% response rate) completed the NHS sustainability survey. The overall mean sustainability score was 64.6 (range 22.8–96.8). Important opportunities for improving sustainability were better staff involvement and training, clinical leadership engagement, and infrastructure for sustainability. Interviews with 25 participants (response rate 60%) showed that factors influencing the sustainability and scalability of Health TAPESTRY emerged across two dimensions: I) Health TAPESTRY operations (development and implementation activities undertaken by the central team); and II) the Health TAPESTRY intervention (factors specific to the intervention and its elements). Resource capacity appears to be an important factor to consider for Health TAPESTRY operations as it was identified across both sustainability and scalability factors; and perceived lack of interprofessional team and volunteer resource capacity and the need for stakeholder buy-in are important considerations for the Health TAPESTRY intervention. We used these findings to create actionable recommendations to initiate dialogue among Health TAPESTRY team members to improve the intervention.
Our study identified sustainability and scalability determinants of the Health TAPESTRY intervention that can be used to optimize its potential for impact. Next steps will involve using findings to inform a guide to facilitate sustainability and scalability of Health TAPESTRY in other jurisdictions considering its adoption. Our findings build on the limited current knowledge of sustainability, and advances KT science related to the sustainability and scalability of KT interventions.
Chronic diseases are a significant public health concern [1,2,3], and were reported by the World Health Organization as the leading cause of death worldwide . Multiple chronic conditions are more prevalent among adults aged 65 years and older, who are not only the fastest growing proportion of our population, but also amongst the highest users of the health care system [4, 5]. Aging is an expensive process, as 10% of seniors who have the most complex health needs account for 60% of the total annual health care spending in many provinces in Canada . By 2030, the increasing number of seniors is projected to cost the Canadian healthcare system $24 billion more annually (50% more than today) . Given these projections, we need to address the delivery of health care services to optimally meet the needs of this population.
Health TAPESTRY (Teams Advancing Patient Experience: Strengthening Quality) was created in response to these challenges and to optimize health care delivery . It uses a novel approach that integrates the involvement of trained volunteers who visit individuals (clients) in their homes and the use e-health technologies via touch screen tablets to identify clients’ health goals and chronic disease risks. Studies suggest that the use of volunteers in health interventions have potential for impact [8,9,10,11,12,13], particularly because this strategy can respond to the need for low cost, sustainable interventions to address the increasing burden of chronic disease, to promote community engagement, and to address social isolation of older adults. In Health TAPESTRY, data gathered by volunteers during the client visits are transmitted to inform appropriate care from an inter-professional healthcare team consisting of physicians, nurses, system navigators and other health care professionals. As such, it is considered a KT intervention (i.e., it facilitates the uptake of knowledge). Health TAPESTRY delivers the overall intervention with an innovative technology-based system integrated with the McMaster family practice electronic KindredPHR© Personal Health Record (PHR) system. Trained volunteer pairs collect health information over several home visits. The health care team uses the report to design an action plan to address identified client goals and health risks. For example, this may involve a wide range of activities by different health care providers such as optimized diabetes management, medication review, promoting clients’ access to health and community-based programs and services with the help of healthcare navigators, and engaging the volunteers in follow up activities or encouraging clients to use the PHR for chronic disease management or communication with the clinic.
The Health TAPESTRY intervention was conceptualized and developed using an integrated knowledge translation (KT) approach . The approach included a process whereby a wide scope of relevant knowledge users including patients, volunteers, health care providers, leaders of community organizations and experts and stakeholders in geriatric medicine, nursing, allied health, volunteer training, knowledge translation, health services research methods, and information technology worked as a team to create and implement Health TAPESTRY. More specifically its development considered: (1) inclusion of components supported by literature on evidence of effectiveness; (2) theoretical models such as the Chronic Care Model, Behaviour Change, Diffusion of Innovation; (3) developmental evaluation; (4) participatory co-development; (5) formal investigation of sustainability and scalability; (6) iterative pilot testing including evaluation of implementation of the intervention; (7) a pragmatic randomized controlled trial to evaluate the effectiveness and cost effectiveness in older adults; and (8) implementation of an adapted approach in other sites/contexts . Health TAPESTRY was implemented first in an interprofessional primary care practice in Hamilton, Ontario, Canada and pilot-tested using an explanatory mixed methods design to explore its feasibility (e.g., the potential of pairing volunteers, client visits, automation of data transfer from tablets to electronic medical records) ; this study involved 11 volunteers, and 11 patients of four family physicians. Findings were used to modify the intervention after which it was evaluated in a large randomized controlled trial (RCT) carried out between 2014 and 2016 with 360 older adults (analysis underway) . We also conducted other smaller feasibility studies to test the potential of Health TAPESTRY in more targeted populations (i.e., people with diabetes and hypertension, high health system users), and utilizing different groups of volunteers (i.e., nursing and medical students who participated as part of a formal for-credit university-level course).
Sustainability is an important implementation outcome and a necessary consideration for scaling up KT interventions such as Health TAPESTRY. In fact, it would be inappropriate to scale-up an intervention that is not sustainable. Identifying the determinants of sustainability is therefore needed to facilitate appropriate scale-up of interventions to ensure that the knowledge that is generated by them can inform optimized decision making for their target end users, and to achieve patient well being and positive health outcomes. Scalability or large-scale adoption is defined as “deliberate effors to increase the impact of health service innovations successfully tested in pilot or experimental projects so as to benefit more people” . It is the vertical diffusion or deliberate, systematic approach to increasing the coverage, range and sustainability of services . Sustainability is commonly defined as: “the degree to which an innovation continues to be used after initial efforts to secure adoption is completed”  or becomes a routine part of care delivery and continues to deliver outcomes [20,21,22]. It has also been defined as an intervention that is in place for more than 1 year after implementation or after the research or project funding period is complete . Sustainability is an important implementation outcome, yet it is seldom considered in research plans or considered early enough in the development process of interventions to be able to address its potential challenges and barriers [24, 25]. Additionally, little research has been done to investigate long-term sustainability of KT interventions [26,27,28,29]. Intervention design is often focused on short-term outputs and rarely addresses long-term outcomes in practice and policy domains [24, 25]. This can lead to worse patient outcomes and quality of care [30,31,32], implementation failure and wasted resources [33,34,35,36].
The overall objectives of the current study were to assess the sustainability and scalability potential of the Health TAPESTRY intervention near the beginning of intervention development (prior to the planned RCT) from the perspective of the team who were involved in its development and pilot testing. Specific objectives were to: identify the specific determinants that influence the implementation, sustainability and scale-up of Health TAPESTRY across Canada, and to create a guidance document (addressing barriers) to enable the team to optimize its potential for impact.
Our objectives were addressed using a sequential mixed-methods approach conducted in two phases. First, we administered a validated sustainability questionnaire developed by the National Health Service (NHS) Institute (PHASE 1)  followed by a qualitative study of one-on-one telephone interviews (PHASE 2). Both phases involved recruiting members of the Health TAPESTRY team (i.e., affiliated researchers, program staff, volunteer organizations) and clinic providers and staff of the McMaster Family Health Team (FHT) in Hamilton, Ontario; volunteers were involved in PHASE 2 only. Research and ethics approval was obtained from the Hamilton Integrated Research Ethics Board as part of the Health TAPESTRY program development and evaluation at McMaster University in Hamilton, Ontario, Canada.
PHASE 1 – Administration of the NHS sustainability survey
The NHS sustainability model and survey was designed to help teams become aware of potential challenges to the sustainability of a new practice change (such as from an implemented intervention). In the context of their environment, teams affected by the change can determine (through self-assessment using the NHS survey), whether the new practice or change is likely to be sustained, and to prompt timely action to facilitate sustainability [22, 37]. The NHS model and survey was developed by the NHS Institute for Innovation and Improvement Program through review of management literature and input from their improvement experts, senior administrative and clinical leaders to identify over 100 factors considered important for sustaining change . These factors were ranked in a series of focus groups involving 250 NHS staff and health care experts, which led to 10 sustainability factors relating to three domains that are important in sustaining change in healthcare (process, staff, organization) : Process explores the benefits of the “change” beyond helping patients (e.g., does the change reduce waste or avoid duplication?), the credibility of the benefits (e.g., do staff believe in the benefits?), adaptability of the improved process (e.g., does the change continue to meet ongoing needs effectively?), and the effectiveness of the system to monitor progress (e.g., is there a feedback system to reinforce benefits and progress and initiate new or further action?); Staff explores the involvement and training of staff to sustain the process (e.g., do staff play a rpart in the innovation, design and implementation of the change?), staff behaviours toward sustaining the change (e.g., are staff encouraged and able to express their ideas regularly and is their input taken into consideration?), senior leadership and clinical leadership engagement and support (e.g., are the senior/clinic leaders trusted, influential, respected and believable? Are they involved in the initiative, do they understand it and promote it?); and Organization explores the fit with the organization’s strategic aims and culture (e.g., are the goal clearly contributing to the overall organizationa strategic aims?); and infrastructure (e.g., are there policies and procedures supporting the new way of working?). The resulting survey is a self-assessment tool that is designed for teams to identify strengths and weaknesses of the implementation plan and to predict the likelihood of sustainability . Respondents are asked to select from one of 4 response levels for each of the 10 sustainability factors (across the 3 domains). The different levels represent what the respondent perceives as the best fit with their current situation: the highest level represents the most favourable sustainability perspectives, the lowest level represents the least favourable perspective, and the remaining two levels are in the middle. The survey accompanies a sustainability guide, which offers practical advice on achieving sustained use of the initiative by providing the means for teams to address identified challenges and prompt discussion and action to address them, particularly across factors with maximum potential for improvement (i.e., largest difference between identified score and maximum potential sustainability score). The process is aimed at raising early awareness of sustainability challenges, and the opportunity for teams to iteratively address these challenges to optimize the new initiative’s potential for impact .
Population and recruitment
We used a non-probability sampling strategy whereby the NHS survey was administered in May 2014 to all members of the Health TAPESTRY team who were actively involved in the development, implementation and pilot evaluation of the intervention (n = 38). The team comprised the following stakeholder groups: Scientific leads (n = 11): Research staff (n = 7), Program staff (n = 5) Clinic staff (n = 12) and, Volunteer organization (n = 3) (Additional file 1). An email invitation was sent to Health TAPESTRY team members with the purpose of the survey, and instructions on how to complete it. We excluded recruitment of volunteers because they were not directly involved in the development of Health TAPESTRY. To enhance response rates, we used Dillman’s strategy (3 follow-up reminders approximately two weeks apart) .
Data collection and analysis
Each member of the team independently completed the sustainability survey on an electronic worksheet developed by NHS (Excel file). The survey represents a diagnostic scoring system designed to bring about the “conversation” around potential barriers using guidance for sustainability . A sustainability score of >55 represents a “reason to be optimistic” (i.e., above the threshold of what is considered a potentially sustainable intervention); and a score of <45 suggests that some action needs to be taken to “increase the likelihood that the improvement initiative will sustain” . Analysis involved a quantitative assessment of the mean scores generated by each respondent’s survey. We aggregated individual mean scores to generate an overall mean team score and corresponding bar graph to highlight scores across each of the 3 sustainability domains and its subdomains. We calculated the largest difference between actual scores and maximum potential score (a pre-specified maximum potential score built into the NHS survey scoring system) to identify best potential for improvement across sustainability factors. Aggregated team results were shared amongst the Health TAPESTRY team during a Scientific Leads meeting in June 2014 to discuss the sustainability factors identified as having the most potential for improvement (i.e., greatest difference between sustainability score and maximum potential score).
Outcomes and outputs
Primary outcomes were mean team sustainability score and factors across three domains with the greatest potential for improvement. Outputs were a description of the factors influencing sustainability of the intervention and consensus-based suggestions by the Health TAPESTRY team for addressing identified sustainability barriers.
PHASE 2: Qualitative telephone interviews
To gain a more in-depth understanding of the factors that influence the implementation, sustainability and scalability potential of the Health TAPESTRY intervention, we conducted semi-structured telephone interviews with selected members of the Health TAPESTRY team between July and September 2014.
Population and recruitment
We used a purposive sampling strategy to recruit 3–5 Health TAPESTRY team members from stakeholder groups that participated in PHASE 1 survey (n = 42): Scientific leads (n = 11); Research staff (n = 7): Program staff (n = 5): Clinic staff (n = 12): and Volunteer organization (n = 3). We also included representation from the Volunteers providing direct service to clients (n = 4). Recruitment involved sending an email invitation describing the purpose of the qualitative interviews to team members. To increase our response rates, we also added a question at the conclusion of the NHS survey (PHASE 1) to invite respondents to participate in qualitative interviews.
Interview guide development
We developed an interview guide using findings from the NHS survey as well as the Theoretical Domains Framework , which is a useful approach to inform questions that best elicit responses on behaviour change domains. Questions were grouped into three categories: participants’ role in Health TAPESTRY, their general perceptions of the determinants of its sustainability and scalability (barriers and facilitators), and suggestions for overcoming potential barriers. Two experienced moderators conducted 30–60-min interviews (Additional file 2).
Interviews were audio recorded and transcribed verbatim. Transcripts were imported into qualitative analysis software (i.e. NVivo 10.0), which was used by two independent reviewers to conduct a multi-step thematic analysis . This involved the reviewers independently reading transcripts and creating an initial list of codes. The reviewers developed a codebook through consensus-based discussions; the codebook was further refined and pilot tested using two interview transcripts. Inter-rater agreement between reviewers was calculated using the Kappa coefficient in NVivo. After each discussion about a discrepancy, one researcher made all appropriate changes to the coding file until a Kappa coefficient of ≥0.8 was achieved for each individual code/transcript combination. The transcript data were divided into five groups, which were coded in sequential rounds using the steps described above. After the first two rounds of coding (n = 10 transcripts), the number of discrepancies between the two reviewers decreased substantially, so the remaining three rounds of coding (n = 15 transcripts) involved a modified coding approach where one reviewer coded all transcripts and the second coder verified coding on one randomly selected audit transcript per round. Inter-rater reliability was computed for the audit transcript and if any discrepancies arose, the reviewers discussed and resolved them until the Kappa coefficient of ≥0.8 was achieved. Once all data were coded, a series of team meetings was held to discuss the main themes of the data.
Outcomes and outputs
Perceptions of stakeholders of the determinants influencing the sustainability and scalability of the Health TAPESTRY intervention. We generated a table of identified barriers and facilitators according to identified themes and mapped the recommendations (as informed by respondents) to address barriers. This output was aimed to help the team make informed decisions about modifying the Health TAPESTRY intervention to maximize its potential for impact.
Integration of quantitative and qualitative results
To facilitate the integration of data from PHASE 1 quantitative survey with PHASE 2 qualitative interviews, a between-method-data triangulation technique entitled the meta-matrix, was applied . The meta-matrix technique allows for secondary-level triangulation, whereby quantitative and qualitative data are analysed individually, followed by plotting commonalities between them into a matrix. This allows a strong visual anchor to recognize patterns in the data and to recognise confirming/corroborating, elaborating/expanding and identifying contradicting/discrepant views between data types . This technique was employed to allow for a more complete understanding of the sustainability and scalability potential of the Health TAPESTRY intervention. For example, we mapped NHS sustainability domains/sub-domains to the factors influencing sustainability that emerged from qualitative findings.
Table 1 shows the demographic characteristics of the NHS survey and qualitative interview participants. Of 38 and 42 Health TAPESTRY team members who were invited to participate in PHASE 1 and PHASE 2, respectively, 20 individuals completed the NHS survey (53% response rate) and 25 individuals completed the qualitative interviews (60% response rate). Participation across the 2 studies was representative of the wide range of stakeholder groups involved in the development and pilot testing of Health TAPESTRY (Table 1).
PHASE 1: NHS sustainability survey
Figure 1 shows the mean sustainability score for each of the three sustainability domains of the NHS model (process, staff and organization). The total mean sustainability score was 64.6 (range 22.8 to 96.9), and 70% of survey participants reported a score ≥ 55 (i.e., above the threshold of what is considered a potentially sustainable intervention and “reason to be optimistic”). Figure 2 shows the mean scores for each of the 10 sustainability factors highlighting the difference between team mean score and maximum potential score. The best opportunities for improvement were in the Staff domain pertaining to the factors: “staff involvement and training to sustain the process” and “clinical leadership engagement”; and in the Organization domain pertaining to the factor: “infrastructure for sustainability”.
PHASE 2: Qualitative interviews
Sustainability and scalability determinants of Health TAPESTRY emerged across two major dimensions: I) Health TAPESTRY operations, and II) Health TAPESTRY intervention. We defined “Health TAPESTRY operations” as any factor related to activities undertaken by the central Health TAPESTRY developer team (scientific leads, research staff, and program staff) to support the development and implementation of Health TAPESTRY, and the processes by which the program and its team members accomplished this. Examples of activities were developing technological components of the intervention, conducting research studies to understand and evaluate Health TAPESTRY, and coordinating the communication between members of the developer team so they work as a cohesive unit. We defined “Health TAPESTRY intervention” as any factor associated with the intervention and its specific elements (i.e., patient home visits by volunteers to collect information on health goals and chronic disease risks, automated transmission of data to primary care inter-professional team using tablet devices, the Health TAPESTRY report outlining an action plan in response to health goals and risks, and patient access to the PHR). Tables 2 and 3 summarizes the sustainability and scalability determinants of Health TAPESTRY across these two dimensions (operations and intervention) matched with actionable recommendations as suggested by interview informants. We also mapped the NHS sustainability model domains and related factor level questions to identified determinants that the Health TAPESTRY team can consider as a starting point to address identified challenges. A more detailed description of our data is in Additional files 3 and 4.
I. Health TAPESTRY operations
Table 2 summarizes the determinants across Health TAPESTRY operations. We identified four themes related to sustainability, and five themes related to scalability:
Program complexity: Health TAPESTRY was perceived as large and complex with many different facets of research, which will result in substantial amounts of data to process and interpret. Health TAPESTRY’s focus and purpose was also perceived as unclear in terms of how all the pieces fit together, and described as “everything moving slowly” due to the large number of people who are involved. They suggested synchronizing all the pieces of Health TAPESTRY, to make sure that various components and timelines fit together to reach goals, and to have a better balance in the number of people involved across stakeholder groups. Program coordination and communication : Health TAPESTRY was described as having experienced Scientific Leads that were well connected to relevant clinical and content experts, and were enthusiastic and knowledgeable to meet program goals. Identified gaps were to not delegating responsibilities, certain skill sets, variation in team involvement, communication between leads, research, IT, volunteers, and the community. Recommendations were to expand the central Health TAPESTRY team, provide training, be transparent in the decision making processes, and provide progress notes or weekly updates distributed to the whole team. Resource capacity: There was concern that the central Health TAPSTRY team may not be able to carry out administrative tasks to oversee the project and manage the sub-studies being conducted, and the long-term funding of the program. Recommendations were to designate a person who would function as an intermediary to facilitate communication between team members, to build resource capacity for IT, and to investigate additional research opportunities to generate additional funding. Stakeholder buy-in: Informants felt that it was important to maintain buy-in from a broad range of disciplines and organizations and to ensure that the inter-professional team is considered as part of the central Health TAPESTRY developer team. They suggested more involvement of this group in discussions, particularly in terms of how best clinicians could respond to the Health TAPESTRY report data, and to highlight the mutual benefits between the project and the clinical and content experts.
Adoption potential : Health TAPESTRY was perceived to have potential for scale-up in a range of different implementation contexts, but only if implementation plans are not too rigid. Scalability challenges were related to how data would be shared between sites if the adopter site has different hardware (e.g., tablet device) and software (e.g., operating system), and how to keep Health TAPESTRY consistent across different adopter sites/provinces. Suggestions were to develop an online implementation guide to provide practical information on how to recruit and train volunteers, how to overcome risk management, and who should be involved at each step. Adopter site characteristics: Scaling up was perceived to be a challenge for sites that don’t have the same understanding of how Health TAPESTRY works as the pilot site, and how it might function in culturally diverse populations including language barriers. Suggestions were to make sure that adopter sites have a complete understanding of Health TAPESTRY and the community in which it will be implemented, and to understand that the partnerships that have been built within the pilot site may look very different at their site. Readiness to scale-up: Many informants indicated that Health TAPESTRY is not yet ready to be scaled up or sustained in other jurisdictions, but recommended sharing learnings from the pilot site, to think about what aspects of the scalability need to be planned, and to work out the systems issues in the most supportive environment first. Resource capacity: Scaling up will be challenging for adopter sites that lack research capacity. Respondents indicated that it is important to convey that the goal of Health TAPESTRY is not to replace or take over existing processes but to help identify which core components are valuable and fit within the adopter setting. Undetermined program efficacy: A major perceived scalability barrier was the currently undetermined effectiveness and cost-effectiveness of Health TAPESTRY. Although they recognized that this work is currently underway, respondents highlighted the importance of sharing this knowledge and evidence with adopter sites so they can see what has worked, and where changes might need to be made to optimize success.
II. Health TAPESTRY Intervention
Table 3 summarizes the determinants across the dimension of the Health TAPESTRY intervention. We identified four themes related to, and three themes related to scalability:
Program complexity: Health TAPESTRY is complex and has many moving parts including IT maintenance, PHR information, regulations for changing coding in the EMR, data collection and update, and integrating Health TAPESTRY within new and existing health care programs (e.g., HealthLinks). Program coordination and communication: Clinic team challenges were a lack of a process for when and how to use the Health TAPESTRY report by providers, who should take care of the patient once the report is generated, lack of information on what resources are available to support activities suggested by the report, and that the report might get lost among other incoming electronic documents. Suggestions were to clarify specific roles and responsibilities on the report, to clearly indicate the expected and correct health care responses, and to adjust care processes so that Health TAPESTRY becomes part of what the clinic team are already doing. At the level of the volunteer program, challenges involved the difficulty with sustaining the volunteer pool with increasing number of enrolled clients, the availability of student or young volunteers, involvement of a well-established volunteer community organization, and the lack of support for coordinating and scheduling client visits. Suggestions were to keep the volunteer numbers reflective of the expanding client numbers, to provide on-going support and updates to volunteers/coordinators on the overall program, to provide constructive feedback about expectations and performance, and to ensure that volunteers are prepared for unknown situations. Resource capacity: Challenges experienced by the clinic team were not having enough resources to keep up with the increased number of clients enrolled in Health TAPESTRY, recruiting enough volunteers, and obtaining sustainable funding. Suggestions were to provide additional support to primary care, consider Health TAPESTRY elements that fit into existing workflows, and to consider opportunity costs. Although the technology behind Health TAPESTRY and its integration (i.e., the automated generation of the report) were described by respondents as innovative, it was perceived as requiring a large amount of resources. It was suggested that a plan is needed that takes into consideration the burden on providers’ time and workflow. Stakeholder buy-in : The inter-professional team was described as enthusiastic, innovative, and open-minded about Health TAPESTRY, but client recruitment was perceived as dependent on clinician buy-in or willingness to review and follow-up on Health TAPESTRY reports. Other factors influencing buy-in were clinical relevance of the data gathering tools, the ability to have input into customizing questions, having competing priorities within primary care, the lack of time and processes for screening patients for inclusion, and the potential for provider research fatigue. Clients’ buy-in were influenced by their comfort level with technology, perception of how the program can add to their healthcare experience, comfort with the clinic team having access to their data, and their reception to volunteers. Community level buy-in was related to fostering and maintaining a relationship with community organizations. Respondents suggested engaging community partners in updates and program progress, providing information on how they can be involved, and for the Health TAPESTRY team to demonstrate early wins to benefit patients and the system.
Adopter site characteristics: Adopter sites need to understand how their environment works, to recognize that other communities are typically very different or have a different culture, and that partnerships that were built in the pilot site may not resemble their own partnerships. Suggestions to facilitate adoption were to get to know the infrastructure, needs and resources that are available at the adopter site including what works and what community services are available, how is it different, and how Health TAPESTRY might fit into it. Respondents felt that it is important for adopter sites to have a full understanding of Health TAPESTRY at the onset of implementation so they can better understand how it might fit into their context, and to understand that it will not replace existing processes. The goal is to help individuals/organizations to identify which core components of the program fit with their needs and setting.
Resource capacity: Respondents were concerned that due to resource constrains experienced by many private, solo, and paper-based primary care practices, Health TAPESTRY may be perceived as overwhelming or difficult to scale up. These practices may lack the volunteer capacity to gather the data and the physicians to address identified problems. Suggestions were to engaging people and organizations outside of the health realm that can support solo practitioners, and to consider Health TAPESTRY as part of the primary care visit. At the level of volunteers, challenges were related to the lack of resources in rural settings to coordinate volunteers (i.e., access to and their transportation), and to train and support them. Additionally, Health TAPESTRY may not work in family health teams that are not linked with volunteer and communication resources. Another identified challenge was how to ensure the quality of volunteer-client visits, particularly as the program expands. Suggestions were to expand the scope of volunteers beyond students such as to consider peer support volunteers (e.g., clients’ relatives, friends, or neighbours). Stakeholder buy-in: It is important to obtain buy-in from the expansion site’s clinic team to ensure that they view Health TAPESTRY as value added. Respondents felt that it was important to show the relative advantage of Health TAPESTRY including improved care for patients, and to persuade other stakeholders in the system such as government and patient advocacy groups of these advantages. Having credible and respected champions in the local primary care practices was also identified as important. At the level of the volunteer team, informants reflected on the pilot stage when this group was smaller and had a stronger sense of community, which facilitated buy-in and engagement from the volunteers. Respondents feared that as Health TAPESTRY expands with an increased number of volunteers, their sense of belonging to the program may be diminished, lead to decreased commitment to the program or feeling under-recognized or under-appreciated.
Our mixed-methods study identified factors that influence the sustainability and scalability of the Health TAPESTRY intervention from the perspective of the team that was actively involved in its development and pilot testing. Our PHASE 1 survey findings helped us to focus a discussion amongst the Health TAPESTRY team about anticipated sustainability challenges, and informed the interview guide that was used for a more in-depth understanding of specific sustainability and scalability determinants. In PHASE 2, our qualitative study of telephone interviews with 25 team members revealed that the sustainability of Health TAPESTRY was influenced by its complexity, coordination and communication, resource capacity, and stakeholder buy-in; while the scalability of Health TAPESTRY was influenced by its adoption potential, adopter site characteristics, readiness to scale-up, resource capacity, stakeholder buy-in, and undetermined efficacy (Fig. 3). These findings helped us to create actionable recommendations (as indicated by informants) that were matched with specific sustainability and scalability challenges. Overall, we found that the resulting themes across the different factor levels of the NHS sustainability model were consistently spread across all three domains (process, staff, organization). The recommendations suggested by informants combined with team discussions (generated by standardized factor-matched questions in the NHS sustainability model) can be used to optimize an intervention such as Health TAPESTRY thereby increasing its potential for impact.
Our overall findings showed that resource capacity appears to be an important factor influencing the sustainability and scalability of Health TAPESTRY both in terms of its operations as well as the intervention itself and its components (Fig. 3). To sustain Health TAPESTRY, informants suggested designating a person who could facilitate communication between team members to oversee the project, carry out administrative tasks, build resource capacity for IT, and to generate additional funding. To ensure that clinic teams have enough resources to keep up with the demand of Health TAPESTRY (i.e., increased number of clients and volunteers), respondents suggested providing additional support to primary care, and to consider only those elements of Health TAPESTRY that fit into existing workflows and provider time, particularly in terms of overcoming the demands to sustain its technology innovation. To scale up Health TAPESTRY, respondents iterated the importance of conveying that the goal of Health TAPESTRY is not to replace or take over existing processes but to help identify which core components are valuable and fit within the adopter setting. To support resource poor primary care practices (i.e., many private, solo and paper-based practices), respondents suggested engaging people and organizations outside of the health realm, and to consider Health TAPESTRY as part of the primary care visit. At the level of volunteers, suggestions were to expand the scope of volunteers beyond students such as to consider peer support volunteers (e.g., clients’ relatives, friends, or neighbours), particularly in rural settings where access to public transportation may be limited.
The investigation of sustainability is seldom considered in research plans or not considered early enough in the development process to be able to address potential challenges. A related challenge is that there is little guidance on how to assess the sustainability of KT interventions. There are frameworks for implementing sustainability interventions and for measuring sustainability [22, 27, 42, 43], but a recent scoping review by Tricco et al. to determine the uptake of such frameworks found no studies that used a framework to consider sustainability of a KT intervention in the context of chronic disease management . Furthermore, very little attention has been paid to studying the sustainability and scale-up and spread of specific health care innovations  even though these are the investigations that are needed to inform practice and policy decision making. Of reported efforts to scale-up interventions, most do not consider theory-based, rigorous methods for studying scale-up processes, and have involved mostly “conceptual or descriptive” analyses with a narrow focus . The lack of consideration of sustainability can lead to implementation failure and wasted resources [34,35,36]. Understanding the determinants of sustainability and taking action to address them involving all knowledge users, has the potential to facilitate implementation efforts of change initiatives, and can help support sustained improvements in health care services and patient outcomes over time.
Our study contributes to advancing the knowledge of systematically investigating the sustainability of complex KT interventions such as Health TAPESTRY. Another strength of our study is that we not only identified potential sustainability and scalability factors, but also attempted to map suggestions and recommendations by respondents to identified factors as well as to the NHS sustainability model, to elucidate what steps may be actionable to overcome them. Individual scores were highly variable (range 22.8–96.9), which is a reflection of the wide range of stakeholders involved in the implementation of Health TAPESTRY. However, the goal of the NHS survey is to derive a team score, which is the level at which we can best reveal opportunities for improvement, to facilitate a common understanding of challenges, and to promote consensus-based solutions to address them. The actionable recommendations are therefore representative of the collective perceptions of the Health TAPESTRY team, making them more relevant for decision making. In fact, informant suggestions were directly applied to make changes to Health TAPESTRY (e.g., to clarify aspects of the automated report; improve volunteer coordination and recruitment) as well as to improve the communication between the various stakeholders involved in Health TAPESTRY (e.g., regular updates via monthly electronic newsletters). Lastly, our findings contribute to the value and feasibility of a structured process related to our study design (i.e., sequential mixed-methods approach) that can be used by teams to iteratively inform change. We suggest that these methods could be applied to any complex health care delivery intervention.
Our study has some limitations. First, we used the NHS sustainability survey, which aims to enable teams to recognize and self-assess key sustainability factors at an early stage of implementation and to prompt discussion and action across domains and sub-domains with the largest potential for improvement . However, some participants found the use of this survey challenging (e.g., the number and understandability of the questions, forced choice statements, and its value in elucidating sustainability barriers). Another study that applied the NHS sustainability survey found similar challenges . However, these challenges may be overcome by providing more instruction to participants on how to complete the survey. Additionally, since the survey was administered to a wide scope of Health TAPESTRY stakeholders, participants who were less familiar with all facets of the intervention and its developmental steps (e.g., volunteer organization) found it difficult to understand and relate the terminology used in the survey to the Health TAPESTRY intervention. Third, the response rate for our survey was moderately low (53%), so our sample and findings may not be representative of the entire implementation team. Furthermore, findings may not be generalizable beyond the sample of Health TAPESTRY-affiliated stakeholder groups who participated in our studies. However, we had a balanced representation of our stakeholder groups across our sample of participants in both phases of our investigation. Fourth, it was a challenge to prioritize within all of the data in terms of clarifying what are the critical factors that could impair the sustainability and scalability of an intervention such as Health TAPESTRY. However, sustainability and scalability are interrelated, and so we suggest that a development team cannot address identified sustainability and scalability factors in isolation. Fifth, our findings could not elucidate which elements need to be consistently applied and how best to balance this with the contextualization that is needed. When research programs are implemented in real-world settings, some degree of adaptation of program components occurs [45,46,47]. There is some evidence that it can be beneficial to allow some adaptation to local context in order to facilitate buy-in and receptivity of program content [48, 49]. This will be an important consideration when developing an implementation guide for Health TAPESTRY for the purpose of scaling up the approach across Canada. It will also be important to evaluate this process to determine what aspects of the approach should change to maintain fidelity, and which components can allow flexibility without impacting outcomes.
Our study identified sustainability and scalability determinants of the Health TAPESTRY intervention that can be used by the Health TAPESTRY team to optimize its potential for impact. Next steps will involve using findings to inform a guide to facilitate sustainability and scalability of Health TAPESTRY in other jurisdictions considering its adoption. Our findings build on the limited current knowledge of sustainability, and advances KT science related to the sustainability and scalability of KT interventions.
Electronic medical record
Family health team
- Health TAPESTRY:
Teams advancing patient experience: strengthening quality
National health service
Personal health record
Statistics Canada. Tables by subject: Diseases and health conditions. Available at: http://www.statcan.gc.ca/tables-tableaux/sum-som/l01/ind01/l3_2966_1887-eng.htm?hili_none. Accessed in July 2017.
The Chief Public Health Officer’s Report on the State of Public Health in Canada, 2014: Public Health in the Future. Public Health Agency of Canada. Available at: http://www.phac-aspc.gc.ca/cphorsphc-respcacsp/2014/index-eng.php. Accessed in July 2017.
World Health Organization. Global status report on noncommunicable diseases. Geneva: WHO. 2010. Available at: http://www.who.int/nmh/publications/ncd_report_full_en.pdf. Accessed in July 2017.
Kone Pefoyo AJ, Bronskill SE, Gruneir A, Calzavara A, Thavorn K, Petrosyan Y, et al. The increasing burden and complexity of multimorbidity. BMC Public Health. 2015;15:415.
Canadian Institute of Health Information (CIHI): Pan-Canadian Forum on High Users of Health Care. Available at: https://secure.cihi.ca/free_products/highusers_summary_report_revised_EN_web.pdf. Accessed in July 2017.
Independence, Activity and Good Health Ontario’s Action Plan for Seniors. ON Seniors’ Secretariat 2013. Available at: http://www.oacao.org/images/ontarioseniorsactionplan-en.pdf. Accessed in July 2017.
Dolovich L, Oliver D, Lamarche L, Agarwal G, Carr T, Chan D, Cleghorn L, Griffith L, Javadi D, Kastner M, Longaphy J, Mangin D, Richardson J, Papaioannou A, Ploeg J, Raina P, Risdon C, Santaguida L, Straus SE, Thabane L, Valaitis R, Price D. A protocol for a pragmatic randomized controlled trial using the Health Teams Advancing Patient Experience: Strengthening Quality (Health TAPESTRY) platform approach to promote person-centred primary health care for older adults. Implement Sci. 2016 (in press).
Jenkinson CE, Dickens AP, Jones K, et al. Is volunteering a public health intervention? A systematic review and meta-analysis of the health and survival of volunteers. BMC Public Health. 2013;13:773.
Casiday R, Kinsman E, Fisher C, Bambra C. Volunteering and health: what impact does it really have? Final report to Volunteering England. London: Volunteering England; 2008.
von Bonsdorff MB, Rantanen T. Benefits of formal voluntary work among older people. A review. Aging Clin Exp Res. 2011;23:162–9.
Cattan M, Hogg E, Hardill I. Improving quality of life in ageing populations: what can volunteering do? Maturitas. 2011;70:328–32.
European Year of Volunteering: Policy Agenda for Volunteering in Europe (PAVE). EYV 2011 Alliance Secretariat. Available at: http://www.kansalaisareena.fi/EYV2011Alliance_PAVE_copyfriendly.pdf. Accessed in July 2017.
United Nations Volunteers: State of the world’s volunteerism report 2011. Universal values for global well-being. Bonn: United Nations Volunteers. 2011.
Canadian Institutes of Health Research: Guide to Knowledge Translation Planning at CIHR: Integrated and End-of-Grant Approaches. Available at: http://www.cihr-irsc.gc.ca/e/45321.html. Accessed in July 2017.
Oliver D, Price D, Agarwall G, Chan D, Goeree R, Griffiths L, Kastner M, Letts L, Mangin D, et al. The use of volunteers, technology and interprofessional team to keep older adults health at home: Findings from the TAPESTRY Pilot Study. Canadian Association for Health Services and Policy Research (CAHSPR) conference 2014. Available at: https://cahspr.ca/web/uploads/conference/2014-05-12_Book_of_Abstracts.pdf. Accessed in July 2017.
Dolovich L, Oliver D, Lamarche L, Agarwal G, Carr T, Chan D, Cleghorn L, Griffith L, Javadi D, Kastner M, Longaphy J, Mangin D, et al. A protocol for a pragmatic randomized controlled trial using the Health Teams Advancing Patient Experience: Strengthening Quality (Health TAPESTRY) platform approach to promote person-focused primary healthcare for older adults. Implement Sci. 2016;11:49.
WHO: Nine steps for developing a scaling up strategy 2010. Available at: http://www.who.int/immunization/hpv/deliver/nine_steps_for_developing_a_scalingup_strategy_who_2010.pdf. Accessed in July 2017.
Eaton J, McCay L, Semrau M, Chatterjee S, Baingana F, Araya R, et al. Scale up of services for mental health in low-income countries and middle-income countries. Lancet. 2011;378:1592–603.
Rogers EM. Diffusion of innovations. 5th ed. New York: Free Press; 2005. p. 429.
Pluye P, Potvin L, Denis J-L. Making public health programs last: conceptualizing sustainability. Eval Program Plann. 2004;27(2):121–33.
Scheirer MA, Dearing JW. An agenda for research on the sustainability of public health programs. Am J Public Health. 2011;101(11):2059–67.
Doyle C, Howe C, Woodcock T, Myron R, Pheko K, McNicholas C, et al. Making change last: Applying the NHS Institute for Innovation and improvement sustainability model to healthcare improvement. Implement Sci. 2013;8:127.
Tricco AC, Cogo E, Ashoor H, Perrier L, McKibbon A, Grimshaw JM, Straus SE. Sustainability of knowledge translation interventions in healthcare decision-making: protocol for a scoping review. BMJ Open. 2013;3:e002970.
Arnold SR, Straus SE. Interventions to improve antibiotic prescribing practices in ambulatory care. Cochrane Database Syst Rev. 2005;2:CD003539.
Ivers N, Jamtvedt G, Flottorp S, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012;4:CD000259.
Chambers LL. Factors for sustainability of evidence-based practice innovations: Part I. Res Theory Nurs Pract. 2015;29(2):89–93.
Chambes DA, Glasgow RE, Strange KC. The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implement Sci. 2013;8:117.
Proctor E, Luke D, Calhoun A, McMillen C, Brownson R, McCrary S, et al. Sustainability of evidence-based healthcare: research agenda, methodological advances, and infrastructure support. Implement Sci. 2015;10:88.
Tricco AC, Ashoor HM, Cardoso R, MacDonald H, Cogo E, Kastner M, Perrier L, McKibbon A, Grimshaw JM, Straus SE. Sustainability of knowledge translation interventions in healthcare decision-making: a scoping review. Implement Sci. 2016;11(1):55.
Scheirer MA. Is sustainability possible? A review and commentary on empirical studies of program sustainability. Am J Eval. 2005;26(3):320–47.
Wiltsey Stirman S, Kimberly J, Cook N, Calloway A, Castro F, Charns M. The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implement Sci. 2012;7:17. doi:10.1186/1748-5908-7-17.
Ham C, Kipping R, McLeod H. Redesigning work processes in health care: lessons from the National Health Service. Milbank Q. 2003;81(3):415–39.
Moher D, Glasziou P, Chalmers I, Nasser M, Bossuyt PM, Korevaar DA, Graham ID, Ravaud P, Boutron I. Increasing value and reducing waste in biomedical research: who’s listening? Lancet. 2015;387(10027):1573–86.
Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M, Julious S, et al. Reducing waste from incomplete or unusable reports of biomedical research. Lancet. 2014;383(9913):267–76.
Al-Shahi Salman R, Beller E, Kagan J, Hemminki E, Phillips RS, Savulescu J, et al. Increasing value and reducing waste in biomedical research regulation and management. Lancet. 2014;383(9912):176–85.
Ioannidis JP, Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, et al. Increasing value and reducing waste in research design, conduct, and analysis. Lancet. 2014;383(9912):166–75.
Maher L, Gustafson D, Evans A. NHS Sustainability Model and Guide. NHS Institute for Innovation and Improvement. 2010. Available at: http://www.institute.nhs.uk/index.php?option=com_joomcart&main_page=document_product_info&products_id=291&cPath=88 (login required). Accessed in July 2017.
Dillman DA. Mail and telephone surveys. The total design method. Hoboken: John Wiley & Sons; 1978.
Cane J, O’Connor D, Michie S. Validation of the theoretical domains framework for use in behavior change and implementation research. Implement Sci. 2012;7:37.
Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101.
Wendler MC. Triangulation using a meta-matrix. J Adv Nurs. 2001;35(4):521–5.
Schell SF, Luke DA, Schooley MW, Elliott MB, Herbers SH, Mueller NB, et al. Public health program capacity for sustainability: a new framework. Implement Sci. 2013;8:15. doi:10.1186/1748-5908-8-15.
Simpson DD. A Framework for Implementing Sustainable Oral Health Promotion Interventions. J Public Health Dent. 2011;71(s1):S84–s94. doi:10.1111/j.1752-7325.2011.00235.x.
Norton WE, McCannon CJ, Schall MW, Mittman BS. A stakeholder-driven agenda for advancing the science and practice of scale-up and spread in health. Implement Sci. 2012;7:118.
Dariotis JK, Bumbarger BK, Duncan LG, Greenberg MT. How do implementation efforts relate to program adherence? Examining the role of organizational, implementer, and program factors. J Commun Psychol. 2008;36:744–60.
Dusenbury L, Branningan R, Falco M, Hansen WB. A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Educ Res. 2003;18:237–56.
Ringwalt CL, Vincus A, Ennett S, Johnson R, Rohrbach LA. Reasons for teachers’ adaptation of substance use prevention curricula in schools with nonwhite student populations. Prev Sci. 2004;5:61–7.
Bernal G, Sáez-Santiago E. Culturally centered psychosocial interventions. J Commun Psychol. 2006;34:121–32.
Edwards N, Barker PM. The importance of context in implementation research. J Acquir Immune Defic Syndr. 2014;67(Suppl 2):S157–62.
The authors greatfully acknowledge the participation of the Health TAPESTRY team members and for their support in this study. MK is supported by a Canadian Institutes of Health Research (CIHR) New Investigator Award; SES is funded by a Tier 1 Canada Research Chair in Knowledge Translation.
The study was funded by Health Canada (6817–06-2013/5570001). The funders had no role in the design, data collection and analysis of the study or the prepration of the manuscript.
Availability of data and materials
The authors make, upon request, associated protocols and data collection tools available to the readers.
Ethics approval and consent to participate
Research and ethics approval was obtained from the Hamilton Integrated Research Ethics Board as part of the Health TAPESTRY program development and evaluation at McMaster University in Hamilton, Ontario, Canada (REB#14–726). All participants provided written consent for participation.
Consent for publication
All participants were made aware that the findings of the research would be prepared for publication in a peer-reviewed journal.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Characteristics of Health TAPESTRY team stakeholder groups. This table shows the breakdown of Health TAPESTRY team member types, the number in each category, and their definition. (DOCX 16 kb)
Qualitative telephone interview semi-structured questionnaire. This is the interview guide that was used for the qualitative telephone interviews. It shows the questions, and how they are mapped to the theoretical domains framework (TDF) and their constructs. (DOCX 21 kb)
Identified challenges to the sustainability of Health TAPESTRY and recommendations to address them as indicated by respondents. This table shows the identified sustainability challenges by theme and sub-theme that emerged from qualitative telephone interviews as well as the recommendations to overcome the barrier (as suggested by respondents) accompanied by relevant quotes. (DOCX 43 kb)
Identified challenges to the scalability of Health TAPESTRY and recommendations to address them. This table shows the identified scalability challenges by theme and sub-theme that emerged from qualitative telephone interviews as well as the recommendations to overcome the barrier (as suggested by respondents) accompanied by relevant quotes. (DOCX 51 kb)
About this article
Cite this article
Kastner, M., Sayal, R., Oliver, D. et al. Sustainability and scalability of a volunteer-based primary care intervention (Health TAPESTRY): a mixed-methods analysis. BMC Health Serv Res 17, 514 (2017). https://doi.org/10.1186/s12913-017-2468-9