- Study protocol
- Open Access
Enhancing implementation of measurement-based mental health care in primary care: a mixed-methods randomized effectiveness evaluation of implementation facilitation
BMC Health Services Research volume 18, Article number: 753 (2018)
Mental health care lags behind other forms of medical care in its reliance on subjective clinician assessment. Although routine use of standardized patient-reported outcome measures, measurement-based care (MBC), can improve patient outcomes and engagement, clinician efficiency, and, collaboration across care team members, full implementation of this complex practice change can be challenging. This study seeks to understand whether and how an intensive facilitation strategy can be effective in supporting the implementation of MBC. Implementation researchers partnering with US Department of Veterans Affairs (VA) leaders are conducting the study within the context of a national initiative to support MBC implementation throughout VA mental health services. This study will focus specifically on VA Primary Care-Mental Health Integration (PCMHI) programs.
A mixed-methods, multiple case study design will include 12 PCMHI sites recruited from the 23 PCMHI programs that volunteered to participate in the VA national initiative. Guided by a study partnership panel, sites are clustered into similar groups using administrative metrics. Site pairs are recruited from within these groups. Within pairs, sites are randomized to the implementation facilitation strategy (external facilitation plus QI team) or standard VA national support. The implementation strategy provides an external facilitator and MBC experts who work with intervention sites to form a QI team, develop an implementation plan, and, identify and overcome barriers to implementation. The RE-AIM framework guides the evaluation of the implementation facilitation strategy which will utilize data from administrative, medical record, and primary qualitative and quantitative sources. Guided by the iPARIHS framework and using a mixed methods approach, we will also examine factors associated with implementation success. Finally, we will explore whether implementation of MBC increases primary care team communication and function related to the care of mental health conditions.
MBC has significant potential to improve mental health care but it represents a major change in practice. Understanding factors that can support MBC implementation is essential to attaining its potential benefits and spreading these benefits across the health care system.
Most modern medical care uses objective measurement to guide and evaluate treatment. For example, blood pressure measurement is routinely used to screen for hypertension, to determine if treatment is indicated, and to guide treatment including behavior-focused efforts. By contrast, modern mental health care typically uses subjective clinician assessment as the most common tool for guiding both psychotherapy and pharmacotherapy treatment decisions. Although reliable and valid patient reported outcome measures (PROM) for mental health conditions are available, their use to guide treatment beyond screening and initial evaluation is relatively infrequent [1,2,3,4,5,6,7]. Routine use of PROM to track patient symptoms, guide treatment decisions and facilitate communication between patients and providers is referred to as measurement-based care (MBC). There are three critical elements to MBC: (1) collection of information using psychometrically-sound self-report instruments repeatedly over the course of treatment; (2) use of that information to guide treatment decisions; and, (3) sharing the information with patients and others on the health care team to support collaborative treatment decision-making.
The evidence for MBC is strong when it is used over time in a systematic way to adjust treatment (pharmacotherapy or psychotherapy) rather than simply fed back to providers or patients (monitoring alone) [8,9,10,11]. In addition to its potential benefit to individual patient outcomes, MBC can improve treatment fidelity [9, 10, 12], improve patient-provider communication [13, 14], and increase patient engagement . At the clinic and program level, MBC can support treatment team communication about mental health conditions [16, 17] and can facilitate quality improvement (QI) efforts  by providing patient outcome data that can be monitored in response to systematic changes in clinical operations. At the population level, MBC can improve the programmatic efficiency of care by identifying patients in need of more treatment and reducing the number of sessions for patients who have improved . Further, MBC does not require new staffing and, while a MBC approach requires providers to alter their clinical practice, it does not add time to patient encounters . As a result, while MBC improves the outcomes of care, it can also increase clinician efficiency . Despite these benefits and calls for the implementation of standard systems of MBC in mental health practice, few health care systems have adopted it as a standard of care [11, 19, 20]. In fact, it is the norm for mental health providers in all clinic settings to rely heavily on clinical interviews focused on the individual experience of patients (ideographic assessment) rather than using standardized instruments that allow comparison to a normative group (nomothetic assessment) [5, 6, 21].
Beginning in 2015, the U.S. Department of Veterans Affairs (VA) began an initiative to increase the use of MBC throughout VA mental health services. The initiative has occurred in phases. In the first phase, leadership representing the broad scope of VA mental health care agreed on a set of standards that would define successful implementation. These standards include: the use of four specific PROMs (the Patient Health Questionnaire-9 (PHQ-9) , the Generalized Anxiety Disorder-7 (GAD-7) , the Brief Addiction Monitor (BAM) , and the Post Traumatic Stress Disorder Checklist-5 (PCL-5) [25, 26]); the requirement that assessment responses be accessible in the electronic health record and not just recorded in progress notes; and the collection of measures at the start and periodically throughout episodes of care. Having laid this groundwork, the next phase included a set of activities that have become standard in the VA system for supporting practice change initiatives. This national support included: developing educational materials for providers and patients, engaging volunteer programs throughout the system to begin implementation, supporting a community of practice for engaged sites, and, upon request, brief coaching for implementation problem-solving. That stage was recently completed with evidence of increased use of PROM throughout the healthcare system. Educational materials, web-based community of practice discussion, and expert consultation remain available. The next phase requires every facility to implement MBC in at least one clinical program. Coincident with these policy changes the Joint Commission is also now requiring the use of MBC in some behavioral health programs such as addiction and residential services .
The VA has been an early adopter of MBC in its Primary Care Mental Health Integration (PCMHI) program. Beginning in 2007, the program integrated mental health services into primary care clinics system-wide  using both care managers in a collaborative care model [29, 30] and embedded behavioral health providers (BHPs) in an integrated primary care model . Collaborative care models of primary care treatment for depression provided early, evidence-based examples of how MBC can improve communication, collaboration, and quality of care [7, 29, 32]. While MBC is a core component of care management, embedded BHPs have typically relied on PROM only as part of an initial assessment; follow-up assessments are most often idiographic [5, 6, 21]. Further, the implementation of the BHP component of PCMHI has been more prevalent than the implementation of care management . As a result, at many sites, the PCMHI program is not attaining the full benefit of MBC.
Even when PROM is used in PCMHI programs, it is often only partially used. For instance, measures may be collected using pen and paper but not transcribed into the electronic medical record. Thus, the data are unavailable for communication, team decision-making, and program improvement efforts. Further, in PCMHI, where providers are intended to be fully integrated members of primary care, this partial use of MBC limits the ability of the rest of the primary care team to coordinate with and support mental health treatments and presents a barrier to interdisciplinary team function . Team communication is a crucial component of team development and is thought to be critical to establishing high performance health care delivery teams [20, 34, 35]. The implementation of MBC has the potential to provide a pathway to improved communication about mental health conditions, allowing professionals from medical and mental health disciplines to work in a more fully integrated manner [36, 37].
In summary, while MBC has significant potential to yield improved patient care and interdisciplinary practice, it is a complex practice that has proven to be challenging to fully implement. Therefore, the current protocol will seek to determine if external facilitation (EF) when combined with an internal QI team will improve the implementation of MBC practice as compared to sites receiving only standard national support. External facilitation, an evidence-based, multi-faceted process of interactive problem-solving and support [38, 39], can incorporate multiple other discrete implementation strategies, e.g., audit and feedback, education, and marketing [40,41,42], to address the challenges of implementing MBC for mental health conditions. Engaging stakeholders and involving them in implementation processes is a core component of facilitation [43, 44] and is critical for implementation success [45, 46]. QI teams in this study will be composed of local stakeholders who will share responsibility for implementation efforts, thus acting as internal facilitators. The Enhancing Implementation of MBC in primary care study therefore has two primary research aims:
Aim 1: To understand whether and how an EF + QI team strategy can be effective in supporting the implementation of MBC as compared to standard VA national support.
Aim 2: To explore the association of MBC practice with primary care and PCMHI communication and team functioning.
This study employs a quasi-experimental multiple case study design , using mixed methods to accomplish the study aims. In addition to addressing the two primary aims, the study is designed to explore factors that may affect successful implementation.
This study was designed as a collaborative effort between VA national leaders and the study team with the shared goal of increasing the use of MBC in PCMHI programs. In order to realize this ongoing collaboration, a Partnership Panel meets regularly with study leaders to inform the project’s work, provide input into study procedures, e.g., site identification and matching, and, discuss emerging findings and their implications for both the project and system-wide implementation. The panel consists of members whose role in the VA includes relevant national leadership responsibilities and three PCMHI site leaders whose facilities have demonstrated high PROM utilization.
The RE-AIM  and integrated Promoting Action on Research Implementation in Health Services (iPARIHS) [49, 50] frameworks guided the design of this protocol. Figure 1 shows how these frameworks work together in the evaluation of the EF + QI team implementation strategy and factors that may affect implementation.
The RE-AIM framework guided the design of the evaluation of the implementation strategy’s effectiveness, i.e., the extent to which MBC is implemented at each site. RE-AIM is a useful framework for evaluating implementation interventions because it addresses issues related to real-world settings and assesses multiple dimensions (Reach, Effectiveness, Adoption, Implementation, and Maintenance). Reach is defined as the absolute number or proportion of eligible individuals who received the evidence-based practice. Effectiveness is defined as the impact of the evidence-based practice on identified outcomes. Adoption is defined as the absolute number or proportion of settings or staff using the evidence-based practice. Implementation is defined as the fidelity of the evidence-based practice or the proportion of its components as implemented in routine care. Maintenance is defined as the degree to which the evidence-based practice is sustained over time [48, 51]. Study measures will address each of the five RE-AIM dimensions to test the implementation strategy’s overall effect .
The iPARIHS framework guided the design of the implementation facilitation strategy and the assessment of factors that may influence MBC implementation. The iPARIHS framework suggests that four dimensions need to be considered when conducting implementation efforts . Characteristics of three of these dimensions can hinder or foster implementation: Innovation, the focus of implementation efforts; Recipients, the individuals and teams who are affected by or who influence the implementation; and Context, factors within the local clinic, organization within which it is embedded, and the outer context, i.e., the wider health system. The fourth dimension, Facilitation, consists of a role, i.e., a designated person serving as a facilitator, and a process, the strategies and actions that a facilitator applies [40, 53]. Facilitation is the active ingredient that can maximize the potential for successful implementation by assessing and addressing challenges and leveraging strengths in each of the other dimensions.
Site selection and recruitment
A rolling site recruitment approach is used so that four sites will begin study participation every 3 months. Site selection and recruitment processes for each cycle consists of three steps: identification of potential sites, stratification of similar sites into groups, and then recruitment of one or more pairs of sites within those groups. Initially, potential sites are drawn from a pool of 21 VA facilities that self-nominated their PCMHI program to participate in the first phase of the VA National MBC Initiative. In collaboration with the Partnership Panel, we selected administrative data variables (e.g., current use of PROM, site size, staffing and PCMHI program metrics [same-day access and population penetration]), and weighting current PROM use more heavily, we used these variables to stratify sites into relatively similar groups. Prior to each new recruitment cycle, we will draw the same administrative data, excluding participating study sites, to refresh the group placement and the Partnership Panel will review and provide input. If we are unable to recruit 12 sites from within the original pool, we will use the same administrative data variables and collaboration process to identify additional potential sites and group them by similarity. We will continue this process until all 12 sites have been selected and recruited into the study. For each cycle, we will recruit four sites whose PCMHI, primary care and mental health leaders have agreed to participate in study activities. Sites will be randomly assigned to either continued standard national support or the study implementation intervention.
Within each of the 12 study sites, study participants will be predominantly VA employees selected based on their role in the clinics: PCMHI leaders, PCMHI providers, primary care providers and nurses serving on their teams, and QI team members. In addition to the PCMHI leader, QI teams will include the site’s primary care and mental health leaders or their designees, other members as identified by the site for inclusion, and Veteran patient representatives who may or may not be VA employees. Age, race/ethnicity, and gender distribution of VA staff will be reflective of those distributions among VA staff at participating clinics.
Implementation facilitation strategy
The implementation facilitation strategy at the six intervention sites will combine expert external facilitation (EF) with an internal QI team to increase the use of MBC for mental health conditions within primary care. A team providing EF at each site will consist of an external facilitator and two to three MBC subject matter experts (SME). The two study external facilitators have expertise in implementation science principles and methods broadly, as well as PCMHI models of care. They each have extensive experience facilitating implementation of evidence-based innovations. Each will lead a facilitation team at intervention sites in supporting QI team members’ efforts to implement MBC.
The facilitation strategy will be applied at each site across three phases, Preparation Phase, Design Phase, and Implementation Phase.
During this approximately 2-month long phase, the site facilitation team will engage leadership, begin to assess site needs and resources, and help identify QI team members.
At the beginning of this approximately 3-month long phase, the external facilitator will conduct an in-person site visit during which he or she will meet with site leaders, the QI team and clinical staff to introduce the study, address gaps in MBC knowledge and perceptions of the evidence, and begin the process of planning for MBC implementation. Although the external facilitator will continue to engage local site leadership throughout this phase, the facilitation team will focus on 1) providing training to the QI team on PROM use, MBC practice and systems change, and 2) helping the QI team develop an implementation plan that is customized to local stakeholder needs, resources and preferences and then assess implementation status and needs. The implementation plan will include a kick-off date and process for the beginning of the next phase.
At the beginning of this phase, site stakeholders, including the QI team, will meet to formally initiate the implementation of their plan. The external facilitator will be available through virtual technology to support the QI team in this kick-off meeting. Throughout this approximately 6-month phase, the QI team, with external facilitator and SME support, will educate, mentor, and encourage clinicians as they change their practice. They will also monitor implementation progress, conduct problem-identification and resolution, and help sites make adjustments to local MBC practices to overcome problems and enhance sustainability of MBC.
Across the intervention strategy phases, the facilitation team will gather information directly from site leaders and personnel and other sources (e.g., a national dashboard developed for the MBC initiative; local program level reports) for use in developing facilitation activities and to provide feedback to the QI team. Results of three baseline study measures, the Team Development Measure, the Organizational Readiness to Change instrument, and the Survey of MBC use, will also be provided to the facilitation team.
Timing of data collection activities
The timing of all data collection activities is based on index dates linked to facilitation activities that demarcate the phases of the facilitation strategy (Preparation Phase, Design Phase, and Implementation Phase) and an additional Sustainment phase, a 6-month observation period following the end of the facilitation intervention (see Fig. 2). Index dates at intervention sites will be used to structure evaluation activities for both the intervention sites and matched comparison sites.
Index Date 1 is defined as the date the external facilitator begins working with intervention site personnel. Index Date 2 is defined as the date of the external facilitator’s in-person site visit. Index Date 3 is defined as the date of the intervention site kick-off meeting. Index Date 4 is defined as the date that the external facilitator reports that the facilitation team has completed active facilitation and has ended regularly scheduled implementation support activities. Index Date 5 is defined as 6 months after the end of the Implementation Phase index date.
Aim 1 data collection
Mixed methods will be used to 1) evaluate the effectiveness of the EF + QI team strategy to improve MBC implementation in primary care clinics and 2) assess factors that may affect MBC implementation.
Effectiveness of EF + QI team strategy
The RE-AIM framework guided the selection of evaluation measures to assess the effectiveness of the EF + QI team implementation strategy. For both intervention and comparison sites, data sources and measures were selected for each RE-AIM framework dimension (see Table 1). For intervention sites, additional data sources will provide supplementary information on the Implementation dimension. Data collection methods for the measures of RE-AIM dimensions follow.
Administrative dataAdministrative data focusing on the mental health services delivered in primary care to patients seen by PCMHI providers at all 12 sites will be pulled from the VA Corporate Data Warehouse to capture measures of Reach, Adoption, Implementation, and Maintenance. The study will access use of PROM and medical care utilization records for three time periods based on the site index dates: baseline, during the implementation, and during sustainment.
Chart reviewsTo assess Effectiveness and Implementation, data will be extracted from randomly selected subsamples of patient records in the baseline and implementation administrative data pulls for all 12 sites (see Fig. 1). Electronic records of PROM administration and progress notes will be reviewed for each patient encounter during the observation period to extract data for the following variables: patient and provider demographics, dates of encounters, clinic locations, diagnostic codes, mental health PROM use and scores, and text pertaining to assessment and measurement administration/utilization. Progress notes will be reviewed for both the presence and absence of MBC elements, such as evidence that the PROM data played a role in clinical decisions and/or was communicated to the patient or other providers.
Provider survey: Measurement-based Care Use and AttitudesOnline surveys will be conducted with PCMHI and primary care providers and nurses to capture MBC use (Effectiveness and Maintenance) at all 12 sites at three time points (see Table 2 and Fig. 1). The survey instrument (Oslin DW, Hoff R, Mignogna J, Resnick SG: Provider attitudes and experience with measurement-based mental health care, submitted), developed for the VA National MBC Initiative and adapted for this study, will assess provider perceptions and attitudes about MBC and frequency of PROM use at key points in care.
Provider survey: Team Development Measure (TDM)The TDM, which has strong psychometric properties, was designed to measure and promote quality improvement in team-based healthcare settings . It will be administered online to providers at two time points (see Fig. 1 and Table 1). TDM data will be used to assess the degree to which health care teams at all sites have and use elements (cohesion, communication, role clarity and goals and means clarity) needed for highly effective teamwork (Implementation).
Measurement-based care qualitative interviewsWe will conduct semi-structured qualitative interviews with PCMHI programmatic leaders at all 12 sites to assess the status of MBC implementation and use of MBC in current practice (Effectiveness and Implementation). These interviews will be conducted at two time points (see Table 2 and Fig. 1). Both interviews will focus on PCMHI leader perceptions of how PCMHI providers are collecting, using and sharing PROM, the evidence for MBC, the importance and value of MBC to site stakeholders, and the barriers and facilitators to MBC implementation. The second interview is designed to assess changes in these perceptions and attitudes, adaptations in clinic MBC practice, and additional barriers and facilitators to implementing MBC experienced since the last interview. Finally, we will ask for PCMHI leader views on potential barriers and facilitators to sustaining MBC if it has been implemented.
Factors that may affect MBC implementation
The iPARIHS framework guides our assessment of factors that may affect implementation. Table 2 shows data sources that will provide information for the assessment of the iPARIHS dimensions. Some of these data sources also address measures for the RE-AIM evaluation of the implementation facilitation (IF) strategy and were previously described.
Assessment of facilitation includes both self-reported documentation of activities by the external facilitators and SMEs, and, data collected from QI team members on IF activities. Because comparison sites will not receive facilitation, assessment of Facilitation will only be conducted at intervention sites.
Documentation of the facilitation strategyFacilitation activities will be documented using two data collection mechanisms. First, we will conduct bi-weekly to monthly debriefing interviews with the external facilitators and SMEs and use summary notes to document on-going facilitation, QI team processes, and factors that might foster or hinder MBC implementation at each intervention site. Second, facilitation team members will document their time, activities, and numbers and roles of site personnel on activity logs in preparation for a time-motion analysis of facilitation activities. Debriefing interview and time-motion data will enable us to go beyond determining whether the EF + QI team strategy is effective and to offer possible explanations for why it is or is not effective within particular organizational contexts and how facilitators can enhance MBC implementation.
QI team interviewsTo assess the facilitation process from the perspective of QI team members, semi-structured qualitative group interviews will be conducted with QI team members at intervention sites at two time points (see Fig. 2). The first interview will assess QI team accomplishments and functioning, the value of the facilitation process, and how it might be improved. The second interview will focus on MBC implementation and factors that affected it, results of and concerns about using MBC, MBC sustainment, and experiences with facilitation.
We will assess select organizational factors at all 12 study sites with two validated instruments to enhance our understanding of contextual factors that may have affected MBC implementation.
Provider survey: Organizational Readiness for Change (ORC)To assess baseline organizational functioning and climate at intervention and comparison sites, we will administer the ORC, which is considered to have good validity . Similar to other VA implementation studies [56, 57], we will minimize participant burden by administering only ten of the ORC subscales across three domains, including: motivation to change subscales (program needs, training needs, and pressures for change), an adequacy of resources subscale (staffing), and, organizational climate subscales (mission, cohesion, autonomy, communication, stress, and, change) .
Provider survey: Primary Care Behavioral Health Provider Adherence Questionnaire-2 (PPAQ-2)The PPAQ-2 is a scale of behavioral health provider protocol adherence to PCMHI. This adherence is thought to be essential to BHPs’ ability to function as part of the primary care team. The original PPAQ has been shown to have strong psychometric properties . The recently revised and expanded version, PPAQ-2, has shown excellent reliability and validity through confirmatory factor analysis . The study will use the following content domains: Practice and Session Management; Referral Management and Care Continuity; Consultation, Collaboration, and Interprofessional Communication; Patient Education, Self-Management Support, and Psychological Intervention; and Panel Management. Higher PPAQ-2 scores indicate high fidelity to PCMHI.
Other data sourcesIn addition to the ORC and PPAQ-2, data sources addressing RE-AIM measures will also inform our understanding of the context for MBC implementation. For example, the team communication interviews and TDM will provide us with additional information about the organizational culture and the MBC qualitative interviews will provide us with information about the contextual barriers to and enablers of change for MBC implementation. At the intervention sites, the debriefing interviews will be particularly rich sources of information about the organizational context across organizational levels.
Innovation and recipients
Some of our data sources will provide us with information about the characteristics of the innovation, e.g., MBC interviews and MBC use surveys (underlying knowledge sources, clarity and usability). They will also provide us with information about the recipients, e.g., Team communication interviews and TDM (collaboration and teamwork); QI team interviews (motivation to change and beliefs) and MBC interviews (values). Again, the debriefing interview data will provide us with a rich description of the recipients and the innovation at intervention sites.
Aim 2 data collection
To study the association between team communication and functioning and MBC implementation, individual semi-structured qualitative interviews will be conducted with primary care (providers and nurses) and PCMHI providers at the facilitation intervention sites. These interviews will be conducted at two time points (see Fig. 1). Interview items reflect constructs related to team functioning in health care settings . These items are tailored specifically for the topic of caring for primary care patients with mental health conditions: communication across primary care and PCMHI providers; impact of MBC implementation on provider communication; cohesiveness among team members; perceptions of professional role clarity; and perceptions of common team goals. Additionally, during the second interview, providers’ attitudes about the acceptability, feasibility and utility of using MBC will be assessed.
The data analysis plan is designed to allow for triangulation across data sources in order to inform the two study aims. General analytic approaches are described first, followed by the planned strategies for addressing the two study aims.
Aim 1 data analysis
Administrative data will be used to compare rates of PROM administration and entry into the EMR across intervention and comparison sites. The analytic strategy will be to compare rates between the intervention and comparison sites at the end of the intervention and after a sustainment period while controlling for the baseline rate of use. Chart review data will be analyzed by calculating descriptive statistics for patient background/demographics, PCMHI service utilization, patient diagnoses, and, PCMHI provider type. Descriptive statistics will also be used to characterize PCMHI provider documentation of MBC elements including administration of measures of interest (i.e., PHQ-9, PCL, BAM, GAD-7, others) and indicators of how scores from each administration were used (e.g., to support a diagnosis, provide feedback to patient). Analysis of variance (ANOVA) will be used to assess for changes in frequency in chart review MBC elements by group (intervention v. comparison site). Provider survey data will be analyzed at the site level as follows. Analysis of the MBC provider surveys will examine differences in attitudes and PROM use by professional group (physicians, nurses, social workers, psychologists, etc.) as well as site. In addition, we will examine change in attitudes over time comparing intervention to comparison sites. As a non-standardized measure, we will be limited to examining frequencies and distributions of responses. Using the TDM, we will calculate ratings for overall team development and the four subfactors of cohesiveness, communication, role clarity, and goals and means. These ratings will be used to assess differences in team development both between and within sites over time. Mean overall clinic/team ORC scores will be calculated as will means on each subscale. Site ORC scores will be considered when interpreting qualitative findings regarding status of MBC implementation. PPAQ-2 data will be analyzed using descriptive statistics and a series of ANOVAs to compare overall site fidelity to the PCMHI model and variations of PPAQ-2 domains across intervention and comparison sites and over time. A descriptive analysis of time data will be conducted, calculating hours spent by the EF and SMEs in support of MBC implementation. We will examine variation in IF time across clinics, types of IF activities, and over time.
Analysis of MBC and QI Team qualitative interview transcripts and facilitation team debriefing notes will be conducted separately using a rapid analysis approach. Text will be coded utilizing qualitative data analysis software, extracted, and summarized for each site to answer target research questions. Site summary results for interviews conducted at two time points will be examined for changes over time.
The application of mixed methods in this design provides the opportunity to advance our understanding of these site changes. Using qualitative data sources, the analysis will yield a more complete description of how MBC is implemented in practice at each site. Further, individual site context factors will be incorporated by using ORC, TDM and PPAQ-2 results during qualitative analysis to help explain differences in site implementation status of MBC. For example, it would be expected that sites with low readiness to change (ORC) or low levels of team development (TDM) would be less likely to successfully implement any new practice, including MBC. Conversely, if the EF + QI Team strategy is successful in addressing team development challenges, a given site may experience greater levels of MBC implementation success. Similarly, sites with low PCMHI model fidelity may struggle to keep up with the demands of fast paced primary care clinics and may find the addition of MBC practice is too challenging. Finally, provider attitudes regarding MBC are known to limit uptake [2, 62, 63]. The distribution of clinician attitude and perception responses at sites will be considered when comparing site implementation status of MBC.
Aim 2 data analysis
Transcripts from Team Communication and Functioning interviews will be analyzed utilizing a different rapid analysis approach which will include a review of each transcript and extraction of data into a summary template in order to compile results for each domain/theme of the interview guide. As there will be multiple Team Communication and Functioning interviews per site, a site summary will be created from the case-by-case summaries. Site summary results for interviews conducted at two time points will also be examined for changes over time. In addition to examining the association of MBC practice with primary care and PCMHI communication and team functioning across varying contexts, individuals and teams, our final analysis will also examine whether there are patterns in responses related to the level of fidelity to PCMHI.
As described, initial potential sites were identified and grouped with Partnership Panel input. From within each of two of these groups, two sites have been recruited. Within these pairs, one site was randomized to the implementation strategy and facilitation activities began in May 2018.
The evidence for the systematic use, over time, of PROM for mental health conditions is strong but MBC is seldom the standard of care, even in VA PCMHI programs where collaborative care management services are designed with MBC as a core component. This study will evaluate an implementation strategy, consisting of external facilitation and an internal QI team, for improving utilization of measurement-based mental health care and explore the associations of MBC practice with team communication and functioning. The EF + QI team strategy combines MBC and implementation science expertise with the expertise and participation of local stakeholders to develop and execute an MBC implementation plan tailored to site needs and resources. Grounding the study in the RE-AIM and iPARIHS frameworks will allow us to document the strategy’s effectiveness, as well as how facilitation was utilized to address barriers [63,64,65] and leverage facilitators related to characteristics of MBC for mental health, site stakeholders, and the organizational context. This knowledge will be helpful to both our VA mental health operations partners and to mental health care systems outside VA seeking to modernize mental health care delivery. Further, as MBC becomes a new standard of care , it will be essential to understand how this practice can be efficiently implemented and sustained.
This project is one of three linked studies focusing on team-based mental health care; the other two studies examine (1) the addition of veteran peer specialists to the primary care team  and (2) the implementation of interdisciplinary behavioral health teams in general mental health clinics . This portfolio of projects, part of a VA Quality Enhancement Research Initiative (QUERI) funded program, QUERI for Team-Based Behavioral Health (BH QUERI; QUE 15-289), is responsive to the Institute of Medicine recommendation that “all health professionals should be educated to deliver patient-centered care as members of an interdisciplinary team, emphasizing evidence-based practice, quality improvement approaches, and informatics” . Interdisciplinary team-based care, while valuable, is complex and challenging to implement . Therefore, this group of projects was designed to study the application of implementation facilitation to address the challenges of implementing these innovations and complex care delivery models, including team-based care [41, 68, 69]. If successful, these projects may individually and collectively advance the understanding of key factors necessary to achieve the Institute of Medicine recommendations on team care. Additionally, in sites where MBC is implemented, this project has potential to advance our understanding of interprofessional team practice as we explore whether the development of a shared system of communication regarding mental health strengthens team-based primary care in integrated care settings.
As is often the case with implementation science studies, the evolving organizational context has been, and likely will continue to be, a challenge to this project. For example, between the time that this project was initially proposed and funding started, the VA national initiative efforts moved from planning to active stages resulting in the need to redesign components of the project. As a result, the project’s site recruitment plan was refocused on PCMHI sites that engaged in the VA National MBC Initiative and our comparison condition became standard national support. Our project was strengthened by these changes as they allow us to compare our resource intensive implementation support intervention to a ‘light touch’ standard support condition. If successful, our project will help us to answer an important question for our operations partners. That is, “How much support is necessary and sufficient to implement a complex new health care practice?” It is only through our close and ongoing partnership with the VA leaders who are responsible for national MBC implementation efforts that we were able to navigate the changing national context. Through continued close collaborations we hope to execute this study in a way that will result in meaningful findings.
Analysis of Variance
Brief Addiction Monitor
Behavioral Health Provider
Electronic Medical Record
Generalized Anxiety Disorder-7
integrated Promoting Action on Research Implementation in Health Services
Organizational Readiness for Change
Post-Traumatic Stress Disorder Checklist-5
Primary Care-Mental Health Integration
Patient Health Questionnaire-9
Patient Reported Outcome Measure
Quality Enhancement Research Initiative
Reach, Effectiveness, Adoption, Implementation and Maintenance
Subject Matter Experts
Team Development Measure
- VA :
U.S. Department of Veterans Affairs
Veterans Health Information Systems and Technology Architecture
Garland AF, Kruse M, Aarons GA. Clinicians and outcome measurement: what’s the use? J Behav Health Serv Res. 2003;30(4):393–405.
Gilbody SM, House AO, Sheldon TA. Psychiatrists in the UK do not use outcomes measures. National survey. Br J Psychiatry. 2002;180(2):101. https://doi.org/10.1192/bjp.180.2.101.
Gilbody SM, House E, Sheldon TA. Routinely administered questionnaires for depression and anxiety: systematic review. BMJ. 2001;322(7283):406. https://doi.org/10.1136/bmj.322.7283.406.
Hatfield DR, Ogles BM. The use of outcome measures by psychologists in clinical practice. Prof Psychol Res Pract. 2004;35(5):485–91. https://doi.org/10.1037/0735-7028.35.5.485.
King PR, Beehler GP, Vair CL, Gass J, Funderburk JS, Lilienthal KR, et al. Identifying measurement-based care practices of VHA co-located, collaborative care providers. Prof Psychol Res Pract. 2017;48(4):236–42.
Beehler GP, King PR, Vair CL, Gass J, Funderburk JS. Measurement of common mental health conditions in VHA co-located, collaborative care. J Clin Psychol Med Settings. 2016;23(4):378–88.
Katon W, Unutzer J, Wells K, Jones L. Collaborative depression care: history, evolution and ways to enhance dissemination and sustainability. Gen Hosp Psychiatry. 2010;32(5):456–64. https://doi.org/10.1016/j.genhosppsych.2010.04.001.
Krägeloh C, Czuba KJ, Billington DR, Kersten P, Siegert RJ. Using feedback from patient-reported outcome measures in mental health services: a scoping study and typology. Psychiatr Serv. 2014;66(3):224–41. https://doi.org/10.1176/appi.ps.201400141.
Lambert MJ, Whipple JL, Hawkins EJ, Vermeersch DA, Nielsen SL, Smart DW. Is it time for clinicians to routinely track patient outcome? A meta-analysis. Clin Psychol. 2003;10(3):288–301.
Guo T, Xiang YT, Xiao L, Hu CQ, Chiu HFK, Ungvari GS, et al. Measurement-based care versus standard care for major depression: a randomized controlled trial with blind raters. Am J Psychiatr. 2015;172(10):1004–13. https://doi.org/10.1176/appi.ajp.2015.14050652.
Valenstein M, Adler DA, Berlant J, Dixon LB, Dulit RA, Goldman B, et al. Implementing standardized assessments in clinical care: now’s the time. Psychiatr Serv. 2009;60(10):1372–5. https://doi.org/10.1176/ps.2009.60.10.1372.
Trivedi MH, Daly EJ. Measurement-based care for refractory depression: a clinical decision support model for clinical research and practice. Drug Alcohol Depend. 2007;88(Suppl 2):61–71.
Carlier IVE, Meuldijk D, Van Vliet IM, Van Fenema E, Van der Wee Nic JA, Zitman FG. Routine outcome monitoring and feedback on physical or mental health status: evidence and theory. J Eval Clin Pract. 2010;18(1):104–10. https://doi.org/10.1111/j.1365-2753.2010.01543.x.
Dowrick C, Leydon GM, McBride A, Howe A, Burgess H, Clarke P, et al. Patients’ and doctors’ views on depression severity questionnaires incentivised in UK quality and outcomes framework: qualitative study. BMJ. 2009;338:b663. https://doi.org/10.1136/bmj.b663.
Hawkins EJ, Lambert MJ, Vermeersch DA, Slade KL, Tuttle KC. The therapeutic effects of providing patient progress information to therapists and patients. Psychother Res. 2004;14(3):308–27. https://doi.org/10.1093/ptr/kph027.
Greenhalgh J. The applications of PROs in clinical practice: what are they, do they work, and why? Qual Life Res. 2009;18(1):115–23. https://doi.org/10.1007/s11136-008-9430-6.
Liu CF, Hedrick SC, Chaney EF, Heagerty P, Felker B, Hasenberg N, et al. Cost-effectiveness of collaborative care depression treatment in a primary care veteran population. Psychiatr Serv. 2003;54(5):698–704.
Chaney E, Bonner L, Vivell S, Cohen AN, Young AS, Rubenstein L. How behavioral healthcare informatics systems interface with medical informatics systems: a work in progress. In: Dewan NA, Luo JS, Lorenzi NM, editors. Information technology essentials for behavioral health clinicians. New York: Springer; 2011. p. 195–204.
Pincus HA, Page AEK, Druss B, Appelbaum PS, Gottlieb G, England MJ. Can psychiatry cross the quality chasm? Improving the quality of health care for mental and substance use conditions. Am J Psychiatr. 2007;164(5):712–9. https://doi.org/10.1176/ajp.2007.164.5.712.
Institute of Medicine. Crossing the quality chasm: a new health system for the 21st century. Washington, DC: National Academy Press; 2001.
Scott K, Lewis CC. Using measurement-based care to enhance any treatment. Cogn Behav Pract. 2015;22(1):49–59.
Kroenke K, Spitzer RL, Williams JBW. The PHQ-9 validity of a brief depression severity measure. J Gen Intern Med. 2001;16:606–13.
Spitzer RL, Kroenke K, Williams JW, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med. 2006;166(10):1092–7. https://doi.org/10.1001/archinte.166.10.1092.
Cacciola JS, Alterman AI, DePhilippis D, Drapkin ML, Valadez C, Fala NC, et al. Development and initial evaluation of the Brief Addiction Monitor (BAM). J Subst Abus Treat. 2013;44(3):256–63.
Blevins CA, Weathers FW, Davis MT, Witte TK, Domino JL. The posttraumatic stress disorder checklist for DSM-5 (PCL-5): development and initial psychometric evaluation. J Trauma Stress. 2015;28(6):489–98.
Bovin MJ, Marx BP, Weathers FW, Gallagher MW, Rodriguez P, Schnurr PP, et al. Psychometric properties of the PTSD checklist for diagnostic and statistical manual of mental disorders-fifth edition (PCL-5) in veterans. Psychol Assess. 2016;28(11):1379–91. https://doi.org/10.1037/pas0000254.
The Joint Commission. New outcome measures standard. https://www.jointcommission.org/accreditation/bhc_new_outcome_measures_standard.aspx. Accessed: 26 Feb 2018.
Zeiss AM, Karlin BE. Integrating mental health and primary care services in the Department of Veterans Affairs Health Care System. J Clin Psychol Med Settings. 2008;15(1):73–8.
Tew J, Klaus J, Oslin DW. The Behavioral Health Laboratory: building a stronger foundation for the patient-centered medical home. Fam Syst Health. 2010;28(2):130–45.
Rubenstein LV, Chaney EF, Ober S, Felker B, Sherman S, Lanto A, et al. Using evidence-based quality improvement methods for translating depression collaborative care research into practice. Fam Syst Health. 2010;28(2):91–113.
Pomerantz AS, Shiner B, Watts BV, Detzer MJ, Kutter C, Street B, et al. The White River model of colocated collaborative care: a platform for mental and behavioral health care in the medical home. Fam Syst Health. 2010;28(2):114–29. https://doi.org/10.1037/a0020261.
Chaney E, Rubenstein L, Liu CF, Yano E, Bolkan C, Lee M, et al. Implementing collaborative care for depression treatment in primary care: a cluster randomized evaluation of a quality improvement practice redesign. Implement Sci. 2011;6(1):121. https://doi.org/10.1186/1748-5908-6-121.
Wray LO, Szymanski BR, Kearney LK, McCarthy JF. Implementation of primary care-mental health integration services in the veterans health administration: program activity and associations with engagement in specialty mental health services. J Clin Psychol Med Settings. 2012;19(1):105–16.
Nelson EC, Godfrey MM, Batalden PB, Berry SA, Bothe AE Jr, McKinley KE, et al. Clinical microsystems, part 1. The building blocks of health systems. Jt Comm J Qual Patient Saf. 2008;34(7):367–78.
Charns MP, Young GJ. Organizational design and coordination. In: Burns LR, Bradley EH, Weiner BJ, editors. Shortell and Kaluzny’s healthcare management: organization, design and behavior. Clifton Park: Delmar Publishing; 2011. p. 64–90.
Felker BL, Chaney E, Rubenstein LV, Bonner LM, Yano EM, Parker LE, et al. Developing effective collaboration between primary care and mental health providers. Prim Care Companion J Clin Psychiatry. 2006;8(1):12–6.
Fickel JJ, Parker LE, Yano EM, Kirchner JE. Primary care - mental health collaboration: an example of assessing usual practice and potential barriers. J Interprof Care. 2007;21(2):207–16.
Stetler CB, Legro MW, Rycroft-Malone J, Bowman C, Curran G, Guihan M, et al. Role of “external facilitation” in implementation of research findings: a qualitative evaluation of facilitation experiences in the veterans health administration. Implement Sci. 2006;1:23. https://doi.org/10.1186/1748-5908-1-23.
Bidassie B, Williams LS, Woodward-Hagg H, Matthias MS, Damush TM. Key components of external facilitation in an acute stroke quality improvement collaborative in the veterans health administration. Implement Sci. 2015;10(1):69. https://doi.org/10.1186/s13012-015-0252-y.
Dogherty EJ, Harrison MB, Graham ID. Facilitation as a role and process in achieving evidence-based practice in nursing: a focused review of concept and meaning. Worldviews Evid-Based Nurs. 2010;7(2):76–89. https://doi.org/10.1111/j.1741-6787.2010.00186.x.
Kirchner JE, Ritchie MJ, Pitcock JA, Parker LE, Curran GM, Fortney JC. Outcomes of a partnered facilitation strategy to implement primary care-mental health. J Gen Intern Med. 2014;29(Suppl 4):904–12. https://doi.org/10.1007/s11606-014-3027-2.
Ritchie MJ, Parker LE, Kirchner JE. Using implementation facilitation to foster clinical practice quality and adherence to evidence in challenged settings: a qualitative study. BMC Health Serv Res. 2017;17:294. https://doi.org/10.1186/s12913-017-2217-0.
Dogherty EJ, Harrison MB, Baker C, Graham ID. Following a natural experiment of guideline adaptation and early implementation: a mixed-methods study of facilitation. Implement Sci. 2012;7:9. https://doi.org/10.1186/1748-5908-7-9.
Lessard S, Bareil C, Lalonde L, Duhamel F, Hudon E, Goudreau J, et al. External facilitators and interprofessional facilitation teams: a qualitative study of their roles in supporting practice change. Implement Sci. 2016;11(1):1–12. https://doi.org/10.1186/s13012-016-0458-7.
Parker LE, de Pillis E, Altschuler A, Rubenstein LV, Meredith LS. Balancing participation and expertise: a comparison of locally and centrally managed health care quality improvement within primary care practices. Qual Health Res. 2007;17(9):1268–79. https://doi.org/10.1177/1049732307307447.
Norris JM, White DE, Nowell L, Mrklas K, Stelfox HT. How do stakeholders from multiple hierarchical levels of a large provincial health system define engagement? A qualitative study. Implement Sci. 2017;12:98. https://doi.org/10.1186/s13012-017-0625-5.
Yin RK. Case study research: design and methods. 5th ed. Thousand Oaks: Sage Publications, Inc.; 2014.
Glasgow RE, McKay HG, Piette JD, Reynolds KD. The RE-AIM framework for evaluating interventions: what can it tell us about approaches to chronic illness management? Patient Educ Couns. 2001;44(2):119–27.
Harvey G, Kitson A. Implementing evidence-based practice in healthcare: a facilitation guide. London: Routledge; 2015.
Harvey G, Kitson A. PARIHS revisited: from heuristic to integrated framework for the successful implementation of knowledge into practice. Implement Sci. 2016;11(1):1–13. https://doi.org/10.1186/s13012-016-0398-2.
Gaglio B, Shoup JA, Glasgow RE. The RE-AIM framework: a systematic review of use over time. Am J Public Health. 2013;103(6):e38–46. https://doi.org/10.2105/AJPH.2013.301299.
Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89(9):1322–7.
Harvey G, Loftus-Hills A, Rycroft-Malone J, Titchen A, Kitson A, McCormack B, et al. Getting evidence into practice: the role and function of facilitation. J Adv Nurs. 2002;37(6):577–88.
Stock R, Mahoney E, Carney PA. Measuring team development in clinical care settings. Fam Med. 2013;45(10):691–700.
Gagnon MP, Attieh R, Ghandour el K, Legare F, Ouimet M, Estabrooks CA, et al. A systematic review of instruments to assess organizational readiness for knowledge translation in health care. PLoS One. 2014;9(12):e114338. https://doi.org/10.1371/journal.pone.0114338.
Hamilton AB, Cohen AN, Young AS. Organizational readiness in specialty mental health care. J Gen Intern Med. 2010;25(Suppl 1):27–31.
Chinman M, Daniels K, Smith J, McCarthy S, Medoff D, Peeples A, et al. Provision of peer specialist services in VA patient aligned care teams: protocol for testing a cluster randomized implementation trial. Implement Sci. 2017;12(1):57.
Lehman WEK, Greener JM, Simpson D. Assessing organizational readiness for change. J Subst Abus Treat. 2002;22:197–209.
Beehler GP, Funderburk JS, King PR, Wade M, Possemato K. Using the primary care behavioral health provider adherence questionnaire (PPAQ) to identify practice patterns. Transl Behav Med. 2015;5(4):384–92. https://doi.org/10.1007/s13142-015-0325-0.
Beehler GP, Funderburk JS, Possemato K, King PR, Maddoux JA. Measuring practice variation in integrated primary care: validation of the PPAQ-2. Poster presented at the Society of Behavioral Medicine’s 39th Annual Meeting and Scientific Sessions, New Orleans, LA. 11-14 Apr 2018.
Salem-Schatz S, Ordin D, Mittman B. Guide to the Team Development Measure: Center for Implementation Practice and Research Support. https://www.queri.research.va.gov/tools/TeamDevelopmentMeasure.pdf. Accessed: 2 Apr 2018.
Harding KJ, Rush AJ, Arbuckle M, Trivedi MH, Pincus HA. Measurement-based care in psychiatric practice: a policy framework for implementation. J Clin Psychiatry. 2011;72(8):1136–43.
Jensen-Doss A, Hawley KM. Understanding barriers to evidence-based assessment: clinician attitudes toward standardized assessment tools. J Clin Child Adolesc Psychol. 2010;39(6):885–96.
Boswell JF, Kraus DR, Miller SD, Lambert MJ. Implementing routine outcome monitoring in clinical practice: benefits, challenges, and solutions. Psychother Res. 2015;25(1):6–19.
Gleacher AA, Olin SS, Nadeem E, Pollock M, Ringle V, Bickman L, et al. Implementing a measurement feedback system in community mental health clinics: a case study of multilevel barriers and facilitators. Admin Pol Ment Health. 2016;43(3):426–40.
Bauer MS, Miller C, Kim B, Lew R, Weaver K, Coldwell C, et al. Partnering with health system operations leadership to develop a controlled implementation trial. Implement Sci. 2016;11(1):22. https://doi.org/10.1186/s13012-016-0385-7.
Mitchell P, Wynia M, Golden R, McNellis B, Okun S, Webb CE, et al. Core principles and values of effective team-based health care. Discussion paper, Institute of Medicine, Washington, DC. https://nam.edu/wp-content/uploads/2015/06/VSRT-Team-Based-Care-Principles-Values.pdf. Accessed 6 Apr 2018.
Berta W, Cranley L, Dearing JW, Dogherty EJ, Squires JE, Estabrooks CA. Why (we think) facilitation works: insights from organizational learning theory. Implement Sci. 2015;10:141. https://doi.org/10.1186/s13012-015-0323-0.
Roderick SS, Burdette N, Hurwitz D, Yeracaris P. Integrated behavioral health practice facilitation in patient centered medical homes: a promising application. Fam Syst Health. 2017;35(2):227–37. https://doi.org/10.1037/fsh0000273.
This study is supported by a grant from the Department of Veterans Affairs, Quality Enhancement Research Initiative (QUERI), QUE 15-289. The funding body had no involvement in the design of the study, in the collection, analysis, and interpretation of data, or in writing the manuscript. The views expressed in this article are those of the authors and do not represent the views of the Department of Veterans Affairs or the United States Government.
Ethics approval and consent to participate
This study was approved by the VA Central Institutional Review Board (#17–25). Prior to conducting data collection activities, we will provide study subjects with the elements of informed consent and will obtain their consent to participate per the approved protocol.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
About this article
Cite this article
Wray, L.O., Ritchie, M.J., Oslin, D.W. et al. Enhancing implementation of measurement-based mental health care in primary care: a mixed-methods randomized effectiveness evaluation of implementation facilitation. BMC Health Serv Res 18, 753 (2018). https://doi.org/10.1186/s12913-018-3493-z
- Measurement-based care
- Integrated primary care
- Mental health
- Primary care