Skip to main content

Proactive integrated virtual healthcare resource use in primary care

A Correction to this article was published on 09 October 2021

This article has been updated



Proactive integrated virtual healthcare resource (VHR) use can improve efficiency, maximize resource capacity for delivering optimal coordinated care and improve patient outcomes. Proactive integrated VHR use is vital for delivering high quality care. Our objectives were to identify proactive integrated VHR use among primary care teams, best practices and targeted implementation strategies to promote proactive integrated VHR use.


This is a mixed-method descriptive study. We employed a community-based participatory approach to collect data and the Consolidated Framework for Implementation Research to analyze and contextualize findings. A cross-sectional sample of primary care team members (n = 65) from a Department of Veterans Affairs medical center participated in focus groups, follow-up interviews (n = 16), and respond to self-report surveys. Operational subject matter experts (n = 15) participated in informant interviews.


Survey data described current use and factors that influenced singular VHR use and were convergent with qualitative findings. Focus group and interview data described no evidence of proactive integrated VHR use. Differences and similarities were identified between both utilization groups, such as facilitators and barriers, recommendations, patient education and preferred implementation strategies. All groups reported issues around VHR availability knowledge and access and functionality. Participants identified the need for best practices that are specific to care tasks and performance measures. Expert informant interviews identified a list of VHR tools that could be proactively integrated across the healthcare continuum.


Health systems are leveraging technologies to proactively integrate VHR to maximize information exchange, clinical decision support and patient engagement. VHR is critical during global pandemics, such as COVID-19, to maintain access to care coordination and delivery while abiding by public health recommendations. Though recent requirements for reducing contact create an intrinsic motivation, cultural change through education and best practices of proactive integrated use across the healthcare continuum is needed to create a culture of VHR super users.

Peer Review reports


Virtual healthcare resources (VHR), also known as eHealth and telehealth [1], are central to strategic priorities for ensuring receipt of timely, integrated, patient-centered care. VHR consist of many resources including patient health portals, mobile technology, electronic health records, and telehealth including video, audio, instant and secure messaging [1]. Healthcare systems are taking strategic steps to promote VHR use to support access, care coordination and delivery needs through an extensive network of resources, to improve patient outcomes and promote efficient system utilization [2]. Healthcare systems have established VHR as an option for care delivery. In light of COVID-19, most systems are now relying on VHR to coordinate and deliver high quality care while reducing contact [3]. The imminent need for proactive integrated VHR use in healthcare systems warrants investigation.

Proactive integrated VHR use is defined as a self-initiated approach to coordinated use of applicable VHR systems for the purposes of coordinating and delivering timely high-quality patient-centered care [4]. Fragmented systems can frustrate primary care teams, and reduce patients’ adoption of VHR and satisfaction with care [5, 6]. VHR is most effective when tools are used in an integrated manner which improves workflow efficiency [7]. Provider promotion and proactive reinforcement are key factors in patient adoption and sustained use of VHR [5, 6]. Understanding best practices of VHR proactive integrated use can optimize VHR implementation efforts.

Primary care typically includes a physician, nurse, clinical and clerical associates [8]. This primary care team can extend to staff members from specialty areas of service, such as pharmacy, nutrition, social work and psychology. Since implementation of primary care teams in the Department of Veterans Affairs (VA), use of VHR has increased, yet little is known about the integrated use of VHR. Currently there are no established best practices to guide clinical practice [9]. The aims of this research were to: (1) examine primary care use and perceptions of VHR among high- and low- utilization groups; (2) determine the level of integration of VHR in primary care, and identify best practices; and (3) identify implementation of strategies to promote primary care use and integration of VHR across the health care continuum [10].

Theoretical context

A community-based participatory research (CBPR) approach guided data collection and the Consolidated Framework for Implementation Research (CFIR) [11] informed analyses. This study was developed using CBPR [12]. During proposal development team members actively developed collaborative partnerships with primary care team members. This included meetings, resource sharing, networking and open discussions of needs and barriers. The study team intentionally hired clinicians with long term relationships with primary care teams and a shared language to interpret communication as needed. This fledgling group helped to further define the study problem in language that all understood. Partnerships were documented through study documents, meeting minutes and regulatory forms. Given the duration of this study and partnerships, open discussions, feedback and data interpretation were common in weekly meetings. Community partners were involved in all dissemination activities including development of ideas, presentations, video production and manuscripts. Primary care team members (i.e. participants) were actively involved in development of study protocols, data collection, and co-creation and implementation of the intervention [13]. CFIR constructs were used to guide data analysis, document primary care team VHR use, and strengthen the research and applicability of findings [14] to broader implementation efforts.


This paper reports findings from an Institutional Review Board approved concurrent mixed-method implementation study [10].


This study was conducted within primary care at one VA medical center, located in the southeastern United States. This VA is comprised of six sites: main hospital, primary care annex, and four community-based outpatient clinics.


A complete description of participant selection and recruitment is published [10]. In short, using outgoing Secure Messages as a proxy for VHR use, primary care teams were ranked “high” or “low” utilization. Rankings were confirmed by clinical experts following a CBPR approach. Active and passive methods were used to recruit primary care team members (N = 267) [15]. Snowball sampling identified extended primary care team members (e.g., pharmacy, social work). Expert informants (n = 15), consisting of operational partners and local leaders, were purposively recruited based on their knowledge of VHR systems. Potential participants who did not participate cited reasons such as lack of time or inappropriate fit to study aims.


Of the 57 existing teams, 19 were purposively identified based on VHR utilization (10 high [n = 10]; low [n = 9]). Twenty-one clinic-held focus groups were conducted: 19 high and low groups and 2 extended primary care team focus groups. Of the 267 primary care team members 65 provided written consent to participate. Approximately four primary care team members (i.e. physician, nurse, clinical and clerical associate) participated in each focus group. Following a CBPR approach, we approached focus group participants who provided insightful information to participate in a follow-up interview to provide additional information regarding integration of VHR among other topics. Follow-up interviews were conducted with focus group participants (n = 16) at the place of their choice (e.g. clinic). Expert informants (n = 15) participated in an individual telephone interview. Only participants and researchers were present for data collection activities.

Data sources

Data collection measures (survey, focus group discussion and interview guides) were developed for this study and have been previously piloted, described and published (6). The demographic questionnaire (Additional File 1) included 16 items and the CFIR-informed self-report survey (Additional File 2) contained 18 items. A CFIR-informed focus group guide (Additional File 3) included a list of implementation strategies [16]. Participants identified preferred VHR implementation strategies. A semi-structured interview guide (Additional File 4) was used for follow-up interviews and expert informants.

Data collection procedures

Prior to data collection, research staff met with all sites to introduce individual researchers (e.g., name, educational background), provide information about the study and explicitly develop rapport. CM had existing professional relationships with several primary care team members. Study demographic and VHR use survey data were collected August through November 2018 and managed using REDCap electronic data capture tools hosted at Veterans Affairs [17, 18]. REDCap (Research Electronic Data Capture) is a secure, web-based software platform designed to support data capture for research studies, providing 1) an intuitive interface for validated data capture; 2) audit trails for tracking data manipulation and export procedures; 3) automated export procedures for seamless data downloads to common statistical packages; and 4) procedures for data integration and interoperability with external sources. Qualitative data collection was conducted by study staff including BC and CM. All interviews and focus groups were audio recorded with permission. December through February 2018 fifty-five minute in-person, clinic-based focus groups were conducted to identify proactive integrated VHR use, best practices and targeted implementation strategies; 30-min follow-up interviews were conducted for clarification and individual perspective. March through May 2019, 30-min expert informant interviews were conducted via telephone to elicit a future vision for an integrated VHR system. Interviewers electronically shared a graphic representation of the healthcare continuum with expert informants (Fig. 1). Expert informants were asked to modify this continuum based on their knowledge of organizational capacity, clinical needs, and patient preferences.

Fig. 1
figure 1

Healthcare continuum and relevant virtual healthcare resources

Data analysis

Descriptive statistics were used by VP to analyze the demographic questionnaire to describe the overall sample and high and low utilization group characteristics. Group means and standard deviations (SD) were also computed for each subscale described in Table 3 by averaging responses across items and across responders; lower scores signified more positive perceptions. Given the size of our samples (High utilization group [n = 23] and Low utilization group [n = 29]) comparative analyses (t-test and chi-square) of survey data were supplemented by effect size calculations. We calculated Cohen’s d and h effect sizes and 95% confidence intervals [19], commonly used to evaluate differences between two independent means and two independent proportions, respectively. Analyses were performed with SPSS Version 25 (SPSS Statistics for Windows, n.d.) to identify group differences between participants associated with high and low utilization user groups. Across all variables, 42.9% had no missing data, 28.6% had < 5 participants with missing data (< 10%), 25.9% had < 10 participants with missing data (< 20%). The remaining 3 variables had 11–18 participants with missing data. Overall, 384 (6.59%) of 5824 values were missing completely at random. Therefore, the default listwise deletion was deemed acceptable.

Qualitative data collection and analysis were conducted by BC and CM and research staff concurrently. Insights from data analyses were used to guide subsequent data collection. A rapid analysis process [20] was used to analyze interview data. Data were managed within Microsoft Excel. Content analysis was conducted by four research team members who read d participant comments, coded the data deductively by CFIR constructs, and merged similar codes to identify themes [21,22,23,24]. Data were stratified by high and low utilization groups and compared to one another. Matrix analysis was used to analyze across-thematic categories [25]. Clinical team members and expert informants verified findings and provided clarification.


Survey findings: utilization patterns and beliefs about VHR

Participants represented all roles on a primary care team and were mainly white females over 46 years old (Table 1). Survey data described participants’ VHR use, perceptions of patient’s preferred methods of communication, and promotion of VHR with their patients.

Table 1 Primary Care Participant Characteristics

Most participants reported using and promoting My HealtheVet, Secure Messaging, and VetLink Kiosks with patients, across low and high user groups. Although telehealth use was reported by nearly 50% of all providers, mobile apps use was the lowest used resource across participant groups (Table 2). More than two thirds in both utilization groups reported using VHR tools daily (Low group = 69% yes daily; High group = 78.3% yes daily; 2 = 0.58, p = 0.57, h (95%CI) = 0.21 (0.14,0.29)) and over two thirds of the providers across groups have been using VHR for over 3–5 years and more (Low group = 65.5% yes 3–5+ years; High group = 65.2% yes 3–5+ years; 2 = 0.65, p = 0.52, h (95%CI) = 0.01 (− 0.07,0.08).

Table 2 Use and promotion of virtual healthcare resources (VHR) among providers from high versus low utilization groups

Providers from both user groups perceived patients’ preferred methods of communication to be primarily telephone, face to face, and Secure Messaging, followed by My HealtheVet tools (below 50%), with telehealth, VetLink Kiosks, and apps reported as the least preferred among patients (Table 2). Although none of the group differences in reported patient’s preferences reached traditional statistical significance threshold, high utilization providers reported more elevated face to face and Secure Messaging preferences among their patients relative to low utilizing providers. These differences did not reach significance levels (p > .05) however the observed medium effect sizes (h > .50) suggest that the analysis was under powered to detect even a medium effect. Survey data suggest promotion of Secure Messaging followed by My HealtheVet was high among all providers, with more than 85% of providers from both high and low utilizing groups endorsing these tools. Even promotion of telehealth and VetLink Kiosk use was relatively high across providers (Table 2).

Participants perceived relative advantage (i.e. improved care delivery, preference over traditional tools), observability (i.e., improved clinical workflow and patient outcomes), and compatibility (i.e. how well VHR fits with primary care team values and workflow systems) for the use of My HealtheVet and Secure Messaging; however, data suggests provider perceptions of telehealth and VetLink Kiosks were more mixed in terms of their perceived relative advantage in patient care and workflow (Table 3). Survey data captured perceptions of complexity (i.e. perceived difficulty of implementing use of VHR) through two subscales: Compatibility and Complexity. Responses suggest while providers from low use groups consistently reported comfort with utilization of My HealtheVet and Secure Messaging, providers from high use groups reported more variability in comfort with both My HealtheVet and Secure Messaging. Although differences between providers from high and low use groups did not reach traditional statistical significance thresholds (i.e., p < .05), effect sizes crossed the medium threshold for My HealtheVet and Secure Messaging suggesting meaningful differences between the groups on perceptions about compatibility of VHR use. Next, the complexity subscale showed that, on average, providers reported ease with integrating all VHR and that they would benefit from education on how to access tools with patients (Table 3). Finally, the context and facilitation (i.e., use reflects role responsibilities, reinforced by workplace) survey subscale highlighted My HealtheVet and Secure Messaging use across both groups; use was reinforced in the workplace; and both were most often used and recommended for patient use. Again, reports varied related to telehealth, VetLink Kiosks, and apps (Table 3).

Table 3 Perceptions about virtual healthcare resources (VHR) among providers from high versus low use teamlets

Focus group and follow-up interview findings

Focus group and follow-up interview findings for primary care and extended primary care members were consistent and thus combined for results presentation. Of key interest was the differences between high and low user groups, surprisingly, differences were minimal, but some did emerge, such that: (1) low user groups tended to report using a more structured process for VHR patient education than high user groups; (2) high user groups identified the need for properly functioning, integrated and accessible equipment, while low user groups recommended policy/system changes such as work load credit, workflow fit, decreased workload, safety nets for communication with patients and dedicated staff to educate patients; (3) high user groups identified factors which inhibit VHR use, such as lack of accessibility and function, while low user groups focused on factors such as individual patient needs and preferences, and staff lack of knowledge about VHR; (4) high user groups identified ways to overcome barriers, such as mandates and awareness of VHR benefits (e.g. less trips to clinic, less phone calls, redistributed work), while low user groups identified process structures (e.g. triage processes, follow-up processes) to overcome barriers; and (5) though both groups identified education and training as preferred implementation strategies, low user groups identified more implementation strategies to support VHR use (Table 4).

Table 4 High and Low Utilization Groups’ Most Useful Recommended Implementation Strategies

Qualitative analysis also identified factors that emerged relevant to VHR use among all focus groups (high and low utilization). Relevant data findings are presented by CFIR constructs: characteristics of individuals, intervention characteristics, inner setting, and outer setting.

Characteristics of individuals

Knowledge and Beliefs about the Intervention was a factor associated with VHR use. Participants identified using Secure Messaging, telehealth and VetLink Kiosks for patient care. Comfort using VHR varied across groups. Those comfortable using VHR use among those with comfort and knowledge created ease in their work flow as reported by one nurse, ‘We use things (VHR) and we don’t know we are using them…it’s like breathing’ .Participants did not report knowing how to proactively integrate VHR. Participants were more likely to report using VHR when they believed it benefited their patients, improved patient-provider relationships, and enhanced their clinical workflow. One physician reported, “Secure messaging allows for problem-focused response”, saving his time and improving workflow. Though the project was focused on all available VHR, participants consistently described using Secure Messaging and described coordinated use of Secure Messaging (Fig. 2). Participants reported the need for education about coordinating proactive integration of VHR.

Fig. 2
figure 2

PACT coordinated use of Secure Messaging. RN = Registered Nurse. LPN = Licensed Practical Nurse

Intervention characteristics

Five intervention characteristics [14] that impact VHR use were: relative advantage, adaptability,complexity, and design quality and packaging, and cost. Relative Advantage was germane to participants, who suggested VHR use provided expedited and improved continuity of care which increased patient satisfaction. One physician reported using VHR made for more efficient work. ‘“(VHR) allows providers to work to the top of their licenses” thus allowing registered nurses, advanced practice registered nurses, and physicians the ability to work to the scope of their license, no longer confined by inefficient processes that limit licensed providers’ actions. Adaptability was relevant, participants indicated a need for VHR to be tailored and integrated to support workflow needs, though several focus groups did report efforts to create processes to integrate VHR into workflow. Complexity was identified as a system-based issue by participants, such that the user experience can be improved through streamlining VHR for ease of navigation and integration. Design quality and packaging was a concern reported among participants. Participants cited usability issues and inability to easily navigate between VHR platforms, suggesting efforts to better package and integrate the suite of available VHR as necessary to support their integrated use. Cost of time was a clear challenge for participants in their motivated use of VHR as evidenced by reports of negative impacts on workflow and the continued need for team member and patient education.

Inner setting

Data gleaned relevant to inner setting, included networks and communications, compatibility, available resources, access to knowledge and information. Networks and communications emerged as relevant to participants, resulting in team-based development of processes to adjust to workflow impacts. Participants indicated VHR can both positively - and negatively - affect workflow. Though participants stated workflow could be improved by using multiple tools, their workflow was also congested because these systems are often siloed, cumbersome to navigate (e.g., number of clicks) in and between systems (e.g., electronic health record, telehealth), and sometimes malfunctioned (e.g., VetLink Kiosk). Compatibility was relevant, as participants reported VHR use aligned with team member and patient norms and values, however the technology was often incompatible within existing workflows. Available resources were noted as an issue, such as the need for dedicated time, training, and education, for team members and patients to support adoption and motivated VHR use. Access to knowledge and information was needed, particularly about best practices and specific use cases/scenarios that are driven by care tasks and performance measures. Several nurses suggested a ‘hear it, see it, and do it’ approach to sharing VHR knowledge and information.

Outer setting

Outer setting constructs relevant to participants’ VHR use were patients’ needs and resources and external policy and incentives. Specific to patients’ needs and resources, participants reported encouraging patients to use patient facing tools, however, they reported patient communication preferences to be age dependent, such that older patients preferred phone calls or walk-in appointments. Most teams identified VHR education as part of their routine patient care. Promoting patient use of VetLink Kiosks and telehealth was rarely coordinated as a team, but instead fell to individual team members. When VHR was not a viable option for a patient or the clinic, teams provided alternatives, such as telephone calls. Some primary care teams reported having limited access to Wi-Fi. These teams tended to rely less on VHR answering patients’ questions at the front desk and during walk-in appointments. External Policy and Incentives emerged as a construct of relevance, though low user groups more often recommended system changes such as workload credit and dedicated staff to educate patients. These incentives were identified as a critical need to promote proactive integrated VHR use.

Implementation strategies for increasing proactive integrated VHR use

Focus group participants identified several implementation strategies to support adoption and VHR use, for example: patient and provider education on VHR; dedicated on-site technical assistance and additional resources to assist with enrollment, and patient identity authentication; and patient-facing test user accounts for providers who may otherwise have no firsthand knowledge of the patient-side of VHR. There were key differences in user groups’ promotion of implementation strategies (Table 4) [16]. In general, the most useful strategies reported were increasing educational approaches (e.g. hands on), changing record systems to assess implementation and clinical outcomes, centralizing technical assistance, assessing locations for readiness, and identifying barriers and facilitators.

Expert informant interviews

Expert informant interviews focused on the identification and review of twenty-three VHR tools that could be used across the healthcare continuum (Fig. 1). The right side of Fig. 1 includes a numbered list of VHR participants reported using at various phases of the healthcare continuum. VHR are further described in Table 5 including whether the VHR is veteran or provider facing. Note some VHR faces both veteran and provider. This numbered list was referenced as experts identified tasks and VHR that could be proactively integrated at each phase of the healthcare continuum (Table 6). The double line in the table delineates additional VHR that are currently available to VA staff. Each of these VHR are designated with an asterisk. It is our hope that this table, though not exhaustive, will provide an inventory of many VHR available. Expert informants suggested proactive integrated use of VHR will be predicated on several important factors: (1) primary care teams must view VHR as therapeutic tools and proactively “prescribe” them accordingly; (2) provider-facing VHR must complement existing documentation requirements and tools; and (3) an integrated system of provider and patient generated health data is critical.

Table 5 Virtual Healthcare Resources.
Table 6 VHR use within the healthcare continuum


Identifying proactive integrated VHR use best practices is imperative. Proactive integrated VHR use is complex with several factors, including coordination across multiple tools, which can be used across the healthcare continuum. This study aimed to understand how primary care team members used VHR through qualitative and quantitative data. Though teams reported using some VHR, teams did not report using multiple VHR in a proactive integrated way. Use of singular systems was the norm; often use was reactive. Differences and similarities were identified between high- and low- utilization groups in the qualitative findings. Similar to previous publications, all groups described concerns with accessibility, functionality and VHR knowledge [26, 27]. Even though best practices for many primary care tasks and performance measures exist in the literature [28,29,30,31,32], participants from both utilization groups identified the need for VHR best practices that are specific to care tasks (e.g., history taking and consultations) and performance measures (e.g., smoking cessation and chronic pain). These VHR best practices have not been established in the literature [9]. Previous research has shown primary care physician’s familiarity and comfort with the electronic patient-portal influence portal use [33]. These previous findings are in alignment with our findings that teams want to know what VHR patients have access to and how they use it. Due to the pandemic, providing remote care has become a necessity. These findings may inform future implementation efforts.

Data suggest a need to identify and disseminate best practices that can be applied across the healthcare continuum. Interview data presented options for proactively integrating VHR as a standard of care across the healthcare continuum. To our knowledge, proactive integration of VHR across the healthcare continuum does not exist in the literature, making this a novel finding. Table 5 is not an exhaustive inventory, but it is representative of the kinds of information and tools available to primary care teams, needed to engage in proactive integrated VHR use. This information can inform the production of materials and training (i.e., toolkits) to increase awareness, motivate VHR use and initiate a culture change.

The final aim of this study was to identify implementation strategies to promote primary care team use and integration of VHR across the healthcare continuum. Participants identified education and training implementation strategies as a critical need. To our knowledge, there is currently no training content that supports the proactive integrated use of VHR or best practices within the VA. Though focus group participants reported perceiving older patients do not prefer using VHR, previous research has found this to be a misconception [6, 34] For patients who are hesitant to adopt VHR, team members will benefit from efforts to support and reinforce patients’ motivated use and perceived value of VHR [5]. Implementation strategies need to focus on awareness building to dispel this misperception.

A common way to package implementation strategies and best practices is development of a toolkit [35,36,37]. Though not highlighted in this paper, the parent study used findings from this study aim to develop, and disseminate, a VA primary care team toolkit. This toolkit is currently being finalized and will be accessible via the VA Office of Connected Care.

Finally, consistent with the literature, participants stated their motivation to work on VHR tasks was negatively influenced by time burden [38]. Workload credit and incentives to proactively integrate VHR into the coordination and delivery of care may provide the necessary motivation for adoption and sustained use over time [6, 39].

Strengths and limitations

This study contributes to the science by providing: (1) current use and potential for proactive integration of VHR into daily workflow; and (2) identification of team focused implementation strategies to increase uptake and proactive integrated use of electronic health resources. The use of CBPR and mixed methods in this study informed primary care team experiences with VHR and incorporated their expertise through the research process. Biases during all phases of the study were limited by integrating participant perspectives, and using mixed methods to triangulate findings [40]. Study team has extensive research experience. BC and CM have worked closely for several years, studying qualitative design and methods. VP has over 5 years’ experience designing innovative data analysis plans, including triangulation. JH worked closely with WM and BZ to conceptualize, design and implement this study.

Limitations should be considered when interpreting findings. First, data represents VHR use at one site, however this site is a large, teaching hospital that treated nearly 100,000 Veterans and conducted over one million outpatient visits at the time of this study. Second, while the sample size was comparable to other qualitative studies [41], based on a representative sample of participants and saturation of data, findings may not be generalizable beyond primary care within comparable healthcare systems. Third, though we purposively recruited primary care members, we may have missed valuable insights from other clinical groups. To address this limitation in part, we expanded recruitment to include primary care pharmacy, nutrition, psychology, and social work. Fourth, though we used CFIR to analyze and contextualize data, it may have biased the lens by which data was interpreted. The CBPR approach and analysis by multiple qualitative experts increases the validity of data collection and findings. Fifth, though utilization groups were sorted by using high and low Secure Messaging as a proxy for VHR use, it is notable they reported no difference in the participant survey – this is likely because the survey queried a dichotomous response of yes/no, not quantity of use. As such, the survey is limited in its usefulness to measure the frequency of Secure Messaging use.

Future research, education and policy

Findings from this project may inform future research on the use of VHR in specialty care (e.g. rehabilitation, mental health, cardiology). Also, COVID-19 presents as a natural experiment to examine the increased motivation and necessity for proactively integrating VHR into the health care delivery process, research efforts should carefully examine VHR use within the new era of care that emphasizes reducing contact and remote access. Future research should include large scale implementation studies that include clinical and implementation outcomes. As implementation efforts continue, research should focus on the identification and dissemination of best practices. There is continued need for dynamic education to support primary care team proactive integrated VHR use across the healthcare continuum, with emphasis on specific care tasks and performance measures [42]. Education is needed to dispel primary care team member misperceptions about patient preferences for VHR, as well as specialized training to support patient adoption of VHR. Finally, workload credit and incentives should be considered in future research and policy efforts [42].


Health systems are leveraging technologies to proactively integrate VHR, and maximize information exchange, clinical decision support and patient engagement. Proactive integrated use of VHR is not common practice among primary care teams. Care teams want to know how to proactively integrate VHR into daily workflow, and they want workload credit for incorporating VHR use into healthcare coordination and delivery. Our work to provide trainings, resources and a toolkit helped to improve care team knowledge of proactive integrated use of VHR. Future efforts should build on this toolkit, making VHR and its proactive integrated use common place across primary care teams. Though recent requirements for reducing contact create an intrinsic motivation, cultural change through education and best practices of proactive integrated use across the healthcare continuum is needed to create a culture of VHR super users. Future efforts can work to create culture change and understand proactive integrated use of VHR while maintaining access to comprehensive high-quality care.

Availability of data and materials

The datasets during and/or analyzed during the current study available from the corresponding author on reasonable request.

Change history



virtual healthcare resource


Department of Veterans Affairs


community-based participatory research


Consolidated Framework for Implementation Research


standard deviation




confidence interval


Computerized Patient Record System


Joint Legacy Viewer


Veterans Health Information Systems and Technology Architecture


  1. Department of Veterans Affairs. (2019, February 19). EHealth/telehealth. HSR&D Research Topics.

  2. Cipriano PF, Bowles K, Dailey M, Dykes P, Lamb G, Naylor M. The importance of health information technology in care coordination and transitional care. Nurs Outlook. 2013;61(6):475–89.

    Article  PubMed  Google Scholar 

  3. Salway RJ, Silvestri D, Wei EK, Bouton M. Using information technology to improve COVID-19 care at new York City health+ hospitals: commentary describes new York City health+ hospitals use of innovative technological solutions to respond to the COVID-19 pandemic. Health Aff. 2020;39(9):1601–4.

    Article  Google Scholar 

  4. Haun JN. Implementation strategies to promote proactive integrated use of virtual resources: overcoming barriers to moving HIT interventions into clinical practice. Virtual: VA RR&D Center Director/REAP meeting; 2021, April 27.

    Google Scholar 

  5. Haun JN, Chavez M, Nazi K, Antinori N, Melillo C, Cotner BA, et al. Veterans’ preferences for exchanging information using veterans affairs health information technologies: focus group results and modeling simulations. J Med Internet Res. 2017;19(10):e359.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Haun JN, Lind JD, Shimada SL, Martin TL, Gosline RM, Antinori N, et al. Evaluating user experiences of the secure messaging tool on the veterans affairs’ patient portal system. J Med Internet Res. 2014;16(3):e75.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Meaker R, Bhandal S, Roberts CM. Information flow to enable integrated health care: integration or interoperability. Br J Gen Pract. 2018;68(668):110–1.

    Article  PubMed  PubMed Central  Google Scholar 

  8. LaVela SL, Hill JN. Re-designing primary care: implementation of patient-aligned care teams. Healthcare. 2014;2(4):268–74.

    Article  PubMed  Google Scholar 

  9. Mohammed SA, Yusof MM. Towards an evaluation framework for information quality management (IQM) practices for health information systems–evaluation criteria for effective IQM practices. J Eval Clin Pract. 2013;19(2):379–87.

    Article  PubMed  Google Scholar 

  10. Haun J, Chavez M, Hathaway W, Antinori N, Melillo C, Cotner BA, et al. Virtual medical modality implementation strategies for patient-aligned care teams to promote veteran-centered care: protocol for a mixed-methods study. JMIR Research Protocols. 2018;7(8):e11262.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implement Sci. 2009;4(1):50.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Israel BA, Eng E, Schulz AJ, Parker, E. A. (Eds.). Methods for community-based participatory research for health. 2nd ed: Jossey-Bass; 2012.

  13. Huang R, Spector JM, Yang J. Educational technology: A primer for the 21st century: Springer; 2019.

  14. Birken SA, Powell BJ, Presseau J, Kirk MA, Lorencatto F, Gould NJ, et al. Combined use of the consolidated framework for implementation research (CFIR) and the theoretical domains framework (TDF): A systematic review. Implement Sci. 2017;12(1):2.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Estabrooks P, You W, Hedrick V, Reinholt M, Dohm E, Zoellner J. A pragmatic examination of active and passive recruitment methods to improve the reach of community lifestyle programs: the talking health trial. Int J Behav Nutr Phys Act. 2017;14(1):7.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the expert recommendations for implementing change (ERIC) project. Implement Sci. 2015;10(1):21.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Harris PA, Taylor R, Minor BL, Elliott V, Fernandez M, O’Neal L, et al. The REDCap consortium: building an international community of software platform partners. J Biomed Inform. 2019;95:103208.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. A metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377–81.

    Article  PubMed  Google Scholar 

  19. Cohen J. Statistical power analysis for the behavioral sciences: Routledge; 1988.

  20. Beebe J. Rapid qualitative inquiry: A field guide to team-based assessment: Rowman & Littlefield; 2014.

  21. Elo S, Kyngäs H. The qualitative content analysis process. J Adv Nurs. 2008;62(1):107–15.

    Article  PubMed  Google Scholar 

  22. Haun JN, Chavez M, Nazi KM, Antinori N. Developing a health information technology systems matrix: A qualitative participatory approach. J Med Internet Res. 2016;18(10):e266.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Lindgren, B.-M., Lundman, B., & Graneheim, U. H. (2020). Abstraction and interpretation during the qualitative content analysis process. International journal of nursing studies, 103632.

  24. Vaismoradi M, Turunen H, Bondas T. Content analysis and thematic analysis: implications for conducting a qualitative descriptive study. Nursing & Health Sciences. 2013;15(3):398–405.

    Article  Google Scholar 

  25. Miles MB, Huberman M. Qualitative data analysis: an expanded Sourceboo. 2nd ed. Inc: Sage Publications; 1994.

    Google Scholar 

  26. Arabadjis, S. D., & Sullivan, E. E. (2020). Data and HIT systems in primary care settings: an analysis of perceptions and use. Journal of Health Organization and Management.

    Google Scholar 

  27. O’Donnell A, Kaner E, Shaw C, Haighton C. Primary care physicians’ attitudes to the adoption of electronic medical records: A systematic review and evidence synthesis using the clinical adoption framework. BMC Medical Informatics and Decision Making. 2018;18(1):1–16.

    Article  Google Scholar 

  28. Begley CE, Hickey JS, Ostermeyer B, Teske LA, Vu T, Wolf J, et al. Best practices: integrating behavioral health and primary care: the Harris county community behavioral health program. Psychiatr Serv. 2008;59(4):356–8.

    Article  PubMed  Google Scholar 

  29. McIvor A, Kayser J, Assaad J-M, Brosky G, Demarest P, Desmarais P, et al. Best practices for smoking cessation interventions in primary care. Can Respir J. 2009;16(4):129–34.

    Article  PubMed  PubMed Central  Google Scholar 

  30. McKinstry B, Campbell J, Salisbury C. Telephone first consultations in primary care. In: Telephone first consultations in primary care; 2017.

  31. Meadows GN, Harvey CA, Joubert L, Barton D, Bedi D, Psych G. Best practices: the consultation-liaison in primary-care psychiatry program: A structured approach to long-term collaboration. Psychiatr Serv. 2007;58(8):1036–8.

    Article  PubMed  Google Scholar 

  32. Ng H. Best practices in LGBT care: A guide for primary care physicians. Cleve Clin J Med. 2016;83(7):531.

    Article  PubMed  Google Scholar 

  33. Chan B, Lyles C, Kaplan C, Lam R, Karliner L. A comparison of electronic patient-portal use among patients with resident and attending primary care providers. J Gen Intern Med. 2018;33(12):2085–91.

    Article  PubMed  PubMed Central  Google Scholar 

  34. Greenwald P, Stern ME, Clark S, Sharma R. Older adults and technology: in telehealth, they may not be who you think they are. Int J Emerg Med. 2018;11(1):1–4.

    Article  Google Scholar 

  35. Barac R, Stein S, Bruce B, Barwick M. Scoping review of toolkits as a knowledge translation strategy in health. BMC Medical Informatics and Decision Making. 2014;14(1):1–9.

    Article  Google Scholar 

  36. Hempel S, Miake-Lye I, Brega AG, Buckhold F III, Hassell S, Nowalk MP, et al. Quality improvement toolkits: recommendations for development. Am J Med Qual. 2019;34(6):538–44.

    Article  PubMed  Google Scholar 

  37. Newnam S, Goode N, Read GJ, Salmon PM. Closing the research-practice gap in healthcare: the development and usability evaluation of a patient handling incident investigation toolkit. Saf Sci. 2020;129:104844.

    Article  Google Scholar 

  38. Arndt BG, Beasley JW, Watkinson MD, Temte JL, Tuan W-J, Sinsky CA, et al. Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations. The Annals of Family Medicine. 2017;15(5):419–26.

    Article  PubMed  Google Scholar 

  39. Ancker JS, Edwards A, Nosal S, Hauser D, Mauer E, Kaushal R. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Medical Informatics and Decision Making. 2017;17(1):1–9.

    Article  Google Scholar 

  40. Birt L, Scott S, Cavers D, Campbell C, Walter F. Member checking: A tool to enhance trustworthiness or merely a nod to validation? Qual Health Res. 2016;26(13):1802–11.

    Article  PubMed  Google Scholar 

  41. Guest G, Bunce A, Johnson L. How many interviews are enough? An experiment with data saturation and variability. Field Methods. 2006;18(1):59–82.

    Article  Google Scholar 

  42. Thompson TG, Brailer DJ. The decade of health information technology: delivering consumer-centric and information-rich health care. Washington, DC: US Department of Health and Human Services; 2004.

    Google Scholar 

Download references


Development of this manuscript was supported in part by the James A. Haley Veterans Hospital Research and Development Service. The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs. The authors would like to express a special thank you to Nancy Wilk for her review. Portions of this paper were at the American Anthropological Association and the American Congress for Rehabilitation Medicine in 2018-2019.


The study was supported by the Department of Veterans Affairs, Veterans Health Administration, Office of Research and Development, Health Services Research and Development Service, (IIR 15–443 - Virtual Medical Modality Implementation Strategies for Patient Aligned Care Teams to Promote Veteran Centered Care) award number I01 HX002010.

Author information

Authors and Affiliations



JH designed the research, secured funding for the study, supervised research conduct and prepared the manuscript. BC collected, analyzed and interpreted the qualitative data and prepared the manuscript. CM collected, analyzed and interpreted the qualitative data and prepared the manuscript. VP collected, analyzed and interpreted the quantitative data and prepared the manuscript. WM designed the research, assisted with recruitment for data collection and prepared the manuscript. SP designed the research and prepared the manuscript. BZ designed the research, assisted with recruitment for data collection and prepared the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Christine Melillo.

Ethics declarations

Ethics approval and consent to participate

This study received ethics approval from the University of South Florida Institutional Review Board. Fully informed written consent was obtained from all participants.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The original version of this article was revised: the article title, the author affiliation 1, Table 2, Table 3 and Figure 2 have been revised.

Supplementary Information

Additional file 1.

PACT Teamlet Member Demographic Survey.

Additional file 2.

PACT Teamlet Member Virtual Medical Modality Survey.

Additional file 3.

Aim 1 - PACT Teamlet Member Focus Group Interview Script.

Additional file 4.

Aim 1 Expert Informant Interview Script.

Additional file 5.

COREQ (COnsolidated criteria for REporting Qualitative research) Checklist.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Haun, J.N., Cotner, B.A., Melillo, C. et al. Proactive integrated virtual healthcare resource use in primary care. BMC Health Serv Res 21, 802 (2021).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: