Skip to main content

Provider reported value and use of virtual resources in extended primary care prior to and during COVID-19



A proactive approach to delivering care using virtual resources, while reducing in-person contact, is needed during the COVID-19 pandemic.


In the current study we describe pre- to post- COVID-19 pandemic onset related changes in electronic delivery of primary care.


A longitudinal, pre-post within-subjects design was used. Patient-aligned care team providers from one VA medical center, a primary care annex, and four affiliated community-based outpatient clinics completed both a baseline and follow up survey (N = 62) or the follow-up survey only (N = 85). The follow-up survey contained questions about COVID-19.


The majority of providers (88%) reported they would continue virtual care once pandemic restrictions were lifted. Most (83%) felt prepared to transition to virtual care when pandemic restrictions began. Use of My HealtheVet, Telehealth, and mobile apps showed a significant increase (22.7%; 31.1%; 48.5%). Barriers to virtual care included (1) internet connectivity; (2) patients’ lack of technology comfort and skills; and (3) technical issues. Main supports to provide virtual care to patients were (1) peers/ colleagues; (2) technology support through help desk; (3) equipment such as laptops and dual screens; (4) being able to use doximety and virtual care manager, and (5) training.


Overall, provider-use and perceptions related to using virtual care improved over time. Providers adapted quickly to providing virtual care during COVID-19 and planned to provide virtual care long-term.

Peer Review reports


Historically, virtual healthcare resources (VHR) have been used to enhance patient access to healthcare, especially in rural areas [1]. The Department of Veterans Affairs (VA) has an enterprise-wide effort to improve access to patient-centered care using VHR. VA’s effort to promote use of VHR includes an extensive suite of resources (Table 1). Though the VA has an ongoing virtual care delivery initiative, the COVID-19 pandemic required healthcare systems to use VHR to expand access to care while reducing in-person contact [2, 3].

Table 1 Virtual healthcare resources

VA has spent the last decade improving patient-centered care through promotion of proactive integrated use of VHR, specifically within primary care. Proactive integrated VHR use is defined as ‘a self-initiated approach to coordinated use of applicable VHR systems for the purposes of coordinating and delivering timely high-quality patient-centered care’ [4]. Provider openness to use VHR was promoted by system preparedness (e.g., IT systems, software, policies) and the necessity brought on by COVID-19 pandemic. For example, between mid-March and late April 2020, the number of VA weekly video-to-home primary care encounters rose over 11-fold and the number of clinicians using video-to-home visits increased 22% [5]. Telephone visits increased by 131% [5].

In this study we evaluated pre- to post-COVID-19-related changes in delivery of primary care at one VA medical center and associated sites (VAMC). “Post-COVID-19” refers to changes in care delivery put into place in response to the pandemic. We conducted a survey of VHR use and perceptions among primary care providers [aka, Patient Aligned Care Teams (PACT)] prior to onset of the pandemic (January 2020) and after COVID-19-related changes to care (October 2020). Understanding the use of VHR before and during the pandemic will provide a baseline of VHR use while acknowledging the influence of the pandemic. This data will allow for long-term assessment of VHR use and inform opportunities to continue VHR use over time.


Parent study

The parent study [6] was designed to evaluate implementation strategies to promote VHR use among primary care providers. After consent, data were collected via paper and online surveys through VA Research Electronic Data Capture (REDCap) [7, 8]. Dillman method [9] was used to increase response rates. Analyses comparing providers who received VHR training (n = 45) versus those who did not (n = 17) on demographics, baseline, and follow-up survey data resulted in no significant group differences (p > .05). Differences in providers’ use and perceived benefits of VHR are attributed to the onset of the COVID-19 pandemic which provided an unexpected opportunity to evaluate experiences of VHR use during widespread disruption of the healthcare system. This research protocol was reviewed and approved by the University of South Florida Institutional Research Board. Informed consent was obtained according to University of South Florida Institutional Research Board guidance.


The current study used a pre-post within-subjects design. Data were collected from providers at two time points: January 2020 and October 2020. Consolidated Framework for Implementation Research (CFIR) [10] constructs were used to guide data collection and analysis.


We recruited PACT providers from one VAMC. Sixty-two providers completed baseline and follow-up surveys; an additional 23 providers completed the follow-up survey only, for a total of 85 providers completing COVID-19 questions in the follow-up survey. A priori power analysis indicated a sample size of 63 to achieve 86% power to detect a medium effect size (Cohen’s d = .5). An evaluation of study attrition indicated no demographic differences (p > .05) between participants who completed a post survey (n = 62) and those missing follow-up data (n = 47).


A 16-item questionnaire assessed demographics; an 18-item self-report survey, previously described [6], was organized into 5 subscales informed by CFIR constructs, including relative advantage (perceptions of improved care delivery, preference over traditional tools); observability (perceptions of improved clinical workflow and patient outcomes); compatibility (self-reported comfort with VHR use to communicate and deliver care); complexity (education is needed on access/use, integration ease in patient care delivery); context and facilitation (perceptions of VHR use reflects role responsibilities and is reinforced by workplace) [11]. Items were scored on a 5-point Likert scale (1 = strongly agree to 5 = strongly disagree) that were averaged at the subscale level. The follow-up survey included nine additional questions related to COVID-19: six regarding use of VHR pre, during, and post (projected) and three open-ended about barriers, supports, and preference for providing virtual care. All questions are aimed at capturing provider perspective of their own and their patient’s use and change in use of VA resources. VHR listed in the survey are described in Table 1.


Descriptive statistics summarize sample demographics. Given the non-independent nature of our within-group sample, we used the following analyses to evaluate pre- to post-changes: 1) nonparametric McNemar’s test to evaluate VHR utilization pre- to post-COVID-19 (Table 3); 2) paired sample t-tests to evaluate changes in provider perceptions of patients’ preferred methods of communication, and promotion of VHR with their patients (Table 4); and 3)Cohen’s d effect sizes with 95% confidence intervals are reported in Table 4. Rapid matrix analysis [12] and content analysis [13] was used to analyze the three open-ended questions.


Provider characteristics

Participants with pre and post surveys (N = 62) represented all PACT roles; nearly half representing nurses (45.2%). The sample was predominately white (77.4%), female (85.5%), with an average age of 46.2 years (Table 2).

Table 2 Primary care participant baseline characteristic

Quantitative findings

Provider reported virtual practice pre, during, and projected post COVID-19

Of the follow-up participants (N = 85, see Table 2), 75% reported 25% or less of their care delivery was conducted virtually prior to COVID-19. Only 1% of providers reported 25% or less of their practice was conducted virtually during COVID-19. When asked to predict whether their practice would continue virtually after COVID-19, 48% predicted less than half of their care being delivered virtually versus 52% predicting more than half of their care being delivered virtually; 88% reported, if possible, they would continue conducting care virtually once COVID-19-related restrictions are lifted. Majority (83%) reported feeling prepared to implement COVID-19-related changes to care. Finally, 39% reported information received during the parent study training helped respond to COVID-19 changes to care.

Changes in provider perception of their and patients’ use and promotion of VHR

While use of Secure Messaging remained elevated over the study period with over 87% of providers reporting use, survey results highlighted changes in use of My HealtheVet, Telehealth VA Video Connect (referred to as Telehealth hereafter), VetLink Kiosks, and mobile apps (Table 3). Despite these changes, providers reported patient preference for most VHR remained stable. In contrast, patient preference for Telehealth doubled (Table 3).

Table 3 Baseline and six-month follow-up use and promotion of virtual medical modalities (virtual resources) among PACT providers (N = 62)

Similarly, provider promotion of patient use and their own use of VHR remained stable, with over 75% reporting continued use or promoting My HealtheVet and Secure Messaging with 50–100% of their patients. However, providers’ use of mobile apps and Telehealth increased (Table 3).

Changes in provider perceptions

Providers’ perceptions of relative advantage for use of My HealtheVet, Secure Messaging, and Kiosks were stable (i.e. small effect sizes) showing providers continued to agree VHR use improved care delivery and was preferred over other tools (e.g., telephone). The two VHR that showed improvements in provider perceptions were Telehealth and mobile apps (Table 4).

Table 4 Perceptions about virtual resources among providers (N = 62) at baseline and six-month follow-up

Two domains that showed moderate to large effect size changes were perceptions of observability and compatibility. Providers reported a moderate effect size increase in perceived observability and compatibility for use of My HealtheVet, a moderate to large effect size increase for use of mobile apps, and a large effect size increase for use of Telehealth (Table 4).

Telehealth was the only VHR that showed large increases in perceived complexity and context and facilitation with its use, with other VHR showing stability over time (Table 4). Providers were more likely to report at follow-up that education is needed in using Telehealth and integrating it into care.

Qualitative findings

Provider reported barriers, facilitators, and changes in VHR preferences

Provider identified main barriers were (1) internet connectivity; (2) patients’ lack of skills using technology; and (3) technical issues. Main supports to provide virtual care to patients were (1) peers/colleagues; (2) technology support through help desk; (3) equipment to support virtual care delivery (e.g., laptops); (4) online networking services, and (5) training. Half of the providers (48%) appreciated virtual care more or were more open to it. Some reasons cited for this preference were improved: confidence, convenience, workflow, and continuity of care.


In this study we described changes in primary care delivery pre- to post- COVID-19 pandemic onset practice changes at one VAMC and affiliated sites. We conducted a survey of VHR utilization and perceptions among PACT members in January 2020 and after pandemic-related changes to care were underway (October 2020). Data representing VHR use before and during the pandemic will provide a baseline of VHR use and inform opportunities to sustain VHR use over time. Our study is unique in that it is one of the first data samples to describe providers’ VHR use and perceptions within the VA both before and after COVID-19 changes in care delivery. These data provide a baseline study for VHR use changes and can inform use patterns over time.

While use of Secure Messaging remained consistent from pre- to post-evaluation, results highlighted changes in use of other tools (e.g., My HealtheVet, mobile apps). These changes are consistent with transition to mainly virtual delivery in March 2020. We recorded an increase in VHR designed for remote care, such as apps, paired with an expected decrease in tools associated with face-to-face visits such as VetLink Kiosks. These findings are consistent with data we collected on provider reports of pre- to post- delivery of care changes in their own practices.

We also found providers’ perceptions of patient preferences changed. Providers reported nearly twice as many patients preferred Telehealth to face-to-face visits. Although we only collected providers’ perceived patient preference in this study, this work was an extension of the authors’ previous works, which focused on patient perspective and experiences with VHR [14,15,16,17,18]. The authors focus on providers’ perspective was driven by patient reports that providers’ recommendations and reinforcement of VHR use drive patient use patterns [17]. Findings align with previous work suggesting patients perceived managing their care with VHR alleviated care burden by improving access to knowledge and care, and improving continuity and experience of care [16, 19]. Findings indicate provider promotion of patient use and their own use of VHR predates COVID-19. Providers and patients were familiar with the tools and primed to implement care changes, which translated to doubling promotion and use of telehealth from pre- to post- COVID-19.

Overall, provider positive perceptions regarding VHR use remained stable over time. Providers reported perceived benefits and preference for using mobile apps to deliver care; and were more likely to agree that use of My HealtheVet, mobile apps, and telehealth was associated with improved clinical workflow, patient outcomes, and workflow at follow-up. Telehealth was the only VHR showing large increases in perceptions that further education is needed to integrate telehealth into care delivery. These findings suggest having access to a wide array of VHR can improve workflow by more flexibly responding to needs; however, providers would desire more training, given the recent rapid shift in using VHR to deliver healthcare [5].

Our COVID-19-related questions shed light on providers’ experiences with adapting to COVID-19-related care changes. Most of the sample reported feeling prepared to implement VHR during COVID-19. This is likely in part due to belonging to a system that had made efforts well before COVID-19 to implement VHR for care delivery. This preparedness could be related to the parent study’s training provided to PACT in the month prior to COVID-19-related changes. Training and targeted implementation strategies are needed to support providers’ proactive integrated VHR use while providing consistently high-quality care [2]. Qualitative findings suggested providers’ preference for virtual care for most types of visits. Finally, the experience of adapting to COVID-19-related changes in delivery of care while abrupt, is likely to have lasting effects, with more than three-fourths of providers reporting desire to continue conducting care virtually once restrictions are lifted. Sustainability of VHR use is needed beyond the recent COVID-19 pandemic context. Future work should assess sustainability of VHR use and quality of patient experiences and outcomes. Previous efforts such as mandates and workload credit were successfully implemented to support uptake [14] and sustained use over time.

Study limitations should be considered when interpreting findings, including the use of self-report, sample size, and the unique VA environment. Authors did not evaluate the medicolegal implications of the usage of VHRs post pandemic as that is beyond the scope of this study. Such evaluation should be addressed in future work as global pandemics and the need for VHR use are likely to remain relevant to health services. These data provide insights into VHR use in an unprecedented context and can be used to inform a subsequent study using a larger sample to assess convergence with these findings. Future research is also warranted to determine if VHR user trends persist post-pandemic, and if PACT providers – and patients – need education to maximize the opportunity to proactively integrate VHR into the healthcare delivery process.


Data indicate providers were prepared to use VHR within their practice, yet many did not embrace implementation of proactive integrated use of VHR prior to COVID-19 related care delivery changes. The pandemic necessitated implementation of proactive use of VHR. Providers reported increased use, improved perceptions, and intention to continue use of VHR, consistent with the VA system wide effort to promote the delivery of care virtually. Future research is needed to build upon the current findings to determine if VHR use sustains post pandemic and assess quality of patient experiences and outcomes.

Availability of data and materials

The datasets during and/or analyzed during the current study available from the corresponding author on reasonable request.


  1. Lum HD, Nearing K, Pimentel CB, Levy CR, Hung WW. Anywhere to anywhere: use of telehealth to increase health care access for older, rural veterans. Public Policy Aging Rep. 2020;30(1):12–8.

    Article  Google Scholar 

  2. Wong A, Bhyat R, Srivastava S, Lomax L, Appireddy R, Wong A, et al. Patient care during the COVID-19 pandemic: use of virtual care. J Med Internet Res. 2021;23(1):e20621. PMID: 33326410.,” J. Med. Internet Res., vol. 23, no. 1, p. e20621, 2021.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Whitelaw S, Mamas MA, Topol E, Van Spall HG. Applications of digital technology in COVID-19 pandemic planning and response. Lancet Digit Health. 2020;2:e435–40.

  4. Haun JN, et al. Proactive integrated virtual healthcare resource use in primary care. BMC Health Serv Res. 2021;21(1):1–14.

    Google Scholar 

  5. Heyworth L, Kirsh S, Zulman D, Ferguson JM, Kizer KW. Expanding access through virtual care: the VA’s early experience with Covid-19. NEJM Catal Innov Care Deliv. 2020;1(4):1–11.

  6. Haun J, et al. Virtual medical modality implementation strategies for patient-aligned care teams to promote veteran-centered care: protocol for a mixed-methods study. JMIR Res Protoc. 2018;7(8):e11262.

  7. Harris PA, et al. The REDCap consortium: building an international community of software platform partners. J Biomed Inform. 2019;95:103208.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. A metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inf. 2009;42(2):377–81.

    Article  Google Scholar 

  9. D. A. Dillman, J. D. Smyth, and L. M. Christian, Internet, phone, mail, and mixed-mode surveys: the tailored design method. 2014. Accessed: Feb. 25, 2016. [Online]. Available:

  10. Damschroder LJ, Goodrich DE, Robinson CH, Fletcher CE, Lowery JC. A systematic exploration of differences in contextual factors related to implementing the MOVE! Weight management program in VA: a mixed methods study. BMC Health Serv Res. 2011;11(1):1 Accessed: Nov. 04, 2016. [Online]. Available:

    Article  Google Scholar 

  11. Helfrich CD, et al. A critical synthesis of literature on the promoting action on research implementation in health services (PARIHS) framework. Implement Sci. 2010;5(1):1 Accessed: Nov. 04, 2016. [Online]. Available:

    Article  Google Scholar 

  12. Beebe J. Rapid qualitative inquiry: a field guide to team-based assessment: Rowman & Littlefield; 2014.

    Google Scholar 

  13. Vaismoradi M, Turunen H, Bondas T. Content analysis and thematic analysis: implications for conducting a qualitative descriptive study. Nurs Health Sci. 2013;15(3):398–405.

    Article  PubMed  Google Scholar 

  14. Haun JN, et al. Evaluating user experiences of the secure messaging tool on the veterans affairs’ patient portal system. J Med Internet Res. 2014;16(3):e75.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Haun JN, et al. Using electronic data collection platforms to assess complementary and integrative health patient-reported outcomes: feasibility project. JMIR Med Inform. 2020;8(6):e15609.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Haun JN, Lind JD, Shimada SL, Simon SR. Evaluating secure messaging from the veteran perspective: informing the adoption and sustained use of a patient-driven communication platform. Ann Anthropol Pract. 2013;37(2):57–74.

    Article  Google Scholar 

  17. Haun JN, et al. Veterans’ preferences for exchanging information using veterans affairs health information technologies: focus group results and modeling simulations. J Med Internet Res. 2017;19(10):e8614.

    Article  Google Scholar 

  18. Haun JN, Chavez M, Nazi KM, Antinori N. Developing a health information technology systems matrix: a qualitative participatory approach. J Med Internet Res. 2016;18(10):e6499.

    Article  Google Scholar 

  19. Kelley L, et al. Exploring how virtual primary care visits affect patient burden of treatment. Int J Med Inf. 2020;141:104228.

    Article  CAS  Google Scholar 

Download references


This study was also supported in part by the Center of Innovation for Disability and Rehabilitation Research at the James A. Haley Veterans Hospital. The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs. The authors would like to express a special thank you to the PACT teams at James A. Haley Veterans Hospital for their interest and support.

The corresponding author, as a US government employee, has the right to grant on behalf of all authors and does grant on behalf of all authors, a nonexclusive worldwide license to the publishers and its licensees in perpetuity, in all forms, formats, and media (whether known now or created in the future) to (1) publish, reproduce, distribute, display, and store the contribution, (2) translate the contribution into other languages, create adaptations, reprints, include within collections and create summaries, extracts and/or, abstracts of the contribution, (3) create any other derivative work(s) based on the contribution, (4) to exploit all subsidiary rights in the contribution, (5) the inclusion of electronic links from the contribution to third-party material wherever it may be located, and (6) licensee any third party to do any or all of the above.


The study was supported by the Department of Veterans Affairs, Veterans Health Administration, Office of Research and Development, Health Services Research and Development Service, (IIR 15–443 - Virtual Medical Modality Implementation Strategies for Patient Aligned Care Teams to Promote Veteran Centered Care) award number I01 HX002010.

Author information

Authors and Affiliations



JH conceptualized and contributed to the development and writing of the manuscript. VP, BC and CM wrote the main manuscript text. VP, HH, CF collected and analyzed the quantitative data, prepared Tables 2, 3 and 4 and contributed to writing of the conclusions. BC and CM collected and analyzed the qualitative data, prepared Table 1 and contributed to writing of the conclusions. BZ and WM provided clinical context for findings and contributed to writing of the conclusions. All authors reviewed the manuscript. The authors read and approved the final manuscript.

Corresponding author

Correspondence to Christine Melillo.

Ethics declarations

Ethics approval and consent to participate

The study was performed in accordance with the Declaration of Helsinki ethical principles for medical research involving human subjects. This study was approved by the University of South Florida Institutional Review Board (Pro00028519). Informed consent to participate was obtained from all participants.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Haun, J.N., Panaite, V., Cotner, B.A. et al. Provider reported value and use of virtual resources in extended primary care prior to and during COVID-19. BMC Health Serv Res 22, 1353 (2022).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: