Skip to main content
Download PDF

Development of a novel instrument to characterize telemedicine programs in primary care

BMC Health Services Research volume 23, Article number: 1274 (2023) Cite this article

Abstract

Background

Given the rapid deployment of telemedicine at the onset of the COVID - 19 pandemic, updated assessment methods are needed to study and characterize telemedicine programs. We developed a novel semi - structured survey instrument to systematically describe the characteristics and implementation processes of telemedicine programs in primary care.

Methods

In the context of a larger study aiming to describe telemedicine programs in primary care, a survey was developed in 3 iterative steps: 1) literature review to obtain a list of telemedicine features, facilitators, and barriers; 2) application of three evaluation frameworks; and 3) stakeholder engagement through a 2-stage feedback process. During survey refinement, items were tested against the evaluation frameworks while ensuring it could be completed within 20–25 min. Data reduction techniques were applied to explore opportunity for condensed variables/items.

Results

Sixty initially identified telemedicine features were reduced to 32 items / questions after stakeholder feedback. Per the life cycle framework, respondents are asked to report a month in which their telemedicine program reached a steady state, i.e., “maturation”. Subsequent questions on telemedicine features are then stratified by telemedicine services offered at the pandemic onset and the reported point of maturation. Several open - ended questions allow for additional telemedicine experiences to be captured. Data reduction techniques revealed no indication for data reduction.

Conclusion

This 32-item semi-structured survey standardizes the description of primary care telemedicine programs in terms of features as well as maturation process. This tool will facilitate evaluation of and comparisons between telemedicine programs across the United States, particularly those that were deployed at the pandemic onset.

Peer Review reports

Background

The onset of the COVID - 19 pandemic accelerated a shift towards virtual care in the United States, particularly in primary care settings, with telemedicine now being routinely offered to patients [1, 2]. Despite widespread availability of telemedicine, challenges remain as telemedicine programs were swiftly deployed, often in immature states, thus forgoing a traditional health information technology (IT) life cycle [3, 4]. The latter is of particular importance as evaluation throughout the entire health IT life cycle is thought to be crucial to ensure that any health IT intervention has time to reach its full potential [5, 6]. Such life cycle phases generally include the following: 1) planning, 2) design and development, 3) implementation and use, and 4) a fully operational phase with monitoring, evaluation, and optimization [3, 4].

The rapid implementation of telemedicine programs in primary care has complicated systematic evaluations with likely substantial variations in experiences between individual practices; for example, acuteness of telemedicine roll-out, available telemedicine features or resources may differ significantly between practices. Indeed, since the deployment of these telemedicine programs, some practices and providers have felt the strain of using hastily-developed, insufficiently mature programs that had not fully considered workflow and human factors [7]. Since the onset of the pandemic, several frameworks to evaluate telemedicine programs have been proposed. However, frameworks have focused on telemedicine quality metrics [8], evaluated telemedicine broadly without considering program differences [9], or focused solely on implementation without first characterizing existing platforms [10]. Additionally, none of these proposed frameworks were developed in consultation with a panel of stakeholders, nor do they offer a usable instrument for obtaining the information needed for evaluation. The current absence of such a usable instrument – one that rigorously captures the process of telemedicine roll-out and systematically describes existing telemedicine program features – prohibits widespread, practical evaluation of telemedicine programs.

Objective

Therefore, in the context of a larger United States telemedicine evaluation project, this study aimed to develop a semi-structured survey informed by existing evidence, relevant frameworks, and input from a variety of stakeholders including patients, clinicians and researchers.

Methods

This study is part of larger study funded by the Patient-Centered Outcomes Research Institute (PCORI) with the objective of describing and evaluating the impact of telemedicine programs in primary care implemented in the wake of the COVID - 19 pandemic. Here, we describe the development of a semi-structured survey to be applied in interviews with primary care practice leaders across the United States. The goal of the survey is to systematically describe practices’ telemedicine features and the process of telemedicine implementation to understand the telemedicine programs’ life cycle.

The survey design process included three iterative steps: 1) a literature review of evidence on barriers to and facilitators of telemedicine, 2) application of three different frameworks, and 3) input from various stakeholders on relevance of items and formulation of questions (Fig. 1). Each step is described in more detail below.

Fig. 1
figure 1

Schematic model of the survey design process

Full size image

Literature review

Two electronic databases (PubMed and Web of Science) were queried by two researchers for relevant articles published until May 2020. The search strategy included the following keywords: Telemedicine, Telehealth, Primary Care, General Practice, Family Doctor, Primary Health Care, Primary Healthcare, Barrier(s), Facilitator(s), Evaluation, Features, Characteristics. Studies were included if they described (evidence on) features of telemedicine programs in terms of being facilitators of or barriers to telemedicine implementation. Collected features were non-exclusive to a particular participant in the telemedicine process (i.e., factors relating to patients, providers, practice, or technology were all considered).

Frameworks

The features of telemedicine identified in the previous stage were matched to elements of two theoretical frameworks: the Triangle Model for health information technology evaluation [11] and Penchansky and Thomas’ access to care framework [12]. A third health IT life cycle framework was applied to describe the evolution of telemedicine programs over time [13].

Triangle model

The Triangle Model, first developed to evaluate health information technology systems such as the electronic health record (EHR), provides a cross-sectional framework that encourages qualitative and quantitative assessment of structure and process factors of the health information technology [11]. This framework identifies seven domains for evaluation: (I) the technology of interest, (II) the organization that adopts this technology, (III) users of the technology, (IV) the patient population served, (V) the relationships between the organization and the technology, (VI) the relationships between the organization and the providers, and (VII) the relationships between users and the technology. Figure 2 provides an overview and application of this model to the context of telemedicine with telemedicine-specific examples. Application of the Triangle model involved grouping identified features of telemedicine into each domain.

Fig. 2
figure 2

Triangle model modified for telemedicine

Full size image

Penchansky and Thomas’ access to care framework

This framework conceptualizes access to healthcare in terms of the fit between features of providers and health services, and patient characteristics and expectations [12, 14]. In other words, access is conceived as the interface between potential users/patients and healthcare resources, influenced by features of those who supply and those who utilize the services. We applied it given telemedicine’s theoretical potential to improve access to care by eliminating physical barriers such as distance or allowing more flexibility in terms of scheduling [15, 16]. Access to care elements in this framework include: A) affordability, B) accommodation, C) availability, D) accessibility, and E) acceptability. Application of the access to care framework involved assessing each identified telemedicine feature in terms of their potential to impact access to care based on relevance to each of the aforementioned five access to care elements.

Health IT life cycle framework (‘maturation’)

This framework was applied as telemedicine programs were rolled out swiftly given the acute onset of the pandemic, and telemedicine programs gradually "matured" after widespread implementation. The unique and acute nature of the pandemic also almost certainly precluded the types of ongoing formative evaluations thought to be crucial for any health information intervention to achieve its full potential [5, 6]. We aimed to quantify the process of telemedicine program "maturation" using the health IT life cycle framework. While various of such frameworks exist, life cycle phases generally include the following: 1) planning, 2) design and development, 3) implementation and use, and 4) a fully operational phase with monitoring, evaluation, and optimization of the health IT application of interest [3, 4, 13]. Application of the health IT life cycle framework involved querying a perceived point of maturation of a telemedicine program and what telemedicine features in particular were associated with this point of maturation.

Stakeholder engagement

To support the parent study of telemedicine programs in primary care, a stakeholder board was established using a previously applied methodology [17]. The main goal of this stakeholder board was to provide critical input at various stages of our research. Overall, 15 members (10 female, 5 male) agreed to serve on the board: five patients with chronic diseases (three aged > 65 years), four primary care providers with telemedicine experience, three health informatics experts, one primary care healthcare administrator, two payer administrators, and one patient advocate. Of note, the stakeholder board did not include family members of patients; inclusion of this group would have likely provided further insights as to the experiences of older patients who require assistance with telemedicine. For this survey development project, stakeholders were invited to provide input on 1) items that are important to consider in evaluations of telemedicine programs, and 2) how to formulate these items as survey questions. More specifically, the survey refinement process involved identifying features of telemedicine that were most relevant to patient experiences and practice operations, combining related features into a single survey item, and removing questions that could be answered via quantitative data provided by the participating institutions. A 2-stage feedback process consisted of one formal meeting with the stakeholder board and the research team as well as empowering members to provide asynchronous feedback (through email) in response to each of the four revised versions of the survey. This extensive engagement process with the stakeholder board occurred longitudinally throughout the survey development, and all feedback and associated survey edits were noted and incorporated. This process continued until a final draft of the survey was ready for administration to primary care practice leaders, designed to be completed in 20–25 - min interviews.

The final survey questions were input into REDcap, a secure online application for capturing data and managing databases [18]. Most questions could be answered with ordinal or nominal responses, and respondents were also encouraged to supplement their interviews with narratives, anecdotes, and elaboration.

Survey administration and data reduction analysis

The developed survey was administered in 20–25-min virtual interviews with primary care practice leaders from four large health systems (Weill Cornell Medicine, Mount Sinai, University of North Carolina, and University of Florida) participating in the Patient-Centered Outcomes Research Institute PCORnet network. Practice leader participation was optional and voluntary.

Data reduction techniques were applied to the collected survey data to determine if there were associations between items of the survey i.e., if items on the survey could be reduced to summary variables / questions. Where needed, continuous variables were transformed into categorical variables; this was either distribution - based or based on natural cutoffs. Levels of categorical variables were inspected manually and re - coded as appropriate. The conversion to categorical variables was a decision that facilitated the data reduction analysis as a means of “pre-summarizing and reducing the data;” the conversion to categorical variables does not preclude the use of continuous data in future uses of this instrument. Finally, the missing values of the continuous variables were imputed using their medians. The dimension reduction technique applied was factor analysis of mixed data (FAMD); here, the representation of continuous variables is constructed as in a principal component analysis (PCA), and categorical variables are constructed as in a multiple correspondence analysis (MCA). Scree plots were used to determine the least number of factors needed to accurately represent the original data. Analyses were conducted in R (R version 3.6.0).

Results

Following the literature review, 35 telemedicine-related features were identified, i.e., factors potentially associated as barriers to or facilitators of telemedicine implementation. These were categorized into Triangle Model domains, and features were assessed against Penchansky & Thomas’ access-to-care framework elements (Appendix 1). The process of stakeholder board engagement added 25 more items to this preliminary list adding up to a total of 60 features (Table 1).

Table 1 Overview of telemedicine features and telemedicine-related features considered to be included in the survey; features are grouped by Triangle model domains. This list is a combination of results from the literature review and feedback from the stakeholder board (see Appendix 1 for an overview of features resulting just from the literature review grouped by Triangle model domains)
Full size table

In the final two rounds of stakeholder board engagement, the research team specifically queried which items would be the most important to include in the survey, while also taking into account a duration of 20–25 min for an interview. The stakeholder board also advised on how questions should be formulated. This resulted in a final survey of 32 items (Table 2; the actual survey can be found in Appendix 2).

Table 2 Survey items and questions, response types, and response options (the final REDcap survey can be found in Appendix 2)
Full size table

The final domains covered practice characteristics, patient population characteristics, technical features, barriers to care, patient services, operational processes, financial investment, and future plans for telemedicine. Each item in the survey also included opportunities to record associated qualitative information. In order to systematically describe the evolution of telemedicine programs (and taking into account the health IT life cycle framework) a question was included on the perceived point of maturation of a telemedicine program. Questions 9–29 on the features of the telemedicine program were asked twice, once for the program at the onset of the pandemic and a second time at the point identified as telemedicine program maturation.

The survey was administered to 33 practice leaders representing 100 unique primary care practices across four states. Characteristics of the result can be found in Table 3. From this data, 18 continuous and 40 categorical variables were extracted for data reduction analysis. Scree plots (Appendices 3 and 4) created from the data reduction analyses indicated that there was no clear “elbow” with a rather smooth plot, i.e., no indication for a suitable number of factors to retain. This was the first indication against the use of a data dimension reduction approach. To further explore, the top ten variables contributing to each of the first four dimensions/factors, representing a total of 33 variables (instead of the 58 variables used in the previous analysis), were selected and the same analysis was performed on the smaller set. Resulting scree plots showed a similar pattern, i.e., no clear elbow.

Table 3 Telemedicine features stratified by March 2020 and point of telemedicine program maturation (n = 100 practices)
Full size table

Discussion

This article described the process of developing a semi - structured survey to be used in a larger study characterizing telemedicine programs in primary care rolled out in response to the COVID-19 pandemic. Via an iterative process that involved querying existing evidence, the application of relevant frameworks, and extensive stakeholder input, a 32-item tool was developed to systematically quantify telemedicine programs vis-à-vis telemedicine and telemedicine-related features as well as the process of telemedicine maturation, while also allowing room for non-structured responses to fully capture practice-specific telemedicine experiences. This tool will facilitate not only evaluations but also comparisons between telemedicine programs across the United States, particularly those that were deployed at the pandemic onset.

The roll-out of telemedicine across the pandemic was an unprecedented implementation of a technology in healthcare. For this reason, there is no gold standard or accepted truth for the question of this study: how telemedicine programs were rolled-out in primary care practices. However, the data reduction techniques applied provides reassurance that the items included in the survey were not redundant and could not be summarized with fewer variables. Each item therefore contributes value to the overall tool; the tool cannot be shortened without losing unique information on practices.

The strength of the developed survey tool lies in the inclusion of a thorough overview of existing evidence, the application of relevant frameworks and, importantly, inclusion of stakeholder feedback, which are crucial recommended steps in survey development [19, 20]. These steps, particularly the latter, are designed to ensure items’ relevance, content validity and clarity. The current lack of an instrument to systematically measure telemedicine programs justifies follow-up research on improving and modifying our survey. For example, future work could include validation of the survey and modifying the survey (e.g., shortening through data reduction techniques or modifying it based on the intended respondent). Following development, to date, this tool has been used in 33 interviews with primary care practice leaders across the United States, and important lessons related to wording of questions and time needed for interviews have been learned. Results of this ongoing process will be reported with the completion of each step.

Lacking systematic and standardized evaluations of health IT interventions is a known limitation in the health IT evaluation literature as it has been identified as a major barrier to between - study comparisons, generalizability of evaluation results, and importantly, reproducibility [21, 22]. This problem is compounded by various factors including heterogeneity in terms of the respective health IT intervention’s features, implementation context, organizational context, site - specific user characteristics, and more generally a lack of standard for best practices on reporting results of health IT intervention evaluations, even though numerous frameworks exist [23, 24]. Most of these limitations are relevant to the current telemedicine literature in primary care especially given the fast pace of roll-out of such programs with the onset of the COVID - 19 pandemic [25,26,27,28,29]. Indeed, most primary care telemedicine evaluation studies - specifically those published since the pandemic onset - either describe single institution (or single health system) experiences or do not clearly report the full extent of telemedicine features employed, with data missing on important contextual factors such as provider training in telemedicine, patient populations’ access to telemedicine technology, or support for patients opting to have a telemedicine visit [25,26,27,28,29]. Moreover, data is lacking on the process of maturation of telemedicine programs as this may have been different between practices in early hotspots of the pandemic (e.g. New York City region) compared to practices in other geographic areas. Perhaps the best parallel can be drawn with the fast adoption of EHR systems, accelerated in part by federally mandated programs such as Meaningful Use (currently known as Promoting Interoperability) supported through the Health Information Technology for Economic and Clinical Health (HITECH) Act [30]. Here, similar concerns have been noted such as lack of contextual information, non-standardized reporting and uncertainty regarding generalization [31, 32]. Application of the survey tool developed in the current study aims to address the aforementioned limitations of the current telemedicine literature, specifically in case of conflicting findings. One such example concerns the hypothesized impact of telemedicine on disparities in access to care; here, some studies point to telemedicine’s potential to reduce disparities [33] while others have noted concerns on a potential increase in disparities [1]. Employing a tool to describe telemedicine programs and their context in a systematic way may prove useful in explaining the mechanism behind such contradicting results.

One important feature of the developed survey is the quantification of the maturation process of telemedicine programs, a crucial aspect to take into account when comparing telemedicine program evaluations, as recently noted [34]. Indeed, various health systems had adopted telemedicine in some form before the pandemic, and it is likely that they were better equipped to manage and underwent different maturation processes relative to health systems that suddenly needed to implement telemedicine programs and infrastructure from scratch at the pandemic onset. More research is needed to quantify the head start these early adopters enjoyed and how this impacted patients’ access to care. Moreover, the dynamic nature of the pandemic – including subsequent COVID - 19 variants and surges coming and going – may require a telemedicine-specific maturation framework as practices and health systems may scale up telemedicine programs and features in response to surges in COVID - 19 cases, with the reverse happening during steady-states.

In part based on stakeholder feedback, our survey explicitly allows for qualitative responses outside of the structured questions. This was deliberate as we expected there to be unique practice-specific and health system-specific experiences regarding the roll-out of telemedicine programs, particularly as a result of the acute need to rapidly deploy and scale individual telemedicine programs. Moreover, other factors such as a practice’s or health system’s culture are hard to capture with structured questions or quantitative data. As stated, we are currently in the process of applying the survey to primary care practice leaders across the United States and extracted qualitative data may also inform future survey modifications.

Importantly, the developed survey does not include any items or variables to assess the quality or value of telemedicine programs. This is an artifact of the timing of this study, as initial steps toward developing the survey began in the immediate post-pandemic era. While value and quality metrics have important implications, many telemedicine programs were far too young to retrieve any meaningful outcomes data. There have been numerous publications demonstrating mixed results regarding specific telemedicine outcomes since the pandemic [35,36,37], and future work examining telemedicine outcomes is certainly needed. Data gleaned from the use of our proposed instrument can provide a comprehensive picture of specific telemedicine programs, necessary context for any future quality or value assessment.

This study has limitations. The survey has not yet been validated. Performance of the survey tool, as it relates to obtaining adequate telemedicine details, may differ in other practices and health systems. A different or larger group of stakeholders might have provided different input. Here, we aimed to balance the practicality of efficiently executing the research against theoretical perfection. Additionally, our tool was developed to be used in 20–25-min virtual interviews with primary care practice leaders; depending on the study objective of interest, intended type of respondent and time available for interviews, this exact tool may not be as useful in other telemedicine research and may require project - specific modifications. Finally, the telemedicine landscape has continued to evolve and change over the course of this project. While this instrument was designed in a way to make it applicable to non - pandemic era telemedicine programs in addition to pandemic - implemented programs, the notable changes in telemedicine utilization [38], guidelines [39], regulation [40], and reimbursement [41] should be noted.

The tool developed in this study has significant implications at the practice level. This standardized evaluation of telemedicine programs provides a framework for practice leaders to assess, compare, and iterate telemedicine programs. Using this tool, practices will be able to undergo continuous quality improvement initiatives more effectively as this tool offers a holistic assessment of programs, as supported by the rigorous development process documented in this manuscript. For practices that are seeking to roll - out new or re - designed telemedicine programs, the tool can act as a comprehensive list of characteristics of telemedicine programs. Administrators of telemedicine programs could reference this survey to ensure that they have considered all of the included features of their telemedicine programs. Additionally, this tool has the ability to highlight factors associated with telemedicine program evolution. Understanding and targeting these areas may accelerate the refinement of telemedicine programs moving forward.

In conclusion, in this paper we describe the process of developing a 32-item semi-structured survey to standardize the description of primary care telemedicine programs in terms of features as well as maturation process. This tool will facilitate not only evaluations but also comparisons between telemedicine programs across the United States, particularly those that were deployed at the pandemic onset.

Availability of data and materials

This data describes the development of a survey which is provided in its entirely in the supplementary files.

References

  1. Chang JE, Lai AY, Gupta A, Nguyen AM, Berry CA, Shelley DR. Rapid transition to telehealth and the digital divide: implications for primary care access and equity in a post - COVID Era. Milbank Q. 2021;99(2):340–68.

    Article  PubMed  PubMed Central  Google Scholar 

  2. Dhaliwal JK, Hall TD, LaRue JL, Maynard SE, Pierre PE, Bransby KA. Expansion of telehealth in primary care during the COVID - 19 pandemic: benefits and barriers. J Am Assoc Nurse Pract. 2021;34(2):224–9.

    Article  PubMed  Google Scholar 

  3. Sittig DF, Wright A, Coiera E, Magrabi F, Ratwani R, Bates DW, et al. Current challenges in health information technology - related patient safety. Health Informatics J. 2020;26(1):181–9.

    Article  PubMed  Google Scholar 

  4. Scott RE. “Pragmatic Evaluation”: a conceptual framework for designing a systematic approach to evaluation of eHealth interventions. IJEHMC. 2010;1(2):1–11.

    CAS  Google Scholar 

  5. Broens TH, Huis in’t Veld RM, Vollenbroek - Hutten MM, Hermens HJ, van Halteren AT, Nieuwenhuis LJ. Determinants of successful telemedicine implementations: a literature study. J Telemed Telecare. 2007;13(6):303–9.

    Article  PubMed  Google Scholar 

  6. Stead WW, Haynes RB, Fuller S, Friedman CP, Travis LE, Beck JR, et al. Designing medical informatics research and library–resource projects to increase what is learned. J Am Med Inform Assoc. 1994;1(1):28–33.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  7. ECRI. Top 10 health technology hazards for 2022 executive brief. New York: Economic Cycle Research Institute; 2022.

  8. Demaerschalk BM, Hollander JE, Krupinski E, Scott J, Albert D, Bobokalonova Z, et al. Quality frameworks for virtual care: Expert panel recommendations. Mayo Clin Proc Innov Qual Outcomes. 2023;7(1):31–44.

    Article  PubMed  Google Scholar 

  9. Zhang T, Mosier J, Subbian V. Identifying Barriers to and Opportunities for Telehealth Implementation Amidst the COVID - 19 Pandemic by Using a Human Factors Approach: A Leap Into the Future of Health Care Delivery? JMIR Hum Factors. 2021;8(2):e24860.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Otto L, Schlieter H, Harst L, Whitehouse D, Maeder A. The telemedicine community readiness model - successful telemedicine implementation and scale - up. Front Digit Health. 2023;5:1057347.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Ancker JS, Kern LM, Abramson E, Kaushal R. The Triangle Model for evaluating the effect of health information technology on healthcare quality and safety. J Am Med Inform Assoc. 2012;19(1):61–5.

    Article  PubMed  Google Scholar 

  12. Saurman E. Improving access: modifying Penchansky and Thomas’s Theory of Access. J Health Serv Res Policy. 2016;21(1):36–9.

    Article  PubMed  Google Scholar 

  13. Khoja S, Durrani H, Scott RE, Sajwani A, Piryani U. Conceptual framework for development of comprehensive e - health evaluation tool. Telemed J E - Health. 2013;19(1):48–53.

    Article  PubMed  Google Scholar 

  14. Penchansky R, Thomas JW. The concept of access: definition and relationship to consumer satisfaction. Med Care. 1981;19(2):127–40.

    Article  CAS  PubMed  Google Scholar 

  15. Donaghy E, Atherton H, Hammersley V, McNeilly H, Bikker A, Robbins L, et al. Acceptability, benefits, and challenges of video consulting: a qualitative study in primary care. Br J Gen Pract. 2019;69(686):e586–94.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Powell RE, Henstenburg JM, Cooper G, Hollander JE, Rising KL. Patient perceptions of telehealth primary care video visits. Ann Fam Med. 2017;15(3):225–9.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Horowitz CR, Shameer K, Gabrilove J, Atreja A, Shepard P, Goytia CN, et al. Accelerators: sparking innovation and transdisciplinary team science in disparities research. Int J Environ Res Public Health. 2017;14(3):225.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)–a metadata - driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377–81.

    Article  PubMed  Google Scholar 

  19. Artino AR Jr, La Rochelle JS, Dezee KJ, Gehlbach H. Developing questionnaires for educational research: AMEE Guide No 87. Med Teach. 2014;36(6):463–74.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Gehlbach H, Artino AR Jr, Durning S. AM last page: survey development guidance for medical education researchers. Acad Med. 2010;85(5):925.

    PubMed  Google Scholar 

  21. Zozus MN, Richesson RL, Walden A, Tenenbaum JD, Hammond WE. Research reproducibility in longitudinal multi - center studies using data from electronic health records. AMIA Jt Summits Transl Sci Proc. 2016;2016:279–85.

    PubMed  PubMed Central  Google Scholar 

  22. Fu S, Leung LY, Raulli AO, Kallmes DF, Kinsman KA, Nelson KB, et al. Assessment of the impact of EHR heterogeneity for clinical research through a case study of silent brain infarction. BMC Med Inform Decis Mak. 2020;20(1):60.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Teixeira L, Santos BS, Saavedra V, Ferreira C, editors. Evaluation Methods to Support Health Information Systems Development: A Framework Supported in Literature and Practical Experience. Digital Human Modeling Applications in Health, Safety, Ergonomics, and Risk Management: Health and Safety. Cham: Springer International Publishing; 2017.

  24. Cresswell K, Williams R, Sheikh A. Developing and applying a formative evaluation framework for health information technology implementations: qualitative investigation. J Med Internet Res. 2020;22(6):e15068.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Lindenfeld Z, Berry C, Albert S, Massar R, Shelley D, Kwok L, Fennelly K, Chang JE. Synchronous home-based telemedicine for primary care: a review. Med Care Res Rev. 2023;80(1):3-15. https://doi.org/10.1177/10775587221093043.

  26. Adepoju O, Liaw W, Chae M, Ojinnaka C, Britton E, Reves S, et al. COVID - 19 and telehealth operations in Texas primary care clinics: disparities in medically underserved area clinics. J Health Care Poor Underserved. 2021;32(2):948–57.

    Article  PubMed  Google Scholar 

  27. Knierim K, Palmer C, Kramer ES, Rodriguez RS, VanWyk J, Shmerling A, et al. Lessons learned during COVID - 19 that can move telehealth in primary care forward. J Am Board Fam Med. 2021;34(Suppl):S196–202.

    Article  PubMed  Google Scholar 

  28. Reed M, Huang J, Graetz I, Muelly E, Millman A, Lee C. Treatment and Follow - up Care Associated With Patient - Scheduled Primary Care Telemedicine and In - Person Visits in a Large Integrated Health System. JAMA Netw Open. 2021;4(11):e2132793.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Ryskina KL, Shultz K, Zhou Y, Lautenbach G, Brown RT. Older adults’ access to primary care: Gender, racial, and ethnic disparities in telemedicine. J Am Geriatr Soc. 2021;69(10):2732–40.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Blumenthal D, Tavenner M. The, “meaningful use” regulation for electronic health records. N Engl J Med. 2010;363(6):501–4.

    Article  CAS  PubMed  Google Scholar 

  31. Haynes RB, Del Fiol G, Michelson M, Iorio A. Context and approach in reporting evaluations of electronic health record - based implementation projects. Ann Intern Med. 2020;172(11 Suppl):S73–8.

    Article  PubMed  Google Scholar 

  32. Tsai CH, Eghdam A, Davoody N, Wright G, Flowerday S, Koch S. Effects of electronic health record implementation and barriers to adoption and use: a scoping review and qualitative analysis of the content. Life. 2020;10(12):327.

    Article  PubMed  PubMed Central  Google Scholar 

  33. Pagán VM, McClung KS, Peden CJ. An observational study of disparities in telemedicine utilization in primary care patients before and during the COVID-19 pandemic. Telemed J E Health. 2022;28(8):1117–25. https://doi.org/10.1089/tmj.2021.0412.

  34. Baker MC, King SL, Sikka N, Krupinski EA, Shipman SA, Haberman M. Trends in adoption and maturation of telehealth programs at teaching hospitals and health systems. Telemed J E Health. 2022;28(4):517–25.

    Article  PubMed  Google Scholar 

  35. Cole MB, Lee EK, Frogner BK, Wright B. Changes in performance measures and service volume at US Federally qualified health centers during the COVID - 19 pandemic. JAMA Health Forum. 2023;4(4):e230351.

    Article  PubMed  PubMed Central  Google Scholar 

  36. Hoppe KK, Smith M, Birstler J, Kim K, Sullivan - Vedder L, LaMantia JN, et al. Effect of a Telephone Health Coaching Intervention on Hypertension Control in Young Adults: The MyHEART Randomized Clinical Trial. JAMA Netw Open. 2023;6(2):e2255618.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Snoswell CL, Chelberg G, De Guzman KR, Haydon HH, Thomas EE, Caffery LJ, Smith AC. The clinical effectiveness of telehealth: a systematic review of meta-analyses from 2010 to 2019. J Telemed Telecare. 2023;29(9):669–84. https://doi.org/10.1177/1357633X211022907.

  38. Apathy NC, Dixit RA, Boxley CL, Adams KT, Booker E, Ratwani RM. Variations in Physician Telemedicine Provision. JAMA Netw Open. 2023;6(7):e2321955.

    Article  PubMed  PubMed Central  Google Scholar 

  39. WHO. Digital Implementation Investment Guide (DIIG): Integrating Digital Interventions into Health Programmes. 2020.

    Google Scholar 

  40. Telehealth policy changes after the COVID - 19 public health emergency: Department of Health and Human Services. 2023. Available from: https:// telehealth.hhs.gov / providers / telehealth - policy / policy - changes - after - the - covid - 19 - public - health - emergency.

  41. Goodman RB, Lacktman NM, Ferrante TB. CMS Finalizes Changes for Telehealth Services for 2023. Available from: https:// www.foley.com / en / insights / publications / 2022 / 11 / cms - finalizes - changes - telehealth - services - 2023.

Download references

Acknowledgements

Not applicable.

Funding

Research reported in this work was funded through a Patient-Centered Outcomes Research Institute (PCORI) Award (COVID - 2020C2 - 10791). The views in this work are solely the responsibility of the authors and do not necessarily represent the views of the Patient - Centered Outcomes Research Institute (PCORI), its Board of Governors or Methodology Committee. The authors gratefully acknowledge input from the Stakeholder Advisory Committee and co-investigators of the overall project, “Evaluating the Comparative Effectiveness of Telemedicine in Primary Care: Learning from the COVID-19 Pandemic”.

Author information

Authors and Affiliations

  1. Icahn School of Medicine at Mount Sinai, 1 Gustave L. Levy Pl, New York, NY, 10029, USA

    Logan D. Cho & Grace Rabinowitz

  2. Department of Population Health Science & Policy, Icahn School of Medicine at Mount Sinai, 1425 Madison Avenue Box 1077, New York, NY, 10029, USA

    Crispin Goytia & Carol Horowitz

  3. Department of Population Health Sciences, Weill Cornell Medical College, 402 E. 67Th Street, New York, NY, 10065, USA

    Katerina Andreadis, Natalie C. Benda & Rainu Kaushal

  4. Institute for Healthcare Delivery Science, Department of Population Health Science & Policy, Icahn School of Medicine at Mount Sinai, 1425 Madison Avenue Box 1077, New York, NY, 10029, USA

    Hsin - Hui Huang & Jashvant Poeran

  5. Department of Orthopedics, Icahn School of Medicine at Mount Sinai, 1425 Madison Avenue Box 1077, New York, NY, 10029, USA

    Hsin - Hui Huang & Jashvant Poeran

  6. Department of Medicine, Division of General Internal Medicine, Icahn School of Medicine at Mount Sinai, 17 E. 102nd Street Box 1087, New York, NY, 10029, USA

    Jenny J. Lin

  7. Department of Biomedical Informatics, Vanderbilt University Medical Center, 2525 West End Ave., Rm 14122, Nashville, TN, USA

    Jessica S. Ancker

  8. Department of Medicine, Icahn School of Medicine at Mount Sinai, 1425 Madison Avenue Box 1077, New York, NY, 10029, USA

    Jashvant Poeran

Authors
  1. Logan D. Cho
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Grace Rabinowitz
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Crispin Goytia
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Katerina Andreadis
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hsin - Hui Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Natalie C. Benda
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jenny J. Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Carol Horowitz
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Rainu Kaushal
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Jessica S. Ancker
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Jashvant Poeran
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

LDC, CG, KA, NCB, JJL, CH, RK, JA, and JP all contributed to study conceptualization and design. LDC and JP finalized the survey. LDC, NCB, and JP administered the survey to practice leaders. LDC managed the collected survey data. HH performed the data reduction analysis. LDC, GR, and JP contributed to primary manuscript writing and editing. All authors edited and approved the final manuscript.

Corresponding author

Correspondence to Jashvant Poeran.

Ethics declarations

Ethics approval and consent to participate

Institutional Review Board approval was obtained from BRANY IRB, applying to all participating institutions. All methods were carried out in accordance with relevant guidelines and regulations. Informed consent was obtained from all survey respondents.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: Appendix 1.

Initially identified features of telemedicine (and healthcare delivery context) based on a review of existing evidence, categorized by Triangle model domains with each feature assessed against Penchansky & Thomas ‘access - to - care’ framework elements.

Additional file 2: Appendix 2.

Final survey.

Additional file 3: Appendix 3.

Scree plot; eigenvalues (y-axis) plotted against factors/principal components (x-axis).

Additional file 4: Appendix 4.

Scree plot; explained variance (y-axis) plotted against factors/principal components (x-axis).

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cho, L.D., Rabinowitz, G., Goytia, C. et al. Development of a novel instrument to characterize telemedicine programs in primary care. BMC Health Serv Res 23, 1274 (2023). https://doi.org/10.1186/s12913-023-10130-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12913-023-10130-5

Keywords

BMC Health Services Research

ISSN: 1472-6963

Contact us