Evaluating diverse electronic consultation programs with a common framework
BMC Health Services Research volume 18, Article number: 814 (2018)
Electronic consultation is an emerging mode of specialty care delivery that allows primary care providers and their patients to obtain specialist expertise without an in-person visit. While studies of individual programs have demonstrated benefits related to timely access to specialty care, electronic consultation programs have not achieved widespread use in the United States. The lack of common evaluation metrics across health systems and concerns related to the generalizability of existing evaluation efforts may be hampering further growth. We sought to identify gaps in knowledge related to the implementation of electronic consultation programs and develop a set of shared evaluation measures to promote further diffusion.
Using a case study approach, we apply the Reach, Effectiveness, Adoption, Implementation and Maintenance (RE-AIM) and the Quadruple Aim frameworks of evaluation to examine electronic consultation implementation across diverse delivery systems. Data are from 4 early adopter healthcare delivery systems (San Francisco Health Network, Mayo Clinic, Veterans Administration, Champlain Local Health Integration Network) that represent varied organizational structures, care for different patient populations, and have well-established multi-specialty electronic consultation programs. Data sources include published and unpublished quantitative data from each electronic consultation database and qualitative data from systems’ end-users.
Organizational drivers of electronic consultation implementation were similar across the systems (challenges with timely and/or efficient access to specialty care), though unique system-level facilitators and barriers influenced reach, adoption and design. Effectiveness of implementation was consistent, with improved patient access to timely, perceived high-quality specialty expertise with few negative consequences, garnering high satisfaction among end-users. Data about patient-specific clinical outcomes are lacking, as are policies that provide guidance on the legal implications of electronic consultation and ideal remuneration strategies.
A core set of effectiveness and implementation metrics rooted in the Quadruple Aim may promote data-driven improvements and further diffusion of successful electronic consultation programs.
The demand for specialty care is growing and is predicted to exceed specialist workforce capacity in the United States by 2025 . Electronic consultation, which provides a way for primary care providers (PCPs) and their patients to obtain specialty expertise without an in-person patient visit to the specialist, has emerged as a mode of specialty care delivery that may address the impending supply-demand mismatch for specialty care.
Studies of individual early adopter electronic consultation programs in North America have demonstrated benefits consistent with the overarching goal of achieving the quadruple aim : high-quality care delivery that improves population health [3,4,5] and achieves high satisfaction among providers [6,7,8], care-team members, and patients,  while introducing cost-savings from new efficiencies [10, 11]. Two systematic reviews that included international programs have also reported that electronic consultation programs are associated with improved access to specialty care and enhanced care coordination, high satisfaction among primary and specialty care providers, and positive patient experience [12, 13].
Despite these data, electronic consultation programs have not achieved widespread use in the United States. The use of variable evaluation metrics across health systems and concerns related to the generalizability (or lack thereof) existing evaluation efforts may be hampering further growth, alongside financial and cultural barriers. Using a case study approach with four very different health delivery systems, we apply the Reach, Effectiveness, Adoption, Implementation and Maintenance (RE-AIM)  and the Quadruple Aim  frameworks of evaluation to highlight the generalizability of electronic consultation program implementation. In so doing, we also identify gaps in knowledge to promote areas of future study and propose a set of shared evaluation measures to promote future study.
Electronic consultations (e-consults) are asynchronous provider-provider consultations that occur within an electronic health record (EHR) or a web-based portal. E-consults allow providers, most commonly PCPs, to access specialist input for clinical questions that may be addressable without an in-person specialty care visit. Electronic consultation systems allow specialists to address consultative needs through chart review and communication with the referring provider, and ensure adequate diagnostic workup prior to a face-to-face patient visit when necessary. There are three main types of electronic consultation systems. (1) They can exist independently or (2) in parallel to referral processes (either electronic – known as e-referrals -- or paper/fax based), which carry the expectation of a specialty care visit. (3) Alternatively, some are integrated into the overall referral process, creating integrated electronic consultation and referral systems, in which all requests for specialty expertise are electronically initiated and specialist reviewers respond to each request with the appropriate form of consultation, including scheduling the patient for an in-person visit.
Study design and data sources
We used a case study design to explore the impact of electronic consultation system implementation across 4 diverse healthcare delivery systems. The systems were selected because they represent varied organizational structures, care for different patient populations, have well-established multi-specialty electronic consultation programs that represent the three types of systems, and have undergone independent evaluations of program implementation and local impact (Table 1). Data sources from each health delivery system included published quantitative data from each system’s electronic consultation database and qualitative data from each system’s end-users. Additionally, some unpublished data from each system were provided by co-authors.
Evaluation frameworks: RE-AIM and quadruple AIM
The five constructs of the RE-AIM framework  (Reach, Effectiveness, Adoption, Implementation, Maintenance) formed the basis for analysis in this study along with the Quadruple Aim framework to further refine Effectiveness measures. RE-AIM is well-suited to our case study as it addresses critical aspects of implementation in real-world settings and identifies facilitating and hindering factors to success, which are particularly important for external validity. Additional file 1: Table S1 summarizes the dimensions of these validated frameworks with associated definitions and evaluation questions pertinent to electronic consultation programs.
The Reach and Effectiveness constructs reflect patient-level measures and together, they measure the impact of an intervention on a given population. Reach considers the extent to which a sample of participants reflects the overall eligible population for that intervention. Effectiveness measures reflect short-term and/or long-term patient-level outcomes, including unintended positive and negative consequences. They can be categorized into 4 domains using the Quadruple Aim framework: Population Health, Patient Satisfaction, Care Team Satisfaction and Financial Implications . Measures of Adoption and Implementation are organizational measures that focus on translatability of an intervention to other settings. Adoption is related to the absolute number, proportion and representativeness of providers who use a new program. Implementation refers to the extent and fidelity of the intervention compared to intended design/workflow, including barriers and enablers to local implementation and program modifications for success. The Maintenance dimension of the framework addresses the extent to which the intervention is sustainably embedded within routine organizational practice (Additional file 1: Table S1).
Four case studies were performed, examining electronic consultation implementation in systems that represent different structures of specialty care delivery: a community setting with non-affiliated individual PCP and specialty care practices; an integrated tertiary care academic medical center; an urban safety-net system with affiliated PCPs and specialists; and a fully integrated public delivery system (Table 1).
Champlain local health integration network (LIHN)
The Champlain BASE™ (Building Access to Specialists through eConsultation) program was developed in 2009. It is based in the Champlain health region of Eastern Ontario, Canada, which has a population of 1.3 million individuals, roughly half of whom reside in the city of Ottawa . Independent PCPs who sign up for the BASE service can submit e-consults to any specialist who has agreed to provide this service, from any internet-connected device via the service’s secure SharePoint platform, attaching any files they deem pertinent (e.g. images, test results). A BASE program manager allocates the e-consult to a specialist in the specialty group selected by the PCP. That specialist responds with expert advice, a recommendation for referral, or a request for additional information. This service exists in parallel to existing referral practices in the Champlain health region and is not embedded into any particular EHR. Specialist reviewers are remunerated for their e-consult activities using the fee schedule for in-person consultation pro-rated to time spent, from a combination of regional, provincial, and research funding.
This is an academic not-for-profit tertiary care academic medical center with three major campuses in Rochester, Minnesota; Scottsdale, Arizona; and Jacksonville, Florida. E-consults for the primary care population were offered to PCPs in 2008. In 2010 the program was expanded to allow Mayo specialists to request e-consults from other Mayo specialists. The electronic consultation program exists in parallel to the electronic referral system that providers use to order face-to-face specialty visits; both systems are embedded into the EHR, so that specialist reviewers can review any background data to complete e-consults. Completed e-consults are available in the EHR and the patient portal for subsequent review by ordering providers and patients, respectively . Specialist reviewers at the Mayo clinic are salaried, do not receive additional compensation to perform e-consults but do receive some protected time.
San Francisco Health Network (SFHN)
The SFHN is San Francisco’s publicly funded safety-net healthcare delivery system. All specialty services are provided at the acute care hospital, Zuckerberg San Francisco General Hospital and Trauma Center (ZSFG). Requests for specialty care come from fourteen SFHN primary care clinics and an affiliated network of twelve community health centers. The referral base consists of approximately 100,000 patients. Since 2007, all requests for ZSFG specialty services are initiated as e-consults through the electronic referral and consultation program that is embedded into the EHR . ZSFG specialists are all salaried; specialist reviewers receive work credit for their e-consult activities.
Veterans administration (VA)
The VA is the largest integrated healthcare system in the United States, caring for over 6 million Veterans annually. Specialists are based at approximately 170 VA Medical Centers (VAMC). Most VAMCs are affiliated with one or more of the 1000 community-based outpatient centers located between a few to 300 miles from the parent facility. The VA launched its electronic consultation program in 2011 with variable implementation across the VAMCs. Among participating VAMCs, any provider with ordering privileges can request an e-consult within the EHR in the same manner as they would a traditional face-to-face referral; the two systems exist in parallel. Patient data is available for review within the EHR. Specialists have the option to convert e-consult requests to face-to-face referrals and vice versa. VA providers are usually salaried and specialists receive workload credit based on the time spent to complete each e-consult.
Application of the RE-AIM and Quadruple Aim frameworks to the evaluation of each electronic consultation implementation is detailed below. Key data from each system pertinent to measures within each RE-AIM dimension and Quadruple Aim domain are also summarized in Table 2.
Across the four systems, the annual number of e-consults ranges from 10,000 to 443,600, representing a range of e-consults/1000-patient lives of 1 to 378. Demographic characteristics of patients who receive e-consults have been sparsely published, though VA data suggest that patients who received an e-consult lived in more rural areas than those who received a face-to-face visit . In SFHN, the population that receives an e-consult is similar to its overall racially/ethnically diverse patient population.
Population Health. Timely access to specialty care is the main measure that has been used to evaluate the effectiveness of each electronic consultation system on population health. Two systems have reported a mean e-consult response time of 1–2 business days; the other two systems report that 91–95% of e-consults were completed within 3 business days. Wait times for in-person specialty care visits, measured by third next available specialty care in-person appointments, a widely-accepted metric of wait time for primary care access published by the Institute for Healthcare Improvement , declined in SFHN as a result of implementing the electronic consultation program . These data are not available for the other systems. The percentage of e-consults associated with an avoidable in-person visit ranges from 37 to 82% in the delivery systems where e-consult is an adjunct to the traditional referral system and is approximately 23% in SFHN, where all requests for non-urgent ambulatory specialty care are initiated as e-consults [5, 20,21,22].
None of the systems in this case study have reported data on patient level outcomes, although providers consistently perceive e-consults as delivering high quality specialty care for patients. In prior survey studies, over 90% of PCPs participating in the BASE and Boston VA electronic consultation systems believed that e-consults provided faster specialty care for their patients and nearly three-quarters of Mayo clinic and SFHN PCPs agreed that e-consults provide “good medical care” [6, 8, 23, 24]. Additional data from each system demonstrate that 50–60% of specialists agree that e-consults provide good medical care [8, 24, 25].
All four healthcare delivery systems have documented positive and negative unintended consequences related to their electronic consultation implementation. In the BASE and SFHN systems, approximately 85–90% of PCPs self-reported high levels of educational value inherent to e-consult communication . Additionally, 3.4% of e-consults in the BASE system resulted in initiation of a face-to-face referral when not originally contemplated by the PCP, a potentially important safety indicator . Potential patient harms associated with e-consults have been evaluated at the Mayo clinic, in two specialties at SFHN, and in 5 specialties in the VA. A chart review of 187 e-consults with recommendations from the most common e-consult specialties at the Mayo clinic demonstrated that specialist e-consult recommendations were not completed 10% of the time . In SFHN, among patients who received an e-consult and were not scheduled for an in-person general surgery or gastroenterology face-to-face appointment, 1–2% of patients experienced emergency department visits or hospital stays related to the content of their e-consult . However, none of these harms were clearly due to lapses in communication during the consultation process. In one VA medical center, only 0.8% of 61,931 e-consults to 5 medical specialties did not have a specialist response. Chart review demonstrated that 6.3% of e-consults did not have appropriate specialist follow-up after the initial communication and 7.4% did not have appropriate documented PCP follow-up .
Patient satisfaction. Patient satisfaction with e-consults has been evaluated in three of the four systems with surveys and focus groups and has been generally positive. Nearly one-half of patients who received an initial in-person endocrinology consultation reported being open to the idea of receiving an e-consult from the BASE system instead, considering it a viable alternative to many but not all face-to-face appointments for diabetes and endocrine conditions . In SFHN, focus group participants also appreciated improved access to specialty care with e-consult but voiced a desire to receive more information about the PCP-specialist communication . A group of 15 Veterans from the Pittsburg VA who completed a survey about e-consults reported a mean satisfaction score of 5/5 on a Likert scale .
Care Team Satisfaction. Published survey data from BASE and VA systems suggest > 80% of PCPs and > 50% of specialists are satisfied with the electronic consultation system [23, 24]. Unpublished data from SFHN and Mayo Clinic are similar. Interviews with PCPs and specialists from three systems corroborate these data [7, 31, 32].
Financial Implications. Only two of the four systems in this case series have available cost data; both have suggested cost savings attributed to decreased face-to-face specialty care appointments and decreased travel costs incurred by patients. Taking into account the technological start-up costs of electronic consultation implementation, administrative and personnel costs associated with system maintenance, specialist remuneration based on time-spent per e-consult communication, and intentionally avoided specialist in-person visits based on PCP report, the BASE system calculated an average cost of $16.71 CAD per e-consult and a break-even point for the system with a total volume of 7818 e-consults . A national VA study examining 217,000 e-consults to 13 different specialties between 2011 and 2013 suggested $2,800,000 savings from avoided patient travel to in-person specialty care visits. The VA evaluation did not take into account any fixed costs associated with implementation of the electronic consultation system .
All four systems are multi-specialty, with over 50 participating medical, surgical, pediatric, women’s health, and mental health specialties. In 2016, the percentage of PCPs that requested e-consults for their patients using the BASE, Mayo and SFHN systems ranged from 75 to 100%, with a greater percentage in the system that mandated the use of e-consults for initial ambulatory specialty care requests. Such data are not readily available from the VA, due to its decentralized implementation. In all U.S, systems, participating PCPs included family physicians, internal medicine physicians, nurse practitioners and physician assistants; pediatricians also used the Mayo and SFHN services [26, 32, 33]. Only family physicians and nurse practitioners submitted e-consults in the BASE system . Demographic characteristics and level of experience of PCPs who submit e-consults versus those who do not are not available.
Poor access to specialty care and existing inefficiencies in the traditional referral process were the common main drivers for implementation across all health systems [23, 24, 35, 36]. Despite this overarching similarity, local reinforcing factors and challenges influenced implementation [36,37,38]. For example, the types of e-consult specialties offered by each system were influenced by requests from local PCPs as well as the presence/absence of local specialty physician champions. Local strategies to ensure success that were similar across all 4 systems included primary care workflow re-design and active involvement of physician champions. Customizable templates and integration with the EHR were local reinforcing factors that were unique to the VA. Implementation barriers common to all four sites included increased and/or different PCP and specialist workload, lack of clinical oversight, and concern about the legal implications of electronic communication.
Each of these four electronic consultation systems started as pilot projects with few participating specialties and referring providers. Submitting e-consults is now routine practice in each system. Common reinforcing factors for growth have included: investment in a core implementation team with a program manager, health information technology expert, PCP champion, and specialist champion; development of fair reimbursement strategies for providers; and a commitment to system quality improvement . In the VA, widespread growth has been attributed to de-centralized activities. Decisions about workflow management and template development have been delegated to local specialty services, allowing hundreds of individual services differing in size, staffing, wait times, and clinical settings, to tailor the e-consult program according to local needs and constraints . This contrasts with the BASE and SFHN system, where e-consult activities have been centrally managed.
Electronic consultation is a disruptive innovation that has challenged the status quo of specialty health care delivery traditionally limited to face-to-face patient-provider encounters [19, 32]. Disruptive innovations often provide the most value to individuals or groups unable to participate in successful status quo processes . It is not surprising that early adopters of electronic consultation systems, including some in this case series, have been motivated by challenges in access to, or efficiency of, specialty care delivery through traditional face-to-face consults alone. As this analysis demonstrates, while the organizational drivers of implementation were similar across our early adopter health systems, unique system-level facilitators and barriers influenced local adoption and implementation, ultimately affecting the final design of each system. Despite these differences, implementation of this disruptive innovation was consistently associated with improved patient access to timely, perceived high-quality specialty expertise and few negative consequences.
Disruptive innovation theory suggests that widespread dissemination of electronic consultation programs will occur only when their positive impact is clearly obvious and the quality of care delivery is acceptable to all populations, including those that were perhaps less vulnerable at the outset. Further dissemination of electronic consultation will thus depend on the collection and publication of data to comprehensively demonstrate its value and to close existing gaps in knowledge related to Reach, Effectiveness and Adoption. In one of their landmark reports, the Institute of Medicine stated that standardized metrics for health and health care could provide benchmarks for progress and system performance . With that guiding principle, we propose a set of metrics for electronic consultation rooted in the Quadruple Aim that stem directly from the implementation analysis depicted in this manuscript (Table 3). These metrics have been vetted by diverse stakeholders in electronic consultation, including leaders in healthcare delivery, payers, and professionals with expertise in quality, and provide an important step forward to promote high-value dissemination. Next steps will need to include validation on a national level through a formal consensus-building process.
By applying the RE-AIM framework for program implementation, we were able to identify characteristics associated with greater Reach and Adoption of electronic consultation programs. Consistent with data from newer programs, local penetration across patient and PCP populations was highest in the delivery system with an integrated electronic consultation and referral system, in which an e-consult was the initial step in the referral pathway . Overall reach was lower in systems with separate electronic consultation and electronic referral systems, though PCP adoption in those systems was higher when a priority was placed on primary care system re-design. By contrast, Effectiveness measures were similar amongst the systems. E-consult response time was excellent in all 4 sites, as was the perceived quality of care inherent to electronic consultation, the educational value to referring providers, and overall program satisfaction by diverse stakeholders. Importantly, these findings from early U.S. and Canadian adopters are consistent with newer U.S. programs [42,43,44,45]. Analyses of European e-consult systems have also reported similar data: levels of provider satisfaction and educational benefit exceeding 90% [46, 47] and response time for humanitarian e-consults for international providers who are geographically isolated from specialists have been reported to be 1–3 days .
The use of validated frameworks for comparative program evaluation facilitates the identification of knowledge gaps. From this analysis, we note a paucity of data about which patient populations are directly benefiting from e-consults. Demographic data for patients who receive an e-consult compared to the general population served by a healthcare delivery system would be helpful for stakeholders concerned about replicating health inequities associated with prior technology-driven programs . Additionally, while electronic consultation systems have improved timely access to specialty expertise, data associating improved clinical outcomes and electronic consultations are lacking. One lone trial has demonstrated decreased emergency department utilization among patients randomized to receive a cardiology e-consult compared to those randomized to participate in the usual cardiology referral process . Since the clinical benefits of in-person specialty care visits have not been robustly demonstrated for all specialties, the lack of a gold standard to which e-consults can be compared may be contributing to the paucity of data and is one reason why we have not included clinical benefits in our list of proposed core evaluation metrics. Nevertheless, this area represents a large gap in the Effectiveness literature and may warrant a large, multi-specialty clinical trial. Similarly, the financial implications and return on investment for electronic consultation implementation need to be further examined.
This study also demonstrates that Implementation and Maintenance could be facilitated by policies that provide guidance on the legal implications of electronic consultation, ways to provide clinical oversight, and ideal remuneration strategies to account for increased provider workload while remaining aligned with the strategic directions of ongoing health care reform. The California Medicaid 2020 waiver  is a promising policy example that promotes the use of e-consult across California, as it provides financial incentives for specialty care re-design among public delivery systems, including funds for non-traditional specialty care encounters.
Despite providing important data about the implementation of electronic consultation programs across diverse healthcare delivery systems, the analysis is limited by the small number of programs included in this case series. Also, this study does not prove a direct relationship between electronic consultation program implementation and patient outcomes, such as specialty care access. Nonetheless, the comparable magnitude of impact across the four programs pertinent to Reach and Effectiveness is reassuring, as are similar data that have emerged from newer programs.
Successful implementation of electronic consultation programs is occurring across diverse health care delivery systems in North America, consistently improving timely access to specialist expertise. The use of an evaluation framework with common metrics, as proposed here, is important to promote ongoing adoption and diffusion. As additional reach, outcome, safety, and cost data emerge from existing and nascent programs, electronic consultation may become a standard mode of specialty care delivery.
Building Access to Specialists through eConsultation
Electronic health record
Local Health Integration Network
Primary care provider
Reach, Effectiveness. Adoption, Implementation, Maintenance
San Francisco Health Network
Veterans Affairs Medical Centers
Zuckerberg San Francisco General Hospital
Dall Tim WT, Ritashree C, Will I. The Complexities of Physician Supply and Demand: Projections from 2014 to 2025. Washginton, DC: IHS Inc. for the Association of American Medical Colleges; 2016.
Bodenheimer T, Sinsky C. From triple to quadruple aim: care of the patient requires care of the provider. Ann Fam Med. 2014;12(6):573–6.
Olayiwola JN, Anderson D, Jepeal N, Aseltine R, Pickett C, Yan J, et al. Electronic consultations to improve the primary care-specialty care Interface for cardiology in the medically underserved: a cluster-randomized controlled trial. Ann Fam Med. 2016;14(2):133–40.
Cruz ML, Gleason N, Wang M, Wrenn K, Gonzales R. Transforming the endocrine consult: asynchronous provider consultations. Endocr Pract. 2015;21(5):514–21.
Tuot DS, Murphy EJ, McCulloch CE, Leeds K, Chan E, Chen AH. Leveraging an electronic referral system to build a medical neighborhood. Healthc (Amst). 2015;3(4):202–8.
Kim Y, Chen AH, Keith E, Yee HF Jr, Kushel MB. Not perfect, but better: primary care providers' experiences with electronic referrals in a safety net health system. J Gen Intern Med. 2009;24(5):614–9.
Liddy C, Afkham A, Drosinis P, Joschko J, Keely E. Impact of and satisfaction with a new eConsult service: a mixed methods study of primary care providers. J Am Board Fam Med. 2015;28(3):394–403.
Angstman KB, Adamson SC, Furst JW, Houston MS, Rohrer JE. Provider satisfaction with virtual specialist consultations in a family medicine department. Health Care Manag (Frederick). 2009;28(1):14–8.
Keely E, Traczyk L, Liddy C. Patients' perspectives on wait times and the referral-consultation process while attending a tertiary diabetes and endocrinology Centre: is Econsultation an acceptable option? Can J Diabetes. 2015;39(4):325–9.
Gleason N, Prasad PA, Ackerman S, Ho C, Monacelli J, Wang M, et al. Adoption and impact of an eConsult system in a fee-for-service setting. Healthc (Amst). 2016.
Liddy C, Deri Armstrong C, Drosinis P, Mito-Yobo F, Afkham A, Keely E. What are the costs of improving access to specialists through eConsultation? The Champlain BASE experience. Stud Health Technol Inform. 2015;209:67–74.
Liddy C, Drosinis P, Keely E. Electronic consultation systems: worldwide prevalence and their impact on patient care-a systematic review. Fam Pract. 2016;33(3):274–85.
Vimalananda VG, Gupte G, Seraj SM, Orlander J, Berlowitz D, Fincke BG, et al. Electronic consultations (e-consults) to improve access to specialty care: a systematic review and narrative synthesis. J Telemed Telecare. 2015;21(6):323–30.
Glasgow RE, Klesges LM, Dzewaltowski DA, Estabrooks PA, Vogt TM. Evaluating the impact of health promotion programs: using the RE-AIM framework to form summary measures for decision making involving complex issues. Health Educ Res. 2006;21(5):688–94.
Pecina JL, North F. Early e-consultation face-to-face conversions. J Telemed Telecare. 2016;22(5):269–76.
Chen AH, Kushel MB, Grumbach K, Yee HF, Jr. Practice profile. A safety-net system gains efficiencies through 'eReferrals' to specialists. Health Aff (Millwood) 2010;29(5):969–971.
Kirsh S, Carey E, Aron DC, Cardenas O, Graham G, Jain R, et al. Impact of a national specialty e-consultation implementation project on access. Am J Manag Care. 2015;21(12):e648–54.
Institute for Healthcare Improvement. Third Next Available Appointment [Available from: http://www.ihi.org/resources/Pages/Measures/ThirdNextAvailableAppointment.aspx].
Chen AH, Murphy EJ, Yee HF Jr. eReferral--a new model for integrated care. N Engl J Med. 2013;368(26):2450–3.
Pecina JL, Frank JM, North F. A retrospective study on how primary care providers manage specialists' recommendations after an e-consultation. SAGE Open Med. 2016;4:2050312116682127.
Tran C, Liddy C, Pinto N, Keely E. Impact of question content on e-consultation outcomes. Telemed J E Health. 2016;22(3):216–22.
Singh H, Esquivel A, Sittig DF, Murphy D, Kadiyala H, Schiesser R, et al. Follow-up actions on electronic referral communication in a multispecialty outpatient setting. J Gen Intern Med. 2011;26(1):64–9.
Liddy C, Rowan MS, Afkham A, Maranger J, Keely E. Building access to specialist care through e-consultation. Open Med. 2013;7(1):e1–8.
McAdams M, Cannavo L, Orlander JD. A Medical Specialty e-Consult Program in a VA Health Care System. Federal Practitioner. 2014;31(5):26–31.
Keely E, Drosinis P, Afkham A, Liddy C. Perspectives of Champlain BASE specialist physicians: their motivation, experiences and recommendations for providing eConsultations to primary care providers. Stud Health Technol Inform. 2015;209:38–45.
Kwok J, Olayiwola JN, Knox M, Murphy EJ, Tuot DS. Electronic consultation system demonstrates educational benefit for primary care providers. J Telemed Telecare. 2017; 1357633X17711822.
Liddy C, Drosinis P, Fogel A. Keely E. Canadian Family Physician: Prevention of delayed referrals through the Champlain BASE eConsult Service; 2017. [in press]
Ulloa JG, Russell MD, Chen AH, Tuot DS. A cohort study of a general surgery electronic consultation system: safety implications and impact on surgical yield. BMC Health Serv Res. 2017;17(1):433.
Olayiwola JN, Knox M, Dube K, Lu EC, Woldeyesus T, James IE, et al. Understanding the potential for patient engagement in electronic consultation and referral systems: lessons from one safety net system. Health Serv Res. 2017.
Rodriguez KL, Burkitt KH, Bayliss NK, Skoko JE, Switzer GE, Zickmund SL, et al. Veteran, primary care provider, and specialist satisfaction with electronic consultation. JMIR Med Inform. 2015;3(1):e5.
Straus SG, Chen AH, Yee H Jr, Kushel MB, Bell DS. Implementation of an electronic referral system for outpatient specialty care. AMIA Annu Symp Proc. 2011;2011:1337–46.
Gupte G, Vimalananda V, Simon SR, DeVito K, Clark J, Orlander JD. Disruptive innovation: implementation of electronic consultations in a veterans affairs health care system. JMIR Med Inform. 2016;4(1):e6.
Pecina JL, North F. A retrospective analysis of medical record use in e-consultations. J Telemed Telecare. 2016.
Liddy C, Deri Armstrong C, McKellips F, Keely E. A comparison of referral patterns to a multispecialty eConsultation service between nurse practitioners and family physicians: the case for eConsult. J Am Assoc Nurse Pract. 2016;28(3):144–50.
Tuot DS, Leeds K, Murphy EJ, Sarkar U, Lyles CR, Mekonnen T, et al. Facilitators and barriers to implementing electronic referral and/or consultation systems: a qualitative study of 16 health organizations. BMC Health Serv Res. 2015;15:568.
Kirsh SR, Ho PM, Aron DC. Providing specialty consultant expertise to primary care: an expanding spectrum of modalities. Mayo Clin Proc. 2014;89(10):1416–26.
North F, Uthke LD, Tulledge-Scheitel SM. Internal e-consultations in an integrated multispecialty practice: a retrospective review of use, content, and outcomes. J Telemed Telecare. 2015;21(3):151–9.
Houston MS, Myers JD, Levens SP, McEvoy MT, Smith SA, Khandheria BK, et al. Clinical consultations using store-and-forward telemedicine technology. Mayo Clin Proc. 1999;74(8):764–9.
Hwang J, Christensen CM. Disruptive innovation in health care delivery: a framework for business-model innovation. Health Aff (Millwood). 2008;27(5):1329–35.
Vital signs. Core metrics for health and health care progress. Washington: The National Academies Press; 2015. IOM (Institute of Medicine)
Barnett ML, Yee HF Jr, Mehrotra A, Giboney P. Los Angeles safety-net program eConsult system was rapidly adopted and decreased wait times to see specialists. Health Aff (Millwood). 2017;36(3):492–9.
Gleason N, Prasad PA, Ackerman S, Ho C, Monacelli J, Wang M, et al. Adoption and impact of an eConsult system in a fee-for-service setting. Healthc (Amst). 2017;5(1–2):40–5.
Lee BJ, Forbes K. The role of specialists in managing the health of populations with chronic illness: the example of chronic kidney disease. BMJ. 2009;339:b2395.
Wasfy JH, Rao SK, Kalwani N, Chittle MD, Richardson CA, Gallen KM, et al. Longer-term impact of cardiology e-consults. Am Heart J. 2016;173:86–93.
Davis A, Gilchrist V, Grumbach K, James P, Kallenberg R, Shipman SA. Advancing the primary/specialty care Interface through Econsults and enhanced referrals. Ann Fam Med. 2015;13(4):387–8.
Patterson V, Wootton R. A web-based telemedicine system for low-resource settings 13 years on: insights from referrers and specialists. Glob Health Action. 2013;6:21465.
Wootton R, Liu J, Bonnardot L. Quality Assurance of Teleconsultations in a store-and-forward telemedicine network - obtaining patient follow-up data and user feedback. Front Public Health. 2014;2:247.
Wootton R, Geissbuhler A, Jethwani K, Kovarik C, Person DA, Vladzymyrskyy A, et al. Long-running telemedicine networks delivering humanitarian services: experience, performance and scientific output. Bull World Health Organ. 2012;90(5):341–7D.
Arcaya MC, Figueroa JF. Emerging trends could exacerbate health inequities in the United States. Health Aff (Millwood). 2017;36(6):992–8.
California Health and Human Services Agency (2016). California Medi-Cal 2020 Demonstration, 11-W-00193/9.
This work was supported by the Blue Shield of California Foundation. This funding body had no role in the study design, data analysis/interpretation or manuscript preparation.
Availability of data and materials
All data presented were obtained with permission from co-authors and leaders from the four delivery systems. These data are not publically available for routine use by other investigators, but may are available from the corresponding author on reasonable request.
Ethics approval and consent to participate
This study was approved by the University of California, San Francisco Institutional Review Board (study number: 11–05827). This study did not include direction patient contact and thus formal consent was not obtained.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Table S1. RE-AIM dimensions, definitions and evaluation questions pertinent to electronic consultation implementation. Description: The 5 RE-AIM domains are listed, with definitions of each domain, and example metrics or questions for each domain pertinent to the implementation of electronic consultation. (DOCX 31 kb)
About this article
Cite this article
Tuot, D.S., Liddy, C., Vimalananda, V.G. et al. Evaluating diverse electronic consultation programs with a common framework. BMC Health Serv Res 18, 814 (2018). https://doi.org/10.1186/s12913-018-3626-4