- Open Access
Patient and family experience with telemedicine and in-person pediatric and obstetric ambulatory encounters throughout 2020, during the COVID-19 epidemic: the distance effect
BMC Health Services Research volume 22, Article number: 659 (2022)
Telemedicine has grown significantly in recent years, mainly during the COVID-19 pandemic, and there has been a growing body of literature on the subject. Another topic that merits increased attention is differences in patient and family experience between telehealth and in-person visits. To our team’s knowledge, this is the first study evaluating pediatric and obstetrics outpatients experience with telemedicine and in-person visit types in an academic maternal and children’s hospital, and its correlation with geographic distance from the medical center throughout 2020, during the COVID-19 crisis.
We aim to evaluate and compare patients’ telemedicine and in-person experience for ambulatory encounters based on survey data throughout 2020, during the COVID-19 pandemic, with particular focus on the influence of distance of the patient’s home address from the medical facility. A total of 9,322 patient experience surveys from ambulatory encounters (6,362 in-person and 2,960 telemedicine), in a maternal and children’s hospital during 2020 were included in this study. The percentage of patients who scored the question “Likelihood to recommend practice” with a maximum 5/5 (top box) score was used to evaluate patient experience. The k-means model was used to create distance clusters, and statistical t-tests were conducted to compare mean distances and Top Box values between telemedicine and in-person models. Logistic regression analysis was used to evaluate the correlation between Top Box scores and patients’ distance to the hospital.
Top Box likelihood to recommend percentages for in-person and telemedicine were comparable (in-person = 81.21%, telemedicine = 81.70%, p-value = 0.5624). Mean distance from the hospital was greater for telemedicine compared to in-person patients (in-person = 48.89 miles, telemedicine = 61.23 miles, p-value < 0.01). Patients who live farther displayed higher satisfaction scores regardless of the visit type (p-value < 0.01).
There is a direct relationship between the family experience and the distance from the considered medical center, during year 2020, i.e., patients who live farther from the hospital record higher Top Box proportion for “Likelihood to Recommend” than patients who live closer to the medical center, regardless of the approach, in-person or telemedicine.
The term Telemedicine is used to define the exchange and use of medical information to deliver medical services from different sites, by using electronic communication networks. [1, 2]. Although telemedicine is not a novel concept [3,4,5], it has grown significantly in recent years, in particular during the COVID-19 pandemic when non-face-to-face healthcare became essential [6, 7]. Consequently, there is increased interest on its outcomes and indications, and there has been a growing body of literature on the subject, as shown on Fig. 1.
Based on data obtained from the Scopus (Elsevier) and Web of Science databases, the pediatrics is one area with few publications (470), as showed at Fig. 2 A. Obstetrics has even less (253), and it is not showed at picture Fig. 2 A.
There has been an increase in publications for these two specifics area, mainly in recent years: in pediatrics, it is noted an increase from 55 publications in 2019 to 174 in 2020, while in obstetrics, it increased from 36 in 2019 to 92 in 2020 (Fig. 2 B).
Another topic that merits increased attention is data from surveys of patient and family experience, but specifically for the two specialties considered here, there are just publications about telemedicine and patient experience. While the ideal methodological approach to measure it is still a matter of discussion [8, 9], proxies such as patients’ experience survey scores are still the most commonly utilized tools. Literature reviews and development of frameworks for telemedicine satisfaction measurements have been proposed [10, 11]. However, to our team’s knowledge, this is the first study evaluating pediatric and obstetrics outpatients experience with telemedicine and in-person visit types in a large US-based academic medical center, and its correlation with geographic distance from the medical center during the COVID-19 crisis.
Study design and population
This is a retrospective analysis of patient’s experience survey data administered at a maternal and children’s hospital and associated clinics. Questionnaires were collected from January to December 2020 for in-person and telehealth visit types. Data included primary care and subspecialized pediatric and obstetric outpatient services.
As part of our institutional oversight of quality, safety, and service – patient experience survey scores are monitored to determine where we are improving and what targeted actions may need to be taken. Following our institutional guidelines, this project met criteria as quality improvement activity and was not considered human subjects research, and as a result did not require approval by our Institutional Review Board.
The 16-question patient experience survey was electronically collected from patients and families who underwent to medical services, in-person or via telehealth. The same 16-question patient experience survey (Table 1) was used in other studies developed at this hospital unit: “The survey is a tailored version of an ambulatory survey created in cooperation with a vendor (Press Ganey, South Bend, IN), validated and used at multiple healthcare systems throughout the United States” .
Each question was evaluated on a scale of 1–5. A grade of five (5/5) was considered Top Box . The organization considers the percentage of Top box scores as a standard metric for patient experience.
For the purposes of this study, the answers to the question 13—Likelihood of your recommending our practice to others (LTR) were considered for further analysis. This metric has been extensively studied, and is frequently utilized in patient experience improvement efforts [13,14,15].
Patient’s distance to the medical facility was calculated by an interface with Google Maps service utilizing the zip code for each patient’s residence. The Top Box data was then stratified by micro-regions, and the averages were weighted by the number of responses from each zip code.
Zip codes were clustered according to distance to the medical center, and a k-means model, a clustering algorithm that associates an identification to each group of elements that share the same cluster, was used .
The distance between the elements and each centroid is calculated using a cartesian plane with clusters on the x-axis and distances on the y-axis. The number of centroids defines the number of desired clusters. If only one centroid is considered, we will have only one cluster. With two or more defined centroids, each element is associated with the centroid closest to it. Since the number of centroids (or clusters) desired must be indicated in the algorithm, a calculation was made to determine the ideal number of clusters to minimize the sum of the distances of each element to the centroids. In a range from 1 to 15 centroids, the amount that minimizes this sum and, therefore, the ideal number of clusters, was three .
As all patients living within the same zip code were considered to have the same distance to hospital, distance outliers were eliminated. To compare mean values for Top Box and distance for telehealth and in-person groups, a statistical t-test was conducted. To evaluate the relationship between distance and visit type (independent variables) with Top Box (dependent variable), a logistic regression model was used. All statistical analysis was made using R Studio .
A total of 9,322 patient experience surveys (for 6,362 in-person and 2,960 telemedicine encounters) were analyzed, from pediatric and obstetrics outpatients encounters during year 2020. They were answered by patients or families living in 30 American states, 406 cities, and 694 zip codes.
Table 2 shows that most micro-regions were located less than 193.77 miles from the center, with a mean of 248.16 and a median of 93.44 miles. This big difference between mean and median is influenced by 126 individual patient surveys, which addresses have the farthest 93 zip codes, with distance greater than 421.61 miles (limit value to be statistically considered an outlier) and although few, significantly influenced the means. Therefore, we excluded the evaluations from these 93 zip codes as they were considered outliers, leaving 9,196 individual patient encounters. Therefore, 98.6% of the patient encounters and 86.6% of the original zip codes were retained in the study.
For clarity, clusters were renamed to represent the average distance from the cluster to the hospital as "Close," "Intermediate," and "Far". According to their distances to the hospital, the cluster compositions are presented in Fig. 3, which is a representation of zip codes and clusters. For “Close”, the median distance from the zip code to the hospital was 46.1 with a range of 0.0 to 104.3 miles. For “Intermediate”, the median distance from the zip code to the hospital was 159.0 with a range of 105.7 to 229.3 miles. For “Far”, the median distance from the zip code to the hospital was 288.0, with a range of 233.7 to 419.3 miles.
The distribution of zip codes and the associated mean score for “likelihood to recommend” are presented in Fig. 4, where each zip code is represented by a circle, and the size of the circle represents the number of evaluations in that micro-region. Note that most experience scores in the “Close” cluster are above the value of 50% for Top Box LTR.
Average patient experience by visit type (in-person or telemedicine), and number of completed surveys are shown in Fig. 5. Despite monthly variations in the Top Box percentages for each modality, the t-test comparing the mean Top Box between two visit types, showed that there was no statistical difference between the average of telemedicine and in-person service evaluations (in-person = 81.21%, telemedicine = 81.70%, p-value = 0.5624) for the time period of calendar year 2020.
Average patient distance by visit type
The relationship between average distance from the patient’s home and the healthcare center by visit type is shown in Fig. 6.
Comparing the average distance of consultations by telemedicine relative to in-person, t-test analyses demonstrate a statistical difference (in-person = 48.90 miles, telemedicine = 61.24 miles, p-value < 0.01) confirming that the average distance of telemedicine services was greater than average distance of patients on in-person visit type, as shown in Table 3.
Patient top box evaluation by distance and visit type
Considering that distance is a relevant factor for telemedicine (the prefix ‘tele’, from the Greek ‘telos’, implies distance ), we analyzed the relation between visit type, distance and Top Box proportion for LTR for patient surveys during 2020. The variable Top Box was created from transformation of variable LTR: if the LTR was equal to 5, Top Box value was defined to 1; if the LTR was less than 5, the Top Box value was defined to 0. Figure 7 shows that there is a statistically significant relation between average patient’s distance from hospital and Top Box proportion for LTR. The greater the average distance, the greater the proportion of Top Box. This trend is valid for both telemedicine and in-person services.
The Top Box proportion for the “Close” cluster was 78.2% for in-person and 79.3% for telemedicine. For the “Intermediate” it was 86.7% for in-person and 84.7% for telemedicine, and for the “Far” cluster, the proportion of Top Box was 91.8% for in-person and 89.3% for telemedicine.
Logistic regression confirmed that there was a statistically significant difference between the Top Box proportion, used as dependent variable, and patient distance and service type considered as the independent variables, all considering 95% confidence interval. This result was found for regression using individual evaluations where Top Box was the dependent variable and patient distance was the independent variable (OR [1.0065; 1.0083] and p-value < 0.01). The residual analysis confirmed a well-adjusted model. This is the same result for regression using cluster Top Box proportion as dependent and cluster average distance as the independent variable: p-value < 0.01 for "Intermediate" (OR [2.5471; 3.8600]) and p-value < 0.01 for "Far" (OR [1.6794; 1.9731]) considering "Close" as the baseline.
In our geographic region, location is highly related to median household income . Our healthcare center is located in the middle of Silicon Valley in which many zip codes have high median household incomes. Patients who travel farther distances to our medical center often come from the Central Valley (Fig. 3). Many of the zip codes in this region have a lower median household income. It is possible that patients and families from areas with lower median household incomes are more appreciative and provide higher patient experience scores. This is speculative but one of the limitations to our evaluation of differences Further studies are required on this topic.
In conclusion, our analysis revealed a direct relationship between the proportion of Top Box evaluations, as a proxy for patient and family experience with pediatric and obstetric services, and the distance from the considered medical center, during year 2020, i.e., patients who live farther from the hospital record higher Top Box proportion for “Likelihood to Recommend” than patients who live closer to the medical center, regardless of the approach, in-person, or telemedicine. The possible reason for this direct relationship was not evaluated and could be analyzed in future research.
Availability of data and materials
The datasets generated and/or analyzed during the current study are not publicly available due to Stanford Medicine's institutional policies, as quality/performance data access is limited, but are available from the corresponding author on reasonable request.
Sood S, Mbarika V, Jugoo S, Dookhy R, Doarn CR, Prakash N, et al. What is telemedicine? A collection of 104 peer-reviewed perspectives and theoretical underpinnings. Telemed e-Health. 2007;13(5):573–90.
Tuckson RV, Edmunds M, Hodgkins ML. Telehealth. N Engl J Med. 2017;377(16):1585–92.
Yan Z, editor. Encyclopedia of Cyber Behavior. Albany: University At Albany; 2012. p. 1632.
ATA - American Telemedicine Association. Telehealth Basics [Internet]. [cited 2020 May 5]. Available from: https://www.americantelemed.org/resource/why-telemedicine/
Chan S, O’Riordan A, Appireddy R. Exploring the Determinants and Experiences of Senior Stroke Patients with Virtual Care. Can J Neurol Sci / J Can des Sci Neurol. 2020;48(1):87–93.
Crawford C. Inside Look at Using Telemedicine During COVID-19 Pandemic [Internet]. American Academy of Family. 2020 [cited 2020 May 5]. Available from: https://www.aafp.org/news/health-of-the-public/20200323covidtelehealth.html
Gadzinski AJ, Gore JL, Ellimoottil C, Odisho AY, Watts KL. Implementing Telemedicine in Response to the 2020 COVID-19 Pandemic. J Urol [Internet]. 2020;204(July 2020):1–3. https://doi.org/10.1097/JU.0000000000001033.
Whitten PS, Mair F. Telemedicine and patient satisfaction: current status and future directions. Telemed J E Health. 2000;6(4):417–23.
Mair F, Whitten P. Systematic review of studies of patient satisfaction with telemedicine. BMJ. 2000;320(7248):1517–20.
Garcia R, Adelakun O. Towards a Conceptual Framework for the Evaluation of Telemedicine Satisfaction. In: Rocha Á, Adeli H, Reis LP, Costanzo S, editors. Trends and Advances in Information Systems and Technologies. Chicago: Springer International Publishing; 2018. p. 1188–97.
Law T, Cronin C, Schuller K, Jing X, Bolon D, Phillips B. Conceptual framework to evaluate health care professionals’ satisfaction in utilizing telemedicine. J Am Osteopath Assoc. 2019;119(7):435–45.
Bomher ST, Wood M, Uhlhorn E, Marques S, Kwiatkowski L, Tekic N, et al. Comparing patient experience survey scores between telehealth and in-person ambulatory pediatric subspecialty visits. Patient Exp J. 2021;8(3):100–8.
Peng FB, Burrows JF, Shirley ED, Rosen P. Unlocking the Doors to Patient Satisfaction in Pediatric Orthopaedics. J Pediatr Orthop. 2018;38(8):398–402.
Camilon PR, Levi JR, Carrion RA, Josephson GD. Physician impact on the patient and family experience in a pediatric otolaryngology practice. Laryngoscope. 2019;129(11):2610–3.
Deitrick LM, Capuano TA, Paxton SS, Stern G, Dunleavy J, Miller WL. Becoming a leader in patient satisfaction: Changing the culture of care in an Academic community Hospital. Health Mark Q. 2007;23(3):31–57.
Skansi S. Introduction to Deep Learning: From Logical Calculus to Artificial Intelligence. Cham: Springer; 2018. p. 191.
RStudio Team. RStudio: Integrated Development Environment for R [Internet]. Boston, MA; 2020. Available from: http://www.rstudio.com/
Zundel KM. Telemedicine: history, applications, and impact on librarianship. Bull Med Libr Assoc. 1996;84(1):71–9.
Tamayo AL, Espinós EL, González ER, Guzmán CT, Álvarez-Maestro M, de Guerín C C, et al. Evaluation of teleconsultation system in the urological patient during the COVID-19 pandemic. Actas Utológicas Esp. 2020;44(9):617–22.
Wood K, Mehri N, Hicks N, Vivoda JM. Family Satisfaction: Differences Between Nursing Homes and Residential Care Facilities. J Appl Gerontol. 2020;40:073346482097152.
Makkar A, Mccoy M, Hallford G, Foulks A, Anderson M, Milam J, et al. Evaluation of Neonatal Services Provided in a Level II NICU Utilizing Hybrid Telemedicine: A Prospective Study. Telemed e-Health. 2020;26(2):176–83.
Hasson SP, Waissengrin B, Shachar E, Hodruj M, Fayngor R, Brezis M, et al. Rapid Implementation of Telemedicine During the COVID-19 Pandemic: Perspectives and Preferences of Patients with Cancer. Oncol [Internet]. 2021;26:e679-85. Available from: http://www.TheOncologist.com.
U.S. Census Bureau. American Community Survey [Internet]. 2019 [cited 2021 Mar 14]. Available from: https://data.census.gov/cedsci/
This research did not have any form of funding from the university or third parties, being fully funded by its authors.
Ethics approval and consent to participate
All methods were carried out in accordance with relevant guidelines and regulations.
All experimental protocols were approved by Stanford University’s Research Compliance Office.
Following our institutional guidelines, this project met criteria as quality improvement activity and was not considered human subjects research by Stanford University’s Research Compliance Office (our Institutional Review Board), and as a result, received exemption of approval documented by e-mail.
Consent for publication
This manuscript does not contain any individual person’s data or image.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
Marques, S., Cruz, J.A.W., da Cunha, M.A.V.C. et al. Patient and family experience with telemedicine and in-person pediatric and obstetric ambulatory encounters throughout 2020, during the COVID-19 epidemic: the distance effect. BMC Health Serv Res 22, 659 (2022). https://doi.org/10.1186/s12913-022-08037-8
- Patient Experience
- Quality improvement
- In-person visits
- Likelihood to Recommend
- Maternal and children’s hospital
- Logistic regression