Skip to main content
Download PDF

Electronic health record–integrated approach for collection of patient-reported outcome measures: a retrospective evaluation

BMC Health Services Research volume 21, Article number: 626 (2021) Cite this article

Abstract

Background

The integration of Patient Reported Outcome Measures (PROMs) into clinical care presents many challenges for health systems. PROMs provide quantitative data regarding patient-reported health status. However, the most effective model for collecting PROMs has not been established. Therefore the purpose of this study is to report the development and preliminary evaluation of the standardized collection of PROMs within a department of orthopedic surgery at a large academic health center.

Methods

We utilized the Users’ Guide to Integrating Patient-Reported Outcomes in Electronic Health Records by Gensheimer et al., 2018 as a framework to describe the development of PROMs collection initiative. We framed our initiative by operationalizing the three aspects of PROM collection development: Planning, Selection, and Engagement. Next, we performed a preliminary evaluation of our initiative by assessing the response rate of patients completing PROMs (no. of PROMs completed/no. of PROMs administered) across the entire department (18 clinics), ambulatory clinics only (14 clinics), and hospital-based clinics only (4 clinics). Lastly, we reported on the mean response rates for the top 5 and bottom 5 orthopaedic providers to describe the variability across providers.

Results

We described the development of a fully-integrated, population health based implementation strategy leveraging the existing resources of our local EHR to maximize clinical utility of PROMs and routine collection. We collected a large volume of PROMs over a 13 month period (n = 10,951) across 18 clinical sites, 7 clinical specialties and over 100 providers. The response rates varied across the department, ranging from 29 to 42%, depending on active status for the portal to the electronic health record (MyChart). The highest single provider mean response rate was 52%, and the lowest provider rate was 13%. Rates were similar between hospital-based (26%) and ambulatory clinics (29%).

Conclusions

We found that our standardized PROMs collection initiative, informed by Gensheimer et al., achieved scope and scale, but faced challenges in achieving a high response rate commensurate with existing literature. However, most studies reported a targeted recruitment strategy within a narrow clinical population. Further research is needed to elucidate the trade-off between scalability and response rates in PROM collection initiatives.

Peer Review reports

Background

Patient-reported outcome measures (PROMs) quantify outcomes that are important to patients such as disease symptoms, health-related quality of life, and patient satisfaction [1]. PROMs can be used to support patient-provider decision-making by standardizing how patient’s health status is reported [2]. Moreover, PROMs play an important role in advancing patient-centered care, such as providing information to guide diagnostic and treatment decisions; in value-based health care [3]; and in patient-centered comparative effectiveness trials [4, 5]. Therefore it is vital for health systems to integrate PROMs collection into routine clinical care to improve patient care, engage in shared decision-making, and complete meaningful patient-centered research [6].

Patient-related outcome information is a prerequisite for patient centered care, which has increasingly been recognized as an ethical imperative in modern health care [7]. However, the integration of PROMs into clinical care presents many challenges for health systems [8], and widespread implementation of routine collection has been limited due to clinician, staff, and patient reluctance, inadequate resources to deal with positive or unexpected results, concerns about how the data will be used, and workflow and technology challenges [9]. However, effective patient-centered care requires the routine integration of PROMs [10], and integration of PROMs into the electronic health record (EHR) is fundamental to advancing clinical care by providing quantitative data regarding patient reported patient health status [11].

Before implementation of PROMs in clinical practice, feasibility of patient burden, logistics of workflow impact, display of results, and administration frequency should be carefully evaluated, just as psychometric properties and applicability of the outcomes [12]. Through a collaboration between the International Society for Quality of Life Research and the Patient-Centered Outcomes Research Institute, researchers from Johns Hopkins University developed the Users’ Guide to Integrating Patient-Reported Outcomes in Electronic Health Records [13]. The guide provides a practical framework and examples for administrators, clinicians, researchers, information technology (IT) professionals, and others wishing to integrate patient-reported outcomes into their EHRs. It offers guidance on 11 key questions related to integration and for each question describes advantages and disadvantages of different options and identifies research gaps. As a follow-up to the Users’ Guide, Gensheimer et al. presented the 11 key questions in 3 thematic groups—Planning, Selection, and Engagement—and described the key questions and their important considerations and takeaways for implementation [14]. Addressing and understanding these key issues around PROMs-EHR integration will improve the adoption and implementation PROMs collection in a given health system.

The purpose of this study is to describe the utilization of the thematic framework by Gensheimer et al. [12], informed by the Users’ Guide [13], to develop and implement a large-scale, EHR-integrated approach to PROMs collection within the orthopedic surgery department at a large academic health center. We utilized this framework to report our PROMs implementation initiative and framed our approach using the three thematic groups: Planning, Selection, and Engagement. We performed a 13-month retrospective evaluation of the PROMs collection implementation initiative by analyzing the scope, scale, and patient collection response rates associated with this initiative. By doing this, we highlight the barriers and facilitators to implementing best practices for EHR-integrated approach to PROMs collection demonstrating the value proposition for PROM collection initiatives for health systems. Moreover, we were able to evaluate the challenges and opportunities of implementing a large-scale, real-world PROMs collection across a heterogeneous clinical department.

Methods

Study setting

The EHR-integrated PROMs collection initiative was conducted in the Department of Orthopaedic Surgery at a university health system in North Carolina. The Department of Orthopaedic Surgery PROMs collection initiative took place across 18 adult clinics (14 ambulatory and 4 hospital-based clinics) and covered 7 specialties (Joint Reconstruction, Spine, Sports Medicine, Trauma, Oncology, Foot and Ankle, and Hand) with over 100 orthopedic providers including surgeons, physician assistants, and nurse practitioners.

In 2017, as part of the Department of Orthopaedic Surgery’s strategic plan, the integration of PROMs collection in routine clinical care was identified as a departmental priority. The previous state of PROMs collection within the department was highly variable and lacked a standardized approach and associated EHR infrastructure. Moreover, access to PROMs data inside and outside the EHR was limited, which hindered a scalable implementation of PROMs collection. Therefore, a multidisciplinary team was assembled to develop a strategy for implementation of standardized PROMs collection within the department. The multidisciplinary team consisted of faculty in the department including clinical researchers with expertise in clinical informatics, data analytics, outcomes, and musculoskeletal research, as well as an orthopedic surgeon and orthopedic practice administrator. Key considerations for the multidisciplinary team when developing the PROMs collection strategy as identified in the strategic plan were cost, scalability, actionability of data, impact on operations, and patient engagement.

Integration of PROMs in the EHR: planning, selection, and engagement

To assess our strategy for standardized PROMs collection, we discussed the 11 key questions detailed in the Users’ Guide [11], as well as Gensheimer et al.’s [14] three thematic groups: Planning, Selection, and Engagement. We utilized this thematic framework to facilitate our discussions regarding our PROMs implementation initiative.

The planning stage for the PROMs implementation involved answering the following key questions: What strategy will be used for PROMs integration? How will the data be governed? Are there ethical and legal issues? How can data from EHR sources be pooled inside and across organizations? The key questions to be addressed during the selection stage for PROMs implementation included: Which populations and patients are most suitable for collection and use of PROMs data, and how can EHRs support identification of suitable patients? Which outcomes are important to measure for a given population? How should candidate PROMs be evaluated? To aid in PROM selection, we administered an electronic survey to identify the PROMs currently being collected and the PROMs they would like to use in the future. Lastly, the providers were asked for what purposes, if any, they would like to use PROMs outside of clinical care. Lastly, the engagement stage was intended for determining how, where, and with what frequency patients would respond to PROMs; how to display PROMs data within the EHR; how clinical teams would act upon PROMs data; and how to train, support, and incentivize clinical teams and patients.

Evaluation of implementation of PROMs collection initiative

After we completed the planning, selection, and engagement phases of the EHR-integrated PROMs collection initiative, we implemented standardized PROMs collection on November 17, 2017. Approximately 13 months after implementation, we performed a preliminary evaluation of the PROMs collection initiative. The primary metric we were interested in evaluating was the response rate of patients completing PROMs, a key indicator of success for PROMS collection. We reported the total number of PROMs completed and mean response rates (no. of PROMs completed/no. of PROMs administered) at baseline across the entire department (18 clinics), ambulatory clinics only (14 clinics), and hospital-based clinics only (4 clinics). Lastly, we reported on the mean response rates for the top 5 and bottom 5 orthopaedic providers to describe the variability in providers. Our PROMs administration strategy relied on the use of the EHR patient portal, MyChart, for PROMs administration. Patients who did not have access to MyChart prior to their appointment could complete the questionnaires via Welcome Tablets. Welcome tablets are EHR integrated tablets or iPads used to collect PROMs and directly communicate with the patient record. During the implementation period, Welcome tablets were only available for use in 2 clinics and not consistently used to collect PROMs. Therefore, we reported response rates in 1) MyChart active patients who filled out PROMs via patient portal only and 2) all patients (i.e., both MyChart active and not active patients). These metrics were provided to indicate the different denominators corresponding with each collection strategy for calculating response rates. All analyses were performed using STATA SE/15.1 and Tableau.

Results

Framework for PROMs collection

Planning

Our process and key questions answered during the EHR-integrated PROMs collection initiative are summarized in Table 1. The multidisciplinary group considered several key issues when exploring options for the PROMs collection strategy, including cost, scalability, access to data, and impact on clinical operations and patient burden. Two viable options for PROMs collection were then considered: 1) third-party PROMs collection with EHR integration and 2) full integration of PROMs collection within the existing EHR. The identified advantages to the third-party approach were the access to data for clinicians and researchers outside the EHR, benchmarking with other orthopedic departments and providers, ability to enroll patients in registries, and wide availability of orthopedic-specific outcome measures. The disadvantages to this approach were high annual cost per user, 2-way integration with EHR systems that would require EHR governance approval and result in potentially lengthy implementation, and concerns about clinician adoption within the workflow. The alternative approach (full integration of PROMs collection) presented minimal or no upfront cost for integration beyond time resources by the local IT team. This approach supported scalability for PROMs administration across the department with low perceived burden to clinicians or staff to access PROMs data during the clinical encounter within the existing EHR workflow. The main disadvantage of the full integration approach was the inflexibility of the EHR for customizing PROMs and data visualizations.

Table 1 Planning phase of PROMs collection implementation initiative
Full size table

Next, we considered how the data would be governed, as well as ethical matters. A centralized governance model would facilitate standardized implementation of PROMs collection, with an emphasis on coordination of data across multiple divisions and centralized ethical oversight, whereas a distributed model could hinder the standardization of PROMs collection and lead to fragmented collection and alignment of PROMs collection goals. Lastly, the multidisciplinary group discussed how the data should be stored, accessed, and used outside the clinical encounter.

After these considerations, we chose to implement a full EHR integration approach to PROMs collection. The full integration approach required that PROMs are collected from the patient through MyChart and reported directly within the EHR (MaestroCare) for providers. By doing this, providers had the ability to communicate about PROMs and results with patients through the existing patient portal. Moreover, with this approach, our goal was to incorporate the collection of PROMs as part of the standard of care. The governance of PROMs collection was to be initially overseen by the multidisciplinary group and followed by a steering committee of clinicians, operations and IT staff, and researchers within the department after the initial implementation. Our goal in planning this approach was to facilitate improved access to PROMs data for providers during the clinical encounter within the existing EHR workflow. Lastly, the full integration approach of PROMs collection limits patient burden by providing a safeguard against redundant data collection within the EHR that can occur with third-party or minimal integration approaches. In collaboration with the local analytics team, the multidisciplinary group designed a data model prototype to routinely extract PROMs directly from the EHR along with other clinically relevant data and a dashboard for data visualization.

Selection of PROMs

The process of identification of outcomes and key questions addressed in the selection of PROMs are detailed in Table 2. Providers surveyed, both operative and non-operative, were most interested in outcomes such as physical function, disability, pain, quality of life and psychosocial disposition. Providers reported PROMs were most important for improving clinical care and to benchmark their patient outcomes and they would like to use PROMs data for quality improvement and research initiatives.

Table 2 Selection phase of PROMs collection implementation initiative
Full size table

Therefore, we chose to implement a standardized population health-based system (as opposed to a tailored approach) for collecting PROMs that would be inclusive of the different provider types within the department (i.e., surgical and nonsurgical, rehabilitation) and allow for longitudinal measurement of PROMs at set time intervals. The common system that was selected was PROMIS® (Patient-Reported Outcomes Measurement Information System) [15].PROMIS is validated for use in multiple orthopedic populations and is anchored to a t-score metric allowing for consistent interpretation of findings, and it can be mapped to many region-specific or legacy measures [16, 17]. For similar reasons as selecting a population-based approach for PROMs collection (scalability, benchmarking across divisions, and actionability of data), we chose generic measures to be collected across the department. PROMIS measures have multiple domains available to quantify each of the outcomes of interest identified by providers in the orthopedics department. In the initial phases of implementation, we selected 8 PROMIS domains to collect using short forms. This decision was pragmatic and was based on the availability of PROMIS measures available within Epic’s Foundation system, which is a collection of all the functionality developed by former Epic customers, and provider preference for constructs to measure. The selection of common PROMs and domains across the department allowed providers within and across divisions to compare outcomes across all orthopedic patients, and provided the standardization needed to engage in value-based care programs and large, pragmatic trials. The long-term plan was to transition to computer adaptive testing PROMIS tools after preliminary implementation and when this technology was available within our instance of Epic.

Engagement

Provider engagement

One aim of our initiative was to engage the department in PROMs collection and use of PROMs in clinical care. To this end, we promoted engagement of providers and patients by leveraging the use of the EHR (Epic MaestroCare) and patient portal (Epic MyChart) to administer PROMs to patients and for providers to view and act upon PROMs data at the point of care. Leveraging the EHR and patient portal allowed providers the ability to communicate with patients about PROMs both during and outside the clinical encounter. Providers and clinical staff were educated on PROMs collection methods and provider EHR interaction with the PROMs data through structured presentations and materials.

To provide feedback on PROMs performance as well as visualization access to patient-level PROMs data, the Epic Synopsis Activity View was created. This allows providers to view trends in PROMs data over time and discuss with patients (Fig. 1). A time-lagged Tableau® dashboard was created to display aggregate PROMs data to be viewed at the provider, clinic location, and division level. Visualization access of standardized PROMs across all divisions allows department members to view and compare data across different patient populations and providers, with a common metric (PROMIS). Providers reported that the synopsis report was helpful for viewing PROMs as well as other clinical data, however for some this was not part of their regular workflow. Providers and PROMIS experts indicated that the strength of the dashboard was the comparison across providers and visualization of PROMs. Providers reported that response rate and regular access to the dashboard was important for continued engagement.

Fig. 1
figure 1

Synopsis activity view of PROMIS data. *Synopsis activity view has been altered to improve visualization of PROM data

Full size image

Patient engagement

PROMs were administered to patients ahead of their scheduled appointment through MyChart, providing the patients the opportunity to complete the PROMs prior to their appointment. If patients did not complete the PROMs prior to their scheduled appointment, EHR-integrated tablet computers (operating the EPIC Welcome module) were available at limited clinical sites to collect PROMs during the clinical encounter. However, the tablets were not used regularly throughout the implementation phase.

We chose to deploy the PROMs via a questionnaire series, rather than anchored to an appointment type such as all new and return visits. This was the option recommended by our EHR IT team to leverage the reporting capabilities within Epic and to reduce patient fatigue. Moreover, this approach allowed patients to complete the questionnaires outside of their clinical visit, reducing the need and potential burden associated with in-clinic collection but allowing longitudinal measurement. The visit anchor for the questionnaire series was a new patient appointment with an orthopedic provider within the 18 designated MaestroCare departments, which are reflective of the clinical locations. Once a patient completed the first PROM, the anchor rule defined by MaestroCare was met and the PROMIS questionnaire series was initiated. Through MyChart, patients would be messaged reminders to complete the follow-up PROMs. If enabled by the patient, an e-mail reminder would also be sent to patients. Follow-up time points began at 6 weeks after the completion of the initial questionnaire and then quarterly for 1 year. This approach allowed for remote collection of PROMs, aiming to decrease the burden on clinical sites and promote routine patient engagement. See Table 3 for a summary of engagement considerations.

Table 3 Engagement phase of PROMs collection implementation initiative
Full size table

Evaluation

Standardized and routine PROMs administration and collection occurred from November 17, 2017, through December 18, 2018. See Fig. 2 for patient selection. The primary metric we were interested in evaluating was the response rate of patients completing PROMs. Our selected approach was a fully integrated approach that leveraged the existing capabilities of the EHR and clinical workflows. Subsequently with this approach, we were reliant on the capabilities of the local EHR, clinical sites, and personnel to collect PROMs and track in-clinic PROMs collection. The response rate to the initial PROMs assessment across the entire department was relatively low, with a 29% (n = 10,951) response rate for all patients and a 42% (n = 9162) response rate for MyChart active-only patients. However, the volume of total patients responses in the time period was substantial (n = 10,951). When comparing ambulatory and hospital-based clinics, the response rates were very similar. In the ambulatory clinics, the response rate for all patients was 29% (n = 8459), and in MyChart active-only patients, the response rate was 42% (n = 7072). In the hospital-based clinics, the response rate for all patients was 26% (n = 2492), and 41% (n = 2090) for MyChart active-only patients. These data are summarized in Table 4.

Fig. 2
figure 2

Patient eligibility flowchart

Full size image
Table 4 Response rates for PROMs assessment
Full size table

Lastly, we sought to determine which providers had the highest initial assessment response rates as well as which ones had the lowest response rates, and assess whether there were common characteristics that were different between these groups. The mean response rate for the top 5 providers was 47.3% (n = 679/1426) for all patients and 54% (n = 652/1218) for only MyChart active patients. The mean response rate for the bottom 5 providers was 14.8% (n = 254/1713) for all patients and 22.8% (n = 252/1058) for only MyChart active patients. The provider characteristics between the top and bottom 5 patient response rates (both MyChart active patients only and all patients) differed. Regarding provider type, the top 5 were represented by primarily orthopedic surgeons, whereas the bottom 5 were mixed between orthopedic surgeons and physician assistants. Provider area of specialty differed between the top and bottom providers in that the ‘total joint’ specialty area was represented in the top 5 but not in the bottom 5. These data are summarized in Table 5.

Table 5 Characteristics of the providers with the 5 highest and lowest initial assessment response ratesa
Full size table

Discussion

We herein described the process of developing and implementing a fully EHR-integrated, population-based approach to PROMs collection using the thematic framework described by Gensheimer et al. [12], informed by the Users’ Guide to Integrating Patient-Reported Outcomes in Electronic Health Records [11]. We explored the thematic phases and addressed the 11 key questions outlined by Gensheimer et al. to determine the implementation approach for PROMs collection in our department. Lastly, we performed an evaluation of our implementation approximately 13 months after beginning the initiative to determine the scope, scale, and uptake of our PROMs collection effort. We found that in following the framework, we were able to develop an implementation strategy that met the goal of standardized PROM collection in our department as part of the strategic plan while also leveraging the existing resources of our local EHR to maximize clinical utility of PROMs and routine collection. With our initial implementation approach, we achieved a large scale of routine collection across 18 clinical sites. The scope of our initiative was broad, covering 7 clinical specialties and capturing outcomes related to visits to orthopedic surgeons, non-operative orthopedic providers, and advanced practice providers (nurse practitioners and physician assistants). Our response rate to the initial questionnaires varied across the department, with the highest single provider response rate reaching an average of 52% and the lowest provider 13%. The response rates for the department were fairly consistent between hospital-based and ambulatory clinics, but differences were seen consistently for MyChart active only versus all patients, where response rates were higher for the MyChart active patients. This is to be expected, as our PROMs collection initiative primarily relied on the use of the patient portal (MyChart). Some clinics had the Welcome tablets available, but they were not consistently used and only accounted for an 6% overall percentage of initial PROMs assessments collected.

Given substantial changes in the data “ecosystems” that support the capture and use of patient-reported outcomes in health care delivery [18], it is important to identify and understand barriers to and facilitators of the successful implementation of PROMS collection [19]. The decision-making process and challenges we faced with our initiative are similar to other studies reporting the implementation of PROMs collection. With our implementation approach we were able to successfully administer, collect, and report PROMs within the EHR and through data visualization applications as other studies in orthopedic settings have demonstrated [20,21,22]. Moreover, we decided to collect PROMIS measures as our standard measure, as opposed to region-specific or legacy measures associated with each clinical division, which allowed us to scale the collection of PROMs across our department while minimizing the complexity of EHR programming for administering and reporting our PROMs. Our approach and findings adds to the body of literature supporting use of PROMIS measures as a routinely collected set of measures in Orthopaedics [23, 24]. By integrating PROMs collection into the EHR and clinical care, we made PROMs data available and actionable for providers for clinical decision-making, to assess the patient’s experience of his or her symptoms or illness, in addition to objective findings [25]. The challenges we faced were consistent with the themes reported in other studies such as logistical and workflow issues, technological barriers, and patient engagement and activation [26, 27]. Our most notable challenge was increasing our overall response rate for PROMs collection; our rate for all patients (29%) was consistent with the statistic for general surveys (33%) [28], and our rate was slightly higher for patients who were MyChart active (42%). The variation we reported in our study is consistent with other PROMs collection initiatives reported in orthopedics (34–77% over a 1-year period [20]. However, the general response rate for PROMs collection initiatives in orthopedics appears to hover around 70–80%, with studies reporting 74.9% [29] and 72.6% [30], and the UK National Joint Registry reporting 64%, the Swedish Hip Arthroplasty Registry 79%, and the New Zealand Joint Registry 70% [31]. A notable difference between our initiative and the above-mentioned studies is that we employed a population-health based approach for PROMs collection solely within the EHR (fully integrated approach), where as these studies collected PROMs via outside collection platforms that rely on email correspondence rather than patient portals for administration. Our response rate (42%) is similar to that of another study that administered PROMIS measures through MyChart (37%) [32], confirming that the mode of administration may affect the response rates. A noted difference in our study compared to similar studies is that we scaled our PROMs collection across a large department (18 clinics) and many specialties (7 specialties). Many of the previous studies of PROMS collection were primarily conducted in one orthopedic patient population or clinical specialty, involved data capture outside the EHR, or had a targeted recruitment strategy with dedicated personnel responsible for achieving response rates of 80% [33]. We faced similar challenges as other studies employing remote PROM collection that reported low response rates (24–36%) [34,35,36,37]. This indicates that perhaps response rates for PROM collection embedded in clinical care are expected to be lower unless resources are provided to increase the rate; however, these approaches have the potential to achieve a higher volume of responses that result in more actionable data at the point of care [38, 39]. The two major barriers that we faced in improving the response rates was the reliance on the EHR to administer and collect PROMs without consistent in-clinic collection strategies and the lack of real-time monitoring of response rates to provide feedback to clinical sites on performance.

The implementation of our PROMs collection has some strengths and limitations. First, the major strengths of our study were the scale and scope of our initiative. By leveraging the existing capabilities of the EHR, we were able to collect a large volume of patient responses over a 13-month period (n = 10,951) with minimal effort by clinicians or staff, and patients were able to complete the questionnaires at their leisure. Moreover, we collected a diverse set of PROMIS measures, allowing us to make future determinations about managing our patient population and inform subsequent treatment decisions using historical patient data. The limitations of our initiative were primarily related to clinical workflow issues, tablet availability, and provider engagement with PROMs collection. Some clinical sites reported difficulties with determining which patients still needed to complete their PROMs. Moreover, each clinical site differed in the clinical workflow; therefore, one approach for PROMs collection was not feasible. During our implementation period, welcome tablets were not available at most sites. Additionally, the Epic Welcome module was a fairly new feature, and there were frequent technological issues with the Welcome tablets, resulting in inconsistent use. Lastly, continued provider engagement was challenging. During the selection process, we selected a generic PROM based on clinician feedback. However, during the implementation period, many clinicians reported still wanting to collect legacy measures or difficulty interpreting PROMIS measures.

Conclusions

We demonstrated that PROMs can be successfully collected at scale through the EHR across a large department with seven clinical orthopaedic specialties. However, we did encounter similar logistical, technical, and engagement issues as have other studies reporting PROMs collection relying primarily on remote collection [26, 27]. The findings of our study add value to the ecosystem of PROMs collection. We are the first study we are aware of to successfully administer standardized PROMIS measures across an entire department with a heterogeneous patient population through primarily remote EHR collection. This study has yielded valuable information regarding the potential trade-off between implementing a scalable, population health based approach and achieving adequate response rates. Therefore, future steps for PROMs collection should focus on improving the robustness of PROMs response rate by updating utilities within the EHR that improve communication with patients and demonstrate how PROMs support shared decision-making with providers.

Availability of data and materials

The datasets generated and/or analysed in this study are not publicly available and restrictions apply to the availability of these data. Data may be requested from the authors upon reasonable request and only after the conclusion of all analyses pertaining to this data.

Abbreviations

PROMs:

patient reported outcome measures

EHR:

electronic health record

IT:

information technology

PROMIS:

patient reported outcome measurement information system

COORDS:

comprehensive outcomes in orthopaedics and rehabilitation data system

References

  1. Anker SD, Agewall S, Borggrefe M, Calvert M, Jaime Caro J, Cowie MR, et al. The importance of patient-reported outcomes: a call for their comprehensive integration in cardiovascular clinical trials. Eur Heart J. 2014;35(30):2001–9. https://doi.org/10.1093/eurheartj/ehu205.

    Article  PubMed  Google Scholar 

  2. Wees PJVANDER, Van Der Wees PJ, Nijhuis-Van Der MW, et al. Integrating the use of patient-reported outcomes for both clinical practice and performance measurement: views of experts from 3 countries. Milbank Q. 2014;92(4):754–75. https://doi.org/10.1111/1468-0009.12091.

    Article  Google Scholar 

  3. Porter ME, Others. What is value in health care. N Engl J Med. 2010;363(26):2477–81. https://doi.org/10.1056/NEJMp1011024.

    Article  CAS  PubMed  Google Scholar 

  4. Ahmed S, Berzon RA, Revicki DA, Lenderking WR, Moinpour CM, Basch E, et al. The use of patient-reported outcomes (PRO) within comparative effectiveness research: implications for clinical practice and health care policy. Med Care. 2012;50(12):1060–70. https://doi.org/10.1097/MLR.0b013e318268aaff.

    Article  PubMed  Google Scholar 

  5. Selby JV, Beal AC, Frank L. The Patient-Centered Outcomes Research Institute (PCORI) national priorities for research and initial research agenda. JAMA. 2012;307(15):1583–4. https://doi.org/10.1001/jama.2012.500.

    Article  CAS  PubMed  Google Scholar 

  6. Deshpande PR, Rajan S, Sudeepthi BL, et al. Patient-reported outcomes: a new era in clinical research. Perspect Clin Res. 2011;2(4):137–44. https://doi.org/10.4103/2229-3485.86879.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Olde Rikkert MGM, van der Wees PJ, Schoon Y, Westert GP. Using patient reported outcomes measures to promote integrated care. Int J Integr Care. 2018;18(2):8. https://doi.org/10.5334/ijic.3961.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Sisodia RC, Dankers C, Orav J, Joseph B, Meyers P, Wright P, et al. Factors associated with increased collection of patient-reported outcomes within a large health care system. JAMA Netw Open. 2020;3(4):e202764. https://doi.org/10.1001/jamanetworkopen.2020.2764.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Basch E, Deal AM, Dueck AC, Scher HI, Kris MG, Hudis C, et al. Overall survival results of a trial assessing patient-reported outcomes for symptom monitoring during routine Cancer treatment. JAMA. 2017;318(2):197–8. https://doi.org/10.1001/jama.2017.7156.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Stover A, Irwin DE, Chen RC, et al. Integrating Patient-Reported Outcome Measures into Routine Cancer Care: Cancer Patients’ and Clinicians' Perceptions of Acceptability and Value. EGEMS (Wash DC) 2015; 3: 1169.

  11. Browne JP, Cano SJ, Smith S. Using patient-reported outcome measures to improve health care: time for a new approach. Med Care. 2017;55(10):901–4. https://doi.org/10.1097/MLR.0000000000000792.

    Article  PubMed  Google Scholar 

  12. Bingham CO, Noonan VK, Auger C, et al. Montreal accord on patient-reported outcomes (PROs) use series – paper 4: patient-reported outcomes can inform clinical decision making in chronic care. J Clin Epidemiol. 2017;89:136–41. https://doi.org/10.1016/j.jclinepi.2017.04.014.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Snyder C, Wu AW. Users’ guide to integrating patient-reported outcomes in electronic health records. Baltimore: Johns Hopkins University, https://dcricollab.dcri.duke.edu/sites/NIHKR/KR/GR-Slides-10-13-17.pdf (2017).

  14. Gensheimer SG, PRO-EHR Users’ Guide Steering Group, Wu AW, et al. Oh, the places We’ll go: patient-reported outcomes and electronic health records. Patient - Patient-Centered Outcomes Res. 2018;11(6):591–8. https://doi.org/10.1007/s40271-018-0321-9.

    Article  Google Scholar 

  15. Cella D, Riley W, Stone A, Rothrock N, Reeve B, Yount S, et al. The patient-reported outcomes measurement information system (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005–2008. J Clin Epidemiol. 2010;63(11):1179–94. https://doi.org/10.1016/j.jclinepi.2010.04.011.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Jildeh TR, Lizzio VA, Meta F, Fidai MS, Kaat AJ, Makhni EC. The correlation between PROMIS pain interference and VAS pain in ambulatory orthopedic patients. Orthopedics. 2018;41(6):e813–9. https://doi.org/10.3928/01477447-20180912-06.

    Article  PubMed  Google Scholar 

  17. Brodke DJ, Saltzman CL, Brodke DS. PROMIS for Orthopaedic outcomes measurement. J Am Acad Orthop Surg. 2016;24(11):744–9. https://doi.org/10.5435/JAAOS-D-15-00404.

    Article  PubMed  Google Scholar 

  18. Segal C, Holve E, Sabharwal R. Collecting and using patient-reported outcomes (PRO) for comparative effectiveness research (CER) and patient-centered outcomes research (PCOR): challenges and opportunities, http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.831.9388&rep=rep1&type=pdf (2013).

  19. Lavallee DC, Chenok KE, Love RM, Petersen C, Holve E, Segal CD, et al. Incorporating patient-reported outcomes into health care to engage patients and enhance care. Health Aff. 2016;35(4):575–82. https://doi.org/10.1377/hlthaff.2015.1362.

    Article  Google Scholar 

  20. Bhatt S, Davis K, Manning DW, Barnard C, Peabody TD, Rothrock NE. Integration of patient-reported outcomes in a Total joint arthroplasty program at a high-volume Academic Medical Center. JAAOS Global Res Reviews. 2020;4(5):e20.00034. https://doi.org/10.5435/JAAOSGlobal-D-20-00034.

    Article  Google Scholar 

  21. Haskell A, Kim T. Implementation of patient-reported outcomes measurement information system data collection in a private orthopedic surgery practice. Foot Ankle Int. 2018;39(5):517–21. https://doi.org/10.1177/1071100717753967.

    Article  PubMed  Google Scholar 

  22. Biber J, Ose D, Reese J, et al. Patient reported outcomes – experiences with implementation in a University Health Care setting. J Patient-Reported Outcomes. 2. Epub ahead of print 2018. DOI: https://doi.org/10.1186/s41687-018-0059-0.

  23. O’Hara NN, Richards JT, Overmann A, et al. Is PROMIS the new standard for patient-reported outcomes measures in orthopaedic trauma research? Injury. 2020;51(Suppl 2):S43–50. https://doi.org/10.1016/j.injury.2019.10.076.

    Article  PubMed  Google Scholar 

  24. Horn ME, Reinke EK, Couce LJ, et al. Reporting and utilization of Patient-Reported Outcomes Measurement Information System® (PROMIS®) measures in orthopedic research and practice: a systematic review. J Orthopaedic Surg Res; 15. Epub ahead of print 2020. DOI: https://doi.org/10.1186/s13018-020-02068-9.

  25. Tatman LM, Obremskey WT. Patient reported outcomes: the Foundation of Value. J Orthop Trauma. 2019;33(Suppl 7):S53–5. https://doi.org/10.1097/BOT.0000000000001613.

    Article  PubMed  Google Scholar 

  26. Rodriguez HP, Poon BY, Wang E, et al. Linking practice adoption of patient engagement strategies and relational coordination to patient-reported outcomes in accountable care organizations. Milbank Q. 2019;97(3):692–735. https://doi.org/10.1111/1468-0009.12400.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Slover JD, Karia RJ, Hauer C, Gelber Z, Band PA, Graham J. Feasibility of integrating standardized patient-reported outcomes in orthopedic care. Am J Manag Care. 2015;21(8):e494–500.

    PubMed  Google Scholar 

  28. What’s the average survey response rate? [2019 benchmark], https://surveyanyplace.com/average-survey-response-rate/ (2019, accessed 7 November 2020).

  29. Horevoorts NJ, Vissers PA, Mols F, et al. Response rates for patient-reported outcomes using web-based versus paper questionnaires: comparison of two invitational methods in older colorectal cancer patients. J Med Internet Res 2015; 17: e111, 5, DOI: https://doi.org/10.2196/jmir.3741.

  30. Zakaria HM, Mansour T, Telemi E, Xiao S, Bazydlo M, Schultz L, et al. Patient demographic and surgical factors that affect completion of patient-reported outcomes 90 days and 1 year after spine surgery: analysis from the Michigan spine surgery improvement collaborative (MSSIC). World Neurosurg. 2019;130:e259–71. https://doi.org/10.1016/j.wneu.2019.06.058.

    Article  PubMed  Google Scholar 

  31. Ho A, Purdie C, Tirosh O, Tran P. Improving the response rate of patient-reported outcome measures in an Australian tertiary metropolitan hospital. Patient Relat Outcome Meas. 2019;10:217–26. https://doi.org/10.2147/PROM.S162476.

    Article  PubMed  PubMed Central  Google Scholar 

  32. Wagner LI, Schink J, Bass M, Patel S, Diaz MV, Rothrock N, et al. Bringing PROMIS to practice: brief and precise symptom screening in ambulatory cancer care. Cancer. 2015;121(6):927–34. https://doi.org/10.1002/cncr.29104.

    Article  PubMed  Google Scholar 

  33. Hendra R, Hill A. Rethinking response rates: new evidence of little relationship between survey response rates and nonresponse bias. Eval Rev. 2019;43(5):307–30. https://doi.org/10.1177/0193841X18807719.

    Article  PubMed  Google Scholar 

  34. Borowsky PA, Kadri OM, Meldau JE, et al. The Remote Completion Rate of Electronic Patient-Reported Outcome Forms Before Scheduled Clinic Visits-A Proof-of-Concept Study Using Patient-Reported Outcome Measurement Information System Computer Adaptive Test Questionnaires. J Am Acad Orthop Surg Glob Res Rev; 3. Epub ahead of print October 2019. DOI: https://doi.org/10.5435/JAAOSGlobal-D-19-00038.

  35. Kortlever JTP, Janssen SJ, van Berckel MMG, et al. What Is the Most Useful Questionnaire for Measurement of Coping Strategies in Response to Nociception? Clin Orthopaedics Related Res®. 2015;473:3511–8.

    Article  Google Scholar 

  36. Nota SPFT, Sjoerd PF, Strooker JA, et al. Differences in response rates between mail, E-mail, and telephone follow-up in hand surgery research. HAND. 2014;9(4):504–10. https://doi.org/10.1007/s11552-014-9618-x.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Schønnemann JO, Larsen K, Hansen TB, Søballe K. Reliability and validity of the Danish version of the disabilities of arm, shoulder, and hand questionnaire in patients with fractured wrists. J Plast Surg Hand Surg. 2011;45(1):35–9. https://doi.org/10.3109/2000656X.2011.554708.

    Article  PubMed  Google Scholar 

  38. Nyirenda J, Sun C, Wright S, Rezek G. Feasibility testing of health information technology: enabled patient-reported outcome measurement in the home health setting. Home Healthc Now. 2019;37(4):222–6. https://doi.org/10.1097/NHH.0000000000000779.

    Article  PubMed  Google Scholar 

  39. Zhang R, Burgess ER, Reddy MC, Rothrock NE, Bhatt S, Rasmussen LV, et al. Provider perspectives on the integration of patient-reported outcomes in an electronic health record. JAMIA Open. 2019;2(1):73–80. https://doi.org/10.1093/jamiaopen/ooz001.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

We would like to acknowledge the Comprehensive Outcomes in Orthopaedics and Rehabilitation Data System (COORDS) group for supporting the infrastructure for data collection. We would also like to acknowledge Jennifer Gagnon for her contribution for data acquisition.

Funding

No researching funding was received for this study.

Author information

Authors and Affiliations

  1. Division of Physical Therapy, Department of Orthopaedic Surgery, Duke University, 311 Trent Drive, Durham, NC, 27710, USA

    Maggie E. Horn

  2. Division of Sports Medicine, Department of Orthopaedic Surgery, Duke University, 3475 Erwin Rd, Durham, NC, 27705, USA

    Emily K. Reinke & Richard C. Mather

  3. Duke-Margolis Center for Health Policy, Duke University School of Medicine, Durham, NC, USA

    Jonathan D. O’Donnell

  4. Department of Orthopaedic Surgery and Duke Clinical Research Institute, Duke University, 200 Morris Street, Durham, NC, 27001, USA

    Steven Z. George

Authors
  1. Maggie E. Horn
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Emily K. Reinke
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Richard C. Mather
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jonathan D. O’Donnell
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Steven Z. George
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

MEH, EKR, RCM, JDO and SZG made substantial contributions to conception and design of the study. MEH and EKR were responsible for the acquisition, analysis and interpretation of the data. MEH and SZG primarily drafted the manuscript. All authors approved the submitted study and agree to be accountable for their contributions.

Corresponding author

Correspondence to Maggie E. Horn.

Ethics declarations

Ethics approval and consent to participate

This study and data analysis was determined as an exempt study by the Duke University Institutional Review Board (Pro00084613).

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Horn, M.E., Reinke, E.K., Mather, R.C. et al. Electronic health record–integrated approach for collection of patient-reported outcome measures: a retrospective evaluation. BMC Health Serv Res 21, 626 (2021). https://doi.org/10.1186/s12913-021-06626-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12913-021-06626-7

Keywords

BMC Health Services Research

ISSN: 1472-6963

Contact us