- Research article
- Open Access
Evaluation of real-time use of electronic patient-reported outcome data by nurses with patients in home dialysis clinics
BMC Health Services Research volume 17, Article number: 439 (2017)
Internationally, the use of patient-reported outcomes (PROs) is increasing. Electronic PROs (ePROs) offer immediate access of such reports to healthcare providers. The objectives of this study were to assess nurses’ perspectives on the usefulness and impact of ePRO administration in home dialysis clinics and assess patient perceptions of satisfaction with nursing care following use of ePROs.
A concurrent, longitudinal, mixed methods study was conducted over 6 months during home dialysis outpatient clinic visits in two cities. Patients (n = 99) provided ePROs using tablet computers when they visited the clinic on two consecutive occasions approximately 3 months apart. Results were scored, printed, and given to nurses before patient appointments. Patients completed satisfaction items from the Comox Valley Nursing Centre Client questionnaire following their appointments. All clinic nurses (n = 11) participated and they were each interviewed twice, three months and six months after the start of the study.
The five themes that emerged from the interviews with the nurses include: enhancing focus of the nurses, directing interdisciplinary follow-up, offering support to patients through the process, interpreting results from the visual display, and integrating into workflow.
Scores on the Client Questionnaire suggested that patients believed that they received excellent care (97%), and that the nurses perfectly understood their needs (90.9%). However, their satisfaction with care did not change over time when ePRO data was repeatedly provided to their nurses.
Nurses reported that sharing ePRO data in real-time informed their practice. Although there was no statistically significant change in patient satisfaction scores over time, some patients reported changes and benefits from the use of ePROs. Further research is needed to provide guidance about how ePRO data could enhance person-centered care.
Internationally, the use of patient-reported outcomes (PROs) such as health-related quality of life (HRQOL) is increasing in clinical settings [1,2,3]. Patients’ reports of their own health are essential and integral components to providing person-centered care. In people with chronic kidney disease (CKD), who experience many symptoms and impairments in HRQOL, these outcomes are particularly important. These symptoms could be assessed more accurately with use of PROs, and results from PROs could be used for collaborative planning to enhance HRQOL. It is possible that use of ePROs can improve communication between patients and their care providers resulting in better more individualized care and greater satisfaction of patients with their care. Hence, the objectives of this study were to assess patient perceptions of satisfaction with nursing care following use of ePROs and to assess nurses’ perspectives on the usefulness and impact of electronic PRO (ePRO) administration in home dialysis clinics.
Electronic PROs (ePROs) [4,5,6,7,8] offer healthcare providers immediate access to reports , and integration of ePROs into routine practice offers instant receipt of results with opportunities to discuss them with patients [10, 11]. Researchers have found that this enhances communication and effectiveness of the multidisciplinary team [6, 9, 12]. With a few exceptions, the usefulness and impact of ePROs for patients themselves, or for nurses, has rarely been explored.
There has been conflicting evidence relating to the benefits of use of PROs in clinical practice. For instance, in a qualitative study, Wolpert et al.  sought the views of young people, mothers and clinicians about use of PROs in mental health and diabetes services. While clinicians perceived that PROs offered opportunities for more individualised care, patients were concerned that treatment would be prioritized over therapeutic relationships. Patients were also concerned about how their results would be used, and questioned the safety of revealing their satisfaction with care in case it reflected poorly on their care providers. In a systematic review of the impact of PRO use on patient outcomes, Boyce and Browne  found weak evidence to support the claim that providing PRO feedback to healthcare professionals positively impacted patient outcomes. These two studies raise the question: how beneficial are PROs to patients?
The potential benefits of the use of PROs may be influenced by how healthcare providers use PROs. In a theory-driven approach, Greenhalgh  described a taxonomy of applications for PROs in clinical practice that included their use as: tools for screening or monitoring, methods to promote person-centered care, decision aids, strategies to facilitate communication in multidisciplinary teams, and means of monitoring quality of care. In cancer care, Greenhalgh et al.  qualitatively studied how physicians referred to PROs in their patient consultations. They found that while use of PRO data may have enabled patients to elaborate on their concerns, physicians did not always know how to respond. Like Edbrooke-Childs et al.,  Greenhalgh et al. used their research to inform the development of training materials for doctors about when and how to use these instruments. Takeuchi et al.  also recommended training for physicians in their follow-up on use of PROs. In their study on patient-physician communication, they found that clinicians tended to focus on symptom intervention, but not on patients’ reports of functional concerns. While nurses’ use of PROs has not been studied in detail, Hilarius et al.  found that incorporating HRQOL assessments in daily nursing practice increased communication related to HRQOL issues and significantly improved nurses’ awareness of patients’ experiences of pain, level of functioning, and overall quality of life (QOL).
While there are numerous studies involving use of PROs in CKD and transplantation [19,20,21], there is a significant gap in our understanding of how nephrology practitioners use PROs in patient care . This gap exists despite the fact that: a) nephrology experts agree that person-centered care is a priority both for quality improvement and research [23,24,25,26]; and b) QOL has been identified as the health outcome that is most valued by CKD patients, and is most useful in dialysis care assessment [25, 27]. To our knowledge, kidney patients’ views on the usefulness of PROs in their care have not been examined. Thus, the focus of our paper is not on PRO results, but rather on nurses’ and patients’ assessment of real-time use of ePROs. The objectives of this concurrent, longitudinal, mixed-methods study [28, 29] were to assess nurses’ perspectives on the usefulness and impact of ePRO administration in home dialysis clinics and assess patient satisfaction with nursing care following use of ePROs in the clinics.
The quantitative ePRO data was gathered from patients who attended outpatient home dialysis clinics in two cities on the west coast of Canada over a 6-month timeframe. Letters were sent to all patients who participated in the pilot phase of the project , inviting them to again participate and encouraging them to arrive early. Inclusion criteria were: being on dialysis at home (either peritoneal or hemodialysis), over 19 years old, and willing to participate. Exclusion criteria included lack of proficiency in English, inability to read, moderate or severe cognitive impairment, or medical crisis.
All nurses in both clinics were invited to participate in the study. They learned about the project through presentations to clinic staff. None of the nurses knew members of the research team before the interviews and learned about the purpose of the study through the presentation.
Measures and measurement
Three survey instruments were used to collect data for the larger study: 1) The Edmonton Symptom Assessment System for renal patients (ESAS-r:Renal) [31,32,33]. 2) The Kidney Disease Quality of Life-36 (KDQOL-36) . (ESAS-r:Renal and KDQOL-36 data are not reported in this article) 3) And 14 satisfaction items that were selected and adapted from the Comox Valley Nursing Centre Client Questionnaire , previously tested for construct validity and internal consistency . This measure was selected to assess whether patients were satisfied with aspects of their nursing care, including the use of ePROs .
Two ePROs were designed as apps to be run using the FileMaker Go iPad app using iOS 7. The KDQOL-36 was completed online using KDQOL Complete (provided by Medical Education Institute ). Further details about the technology, design, measurement equivalence , and internet access are outlined in an article on the pilot phase of the study .
E-PRO intervention and procedure
Patients typically attend a clinic every 3 months; all home dialysis patients from these two clinics were invited to participate by providing ePROs on two consecutive occasions. A $10 gift card from a grocery store was provided both times they participated.
In order to keep the clinics on schedule, participants were asked to arrive 10–15 min early to participate in the study so as to not delay appointments. When patients came for their regularly scheduled appointments, those who met the inclusion criteria were invited to participate by a third party. If the patient was interested, a research assistant in the waiting room obtained informed consent and showed them how to use the tablet computer (iPad™). No other assistance was provided in completing the measures.
After completion of the KDQOL-36 and ESAS-r:Renal on the tablet computer, results were printed for the nurses who reviewed the graphically depicted scores with the patients (see Fig. 1 for sample display). Nurses were instructed to review the ePRO scores with the patients and discussions focused on the interpretation of results. They had received education about use of the ePROs, but specific norms or guidelines for responses to scores were not suggested. Patients were provided with their own PRO results, as well as general patient education materials available from the KDQOL-Complete platform if requested (see https://www.kdqol-complete.org/pdfs/KDQOL-Complete-Brochure.pdf).
Following their appointments, patients completed the Client Questionnaire on the tablet computer. This measure was selected to assess whether patients were satisfied with aspects of their nursing care, including the use of ePROs . (See Fig. 2 for summary of procedures used).
Data collection from nurses: Qualitative component
Interviews were conducted by two female research assistants, both master’s prepared nurses with prior qualitative research experience who were enrolled in doctoral programs. All nurses in the two clinics were invited to participate in two interviews, one mid-way through the study (3 months) and another at the end of the study (6 months). All 11 clinic nurses participated. One nurse was not available for the 2nd interview; in total, 21 interviews were conducted. Interviews ranged in length from 30 to 60 min, and they were conducted in a private location in a clinic at a time convenient for the nurses, often on a lunch break or after work. The interviews were semi-structured (not pilot tested) and started with a guiding question such as, “Can you please describe your experience with use of the ePROs in the clinic?” Nurses were asked to provide examples from their practice of using the ESAS-r:Renal and/or KDQOL-36. In the second interview, nurses were asked to describe examples of how they may have initiated interdisciplinary follow-up based on a patient’s ePRO responses, how their practice may have been affected by receiving ePRO results, what the perceived benefits were for patients, and how they would like ePRO information to be integrated into existing and future work structures. Participants received a $10 gift card for a grocery store as a token of appreciation. Interviews were digitally recorded and transcribed verbatim by a professional transcriptionist. The interviewer took field notes regarding important issues for follow-up in the second interviews or in analysis.
Qualitative component: Data analysis and rigor
Qualitative data analysis was undertaken using interpretive description, a qualitative methodology used in applied clinical fields [38, 39]. An inductive approach to analysis was employed and data were saturated. The first author, assisted by two research assistants, conducted the analyses. To facilitate the analysis, quotes were highlighted in the electronic transcripts, reflexive notes were created, and quotations were organized into groupings. An audit trail was maintained tracking the analytic decisions. Preliminary findings were shared with nurse participants, and they were invited to provide either written or verbal feedback.
Quantitative data analysis
Descriptive statistics (means, standard deviations) were calculated for each satisfaction item. Paired t-tests and Hotelling’s T-square (MANOVA) were used to compare changes in each satisfaction item over time. With Hotelling’s T-square, the item “satisfaction with help received” was held as the fixed variable. SPSS (version 21) was used.
IRB approval was obtained from the University of Alberta Health Research Ethics Board (Approval # Pro 00040538) and the Vancouver Island Health Authority (Approval # H2013–065) for all components of the study. All participants provided a written signed informed consent, verbally confirmed at the start of the study and prior to each interview or survey. Consent for publication of individual data was obtained from all participants.
Characteristics of the patient participants are presented in Table 1. The mean age was 67 years (range from 34 to 88 years), 83% identified as Caucasian, 68% were retired, and 86% were on peritoneal dialysis. Forty-two percent reported previous tablet use.
On average, the 11 clinic nurses had 19 years of nursing experience, 14 years of experience working in renal settings (other than home dialysis), 4 years of experience in home dialysis, and national certification in nephrology nursing (55%). Most held a RN diploma (64%) and 46% worked permanent part-time. In Table 2, additional characteristics of the nurse participants are provided.
Qualitative component: Nurse interviews
Themes identified in the nurse interviews include: provides focus: “be patient-centred”; directs interdisciplinary team follow-up: “it’s a team thing”; offers support: “brings an awareness”; visual display: “it jumps out at you”; and integration with workflow: “long-term picture”. The themes are described below and additional sample quotes are provided in Table 3.
Provides focus: “be patient-centred”
Many nurses said that they looked at the ePRO data immediately so that they could focus on the priorities or concerns of the patient. Nurse 5 said, “I look at it [ePROs] before I start the interview. I focus on it first - those are the things that are important to the patient, rather than focus on what's important to me.” Keeping the patient “in the middle of the visit” was a way for nurses to “be patient-centred.” The nurses emphasized the need to address patients “as a whole person” while simultaneously focusing on a specific physical symptom or a psychosocial issue.
The nurses saw the ePROs as “tools” that helped them focus their nursing assessment. As Nurse 2 said, the ePRO results “trigger us, as nurses, to dig into it a bit more, ask more questions, and maybe investigate a little bit more.” As the nurses narrowed their clinical assessment, the ePRO data helped them identify areas that otherwise might have been missed. Nurse 5 said “I might not ever ask them about something that was rated really high on the survey…if I hadn't seen that, I probably would not have focused in on that one item.” They believed that by focusing their assessments, they saved time and altered how they did their work. And as Nurse 8 commented, “has it changed your practice? I think it does.”
Directs interdisciplinary team follow-up: “It’s a team thing”
The nurse participants worked in interdisciplinary home dialysis clinics that involved social workers, dietitians, and nephrologists. Each patient came to an outpatient clinic for 2–2.5 hours and met individually with each clinician. Patient-nurse interviews averaged 20 minutes, but if required, they ran up to 1 hour. While the nurses emphasized they had always worked collaboratively with their colleagues, they spoke in great detail about how they highlighted patients’ priorities to their colleagues for interdisciplinary follow-up. They communicated significant findings from the ePRO data with their team members either verbally, in writing, or via email.
Nurses included patients in this team follow-up. As Nurse 4 said, “I'll give a verbal report with the patient too because they're part of that discussion and things that we have highlighted.” The clinic nurses also encouraged patients to initiate conversations about their own needs with any one of the other clinicians. Nurse 6 said, “people will speak to who they're comfortable speaking to.” Seeing patients as an integral part of the team, Nurse 1 explained, “because it's a team thing we need - everybody has to be on the same page so we can all help the patient work through the particular concerns that they're having.”
Offers support: “brings an awareness”
The nurses believed that the process of completing the ePROs provided support to patients. In the clinic waiting rooms, completion of the ePROs sparked conversations between patients, and between patients and family members. Nurse 8 recounted, “Sometimes you see the dynamic in the waiting room where the spouse will be making comments, looking over their shoulder saying, ‘No, I wouldn't put that!’” The nurses also believed that this process facilitated patient engagement in their own personal health. Nurse 10 said, “I think it brings an awareness to them in that moment and makes them think about each individual thing so it's like right at the front of their brain when they go in to talk to the nurse.” The nurses perceived that this “patient-driven” process provided patients with “support”, “acknowledgement”, “improved quality of life”, and the sense of “feeling heard.”
The nurses believed that it may have been easier for the patients to initially complete the ePROs rather than talk about the items, especially for challenging topics such as depression, anxiety, sex, or pain. Nurse 2 wondered, “maybe they're embarrassed a little bit or it's easier to put it on a screen maybe than talk about it?” The nurses thought that the numeric scores offered “safety” to the patient to express concerns in numbers and not words. Nurse 3 explained, “It's easier for them (pause) on a scale like than to say, ‘like this is really bothering me’ because sometimes when they use descriptive words it doesn't sound as (pause) bothersome as when you put in a number.” The nurses believed that many patients downplayed their concerns verbally, often saying “I’m fine, I’m fine”, and then scoring these issues much higher on ePROs.
Visual display: “it jumps out at you”
The nurses emphasized that their use of ePRO data was impacted by how the data were visually displayed. ePRO results were printed in color: blue bars for the ESAS-r:Renal (see Figure 1), and red, yellow, and green pie charts for the KDQOL-36 domains. Nurse 11 explained, the colors “gave me a quick glance of things I should address.” And Nurse 6 said, “it jumps out at you more 'cause of the highlighting.”
Many nurses offered suggestions for how the ePRO result display could have been presented in a more user-friendly format. Some requested that data be displayed in one column (instead of two), and others wished that the font had been larger. The visual display of the results was very important to their ability to interpret results quickly and integrate them into their clinical assessments and follow-up.
Integration with workflow: “long-term picture”
The nurse clinicians explained that their use of the ePRO data was impacted by whether or not the data was integrated with their workflow. They wished that ePRO data could be displayed longitudinally, in graphs, to see trends over time. Instead, they found themselves reading through charts to gain historical perspective. “You kind of like to have a general long-term picture, not just like one, 20-minute interview” (Nurse 3). Seeing those trends were particularly important to nurses caring for patients whose health was declining or were “at their end of life.”
The nurses admitted that they spent more time focusing on the ESAS-r:Renal (rather than the KDQOL-36) because they were familiar with it and it was required to be part of the health authority data registry. The nurses’ lack of familiarity with the KDQOL-36 necessitated that they spend more time with it, and sometimes that time wasn’t available. Nurse 8 explained, “If I had two things to look at, I'm going to choose the one that I'm familiar with because it takes me more time to figure that [KDQOL-36] out.” The nurses also pointed out that there was a lot of “overlapping” between the mandated ESAS-r:Renal, their nursing assessment checklists, and unit charting documents. Nurse participants were hopeful that when ePRO results were shared between clinicians, then they could “reduce the number of questions” that were repeatedly asked to each patients during each clinic visit.
Quantitative component: Satisfaction with care
Over the 6-month period, 126 home dialysis patients attended the two outpatient clinics. Ninety-nine patients participated (79% response rate) during the first 3 months. Sixty-nine patients participated at the second point of data collection, 3 months later. Reasons for not participating a second time included not having a second scheduled visit (n = 18), declining health or death (n = 9), moving to a different city (n = 1), changing appointments to telehealth modality (n = 1), and withdrawing from the study (n = 1).
At the second point of data collection (3 months later), after providing ePROs, patients reported being very satisfied with the amount of help they received (86.4%), as well as the services they received at the clinic (92.4%). They believed that the services they received helped them deal more effectively with their concerns or problems (71.2%). Patients rated the quality of the services they received as excellent (97%), and reported that the nurses perfectly understood the kind of help that they wanted (90.9%).
Patient participants were asked to report on the impact of providing ePROs to their nurses. The second point of satisfaction data collection, 3 months later, is presented in Table 4. Forty-one and a half percent (41.5%) of patients identified that providing ePROs to their nurses was “just the same” as in previous clinics, and 44% reported that their experiences of care were also “just the same”. Also, some patients reported that they felt a great deal healthier (31.8%), more hopeful (31.8%), and more supported (65.2%) due to their visit to the clinic. Some (38.2%) reported using the emergency room “a great deal” less, 39.4% reported being “a great deal” more aware of what they could do to improve their situation, and 42.4% reported having a better understanding of their situation. After providing ePROs to their nurses over time, there was no significant difference in patients’ reports of change (.98 at Time 1, 1.02 at Time 2; t (61) = −.214, p = .83), nor did they perceive difference in their received care (.73 at Time 1, .92 at Time 2; t (62) = −1.23, p = .22). Specifically related to the item pertaining to satisfaction with care received, there was no significant difference on level of satisfaction over time (3.92 at Time 1, 3.88 at Time 2) following the use of ePROs (t (64) = 0.83, p = .41).
In summary, the five themes that emerged from the interviews with the nurses include: enhancing focus of the nurses, directing interdisciplinary follow-up, offering support to patients through the process, interpreting results from the visual display, and integrating into workflow. Scores on the Client Questionnaire suggested that patients believed that they received excellent care (97%), and that the nurses perfectly understood their needs (90.9%). Although there was no statistically significant change in patient satisfaction scores over time, 38.5% of the patients reported moderate or a great deal of change as a result of the use of ePROs.
Unfortunately, we did not explore the nature of those changes. There are many contextual, demographic, and clinical factors that may have influenced satisfaction. Other researchers have pointed out that satisfaction scores usually tend to be positively skewed because respondents fear that the results will reflect badly on their care providers . Given the longstanding relationships between nurses and patients in these outpatient settings, patients may indeed have felt protective of their nurses. Effective interprofessional care may have also influenced these perceptions [40, 41].
Despite the assumption that provision of PROs in routine clinical practice may lead to improved patient satisfaction , there is little evidence that satisfaction increases when PROs are fed back to healthcare providers [16, 43, 44]. This study suggests that use of ePROse may have an impact for some patients. This should be explored further in future studies.
It has previously been found that people with kidney disease find it difficult to discuss some topics [45, 46], and some of the items in the ePROs, such as feelings of well-being, health, depression, being a burden to one’s family, dependence on healthcare practitioners, and stress caused by kidney disease may be difficult to discuss [45, 46]. Wolpert et al.  noted that patients were unwilling to reveal their ratings of some areas via PROs. In this study, nurse participants believed that providing a numerical score on an ePRO was perceived as safer by patients and facilitated deeper discussion. It would be important to explore this perception with patients to see if they also perceive that the ePROs enhance discussion.
Wolpert et al.  also reported that patients thought that their PRO instruments (PedsQL and a symptom checklist) did not fully capture patient concerns, but perhaps their measures were less comprehensive than those used in our study. While patients in this project were not asked about willingness to reveal their ratings, most (90.9%) reported that nurses perfectly understood the kind of help that they wanted. Other researchers [4, 5] support the reports of nurses in this study who believed that ePROs increased patients’ willingness to report sensitive information.
Our findings suggest that ePROs can be valuable tools to facilitate engagement of patients in discussions about their health and well-being. Further, the process of completing the questionnaires facilitated dialogue that was perceived by nurses to provide support to patient participants. It may be that patients’ experiences of satisfaction were influenced by these experienced nurses who appropriately focused on the patients rather than the technology. The nurses’ prioritization of patients’ needs and use of the reports in the interdisciplinary team follow-up may have also influenced patient’s reports of health, and fostered better understanding and awareness of the patients’ needs. Previous research has identified that renal patients desire engagement in their own care through a person-centered approach [23, 25].
Future researchers and practitioners planning to provide ePROs in real-time may want to consider the visual display of data. Nurses in this study emphasized that the visual (versus numerical) display enhanced efficiency and use of their time. While other researchers have identified that integration of ePROs requires upfront planning and training [3, 8, 46], use of ePROs has been found to be more economical in terms of time and resources [5, 6, 8, 46,47,48,50]. Integration of ePROs into the clinicians’ workflow is essential, not only for buy-in and support by practitioners, but also to avoid duplication of data entry. Further education on use of ePROs in practice, perhaps drawing on clinical guidelines where they are available, may be useful to obtain more positive outcomes. Designing longitudinal displays of ePRO data may also facilitate assessment of change over time.
This study has a number of limitations. We had a relatively small sample size and did not collect baseline patient satisfaction data. The descriptive design does not enable one to draw causal inferences. Patient participants were also not interviewed about the benefits of use of ePROs. Such interviews could yield rich data about the way that ePROs could change patient care. The patient responses were collected in an open waiting room, so the presence or comments of others (family members or other patients) may have influenced the responses.
Different questions were asked of patients and nurses regarding perceived usefulness of ePROs, so that may explain why there were different perceptions regarding utility of the measures. The nurses also had limited time (meeting with a patient once every three months) for nursing interventions to impact patient care. Limited training was provided to nurse participants on use of ePROs in clinical practice and it may be that they, or members of the interprofessional team, did not fully understand the ePRO results. Further training of the entire team on how to respond to scores may result in different outcomes.
Nurses reported that sharing ePRO data in real-time informed their practice. Although patients did not report any change in satisfaction with care over time, some reported that use of ePROs changed their care. As use of PROs is growing, further work is needed to provide guidance about how and why use of this type of data can enhance person-centered care. Replication of the study in other sites would help build our knowledge base about use of ePROs in clinical practice. Further study of the process of integration of ePROs into nurses’ or other professionals’ practice over time would help guide the planning of similar processes. A randomized controlled trial allocating patients into different groups may be helpful in assessing whether use of ePROs provides documented benefits over other traditional means of patient assessment in offering quality care and improving patient outcomes.
Chronic kidney disease
Electronic patient reported outcome
Revised Edmonton Symptom Assessment System for renal patients
Health-related quality of life
Kidney Disease Quality of Life-36
Patient reported outcome
Quality of life
International Society for Quality of Life Research. User’s guide to implementing patient-reported outcomes assessment in clinical practice. International Society for Quality of Life Research. 2011. http://www.isoqol.org/UserFiles/file/UsersGuide.pdf . Accessed March 13, 2017.
Canadian Institute for Health Information (CIHI). CIHI PROMs forum: PROMs background document. Ottawa: ON: CIHI; 2015.
Hjollund NHI, Larsen LP, Biering K, Johnsen SP, Riiskjaer E, Schougaard LM. Use of patient-reported outcome (PRO) measures at group and patient levels: experience from the generic integrated PRO system. WestChronic Interact J Med Res. 2014;3(1):e5. doi:10.2196/ijmr.2885.
Basch E, Goldfarb S. Electronic patient-reported outcomes for collecting sensitive information from patients. J Support Oncol. 2009;7:98–9.
Dupont A, Wheeler J, Herndon JE II, Coan A, Zafar SY, Hood L, et al. Use of tablet personal computers for sensitive patient-reported information. J Support Oncol. 2009;7:91–7.
Smith SK, Rowe K, Abernethy AP. Use of an electronic patient-reported outcome measurement system to improve distress management in oncology. Palliat Support Care. 2014;12:69–73.
Zbrozek A, Hebert J, Gogates G, Thorell R, Dell C, Molsen E, et al. Validation of electronic systems to collect patient-reported outcome (PRO) data – recommendations for clinical trial teams: report of the ISPOR ePRO systems validation good research practices task force. Value Health. 2013;16:48–489.
Schick-Makaroff K, Molzahn A. Strategies to use tablet computers for collection of electronic patient-reported outcomes. Health Qual Life Out. 2015;13(1, 2) doi:10.1186/s12955-014-0205-1.
Bennett AV, Jensen RE, Basch E. Electronic patient-reported outcome systems in oncology clinical practice. CA-Cancer J Clin. 2012;62:337–47.
Hilarius DL, Kloeg PH, Gundy CM, Aaronson NK. Use of health-related quality-of-life assessments in daily clinical oncology nursing practice: a community hospital-based intervention study. Cancer. 2008;113:628–37.
Velikova G, Booth L, Smith AB, Brown PM, Lynch P, Brown JM, et al. Measuring quality of life in routine oncology practice improves communication and patient well-being: a randomized control trial. J Clin Oncol. 2004;22:714–24.
Dy SM, Roy J, Ott GE, McHale M, Kennedy C, Kutner JS, et al. Tell us™: a web-based tool for improving communication among patients, families, and providers in hospice and palliative care through systematic data specification, collection and use. J Pain Symptom Manag. 2011;42:526–34.
Wolpert M, Curtis-Tyler K, Edbrooke-Childs J. A qualitative exploration of patient and clinician views on patient reported outcome measures in child mental health and diabetes services. Adm Policy Ment Health. 2014; doi: 10.1007/s10488-014-0586-9
Boyce MB, Browne JP. Does providing feedback on patient-reported outcomes to healthcare professionals result in better outcomes for patients? A systematic review. Qual Life Res. 2013;22(9):2265–78. doi:10.1007/s11136-013-0390-0.
Greenhalgh J. The applications of PROs in clinical practice: what are they, do they work, and why? Qual Life Res. 2009;18:115–23.
Greenhalgh J, Abhyankar P, McCluskey S, Takeuchi E, Velikova G. How do doctors refer to patient-reported outcome measures (PROMS) in oncology consultations? Qual Life Res. 2013;22:939–50.
Edbrooke-Childs J, Wolpert M, Deighton J. Using patient reported outcome measures to improve service effectiveness (UPROMISE): Training clinicians to use outcome measures in child mental health. Adm Policy Ment Health. 2014; doi: 10.1007/s10488-014-0600-2.
Takeuchi EE, Keding A, Awad N, Hofmann U, Campbell LJ, Selby PJ, et al. Impact of patient-reported outcomes in oncology: a longitudinal analysis of patient-physician communication. J Clin Oncol. 2011;29:2910–7.
Kumnig M, Rumpold G, Höfer S, König P, Holzner B, Giesinger J, et al. Patient-reported outcome reference values for patients after kidney transplantation. Wien Klin Wochenschr. 2014;126:15–22.
Purnell TS, Auguste P, Crews DC, Lamprea-Montealegre J, Olufade T, Greer R, et al. Comparison of life participation activities among adults treated by hemodialysis, peritoneal dialysis, and kidney transplantation: a systematic review. Am J Kidney Dis. 2013;62:953–573.
Snyder CF, Aaronson NK, Choucair AK, Elliott TE, Greenhalgh J, Halyard MY, et al. Implementing patient-reported outcomes assessment in clinical practice: a review of the options and considerations. Qual Life Res. 2012;21:1305–14.
Finkelstein FO, Wuerth D, Finkelstein SH. Health related quality of life and the CKD patient: challenges for the nephrology community. Kidney Int. 2009;76:946–52.
Bear RA, Stockie S. Patient engagement and patient-centered care in the management of advanced chronic kidney disease and chronic kidney failure. Can J Kidney Health Dis. 2014;1(1):24. doi:10.1186/s40697-014-0024-7.
Cavanaugh KL. Prioritizing patient-centered care implementation and research for patients with kidney disease. Sem Dialysis. 2014; doi: 10.1111/sdi.12326.
Manns B, Hemmelgarn B, Lillie E, Crowe S, Cyr A, Gladish M, et al. Setting research priorities for patients on or nearing dialysis. Clin J Am Soc Nephrol. 2014;9:1813–21.
Perrone RD, Coons SJ, Cavanaugh K, Finkelstein F, Meyer KB. Patient-reported outcomes in clinical trials of CKD-related therapies: report of a symposium sponsored by the National Kidney Foundation and the US Food and Drug Administration. Am J Kidney Dis. 2013;62:1046–57.
Krishnan M, Brunelli SM, Maddux FW, Parker TF, Johnson D, Nissenson AR, et al. Guiding principles and checklist for population-based quality metrics. Clin J Am Soc Nephrol. 2014;9:1124–31.
Klassen AC, Cresswell J, Plano Clark VL, Smith KC, Meissner HI. Best practices in mixed methods for quality of life research. Qual Life Res. 2012;21:377–80.
Creswell JW. Mapping the developing landscape of mixed methods research. In: Tashakkori A, Teddlie C, editors. The sage handbook of mixed methods in social and behavioral research. 2nd ed. Thousand Oaks: Sage; 2010. p. 1–44.
Schick-Makaroff K, Molzahn A. Brief communication: patient satisfaction with the use of tablet computers: a pilot study in two outpatient home dialysis clinics. Can J Kidney Health Dis. 2014;1(1):22. doi:10.1186/s40697-014-0022-9.
Eduardo B, Kuehn N, Miller MJ, Selmser P, Macmillan K. The Edmonton symptom assessment system (ESAS): a simple method for the assessment of palliative care patients. J Palliat Care. 1991;7:6–9.
Davison SN, Jhandri GS, Johnson JA. Cross-sectional validity of a modified Edmonton symptom assessment system in dialysis patients: a simple assessment of symptom burden. Kidney Int. 2006;69:1621–5.
Davison SN, Jhandri GS, Johnson JA. Longitudinal validation of a modified Edmonton symptom assessment system (ESAS) in haemodialysis patients. Nephrol Dial Transpl. 2006;21:3189–95.
Hays RD, Kallich JD, Mapes DL, Coons SJ, Carter WB. Development of the kidney disease quality of life (KDQOL) instrument. Qual Life Res. 1994;3:329–38.
Hilton BA, Budgen C, Mozahn AE, Attridge CB. Developing and testing instruments to measure client outcomes at the Comox Valley nursing center. Public Health Nurs. 2001;18:327–39.
Kidney Disease Quality of Life (KDQOL) Complete. http://www.kdqol-complete.org. Accessed March 13, 2017.
Coons SJ, Gwaltney CJ, Hays RD, Lundy JJ, Sloan JA, Revicki DA, et al. Recommendations on evidence needed to support measurement equivalence between electronic and paper-based patient-reported outcome (PRO) measures: ISPOR ePRO good research practices task force report. Value Health. 2009;12:419–29.
Thorne S. Interpretive description. Walnut Creek: Left Coast Press; 2007.
Sandelowski M, Barroso J. Classifying the findings in qualitative studies. Qual Health Res. 2003;13:905–23.
Noble H, Kelly D, Rawlings-Anderson K, Meyer J. A concept analysis of renal supportive care: the changing world of nephrology. J Adv Nurs. 2007;59:644–53.
Ivanitskaya L, Clark D, Montgomery G, Primeau R. Interdisciplinary learning: process and outcomes. Innov Higher Educ. 2002;27:95–111.
Greenhalgh J, Long AF, Flynn R. The use of patient reported outcome measures in routing clinical practice: lack of impact or lack of theory? Soc Sci Med. 2005;62:833–43.
Marshall S, Haywood KL, Fitzpatrick R. Impact of patient-reported outcome measures on routine practice: a structured review. J Eval Clin Pract. 2006;12:559–68.
Taenzer P, Bultz BD, Carlson LE, Speca M, DeGagne T, Olson K, et al. Impact of computerized quality of life screening on physician behaviour and patient satisfaction in lung cancer outpatients. Psychooncology. 2000;9:203–13.
Schick-Makaroff K, Sheilds L, Molzahn A. Stories of chronic kidney disease: listening for the unsayable. J Adv Nurs. 2013;69:2644–53.
Schick-Makaroff K, Sheilds L, Molzahn A. Symbolic representations of living with chronic kidney disease. Nephrol Nurs J. 2013;40:517–26.
Holzner B, Giesinger JM, Pinggera J, Zugal S, Schöpf F, Oberguggenberger AS, et al. The computer-based health evaluation software (CHES): a software for electronic patient-reported outcome monitoring. BMC Med Inform Decis Mak. 2012;9:126–37.
Chang Y-J, Chang C-H, Peng C-L, Wu H-C, Lin H-C, Wang J-Y, et al. Measurement equivalence and feasibility of the EORTC QLQ-PR25: paper-and-pencil versus touch-screen administration. Health Qual Life Out. 2014;12(1):23. doi:10.1186/1477-7525-12-23.
Coons SJ. ePRO systems validation: clearly defining the roles of clinical trial teams and ePRO system providers. Value Health. 2013;16:457–8.
Ashley L, Jones H, Forman D, Newsham A, Brown J, Downing A, et al. Feasibility test of a UK-scalable electronic system for regular collection of patient-reported outcome measures and linkage with clinical cancer registry data: the electronic patient-reported outcomes from cancer survivors (ePOCS) system. BMC Med Inform Decis Mak. 2011;11(1):66. doi:10.1186/1472-6947-11-66.
We would also like to acknowledge our project coordinator, Marcy Antonio, and research assistants, Olga Petrovskaya, Judy Bandsmer, Diane Butcher, and Kristina Maguire for their immense help with the data collection.
This study was funded by a KRESCENT (Kidney Research Scientist Core Education and National Training Program) postdoctoral fellowship and the Faculty of Nursing, University of Alberta in Canada.
Availability of data and materials
Ethics Committee approvals were not obtained for sharing of datasets outside of the research team.
KSM co-designed the study, wrote ethics applications, collaborated with practice partners, analyzed data, and drafted/revised the manuscript. AM substantially contributed to the design of the study and helped write and revise the manuscript for intellectual content. Both authors read and approved the final manuscript.
KSM is an Assistant Professor in the Faculty of Nursing at the University of Alberta, Canada. She is currently a Can-SOLVE CKD - KRESCENT New Investigator (Canadians Seeking Solutions & Innovations to Overcome Chronic Kidney Disease; Kidney Research Scientist Core Education & National Training Program).
AM is Dean and Professor in the Faculty of Nursing at the University of Alberta, Canada.
The authors declare that they have no competing interests.
Consent for publication
Consent for publication of individual data was obtained from all participants.
Ethics approval and consent to participate
Ethical approval was obtained from the University of Alberta Health Research Ethics Board (Approval # Pro 00040538) and the Vancouver Island Health Authority (Approval # H2013–065) for all components of the study. All participants provided a written signed informed consent to participate, verbally confirmed at the start of the study and prior to each interview or survey.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Schick-Makaroff, K., Molzahn, A.E. Evaluation of real-time use of electronic patient-reported outcome data by nurses with patients in home dialysis clinics. BMC Health Serv Res 17, 439 (2017). https://doi.org/10.1186/s12913-017-2377-y
- Patient reported outcomes (PRO)
- Chronic kidney disease
- Home hemodialysis
- Peritoneal dialysis
- Electronic patient reported outcomes (ePRO)
- Patient satisfaction
- Quality of life
- Symptom assessment