Skip to main content

A framework to assess patient-reported adverse outcomes arising during hospitalization

  • The Erratum to this article has been published in BMC Health Services Research 2017 17:563



The assessment of adverse events from a patient-centered view includes patient-reported adverse outcomes. An adverse outcome refers to any suboptimal outcome experienced by the patient; when adverse outcomes are identified through a patient interview these are called patient-reported adverse outcomes. An adverse event is an adverse outcome that is more likely due to the processes of medical care rather than to the mere progression of disease. In the context of a large-scale study assessing post-hospitalization adverse events, we developed a conceptual framework to assess patient-reported adverse outcomes (PRAOs). This methodological manuscript describes this conceptual framework.


The PRAO framework builds on a validated adverse event ascertainment method including three phases: Phase 1 involves an inquiry to ascertain the occurrence of any patient-reported adverse outcome. It is completed by a structured telephone interview to obtain details – from a patient perspective – on symptoms that developed and/or worsened after hospitalization. Phase 2 involves the classification of PRAOs by physicians not involved in the patient care. Physician-reviewers then rate the PRAOs using well-adopted scales to determine whether the occurrence was the natural progression of the underlying illness or due to medical care. When the PRAO is rated as “due to medical care”, it is then classified as an “adverse event”. Phase 3 involves the classification of adverse events as preventable or ameliorable.


Out of the 1347 patients contacted at 1-month post-discharge, 469 reported AOs and after reviewing 369 cases, 29 were classified as AEs. Observed agreement levels between raters were 87.3, 85.5, and 85.2 % respectively displaying a good agreement (k > 0.60).


The framework incorporates PRAOs as a way to identify cases that need to be evaluated for adverse events. Further validation of this framework is warrant with the final aim of implementation at larger scale. The implementation of this framework will enable clinicians, researchers and healthcare institutions to compare outcome rates across providers and over time.

Peer Review reports


Since the late 1990s when the Institute of Medicine released its influential report “To err is human”, there have been increased efforts to enhance patient safety and to quantify the rate of adverse events (AEs) [1]. The reported rate of AEs for in-hospital patients ranges from 2.5 to 7.5 % [2, 3], although recent literature suggests that these rates may be a gross underestimation, with the actual rate ten times greater than previously reported [4]. Two recent North American studies found an incidence of post-hospital discharge AEs ranging from 19 to 23 % [57]. The discrepancies in these numbers highlight the lack of a gold standard technique for determining and assessing AEs.

Indeed, various methods exist within the literature for the collection and ascertainment of AEs, including self [810] and incidence reporting, conduction of full retrospective patient record reviews, malpractice claims analyses and mortality and morbidity reviews [11, 12]. The problem with these methods is that they are time intensive, resource heavy, often identify different types of AEs and miss injuries occurring in the post discharge period.

There has been a movement within the literature to turn towards patients to help identify adverse events [1318]. Zhu et al., demonstrated that patients recognize and accurately identify in-hospital adverse events [13]. In addition, Weingart et al. showed that many patient identified events were not captured by traditional incident reporting systems [14]. Furthermore, King et al. [18] discussed how the integration of patients’ perspectives facilitates the understanding of adverse events and suggest the inclusion of patients’ perspective as a complementary measuring tool.

Forster et al. [7] identified that one in five patients experienced an adverse event in the transition from hospital to home, otherwise known as the post discharge period, a period that is missed by the traditional methods of adverse event ascertainment. These studies show the necessity of involving patients to improve patient safety.

Forster et al. [7] developed a method where the patient identified potential adverse events. Patients were contacted within 2–5 weeks after discharged and were asked to identify any new or worsening conditions, or otherwise known as a patient reported adverse outcomes (PRAO), as well as health services utilized. Patients who identified a PRAO had a case summary prepared that was reviewed by two board certified internists, to determine if an adverse event occurred [7]. In a follow up study, the authors assessed the reliability of the peer review process for the ascertainment of adverse events and concluded that for the best agreement, at least three similarly-trained reviewers are required [16].

This methodological paper describes the framework developed for the assessment of patient-reported adverse outcomes (PRAOs) occurring at the transition between the hospital and the community. The framework is an extension of the method developed by Forster et al. This modified method substitutes the medical chart review, for the review of patient cases that include clinical information from discharge summaries and the patient reports [6, 7, 16, 1921].


The present study was nested in a large-scale randomized clinical trial (RCT). This RCT assessed the efficacy of a structured web-accessible electronic discharge summary being compared to usual care discharge summary dictations [22, 23]. The full details of this RCT has been described elsewhere [22]. This study uses data collected during the RCT.

In this section we present the multi-phased method included in the framework overview.

Framework overview

The conceptual framework for assessing PRAOs was developed to aid in the understanding of concepts and to act as a guide for future researchers conducting similar research [24]. Using an iterative process, we made modifications to the approach designed by Forster et al. [6, 7, 16, 20, 21] to enable the assessment of PRAO that could be performed using hospital discharge summaries. Figure 2 displays the framework, and its three phases of information including, Phase 1 describes the identification of PRAOs; Phase 2 includes two steps: 1) evaluating causation and severity of the PRAO. When the PRAO is likely to be related to medical care (omission or commission) then the PRAO is considered an adverse event; 2) using a modified 6-point scale to assess whether the AE was due to treatment/suboptimal medical care. [16]; Phase 3 classifies the identified AE as ameliorable, preventable or neither.

We have clarified the terminology used to describe the conceptual framework (Table 1).

Table 1 Definition of terms

The most important terms underlying the framework are: adverse outcomes, patient-reported adverse outcomes (PRAO), adverse events (AE), ameliorability, and preventability. Figure 1 is a schematic representation of how the concepts relate to one another. An adverse outcome is defined as any suboptimal outcome experienced by the patient, including a new or worsening symptom, an unanticipated visit to a health facility, or death [6, 7]. Adverse outcomes can be identified by medical record review or through a patient interview. If captured only through the latter, we define the outcomes as “patient reported adverse outcomes” or PRAOs. An adverse event is an adverse outcome that is more likely due to the processes of medical care rather than by the mere progression of disease, where medical care refers to all aspects of care including the commission or omission of required acts [6, 7]. An ameliorable adverse event is an outcome whose severity could have been substantially reduced if different actions or procedures had been performed or followed [6, 7]. A preventable adverse event, meanwhile, is an outcome that could have been potentially avoided.

Fig. 1

Schematic representation of how the concepts described and assessed in the paper relate to one another

Phase 1: PRAO identification

In our study [22, 23], PRAO identification occurs at one month post-hospitalization [24]. A trained interviewer administers a brief structured interview over the telephone that takes approximately ten minutes to complete. The questionnaires (see Additional files 1, 2 and 3) gives the patient an opportunity to describe new or worsening health conditions as symptoms. We ask patients to describe: 1) timing of the onset of symptoms; 2) duration of symptoms; 3) severity of the symptoms (on a 10-point scale; 1 = not bothersome; 10 = bother a great deal). Patients also explain the actions taken to alleviate the symptoms, including their use of health services and report if and how the symptom limited their usual physical activities. The PRAO data generates a narrative interview summary. This summary is used in subsequent phases.

Phase 2: adverse event determination

The narrative interview summary is combined with information obtained from the electronic medical record (specifically, the transfer-of-care summaries, otherwise known as discharge summaries, related to the index hospital encounter and subsequent visits) [6, 7, 20] generating a case report. This case report is used by the physician raters to determine the cause of the adverse event. Two physicians review the report independently to determine whether the cause of the outcome was due to the natural progression of the patient’s health condition or to the medical care received. Our team of reviewers consists of three academic physicians; two Royal College certified internists and one College-certified family physician.

During this phase, the physicians perform their classification of causation based on a simple dichotomous choice (i.e. the outcome was likely caused by medical care or not). If the PRAO is rated by the reviewing physician as being likely to have been caused by medical care, it is then classified as an adverse event. Once a PRAO is classified as an AE, the physician reviewers rate their type. Type includes the following groups: adverse drug events, procedure-related injuries, nosocomial infections, care related falls, therapeutic errors, diagnostic errors, and other. Severity is measured in terms of the impact on the patient (how it affected the patient and how the patients felt and functioned) and in terms of the impact on health services used (e.g. visit to the emergency department).

Recognizing that a dichotomous assessment of relation of a PRAO to medical care may be difficult, we also presented reviewers with a more nuanced 6-point scale for judging the link of a PRAO to quality of medical care. The 6-point scale’s response options are:

  1. 1 =

    No evidence that outcome was due to treatment and/or suboptimal medical care;

  2. 2 =

    Little evidence that outcome was due to treatment and/or suboptimal medical care;

  3. 3 =

    Outcome was possibly due to treatment and/or suboptimal medical care (50/50 chance) but was more likely due to disease;

  4. 4 =

    Outcome was possibly due to treatment and/or suboptimal medical care (50/50 chance) and was more likely due to treatment and/or medical care than disease;

  5. 5 =

    Outcome was probably due to treatment and/or suboptimal medical care;

  6. 6 =

    Outcome was definitely due to treatment and/or suboptimal medical care.

Phase 3: preventability and ameliorability determinations

We then brought forward any cases with a rating of ≥5 on the above-mentioned 6-point scale to phase 3 for a final evaluation step using Additional file 4. In this final step, the physician reviewers were asked to classify whether the adverse event is preventable, ameliorable or neither, with definitions of these constructs as outlined earlier.

Ascertainment of agreement

Disagreements among reviewers can occur during phases 2 and 3 of this multi-phased process. In phase 2, when a disagreement between raters occurs, the raters can meet face-to-face to come to a consensus. If a disagreement persists, a third rater can act as a tiebreaker.

When using the 6-point scale as the primary tool for determining link of a PRAO to medical care, reviewers similarly need to review and discuss cases when their ratings are disparate. This is needed when one physician rates ≥5, while the other physician rates ≤3. If one physician rates ≥5 and the other rates 4, the higher rating is endorsed and this is considered an AE. Similarly, when one physician rates ≤3 while the other rates 4, the lower rating is endorsed and this is not considered an AE.


All patients admitted to a medical teaching unit in a Canadian tertiary hospital were invited to participate in the study [22]. During a period of 18 months, 1347 patients were interviewed to report any adverse outcome that rose at one month after discharge from hospital. Of these patients 469 reported adverse outcomes, 369 cases were reviewed of which 29 were rated as adverse events by the raters with an overall adverse event rate of 7.9 %. Out of these, 11.5 % were preventable, 69.3 % were ameliorable and 19.2 % were classified as neither preventable nor ameliorable.

All 369 cases were reviewed by at least 2 reviewers. Reviewers 1, 2, and 3 each reviewed 266, 199, and 273 cases, respectively. Reviewers 1 and 2 rated a total of 96 cases between themselves; reviewers 2 and 3 rated 103 cases between themselves; and reviewers 1 and 3 rated a total of 170 cases between themselves. Table 2 shows the phase 2 agreements between raters prior to any re-rating efforts, with observed agreement levels of 65.3, 75.2, and 52.4 % respectively, and corresponding kappa (k) values of 0.30, 0.33 and 0.14, respectively. These kappa levels indicate only slight to fair agreement [25]. After re-rating, the observed agreement levels were 87.3, 85.5, and 85.2 % respectively, and corresponding kappa (k) values of 0.62, 0.76 and 0.67 (p < 0,001) [25].

Table 2 Phase 2 agreements between reviewers prior to re-rating on the dichotomous choice of whether the outcome was due to medical management or not. (N = 369)

Table 3 shows the agreement between reviewers after re-rating occurred using the 6-point scale of the PRAO framework (Fig. 2). Reviewers 1 and 2 had a 94.8 % agreement with k = 0.82 (p = 0.001). Reviewers 2 and 3 had a 91.3 % agreement, k = 0.81 (p = 0.001) and reviewers 1 and 3 had a 91.8 % agreement, k = 0.74 (p = 0.001). There were 21 cases that reviewers had full agreement on after re-rating as being classified an AE due to medical care (i.e. both reviewers rated the PRAO between a 5 and 6), which indicates an initial adverse event rate of 5.7 %.

Table 3 Phase 3 ratings from reviewers using the 6-point scale of the PRAO framework after re-rating
Fig. 2

Flow chart describing the evaluation process of a patient-reported adverse outcome (PRAO)

There were 30 instances in which reviewers still disagreed after meeting and reassessing cases (i.e. one reviewer rated that PRAO between 1 and 3, while the other rated it between 5 and 6). In such instances, a third reviewer assessed the case and this rating was taken as final (Table 4).

Table 4 Number of PRAOs that still resulted in disagreement between two reviewers, which were rated by a third reviewer

To further illustrate the PRAO rating process, we have provided real examples that can be found in Additional file 5 (a, b, c). These examples include full agreement on AE, agreement on not an AE, and an example of disagreement.


In this paper, we have refined and presented a conceptual and methodological framework for the assessment of PRAOs. The framework builds on a validated adverse event ascertainment method, with modifications to streamline the process by including only the discharge summary as well as the patient report [6, 7, 19, 20]. This method identifies PRAOs using a structured interview via the telephone, classifies the cause of the PRAOs and determines if it was due to medical care and whether the PRAO was ameliorable or preventable. The framework presents a potential tool to assess patient safety that can be used by other groups such as healthcare quality improvement management, and with wide spread adoption, it can help standardize the ascertainment of AEs.

The importance of using PRAO in the ascertainment of adverse events is highlighted in several studies [1318, 2628]. Traditionally it has been argued that patients lack the expertise to evaluate care received [28]. However, patients’ experiences and perceptions are related to indicators such as access to medical care, patient-provider communication, and transitions of care [9, 27]. Positive patient experiences are linked to patient health status [27], while negative patient experiences have been linked to adverse health outcomes [5]. In addition, patients are aware of adverse events and are often accurate in reporting when adverse events have occurred. It has also been found that patients report events that would have been missed using standard chart review techniques [14, 26, 27, 29, 30]. Utilizing PRAO can be a less costly method for the identification of adverse events as compared to the traditional methods. The novelty of this framework is the incorporation of PRAOs as a way to identify cases that needed to be evaluated for adverse events [7]. To our knowledge, no framework has been developed describing the assessment of adverse events utilizing electronic transfer-of-care discharge summaries and PRAO.

Many different methods exist within the literature for the collection and ascertainment of AEs, but each method is faced with limitations [3, 10, 3133]. Self-reporting of adverse events has been one common method utilized in institutions, although voluntary reporting often underreports the rate of AEs [1012, 34]. Conducting full retrospective patient record reviews has been regarded by some groups as the gold standard for the assessment of AEs [33, 35]. This method typically consists of a two-phase chart review process, involving a research nurse and a physician rating adverse events [3, 10, 3133]. This is an onerous process that requires a significant amount of time and resources, and relies on events having already occurred or a healthcare professional identifying an event [10]. Furthermore, this method does not capture post-hospital AEs and excludes patients’ reports, which we know are invaluable and capture missed events [13, 14].

Adverse event ascertainment is often time and resource intensive, due to amount of documentation required from the chart. In order to streamline the adverse event ascertainment process, our framework proposes the use of hospital discharge summaries as a key source of information for PRAO assessments, supplemented by direct interviews with patients. Previous studies have shown that discharge summaries contain the necessary information required to assess adverse events [36]. When Melton and Hripcsak [37] compared assessments of adverse events using full chart reviews versus discharge summaries, their results showed that the assessments were similar. Furthermore, some studies have shown the potential for the use of automated detection systems that are able to scan discharge summaries for adverse event triggers [3640]. Electronic transfer-of-care summaries can provide a platform for this technology.

There are four limitations to this method of AE detection. Firstly, patients can report a large variation in the amount of outcomes, not all of which are related to adverse events. Previous studies have shown that approximately 50 % of patients will report an adverse occurrence to an interviewer after being discharged from the hospital, although the actual AE rate was 9 per 100 admissions [14]. This high rate of PRAOs leads to a large volume of cases that need to be evaluated by reviewers, increasing the burden to raters. Faced with this challenge, the use of well-trained research assistants conducting patient follow up calls is a potential solution.

Secondly, as with any retrospective study utilizing an interview technique, there is a potential for recall bias. To minimize recall bias, interviewers provide patients with probes to assist recall along with possible responses that offer patients set time frames around which they can structure their answers [4144]. For example, for symptom duration, patients can report less than one day, less than one week, less than two weeks, two weeks or more, or still occurring. By proposing a standard process for data collection and review, we hope that the effect of this bias will be minimized. If all institutions and researchers perform this work in the same way, then the impact of this recall bias will be consistent.

Novick suggests that the traditional bias against telephone interviews, low quality of data is not supported by robust evidence [45]. This bias had been attributed to the lack of visual cues during telephone interviews that could potentially result in probing and interpretation of responses. However on the other hand, telephone-interviews may allow respondents to feel relaxed and able to disclose sensitive information [4547].

Thirdly, a known limitation within the traditional methodology to ascertain AEs is the inter-rater reliability within the point scale [6, 16, 48]. It has been our observation that different raters have varying exposures to AEs resulting in each individual viewing a case differently. It is for this reason that we have suggested incorporating an “all-or-none” phenomenon, which leaves little room for disagreement. However, if a point scale is used, we suggest that reviewers are carefully chosen to have similar training, exposure to AEs, are from the same institution and that regular and frequent meetings are conducted to ensure all reviewers have a firm understanding of the definitions associated with AEs and what exactly is being rated. Furthermore, we suggest that reviewers undergo a training program similar to that used by Brennan and colleagues to help improve inter-rater reliability [49]. While there is debate within the literature concerning the reliability of using one rater versus two or more, most studies suggest that more than one reviewer is required to agree on a case before the event can be classified as an AE [15, 48]. These are all factors that can be modified as our knowledge of optimal rating procedures improves.

Finally, we found in our study that in some instances reviewers had difficulty deciding between a 3 (the outcome was possibly due to treatment and/or suboptimal medical care (50/50 chance) but was likely due to disease) or a 4 (outcome was possibly due to treatment and/or suboptimal medical care (50/50 chance) and was more likely due to treatment and/or medical management than disease) response options on the 6-point scale. However, raters recognized that the inclusion of both options was important, as in ‘real world’ there are situations in which is difficult to differentiate between disease progression and treatment/medical management and the two options make raters to consider this difference. Perhaps, testing a modify version of this scale, compressing the scale to a 5-point scale (merging options 3 and 4) could be consider. Any such adjustment of the 6-point scale to a 5-point scale would require some retesting of the performance of the revised scale in inter-observer agreement.

Adverse events are an important patient safety issue and have become a focal point for research over the past 30 years. Despite the increased awareness surrounding AEs, recent studies have pointed to the disappointing numbers in AEs trends: rates have not been decreasing significantly [50, 51]. A caveat to these findings, however, is that standardized methodologies for the detection and measurement of AEs need further development and study. Our proposed framework can offer a standardized method that incorporates patients’ views in the assessment of adverse events. Future studies can focus on the validity and reliability of this framework. Widespread adoption of this tool can create a platform for future researchers to compare AE statistic following patient safety interventions.


The framework presented here is an approach to determining AEs that can be used as a template by clinicians, researchers and health care institutions. The data collected using this framework could help health care systems in their efforts to improve patient safety.


AE(s), adverse event(s); AO, adverse outcome; PRAO, patient –reported adverse outcome; RCT, randomized clinical trial.

Change history

  • 16 August 2017

    An erratum to this article has been published.


  1. 1.

    Institute of Medicine. To Err is human: building a safer health system. Washington, DC: The National Academies Press; 1999.

    Google Scholar 

  2. 2.

    Thomas EJ, Studdert DM, Newhouse JP, et al. Costs of medical injuries in Utah and Colorado. Inquiry. 1999;36:255–64.

    CAS  PubMed  Google Scholar 

  3. 3.

    Leape LL, Brennan TA, Laird N, et al. The nature of adverse events in hospitalized patients: results of the harvard medical practice study II. N Engl J Med. 1991;324:377–84.

    CAS  Article  PubMed  Google Scholar 

  4. 4.

    Classen DC, Resar R, Griffin F, et al. ‘Global trigger tool’ shows that adverse events in hospitals may be ten times greater than previously measured. Health Aff (Millwood). 2011;30:581–9.

    Article  Google Scholar 

  5. 5.

    Baker D, Stevens C, Brook R. Patients who leave a public hospital emergency department without being seen by a physician. J Am Med Assoc. 1991;266:1085–90.

    CAS  Article  Google Scholar 

  6. 6.

    Forster AJ, Clark HD, Menard A, et al. Adverse events among medical patients after discharge from hospital. Can Med Assoc J. 2004;170:345–9.

    Google Scholar 

  7. 7.

    Forster AJ, Murff HJ, Peterson JF, et al. The incidence and severity of adverse events affecting patients after discharge from the hospital. Ann Intern Med. 2003;138:161–74.

    Article  PubMed  Google Scholar 

  8. 8.

    Bjertnaes O, Skudal KE, Iversen HH, Lindahl AK. Patient-reported incident in hospital instrument – PRIH. BMJ Qual Saf. 2013;22:743–51.

    Article  PubMed  PubMed Central  Google Scholar 

  9. 9.

    Kemp K, Warren S, Chan N, McCormack B, Santana M, Quan H. Qualitative complaints and their relation to overall hospital rating using an H-CAHPS-derived instrument. BMJ Qual Saf 2015 Oct 28. doi:10.1136/bmjqs-2015-004371.

  10. 10.

    Murff HJ, Patel VL, Hripcsak G, et al. Detecting adverse events for patient safety research: a review of current methodologies. J Biomed Inform. 2003;36:131–43.

    Article  PubMed  Google Scholar 

  11. 11.

    World Health Organization. WHO Draft Guidelines for Adverse Event Reporting and Learning System. WHO Press. 2005. (accessed 27 Sep2014).

  12. 12.

    Olsen S, Neale G, Schwab K, et al. Hospital staff should use more than one method to detect adverse events and potential adverse events: incident reporting, pharmacist surveillance and local real-time record review may all have a place. Qual Saf Health Care. 2007;16:40–4.

    Article  PubMed  PubMed Central  Google Scholar 

  13. 13.

    Zhu J, Stuver SO, Epstein AM, et al. Can we rely on patients’ reports of adverse events? Med Care. 2011;49:948–55.

    Article  PubMed  Google Scholar 

  14. 14.

    Weingart SN, Pagovich O, Sands DZ, et al. What can hospitalized patients tell us about adverse events? Learning from patient-reported incidents. J Gen Intern Med. 2005;20:830–6.

    Article  PubMed  PubMed Central  Google Scholar 

  15. 15.

    Weissman JS, Schneider EC, Weingart SN, et al. Comparing patient-reported hospital adverse events with medical record review: do patients know something that hospitals do not? Ann Intern Med. 2008;149:100–8.

    Article  PubMed  Google Scholar 

  16. 16.

    Ward JK, Armitage G. Can patients report patient safety incidents in a hospital setting? A systematic review. BMJ Qual Saf. 2012;21(10):685–99.

    Article  PubMed  Google Scholar 

  17. 17.

    Massó Guijarro P, Aranaz Andrés JM, Mira JJ, et al. Adverse events in hospitals: the patient’s point of view. Qual Saf Health Care. 2010;19(11):144–7.

    Article  PubMed  Google Scholar 

  18. 18.

    King A, Daniels J, Lim J, et al. Time to listen: a review of methods to solicit patient reports of adverse events. Qual Saf Health Care. 2010;19:148–57.

    CAS  Article  PubMed  Google Scholar 

  19. 19.

    Forster AJ, Taljaard M, Bennett C, et al. Reliability of the peer-review process for adverse event rating. PLoS One. 2012;7:e41239.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  20. 20.

    Forster AJ, Asmis TR, Clark HD, et al. Ottawa hospital patient saftey study: incidence and timing of adverse events in patients admitted to a Canadian teaching hospital. Can Med Assos J. 2004;170:1235–40.

    Article  Google Scholar 

  21. 21.

    Jabareen Y. Building a conceptual framework: philosophy, definitions, and procedure. 2009;49–62.

  22. 22.

    Okoniewska BM, Santana MJ, Holroyd-Leduc J, et al. The Seamless Transfer-of-Care Protocol: a randomized controlled trial assessing the efficacy of an electronic transfer-of-care communication tool. BMC Health Serv Res. 2012;12:414.

    Article  PubMed  PubMed Central  Google Scholar 

  23. 23.

    Santana MJ, Holroyd-Leduc J, Flemons WW, et al. The Seamless Transfer of Care: A Pilot Study Assessing the Usability of an Electronic Transfer of Care Communication Tool. Am J Med Qual Published Online First: 2013.

  24. 24.

    Michel P, Quenon JL, de Sarasqueta AM, et al. Comparison of three methods for estimating rates of adverse events and rates of preventable adverse events in acute care hospitals. BMJ. 2004;328:199.

    Article  PubMed  PubMed Central  Google Scholar 

  25. 25.

    Goldman RL. The reliability of peer assessments of quality of care. J Am Med Assoc. 1992;267:958–60.

    CAS  Article  Google Scholar 

  26. 26.

    Blendon RJ, Desroches CM, Brodie M, et al. Views of practicing physicians and the public on medical records. N Engl J Med. 2002;347:1933–40.

    Article  PubMed  Google Scholar 

  27. 27.

    Davis K, Schoenbaum SC, Collins KS, et al. Room for improvement: patients report on the quality of their health care. 2002.

  28. 28.

    Hofer TP, Asch SM, Hayward RA, Rubenstein LV, Hogan MM, Adams J, Kerr EA. Profiling quality of care: Is there a role for peer review?. BMC Health Serv Res. 2004;4(1):1.

  29. 29.

    Weingart SN, Gandhi TK, Seger AC, et al. Patient-reported medication symptoms in primary care. Arch Intern Med. 2005;165:234–40.

    Article  PubMed  Google Scholar 

  30. 30.

    Tsilimingras D, Bates DW. Addresing postdischarge adverse events: a neglected area. Jt Comm J Qual Patient Saf. 2008;34:85–97.

    Article  PubMed  Google Scholar 

  31. 31.

    Thomas EJ, Studdert DM, Burstin HR, et al. Incidence and types of adverse events and negligent care in Utah and Colorado. Med Care. 2000;38:261–71.

    CAS  Article  PubMed  Google Scholar 

  32. 32.

    Mills DH. Medical Insurance Feasibility Study. West J Med. 1978;128:360–5.

    CAS  PubMed  PubMed Central  Google Scholar 

  33. 33.

    Hiatt H, Barnes B, Brennan T, et al. A study of medical injury and medical malpractice. N Engl J Med. 1989;321:480–4.

    CAS  Article  PubMed  Google Scholar 

  34. 34.

    Cullen DJ, Sweitzer BJ, Bates DW, et al. Preventable adverse drug events in hospitalized patients: a comparative study of intensive care and general care units. Crit Care Med. 1997;25:1289–97.

    CAS  Article  PubMed  Google Scholar 

  35. 35.

    Mendes W, Martins M, Rozenfeld S, et al. The assessment of adverse events in hospitals in Brazil. Int J Qual Heal care. 2009;21(31):279–84.

    Article  Google Scholar 

  36. 36.

    Murff HJ, Forster AJ, Peterson JF, et al. Electronically screening discharge summaries for adverse medical events. J Am Med Informatics Assoc. 2003;10:12–4.

    Article  Google Scholar 

  37. 37.

    Melton GB, Hripcsak G. Automated detection of adverse events using natural language processing of discharge summaries. J Am Med Informatics Assoc. 2005;12(33):448–58.

    Article  Google Scholar 

  38. 38.

    Forster AJ, Andrade J, Van Walraven C. Validation of a discharge summary term search method to detect adverse events. J Am Med Informatics Assoc. 2005;12:200–7.

    Article  Google Scholar 

  39. 39.

    Bates DW, Evans RS, Murff H, et al. Detecing adverse events using information technology. J Am Med Informatics Assoc. 2003;10:115–29.

    Article  Google Scholar 

  40. 40.

    Linder JA, Haas JS, Iyer A, et al. Secondary use of electronic health record data: spontaneous triggered adverse drug event reporting. Pharmacoepidemiol Drug Saf. 2010;19:1211–5.

    Article  PubMed  Google Scholar 

  41. 41.

    Corwin RG, Krober M, Roth HP. Patients’ accuracy in reporting their past medical history, a study of 90 patients with peptic ulcer. J Chronic Dis. 1971;23:875–9.

    CAS  Article  PubMed  Google Scholar 

  42. 42.

    Preston-Martin S, Bernstein L, Maldonado A, et al. A dental x-ray validation study. Am J Epidemiol. 1985;121:430–9.

    CAS  Article  PubMed  Google Scholar 

  43. 43.

    Sudman S, Bradburn N. Response effects in surveys: a review and synthesis. Chicago: Aldine; 1974.

    Google Scholar 

  44. 44.

    Feinstein A. Clinical Epidemiology: the architecture of clinical research. Philadelphia: W.B. Saunders Co; 1985.

    Google Scholar 

  45. 45.

    Novick G. Is there a bias againt telephone interviews in qualitative research? Res Nurs Health. 2008;31(4):391–8.

    Article  PubMed  PubMed Central  Google Scholar 

  46. 46.

    Burke L, Miller M. Phone interviewing as a means of data collection: Lessons learned and practical recommendations [Online journal] Forum: Qualitative Social Research. 2001;2. Retrieved August 28, 2007 from

  47. 47.

    Burnard P. The telephone interview as a data collection method. Nurse Educ Today. 1994;14:67–72.

    CAS  Article  PubMed  Google Scholar 

  48. 48.

    Zegers M, de Bruijne MC, Wagner C, et al. The inter-rater agreement of retrospective assessments of adverse events does not improve with two reviewers per patient record. J Clin Epidemiol. 2010;63:94–102.

    Article  PubMed  Google Scholar 

  49. 49.

    Brennan TA, Localio RJ, Laird NL. Reliability and validity of judgments concerning adverse events suffered by hospitalized patients. Med Care. 1989;27:1148–58.

    CAS  Article  PubMed  Google Scholar 

  50. 50.

    Shojania KG, Thomas EJ. Trends in adverse events over time: why are we not improving? BMJ Qual Saf. 2013;22:273–7.

    Article  PubMed  Google Scholar 

  51. 51.

    Vincent C, Aylin P, Franklin BD, et al. Is health care getting safer? BMJ. 2008;337:a2426.

    Article  PubMed  Google Scholar 

Download references


We would like to acknowledge the contributions of Chloe de Grood for her support with data collection.


Funding was received from the Canadian Institute for Health Research.

Availability of data and materials

The dataset supporting the conclusions of this article is presented in the manuscript.

Authors’ contributions

BO participated in the data collection and analysis. She interpreted the data and drafted the manuscript. MJS participated in the data collection and analysis. She interpreted the data and drafted the manuscript. JHL was one of the three raters. She also contributed to the analysis and interpretation of the results and helped to draft the manuscript. WF was one of the three raters. He contributed to the analysis and interpretation of the results and helped to draft the manuscript. MO was one of the three raters. She also contributed to the analysis and interpretation of the results and helped to draft the manuscript. DW contributed to the study planning and design and helped to draft the manuscript. WO contributed to data collection and interpretation of results. GW contributed to the design of the study and interpretation of results. AJF contributed to the design of the study and interpretation of results, and helped to draft the manuscript. All authors read and approved the final manuscript.

Competing interests

The authors declare that they have no competing interests.

Consent for publication

Not applicable.

Ethics approval and consent to participate

Ethics approval was obtained from the Ethics board at the University of Calgary. Written consent was obtained from all participants.

Author information



Corresponding author

Correspondence to Santana Maria Jose.

Additional information

An erratum to this article is available at

Additional files

Additional file 1:

Patient Reported Adverse Outcome Questionnaire. (DOC 25 kb)

Additional file 2:

Health Services Utilization Questionnaire. (DOCX 56 kb)

Additional file 3

Patient Reported Outcome (PRAO) Assessment. (DOCX 53 kb)

Additional file 4:

Type of Adverse Event. (DOCX 61 kb)

Additional file 5:

Cases reported. (DOCX 989 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Barbara, O., Jose, S.M., Jayna, H. et al. A framework to assess patient-reported adverse outcomes arising during hospitalization. BMC Health Serv Res 16, 357 (2016).

Download citation


  • Patient-reported adverse outcomes
  • Patient safety
  • Medical informatics
  • Transitions of care
  • Electronic health records