Open Access
Open Peer Review

This article has Open Peer Review reports available.

How does Open Peer Review work?

Estimating the incidence of adverse events in Portuguese hospitals: a contribution to improving quality and patient safety

  • Paulo Sousa1Email author,
  • António Sousa Uva1, 2,
  • Florentino Serranheira1, 2,
  • Carla Nunes1 and
  • Ema S Leite1, 2, 3
BMC Health Services Research201414:311

DOI: 10.1186/1472-6963-14-311

Received: 14 September 2013

Accepted: 11 July 2014

Published: 18 July 2014



Several review studies have shown that 3.4% to 16.6% of patients in acute care hospitals experience one or more adverse events. Adverse events (AEs) in hospitals constitute a significant problem with serious consequences and a challenge for public health. The occurrence of AEs in Portuguese hospitals has not yet been systematically studied. The main purpose of this study is to estimate the incidence, impact and preventability of adverse events in Portuguese hospitals. It is also our aim to examine the feasibility of applying to Portuguese acute hospitals the methodology of detecting AEs through record review, previously used in other countries.


This work is based on a retrospective cohort study and was carried out at three acute care hospitals in the Administrative Region of Lisbon. The identification of AEs and their impact was done using a two-stage structured retrospective medical records review based on the use of 18 screening criteria. A random sample of 1,669 medical records (representative of 47,783 hospital admissions) for the year 2009 was analyzed.


The main results found in this study were an incidence rate of 11.1% AEs, of which around 53.2% were considered preventable. The majority of AEs were associated with surgical procedures (27%), drug errors (18.3%) and hospital acquired infections (12.2%). Most AEs (61%) resulted in minimal or no physical impairment or disability, and 10.8% were associated with death. In 58.6% of the AEs’ cases, the length of stay was prolonged on average 10.7 days. Additional direct costs amounted to €470,380.00.


The magnitude of these results was critical, reinforcing the need of more detailed studies in this area. The knowledge of the incidence and nature of AEs that occur in hospitals should be seen as a first step towards the improvement of quality and safety in health care.


Patient safety Medical errors Hospitals Quality of care Adverse events


Adverse events (AEs) occur with alarming frequency in healthcare. These events represent significant losses from a clinical, economic and social perspective [13]. To learn from these events and improve safety, they must be identified, measured and their causes found. Healthcare providers and researchers are searching for an accurate, reliable and low cost method to identify and measure AEs in hospital and other settings. The method developed in the Harvard Medical Practice Study, in the 90s, is the one most often used for national AE studies [4].

Many countries, like Portugal, have taken initiatives over the past decade to address safety problems in health care [3, 5, 7, 8]. The US report “To Err is Human” marked an acceleration in programmes and actions for increased patient safety initiated by health care policy makers, health care professionals and managers [9]. As far as we know there have been no studies to date on the occurrence, nature, preventability and impact of AEs in Portuguese hospitals.

Studies based on the “Harvard method” have been carried out in several countries with different results, finding evidence of 3.4% to 16.6% of patients in acute care hospitals experiencing one or more AEs [1017]. The results of these studies give data on a critical aspect of hospital performance and stimulate patient safety improvement of each country.

The aims of this study were to estimate the incidence, nature, preventability, cost and impact of adverse events in Portuguese hospitals. This was an exploratory study that also intended to examine the feasibility of applying to Portuguese acute hospitals the methodology previously used in other countries.


This work followed a retrospective cohort study design. The methods were based on the protocol used in the Harvard Medical Practice Study [4, 11, 18] with modifications introduced in subsequent studies undertaken in the United Kingdom, New Zealand, Canada, the Netherlands, Sweden, Brazil, and, more recently, in a sample of 26 hospitals from eight developing and transitional countries [1012, 1518].

The study was carried out in three public hospitals of the Lisbon Administrative Region (these hospitals are acute care hospitals; with 785 beds, 450 beds and 220 beds respectively; an emergency department, intensive care units and a high surgery volume). Although the participating hospitals were selected by convenience, they reflect the major characteristics of other public hospitals in Portugal regarding dimension (number of beds), emergency department 24 hours per day, intensive care units, medical and surgical departments and casemix index of patients treated. No specialty hospitals (e.g. Pediatric, Oncology, Obstetric) were included in the study. A global random sample of 1,669 medical records was used, representative of 47,783 (3.5%) hospital admissions, between 01 January 2009 and 31 December 2009, fulfilling the inclusion criteria for this study. At each hospital a simple random sample was selected assuming the number of hospital admissions, an incidence of AEs of 8% (based on The Canadian Adverse Events Study) and a confidence level of 95% [12]. The sampling frame included all admissions of patients over 18 years old who had a minimum stay in hospital of 24 hours. Hospital admissions with a primary diagnosis related to psychiatry were excluded. Oversampling was carried out with the expectation that 10% of medical records would be unusable. A two-stage structured retrospective medical records review was carried out based on the use of 18 screening criteria (Table 1).
Table 1

Screening criteria and number of medical records showing evidence of one category of adverse event




Hospital-incurred patient injury



Unplanned readmission after discharge from index admission (12 months)



Unplanned admission related to previous healthcare management



Hospital-acquired infection or sepsis



Adverse drug reaction



Unplanned return to the operating room



Any other undesirable outcomes not covered in this list of criteria



Unexpected death



Unplanned transfer from general care to intensive care



Other patient complications



Cardiac or respiratory arrest



Unplanned removal, injury or repair of an organ during surgery



Unplanned transfer to another acute care hospital



Development of neurological deficit not present on admission



Inappropriate discharge to home



Dissatisfaction or correspondence indicating litigation



Injury related to abortion or delivery



Dissatisfaction with care documented in the patient’s medical record






The criteria are not mutually exclusive; i.e. one patient can have more than one criterion (in 365 positive medical records there were 451 criteria).

In the first stage, a group of six nurses (two from each hospital, with a minimum of five years experience in clinical audits) assessed the medical records, looking for the presence of, at least, one of the 18 criteria for the presence of a potential adverse event. In stage two, a group of five physicians (one Cardiologist, one Neurologist, two Surgeons, one Internal Medicine, with a minimum of five years experience in clinical codes and in clinical audits) reviewed each positive record in order to confirm the presence of an adverse event, to estimate its impact and determine its preventability, according to the definition established previously. The degree of agreement between the reviewers in each stage was calculated (assessed on a random sample of 10% of medical records) by using kappa coefficient. Neither nurses nor doctors knew the previous classification of their colleagues.

The timing of the AEs in relation to the hospital admission is an important methodological issue. We considered AEs that occurred during the index hospital admission and that were detected during either the index or subsequent hospital admissions over the following 12-month period (in the same hospitals).

Similarly to other studies, physicians estimated, based on evidence in the medical record and their professional judgment, the impact of AEs in two different ways: i) the degree of physical impairment or disability at discharge (minimal, moderate or permanent), or death; ii) the number of additional hospital days directly attributable to AEs.

The costs related to the additional length of stay were estimated based on official accounting data from all the NHS hospitals, providing information on daily costs. This value includes hospital daily costs, namely physicians' and nursing staff, lab tests and exams, medication, housing and overhead [19].

Using professional judgment and based on the information of medical records, the physician reviewers also classified the preventability of each AE using a six-point scale (1- virtually no evidence of preventability; 2- slight to modest evidence of preventability; 3- Preventability not quiet likely less than 50/50, but “close to call”; 4-preventability more than likely 50/50; 5- strong evidence of preventability; and 6 -virtually certain evidence of preventability. The preventability of an adverse event was considered with a score > = 4).

SPSS (version 19) was used for data processing and for statistical analysis. The study was approved by the Ethics Committee of participating hospitals.


One or more of the criteria for an adverse event were identified in 365 out of the 1.669 medical records reviewed (22%). Of these 365 records, those which passed to the second screening, 186 (51%), were confirmed as an AE, an overall incidence rate of 11.1% (186/1669) with a 95% CI (9.6%; 12.6%). The highest proportions of AEs identified were related to surgical procedures (27.0%), drug errors (18.3%) and hospital-acquired infection (12.2%). Most of the AEs (59.2%) occurred in patients aged 65 or over (Figure 1).
Figure 1

Percentage distribution of adverse events by age group.

Most (61%) of the AEs resulted in no physical or minimal impairment or disability, and were satisfactorily resolved during the admission, or within one month from discharge. Nevertheless, the criteria applied estimated that 5.4% of the AEs resulted in permanent disability according to the definition and 10.8% associated with death (Table 2).
Table 2

Degree of physical impairment or disability at discharge




Minimal impairment or disability, recovery within 1 month



Moderate impairment or disability, recovery within 1-12 months



Permanent impairment or disability






Unable to determine



Total (of AEs)



We found that most of the patients, 109/186 (58.6%), who experienced AEs incurred extra bed days in hospital (a total of 1,166 extra days, an average of 10.7 days per patient, ranging from 1-70 days), with additional total costs of €470,380.00 for all three hospitals together.

Concerning preventability, 53.2% were classified as preventable (Table 3).
Table 3

Level of preventability of the AEs





Virtually no evidence of preventability



46.8% not preventable

Slight to modest evidence of preventability



Preventability not quite likely (less than 50/50, but “close call”)



Preventability more than likely (more than 50/50 but “close call”



53.2% Preventable

Strong evidence of preventability



Virtually certain evidence of preventability







The reliability of the assessment of the screening criteria performed by nurses (first screening) was considered good – substantial agreement (k = 0.63; IC 0.43; 0.79 and p < 0.001; 83.5% agreement). Among doctors (second screening), the reliability of determination of AEs was also good (k = 0.78; IC 0.49; 1 and p < 0.001; 86% agreement) and that of preventability was considered fair (k = 0.58; IC 0.23; 0.94 and p = 0.009; 79% agreement) (Table 4).
Table 4

Reliability of assessment between reviewers in the first and the second screening



K statistic

CI 95%


Presence of one or more positive criteria in the medical record analyzed

Screening 1

0.63 (83.5% agreement)

(0.43; 0.79)

< 0.001

AE confirmed by physician

Screening 2

0.78 (89% agreement)

(0.49; 1)

< 0.001

Level of preventability (scale 1–6)

Screening 2

0.58 (79% agreement)

(0.23; 0.94)



We used retrospective medical record review following the HMPS methodology in order to assess the nature, incidence, and clinical and/or economic impact of adverse events and to provide some information on their causes. Studies in many countries have followed the same methodology and have come to broadly similar conclusions [1012, 1417]. Rates of AEs in most recent studies lie between 8% and 12%, a range now accepted as being common in the healthcare systems of developed countries.

In this study, we found an incidence of AEs of 11.1% of which 53.2% were considered as preventable. The assessment of preventability is one challenge for these studies, and indicates the potential gains to be achieved by improvements. For this reason, analysis of each case is needed to decide prevention strategies. On the other hand, the classification of preventable AEs, while using a clear criterion and standard, still involves a subjective element and may vary with the expertise, practical experience of the physician and the way the data is registered in the medical record. Some authors argue that knowing the outcome and its severity may influence the judgment of causation and preventability and that the bias element is likely to be in the direction of overestimation of the rate of “preventable” AEs (hindsight bias) [2022]. Nevertheless, our results are similar to the findings of previous studies, particularly those of the UK (incidence10.8% and 52% preventable), New Zealand (incidence 11.3% and 61.6% preventable) and the Danish study (incidence 9.0% and 40.4% preventable) [10, 11, 23].

One finding was that 59.2% of patients who experienced an AE were 65 years old or older. Relatively little attention has been paid to patient safety in older people although they are particularly vulnerable to healthcare error and harm [24, 25]. Moreover, older people are more likely to suffer from multiple health conditions, receive multiple treatments, and stay longer in hospital. A longer stay increases the risk of all complications of hospitalization. In the last decades more people are living longer and this trend is likely to continue. For all these reasons, it is our opinion that safety for older patients should receive higher priority in research and in safety prevention strategies.

The majority of AEs (61%) were not documented as causing serious consequences for the patient. They did not result in any significant physical impairment or disability, and were resolved during the admission or, within one month from discharge. This is a similar percentage to that reported in other studies [3, 10, 12, 13, 17]. However, a meaningful proportion of patients died (10.8%) or experienced a permanent disability as a result of their AEs (5.4%).

The economical impact of adverse events has been gaining particular attention in the last years [26, 27]. The report of the Institute of Medicine estimates that, in the United States, the total national costs associated with adverse events represent approximately 4% to 6% of national health expenditures [9]. In Britain, the cost of preventable AEs, in lost bed days alone, was estimated in one billion (Pounds) per year [24]. In the Netherlands (the Dutch adverse events study) the authors concluded that 3% of all bed days and 1% of the total health budget could be attributed to preventable AEs (costs of direct hospital care, mainly additional time in hospital) [20]. In our study, most of the patients (58.6%) who experienced AEs prolonged the length of stay in hospital on average 10.7 days, with additional direct costs of €470,380.00. Extrapolating this estimated value to the population of the study (all admission of the three hospitals in 2009) the costs varied between €1,290.310 and €1,691.643. The overall real costs are higher, as these estimates do not include additional treatments in ambulatory care or other hospitalizations (which are related to the same AE), costs to patients (e.g. medication in ambulatory) or any societal costs (e.g. absence from work, premature death). The costs of prevention, however, may be significant, especially if organizations or their health systems do not have effective implementation capacity [27].

In addition to the costs of unnecessary suffering caused to patients and families, the costs and savings of adverse events are a critical issue in Portugal, which will be experiencing a challenging economic situation for at least the next five years [28] . As with some other European countries, the economic rationale is increasing for selecting effective strategies and implementing them in a context which rewards safety improvement [27]. Including an economic dimension to the knowledge gained from research into quality and safety is an important part of the health system reform that is being implemented in Portugal [5, 12, 29, 30]. Moreover, it could be a starting point for specific interventions in the improvement of patient safety and it may help to prioritise research areas for the near future [6, 20, 27, 31].

There are limitations to the study. These include those applying to retrospective studies, such as information bias and hindsight bias [15, 21, 22]. We observed that the quality of patient records could be better, although this did not significantly limit the study. Retrospective patient record studies still represent the “gold-standard” method for assessing incidence and monitoring the frequency of AEs [13, 20, 31, 32] and the use of similar methods to those used elsewhere allows comparisons to be made. However, for a more comprehensive assessment of safety, other methods also need to be used, including reporting and learning systems, root-cause analyses, failure mode effects analyzes and morbidity and mortality reviews [13, 33].

The reliability of the review process is another critical element in these studies [34, 35]. In our study, the level of agreement (Kappa statistic – k value) between nurses was considered good - substantial agreement (k value 0.63) and is related to the identification of the presence of at least one positive criterion in the first screening. This value is in line with those found in other studies [1113, 15]. Among doctors (in the second screening), the reliability of determination of AEs was also considered good - substantial agreement with k value of 0.78 and fair concerning their preventability, with k value of 0.58. For the second screening, the k value is slightly higher than those found in other studies undertaken at national level (e.g. Canada, New Zealand, Sweden and The Netherlands). These results must be carefully interpreted, due to the small number of reviewers, cases and hospitals involved in this study [36, 37].


In Portugal, there is an overall awareness and a growing concern about patient safety issues. This study suggests that AEs in three Portuguese hospitals affect nearly one in ten patients and results in considerable avoidable suffering and economic costs. With local evidence of the size of the problem, staff is more motivated to act, especially if effective interventions to reduce adverse events can be selected and implemented to target those AEs that are prioritized, namely those resulting from surgical procedures, drug errors and health-acquired infections.

If interventions can be implemented and demonstrated to be cost-effective, then more healthcare managers and policy makers may begin to view improvements in safety as an investment rather than an expense. This can also speed changes in incentives so as to reward safety and quality. Local evidence of the size and nature of the problem and its clinical and cost impact is one of the first steps toward this desirable change.



We would like to thank everyone who contributed to the study, namely the staff of the three participating hospitals who facilitated the access to patient records and the nurses and physicians for reviewing the medical records. This work was supported by the Fundação Calouste Gulbenkian.

Authors’ Affiliations

National School of Public Health, Universidade Nova de Lisboa
CMDT – Centro de Investigação em Malária e Doenças Tropicais – Saúde Pública
Centro Hospitalar de Lisboa Central


  1. Thomas E, Petersen L: Measuring errors and adverse events in health care. J Gen Intern Med. 2003, 18: 61-67. 10.1046/j.1525-1497.2003.20147.x.View ArticlePubMedPubMed CentralGoogle Scholar
  2. Landrigan CP, Parry GJ, Bones CB, Hackbarth AD, Goldmann DA, Sharek PJ: Temporal trends in rates of patient harm resulting from medical care. N Engl J Med. 2012, 363 (22): 2124-2134.View ArticleGoogle Scholar
  3. de Vries EN, Ramrattan MA, Smorenburg SM, Gouma DJ, Boermeester MA: The incidence and nature of in-hospital adverse events: a systematic review. Qual Saf Health Care. 2008, 17: 216-223. 10.1136/qshc.2007.023622.View ArticlePubMedPubMed CentralGoogle Scholar
  4. Leape LL, Brennan TA, Laird N, Lawthers AG, Localio AR, Barnes BA, Hebert L, Newhouse JP, Weiler PC, Hiatt H: The nature of adverse events in hospitalized patients: results of the Harvard medical practice study II. N Engl J Med. 1991, 324: 377-384. 10.1056/NEJM199102073240605.View ArticlePubMedGoogle Scholar
  5. Sousa P, Uva A, Serranheira F, Pinto F, Øvretveit J, Klazinga , Sunol R, Terris D: The patient safety journey in Portugal: challenges and opportunities from a public health perspective. Revista Portuguesa de Saúde Pública. 2009, Especial 25 anos: 91-106.Google Scholar
  6. Hauck K, Zhao X, Jackson T: Adverse event rates as measures of hospital performance. Health Policy. 2012, 104: 146-154. 10.1016/j.healthpol.2011.06.010.View ArticlePubMedGoogle Scholar
  7. Sousa P: Patient safety: a necessidade de uma estratégia nacional. Acta Med Port. 2006, 19: 309-318.PubMedGoogle Scholar
  8. Garrouste-Orgeas M, Philippart F, Bruel C, Max A, Lau N, Misset B: Overview of medical errors and adverse events. Annals of Intensive Care. 2012, 2: 2-10.1186/2110-5820-2-2. doi:10.1186/2110-5820-2-2View ArticlePubMedPubMed CentralGoogle Scholar
  9. Institute of Medicine (IOM): To err is human: building a safer health system. 2000, Washington, DC: National Academy PressGoogle Scholar
  10. Vincent C, Neale G, Woloshynowych M: Adverse events in British hospitals: preliminary retrospective record review. BMJ. 2001, 322: 517-519. 10.1136/bmj.322.7285.517.View ArticlePubMedPubMed CentralGoogle Scholar
  11. Davis P, Lay-Yee R, Briant R, Ali W, Scott A, Schug S: Adverse events in New Zealand public hospitals I: occurrence and impact. N Z Med J. 2002, 115: 203-209.Google Scholar
  12. Baker GR, Norton PG, Flintoft V, Blais R, Brown A, Cox J, Etchells E, Ghali WA, Hébert P, Majumdar SR, O'Beirne M, Palacios-Derflingher L, Reid RJ, Sheps S, Tamblyn R: The Canadian adverse events study: the incidence of adverse events among hospital patients in Canada. CMAJ. 2004, 170: 1678-1686. 10.1503/cmaj.1040498.View ArticlePubMedPubMed CentralGoogle Scholar
  13. Michel P, Quenon JL, de Sarasqueta AM, Scemama O: Comparison of three methods for estimating rates of adverse events and rates of preventable adverse events in acute care hospitals. BMJ. 2004, 328: 199-10.1136/bmj.328.7433.199. doi: View ArticlePubMedPubMed CentralGoogle Scholar
  14. Aranaz-Andrés JM, Aibar-Remón C, Vitaller-Murillo J, Ruiz-López P, Limón-Ramírez R, Terol-García E: Incidence of adverse events related to health care in Spain: results of the Spanish National Study of Adverse Events. J Epidemiol Community Health. 2008, 62 (12): 1022-1029. 10.1136/jech.2007.065227.View ArticlePubMedGoogle Scholar
  15. Zegers M, de Bruijne MC, Wagner C, Hoonhout LH, Waaijman R, Smits M, Hout FA, Zwaan L, Christiaans-Dingelhoff I, Timmermans DR, Groenewegen PP, van der Wal G: Adverse events and potentially preventable deaths in Dutch hospitals: results of a retrospective patient record review study. Qual Safe Health Care. 2009, 18: 297-302. 10.1136/qshc.2007.025924.View ArticleGoogle Scholar
  16. Soop M, Fryksmark U, Köster M, Haglund B: The incidence of adverse events in Swedish hospitals: a retrospective medical record review study. Int J Qual Health Care. 2009, 22: 324-333.Google Scholar
  17. Mendes W, Martins M, Rozenfeld S, Travassos C: The assessment of adverse events in hospitals in Brazil. Int J Qual Health Care. 2009, 2: 279-284.View ArticleGoogle Scholar
  18. Wilson RM, Michel P, Olsen S, Gibberd RW, Vincent C, El-Assady R, Rasslan O, Qsous S, Macharia WM, Sahel A, Whittaker S, Abdo-Ali M, Letaief M, Ahmed NA, Abdellatif A, Larizgoitia I: Patient safety in developing countries: retrospective estimation of scale and nature of harm to patients in hospitals. BMJ. 2012, 324: e832-doi:10.1136/bmj.e832View ArticleGoogle Scholar
  19. da Saúde M: ACSS. Contabilidade analítica dos hospitais do SNS. 2011, Administração Central do Sistema de Saúde: LisboaGoogle Scholar
  20. Zegers M, De Bruijne MC, Spreeuwenberg P, Wagner C, Van Der Wal G, Groenewegen PP: Variation in the rates of adverse events between hospitals and hospitals departments. Int J Qual Health Care. 2011, 23 (2): 126-133. 10.1093/intqhc/mzq086.View ArticlePubMedGoogle Scholar
  21. Zegers M, de Bruijne MC, Wagner C, Groenewegen PP, Waaijman R, van der Wal G: Design of a retrospective patient record study on the occurrence of adverse events among patients in Dutch hospitals. BMC Health Serv Res. 2007, 7: 27-34. 10.1186/1472-6963-7-27.View ArticlePubMedPubMed CentralGoogle Scholar
  22. Henriksen K, Kaplan H: Hindsight bias, outcome knowledge and adaptive learning. Qual Saf Health Care. 2003, 12 (Suppl 2): ii46-ii50.PubMedPubMed CentralGoogle Scholar
  23. Schioler T, Lipczak H, Pedersen BL: Incidence of adverse events in hospitals: a retrospective study of medical records: Danish study of adverse events. Scan J Public Health. 2000, 32: 324-333.Google Scholar
  24. Vincent C: Patient Safety. 2010, Oxford: Wiley-Blackwell, 2View ArticleGoogle Scholar
  25. Tsilimingras D, Rosen AK, Berlowitz DR: Patient safety in geriatrics: a call for action. J Gerontol. 2003, 58: 813-819. 10.1093/gerona/58.9.M813.View ArticleGoogle Scholar
  26. Brown P, McArthur C, Newby L, Lay-Yee R, Davis P, Briant R: Cost of medical injury in New Zealand: a retrospective cohort study. J Health Serv Res Policy. 2002, 7 (Suppl 1): S29-S34. 10.1258/135581902320176449.View ArticlePubMedGoogle Scholar
  27. Øvretveit J: Does Improving Quality Save Money?: A Review of Evidence of which Improvements to Quality reduce Costs to Health Service Providers. 2009, London: The Health FoundationGoogle Scholar
  28. Commission E: The Economic Adjustment Programme for Portugal. June 2011, European Commission: BrusselsGoogle Scholar
  29. Parand A, Benn J, Burnett S, Pinto A, Vincent C: Strategies for sustaining a quality improvement collaborative and its patient safety gains. Int J Qual Health Care. 2012, 24 (4): 380-390. 10.1093/intqhc/mzs030.View ArticlePubMedGoogle Scholar
  30. Ovretveit J, Klazinga N: Learning from large-scale quality improvement through comparisons. Int J Qual Health Care. 2012, 24 (5): 463-469. 10.1093/intqhc/mzs046.View ArticlePubMedGoogle Scholar
  31. O'Leary KJ, Devisetty VK, Patel AR, Malkenson D, Sama P, Thompson WK, Landler MP, Barnard C, Williams MV: Comparison of traditional triggers tool to data warehouse based screening for identifying hospital adverse events. BMJ Qual Saf. 2012, 22 (2): 130-138.View ArticlePubMedGoogle Scholar
  32. Hogan H, Healey F, Neale G, Thomson R, Vincent C, Black N: Preventable deaths due to problems in care in English acute hospitals: a retrospective case record review study. BMJ Qual Saf. 2012, 21 (9): 737-745. 10.1136/bmjqs-2011-001159.View ArticlePubMedPubMed CentralGoogle Scholar
  33. Parry G, Cline A, Goldmann D: Deciphering harm measurement. JAMA. 2012, 307: 2155-2156.View ArticlePubMedGoogle Scholar
  34. Zegers M, de Bruijne MC, Wagner C, Groenewegen PP, van der Wal G, de Vet HC: The inter-rater agreement of retrospective assessment of adverse events does not improve with two reviewers per patient record. J Clin Epidemiol. 2010, 63: 94-102. 10.1016/j.jclinepi.2009.03.004.View ArticlePubMedGoogle Scholar
  35. Naessens JM, O'Byrne TJ, Johnson MG, Vansuch MB, McGlone CM, Huddleston JM: Measuring hospital adverse events: assessing inter-rater reliability and trigger performance of the global trigger tool. Int J Qual Health Care. 2010, 22 (4): 266-274. 10.1093/intqhc/mzq026.View ArticlePubMedGoogle Scholar
  36. Viera A, Garrett J: Understanding interobserver agreement:the Kappa Statistic. Fam Med. 2005, 37 (5): 360-363.PubMedGoogle Scholar
  37. de Vet HC, Mokkink LB, Terwee CB, Hoekstra OS, Knol DL: Clinicians are right not to like Cohen’s K. BMJ. 2013, 12: 346:f2125-doi:10.1136/bmj.f2125Google Scholar
  38. Pre-publication history

    1. The pre-publication history for this paper can be accessed here:


© Sousa et al.; licensee BioMed Central Ltd. 2014

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated.