Impact of date stamping on patient safety measurement in patients undergoing CABG: Experience with the AHRQ Patient Safety Indicators

The Institute of Medicine (IOM) report To Err is Human [1] has fundamentally changed the landscape of American medicine by challenging us to examine and improve the safety and quality of the health care system in this country [2]. Seven years after the release of this seminal report, "the quality chasm in health care remains wide [3]." A recent study from RAND documented that adults in the United States receive only 55% of recommended care [4]. In its blueprint for "Crossing the Quality Chasm", the IOM recommended the development of performance and outcome measurements [5]. In response, the Agency for Healthcare Research and Quality (AHRQ) has constructed Patient Safety Indicators (PSI) [6] to serve as the "state-of-the art in measuring the safety of hospital care through the analysis of inpatient discharge data [6]."

The AHRQ PSIs provide information on risk-adjusted rates for potentially preventable adverse events such as decubitus ulcers, postoperative hemorrhage, and postoperative sepsis. In theory, a hospital can benchmark its performance using these safety indicators and target specific areas of poor performance for improvement. This 'global' approach to quality improvement is more likely to be effective than the traditional approach of narrowly focusing on the unsafe actions of a few physicians [7]. Traditional quality assurance (QA) efforts, which are triggered by individual case reviews, may divert limited hospital resources to clinical areas where overall hospital performance is not problematic, while ignoring system-level problems. Using PSIs to guide QI efforts may force hospitals to focus their efforts on the development of systems which will reduce the incidence and mitigate the impact of dangerous medical errors [7].

The Patient Safety Indicators were designed to be used with administrative data and are currently being used by public and non-public organizations to assess hospital quality of care [8]. Basing the PSIs on administrative data makes it possible for virtually any hospital to examine its safety performance. Hospitals can download the PSI software from the AHRQ web site and use their own administrative data to construct risk-adjusted PSI rates and compare themselves to a national benchmark. Because these indicators are based on administrative data, there is no added data collection burden. However, administrative data have well recognized limitations, including the lack of precise definitions and coding differences across hospital [9]. Several investigators have questioned the accuracy of using administrative data to identify complications [10, 11], while others have concluded they may be useful for identifying surgical complications [12].

One of the primary problems with using administrative data to identify adverse events is the failure of most ICD-9-CM codes to distinguish between pre-existing conditions and complications that occur after hospital admission. Counting pre-existing conditions as adverse events may falsely elevate adverse event rates and bias these safety indicators against hospitals which care for sicker patients. In designing the PSIs, AHRQ researchers attempted to exclude ICD-9-CM codes that could represent either a pre-existing condition or a complication [13]. In theory, the use of a present-on-admission (POA) modifier would have made it possible to avoid mis-classifying pre-existing conditions as adverse events. Several states (California, New York State, Florida, and Wisconsin) require hospitals to include a POA modifier in their administrative data. However, AHRQ researchers were initially constrained to develop a set of indicators which could be used by hospitals in all States, and thus were not able to incorporate the POA indicator in the PSI algorithms [6]. The latest version (2007) of the AHRQ PSIs incorporates the POA indicator and gives users the options of using this information to calculate the AHRQ PSIs [14].

Using the POA modifier to differentiate pre-existing conditions from complications, we will examine the accuracy of the AHRQ PSIs using discharge abstracts for patients undergoing coronary artery bypass graft (CABG) surgery in the California State Inpatient Database. Previous studies have shown that the AHRQ PSIs over-estimate the number of patient safety events when pre-existing conditions are incorrectly counted as complications [15, 16]. One of these studies, by Houchens and colleagues [16], specifically examined the impact of counting pre-existing conditions as complications on hospital PSI rates based on all patient admissions. However, both studies are based on a broad range of patient diagnoses and procedures. Although it is likely that there is some overlap in the care processes that prevent certain complications across different surgical procedures and medical conditions, it is also likely that specific best practices for preventing complications may be unique to different patient populations. For example, it is unlikely that one hospital-wide approach to preventing postoperative hemorrhage will be equally effective in patients undergoing hysterectomy compared to CABG patients, because the causes of postoperative hemorrhage in these two groups are so different. Just as procedure-specific mortality rates (e.g. CABG) are more informative, and possibly more useful, than overall hospital mortality rates, we believe that diagnosis-specific PSI rates may prove more useful that hospital-wide PSI rates.

We have previously shown that the addition of date stamp information to administrative data has a substantial impact on the ranking of hospital quality based on risk-adjusted mortality rates [17, 18]. In the current study, our goal is to examine whether the AHRQ PSIs accurately measure the rates of adverse events, using the data with the POA modifier as the gold standard for comparison. Under the Deficit Reduction Act of 2005, the Centers for Medicare and Medicaid Services will require hospitals to add POA indicators to Medicare claims starting in 2007 [19]. The results from this study will help to inform policy-makers as to the value of requiring the inclusion of the POA modifier in non-Medicare claims as well.

In this study we find that the present-on-admission (POA) indicator has a significant impact on some of the AHRQ Patient Safety Indicators (PSI) rates in patients undergoing CABG surgery. The PSIs are one component of the quality toolbox developed by AHRQ to facilitate quality improvement and provide hospitals with the opportunity to benchmark their performance [29, 30].

In practice, quality assurance is usually triggered by case reviews and focuses on the perceived failures of individual physicians and providers. Medical errors are attributed to "aberrant mental processes such as forgetfulness, inattention, poor motivation, carelessness, negligence, and recklessness [7, 31]." Since individual cases selected for examination are often reviewed in isolation, as opposed to being reviewed as part of a cohort of similar cases, the critical role of health care systems in causing medical errors is frequently ignored. Because PSI rates are, by construction, a measure of global hospital performance, they shift the focus of error analysis from the individual provider to the level of the health care system. For example, a high rate of postoperative sepsis after CABG surgery across cardiac surgeons is more likely to improve with better adherence to patient safety practices, such as the use of maximum sterile barriers during catheter insertion or the use of antibiotic-impregnated catheters [32], than by the act of "disciplining" a single physician. Thus, PSIs may provide the impetus for a hospital's leadership to examine the "latent conditions" that lead to medical errors – production pressure, inadequate staffing, fatigue – and help set the stage for the adoption of a true "systems approach" to reducing medical error and improving health care quality.

The AHRQ PSI rates have the advantage of being based on administrative data, which are collected by virtually all hospitals in computerized form, and thus is readily available at low cost. Furthermore, the availability of the AHRQ PSI software in the public domain provides all hospitals with the opportunity to benchmark and track their PSI rates. However, the use of administrative data to monitor complications also has important limitations. In particular, the under-reporting of complications using ICD-9-CM codes, in addition to variability in coding practices across institutions, raises questions regarding the validity of using ICD-9-CM codes to report complications [33] and creates concerns that public reporting of PSI rates may unfairly penalize those hospitals with more accurate reporting practices. The primary limitation of this study is the assumption that the POA indicator accurately distinguishes complications from pre-existing conditions. Parker and colleagues [23] recently examined the accuracy of administrative data from California, using the POA indicator to exclude complications, with a clinical registry for CABG patients. Using the clinical data as the gold standard, the sensitivity of the risk factors in the administrative data ranged between 22% to 95%, with most above 50%. For most risk factors, specificity exceeded 90%. AHRQ has recently released a report summarizing the evidence supporting the value and validity of the POA indicator [19].

However, these well recognized limitations of administrative data for error reporting should not prevent individual hospitals or hospital systems from using non-public reports based on the AHRQ PSI to facilitate quality improvement. Despite the inherent limitations of risk-adjustment for "leveling the playing field" [34, 35], public and non-public reporting of hospital mortality rates have been associated with significant decreases in mortality for cardiac [36, 37] and non-cardiac surgery [38].

Our findings in this study examine one of the known limitations of administrative data for error reporting, namely, the inability of administrative data to effectively distinguish between pre-existing conditions and complications. Despite the fact that the AHRQ PSI were designed to "emphasize specificity over sensitivity", we found significant numbers of false positives for some of the PSIs. The planned expansion of the use of the POA indicator to all Medicare claims, beginning in 2007, could improve the validity of the AHRQ PSIs if the AHRQ algorithm were revised to include the POA indicator. Recent research in the private sector has led to the development of a system to identify Potentially Preventable Complications (PPC) [21]. By incorporating the POA indicator into its algorithms for the PPC groups, it was possible to expand the number of diagnoses that could be considered complications without sacrificing specificity. This expansion in scope of error monitoring, predicated on the use of the POA indicator to distinguish complications from pre-existing conditions, may be the "next step" in the evolution of the AHRQ PSI. This "next-generation" complication reporting system may provide greater opportunities for reducing medical errors and improving health care quality. Our study, by showing significant number of false positives using the AHRQ PSIs, further reinforces the need for the widespread adoption of the POA indicator which will make it possible for revised PSI systems, such as the PPC system, to be widely adopted.

Two recent studies have investigated the impact of the POA indicator on patient safety events. The first, by Naessens and colleagues [15], was based on hospital discharges from the Mayo Clinic Rochester hospitals. This study found that after eliminating secondary diagnoses that were present on admission, the overall rate of patient safety events decreased by nearly 50%. The second, by Houchens and colleagues [16], used data from California and New York State Inpatient Databases to examine the impact of the POA indicator. This study found that three of the 13 PSIs greatly over-estimated the number of patient safety events when information from the POA indicator was not used to differentiate pre-existing conditions from complications. For these three PSIs, there were significant discrepancies between hospital risk-adjusted PSI rates before and after excluding pre-existing conditions. Our study adds to the existing literature by focusing on a single medical condition, CABG surgery, as opposed to basing the analysis on all inpatient admissions. We believe that PSIs will be useful only insofar as they allow physicians and hospitals to identify problems and, then to focus QI efforts for specific hospital departments, as opposed to solely providing hospitals with a global measure of patient safety events. In this light, studies evaluating the validity of the AHRQ PSIs should assess disease-specific performance, in addition to global performance. In addition, our previous work has shown that the extent to which complications are mis-identified as pre-existing conditions varies substantially across patient populations (e.g. CABG, abdominal aortic aneurysm repairs, stroke patients) [22]. Thus, it is likely that the accuracy of PSIs would also vary across patient groups. The accuracy of the AHRQ PSI in CABG patients may be of particular interest to hospitals seeking to improve CABG outcomes that only have access to administrative data without the POA indicator.

Increasingly, private payers and Medicare are promoting the use of financial incentives to improve the quality of care through pay for performance initiatives. Nationally, over fifty-percent of Health Maintenance Organizations covering greater than 80% of enrolled patients, have pay-for-performance programs in place [39]. Under the Deficit Reduction Act of 2005, the reduction in hospital Medicare payments to hospitals not reporting quality data will increase five-fold from 0.4 percent to 2 percent, and infectious complications will no longer entitle hospitals to higher reimbursement rates [40]. However, the actual impact of pay-for-performance on quality is largely unknown [41], although recent work suggests that financial incentives has a relatively modest effect on adherence to process measures [42]. Even if financial incentives were found to significantly improve adherence to process measures, recent work suggests that adherence to "best practices" has only a relatively modest impact on risk-adjusted 30-day mortality rates for patients with acute myocardial infarctions (0.6%), heart failure (0.1%) and pneumonia (0.1%) [43]. In light of the weak association between many processes of care and outcome, direct outcome measures, such as PSI and related measures of adverse events, may have an important role in future efforts to improve health care quality.

For some of the Patient Safety Indicators, there are significant differences in the rates of adverse events depending on whether the POA indicator is used to distinguish between pre-existing conditions and complications. The use of the POA indicator will increase the accuracy of the AHRQ PSIs as measures of adverse outcomes, and will make the future implementation of more comprehensive measures of complications more feasible.