A number of private and public companies calculate and publish proprietary hospital patient safety scores based on publicly available quality measures initially reported by the U.S. federal government. This study examines whether patient safety culture perceptions of U.S. hospital staff in a large national survey are related to publicly reported patient safety ratings of hospitals.
The Agency for Healthcare Research and Quality Hospital Survey on Patient Safety Culture (Hospital SOPS) assesses provider and staff perceptions of hospital patient safety culture. Consumer Reports (CR), a U.S. based non-profit organization, calculates and shares with its subscribers a Hospital Safety Score calculated annually from patient experience survey data and outcomes data gathered from federal databases. Linking data collected during similar time periods, we analyzed relationships between staff perceptions of patient safety culture composites and the CR Hospital Safety Score and its five components using multiple multivariate linear regressions.
We analyzed data from 164 hospitals, with patient safety culture survey responses from 140,316 providers and staff, with an average of 856 completed surveys per hospital and an average response rate per hospital of 56%. Higher overall Hospital SOPS composite average scores were significantly associated with higher overall CR Hospital Safety Scores (β = 0.24, p < 0.05). For 10 of the 12 Hospital SOPS composites, higher patient safety culture scores were associated with higher CR patient experience scores on communication about medications and discharge.
This study found a relationship between hospital staff perceptions of patient safety culture and the Consumer Reports Hospital Safety Score, which is a composite of patient experience and outcomes data from federal databases. As hospital managers allocate resources to improve patient safety culture within their organizations, their efforts may also indirectly improve consumer-focused, publicly reported hospital rating scores like the Consumer Reports Hospital Safety Score.
Since the Institute of Medicine published To Err is Human in 2000 , there has been an increase in federal reporting requirements for hospitals. In turn, consumers’ access to a wide range of hospital safety and quality measures, including infection rates, preventable readmission rates, and data on patient experience, has also increased. Multiple private reporting services in the U.S. have aggregated these hospital quality measures into single, easy-to-use hospital scores to help consumers interpret the data when deciding where to seek care—for example, the Consumer Reports (CR) Hospital Safety Score , U.S. News and World Report Hospital Rankings , and Leapfrog Hospital Safety Score . However, these reporting systems often provide conflicting information [5, 6]. Each rating system uses its own rating methods, focuses on different aspects of quality of care, and measures different areas of performance. It is important for consumers to understand the differences between these systems and for hospitals to understand drivers or ways to improve their scores.
Many hospitals currently assess patient safety culture to identify areas for improvement. Patient safety culture refers to providers’ and staff values, beliefs, and norms about what is important in a healthcare organization, how organization members are expected to behave, what attitudes and actions are appropriate, and what processes and procedures are rewarded and punished with regard to patient safety .
CR has more than 3 million paying online subscribers who have access to U.S. Medicare-certified hospital safety and quality measures, including the Hospital Safety Score . The Hospital Safety Score is a composite rating based on publicly available federal data regarding infection and readmissions rates, communication between staff and patients, use of scanning, and mortality rates, and gives consumers a way to compare hospitals on patient safety. Non-subscribers can view the top 10 scoring and bottom 10 scoring hospitals for free online . In addition, local and national news outlets annually cover CR’s release of the Hospital Safety Score, which pressures hospital leaders to publicly respond to their respective ratings and national rankings [10–12].
Given the local and national attention that the CR Hospital Safety Score receives, hospitals may be motivated to improve quality measures that are used to calculate their Safety Score. Focusing on patient safety culture may do just that. Recent studies found relationships between patient safety culture and patient safety indicators and readmission rates, which are used to calculate the CR Hospital Safety Score. Singer et al. found that higher staff perceptions of patient safety culture were related to fewer patient safety adverse events . Similarly, Mardon et al. found that higher patient safety culture scores were associated with fewer adverse events in hospitals . Another study reported lower non-management staff perceptions of safety climate were associated with higher readmission rates for acute myocardial infarction and heart failure, as well as lower rates of mortality . Birkmeyer et al. found that rates of serious complications were significantly lower among hospitals receiving an overall safety rating of “Excellent” from nurses, compared with those receiving a “Very good” or “Acceptable” rating .
Patient safety culture has also been linked with patient experience, which is another quality measure used to calculate the CR Hospital Safety Score. Sorra et al. found that hospitals with higher patient safety culture scores tended to have more positive assessments of care from patients on the Consumer Assessment of Healthcare Providers and Systems (CAHPS®) Hospital Survey . Patients’ perceptions of communication with nurses (e.g. explained things so that patients could understand) and responsiveness of hospital staff were most often positively related to staff perceptions of patient safety culture as measured by the Agency for Healthcare Research and Quality (AHRQ) Surveys on Patient Safety Culture (SOPSTM) Hospital Survey.
The purpose of this research is to examine the relationship between one of the publicly reported, consumer-oriented hospital patient safety rating systems, the CR Hospital Safety Score, and patient safety culture scores as measured by AHRQ’s Hospital SOPS. We hypothesized that hospitals with higher Hospital SOPS scores would have a higher CR Hospital Safety Score. Furthermore, we hypothesized that hospitals with higher Hospital SOPS scores would be related to the CR Hospital Safety Score components in the following ways: 1) lower hospital acquired infection rates, 2) lower hospital-wide readmission rates, 3) better communication about medications and discharge with patients, 4) more appropriate use of scanning, and 5) lower mortality rates.
Data sources and measures
AHRQ Hospital Survey on Patient Safety Culture (Hospital SOPS)
The Hospital SOPS assesses provider and staff perceptions about patient safety issues, medical errors, and event reporting. The development team reviewed research pertaining to safety, patient safety, health care quality, medical errors, error reporting, and organizational climate and culture. In addition, existing safety climate and culture instruments were examined. Then, key dimensions of patient safety culture were identified and survey items were developed. The survey was pilot tested, revised, reviewed by technical experts and then AHRQ released it in 2006. The survey is psychometrically sound at the individual, unit, and hospital levels of analysis .
The survey includes 42 items that measure 12 composites of patient safety culture. Each of the 12 patient safety culture composites is listed and defined in Table 1. The items that make up these composites and the overall patient safety rating item can be found on the AHRQ Web site . AHRQ funds a comparative database to enable hospitals to compare their survey results with other hospitals . Hospitals voluntarily submit their data for inclusion in the database.
The Hospital SOPS items use 5-point response scales of agreement (“Strongly disagree” to “Strongly agree”) or frequency (“Never” to “Always”). Each item’s percent positive score consists of the percentage of positive responses (“Agree” or “Strongly Agree”, “Most of the time” or “Always” to positively worded items) within a hospital. Prior to analyses, negatively worded items were reverse coded so that higher scores represent positive responses. Percent positive scores were calculated for composites by taking the average of the percent positive scores for the 3 or 4 items that make up the composite . Rarely, hospitals exclude composite items from their survey or have such a small sample size that they do not receive a respective composite score. When an item is excluded, the composite score that includes that item is not calculated for that hospital.
The survey also includes a single-item measure that asks providers and staff to give their work area/unit a patient safety grade on a 5-point scale ranging from “Excellent” to “Failing”. The percent positive patient safety grade was calculated as the percentage of respondents who gave their work area/unit either a grade of “Excellent” or “Very good”. For these analyses we create a Hospital SOPS composite average index scores, which is the mean value of the 12 composite measures. All Hospital SOPS measures had the potential to range between 0 and 100% positive. This study utilized the Hospital SOPS 12 patient safety culture composites, the Hospital SOPS composite average score, and the overall patient safety rating.
Consumer Reports Hospital Safety Score
Consumer Reports (CR) is a U.S. based non-profit organization that provides its subscribers access to hospital ratings based on patient experience survey data and outcomes data provided by the Centers for Medicare and Medicaid Services (CMS) the Centers for Disease Control (CDC) and Prevention. Both CMS and CDC are federal agencies within the U.S. Department of Health and Human Services, which has the overall goal of protecting health of all Americans and providing essential health and human services. This data is used to reimburse and reward providers and health care settings that adhere to government policy. CR, using elements of this data, annually calculates a Hospital Safety Score based on five standardized component measures: 1) Hospital acquired infections, 2) Hospital-wide readmissions, 3) Communication about medications and discharge, 4) Appropriate use of scanning, and 5) Mortality. CR enlists the help of external expert reviewers for feedback on measure methodology and on how to turn raw federal data into hospital ratings. The technical details for how each measure is calculated can be found on the Consumer Reports Web site . Higher scores indicate lower infection and readmissions rates, better communication about medications and discharge between staff and patients, more appropriate use of scanning, and lower mortality rates.
The CR Hospital Safety Score is the mean of these five equally weighted components. That mean is then linearly transformed to a scale of 0 to 100 (100 being the safest). CR chose these five measures to serve as indicators of a hospital’s commitment to patient safety and deliberately did not include process measures. Furthermore, the five components were selected because they represent a broad range of patient outcomes. When possible, federal data were statistically adjusted prior to public reporting to minimize differences among hospitals due to the types of patients they served.
The analysis dataset consisted of 164 hospitals out of the 419 potential hospitals that voluntarily submitted survey data to the AHRQ Hospital SOPS 2014 Comparative Database and were in the CR 2014 Hospital database. The 419 hospitals that existed in both databases were provided with a Data Use Agreement describing the proposed analyses and requesting permission to use their data. These analyses only include data from the 164 hospitals that authorized use of their data for this study. The Hospital SOPS data from these hospitals were collected from 2011 to 2013 and the CR data were collected between 2009 and 2013. All data were aggregated to the hospital level for analyses.
The 164 hospitals in the analytic dataset differed slightly in characteristics from the 6407 U.S. hospitals registered in the 2011 American Hospital Association (AHA) Annual Survey of Hospitals. Hospitals in the analytic dataset tended to be larger (i.e., more beds) compared with AHA-registered U.S. hospitals (Table 2). The majority of the hospitals in the analytic dataset were nonteaching (55%), though this was lower than the percentage of nonteaching AHA-registered U.S. hospitals (76%). Finally, the hospitals overrepresented nongovernment hospitals (93 versus 75% in the AHA dataset).
Hospital characteristics as covariates
Hospital characteristics obtained from AHA data were examined as covariates or control variables: bed size (dummy coded as a categorical variable, with 49 beds or less as the comparison group), teaching status (coded as 1 for teaching and 0 for non-teaching), and ownership (coded as 1 for government and 0 for nongovernment). It is important to control for the impact of these variables because hospital characteristics have shown consistent associations with Hospital SOPS scores and also may be associated with the CR Safety Score .
Multivariate linear regressions were conducted to estimate the relationship between Hospital SOPS measures and the CR Hospital Safety Score, controlling for bed size, teaching status, and ownership. Regressions were conducted to estimate the relationships between Hospital SOPS measures and each of the five CR component measures: 1) Hospital acquired infections, 2) Hospital-wide readmissions, 3) Communication about medications and discharge, 4) Appropriate use of scanning, and 5) Mortality. Multivariate regression allows for the simultaneous analysis of multiple predictor variables (i.e., Hospital SOPS scores) and controls (bed size, teaching status, and ownership) on the dependent variable of interest (e.g., CR Safety Score). Using one Hospital SOPS measure per model avoids multicollinearity, which would occur if we included all Hospital SOPS measures in a single regression model. Multicollinearity can lead to biased standard errors and unreliable estimates of standardized regression coefficients. All analyses were completed using SAS 9.3.
The mean number of Hospital SOPS respondents per hospital was 856 (minimum, 34; maximum, 7,806). The mean Hospital SOPS response rate per hospital was 56% (minimum, 12%; maximum, 100%). As shown in Table 3, Hospital SOPS scores ranged from 43% positive response (Handoffs and transitions) to 81% positive response (Teamwork within units). The Hospital SOPS scores of the hospitals included in our analysis tended to be slightly lower than the full set of 653 hospitals in the 2014 Hospital SOPS Comparative Database (top of Table 3, last column).
CR Hospital Safety Scores in the analysis dataset ranged from 29.86 to 74.98 with an average of 54.14, which was slightly higher than the overall CR database average of 50.91. Furthermore, all of the CR component measures in the analysis dataset, except the hospital acquired infections component measure, were higher than the CR database average (bottom of Table 3, last column).
Across the multiple regressions, the CR Hospital Safety Score and its five components were often negatively associated with teaching hospitals, such that teaching hospitals tended to have lower CR ratings. Ownership status and categorical bed size were generally not significantly related to the CR ratings.
Relationships with CR Hospital Safety Score
Table 4 displays multivariate regression results. Higher Hospital SOPS composite average scores were associated with higher CR Hospital Safety Scores, controlling for bed size, ownership, and teaching status (β = 0.24, p < 0.05). Furthermore, nine of the 12 Hospitals SOPS composites were positively related with the CR Hospital Safety Score ranging from β = 0.17 to β = 0.29. The Hospital SOPS composite with the largest standardized coefficient was Teamwork within units, signifying that more positive staff perceptions of teamwork within units was associated with higher CR Hospital Safety Scores. The Hospital SOPS composites Teamwork across units and Overall perceptions of patient safety had the next largest relationships with CR Hospital Safety Scores (β = 0.23). The percentage of providers and staff giving their hospital a patient safety grade of “Excellent” or “Very good” was also related with higher CR Hospital Safety Scores (β = 0.22).
Relationships with the five CR components
Higher hospital SOPS composite average scores were associated with better communication about medications and discharge (β = 0.30) (Table 4). For 10 of the 12 Hospital SOPS composites, higher patient safety culture scores were also associated with better communication about medications and discharge component (range β = 0.18 to 0.35); the strongest relationship was with Teamwork within units. Higher patient safety culture scores for four composites were also associated with lower hospital acquired infections, ranging from β = 0.19 to β = 0.24.
Two Hospital SOPS composites were related to the CR component Hospital-wide readmissions: 1) More frequent reporting of patient safety events was associated with higher readmission rates (β = −0.21), which was in a direction that was opposite of our hypothesized relationship, and 2) better Teamwork within units was associated with more appropriate use of scanning (β = 0.17). None of the Hospital SOPS measures were related to the CR component Mortality.
Relationships with CR Communication about medications and discharge subcomponents
Of the five CR components, the patient experience component on Communication about medications and discharge had the strongest and most consistent relationships with Hospital SOPS measures. Therefore, we conducted additional analysis to better understand what was driving these relationships.
The CR component Communication about medications and discharge is made up of two patient experience subcomponents: 1) Communication about medications and 2) Communication about discharge; each subcomponent is derived from two CAHPS survey items. The CR subcomponent Communication about medications was derived from the following CAHPS items: 1) Before giving you any new medicine, how often did hospital staff tell you what the medicine was for? 2) Before giving you any new medicine, how often did hospital staff describe possible side effects in a way you could understand?
The CR Communication about discharge was derived from the Hospital CAHPS items: 1) During this hospital stay, did doctors, nurses or other hospital staff talk with you about whether you would have the help you needed when you left the hospital? 2) During this hospital stay, did you get information in writing about what symptoms or health problems to look out for after you left the hospital? Information on how these subcomponents were calculated can be found on the Consumer Reports Web site .
We conducted post hoc linear regression analysis to determine which, if either, of the two communication subcomponents 1) Communication about medications and 2) Communication about discharge, was driving the relationship with the Hospital SOPS measures, even when controlling for bed size, ownership, and teaching status (Table 5). All 12 Hospital SOPS composite scores (range: β = 0.24 to β = 0.44) and the Hospital SOPS patient safety grade (β = 0.44) were positively related to the CR subcomponent Communication about medications indicating that higher staff perceptions of patient safety culture were related to higher patient perceptions of how well hospital staff communicate about medication. Five Hospital SOPS composites were positively related to the CR subcomponent Communication about discharge, ranging from β = 0.17 to β = 0.24.
This is the first study that has explored the relationship between hospital providers’ and staff perceptions of patient safety culture and the publicly reported, consumer-oriented CR Hospital Safety Score, which was derived from publicly available hospital data. We found that hospitals where providers and staff have more positive perceptions of patient safety culture tended to have higher CR Hospital Safety Scores. Our results also show that higher frequency of events reported is related to higher hospital-wide readmission rates. For the most part, these findings suggest that both the Hospital SOPS and the CR Hospital Safety Score are valid measures that generally complement each other. Given that Medicare-certified hospitals are reimbursed for providing high quality services and, in particular, lower readmission rates, future research might investigate if improving one’s patient safety culture might also lead to improvements in aspects of care that would lead to higher future CMS reimbursement rates.
Drilling down further to help determine what drives these findings, the better communication about medications with patients was most often related to higher Hospital SOPS measures. These findings are consistent with Sorra et al. , which found significant relationships between 10 of the 12 Hospital SOPS composites and the Hospital CAHPS Communication about medicine item while only one Hospital SOPS composite, Frequency of events reported, was related to the Communication about discharge item. In addition, more positive Teamwork within units was related to lower hospital-wide readmissions, better communication about medications and discharge, and more appropriate use of scanning.
While not the primary focus of this research, it is interesting to note that teaching hospitals tended to have lower CR Hospital Safety Scores. These findings are consistent with previous studies that found AHRQ patient safety indicator rates, which distinguish adverse events resulting from medical intervention, are higher among teaching hospitals [14, 22]. In addition, teaching hospitals have historically reported lower patient safety culture scores .
There are several limitations to this analysis. We were able to conduct analysis on only 164 hospitals that voluntarily agreed to participate in the study and also independently administer the Hospital SOPS survey. These hospitals may be more actively addressing patient safety and quality issues compared with hospitals that declined participation or do not administer Hospital SOPS. However, this is less likely given the broad range of Hospital SOPS composite scores. Larger, non-government hospitals are overrepresented compared to non-participating hospitals in this analysis, so findings may not be generalizable to the entire U.S. hospital population.
In addition, hospitals do not provide staff position census numbers when submitting survey data to the Comparative Database and collect survey data using either paper, online, or both modes. It is also unknown if hospitals equally targeted or incentivized staff to complete surveys. Therefore, submitted staff responses may not be representative of their respective hospitals.
A third limitation is that this study was cross-sectional, and we cannot directly measure any causal relationship between patient safety culture and the outcomes aggregated in the CR Safety Score. Since we hypothesized that patient safety culture would be related to hospital outcomes, ideally patient safety culture results would be collected preceding or concurrently with hospital outcomes data. However, a portion of the outcomes data used to calculate CR Safety Scores was collected up to two years earlier than the Hospital SOPS data; furthermore, a portion of the mortality component was collected up to four years prior to Hospital SOPS administration. This measurement “noise” could have affected our findings or lack thereof. However, the Hospital SOPS timeframe does coincide with the CR component measures that had the strongest relationships (i.e., Communication about medications and discharge and Hospital acquired infections).
Future analysis should more closely examine how nationally standardized outcome measures, as well as other CR ratings, relate to Hospital SOPS measures. In addition, research examining the relationships between patient safety ratings from other companies (e.g., Leapfrog, U.S News and World Report, and Healthgrades) and provider and staff perceptions of patient safety culture might help patients and hospital leaders further understand how these different measures of patient safety are related patient safety and quality . Additional research on the relationships between medical students, patient safety culture, and preventable adverse events would be prudent.
In conclusion, we have discovered relationships between AHRQ Hospital SOPS measures and the CR Safety Score and its components. We found statistically significant relationships showing that higher Hospital SOPS composite scores are associated with higher CR Hospital Safety Scores. Furthermore, our analysis of the CR Hospital Safety Score components suggests that the main driver of these relationships was the patient experience communication component collected by the CAHPS Hospital survey. This analysis may lay the groundwork for future research examining the relationships between hospital provider and staff perceptions of patient safety culture and publicly reported patient safety scores.
American Hospital Association
The Agency for Healthcare Research and Quality
Consumer Assessment of Healthcare Providers and Systems (CAHPS®)
The Centers for Disease Control
The Centers for Medicare and Medicaid
Surveys on Patient Safety Culture (SOPSTM)
Institute of Medicine (IOM). In: Kohn LT, Corrigan JM, Donaldson MS, editors. To Err is human: building a safer health system. Washington, D.C: National Academy Press; 2000.
Austin JM, Jha AK, Romano PS, Singer SJ, Vogus TJ, Wachter RM, Provonost PJ. National hospital ratings systems share few common scores and may generate confusion instead of clarity. Health Aff. 2015;34(3):423–30.
Rivard PE, Elixhauser A, Christiansen CL, Zhao S, Rosen AK. Testing the association between patient safety indicators and hospital structural characteristics in VA and nonfederal hospitals. Med Care Res Rev. 2010;67(3):321–41. doi:10.1177/1077558709347378.
Famolaro T, Yount ND, Burns W, et al. Hospital Survey on Patient Safety Culture 2016. Comparative Database Report, Appendixes, Parts II and III (Appendixes for AHRQ Publication No.16-0021-EF). (Prepared by Westat, Rockville, MD, under contract No. HHSA290201300003C). Rockville: Agency for Healthcare Research and Quality; 2016.
This research was funded by the Agency for Healthcare Research and Quality: Contract # HHSA290201300003C.
Availability of data and materials
Hospitals agreed to release their data for this analysis with the stipulation that individual results would not be available to the public. Therefore, we are unable to make an analysis dataset available.
SS and NY conceived and designed the analysis. SS performed the analysis. SS and NY wrote the paper. JS revised critically the paper for important intellectual contents. All authors read and approved the final manuscript.
The authors declare that they have no competing interests.
Consent for publication
Ethics approval and consent to participate
Not applicable. Hospitals included in this study voluntarily submitted respondent-level de-identified data to the Agency Healthcare Research & Quality Hospital (AHRQ) Survey on Patient Safety Culture Comparative Database. AHRQ reviewed the study design and allowed the authors of this study to reach out to hospitals to release their hospital-level data for this study. Subsequently, an official of each hospital included in this study signed a consent form allowing the use of their organization’s data for this specific research. Consumer Reports reviewed the study design and provided a national database to link their data to the hospitals’ patient safety culture data for analysis; however, they did not participate in the design of the study, analysis of the data, or the writing or editing of the manuscript. The 2011 American Hospital Association dataset was purchased for both commercial and research purposes.
Authors and Affiliations
Westat, 1600 Research Blvd, Rockville, MD, 20850, USA
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.