COVID-19 inpatient care performance in the unified health system, São Paulo state, Brazil: an application of standardized mortality ratio for hospitals’ comparisons

Abstract

Objective

To evaluate the variation in COVID-19 inpatient care mortality among hospitals reimbursed by the Unified Health System (SUS) in the first two years of the pandemic in São Paulo state and make performance comparisons within periods and over time.

Methods

Observational study based on secondary data from the Hospital Information System. The study universe consisted of 289,005 adult hospitalizations whose primary diagnosis was COVID-19 in five periods from 2020 to 2022. A multilevel regression model was applied, and the death predictive variables were sex, age, Charlson Index, obesity, type of admission, Brazilian Deprivation Index (BrazDep), the month of admission, and hospital size. Then, the total observed deaths and total deaths predicted by the model’s fixed effect component were aggregated by each hospital, estimating the Standardized Mortality Ratio (SMR) in each period. Funnel plots with limits of two standard deviations were employed to classify hospitals by performance (higher-than-expected, as expected, and lower-than-expected) and determine whether there was a change in category over the periods.

Results

A positive association was observed between hospital mortality and size (number of beds). There was greater variation in the percentage of hospitals with as-expected performance (39.5 to 76.1%) and those with lower-than-expected performance (6.6 to 32.3%). The hospitals with higher-than-expected performance remained at around 30% of the total, except in the fifth period. In the first period, 64 hospitals (18.3%) had lower-than-expected performance, with standardized mortality ratios ranging from 1.2 to 4.4, while in the last period, only 23 (6.6%) hospitals were similarly classified, with ratios ranging from 1.3 to 2.8. A trend of homogenization and adjustment to expected performance was observed over time.

Conclusion

Despite the study’s limitations, the results suggest an improvement in the COVID-19 inpatient care performance of hospitals reimbursed by the SUS in São Paulo over the period studied, measured by the standardized mortality ratio for hospitalizations due to COVID-19. Moreover, the methodological approach adapted to the Brazilian context provides an applicable tool to follow-up hospital’s performance in caring all or specific-cause hospitalizations, in regular or exceptional emergency situations.

Introduction

How much the COVID-19 pandemic has challenged the world’s health systems, exposing their strengths and weaknesses [1,2,3,4], has been widely documented. Among the main difficulties was the need to adapt hospital units’ installed capacity and protocols to provide the care required by patients with more severe disease manifestations. Moments of collapse were witnessed in hospitals due to high case incidence waves, compromising the response to the demands posed by COVID-19 and those regularly caused by other health problems. Brazilian hospital services of the Unified Health System (SUS) were exhausted [5], with different hospital reorganizations in the face of the pandemic, possibly reflecting on varying use and results of health care. Scrutinizing this experience could be useful in the future to ensure greater effectiveness, efficiency, patient and professional safety and health system resilience for long-term sustainability.

Given COVID-19’s high lethality, analyses of individual factors predicting hospital death [6] emerged, as well as analyses of hospital mortality variations – a measure for tracking health care quality –, explained by patients’ risk factors, and provision, access to and appropriateness of health care provided [7]. Many studies aimed to understand the effects of patients’ sociodemographic characteristics, geographic location, and clinical condition, expressed in severity indices and comorbidities, on hospital mortality by COVID-19. Some also explored the effects of the care process, such as the use of ventilatory support or intensive care [8,9,10,11]; hospital structure – number of beds [12], volume of hospitalizations [13] and daily admissions due to COVID-19¹⁴, among other context-dependent aspects. Furthermore, there was increasing evidence of declining deaths considering factors at the individual and hospital level over time due to increased knowledge and experience in handling the disease, the emergence of new therapeutic resources, and anti-COVID-19 vaccines [15,16,17].

However, the application of multilevel models to classify hospital performance during the COVID-19 pandemic was more uncommon in the literature and not considered in the Brazilian context at all. It discriminates hospitals that overperform or underperform over time, considering case mix, as in a study that evaluated the variation in mortality rates standardized across hospitals in England during the first pandemic wave [14]. The method requires adjustments to contextual factors and may be explored for monitoring healthcare quality in multiple situations, including public health emergencies.

The SUS is universal, public, and the only healthcare provider for about 75% of the Brazilian population. On the one hand, Brazilian studies pointed to the worse performance of the SUS hospital network against the private sector during the pandemic [13, 18], resulting from many years of low investment and public system scrapping. On the other hand, it is also documented that the SUS played a central role in combating COVID-19 [19] on several fronts, accounting for at least 70% of hospitalizations nationally [13]. Methods for monitoring SUS health care performance are fundamental in conjunction with the necessary investments in its structures, personnel, and overall strengthening.

When strictly evaluating the variation in hospital mortality related to COVID-19, the work developed by Bottle et al. [14] sought to measure how case management learning improved care, providing subsidies to reorganize the hospital service network, including potentially modifiable structural characteristics in the face of future threats [20]. From this perspective, this study comprises a methodological exploration to analyze this classic hospital performance indicator – death – and aims to study the variation in COVID-19 inpatient care mortality among hospitals reimbursed by the SUS in the first two years of the pandemic in São Paulo state (Brazil) and to make comparisons of hospital performance within periods and over time.

Methods

Study design

Observational, secondary data-based study to analyze the variation in the Standardized Mortality Ratio (SMR) based on hospital admissions of adults with a primary diagnosis of COVID-19, coded, according to the recommendation from the Brazilian Ministry of Health (MS), as B34.2 from ICD-10 [International Statistical Classification of Diseases and Related Health Problems, tenth revision]. The SMR expresses the quotient between the number of observed and predicted deaths (O/P), estimated by a risk adjustment model to consider the hospital variability.

Data source and study universe

The primary source was the Hospital Information System (SIH), a publicly accessible administrative database provided by the Ministry of Health, which contains information on hospitalizations reimbursed by the SUS. We identified 389,156 hospitalizations from COVID-19 in individuals aged 18–99 in São Paulo state from February 2020 to February 2022. The geographic scope was justified because it represented the highest volume of COVID-19 hospitalizations in a Brazilian state and, above all, due to the best registration of secondary diagnoses (comorbidities) in comparison to the other Federation Units (UF). In addition, São Paulo state is the most populous and richest in the country and was one of the less vulnerable in terms of the availability of complex healthcare resources during the pandemic in Brazil. We adopted the National Registry of Health Establishments (CNES) to identify hospitalizations in general hospitals in operation since the onset of the pandemic; in this sense, we considered only establishments active in January 2020 classified as general hospitals, according to CNES. The match between SIH observations and the list of general hospitals selected 295,124 admissions. We excluded 3,954 hospitalizations whose reason for leaving was administrative closure and 2,165 that occurred in hospitals with less than 100 admissions in the first two years of the pandemic (0.7% of admissions). At the end of this process, 289,005 hospitalizations made up the study universe.

Data analysis

The analysis plan established five periods from February 2020 to February 2022, distinguishing three COVID-19 waves, as indicated by the FIOCRUZ COVID-19 Observatory [21]. The first period, February-August 2020, included the first wave of the pandemic identified in the national territory and São Paulo state. The second period was from September to November 2020. The third, representing the second and largest COVID-19 wave in the state and country, occurred from December 2020 to June 2021. The fourth period was from July to November 2021, and the fifth and final period, which defined the third wave, occurred from December 2021 to February 2022.

A risk adjustment model was built using multilevel logistic regression with random intercepts for hospitals for each period. The dichotomic dependent variable was death vs. not death (survival). The final model for each of the five periods was tested for discriminative capacity based on the C statistic, for which values from 0.7 to 0.8 are reasonable and higher, good [22]. The Hosmer-Lemeshow plots were used to evaluate the calibration [23] (see Additional file 1 to Additional file 5). At the patient level, the predictive variables included in the final risk model were sex, age (18–29; 30–39; 40–49; 50–59; 60–69; 70–79; 80–89 and 90–99), Charlson Index (0; 1; ≥ 2), obesity (ICD-10: E66), type of admission (elective/urgent), the Brazilian Deprivation Index (quintiles 1, 2, 3, and 4/5), and month of admission. The Charlson Index (CI) was calculated based on the algorithm proposed by Quan et al. [24], considering comorbidities registered at admission among the nine secondary diagnoses in the SIH. The Brazilian Deprivation Index (BrazDep) of the patient’s residence municipality [25] was included in the model by classifying it into quintiles, where the first quintile had the lowest deprivation and the last had the highest. The fourth and fifth quintiles were grouped for analysis purposes. The variables ethnicity/skin color and secondary diagnosis of hypertensive disease (ICD-10: I10, I11, I12, I13, and I15) were tested at the patient level, but the variables were not included in the final model. The ethnicity/skin color variable presented a high percentage of unknown information. The origin/source of admission (home or another hospital), a variable frequently included in the risk adjustment for hospital mortality, was not tested as it is not included in the SIH.

At the hospital level, only hospital sizes (up to 49 beds; 50–149; 150–299; and 300 or more) were included in the final risk model. However, the number of daily COVID-19 hospitalizations in the period was tested (less than 2 hospitalizations per day; 2–4; and above four hospitalizations) but was not statistically significant. Due to the low number of admissions to private hospitals, which represented 0.6% of admissions, this variable was excluded from the final model; these hospitalizations occurred in only three (0.8%) of the 365 hospitals evaluated.

The number of deaths expected at the hospitalization level was calculated from the multilevel model’s fixed effects component, excluding random (hospital) effects. Thus, an aggregate database per hospital was constructed for each period, where the total number of observed deaths and the total number of predicted deaths were computed to estimate each hospital’s SMR in each period.

Each model evaluated the intraclass correlation coefficient (ICC), which expresses how much of the total patient mortality variation is attributed to hospitals [26]. Funnel plots with thresholds of two standard deviations for the five different periods were applied to identify hospitals with discrepant results in the SMR, classifying them into three performance categories (higher-than-expected performance, as-expected performance, and lower-than-expected performance). These categories correspond, respectively, to hospitals with lower-than-expected mortality, plotted below the funnel’s inferior dotted line; hospitals with expected mortality, plotted between the funnel’s inferior and superior dotted lines; and hospitals with higher-than-expected mortality, plotted above the funnel’s superior dotted line. The funnel plot also presented the 99.8% thresholds (three standard deviations) based on the Poisson distribution.

Once the hospitals were classified into the three performance categories in each pandemic period, descriptive statistics for the number of hospitals, admissions, observed deaths, expected deaths, crude mortality, adjusted/standardized mortality, and standardized mortality ratio were obtained for all combined cases of performance class and period. Furthermore, considering that the number of hospitals involved in COVID-19 inpatient care varied expressively across the pandemic periods, we selected those consistently involved (327 hospitals) to compare how they overall performed over time concerning the average inpatient care performance under the different contextual conditions related to the pandemic periods. As an illustration, we registered the transitions of these hospitals between the extreme periods in the analyses – from the first to the last pandemic period –, partially expressing their compared to average performance trends.

Additionally, sensitivity analyses excluded hospitals that did not record deaths during the periods analyzed (see Additional file 6 to Additional file 8). In total, 62, 75, 1, 61, and 38 hospitals were excluded in the first, second, third, fourth, and fifth periods, respectively. The analyses and funnel plots were built in the R statistical program.

Results

A total of 80,487 deaths by COVID-19 were observed in the study universe (N = 289,005). In all periods, crude mortality was higher among men, in older age groups, among those with comorbidities (hypertensive disease and obesity), and higher CI scores (Table 1). A progressive drop in hospital mortality was observed from the beginning of the pandemic (February-March/2020) until November/2020. From December 2020, the beginning of the second wave and the most critical pandemic period, crude hospital mortality rose again, reaching its peak in March 2021 (33.3%), with 34,641 recorded hospitalizations. In the fourth period, crude hospital mortality remained between 23.7% and 24.7% and increased in the fifth period evaluated (third wave) (Table 1).

Table 1 Characteristics of COVID-19 hospital admissions in the SUS, São Paulo state, Brazil, February 2020–2022

A downward mortality trend was observed in the first and second periods after risk adjustment based on multilevel modeling, although they are not directly comparable. In the third period (second wave), hospital deaths in March 2021 were 2.05 more likely than those observed in December 2020, decreasing in the subsequent months. Although it seems possible to identify a new increase in cases and crude hospital mortality in January and February 2022, the adjusted model did not show a statistically significant association between hospital death and month of admission in the last period evaluated (Table 2).

Table 2 Multilevel logistic regression model for COVID-19 inpatient in the SUS, São Paulo, Brazil, February 2020–2022

Statistically significant predictors for all periods were age, sex, CI, and size (number of beds) (Table 2). The obesity variable was maintained in all models, although it was not statistically significant in the last period (OR = 1.26; 95%CI:0.88–1.82); the type of admission was not statistically significant in the second (OR = 1.22; 95% CI:0.91–1.64) and fifth periods (OR = 1.22; 95% CI:0.87–1.69). In the third period, there was a statistically significant association between deprivation (BrazDep) and mortality. The month of admission was statistically significant in all periods except the last one (January/2022, OR = 1.14; 95%CI:0.98–1.32; February/2022, OR = 1.10; 95%CI: 0.94–1.28). Hospital size had a strong and statistically significant association with mortality in all periods (Table 2).

The final models for each period showed a C statistic equal to 0.75, 0.76, 0.73, 0.75, and 0.72 from the first to the last period, suggesting adequate discriminatory capacity. The ICCs of the null models indicated that 17–21% of the total variance was at the hospital level from the first to the fourth periods; that is, it was due to the hospitals and not to individual patients within the hospitals. In the last period, the third COVID-19 wave, the ICC was 11% (Table 2).

The number of hospitals underperforming (lower-than-expected performance) varied among periods, with 64 (18.3%) in the first period and only 23 (6.6%) in the last period. Crude mortality among the lower-than-expected performance hospitals varied from 19.0 to 73.7% in the first period and from 25.9 to 65.6% in the last period. The SMR showed high and moderate variation among hospitals with lower-than-expected performance in the first period/wave – 1.2 to 4.4 – and in the last period (third wave) – 1.3 to 2.8 – respectively (Table 3). There was greater variation in the percentage of hospitals with as-expected performance (39.5 to 76.1%) and those with lower-than-expected performance (6.6 to 32.3%). The hospitals with higher-than-expected performance remained at around 30% of the total, falling to 17% only in the fifth period (Table 3; Fig. 1).

Table 3 Classification of hospital performance, according to funnel plot limits. São Paulo state, Brazil, February 2020–2022

Fig. 1

Funnel plot of the hospital standardized mortality ratio from the first to the fifth period

SMR: standardized mortality ratio

Table 4 presents the overall transitions of 327 hospitals consistently involved in COVID-19 inpatient care in the whole pandemic period considered among the performance categories (higher-than-expected, as-expected, or lower-than-expected) from the first to the last COVID-19 pandemic period. Overall, looking at them, 99.7% (95 out of 96) of the higher-than-expected performance hospitals in the first wave performed as either as-expected or higher-than-expected in the third wave, in contrast to 94.1% (160 out of 170) and 80.3% (49 out of 61) of those initially classified as as-expected and lower-than-expected performers, respectively; about 93.0% (304 out of 327) of the hospitals performed as-expected or better-than-expected in the last period, against 81.3% (266 out of 327) in the first period.

Table 4 – transitions among performance categories, from the first to the last pandemic period, of hospitals consistently involved in COVID-19 inpatient care (N = 327) over time. São Paulo state, Brazil, February 2020–2022

In the sensitivity analysis, the regression coefficients at the patient level remained practically unchanged, with differences observed at the hospital level. The magnitude of the association with hospital size was smaller than that observed in the primary analysis (see Additional file 7). As expected, the variation in SMR in each period was smaller in the sensitivity analysis than in the primary analysis (see Additional file 8). However, the percentage of hospitals classified as lower-than-expected performers in both analyses was similar (see Additional file 8).

Discussion

In this study, we built a model for each period, given the substantive disease’s varying behavior throughout the pandemic, reflecting, among other factors, differences in infection rates, the pressure on the healthcare system, the knowledge to deal with the disease, and the vaccine availability [27, 28]. This approach sought to compare hospitals’ performances within each period, recognizing that relevant contextual conditions constituted specific scenarios. The rationale for the periods’ definition relied, therefore, on the pandemic dynamics, implying, naturally, in a vastly varying number of hospitals involved in COVID-19 inpatient care, admissions, and even duration. These variations are themselves expressions of differentiated levels of pressure on the healthcare system, an important factor affecting how it performs.

On the other hand, despite limitations resulting from the varying number of hospitals involved in COVID-19 inpatient care and admissions over time, the study also sought for comparisons within a subset of consistently involved hospitals about their performance level in reference to the average inpatient care performance across periods. Overall, this provided a trend view of the hospitals’ performances from the beginning to the last pandemic period considered, as treated below. Moreover, although we have illustrated only transitions from the first to the last period, the methodology allows for looking at transitions between consecutive periods.

The results of this study point to the high magnitude of the second wave of the pandemic (December/2020-June/2021) in São Paulo state, when occurred the highest number of admissions (34,641 in March 2021), as well as the highest mortality. Crude hospital mortality in the first pandemic months (February-April/2020) was 31.4%, and reached 33.3% in the second wave period. These mortality levels are comparable to those observed in the English pandemic onset [29], and lower than those observed in Mexico in 2020. The Mexican study, which included 71,189 discharges of patients diagnosed with COVID-19 in 2020, identified that hospital fatality was 38,5%, 40.7% in men and 35.0% in women [30].

The findings of this study also reinforce the declining hospital mortality trend from the second half of 2021, even during the third wave (December/2021-February/2022), as observed in a national analysis based on SIVEP-Gripe [13]. Additionally, the individual risk factors for in-hospital mortality, such as biological sex, age, and comorbidities, were consistent with those observed in a national and international study [9, 10].

At the hospital level, only hospital size (number of beds) remained in the final models. A positive association was observed between hospital mortality and size, with a smaller magnitude of association in the last period analyzed. An American study [12] analyzed 158,569 admissions due to COVID-19 in 181 hospitals and identified a higher mortality risk in hospitals with 100 to 349 beds (OR = 1.77; 95% CI:1.46–2.14) and 350 beds or more (OR = 1.85; 95% CI:1.19–2.89) against those with less than 100 beds. Possibly, size does not reflect the hospital’s experience in caring for COVID-19, depending on the proportion of beds allocated to care for the disease. A national study [13] found a similar association between hospital mortality and size in the bivariate analysis. However, it lost significance in the multivariate analysis, as the volume of admissions due to COVID-19 was included, with a statistically significant protective effect on hospital deaths.

Particularly, São Paulo had the first recorded case of COVID-19 in Brazil and experienced a rapid increase in the spread of cases and hospitalizations related to COVID-19. The state has 20% of the Brazilian population and the largest public hospital complex in Latin America [31]. São Paulo was one of the less vulnerable Brazilian states in relation to socioeconomic indices, to the proportion of the population with health risk factors, and to the number of SUS intensive care beds to respond to COVID-19³. São Paulo has also distinguished itself for its strong regional state capacity in regulating and organizing hospital supply and the use of the SUS, including an effective system for regulating and managing the distribution of COVID-19 patients among hospitals [32]. However, the public health system did not conduct so much RT-PCR testing and genomic surveillance, especially in areas with higher population density and dependence on the SUS [33].

The hospital mortality variation highlighted the importance of small hospitals (SH) in the state throughout the pandemic. Added to the debate about their efficiency [34] is that SH can provide a strategic response to health emergencies requiring proximity and opportunity in providing care. Thus, its high capillarity can be used for responses with technological, professional, financial, and informational support and greater integration into the network for the necessary care [35].

The ICC of the first and last periods’ null models was 21% and 11%, respectively. The high ICC values reinforce the importance of the multilevel approach and reveal that most of the total variance is at the level of SUS hospitals, which is not observed in international studies. Studies conducted with administrative data from the United Kingdom’s National Health Service in the first year of the pandemic, when little was known about COVID-19 patient care, indicated an ICC between 1.4% and 2.9%^10,14,29. In Japan, a study with adult patients discharged between January/2020 and February/2021 observed that 7.9% of the total variance in mortality was attributable to variance between hospitals [36]. Only Illinois (USA) reported an ICC similar to the last period of this study (0.11; 95% CI: 0.08–0.14) [12]. The sensitivity analysis identified the reductions of ICC when excluding hospitals that did not record deaths in the period.

Considering the variations in results between the first wave (first period) and the third wave (last period) for those with the worst performance (lower-than-expected performance), in terms of the ranges of crude mortality (19.0-73.7% vs. 25.9–65.6%) and SMR (1.2–4.4 vs. 1.3–2.8), as well as the percentage of hospitals classified in this performance subgroup (18.3% vs. 6.6%), we underline the reduction of their variation and the suggestion that healthcare quality improved throughout the pandemic, probably reflecting changes in care protocols, emergency allocation of financial resources, or expanded vaccination coverage.

Bottle et al. [29] also observed significant variation in gross hospital mortality among English hospitals in the first year of the pandemic. The analyses showed that the crude mortality rate ranged from 14.4 to 42.7% in the first wave (March-July/2020) and from 9.3 to 34.8% in the second wave (August-March/2021). However, the SMR ranged from 0.6 to 1.35 and 0.5 to 1.35 between hospitals in the first and second wave, respectively, suggesting modest variation in mortality between hospitals after risk adjustment, different from the first period of this study. These authors [29] identified that the second wave had more outliers than the first, with 11 hospitals (9.0%) presenting SMR above control limits of 99.8% compared to 3 (2.5%) in the first wave. Notably, the authors analyzed a shorter period, with a possible lower effect of learning on COVID-19 patient care than in this study, besides applying control thresholds of three standard deviations and covering a period before vaccination, which started in England in December/2020.

This study has strengths and limitations. We cannot ignore that the approach adopted, with a large variation in the number of observations between periods, could lead to different statistical power to detect outliers. Thus, the larger the number of observations, the greater the power to discriminate hospitals with higher-than-expected and lower-than-expected performance.

On the other hand, in-hospital mortality predictions based on risk-adjusted models built for each period are likely to allow for more precise estimates in the specific contexts observed in the pandemic periods despite the difficulties of directly comparing a hospital’s performance individually over time. Performance is not treated as an element intrinsic to the hospital but is context-dependent, considering, among other aspects, the different composition of COVID-19 hospitals’ networks over time.

Despite the possibility of classification error, funnel plots have been recommended as a useful tool for communicating outcome comparisons [37, 38]. In mortality studies, funnel plots are useful for deciding, among other information, which hospitals merit further investigation. In this study, we mainly sought to identify large variations over time that could suggest improvement or worsening in hospital performance throughout the pandemic. We should highlight that the mean pattern of results changed positively during the pandemic. Overall, there was a reduction in the outcomes’ variation over time, and the analyses on the hospitals consistently involved in COVID-19 care indicated an increase in the proportion adjusted or positively extrapolating expectations.

An additional limitation of this study is that SIH data quality can affect the model’s adjustment. So, hospital variations can also be due to case severity or the information source’s completeness and scope. Although this is an old issue with Brazilian health information systems, São Paulo state SIH records’ completeness is better than the rest of the country, allowing a better adjustment for the patient’s risk factors despite keeping some problems. It is recognized that the poor registration of comorbidities is very likely to affect the procedure, incurring confounding with structural and healthcare process variables.

It was also impossible to adjust for the origin/source of admission (home or other hospital) as this was not available in the dataset. However, it is expected that in the case of COVID-19 hospitalizations, the main sources would be emergency rooms and, eventually, primary healthcare units through a regulatory system.

The municipal Brazilian Deprivation Index is also limited because it cannot be used at the census sector level from nationally publicly accessible databases. Furthermore, it was constructed with data from the 2010 Census. Notably, the option to include only general hospitals active in January 2020 deliberately excluded field hospitals implemented in the period.

In addition, comparability between studies is certainly difficult in the face of contextual barriers, availability and quality of information, and analysis plan design, especially in Latin America and the Caribbean, where studies in the field of health services research are scarce.

Conclusion

In short, measured by SMR, hospital performance improved throughout the period studied among hospitals reimbursed by the SUS in the São Paulo state (Brazil). However, the results may include residual confusion due to measuring the complex profile of treated cases, differences in applying treatment protocols over time, and a reflection of vaccination coverage that reduced the disease’s severity.

It is important to highlight the selection of São Paulo state given the availability of better data. The state’s economic conditions, resources, and SUS functioning display a different picture than those of others in the country. Data quality improvement is fundamental for amplifying the application of SMR in Brazil. At the same time, using similar approaches in other geographic areas to assist health system managers would be informative about the need for financial resources and improvements in the structure, appropriateness, and effectiveness of hospital care.

In the scenario of post COVID-19 epidemic, it is worthy note, additionally, the fact that this work provides a methodological approach adjusted to the Brazilian context, which may be applicable to follow-up hospital’s performance in caring general or specific-cause hospitalizations, in regular or exceptional emergency situations.