Setting

For the purpose of this model analysis the DHD group gave its formal permission to use an anonymised LMR dataset from 89 Dutch hospitals covering the years 2005 to 2009. Nineteen hospitals were excluded from the calculation because the quality of their patient registration data was insufficient.

This study excluded any experimental research, either on human or animal subjects.

The HSMR model used in this study

We performed HSMR calculations using two models, shown as model 1 and model 2.

For model 1, we used the Dr Foster model [1], applied in the Netherlands in 2010. This Dutch model included clinical admissions only, so no day cases. Furthermore only in-hospital deaths were counted. There are differences between the Dutch and the UK model. The Dutch model used 50 Clinical Classifications Software (CCS) groups based on an ICD-9 coding, whereas the UK used 56 CCS groups based on ICD-10 coding, of which 42 groups were the same as in the Dutch model. Furthermore, the UK model adjusted for palliative care and for the number of previous emergency admissions within one year. The Dutch model did not adjust for either of these.

HSMR in brief

The HSMR ('hospital standardised mortality ratio')is the ratio between the observed number of in-hospital deaths and the predicted number of deaths, determined by comparing the patient casemix with the national average. The outcome is standardised around 100. A value above 100 indicates higher mortality than average, below 100, lower mortality. If the calculation is applied to one of the 50 diagnostic groups, then we speak of the SMR ('standardised mortality ratio').

The Dutch HSMR model of 2010 adjusts for the following casemix properties: year of discharge, sex, age at admission, admission type, comorbidity (charlsonindex), social deprivation, month of admission, source of referral, diagnostic group, and, in part, for the casemix on the primary diagnostic level.

Each diagnostic group is composed of a number of underlying diseases: ICD-9 codes (International Classification of Diseases Ninth edition) as determined by the Clinical Classification Software (CCS). This tool clusters various ICD-9 diagnoses into a manageable number of meaningful categories [12], indicated as 'CCS diagnostic groups'. The selected CCS diagnostic groups have a relatively high mortality and together cover over 80% of the total number of hospital deaths.

For model 2, we took model 1 and added an adjustment for the frequency of readmission, as described by Van den Bosch WF, et al [10]. The authors demonstrated that frequently admitting patients was associated with lower mortality ratios per admission, for which an additional correction would be needed. In their publication the patients were grouped into 'patient view' classes P(m). We have also applied this to our study. Here the admission frequency m was equal to the number of times a patient was admitted to the same hospital during the five-year study period.

For example, a patient who was admitted ten times during the five-year period to hospital X, to be treated for one or more diseases, possibly with different diagnoses, was part of the patient view class P(10) and not part of any other patient view class. This patient contributed ten admissions to P(10) which were all accounted for in the regression calculation.

We were able to retrieve the number of times that a patient was admitted through the unique patient identification number that all of the 70 hospitals included were using. We have grouped the patient view classes into eight categories: m = 1, m = 2, m = 3, m = 4, m = 5-6, m = 7-9, m = 10-20 and m > 20. This is in order to limit the number of categories for the regression calculation and to avoid categories becoming too small.

Agreement, or otherwise of the two models

Quality metrics of the two models

Sensitivity of the HSMR model to adjustment for readmission

We have investigated to what extent the model was sensitive to variations in the length of the period under review. We took three scenarios: a period of review of one year (2009), two years (2008-2009) and five years (2005-2009). We then examined the statistical distance between model 1 and model 2, and the three model quality metrics on the HSMR level.

Agreement between the two models

Figure 1 shows the frequency distribution of the relative changes of the 3500 SMRs, in going from model 1 to model 2. Similarly the changes of the 70 HSMRs of the hospitals are shown in Figure 2. The SMR changes ranged from -39% to +110% and the HSMR changes ranged from -12% to +11%. The standard deviation of the frequency distribution of the changes amounted to 12.2% (roughly 12 SMR points) for the SMRs and to 4.1% (roughly 4 HSMR points) of the HSMRs.

Finally, we calculated per SMR for how many hospitals a significantly high SMR score changed to not significantly high from model 1 to model 2 and vice versa; see Table 1. Model 1 indicated 328 SMRs in total as "higher than expected" of which 64 were not indicated by model 2 (20%). On the other hand, model 2 indicated 313 SMRs in total as "higher than expected" of which 49 were not indicated by model 1 (16%). Across hospitals, model 1 indicated that 23 hospitals recorded a higher than expected HSMR of which four were not indicated by model 2 (17%). And, alternatively, model 2 indicated that 23 hospitals were having a higher than expected HSMR of which four were not indicated by model 1 (17%).

H/SMR model quality metrics

For both models the 'c-statistic' per CCS diagnostic group was calculated, see Table 1. Model 2 scored a higher c-statistic compared to model 1 in 47 groups out of 50. For eighteen of these groups the scores of model 2 exceeded the scores of model 1 by between 0.03 and 0.08. Model 1 had one diagnostic group for which the c-statistic was at least 0.03 better compared to model 2. The c-statistic of the HMSR across all hospitals amounted to 0.852 for model 1 and to 0.867 for model 2 (favourable).

Finally the overall explanatory power expressed in pseudo R² statistics were calculated. These amounted to 0.259 for model 1 versus 0.282 for model 2 and so favourable to model 2.

Sensitivity of the model to the review period of adjustment

The overall agreement between models 1 and 2 varied, depending on the review period, from 0.92 (one year) to 0.89 (two years) to 0.86 (five years). So a longer review period corresponded with an increasing divergence between the two models.

How should we interpret the differences in model outcomes?

In this study we have calculated and compared the H/SMR outcomes of two models. The disagreement between the two models was substantial, according to the relative changes in HSMRs and SMRs.

The standard deviation of the frequency distribution of HSMR-change amounted to 4 HSMR points, which was substantial compared to the standard deviation of the HSMR-frequency distribution amounting to 14 HSMR points. Moreover, these shifts introduced an uncertainty into the HSMR outcomes that exceded the uncertainty due to chance as indicated by the 95% confidence intervals (CIs): The 95% CI of a HSMR equalled [-2σ, 2σ]. On average for the 70 hospitals over five years, the standard deviation σ of the CI distribution equalled 2.5 HSMR points.

With respect to the SMRs: the statistical 'distance' between SMRs of model 1 and model 2 expressed in R² coefficients, varied from 0.60 to 0.95. Low scores indicated susceptibility to adjustment for readmissions. On average chronic diseases scored lower compared to acute diseases as to be expected since chronic diseases in particular often result in readmissions. There was no particular peak among any of the main diagnostic groups. The major diagnostic groups like cancer and heart and lung diseases all showed substantial disagreement between model 1 and model 2 due to adjustment for readmission. The results also showed disagreement between the models in designating hospitals as having higher than expected H/SMRs. The percentages of disagreement varied from 16% to 20%.

Bottom-line: all differences in H/SMR outcomes that we found between the two models, cannot be attributed to differences in the quality of care neither to 'chance'. They have to be attributed to the difference in the type of model, because this is the only distinction applied in producing the results from the same set of admissions. This implies that publicised good and bad practices, linked to hospital rankings based on HSMRs, can differ substantially depending on the choice of model.

How do readmissions affect H/SMRs and how do they occur?

We asked: how did readmissions impact the H/SMR, how did readmissions occur and why did hospitals differ in this respect?

For example: endlessly readmitting the same patient in the same hospital continuously increased the denominator (predicted death) of the calculated HSMR for that hospital. We have observed numerous cases where the contribution to the denominator on behalf of a single patient over the years aggregated to numerical values of 3 to 4. However a patient could only die once and so any patient could maximally contribute a numerical value of 1 to the numerator. Consequently this effect lowered the HSMR ratio and favoured hospitals which had many frequently readmitted patients.

There are many types of mechanisms that explain the variation in the frequency of clinical readmission. These could have been administrative in nature, for example differences in administrating chemotherapy [10]. There were also many systematic differences in treatment practice. For example one hospital applied and recorded a different number of clinical admissions than another hospital did for the same combination of diagnosis and treatment applied to the same patient. Differences also occurred as a consequence of transferring patients back and forth. Poor care during the initial admission may also have triggered readmissions later. As a consequence all these examples of increased readmissions were being 'rewarded' by model 1 while model 2 was correcting for this phenomenon.

The analysis showed that the combined effect of all these mechanisms was not restricted to some of the diagnostic groups, but had an impact upon most of them. Clearly the HSMR model was susceptible to an adjustment for the frequency of readmissions in the way we have defined it.

The analysis also showed that reducing the review period from five to two years and from two to one year, resulted in a substantial reduction of this susceptibility. This phenomenon may be explained as follows: It may take several years before the readmissions of a patient constitute a substantial sequence of admissions. Furthermore a sequence might also have started long before the analysis period started. In these cases apparently one year was not a sufficiently long time period to capture sequences with a substantial amount of admissions. By looking at various years these sequences became better visible.

Choice of model: consequences for usage of H/SMR as indicator

We considered which of the two models would be more favourable to use and why. Since the model characteristics with regard to discrimination, calibration and explanatory power all were in favour of model 2, we recommend using model 2, although this preference does not necessarily invalidate model 1. HSMR models in other countries may need a similar additional adjustment. The impact of readmissions on the current HSMR in the UK is not clear. The UK model adjusts for readmissions for emergency cases. However for non-emergency cases the model does not adjust for readmissions. Moreover, admissions with different primary diagnoses are not counted as readmissions as well. Furthermore the UK model is restricted to a maximum review period of one year, which is too short for the readmission effect to become visible as demonstrated in our study.

A more fundamental issue than the choice of model however, is the fact that two comparable models, such as model 1 and model 2, may deliver such divergent outcomes. Since there is no 'gold standard' one cannot state that the one is false and the other one is true. This finding reveals an uncertainty in HSMR outcomes that exceeds the amount of uncertainty introduced by chance expressed in 95% confidence intervals over the five-year period.

HSMRs and SMRs are increasingly used by physicians to improve the quality of care in their hospitals. In particular diagnostic groups with higher than expected mortality are the subjects of investigation. However, in a lot of cases, the designated significance of these groups may depend on the regression model used, not only on the confidence intervals. The same is applicable for the current HSMR scores in the Netherlands. Physicians and hospital managers should be aware of this phenomenon.

Limitations of the study

A limitation of this study was the fact that we had to exclude 19 hospitals and so the HMSR calculation was based on roughly 80% of the Dutch hospitals. Nevertheless, we think the results of our study were representative of the Netherlands and have clearly demonstrated the impact upon the HSMR of not adjusting for frequency of readmissions.

Another limitation was the fact that the Dutch HSMR only accounts for in-hospital mortality and not for 30-day mortality, which may favour hospitals with shorter lengths of stay [14]. However since the effects, demonstrated in our study, particularly concerned the phenomenon of the frequently returning patient, and so no 30-days deaths, we think this potential bias does not significantly change our conclusions.