Skip to main content
  • Research article
  • Open access
  • Published:

Predictive performance of comorbidity measures in administrative databases for diabetes cohorts



The performance of comorbidity measures for predicting mortality in chronic disease populations and using ICD-9 diagnosis codes in administrative health data has been investigated in several studies, but less is known about predictive performance with ICD-10 data and for other health outcomes. This study investigated predictive performance of five comorbidity measures for population-based diabetes cohorts in administrative data. The objectives were to evaluate performance for: (a) disease-specific and general health outcomes, (b) data based on the ICD-9 and ICD-10 diagnoses, and (c) different age groups.


Performance was investigated for heart attack, stroke, amputation, renal disease, hospitalization, and death in all-age and age-specific cohorts. Hospital records, physician billing claims, and prescription drug records from one Canadian province were used to identify diabetes cohorts and measure comorbidity. The data were analysed using multiple logistic regression models and summarized using measures of discrimination, accuracy, and fit.


In Cohort 1 (n = 29,058), for which only ICD-9 diagnoses were recorded in administrative data, the Elixhauser index showed good or excellent prediction for amputation, renal disease, and death and performed better than the Charlson index. Number of diagnoses was a good predictor of hospitalization. Similar results were obtained for Cohort 2 (n = 41,925), in which both ICD-9 and ICD-10 diagnoses were recorded in administrative data, although predictive performance was sometimes higher. For age-specific models of mortality, the Elixhauser index resulted in the largest improvement in predictive performance in all but the youngest age group.


Cohort age and the health outcome under investigation, but not the diagnosis coding system, may influence the predictive performance of comorbidity measure for studies about diabetes populations using administrative health data.

Peer Review reports


Administrative health data are frequently used for surveillance and research in chronic disease populations. These data contain medical records generated for management and remuneration purposes at the time of hospital discharge or provision of services [1]. Besides providing timely and cost-effective information, their popularity stems from the fact that they are population-based and capture both utilization and diagnostic information. However, to obtain unbiased conclusions from observational chronic disease studies using administrative data requires control of confounding factors that may differ among populations and are associated with the health utilization or outcome measure under investigation. Demographic and socioeconomic variables are included as risk-adjustment measures in most observational studies. Comorbid conditions, pre-existing conditions that co-occur with the index disease, [2] are also commonly considered.

A number of comorbidity measures are available for administrative health data. These include both general-purpose and disease-specific comorbidity measures; [3, 4] general-purpose measures are advantageous because they can be used to compare comorbidity characteristics across different chronic disease populations. Some general measures are based on simple counts of the number of diagnoses or prescription drugs for an individual [5]. Others are based on specific sets of diagnosis codes or prescription drug codes. The Chronic Disease Score (CDS), for example, is based on a set of codes for prescription drugs used to treat major chronic conditions [6]. Diagnosis-related measures, such as the Charlson and Elixhauser indices, use International Classification of Disease (ICD) diagnosis codes to identify major comorbid conditions [7, 8]. Both the Charlson and Elixhauser indices were originally used to predict mortality for in-hospital populations, but they have also been applied to outpatient populations and to some other health outcomes [911]. The Elixhauser index was developed using the clinical modification of the 9th revision of ICD (i.e., ICD-9-CM); the Charlson index was also proposed using this classification system. However, many countries, including Canada, Australia, New Zealand, Japan, China, and some European countries have now implemented the 10th revision of ICD (i.e., ICD-10), which covers a broader range of clinical information. Quan et al. [12] extended the Charlson and Elixhauser indices to ICD-10 codes, but only a few studies have compared the predictive performance of comorbidity algorithms based on ICD-9 and ICD-10 codes. Li et al. [13] observed good predictive performance for in-hospital mortality using both the Charlson and Elixhauser indices with the two coding systems in Canadian data. Sundararajan et al. [14] found similar results using Australian data and when the outcome was in-hospital mortality. However, the authors noted that predictive performance for other outcomes could be investigated.

Several studies have used administrative data to study health outcomes and healthcare use in diabetes populations. Diabetes places a significant burden on the health care system [1518], and therefore is of great interest to clinicians and policy analysts. It is responsible for vascular and neurologic complications such as acute myocardial infarction (AMI), stroke, lower extremity amputation (LEA), end stage renal disease (ESRD), and retinopathy [1520]. De Berardis et al. found that the hospitalization rate in diabetics is twice that of the general population, accounting for an excess of 12,000 hospital admissions per 100,000 person years [15]. Studies that have investigated comorbidity measures in diabetes populations using administrative data have been limited, although Quail et al. [21] found that the predictive performance of different comorbidity measures for mortality and hospitalization outcomes was variable in study cohorts with diabetes.

Only a few studies have compared the predictive performance of comorbidity measures in different age groups, although these groups may differ in their comorbidity characteristics. Studies that have investigated risk-adjustment tools have often focused on older populations [3, 22, 23]. In contrast, Quail et al. compared an age-inclusive cohort (i.e., 20+ years) to an age-restricted cohort (i.e., 65+ years) and found diminished performance of comorbidity measures in predicting mortality and hospitalization in the latter [21].

Given this background, the study purpose was to investigate predictive performance of comorbidity measures in diabetes cohorts defined from administrative health data. The objectives were to compare performance for: (a) disease-specific and general health outcomes, (b) data based on ICD-9 and ICD-10 diagnoses, and (c) different age groups.


Data sources

Study data were from the Canadian province of Saskatchewan, which has a population of approximately 1.1 million according to the 2006 national census. About 40 per cent of residents live in one of two major urban centres, while the remainder of residents live in rural communities [24].

Like other Canadian provinces and territories, Saskatchewan has a universal health care system. All hospital records and virtually all physician billing claims and outpatient prescription drug records are captured for residents eligible to receive health insurance benefits. Records of hospital, physician, and prescription drug services are collected in electronic databases that can be anonymously linked, via a unique personal health number, to the population health insurance registration file [25]. The registration file captures dates of health insurance coverage, demographic characteristics and location of residence.

A hospital record is completed upon patient discharge. Diagnoses are recorded using ICD-9 up to and including the 2000/01 fiscal year (a fiscal year extends from April 1 to March 31). Beginning in 2001/02, ICD-10-CA diagnoses were used. Between three and 16 diagnoses are captured in the data prior to the introduction of ICD-10-CA and up to 25 diagnoses are captured subsequently.

Physicians paid on a fee-for-service basis submit billing claims to the ministry of health for payment purposes. A single diagnosis is recorded on each claim using three-digit ICD-9 codes.

Prescription drug files contain records of outpatient drugs dispensed to residents eligible for coverage. Registered Indians, who represent about 9 per cent of the population, have their prescription drug benefits paid by the federal government rather than the province so their records are not available in the provincial database. Prescription drug records include the date of dispensation and national drug identification number (DIN). DINs are linked to codes in the American Hospital Formulary System (AHFS) Pharmacologic-Therapeutic Classification System ( The AHFS is used to group drugs with similar pharmacologic, therapeutic, and/or chemical characteristics using a hierarchical system with four levels.

The accuracy and completeness of Saskatchewan’s administrative data for research has been documented in multiple studies [2628]. Ethics approval for this research was received from the University of Saskatchewan Biomedical Research Ethics Board. Data were accessed and analysed at the Health Quality Council in accordance with a standing data sharing agreement between that organization and the provincial health ministry.

Study cohorts

To permit comparisons of ICD coding systems, two cohorts were defined using the diabetes case definition developed by the National Diabetes Surveillance System, [29] which has been validated in previous research [30, 31]. This case definition identifies all individuals with a diabetes diagnosis (ICD-9 250; ICD-10-CA E10-E14) in at least one hospital record or two physician claims within a two-year period. The diagnosis index date is the date of hospitalization or the date of the last of the two physician visits.

Cohort 1 was composed of residents aged 20 years and over at their diagnosis index date who satisfied the diabetes case definition and had uninterrupted health coverage from 1997/98 to 1999/00 or until death. The ICD-9 coding system was exclusively used during this period to record diagnoses in both hospital and physician databases. Cohort 2 was composed of residents aged 20 years and over at their diabetes index date who satisfied the diabetes case definition and had uninterrupted health coverage from 2001/02 to 2003/04 or until death. ICD-9 (for physician data) and ICD-10 (for hospital data) coding systems were in use during this latter period. Data from fiscal year 1996/97 onward, the first year of available data, were used to ascertain diabetes cases.

Comorbidity measures

Five measures were investigated (Table 1): number of different diagnoses, Charlson index, Elixhauser index, number of different drugs, and CDS. For Cohort 1, data from 1997/98 were used to define each measure while for Cohort 2, data were from 2001/02.

Table 1 Description of comorbidity measures

The number of different diagnoses in hospital and physician data [32] was based on codes recorded to the third digit in both ICD-9 (e.g., 812) and ICD-10 (e.g., S42). Diagnoses related to pregnancy, childbirth, or abortions were excluded. The Charlson index was originally developed using data abstracted from hospital charts, [7] subsequently adapted to hospital data coded with ICD-9, and then extended to data coded with ICD-10 [12]. The index is based on diagnoses for 17 conditions; each condition is assigned a weight from one to six. A summary score is computed ranging from 0 to 32, where a higher score indicates greater comorbidity. In accordance with previous research, [33] the Charlson index was computed using diagnosis codes in both hospital and physician data. For hospital data, only those conditions present on admission were included. The Elixhauser index [8] has also been extended from its original formulation using ICD-9 to ICD-10 [12]. Each of the 31 conditions comprising the index is coded as a binary indicator. This index was also implemented using diagnosis codes from both hospital and physician data. The number of different prescription drugs has also been shown to be a valid measure of comorbidity [11]. This measure was based on six-digit AHFS codes, of which there are 125. The CDS [6] was originally developed for predicting mortality and hospitalization outcomes. It uses the first four digits of AHFS codes to identify prescription drugs for treating 17 conditions. Each disease treatment group is assigned a weight and the weighted category scores are used to produce a single summary score from 0 to 35, where a high score indicates greater comorbidity.

Outcome variables

Study outcomes were identified using data from the two years following the comorbidity measurement year of each cohort. Thus, outcomes were measured for 1998/99 to 1999/00 for Cohort 1 and 2002/03 to 2003/04 for Cohort 2. Outcomes of AMI, stroke, LEA, and ESRD were based on previously-defined algorithms [3436]. Specifically, AMI cases were identified with an ICD-9 code of 410 or ICD-10 code of I21 in the most responsible (i.e., primary) diagnosis field in hospital records. AMI hospitalizations that occurred within a one-year period following a previous AMI were excluded, to ensure that only incident cases were captured. Stroke cases were identified using ICD-9 codes 430, 431, 434, 436, and 362.3 and ICD-10 codes I60, I61, I63, I64, and H34.1 in the most responsible diagnosis field in hospital records. The LEA case definition was based on procedure codes in hospital records. It captured both minor (toes, forefoot, foot below ankle), and major (ankle, below knee, above knee) amputation procedures not related to trauma or malignancy. Individuals with ESRD were identified from service codes for chronic dialysis and renal transplantation. The full list of procedure and service codes for LEA and ESRD is available from the authors.

Other outcomes that were investigated included death, hospitalization for any reason, and hospitalization for diabetes (ICD-9 250; ICD-10-CA E10-E14). Deaths were identified from the population registry. Hospitalizations associated with pregnancy, childbirth, or abortions were excluded. Transfers between facilities and hospital re-admissions within 24 hours of discharge were considered part of the initial hospital admission [35].

Other study variables

The cohorts were described on age, sex, recent diabetes diagnosis, region of residence, and income quintile. Age and sex were defined from the population registration file. Individuals who had a diagnosis index date in 1997/98 or 2001/02 for Cohorts 1 and 2, respectively, were identified as recently diagnosed. Individuals with a prior index date were defined as previously diagnosed. Urban and rural region of residence was defined based on postal code in the registration file; urban residents were those living in the health regions of Saskatoon and Regina.

Income quintile was defined using a method based on average household income from the 2001 Statistics Canada Census [37]. Each individual’s postal code was assigned to a dissemination area (DA), the smallest geographic unit for which Census data are reported. Income ranges were determined such that the entire Saskatchewan population was divided into five approximately equal groups. Residents were assigned an income quintile according to their DA average household income. Some residents could not be assigned a quintile because income measures are suppressed for some DAs, usually because of small population size. Approximately 14 per cent of the total population had a missing income quintile. A method was used that first developed a predictive model for the missing quintiles based on socio-demographic variables that are generally not suppressed, including marital status, ethnicity, and unemployment. A multiple imputation approach was then used to assign income quintile, taking the average of the multiple imputed values, [38] for all DAs that did not have missing information on one or more of these socio-demographic variables and for all individuals who did not have a missing postal code. Average total income, reported by Statistics Canada in 2001, was $12,700 for the lowest quintile, $29,700 for the second quintile, $47,200 for the third quintile, $71,800 for the fourth quintile, and $128,700 for the highest quintile (

Statistical analysis

Frequencies, means, and standard deviations were used to describe the cohorts’ characteristics. Performance of each comorbidity measure was assessed using multiple logistic regression by fitting base and full models to the data [32]. For each cohort, the base model contained the following variables: age (centred on the mean), a quadratic age term, sex, region of residence (urban [reference], rural), income quintile (Q1/Q2, Q3 [reference], Q4/Q5), and recent diabetes diagnosis (prior [reference], recent). The full model contained all variables in the base model in addition to a comorbidity variable(s), which were modelled as categorical variables. The variable categorization adopted for each comorbidity measure is provided in Table 1.

The base and full models were compared using the c-statistic, a measure of discrimination that is equivalent to the area under the receiver operating characteristic curve for dichotomous outcomes [39, 40]. The c-statistic ranges from zero to one, with a value of one representing perfect prediction and a value of 0.5 representing chance prediction. A value between 0.7 and 0.8 is considered to demonstrate acceptable predictive performance, while a value greater than 0.8 demonstrates excellent predictive performance. The 95% confidence intervals (CIs) were computed. The difference in c-statistics for the base and full models (i.e., Δc) was tested for statistical significance using the method of DeLong et al. [41]. The percentage change in the c-statistic between base and full models was also computed.

To investigate model fit, a likelihood ratio test (LRT) was also conducted for the base and full models [42]. The LRT statistic asymptotically follows a χ2 distribution, with the degrees of freedom (df) for this statistic equal to the difference in df for the base and full models. A statistically significant LRT statistic indicates that the inclusion of the comorbidity measure results in an improvement in model fit. Each test was conducted at the α = .01 significance level, to reduce the overall probability of a Type I error.

The Brier score, which combines information about model calibration (i.e., accuracy) and discrimination, was also computed. The Brier score ranges from zero to one, [43] with lower values indicating less prediction error. Given that a score of 0.25 can be achieved by assigning an event probability of 0.5 to each individual, [43] a value less than 0.25 was considered to represent acceptable prediction error.

All analyses were conducted using SAS software [44]. Separate analyses were conducted for Cohorts 1 and 2 and for age-specific groups within each cohort. The age-specific groups were: 20 to 44 years, 45 to 64 years, 65 to 74 years and 75+ years.

Results and discussion

Cohort 1 consisted of a total of 29,058 individuals and Cohort 2 consisted of 41,925 individuals. In total, 1,106 (3.7%) individuals were excluded from Cohort 1 because they did not have health insurance coverage throughout the study observation period; this percentage was similar for Cohort 2 (3.9%). Table 2 describes the age-specific demographic, health outcome, and comorbidity characteristics for both cohorts. The youngest age group was more likely to be comprised of urban residents while the oldest age group was more likely to contain rural residents. In Cohort 1, close to one-third of individuals in the youngest age group were in the lowest income quintile, compared to 27.1% of individuals in the 75+ age group. Similar results were observed for Cohort 2. Overall, individuals in Cohort 1 were more likely to have a recent diabetes diagnosis compared to Cohort 2. The overall percentage of individuals experiencing each health outcome was higher in Cohort 1 than Cohort 2 with the exception of ESRD. Cohort 1 had lower mean scores for the number of diagnoses, number of drugs and CDS, but not the Charlson index score, for which both cohorts had the same mean score. As expected, the average scores for the comorbidity measures increased with age.

Table 2 Description of diabetes cohorts

Tables 3 and 4 describe the comorbidities comprising the Charlson and Elixhauser indices, respectively. For the Charlson index, the most common comorbidities in both cohorts were uncomplicated diabetes and chronic pulmonary disease. For the Elixhauser index, the most common comorbidities, in addition to uncomplicated diabetes, were uncomplicated hypertension and chronic pulmonary disease. More than 60% of individuals in both Cohorts 1 and 2 had at least one of the Elixhauser comorbidities.

Table 3 Charlson index comorbidities (%) for study cohorts
Table 4 Elixhauser index comorbidities (%) for study cohorts

Table 5 reports the modelling results for both cohorts when age-inclusive analyses were conducted. The LRTs were statistically significant for all comorbidity measures and for all outcomes, except for the CDS for AMI and the Elixhauser index for stroke in Cohort 2. These results indicate that the comorbidity measures almost always resulted in an improvement in model fit. Therefore, the focus of the remainder of this section is on the c-statistics and Brier scores.

Table 5 Model comparisons for health outcomes in all-age diabetes cohorts

For AMI, the base models had c-statistics of 0.66 (95% CI: 0.64, 0.68) and 0.68 (95% CI: 0.66, 0.70) in Cohorts 1 and 2, respectively and both had a Brier score of 0.02, indicating poor predictive performance and low error. The addition of a comorbidity measure was associated with, at most, a 2.95% increase in the c-statistic. None of the full models had c-statistics that exceeded 0.70.

The base model for stroke in Cohort 1 had a c-statistic of 0.70 (95% CI: 0.68, 0.72) and a Brier score of 0.02, indicating good discrimination and low error. The improvement in the c-statistic was only statistically significant for the Elixhauser index (2.56%). The c-statistic for the base model in Cohort 2 was similar but the percentage change in this measure was not statistically significant for any of the full models.

For LEA, the c-statistic for the base model in Cohort 1 was below 0.70; each comorbidity measure resulted in a statistically significant increase in the c-statistic. The largest improvement was for the Elixhauser index (20.71%), followed by the Charlson index (14.06%). Both indices had low Brier scores (0.01). Similar results were found for Cohort 2, although the c-statistics were higher for the full models, and the change in the c-statistics was, overall, smaller than for Cohort 1.

Similar findings were observed for ESRD. In Cohort 1, there was excellent discrimination for the full model containing the Elixhauser index (c = 0.84; 95% CI: 0.80, 0.89). In Cohort 2, both the Elixhauser index (c = 0.87; 95% CI: 0.84, 0.90) and Charlson index (c = 0.82; 95% CI: 0.79, 0.86) resulted in full models with excellent predictive performance.

The base models for the two hospitalization outcomes had lower c-statistics and higher Brier scores than the base models for disease-specific outcomes. While all of the comorbidity measures resulted in statistically significant improvements in the c-statistic, none of the full models had values greater than 0.70. For hospitalization for any reason, the largest improvement was observed for the number of different diagnoses. For diabetes hospitalization, the largest improvement was observed for the Elixhauser index, but it was similar to the value for the number of diagnoses.

For death, the c-statistic of the base model for Cohort 1 was 0.77 (95% CI: 0.76, 0.78) and the Brier score was 0.08, indicating good discrimination and low prediction error. Results were similar for Cohort 2. All comorbidity measures were associated with statistically significant increases in the c-statistic. In Cohort 1, the largest increase was for the Elixhauser index (c = 0.83; 95% CI: 0.82, 0.83) followed by the Charlson index (c = 0.82; 95% CI: 0.81, 0.82). Similar results were found for Cohort 2, although the percentage change in the c-statistics were smaller than for Cohort 1.

The age-specific results are reported in Table 6. We conducted the analyses for death only, to limit the number of model comparisons and also because for some outcomes, age-specific models could not be fit to the data given the low numbers of health events. LRT statistics for all models were statistically significant, except for the model for number of drugs in the 20 to 44 age group in Cohort 1.

Table 6 Model comparisons for death in age-specific diabetes cohorts

For each age group, the base model c-statistic was consistently below 0.70. Brier scores were smallest for the youngest age group and largest for the oldest age group. For Cohort 1 in the youngest age group, only the Charlson index resulted in a significant increase in the c-statistic. In Cohort 2 in the youngest age group, the Charlson index, number of different diagnoses, Elixhauser index and number of different prescription drugs resulted in statistically significant increases in the c-statistic. The results for the other age groups were similar in the two cohorts. The addition of each comorbidity measure to the base model was associated with a statistically significant increase in the c-statistic. The Elixhauser index consistently resulted in the largest increase in the c-statistic, followed by the Charlson index.


This study of comorbidity measures in population-based cohorts with diagnosed diabetes had the following key findings. First, there were substantial differences in the predictive performance of the base set of risk-adjustment variables selected for this study. Performance was lowest for hospitalization measures and highest for death and stroke. Improvements in model fit were often observed when a comorbidity measure was included in the model. However, for the health outcomes of AMI and stroke, there was limited utility associated with the inclusion of a comorbidity measure in the risk-adjustment model, based on model discrimination (i.e., c- statistic). For the other health outcomes, there was always a statistically significant improvement in the c-statistic for the full models. ESRD and death were the outcomes for which the comorbidity measures resulted in the greatest improvement in predictive performance. The model containing the Elixhauser index had the best predictive performance for all outcomes except for hospitalization for any reason, where number of diagnoses performed well. However, this was not always the case when age-specific cohorts were investigated. Similar changes in the c-statistics were observed for the diagnosis-based comorbidity measures regardless of whether the measures were based on ICD-9 codes only, or both ICD-9 and ICD-10 codes. The comorbidity measures based on prescription drugs had similar changes in the c-statistic values in the two cohorts to those observed using the diagnosis-based measures. Overall, however, comorbidity measures based on diagnosis codes performed better than comorbidity measures based on prescription drug codes.

The findings that the Charlson and Elixhauser indices performed well for predicting general measures of hospital utilization and mortality concurs with previous research [10, 45, 46]. However, this research has also shown that predictive performance of these comorbidity measures tends to be lower for healthcare utilization than for mortality, but still greater than when the predictive model is limited to socio-demographic variables and recency of diagnosis. Farley et al. [45] found that for predicting healthcare expenditures in the general population, simple count measures, such as counts of the number of diagnosis clusters, performed better than the Charlson and Elixhauser indices, which is consistent with most of the findings of the current study.

Schneeweiss et al. [47] observed that in a population of older adults, comorbidity measures based on medication codes had poorer performance than measures based on diagnosis codes. We also observed this for the age-specific analyses. The percentage change in the c-statistic for the full models containing the CDS and number of different drugs was larger in younger than in older age groups. The poor performance of measures based on prescription drug codes may arise because we focused on short-term outcomes and some drugs are used by individuals primarily for preventive therapy, as opposed to being used for treatment of chronic conditions. Finally, the addition of new drug classes to the marketplace since the CDS was originally developed may also contribute to its poorer predictive performance.

An interesting finding was that members of Cohort 1 had fewer comorbid conditions but were more likely to experience a health outcome compared to members of Cohort 2, who had a greater burden of comorbidity but were less likely to experience a health outcome. This observation of greater comorbidity could potentially be explained by the increase in the number of diagnostic fields from three in 1997/98 to 25 in 2001/02 in hospital administrative data. However, this does not appear to have affected the overall predictive performance of the comorbidity measures. The finding that predictive performance of comorbidity measures was not substantially different when diagnoses were based on ICD-9 only compared to when they were based on both ICD-9 and ICD-10 is consistent with previous research [12, 13].

There are some limitations to this study. Comorbidity was defined using only a single year of data and was based on data from the year immediately prior to the outcome observation period. However, this methodology parallels one adopted in a similar study involving a general elderly population [10]. Moreover, previous research found that varying the time frame for measurement of comorbidity had a trivial effect on predictive performance [48, 49]. The study cohorts were not independent; 80% of the individuals in Cohort 1 were also present in Cohort 2. It would have been preferable to examine predictive performance in independent cohorts defined over the same period of time, with different ICD coding systems being used in parallel, to avoid the potential confounding effects of cohort aging and changes in ICD coding on predictive performance. Sundarajan et al. [14] also noted the potential for temporal confounding in their investigation of changes in ICD coding. However, a study design that used independent cohorts was not possible to implement to evaluate the potential effects of the change in diagnosis coding. We observed that the prevalence of comorbid conditions was similar in both cohorts, with the exception of uncomplicated hypertension, suggesting little change in capture of major comorbidities with a change in diagnosis coding. Other comorbidity measures could have been included in the analysis. For example, an updated version of the CDS has been developed, [50] although Schneeweiss et al. [47] found that this revision did not result in improved predictive performance when compared to the original CDS in an elderly population. Another limitation is that some of the investigated outcomes were sparse in the cohorts, which can reduce the power of Delong’s [41] test for differences in discriminative performance of the models [51]. Finally, it is generally recognized that when working with administrative data, misclassification may arise due to inaccuracies in the assignment of diagnostic codes [32]. For example, rule-out diagnoses, which are used to indicate that an individual does not have a condition, may be incorrectly classified as comorbidities.

Major strengths of this study are the investigation of multiple outcome measures, several commonly-used general measures of comorbidity and measures based on both diagnosis and prescription drug codes. As well, we conducted age-specific analyses as well as analyses for all-ages cohorts to assess the generalizability of performance of comorbidity measures across the population. Using population-based data as opposed to data for a specific clinical cohort improves generalizability of the study results. Finally, our base model included a variety of variables that can be validly defined using administrative data and a broad range of potential risk variables.

In summary, our study suggests that the predictive performance of comorbidity measures based on administrative health data in population-based diabetes cohorts will vary with the outcome measure under investigation, although the Elixhauser index performed well overall. Predictive performance of all measures may not be equivalent for all age groups. At the same time, changes in the diagnosis coding system used in hospitalization data do not appear to affect predictive performance over time.


  1. Virnig BA, McBean M: Administrative data for public health surveillance and planning. Annu Rev Public Health. 2001, 22: 213-230. 10.1146/annurev.publhealth.22.1.213.

    Article  CAS  PubMed  Google Scholar 

  2. Valderas JM, Starfield B, Sibbald B, Salisbury C, Roland M: Defining comorbidity: implications for understanding health and health services. Ann Fam Med. 2009, 7: 357-363. 10.1370/afm.983.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Grunau GL, Sheps S, Goldner EM, Ratner PA: Specific comorbidity risk adjustment was a better predictor of 5-year acute myocardial infarction mortality than general methods. J Clin Epidemiol. 2006, 59: 274-280. 10.1016/j.jclinepi.2005.08.007.

    Article  PubMed  Google Scholar 

  4. St Germaine-Smith C, Liu M, Quan H, Wiebe S, Jette N: Development of an epilepsy-specific risk adjustment comorbidity index. Epilepsia. 2011, 52: 2161-2167. 10.1111/j.1528-1167.2011.03292.x.

    Article  PubMed  Google Scholar 

  5. Schneeweiss S: Sensitivity analysis and external adjustment for unmeasured confounders in epidemiologic database studies of therapeutics. Pharmacoepidemiol Drug Saf. 2006, 15: 291-303. 10.1002/pds.1200.

    Article  PubMed  Google Scholar 

  6. Von Korff M, Wagner EH, Saunders K: A chronic disease score from automated pharmacy data. J Clin Epidemiol. 1992, 45: 197-203. 10.1016/0895-4356(92)90016-G.

    Article  CAS  PubMed  Google Scholar 

  7. Charlson ME, Pompei P, Ales KL, MacKenzie CR: A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis. 1987, 40: 373-383. 10.1016/0021-9681(87)90171-8.

    Article  CAS  PubMed  Google Scholar 

  8. Elixhauser A, Steiner C, Harris DR, Coffey RM: Comorbidity measures for use with administrative data. Med Care. 1998, 36: 8-27. 10.1097/00005650-199801000-00004.

    Article  CAS  PubMed  Google Scholar 

  9. Dominick KL, Dudley TK, Coffman CJ, Bosworth HB: Comparison of three comorbidity measures for predicting health service use in patients with osteoarthritis. Arthritis Rheum. 2005, 53: 666-672. 10.1002/art.21440.

    Article  PubMed  Google Scholar 

  10. Perkins AJ, Kroenke K, Unutzer J, Katon W, Williams JW, Hope C, Callahan CM: Common comorbidity scales were similar in their ability to predict health care costs and mortality. J Clin Epidemiol. 2004, 57: 1040-1048. 10.1016/j.jclinepi.2004.03.002.

    Article  PubMed  Google Scholar 

  11. Schneeweiss S, Seeger J, Maclure M, Wang P, Avorn J, Glynn RJ: Performance of comirbidity scores to control for confounding in epidemiologic studies using claims data. Am J Epidemiol. 2001, 154: 854-865. 10.1093/aje/154.9.854.

    Article  CAS  PubMed  Google Scholar 

  12. Quan H, Sundararajan V, Halfon P, Fong A, Burnand B, Luthi JC, Saunders LD, Beck CA, Feasby TE, Ghali WA: Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Med Care. 2005, 43: 1130-1139. 10.1097/01.mlr.0000182534.19832.83.

    Article  PubMed  Google Scholar 

  13. Li B, Evans D, Faris P, Dean S, Quan H: Risk adjustment performance of Charlson and Elixhauser comorbidities in ICD-9 and ICD-10 administrative databases. BMC Health Serv Res. 2008, 8: 12-10.1186/1472-6963-8-12.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Sundararajan V, Henderson T, Perry C, Muggivan A, Quan H, Ghali WA: New ICD-10 version of the Charlson comorbidity index predicted in-hospital mortality. J Clin Epidemiol. 2004, 57: 1288-1294. 10.1016/j.jclinepi.2004.03.012.

    Article  PubMed  Google Scholar 

  15. De Berardis G, D’Ettorre A, Graziano G, Lucisano G, Pellegrini F, Cammarota S, Citarella A, Germinario CA, Lepore V, Menditto E, et al: The burden of hospitalization related to diabetes mellitus: A population-based study. Nutr Metab Cardiovasc Dis. 2012, 22: 605-612. 10.1016/j.numecd.2010.10.016.

    Article  CAS  PubMed  Google Scholar 

  16. Flores-Le Roux JA, Comin J, Pedro-Botet J, Benaiges D, Puig-de DJ, Chillaron JJ, Goday A, Bruguera J, Cano-Perez JF: Seven-year mortality in heart failure patients with undiagnosed diabetes: an observational study. Cardiovasc Diabetol. 2011, 10: 39-10.1186/1475-2840-10-39.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Meisinger C, Heier M, Von SW, Kirchberger I, Hormann A, Kuch B: Gender-Specific short and long-term mortality in diabetic versus nondiabetic patients with incident acute myocardial infarction in the reperfusion era (the MONICA/KORA Myocardial Infarction Registry). Am J Cardiol. 2010, 106: 1680-1684. 10.1016/j.amjcard.2010.08.009.

    Article  PubMed  Google Scholar 

  18. Ovbiagele B, Markovic D, Fonarow GC: Recent US patterns and predictors of prevalent diabetes among acute myocardial infarction patients. Cardiol Res Pract. 2011, 2011: 145615.

    PubMed  PubMed Central  Google Scholar 

  19. Escobar C, Blanes I, Ruiz A, Vinuesa D, Montero M, Rodriguez M, Barbera G, Manzano L: Prevalence and clinical profile and management of peripheral arterial disease in elderly patients with diabetes. Eur J Intern Med. 2011, 22: 275-281. 10.1016/j.ejim.2011.02.001.

    Article  PubMed  Google Scholar 

  20. Kramer CK, Rodrigues TC, Canani LH, Gross JL, Azevedo MJ: Diabetic retinopathy predicts all-cause mortality and cardiovascular events in both type 1 and 2 diabetes: meta-analysis of observational studies. Diabetes Care. 2011, 34: 1238-1244. 10.2337/dc11-0079.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Quail JM, Lix LM, Osman BA, Teare GF: Comparing comorbidity measures for predicting mortality and hospitalization in three population-based cohorts. BMC Health Serv Res. 2011, 11: 146-10.1186/1472-6963-11-146.

    Article  PubMed  PubMed Central  Google Scholar 

  22. Camilloni L, Farchi S, Giorgi RP, Chini F, Borgia P: Mortality in elderly injured patients: the role of comorbidities. Int J Inj Contr Saf Promot. 2008, 15: 25-31. 10.1080/17457300701800118.

    Article  PubMed  Google Scholar 

  23. Gagne JJ, Glynn RJ, Avorn J, Levin R, Schneeweiss S: A combined comorbidity score predicted mortality in elderly patients better than existing scores. J Clin Epidemiol. 2011, 64: 749-759. 10.1016/j.jclinepi.2010.10.004.

    Article  PubMed  PubMed Central  Google Scholar 

  24. Statistics Canada. 2011, 2006 community profiles,

  25. Downey W, Beck C, McNutt M, Stang M, Osei W, Nichol J: Health databases in Saskatchewan. Pharmacoepidemiology. Edited by: Strom BL. 2000, New York: Wiley, 325-345. 3

    Chapter  Google Scholar 

  26. Edouard L, Rawson NS: Reliability of the recording of hysterectomy in the Saskatchewan health care system. Br J Obstet Gynaecol. 1996, 103: 891-897. 10.1111/j.1471-0528.1996.tb09908.x.

    Article  CAS  PubMed  Google Scholar 

  27. Liu L, Reeder B, Shuaib A, Mazagri R: Validity of stroke diagnosis on hospital discharge records in Saskatchewan, Canada: implications for stroke surveillance. Cerebrovasc Dis. 1999, 9: 224-230. 10.1159/000015960.

    Article  CAS  PubMed  Google Scholar 

  28. Rawson NS, D’Arcy C: Assessing the validity of diagnostic information in administrative health care utilization data: experience in Saskatchewan. Pharmacoepidemiol Drug Saf. 1998, 7: 389-398. 10.1002/(SICI)1099-1557(199811/12)7:6<389::AID-PDS380>3.0.CO;2-S.

    Article  CAS  PubMed  Google Scholar 

  29. Canada H: Responding to the challenge of diabetes in. 2003, First report of the National Diabetes Surveillance System. Ottawa, ON: Canada

    Google Scholar 

  30. Blanchard JF, Ludwig S, Wajda A, Dean H, Anderson K, Kendall O, Depew N: Incidence and prevalence of diabetes in Manitoba, 1986–1991. Diabetes Care. 1996, 19: 807-811. 10.2337/diacare.19.8.807.

    Article  CAS  PubMed  Google Scholar 

  31. Hux JE, Ivis F, Flintoft V, Bica A: Diabetes in Ontario: determination of prevalence and incidence using a validated administrative data algorithm. Diabetes Care. 2002, 25: 512-516. 10.2337/diacare.25.3.512.

    Article  PubMed  Google Scholar 

  32. Schneeweiss S, Maclure M: Use of comorbidity scores for control of confounding in studies using administrative databases. Int J Epidemiol. 2000, 29: 891-898. 10.1093/ije/29.5.891.

    Article  CAS  PubMed  Google Scholar 

  33. Klabunde CN, Potosky AL, Legler JM, Warren JL: Development of a comorbidity index using physician claims data. J Clin Epidemiol. 2000, 53: 1258-1267. 10.1016/S0895-4356(00)00256-0.

    Article  CAS  PubMed  Google Scholar 

  34. Metcalfe A, Neudam A, Forde S, Liu M, Drosler S, Quan H, Jette N: Case definitions for acute myocardial infarction in administrative databases and their impact on in-hospital mortality rates. Health Serv Res. 2013, 48: 290-318. 10.1111/j.1475-6773.2012.01440.x.

    Article  PubMed  Google Scholar 

  35. Klomp H, Chan BTB, Cascagnette P, Teare G, Sidhu N: Quality of diabetes management in Saskatchewan: technical appendix. 2006, Saskatoon, SK: Health Quality Council

    Google Scholar 

  36. Canadian Stroke Strategy Information and Evaluation Working Group: Canadian Stroke Strategy performance measurement manual. 2008, Ottawa, ON: Canadian Stroke Network

    Google Scholar 

  37. Roos NP, Mustard CA: Variation in health and health care use by socio-economic status in Winnipeg, Canada: the system works well? Yes and no. Milbank Q. 1997, 75: 89-111. 10.1111/1468-0009.00045.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  38. Reiter JP, Raghunathan TE: The multiple adaptations of multiple imputation. J Am Stat Assoc. 2007, 102: 1462-1471. 10.1198/016214507000000932.

    Article  CAS  Google Scholar 

  39. Ikeda M, Ishigaki T, Yamauchi K: Relationship between Brier score and area under the binormal ROC curve. Computer Meth Prog Biomedicine. 2002, 67: 187-194. 10.1016/S0169-2607(01)00157-2.

    Article  Google Scholar 

  40. Harrell FE, Lee KL, Mark DB: Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996, 15: 361-387. 10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4.

    Article  PubMed  Google Scholar 

  41. Delong ER, DeLong DM, Clarke-Pearson DL: Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988, 44: 837-845. 10.2307/2531595.

    Article  CAS  PubMed  Google Scholar 

  42. Steyerberg EW, Harrell FE, Habbema JDF: Prognostic modelling with logistic regression analysis: a comparison of selection and estimation methods in small data sets. Stat Med. 2000, 19: 1059-1079. 10.1002/(SICI)1097-0258(20000430)19:8<1059::AID-SIM412>3.0.CO;2-0.

    Article  CAS  PubMed  Google Scholar 

  43. Redelmeier DA, Bloch DA, Hickam DH: Assessing predictive accuracy: how to compare Brier scores. J Clin Epidemiol. 1991, 44: 1141-1146. 10.1016/0895-4356(91)90146-Z.

    Article  CAS  PubMed  Google Scholar 

  44. SAS Institute Inc: SAS/STAT user’s guide. 2004, Cary, NC: SAS Institute Inc

    Google Scholar 

  45. Farley JF, Harley CR, Devine JW: A comparison of comorbidity measurements to predict healthcare expenditures. Am J Manag Care. 2006, 12: 110-119.

    PubMed  Google Scholar 

  46. Maciejewski ML, Liu CF, Derleth A, McDonell M, Anderson S, Fihn SD: The performance of administrative and self-reported measures for risk adjustment of veterans affairs expenditures. Health Serv Res. 2005, 40: 887-904. 10.1111/j.1475-6773.2005.00390.x.

    Article  PubMed  PubMed Central  Google Scholar 

  47. Schneeweiss S, Wang PS, Avorn J, Maclure M, Levin R, Glynn RJ: Consistency of performance ranking of comorbidity adjustment scores in Canadian and U.S. utilization data. J Gen Intern Med. 2004, 19: 444-450. 10.1111/j.1525-1497.2004.30109.x.

    Article  PubMed  PubMed Central  Google Scholar 

  48. Radley DC, Gottlieb DJ, Fisher ES, Tosteson ANA: Comorbidity risk-adjustment strategies are comparable among persons with hip fracture. J Clin Epidemiol. 2008, 61: 580-587. 10.1016/j.jclinepi.2007.08.001.

    Article  PubMed  PubMed Central  Google Scholar 

  49. Wang PS, Walker A, Tsuang M, Orav EJ, Levin R, Avorn J: Strategies for improving comorbidity measures based on Medicare and Medicaid claims data. J Clin Epidemiol. 2000, 53: 571-578. 10.1016/S0895-4356(00)00222-5.

    Article  CAS  PubMed  Google Scholar 

  50. Clark DO, Von KM, Saunders K, Baluch WM, Simon GE: A chronic disease score with empirically derived weights. Med Care. 1995, 33: 783-795. 10.1097/00005650-199508000-00004.

    Article  CAS  PubMed  Google Scholar 

  51. Peek N, Arts DG, Bosman RJ, van der Voort PH, De Keizer NF: External validation of prognostic models for critically ill patients required substantial sample sizes. J Clin Epidemiol. 2007, 60: 491-501.

    Article  CAS  PubMed  Google Scholar 

Pre-publication history

Download references


This research was supported in part by a Canadian Institutes of Health Research New Investigator Award, a CIHR Operating Grant (Funding Reference #86575) and funding from the Centennial Chair Program, University of Saskatchewan, to the first author. The authors are indebted to Nedeene Hudema of the Health Quality Council for assistance with data extraction and analysis. This study is based in part on de-identified data provided by the Saskatchewan Ministry of Health. The interpretation and conclusions contained herein do not necessarily represent those of the Government of Saskatchewan or the Ministry of Health.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Lisa M Lix.

Additional information

Competing interests

The authors declare that they have no competing interests.

Authors’ contributions

LML and JQ conducted the data extraction and carried out the statistical analyses. OF summarized the data. LML, JQ, and OF drafted the manuscript. LML, JQ, and GFT conceived the study and participated in its design. All authors read and approved the final manuscript.

Rights and permissions

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article

Lix, L.M., Quail, J., Fadahunsi, O. et al. Predictive performance of comorbidity measures in administrative databases for diabetes cohorts. BMC Health Serv Res 13, 340 (2013).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: