Pre-existing conditions or comorbidities have been used in risk adjustment for health outcomes research [1, 2]. The number and type of comorbidities can have a significant impact on patient outcomes and may explain some of the observed variations [3–9]. Traditionally, medical charts were used to obtain information on comorbidities. This is a very laborious process. With the advent of electronic health care databases that capture financial data for the purpose of claims, such administrative data have provided an alternative for health services researchers to obtain comorbidity information for outcomes research [10–13].

While information obtained from administrative databases has been used for risk adjustment [14–16], the evidence on accuracy of secondary diagnoses from administrative databases as a substitute for comorbidities has been mixed [17–30]. Previous studies have found that administrative databases tend to under-report some comorbidities but overestimate the prevalence of others when compared with information from medical charts [17–21, 24, 28–31]. Comorbidities that showed better agreement between both data sources were solid tumor [17], diabetes mellitus [25], connective tissue disease [17, 19], and chronic pulmonary disease [25]. The conditions with poor agreement were renal disease [17], dementia [17], hypertension [19], diabetes with complications [17, 25] and peripheral vascular disease [20].

Reasons for the discordance of comorbidity assignment from both sources of information have been offered. Romano et al [24] and Powell et al [21] found that conditions which were asymptomatic tended to be under-reported in administrative data. Iezzoni et al [18] suggested that some acute medical conditions or complications were deemed by coders to be more important than others, thereby creating coding bias.

Humphries et al [25] found that although the agreement between comorbdities from two different sources as measured by kappa statistics was only fair, there was no significant difference in the predictive value for all-cause mortality in a group of patients who have undergone percutaneous coronary intervention. Newschaffer et al [29] found similar results in a population of patients with breast cancer. Van Doorn [32] showed the same findings in a population of older adults. Susser et al [33] found that the Charlson Comorbidity Index (CCI) [16] constructed with comorbidities obtained from either administrative data or self-report had similar predictive validity for functional decline and health services utilization.

To date, there has not been any study that identified patient characteristics associated with the likelihood of over- or under-reporting of comorbidities in administrative data, particularly in older populations where the prevalence of comorbidities is higher. This study aimed to (a) quantify the agreement between administrative data and medical charts review across a set of comorbidities in older hospitalized persons; and (b) examine the factors associated with under- or over-reporting of comorbidities by the administrative data. We hypothesized that patients with high number of comorbidities were more likely to have under-reporting of comorbidities in the administrative data, while those who had longer lengths of hospital stay were more likely to have over-reporting.

A total of 3517 hospital admissions for pneumonia that satisfied the criteria for the study population were identified. Of these, 46 admissions (1.3%) were excluded from the study as their medical charts were not available during the period of review (Figure 1). The characteristics of the remaining 3471 admissions are summarized in Table 1.

The overall agreement between both data sources using kappa statistics ranged widely from 0.01 (poor agreement) to 0.78 (excellent agreement). Diabetes mellitus (uncomplicated and complicated), metastatic cancer, chronic pulmonary disease, lymphoma and alcohol abuse reported the highest kappa statistic values of 0.53 to 0.78.

Hierarchical modeling obtained very similar odds ratios and their 95% confidence intervals for factors explaining over- and under-reporting of comorbidities by the administrative data.

To the best of our knowledge, this is the first study that has evaluated the reliability of secondary diagnoses in the administrative data as a surrogate for comorbidities in Singapore. The results were consistent with other studies where the prevalence of comorbidities obtained from administrative data was lower than that obtained from medical charts for most conditions [17, 19–21, 24, 38–40]. This is despite the fact that both the research nurse who abstracted information from medical charts and the clinical coders responsible for assigning the ICD-9CM codes in the administrative database had obtained their information from the same source. The discordance can be explained by examining the reasons for which the information was collected. Pre-existing conditions were documented in medical charts to assist clinicians in clinical care decisions. Secondary diagnoses codes from administrative data do not differentiate between pre-existing conditions or complications that occurred during the hospitalization [41]. There is no flag in the administrative data indicating that the conditions existed on or after admission. Therefore, conditions such as fluid and electrolyte disorders, deficiency anemias, blood loss anemia, diabetes with complications, and coagulopathy reported higher prevalence rates in administrative data than in medical charts because it is highly likely that these conditions arose during hospitalization as a result of worsening medical condition, complications of treatment, or confirmation of new medical conditions through laboratory tests. Clinical coders extract information for reimbursement purposes, and were more likely to include any conditions that have an impact on the utilization of resources during the episode of hospitalization. Hence, these conditions would be coded as secondary diagnoses in the administrative data, whilst the research nurse abstracting the information from the medical charts would have excluded them. If these conditions were used in risk adjustment to compare outcomes, these could potentially cause an "over-adjustment" of risk [42–48]. On the other hand, chronic conditions such as depression, psychoses, peptic ulcer disease, paralysis, or renal failure [49] were under-reported. These conditions may not have been active problems during hospitalization for pneumonia and therefore, did not contribute to increased resource utilization. It is most likely that the patients with history of drug abuse as documented in the medical charts, were no longer receiving treatment for it, and the condition would not be coded by the clinical coders. Similarly, obesity was not coded as a secondary diagnosis as the patients were not likely to have received treatment for obesity during their hospitalization for pneumonia.

The kappa statistics for diabetes (uncomplicated and complicated), metastatic cancer and COPD showed substantial agreement, as was found in other studies [17, 21, 25]. Similarly, Quan and colleagues [40] reported that comorbidities obtained from ICD-9-CM data had sensitivity ranging from 9.3% to 83.1%. The wide range for PPV was a reflection of the different prevalence rates for individual comorbidities. Paralysis, blood loss anemia and weight loss with the lowest PPV had the lowest prevalence in the medical charts as well.

Factors associated with over-reporting were length of hospitalization, severity of illness, and inhospital death. This is not unexpected as these factors were likely to be associated with an increase in the number of investigations and interventions during the hospital episode, resulting in identification of additional concurrent conditions and complications. These conditions were more likely to be coded as secondary diagnoses in the administrative data because they were related to increased resource utilization. Other researchers have previously found that some medical conditions or complications of treatment were judged more important than other chronic conditions when patients were critically ill or when they died [18, 21, 23, 50].

Age, previous hospital admission and number of comorbidities were associated with under-reporting of comorbidities in the administrative database., Although there is no limit to the number of secondary diagnoses that can be coded, having a higher number of concurrent comorbidities may result in less important comorbidities being disregarded. Advancing age [8] and recent hospitalization are themselves associated with increased number of comorbidities. The association of these two factors with under-reporting may be a reflection of residual confounding of number of comorbidities that is not accounted for due to the specification of our regression models.

For the individual comorbidities, the same factors associated with over- or under-reporting for the whole study population were represented for most of the important ones we selected for detailed analyses. This was most clearly seen with number of concurrent comorbidities being associated with under-reporting for all 10 conditions. For length of hospitalization, severity of illness, inhospital death, age, and hospital admission in the previous 90 days, smaller sample sizes or lack of true effect may have accounted for the absence of association observed for some comorbidities. Nevertheless, we argue that the overall picture supports the results for the whole study population.

The main strength of this study is its large sample size. In addition, as there is no limit to the number of secondary diagnoses that can be coded in the administrative data, it is unlikely that the lower prevalence of comorbidities in the administrative data were due to restrictions imposed by the prevailing health information system.

There are several limitations of the study. Firstly, the findings of this study may not be generalized to older patients hospitalized for other acute illnesses because we only studied those with pneumonia. The use of DRG codes for pneumonia to identify cases could have included cases that did not meet the BTS criteria for pneumonia as DRG codes were assigned to reflect resource utilization. However, as the main objectives of this study were to compare the comorbidities obtained from two sources, and to identify the possible factors that may be associated with under or over-reporting of the comorbidities, and not on the outcomes for patients with pneumonia, the inclusion of such was not likely to have affected the findings. Secondly, documentation of comorbidities in the medical charts within the first 24 hours of admission may be incomplete in a busy ward environment with many patients waiting to be reviewed and therefore, may not be the ideal "gold standard" where comorbidities are concerned. To address this, we ensured that the research nurse was familiar with documentation in the medical charts and was trained to abstract very specific conditions. Attending physicians also had access to an electronic medical records system that contained information on known comorbidities. Therefore, we believe that the likelihood of documentation of important comorbidities being omitted is low. Thirdly, this study involved documentation of comorbidities in a single health care system and may not necessarily mirror the situation in other systems. Further research on factors associated with under- and over-reporting of comorbidities by administrative data in other health systems is needed to confirm our findings. Fourthly, we acknowledge that there may be other factors that could be associated with over- or under-reporting of the comorbidities but were not measured in our study. Although there were little prior research on this subject to guide us, we have included plausible factors on the basis of clinical opinion.

Our study confirmed that the prevalence of almost all comorbidities obtained from administrative data was lower than that obtained from medical chart review. The validity of secondary diagnoses from administrative data varies with the specific comorbidity in question, and is influenced by factors such as age, number of comorbidities, hospital admission in the preceding 90 days, severity of illness, length of hospitalization, and whether inhospital death occurred. While some comorbidities were reported as secondary diagnoses with a reasonable level of accuracy, there were several that may not be used interchangeably. Researchers should be cautious and take these findings into account when using this source of information as an alternative to medical chart reviews for the purpose of measuring comorbidity burden. This may also affect policy decisions by hospital administrators as it may underestimate the true burden of illnesses. Further research is needed to confirm our findings in other patient populations.