A comparison of a multistate inpatient EHR database to the HCUP Nationwide Inpatient Sample

Abstract

Background

The growing availability of electronic health records (EHRs) in the US could provide researchers with a more detailed and clinically relevant alternative to using claims-based data.

Methods

In this study we compared a very large EHR database (Health Facts©) to a well-established population estimate (Nationwide Inpatient Sample). Weighted comparisons were made using t-value and relative difference over diagnoses and procedures for the year 2010.

Results

The two databases have a similar distribution pattern across all data elements, with 24 of 50 data elements being statistically similar between the two data sources. In general, differences that were found are consistent across diagnosis and procedures categories and were specific to the psychiatric–behavioral and obstetrics–gynecology services areas.

Conclusions

Large EHR databases have the potential to be a useful addition to health services researchers, although they require different analytic techniques compared to administrative databases; more research is needed to understand the differences.

Background

Quality measurement and many health services research studies traditionally rely on administrative claims data. Discrepancies between medical claims-based data and clinical data within the patient medical record are well established [1–3], and claims data has been more consistently available than clinical data. However, recent advances in electronic health record (EHR) and health information exchange technology adoption may increase the availability of clinical data for research.

The Congressional Budget Office estimates that 90 % of physicians and 70 % of hospitals in the US will have electronic health records by 2019 [4]. Although electronic health records are primarily intended to support patient care, there is a high level of interest in using EHR data for secondary purposes such as biomedical and outcomes research, clinical process improvement, and epidemiological monitoring [5–8]. Despite some differences in EHR data quality compared to expert review and self-reported data [9], many health services researchers consider the EHR record to be more accurate compared to claims data and are calling for a shift away from claims-based measures to using clinical measures derived from the EHR [3, 10] for research, quality measurement, and performance monitoring. This transition away from claims data is demonstrated by the recent HITECH Act, which requires providers to report key clinical performance metrics directly from their EHR [11]. Limited information has been published on the benefits and limitations of using EHR data compared to claims-based data [12], yet it is critically important to understand these differences as we begin to rely on EHR data for research and payment in addition to supporting clinical care.

The potential advantages of using EHR data are numerous and are generally related to the detailed nature of the data as well as the benefits of being on a computer-based platform, yet the challenges posed are equally significant. The growing availability of electronic health records in the US could provide researchers with faster, less resource-intensive access to data, larger population samples, more data measurements, and additional types of data compared with primary data collection methods and claims data [6, 13]. However, researchers have noted significant challenges with using EHR data including privacy issues, variation in EHR data sources, inconsistent case definitions, poor data quality, and questionable representativeness of patient populations [8, 14, 15]. Many of these comparisons are based on public health or research data capture models, in which the data capture instrument and process were designed to capture comprehensive and analysis-ready data, while EHRs are generally implemented to provide clinically effective and rapid workflows with a higher tolerance for missing or absent data.

Notable platform-specific EHR databases that are frequently referenced in publications include the Veterans Health Administration databases [16, 17], the Kaiser Permanente Northern California Research Database [18], and the GE Centricity EMR Database [19]. Although valuable for many types of inquiry, using databases such as these to make population estimates may be problematic due to unique regional variations, inpatient or outpatient bias, or limited demographic population captured in the data.

A large EHR database, however, which is widely dispersed among different populations, hospital types, and care settings, may be used to make estimates of health at the population level or for broader groups such as the national universe of inpatients. Moreover, large EHR databases that contain detailed clinical data not found in claims databases could be very valuable to population health management if it were accurate, consistent, and representative of the population. As large EHR databases grow in numbers and in research utility, it becomes even more important to assess their strengths and limitations compared to existing data sources.

In this study, we compare a very large EHR database (Health Facts®) to a well-established population estimate (Nationwide Inpatient Sample). Although we are primarily concerned with the issue of representativeness, other considerations are relevant to our results. To assess representativeness, we calculate adjusted population estimates for encounter demographics, diagnoses, and procedures from both data sources, as well as the relative differences and t-values between the Nationwide Inpatient Sample and Health Facts® EHR populations.

Cerner health facts®

Cerner Corporation, a leading EHR vendor, maintains one of the largest vendor-specific EHR databases, called Health Facts® (HF). Contributing organizations receive quality and benchmarking reports based on internal and external data. HF data is de-identified and HIPAA-compliant to protect both patient and organization identity. The patient-level data in HF includes encounter, medication, diagnosis, laboratory orders and results, pharmacy, and procedure values. These records are comprehensive and include over 300 data elements. For example encounter information includes details such as payer, discharge and admission sources, and care setting. Laboratory data includes collection sites, reference ranges, and the timing of collection and results. The database is longitudinal and hospital patients can be followed post discharge if they return to care settings within the same health system. At the time of this writing, HF contains data from more than 500 health care facilities across the United States representing 133 million encounters and 84 million patients over the past two decades. Health Facts® data is a publicly available resource from Cerner Corporation (Kansas City, MO).

AHRQ HCUP Nationwide inpatient sample

The Agency for Healthcare Research and Quality (AHRQ) Healthcare Cost and Utilization Project (HCUP) Nationwide Inpatient Sample (NIS) is a valuable resource primarily intended for studies of inpatient hospital utilization and charges (http://www.hcup-us.ahrq.gov/nisoverview.jsp). It is the largest publicly available all-payer inpatient care database for the United States; in 2010, the NIS sampled approximately 20 % (1051) of all hospitals in 45 states. National estimates are based on a validated method of weights [20]. The NIS is released annually and contains standardized data elements reporting hospital characteristics, diagnosis and procedure codes, and other important data that can support research on medical treatment and effectiveness, quality of care and patient safety, impact of health policy changes, and use of hospital services within the United States. The research footprint and ultimate value of the HCUP NIS and other HCUP all-payer databases is significant, with thousands of studies publishing on HCUP data in peer-reviewed journals including hundreds of citations in top journals [21].

Methods

Data sources

Both the HF EHR database and the NIS data set capture patient-level data elements linked to inpatient encounters. Both data sources are available to the public at www.cerner.com/lifesciences and https://www.hcup-us.ahrq.gov/nisoverview.jsp respectively.

For this analysis we used an extract from the HF database made in January 2012. To make the HF data comparable to the 2010 NIS data set, we included only acute care discharges made during 2010 and excluded any stay longer than 365 days. We then created a weighting scheme similar to the weighting scheme used in the NIS database to make population estimates. First, we used 2010 American Hospital Association data stratified by region and hospital size to determine how many hospitals and discharges there were in the universe. We then assigned a weight to each hospital in HF that reflected its representation in the stratification. The formula for weight used is: (# in AHA universe/# in HF for each stratum). Strata used are region and bed size. For example, each 200–299 bed hospital in the Northeast in the HF sample is weighted to be equivalent to 6.23 hospitals in the universe. Because of limitations as a convenience sample, this weighting scheme is not balanced (i.e., we did not try to have the same size groupings) and does not take into account teaching hospitals as the NIS does. We used HCUP NIS data for 2010 for the comparison with no additional modifications.

Analytic methods

Population estimates were calculated using SAS9 (SAS Institute Inc., Cary, NC, USA). Weighted estimates of diagnoses were aggregated using Major Diagnostic Category (MDC) groupings and included primary (i.e., first listed) diagnoses recorded for an encounter in the respective database. Procedures were aggregated according to high-level Clinical Classification Software (CCS) groupings and also included primary (first listed) procedure codes. ICD-9-CM code mappings for MDC and CCS groupings can be found on the HCUP website.

Standard errors were calculated and presented along with the NIS-weighted count estimates. HF provides complete enumeration of discharges from the source; therefore, no standard errors are presented. To compare the two data sets, we use relative difference and present the t-value. Relative difference is represented by the absolute value of (NISest-HFest)/NISest. The t-value is calculated as the difference between the estimations divided by the standard error (NIStot-HFtot)/(NISeststd err). A hypothesis-driven comparison of two data sets consisting of very large data sets such as these would result in statistical significance with very little difference. Therefore, we assess the significance of the t-value at 95 % confidence and, due to such large sample sizes, also at 99.9 % confidence. For those that are significantly different, we also provide the relative difference to assess the magnitude.

Results

Encounters and demographics are presented in Table 1.

Table 1 Encounters and demographics: comparison of nationwide inpatient sample (NIS) and health facts® (HF) EHR population

At the discharge level, the regional counts of discharges are generally comparable between the NIS data set and the HF EHR population. However, the HF population tends to be slightly younger (by approximately 1 year) and report fewer in-hospital deaths (1.63 vs. 1.90 %) compared to the NIS data set.

Tables 2 and 3 present comparisons on diagnoses and procedures, respectively.

Table 2 Diagnoses: comparison of nationwide inpatient sample (NIS) and health facts® (HF) EHR population

Table 3 Procedures: comparison of nationwide inpatient sample (NIS) and health facts® (HF) EHR population

Nearly all of the encounters for both the NIS and HF had primary (i.e., first listed) diagnoses, equaling roughly 39 million encounters for each. In all but two of the MDC disease categories (14, 15), the percentage of the overall proportion of discharges in the NIS data set is within 2 % of the overall proportion of the HF population, indicating that the distribution of disease is very similar between the two data sources. Actual counts of encounters follow a similar pattern on comparability. With 95 % confidence in detecting a difference, 15 of the 25 MDC show statistically significant differences between the data sets. The relative difference is smaller than 5 % in one category, and smaller than 10 % in 5 categories. Although many MDC groups are consistent between the NIS data set and HF population, there are significant differences in the number of primary diagnoses in obstetrics and gynecological diagnoses (MDC 13, 14, 15), as well as psychiatric and behavioral diagnoses (MDC 19, 20).

About two-thirds of the NIS data set and one-half of the HF population had procedure codes associated with the encounter. Despite representing significantly fewer procedures overall, the proportions of procedures done in each CCS category in the HF EHR population were very similar to the proportions in the NIS data set. With 95 % confidence, 10 out of 16 NIS CCS categories are different between the data sets. However, only four of the procedure categories (9, 11, 13, 14) had proportions that differed by more than 2 % between the NIS data set and the HF population.

Discussion

Regarding the demographic data on encounters, the comparison of the NIS data set with the HF EHR population shows strong similarities. The HF EHR population is slightly younger by about a year and reported slightly fewer in-hospital deaths. The small difference in age and mortality could possibly be explained through the distribution of hospitals in the HF database. Compared to the NIS data set, small hospitals (<99 beds) are overrepresented in the HF EHR database. If larger hospitals tend to have older patients and increased mortality compared to much smaller hospitals, then this may be reflected in the results. The weighting system applied to HF for this analysis is designed to adjust for much of this variation, yet some difference may remain.

Comparisons of the NIS data set and the HF population for diagnoses and procedures show mixed results. With the exception of a few diagnoses, MDC categories in the HF EHR population were consistent with the NIS data set. Psychiatry and behavioral (Alcohol/Drug Use & Alcohol/Drug Induced Organic Mental Disorders, Mental Diseases & Disorders) diagnoses as well as obstetrics and gynecology (Diseases & Disorders Of The Female Reproductive System, Pregnancy, Childbirth & The Puerperium, Newborns & Other Neonates With Conditions Originating In Perinatal Period) diagnoses are much less frequently represented in the HF EHR population compared to the NIS data set. It is possible that these differences could be attributed to selection bias of hospitals in the HF database resulting in hospitals that see fewer psychiatry and obstetrics and gynecology cases. For example, hospitals who adopted this specific EHR may see fewer of these kinds of cases. Another, more likely, explanation is that many of those services were still using either paper-based clinical records or a separate EHR system during the study period. This would result in fewer diagnoses in these categories within the hospital EHR. Mental health, substance abuse, and reproductive/sexual health are identified as are identified as “sensitive health information” in the recommendations of the National Committee on Vital and Health Statistics to the Secretary of Health and Human Services [22]. Some participating institutions may have compartmentalized this data from the less sensitive EHR data or embraced emerging security features of the EHR that allow patients to have greater control over the privacy of their health data. Information about the frequency of procedures was generally less consistent between the data sources compared to diagnoses. Although 11 of the 15 procedure categories in the NIS data set are proportionally within 2 % of the HF population, the HF population reports many fewer procedures overall compared with the NIS data set. Prior research has identified that EHR databases have tended to capture fewer data elements related to provider orders [23], which may explain the difference. Another possible explanation is that not all contributing sites are able to supply procedure codes to the database.

There are overarching considerations to the differences between EHR databases and claims-based samples identified in our findings. First and perhaps primarily, they are different resources created and predominantly used for distinctly different purposes. Prior research has shown claims coding can be markedly different than patient problems captured by providers [1, 24, 25]. Neither claims nor medical records are primarily collected for research so one is not necessarily more ‘correct’ than the other. Claims data in general has the advantage of years of validation and research use. Comparably, questions of internal validity and data integrity surround EHR data. Researchers are beginning to address many of these issues by developing innovative data infrastructure and study designs [14, 26]. In contrast, the HCUP NIS is a validated and trusted data source; however, it lacks much of the detailed clinical information that is available through large, longitudinal, detailed EHR databases.

Differences between claims-based data and EHR data found in our analyses may have implications for quality of care reporting since many federal measures are derived from claims data. Many federal-based quality reporting measures are based on claims data and the significance of the differences in reporting quality measures through the EHR are not well understood. For example, EHRs typically have more diagnosis fields (we found nearly 100 in this sample) compared to claims data, which may be significant when excluding cases from quality measures. Also, concepts are documented in claims data that are not documented in EHRs, and vice versa.

Additionally, EHR data may capture health care provided to patients yet not billed for that may not be reflected in the claims-based data, which could affect the comparison between the two data sets. The EHR sample included uninsured and self-pay patients, which the claims data do not have. Although EHRs may be more comprehensive in this regard, it complicates the comparison between the two. Finally, this analysis focused on in-patient hospitalizations; comparison of outpatient data might yield different results.

EHR data is a new type of data with new research possibilities, rather than being used as a supplement or substitute for claims based study designs. For example, EHR data may be preferred to claims data when clinical outcomes such as laboratory tests are required, or a level of clinical specificity is needed. EHR data may be preferred when the complete lists of diagnosis are needed, as many claims data will use only the first 25 or so diagnosis. Some data from the EHR such as problem lists, reflect clinician assigned diagnoses as they work the case, rather than codes assigned by a professional medical coder.

Limitations

Some limitations of this study stem from the methodological differences between the NIS and HF data sets. NIS sampling is based on sampling techniques validated to represent hospitals across the country. Although the weighting scheme applied to HF is adjusted somewhat to reflect a national population of hospitals, it was not as detailed as the NIS weighting scheme and likely prone to error, particularly in the western region of the US where fewer hospitals contribute to the HF database. These data were collected through a single EHR vendor and it is unknown how they may differ (if in any way) from data obtained elsewhere. In addition, there may be differences between the data sets in the primary diagnosis criteria and definition. There may also be differences due to the inclusion of self-pay and uninsured patients in the EHR data. It is also possible that there were structural or service line differences between the hospitals in each sample. For example, some hospitals may not have Obstetrics or Psychiatry. Significant differences in either of these may skew the results.

A second limitation of the study reflects the transition from paper-based records to EHRs occurring during the study period. It is very likely that in the study year (2010) many of the hospitals contributing to the HF EHR database were not completely EHR-based throughout the hospital, potentially resulting in missing information within the EHR database. More research is needed to compare EHR populations to claims samples to detect changes in these differences over time.

A third important limitation of the study is that the analysis was done on hospital data and not outpatient data. Inpatient and outpatient EHR adoption rates varies, as does the documentation and coding practices. Prior research has found variation between outpatient EHR data and Medicaid claims [27], but promising methods for applying claims-based quality measures to outpatient EHRs are being developed [28]. More research is needed into understanding the developing role of outpatient EHR data in health services research and quality measurement.

As already described, HF is not a random sample and the EHR population at the time of this analysis consisted of proportionately more small (<99 bed) hospitals compared with the NIS, with a greater concentration of hospitals in the northeast region. Addressing these differences required an unbalanced weighting scheme with weights that varied by a factor of 100. Also, there may be differences in health care organizations that choose this specific EHR vendor compared to those who choose another vendor.

Conclusions

This study compared demographic variables, diagnosis groups, and procedure categories in the HCUP NIS data set and HF EHR population. Compared to the NIS, the HF EHR population is slightly younger with lower in-hospital mortality. There tends to be about the same distribution of principal diagnoses between the groups, with the HF EHR sample reporting fewer psychiatry, behavioral, obstetrics, and gynecology diagnoses. Compared to the NIS data set, HF captured fewer procedures overall, and also in relation to surgeries involving male and female genital organs and obstetric procedures.

These findings improve our understanding of the differences between established claims-based databases and large EHR databases and support evidence that large EHR databases have the potential to be a useful addition to health services researchers.

More research is needed to understand the internal and external validity of large EHR databases as it pertains to population health research, specifically in the interpretation of results from EHR databases. There is also a significant need for the development of innovative methodologies that may be required to fully utilize these rapidly growing data sets.

Appendix

Table 4 Weighting scheme