The accuracy and completeness for receipt of colorectal cancer care using Veterans Health Administration administrative data

Background The National Comprehensive Cancer Network and the American Society of Clinical Oncology have established guidelines for the treatment and surveillance of colorectal cancer (CRC), respectively. Considering these guidelines, an accurate and efficient method is needed to measure receipt of care. Methods The accuracy and completeness of Veterans Health Administration (VA) administrative data were assessed by comparing them with data manually abstracted during the Colorectal Cancer Care Collaborative (C4) quality improvement initiative for 618 patients with stage I-III CRC. Results The VA administrative data contained gender, marital, and birth information for all patients but race information was missing for 62.1 % of patients. The percent agreement for demographic variables ranged from 98.1–100 %. The kappa statistic for receipt of treatments ranged from 0.21 to 0.60 and there was a 96.9 % agreement for the date of surgical resection. The percentage of post-diagnosis surveillance events in C4 also in VA administrative data were 76.0 % for colonoscopy, 84.6 % for physician visit, and 26.3 % for carcinoembryonic antigen (CEA) test. Conclusions VA administrative data are accurate and complete for non-race demographic variables, receipt of CRC treatment, colonoscopy, and physician visits; but alternative data sources may be necessary to capture patient race and receipt of CEA tests.


Background
Approximately 3,400 colorectal cancers (CRC) are diagnosed in the Veterans Health Administration (VA) each year [1]. Considering CRC's 5 year survival rate of 65.2 % [2], most of these veterans will become longterm survivors. As such, the receipt of CRC-related care, which includes curative intent treatment for CRC (i.e. surgical resection, chemotherapy, and radiation) [3,4] as well as tests for assessment and surveillance after the diagnosis of CRC (i.e. colonoscopy, physician visit, and carcinoembryonic antigen (CEA) test) [5], is of broad interest for both veteran and non-veteran populations [6][7][8][9].
An accurate and efficient method is needed to measure receipt of CRC-related care. Potential data sources include surveys [10], health record review [11], and health claims (billing) data [10,[12][13][14][15][16][17]. The VA uses an electronic health record system, known as the Computerized Patient Record System (CPRS), which allows searching for procedure notes and other services within a given patient record to determine what care has been provided [18]. However, review of health records for research purposes typically occurs on a facility by facility basis because patient health records are not routinely shared between medical centers except for direct patient care. Use of health claims data has practical advantages in terms of access to large samples across multiple sites and low incremental costs. Thus, in this methodologic study, our objective was to compare the receipt of CRC-related care in VA according to administrative data compared with data from a manual chart abstraction of the electronic health record performed during a quality improvement collaborative (EHR/QI) called the Colorectal Cancer Care Collaborative (C4). C4 was undertaken to improve processes of care delivery for CRC [7,19].
Administrative data has been compared with health record review for receipt of cancer surveillance tests [12] and colonoscopy [10] by Medicare patients; receipt of colonoscopy prior to CRC diagnosis [17] and CRC surgery [16] in Alberta, Canada; and receipt of colonoscopy, surgery, chemotherapy, and radiation by CRC patients in New South Wales, Australia [15]. But this study is the first to attempt to validate the use of VA administrative data for measuring the receipt of CRC treatment and surveillance care. The objectives of this study are to identify types of care in which the use of VA administrative data to measure receipt of CRC-related care is reasonably complete and accurate as well as measures in which the VA administrative data may not be sufficiently complete or accurate so alternative methods may be necessary to collect this data in studies involving VA data.

Methods
The study was approved by the Institutional Review Board at Indiana University (0805-61), the VA research committee at the Roudebush VA Medical Center, and the Institutional Review Board at Louisiana Tech University (HUC 1048). The research is in compliance with the Helsinki Declaration.

Data sources
C4 was a national quality improvement project involving teams from 28 VA medical centers (VAMCs) with at least one team from each of the 21 regional Veterans Integrated Service Networks (VISNs) covering the entire U.S. (see Jackson et al., 2013 [9] for a detailed description of the C4 manual data abstraction.). Briefly, C4 sites used local cancer registries or clinical patient lists to identify a cohort of newly diagnosed CRC patients. C4 sites used an abstraction tool, called the Cancer Care Quality Measurement System (CCQMS), to systematically guide retrospective manual abstraction of approximately 230 data elements from the VA's Computerized Patient Record System (CPRS). These elements were used to evaluate compliance with 25 quality indicators, based upon National Comprehensive Cancer Network (NCCN) guidelines, for colon and rectal cancer treatment. These quality indicators include receipt of curative intent CRC treatment and dates of CRC-related tests received by individual patients (which the current study uses to quantify the accuracy and completeness of VA administrative data for measuring the receipt of this care). Abstractors were cancer registrars, clinical, or administrative staff. The level of training and number of abstractors varied between sites but data variability was limited by a standard computerized data abstraction tool, data definition documents, availability of a centralized help desk, and training of abstractors [9]. A validation of inter-observer health record abstraction accuracy was not performed for the C4 study, but its data abstractors reported a high level of confidence in knowing what data needed to be entered, where to find the data in medical records, and that the instructions were clear [9].

Study population
The study cohort included patients identified during C4 with an incident diagnosis of non-metastatic (AJCC collaborative stage I, II, or III) CRC between January 1, 2004 and October 1, 2006 with complete data collection in which all NCCN quality measures and surveillance tests were evaluated by C4 abstractors. Follow-up care data were collected by October 1, 2007. But, because there was also no documentation of when each manual search of the EHR/QI was conducted, we assumed that the final search for a patient was conducted on the last documented date in the EHR/QI data for that patient. Patients with stage IV CRC were excluded from the current study because these patients may not be eligible for curative intent treatment and the American Society of Clinical Oncology (ASCO) guidelines for CRC surveillance care are more uniformly defined for patients diagnosed with stage I-III CRC [5].

Electronic health record / quality improvement (EHR/QI) data
While all C4 study patients were eligible for data collection for at least 12 months after CRC diagnosis, the EHR/ QI data collection was non-systematic. Data abstractions were not done at a fixed period of time after diagnosis. Therefore, a patient's data were censored at the latest date recorded by the EHR/QI abstractor so only the period of care reviewed by the abstractors was used for analysis.

Administrative data
VA administrative data were drawn from inpatient (acute, extended, observation, and non-VA care) and outpatient files from January 1, 2004 through October 1, 2007. This period spans the EHR/QI data collection from the earliest diagnosis of CRC in EHR/QI to one year after the latest diagnosis of CRC when EHR/QI surveillance data collection ended. For the inpatient file, the main, procedure, and surgery datasets were included. CRC-related care events were identified using relevant Current Procedure Terminology (CPT-4) procedure codes and International Classification of Diseases, 9 th revision, Clinical Modification (ICD-9-CM) diagnosis codes ( Table 1). The ICD-9 codes are used in both inpatient and outpatient administrative files and CPT-4 codes are used in the outpatient administrative data files so both types of codes are required to identify all diagnoses and procedures. VA administrative data are based on information entered into CPRS and coded (e.g., ICD-9 and CPT-4 codes) at the time of the patient encounter. VA administrative data are then stored in a centralized location from across the VA healthcare system. We obtained the VA administrative data for this study by querying the VA national data system (NDS) for records containing any of the codes listed in Table 1 during the study period for the study patients. Demographic variables were identified from the visit dataset in the outpatient file. In addition, demographic characteristics were also drawn from the VA Central Cancer Registry (VACCR) for a separate comparison with the EHR/QI data.

Variables collected
CRC-related care included both curative intent treatments and surveillance tests done after CRC diagnosis to detect CRC recurrence. Receipt of the following curative intent treatments were drawn from the EHR/ QI and VA administrative data: surgical resection, neoadjuvant chemotherapy, and adjuvant chemotherapy, and neoadjuvant radiation, and adjuvant radiation therapies. In VA administrative data, chemotherapy or radiation prior to the date of surgical resection in EHR/QI data were considered neoadjuvant treatment and chemotherapy or radiation after this date was considered adjuvant treatment; the completion of treatment courses was not evaluated.
Receipt of the following surveillance tests [20] were compared between the EHR/QI and VA administrative data: colonoscopy [21], CEA test [5], and physician visits. Two administrative records were considered to be the same event if both events 1) were of the same type and 2) occurred within a specified time window of each other. Colonoscopies and surgeries that occurred within 30 days of each other were considered the same event and CEA tests and physician visits that occurred within 7 days of each other were considered the same event [22]. In these cases, the event that occurred first was considered the "true" event because, for a short time window, the repeat test is likely a completion or followup to the first test and not an independent event. These time windows were applied to every instance of every test.
The demographic variables of date of birth, date of death, gender, race, and marital status were collected from EHR/QI and VA administrative data as well as the VACCR.

Analysis
Patients in the VA administrative data were matched with patients in the EHR/QI by their social security number. For demographic variables, values from VA administrative data were compared to EHR/QI data for accuracy and completeness (see Table 2). The accuracy of a variable was assessed using the percent agreement between the values in EHR/QI and VA administrative data for patients who have non-missing observations in both sources. The completeness of a variable was the percentage of patients with an observation in the EHR/QI data who also had the observation in VA administrative data. While this metric misses EHR/QI false negative tests, it does demonstrate the ability of the VA administrative data to identify outcomes that would also be identified by manual abstraction.
To evaluate the receipt of individual surveillance test events, the accuracy of VA administrative data relative to the EHR/QI data for events of each type of postdiagnosis test was assessed using the positive predictive value relative to the EHR/QI data. The advantage of the positive predictive value is that it assesses the overall  [33] identified the ICD-9 and CPT-4 codes associated with the 4 levels of surgery documented in SEER: biopsy, local excision, bypass surgery, and resection. Only the codes associated with "resection" were used  [23]. The sensitivity for identifying whether a type of CRC-related care was received at any time during the eligible time period was the fraction of patients documented by the EHR/QI data as receiving a type of care who also had documentation of that care in VA administrative data.
All analyses were performed using R 2.15.2.

Results
The EHR/QI project completed chart abstraction for 624 patients with AJCC collaborative stage I, II, or III CRC; as shown in Table 3, the demographic characteristics of the EHR/QI cohort are similar to those of all patients diagnosed with AJCC collaborative stage I-III CRC in the VACCR during the study period. A total of 618 patients (99.0 %) identified in the EHR/QI process were also identified in VA administrative data. A total of 413 of these patients were also present in the VACCR with AJCC collaborative stage I, II, or III cancer. According to the EHR/QI data abstraction, the average age at the diagnosis date was 69.0 years and the population was predominantly male (98.1 %) and white (80.6 %) as shown in Table 3

Percent agreement and completeness of demographic characteristics
For the 618 of 624 patients in EHR/QI data also in the VA administrative data, the agreement between the EHR/QI and VA administrative data for demographic and CRC variables was 98.7 % (610/618) (Table 4). A lower positive predictive value indicated that more events were identified in the administrative data that were not also identified by C4.   The supplemental accuracy measures of the specificity, positive predictive value, and negative predictive value of receipt of a type of care during the eligible period for VA administrative data versus the EHR/QI data are provided in Table 6.

Discussion
The use of cancer registries linked with administrative data represents a widely accepted methodologic approach in cancer health services research to identify cancer treatment and surveillance care that may not be captured by tumor registries alone. Linked SEER-Medicare databases have been used to develop relapse algorithms [24]; evaluate receipt of CRC surgical resection [25], chemotherapy [26], and radiation [27]; and study cancer screening rates [28]. Similarly, linkages between VA cancer registries and VA administrative data could be used to evaluate care received within the VA system if this care is adequately captured in administrative data. The methodologic study presented here evaluated the completeness and accuracy of VA administrative data for CRC-related care by comparing procedures identified in the administrative data  A lower positive predictive value indicated that more events were identified in the administrative data that were not also identified by C4 with those from a manual abstraction of electronic health records performed during the course of a quality improvement collaborative.
There was good agreement between VA administrative and EHR/QI data for demographic variables (91.4 % to 100 % agreement) and CRC treatments received (80.3 % to 92.9 % agreement). The agreements for treatment are similar to what has been observed in Medicare vs. record review for receipt of chemotherapy (88 % colon and 91 % rectal) [26] and radiation (93.5 %) [27]. As previously noted by others [29,30], there is considerable missingness in VA administrative data for race (62.1 % missing). Our result is consistent with what is known about race in VA administrative data; according to the VIReC Technical Report: VA Race Data Quality, race is missing from 33-37 % of VA administrative data records for 2004-2005, although the quality of the data has been improving with time [31]. The missingness of race in the VA cancer registry was considerably lower (2.2 %) and may be considered as the primary source of race for studies involving VA patients with cancer.
VA administrative data were reasonably complete for post-diagnosis colonoscopy (76 %) and physician visit (84.6 %) events but the completeness for the CEA (26.3 %) events was considerable lower. This lower completeness for the CEA tests may be due to the coding of many of these procedures in laboratory data, which was not considered here, or a lower rate of coding laboratory tests in the administrative data. The agreement for receipt of colonoscopy (κ = 0.66) was moderate but lower than the almost perfect agreement that has been observed between Medicare data and health record review (κ = 0.89) [10]. But, as discussed below, the majority of the disagreements in VA data were due to colonoscopies in the VA administrative data that were not documented by the EHR/QI. There were a significant number of procedures of all types identified in VA administrative data but not by the EHR/QI data manual abstraction process. For example, the positive predictive value for colonoscopy after diagnosis was 61.1 %, or 111 of the 285 colonoscopy events in VA administrative data were not in the EHR/ QI data. This difference between administrative claims data and chart abstraction can be explained by either "false-positive" claims or incomplete/inaccurate abstraction. The latter explanation is the more compelling. Specifically, it was unclear whether the EHR/QI data continued to collect surveillance events once the recommended surveillance was achieved; this would lead to apparent false positives in the VA administrative data which would lower the positive predictive value. Also, differences between administrative data and manual abstraction may occur if abstractors misinterpreted coding directions (e.g., difficulty in defining a physician visit was noted as a limitation of the EHR/QI [9]), did not collect all procedures (e.g., local site variations [32] or patient met the indicator criteria so no additional procedures were collected), or simply missed procedures. Because the C4 was undertaken as part of a quality improvement effort rather than a systematic CRC surveillance study, these abstraction errors are likely to be more prevalent in the C4 data than traditional research studies. Previous studies stemming from the C4 have identified the greatest opportunity for improvement as timely surveillance of cancer survivors, however due to missing data, previous studies may have overestimated the size of the quality gap [7,9]. Also of note is that colon and rectal cancer were not differentiated using VA administrative data whichbecause, in the EHR/QI, there are indicator variables for neoadjuvant chemotherapy and neoadjuvant radiation of rectal but not colon cancer as well as no indicator variable for adjuvant radiation [9] likely led to many of the "false-positives" for these variables in VA administrative data.
There are several potential limitations to this study. While the receipt of CRC surveillance procedures was compared, the indication for the procedure (i.e. whether the procedure was for surveillance or diagnostic) was unknown. Without the indication to identify whether procedures were related, time windows were used to identify related procedures but this may combine two unrelated procedures together. There was also no documentation of when each manual search of the EHR/QI was conducted. We assumed that the final search for a patient was conducted on the last documented date in the EHR/QI data for that patient, but it is possible that subsequent searches were conducted that found no new information. We also assumed that all EHR/QI variables were searched for during each manual abstraction, but it is possible that not all variables were searched for during each examination of the health record. Of note, our study represents a comparison of administrative data and electronic health records in the VA system. While these methods may serve as a guide, they should be validated in other healthcare systems. In addition, the VA CRC patient population is predominantly male and diagnosed at an earlier stage than the general population [1] so the study cohort may not be generalizable. Finally, the comparison on patient characteristics in Table 3 was based patients whose data was not missing so there is the potential for selection bias.

Conclusions
In summary, national Veterans Affairs administrative data can be a powerful tool to assess the quality of CRC surveillance care. However, before measures of care quality can be instituted, the accuracy of these data should be examined. Results from this study suggest that VA administrative data are accurate and complete for non-race demographic variables, receipt of CRC treatment, colonoscopy, and physician visits and may be a potential alternative to manual abstraction for the measurement of CRC care. However alternative data sources will be necessary to capture patient race and receipt of CEA tests.