- Research article
- Open Access
- Open Peer Review
External validation of a claims-based algorithm for classifying kidney-cancer surgeries
BMC Health Services Researchvolume 9, Article number: 92 (2009)
Unlike other malignancies, there is no literature supporting the accuracy of medical claims data for identifying surgical treatments among patients with kidney cancer. We sought to validate externally a previously published Medicare-claims-based algorithm for classifying surgical treatments among patients with early-stage kidney cancer. To achieve this aim, we compared procedure assignments based on Medicare claims with the type of surgery specified in SEER registry data and clinical operative reports.
Using linked SEER-Medicare data, we calculated the agreement between Medicare claims and SEER data for identification of cancer-directed surgery among 6,515 patients diagnosed with early-stage kidney cancer. Next, for a subset of 120 cases, we determined the agreement between the claims algorithm and the medical record. Finally, using the medical record as the reference-standard, we calculated the sensitivity, specificity, and positive and negative predictive values of the claims algorithm.
Among 6,515 cases, Medicare claims and SEER data identified 5,483 (84.1%) and 5,774 (88.6%) patients, respectively, who underwent cancer-directed surgery (observed agreement = 93%, κ = 0.69, 95% CI 0.66 – 0.71). The two data sources demonstrated 97% agreement for classification of partial versus radical nephrectomy (κ = 0.83, 95% CI 0.81 – 0.86). We observed 97% agreement between the claims algorithm and clinical operative reports; the positive predictive value of the claims algorithm exceeded 90% for identification of both partial nephrectomy and laparoscopic surgery.
Medicare claims represent an accurate data source for ascertainment of population-based patterns of surgical care among patients with early-stage kidney cancer.
The introduction of partial nephrectomy (also called kidney-sparing surgery) and laparoscopy (also called minimally-invasive surgery) changed the paradigm for surgical management of patients with early-stage kidney cancer in two important ways. First, partial nephrectomy – which achieves equal rates of cure while better preserving long-term renal function – replaced radical nephrectomy (complete excision of the tumor-bearing kidney) as the favored treatment for patients with small renal tumors.[2, 3] Second, for many patients undergoing radical nephrectomy, laparoscopy became the recommended surgical approach because it affords easier convalescence than conventional incisional surgery without compromising long-term cancer control.
Despite their potential benefits to patients, surgeon adoption of partial nephrectomy and laparoscopy has been gradual and assymetric, [5–7] suggesting an opportunity to achieve population-level improvements in the quality of surgical care for patients with kidney cancer. In an effort to understand this protracted dissemination, we recently used linked data from the National Cancer Institute's Surveillance, Epidemiology and End Results (SEER) program and the Centers for Medicare and Medicaid Services (Medicare) to estimate the magnitude of surgeon- versus patient-attributable variance in the use of partial nephrectomy and laparoscopy. For that study, we implemented a Medicare-claims-based algorithm to identify and classify the procedure received by each subject treated surgically for early-stage kidney cancer. The claims-based algorithm included decision-rules to resolve discrepancies between hospital and physician procedure claims (e.g., a hospital claim for partial nephrectomy coupled with a physician claim for radical nephrectomy).
Unlike prostate, lung, and breast cancer, [8, 9]however, no published literature supports the accuracy of claims data for identifying surgical treatments among patients with kidney cancer. Moreover, the initial absence of specific administrative codes for laparoscopic kidney cancer surgeries presented a special challenge for claims-based assessments of evolving surgical practice patterns. The primary goal of this study, therefore, was to validate our Medicare-claims-based algorithm for classifying kidney cancer surgeries by comparing procedure assignments based on Medicare claims with the type of surgery specified in SEER registry data and actual clinical operative reports. Our findings will, in turn, further clarify the integrity of claims data as a tool for studying patterns of surgical care among patients with kidney cancer.
Identification of study cohort and development of claims algorithm
We sought to validate a Medicare-claims-based algorithm for surgical procedure assignment employed in a previously-published study (referred to as the parent study) that used linked SEER-Medicare data to study the utilization of partial nephrectomy and laparoscopic radical nephrectomy among Medicare beneficiaries diagnosed with kidney cancer from 1997 through 2003.SEER is a population-based cancer registry that collects data regarding incidence, treatment, and mortality. The demographic composition, cancer incidence, and mortality trends in the SEER registries are representative of the entire United States population.The Medicare program provides primary health insurance for 97% of the United States population ages ≥ 65 years. Successful linkage with Medicare claims is achieved for > 90% of Medicare patients whose cancer-specific data are tracked by SEER.
From the SEER data included with linked SEER-Medicare files, we identified 6,515 eligible subjects with a new diagnosis of localized/regional, non-urothelial kidney cancer (see Additional File 1). For each case in this preliminary cohort, we searched Medicare inpatient (International Classification of Diseases, 9th revision, Clinical Modification (ICD-9) codes) and physician (American Medical Association Current Procedural Terminology (CPT) and ICD-9 codes) claims for kidney cancer-specific diagnosis and surgical procedure codes (see Additional File 2). Using this approach, we identified an analytic cohort comprising 5,483 patients (84.2% of the preliminary cohort) with Medicare claims specifying the occurrence of cancer-directed surgical therapy (see Additional File 1).
We then applied a claims-based algorithm (defined a priori) to classify the specific type of surgical therapy received by each patient (see Additional File 3). The algorithm included explicit rules for procedure assignment for cases with multiple surgical claims and/or discrepancies between the inpatient and physician claims (e.g., a hospital claim for partial nephrectomy coupled with a physician claim for radical nephrectomy). We used both direct (CPT) and indirect (CPT and ICD-9) codes to identify cases performed laparoscopically (see Additional Files 2 and 3). As part of this algorithm, we also ascribed a laparoscopic approach to patients whose length of hospital stay was ≤ 2 days following radical or partial nephrectomy. By employing this algorithm, we assigned each case to one of four mutually exclusive surgical categories: (1) open radical nephrectomy (ORN); (2) open partial nephrectomy (OPN); (3) laparoscopic radical nephrectomy (LRN); (4) laparoscopic partial nephrectomy (LPN). Because we anticipated small numbers during the specified study interval, we included patients undergoing kidney-sparing, probe-ablative (i.e., cryo or radiofrequency) therapies in the partial nephrectomy cohorts.
Validation of claims algorithm
After obtaining Institutional Review Board approvals, we undertook a two-step approach to assessing the validity of our claims-based algorithm for kidney cancer surgical procedure assignment. First, we assessed the level of concordance between Medicare claims and SEER data for identification of cancer-directed surgery (i.e., ablative surgery, partial nephrectomy, or radical nephrectomy) among 6,515 patients in the preliminary cohort from our parent study (see Additional File 1).[8, 9] We did this by calculating both the observed agreement between the two data sources (i.e., the number of cases in agreement divided by the total number of cases) and the corresponding kappa statistic, which quantifies the degree of agreement, adjusting for chance. We used the following scale for assessing strength of agreement based on kappa statistics: κ = 0.81–1.00 almost perfect agreement; κ = 0.61–0.80 substantial agreement; κ = 0.41–0.60 moderate agreement; κ = 0.21–0.40 fair agreement. For cases where cancer-directed surgery was specified in both Medicare claims and SEER data, we used the same approach to assess the level of agreement between data sources for identification of partial versus radical nephrectomies. Using SEER data as the reference-standard, we additionally calculated the sensitivity, specificity, and positive (PPV) and negative (NPV) predictive values for the claims algorithm (both for the identification of any cancer-directed surgery and for the identification of partial nephrectomies). All statistical testing was completed using computerized software (SAS v9.1, SAS Institute, Cary, NC).
For the second step, we collaborated with the SEER registry for Los Angeles County, the Cancer Surveillance Program (CSP) at the University of Southern California, to carry out a medical-records-based validation of our claims algorithm. In order to retrieve medical records, we provided the CSP with a de-identified list of patient codes for Los Angeles cases included in the analytic cohort for the parent study (n = 549). These 549 cases from the parent study matched with 605 records in the CSP (for patients treated at more than one hospital a report is collected from each). After the initial match – and before sampling cases – CSP staff excluded 14 LA County residents who were diagnosed and/or treated in facilities outside of LA County, 15 cases from facilities that are now closed, and 8 cases whose CSP records did not include a code for kidney cancer surgery. Anticipating that future studies in this area will employ currently-available procedure codes, we decided a priori to stratify the remaining eligible cases according to treatment year (2000–2003 vs 1997–1999); CSP staff then sampled 5 cases from 2000–2003 for every 1 case from 1997–1999. CSP staff requested information for 141 cases before successfully retrieving records for the 120 cases (22% of Los Angeles cases, 2% of the total sample from the parent study) comprising our final validation sample (i.e., medical records were not available for 21 patients). Detailed patient characteristics for each of the previously described cohorts and sub-cohorts are presented in Additional File 4.
Upon receipt of records from treating facilities, CSP chart abstractors identified and photocopied the kidney cancer surgery operative report (or the relevant hospital discharge summary for one case). Next, they removed all patient and/or provider-level identifiers, replaced these with the corresponding unique identifiers from the parent study, and sent the de-identified, photocopied medical records back to UCLA for review and data abstraction. One study author (DCM) reviewed the medical records (operative note for 119 cases, discharge summary for 1 case) to determine the surgical procedure received by each patient (ORN, OPN, LRN, LPN).
In analyses for the subset of cases with medical records data, we considered the procedure specified in the operative note to be the reference-standard. We first calculated the overall concordance or observed agreement rate between the claims algorithm and the medical record (i.e., the number of cases where procedure assignment was the same for the claims algorithm and the operative report divided by the total number of cases in the validation sample). We performed separate calculations for overall procedure assignments (i.e., ORN vs OPN vs LRN vs LPN), partial versus radical nephrectomy, and open versus laparoscopic surgical approach. Finally, we calculated the sensitivity, specificity, PPV and NPV of the claims algorithm for identification of open surgery versus laparoscopy, and partial versus radical nephrectomy.
Among 6,515 Medicare beneficiaries with a new diagnosis of localized/regional, non-urothelial kidney cancer, Medicare claims and SEER data identified 5,483 (84.1%) and 5,774 (88.6%) patients, respectively, who underwent cancer-directed surgery. For cases with both hospital and physician claims specifying the occurrence of cancer-directed surgery, we observed 98% agreement (κ = 0.88, 95% CI 0.86 – 0.90) for classification of partial versus radical nephrectomy, and 94% agreement (κ = 0.45, 95% CI 0.40 – 0.50) for classification of laparoscopic versus open surgery.
As presented in Table 1, agreement between the two data sources exceeded 90% (κ = 0.69, 95% CI 0.66 – 0.71). The sensitivity of the claims algorithm for identifying the occurrence of kidney cancer-directed surgery was 93.3% (95% CI 92.7%–94.0%), with a PPV of 98.3% (95% CI 97.9%–98.6%) (Table 1). Among the 5,389 cases where both the claims algorithm and SEER data specified the occurrence of cancer-directed surgery, we observed 97% agreement between data sources for classification of partial versus radical nephrectomy (κ = 0.83, 95% CI 0.81 – 0.86) (Table 2).
Among the 120 subjects whose medical records were reviewed, 100 (83%) and 20 (17%) underwent surgery in 2000–2003 and 1997–1999, respectively. The claims-based algorithm correctly classified the specific surgical procedure for 116 of 120 cases (observed agreement = 96.7%) (Table 3). The four misclassified cases include the following: 1) one case classified as an open radical nephrectomy was actually a laparoscopic radical nephrectomy; 2) one case classified as a laparoscopic radical nephrectomy was actually an open radical nephrectomy (although it had started as a laparoscopic case before the surgeon converted to conventional open surgery); 3) one case classified as a laparoscopic radical nephrectomy was actually a laparoscopic partial nephrectomy; and 4) one case classified as a laparoscopic partial nephrectomy was actually a percutaneous ablative procedure (for analytic purposes we assigned this case to the open partial nephrectomy group since the misclassification was based only on surgical approach). For the single case where we reviewed only the hospital discharge summary, both the claims algorithm and the medical record specified an open partial nephrectomy.
Tables 4 and 5 compare case classifications based on the claims algorithm and operative reports for partial vs radical nephrectomy and for laparoscopic vs open surgery, respectively. The PPV of the claims algorithm exceeded 90% for identification of both partial nephrectomy (Table 4) and laparoscopic (Table 5) surgeries.
Administrative and tumor registry data are useful tools for researchers interested in the delivery and outcomes of cancer care. For instance, linked SEER-Medicare data can be used to study population-based trends in cancer screening, diagnosis, treatment, and survivorship among Medicare beneficiaries in the United States. In this study, we evaluated the validity of Medicare claims for determining the type of surgical therapy received by patients with early-stage kidney cancer. We report three principal findings. First, we observed substantial agreement (κ = 0.69) between Medicare claims and SEER registry data for the identification of cancer-directed surgery among patients with early-stage kidney cancer. Second, our claims-based algorithm assigned the correct surgical procedure (ORN, OPN, LRN, or LPN) for 97% of the subset of cases for whom we performed medical record review. Third, our claims algorithm demonstrated a high positive predictive value for identification of partial nephrectomy (PPV 94%) and laparoscopic kidney cancer surgeries (PPV 92%). The consequent inference is that Medicare claims represent an accurate data source for ascertainment of population-based patterns of surgical care among patients with early-stage kidney cancer.
Our findings augment existing literature that describes the fidelity of claims data for evaluating practice patterns in surgical oncology.[8, 13] Prior work has demonstrated good agreement between SEER and Medicare claims for identification of patients undergoing major surgical resections for breast, colorectal, endometrial, lung, pancreatic or prostate cancer. Moreover, medical record reviews have confirmed the high sensitivity and positive predictive value of Medicare claims for detection of patients undergoing inpatient colectomy, mastectomy, or hysterectomy.Data from the current study suggest that Medicare claims can also be used to classify accurately patients undergoing partial or radical nephrectomy for kidney cancer (by either an open or laparoscopic approach). The ability to distinguish laparoscopic from open surgical cases is particularly relevant since these data are not routinely available from the SEER registries. Accordingly, medical claims (available from Medicare and other payers) represent a valid and reliable source of data for monitoring – at a national level – the impact of ongoing initiatives that seek to increase surgeons' use of partial nephrectomy and laparoscopy.
Our study has several limitations. First, the small sample size for the medical record validation may raise concerns about the generalizability of our findings for the greater population of Medicare beneficiaries with kidney cancer. Second, because CSP abstractors were unable to locate medical records for some patients, it is possible that we have overestimated the concordance between the claims algorithm and surgeon operative notes. Third, energy ablative therapies (i.e., cryosurgery and radiofrequency ablation) are now used more frequently as alternatives to partial nephrectomy. Because we anticipated (and observed) very small numbers of these cases in the parent study, we included ablative therapies with the partial nephrectomies. In the future, updated claims algorithms may be needed to separately identify patients treated with ablative therapies. Finally, the generalizability of our results may be limited by a sample that includes only patients ≥ 66 years of age at the time of diagnosis. We do not believe, however, that patterns of treatment are systematically different for 65-year-olds versus older Medicare beneficiaries ages 66 and older.
Despite these limitations, this study confirms a high level of agreement between Medicare claims and clinical operative reports for the identification of specific kidney cancer surgeries. Claims data, therefore, may be considered a valid substrate for studying patterns of surgical care among patients with kidney cancer.
Open Radical Nephrectomy
Open Partial Nephrectomy
Laparoscopic Radical Nephrectomy
Laparoscopic Partial Nephrectomy
Cancer Surveillance Program (Los Angeles)
Surveillance, Epidemiology, and End Results
Positive Predictive Value
Negative Predictive Value
Novick AC: Laparoscopic and partial nephrectomy. Clinical Cancer Research. 2004, 10: 6322-6377. 10.1158/1078-0432.CCR-050003.
Fergany AF, Hafez KS, Novick AC: Long-term results of nephron sparing surgery for localized renal cell carcinoma: 10-year followup. J Urol. 2000, 163: 442-445. 10.1016/S0022-5347(05)67896-2.
Huang WC, Levey AS, Serio AM, Snyder M, Vickers AJ, Raj GV, et al: Chronic kidney disease after nephrectomy in patients with renal cortical tumours: a retrospective cohort study. Lancet Oncol. 2006, 7: 735-740. 10.1016/S1470-2045(06)70803-8.
Dunn MD, Portis AJ, Shalhav AL, Elbahnasy AM, Heidorn C, McDougall EM, et al: Laparoscopic versus open radical nephrectomy: a 9-year experience. J Urol. 2000, 164: 1153-1159. 10.1016/S0022-5347(05)67131-5.
Miller DC, Hollingsworth JM, Hafez KS, Daignault S, Hollenbeck BK: Partial nephrectomy for small renal masses: an emerging quality of care concern?. J Urol. 2006, 175: 853-857. 10.1016/S0022-5347(05)00422-2.
Miller DC, Taub DA, Dunn RL, Wei JT, Hollenbeck BK: Laparoscopy for renal cell carcinoma: diffusion versus regionalization?. J Urol. 2006, 176: 1102-1106. 10.1016/j.juro.2006.04.101.
Miller DC, Saigal CS, Banerjee M, Hanley J, Litwin MS: Diffusion of surgical innovation among patients with kidney cancer. Cancer. 2008, 112: 1708-1717. 10.1002/cncr.23372.
Cooper GS, Virnig B, Klabunde CN, Schussler N, Freeman J, Warren JL: Use of SEER-Medicare data for measuring cancer surgery. Med Care. 2002, 40 (Suppl 8): 43-48.
Cooper GS, Yuan Z, Stange KC, Dennis LK, Amini SB, Rimm AA: Agreement of Medicare claims and tumor registry data for assessment of cancer-related treatment. Med Care. 2000, 38: 411-421. 10.1097/00005650-200004000-00008.
Hankey BF, Ries LA, Edwards BK: The surveillance, epidemiology, and end results program: a national resource. Cancer Epidemiology, Biomarkers & Prevention. 1999, 8: 1117-21.
Warren JL, Klabunde CN, Schrag D, Bach PB, Riley GF: Overview of the SEER-Medicare data: content, research applications, and generalizability to the United States elderly population. Med Care. 2002, 40 (Suppl 8): 3-18.
Fleiss JL: Statistical Methods for Rates and Proportions. 1981, New York, John Wiley & Sons
McClish DK, Penberthy L, Whittemore M, Newschaffer C, Woolard D, Desch CE, et al: Ability of Medicare claims data and cancer registries to identify cancer cases and treatment. Am J Epidemiol. 1997, 145: 227-233.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6963/9/92/prepub
This work was supported by the National Institute of Diabetes and Digestive and Kidney Diseases (N01-DK-1-2460), the Urologic Diseases in America Project. David C. Miller, MD, MPH is additionally supported by grants from the National Cancer Institute (NIH-1-F32 CA123819-01), the American Cancer Society (PF CPHPS-112124) and the American Urological Association Foundation Research Scholar Program.
The collection of cancer data by the Los Angeles Cancer Surveillance Program is supported by the California Department of Health Services as part of the statewide cancer reporting program mandated by California Health and Safety Code Section 103885; the National Cancer Institute's Surveillance, Epidemiology and End Results Program under contract 00261200306D261035139 and NCI Control N01-PC-35139 awarded to the University of Southern California, and contract N02-PC- 15105 awarded to the Public Health Institute; and the Centers for Disease Control and Prevention's National Program of Cancer Registries, under agreement U55/CCR921930 awarded to the Public Health Institute. The ideas and opinions expressed herein are those of the author(s) and endorsement by the State of California, Department of Health Services, the National Cancer Institute, and the Centers for Disease Control and Prevention or their contractors and subcontractors is not intended nor should be inferred.
The funding bodies had no role in the study design; in the collection, analysis, and interpretation of data; in the writing of the manuscript; or in the decision to submit the manuscript for publication.
This study used the linked SEER-Medicare database. The interpretation and reporting of these data are the sole responsibility of the authors. The authors acknowledge the efforts of the Applied Research Program, NCI; the Office of Research, Development and Information, CMS; Information Management Services (IMS), Inc.; and the Surveillance, Epidemiology, and End Results (SEER) Program tumor registries in the creation of the SEER-Medicare database.
The authors declare that they have no competing interests.
DM conceived of the study, participated in its design, coordination, execution and analyses, and drafted the manuscript. CS conceived of the study, participated in its design, coordination and analyses, and helped to draft the manuscript. JW participated in the study design and analyses, and helped to draft the manuscript. ML participated in the study design and coordinated ascertainment of cases from the LA Tumor Registry. DD participated in the study design and coordination, supervised ascertainment of cases from the LA Tumor Registry, and helped to draft the manuscript. MB participated in the study design and statistical analyses and helped to draft the manuscript. JL and JH performed statistical analyses for the study. ML conceived of the study, participated in its design, coordination and analyses, and drafted the manuscript. All authors read and approved the final manuscript.