Because they are easily accessible and provide readily-available information on a large number of patients receiving care in real-world clinical practice, private and public healthcare claims have been used extensively to date in health-economic and outcomes research. Such databases, however, are not without limitations, since they are not designed to capture certain components of care (e.g., laboratory results) and thus may lack data on potentially clinical parameters, which oftentimes are important for case-ascertainment. Inaccuracies in case-ascertainment may confer unknown biases in the estimation of study results.
In this study, we evaluated the accuracy of operational definitions previously used to identify chemotherapy-induced hospitalization FN in healthcare claims databases. We found that PPV exceeded 80%--the assumed minimum acceptable threshold--for the definitions including the diagnosis code for neutropenia in the principal position (87%) and diagnosis codes for neutropenia plus fever (100%), and that PPV was close to the acceptable threshold for the definition including a diagnosis code for neutropenia in any position (77%). Among these definitions, sensitivity was highest for the last one (67%), although this figure is somewhat lower than that (81%) reported for a similar definition in the aforementioned evaluation of older adults with non-Hodgkin’s lymphoma . While sensitivity was improved in our study by including codes for fever and infection (87%), PPV (35%) was lower. We note, however, that this claims-based definition has most often been employed in comparative effectiveness studies of CSF agents, in which the focus was not on FN per se but all neutropenic complications (including severe neutropenia, afebrile infections in neutropenic patients, as well as FN) resulting in hospitalization that may be prevented with prophylaxis. When modifying the gold standard (i.e., ANC was < 1.5 × 109/L) to capture additional neutropenic complications--irrespective of whether the patient was febrile or received antimicrobials--PPV (45% [38–53]) improved, but sensitivity was lower (81% [74–88]).
Because no one definition was found to be associated with both high PPV and high sensitivity, the selection of a specific definition for identifying hospitalized FN should be guided by the potential impact of misclassification on study outcomes. For example, in studies evaluating the comparative effectiveness of CSFs in clinical practice, a premium should be placed on a high PPV and limiting the inclusion of false-positive cases. Since CSFs can only reduce the risk of infections associated with true neutropenia, a low PPV would result in a dilution of the estimated effectiveness of CSF prophylaxis and estimated relative differences in effectiveness among agents being studied. A high PPV alone may not be sufficient, however, since if sensitivity is low, analyses may not be adequately powered to evaluate study objectives. For other studies, however, such as those in which the outcome is the cost of FN hospitalization, some misclassification may be acceptable to the extent that there are offsetting effects (e.g., if sensitivity and PPV are comparable; and either costs of false-positives, false-negatives, and true-positives are similar or costs of false-negatives and false-positives differ in the opposite direction by the same degree). The same also may be true for studies characterizing the clinical risk (i.e., incidence) of FN, for which--given the limitations of the alternative definitions--the goal should be to select one that is adequately sensitive and/or yields a balance between the number of patients who are misclassified (i.e., as false-positives and false-negatives). It therefore is essential to consider the importance of the individual test and performance characteristics of a measure, relative to the objectives of the study, before selecting one definition for use, and we recommend considering several alternative definitions.
Our study has a number of limitations. First, we excluded from the study population all patients who were not hospitalized in a GHS facility (because ANC data were not available for these patients) and those hospitalized in a GHS facility who did not have ANC data proximal to hospital admission. Only 35% of patients in the source population who were hospitalized (including patients treated inside and outside GHS) had ANC data during the study time period and thus could be included in the study population. For these reasons, the number of patients in the study population was limited to those treated at a GHS facility and who had ANC data. To the extent that excluded patients actually had FN and differed from the study population in important respects, our results may not be reflective of the larger population. In addition, because patients who had ANC data proximal to hospital admission may be selectively different from those who did not (because of, for example, a history of FN), our study population may be “enriched” and subject to verification bias. Second, even for some patients included in the study population, critical data--especially body temperature--was missing and thus we used a gold standard based on ANC and body temperature or administration of antibiotic or antiviral therapy. Because we used an “imperfect” gold standard, however, and thus patients’ true status in terms of FN was unknown, the presence of errors in the classification of patients by the gold standard could result in biased estimates of PPV, sensitivity, and other markers. Third, since the data for the study come from a single health plan and coding practices may vary across plans and geographic regions of the country, the generalizability of our findings is uncertain. Fourth, the accuracy of claims-based definitions in identifying FN may vary across subgroups of patients (e.g., those defined on the basis of type of cancer or regimen), and thus caution should be employed when generalizing study results to specific populations; such subgroup analyses, however, were not possible given the relatively small size of the study population. Fifth, while acknowledging that culture data are less than perfectly sensitive in identifying infection, the lack of such information in the data extract precluded any evaluation of the presence of infection in true FN patients based on positive cultures. Finally, because chemotherapy-induced FN almost never occurs earlier than the fifth day of a chemotherapy cycle, and because some of the aforementioned prior studies did not consider hospitalizations during this period, only hospitalizations occurring on or after the fifth day of a chemotherapy cycle were included. Results were not sensitive to this methodological feature as <5% of all hospitalizations occurred during the first five days of a chemotherapy cycle.