Detection of adverse drug events in e-prescribing and administrative health data: a validation study

Background Administrative health data are increasingly used to detect adverse drug events (ADEs). However, the few studies evaluating diagnostic codes for ADE detection demonstrated low sensitivity, likely due to narrow code sets, physician under-recognition of ADEs, and underreporting in administrative data. The objective of this study was to determine if combining an expanded ICD code set in administrative data with e-prescribing data improves ADE detection. Methods We conducted a prospective cohort study among patients newly prescribed antidepressant or antihypertensive medication in primary care and followed for 2 months. Gold standard ADEs were defined as patient-reported symptoms adjudicated as medication-related by a clinical expert. Potential ADEs in administrative data were defined as physician, ED, or hospital visits during follow-up for known adverse effects of the study medication, as identified by ICD codes. Potential ADEs in e-prescribing data were defined as study drug discontinuations or dose changes made during follow-up for safety or effectiveness reasons. Results Of 688 study participants, 445 (64.7%) were female and mean age was 64.2 (SD 13.9). The study drug for 386 (56.1%) patients was an antihypertensive, and for 302 (43.9%) an antidepressant. Using the gold standard definition, 114 (16.6%) patients experienced an ADE, with 40 (10.4%) among antihypertensive users and 74 (24.5%) among antidepressant users. The sensitivity of the expanded ICD code set was 7.0%, of e-prescribing data 9.7%, and of the two combined 14.0%. Specificities were high (86.0–95.0%). The sensitivity of the combined approach increased to 25.8% when analysis was restricted to the 27% of patients who indicated having reported symptoms to a physician. Conclusion Combining an expanded diagnostic code set with e-prescribing data improves ADE detection. As few patients report symptoms to their physician, higher detection rates may be achieved by collecting patient-reported outcomes via emerging digital technologies such as patient portals and mHealth applications. Supplementary Information The online version contains supplementary material available at 10.1186/s12913-021-06346-y.


Background
Drug expenditures have been growing at a faster rate than all other costs in health care [1,2]. While modern drug therapy plays an important role in managing health problems, adverse drug events (ADEs) are frequent and costly, with up to 16% of emergency department (ED) visits and 7% of hospital admissions being medicationrelated [3][4][5][6]. The frequency of these events can be explained in part by established processes for drug approval, which test drugs in tightly controlled settings and in a limited number of patients who infrequently represent those typically prescribed the drug after approval. In addition, once drugs have entered the market, they are often prescribed off-label for conditions for which they have not been approved [7][8][9][10], which previous research has shown increases the risk of ADEs [11]. Consequently, robust methods of post-marketing surveillance have emerged as a requirement to monitor the safety and effectiveness of medications after they have been approved for sale [12][13][14].
Methods of post-marketing drug surveillance include voluntary systems for spontaneous reporting of ADEs [15] and prescription event monitoring [16], both of which suffer from systematic under-reporting [16][17][18][19]. The data gaps in these reporting systems have been increasingly supplemented with computerized administrative health data, which are timelier, more reliable, and easier to collect. Diagnostic codes from physician billing data are often used to detect ADEs, with International Classification of Disease (ICD) code sets developed for this purpose [20][21][22][23][24][25][26][27][28][29][30]. However, most of these codes have only been validated in the in-patient hospital setting [21,22,[24][25][26]31]. Few studies have evaluated the accuracy of ICD code sets for ADE detection in ED and outpatient settings, where the majority of medications are prescribed and resulting ADEs diagnosed [32]. Moreover, code sets evaluated in ED and outpatient settings have not been very helpful, with a reported sensitivity of only up to 28% in detecting ADEs [20,23,27,28]. Many codes are external cause codes that describe precise causes of ADEs (e.g. ICD-10 Y40-Y59 drugs, medication, and biological substances causing adverse effects in therapeutic use) or codes that relate to drug-induced diagnoses (e.g. ICD-10 G44.4 drug-induced headache). ADE detection which relies on these codes requires that the ADE is both recognized by the busy clinician and coded and recorded in administrative health data, activities which are sub-optimally achieved [4,20,33,34].
In contrast, automated systems that search electronic medical notes for phrases representing known adverse effects of specific drug classes are much better at detecting ADEs [23,35]. This approach relies on the documentation of patient symptoms, which is routinely done, rather than on a clinician's recognition of an ADE. To develop an enhanced ADE detection system for use in administrative health data, we adapted the strategy of searching for drug side effects by creating an expanded, therapeutic class-targeted ICD code set. This code set includes side effects of specific therapeutic drug classes (e.g. sexual dysfunction for antidepressants), in addition to the conventional external cause codes and druginduced diagnoses.
One missed opportunity for ADE detection is the use of data from computerized prescribing systems. These systems are ideal for identifying when new drugs are started, when doses are changed, or when drugs are discontinued because of adverse effects or lack of therapeutic effectiveness [36,37]. Moreover, electronic prescribing systems are commonly used, if not required, in many jurisdictions and the majority of these systems provide highly structured drug data. To further capture ADEs recognized by physicians but underreported in administrative data, we took advantage of an e-prescribing system that requires physicians to record the reasons for changing dose or discontinuing a drugdata that have previously been shown to be accurately recordedto determine if e-prescribing data could increase ADE detection.
We assessed the accuracy of an expanded administrative data ICD code set and e-prescribing data, individually and combined, for ADE detection among patients receiving a newly prescribed medication in primary care.

Research design
We conducted a prospective cohort study to assess the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of using treatment change orders in e-prescribing data and diagnostic codes in administrative health data to detect adverse events attributable to medications prescribed in a primary care setting. We assessed the accuracy of ADE detection among new users of antidepressant and antihypertensive medications, as these are among the mostly frequently prescribed medication classes and account for a significant proportion of adverse events [8,23,[38][39][40][41]. New users were identified through the e-prescribing system, which had information on all drugs prescribed and dispensed in the past year. New users were then followed for a period of 2 months, starting from the date the medication was first dispensed. Gold standard ADEs were defined as new or worsening symptoms reported by patients and adjudicated as medication-related by a clinical expert using the Naranjo criteria [42].

Study context
This study was conducted in Quebec, Canada, where the healthcare system provides complete coverage for medical services. Prescription drug coverage in Quebec is mandatory and provided by the provincial health insurance agency (Régie de l'assurance maladie du Québec -RAMQ) to individuals who are above the age of 65, are welfare recipients, or are not covered by their employer.
Patients were recruited into the study between 2007 and 2015 through MOXXI (Medical Office of the XXIst Century), an experimental electronic health record platform used by 110 primary care physicians for approximately 90,000 of their patients [43]. The system, which integrates information drawn in real-time from administrative health databases into patient profiles (e.g. medications dispensed, current and past health problems, recent ED visits and hospital admissions), also functions as an e-prescribing tool. Physicians using MOXXI must enter a treatment indication for every new medication prescribed by selecting an item from a drop-down listing of all known on-and off-label conditions for that drug. If a medication is discontinued or its dose is changed, the prescriber is required to record a reason for their action by selecting an option from a drop-down listing that includes: adverse drug reaction, ineffective treatment, drug interaction, allergic response, adjusting dose to optimize treatment, error in prescribing, incorrect medication dispensed, end of treatment, and substitution for less expensive drug (Fig. 1). The medication change and associated reason are also printed on the prescription and recorded in the treatment history of the health problem for which the drug was prescribed. A previous study showed that MOXXI had a specificity of 99.7% and sensitivity of 67.0% in documenting treatment discontinuations and dose changes, as well as high concordance between treatment change reasons recorded in the application and those obtained from chart-facilitated physician interview [37].

Study population
Patients participating in MOXXI were eligible to be included in this study if they 1) had prescription drug insurance through the RAMQ, 2) were prescribed and dispensed an antidepressant or antihypertensive medication by a MOXXI physician between 2007 and 2015, and 3) had no prior prescription or record of dispensed medication for that same drug in the preceding 12 months. We chose to focus on incident users as adverse events are more likely to occur shortly after starting or stopping therapy [44][45][46][47].
After enrollment in the study, patients were followed for 2 months, starting from the date the antidepressant or antihypertensive medication was first dispensed. During this period, data were collected on medical services received, dose changes or discontinuations of the study drug, and patient-reported outcomes.

Data sources Administrative health data
Real-time patient demographic, drug, and clinical information was gathered using MOXXI's dedicated, secure interface to RAMQ databases, with records linked using patients' unique health insurance numbers. Patient demographic information included date of birth, sex, and date of death (if applicable). Drug information included pharmacy claims data for all drugs dispensed to a patient following the visit in which they received the new prescription, as well as in the preceding 12 months. This included the date of dispensing, drug identification number, quantity dispensed, and prescription duration. The validity of these drug data has been previously demonstrated [48].
Medical fee-for-service billing data for services provided by any licensed physician in the province to study patients during the 2-month follow-up period and the 12 months prior to enrollment were also obtained via the RAMQ interface. These data included the date of service delivery, the service location (critical care, inpatient ward, ED, outpatient clinic, private office), and the ICD-9 diagnosis for the visit. In addition, data from the provincial hospitalization database were retrieved and linked to study patients by unique medicare number. For each hospitalization during the follow-up period and the 12 months prior, we retrieved admission and discharge dates, ICD-10 codes of the principal discharge

MOXXI electronic prescribing data
Data from the MOXXI e-prescribing tool were used to identify patients with new prescriptions of antihypertensive and antidepressant medications between 2007 and 2015. Prescription information included the patient's unique health insurance number, date of the prescription, drug identification number, treatment indication, quantity, duration, directives, and number of refills. For each patient with a new prescription for one of the study drugs, we also retrieved data on subsequent discontinuations or dose changes made during the 2 months following first dispensation of the drug, along with the documented reason(s) for the change.

Patient interview data
A modified version of the Australian adverse reaction and drug event report was used to collect patient feedback on adverse events (Additional file 1: Patient Questionnaire) [49]. The interview was administered via telephone by a trained research assistant within 3 weeks following the first dispensation of the study drug. Patients were first asked to report any new or worsening symptom or health problem experienced in the past 3 weeks. Standardized system-related questions were then used to identify any body system changes patients may have experienced since starting the drug, and drugspecific standardized questions were used to determine if patients had experienced any of the known effects of the study drug.

Patient characteristics
Demographic and clinical characteristics used to characterize the study population included age, sex, the therapeutic class of the study drug, the treatment indication of the study drug, the number of concurrent medications patients were taking, and the presence or absence of the 17 conditions included in the Charlson Comorbidity Index [50,51]. The latter was determined using diagnostic codes for medical services that patients received in the 12 months prior to enrollment in this study.

Adverse drug event (ADE) definitions
ADEs in administrative health data: Expanded therapeutic class-targeted code set & standard code set The expanded, therapeutic class-targeted ICD code set was composed of ICD codes for known adverse effects of antihypertensive and antidepressant medications (e.g. sexual dysfunction for antidepressants), as well as external cause codes specifying an adverse reaction to those medication classes. Adverse effects for every drug group have been defined by the Oregon Drug Effectiveness Review Project [52] and those relevant to antidepressants and antihypertensives were mapped to corresponding ICD-9 and ICD-10 codes (Additional file 2: Appendices 1-2).
The narrower, standard code set consisted of diagnostic codes included in previously reported ADE code sets [20,23,27,28]. This code set was comprised of external cause codes and drug-induced diagnoses specifying an adverse reaction to antihypertensive and antidepressant medications (Additional file 2: Appendix 3).
To assess ADEs using the standard and expanded code sets, ICD codes recorded in medical services claims for all physician visits (office, ED, outpatient, inpatient) and hospital admissions during the follow-up period were retrieved. Patients were classified as having a potential ADE if one or more ICD codes from the a) standard code set or b) expanded code set were documented as a reason for the physician visit or hospital admission.

ADEs in MOXXI electronic prescribing data
Potential ADEs in MOXXI e-prescribing data were defined as study drug discontinuations or dose changes made by the study physician during the follow-up period for safety (adverse drug reaction, allergy, drug interaction) or effectiveness reasons (treatment ineffective). Treatment changes made for 'ineffectiveness' reasons could indicate underlying problems related to medication safety, such as interactions, and because patients often discontinue medication due to adverse reactions [53][54][55].

Gold standard assessment of ADEs
New or worsening symptoms and health problems reported during patient interviews were reviewed by a clinical expert. The expert was also presented with information regarding other medications the patient was prescribed and dispensed, the patient's medical problem list as documented in MOXXI, and medical services received by the patient (ED visits and hospital admissions). The Naranjo criteria for ADE assessment were used to assess the likelihood that the symptom(s) reported during the interview were related to the newly started therapy [42]. The Naranjo algorithm assesses the presence or absence of ten criteria related to the adverse event, each of which is given a specific weight based on its relative importance. The sum of weights is then used to determine an overall score, which places the probability that the event was medication-related into one of four categories: doubtful, possible, probable, or definite. Gold standard ADEs were defined as patient-reported adverse events whose Naranjo score was classified as probable or definite.

Outcome definitions
Sensitivity was defined as the proportion of true ADEs, as defined by the gold standard, which were correctly identified using e-prescribing and/or administrative health data. Specificity was defined as the proportion of true negatives that were identified as such using eprescribing and/or administrative health data. The PPV was defined as the proportion of potential ADEs identified using e-prescribing and/or administrative health data that were true ADEs, and the NPV as the proportion of patients who did not have an ADE based on eprescribing and/or administrative health data who were true negatives.

Statistical analysis
Descriptive statistics were used to characterize the study population and to estimate ADE rates, overall and by therapeutic class, using the gold standard definition, standard and expanded ICD code sets, and MOXXI e-prescribing data. The sensitivity, specificity, PPV and NPV of using administrative health data and e-prescribing data, individually and combined, to detect ADEs were also estimated, overall and by therapeutic class. 95% confidence intervals (CIs) around these estimates were constructed using the exact method for binomial proportions. All analyses were conducted using SAS version 9.4 (SAS Institute Inc., Cary, NC, USA).

Study cohort
Between July 2007 and June 2015, 5012 patients had a new prescription for an antihypertensive or antidepressant medication (Fig. 2). Of these, 1133 (22.6%) had insufficient RAMQ prescription drug coverage during the study period, 1393 (27.8%) declined to participate, and 1215 (24.2%) were refused enrollment by the MOXXI physician. A total of 1271 patients consented and were enrolled in the study. Compared to patients who
More than half (51.5%) of patients in this study had at least one comorbidity, with a larger proportion of antihypertensive users having a comorbidity (56.2%) compared to antidepressant users (45.4%) ( Table 1). The most common comorbidities included chronic pulmonary disease (21.7%), diabetes (without complications) (15.3%), and renal disease (9.6%). In the one month following the visit in which the study drug was prescribed, patients were dispensed an average of 3.7 (SD 2.9) medications other than the study drug (Table 1). There was no difference between antidepressant and antihypertensive users in the number of medications dispensed.
Using the gold standard definition, 114 (16.6%) study patients experienced an adverse event attributable to the study drug during the study follow-up period ( Table 2).
Assessing the accuracy of these approaches by therapeutic class revealed that while the sensitivity of eprescribing data was higher among antidepressant users compared with antihypertensive users (13.5% vs 2.5%, respectively), the reverse was true regarding the sensitivity Defined as the number of medications besides the study drug that were dispensed in the 1 month following the visit in which the study drug was prescribed of our expanded ICD code set (5.4% vs 10.0%, respectively) ( Table 3).

Symptoms reported to physicians
Interview data indicate that 73% of study patients who reported new or worsening symptoms and health problems during their interview did not inform their doctor of these issues. Compared to patients who reported all their symptoms to their physician, patients who did not report symptoms experienced less severe symptoms (27.0% of symptoms were rated severe or very severe vs 51.9% of symptoms), were younger (mean age 61.5 vs 66.3), and were more likely to be male (29.6% vs 25.3%) (Additional file 2: Appendices 8-9). We also found that the sensitivity of our combined approach was improved almost two-fold when the analysis was restricted to patients who had reported symptoms to their physician (25.8% vs 14.0%).

Discussion
In this validation study, we found that treatment change orders in e-prescribing data and diagnostic codes in administrative health data, individually and combined, enhanced the identification of ADEs caused by antidepressant and antihypertensive medications prescribed in a primary care setting compared to the standard code set. The higher sensitivity of our expanded ICD code set compared to the standard code set suggests that including diagnostic codes for known adverse effects of therapeutic drug classes improves ADE detection. However, the sensitivity of the expanded code set was still low, as even a comprehensive diagnostic code set will fail to Gold standard assessment of ADEs was based on patient interview data, adjudicated by a clinical expert to assess causality based on the Naranjo criteria. b Potential ADEs in electronic prescribing data were defined as study drug discontinuations or dose changes due to safety or effectiveness reasons. c Potential ADEs in administrative health data using the expanded ICD code set were defined as a physician visit, ED visit, or hospital admission during follow-up for which the recorded ICD code was (1) a relevant external cause code or (2) an adverse effect of the study drug. d Potential ADEs in administrative health data using the standard ICD code set were defined as a physician visit, ED visit, or hospital admission during follow-up for which the recorded ICD code was (1) a relevant external cause code or (2) an adverse effect of the study drug and (3) was included in previously validated code sets. capture ADEs if ICD codes are not reported in administrative health data. To fill this gap, we supplemented our expanded code set with e-prescribing data on change or discontinuation orders for safety or effectiveness reasons. Although this doubled the sensitivity, the sensitivity of our combined approach did not surpass 14.0%. Our results can be explained, at least in part, by low rates of symptom reporting by patients to physicians. Indeed, 73% of study patients indicated in their interviews that they had not reported any of their symptoms to a physician. This suggests that e-prescribing and administrative health data alone may not detect mild ADEs for which patients do not seek medical attention. Even when patients do seek medical attention, there is discordance between the symptoms patients report and those documented in the electronic health record [56], which suggests that physician-reported data may not provide an accurate picture of patients' symptoms. This could explain why, even when restricting analysis to patients who had reported symptoms to their physicians, the sensitivity of our combined approached remained relatively low. Thus, a potential alternative for ADE detection may lie in patient-reported outcomes (PROs), which have emerged as a necessary and increasingly promoted means of obtaining information from patients regarding their health status, including adverse events [57]. PROs have potential as a feasible and effective approach for ADE detection [58,59], particularly if integrated into patient-centered digital health tools such as patient portals, mobile health (mHealth) applications, or even online reporting sites [60].
Interestingly, our subgroup analyses indicated that the sensitivity of treatment change orders in e-prescribing data was higher among antidepressant users compared with antihypertensive users. One potential explanation is that patients presenting to primary care physicians with an ADE related to new antidepressant use may be more likely to have their medication discontinued or changed. In contrast, patients with ADEs related to antihypertensive use may be prescribed an additional medication that targets the ADE, creating a medication cascade [61]. This is supported by a closer look at diagnostic codes of potential ADEs in administrative health data, the most common of which among antihypertensive users were for hyperglycemia. Physicians faced with a hyperglycemic patient may be more likely to prescribe a hypoglycemic medication than discontinue the antihypertensive [62]. This could explain the lower sensitivity of e-prescribing data among antihypertensive users.
Although our expanded, therapeutic class-targeted ICD code set was more sensitive than the standard code set in detecting ADEs in our cohort, its accuracy was similar to previously reported estimates for narrower code sets. In a 2001 study conducted in an outpatient setting, Honigman reported a PPV of 2% for the use of ICD-9 codes in identifying ADEs [23]. Similarly, Field found that computer-generated signals that included ICD-9 codes flagged true ADEs 7% of the time [27]. Results of both studies suggested low sensitivities of diagnostic codes in detecting ADEs, as did an investigation by Hohl in 2013 of the sensitivities of two ICD-10 code sets in detecting ADEs in the emergency department (6.8 and 28.1%) [20]. Despite using codes similar to those in our narrow standard code set, the sensitivities reported in these studies are more similar to that of our expanded code set. This may be explained by the data sources from which ICD codes were obtained in these studies, which differ from those used in our investigation. Electronic medical record data were used by the Field and Honigman studies, whereas Hohl used ED data from the National Ambulatory Care Reporting System (NACRS). The latter assigns ICD codes to ED discharge diagnoses by mapping selections made from a shortlist of 800 common term diagnoses to ICD-10-CA codes. Compared to the billing data used in our study, this process likely assigns ICD codes that more accurately reflect patients' diagnoses. This could explain the higher sensitivity reported in Hohl despite their use of a narrower code set. In addition, as part of their process to identify gold standard ADEs, Hohl interviewed treating emergency physicians using a standardized questionnaire to determine patients' working diagnoses. This may have made it more likely that treating physicians recognized and documented ADEs in the ED visit note, which in turn is used to assign ICD codes in NACRS, thus potentially overestimating the sensitivity of their code set.
Our study has several limitations. First, gold standard ADEs were often adjudicated based on incomplete information since lab test results and information on whether symptoms improved after medication discontinuation were often unavailable. These missing data impact on the Naranjo score calculation since item responses are recorded as "don't know". This uncertainty with missing information is a well-documented issue with the Naranjo algorithm, which nevertheless remains a frequently used method for ADE adjudication [63]. However, the ADE rates produced by our gold standard are similar to previously reported rates of adverse events attributable to antidepressant and antihypertensive medications [64][65][66], suggesting that the Naranjo algorithm produced accurate estimates in our sample. Second, the 3-week recall period used in patient interviews may have led to an underestimation of true ADE rates as patients may not have recalled mild symptoms that occurred earlier in this period. However, we expect such bias to be minimal as adverse effects of medications typically occur days or even weeks following the start of medication use [67,68]. The 3-week window we used is meant to allow time for adverse effects to occur prior to the interview while minimizing poor recall. It is similar to (and in some cases even shorter than) recall periods used in previously published studies [45,[69][70][71]. In addition, the mean age of the study population was 64.2 years, of whom a low proportion would be expected to have significant cognitive problems impacting recall, although this was not specifically addressed. Third, administrative health data from the RAMQ record only the first 4 digits of ICD-9 codes, whereas many of the codes we identified as potential adverse effects of study drugs had up to 5 digits (and thus had to be truncated when applied to our data). This could have led to false positive ADEs detected using this approach. However, given the high specificity of administrative health data in detecting ADEs, this did not seem to be an issue in our particular sample. Finally, the generalizability of our results is unknown, as the accuracy of our ICD code set in detecting ADEs is largely dependent on how consistently and accurately ICD codes are recorded in billing data (since there are no systematic checks for data quality), which may vary between jurisdictions.

Conclusion
The results of this validation study suggest that supplementing diagnostic codes in administrative health data with treatment change orders in electronic prescribing data doubles sensitivity. However, even when combined, these approaches do not possess sufficient accuracy to detect most ADEs resulting from medications prescribed in the community. Further research should be conducted to investigate the utility of patient-reported outcomes in detecting potential ADEs via emerging digital health technologies such as patient portals and mobile health applications. Expanded, Therapeutic Class-Targeted ICD Code Set. Appendix 3. Standard ICD Code Set. Appendix 4. Characteristics of consented patients, patients who declined consent, patients who were not dispensed the study drug, patients who were dispensed the study drug, and patients who were included in the final analysis. Appendix 5. Symptoms reported during interview, overall and by study drug therapeutic class, and among patients who reported all, none, or some of their symptoms to their physician. Appendix 6. Potential medication-related adverse effects experienced by antidepressant and antihypertensive users based on expanded, therapeutic class-targeted codeset. Appendix 7. Details of treatment changes among patients with a potential ADE based on eprescribing data. Appendix 8. Characteristics of patients who reported all, none, or some of the symptoms they indicated in the interview to their physician. Appendix 9. Severity of symptoms experienced, as reported during interviews, by patients who reported all, none, or some of their symptoms to their physician.