- Research article
- Open Access
Evaluation of drug administration errors in a teaching hospital
BMC Health Services Research volume 12, Article number: 60 (2012)
Medication errors can occur at any of the three steps of the medication use process: prescribing, dispensing and administration. We aimed to determine the incidence, type and clinical importance of drug administration errors and to identify risk factors.
Prospective study based on disguised observation technique in four wards in a teaching hospital in Paris, France (800 beds). A pharmacist accompanied nurses and witnessed the preparation and administration of drugs to all patients during the three drug rounds on each of six days per ward. Main outcomes were number, type and clinical importance of errors and associated risk factors. Drug administration error rate was calculated with and without wrong time errors. Relationship between the occurrence of errors and potential risk factors were investigated using logistic regression models with random effects.
Twenty-eight nurses caring for 108 patients were observed. Among 1501 opportunities for error, 415 administrations (430 errors) with one or more errors were detected (27.6%). There were 312 wrong time errors, ten simultaneously with another type of error, resulting in an error rate without wrong time error of 7.5% (113/1501). The most frequently administered drugs were the cardiovascular drugs (425/1501, 28.3%). The highest risks of error in a drug administration were for dermatological drugs. No potentially life-threatening errors were witnessed and 6% of errors were classified as having a serious or significant impact on patients (mainly omission). In multivariate analysis, the occurrence of errors was associated with drug administration route, drug classification (ATC) and the number of patient under the nurse's care.
Medication administration errors are frequent. The identification of its determinants helps to undertake designed interventions.
Since the release of the report To Err Is Human, patient safety has risen to the forefront of healthcare issues . Medication errors can occur during any of the three steps of the medication use process: from prescription, medication delivery to dispensing to the patient. Reviews on medication errors [2–10], prescription errors  or dispensing errors  are numerous. The evaluation and improvement of drug administration process are key elements in patient safety. A review of drug administration errors detected by the observation technique  revealed methodological limitations in studies evaluating the administration process: no standardized definition of error types and error rate, lack of information about the selection method of nurses observed and number of nurses observed, the level of experience of nurses, the number of patients or information on the observation technique [14–28].
To overcome the limits described above, we aimed to assess the frequency, type, potential clinical significance and determinants of drug administration errors detected by direct observation in adult in-patients.
The study was conducted in four adult wards (90 beds) at a teaching hospital in Paris, France (800 beds): immunology-cardiology ward, nephrology ward, vascular medical ward and cardiovascular surgical ward. These four wards are the only ones for which the drug doses are prepared daily by the pharmacy. Prescriptions were written by physicians, using the same computerized physician order entry system (DxCare®, Medasys). The hospital pharmacy validated orders and delivered the prescribed drugs to each ward, on a unit-dose basis except for drugs prescribed as needed. If necessary, the nurses obtained the drugs from secure and automated medication cabinets (Omnicell® Inc.) available in each ward. Each nurse was in charge of 6 to 8 patients.
Disguised direct observation was used for the detection of drug administration errors , because this approach gives more efficient, objective, and reliable results than spontaneous reporting or patient chart reviews [29–31]. Head nurses and physicians were informed about the objectives and the nurses who were observed were told that a pharmacist was evaluating the process of drug administration with the aim of improving it. We used a single observer, to avoid problems of interobserver variability. At the start of the observation period, each nurse gave written consent for observation and had the option of refusing to be observed. The observer was a clinical pharmacist and received a one-month training with a senior pharmacist before the start of the study.
For each observation round and ward, the observation order of nurses was randomized for identification of the nurse to be observed. Patients were not selected. The observer watched the selected nurses preparing and administering medication. Observations were carried out on six consecutive days per ward (including Saturday), for three drug rounds per day (8 am, 12 pm, 6 pm). Recorded observations were compared immediately after observation with the physician orders. Because the observer saw the order after drug administration, she was unable to prevent some errors. But if she was aware of an imminent potential error, she intervened to prevent it.
Emergency drugs, parenteral nutrition and drugs prescribed as needed were not observed as the delivery of these medications did not follow the unit-dose process described above. Non-permanent nurses were not included in the study.
The following data were collected: characteristics of the nurse (age, sex, years of experience and years in the unit); nurse-to-patient ratio; nurse workload (number of patients under the care of each nurse, including patients who were admitted on that day) and number of interruptions during the drug administration round, and characteristics of the drugs (Anatomical Therapeutic Chemical (ATC) classification, unit-dose prepared by the pharmacy, route, pharmaceutical form, dose and time of drug administration).
The study was not considered as research but as routine care. It was part of an audit for quality improvement and was considered exempted from ethical approval. All data were studied anonymously.
The main outcome assessed was error rate. Error rate without wrong time errors, types of errors, severity of errors and risk factors were also evaluated.
To overcome the limits found in the review we conducted, we standardized the drug administration error rate using the same denominator (see below) and the types of errors defined by the American Society of Health-system Pharmacists (ASHP). Drug administration errors were classified into the nine categories of the ASHP: omission error (failure to administer an ordered dose to a patient), wrong time error (administered more than one hour before or after the specified time), unauthorized drug error (dose given to the wrong patient, unordered drugs), wrong dose error, wrong dosage-form error, wrong drug-preparation error (incorrect dilution or reconstitution, mixing drugs that are physically incompatible and inadequate product packaging), wrong administration technique error (doses administered via the wrong route (different from the route prescribed), via the correct route but at the wrong site, and at the wrong rate of administration), deteriorated drug error (use of expired drugs or improperly stored drugs) and other medication error (included any drug administration errors not fitting into the above predefined categories) . The error rate was calculated using the Total Opportunities for Errors (TOE), which is the sum of all doses ordered plus all the unordered doses given . The drug administration error rate was then calculated as the number of administrations with one or more errors divided by the TOE and multiplied by 100. We also calculated the error rate without wrong time errors, as this type of error is a matter of much debate. Some authors recommended that studies on medication errors should report both the error rates with and without timing errors [29, 34]. For injectable drugs, an administration of drug and solvent corresponded to one opportunity for error. Any given administration could be subject to several types of error, so it was possible that the sum of errors types was greater than the total number of administrations with at least one error. But an error could only be classified in one category of error.
A panel of senior experts composed of four physicians, three head nurses and two pharmacists evaluated the severity of each error anonymously according to a three-category scale: no clinical impact, serious or significant clinical impact, life-threatening impact on the patient .
Categorical variables are reported as frequencies (percentages) and numerical variables are reported as medians (minimum and maximum). The drug administration error rate was calculated with and without wrong time errors. We investigated the relationship between the occurrence of errors (error rate and error rate without wrong time errors) and potential risk factors (characteristics of the nurse and drug), using logistic regression models with random effects (intercepts) to take multiple observations for the same patient and the same nurse into account. All risk factors were analyzed in univariable and multivariable analyses. The final model was obtained by removing all factors not significant at the 5% level. Results are expressed as odds ratios (OR), with the 95% confidence intervals (CI). Data were analyzed with SAS® version 9.1 (SAS Institute, Cary, NC, USA).
Disguised direct observation was carried out during 72 drug rounds. All the nurses agreed to participate. Table 1 lists the characteristics of the nurses. There were 28 nurses (all female): six in the immunology-cardiology ward, seven each in the nephrology ward and cardiovascular surgical ward and eight in the vascular medical ward. Half the nurses were observed at least twice (min-max: 1-6 observation periods). Drug rounds lasted a median of 1 h 12 min (min-max: 15 min-2 h 45 min). During the study period, 108 patients were under the care of the nurses studied. A median of six patients per drug round was observed (min-max: 2-9).
In total, we recorded 1501 TOE. At least one error was detected in 415 of the 1501 TOE (error rate = 27.6%), with 430 errors identified in total. For 13 administrations, 2 errors were observed and for one administration, 3 errors were observed. Of the 14 administrations with more than one error, 10 included a wrong time error (see details in footnotes of Table 2). After exclusion of the 312 wrong time errors, 113 of the 1501 administrations (TOE) remained erroneous (415-312 + 10) corresponding to an error rate without wrong time error of 7.5%.
Types of errors
Wrong time errors were the principal type of errors observed (n = 312, 72.6%), followed by errors of omission (n = 60, 14.0%), and unauthorized drug errors (n = 16, 3.7%). There were 10 errors (2.3%) with the type "Other medication error" corresponding to administration of thyroid hormones with food, whereas these drugs should be administered to fasting patients. Wrong dose errors, wrong dosage-form errors, wrong drug-preparation errors, and wrong administration technique errors were rare (n = 8 for each type, 1.9%). There was no deteriorated drug error (see additional file 1).
The type of drugs administered and the error rate according to ATC drug classification are described in Table 2 and Additional file 1. The most frequently administered drugs were the cardiovascular drugs (425/1501, 28.3%) (C in ATC classification), the central nervous system drugs (279/1501, 18.6%) (N in ATC classification), followed by the gastrointestinal drugs (270/1501, 18.0%) (A in ATC classification).
More than a half of administrations of dermatological drugs (D in ATC classification) and sensory organs drugs (S in ATC classification) had an error but they were rarely prescribed (0.5% and 0.7% respectively). The most frequently administered drugs (C, N and A in ATC classification) had error rates of 20.5%, 33.7% and 31.8% respectively.
The administration or omission of 182 different drugs were incorrect. The first ten most administered drugs were oral acetaminophen (37% of administrations had an error), esomeprazol (37%), acetylsalicylic acid (22%), oral furosemide (9%), bisoprolol (28%), atorvastatin (9%), calcium heparin (32%), oral tramadol (22%), amlodipine (16%) and oral potassium chloride (31%).
Severity of errors
The observer intervened three times to prevent errors from occurring: corresponding to 5 drugs almost administered to the wrong patient, one ganciclovir administered at the wrong dose (100 mg instead of 300 mg) and one amoxicillin/clavulanic acid drug almost administered instead of prescribed amoxicillin. No patient harm was observed.
The expert panel classified 406 (94%) of the 430 errors as having no clinical impact on the patient and 24 (6%) as having serious or significant impact. Most of these errors were omissions. No potentially life-threatening errors were witnessed.
Risk factors for errors
Univariable logistic regression analysis indicated that the occurrence of errors was significantly associated with administration route, drug ATC and number of patients under the nurse's care (Table 3). There was a non significant trend towards an increase in the occurrence of errors towards the end of the week. There was a non significant trend towards a decrease in the occurrence of errors with increase age of the nurse (OR associated with a five-year increase in age: 0.81 [95% CI: 0.62 to 1.06], p = 0.127). No significant effects of clinical unit, week day, drug round, unit-dose preparation by the pharmacy, interruptions and number of patients receiving drugs were found. The removal of medication from secure medication cabinets, by the nurses, was not associated with errors.
In the multivariable analysis (Table 3), the same three factors were associated with the occurrence of errors. The risk of error was higher for administrations by injection (OR versus the oral route: 3.30 [95% CI: 2.01 to 5.44], p < 0.001) and for nurses with larger numbers of patients under their care (OR associated with a one-patient increase: 1.22 [95% CI: 1.04 to 1.42], p = 0.013). The highest risks of error in a drug administration were for dermatological drugs (D in ATC classification) and sensory organs drugs (S in ATC classification) but the confidence intervals were large due to small sizes. The risk of error was significantly higher for respiratory system drugs (R in ATC classification), systematic hormonal drugs (H in ATC classification), anti-infective drugs for systemic use (J in ATC classification), central nervous system drugs (N in ATC classification) and gastrointestinal drugs (A in ATC classification) than for cardiovascular drugs, the most frequently administered drugs (C in ATC classification). Analysis of risk factors for errors excluding wrong time errors highlighted the same factors. Since the frequency of event was more rare (113/1501), the drug ATC (14 levels) could not be evaluated as a risk factor in a model with random effects. Drug ATC was highly significant in the univariate analysis (logistic regression model without random effects: p < 0.001). The multivariable analysis (without drug ATC) indicated that the risk of error (wrong time errors excluded) was higher for administrations by injection (OR versus the oral route: 6.89 [95% CI: 4.06 to 11.70], p < 0.001), tended to be lower with increase age of the nurse (OR associated with a five-year increase in age: 0.78 [95% CI: 0.60 to 1.02], p = 0.069), and surprisingly there was a non significant trend towards a decrease of risk of error (wrong time errors excluded) with an increase of the number of patients under the nurse's care (OR associated with a one-patient increase: 0.75 [95% CI: 0.56 to 1.01], p = 0.060).
Drug administration errors were common in the wards studied. An error rate of 27.6% was found, decreasing to 7.5% when wrong time errors were excluded. Most of the errors (94%) were unlikely to cause lasting harm, but 6% were serious. By extrapolation to the whole hospital (800 beds), a rate of 6% serious errors would have meant more than 200 such errors every month. The factors associated with errors were administration route, drug ATC and number of patients under the nurse's care. Unit-dose preparation by the pharmacy was not associated with a higher occurrence of errors than the removal of the drug from a secure medication cabinet.
In the literature, there is a high heterogeneity in the methodologies, leading to heterogeneity in the results. In the systematic review we conducted, wrong time errors followed by omissions are the most frequent types of errors reported. In addition, the error rate without wrong time errors ranges from 1% [36, 37] to 48% . In a previous study in two adult units (geriatric and cardiovascular-thoracic surgery unit), error rates reached 14.9% with wrong time errors and 11% without wrong time errors . The total error rate in our study was higher than those in the study by Chua and colleagues (11.4% in an adult hematology unit), but the error rates without wrong time errors was similar to that reported here (8.7%) .
We did not find an association with the unit, the day of observation and the drug rounds. As our study, Prot and colleagues found an association between errors and drug ATC classification (cardiovascular, anti-infective and central nervous system drugs) . Chua and colleagues showed an association between errors and injectable route administration compared to oral route . Finally, nurse workload was a risk factor of medication administration errors in the study by Tissot and colleagues whereas injectable administration was not associated with errors .
We chose to use disguised observation technique in this study. The observation period was relatively long. We used a single observer specifically trained for this study. To overcome the limits found in our review, we standardized the drug administration error rate using the same denominator (TOE) and the types of errors (ASHP classification). We reported the number of drug administration's with one or more errors in order to calculate an error rate excluding some types of errors. Finally, characteristics of hospital are presented (country, types of units, delivery system, characteristics of nurses observed). Our study has several potential limitations. First, it was a single-center study. Observation is very time-consuming and can therefore be carried out for very long periods of time. We did not observe nurses during the Sundays, and thus the applicability of the results for work at these times is unknown. It is also possible that nurses changed their behaviors when observed because they were aware that they were being observed to identify problems in the medication use process. However, Allan and Barker showed that disguised observation decreases the Hawthorne effect on observed nurses . We did not observe non-permanent nurses as their agreement could not be obtain. However they represented less than 10% of the nurses during the observation period. Medication with high risk like chemotherapy drugs were rarely prescribed in the 4 units observed therefore no error of such administration was detected.
Different interventions have been proposed to improve the drug administration process. This study shows that those interventions should be adapted to the local context and the type of errors observed. For example, introduction of bar-code medication administration systems together with awareness of nurses could reduce some errors like omissions and wrong time (the two most frequent errors found in our study). Wrong administration technique including injectable drugs could be decrease with nurse training and awareness to manipulate injectable drugs.
Medication safety issues are an important element of the medication use process in hospitals. Drug administration errors are frequent. Standardization of drug administration error rate using the same denominator (TOE) and types of errors remains essential for further studies.
Institute, of, Medicine: To err is human: building a safer health system. 1999, Washington, DC: National Academy Press edn
Krahenbuhl-Melcher A, Schlienger R, Lampert M, Haschke M, Drewe J, Krahenbuhl S: Drug-related problems in hospitals: a review of the recent literature. Drug Saf. 2007, 30: 379-407. 10.2165/00002018-200730050-00003.
Ghaleb MA, Barber N, Franklin BD, Yeung VW, Khaki ZF, Wong IC: Systematic review of medication errors in pediatric patients. Ann Pharmacother. 2006, 40: 1766-1776. 10.1345/aph.1G717.
Wong IC, Ghaleb MA, Franklin BD, Barber N: Incidence and nature of dosing errors in paediatric medications: a systematic review. Drug Saf. 2004, 27: 661-670. 10.2165/00002018-200427090-00004.
Maidment ID, Lelliott P, Paton C: Medication errors in mental healthcare: a systematic review. Qual Saf Health Care. 2006, 15: 409-413. 10.1136/qshc.2006.018267.
Miller MR, Robinson KA, Lubomski LH, Rinke ML, Pronovost PJ: Medication errors in paediatric care: a systematic review of epidemiology and an evaluation of evidence supporting reduction strategy recommendations. Qual Saf Health Care. 2007, 16: 116-126. 10.1136/qshc.2006.019950.
Moyen E, Camire E, Stelfox HT: Clinical review: medication errors in critical care. Crit Care. 2008, 12: 208-10.1186/cc6813.
Evans J: Prevalence, risk factors, consequences and strategies for reducing medication errors in Australian hospitals: a literature review. Contemp Nurse. 2009, 31: 176-189. 10.5172/conu.618.104.22.168.
Snijders C, van Lingen RA, Molendijk A, Fetter WP: Incidents and errors in neonatal intensive care: a review of the literature. Arch Dis Child Fetal Neonatal Ed. 2007, 92: F391-F398. 10.1136/adc.2006.106419.
van den Bemt PM, Egberts TC: de Jong-van den Berg LT, Brouwers JR: Drug-related problems in hospitalised patients. Drug Saf. 2000, 22: 321-333. 10.2165/00002018-200022040-00005.
Lewis PJ, Dornan T, Taylor D, Tully MP, Wass V, Ashcroft DM: Prevalence, incidence and nature of prescribing errors in hospital inpatients: a systematic review. Drug Saf. 2009, 32: 379-389. 10.2165/00002018-200932050-00002.
James KL, Barlow D, McArtney R, Hiom S, Roberts D, Whittlesea C: Incidence, type and causes of dispensing errors: a review of the literature. Int J Pharm Pract. 2009, 17: 9-30. 10.1211/ijpp.17.1.0004.
Barker KN, McConnell WE: The problems of detecting medication errors in hospitals. Am J Hosp Pharm. 1962, 19: 361-369.
Anselmi ML, Peduzzi M, Dos Santos CB: Errors in the administration of intravenous medication in Brazilian hospitals. J Clin Nurs. 2007, 16: 1839-1847. 10.1111/j.1365-2702.2007.01834.x.
Barker KN, Flynn EA, Pepper GA, Bates DW, Mikeal RL: Medication errors observed in 36 health care facilities. Arch Intern Med. 2002, 162: 1897-1903. 10.1001/archinte.162.16.1897.
Calabrese AD, Erstad BL, Brandl K, Barletta JF, Kane SL, Sherman DS: Medication administration errors in adult patients in the ICU. Intensive Care Med. 2001, 27: 1592-1598. 10.1007/s001340101065.
Chua SS, Chua HM, Omar A: Drug administration errors in paediatric wards: a direct observation approach. Eur J Pediatr. 2009, 169: 603-611.
Chua SS, Tea MH, Rahman MH: An observational study of drug administration errors in a Malaysian hospital. J Clin Pharm Ther. 2009, 34: 215-223. 10.1111/j.1365-2710.2008.00997.x.
Fahimi F, Ariapanah P, Faizi M, Shafaghi B, Namdar R, Ardakani MT: Errors in preparation and administration of intravenous medications in the intensive care unit of a teaching hospital: an observational study. Aust Crit Care. 2008, 21: 110-116. 10.1016/j.aucc.2007.10.004.
Haw C, Stubbs J, Dickens G: An observational study of medication administration errors in old-age psychiatric inpatients. Int J Qual Health Care. 2007, 19: 210-216. 10.1093/intqhc/mzm019.
Le Grognec C, Lazzarotti A, Marie-Joseph DA, Lorcerie B: Medication errors resulting from drug preparation and administration. Thérapie. 2005, 60: 391-399.
Lisby M, Nielsen LP, Mainz J: Errors in the medication process: frequency, type, and potential clinical consequences. Int J Qual Health Care. 2005, 17: 15-22. 10.1093/intqhc/mzi015.
Prot S, Fontan JE, Alberti C, Bourdon O, Farnoux C, Macher MA, Foureau A, Faye A, Beaufils F, Gottot S, Brion F: Drug administration errors and their determinants in pediatric in-patients. Int J Qual Health Care. 2005, 17: 381-389. 10.1093/intqhc/mzi066.
Ridge KW, Jenkins DB, Noyce PR, Barber ND: Medication errors during hospital drug rounds. Qual Health Care. 1995, 4: 240-243. 10.1136/qshc.4.4.240.
Taxis K, Barber N: Ethnographic study of incidence and severity of intravenous drug errors. BMJ. 2003, 326: 684-10.1136/bmj.326.7391.684.
Taxis K, Dean B, Barber N: Hospital drug distribution systems in the UK and Germany-a study of medication errors. Pharm World Sci. 1999, 21: 25-31. 10.1023/A:1008616622472.
Tissot E, Cornette C, Demoly P, Jacquet M, Barale F, Capellier G: Medication errors at the administration stage in an intensive care unit. Intensive Care Med. 1999, 25: 353-359. 10.1007/s001340050857.
Wirtz V, Taxis K, Barber ND: An observational study of intravenous medication errors in the United Kingdom and in Germany. Pharm World Sci. 2003, 25: 104-111. 10.1023/A:1024009000113.
Allan EL, Barker KN: Fundamentals of medication error research. Am J Hosp Pharm. 1990, 47: 555-571.
Meyer-Massetti C, Cheng CM, Schwappach DL, Paulsen L, Ide B, Meier CR, Guglielmo BJ: Systematic review of medication safety assessment methods. Am J Health Syst Pharm. 2011, 68: 227-240. 10.2146/ajhp100019.
Flynn EA, Barker KN, Pepper GA, Bates DW, Mikeal RL: Comparison of methods for detecting medication errors in 36 hospitals and skilled-nursing facilities. Am J Health Syst Pharm. 2002, 59: 436-446.
ASHP guidelines on preventing medication errors in hospitals. Am J Hosp Pharm. 1993, 50: 305-314.
Allan BL: Calculating medication error rates. Am J Hosp Pharm. 1987, 44: 1044-1046.
Pepper GA: Errors in drug administration by nurses. Am J Health Syst Pharm. 1995, 52: 390-395.
Estellat C, Colombet I, Vautier S, Huault-Quentel J, Durieux P, Sabatier B: Impact of pharmacy validation in a computerized physician order entry context. Int J Qual Health Care. 2007, 19: 317-325. 10.1093/intqhc/mzm025.
Conroy S, Appleby K, Bostock D, Unsworth V, Cousins D: Medication errors in a children's hospital. Paediatric and Perinatal Drug Therapy. 2007, 8: 18-25. 10.1185/146300907X167790.
Pasto-Cardona L, Masuet-Aumatell C, Bara-Olivan B, Castro-Cels I, Clopes-Estela A, Paez-Vives F, Schonenberger-Arnaiz JA, Gorgas-Torner MQ, Codina-Jane C: Incident study of medication errors in drug use processes: prescription, transcription, validation, preparation, dispensing and administering in the hospital environment. Farm Hosp. 2009, 33: 257-268. 10.1016/S1130-6343(09)72465-1.
Bourlon S, Baronnet A, Provost S, Meunier P: Evaluation des erreurs médicamenteuses dans une unité de soins pédiatriques. J Pharm Clin. 2006, 25: 23-31.
Tissot E, Cornette C, Limat S, Mourand JL, Becker M, Etievent JP, Dupond JL, Jacquet M, Woronoff-Lemsi MC: Observational study of potential risk factors of medication administration errors. Pharm World Sci. 2003, 25: 264-268.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6963/12/60/prepub
We thank the doctors, the head nurses and nurses of the four wards studied. We particularly thank Ali Al-Khafaji.
No funding source.
All authors declared that they have no competing interests: no support from any organization for the submitted work; no financial relationships with any organization that might have an interest in the submitted work in the previous five years; no other relationships or activities that could appear to have influenced the submitted work. SB as PhD student was funded by HMR Expert and a grant from AP-HP (from November 2010 to October 2011) but none of these funds has influenced the submitted work.
SB, BS, FG, TC, PP and PD conceived and planned the work that led to the manuscript. SB made substantial contributions in the acquisition, analysis and interpretation of the data. FG performed data analysis made substantial contributions in the interpretation of the data. BS and PD made substantial contributions in the interpretation of the data.
SB wrote the first draft of the report. All authors were involved in drafting and revising the various drafts of the report with substantive suggestions. All authors read and approved the final manuscript.
Electronic supplementary material
About this article
Cite this article
Berdot, S., Sabatier, B., Gillaizeau, F. et al. Evaluation of drug administration errors in a teaching hospital. BMC Health Serv Res 12, 60 (2012). https://doi.org/10.1186/1472-6963-12-60
- Hospital care
- Medication errors
- Direct observation