Patient group
Patients arriving at the hospital suffering from shortness of breath (dyspnea) who were seen by a physician (mostly residents) of the participating specialties were included in the study. As the study focuses on suboptimal cognitive acts a specific patient group was selected since it is easier to discover deviations from the optimal process when the group is homogenous and the optimal diagnostic process can be defined more clearly. As this study focuses on the diagnostic process and patients were included prospectively it was crucial to select patients based on a symptom rather than a disease. A large variety of diseases can cause shortness of breath and the levels of difficulty of establishing a correct diagnosis vary.
Hospital selection
The study took place in five hospitals in the Netherlands in the departments of internal medicine, cardiology and pulmonology between May 2007 and May 2008. A random sample of hospitals located within a reasonable traveling distance (2 hours) of the researcher was invited to participate in the study. All participating hospitals were acute care hospitals, one university hospital, two tertiary medical teaching hospitals and two general hospitals. The organization of the departments varied between the hospitals e.g. in some hospitals residents from the pulmonology and cardiology were not exchangeable with the residents of the internal medicine department while in other hospitals all residents worked alternating shifts in the departments of internal medicine, pulmonology as well as cardiology.
1. Determining suboptimal cognitive acts in diagnostic process and patient outcomes
The diagnostic process was assessed by different data sources which together added up to a more complete assessment of the diagnostic process. First, information of the differential diagnoses was gathered from the treating physician concurrently to diagnosing the patient. The second data source was a retrospective record review of the diagnostic process and patient outcomes. The questionnaire used in the record review was based on an optimal diagnostic process for dyspnea patients developed using a Delphi method. The record review revealed suboptimal cognitive acts, which are cognitive errors using a very liberal threshold and therefore includes a lot of minor deviations from the optimal process. The record review was followed by a short meeting with the treating physician to elucidate the suboptimal cognitive acts as assessed by the expert reviewers to have occurred during the diagnostic process. The data sources are described in detail below and see Figure 1 for an overview of the data gathering process.
Information about differential diagnosis
The treating physicians were asked to note down their differential diagnoses in order of probability. After the treating physician decided to admit the patient to the hospital they were asked to fill out a small questionnaire (see Figure 2). They noted down what possible underlying diagnoses could cause the dyspnea in the patient, subsequently they indicated how likely they thought these diagnoses were.
Record review of diagnostic process
Optimal diagnostic process
Seven expert internists participated in developing the optimal diagnostic process for dyspnea patients in a two stage Delphi method. First, all participating expert internists received a questionnaire with twelve cases of patients who were admitted to the hospital suffering from dyspnea. The questions focused on the important decisions during the diagnostic process. After analyzing the questionnaires a first draft of the optimal process was developed by a researcher (LZ) and an internist (AT). During a second meeting with the expert internists, the draft of the optimal diagnostic process was discussed and commented on by the expert internists and a researcher (DT). The comments of the internists and researcher were processed and the final version of the optimal diagnostic process was completed and agreed upon by the expert internists.
Review questionnaire
The questionnaire used to review the patient records was based on the optimal diagnostic process. The diagnostic process was assessed according to the stages of the diagnostic process; (1) History taking, (2) physical examination, (3) laboratory results, (4) imaging techniques, (5) outlining a diagnosis, (6) starting the treatment and (7) verification of diagnosis and treatment during the patients' stay. The first four stages are information gathering stages, and the last three are stages of information integration. The information gathering stages were reviewed on two criteria (1) whether the correct information was gathered and (2) whether the information was interpreted correctly. The information interpreting stages were reviewed on interpretation only.
Information gathering stages
For the information gathering stages (history taking, physical examination, laboratory results and imaging techniques) all relevant aspects for patients with dyspnea were reviewed (i.e. weight loss, medication use, allergies, temperature, listening to the lungs, CT scan). For each aspect the expert reviewers checked the patient record and answered the questions in Figure 3.
After every information gathering stage several questions about the interpretation of the gathered information were asked which included the questions in Figure 4.
Information integration stages
The review questionnaire assessed the information integration stages systematically by questions about the decisions of the physician and the decisions made on the diagnosis and treatment. Examples of the questions were 'Did the treating physician verify the outcomes of the treatment on time?' and 'Did the treating physician integrate all information into a correct diagnosis?'. Furthermore, some questions to obtain additional information were asked in order to obtain more information about the circumstances under which the suboptimal cognitive acts occur, such as: patient factors, supervision and the number of physicians involved in the diagnostic process.
Reviewer selection and reviewing process
Four of the expert internists were invited to participate as reviewers in the reviewing process of the patient records. These expert internists were recruited from a group of fifty-five physicians who participated as reviewers in a large retrospective study on the occurrence of adverse events in the Netherlands[22]. The selection criteria were:
-
At least 10 years of post-graduate work experience in internal medicine
-
A good reputation amongst colleagues
-
Retired for no longer than 5 years at time of selection as a reviewer
-
Interest in and experience with patient safety research
The reviewers attended a one-day training program which was led by a researcher (LZ) and an internist (AT). The topics addressed were the study protocol (e.g. background and research goals), the review questionnaire which was used to review the diagnostic process and the computer program in which the review questionnaire was programmed. Subsequently, two half-day practice meetings were organized during which the expert internists practiced with the review questionnaire and the computer system. In addition, after reviewing 10–15 records in the hospitals, another training session was organized to discuss questions and difficulties the reviewers experienced. The expert internist reviewers were paid a small fee for their work hours and compensated for their expenses.
The record review procedure took place in the hospitals. The review questionnaire was transformed into an electronic template with the program Blaise [23]. The expert reviewers entered the data directly into the program which was installed on laptop computers. A researcher was always present during the reviewing process to answer questions about the study protocol and for technical support.
Information from the treating physician and classification of suboptimal cognitive acts
Two days to eight months after the patients' discharge the patient records were reviewed, a researcher (LZ) discussed the suboptimal cognitive diagnostic acts found in the record review with the physicians who treated the patients. In a few cases the meeting was eight months after the patients' discharge due to the time consuming process of obtaining and reviewing the records and subsequently planning a meeting with the treating physician. During this meeting with the treating physicians, the physicians clarified their decisions and provided additional information on the assessed suboptimal cognitive acts. The researcher (LZ) was instructed by the expert reviewers about the questions that needed to be asked. Based on the findings of the record review and the interview with the treating physician the suboptimal cognitive acts were classified by a researcher (LZ) and an internist (AT) into the error categories: slip, lapse, mistake and violation of the model of unsafe acts [18].
Suboptimal cognitive acts
The main outcome measurements were the suboptimal cognitive acts found in the record review. An act was defined as suboptimal if (1) it was indicated as such by the internist reviewers in the record review and (2) it was not contradicted during the meeting with the physician. Suboptimal cognitive acts were for instance: not ordering a specific test or ordering an unnecessary test, misinterpreting test results etc. Subsequently, after defining the suboptimal cognitive acts a researcher (LZ) and an internist (AT) described all suboptimal cognitive acts. For all suboptimal acts it was determined in which diagnostic stage of the diagnostic process they occurred and what the cause of the act was using the taxonomy of unsafe acts[18].
2. Measurement of workload, fatigue and work experience
The second source of data was measurements of workload, fatigue and work experience of the treating physician, measured during the patients hospital stay.
Workload
Workload was assessed in two ways. First, at the patients' admission the physician answered a question (scale 1 to 20) about the level of the subjective workload (see Figure 2).
Second, the workload of the physicians during the course of the day was measured. In each hospital every two hours physicians filled out the level of their subjective workload using the NASA tlx [24] questionnaire (short version) on a handheld computer. The physicians were notified by an alarm on the handheld computer when to fill out the questionnaire. The NASA tlx exists of 6 subjective dimensions: mental demand, physical demand, temporal demand, performance, effort and frustration (all with scale 1–20).
Fatigue
When the physicians filled out the NASA tlx workload questionnaire on the handheld computer, they also answered one question about their level of fatigue (scale 1–20) to measure the fluctuation of the level of fatigue during the working day.
Work experience
All physicians who included patients in the study or were involved in the care for the patients were asked about their work experience by 4 questions.
How long have you worked:
-
1.
as a physician after you obtained your medical degree?
-
2.
in the hospital you are currently employed?
-
3.
in the hospital department you are currently employed?
-
4.
in your current specialization (internal medicine, cardiology or lung diseases)?
Sample size
As various research questions are studied and data analysis is mainly descriptive we based our sample size on the distribution of causes of the suboptimal cognitive acts. In order to obtain a substantial number of suboptimal cognitive acts in each category (slip, lapse, mistake, violation and patient record problem) a confidence interval was calculated assuming an equal distribution of suboptimal cognitive acts in each category (20% of the suboptimal cognitive acts in each category). Based on a pilot review of several patient records by an internist (AT), it was expected to find an average of 1,5 suboptimal acts per patient record which would mean that 250 patient records (= 375/1,5) are required in the study.
Using a binomial distribution and 375 suboptimal acts, the confidence interval is as follows:
CI = 0,20 ± 0,040 ≈ 0,16 - 0,24. This is a respectable confidence interval.
Confidentiality and ethical approval
The review board of the VU medical center approved the research protocol. All participating hospitals granted approval to participate. The reviewers and researchers involved in the data collection signed a confidentiality agreement to maintain the secrecy of the data. Furthermore, the expert reviewers did not review records in hospitals where they had been employed in the course of their career. All patients who were included in the study gave informed consent to review their patient record. All patients were assigned to a unique study number in order to keep the patients' identity confidential. Patient identifiers were kept in a dataset separately from the primary database. Patients' names were not included in the database and after completion of the data collection and analysis, medical record identifiers were destroyed.
Reliability study
To assess the reliability of the review process, a random sample of 5% of the records were independently reviewed by two expert reviewers.