Low agreement for assessing the risk of postoperative deep venous thrombosis when deciding prophylaxis strategies: a study using clinical vignettes
© O'Flaherty et al; licensee BioMed Central Ltd. 2002
Received: 19 April 2002
Accepted: 16 August 2002
Published: 16 August 2002
Several clinical practice guidelines (CPG) on antithrombotic prophylaxis in surgical patients help to decide about the prophylaxis strategy based on the patient risk of deep venous thrombosis (DVT). However, the physician risk estimates of DVT could have little inter-observer reproducibility, which could lead to different individual prophylaxis practices.
Physicians were asked to evaluate DVT risk in eight clinical vignettes, describing actual patients cared for in our hospital. The vignettes included all possible levels of DVT risk.
The degree of prophylaxis strategies accuracy was 63% (95% CI 523–75%). Overall agreement was 0.32 (z = 7.61, p < 0.001) and for each level of risk kappa was 0.38 (z = 6.50, p < 0.001); 0.1 (z = 1.65, p < 0.049) and 0.5 (z = 8.45, p < 0.001) for small, moderate and high risk group respectively
Our results showed that there is poor agreement when physicians have to evaluate the risk for postoperative DVT, and in the cases of low and moderate risks of DVT there is the smallest agreement. In addition, the data also showed that the overall accuracy of DVT prophylaxis strategy was only moderate and the risk evaluation did not correlate to the selection of the strategy. The issue of inter-observers variability should be taken into account when CPG performance are analysed, especially when considering the risk-evaluation to choose the appropriate actions.
Deep venous thrombosis (DVT) prophylaxis is an effective intervention in the postoperative period . Several clinical practice guidelines (CPG) on the topic have been published, with the aim to increase the utilisation and appropriateness of this practice. Usually, these guidelines established the choice of the prophylaxis strategies based on the type of surgery, as well as on the presence of specific patient's risk factors [1, 2]
To establish an individual patient risk level is a complex task that could lead to substantial variations between observers. The integration of several cognitive inputs and patient data under real work pressures is a task where the physicians' performance is sub-optimal .
In order to determine if there are variations in the postoperative DVT risk perception among physicians, we performed a study using clinical vignettes.
Clinical practice guideline
Our guideline on DVT prophylaxis is mostly based in the 5th ACCP consensus conference recommendation , where risk-assessment approaches are used to decide the prophylaxis strategy.
In our guideline, the postoperative DVT risk is evaluated basing on the items that follow: the age, the presence of venous thrombosis risk factors and the duration of surgery. With this information we could classify the patients in low, moderate and high-risk groups. Early mobilisation or graded compression stockings are prescribed in the case of low risk group patients. Moderate risk patients are prescribed fixed subcutaneous calcium heparin and graded compression stockings could replace heparin if high haemorrhagic risk exists. High-risk patients received nadroparin but when it was deemed inappropriate due to a high haemorrhagic risk, intermittent pneumatic calf compression (IPCC) was indicated. Both nadroparin and IPCC may be used in selected high-risk patients. The guideline has been widely publicized throughout the institution and it is available in the Hospital Intranet.
Clinical Vignettes summary with risk level and proposed prophylaxis strategyin the CPG.
Clinical Vignettes Summary
Post-operative DVT risk
Case 1: 31- years female post rhinoplasty, no past medical history, uneventful operation.
Case 2: 25 years male insulin dependent diabetes after a correction of hallux valgus. Developed postoperative hyperglycemia.
Case 3: 63- years old female post hysterectomy with non-insulin dependent diabetes. After returning from the operating room a blood pressure of 225/130 was noted responding to analgesia.
Case 4: 42- years old obese male with chronic stable angina history. Resection of melanoma in the malar region.
Case 5: 50- years old male with a 2-month history of macroscopic hematuria and pain in the right lumbar fossa. CT scan showed a right renal mass. He was neprhectomised
IPCC** not indicated since hemorrhage source was controlled
Case 6: 88- years old female with a history of duodenal ulcer in her youth admitted due to hip fracture and operated 24 hs later. She was stable in the immediate postoperative period.
Case 7: 38- year old male with an upper gastrointestinal haemorrhage gastrectomised after failure of medical therapy. An ulcerated leiomioma was diagnosed.
IPCC could be acceptable
Case 8: 46 years-old obese male with hip septic arthritis and previous pulmonary embolism diagnosis. A resection of the femoral head was performed.
We randomly chose 10 physicians from the Internal Medicine and the Intensive Care Unit to read the vignettes. Then we asked them: 1) to estimate the inhospital complications risks (nosocomial infection, cardio-respiratory events, DVT risk and hospital mortality) and to classify them in three levels: low, moderate and high risk; and 2) to order a complete diagnostic and therapeutic plan for each vignette.
Percentages, along with 95% confidence intervals, were calculated for the accuracy of the strategy selection compared to the intended level of risk for the vignette, as decided by the investigators.
The inter-observer agreement was evaluated using the kappa statistic for multiple raters and for multiple categories. Statistical calculations were performed with STATA 6.0.
The degree of appropriateness of prophylaxis strategies used were 63% (95% CI 52–75%).
There was no correlation found between the accuracy of the prophylaxis strategies and the DVT risk evaluated by the physician (χ2 for trend 0.54, p = 0.46). Although no statistically significant, the low risk strata showed a higher rate of strategy inappropriate selection compared to the moderate and high risk ones (72%, 61% and 61.5 respectively, x2 = 0.7, p = 0.7). Overall agreement was 0.32 (z = 7.61, p < 0.001) and kappa by risk category was 0.38(z = 6.50, p < 0.001); 0.1(z = 1.65, p < 0.049) and 0.5 (z = 8.45, p < 0.001) for the low, moderate and high risk group respectively.
All the DVT risk wrong estimates in the low risk group were rated as "moderate" and in the moderate risk group as "low risk". In the high-risk group, 70% of the wrong estimates were "low risk" and the rest were classified as "moderate risk".
Our results showed a poor agreement for assessing the risk of postoperative DVT, especially in the low and moderate risk groups. Wrongly classified patients who belong to moderate risk groups were classified in lower risk category. In addition, the data also showed that overall DVT prophylaxis strategy was not good enough and the risk evaluation did not correlate to the strategy selection.
As stated by Weed et al , our example on DVT risk assessment showed that integration of cognitive inputs and patient data must be systematically explored for different clinical settings since considerable variation may occur, especially when the physician behaviour is assessed.
The evaluation of the physician performance is a difficult task, as several factors from physicians themselves, healthcare structure and case complexity could modify the evaluation. Clinical vignettes are a validated measurement of the physician performance , as compared both to utilisation review and simulated patient-doctor encounters. This method permits controlling case-mix, an important confusing variable when the physician performance is evaluated.
A standardised way of risk evaluation could improve the quality of a key point in the decision process of the CPG, such as risk assessment. Computerized decision support systems have shown benefits in increasing usage and accuracy of DVT prophylaxis .
The issue of inter-observers variability should be taken into account to analyse variations in CPG performance, especially when considering risk-evaluation to select the appropriate actions.
We thank Dr. M. Pellizzari, Dr G. Amestoy and Dr. R. Martin for their helpful revision of the manuscript. We also want to thank our colleagues who kindly accepted to participate in this study and Dr P Casais for her work with the clinical practice guideline.
- Dalen JE, Hirsh J, et al: Fifth ACCP Consensus Conference on Antithrombotic Therapy. Chest. 1998, 114 (suppl): 439s-769s.View Article
- Ravaud P, Durieux P, Fourcade A, and the Comité Scientifique Thrombose: Prophylaxie des thromboses veineuses post-operatories: recommendations de la Ássitance Publique-Hopitaux de Paris. Sang Thromb Vaisseaux. 1995, 7: 119-129.
- Weed LL, Weed L: Opening the black box of clinical judgement-an overview. BMJ. 1999, 319: 1279.PubMed CentralView ArticlePubMed
- Peabody JW, Luck J, Glassman P, Dresselhaus TR, et al: Comparison of vignettes, standardised patients, and chart abstraction: a prospective validation study of 3 methods for measuring quality. JAMA. 2000, 283: 1715-1722. 10.1001/jama.283.13.1715.View ArticlePubMed
- Durieux P, Rémy N, Philippe R, et al: A Clinical Decision Support System for Prevention of Venous Thromboembolism: Effect on Physician Behaviour. JAMA. 2000, 283: 2816-2821. 10.1001/jama.283.21.2816.View ArticlePubMed
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6963/2/16/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article: verbatim copying and redistribution of this article are permitted in all media for any purpose, provided this notice is preserved along with the article's original URL.