- Research article
- Open access
- Published:
Low agreement for assessing the risk of postoperative deep venous thrombosis when deciding prophylaxis strategies: a study using clinical vignettes
BMC Health Services Research volume 2, Article number: 16 (2002)
Abstract
Background
Several clinical practice guidelines (CPG) on antithrombotic prophylaxis in surgical patients help to decide about the prophylaxis strategy based on the patient risk of deep venous thrombosis (DVT). However, the physician risk estimates of DVT could have little inter-observer reproducibility, which could lead to different individual prophylaxis practices.
Methods
Physicians were asked to evaluate DVT risk in eight clinical vignettes, describing actual patients cared for in our hospital. The vignettes included all possible levels of DVT risk.
Results
The degree of prophylaxis strategies accuracy was 63% (95% CI 523–75%). Overall agreement was 0.32 (z = 7.61, p < 0.001) and for each level of risk kappa was 0.38 (z = 6.50, p < 0.001); 0.1 (z = 1.65, p < 0.049) and 0.5 (z = 8.45, p < 0.001) for small, moderate and high risk group respectively
Conclusions
Our results showed that there is poor agreement when physicians have to evaluate the risk for postoperative DVT, and in the cases of low and moderate risks of DVT there is the smallest agreement. In addition, the data also showed that the overall accuracy of DVT prophylaxis strategy was only moderate and the risk evaluation did not correlate to the selection of the strategy. The issue of inter-observers variability should be taken into account when CPG performance are analysed, especially when considering the risk-evaluation to choose the appropriate actions.
Background
Deep venous thrombosis (DVT) prophylaxis is an effective intervention in the postoperative period [1]. Several clinical practice guidelines (CPG) on the topic have been published, with the aim to increase the utilisation and appropriateness of this practice. Usually, these guidelines established the choice of the prophylaxis strategies based on the type of surgery, as well as on the presence of specific patient's risk factors [1, 2]
To establish an individual patient risk level is a complex task that could lead to substantial variations between observers. The integration of several cognitive inputs and patient data under real work pressures is a task where the physicians' performance is sub-optimal [3].
In order to determine if there are variations in the postoperative DVT risk perception among physicians, we performed a study using clinical vignettes.
Methods
Study design
Clinical practice guideline
Our guideline on DVT prophylaxis is mostly based in the 5th ACCP consensus conference recommendation [1], where risk-assessment approaches are used to decide the prophylaxis strategy.
In our guideline, the postoperative DVT risk is evaluated basing on the items that follow: the age, the presence of venous thrombosis risk factors and the duration of surgery. With this information we could classify the patients in low, moderate and high-risk groups. Early mobilisation or graded compression stockings are prescribed in the case of low risk group patients. Moderate risk patients are prescribed fixed subcutaneous calcium heparin and graded compression stockings could replace heparin if high haemorrhagic risk exists. High-risk patients received nadroparin but when it was deemed inappropriate due to a high haemorrhagic risk, intermittent pneumatic calf compression (IPCC) was indicated. Both nadroparin and IPCC may be used in selected high-risk patients. The guideline has been widely publicized throughout the institution and it is available in the Hospital Intranet.
Clinical Vignettes
We have written eight comprehensive clinical vignettes including patients that cover the entire risk spectrum in order to evaluate the inter-observer variability about the risk assessment,. The information was mostly taken from actual patients cared for at our hospital. Specific details were changed in order to respect patient confidentiality and to avoid clueing physicians about the appropriate decisions. The vignettes are summarised in the table. No consent is required in our hospital to use clinical record data if no breach to patient confidentiality is possible. One of us (DG) independently reviewed the clinical vignettes for accuracy and validation regarding the study objectives.
We randomly chose 10 physicians from the Internal Medicine and the Intensive Care Unit to read the vignettes. Then we asked them: 1) to estimate the inhospital complications risks (nosocomial infection, cardio-respiratory events, DVT risk and hospital mortality) and to classify them in three levels: low, moderate and high risk; and 2) to order a complete diagnostic and therapeutic plan for each vignette.
Statistical analysis
Percentages, along with 95% confidence intervals, were calculated for the accuracy of the strategy selection compared to the intended level of risk for the vignette, as decided by the investigators.
The inter-observer agreement was evaluated using the kappa statistic for multiple raters and for multiple categories. Statistical calculations were performed with STATA 6.0.
Results
The degree of appropriateness of prophylaxis strategies used were 63% (95% CI 52–75%).
There was no correlation found between the accuracy of the prophylaxis strategies and the DVT risk evaluated by the physician (χ2 for trend 0.54, p = 0.46). Although no statistically significant, the low risk strata showed a higher rate of strategy inappropriate selection compared to the moderate and high risk ones (72%, 61% and 61.5 respectively, x2 = 0.7, p = 0.7). Overall agreement was 0.32 (z = 7.61, p < 0.001) and kappa by risk category was 0.38(z = 6.50, p < 0.001); 0.1(z = 1.65, p < 0.049) and 0.5 (z = 8.45, p < 0.001) for the low, moderate and high risk group respectively.
All the DVT risk wrong estimates in the low risk group were rated as "moderate" and in the moderate risk group as "low risk". In the high-risk group, 70% of the wrong estimates were "low risk" and the rest were classified as "moderate risk".
Discussion
Our results showed a poor agreement for assessing the risk of postoperative DVT, especially in the low and moderate risk groups. Wrongly classified patients who belong to moderate risk groups were classified in lower risk category. In addition, the data also showed that overall DVT prophylaxis strategy was not good enough and the risk evaluation did not correlate to the strategy selection.
As stated by Weed et al [3], our example on DVT risk assessment showed that integration of cognitive inputs and patient data must be systematically explored for different clinical settings since considerable variation may occur, especially when the physician behaviour is assessed.
The evaluation of the physician performance is a difficult task, as several factors from physicians themselves, healthcare structure and case complexity could modify the evaluation. Clinical vignettes are a validated measurement of the physician performance [4], as compared both to utilisation review and simulated patient-doctor encounters. This method permits controlling case-mix, an important confusing variable when the physician performance is evaluated.
A standardised way of risk evaluation could improve the quality of a key point in the decision process of the CPG, such as risk assessment. Computerized decision support systems have shown benefits in increasing usage and accuracy of DVT prophylaxis [5].
Conclusions
The issue of inter-observers variability should be taken into account to analyse variations in CPG performance, especially when considering risk-evaluation to select the appropriate actions.
References
Dalen JE, Hirsh J, et al: Fifth ACCP Consensus Conference on Antithrombotic Therapy. Chest. 1998, 114 (suppl): 439s-769s.
Ravaud P, Durieux P, Fourcade A, and the Comité Scientifique Thrombose: Prophylaxie des thromboses veineuses post-operatories: recommendations de la Ássitance Publique-Hopitaux de Paris. Sang Thromb Vaisseaux. 1995, 7: 119-129.
Weed LL, Weed L: Opening the black box of clinical judgement-an overview. BMJ. 1999, 319: 1279.
Peabody JW, Luck J, Glassman P, Dresselhaus TR, et al: Comparison of vignettes, standardised patients, and chart abstraction: a prospective validation study of 3 methods for measuring quality. JAMA. 2000, 283: 1715-1722. 10.1001/jama.283.13.1715.
Durieux P, Rémy N, Philippe R, et al: A Clinical Decision Support System for Prevention of Venous Thromboembolism: Effect on Physician Behaviour. JAMA. 2000, 283: 2816-2821. 10.1001/jama.283.21.2816.
Pre-publication history
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6963/2/16/prepub
Acknowledgements
We thank Dr. M. Pellizzari, Dr G. Amestoy and Dr. R. Martin for their helpful revision of the manuscript. We also want to thank our colleagues who kindly accepted to participate in this study and Dr P Casais for her work with the clinical practice guideline.
Author information
Authors and Affiliations
Corresponding author
Additional information
Competing interests
None declared
Authors' contributions
MO y DG designed the study and created the clinical vignettes. MO analyzed the data and wrote the manuscript. KL and PM interviewed the physicians. All authors read and agreed with the final manuscript.
Rights and permissions
This article is published under license to BioMed Central Ltd. This is an Open Access article: Verbatim copying and redistribution of this article are permitted in all media for any purpose, provided this notice is preserved along with the article's original URL.
About this article
Cite this article
O'Flaherty, M., Lerum, K., Martin, P. et al. Low agreement for assessing the risk of postoperative deep venous thrombosis when deciding prophylaxis strategies: a study using clinical vignettes. BMC Health Serv Res 2, 16 (2002). https://doi.org/10.1186/1472-6963-2-16
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/1472-6963-2-16