Low agreement for assessing the risk of postoperative deep venous thrombosis when deciding prophylaxis strategies: a study using clinical vignettes

Abstract

Background

Several clinical practice guidelines (CPG) on antithrombotic prophylaxis in surgical patients help to decide about the prophylaxis strategy based on the patient risk of deep venous thrombosis (DVT). However, the physician risk estimates of DVT could have little inter-observer reproducibility, which could lead to different individual prophylaxis practices.

Methods

Physicians were asked to evaluate DVT risk in eight clinical vignettes, describing actual patients cared for in our hospital. The vignettes included all possible levels of DVT risk.

Results

The degree of prophylaxis strategies accuracy was 63% (95% CI 523–75%). Overall agreement was 0.32 (z = 7.61, p < 0.001) and for each level of risk kappa was 0.38 (z = 6.50, p < 0.001); 0.1 (z = 1.65, p < 0.049) and 0.5 (z = 8.45, p < 0.001) for small, moderate and high risk group respectively

Conclusions

Our results showed that there is poor agreement when physicians have to evaluate the risk for postoperative DVT, and in the cases of low and moderate risks of DVT there is the smallest agreement. In addition, the data also showed that the overall accuracy of DVT prophylaxis strategy was only moderate and the risk evaluation did not correlate to the selection of the strategy. The issue of inter-observers variability should be taken into account when CPG performance are analysed, especially when considering the risk-evaluation to choose the appropriate actions.

Background

Deep venous thrombosis (DVT) prophylaxis is an effective intervention in the postoperative period [1]. Several clinical practice guidelines (CPG) on the topic have been published, with the aim to increase the utilisation and appropriateness of this practice. Usually, these guidelines established the choice of the prophylaxis strategies based on the type of surgery, as well as on the presence of specific patient's risk factors [1, 2]

To establish an individual patient risk level is a complex task that could lead to substantial variations between observers. The integration of several cognitive inputs and patient data under real work pressures is a task where the physicians' performance is sub-optimal [3].

In order to determine if there are variations in the postoperative DVT risk perception among physicians, we performed a study using clinical vignettes.

Methods

Study design

Clinical practice guideline

Our guideline on DVT prophylaxis is mostly based in the 5^th ACCP consensus conference recommendation [1], where risk-assessment approaches are used to decide the prophylaxis strategy.

In our guideline, the postoperative DVT risk is evaluated basing on the items that follow: the age, the presence of venous thrombosis risk factors and the duration of surgery. With this information we could classify the patients in low, moderate and high-risk groups. Early mobilisation or graded compression stockings are prescribed in the case of low risk group patients. Moderate risk patients are prescribed fixed subcutaneous calcium heparin and graded compression stockings could replace heparin if high haemorrhagic risk exists. High-risk patients received nadroparin but when it was deemed inappropriate due to a high haemorrhagic risk, intermittent pneumatic calf compression (IPCC) was indicated. Both nadroparin and IPCC may be used in selected high-risk patients. The guideline has been widely publicized throughout the institution and it is available in the Hospital Intranet.

Clinical Vignettes

We have written eight comprehensive clinical vignettes including patients that cover the entire risk spectrum in order to evaluate the inter-observer variability about the risk assessment,. The information was mostly taken from actual patients cared for at our hospital. Specific details were changed in order to respect patient confidentiality and to avoid clueing physicians about the appropriate decisions. The vignettes are summarised in the table. No consent is required in our hospital to use clinical record data if no breach to patient confidentiality is possible. One of us (DG) independently reviewed the clinical vignettes for accuracy and validation regarding the study objectives.

Table 1 Clinical Vignettes summary with risk level and proposed prophylaxis strategyin the CPG.

We randomly chose 10 physicians from the Internal Medicine and the Intensive Care Unit to read the vignettes. Then we asked them: 1) to estimate the inhospital complications risks (nosocomial infection, cardio-respiratory events, DVT risk and hospital mortality) and to classify them in three levels: low, moderate and high risk; and 2) to order a complete diagnostic and therapeutic plan for each vignette.

Statistical analysis

Percentages, along with 95% confidence intervals, were calculated for the accuracy of the strategy selection compared to the intended level of risk for the vignette, as decided by the investigators.

The inter-observer agreement was evaluated using the kappa statistic for multiple raters and for multiple categories. Statistical calculations were performed with STATA 6.0.

Results

The degree of appropriateness of prophylaxis strategies used were 63% (95% CI 52–75%).

There was no correlation found between the accuracy of the prophylaxis strategies and the DVT risk evaluated by the physician (χ² for trend 0.54, p = 0.46). Although no statistically significant, the low risk strata showed a higher rate of strategy inappropriate selection compared to the moderate and high risk ones (72%, 61% and 61.5 respectively, x2 = 0.7, p = 0.7). Overall agreement was 0.32 (z = 7.61, p < 0.001) and kappa by risk category was 0.38(z = 6.50, p < 0.001); 0.1(z = 1.65, p < 0.049) and 0.5 (z = 8.45, p < 0.001) for the low, moderate and high risk group respectively.

All the DVT risk wrong estimates in the low risk group were rated as "moderate" and in the moderate risk group as "low risk". In the high-risk group, 70% of the wrong estimates were "low risk" and the rest were classified as "moderate risk".

Discussion

Our results showed a poor agreement for assessing the risk of postoperative DVT, especially in the low and moderate risk groups. Wrongly classified patients who belong to moderate risk groups were classified in lower risk category. In addition, the data also showed that overall DVT prophylaxis strategy was not good enough and the risk evaluation did not correlate to the strategy selection.

As stated by Weed et al [3], our example on DVT risk assessment showed that integration of cognitive inputs and patient data must be systematically explored for different clinical settings since considerable variation may occur, especially when the physician behaviour is assessed.

The evaluation of the physician performance is a difficult task, as several factors from physicians themselves, healthcare structure and case complexity could modify the evaluation. Clinical vignettes are a validated measurement of the physician performance [4], as compared both to utilisation review and simulated patient-doctor encounters. This method permits controlling case-mix, an important confusing variable when the physician performance is evaluated.

A standardised way of risk evaluation could improve the quality of a key point in the decision process of the CPG, such as risk assessment. Computerized decision support systems have shown benefits in increasing usage and accuracy of DVT prophylaxis [5].

Conclusions

The issue of inter-observers variability should be taken into account to analyse variations in CPG performance, especially when considering risk-evaluation to select the appropriate actions.