- Research article
- Open Access
An audit tool for assessing the appropriateness of carotid endarterectomy
BMC Health Services Research volume 4, Article number: 17 (2004)
To update appropriateness ratings for carotid endarterectomy using the best clinical evidence and to develop a tool to audit the procedure's use.
A nine-member expert panel drawn from all the Canadian Specialist societies that are involved in the care of patients with carotid artery disease, used the RAND Appropriateness Methodology to rate scenarios where carotid endarterectomy may be performed. A 9-point rating scale was used that permits the categorization of the use of carotid endarterectomy as appropriate, uncertain, or inappropriate. A descriptive analysis was undertaken of the final results of the panel meeting. A database and code were then developed to rate all carotid endarterectomies performed in a Western Canadian Health region from 1997 to 2001.
All scenarios for severe symptomatic stenosis (70–99%) were determined to be appropriate. The ratings for moderate symptomatic stenosis (50–69%) ranged from appropriate to inappropriate. It was never considered appropriate to perform endarterectomy for mild stenosis (0–49%) or for chronic occlusions. Endarterectomy for asymptomatic carotid disease was thought to be of uncertain benefit at best. The majority of indications for the combination of endarterectomy either prior to, or at time of coronary artery bypass grafting were inappropriate. The audit tool classified 98.0% of all cases.
These expert panel ratings, based on the best evidence currently available, provide a comprehensive and updated guide to appropriate use of carotid endarterectomy. The resulting audit tool can be downloaded by readers from the Internet and immediately used for hospital audits of carotid endarterectomy appropriateness.
Evidence for efficacy of an intervention may be clearly established, but its expected effectiveness outside of a clinical trial setting, the values of decision makers, and resources available at a given point in time may all affect how that intervention is brought into generalized clinical care . These may vary across different countries and decision-makers and this may lead to different interpretations of the evidence [2–5]. Hence different criteria, guidelines and appropriateness ratings often need to be developed in different countries, which may then guide the clinical audit of interventions and outcomes.
The first appropriateness study on the use of carotid endarterectomy (CE), prompted by concerns of its unnecessary use, suggested that only a third of CE procedures were appropriate . It was published in 1988, prior to the large randomized controlled trials [7–10] that provide our current evidence base for the majority of indications for this procedure. In spite of the proliferation of the highest level of evidence available to guide patient management, controversy still exists as to what constitutes appropriate use of the procedure, especially in those with asymptomatic disease [11, 12]. This may explain some of the marked variation in the rate of usage of CE between countries and regions [13, 14].
The objectives of this paper were: firstly, to describe the development of contemporary appropriateness ratings for carotid endarterectomy according to a panel of Canadian experts; and secondly, to develop an audit tool for chart review based on these ratings. The ratings generated from this process are likely to be of particular relevance to health systems that resemble Canada's single payer system, and the tolls that we provide will facilitate the conduct of CE appropriateness audits as a means of ensuring optimal utilization of this procedure.
The RAND/ UCLA Appropriateness methodology has been described in depth elsewhere [1, 2, 15–18]. In brief, a comprehensive literature review is conducted collecting all available evidence for a given diagnostic or therapeutic intervention. A set of scenarios is developed covering all possible clinical situations under which that intervention might be performed. A panel of experts is asked to independently rate each scenario guided by the evidence presented. The rating scale ranges from 1 to 9; 1 to 3 indicating an inappropriate indication, 4 to 6 an uncertain indication, and 7 to 9 an appropriate indication. (For all definitions see Appendix) These initial ratings are collated. The panel is then convened and the participants provided with their initial rating and the median rating of the group for each scenario. Under the guidance of a chair, the group discusses each scenario with the relevant evidence and individually re-rates each scenario. The median of these final ratings is used to determine the appropriateness of each scenario according to the schema as presented above. Given that none of the scenarios, provoked disagreement among the panellists according to the RAND definition this played no role in the allocation of appropriateness.
The Medline database was used as the primary route for searching for relevant material for the literature review for this study. Evidence was rated as described elsewhere . Where evidence was available for a given indication from randomized controlled trials, no further literature was presented. If this level of evidence was unavailable, literature from the next available evidence hierarchy was presented.
The nine panel members were selected from nominations made by presidents of the various Canadian national societies representing stroke neurology (two members), neurosurgery (two), vascular surgery (two), internal medicine (one), family medicine (one) and neuroradiology (one). The consensus conference proceedings were taped and transcribed.
A descriptive analysis of the final appropriateness ratings of both symptomatic and asymptomatic stenoses by degree of internal carotid artery stenosis was performed. For all scenarios in this analysis, conventional angiography was the imaging modality assumed to have determined the degree of stenosis. The transcribed discussions of the panel were used to help to identify the 'key elements of reasoning' behind the panel's decisions. These 'key elements of reasoning' are presented in narrative form in the text of the Results section to accompany the quantitative reporting on panel ratings for the clinical scenarios.
A Microsoft Access® Database was developed to enable easy collection of the necessary clinical data to determine appropriateness from patient charts. Data can be entered directly into this database. In parallel, we developed an algorithm written in SAS version 8 for transforming the raw data gathered from chart review to final appropriateness ratings. We make both the Microsoft Access® Database and the SAS code available for download from the Internet .
A reviewer used the database to collect the clinical data by chart review on all carotid endarterectomies performed in the Calgary Health Region in the period of 1997 – 2001, a total of 801 cases. The data collected were then run with the SAS code and its ability to classify the cases tested.
Results and Discussion
Symptomatic carotid stenosis
The final appropriateness ratings for symptomatic internal carotid artery stenoses are summarised in Table 1. The ratings for severe (70 – 99%) symptomatic stenosis are the highest of any group. The panel felt that a majority of patients will benefit from treatment, because the magnitude of benefit conferred by surgery over medical treatment  has been shown to be so great that the procedure is likely to be of benefit even when performed by surgeons with a higher complication rate than in the trials.
Appropriateness ratings ranged from appropriate to inappropriate for moderate (50 – 69%) symptomatic stenosis. This reflected the reduced margin of benefit for CE when compared to severe stenosis in the trials [3, 4], and the higher surgical complication rates seen in real-life practice . Remote transient events, either hemispheric or retinal, and early operation after moderate stroke were considered to be of uncertain benefit. The presence of a contralateral occluded artery tended to lead to an indication being deemed less appropriate.
CE for those with mild (0–49%) stenoses was never rated as appropriate. Indications thought to be of uncertain benefit were multiple or crescendo hemispheric TIAs in association with an ulcerated plaque. The evidence showing that the presence of a symptomatic, irregular plaque was associated with a higher risk of recurrent stroke was a major factor that guided the panel in this decision . However, there was a disparity of opinion among panellists, which did not meet the RAND definition of disagreement, as to rating of these particular indications, displaying the concerns raised by unplanned subanalyses of randomized controlled trials, which may be misleading .
Symptomatic acute occlusions were rated as borderline uncertain/ inappropriate, whereas operating on a chronic occlusion was always deemed inappropriate. The difference between these two indications relates to the limited evidence from case series showing the possibility of improvement in outcome from stroke with emergency endarterectomy for acute occlusion .
Asymptomatic carotid stenosis
Table 2 shows the final appropriateness ratings for asymptomatic internal carotid artery stenosis. For asymptomatic stenoses less than 60%, all agreed that CE was inappropriate. There was a wide range of opinion on the ratings for endarterectomy for stenosis greater than 60% and the scores changed considerably between the two rating sessions, moving from borderline appropriate to a final rating of uncertain benefit. Only the rating of severe asymptomatic stenosis with brain infarct present on brain imaging approached the definition but did not meet the RAND definition of disagreement. Otherwise only a single rater viewed these scenarios as an appropriate. This relates to concerns with the generalization of the results of the Asymptomatic Carotid Atherosclerosis Study (ACAS) . The rate of stroke and/or death in the surgical group of that trial was 2.3%. Case series have been unable to match this exceptionally low complication rate, with rates ranging from 2.5%  to 5.6% . Most of the panel felt that in real life the marginal benefit seen in the ACAS trial would not be seen in real life. However, in view of the positive randomized trial evidence, they felt that at worst this be rated as an uncertain indication. It was felt that the results of the then ongoing Asymptomatic Carotid Surgery Trial would help to resolve the place of carotid endarterectomy for asymptomatic stenosis.
Carotid endarterectomy associated with coronary artery bypass grafting
Table 3 shows the final appropriateness ratings where carotid endarterectomy is performed either with or prior to coronary artery bypass grafting. The imperative to combine procedures is derived from anecdotal experience that those patients with asymptomatic carotid stenosis are at higher risk of stroke following coronary artery bypass grafting. However, case series in the literature both support and refute this [27–30]. The panel discussed a report that the combination of both procedures in a recent large case series appeared to put patients at an unacceptable risk of stroke or death (17.4%) . Therefore, in all but the highest degree of asymptomatic stenosis, the panel thought it was an inappropriate indication for surgery, and even in that situation, an uncertain indication only.
Performance of the appropriateness rating algorithm
Of the 801 CE cases examined, 145 (18.1%) were performed for asymptomatic carotid stenosis. When these data were run with the Access database and SAS code that we developed, 541 were deemed appropriate (67.5%), 61 inappropriate (7.6%), 175 uncertain (21.9%). The most common reasons for inappropriate CE were as follows: asymptomatic stenosis less than 60% (28 cases), presence of concurrent medical co-morbidity (dementia, recent myocardial infarction) (18 cases), symptomatic stenosis <50% (five cases), and chronic occlusion (eight cases).
Only 24 cases (3%) were not classified. The reasons for the unclassified cases are as follows: no carotid artery imaging results available (thirteen cases), no information available regarding the degree of contralateral carotid artery stenosis in the presence of a moderate symptomatic stenosis (five cases), endarterectomy ipsilateral to ischaemic symptoms or side not identified (six cases).
The current study involved an all-Canadian panel of experts from the whole spectrum of specialties (Family Medicine, Internal Medicine, Neuroradiology, Neurology, Neurosurgery and Vascular Surgery) that are involved in the care of patients with carotid artery disease. Ratings of a balanced panel have been shown to reflect the opinion in the broader physician community  and to avoid the bias inherent in having only practitioners of the intervention doing the rating . Each panellist was nominated by his/her respective specialty society and represented Canada geographically well with panellists from Nova Scotia to British Columbia. Their interpretation of the evidence could therefore be expected to have included a representative view of the current values and resource implications of the Canadian healthcare system.
These ratings identify ongoing areas of uncertainty that could be used to identify priority areas for research. The combination of coronary artery bypass grafting in association with carotid endarterectomy requires further close examination, especially given the high complication rates seen in the Multistate utilization study . CE for asymptomatic carotid stenosis remains an area of uncertainty. Hopefully, the completed Asymptomatic Carotid Surgery Trial will shed further light on the efficacy of CE for asymptomatic disease, but it may be that further work is merited to risk stratify those for whom CE is appropriate. The ratings presented here, however, represent the Canadian viewpoint on the decision to perform CE for asymptomatic carotid disease until the Asymptomatic Carotid Surgery Trial is published. These will remain valid in determining the appropriateness of all CE performed prior to the date of its publication and for some time afterward, until clinical experts and guideline-producing bodies fully reappraise the role of CE in asymptomatic carotid disease after the new trial's publication.
This ongoing area of uncertainty with regard to CE for asymptomatic internal carotid artery stenosis led to the conservative ratings of this Canadian Panel, rating it an intervention of uncertain benefit at best. This is in keeping with international opinion leaders . In contrast, this is discrepant from guidelines , appropriateness reviews [33–35] and practice patterns [20, 36] from the United States, where the approach toward asymptomatic disease is more aggressive with surgical intervention being seen as standard of care. The synthesis of the evidence by this panel in the form of the appropriateness ratings may therefore be of considerable value to clinicians and policy-makers in health care systems that resemble Canada's single payer system, as opposed to the more procedurally aggressive American system.
Perhaps most importantly, this paper provides access to downloadable audit tools that readers can use to rapidly implement carotid endarterectomy appropriateness audits in their own clinical setting . Both the data collection template and accompanying SAS code are easy to use, and will enable interested parties to perform such CE appropriateness work to compare their local standards of care with the latest Canadian interpretation of the available evidence.
Appropriate – where a procedure is worth doing if the expected medical benefit to the patient (health status, quality of life, longevity) exceeds the expected negative consequences to the patient (pain, disability, risk of death).
Inappropriate – where a procedure should not be performed if the expected negative consequences to the patient exceeds the expected medical benefit to the patient.
Uncertain – where the net result of a procedure's expected medical benefit and negative consequences are equivocal.
Agreement – scenarios where no more than two of the ratings are outside of the 3-point region containing the median result.
Disagreement – scenarios where three or more readings are in the 1-to-3 region and three or more readings are in the 7-to-9 regions.
Asymptomatic carotid artery stenosis – Carotid stenosis that has been identified through imaging, which on thorough history and examination has no ascribable symptoms.
Symptomatic carotid artery stenosis – Carotid artery stenosis, which has been identified following either transient monocular blindness (retinal TIA) or retinal infarction, or transient cerebral ischemia (hemispheric TIA) or stroke.
Crescendo TIA – recurrent transient cerebral or retinal ischaemia in the distribution of the carotid artery, characterized by a definite change in pattern, such as, (1) increased frequency, multiple episodes in a single day, over several days, or a cluster of spells over the course of several days, (2) increased duration, that is, spells lasting longer than the primary event, or several hours in duration, or (3) increased severity, that is, spread in the distribution of ischaemia with greater or new motor, sensory, speech, or visual defects.
Multiple TIA – multiple TIAs that do not match the definition of crescendo TIAs.
Moderate stroke – corresponds to a disability of 3,4 or 5 on the modified Rankin Scale.
Mild stroke – corresponds to a disability of 1 or 2 on the modified Rankin Scale.
Muir Gray JA: Evidence Based Healthcare. 1997, Churchill Livingstone: London
Brook RH, Kosecoff JB, Park RE, Chassin MR, Winslow CM, Hampton JR: Diagnosis and treatment of coronary disease: Comparison of Doctors' attitudes in the USA and UK. Lancet. 1988, 1: 750-753. 10.1016/S0140-6736(88)91550-4.
McGlynn EA, Naylor CD, Anderson GM, Leape LL, Park RE, Hilborne LH, et al: Comparison of the Appropriateness of Coronary Angiography and Coronary Artery Bypass Graft Surgery between Canada and New York State. JAMA. 1994, 272: 934-940. 10.1001/jama.272.12.934.
Fraser GM, Pilpel D, Kosecoff J, Brook RH: Effect of Panel Composition on Appropriateness Ratings. Int J Qual Health Care. 1994, 6: 251-255.
Leape LL, Park RE, Kahan JP, Brook RH: Group Judgements of Appropriateness: The effect of Panel Composition. Qual Assur Health Care. 1992, 4: 151-159.
Winslow CM, Kosecoff JB, Chassin M, Kanouse DE, Brook RH: The appropriateness of carotid endarterectomy. N Engl J Med. 1988, 318: 721-727.
North American Symptomatic Carotid Endarterectomy Trial Collaborators: Beneficial effect of carotid endarterectomy in symptomatic patients with high-grade stenosis. N Engl J Med. 1991, 325: 445-453.
Barnett HJ, Taylor DW, Eliasziw M, Fox AJ, Ferguson GG, Haynes RB, Rankin RN, Clagett GP, Hachinski VC, Sackett DL, Thorpe KE, Meldrum HE, Spence JD: Benefit of carotid endarterectomy in patients with symptomatic moderate or severe stenosis. N Engl J Med. 1998, 339: 1415-1425. 10.1056/NEJM199811123392002.
European Carotid Surgery Trialists' Collaborative Group: Randomized trial of endarterectomy for recently symptomatic carotid stenosis: final results of the MRC European Carotid Surgery Trial (ECST). Lancet. 1998, 351: 1379-1387.
Executive Committee for the Asymptomatic Carotid Atherosclerosis Study: Endarterectomy for Asymptomatic Carotid Artery Stenosis. JAMA. 1995, 273: 1421-1428.
Barnett HJM, Meldrum HE, Eliasziw M: The Appropriate Use of Carotid Endarterectomy. CMAJ. 2002, 166: 1169-1179.
Warlow CP: Carotid endarterectomy for asymptomatic carotid stenosis. Better data, but the case is still not convincing. BMJ. 1998, 317: 1468.
Tu JV, Hannan EL, Anderson GM, Iron K, Wu K, Vranizan K, Popp AJ, Grumbach K: The fall and rise of carotid endarterectomy in the United States and Canada. N Engl J Med. 1998, 339: 1441-1447. 10.1056/NEJM199811123392006.
Feasby TE, Quan H, Ghali WA: Geographic variation in the rate of carotid endarterectomy in Canada. Stroke. 2001, 32: 2417-2422.
Kahn KL, Kosecoff J, Chassin MR, Flynn MF, Fink A, Pattaphongse N, Solomon DH, Brook RH: Measuring the clinical appropriateness of the use of a procedure: Can we do it?. Med Care. 1988, 26: 415-422.
Brook RH: The RAND/UCLA appropriateness method. In Methodological Perspectives. 1995, AHCPR Pub No. 95-0009, Rockville, MD, 59-70.
Kosecoff J, Fink A, Brook RH, Chassin MR: The appropriateness of using a medical procedure: Is information in the medical record valid?. Med Care. 1987, 25: 196-201.
McGlynn EA, Kosecoff J, Brook RH: Format and conduct of consensus development conferences, multination comparison. Int J Technol Assess Health Care. 1990, 6: 450-469.
Oxford Centre for Evidence-based Medicine Levels of Evidence. [http://cebm.net/levels_of_evidence.asp#levels]
Appropriateness Database, SAS Algorithun Instructions for use. At the Centre for Health and Policy Studies home page, select "Research and Publications" and then "Current Projects" to access database., [http://www.chaps.ucalgary.ca/index.html]
Kresowik TF, Bratzler D, Karp HR, Hemann RA, Hendel ME, Grund SL, Brenton M, Ellerbeck EF, Nilasena DS: Multistate utilization, processes and outcomes of carotid endarterectomy. J Vasc Surg. 2001, 33: 227-235. 10.1067/mva.2001.111881.
Rothwell PM, Gibson R, Warlow CP: Interrelation between Plaque Surface Morphology and Degree of Stenosis on Carotid Angiograms and the Risk of Ischaemic Stroke in Patients with Symptomatic Carotid Stenosis. Stroke. 2000, 31: 615-621.
Oxman A, Guyatt G: Summarizing the evidence: when to believe a subgroup analysis. In Users Guides to the Medical Literature: A Manual for Evidence-Based Clinical Practice. Edited by: Guyatt G, Rennie D. 2002, AMA Press, Chicago, Illinois, 553-565.
Welling RE, Cranley JJ, Krause RJ, Hafner CD, Arbaugh JJ, Roedersheimer LR: Surgical therapy for recent total occlusion of the internal carotid artery. J Vasc Surg. 1984, 1: 57-61. 10.1067/mva.1984.avs0010057.
Cebul RD, Snow RJ, Pine R, Hertzer NR, Norris DG: Indications, outcomes, and provider volumes for carotid endarterectomy. JAMA. 1998, 279: 1282-1287. 10.1001/jama.279.16.1282.
Hartmann A, Hupp T, Koch HC, Dollinger P, Stapf C, Schmidt R, Hofmeister C, Thompson JL, Marx P, Mast H: Prospective study on the complication rate of carotid surgery. Cerebrovasc Dis. 1999, 9: 152-156. 10.1159/000015945.
Safa TK, Friedman S, Mehta M, Rahmani O, Scher L, Pogo G, Hall M: Management of coexisting coronary artery and asymptomatic carotid artery disease: report of a series of patients treated with coronary bypass alone. Eur J Vasc Endovasc Surg. 1999, 17: 249-252. 10.1053/ejvs.1998.0752.
Barnes RW, Liebman PR, Marszalek PB, Kirk CL, Goldman MH: The natural history of asymptomatic carotid disease in patients undergoing cardiovascular surgery. Surgery. 1981, 6: 1075-1083.
Brener BJ, Brief DK, Alpert J, Goldenkranz RJ, Parsonnet V: The risk of stroke in patients with asymptomatic carotid stenosis undergoing cardiac surgery: a follow-up study. J Vasc Surg. 1987, 5: 269-277. 10.1067/mva.1987.avs0050269.
Darling RC, Dylewski M, Chang BB, Paty PS, Kreienberg PB, Lloyd WE, Shah DM: Combined carotid endarterectomy and coronary artery bypass grafting does not increase the risk of perioperative stroke. Cardiovasc Surg. 1998, 6: 448-452. 10.1016/S0967-2109(98)00030-1.
Ayanian JZ, Landrum MB, Normand SL, Guadagnoli E, McNeil BJ: Rating the appropriateness of coronary angiography – do practicing physicians agree with an expert panel and with each other?. N Engl J Med. 1998, 338: 1896-1904. 10.1056/NEJM199806253382608.
Biller J, Feinberg WM, Castaldo JE, Whittemore AD, Harbaugh RE, Dempsey RJ, Caplan LR, Kresowik TF, Matchar DB, Toole JF, Easton JD, Adams HP, Brass LM, Hobson RW, Brott TG, Sternau L: Guidelines for Carotid Endarterectomy. Circulation. 1998, 97: 501-509.
Karp HR, Flanders WD, Shipp CC, Taylor B, Martin D: Carotid Endarterectomy among Medicare Beneficiaries. Stroke. 1998, 29: 46-52.
Bratzler DW, Oehlert WH, Murray CK, Bumpus LJ, Moore LL, Piatt DS: Carotid endarterectomy in Oklahoma Medicare beneficiaries: patient characteristics and outcomes. J Okla State Med Assoc. 1996, 89: 423-429.
Halm EA, Chassin MR, Tuhrim S, Hollier LH, Popp AJ, Ascher E, Dardik H, Faust G, Riles TS: Revisiting the appropriateness of carotid endarterectomy. Stroke. 2003, 34: 1464-1471. 10.1161/01.STR.0000072514.79745.7D.
Chaturvedi S, Meinke JL, St Pierre E, Bertasio B: Attitudes of Canadian and US Neurologists Regarding Carotid Endarterectomy for Asymptomatic Stenosis. Can J Neurol Sci. 2000, 27: 116-119.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6963/4/17/prepub
Supported by a research grants from the Heart and Stroke Foundation of Alberta, the Northwest Territories and Nunavut, and the Canadian Institutes of Health Research. Dr. Kennedy holds a salary award from the Canadian Stroke Network, the Heart and Stroke Foundation of Canada, the Canadian Institutes of Health Research and AstraZeneca. Dr. Ghali is supported by a Health Scholar Award from the Alberta Heritage Foundation for Medical Research and holds a Government of Canada Research Chair in Health Services Research.
Janet Bennett transcribed the tapes of the Appropriateness Conference. Tara Lye performed the literature search and helped organize the appropriateness conference. Louis Girard developed the database programming. Marie McClelland performed the chart review. Danielle Southern developed the website.
The Expert Panel comprised: Dr Walter Montanero, neuroradiologist, Toronto; Dr. M Christopher Wallace, neurosurgeon, Toronto; Dr. Steven J Phillips, neurologist, Halifax; Dr. Louise-Helene Lebrun, neurologist, Montreal; Dr. Daryl S Kucey, vascular surgeon, Toronto; Dr. Anthony JG Salvian, vascular surgeon, Vancouver; Dr. Rupinder K Soin, family physician, Edmonton; Dr. Anthony M Kaufmann, neurosurgeon, Winnipeg; Dr. Finlay A McAllister, general internist, Edmonton.
JK prepared evidence summaries for the appropriateness conference, analysed the data and wrote the initial manuscript. HQ contributed to planning the study and editing the manuscript. TEF wrote the application for funding, planned the study, participated in the analysis, and edited the manuscript. WAG was involved in writing the application for funding, planning the study, performing the data analysis and editing the manuscript. All authors read and approved the final manuscript.
Rights and permissions
About this article
Cite this article
Kennedy, J., Quan, H., Feasby, T.E. et al. An audit tool for assessing the appropriateness of carotid endarterectomy. BMC Health Serv Res 4, 17 (2004). https://doi.org/10.1186/1472-6963-4-17
- Coronary Artery Bypass Grafting
- Carotid Endarterectomy
- Asymptomatic Carotid
- Carotid Artery Disease
- Audit Tool