Development of appropriateness explicit criteria for cataract extraction by phacoemulsification
© Ma Quintana et al; licensee BioMed Central Ltd. 2006
Received: 07 April 2005
Accepted: 02 March 2006
Published: 02 March 2006
Consensus development techniques were used in the late 1980s to create explicit criteria for the appropriateness of cataract extraction. We developed a new appropriateness of indications tool for cataract following the RAND method. We tested the validity of our panel results.
Criteria were developed using a modified Delphi panel judgment process. A panel of 12 ophthalmologists was assembled. Ratings were analyzed regarding the level of agreement among panelists. We studied the influence of all variables on the final panel score using linear and logistic regression models. The explicit criteria developed were summarized by classification and regression tree analysis.
Of the 765 indications evaluated by the main panel in the second round, 32.9% were found appropriate, 30.1% uncertain, and 37% inappropriate. Agreement was found in 53% of the indications and disagreement in 0.9%. Seven variables were considered to create the indications and divided into three groups: simple cataract, with diabetic retinopathy, or with other ocular pathologies. The preoperative visual acuity in the cataractous eye and visual function were the variables that best explained the panel scoring. The panel results were synthesized and presented in three decision trees. Misclassification error in the decision trees, as compared with the panel original criteria, was 5.3%.
The parameters tested showed acceptable validity for an evaluation tool. These results support the use of this indication algorithm as a screening tool for assessing the appropriateness of cataract extraction in field studies and for the development of practice guidelines.
Cataract is very prevalent in developed countries, and the incidence is expected to increase as the result of the aging of the population and increased life expectancies . In addition, cataract extraction is the most common surgical intervention in developed countries . Due to the high prevalence of this condition, clinicians, managers, and health payers must decide who should undergo cataract extraction. Variations in the use of cataract extraction also have been reported, which led to the need to study if overuse or underuse of the procedure was occurring [3–5]. The final goals are to increase the quality of care and the cost-effectiveness in our medical system, which depends on the development of appropriateness criteria to make appropriate decisions.
A way to develop explicit criteria is the methodology developed in the 1980s by the RAND-UCLA group . This is a very popular methodology that has been used in many different diagnostic and therapeutic procedures since then, including cataract extraction. RAND chose cataract extraction as one of the first procedures for which to develop appropriateness criteria . In the1990's, Tobacman et al. applied explicit criteria for cataract extraction following this methodology . Although phacoemulsification has been done since the 1970s and some of the cataract extractions that were reported previously in assessment of appropriateness were performed by phacoemulsification, at that time, phacoemulsification was just beginning to be performed, but now it is the primary surgical technique used for uncomplicated cataract extraction in developed countries . As happens when a new treatment is introduced, the criteria to perform the intervention may change, because new variables should be considered in the decision making process, and some variables that previously were important are no longer so.
The purpose of this study was to update the appropriateness criteria, using the RAND method, for patients undergoing cataract extraction by phacoemulsification exclusively.
Explicit criteria development
The criteria for measuring the appropriateness of cataract surgery were developed according to a previously described explicit method , i.e., the RAND appropriateness method, which consists of the following steps.
First, an extensive literature review was performed to summarize existing knowledge on the efficacy, effectiveness, risks, costs, and opinions about the use of phacoemulsification.
Second, from this review, a comprehensive and detailed list of mutually exclusive and clinically specific scenarios (indications) was developed in which cataract surgery by phacoemulsification might be performed. This list contained 765 indications in three categories: simple cataract (cataract with no other ocular pathologies that may affect the visual prognosis), cataract with diabetic retinopathy, and cataract with other ocular pathologies that may affect the visual prognosis. Each indication was specified in sufficient detail that patients within a given indication were reasonably homogeneous. The indications included the following variables. For patients with simple cataract, best-corrected visual acuity in the cataractous eye (three subgroups: ≥0.5, 0.2–0.4, ≤0.1), best-corrected visual acuity in the contralateral eye (three subgroups: ≥0.5, 0.2–0.4; ≤0.1), visual function (four categories: no impairment, glare, difficulty with recreational activities, or difficulty with activities of daily living); surgical complexity of the cataract procedure (three categories: a) No surgical complications or minor complexity anticipated, as the presence of narrow anterior chamber (corneal amplitude-iris <=2), deep-set eyes, extreme myopia without retinal involvement, posterior synechiae, or a small pupil. b) Medium complexity anticipated: Pseudoexfoliation with mydriasis >3 mm and without subluxation of the crystalline lens, dense cataract, poor pupil dilatation (mydriasis >3 mm, according to the dilatation guidelines), vitrectomized eye, poor patient cooperation during examination, and the presence of two or more minor factors. c) High complexity anticipated. Subluxation of the crystalline lens, fibrosis of the anterior capsule of the crystalline lens, brunescent cataract, posterior polar cataract, and the presence of two or more factors of medium complexity); and laterality of cataract (unilateral or bilateral).
For patients with diabetic retinopathy that may affect the visual prognosis and for patients with other ocular pathologies, the same variables were studied plus the anticipated visual acuity after intervention (in three subgroups, ≥0.5, 0.2–0.4; ≤0.1).
The 765 indications resulted from all possible combinations of the variables described and the respective categories. Additional file 1 (Appendix 1) contains a description of the variables and their categories. Cases in which phacoemulsification was performed in combination with other ophthalmic surgical techniques were excluded.
Third, we compiled a national panel of ophthalmologists (doers and non doers of cataract extraction) recognized in the field, the names of whom were provided by their respective medical societies and members of our research team. The panelists were provided with the literature review and the list of indications, and they rated each indication for the appropriateness of performing phacoemulsification, considering the average patient and average physician in the year 2004. Appropriateness was defined as meaning that the "expected health benefit exceeds the expected negative consequences by a sufficiently wide margin to make cataract surgery worth performing."
Ratings were scored on a 9-point scale. Cataract surgery for a specific indication was considered appropriate if the panel's median score was between 7 and 9 without disagreement, inappropriate if the value was between 1 and 3 without disagreement, or uncertain if the median rating was between 4 and 6 or if the members of the panel disagreed. Disagreement was defined as occurring when at least four panelists rated an indication from 1 to 3 and at least another four rated it from 7 to 9. Agreement if less than four panelists rated the indication outside the 3-point region (1–3; 4–6; 7–9) containing the median; and indeterminated if agreement nor disagreement was found. This method did not attempt to force panelists to reach agreement on appropriateness.
The ratings were confidential and took place in two rounds, using a modified Delphi process. The first round was performed by mail before the members of the panel met. The results were collated and presented to the 12 panelists at the 1-day second-round meeting. Each panelist also received the anonymous ratings of the other panelists and a reminder of his or her own ratings. After extensive discussion, the panelists revised the indications according to the above-mentioned definition of appropriateness. Each panelist rated 765 separate indications.
To determine the use of all theoretical indications created in clinical practice, data related to the algorithm variables were gathered for 1,053 patients on a waiting list to undergo cataract extraction by phacoemulsification from six ophthalmologic services at six area hospitals. These data were collected prospectively by the ophthalmologists of each center. The number of theoretical indications used in clinical practice was calculated for each of the three diagnostic groups.
The mean appropriateness ratings of all indications and the mean change from rounds 1 to 2 were calculated for each panelist. The mean difference from each panelist's score for each indication from the panel median of each indication also was measured for both rounds. A "conformity score" , describing each panelist's tendency to change his or her ratings in the direction of the round 1 panel median rating also was calculated. This score was defined as a decrease in mean absolute deviation from the round 1 median between rounds 1 and 2. The higher the conformity score, the more the individual's round 2 rating shifted toward the median of the round 1 rating.
We studied the reliability of the 12 panelists scores at 2nd round by performing an intraclass correlation coefficient.
Study of the validity of the explicit criteria: Determinants of appropriateness scores and their contribution to the model explanation were assessed with the least-squares regression model, with the median of the panelists' ratings being the dependent variable for each indication, and the variables in the algorithm being the covariates. Ordinal logistic regression also was used, and the classification of the panelists' scores in the categories of appropriate, uncertain, or inappropriate was the dependent variable. Both models were compared regarding the degree of variability explained by each variable. R-square and -2 log L statistics were used, respectively.
Algorithms in decision tree form, which should permit rapid estimation of appropriateness in practice, were compiled from the final results by classification and regression trees (CART) analysis . CART was used to build a classification tree with the appropriateness score of the panel ratings as a dependent variable, as categorical variable (appropriate, uncertain or inappropriate). Misclassification error of the CART, compared to the original panel classification, as the gold standard, was calculated as the ratio of the number of indications erroneously classified by the classification tree divided by the total number of indications.
All statistical analyses were performed using the SAS for Windows, version 8, except for the CART analysis with which we used S-Plus 2000 (MathSoft Inc., 1999) statistical software.
Agreement and appropriateness judgment of the panel of experts by round.
Appropriateness and agreement during second round of the panel of experts, by category.
Cataract with Diabetic Retinopathy
Cataract with Other Pathologies
Panelists scores during both rounds.
Contribution of variables to the appropriateness scoring, by chapter.
Variable by Category
-2 log L
Type of Cataract
Contralateral Visual Acuity
Cataract with Diabetic Retinopathy
Expected Visual Acuity
Contralateral Visual Acuity
Laterality of Cataract
Cataract with Other Pathologies
Anticipated Visual Acuity
Contralateral Visual Acuity
Laterality of Cataract
Finally, the criteria developed by the panel of experts were summarized by CART analysis. Figure 1 (Additional file 2) shows the decision tree for simple cataract. The indications considered appropriate were: 1) visual acuity in the cataractous eye lower than 0.5 and difficulty with activities of daily living, 2) visual acuity of 0.1 or less and glare or difficulty with recreational activities, 3) visual acuity between 0.2 to 0.4 and difficulty with recreational activities (if low or medium surgical complexity), or in cases in which glare is the patient complaint, low surgical complexity, or if medium surgical complexity a contralateral visual acuity between 0.2–0.4, and 4) visual acuity higher than 0.4 if there is difficulty with activities of daily living and low or medium surgical complexity. Globally, cataract extraction was considered inappropriate in patients without impaired visual function with a visual acuity higher than 0.1, or if in the presence of glare or difficulty with recreational activities the visual acuity was higher than 0.4, depending on surgical complexity.
In patients with diabetic retinopathy (Figure 2, Additional file 2), the indications considered appropriate were: 1) visual acuity in the cataractous eye of 0.1 or less and difficulty with activities of daily living and an anticipated postoperative visual acuity higher than 0.1, or unilateral cataract with low-to-medium surgical complexity and difficulty with recreational activities or glare and an anticipated postoperative visual acuity between 0.2 to 0.4, or difficulty with recreational activities or glare and an anticipated postoperative visual acuity higher than 0.4. In addition, patients with a visual acuity between 0.2 and 0.4 and an anticipated postoperative visual acuity higher than 0.4 with difficulty with activities of daily living, or with difficulty with recreational activities but with low or medium surgical complexity.
In patients with other ocular pathologies (Figure 3, Additional file 2), the indications considered appropriate were: 1) visual acuity in the cataractous eye of 0.1 or lower with glare, with low surgical complexity, and with an anticipated postoperative visual acuity higher than 0.4, 2) difficulty with recreational activities and an anticipated visual acuity higher than 0.1, 3) difficulty with activities of daily living and an anticipated postoperative visual acuity higher than 0.1, 4) visual acuity between 0.2 and 0.4 and an anticipated postoperative visual acuity higher than 0.5 with difficulty with activities of daily living or recreational activities but low surgical complexity.
Comparison of the classification by the decision trees with the original panel of experts' scores.
Panel of Experts
Cataract with Diabetic Retinopathy
Cataract with Other Pathologies
We collected information on 1,053 patients to identify how many of the theoretical indications were used in clinical practice. Globally, 205 of 765 theoretical (26.8%) indications were assigned to 1,053 patients. By category, for simple cataract, 98 of 153 theoretical (64%) indications were assigned to 815 patients with that diagnosis. Of those with diabetic retinopathy, 22 of 306 theoretical (7.2%) indications were assigned to 27 patients. Of those with other ocular pathologies, 85 of 306 theoretical (27.8%) indications were assigned to 211 patients within that category.
RAND appropriateness methodology had been used to create explicit criteria to evaluate the appropriateness of cataract extraction [7, 8, 13, 14]. Nevertheless, as the developers of the RAND methodology pointed out, this methodology cannot capture changes over time with the advent of new diagnostic or therapeutic techniques . This also is the case with cataract extraction. Phacoemulsification is now the most frequently performed technique in developed countries for cataract extraction. Our study incorporates new explicit criteria that can be used with patients undergoing only phacoemulsification.
The work of our panel of experts showed a low disagreement rate among panelists in the second round, compared with other studies in which a similar methodology was used . Also, the intraclass correlation coefficient showed acceptable results. This partially support the reliability of the tool. This can be due to the fact that our panel of experts included only ophthalmologists. The agreement rate was slightly higher for simple cataract than for the other two categories and the proportion of indications scored as appropriate also was slightly higher. This is important because this was the most frequently encountered group in the pilot field study. As shown, previous panels comprised of doctors of one specialty tended to consider more scenarios appropriate than multidisciplinary panels . However, we assembled a mono-specialists panel of expert ophthalmologists because in our health system ophthalmologists alone make the decision to perform cataract surgery. In addition to their interest in our conclusions, our criteria should be available to other physician who may play a role in the referral of these patients to the ophthalmologist so that those patients can be considered for surgery if appropriate.
Our results indicated that among the variables included in the criteria, a few play an important role, as reflected by the linear and logistic regression of all the scores. In addition, the decision trees based on CART analysis used only a few variables. The three important variables were: with simple cataract, the preoperative visual acuity in the cataractous eye, as evaluated by the ophthalmologist; the subjective visual function reported by the patient; and in some cases the surgical complexity. In the other two diagnostic groups (cataract with diabetic retinopathy or with other ocular pathologies), the most important variables were the preoperative visual acuity in the cataractous eye, the subjective visual function and the anticipated postoperative visual acuity in the cataractous eye. Here, again, the preoperative visual acuity in the operated eye was the most important variable, while the visual function was third. These results partially support the face and construct validity of our criteria, since those variables have been identified in different studies as the ones that should play an important role when deciding the appropriateness of cataract extraction. However, studies that did not use the RAND methodology identified the same variables as relevant. Therefore, visual function has been reported by different studies as a key variable from the patient perspective . From the ophthalmologist perspective, the visual acuity in the cataractous eye is the primary variable, followed by others such as the presence of ocular comorbidities or the visual acuity in the contralateral eye .
The first study to create explicit criteria following the RAND method developed 1,953 indications during the first round that increased to 2,905 in the second round . The number of diagnoses (ocular comorbidities) included was greater than in our study. In our case, we collected information about the prevalence in clinical practice of all possible ocular pathologies in patients presenting with the need for cataract extraction before we created the indications. We gathered the information from the literature, our administrative databases, and the ophthalmologists of our research team. Since some diagnoses were extremely uncommon, we focused on those most frequently encountered: simple cataract, cataract with diabetic retinopathy, and a new global category called cataract with other ocular pathologies. The last group included some diagnoses that also had been included in the previous RAND studies  as separate groups, which led those authors to have a high number of indications. We included them in a single group due to the criteria of our ophthalmologists who considered first, from the standpoint of surgical intervention, that all the diagnostic groups included in this category have in common that affect the anticipated postoperative visual acuity. Second, they considered that there was no other additional criteria-variable- for each of those diagnostic groups who may force us to have them separately.
RAND methodology has some limitations. Among them, the excessive number of indications that were developed. In our case 765 were far fewer than those in previous studies of cataract surgery appropriateness or even in other RAND studies of other non-ophthalmic procedures [16, 20]. We considered other diagnostic groups, such as Fuchs' corneal dystrophy or uveitis, but decided not to included Fuchs' corneal dystrophy in this study due to the low prevalence; however, uveitis was included as a surgical complexity. A large number of indications compromised the work of the panel of experts because it forced them to spend considerable time scoring all indications. An additional problem related to the previous one is that the number of theoretical indications used in clinical practice were very few. We found that only 27% of all indications were used, and the percentages varied considerably among the three diagnosis groups, from 73% in simple cataract to 7.2% in cataract with diabetic retinopathy. This means that we asked the panel of experts to score an large number of indications that were unlikely to be present in clinical practice. This can bias the scoring of such indications because no evidence can be found in the literature about the efficacy of phacoemulsification for those indications due to the low prevalence, and because the panelists probably have no experience with them . For this reason, we recommend to future developers of explicit criteria using this methodology to, first, study the prevalence of the different diagnostic groups that lead to the intervention and then develop the theoretical indications that are more likely to be found in clinical practice. Additional criteria also should be the presence of important variability in the use of the procedure or the absence of evidence.
There are different ways to use these explicit criteria. As other authors have done with other procedures, and even with cataract extraction, the explicit criteria can be used for utilization review and health services research studies and can be converted to practice guidelines for use by clinicians, managers, or health payers. However, the criteria should be used cautiously, because as the developers of the RAND method pointed out, other patient circumstances not included in these criteria may play an important role in clinician decision making .
In conclusion, this study updates previous work that created explicit criteria for cataract extraction following the RAND methodology. The criteria created are based on variables identified by most studies as relevant, which partially support their validity. Finally, we summarized the results in accurate decision trees, which allow use of the criteria in clinical practice or in the development of practice guidelines.
This study was supported by a grant from the Fondo de Investigación Sanitaria of the thematic networks (G03/202). We also thank the following people for their contributions to this study: Eduardo Cortés (Hospital Severo Ochoa, Madrid); Javier Ferro (Hospital de Txagorritxu, Vitoria); José López-Garrido (Hospital de Galdakao, Bizkaia); Josep Gracia Martínez (Corporació Parc Taulí, Sabadell), Ma Fernanda Aranda del Prado, Rafael Márquez (Hospital Universitario Virgen del Rocío, Sevilla); Nerea Martínez Alday (Hospital de Cruces, Bizkaia); Santiago Mendialdea (Hospital Virgen de las Nieves, Granada); Javier Mendicute (Hospital Donostia, Donostia); Daniel Pascual Allén (Hospital de Móstoles, Madrid); Carmen Rodríguez Bermejo (Fundación Jimenez Díaz, Madrid); and Josep Visa Nasarre (Corporació Parc Taulí, Sabadell). The IRYSS-Appropriateness Cataract Group included the following co-investigators: Jesús Martínez-Tapias, Eduardo Aguayo (Hospital Universitario Virgen de las Nieves, Granada); Emilio Perea-Milla (Hospital Costa del Sol, Málaga); Juan Ramón Lacalle (Universidad de Sevilla); Marisa Baré, Gemma Navarro (Corporació Parc Taulí, Sabadell); Elena Andradas, Juan A. Blasco (Agencia Laín Entralgo, Madrid); Txomin Alberdi (Servicio de Oftalmología, Hospital de Galdakao, Bizkaia); Sandra de Fernando (Servicio de Oftalmología, Hospital de Cruces, Bizkaia); Fabiola Eder (Servicio de Oftalmología, Hospital de Donostia, Gipuzkoa); Jose Ma Beguiristain, Belén Elizalde (Dirección Territorial de Gipuzkoa); Idoia Garai (Dirección Territorial de Bizkaia); Joseba Pérez de Arriba (Dirección Territorial de Alava); Jose Ignacio Pijoan (Unidad de Investigación del Hospital de Cruces, Bizkaia); Felipe Aizpuru (Unidad de Investigación del Hospital de Txagorritxu, Alava); Amaia Bilbao (BIOEF); Nerea González, Iratxe Lafuente, Urko Aguirre; Susana Chic and Susana García (Unidad de Investigación del Hospital de Galdakao, Bizkaia).
- Kocur I, Resnikoff S: Visual impairment and blindness in Europe and their prevention. Br J Ophthalmol. 2002, 86: 716-722. 10.1136/bjo.86.7.716.View ArticlePubMedPubMed CentralGoogle Scholar
- Congdon N, Vingerling JR, Klein BE, West S, Friedman DS, Kempen J, O'Colmain B, Wu SY, Taylor HR: Prevalence of cataract and pseudophakia/aphakia among adults in the United States. Arch Ophthalmol. 2004, 122: 487-494. 10.1001/archopht.122.4.487.View ArticlePubMedGoogle Scholar
- Goldzweig CL, Mittman BS, Carter GM, Donyo T, Brook RH, Lee P, Mangione CM: Variations in cataract extraction rates in Medicare prepaid and fee-for-service settings. JAMA. 1997, 277: 1765-1768. 10.1001/jama.277.22.1765.View ArticlePubMedGoogle Scholar
- Kellie SE, Kelly JT: Medicare Peer Review Organization preprocedure review criteria. An analysis of criteria for three procedures. JAMA. 1991, 265: 1265-1270. 10.1001/jama.265.10.1265.View ArticlePubMedGoogle Scholar
- Norregaard JC, Schein OD, Anderson GF, Alonso J, Dunn E, Black C, Andersen TF, Bernth-Petersen P, Bellan L, Espallargues M, Norregaard JC, Schein OD, Anderson GF, Alonso J, Dunn E, Black C, Andersen TF, Bernth-Petersen P, Bellan L, Espallargues M: International variation in ophthalmologic management of patients with cataracts. Results from the International Cataract Surgery Outcomes Study. Arch Ophthalmol. 1997, 115: 399-403.View ArticlePubMedGoogle Scholar
- Brook RH, Chassin MR, Fink A, Solomon DH, Kosecoff J, Park RE: A method for the detailed assessment of the appropriateness of medical technologies. Int J Technol Assess Health Care. 1986, 2: 53-63.View ArticlePubMedGoogle Scholar
- Kahan JP, Park RE, Leape LL, Bernstein SJ, Hilborne LH, Parker L, Kamberg CJ, Ballard DJ, Brook RH: Variations by specialty in physician ratings of the appropriateness and necessity of indications for procedures. Med Care. 1996, 34: 512-523. 10.1097/00005650-199606000-00002.View ArticlePubMedGoogle Scholar
- Tobacman JK, Lee P, Zimmerman B, Kolder H, Hilborne L, Brook R: Assessment of appropriateness of cataract surgery at ten academic medical centers in 1990. Ophthalmology. 1996, 103: 207-215.View ArticlePubMedGoogle Scholar
- Lundstrom M, Barry P, Leite E, Seward H, Stenevi U: 1998 European Cataract Outcome Study: report from the European Cataract Outcome Study Group. J Cataract Refract Surg. 2001, 27: 1176-1184. 10.1016/S0886-3350(01)00772-6.View ArticlePubMedGoogle Scholar
- S W: Applied Linear Regression. 1985, New York, Wiley, 2nd ed.Google Scholar
- Bagley SC, White H, Golomb BA: Logistic regression in the medical literature: standards for use and reporting, with particular attention to one medical domain. J Clin Epidemiol. 2001, 54: 979-985. 10.1016/S0895-4356(01)00372-9.View ArticlePubMedGoogle Scholar
- W.N. V, B.D. R: 1994, New York, NY, Springer-Verlag, 329-347. Modern applied statistics with S-Plus, Tree-Based Methods.
- Choi YJ, Hong YJ, Kang HG: Appropriateness ratings in cataract surgery. Yonsei Med J. 2004, 45: 396-405.View ArticlePubMedGoogle Scholar
- Leshno M, Reuveni H: Inappropriateness of cataract extraction: an analysis in two Israeli hospital settings. Clin Perform Qual Health Care. 1999, 7: 23-27.PubMedGoogle Scholar
- Kennedy J, Quan H, Feasby TE, Ghali WA: An audit tool for assessing the appropriateness of carotid endarterectomy. BMC Health Serv Res. 2004, 4: 17-10.1186/1472-6963-4-17.View ArticlePubMedPubMed CentralGoogle Scholar
- Park RE, Fink A, Brook RH, Chassin MR, Kahn KL, Merrick NJ, Kosecoff J, Solomon DH: Physician ratings of appropriate indications for six medical and surgical procedures. American Journal of Public Health. 1986, 76: 766-772.View ArticlePubMedPubMed CentralGoogle Scholar
- Leape LL, Park RE, Kahan JP, Brook RH: Group judgments of appropriateness: the effect of panel composition. Qual Assur Health Care. 1992, 4: 151-159.PubMedGoogle Scholar
- Lundstrom M, Roos P, Jensen S, Fregell G: Catquest questionnaire for use in cataract surgery care: description, validity, and reliability. J Cataract Refract Surg. 1997, 23: 1226-1236.View ArticlePubMedGoogle Scholar
- Park RE, Fink A, Brook RH, Chassin MR, Kahn KL, Merrick NJ, Kosecoff J, Solomon DH: Physician ratings of appropriate indications for three procedures: theoretical indications vs indications used in practice. Am J Public Health. 1989, 79: 445-447.View ArticlePubMedPubMed CentralGoogle Scholar
- Hemingway H, Crook AM, Dawson JR, Edelman J, Edmondson S, Feder G, Kopelman P, Leatham E, Magee P, Parsons L, Timmis AD, Wood A: Rating the appropriateness of coronary angiography, coronary angioplasty and coronary artery bypass grafting: the ACRE study. Appropriateness of Coronary Revascularisation study. J Public Health Med. 1999, 21: 421-429. 10.1093/pubmed/21.4.421.View ArticlePubMedGoogle Scholar
- Shiffman RN, Leape LL, Greenes RA: Translation of appropriateness criteria into practice guidelines: application of decision table techniques to the RAND criteria for coronary artery bypass graft. Proceedings - the Annual Symposium on Computer Applications in Medical Care. 1993, 248-252.Google Scholar
- Shekelle PG, Kahan JP, Bernstein SJ, Leape LL, Kamberg CJ, Park RE: The reproducibility of a method to identify the overuse and underuse of medical procedures. [see comments]. N Engl J Med. 1998, 338: 1888-1895. 10.1056/NEJM199806253382607.View ArticlePubMedGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6963/6/23/prepub