Explicit criteria development
We based our development of priority criteria on a modification of the RAND appropriateness method. We first developed criteria to measure the appropriateness of the use of cataract surgery, according to the following steps.
First, an extensive literature search was performed to summarize existing knowledge concerning efficacy, effectiveness, risks, costs, and opinions about the use of phacoemulsification for cataract extraction.
Second, from this review, a comprehensive and detailed list of mutually exclusive and clinically specific scenarios (indications) was developed in which cataract surgery might be performed using phacoemulsification. Regarding cataract surgery, these indications included the following variables: presence of ocular comorbidities (simple cataract, cataract with diabetic retinopathy, or cataract with other ocular pathologies that may affect the visual prognosis), visual acuity in the cataractous eye and the contralateral eye, visual function, expected visual acuity after surgery, surgical technical complexity, and type of cataract. A total of 765 indications resulted from all possible combinations of the variables described previously and the respective categories. A description of all variables and their categories was reported previously [10].
Third, we compiled a national panel of expert ophthalmologists who were nationally recognized specialists in the field. Their names were provided by their respective medical societies and members of our research team.
The appropriateness ratings were confidential and took place in two rounds, using a modified Delphi process. Cataract surgery for a specific indication was considered appropriate if the panel's median score was between 7 and 9 without disagreement, inappropriate if the value was between 1 and 3 without disagreement, or uncertain if the median rating was between 4 and 6 or if the members of the panel disagreed. Disagreement was defined as occurring when at least one third of the panelists rated an indication from 1 to 3 and at least another third rated it from 7 to 9.
In a third round, we selected the scenarios judged in the second round as appropriate or uncertain. We selected the following previous variables for this priority round: appropriateness, presence of ocular comorbidities, visual acuity in the cataractous eye, visual function, visual acuity in the contralateral eye, expected visual acuity after the intervention, the type of cataract (laterality), and the new variable social dependence. Selection of the variables was based on the review of the bibliography and the research team best judgment. Figure 1 shows the variables included in this priority round and Appendix 1 includes a definition of all variables and their categories.
We requested that the same panelists score the 310 scenarios. Ratings also were scored on a 9-point scale, with 9 indicating the highest priority and 1 the lowest compared to other scenarios. Priority in the context of cataract extraction was defined as the benefit that the patient may obtain from undergoing surgery. The higher the benefit was for the patient, for a similar risk of complications, the higher the priority of the intervention was. Benefit was defined in terms of quality-of-life improvement. Three categories, from higher to lower priority, were established. The priority of cataract surgery was considered high for a specific indication if the panel's median rating was between 7 and 9 without disagreement, low if the value was between 1 and 3 without disagreement, or intermediate if the median rating was between 4 and 6 or if the panel members disagreed. Disagreement was defined as occurring when at least one third of the panelists rated an indication from 1 to 3 and at least another third rated it from 7 to 9. This method did not attempt to force panelists to reach agreement on the priority.
To determine the use of all theoretical indications created in clinical practice, data related to the algorithm variables were gathered for 936 patients on a waiting list to undergo cataract extraction exclusively by phacoemulsification from the ophthalmologic services of six hospitals of our area. The number of theoretical indications used in clinical practice was calculated for each of the three main diagnostic groups; simple cataract, cataract with diabetic retinopathy, or cataract with other ocular pathologies that may affect the visual prognosis.
Statistical analysis
We estimated the mean priority rating of all indications for each panellist and the median and the deviation from the panel mean. We created a continuous priority score, which is the sum of the ratings of the 11 panelists, standardised to a 0 to 100-point scale.
Determinants of priority scores and their contribution to the model explanation were assessed with the general linear model [11], with the priority score the dependent variable and the variables in the algorithm the covariates. Ordinal logistic regression also was used, which was the classification of the panelists' scores in the categories of high, intermediate, or low priority of the dependent variable (12). In each model, we studied the degree of variability explained by each variable by means of the R-square and -2 log L statistics, respectively. Because the appropriateness variable is a combination of some of the other variables, we forced it to the enter the last in both models.
We used two methods to determine priority coefficients for the categories of each variable that should permit rapid estimation of a priority score in practice. In the first model, the optimal scaling method [13, 14] considered the priority weighted score as the dependent variable and the variables in the algorithm as the independent variables. Optimal scaling was used to quantify the categorical data by assigning numerical values to the categories in such a way that the fit is optimal. Standard linear regression analysis then was conducted using the numerical explanatory variables obtained with the optimal scaling method. The weights for each variable were calculated based on the t-values from linear regression, with weights distributed across variables as the corresponding proportion of the total model t-value. To determine the weights for the categories within each variable, the numerical values for the categories obtained with the optimal scaling method were rescaled linearly to between 0 and 1; these values then were multiplied by the weight for the variable to which the levels belong.
In the second model, we performed the same estimation using a general linear model in which we created dummy variables from the categories of the independent variables, with the priority score the dependent variable. Weights for categories within each variable were based on beta values from the general linear model, with weights transformed to a 0 to 100 point scale distributed across variables as the corresponding proportion.
Therefore, in both cases weights are apportioned among variables so that the scores range from 0 to 100. The scores obtained with both methods for all scenarios were compared with the original panel priority scores using Pearson and Spearman correlation coefficients.
Algorithms in decision-tree form were compiled by means of classification and regression trees (CART) analysis [15]. CART was used to build a regression tree with a dependent variable as the priority score.
All statistical analyses were performed using the SAS for Windows, version 8, except for CART analysis in which we used S-Plus 2000 software (MathSoft Inc., 1999), and the optimal scaling analysis in which we used SPSS v.12 statistical software.