In this paper, a method for identification of external factors relevant for risk adjustment using quantitative test criteria is proposed and demonstrated by using data from a multi-centre register database for outcome quality in cataract surgery. Risk adjustment serves to establish comparability of different medical facilities and is an important tool for quality assurance. While methodological risk adjustment procedures have already been adopted and are well established , little is known about how to identify the factors relevant for risk adjustment. Clinical relevance is discussed in some publications as a relevant characteristic in addition to statistical significance [5, 6]. However, published literature does not include suggestions for the operationalisation of these criteria for specific indications and databases. In the present study, test criteria are qualitatively defined and parameterised using statistical methods for association analysis and by context-specific cut-off proposals.
Comprehensive analysis for identification of external factors relevant for risk adjustment should also consider the test criterion “uneven distribution among care providers”. We did not apply this criterion to the MONIKA database since the study centres are structurally similar (outpatient specialist providers) and, therefore, do not provide a true representation of the distortion in distribution of external factors throughout different levels of healthcare.
An external factor has clinical relevance if patients are expected to experience disadvantages if it is present. As a measure of clinical relevance, the (absolute) deviation of the success rates with or without presence of the external factor was utilised in this analysis. Clinical relevance can be considered to be greater, the higher the deviation in success rates in the presence or absence of the external factor. In this investigation, taking into account the nature of the actual cataract surgery outcome indicator, a threshold limit value of at least 10 percent points was proposed to classify clinically relevant association between external factors and outcome quality indicators.
However, these cut-off points defining clinical relevance are based on normative settings and inherent reliability is not specified. Therefore, cut-off limits must be determined according to the actual indicators and their context in quality assurance procedures. For example, if an endpoint such as mortality is quantified by means of an indicator, a notably lower threshold limit value for the deviation in prevalences may be appropriate. An example for cataract surgery is the occurrence of postoperative endophthalmitis as (negative) indicator of the outcome quality, with a reported incidence between 0.5% and 0.015% . Due to the high clinical relevance (high risk of blindness), a minor prevalence difference of predisposing external factors would be sufficient to declare clinical relevance for risk adjustment. In terms of this, utilisation of the above clinical relevance criterion based on the “absolute” prevalence difference must be critically discussed.
For positive endpoints like those choosen here determination of the cut-off value is a weighting and not justifiable by objective arguments: A cut-off point for clinical relevance of 5% rather than 10% would have resulted in two additional confirmed external factors (“severe farsightedness” and “presence of at least one surgically relevant ocular risk factor”). Lowering threshold means to consider more external factors and improve therefore the surrounding for a suitable quality comparison. The complexity of documentation, analysis, error rate and susceptibility to manipulation increase on the other hand.
A relative clinical relevance criterion would be another possible parameterization especially in studies with rare outcome variable. A relative risk scale might be considered in a study looking at relevant external factors responsible for rare events, e.g. endophthalmitis. However, in an interdisciplinary clinical setting an absolute excess risk scale is more acceptable. When being presented with an absolute difference most clinicians find such information easier to understand and more straight-forward to interpret as compared to a relative difference.
In the present study, statistical significance was deliberately parameterised by means of two established methodological approaches: first, univariate stratification association and second, multiple (logistic) regression analysis. Although both approaches allow for centre-corrected estimation of the association between each external factor and each outcome indicator, the univariate analysis of the association of external factors and the measured outcome quality did not allow simultaneous comparison of the influence of the surgical facility.
On the other hand, whereas logistic regression modelling enabled several external factors to be simultaneously related to one outcome indicator, model goodness of fit for this approach was crucial: although the univariate and the multiple approach identified the same external factors as those significantly associated with the refractive and visual outcome indicators, the extremely low model fit ≤ 12% for both models did not allow these modelling results to confirm the univariate approach in an analysis method perspective. The model fit might be acceptable with a (negative) endpoint occurring only rarely – e.g. complications after surgery – but not for binary, well confined positive endpoints. The type of variable – dichotomized instead of continuous - might have had negative impact on the model fit. Based on the results of this cataract example, we propose applying the univariate stratification approach in the first instance, despite its rather exploratory nature - and to consider the results from the regression modelling approach as additional information providing confirmation if there is sufficient model fit. It is the task of future studies and discussion to estimate to what extent (as measured by Nagelkerke’s R2) logistic regression modelling can be expected to explain the variation in the results of cataract surgery or other indications.
Both approaches must be discussed further with respect to multiple testing. Whereas both approaches accounted for centre heterogeneity by either stratification or adjustment for centres, both methods estimated eight external factor associations with each outcome quality indicator. As a consequence, a total of 2 × 8 associations were tested for significance, formally requiring multiplicity adjustment. However, regarding the exploratory nature of identifying external factors requiring risk adjustment in future analysis/reports, the above results should be considered in terms of locally significant association findings instead of formally correcting their confidence interval levels for multiplicity. Nevertheless, researchers should be aware of this “local” significance interpretation underlying the above significance criterion — and should reduce the number of external factors to be considered for risk adjustment.
Factors relevant for risk adjustment
The selection of the external factors considered in this study is based on information in published literature on external factors demonstrating influence on success rates [9, 13–21]. However, regardless of this discussion, risk adjustment in established quality assurance procedures for cataract surgery was carried out only for one indicator (visual rehabilitation) and for one factor (pre-existing conditions reducing visual acuity) [18, 22–24]. The results of the analyses described here suggest that this limited risk adjustment does not sufficiently ensure a valid basis of comparison for the selective medical contribution to outcome quality in the context of quality assurance procedures.
The external factors considered in this study are categorized as sociodemographic and quantitatively measurable and qualitative patient-related characteristics documented on the basis of clinical findings. The objective of this categorisation was discovery of potential structural differences that can be attributed to the factor properties. As expected, the differences in prevalence between centres are smallest, the less complex and the more standardised the survey of a factor is. Identification of external factors based on qualitative clinical findings of varying complexity and standardisation therefore constitutes an intrinsic source of bias for appropriate risk adjustment for quality assurance procedures.
Methodological considerations of the data quality of the MONIKA database
Adequate data quality is a prerequisite for transferability of the results from the MONIKA database to other quality assurance procedures. The time-shifted participation of the study centres could be regarded as a potential source of bias (Table 1). During the entire survey period there was no significant change in diagnostic or therapeutic methods employed, so that a systematic effect of the divergent entry period on the data quality seems unlikely. The incompleteness of the data could also be considered as a source for bias. However, reduced completeness has also been reported in other voluntary studies. The rates in the three centres of the MONIKA database are comparable with those of other publications with an equivalent study environment . Lower response rates resulting from reduced commitment do not allow a conclusion on systematic bias .
Internal and external consistency of the documentation
The internal consistency of the frequencies of quantitative to qualitative factors points to good data quality for the three study centres (Table 2). Generally, in these centres, a large proportion of elderly patients is associated with a large number of pre-existing conditions reducing visual acuity (and vice versa). In addition, the relationship of low baseline visual acuity and pre-existing conditions reducing visual acuity was described in all centre cohorts. Lastly, an association between refraction anomalies (“high myopia” or “high hyperopia”) and at least one previous ocular surgery or surgically relevant ocular risk factors is present in all centres.
Another approach for evaluation of the data quality is the comparison of the reported prevalence of external factors according to the MONIKA database (Table 3) with the literature. This analysis also confirms good documentation quality in the MONIKA database. The average age at surgery in the MONIKA database (73 years) is in the middle of the range of reported averages (67 years  to 76 years ). This also applies to the reported proportion of women (MONIKA: 59.4%; range of other publications 53%  to 66% ). The total prevalences documented in the MONIKA database for severe nearsightedness and severe farsightedness lie only slightly below those of a publication by a German university hospital .
Due to inconsistent definitions in the published literature, the comparison of prevalence of qualitative documented factors is more difficult. In the MONIKA database, the two factors “pre-existing conditions reducing final visual acuity” and “history of previous ocular surgery” are documented without quantification of the impact on attainable final visual acuity. In contrast, Murphy et al. use “ocular comorbidity that was expected to reduce postoperative acuity to 6/12 or worse”  as a definition and Jaycock et al. the limit “ocular copathology, identified as a reason for a guarded visual prognosis in the operated eye” . As expected with this definition heterogeneity, the reported frequencies deviate strongly and range from 21.2%  to 41% , as compared to the cumulative reported prevalence for both factors reported in the MONIKA database of 36.1%.
In summary the data from the study centres show adequate consistency. The data quality can be considered sufficient for evaluation of the test criteria to identify relevant external factors for risk adjustment of outcome indicators in cataract surgery.