Data Collection

We obtained information on clinical indicators and incentive payments for practices from the GMS contract documentation. We obtained data on disease prevalence and on exception reporting for English practices from the NHS Information Centre [11]. We obtained data on estimated health gain in terms of lives saved and QALYs using published estimates of the likely maximum expected health gains for 28 clinical indicators which are incentivised in the GMS contract [14, 18, 21]. QALY estimates were based on lifetime QALY gains, on the assumption that appropriate treatment will continue beyond the current year. There are three ways in which to express health gain in a practice population. The 'maximum expected health gain' is the expected number of lives saved if all patients with the condition received indicated care. The 'maximum achievable health gain' is less, as it excludes those patients considered by their general practitioner to be not suitable for the intervention by exception reporting. For example, 13% of patients were excluded by exception reporting from receiving influenza immunization in 2005. The 'incentivised health gain' is the health gain at the performance threshold for maximum payment. This is even less as it is the expected number of lives saved when the target threshold for full payment is reached (for example full incentive payment is received in influenza immunization when 85% of eligible patients have received treatment).

Analysis

We estimated health gain for an average sized practice of 6411 patients. The data for lives saved were expressed for an English population of 100,000 so we multiplied this by 0.06411 to reach the expected lives saved for a practice of size 6411. Data for QALYs gained were expressed for each patient treated, so we multiplied this figure by the number of patients that a practice of 6411 would have with that particular condition, using prevalence data at a national level from English practices in QOF [21].

For our primary analysis, we examined the relationship between the marginal increase in payment and health gain resulting from a one percent point improvement in performance for an average size general practice. For our secondary analysis we examined the relationship between the total payment and health gained at the performance target for maximum payment. Histograms showed that the data were not normally distributed; therefore a non-parametric Spearman rank correlation test was used in preference to a Pearson correlation test. A two-sided test was used to test the null hypothesis of no relationship between incentive payments and measures of health gain versus the alternative hypotheses of a positive or negative relationship. We tested this potential relationship for both the 2004 and 2006 versions of the GMS contract, and also for both health gain in terms of lives saved in one year and QALYs gained.

We conducted two sensitivity analyses. First we conducted a sensitivity analysis using the maximum number of achievable lives saved as practice performance often exceeded the upper threshold target for maximum payment. Second we included clinical indicators with evidence for health gain that were derived from randomised controlled trials only, excluding indicators with a lower strength of evidence. All tests were carried out in SPSS version 18.

Primary analysis

Secondary and sensitivity analyses

In our secondary analysis no associations were found between the size of financial payment for achievement of an indicator and the expected health gain at the performance threshold for maximum payment measured in lives saved or quality adjusted life years (Table 3). Our sensitivity analyses, firstly using 'maximum achievable health gain' rather than 'incentivised health gain' and secondly using evidence for health gain only from randomised controlled trials, did not substantially alter these findings (Tables 3 & 4). Scatter plots presented in Figures 1 and 2 show one outlier, treatment of hypertension (label 3, indicator BP5). Reanalysis of the data after excluding this outlier also shows no significant associations. In summary no statistically significant associations were found between any measure of health gain and incentive gained in either the 2004 or 2006 version of the Quality and Outcomes Framework. We therefore cannot reject the null hypothesis of no relationship between incentive pay and health gain for all areas in both the 2004 and 2006 GMS contract.

Measure of health gain	2004 QOF payment (rcts only)	2006 QOF payment (rcts only)	2004 QOF payment (without BP5)	2006 QOF payment (without BP5)
1% improvement in achievable lives saved and incentive gained	Rho = 0.058 p = 0.111 n = 11	Rho = 0.058 p = 0.111 n = 11	Rho = 0.137 p = 0.553 n = 21	Rho = -0.147 p = 0.561 n = 18
1% increase in achievable QALYs and incentive gained	Rho = 0.450 p = 0.310 n = 7	Rho = 0.611 p = 0.108 n = 8	Rho = 0.180 p = 0.670 n = 8	Rho = 0.178 p = 0.601 n = 11
lives saved to target and maximum incentive payment	Rho = 0.381 p = 0.247 n = 11	Rho = 0.263 p = 0.434 n = 11	Rho = 0.095 p = 0.683 n = 21	Rho = -0.326 p = 0.186 n = 18
QALYs gained to target and maximum incentive payment	Rho = 0.148 p = 0.751 n = 7	Rho = 0.366 p = 0.373 n = 8	Rho = -0.098 p = 0.818 n = 8	Rho = -0.064 p = 0.851 n = 11
Maximum achievable lives saved and maximum incentive payment	Rho = 0.428 p = 0.189 n = 11	Rho = 0.412 p = 0.208 n = 11	Rho = 0.130 p = 0.572 n = 21	Rho = -0.223 p = 0.373 n = 18
Maximum achievable QALYs gained and incentive payment	Rho = 0.150 p = 0.749 n = 7	Rho = 0.368 p = 0.370 n = 8	Rho = -0.098 p = 0.817 n = 8	Rho = -0.065 p = 0.851 n = 11

Strengths and limitations

Strengths of this study include that robust and up-to-date measures of health gain were used which were computed specifically for the clinical indicators in the general practice contract, and included two different measures of health gain. The estimates of health gain are quite robust to measurement error, since all that is required for Spearman correlation tests in is that the estimates reflect the rankings of interventions by size of health gain. Sensitivity analyses excluding trials with lower levels of evidence and using maximum achievable health gain instead of incentivised health gain, show similar results.

Limitations include that we were only able to identify measures of health gain for a subset of 28 out of a total of 98 indicators in the 2004 and 2006 QOF including an additional area of cervical screening. However, these 28 are important indicators with measurable health outcomes and are all considered clinically important interventions, which account for an achievable 1,085 QALYs gained and 22 lives saved in one year for an average sized practice, and yield 41% of the maximum possible payment for clinical interventions. Of the clinical indicators that were not included in this study, a further 27 were processes which were related to achievement of these 28 indicators (additional file 1).

Not all QOF indicators may be mutually independent. For example a practice which has a successful influenza management system may be more likely to target all eligible patients for immunisation irrespective of which chronic disease they have. Furthermore if a patient has co morbidities such as diabetes and heart disease which include the same intervention, for example influenza immunisation (CHD 12 and DM 18) or hypertension control (CHD 6 and DM12), then there will be an inter dependency between these indicators in different disease domains. QOF interventions are selected by the NHS in agreement with the BMA and in consultation with stakeholder groups. The actions of these agencies at the primary care level may influence the priority that primary care gives to certain medical conditions in a number of ways in addition to inclusion in the QOF.

This research does not capture baseline performance, so our analysis is limited to examining pay for the level of performance, rather than for performance improvement. Our estimates of marginal incentive for a one percent point improvement in health gain are not affected by this problem, so long as baseline performance is below the target for maximum payment. We only considered the health benefits of the interventions in relation to the size of the QOF payments made. The costs of the interventions themselves and their effect on other health service costs have not been considered. Such costs would have to be considered if we were to evaluate the cost-effectiveness of the QOF. For example in another study it was found that two of the indicators considered actually reduced overall costs to the NHS (CHD 10- aspirin in heart disease and DM 15- ACE inhibitor drugs in diabetic renal disease) [21]. The net health impact of cost saving interventions could be higher than the average health impact we have estimated, since the resources saved can be used elsewhere for treating patients and delivering health gains. There will be a small number of individual practices with atypical populations where a closer national alignment of QOF incentives and expected health gain does not fit the health need for those particular practices. We have made the assumption that health gains are distributed evenly across all percentage increases in performance, which may not be the case.

Comparison with previous research

Several studies have examined the expected health gain, effectiveness and cost effectiveness of QOF, and one examined the relationship between financial incentives and health gain [12, 14, 18, 21]. The findings of this small study were similar to our study, finding no association between the size of the financial incentive and expected health gain [13].

Implications

The main implication for policy makers is that the lack of an association between the size of the incentive and the expected health gain may risk skewing activity towards areas with high workload but relatively low benefit to health [3]. Other areas which receive little or no incentive may be relatively ignored [17, 18]. Indications for further research include a systematic review of interventions across the spectrum of primary care to identify the evidence base in terms of QALYs to inform both the selection of new indicators and the relative size of financial incentives.