Apoorva Stull, National Committee for Quality Assurance
27 July 2010
To the Editor:
The recent study by Adams et al entitled Incorporating statistical uncertainty in the use of physician cost profiles adds significant information to our understanding of reliability related to the use of measures of cost and resource use in physician practices. NCQA agrees overall with their conclusion that setting standards for reliability of physician cost profiles is important. We developed NCQA’s Physician and Hospital Quality (PHQ) certification program, which aligns with standards for physician level measurement set forth in the Patient Charter for Physician Measurement of the Consumer Purchaser Disclosure Project. This charter resulted from the 2008 New York Attorney General’s settlement of court action with health plans over their doctor profiling programs. While the authors cite the 2008 version of our PHQ standards, their description of the degree of statistical testing is actually from a prior, 2006 version and thus misrepresents current standards.
The PHQ standards require organizations to be transparent about how they use statistical testing and sets the actual standards required for testing. Specifically, PHQ standards require that plans using physician cost profiles as the basis for public reporting, cost sharing and network arrangements must: take quality data into account as well as cost data and take action on cost data when comparisons are based on a confidence interval of 90% or measure reliability of at least .70 based on methodology developed by John Adams and others. For cost measurement, there is no option in our current (2008) standards to use only sample size as a substitute for confidence intervals or a formal test of reliability. NCQA does a formal review of the methodology used by plans in meeting these requirements.
The assertion of Adams et al, that few plans use formal statistical testing in their physician assessment programs is not consistent with the fact that well over a hundred plans have been certified by NCQA as meeting the PHQ standards including the use of confidence intervals or formal reliability testing, which provides significant protection against the misclassification potential that RAND has identified. A complete list of certified organization is available at www.ncqa.org. Thus while the article adds important information on reliability of measures of cost and resource use, the approach of many health plans has changed substantially since the citations used by the authors, at least in part due to the introduction of the NCQA PHQ standards.
Rand Study Response from NCQA
27 July 2010
To the Editor:
The recent study by Adams et al entitled Incorporating statistical uncertainty in the use of physician cost profiles adds significant information to our understanding of reliability related to the use of measures of cost and resource use in physician practices. NCQA agrees overall with their conclusion that setting standards for reliability of physician cost profiles is important. We developed NCQA’s Physician and Hospital Quality (PHQ) certification program, which aligns with standards for physician level measurement set forth in the Patient Charter for Physician Measurement of the Consumer Purchaser Disclosure Project. This charter resulted from the 2008 New York Attorney General’s settlement of court action with health plans over their doctor profiling programs. While the authors cite the 2008 version of our PHQ standards, their description of the degree of statistical testing is actually from a prior, 2006 version and thus misrepresents current standards.
The PHQ standards require organizations to be transparent about how they use statistical testing and sets the actual standards required for testing. Specifically, PHQ standards require that plans using physician cost profiles as the basis for public reporting, cost sharing and network arrangements must: take quality data into account as well as cost data and take action on cost data when comparisons are based on a confidence interval of 90% or measure reliability of at least .70 based on methodology developed by John Adams and others. For cost measurement, there is no option in our current (2008) standards to use only sample size as a substitute for confidence intervals or a formal test of reliability. NCQA does a formal review of the methodology used by plans in meeting these requirements.
The assertion of Adams et al, that few plans use formal statistical testing in their physician assessment programs is not consistent with the fact that well over a hundred plans have been certified by NCQA as meeting the PHQ standards including the use of confidence intervals or formal reliability testing, which provides significant protection against the misclassification potential that RAND has identified. A complete list of certified organization is available at www.ncqa.org. Thus while the article adds important information on reliability of measures of cost and resource use, the approach of many health plans has changed substantially since the citations used by the authors, at least in part due to the introduction of the NCQA PHQ standards.
Margaret E. O’Kane
President
Competing interests
None declared