Assessing methods for measurement of clinical outcomes and quality of care in primary care practices

Primary care, the first point of contact between patients and the health care system, includes disease prevention, health promotion, and chronic disease management. For over a decade, improving primary care has been a key element of health system reform around the world to improve health and health outcomes and reduce the cost of health care [1–9] A program of research to evaluate the impacts of such reforms is essential [10, 11]. The Canadian Institute for Health Information (CIHI) has proposed a comprehensive set of primary care outcome indicators [12], and also noted significant gaps in the availability and quality of the data sources to populate these indicators [13]. There is a need for a set of validated field tested measurement tools to facilitate the population of these indicators [14]. These tools could also be used for quality improvement initiatives in primary care practices and to enhance accurate reporting of performance for pay for performance incentives [15]. Administrative data, electronic health records, chart audits, patient surveys, provider surveys, population level surveys and direct observation have all been used for performance measurement in primary care, and could be used in the evaluation of primary care reform initiatives [14, 16–28].

Survey and audit data were entered into SPSS version 16. Statistical analysis of comparisons between survey and chart audit data was also conducted in SPSS. Any analysis including comparisons to administrative data was conducted in SAS version 9.2. The results obtained from each data source were compared directly using both a comparison of proportion of concordant pairs and the kappa statistic. The clinical significance of the presence of discordant items, the likely reasons underlying discordant data, and the magnitude of the difference in result between methods were also considered [30, 31]. To facilitate analysis we made some minor modifications to some of the definitions established by CIHI (for example looking at the individual components of composite measures, modifying time frames to match standard administrative data analysis algorithms). We did not assume that the chart was the “gold standard” for each item, using clinical experience and knowledge of the limitations of each data source to instead consider why the data sources did not always agree.

Ethics review and approval was obtained from the Research Ethics Boards at Queen’s University, The University of Ottawa, The Ottawa Hospital, and Sunnybrook Hospital (ICES). All study procedures underwent privacy review and approval at ICES.

Table 4 outlines the agreement levels observed in health promotion activities between the patient survey replies and the information found in their chart only as this information is not available through billing records. The only one that had concordance over 75% was current smoking status. Past smoking status showed less than 50% agreement and provision of advice on diet and exercise showed 70-75% concordance with kappa levels all <.2. The levels of agreement between the administrative and chart data as well as between the patient survey and the chart are shown for the presence of the index conditions of interest and use of two broad categories of medications are presented in Table 5. Table 6 supplements this with a comparison between the survey data and administrative data for medication use on patients over 65. There was >85% agreement on the presence of the diagnosis. For medication use, concordance between the chart data and patient surveys was >75%, while the concordance between administrative and chart data was >75% for antihypertensives and only 57% for anti-lipidemics, with much higher rates of lipid lowering agent use noted in the charts. In Table 6 the level of agreement between administrative data and the patient between 70-75%, which is slightly lower than level of agreement between the chart and the patient. The kappa statistic was higher for antihypertensives, than for anti-lipidemic drugs.

Table 7 presents the comparison for a number of disease specific recommendations for the chronic diseases we examined, including a more detailed review of medication usage. Documentation of advice or resources provided has a level of agreement of <70%, with fewer than half of the events reported by patients being noted in the chart and a kappa <.4. For relatively less common but important key events such as MI and hospitalization, while agreement was >90% overall, there was poor agreement between the chart and patients who report having had these events, with less than half being identified in the charts. For the remaining process of care measures there was a mixed result, with levels of agreement being >75% for FBS, Lipid profiles, control of hyperlipidemia, AIC and foot exams and <75% for the remainder. Kappa values were <.4 for all process of care outcomes other than medications. The more detailed medication profiles showed levels of agreement >85% with kappas >.6 with the exception of ASA and Statins. ASA is an over the counter drug and despite a 78% agreement rate and kappa >.4, was only recorded in the chart in about 2/3 of patients who reported using it. In contrast, statins had only 54% agreement and a kappa <.2 due to patients reporting not taking them despite the drug being noted as active in their chart.

There are relatively few studies in primary care performance measurement which have examined the differences in results obtained through different methods of measurement, especially involving administrative data [17, 20, 32, 33]. This study examined the validity of data on primary care performance indicators obtained by various methods, as well as the acceptability, feasibility and potential biases of using a practice-based recruitment approach for the collection of linked data from a range of different methods. Our ability to collect data on multiple measures by audit, survey, and use of administrative data with good participation rates, high agreement rates between chart assessors, high rates of completion for surveys, and little objection to data linkage, shows that the collection of linked data from multiple sources is both acceptable and feasible. This study used sequential recruitment of patients presenting for care in participating practices [22]. Participants had 50-100% higher rates of chronic disease and multi-morbidity than the practice population, children were under-represented, and women and the elderly were overrepresented. Studies seeking a representative sample of all practice patients may wish to use other methods of recruitment,.Studies on the care experiences of chronic disease patients, workload or work process issues or the daily experiences of practitioners may find this to be both appropriate and efficient. As our study was focused on the concordance between data collection methods this design was unlikely to impact our results.

Significant differences in performance were found using the different data collection methods for many indicators. No single data collection method emerged as consistently the most valid across all performance indicators. A only a limited number of indicators had kappa statistics >.4 (moderate or better agreement) [30] however in come cases these were much worse than the degree of concordance estimated by proportion of concordant pairs. When interpreting our data both kappa and the degree of concordance should be considered [31]. With the increasing use of administrative data for primary care performance reporting [19, 34, 35], remuneration and funding decisions [28], disease registries [27] and public health reporting [36] the comparison of administrative data results to multiple different data sources is especially important for guiding the use and interpretation of administrative data results in future research, policy, and planning [27, 37, 38].

Good agreement across measurement methods was seen for preventive health imaging such as mammograms and bone mineral density scans. Previous research has found similar mammogram screening rates using patient surveys and chart audits [17, 32]. This study adds the use of administrative data to this comparison and finds it to be a reasonable alternative. Notable areas of discordance between administrative data and other methods of data collection included Pap smears, colon cancer screening and influenza vaccination, all of which had much lower rates noted in the administrative data. Manoeuvres like Pap tests and colon cancer screening may be completed in contexts other than the delivery of primary care. Where services that are not included in routine administrative data sources, and recourse to them is more widespread administrative data may be less accurate in capturing care received by the patient than other methods. Thus, the local context of care may significantly alter the validity of administrative data results for primary care performance.

Important differences between patient survey reports and chart audits were also found. Pap smears and influenza vaccination were reported at higher rates in the surveys. Patient over-reporting of Pap test rates has been previously reported [32]. Influenza vaccination in Ontario often occurs in public flu clinics and is therefore not always noted in the chart, so higher rates would be expected for this outcome. Patients who report having received diet and exercise advice had a notation to this effect in their chart over 90% of the time. The more significant difference was in the opposite direction, with 25-30% of patients whose charts indicated this was discussed failing to report this in the survey. This finding contradicts previous research comparing chart audit and patient survey to actual observation of patient visits, which found patients reported receiving advice more frequently than was noted in the chart [17]. This could be due in part to the multidisciplinary nature of these family health teams, where allied health providers would be contributing to the delivery of these services and documentation of this in the patient record. However, this finding may also reflect the difficulties in communicating messages on health promotion in ways that are memorable or retained by patients.

For most medications there was good agreement between chart and survey responses, and moderate agreement between survey and administrative data. There were two exceptions. For statins agreement between the chart and survey was poor, while there was much better agreement with administrative data. These discrepancies may represent issues of medication adherence, or poor understanding or recognition of a medication. Aspirin, which is available over the counter (OTC), was reported more frequently by patients than noted in the charts, likely representing lack of documentation in the chart. These findings suggest that patient survey may be the more accurate source for medications used, however this source is not without its potential biases. In terms of clinical outcomes, patients were more likely to report that their blood pressure and lipids were at target than was reported in the chart. These findings highlight the importance of importance of clear communication between patients, their physicians and other providers on issues such as medication use and management targets.

Recent efforts to assess the quality of chronic disease management (CDM) has relied on administrative data, EHR audits and population based surveys to identify the patient population with the condition of interest [36, 38]. For identifying the population with diabetes, there was strong agreement across measurement methods. For hypertension, the level of agreement was still good, but not as much as for diabetes. For estimation of prevalence in the sample the margin of difference is fairly small (39% vs 36%), but there are large numbers of discordant pairs. In these situations, administrative data identify more cases than in the charts, perhaps a reflection of care that occurs in specialist or hospital settings and not captured in the primary care chart. For the purposes of registry generation, either method would be appropriate, with the cost of administrative data being much lower. Performance measurement in primary care increasingly forms the basis of quality improvement investments, performance bonuses, population health planning and reporting [15]. Others have expressed concerns about the unintended consequences of pay for performance systems and on the impact of performance measurement more generally on good clinical practice. [38, 39] Our results indicate that careful consideration needs to be given to the methods used for assessing performance if these concerns are to be minimized. Future performance reporting should account for potential bias in results based on the data collection method and indicators measured.

For many measures of technical quality of care, chart audit remains the most accurate method of data collection. Patient surveys are required for more accurate assessment of indicators such as immunizations, chronic disease advice/information dispensed, medication use, and some general health promotion items. Consecutive sampling of patients in the waiting room samples a population that is sicker, older and more likely to be female compared to the practice population. Administrative data appears useful for a number of indicators including several aspects of screening and chronic disease diagnosis. Administrative data are much less costly than other methods of data collection and can cover entire populations. Recruitment rates of physicians and patients remained high while requesting permission to link the data collected at the practice to the provincial health administrative databases. A comprehensive understanding of primary care performance will require the use of multiple data collection methods for the foreseeable future. The choice of which methods are best for any one particular study or quality improvement initiative requires careful consideration of the biases that each method might introduce into the results. Future studies should also consider assessing the reasons underlying divergence between decisions made at the individual patient level and recommended guidelines.

Portions of this manuscript content have been presented at the Canadian Institutes for Health Research Primary Care Summit, Toronto, Jan 18, 2010, the Canadian Association for Health Services and Policy Research Annual Meeting, Calgary May 12, 2009 and the North American Primary Care Research Group, San Juan Puerto Rico, Nov 16, 2008.

This project was funded by a grant from the Ontario Ministry of Health and Long Term Care. ICES and the Centre for Health Services and Policy Research are also supported in part by the Ontario Ministry of Health and Long Term Care. The opinions, results and conclusions reported in this paper are those of the authors and are independent from the funding sources. No endorsement by ICES or the Ontario MOHLTC is intended or should be inferred.