Correlates of health and healthcare performance: applying the Canadian health indicators framework at the provincial-territorial level

Abstract

Background

Since, at the health system level, there is little research into the possible interrelationships among the various indicators of health, healthcare performance, non-medical determinants of health, and community and health system characteristics, we conducted this study to explore such interrelationships using the Canadian Health Indicators Framework.

Methods

We conducted univariate correlational analyses with health and healthcare performance as outcomes using recent Canadian data and the ten Canadian provinces and three territories as units of the analyses. For health, 6 indicators were included. Sixteen healthcare performance indicators, 12 non-medical determinants of health and 16 indicators of community and health system characteristics were also included as independent variables for the analysis. A set of decision rules was applied to guide the choice of what was considered actual and preferred performance associations.

Results

Health (28%) correlates more frequently with non-medical determinants than healthcare does (12%), in the preferred direction. Better health is only correlated with better healthcare performance in 13% of the cases in the preferred direction. Better health (24%) is also more frequently correlated with community and health system characteristics than healthcare is (13%), in the preferred direction.

Conclusion

Canadian health performance is a function of multiple factors, the most frequent of which may be the non-medical determinants of health and the community characteristics as against healthcare performance. The contribution of healthcare to health may be limited only to relatively small groups which stand to benefit from effective healthcare, but its overall effect may be diluted in summary measures of population health. Interpreting multidimensional, multi-indicator performance data in their proper context may be more complex than hitherto believed.

Background

In June 2000, in its ambitious comparison of 191 countries in terms of their ability to meet three main goals – improving health, increasing responsiveness to meet the legitimate demands of the population, and ensuring that financial burdens are distributed fairly – the World Health Organization (WHO) ranked Canada 30^th in overall health system performance [1]. This was considered a further blow to an already shaken collective psyche in Canada [2]. Canada, which WHO also ranked 35^th on health level performance, has taken these rankings to be indicative of serious performance problems, despite the methodological criticisms leveled against the rankings [3]. In its letter to the then minister of health, the Canadian Medical Association called the report "a serious wake-up call" [3]. By September 2000, Canada's Prime Minister and the First Ministers had made a commitment to produce regular public reports on the performance of their health system [4].

As such, the Canadian government has invested heavily in measuring and reporting on the performance of its health system at various levels [4–6]. In doing so and in line with its longstanding 'health determinants' approach to national health policy [7–9], Canada takes a broad health performance approach to quantifying health and healthcare progress [10]. This has entailed the development and use of a multi-dimensional 'health indicators framework' [10] (see Figure 1). This Canadian Health Indicators Framework (CHIF) has four main tiers, namely (a) health status (4 fields); (b) non-medical determinants of health (4 fields); (c) health system performance, or more appropriately referred to as healthcare performance in this paper, (8 dimensions or fields); and (d) community and health system characteristics (3 fields). Many of the fields or dimensions within the framework have so far been populated with indicators. The choice of a health performance framework built on the Lalonde health determinants model should come as no surprise since this was the country that introduced the Lalonde model to the world about three decades ago [9].

It is expected that the multi-dimensional framework will aid the gauging of health progress in a fair and balanced manner. Particularly, it is often assumed that the various fields are interlinked in complex ways that contribute to health and healthcare performance [11]. Healthcare functioning is also taken to be an important contributor to health, notably for those specific populations that stand to benefit from healthcare services. Yet, there has been no research into whether such links exist within the CHIF. This paper aims to examine such links. Using the CHIF as a linked model, this study analyzes the possible relations between, on the one hand, indicators of health (and healthcare performance) and, on the other hand, indicators of non-medical determinants of health, community and health system characteristics, and healthcare performance. Thus, this study poses the question whether the performance of the Canadian provinces/territories on a health (or healthcare) indicator is related to their performance on an indicator of non-medical determinants, healthcare or community characteristic. This study provides an illustrative interlinking of multi-dimensional performance at the provincial/territorial level, but not necessarily the causal interrelationships between and among indicators. We define 'health performance' as the ultimate health outcomes (measured as health status, morbidity or mortality) of a society given its mix of healthcare and non-medical determinants of health. We also define 'healthcare performance' as the degree of maintenance of healthcare system functioning (measured in terms of dimensions such as effectiveness, patient-centeredness and so forth) that is in keeping with the system's societal, professional and user goals and norms. Therefore, healthcare performance should, in principle, contribute to health performance.

Methods

Study population, data and measures

We used recent (2001 to 2003) secondary data on the performance of the thirteen Canadian provinces (ten in number) and territories (a total of 3) usually reported on by the government [12–16] (see Table 1). The ten provinces are Newfoundland & Labrador, Prince Edward Island, Nova Scotia, New Brunswick, Quebec, Ontario, Manitoba, Saskatchewan, Alberta, and British Columbia. The three territories comprise Yukon Territory, Northwest Territories, and Nunavut. These are the 13 jurisdictions that have constitutional responsibility for Canadian health and healthcare. The provinces and territories are of varying sizes, demographic constitution and capacities. The data underpin the annual Health Indicator publications which accompany the national Health Care in Canada reports [12, 15]. The data which cover 95% of the Canadian population are appropriately age, population and gender weighted. Their primary collection sources include the Canadian Community Health Survey (in 2000/01 and 2003), Hospitality Mortality Database (CIHI), Discharge Abstract Database (CIHI) and the databases of the Centre for Infectious Disease Prevention and Control (Health Canada).

Table 1 Descriptive statistics for the Canadian provinces and territories, in 2001 [12-14]

To address the question whether provinces and territories that are better off in one indicator are also better off in the preferred direction in other indicators, we chose two main outcome tiers from the CHIF. First, for the outcome 'health performance' (health) we included 6 indicators from the health status tier (see Table 2): well-being (1 indicator), health conditions (3 indicators), human function (1 indicator), and deaths (1 indicator). Second, for the outcome 'healthcare performance' (healthcare) we included sixteen indicators for healthcare performance tier to cover acceptability, accessibilty, appropriateness, effectiveness, safety, and health surveillance dimensions. There are currently little or no routinely reported indicators for the competence, continuity and efficiency dimensions. The choice of indicators from the entire set of measures within the CHIF was guided by data availability and completeness (for at least ten provincial/territorial units). The independent variables consisted of indicators of non-medical determinants of health, community and health system characteristics, and healthcare performance. For the non-medical determinants of health (see Additional file 1), twelve indicators were chosen covering health behaviors, living and working conditions, personal resources and environmental factors. In addition, sixteen indicators were selected from the community and health system characteristics tier to cover the community, health system and resource fields. The details of these explanatory variables or indicators are available on request from the first author, and can also be found in the reference [12–16]. An indicator was considered positive (negative) if higher (lower) values of it would be preferred in reality. Whenever it was unclear whether a higher or lower level of an indicator would be preferred, we labeled it as both positive and negative (+/-).

Table 2 Health and healthcare performance indicators: descriptions, estimates, Canadian averages, and data sources [12-16]

Analysis

Pearson's correlation coefficient r was used to estimate the univariate associations between the indicators of the 2 main outcomes (health and healthcare) and the indicators of non-medical determinants, community and health system characteristics, and healthcare performance. The unit of analysis was each of the thirteen Canadian provinces and territories. The indicators were log-transformed to avoid spurious correlations between rate-based indicators. Furthermore, given the small sample size (N = 13), we used critical (cut-off) values of r to ensure that the correlation was real and to minimize the chances of committing type I error, that is incorrectly rejecting a true statistical null hypothesis. At a significance level of 5%, the critical values of r for small sample sizes are as follows: r* ≥ 0.552, 0.576, 0.602 or 0.631 for N = 13, 12, 11, or 10 respectively. At a significance level of 1%: r* ≥ 0.683, 0.707, 0.734 or 0.764 for N = 13, 12, 11, or 10 respectively [17]. Another conservative statistical choice was that we used two-sided p-values to assess the significance of the correlations.

We applied decision rules to aid the interpretation of significant correlations. A correlation between any two indicators i and j was considered a significant preferred performance association if the coefficient r was positive when both i and j were positive or when both were negative. However, if one indicator was positive while the other was negative, the correlation between them was considered a significant preferred performance association if the coefficient r was negative. In all cases, the abovementioned critical value (r*) requirement must have been met. Assuming a null hypothesis H₀ that r = 0, the following decision rules were applied to the univariate correlations:

• (a) if indicators i and j were both either positive (+) or negative (-), then the following decision rule applied to their preferred positive correlation r _ij:

r ≥ r* → reject H₀

r <r* → do not reject H₀

• (b) if only one of the indicators i and j was positive (+), then the following decision rule applied to their preferred negative correlation r (where r* took on a negative value):

r ≤ r* → reject H₀

r > r* → do not reject H₀

Whenever the correlation was significant but not in the preferred direction, the result was considered a suboptimal performance association (termed in this paper as not preferred performance) that could point towards possibilities for improvement. To estimate the uncertainty around each estimated r, we wrote a simple spreadsheet for calculating the 95% confidence interval (C.I.) for each sample size. (This spreadsheet is available on request from the first author.)

As far as the decision rules were concerned, caution was exercised in applying them to indicators for which it was unclear whether higher or lower values would be preferred. For this reason, the decision rules were not applied to three indicators of community characteristics, namely population size, elderly population, and urban population.

All analyses were carried out using SPSS version 12.0.2 (SPSS Inc., Chicago, IL) and Microsoft^® Excel 2002 SP3 (Microsoft Corporation, Redmond, WA).

Results

Table 3 shows that significant 'preferred' correlations between health and non-medical determinants of health range from -0.853 (95% C.I.: -0.955 to -0.570; P < 0.05) for the association between body mass index and dietary practices to 0.836 (95% C.I.: 0.528 to 0.950; P < 0.01) for the association between unemployment rate and diabetes rate. Smoking status, having high proportions of high school and post-secondary graduates displayed unfavorable ('not preferred') associations with health indicators (Table 3).

Table 3 Correlations between health (status) indicators and non-medical determinants of health

Table 4 shows the correlations between health and healthcare performance indicators. Here, the significant 'preferred' correlations range from -0.782 (95% C.I.: -0.932 to -0.406; P < 0.01) for the association between provincial/territorial performance on diabetes rate and vaginal birth after Caesarean section to 0.754 (95% C.I.: 0.347 to 0.922; P < 0.01) for the association between provincial/territorial performance on diabetes rate and in-hospital 30-day stroke mortality. Table 5 shows the correlations of health indicators with community and health system characteristics. Again, significant 'preferred' correlations range from -0.893 (95% C.I.: -0.968 to -0.673; P < 0.01) for the association between knee replacement and functional health status (as perfect or very good) to 0.810 (95% C.I.: 0.468 to 0.941; P < 0.01) for the association between provincial/territorial performance on diabetes rate and hysterectomy. Likewise, several 'not preferred' performance associations exist between health indicators and mostly (health system) resource indicators (Table 5).

Table 4 Correlations between health (status) indicators and healthcare performance indicators

Table 5 Correlations between health (status) indicators and community & health system characteristics

The table in Additional file 2 gives an overview of the correlations between healthcare performance, on the one hand, and non-medical determinants of health and community and health system characteristics, on the other hand. The significant 'preferred' correlations range from -0.863 (95% C.I.: -0.958 to -0.595; P < 0.01) for the association between provincial/territorial performance on screening mammography and its frequency of heavy drinking to 0.944 (95% C.I.: 0.819 to 0.983; P < 0.01) for the performance association between smoking status and chlamydia cases per unit population. Several 'not preferred' associations also exist for the healthcare performance outcome. For instance, the share of public sector health expenditure per capita shows several strong correlations with healthcare indicators: ranging from -0.716 (95% C.I.: -0.909 to -0.273; P < 0.01) for being satisfied with family doctor to 0.851 (95% C.I.: 0.565 to 0.954; P < 0.01) for chlamydia cases per unit population. Preferred inter-correlations among healthcare performance indicators ranged from -0.867 (95% C.I.: -0.960 to -0.605; P < 0.01) for Caesarean section rate versus vaginal birth after Caesarean section to 0.716 (95% C.I.: 0.273 to 0.909; P < 0.05) for the association between being satisfied with family doctor and being satisfied with healthcare services.

Tables 6 and 7 present the summaries of the correlates of health and healthcare performance. Table 6 shows that there are relatively more 'preferred' associations between health and non-medical determinants (that is, 20 out of 72 correlations, or 28%) than between health and healthcare performance indicators (that is, 12 out of 96 correlations, or almost 13%). Similarly, there are relatively fewer 'not preferred' correlations between health and non-medical determinants (that is, 6 out of 72 correlations, or 8%) than between health and healthcare performance indicators (that is, 11 out of 96 correlations, or 11%). Also, the associations between health and community/health system characteristics out-number those between health and healthcare performance. There are 19 significant 'preferred' associations out of 78 correlations between health and community/health system characteristics (that is, almost 24%), while there are 18 significant 'not preferred' associations (that is, 23%). Table 7 shows that out of 190 correlations between healthcare and non-medical determinants, the significant 'preferred' and 'not preferred' associations are respectively 23 (12%) and 18 (9%). Based on 208 correlations between healthcare and community/health system characteristics, there are 27 (13%) and 35 (17%) significant 'preferred' and 'not preferred' associations respectively. Interrelationships among healthcare indicators are few in number within and between dimensions (details not shown but summarized in Table 7; see Additional file 2).

Table 6 Summary of significant 'preferred' and 'not preferred' performance correlates of health (status) indicators^†

Table 7 Summary of significant 'preferred' and 'not preferred' correlates of healthcare performance indicators^†

Discussion

This is the first study to estimate the correlates of health and healthcare performance of Canadian provinces/territories. It suggests that relatively better performance on non-medical determinants of health is related to better health. Healthcare performance is, however, less frequently related to health. In addition, health is relatively better associated with community/health system characteristics than healthcare performance is.

Provincial/territorial healthcare performance shows relatively more 'preferred' than 'not preferred' associations with non-medical determinants. Healthcare correlations with community and health system characteristics show the reverse picture, with relatively more 'not preferred' associations than 'preferred associations'. This again suggests there is still more to be desired in how provinces/territories simultaneously optimize their performance in terms of health and healthcare, given their community and health system characteristics. Interrelationships between healthcare performance indicators suggest that how the healthcare system performs in terms of one indicator is often not related to its performance in terms of another indicator. Importantly, there are at least two ways of looking at the correlations. First, the correlations can be interpreted as possible associations between the epidemiological factors that underlie the indicators (that is, epidemiological associations) in ideal circumstances; for example, body mass index is associated with dietary practices in an epidemiological sense. Second, the correlations could be viewed as no more than associations between the actual performance attainments of provinces/territories in terms of different indicators (that is, performance associations) in everyday circumstances. Although both views are related and implied in this study, we recognize that the latter can undermine the former when real "epidemiological associations" are not observed in sub-optimal or "not preferred" performance scenarios.

Explanation of results

Health is a function of multiple factors or determinants that work in complex, sometimes unclear ways [7–9, 18]. The results of this study, although based on multiple univariate correlations, support this notion. In a similar correlational analysis used in a study of 311 local administrative units covering 70 million populations in Japan, a group of nine health determinant indices (namely, healthcare resources, preventive health activities, environmental quality, housing urban clutter, local economy, employment, income, and education) explained almost 52% of the variances in health index levels in the cities studied [19]. In our study, it could be shown that, if independently assessed, non-medical determinants could explain between 40% and 67% (calculated from r-squared) of the variance in life expectancy as a measure of health at the provincial/territorial level (see Table 3). Similarly, healthcare performance indicators could independently account for 44% to 57% of the variance in life expectancy.

Unsurprisingly, health has relatively more associations with non-medical determinants than with healthcare indicators. Studies looking at healthcare inputs and resources to explain variations in the health of countries or other ecological units of analysis have mostly failed to demonstrate any or consistent associations [20–22]. However, it is also possible that the non-medical determinants correlate more frequently with health levels because they represent factors (such as dietary practices, smoking and so on) that are more or less related to disease risk profiles, prevalence and incidence in a general population. Healthcare factors reflect mostly corrective or management measures that marginally influence the prevalence or incidence of chronic ill-health or diseases, particularly in the face of co-morbidities in risk populations. This does not imply that healthcare is not life-saving for those who need it, when they need it. The point is that the contribution of healthcare to the general health of a population may be limited only to relatively small groups, in time and space, which stand to benefit from effective healthcare, but its overall effect will be diluted in summary measures of population health or well-being. Therefore, using the prevalence of diagnosed health conditions such as diabetes rates as indicators of health performance can only yield disappointing results in relation to healthcare performance. This explanation may be relevant to the results detailed in Table 4 when compared to Table 3 findings.

There are also results (the significant 'not preferred' correlations) which suggest that provinces may be struggling with optimizing their health performance given their health determinants, and healthcare and community profiles. For example, the higher the percentage of high school or post-secondary graduates in a province/territory, the higher the asthma rate (r = 0.855 or 0.711, P < 0.01, in Table 3). This may be expected given that asthma is more prevalent among the younger age groups. The comparable frequency with which health displays both 'preferred' and 'not preferred' associations with community and health system characteristics also points to the possibility that health levels are shaped community needs in complex ways that this study cannot disentangle.

Moreover, it is difficult to say which indicator precedes the other in this study. Healthcare indicators may just be current responses to perceived previous shortcomings in health. For instance, the negative correlation (r = -0.773, P < 0.01, in Table 5) between health expenditure and life expectancy could be due to an increase in total healthcare spending in provinces/territories with a long history of lower health levels. Nunavut, for example, is a collection of 26 communities with 28,000 inhabitants living in a vast territory about one-fifth the size of Canada. Nunavut is only accessible by air or sea, and has substantial difficulties in recruiting and retaining health professionals although it spends twice as much as the Canadian average on health per capita (Table 1). Given, the poorer-than-Canadian-average health in Nunavut, the government has been investing a lot in health and healthcare there. Therefore, health expenditure will understandably display a negative association with life expectancy.

In this study, we assumed that the lower rates of knee replacement or bypass surgery and other contextual health system indicators, when seen in the context of higher health outcomes, will be preferred. It could, however, be argued that lower rates of such contextual indicators could be indicative of unmet needs (locally). Given such interpretations, we would have to reverse the interpretations and associated correlations to reflect the possibility of under-use of needed healthcare in such communities. Nonetheless, our current interpretations allow for the possibilities of over-use of appropriate healthcare in communities where health outcomes are already high. These considerations further highlight the often overlooked difficulties that are inherent in understanding published performance data, regardless of the amount of contextual information provided. In a sense, the meaning and excellence of performance may be in the eye of the beholder.

Implications

Recommending policies on health and healthcare in the provinces and territories must take into account the responsibility structure, organization, delivery and funding of healthcare in the concerned areas. Blanket recommendations will probably miss the point by being too generic and insensitive to local needs. The Canadian healthcare system is mainly publicly funded (Medicare) [4]. The provinces and territories have primary constitutional responsibility for health, and the management and delivery of health services, although they all adhere to a set of federal principles in view of the Canadian history of fiscal transfers from the federal to the provincial governments [23]. A number of interlocking general revenue-financed health insurance plans cover hospital in-patient and out-patient services, pharmaceutical products, physician services and public health services. Therefore, there is, in principle, a lot of improvement leverage points that provincial, territorial and federal governments can use to better the health of Canadians. Nevertheless, there are serious challenges and tensions posed by the varying needs of multiple stakeholders and the use of broad summary indicators or a parsimonious set of indicators. It is advisable that governments invest more in investigating and interpreting possible linkages among performance results in order to aid learning and to facilitate the simultaneous optimization of different performance dimensions and indicators.

That said, it is also becoming increasingly clearer that investing in public health, especially in promoting healthy life style, disease prevention and health protection, may still offer new avenues for dealing with population health in western societies [24–26]. The current narrow focus on technical care may not be the best approach to improving and maintaining population health when most gains are to be made by living and working well, for example. The social choice arrangements needed to achieve better health for Yukon Territory, Northwest Territories and Nunavut will be have to be ambitious. There are signs that some provinces are already investing more in both health and healthcare performance of their communities [27–29].

Study limitations

The data used in this study come from multiple sources with different data elements and quality. Although the Canadian government continues to invest in the quality and coverage of the data used in constructing the indicators, the system is still not perfect [16]. To ensure data quality and comparability in performance reporting, the Canadian First Ministers have been giving policy support to the federal government and the 13 provincial/territorial jurisdictions since September 2000 [4, 16]. In February 2003, the First Ministers' Accord on Health Care Renewal directed Health Ministers to further develop indicators to supplement the work on comparable indicator reporting. So far, about 70 indicators have been standardized for comparable reporting at the provincial/territorial and federal government levels.

Furthermore, it is beyond the scope of this study to ascertain which indicator is really a cause or an effect. Correlation does not imply causality, and this is troublesome in ecological observational study designs [30]. Bearing this in mind, we only hope to speak to the (sub)optimization of a pair of indicators based on the attained performance. We also realize that statistical significance does not necessarily imply substantive significance of performance. Unconfounded associations, temporality, and real-world translation of the performance associations, particularly causal ones, are more appropriate criteria for assessing importance of the association. Such assessments will also entail value judgments pertaining to how good the performance levels may be. Besides, it is quite possible there are important lag effects of health determinants and other indicators on health and healthcare performance that this study will be unable to pick up, given its contemporaneous cross-sectional ecological design.

A vexing limitation is the issue of multiple correlations and significance testing, given also that the health and healthcare indicators are not independent. At a significance level of 5%, there is a 1 in 20 chance of getting a spurious significance, just by chance. Thus, given the large number of correlations conducted in this study, it is quite likely that some correlations occurred by mere chance. However, given the rather low to moderate number of significant results and the fact that we actually pre-specified our paired analyses, it is likely that the magnitude of errors introduced by the multiple independent correlations using the same sample file will be relatively minimal. Besides, our decision rules were rather conservative. We could have set a stricter significance level by, say, dividing 0.05 by the number of anticipated tests (that is, using the Bonferroni method) [31]. Although this would minimize our type I error rate, it would have depleted our statistical power, thus giving a higher probability of type II error (that is, the probability of rejecting an association that actually exists). A trade-off between committing type I error and having enough statistical power was thus necessary, particularly given our already small sample size.

Conclusion

The results of this exploratory study should serve as a provocative basis for future research into performance interrelationships. The prevailing assumption that publishing a comprehensive battery of indicators will automatically lead to clearer understanding and contextualization of performance is not tenable. This study forces us to take a closer look at how we actually interpret such indicators in relation to one another, and we have seen that there are no easy rules for understanding possible links between what a community attains in one indicator and what it achieves in another. Since performance is interventionist in nature, health and healthcare performance can be influenced by those who have the ability and resources to do so. This study suggests that indicators which are correlated with how well Canadian provinces and territories perform in terms of health and healthcare can act as leverage points for improving health. This study has many implications for further research on linkages within performance frameworks now being used in several industrialized countries, and for choosing a national performance framework. For instance, a framework that focuses mostly on healthcare performance (e.g. the US National Healthcare Quality Report framework or the now old UK NHS performance assessment framework) does so at the expense of understanding the links between non-healthcare determinants and population health. Further elucidation of the meanings, nature, and extent of the interrelationships among the different fields and domains of the Canadian or any other performance framework will aid actual performance improvement by pointing the responsible governments in the right direction.