The multi-dimensionality of quality
Our initial problem arises because quality is multidimensional, making it difficult to summarize the quality of care in individual nursing home facilities. In most measurement situations, we operate within the confines of a traditional measurement model. We deal with some unmeasured entity (quality) represented by relatively highly correlated indicators. In the world of nursing home care our situation is much more analogous to what Bollen and Lennox [8] identify as a causal model of measurement. Changes in quality do not so much move correlated indicators of quality as changes in marginally, or uncorrelated, indicators move the larger unmeasured construct of quality (see Figure 1). A home may improve skin care, theoretically increasing the value of our unmeasured construct of quality, but that change will have no effect on that home's use of feeding tubes, another reasonable and appropriate indicator for that larger unmeasured construct.
As an illustration, in one study, the falls quality indicator from the Minimum Data Set for Nursing Home Resident Assessment and Care Screening (MDS) was correlated with 24 other quality indicators (QIs). The average correlation of these 24 indicators with the falls indicator was 0.06, with seven of the correlations being negative [2]. John Hirdes and colleagues [1] used an interesting visual display that illustrates the same lack of high correlations even more vividly. They presented QI percentile ranking for multiple QIs for a single home in a radar graph (analogous to a circular bar graph). Ideally, the multiple rankings would form a perfect circle or, at least, an oval. Instead, far more often, the result was an uneven field of peak and valleys, giving one little hope of high correlations among the quality indicators for a single facility. Were such high correlations present, it would be much easier to think of how one might develop some single scale based on diverse quality indicators that would allow one to talk about overall quality of care in homes.
At this point one may wonder whether quality measurement at the home level is a useful endeavor. If we can't find a set of highly correlated indicators, then some might argue that the logic of scientific inquiry suggests that we look for a less general conceptualization, or a series of dimensions of quality, that might be composed of more highly correlated items. Possibly, as some suggest, deeper inquiry into the dimensionality of quality will provide assistance [5] as might being more conceptual in our thinking about quality [9].
This might, in fact, be a useful exercise in a largely academic inquiry into quality. In the applied world of nursing home operations, however, these less than ideal, poorly correlated indicators; the care problems they may represent; and the residents who may be at risk in each care area all appear in a single home. Residents can't get their skin care from one home because of its superior performance on this indicator and get their incontinence care from a different home because of its performance on this quality indicator. Residents receive all their care in a single home.
But, it is not only residents and potential residents who need information on facility-level quality. Regulators may wish to develop monitoring systems that involve more serious scrutiny for more problematic providers. Policy-makers are also becoming more interested in payment-for-performance, which will demand an assessment of facility-level quality [10]. In all of these instances, entire homes must be discussed. However, what remains unclear, due in part to the multi-dimensionality of quality, is how much one can or should say about individual homes?
The multi-dimensionality of the nursing home population
One potential solution for concerns about the multi-dimensionality of quality, at least as it concerns consumers, has been to provide a wide array of information [7]. For homes across the country, consumers can now acquire online data on deficiencies in multiple surveys, staffing levels, financial data, and quality indicators. What many consider the best of these sites tells a visitor whether the information indicates the home is average, above average, or below average on some indicator. But, is average care good or bad? How do consumers integrate such data on multiple, potentially uncorrelated, indicators?
Even if these systems are used and consumers can integrate the disparate pieces of information they received about a home, one faces a second problem, which is the multidimensionality of the nursing home population. Part of the difficultly in disseminating quality information is dealing with the array of very diverse residents that populate nursing homes. The NH Compare site operated by CMS differentiates between quality indicators for post-acute and for long-stay residents. This is an important step in such differentiation, but it glosses over much of the important variation among residents.
If a resident is bedfast and incontinent, family members, surveyors, and ombudsmen should care tremendously about skin and continence care. If the resident suffers from dementia but is physically active, they should care much more about safety, restraint policy, and activities. Robert Kane [3] has identified at least five different groups of residents who come into nursing homes. These groups include residents recovering from an acute episode and who are likely to return home, residents who are terminally ill, residents who are cognitively impaired, resident who are cognitively intact but suffer from physical challenges, and residents in vegetative states. All of these residents have different needs and different dimensions of quality will vary in their importance across these groups.
Discussing quality becomes even more complicated when we recognize that even within these five groups, residents themselves may entertain varying definitions of quality. As Barbara Bowers and her colleagues [11] indicate, some residents define quality in terms of service, others in terms of attention to comfort, while others define quality in terms of the nature of their interpersonal relationships with staff. Yet, our current level of sophistication in the targeting of quality indicators to special populations goes not deeper than recognizing the distinction between short-stay and long-stay residents.
Home-related variation in quality indicators
Another issue demanding much of our attention is resident outcomes. Unfortunately we have little idea just how much of the variation in resident outcomes is driven by a home's performance. A recent analysis of quality of life indicators [12] implies that such measures can distinguish among homes. However, the analyses indicated that that home characteristics such as size or ownership, explained less then 10% of the variation in quality of life. The analyses of MDS quality indicators considered for inclusion in the CMS NH Compare website required that nursing home characteristics explain at least 21% (r = .45) of the variation in a quality indicator [13] to meet their criteria for validity. It is a less than heartening situation when home characteristics explain just over one-fifth of the variation in an outcome or quality indicator, and we consider that indicator a "valid" measure of homes' performance.
As a group, researchers in nursing home quality seem to have largely avoided dealing with a critical issue. How much variation in a quality indicator must home performance explain in order for the research community to consider the indicator useful? While there is no real answer to such a question, since at this point it is fundamentally a matter of "taste," one can ask a similar question that is much more answerable. Which of our array of potential quality indicators are most affected by home performance? If 90% of the variation in Quality Indicator #1 is determined by factors outside the control of a home and only 35% of the variation in Quality Indicator #2 is determined by factors outside the control of the home, then Quality Indicator #2 may be more useful as an indicator of home performance. Unfortunately, this is not a question that is being addressed by the research or policy communities.
Risk adjustment and the validity of quality indicators
The face, content, and construct validity of the measures developed for nursing homes are relatively rarely points of debate, though a great deal of effort has been expended to ascertain how valid specific measures are when compared to direct observations of care [14–17]. In fact, much of this work has shown that current indicators poorly reflect observed care in homes. However, the validity that might be gained by substituting much more costly observations of care for the current indicators is not clear at this time.
The most often raised issue concerning the validity of the quality measures seems to concern the degree to which one can successfully risk adjust these measures [6, 5]. However, prior to the successful risk adjustment, one needs some clearer sense about the degree of covariance between homes and residents before beginning to understand how important risk adjustment may be. If only 21% of the variation in an indicator can be explained by home characteristics, then how much of the residual can be driven by the covariation of home and individual characteristics? How high must that covariance be in order to demand risk-adjustment?
Even when we make the decision to risk-adjust quality indicators, the task is far from easy. In essence, we are like Goldilocks with only two bears. For everything Goldilocks tried when she was engaged in breaking and entering at the Bear residence, whatever was meant for the one bear was too much, while whatever was meant for another bear was not enough. Only when she tried the bed or porridge of the third bear, was it "just right." When we risk adjust, we must consciously choose whether to go with Bear #1 and "over-adjust" or with Bear #2 and "under-adjust." Unfortunately, there is no third bear conveniently available to offer us the risk-adjustment model that is "just right."
Fortunately, our adjustments will rarely be so powerful that they make the truly good homes look bad or the truly bad homes look good. However, when we over-adjust, we run the risk of making bad homes look mediocre, and under-adjusting may make good homes look mediocre. When we compare the cost of erring in either direction, it seems that the most reasonable course is to consciously under-adjust. Given the option, we provide better service to consumers and other stakeholders (except those facilities that are mediocre and identified as bad) by failing to give a few mediocre homes their due than to make anyone think that a bad home is just mediocre.
