This study has supported the feasibility of the RAI-MH for developing MHQIs based on three dimensions. However, the findings point to a need for further validation and risk adjustment in order for to support ongoing quality monitoring. The RAI-MH sub-scales used to derive MHQIs for depressive symptoms (the DSI) and cognitive performance (CPS) had acceptable psychometric properties and variability among persons receiving adult inpatient mental health services. Unadjusted rate and prevalence MHQIs derived in this study showed substantial variation among hospitals in both the pilot and OMHRS data and no evidence of ceiling or floor effects were identified. The lower mean rates of improvement for cognitive performance compared to depressive symptoms may be related to the more intensive nature of interventions for cognitive issues (e.g., occupation rehabilitation) compared to faster acting interventions (e.g., pharmaceutical) available for management of depressive symptoms. This finding combined with the variability found among hospitals within each MHQI supports the potential utility of the MHQIs for identifying variations in quality. A lack of variability and consistently high or low rates would indicate that the MHQIs are not responsive to variations in practice between hospitals.
The substantial variance in facility MHQI scores, particularly among indicators measuring changes in cognitive performance, may also be related to the inherent differences in the patient case mix among Ontario hospitals. Evidence was limited for the utility of the SCIPP CMI as a risk adjuster for the five MHQIs explored in this study. Conceptually, the SCIPP CMI would seem to be an appropriate risk adjustment variable since it is a composite measure of diagnosis and patient characteristics that are organized to produce scores reflecting greater resource intensity. Since it is used for reimbursement, adjusting based on the SCIPP CMI would have potential to protect against inappropriately rewarding or penalizing facilities for treating either higher or lower risk patients. Although significant relationships were found between the SCIPP CMI and MHQIs for cognitive performance and restraints, adjustment using the SCIPP CMI led to virtually no difference between unadjusted and adjusted hospital MHQI rates or prevalence. The lack of effect in risk adjustment may be related to the composition of the SCIPP CMI. It also suggests that different MHQIs are likely to require different risk adjusters and that at least some of these will require more specific, rather than global, measures of risk. In addition, although SCIPP includes variables that would be expected to affect risk adjustment (e.g., diagnosis) the variables that largely drive resource intensity (e.g., behaviours, suicidality, day of stay, positive symptoms) are also those that are likely to improve rapidly once treatment is initiated. Thus, while they are related to greater resource utilization at a point in time they may not be related to the likelihood of good or poor outcome over time. Instead, case mix adjustment of MHQIs should focus on the use of singular clinical constructs (e.g., items or symptom scales) that impede clinical efficacy. The broad set of clinical information assessed by the RAI-MH will be useful for improving the effectiveness of risk adjustment of MHQIs. Further research is needed to identify case mix adjusters of MHQIs using variables representing singular clinical constructs (e.g., items or symptom scales) rather than composite indices of resource intensity.
In addition to case mix differences, the variability in MHQI scores may also reflect differences in care processes between hospitals. Detecting such differences is an important attribute for quality indicators as it links the measurement of quality to specific opportunities for quality improvement. The variability in preliminary MHQI findings between Ontario hospitals could be related to differences in measurement properties, ascertainment, policy, or practice. Further analysis of the sensitively to change of the RAI-MH scales are needed to determine if variability in the MHQIs is attributable to the sensitivity to change of the scales or actual variations quality. Regarding ascertainment, a thorough training program for completing the RAI-MH is provided to hospital clinicians by CIHI, “refresher” training may ensure clinical staff are able to appropriately ascertain clinical dimensions of these and future MHQIs. Ongoing data quality monitoring should examine inter-rater reliability in periodic sub-sample evaluations to identify opportunitiesion for the RAI-MH. Practice issues may be explored through a comparison between the MHQIs and other established quality indicators. Once risk adjustment is improved, establishing a relationship between the MHQIs and technical process indicators (e.g., staffing levels, use of evidence based practices, availability of ongoing assessment training, etc.) may identify other contextual factors related to differences in MHQI scores between facilities. More importantly, linking the quality indicators to specific care planning activities may be more effective in understanding how care processes relate to outcomes. The RAI-MH includes care planning applications called Clinical Assessment Protocols (CAPs; formerly Mental Health Assessment Protocols) [52, 53] that identify patient problems or needs to staff and include a series of guidelines that staff can use to intervene if deemed necessary. Several CAPs may relate to potential domains of outcome and quality including aggressive behavior and violence, financial and medication management, activities of daily living, pain, interpersonal conflict, acute control medication and physical restraint use. Examining the relationship between use of the CAPs and outcomes based on the MHQI will be important to further validate the MHQIs and provide a mechanism for engaging clinical staff in the quality measurement and improvement process directly at the point of care.
The methodology used in this study could be applied to derivation of MHQIs based on other domains of quality and outcome. In addition to depressive symptoms, cognitive performance, and restraints, the RAI-MH includes a number of other domains of symptoms, behaviours, functioning, and safety. For instance, the RAI-MH includes a positive symptoms scale, pain scale , aggressive behavior scale , activities of daily living scales , and items measuring social functioning, violent and disruptive behavior, financial and medication management, and chemical restraint use are also available [28, 41]. Given the feasibility of the RAI-MH for designing quality indicators, these dimensions should be explored for the development of an inventory of RAI-MH MHQIs.
Several limitations of this study should be noted. First, the pilot data were drawn from only seven facilities that used the RAI-MH prior to the provincial mandate; however, the availability of OMHRS data made it possible to replicate the findings guiding initial MHQI selection. Second, the MHQIs were derived from data that excluded patients with stays of less than 6 days and had only one assessment available. Establishing outcome MHQIs for these may be limited to prevalence indicators of events such as self harm, harm to others, and control procedures. Third, several potential MHQIs measuring the appropriateness of medication use were excluded because medication data was unavailable among all facilities. While all interRAI instruments include sections on medication use, the OMHRS data requirements do not include mandated submission of medication data. Given the importance of pharmaceutical therapies as part of psychiatric services, the lack of these data is an important limitation of the OMHRS database. Without medication data inferences cannot be made about differences in quality between facilities that could be related to the appropriate use of medications.
The current dichotomy used in the MHQIs to measure change does not account for the magnitude of change in a given domain. Indicators assessing magnitude of change would complement the MHQIs by providing information on the clinical efficiency of mental health services in addition to effectiveness. Research is currently underway by the authors to examine the sensitivity to change of the RAI-MH sub-scales, including the DSI and CPS, in order to develop indicators measuring clinical efficiency based on the magnitude of change in the MHQI domains over time. In combination, the MHQIs and efficiency indicators could improve the accuracy of facility rankings by combining information on the proportion of patients who change with the magnitude of that change.
The MHQIs, once validated, carry several advantages for quality improvement at the person and organization levels. The MHQIs examined in this study measure domains that are important for the person’s recovery process and safety. Supporting improvements in cognitive functioning, for instance, may help the person achieve and sustain integration back into community settings. The measurement of such outcomes in accountability frameworks and report cards reinforces interventions directed at improvement, thus benefiting patients with such needs. At the organizational level, the inclusion of improvement as well as incidence/failure to improve in many MHQI domains emphasizes success and opportunities for improvement. The use of prevalence indicators for restraints emphasizes patient safety. Combined, these dimensions will be useful for supporting quality improvement by identifying and sharing best practices between higher and lower ranking facilities based on theses and, potentially, other MHQIs.
Once further refinement is complete, the MHQIs may also be applicable for quality measurement across health sectors, from inpatient to community mental health and beyond. All interRAI instruments include core items that are consistent across all assessments as well as items that are specific to the sector being measured . The interRAI Community Mental Health (CMH), for instance, contains 60% of the items used in the RAI-MH. Only the restraints MHQI cannot be measured using the interRAI CMH. Completion of the interRAI CMH 30 days after discharge, for instance, could serve as a third follow-up assessment for inpatient MHQIs and a baseline assessment for community MHQIs. The interRAI CMH has been pilot tested in Canada (Ontario, Newfoundland), Iceland, Cuba, and Chile, but it has not yet been mandated for regular use.