Development and validity testing of a matrix to evaluate maturity of clinical pathways: a case study in Saskatchewan, Canada

Abstract

Background

Healthcare systems are transforming into learning health systems that use data-driven and research-informed approaches to achieve continuous improvement. One of these approaches is the use of clinical pathways, which are tools to standardize care for a specific population and improve healthcare quality. Evaluating the maturity of clinical pathways is necessary to inform pathway development teams and health system decision makers about required pathway revisions or implementation supports. In an effort to improve the development, implementation, and sustainability of provincial clinical pathways, we developed a clinical pathways maturity evaluation matrix. To explore the initial content and face validity of the matrix, we used it to evaluate a case pathway within a provincial health authority in Saskatchewan, Canada.

Methods

By using iterative consensus-based processes, we gathered feedback from stakeholders including patient and family partners, policy makers, clinicians, and quality improvement specialists, to rank, retain, or remove enablers and sub-enablers of the draft matrix. We tested the matrix on the Chronic Pain Pathway (CPP) for primary care in a local pilot area and revised the matrix based on feedback from the CPP development team leader.

Results

The final matrix contains five enablers (i.e., Design, Ownership and Performer, Infrastructure, Performance Management, and Culture), 20 sub-enablers, and three trajectory definitions for each sub-enabler. Supplemental documents were created for six sub-enablers. The CPP scored 15 out of 40 possible points of maturity. Although the pathway scored highest in the Design enabler (10/12), it requires more attention in several areas, specifically the Ownership and Performer and the Performance Management enablers, each of which scored zero. Additionally, the Infrastructure and Culture enablers scored 2/4 and 3/8 points, respectively. These areas of the CPP are in need of improvement in order to enhance the overall maturity of the CPP.

Conclusions

We developed a clinical pathways maturity matrix to evaluate the various dimensions of clinical pathways’ development and implementation. The goals of this initial work were to develop and validate a tool to assess the maturity and readiness of new or existing pathways and to track pathways' revisions and improvements.

Background

Healthcare systems are transforming to become learning health systems (LHS) in which quality care and value are achieved by demonstrating improvements to patient experiences and health outcomes, provider experiences, population health, and health system costs [1]. In LHS, continuous improvement is achieved by data-driven and research-informed approaches [1,2,3,4,5]. One of these approaches is the utilization of clinical pathways [6, 7]. Clinical pathways, also known as critical or integrated care pathways [8], are operationally defined as tools to standardize care for a specific population, translate guidelines or evidence into local structure, create a structured multidisciplinary care plan, and detail a care plan in an inventory of actions [9,10,11,12]. Clinical pathways can improve patients' and providers' experience and satisfaction, resource utilization, and inter-professional teamwork while reducing knowledge transition gaps, healthcare team burnout, costs, and variation in care [11, 13,14,15]. Further, clinical pathways can be utilized to improve the domains of healthcare quality including safety, effectiveness, patient-centeredness, timeliness, efficiency, and equitability [14, 16, 17].

Implementation barriers such as knowledge users’ awareness, stakeholders’ engagement, information technology (IT) infrastructure, and performance management have been shown to impede optimal integration of clinical pathways into healthcare systems [11,12,13, 17,18,19, 20]. To increase their impact, the development and implementation of clinical pathways should be guided by theories, models, or frameworks [12, 21]. Despite outlining the development and implementation of clinical pathways, many frameworks do not specify how to evaluate the maturity of pathways. For our purposes, we defined maturity as a dynamic state of planning, development, and readiness for a pathway to be implemented, replicated or scaled up, and sustained in its intended clinical settings in which the goals or outcomes of the pathway are achieved. Evaluating the maturity of clinical pathways can inform pathway development teams and health system decision makers about required pathway revisions or implementation supports to improve implementation outcomes such as acceptability, fidelity, feasibility, adoption, appropriateness, and sustainability [22]. Further, evaluating the maturity of clinical pathways enhances the effectiveness of clinical pathways by ensuring they are functioning as intended and achieving the planned effects at the patient, provider, and system levels [11]. To our knowledge, only one paper has been published that describes a formal and standardized process to evaluate the maturity of clinical pathways [13]. Although the maturity model described by Schriek et al. [13] provides a foundation for pathway evaluation, the Saskatchewan Health Authority (SHA) Clinical Pathways Core Team (CPCT) aimed to ensure that the purpose of the model, its enablers and sub-enablers and their definitions, and their trajectory definitions are compatible with the SHA environment. The process and results of verifying content and face validity of the proposed matrix through key stakeholders’ engagement and testing the matrix with a case pathway prototype within the SHA in Canada are described.

Methods

Setting

Serving a diverse population of 1.2 million residents with over 45,000 employees and physicians, the SHA is responsible for delivery of the majority of publicly funded health services throughout the province of Saskatchewan [23]. The SHA was launched in December 2017 through the amalgamation of 12 former health regions. The Clinical Excellence portfolio of the SHA is responsible for the development, implementation, and evaluation of new clinical pathways that guide clinical care for targeted conditions [24]. The SHA currently has clinical pathways for Acute Stroke, Bariatric Surgery, Chronic Pain, Fertility Care, Hip and Knee Replacement Surgery, Lower Extremity Wounds, Multiple Sclerosis, Pelvic Floor, Prostate Cancer, and Spine [24]. Additionally, there are pathways for Diabetes, Chronic Obstructive Pulmonary Disease (COPD), and Long COVID that are in development (Table 1). These pathways have been or are being developed by multidisciplinary stakeholder teams consisting of operational leaders, clinical experts, and patient and family partners (PFPs) [24].

Table 1 List of clinical pathways in Saskatchewan

In 2021, nine pathways developed by the Saskatchewan Ministry of Health (MoH) were transitioned to the SHA, for a total of 13 pathways that fall within SHA accountability (Table 1). At a provincial level, this accountability includes the responsibility of supporting development and implementation, maturing of clinical pathways, and progress reporting to the MoH. As the former MoH pathways were developed without a standardized approach, they varied in their design and scale (provincial versus local settings). Gaps were recognized in that no processes, tools, or methods existed to validate the maturity of each pathway, to compare the pathways to one another, and to provide progress reporting to the MoH. The SHA CPCT planned to develop a maturity evaluation matrix to bridge these gaps by providing a tool that could measure levels of maturity via design, awareness, usage, metrics inclusion, owner engagement and participation, and provincial replicability of the clinical pathways.

Developing the maturity matrix

A search of published English language literature in MEDLINE via Pubmed, CINAHL, Cochrane Library, and Google Scholar for maturity evaluation matrices or models for clinical pathways resulted in identification of only one relevant publication [13]. The maturity matrix published by Schriek et al. [13] contained five enablers and 19 weighted sub-enablers with four trajectory definitions (low, moderate, high, and top) for each sub-enabler. The matrix was initially examined and evaluated by the SHA CPCT to determine its compatibility within the specific context of Saskatchewan. Our CPCT included members with various backgrounds (medicine, quality improvement, implementation science, learning health systems, research, and psychology) as well as a pathway development team leader. The assessment of Schriek et al.’s matrix revealed the need to modify it based on the current knowledge in the fields of quality improvement, implementation science, and evaluation. For example, sub-enabler definitions and their trajectories required revisions to improve their clarity, potential translation issues were addressed and language was refocused to be patient-centered, a dedicated sub-enabler was added to capture the intricacies of the patient journey, and the complexity of the maturity scoring was simplified allowing evaluators to better distinguish between levels.

Using an iterative consensus-based process, email invitations (one initial and one reminder email two weeks later) were sent to SHA and MoH stakeholders with differing levels of experience in clinical pathway development and implementation as well as PFPs. Both emails were sent from the Director of Clinical Excellence in July 2022 with an attached copy of the draft maturity matrix (Fig. 1). We used purposeful and snowball sampling methods to identify the stakeholders from SHA and MoH. They came from diverse disciplines within the SHA, including nursing, executive directors, managers, clinical department heads, physicians, administrators, and a pathway developer from the MoH. Knowledge of pathway development among stakeholders ranged from those that had been involved with development and utilization of pathways to those that had moderate to no exposure in this area. To identify the PFPs, we asked the SHA or MoH stakeholders to recommend PFPs who they had previously worked with as well as contacted the SHA’s Patient and Client Experience (PCE) department [25, 26]. All PFPs were registered with the SHA’s PCE department and were compensated as per the organization’s PFP policy [26, 27]. Knowledge of pathway development among the PFPs ranged from involvement with pathway development and related concepts to no previous exposure in this area. Beyond diversity in professional roles, the stakeholders and PFPs included individuals of different ages, genders, ethnicities, and immigration backgrounds, reflecting a range of lived experiences and perspectives.

Fig. 1

Development of maturity matrix and iterative consensus-based processes

The stakeholders and PFPs were asked to review the draft maturity matrix, determine which enablers and sub-enablers of the matrix should be kept and weighted more importantly on a 10-point Likert scale, choose which sub-enablers to be removed, and complete a REDCap (Research Electronic Database Capture) questionnaire (Supplementary file 1) [28, 29]. Sub-enablers with a mean of 7 to 10 were considered important for inclusion.

After receiving the feedback, all potential participants were invited to attend virtual follow-up meetings in September 2022 via Webex platform [30]. Participation was voluntary and no identifiable information was collected during the virtual meetings (Fig. 1).

During the sessions with the SHA and MoH stakeholders, questions were posed to participants about the inclusion of categories integral to pathway development, including pathway ownership (i.e., owner identity), patient involvement (e.g., ongoing stakeholder engagement), provincial integration (e.g., network of pathways) and replication (e.g., capacity monitoring). Content and face validity related to the relevance, appropriateness, and utility of the tool were explored and verified through discussions regarding the purpose of the tool and potential end users.

The session with PFPs had a series of seven questions seeking patients’ perspectives (five general questions and two questions related to the importance of sub-enablers) (Table 2). The questions were designed in consultation with the SHA’s PCE department and based on the SHA’s “Setting the stage for successful meetings with patient family partners (PFPs)” guidelines (internal document). Highlights of the guidelines include building in sharing time (ice breakers, stories), avoiding medical jargon or acronyms, and listening to PFPs stories, even if they are about care that did not go well. We started with open ended questions about PFPs’ experience in pathways or Saskatchewan’s health system and asked questions related to what is important to them in their care (i.e., pathway outcomes). To ensure the use of plain language and avoid using jargon, the CPCT conducted a readability analysis on the matrix. Results indicated that the matrix was at the university graduate level on the Flesch Readability Scale. Given this, the CPCT decided to focus on overarching concepts instead of one by one sub-enabler review. This was done to promote PFP’s engagement in an open and inviting discussion.

Table 2 List of questions asked during a session with patient and family partners

Notes were taken during the meetings and summarized to participants who then provided additional feedback or context and validated the summary.

It is important to note that, while we chose to begin testing the use of the maturity matrix after the above rounds of development and refinement, the document is intended to be dynamic and continuously evolving based on feedback, context, and experience. While we have finalized the tool for current use, we remain open to future revisions to ensure its efficacy in improving pathways. The current goal is to maintain stability for a period of time to facilitate practical application, testing, and evaluation.

Applying the maturity matrix

We chose the Chronic Pain Pathway (CPP) because it was a newly developed pathway that had not been put into practice across the province. Using the Webex platform, the CPCT and the CPP development team leader met virtually from June to September 2022 (first round) and used the draft matrix to evaluate the CPP. The draft matrix contained 19 sub-enablers, each with four trajectories (low, moderate, high, and top). The purpose of the first round evaluation was to focus on the utility, clarity, and applicability of the draft matrix’s various components and scoring definitions (Fig. 1). From February to March 2023 (second round), the CPCT asked the pathway leader for her input on the elements of the revised maturity matrix, which contained 20 sub-enablers with three trajectories (low, moderate, and high). For this round of evaluation, we wanted to know if the terminology was relevant and if the matrix could be effectively used to rate the development of the pathway.

Based on our experience, we recommend the assembly of a diverse assessment team comprising of individuals with various backgrounds to complete a clinical pathway evaluation. This team may include patient and family partners, clinical experts, pathway developers, administrators, operational staff, quality improvement experts, implementation experts, researchers, administrative personnel, information technology experts, and policy makers. Assessments should adopt an iterative process of current state assessment and should be conducted regularly (e.g., annually) through a series of collaborative meetings where team members systematically review components of the pathway matrix and discuss the status of the pathway for each sub-enabler based on data collected over the year. Supplementary documents should be reviewed for relevance and teams should score the sub-enablers based on the defined trajectories. In the event of a disagreement among team members regarding a particular score, we recommend employing consensus-based approaches to resolution, ensuring that divergent viewpoints are acknowledged and reconciled through constructive dialogue.

Results

Design and structure of the maturity matrix

Thirty-seven people (32 SHA and MoH stakeholders and five PFPs) were invited to participate via email. Fourteen responses were received from the REDCap questionnaire (response rate = 38%). The mean score for importance of all sub-enablers was 7.9/10 (mean range: 7 – 9, SD: 1.8).

Two virtual follow-up meetings (two hour sessions) with 11 SHA and MoH stakeholders (all 32 stakeholders were invited) and one (two hour session) with three PFPs (all five PFPs were invited) were held. During the follow-up meetings with SHA and MoH stakeholders, the attendees emphasized the importance of and equal weighting for all sub-enablers in the maturity matrix, resulting in the inclusion of all in the final matrix. During the follow-up meeting with PFPs, there was agreement that all sub-enablers were of equal relevance. The importance of categories representing patient preferences was highlighted, including multidisciplinary care (Ongoing Stakeholder Engagement sub-enabler), standardization in care (Design Approach sub-enabler), evidence based approaches (Compliance sub-enabler), and ease of navigation (Clarity in the Decision Criteria sub-enabler), all of which had been considered during the development of the matrix.

Based on the feedback received during the follow-up sessions, the CPCT added a “Patient Journey Map” sub-enabler under the “Design” enabler (resulting in 20 sub-enablers), reduced trajectory categories to three (low, moderate, and high), rearranged the order of sub-enablers, and modified the definitions of two enablers and eight sub-enablers to align with the needs of the SHA context (Table 3). A full list of maturity matrix enablers, sub-enablers, their definitions, and the three trajectory definitions are presented in Supplementary file 2.

Table 3 Summary of maturity matrix enablers, sub-enablers, and their definitions

Since all sub-enablers of the maturity matrix were considered to be of equal importance by the stakeholders and PFPs, we did not incorporate weighting the sub-enablers as had been done in the maturity matrix that we modelled our work on [13]. Therefore, we used a simple sum of the sub-enablers’ maturity levels (low = 0, moderate = 1, and high = 2) to score maturity of a pathway. This results in minimum and maximum scores of 0 and 40, respectively.

During the revisions of the maturity matrix by the CPCT and the feedback received from stakeholders, the need was identified to develop supplemental documents for six sub-enablers (Pathway Objective Alignment, Compliance, Design Approach, Network of Pathways, Capacity Monitoring, and Adaptability). These documents were either adapted from other sources (e.g., SHA’s measurement planning templates), or templates were developed ad hoc (Table 4). The supplemental document templates are provided in Supplementary files 3 to 8.

Table 4 Summary of maturity matrix enablers and sub-enablers with supplemental documents

Scores of chronic pain pathway evaluation

The CPP development team leader and the CPCT met virtually in two rounds of meetings (i.e., seven meetings total) (Fig. 1) and scored the pathway twice. The first round of CPP scoring (four meetings, 6.5 h in total) resulted in a score of 18/57 (19 sub-enablers with four trajectory definitions [low = 0, moderate = 1, high = 2, and top = 3]) (Table 5). The CPP scored highest in Design (11/15), followed by Culture (4/12), Infrastructure (1/6), Owner and Performer (1/6), and Performance Management (1/18). The score was 15/40 in the second round (three meetings, 4.5 h in total), with the highest score in Design (10/12), followed by Infrastructure (2/4), Culture (3/8), Owner and Performer (0/4), and Performance Management (0/12) (Table 5).

Table 5 Chronic pain pathway first and second rounds scores, and reasons for score differences

Discussion

We developed a maturity evaluation matrix for clinical pathways based on a previously published matrix in which a generic business process maturity model was utilized [13]. We refined the previous matrix using iterative consensus-based processes that included a questionnaire and multiple group discussions with PFPs, policy makers, clinicians, and quality improvement specialists. All enablers from the previous matrix were retained, but 16/19 sub-enablers were modified and one sub-enabler (i.e., Patient Journey Map) was added.

The existing literature on this topic is limited, which has underscored a significant gap concerning the absence of a comprehensive tool for evaluating the maturity of clinical pathways. This proposed maturity matrix is specifically designed to support clinical pathway development and implementation teams in assessing various aspects of pathway maturity. These aspects include a) Pathway design: This includes factors such as clinical components, objectives of care delivery, adherence to evidence-based practices, and the extent of stakeholder involvement; b) Ownership: This category involves aspects such as leadership engagement and involvement, the assessment of role awareness, and the functionality of different roles within the pathway; c) Infrastructure: This pertains to the integration of infrastructure, both internally and externally, for disseminating information and the connectivity of the pathway to other relevant pathways; d) Performance Management: This encompasses the selection and utilization of metrics, the availability and collection of data, data usage, and planning for provincial replication; and e) Culture: This focuses on elements like pathway awareness, stakeholder engagement, the capability for provincial implementation, and the audit and evaluation process.

To facilitate this evaluation, completion of the supplemental documents contained within the matrix is required. For instance, the Clinical Pathway Listing document aids in assessing whether connections or overlap with other pathways were considered during the development and implementation of the pathway currently under review.

By using our proposed scoring tool, a clinical pathway development team can compare the score of the pathway with previous scores to ensure that the score is improving over time. Further, health system decision makers are able to compare different pathways or examine low scores for commonalities amongst multiple pathways to identify resource needs and systemic issues. For example, if all pathways score low in the Owner (Identity) sub-enabler, it may indicate sponsorship constraints for clinical pathways in an organizational structure that may impact the sustainability of pathways. Further research is needed to understand the interpretation of the total score and whether a threshold score can be identified for satisfactory maturity. At present, pathway development teams are encouraged to make decisions based on individual sub-enabler scores and to use total scores as an overall measure of pathway maturity.

It is important for all healthcare interventions to incorporate aspects of equity, diversity, and inclusion (EDI) into their development and implementation. While these are not explicitly sub-enablers in the current version of the maturity matrix, several components within the matrix incorporate considerations for EDI. For example, the patient journey map, provincial service planning (using supplemental tools such as capacity monitoring), and engagement of diverse patient and family partners and a multidisciplinary team for pathway development inherently encompass aspects of EDI. As future iterations of the maturity matrix are evaluated and modified, inclusion of EDI as a specific sub-enabler may be considered.

Our evaluation showed areas in which the CPP can be improved as well as areas that the matrix can guide further development of the CPP. For example, the CPP scored highest in Design, which may reflect the status of the pathway during the assessment. The CPP has been fully developed but only implemented within one local setting, with plans to be implemented provincially. In addition, the pathway was scored during a time of leadership change within the SHA, leaving a temporary gap in pathway ownership. This status impacted the pathway’s scores for elements such as sponsorship, owner identity, role awareness, connectivity, data collection, provincial replication, and ongoing adaptability. Low scores in the CPP infrastructure and performance management may reflect gaps in organizational investment in resources to support implementation and evaluation of clinical pathways.

During the follow-up sessions, the participants indicated that all sub-enablers held equal significance. This differs from the findings of Schriek et al. wherein weights were incorporated into the analysis through stakeholder consultations [13]. The difference in weighting may be attributed to various factors, such as the revisions we made to the original maturity matrix, differences in methodological approaches (our study's utilization of the consensus-based approach versus the Delphi approach employed by Schriek et al.), and variations in the stakeholders involved. While opting for a non-weighted maturity scoring approach offers simplicity, it may not fully reveal the nuanced distinctions between the sub-enablers. Future studies could play a crucial role in unraveling the potential benefits of adopting a weighted scoring approach.

Limitations

Several limitations should be noted in our study. First, we did not include developers from different clinical pathways in the development of the matrix, which may limit the generalizability of our results. Additionally, we did not have a PFP in our CPCT, which could have provided valuable input from the patient perspective. To mitigate these limitations, we used an iterative consensus-based approach to gather input from a diverse group of stakeholders in developing the matrix.

We considered higher scores as improvements in pathways’ maturity. However, this may not be a reflection of reality. Currently, there is no gold standard by which to measure the accuracy of enablers or sub-enablers of our maturity matrix. At this stage, it was considered critical to ensure that stakeholders agreed on what enablers or sub-enablers were important to observe and how to differentiate between strong and weak performance in those attributes (i.e., content and face validity). Stakeholders agreed that all relevant elements of pathway maturity were included in the enablers and sub-enablers and that measurement trajectories were appropriate. Reviewing the CPP with the CPP development team leader seemed to confirm the face validity of the matrix because it was considered by a targeted end user as effective in measuring maturity (i.e., dynamic state of planning, development, and readiness for a pathway to be implemented, replicated or scaled up, and sustained in its intended clinical settings).

Future direction

There are several areas for future research related to our clinical pathways maturity matrix. First, we did not perform test–retest or inter-rater reliability testing of the matrix, and therefore future studies should evaluate the matrix’s reliability and validity. Further evaluation is needed to determine if the matrix is able to predict pathway progression to future improved state and successful implementation. To do this, the CPCT will monitor pathway development and implementation using the matrix and whether additional elements of pathway maturity emerge with more widespread use of the matrix.

Second, our study highlights the need for standardized measures for performance management of pathways (e.g., length of stay, patient reported experience measures, and patient reported outcome measures). However, IT support is needed to access data. Future research should explore the data access barrier and examine its impact on pathway implementation.

Finally, future versions of the matrix could include implementation, service, and client outcomes, such as pathway adoption, sustainability, or stakeholder satisfaction [22]. These outcomes would provide a more comprehensive picture of the pathway's maturity and impact. It is worth noting that several tools can supplement the clinical pathways maturity matrix, adding complexity, sophistication, and efficacy to the evaluation process. While the focus of this study was on the development and testing of the matrix, it is important to acknowledge the value of these supplementary tools. Future research could explore the integration of these tools into the evaluation process and their impact on the accuracy and utility of the matrix.

Conclusions

The SHA Clinical Pathways Core Team (CPCT) has developed a maturity matrix that can serve as a tool for evaluating both new and existing clinical pathways. This matrix plays a role in evaluating the design quality of pathways and identifying gaps and limitations in their implementation and replication. We believe that our matrix enables development and implementation teams to monitor clinical pathways over time to ensure they are achieving their intended effects at multiple levels, including the patient, provider, and system levels. This comprehensive evaluation warrants that clinical pathways align with their objectives and deliver value across the healthcare systems.

Further research will be necessary to determine the real-world impact of implementing this matrix. We aim to investigate whether utilizing the matrix leads to improved clinical pathways and whether it can effectively identify when a pathway is ready for implementation. By doing so, we hope to contribute to the ongoing improvement of clinical care, enhancing patient outcomes, provider satisfaction, and the efficiency of healthcare delivery.