Successful implementation of a combined learning collaborative and mentoring intervention to improve neonatal quality of care in rural Rwanda

Abstract

Background

Globally, neonatal mortality remains high despite interventions known to reduce neonatal deaths. The All Babies Count (ABC) initiative was a comprehensive health systems strengthening intervention designed by Partners In Health in collaboration with the Rwanda Ministry of Health to improve neonatal care in rural public facilities. ABC included provision of training, essential equipment, and a quality improvement (QI) initiative which combined clinical and QI mentorship within a learning collaborative. We describe ABC implementation outcomes, including development of a QI change package.

Methods

ABC was implemented over 18 months from 2013 to 2015 in two Rwandan districts of Kirehe and Southern Kayonza, serving approximately 500,000 people with 24 nurse-led health centers and 2 district hospitals. A process evaluation of ABC implementation and its impact on healthcare worker (HCW) attitudes and QI practice was done using program documents, standardized surveys and focus groups with facility QI team members attending ABC Learning Sessions. The Change Package was developed using mixed methods to identify projects with significant change according to quantitative indicators and qualitative feedback obtained during focus group discussions. Outcome measures included ABC implementation process measures, HCW-reported impact on attitudes and practice of QI, and resulting change package developed for antenatal care, delivery management and postnatal care.

Results

ABC was implemented across all 26 facilities with an average of 0.76 mentorship visits/facility/month and 118 tested QI change ideas. HCWs reported a reduction in barriers to quality care delivery related to training (p = 0.018); increased QI capacity (knowledge 37 to 89%, p <  0.001); confidence (47 to 89%, p <  0.001), QI leadership (59 to 91%, p <  0.001); and peer-to-peer learning (37 to 66%, p = 0.024). The final change package included 46 change ideas. Themes associated with higher impact changes included provision of mentorship and facility readiness support through equipment provision.

Conclusions

ABC provides a feasible model of an integrated approach to QI in rural Rwanda. This model resulted in increases in HCW and facility capacity to design and implement effective QI projects and facilitated peer-to-peer learning. ABC and the change package are being scaled to accelerate improvement in neonatal outcomes.

Background

Despite declines in under-five mortality in Rwanda and globally, neonatal mortality remains high [1]. Contributors to neonatal mortality include poor coverage and quality delivery of evidence-based interventions known to improve maternal and newborn outcomes [2]. To date, approaches to improve delivery of these interventions have included improving individual provider skills through mentoring and skills-training; health systems strengthening to ensure availability of essential equipment; and quality improvement (QI) interventions, such as learning collaboratives, to drive system change [3, 4].

A learning collaborative brings together QI teams from healthcare facilities to seek improvement in a focused topic area through QI methods and peer learning [5]. Collaboratives often create change packages, collections of high impact change ideas tested during a collaborative, which are used to accelerate quality improvement by spreading potential solutions to common challenges [6,7,8,9,10,11]. The change package includes change concepts, grouping of similar change ideas into a broader conceptual categories [12].

Building on previous success of mentoring and learning collaboratives [11, 13], the Rwanda Ministry of Health (MOH) in partnership with Partners In Health (PIH), a US-based non-governmental organization, designed and implemented the All Babies Count Initiative (ABC) in 2013. ABC was an 18-month district-wide QI learning collaborative and mentorship effort designed to reduce neonatal mortality by improving health system performance and individual provider behavior to prevent newborn death.

ABC was implemented in two rural Rwandan districts which have been supported by the MOH-PIH partnership since 2005, with additional work starting in 2009 through a health systems strengthening project funded through the Doris Duke Charitable Foundation Africa Health Initiative [14]. The two districts serve a catchment population of approximately half a million and included 24 nurse-led health centers and two district hospitals at the start of ABC [15].

This paper reviews the implementation process and implementation outcomes of the ABC initiative including feasibility and fidelity, acceptability, self-reported changes in health care worker (HCW) attitudes and practice of QI, QI project implementation and the resulting change package.

Methods

The ABC intervention

ABC had three primary components. First, based on routine facility data on infrastructure and supplies, baseline gaps in essential neonatal care medical equipment were identified, and essential equipment was provided to close critical needs [16]. Second, utilizing PIH and MOH staff, ABC provided targeted clinical skills training on national neonatal care protocols which included essential newborn care, neonatal resuscitation (using Helping Babies Breathe); and advanced neonatal care for hospital staff. Third, these components were supported by a district-wide 18-month Learning Collaborative organized by ABC mentors with integrated onsite clinical and QI mentorship to improve HCW skills and support system-level QI projects in neonatal and maternal care. Additional program description details can be found elsewhere [17].

The learning collaboratives included four learning sessions during the active collaborative and a final “Harvest Session”. Between the learning sessions, facilities were visited monthly by an ABC mentor who provided individual clinical and team QI mentorship to support improvement in clinical care delivery and in QI project implementation to address identified gaps in the core processes of maternal/newborn health [antenatal care (ANC), delivery management, and postnatal care (PNC)]. ABC mentors were experienced nurses in neonatal care delivery with additional training PIH Pediatric Program Director in QI methods, learning collaborative facilitation, data collection and analysis and coaching. To facilitate sustainability, mentoring visits were conducted jointly with existing, district-based MOH mentors whenever possible.

QI teams were formed at each facility, composed of 3–4 strategic team members, usually including a maternity, antenatal, or neonatal charge nurse, the community health officer, and the facility data manager. QI teams were supported to review their performance in eight core indicators and develop QI projects based on the results. QI projects were designed to test specific interventions (change ideas), using standard “Plan, Do, Study, Act” (PDSA) cycles based on the Model for Improvement [18].

Core learning collaborative indicators were chosen through a combination of literature review, expert consultation and review of data available in the Rwandan National Health Management Information System (HMIS) (Table 1).

Table 1 ABC Learning Collaborative Core Indicators

Core indicators included those available from routinely reported data (HMIS) and ones which required review of facility registers (non-HMIS). Non-HMIS indicators were extracted from registers by the mentors and facility QI teams. HMIS indicators were extracted from existing reports with data validation done with facility QI teams by comparing HMIS reports with facility registers. Indicators were collected retrospectively at baseline (minimum 3 months prior to QI project start for non-HMIS indicators and 6 months for HMIS indicators) and monthly during the 18-month implementation. Data were entered into excel databases for review with facility teams, programmatic monitoring and QI activity tracking. A change package for the three care domains of ABC was created at the end of the collaborative based on quantitative and qualitative data collected from the QI projects and participants.

Data sources

Collaborative implementation

Implementation data were collected using routine program monitoring tools. These data included: learning session attendance, duration, and frequency; mentor visit content and frequency; and QI team characteristics.

Healthcare worker attitudes and practice of quality improvement

Participant surveys were completed before the first learning session and after the Harvest Session with shorter surveys before other learning sessions. The surveys were adapted from previously used surveys in other quality improvement programs. They included answers on a 5 item Likert scale and open-ended questions used for immediate programmatic management. The scaled questions were included in this analysis. Questions included: confidence in QI methods, leadership, teamwork, data use, peer-to-peer learning, motivation, and work environment using a Likert scale as well as presence of barriers to care such as knowledge, time, and resources.

Change package

Change ideas generated and tested by facility teams were captured by mentors using a QI tracking tool. Quantitative pre-post mean analysis of change in the targeted measure was used to identify successful change ideas for inclusion in the change package. During the Harvest Session, each team was asked to rank all tested change ideas using a priority matrix which included: potential impact on health outcomes; volume of patients affected; time to impact; feasibility; and level of other support required. Results of the quantitative analysis and qualitative ranking were used to generate district-wide lists of higher and lower ranked change ideas in each care domain to guide focus group discussions.

During the Harvest Session, one focus group discussion was conducted on each care domain - ANC, PNC and delivery care - with ABC participants in each district for a total of 6 focus group discussions. Discussions were used to capture feedback on the prioritized change ideas and lessons learned to inform future ABC program design and scale. Focus groups were conducted in Kinyarwanda and French using a standardized guide by a focus group facilitator and two note-takers. Follow-up structured debriefs of the facilitator and note takers were conducted by a qualitative expert to extract key themes of the discussions.

Analysis

Collaborative implementation

We assessed the fidelity and completeness of the ABC initiative implementation comparing key activities including mentor visit frequency, site participation and QI activities with the program design. Qualitative data on facilitators and challenges to ABC success were also collected through the focus groups described above.

HCW attitudes and practice of quality improvement

Learning session surveys were entered into a database using EpiInfo version 7.1.5. Individual surveys with < 20% of questions completed were classified as incomplete and excluded. Likert scale questions responses were converted into dichotomous variables (4 or 5 versus < 3). Descriptive statistics were used with significance testing using chi-squared or two-tailed t-tests to measure difference between the baseline (pre-learning session) and endpoint (Harvest Session) for dichotomous and continuous variables respectively. Missing responses to individual questions were excluded and results with a p <  0.05 were considered significant. Quantitative analyses were conducted in Stata v14 (College Station, TX: StataCorp LP).

Change package

Quantitative success of QI projects was defined as significant change (p <  0.05) from the mean facility baseline versus QI project endpoint for the targeted core indicator using two-tailed t-tests.

Qualitative success for QI projects was based on QI team priority matrix rankings of change ideas as noted above and on the thematic coding of the focus group discussions. Both deductive and inductive approaches were used to determine underlying themes. To reduce reporting bias, the codes extracted from the interviews were validated by an expert in ABC implementation (ABC mentor) and by an expert in qualitative analysis.

Qualitative and quantitative data were subsequently integrated to determine which QI projects and change ideas warranted inclusion in the change package using rules summarized in Table 2.

Table 2 Rules for Determining Change Package Inclusion

Results

During the collaboratives, learning sessions were conducted every 3–5 months with an average of 40 participants per session and 98% of all health facilities represented at each session. Each health facility received, on average, 0.76 mentorship visits per month (range 0.6–0.9), slightly less than the planned 1 visit per month. Clinical trainings were implemented on average every 12.5 months with > 92% of facilities achieving the goal of retaining ≥2 trained nurses at the end of ABC (Table 3).

Table 3 ABC Learning Collaborative vs Typical Learning Collaborative Implementation Comparison [7, 10, 17]

HCW attitudes and practice of quality improvement

Self-reported QI capacity increased significantly by the end of the collaborative. This included increase in self-rated QI knowledge (37% vs 89%, p <  0.001), confidence (47% vs 89%, p <  0.001) and leadership (59% vs 91%, p <  0.001) (Table 4). QI team members also reported an increase in being asked for advice to improve neonatal care at the health facility (64 to 93%, p <  0.001).

Table 4 Results from Participant Survey Pre and Post Learning Collaborative^{a, b}

Other significant improvements included QI team member engagement in activities to measure the quality of neonatal care (56 to 87%, p <  0.001) and peer-to-peer learning (36 to 66%, p = 0.024). Surveys showed no significant change in motivation at work or use of routine reports (50 to 59%, p = 0.32).

Reported leadership interest in measuring and improving quality of care increased during ABC (63 to 95%, p <  0.001), although leadership interest in hearing health care worker input on QI remained unchanged (57 to 67% p = 0.2).

In addition to self-reported increases in peer collaboration, 61% of change concepts from the change package were implemented successfully by more than one facility. Focus group discussions revealed that the project also fostered intra-facility collaborations: “through [QI teams] implementing their QI projects, it made more services within the health facilities develop their own projects.” This led to the spread of QI beyond ABC core indicators to lab services, pharmacy, vaccination, family planning and vitamin K administration.

Perception of availability of adequate equipment to provide care and services increased (66 to 84%, p = 0.03). However, although perception of training as a barrier improved (39 to 20% p = 0.018) the top four reported barriers to quality care delivery remained unchanged, including: high patient volume, inadequate staffing, socioeconomic challenges of patients, and staff training.

ABC change package

Facilities initiated a total of 52 QI projects spanning all 3 care domains, testing 118 change ideas. Of the ideas tested, 63% were in ANC, 27% were in delivery management, and 10% were in PNC (Table 5). Six of the eight core indicators were addressed. No projects explicitly address asphyxia (which was covered during clinical trainings) and due to changes in documentation, QI projects promoting placing newborns skin-to-skin with mothers immediately after delivery which were implemented early in ABC were not captured by mentors in the QI tracking templates. Forty-six (38.9%) change ideas were determined to be high impact through quantitative or qualitative criteria and were summarized into 17 change concepts for the change package (Table 6).

Table 5 Total QI Projects, Change Ideas and Care Domain Associated High Impact Themes

Table 6 All Babies Count Change Package

Successful change concepts included improving access to and convenience of ANC services, improving community engagement and awareness of ANC service importance, and leadership engagement. Interventions targeting delivery included improving coordination of care to reduce time to caesarian section, preterm labor management included refresher trainings, collaborations with pharmacy to ensure stock of necessary medications and gestational age calculation training and verification. QI projects focused on PNC included danger sign recognition within 24 h of delivery and return for postnatal check-up after discharge. Successful PNC change concepts included patient register modification to act as a clinical reminder for HCW to check mother-baby dyad and integration of postnatal check-ups into routine vaccination services. Participants identified areas which could not be addressed by the ABC scope such as “retaining patients for 72 hours [for PNC care] was difficult and unsuccessful because of high volume of patients. They don’t have enough rooms, staff, or beds.”

Table 5 describes six general themes of high impact projects identified by focus group participants. One theme was the importance of the integration of services. For example, an intervention to improve ANC visit attendance was reported as working well because it “integrate[d] the tracking of women who tested positive for pregnancy when they came in to other services.” Similarly, PNC service attendance was improved by taking a “strategy to attract more mothers [to post-natal services by getting] them when they came for vaccination clinics.” Projects were noted to be more sustainable when there was integration, “... between health system, local leaders and community health workers.”

Teamwork and communication were also identified as important themes for success across all care domains. “[Teamwork] helps make a project be successful. Working together and learning from each other are what make projects successful.” Leadership support emerged as another key element to high impact project implementation with debriefs mentioning that “Local leadership collaboration allowed for QI projects to be successful.” Integrated clinical and QI mentorship were also identified as important facilitators of high impact QI projects. “The mentors would come to health facilities while they worked and correct them on the job. They would see where they needed more tools or supplies…this would keep the participants active and working on their QI projects.”

Clinical training where the “staff gained skills through (ABC) to implement QI projects” on key practices between or during learning sessions was also reported as valuable. Availability of essential equipment was also noted to be key to the success, as “equipment and materials were a problem at in the beginning of the QI projects but then they received the materials from ABC”. Finally, participants reported the importance of data utilization to QI efforts “[the ability to do] data monitoring in order to track their progress in implementation” was key to the execution of high impact projects.

Discussion

The ABC initiative successfully integrated clinical and QI mentorship with a learning collaborative and targeted training and equipment support designed to improve newborn care in district health care facilities in rural Rwanda. ABC was implemented with strong fidelity and acceptability to participants, resulting in wide-spread QI activities at the health center and district hospital levels. The initiative also increased district-wide QI capacity and peer-to-peer learning. The program success was accomplished by engaging individuals and teams through mentorship and learning sessions to improve individuals’ and system’s ability to create, implement and spread QI projects across the district.

Our finding that a multi-faceted intervention centered around a learning collaborative was an effective model in building QI capacity and practice improvement was similar to Franco’s findings from a review of learning collaboratives in resource limited settings [5]. Quality delivery of evidence-based interventions is dependent on healthcare worker skills and availability of supplies, especially in resource limited settings [19,20,21]. However, studies have found that addressing facility readiness in isolation does not lead to improved delivery of high quality care. ABC’s explicit attention to essential equipment at the start of the project and baseline training in essential care practices to address potential barriers of knowledge and skills was identified as contributing to the successful implementation and acceptability of ABC in addition to the ongoing mentoring and QI. These findings emphasize the importance of comprehensive interventions, such as ABC, which have the ability to address facility readiness as well as continuous learning through on-going training and effective supportive supervision or mentoring [22,23,24,25,26].

The success of ABC in building QI capacity was evident in the reported increase in QI confidence and the range and scope of change ideas implemented. Additionally, work to increase data use for performance measurement, a key step of PDSA cycles, was emphasized in some of the change ideas. For example, maternity and neonatal registers were compared to identify cases of inadequate preterm labor screening, and stock checks for essential medications were integrated into routine activities.

Change packages generated through collaboratives are an important product for spreading change [27]. However, there is little consistency or description of how change packages are created [6] with previous collaboratives in LMIC settings describing a range of methods. Project Fives Alive! in Ghana took a strictly quantitative approach using run chart rules [28]. Others, such as projects supported by the USAID ASSIST [12, 29, 30] describe a more mixed-methods approach to change package creation. We chose a mixed-methods approach to incorporate front line HCWs and identify projects which may not be seen through quantitative analysis, but which were feasible, high impact and applicable to the setting. This addition of qualitative data was particularly relevant when events were rare so statistical significance was unlikely to be reached, yet had potentially critical impact on patient outcomes (i.e. antibiotics for prolonged rupture of membranes), or the number of data points too few to use run charts or statistical process control charts, other methods commonly used in QI [31].

Our study had a number of limitations. Changes in capacity and peer-to-peer learning were solely based on HCW self-report. We also could not analyze change at the individual level since some providers’ attending of the learning sessions changes over time due to staff turnover. We also did not have any counterfactual evidence, so could not prove that changes in QI capacity and activities were due to the ABC intervention nor which components of ABC were most important in driving those changes. However, we do know that no other independent neonatal-focused or other QI initiatives were newly active in the districts during ABC.

The quality of the data and documentation likely improved over the course of ABC implementation, potentially contributing to some of the measured improvement associated with change ideas. We also did not have sufficient data subgroups at the change idea level for the application of run chart rules. Therefore, we used pre-post means of project level measures and included qualitative assessment of change ideas. Impact on quality and cost-effectiveness are not included and will be published separately.

Conclusion

Incorporating clinical mentorship and facility readiness support in the targeted areas into district-wide learning collaboratives was a feasible and effective strategy to support the development of a culture and capacity for QI in rural districts in Rwanda. Including evaluation of implementation contributed to the ability to proceed with scale-up of the intervention including timely application of the change package in Rwanda. Final analysis of ABC impact on care outcomes and sustainability one year post-ABC is underway.