Approaches to ensuring and improving quality in the context of health system strengthening: a cross-site analysis of the five African Health Initiative Partnership programs

Reduction in population morbidity and mortality can only be realized through increasing not just access to needed services, but also the quality of services provided. Quality health services are defined as “effective, safe, centered on the patients needs and given in a timely fashion [1].” As work continues to focus on strengthening health systems, understanding how to best measure and improve quality is critical. The World Health Organization (WHO) focuses on quality as a central component of their framework on health system strengthening (HSS) and a key driver to ensuring that work to strengthen systems translates to improvement in health (Figure 1). Ensuring that quality is incorporated as a critical component of each of the six health system building blocks (service delivery; health workforce; health information system; medical products, vaccines and technologies; health financing; and leadership and governance), either implicitly or explicitly, will be essential to achieving potential health impact.

Other organizations and programs have highlighted the role and relevance of quality in both HSS and in achieving the overall goal of improved health. The United States Agency for International Development (USAID)-funded Health Systems Strengthening II project includes quality of care as one of its key components, with defined elements of quality going beyond care to include other key building blocks of the health system. These include improved planning, presence of supervision and management, supply chain, and strengthened referral systems [2]. Work to strengthen knowledge management, including data quality for evidence-based decisions and identification of gaps has also been identified as a core quality area in the context of HSS [3].

Despite the call for increased HSS efforts and the growing literature on quality improvement, there has been less written about feasible and effective approaches to measuring and improving quality in the context of HSS, particularly in areas outside of clinical care, data management, and supply chain. To help with measurement, a critical step for quality improvement, WHO provides a framework for monitoring and evaluating HSS efforts, which includes quality indicators for each building block [7]. However, there is not much known about the optimal set of quality measures to ensure that the strengthened systems and their associated services will be effective in improving population health.

In 2009, the Doris Duke Charitable Foundation’s African Heath Initiative (AHI) funded Population Health Implementation and Training (PHIT) Partnerships in five countries (Ghana, Mozambique, Rwanda, Tanzania, and Zambia) to improve population health through health system strengthening. While each Partnership developed a different approach to HSS, they were all designed to improve population health through increased quality and utilization of essential primary health care services within the public sector. The PHIT funding supported implementation of the different interventions and rigorous monitoring and evaluation, including measuring indicators of quality. A core set of AHI measures were chosen for tracking across the five sites, complemented by a unique set of measures developed by each Partnership, reflecting the main areas of focus and anticipated outcomes of their interventions.

We describe both the shared and program-specific approaches to measuring and improving quality among PHIT grantees. Understanding how the Partnerships are integrating the measurement and improvement of quality into their HSS efforts will be of value for other programs focused on improving health systems and population health at the district, provincial, or national levels.

The Partnership implementation plans were all focused on the WHO building blocks and are described in more detail in other papers in this supplement [8–12]. Therefore, we use the WHO HSS building blocks framework to explore the Partnerships’ approaches to ensuring quality as an essential outcome required to achieve improvements in population health. The framework was developed based on a review of selected literature on quality in the context of health system strengthening and authors’ experience in this field. Information from each Partnership program was obtained through a number of methods: PHIT project document review, semi-structured interviews with Partnership leads, and feedback during a consultative retreat with the Partnership principal investigators.

Each country Partnership is implementing a set of interventions largely based in the public sector and designed to strengthen health services and improve population health. The study designs are described in detail in related articles.[8–12]. The level of the health system targeted (community, facility, district, province) and planned activities vary across the Partnership. All Partnerships have identified and prioritized quality in their program implementation as both a primary outcome and as a critical step towards achieving health system improvements and public health impact. Work to measure and improve quality is integrated into the work to strengthen health systems and crosses most or all of the WHO building blocks.

The Partnership impact evaluation designs target the measurement of improvements in population health, including under-5 mortality. The designs also measure changes in quality related to the improvement initiatives implemented [8–12]. Measuring the relative importance of improving quality in achieving these population health gains is also integrated into the impact evaluation in most of the Partnerships. For example, in the Rwanda project, the impact evaluation framework is based on a theory of change where quality is a necessary component of each step, from input to processes to outcomes, to achieve improved population health and strengthened health systems. Areas where impacts on improving quality are measured include quality of care provided as well as quality of the health systems targeted for strengthening.

At the start of the AHI initiative, agreement was reached to collect a selected number of core measures of quality across all Partnerships (described in more detail in the paper by Bryce et al. in this supplement) [14]. Two of these – health workers per capita and continuous stocks of essential commodities – reflect WHO guidance on monitoring and evaluation of HSS [7]. Other core measures that capture aspects of quality include specific coverage measures (e.g., intermittent preventive treatment for malaria in pregnancy, skilled attendant at birth, and appropriate management of selected childhood illness). Despite significant discussion and brainstorming, there remain a number of areas where consensus on specific indicators of quality within one or more of the HSS blocks could not be reached. As a consequence, the teams agree to report on their Partnership-specific activities to improve quality of care and measure quality annually.

In each project, quality is defined, measured, and targeted for improvement across many or all of the six WHO HSS building blocks, going well beyond the traditional focus on service quality. The measurement of the impact of these improvement efforts on quality is integrated into each Partnership impact evaluation plan. Data sources in use by the PHIT Partnerships for measuring quality fall into three main categories: 1) use of routinely available data (e.g., facility reports and health management information system (HMIS); 2) data collection tools developed within the intervention and integrated into routine project monitoring (e.g., facility surveys and mentor reports); 3) and measurement solely for interim and summative impact evaluations designed for baseline measurement, mid-course correction, and end-of-project assessment.

Despite the heterogeneity of interventions and independence of evaluation designs, there is significant overlap in approaches targeting the definition of quality (Table 3). While some of these measures overlap with WHO recommendations on measuring components of quality, others reflect the specific pathways through which each Partnership is working to strengthen the health system and improve population health.

A major theme across the Partnerships is the increase of service quality through strengthening data utilization for evidence-based decision making, as well as the identification and addressing of gaps and allocation of resources more efficiently (Table 4). Work includes efforts such as ensuring strong feedback loops to increase capacity of end-users for data analysis, interpretation, and communication. However, every project also recognizes the need to ensure the robustness of HMIS data as a core component in this approach to generating and ensuring quality.

As Leatherman points out, QI can serve to decrease the “gap between actual and achievable practice” for service delivery and to enhance “individual performance, satisfaction and retention” for health workers [6]. Every project includes a component of supportive supervision or mentoring as a means to directly bridge that gap while building capacity among staff at the health facility, district, or province level to better respond to identified gaps.

In Rwanda, the project includes a health center nurse mentoring and supervision program, which provides direct feedback through observation of care in four main areas: outpatient under-5 care; outpatient adult care; HIV and TB care; and maternal health care, focusing on antenatal, labor, and delivery services [10]. The nurse-mentor also provides coaching to the care teams at the health centers to identify and address gaps in quality using systems-based quality QI methodology. Program managers are also given training and mentoring in QI methodology to drive improvement in other areas including inpatient and specialty care and support systems (e.g., pharmacy, information systems etc.). In Zambia, onsite clinical mentoring of health care workers is among the core essentials of the intervention. A team of qualified clinicians uses automatically generated performance indictors to support data utilization. This, in turn, improves the quality of provider-delivered care [8].

The Mozambique Partnership focuses on increasing supervision quality and frequency to better integrate and improve service management at the facility, district and provincial levels. Steps to improve supervision quality include the introduction of a data collection guide for supervision visits to ensure that data quality weaknesses and primary health care utilization gaps are identified. Together with additional data summary tools, these results are discussed with district and facility managers to identify priority problems and define an action plan which is also used for follow-up in subsequent supervision visits.

The Tanzania Partnership links each CHA with both a clinical supervisor based at the health center and a community-based supervisor. Supervisors meet with the CHAs regularly to ensure that CHA responsibilities are met and supported by a strong feedback loop of both activity and interim evaluation data on activities. The process of supervision is improved through feedback of results from interviews with CHA and checklists to measure the quality and frequency of this activity. The results are used to guide the project on planning, management, supervisory decisions, and steps needed to address gaps in the quality of community-based primary healthcare and in the supervision. In the Ghana Partnership, supervisor training is focused on improving supervision capacity. Relevant data are also fed back to the region and then to the districts or sub-districts to further strengthen supervision, ensuring that the district team lead is an important part of the monitoring and supervision processes.

The Partnerships incorporated operational research into the program design to measure and provide insights on the level of quality in the building blocks, the factors associated with variations of quality, and the impact of this variability in achieving population health goals. In Ghana, pilot research is used to guide implementation activities. For instance, a referral scheme to improve maternal and child health is being implemented in one sub-district in one of the intervention districts. Lessons learned from this pilot are guiding scale up efforts in different districts. In Mozambique and Rwanda, this work is combined with in-country research capacity building efforts. In Mozambique, applied research methodologies are used to understand and test innovations to overcome system bottlenecks. The Mozambique Partnership provides financial and technical support to the Ministry of Health’s Beira Operations Research Center (CIOB), as well as masters-level public health training, to build sustainable applied research capacity [9, 10]. In Rwanda, scholarships are provided to field- or MOH headquarter-based implementers to pursue advanced degrees while completing research projects. Research project address factors associated with differences in the quality and the effectiveness of approaches to improve quality of care, services, human resources, and other components of HSS.

While the PHIT initiative is focused on HSS to improve population health, all projects focus on quality across the WHO building blocks as a key outcome and necessary step to achieving the longer term impact. Each of the Partnerships uses different approaches and theoretical frameworks to strengthen health systems and ensure quality; however, they all include work on measuring quality, improving data quality and building capacity of end users so that results identify and address gaps in quality.

Many of the efforts to measure quality are integrated into routine work and monitoring and evaluation rather than existing as separate data collection efforts for performance measurement or evaluation. As a result, all the Partnerships recognize the need and challenge of ensuring that these data are of adequate quality – a priority reflected in the work of building systems and capacity for data quality. In addition, using these data for evidence-based decision making is used as a strategy to strengthen other building blocks, most prominently in leadership and governance. The approach to improve identified gaps in quality varied across Partnership projects, but all recognize the need for strong, supportive supervision, and mentoring. This includes direct mentoring of care providers, mentoring and training of managers to respond to system gaps – including resource allocation – and ongoing adaptations of the interventions to increase quality and in turn improve expected impact. Comparing the approaches with the Leatherman description of the role of QI in HSS, the cross-building block- QI approach to strengthen health systems captures almost all of the potential benefits regardless of the program analyzed [6]. The Partnerships measures capture these areas of improvement as well as additional areas within the building blocks identified by the projects as critical to achieving a quality health system able to improve population health. A challenge for the PHIT Partnerships is to identify areas of common intervention and measurement where lessons learned can be shared and approaches compared within the constraints of the different intervention models and settings. Work to develop focused cross-Partnership evaluations in areas such as mentoring and strengthening data utilization are under discussion.

In conclusion, measuring and ensuring quality beyond the health care delivery building block is identified by all PHIT Partnerships as core to success, with an integrated approach to ensuring a robust feedback loop for improvement. Projects are prioritizing different areas and levels for this work but core themes include a focus on data quality and building capacity to use these data to identify gaps and improve quality more effectively and efficiently. The impact evaluation in each Partnership is designed to capture improvement in quality across multiple HSS building blocks in the context of the implementation plans. Results from the operational research and impact evaluations planned by each Partnership will help increase the understanding of the effectiveness of HSS efforts on improving quality in care delivery and such other critical areas as leadership and health information systems. The Partnership projects will also contribute to the documentation of the relative importance of these improvements on measured changes in population health. The challenge for each country project will be to identify the optimal approaches to improving and maintaining quality within each of the building blocks in order to achieve an effective, strengthened and sustainable health system.