The PHIT implementation program is complemented by a systematic evaluation designed to 1) measure the population health impact of the district-based health systems strengthening intervention; 2) assess the total cost per capita of the health system in the intervention area; 3) study the efficacy of component interventions, including targeted instrumental support to health facilities, the MESH quality improvement program, strengthening of M&E and data quality systems, and the enhanced community health program; and 4) conduct operational research on the implementation of the program (Table 4).
Overall impact evaluation
The impact evaluation includes four major components: an analysis of the difference in differences in key population health indicators between 2010 and 2015; assessment of trends in routinely collected indicators from the national HMIS; assessment of changes in the causes of death in children measured through verbal autopsies; and analysis of contextual factors that may impact district-level differences.
Analysis of DHS data
We are measuring temporal changes in coverage, health indicators, and contextual factors through the routine DHS conducted in 2010 and 2015, and comparing these differences in the intervention area to those in other rural areas in Rwanda. The DHS are standardized household surveys that use a cluster sampling strategy to generate representative population, health, and nutrition indicators at the national and sub-national (DHS Region) level for monitoring and evaluation activities [18].
In both the baseline and final survey, we oversample our intervention area using national DHS methodology [19] to collect an adequately powered sample to estimate difference-in-differences in under-5 mortality and other PHIT collaborative core indicators between the intervention and rural control areas from 2010 to 2015. We will conduct a hierarchical regression analysis that includes the PHIT intervention as one of multiple province- and district-level variables expected to impact under-5 mortality.
The DHS data will also provide information on changes within the districts on other key indicators of population health, including stunting, wasting, and total fertility rate. The data are also the source for information on health access indicators: contraceptive prevalence, unmet need for family planning, antenatal care utilization, treatment of childhood illness, and immunization coverage.
Trends in facility and community health HMIS data
Analysis of routinely collected HMIS data will complement the DHS analysis in two ways. First, comparison of key indicators assessed through DHS and HMIS may mitigate the potential heterogeneity of DHS data when used for sub-national analysis [20]. Second, HMIS data will be used to assess PHIT collaborative indicators not collected by the DHS.
Data from the community health center and district hospital levels are aggregated and submitted monthly by districts to a national HMIS database. We will analyze trends in service delivery and coverage at facility and community levels, comparing key HMIS-derived indicators before, during, and after the start of the intervention. HMIS data will also be used to measure trends in service volume and selected process indicators not captured in DHS, comparing intervention area facilities with those in the rural control area.
Verbal autopsy
The MOH has initiated a national program of routine verbal autopsy (VA) for all maternal and child deaths and is currently implementing this approach nationally, including in the intervention area. The PHIT intervention supports this effort by developing and implementing trainings in VA, and by introducing innovative methods for analyses of VA data. VA will allow for more precise analysis of trends in under-5 deaths and gain a better understanding of under-5 mortality data, which may have important programming implications.
Exogenous and contextual factors
In order to control for potential external confounders, we are assessing exogenous factors expected to have an impact on population health and selected service delivery indicators. We are working with the relevant ministries to develop a database to track such catastrophic events as epidemics and humanitarian crises. Other relevant contextual factors will include sanitation, equity, women’s education, and HIV prevalence, all of which are available through the DHS. These exogenous factors will be included as potential confounders in our regression modeling of district level health outcomes.
Economic and costing analysis
The main purpose of the economic analysis is to understand how the PHIT program contributes to local health system financing. The evaluation measures 1) the costs per capita of PHIT investment in each of the six WHO health system building blocks; 2) total cost per capita of the health system in the two intervention districts; 3) health facility spending by WHO building block; and 4) financial contributions to the local health system made by government, the PHIT Partnership, other partners, and patients. We have designed survey instruments for each facility type to measure funding sources, facility expenditure, and existing capital. Cost data have been collected for the baseline year (2010) and the surveys will be repeated annually through 2014.
We will conduct trend analyses in key variables over time, including the level and trends of total health system and PHIT spending per capita in the two districts; percentage of total health expenditures contributed by PHIT, government investment, and other sources; and the level and trends of expenditure on each of the six WHO building blocks by funding source. We will use regression analysis to evaluate the impact of PHIT spending on health care provision using the provision of child health services as the outcome variable.
Component evaluations of the PHIT intervention
Operational evaluations will focus on key components of the PHIT intervention — targeted instrumental support to health facility building blocks, quality improvement through the MESH program and strengthened M&E systems, and enhancements to the community health system. These evaluations will serve two purposes: use of the data for modification and improvement of the interventions throughout the implementation period and exploration of the relative contributions of each component intervention on health outcomes to inform replication and scale-up of the Rwanda PHIT intervention.
Health facility instrumental support
As described above, a standardized health facility survey has been developed and administered before the initiation of the health center instrumental support, with plans to re-administer quarterly. Changes in domain scores and overall facility scores will be tracked and compared over time. Differences in achieving fully capacitated and high-quality systems across the sites will be explored to identify factors associated with more robust or rapid success and to identify areas where the strategy has been less effective. Multivariate models will adjust for potential confounders including site size, duration of PIH support, and remoteness of health facilities.
Mentorship and enhanced supervision (MESH) program evaluation
The evaluation of the MESH program will measure the change in quality of care delivered at the health center level. The main evaluation focuses on quality of child health care delivered by MESH-supported nurses and the primary outcome is the integrated management of childhood illness (IMCI) integrated assessment index, derived from a checklist of protocol-defined assessment tasks that should be performed for all children [21]. Secondary outcomes include individual assessment components, classification, treatment, and IMCI coverage indicators.
Assessment of data quality and utilization
Because the validity of analyses using routinely reported data depends on the quality of the data collected, we are studying the reliability of CHW data using Lot Quality Assurance Sampling (LQAS). LQAS allows for rapid classification of CHW reports as having high or low quality, defined as concordance between reports aggregated at the health center level and the original registries. This approach will provide a rapid and inexpensive methodology to target data quality improvement efforts and measure their impact [22].
We will also measure changes in data quality of standard health facility reports. Work is ongoing to further develop mixed method evaluation of change in capacity for data interpretation, utilization of resources to address identified gaps, self-reported knowledge in data management, and interpretation.
Assessment of the strengthened network of community health workers
Using routinely collected data from monthly community HMIS reports, we will assess health indicators from intervention area sectors compared to other rural sectors. In a second approach, we will leverage an ongoing World Bank-sponsored study that randomized 200 community health cooperatives (all CHWs in a health center catchment from one cooperative) to one of four different incentive models by applying its results to our interventions. We will, thus, pool data from the intervention area cooperatives and compare them to the non-PHIT cooperatives adjusting for the intervention arm assigned.
Research capacity building through operational research
In order to address a critical skills gap in Rwanda, an operational research capacity building effort has been embedded within the PHIT evaluation plan. Led by National University of Rwanda School of Public Health and Harvard Medical School, we launched an MPhil and PhD research training program, known as Rwanda PHIT Scholars, for 10 Rwandan health professionals involved in the PHIT Partnership. Candidates underwent a competitive selection process that involved the development of operational research proposals to complement the PHIT impact evaluation and intervention component evaluations. Each student has been assigned an advisory team, including a NUR-SPH advisor, an HMS faculty advisor, and a PHIT Rwanda site supervisor.
Among the student-led operational projects are assessments of family planning uptake, performance of the electronic medical record (EMR) currently under development by the PHIT Partnership, the impact of Rwanda’s e-health architecture, the impact of HIV care delivery on maternal and child health services, the impact of the PHIT intervention on childhood nutritional status and health facility staff retention, and ethnographic assessment of family planning utilization.