Skip to main content

Estimating population-based coverage of reproductive, maternal, newborn, and child health (RMNCH) interventions from health management information systems: a comprehensive review

Abstract

Background

Routinely collected health facility data usually captured and stored in Health Management Information Systems (HMIS) are potential sources of data for frequent and local disaggregated estimation of the coverage of reproductive, maternal, newborn, and child health interventions (RMNCH), but have been under-utilized due to concerns over data quality. We reviewed methods for estimation of national or subnational coverage of RMNCH interventions using HMIS data exclusively or in conjunction with survey data from low- and middle-income countries (LMICs).

Methods

We conducted a comprehensive review of studies indexed in PubMed and Scopus to identify potential papers based on predefined search terms. Two reviewers screened the papers using defined inclusion and exclusion criteria. Following sequences of title, abstract and full paper reviews, we retained 18 relevant papers.

Results

12 papers used only HMIS data and 6 used both HMIS and survey data. There is enormous lack of standards in the existing methods for estimating RMNCH intervention coverage; all appearing to be highly author dependent. The denominators for coverage measures were estimated using census, non-census and combined projection-based methods. No satisfactory methods were found for treatment-based coverage indicators for which the estimation of target population requires the population prevalence of underlying conditions. The estimates of numerators for the coverage measures were obtained from the count of users or visits and in some cases correction for completeness of reporting in the HMIS following an assessment of data quality.

Conclusions

Standard methods for correcting numerators from HMIS data for accurate estimation of coverage of RMNCH interventions are needed to expand the use of these data. More research and investments are required to improve denominators for health facility-derived statistics. Improvement in routine data quality and analytical methods would allow for timely estimation of RMNCH intervention coverage at the national and subnational levels.

Background

Many countries have made significant progress in increasing the coverage and quality of maternal, newborn and child health services during the era of Millennium Development Goals (MDGs) and this is also true for countries that did not achieve the MDG 4 and 5 [1, 2]. As low- and middle-income countries (LMICs) strive to maintain the gains and progress towards achieving the Sustainable Development Goal 3 (SDG 3), there is a need to use rigorous analytical methods to analyze readily available routine health facility data to track coverage of key health indicators both at the national and subnational levels.

The coverage of a reproductive, maternal, newborn, and child health (RMNCH) intervention is defined as the proportion of the population in need of the intervention or service that actually receives it[3]. The percentage of the target population that received the defined RMNCH intervention (RMNCH intervention coverage) is a crucial measure in public health[4]. At the global and country level, RMNCH intervention coverage indicators are used to monitor progress, identify coverage gaps, allocate resources, plan future interventions and guide health policies[5]. Additionally, RMNCH intervention coverage indicators are also used to determine countries eligibility for global support programs such as performance-based financing[6,7,8], support from the GAVI Alliance for the introduction of new vaccines [9], the Millennium Challenge Account assistance[10] and other international support programs[11].

The number of individuals who received the RMNCH intervention (numerator) can be used to examine trends over time and difference between subnational units, but the estimation of coverage requires a denominator. In most LMICs, RMNCH intervention coverage based on health management information systems (HMIS) data is estimated using aggregate routine health facility reports and the estimated target population from census projections[4]. When a person visits a health facility for routine RMNCH care, the health worker records the service provided in their clinic register (facility-held record), a tally sheet and often also on a home-based record that the individual keeps with them[12]. Health workers at each health facility compile data manually at the end of each month from the facility registers and tally sheets, using standardized reporting forms which are forwarded to their subnational unit (such as district) health department. The latter office enters the data into the computer and a web-based system (HMIS) such as DHIS2 which is used to check data quality, produce scorecards, and transfer data to regional and national Ministry of Health.

The recording, transcription, compilation and reporting of RMNCH data from health facilities is highly prone to errors and misreporting[12,13,14,15,16,17,18]. Reports based on health facility data may underestimate population level intervention coverage due to incomplete reporting from reporting facilities or non-reporting from other RMNCH intervention providers (e.g. private sector, non-governmental organizations)[12, 19]. Similarly, reports may overestimate coverage due to overreporting from facilities (due to inclusion of people outside the target group for the intervention) or movement of people across administrative boundaries (districts, regions)[19, 20]. The further use of RMNCH intervention coverage to determine eligibility for international support also promotes inflation of administrative coverage estimates[21]. For some RMNCH interventions such as immunization, doses distributed (not the actual number of people receiving the intervention) are sometimes used as the numerator which does not take wastage into account[20, 22,23,24].

In addition to numerator challenges, denominators derived from census and civil registration and vital statistics (CRVS) are often grossly inaccurate[14]. The availability of census data from multiple time points together with accurate CRVS data is necessary for accurate population projections. However, the availability of these data are limited particularly in Sub-Saharan Africa and Asia. Globally, it is estimated that only 59 % of infants younger than 12 months have their births registered and only 33 % in Sub-Saharan Africa and Asia[25]. Globally, 53 % of deaths are not registered and the proportion of unregistered deaths is likely to be much higher in LMICs[25]. Additionally, there is limited information on migration in LMICs making it impossible to adjust for that in estimating target population for the various RMNCH interventions.

The availability of recent and appropriate population level census data is rare in many LMICs[26]. For example, the Democratic Republic of Congo[27], Ethiopia[28], Burkina Faso[29], and Afghanistan[30] held their latest population census in 1984, 2007, 2006 and 1979, respectively. In other countries like Nigeria, not only is the census data old, but the accuracy is highly contested[31, 32]. Given these challenges, the population projection for key RMNCH indicators using census data in LMICs is problematic. When the denominator for a defined intervention is overestimated or under-estimated, the resulting coverage becomes unexpectedly low or high. It is common to see estimated RMNCH indicator coverage from LMICs exceeding 100 %, partly as a result of incorrectly estimated target population (denominator too low) or numerator (too high)[5, 26, 33].

Given the limitations on coverage estimates derived from health facility data, household surveys are considered the ‘gold standard’ for the measurement of RMNCH intervention coverage and are also used to validate coverage estimations from health facility data[3]. However, household surveys have their own biases and limitations[20]. They are infrequent, expensive, time and resource intensive, and are generally not powered for finer subnational coverage estimates limiting their usefulness[3, 5, 20]. They are limited in measuring some conditions (e.g. pneumonia in children) and their treatment coverage, and are also affected by inaccuracies in the measurement of several maternal and newborn health coverage indicators[34,35,36,37].

Policy makers and public health officials across LMICs rely on RMNCH intervention coverage estimates from routine data to timely identify coverage gaps. These data are also used to ascertain the effects of unexpected events on service use such as the current Covid-19 pandemic. It is becoming increasingly necessary to identify reliable analytical methods for measuring the coverage of RMNCH interventions using HMIS data. Inaccurate estimation of the numerator and/or denominator will result in wrong coverage estimates, misinform national and local health policies and result in wrongful allocation of scarce resources. There is limited information on the most up-to-date available analytical techniques that are used to address the limitations of HMIS data and to improve the estimate of RMNCH intervention coverage using HMIS data.

In this paper, we carried out a comprehensive review of the literature to identify approaches that have been used for estimation of national or subnational coverage of RMNCH interventions using HMIS data exclusively or in conjunction with survey data in LMICs. We describe these methods and highlight their strengths and limitations. This information will be useful to researchers, ministries of health and local health departments in LMICs who continuously strive to improve the accuracy of RMNCH intervention coverage estimates using their local HMIS data. To the best of our knowledge, this is the first comprehensive review focusing on analytical methods to improve estimation of RMNCH intervention coverage using routine data.

Methods

We focused on methods used to assess internal and external consistency of HMIS data for the estimation of RMNCH intervention coverage, together with ways of handling incomplete reporting, estimating the target population, and handling of data inconsistencies. We classified denominator estimation methods into two groups: census-based projections and non-census-based projections. Census-based projection is the estimation of a target population using population-based projections (e.g. population censuses, CRVS) with or without further adjustments. Non-census-based projections include the use of data from other sources besides census-based projections such as survey data, reports from other indicators or from previous coverage information.

Search strategy

To reduce the potential of missing relevant articles, broad search terms were used. We searched PubMed and Scopus using several search terms (Table 1). Titles and abstracts containing terms such coverage, routine health data or district health management information, national, or subnational that are indexed in PubMed or Scopus were included in the search. All relevant articles published in any language (with titles or abstracts translated in English) prior to September 2019 were reviewed.

Table 1 Search Strategy

Inclusion and exclusion criteria, and review procedure

Original articles were included in the review if they estimated coverage of any RMNCH related indicator such as immunization, antenatal, stillbirth, delivery and postnatal services, family planning services, management of childhood illness, prevention of mother-to-child transmission of HIV, and using routine data alone or together with any survey data. Letters to the editor, editorials, reviews or systematic reviews, meta-analysis, case reports and articles that used administratively reported coverage without indicating how the numerator or the denominator was estimated for the reported coverage were excluded.

The title and abstract of each article were first reviewed, followed by a full text review. Each article was reviewed by two independent reviewers and when discrepancies occurred, the reviewers met to discuss and resolve all disagreements. Covidence software was used to manage the review of the articles[38].

Data extraction

The following information was extracted from each of the articles included in the review: year of publication, indicator for which coverage was estimated, location of study, whether the estimated coverage was at the national or subnational level, whether the coverage estimate was exclusively from routine data or used a combination of information from routine and survey data, estimation of target population (denominator) and numerator. Other key features identified were documented as well. We focused on the methods to adjust for the denominators and numerators, missing data, over and under reporting and the overall coverage estimate. In cases where the study had multiple indicators, only the methods applied to the estimate of RMNCH related indicator coverage were considered.

The coverage estimation methods were divided into two groups based on whether the coverage was for preventive or curative indicator as these typically require different approaches for denominator estimation. Curative-based coverage refers to indicators for which the denominator refers to a target population with a specific disease or condition presented at the clinic. These include, for example, antiretroviral treatment for HIV positive pregnant women for the purpose of preventing mother-to-child transmission or treatment of illnesses in children under five years of age. Preventive-based intervention coverage refers to coverage for interventions that are not related to treatment of an active disease or condition; examples include immunization of children, family planning services, vitamin A distribution, and antenatal and postnatal services.

Results

Figure 1 shows the screening process. The initial search identified 668 articles. Of the 668 articles 621 (93 %) were excluded after title and abstract review because they were not related to RMNCH or coverage estimation. Full review of the remaining 47 articles resulted in the exclusion of 29 (62 %) articles because they did not calculate coverage or they used reported coverage from other sources without providing any additional information on how the numerator and the denominator of the coverage were estimated. Of the 18 articles included in this review, 12 (67 %) used only routine data and 6 (33 %) used both routine and survey data to estimate coverage for RMNCH indicators. Table 2 provides a summary of all studies included in the review.

Fig. 1
figure 1

Flow chart showing the screening process

Table 2 Characteristics of papers identified during the literature search

Estimation of denominators

Seven studies took the denominator as reported by the subnational health administration or the national institute of statistics without providing details on how it was estimated[5, 22, 39,40,41,42,43], 8 studies used the census based estimation methods[23, 24, 33, 44,45,46,47,48], 4 used non-census-based estimation approach[5, 24, 49, 50] and 3 used a combination of census and non-census-based estimation approach depending on the indicator and the available data[24, 51]. Denominator estimation for curative-based interventions were limited to only non-census-based methods[49, 50].

Census-based projections: Census based parameters including population growth rate, crude birth rate and mortality rate were used independently or together to estimate the expected population in need of a defined RMNCH service (denominator) at the subnational level. In census based denominator projections, population based parameters such as growth rate and fertility rate are applied to the most up to data population census figures derived from a single time point to project the size of the population at subsequent time points. Census-based denominator estimation methods were mostly limited to preventive services for which the target population can be generated from population census data. These include, for example, births for ANC visits, facility deliveries or surviving infants for immunization rates. Studies that used population growth rate or crude birth rate to estimate the target population did not indicate whether such rates were national or subnational parameters and did not indicate how the growth rate was estimated[23, 47, 50].

Depending on the RMNCH intervention of interest, several studies used different census-based parameters and approaches to estimate the target population. As described in Table 1 the population growth rate is applied to the population reported from the most recent census to obtain the projected population of the time of interest. The reported proportion of the target population from the most recent census is multiplied by the projected population to determine the target population for the time of interest[24, 44,45,46].

Instead of applying the growth rate to the total population, some authors applied the growth rate to the target population reported from the most recent census (proportion of the total population that falls within the target population). This approach was used by Haddad et al. to estimate the number of children less than 12 months of age for the purpose of estimating DPT3 and measles coverage for year 2002/2003 across multiple districts in Burkina Faso. The authors used the national population growth rate estimated from the 1996 census to project the population of children less than 12 months of age in each district, thus ignoring between district variations in both the growth rate and the target population. The population growth rate and the proportion of the target population were considered constant from when the census was done and this was true for other papers[23, 46, 48]. Dunkle et al. used a similar approach to estimate the coverage of DPT3 and measles in Nigeria but varied the proportion of surviving infants each year using state specific infant mortality (or proportion of surviving infants) data as reported by the state health department[33].

For reproductive health indicators (coverage of ANC attendance, antenatal HIV and syphilis testing, facility deliveries), Delvaux et al. multiplied the estimated crude birth rate from survey data by the census projected population of the subnational unit of interest to determine the number of expected pregnancies[48].

Non-census-based estimation methods: Non-census-based denominator estimation approaches relied on other sources using different techniques to estimate the population in need of the service. Combined and non-census-based denominator estimation methods are described in Table 3 below. Denominators were derived using related RMNCH interventions (e.g. DTP1 and DTP3), reported health facility users, previous coverage information, and combining previous coverage with census projections. This approach was mostly used to estimate denominators at the subnational level.

Table 3 Combined and non-census-based denominator estimation methods

Estimation of numerators

The methods used for the estimation of numerators were very similar across indicators. For most of the RMNCH indicators, the numerator was simply taken from the health facility, district or province as reported number of people who received the service with or without any further quality checks or adjustments. This was done mainly for interventions that are largely delivered at the facility level or for which service statistics are well compiled. Numerators for the same indicators were defined differently by study authors. For immunization coverage, for example, some authors used the counted number of children immunized with the specific antigen of interest as the numerator[39, 40, 44] and others used the number of doses distributed to estimate the numerator[22,23,24]. Some studies did not assess for quality of the reported numerator and for those that did, nothing was done about the identified problems[33, 42, 45, 46]. However, a few studies used different techniques to assess and address the identified quality issues[5, 48].

Assessing and addressing data quality issues in numerators

Table 4 provides details on some of the approaches used at the analytical stage to address data quality challenges. Several studies acknowledged the challenges with using routine data to estimate coverage but did not provide any suggestions on how to handle such challenges at the analytical stage. However, few studies (5/18) used the World Health Organization (WHO) recommended facility data quality review tool[52] together with additional methods to assess and adjust for challenges such as missing/incomplete data, extremely high or low coverage estimates (outliers) and inconsistent reporting. These challenges were addressed differently when estimating coverage at the national and subnational level. Incomplete reporting happens when a facility fails to submit their monthly report to the health administration for a given period. However, incomplete reporting can also occur when a facility provides RMNCH services but is not required or does not send reports to the sub-district health office[5]. Two methods were used to assess for internal consistency of the reported indicator: examining trends of the reported numerator over time or comparing the reported numerator between indicators that are expected to be similar (e.g. ANC1 and DPT1) or monotonic ( e.g. DPT1 vs. DPT3)[5, 41]. There were two main approaches to handling outliers and missing reports.

The first method is to make assumptions about the incomplete/missing data and then adjust for the reported data based on the assumption. The second approach is to define a threshold for missingness and apply different rules for missingness above or below the set threshold. If missingness for a given facility is greater than the threshold, they are removed from the analysis. If missingness is less than the threshold, replace missing reports with an average of the months where data was reported. For outliers, first define an outlier with respect to the reported data (any reported data that is 2 or 3 standard deviations from the mean of all reported data for the period). Then state how outliers will be handled; for example, replace with mean reported value, remove from analysis, etc.

Table 4 Addressing data quality challenges

Estimation of subnational and national RMNCH intervention coverage

In many instances (14/18) the numerator reported by the facility was divided by the estimated denominator (mostly from census projections) to calculate the coverage of the indicator of interest without any further adjustment. However, other authors used a combination of techniques and data sources to estimate coverage for RMNCH indicators. The methods for estimating coverage at the subnational level were slightly different from the methods used for national level coverage estimates. Surveys such as the Demographic and Health Survey and the Multiple Indicator Cluster Survey are powered to estimate coverage at both national and the first administrative level (region/province). As a result, national RMNCH intervention coverage estimation combined routine coverage estimates from the subnational level with survey estimates to produce a national estimate. Subnational level coverage estimates depend mainly on routine facility data and census projections with limited options or availability of finer subnational (district level) survey data.

Jeffery et al. proposed the annealing technique which uses a hybrid estimator, a weighted average of the estimated coverage from administrative and survey data. The hybrid estimator produces standard errors and 95 % confidence bounds around the coverage estimate[47]. Maina et al. used the adjusted numerator and denominators from multiple sources described above together with several assumptions (census projected denominators, denominators from indicators with high coverage and survey projected denominators) to estimate different coverage for each intervention and to compare the estimated coverage across estimated denominators to determine the most plausible coverage estimate[5].

Bhatnagar et al. used an approach similar to the World Health Organization/UNICEF (WHO/UNICEF) Estimates of National Immunization Coverage (WUENIC) approach to estimate national level coverage of immunization. The WUENIC method uses data from multiple sources including administratively reported data, survey data and expert panel input to determine a national estimated coverage for immunization. In this approach, the administrative coverage data from the subnational level are used in the national estimate if there is no coverage estimate from other sources or coverage data that differed by 10 % points from the subnational administratively estimated coverage. Data from any subnational unit (state) that reported over 100 % coverage or had high year-to-year variations in coverage (unexpectedly low or high coverage) were considered inaccurate unless there was a justification from the state as to why there were such variations (vaccine stock out, health workers on strike, etc.). The authors made several other assumptions on the different data quality challenges[43]. To estimate the national immunization coverage, a subnational weighted coverage estimate was calculated by weighting the subnational level coverage with population weights defined as the number of reported births from each subnational unit divided by the total number of reported births in the country.

Discussion

The purpose of this review was to identify the methodologies that have been used for estimation of national or subnational coverage of RMNCH indicators using HMIS data exclusively or in conjunction with survey data from LMICs in order to inform local public health officials on available coverage estimation options. In this review, the methods used to estimate the population in need (target population) and the population who received the intervention (numerator) were different even when the geographic scope and the indicator for which coverage was estimated were the same. Similarly, methods to assess data quality, adjust for incomplete/missing data and handling of data inconsistencies also varied across studies. Almost all reviewed articles acknowledged the challenges of using routine data to estimate RMNCH indicator coverage, particularly at the subnational level. The findings from this review demonstrate that while there may not be a perfect ‘gold standard’ method available to fix the quality issues associated with routine data at the analytical stage, there are existing tools that can be applied to improve the quality of administrative data to ensure that coverage estimates from such data accurately represent the true coverage at the population level.

The current WHO guidelines provide tools on the definition of indicators and how to assess data quality including inconsistencies, missing/incomplete reporting, over/under reporting and how to easily identify errors, especially for reported numerators[53, 54]. However, there are limited guidelines and standard on how to handle such challenges at the analytical stage. This lack of standards leads to varying and subjective assumptions resulting in different coverage estimates. Depending on the challenge, some authors excluded troubling data from the analysis or applied subjective assumptions that may not be realistic and reproducible. For instance, some authors excluded all facilities missing more than 8 monthly reports and made several assumptions on facilities missing less than 9 monthly reports. Additionally, some authors defined outliers as 2 standard deviations while others defined outliers as 3 standard deviations from the mean reported data for the reporting period. Using expert panel input and making assumptions and definitions on outliers limit the ability to appropriately compare immunization coverage across subnational units or trends over time since different year to year assumptions will result in different coverage estimates. Similarly, when subnational units make different assumptions or handle data quality issues differently, the resulting coverage estimate may not be comparable across subnational units. There is the need to standardize how to handle the challenges with routine data at the analytical stage which will allow for appropriate trend analysis and comparison between subnational units.

Notable challenges remain in the definition and estimation of RMNCH indicator numerators. Some authors used the reported dose as the numerator to estimate immunization coverage. Using the number of doses administered does not account for vaccine wastage and may result in over-estimation of the numerator.[3] With indicators like ANC attendance, family planning services, and antiretroviral treatment for HIV positive pregnant women the reported numerator from the facility level is used without any modification[48]. In situations where such services are provided by other non-reporting sources such as drug stores or private entities, using the reported numerators from government facilities could result in gross underestimation of the intervention coverage in the population.

A troubling assumption made by some authors is the idea that any reported RMNCH indicator coverage that exceeds 100 % is a sign of poor data quality. Indeed, some authors excluded all reported subnational coverage that exceeded 100 % (without explanation) from the estimation of RMNCH indicator coverage at the national level. It is acknowledged that routine data may be inflated particularly when high coverage attracts rewards or incentives[20, 21], however, not all high coverage may indicate poor data quality or inflated data. Internal migration (due to conflict, hunger, etc.), providing service to people outside the recommended target population, and mismatch between the distribution of health facilities and administratively defined geographic boundaries (districts, regions) can result in estimated RMNCH intervention coverage exceeding 100 %[19, 24]. In several LMICs, children or pregnant women can receive services such as immunizations from any part of the country. For instance, when a pregnant woman or a child receives immunization or other RMNCH intervention from a district/region outside their area of residence, they are counted in the numerator but not in the denominator of the reporting subnational unit. Health facilities and subnational units do not report or capture the number of people receiving RMNCH related service who are coming from outside the reporting subnational unit. Not reporting or accounting for such individuals would not be a problem if the movement across administrative boundaries was even, which is unlikely. In an ideal situation where the target denominator is correct, and a subnational unit is able to deliver the specific RMNCH intervention to all its target population plus individuals not accounted for in their denominator (people from other administrative areas), the resulting coverage will correctly exceed 100 %. This scenario is applicable to all RMNCH interventions.

Hence, there is the need for future studies to examine the movement of people across subnational units and to determine if the number of individuals accessing RMNCH related services out of a defined area is equivalent to the number coming into the specific subnational unit from other catchment areas to access the same service. This will make it possible to account for the imbalanced movement of people across catchment boundaries and will also help to delineate genuine high coverage from over-reporting or poor data quality related coverage estimates.

There has been some advancement in improving the estimation of denominators for RMNCH indicators using direct census projections or indirect estimation methods. Direct census-based projections will provide a reliable estimate of the target population if the census data is recent, segregated and the projections are done correctly. However, if the population census is old, which is common in LMICs, the projections can be unreliable, particularly at the subnational level, resulting in over- or under-estimation of the target population[5]. Treating growth rate as constant or linear without considering the variation across years is problematic. Similarly, the application of national or regional growth rate, crude birth rate or proportions to a subnational unit to estimate a target population regardless of local events or contexts will result in inaccurate denominator estimates.

The emergence of methods that rely on survey and high coverage indicators to estimate denominators for other RMNCH indicators is promising but also comes with notable limitations. These methods do not directly rely on population census projections and in settings where there is high coverage and reliable data, these methods can produce highly reliable and improved denominators [5]. However, in situations where there is low coverage or poor data quality of the related indicators, this method may not work. Completeness for this approach is defined as submission of report for a given month with less emphasis on the quality of the submitted report. The adjustment factor used by these methods is arbitrarily determined. The use of regional level coverage from surveys to estimate denominators for coverage at the district level does not account for the coverage inequalities across districts.

Denominator estimation for curative-based indicators rely on, and is limited to, individuals who use health facilities. If such groups are systematically different from people who do not access services from health facilities, the estimated coverage will not reflect the true coverage of the intervention in the population of interest.

The available methods are promising but have several limitations. There is the need for the exploration of alternative modern technology-based population estimation methods such as satellite imagery, geo-positioning and mobile phone call records to estimate the population size and migration at the subnational level[26, 55]. These methods are particularly useful in LMICs with large seasonal population migration and where traditional census-based projections may be inaccurate[26, 55]. Satellite imagery methods can be done remotely and can be used to validate or complement the existing methods to yield more accurate population estimates[26, 55] for RMNCH indicator denominators.

The division of a reported numerator by the census projected denominator without further adjustment could result in extremely high or low coverage estimates. Different methods were used to further adjust for extremely low or high RMNCH coverage indicators. Overall coverage adjustment was usually done when trying to estimate coverage at a high administrative level such as regions/provinces or national using coverage data from a lower level (district/county level). In some instances, the overall coverage was adjusted without necessarily determining whether the coverage issue relates to an inaccurately estimated numerator or denominator. It is important for future studies to delineate numerator related quality issues from denominator related quality challenges. The data source for numerators is different from denominators and the solution to address quality issues will be different between the two data points.

The annealing method, which uses weighted coverage from administrative and survey data to estimate coverage, is appealing because it produces standard errors making it possible to develop confidence bounds around the final estimated RMNCH indicator coverage. This method, according to the authors, produces an estimate that is more accurate and reliable compared to estimates from either data source alone. However, the authors did not indicate specifically which component of routine data challenge the method addresses or how to adjust for denominator and numerator issues in routine data. The authors did not state how missing data, over and under reporting and incomplete reporting in administrative data should be handled prior to applying the method.

This review has some limitations that should be noted. Titles and abstracts that were not published in English or indexed in PubMed or Scopus reviews were not captured. To the best of our knowledge, this is the first systematic review to examine the analytical methods that have been used to improve estimates of RMNCH intervention coverage using routine data from LMICs. This review provides a comprehensive overview of all the available analytical methods, their strengths and limitations which can be used by health departments across LMICs. This review also draws attention to the need to improve and develop new analytical methods that can increase the usage of routine data to inform local policies and RMNCH implementation plans.

Conclusions

This paper summarizes the literature on the available analytical methods that can be used to estimate RMNCH intervention coverage in LMICs using routine data exclusively or together with survey data. The findings from this review show some advances in RMNCH coverage estimation methods using HMIS data. However, there are remaining gaps, particularly at the analytical stage, to accurately estimate the needed numerators, denominators, and coverage of RMNCH indicators using routine data. As countries continue to strive to close the gap and inequality in RMNCH coverage and meet SDG Goal 3, the consistent use of rigorous analytical methods that allow for the use of HMIS data to estimate RMNCH coverage at national and subnational levels is urgently needed. Improvement in routine data quality and analytical methods, together with advances in electronic information systems (data collection, reporting and transmission) and satellite imaging in LMICs will make HMIS data increasingly useful and efficient for the timely estimation of RMNCH intervention coverage at the national and subnational level. This will enable governments to identify gaps and develop policies and interventions to address such gaps in RMNCH indictor coverage. Even though the available methods are far from perfect, they pave the way for more research in this field.

Availability of data and materials

Not applicable.

Abbreviations

ANC:

Antenatal care

ART:

Anti-retroviral therapy

BCG:

Bacillus Calmette-Guerin

CRVS:

Civil Registration and Vital Statistics

DTP:

Diphtheria, Tetanus, Pertussis

HIV:

HIV:Human Immuno-deficiency Virus

HMIS:

Health Management Information System

LMICs:

Low and Middle Income Countries

MDGs:

Millennium Development Goals

OPV:

Oral Polio Vaccine

RMNCH:

Reproductive Maternal Neonatal and Child Health

WHO:

World Health Organization

References

  1. Kumar S, Kumar N, Vivekadhish S. Millennium development goals (MDGS) to sustainable development goals (SDGS): Addressing unfinished agenda and strengthening sustainable development and partnership. Indian journal of community medicine: official publication of Indian Association of Preventive Social Medicine. 2016;41(1):1.

    Article  Google Scholar 

  2. Victora CG, Requejo JH, Barros AJ, Berman P, Bhutta Z, Boerma T, et al. Countdown to 2015: a decade of tracking progress for maternal, newborn, and child survival. The Lancet. 2016;387(10032):2049–59.

    Article  Google Scholar 

  3. Edelstein M. Measuring vaccination coverage better will help achieve disease control. International health. 2017;9(3):142–4.

    Article  PubMed  Google Scholar 

  4. Hadler S, Cochi S, Bilous J, Cutts F. Vaccination programs in developing countries. Vaccines. 4th ed. Philadelphia: Elsevier; 2004.

    Google Scholar 

  5. Maina I, Wanjala P, Soti D, Kipruto H, Droti B, Boerma T. Using health-facility data to assess subnational coverage of maternal and child health indicators, Kenya. Bull World Health Organ. 2017;95(10):683.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Grittner AM. Results-based Financing: Evidence from performance-based financing in the health sector: Discussion Paper; 2013.

  7. Oxman AD, Fretheim A. Can paying for results help to achieve the Millennium Development Goals? A critical review of selected evaluations of results-based financing. Journal of Evidence‐Based Medicine. 2009;2(3):184–95.

    Article  PubMed  Google Scholar 

  8. Brenzel L, Measham A, Naimoli J, Batson A, Bredenkamp C, Skolnik R. Taking stock: World Bank experience with results-based financing (RBF) for health. Washington: The World Bank; 2009.

    Google Scholar 

  9. Naimoli JF. Global health partnerships in practice: taking stock of the GAVI Alliance’s new investment in health systems strengthening. Int J Health Plan Manag. 2009;24(1):3–25.

    Article  Google Scholar 

  10. Knoll M, Zloczysti P. The good governance indicators of the millennium challenge account: How many dimensions are really being measured? World Dev. 2012;40(5):900–15.

    Article  Google Scholar 

  11. Vujicic M, Weber SE, Nikolic IA, Atun R, Kumar R. An analysis of GAVI, the Global Fund and World Bank support for human resources for health in developing countries. Health policy planning. 2012;27(8):649–57.

    Article  PubMed  Google Scholar 

  12. Cutts FT, Izurieta HS, Rhoda DA. Measuring coverage in MNCH: design, implementation, and interpretation challenges associated with tracking vaccination coverage using household surveys. PLoS medicine. 2013;10(5).

  13. Bosch-Capblanch X, Ronveaux O, Doyle V, Remedios V, Bchir A. Accuracy and quality of immunization information systems in forty‐one low income countries. Tropical Med Int Health. 2009;14(1):2–10.

    Article  Google Scholar 

  14. Lim SS, Stein DB, Charrow A, Murray CJ. Tracking progress towards universal childhood immunisation and the impact of global initiatives: a systematic analysis of three-dose diphtheria, tetanus, and pertussis immunisation coverage. The Lancet. 2008;372(9655):2031–46.

    Article  Google Scholar 

  15. Bhattacharya AA, Umar N, Audu A, Felix H, Allen E, Schellenberg JR, et al. Quality of routine facility data for monitoring priority maternal and newborn indicators in DHIS2: A case study from Gombe State, Nigeria. PLoS One. 2019;14(1):e0211265.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  16. Day LT, Gore-Langton GR, Rahman AE, Basnet O, Shabani J, Tahsina T, et al. Labour and delivery ward register data availability, quality, and utility-Every Newborn-birth indicators research tracking in hospitals (EN-BIRTH) study baseline analysis in three countries. BMC Health Serv Res. 2020;20(1):1–14.

    Article  Google Scholar 

  17. Mavimbe JC, Braa J, Bjune G. Assessing immunization data quality from routine reports in Mozambique. BMC Public Health. 2005;5(1):108.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Mate KS, Bennett B, Mphatswe W, Barker P, Rollins N. Challenges for routine health system data management in a large public programme to prevent mother-to-child HIV transmission in South Africa. PloS one. 2009;4(5):e5483.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  19. World Health Organization. Immunization, Vaccination and Biologicals 2019 [updated 8/2019; cited 2020 4/13/2020]. Available from: https://www.who.int/immunization/monitoring_surveillance/routine/coverage/en/index1.html.

  20. Cutts FT, Claquin P, Danovaro-Holliday MC, Rhoda DA. Monitoring vaccination coverage: defining the role of surveys. Vaccine. 2016;34(35):4103–9.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Salami L, Ouendo E-MD, Fayomi B. Effects of Results Based Financing Models on Data Quality Improvement in Benin on 2014. 2016.

  22. Lacapère F, Magloire R, Danovaro-Holliday MC, Flannery B, Chamoulliet H, Celestin E-P. The use of rapid coverage monitoring in the national rubella vaccination campaign, Haiti 2007–2008. J Infect Dis. 2011;204(suppl_2):698–705.

    Article  Google Scholar 

  23. Nanyunja M, Lewis RF, Makumbi I, Seruyange R, Kabwongera E, Mugyenyi P, et al. Impact of mass measles campaigns among children less than 5 years old in Uganda. Journal of infectious diseases. 2003;187(Supplement_1):63-S8.

    Google Scholar 

  24. Zuber PL, Yaméogo KR, Yaméogo A, Otten MW Jr. Use of Administrative Data to EstimateMass Vaccination Campaign Coverage, Burkina Faso, 1999. J Infect Dis. 2003;187(Supplement_1):86–90.

    Article  Google Scholar 

  25. World Health Organization. World health statistics 2016: monitoring health for the SDGs sustainable development goals. World Health Organization; 2016.

  26. Tatem AJ. Mapping the denominator: spatial demography in the measurement of progress. International health. 2014;6(3):153–5.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Brandt CO, De Herdt T. On the political economy of data collection: lessons from the unaccomplished population census (Democratic Republic of the Congo, 2006–2018). 2019.

  28. Office of the Population Housing Census Commission. Summary and statistical report of the 2007 population and housing census: population size by age and sex. Federal Democratic Republic of Ethiopia, Population Census Commission; 2008.

  29. Ouedraogo I, Tigabu M, Savadogo P, Compaoré H, Odén P, Ouadba J. Land cover change and its relation with population dynamics in Burkina Faso, West Africa. Land Degradation Development. 2010;21(5):453–62.

    Article  Google Scholar 

  30. Khalidi NA. Afghanistan: demographic consequences of war, 1978–1987. Central Asian Survey. 1991;10(3):101–26.

    Article  CAS  PubMed  Google Scholar 

  31. National Population Commission. Final results of 2006 population census of Nigeria. Nigeria: NPC, Abuja; 2006.

    Google Scholar 

  32. Idike A, Eme OI. Census politics in Nigeria: an examination of 2006 population census. Journal of Policy Development studies. 2015;289(1851):1–26.

    Google Scholar 

  33. Dunkle SE, Wallace AS, MacNeil A, Mustafa M, Gasasira A, Ali D, et al. Limitations of using administratively reported immunization data for monitoring routine immunization system performance in Nigeria. The Journal of infectious diseases. 2014;210(suppl_1):523-S30.

    Google Scholar 

  34. Campbell H, el Arifeen S, Hazir T, O’Kelly J, Bryce J, Rudan I, et al. Measuring coverage in MNCH: challenges in monitoring the proportion of young children with pneumonia who receive antibiotic treatment. PLoS Med. 2013;10:5.

    Article  Google Scholar 

  35. Stanton CK, Rawlins B, Drake M, dos Anjos M, Cantor D, Chongo L, et al. Measuring coverage in MNCH: testing the validity of women’s self-report of key maternal and newborn health interventions during the peripartum period in Mozambique. PloS one 2013;8:5.

  36. McCarthy KJ, Blanc AK, Warren CE, Mdawida B. Women’s recall of maternal and newborn interventions received in the postnatal period: a validity study in Kenya and Swaziland. Journal of global health. 2018;8(1).

  37. Blanc A, Warren C, McCarthy K, Kimani J, Ndwiga C. RamaRao. Assessing the validity of indicators of the quality of maternal and newborn health care in Kenya. J Glob Health. 2016;6:010405.

    Article  PubMed  PubMed Central  Google Scholar 

  38. Veritas Health Innovation. Covidence systematic review software. Veritas Health Innovation Melbourne, VIC; 2016.

  39. Wandera EA, Mohammad S, Bundi M, Nyangao J, Galata A, Kathiiko C, et al. Impact of rotavirus vaccination on rotavirus hospitalisation rates among a resource-limited rural population in Mbita, Western Kenya. Tropical Med Int Health. 2018;23(4):425–32.

    Article  Google Scholar 

  40. Wandera EA, Mohammad S, Ouko JO, Yatitch J, Taniguchi K, Ichinose Y. Variation in rotavirus vaccine coverage by sub-counties in Kenya. Tropical medicine health. 2017;45(1):9.

    Article  PubMed  PubMed Central  Google Scholar 

  41. Assegaai T, Reagon G, Schneider H. Evaluating the effect of ward-based outreach teams on primary healthcare performance in North West Province, South Africa: A plausibility design using routine data. South African Medical Journal. 2018;108(4):329–35.

    Article  CAS  PubMed  Google Scholar 

  42. Agyepong IA. Reforming health service delivery at district level in Ghana: the perspective of a district medical officer. Health Policy Plann. 1999;14(1):59–69.

    Article  CAS  Google Scholar 

  43. Bhatnagar P, Gupta S, Kumar R, Haldar P, Sethi R, Bahl S. Estimation of child vaccination coverage at state and national levels in India. Bull World Health Organ. 2016;94(10):728.

    Article  PubMed  PubMed Central  Google Scholar 

  44. Vivancos R, Martinez R. Performance assessment of the Ugandan National Programme of Immunization in Masindi: analysis of routine data. J Trop Pediatr. 2008;54(1):62–5.

    Article  PubMed  Google Scholar 

  45. Borgdorff MW, Walker GJ. Estimating coverage from routine information. Tropical doctor. 1988;18(3):130–4.

    Article  CAS  PubMed  Google Scholar 

  46. Haddad S, Bicaba A, Feletto M, Fournier P, Zunzunegui MV. Heterogeneity in the validity of administrative-based estimates of immunization coverage across health districts in Burkina Faso: implications for measurement, monitoring and planning. Health policy planning. 2010;25(5):393–405.

    Article  PubMed  PubMed Central  Google Scholar 

  47. Jeffery C, Pagano M, Hemingway J, Valadez JJ. Hybrid prevalence estimation: Method to improve intervention coverage estimations. Proceedings of the National Academy of Sciences. 2018;115(51):13063–8.

    Article  Google Scholar 

  48. Delvaux T, Samreth S, Barr-DiChiara M, Seguy N, Guerra K, Ngauv B, et al. Linked response for prevention, care, and treatment of HIV/AIDS, STIs, and reproductive health issues: results after 18 months of implementation in five operational districts in Cambodia. JAIDS Journal of Acquired Immune Deficiency Syndromes. 2011;57(3):e47–55.

    Article  PubMed  Google Scholar 

  49. Audureau E, Kahn JG, Besson M-H, Saba J, Ladner J. Scaling up prevention of mother-to-child HIV transmission programs in sub-Saharan African countries: a multilevel assessment of site-, program-and country-level determinants of performance. BMC Public Health. 2013;13(1):286.

    Article  PubMed  PubMed Central  Google Scholar 

  50. Saito S, Chung H, Mahy M, Radin AK, Jonnalagadda S, Hakim A, et al. Pediatric HIV treatment gaps in 7 East and Southern African Countries: examination of modeled, survey, and routine program data. JAIDS Journal of Acquired Immune Deficiency Syndromes. 2018;78:134-S41.

    Article  Google Scholar 

  51. Mensah K, Heraud J, Takahashi S, Winter A, Metcalf CJE, Wesolowski A. Seasonal gaps in measles vaccination coverage in Madagascar. Vaccine. 2019;37(18):2511–9.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  52. Organization WH. Data quality review: module 2: desk review of data quality. 2017.

  53. World Health Organization. Assessing and Improving the Accuracy of Target Population Estimates for Immunization Coverage. Working draft. 2015.

  54. World Health Organization. Analysis and use of health facility data 2020 [cited 2020 9/1]. Available from: https://www.who.int/healthinfo/tools_data_analysis_routine_facility/en/.

  55. Bharti N, Djibo A, Tatem AJ, Grenfell BT, Ferrari MJ. Measuring populations to improve vaccination coverage. Scientific reports. 2016;6:34541.

    Article  CAS  Google Scholar 

Download references

Acknowledgements

The authors would like to thank the Countdown to 2030 team for their input and support.

About this supplement

This article has been published as part of BMC Health Services Research Volume 21 Supplement 2 2021: Health facility data to monitor national and subnational progress – Part 2. The full contents of the supplement are available at https://bmchealthservres.biomedcentral.com/articles/supplements/volume-21-supplement-2.

Funding

This work was supported by the Bill & Melinda Gates Foundation through a grant to the Countdown to 2030.

Author information

Authors and Affiliations

Authors

Contributions

GM and AA made substantial contributions to the conception, acquisition of data, analysis, and interpretation of data and were major contributors in writing and revising the manuscript. EH contributed to the acquisition of data, analysis, and interpretation of data and was involved in drafting and critically revising the manuscript for important intellectual content. AM contributed to drafting and critically revising the manuscript for important intellectual content. All authors read and approve the final manuscript for publication.

Corresponding author

Correspondence to Agbessi Amouzou.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mwinnyaa, G., Hazel, E., Maïga, A. et al. Estimating population-based coverage of reproductive, maternal, newborn, and child health (RMNCH) interventions from health management information systems: a comprehensive review. BMC Health Serv Res 21 (Suppl 2), 1083 (2021). https://doi.org/10.1186/s12913-021-06995-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12913-021-06995-z

Keywords