A simple metric for a complex outcome: proposing a sustainment index for health indicators

Identifying a dependent variable

We now present the construction of the sustainment index for health indicators, as the dependent variable against which different models can ultimately be tested, and three analytical steps, which we have taken to validate its construct. Operationally, we want to measure how a health indicator (Y) trend from time T0 to T1 continued from time T1 to T2. The sustainment index is a simple-to-calculate approximation of the derivative of Y over time (T0: baseline, T1: endline, and T2: post-project), based on the ratio of the slope of Y_T1-T2 over Y_T0-T1. The sustainment index is a quantification of a trend change over two period for a health indicator—it carries no assumptions about human agency, program contribution, attribution, secular trend, or other (see Discussion).

Part 1 - development of the sustainment index

We start with a draft index, which we then modify for boundary conditions. Figure 1 presents the evolution of the value of Y, independent of causal interferences, across three points in time:

• b = Y_T0: baseline value of Y;

• e = Y_T1: endline value of Y; and

• p = Y_T2: post-project value of Y.

Values of the Sustainment Index above 2.0 and below 0.0 are possible, and would correspond respectively to an acceleration of progress beyond mere sustainment, or a reversal of progress beneath baseline conditions.

Two “boundary conditions” of the indicators under study could affect the value of the Sustainment Index and need to be addressed.

Natural boundary A

As an illustration, if b = 70% and e = 90%, then a sustainment index value of 2.0 would require p = 110%, which is impossible. In this case, we recalibrate our measure so that the maximum plausible value of p, pmax = 100% (at T2) gives us the maximum plausible value of Sustainment Index (2.0). This natural boundary A is encountered when two conditions are present at the same time:

1. [C2] -

   First condition for natural boundary A: \( \left(e-b\right)>\left( pmax-e\right)\ast \left(\frac{\Delta 1}{\Delta 2}\right) \), and

    
2. [C3] -

   Second condition: Sustainment Index value is above 1.0 (otherwise, recalibration is not needed): p > e

    

If C2 and C3 are respected then the sustainability index term \( \frac{\left(p-e\right)}{\left(e-b\right)} \) needs to be replaced by \( \left(\frac{p-e}{pmax-e}\right) \). Figure 2 illustrates this visually.

Part 2 – Testing the sustainment index

Correlation of the sustainment index with an independent practitioners’ panel rating

We opportunistically selected a panel of seventeen health professionals, each with a Master’s in Public Health, MD or PhD degree, and with five to 30 years of global health work with various bilateral and government agencies, non-governmental organizations (NGOs), and in-country partners, to provide expert ratings and correlate those with computed values of the Sustainment Index.

We divided the experts into groups of two or three and assigned to each group a set of baseline and endline maternal, newborn and child health indicators, and hypothetical post-project indicator data from a hypothetical district. Panel members had no information on why an indicator had improved in the first place, or on why it progressed in a certain direction during the second phase of measurement. Each group only had to answer how much improvement of an indicator from T0 to T1 had been sustained from T1 to T2, using a graduated visual scale ranging from 0 (return to baseline) to 2 (unchanged trend). We provided the same two examples to each group to calibrate overall responses (Fig. 3). Experts provided ratings on a total of 61 indicator sets, comprised of baseline, endline, and hypothetical post project values. We used the Pearson correlation coefficient to compare the expert group rating with our measure.

Application of the sustainment index metric to a published pre- and post-project dataset in Bangladesh

We then used actual and published baseline, endline, and five-year post-project data on an urban health project carried out in two Bangladeshi municipal health departments (Saidpur and Parbatipur) [16]. We applied the Sustainment Index construct to 13 reported health indicators, and observed the distribution of the Sustainment Index values.

Application of the sustainment index metric to demographic and health survey datasets

We used data from 22 countries in Africa with at least three waves of DHS surveys conducted between 1994 and 2014 (Table 2) [17]. The three waves of DHS surveys were treated as baseline (T0 – earliest survey), endline (T1), and post-project (T2 – latest survey) values for each indicator. We kept the convention of referring to indicator values as b, e, and p, even though the terms “baseline”, “endline”, and “post-project” do not apply to these national longitudinal data. Our goal was simply to describe the behavior of the metric. We set pmax at 100% for immunization. For CPR we used pmax = 65% based on the highest observed values in African countries [18].

Correlation of the sustainment index with an independent practitioners’ panel rating

The Pearson correlation coefficient (r²) between the expert score and our first draft measure was 0.621. After re-calibrating our measure for boundary conditions C2 and C3 (final Sustainment Index) we obtained an r² value of 0.922 (Fig. 4), which indicated strong correlation.

Application of the sustainment index metric to a published pre- and post-project dataset in Bangladesh

Qualitative data in the source publication address the question of attribution, but the Sustainment Index helps us identify a distribution of indicators, with different degrees of sustainment. In effect, our index shows potential for helping us ask why different indicators behaved differently in a post-project period, and helps us discriminate between relatively higher or lower levels of sustainment.

Proper identification and treatment of Acute Respiratory Infection (ARI) is an outlier with a Sustainment Index of 2.8, which corresponds to a phase of accelerated improvement where gains sustained continue to increase in the post-project period. The initial publication identified this indicator as based on a relatively small sub-sample of children, thus putting into question whether this measured acceleration is realistic or a measurement artifact of the original indicator. The reliability of the underlying indicators is central to the reliability of the Sustainment Index.

Application of the sustainment index metric to demographic and health survey datasets

Three countries did not meet the conditions of applicability C1 for computing a Sustainment Index for either indicator (Benin, Kenya, Lesotho); three more did not meet condition C1 for immunization (Uganda, Zambia, Zimbabwe), and Malawi did not meet the condition for CPR. Our recalibration for boundary conditions was not needed for this dataset.

Further observations are made by looking at each indicator closely.

The Sustainment Index values for CPR in 18 countries ranged from − 4.60 for Zimbabwe to 25.0 for Niger (Fig. 5). One country (Zimbabwe) indicated absolutely no sustainment of CPR and actually showed a reversal to below T0 values, three countries showed limited sustainment (Guinea, Mozambique, Ghana), three more had near-plateau sustainment (Zimbabwe, Nigeria, Namibia), three showed stronger sustainment (Cameroon, Cote d’Ivoire, Burkina Faso), and the remaining eight showed accelerated progress (Zambia, Ethiopia, Rwanda, Madagascar, Mali, Senegal, Tanzania, Malawi, Niger).

Sixteen countries out of 22 had results that allowed for computation of a Sustainment Index for full vaccination coverage. One showed limited sustainment (Mali), five had near-plateau sustainment (Guinea, Namibia, Cote d’Ivoire, Mozambique, Nigeria), five indicated strong sustainment (Cameroon, Ethiopia, Uganda, Madagascar, Senegal) and five showed accelerated progress (Ghana, Burkina, Tanzania, Rwanda, Niger) (Fig. 6).

Limitations

This paper offers a metric and presents data to support its face, construct, and computation validation, but it clearly faces a number of limitations.

We are able to understand distribution of level of sustainment, and discriminate between different levels of performance, but the index does not address what allowed progress to be sustained at the measured level. Further research will indicate how much the Sustainment Index adds to our toolbox of measures.

Excessive sensitivity of the measure to small variation between b and e should be mitigated by respecting conditionality [C1], which stipulates that e must be greater than b in a statistically significant manner.

One must remember that the index is not a measure of absolute good, but relative inflection of a trend between T0 and T1. It is insensitive to the actual values of the indicator. Progress on coverage from 20 to 40% to 60% would yield the same exact Sustainment Index (2.0) as progress from 20 to 25% to 30%. In other words, the sustainment index measures the level of sustainment of a performance (how the slope of progress is sustained), not the performance of the indicator (the inclination of the trend).

Value of the sustainment index for future research

The application of a Sustainment Index in Bangladesh for a local intervention with clear start and end dates, and with a rigorous evaluation, allowed for a simple interpretation of the Sustainment Index. Application to national datasets would require different types of research questions with in-depth review of multiple programs, funding streams, and externalities. Where documentation has already been made of project achievements on national scale, the Sustainment Index could be suggested to expand learning. While questions of attribution versus contribution fit the definition of a “wicked” problem when it comes to sustainment, comparisons to secular trend lines have been used in the literature and would allow researchers to strengthen the evidence [9, 19].

The Sustainment Index provides a measure, which can discriminate between different dynamics of change of an indicator from one period of time to the next. It allows comparisons of the behavior of different indicators in the same context, as well as the comparison of the behavior of the same indicator in different contexts—something evidently more useful if the researcher knows something about the contexts and factors at play. Its merit is in providing a quantification of an outcome, which appears desirable – continuation of progress under changing conditions – and in allowing comparison within sensible and empirical parameters. A sustainment index of 1.48 is not “good” or “bad”, but under empirical conditions, ceteris paribus, it is better than an index value of 0.98. The purpose of research is to ask ‘why?’ Ultimately, the sustainment index can be at best one more tool in our analytical toolbox, starting with observations, comparisons, and raising more questions.

In the case of post-project (ex-post) evaluation studies, it will be interesting to systematically examine the distribution of the Sustainment Index value across contexts, projects, interventions and indicators. Large datasets are not likely to be available, but case study post-project evaluation could be enriched by reference to a common metric. This should encourage an increase in the number of post-project evaluations so far carried out and better aggregation of their findings [6, 9–12].

Studies could also examine how progress is sustained after the end of external investments across different groups of population, based on income, residence, ethnic identification, or gender, in order to identify pro-equity policies and design factors.

Ultimately, quantifying the degree of sustainment of program investments only makes sense if research is sought to guide policy and program investments. This would make greater sense to development assistance partners seeking to support lower and middle income countries, or to countries seeking to support more poorly performing regions or districts. It would have limited value when conditions are close to satisfactory and at equilibrium.