Identifying a dependent variable
We now present the construction of the sustainment index for health indicators, as the dependent variable against which different models can ultimately be tested, and three analytical steps, which we have taken to validate its construct. Operationally, we want to measure how a health indicator (Y) trend from time T0 to T1 continued from time T1 to T2. The sustainment index is a simple-to-calculate approximation of the derivative of Y over time (T0: baseline, T1: endline, and T2: post-project), based on the ratio of the slope of YT1-T2 over YT0-T1. The sustainment index is a quantification of a trend change over two period for a health indicator—it carries no assumptions about human agency, program contribution, attribution, secular trend, or other (see Discussion).
Part 1 - development of the sustainment index
We start with a draft index, which we then modify for boundary conditions. Figure 1 presents the evolution of the value of Y, independent of causal interferences, across three points in time:
-
b = YT0: baseline value of Y;
-
e = YT1: endline value of Y; and
-
p = YT2: post-project value of Y.
We treat the evolution of Y across the two periods as a linear trend, and calculate the ratio of the slopes (YT1-T2 / YT0-T1). This assumes that the endline estimate was greater than the baseline in a statistically significant manner (if the original indicator trend was negative, the issue would be how to take corrective measures, not how to sustain). We add 1 to calibrate the sustainment index around three benchmarks for the value of p (Fig. 1):
-
p1: Sustainment Index = 2.0. Flawless continuation of progress from T0 to T1 until T2.
-
p2: Sustainment Index = 1.0. Plateau reached after T1.
-
p3: Sustainment Index = 0. Return to baseline conditions, no sustainment.
This leads us to a first draft of the index:
-
[C1] -
Sustainment Index Condition of Applicability: e > b:
$$ \mathrm{Index}\ \mathrm{Draft}=1+\frac{\left(p-e\right)/\Delta 2}{\left(e-b\right)/\Delta 1} $$
With ∆1 = T1-T0 and ∆2 = T2-T1
Values of the Sustainment Index above 2.0 and below 0.0 are possible, and would correspond respectively to an acceleration of progress beyond mere sustainment, or a reversal of progress beneath baseline conditions.
Two “boundary conditions” of the indicators under study could affect the value of the Sustainment Index and need to be addressed.
Natural boundary A
As an illustration, if b = 70% and e = 90%, then a sustainment index value of 2.0 would require p = 110%, which is impossible. In this case, we recalibrate our measure so that the maximum plausible value of p, pmax = 100% (at T2) gives us the maximum plausible value of Sustainment Index (2.0). This natural boundary A is encountered when two conditions are present at the same time:
-
[C2] -
First condition for natural boundary A: \( \left(e-b\right)>\left( pmax-e\right)\ast \left(\frac{\Delta 1}{\Delta 2}\right) \), and
-
[C3] -
Second condition: Sustainment Index value is above 1.0 (otherwise, recalibration is not needed): p > e
If C2 and C3 are respected then the sustainability index term \( \frac{\left(p-e\right)}{\left(e-b\right)} \) needs to be replaced by \( \left(\frac{p-e}{pmax-e}\right) \). Figure 2 illustrates this visually.
Natural boundary B
For some indicators, pmax cannot feasibly be 100%. This can be seen in the example of the Contraceptive Prevalence Rate (CPR), for which a maximum of 65% can be set for all practical purposes. The same should apply to other indicators.
This provides us with the final conditional equations for the Sustainment Index, whereby accounting for conditions C1 to C3 addresses the two possible boundary conditions.
Conditional Equations for the Sustainment Index:
-
[C1]:
e > b
-
[C2]:
\( \left(e-b\right)>\left( pmax-e\right)\ast \left(\frac{\Delta 1}{\Delta 2}\right) \)
-
[C3]:
p > e
-
If C1 = false, the Sustainment Index is not applicable
-
If C2 = true and C3 = true\( \mathrm{Sustainment}\ \mathrm{Index}=1+\left(\frac{p-e}{pmax-e}\right) \)
-
If C2 = false or C3 = false\( \mathrm{Sustainment}\ \mathrm{Index}=1+\left(\frac{p-e/\Delta 2}{e-b/\Delta 1}\right) \)
Part 2 – Testing the sustainment index
We then tested the validity of the Sustainment Index metric through:
-
A face validity validation exercise;
-
Application to a real post-project dataset; and
-
Application to Demographic and Health Surveys (DHS).
Correlation of the sustainment index with an independent practitioners’ panel rating
We opportunistically selected a panel of seventeen health professionals, each with a Master’s in Public Health, MD or PhD degree, and with five to 30 years of global health work with various bilateral and government agencies, non-governmental organizations (NGOs), and in-country partners, to provide expert ratings and correlate those with computed values of the Sustainment Index.
We divided the experts into groups of two or three and assigned to each group a set of baseline and endline maternal, newborn and child health indicators, and hypothetical post-project indicator data from a hypothetical district. Panel members had no information on why an indicator had improved in the first place, or on why it progressed in a certain direction during the second phase of measurement. Each group only had to answer how much improvement of an indicator from T0 to T1 had been sustained from T1 to T2, using a graduated visual scale ranging from 0 (return to baseline) to 2 (unchanged trend). We provided the same two examples to each group to calibrate overall responses (Fig. 3). Experts provided ratings on a total of 61 indicator sets, comprised of baseline, endline, and hypothetical post project values. We used the Pearson correlation coefficient to compare the expert group rating with our measure.
Application of the sustainment index metric to a published pre- and post-project dataset in Bangladesh
We then used actual and published baseline, endline, and five-year post-project data on an urban health project carried out in two Bangladeshi municipal health departments (Saidpur and Parbatipur) [16]. We applied the Sustainment Index construct to 13 reported health indicators, and observed the distribution of the Sustainment Index values.
Application of the sustainment index metric to demographic and health survey datasets
The DHS [17] provides internationally standardized nationally-representative population-based datasets. We applied the Sustainment Index to two maternal and child health outcome indicators:
-
1.
Contraceptive Prevalence Rate (CPR): percent of currently married women using any family planning method; and
-
2.
Percent of children aged 12–23 months who have received all basic vaccines by two years of age (Full Vaccination).
We used data from 22 countries in Africa with at least three waves of DHS surveys conducted between 1994 and 2014 (Table 2) [17]. The three waves of DHS surveys were treated as baseline (T0 – earliest survey), endline (T1), and post-project (T2 – latest survey) values for each indicator. We kept the convention of referring to indicator values as b, e, and p, even though the terms “baseline”, “endline”, and “post-project” do not apply to these national longitudinal data. Our goal was simply to describe the behavior of the metric. We set pmax at 100% for immunization. For CPR we used pmax = 65% based on the highest observed values in African countries [18].