An Integrated eDiagnosis Approach (IeDA) versus standard IMCI for assessing and managing childhood illness in Burkina Faso: a stepped-wedge cluster randomised trial

Abstract

Background

The Integrated eDiagnosis Approach (IeDA), centred on an electronic Clinical Decision Support System (eCDSS) developed in line with national Integrated Management of Childhood Illness (IMCI) guidelines, was implemented in primary health facilities of two regions of Burkina Faso. An evaluation was performed using a stepped-wedge cluster randomised design with the aim of determining whether the IeDA intervention increased Health Care Workers’ (HCW) adherence to the IMCI guidelines.

Methods

Ten randomly selected facilities per district were visited at each step by two trained nurses: One observed under-five consultations and the second conducted a repeat consultation. The primary outcomes were: overall adherence to clinical assessment tasks; overall correct classification ignoring the severity of the classifications; and overall correct prescription according to HCWs’ classifications. Statistical comparisons between trial arms were performed on cluster/step-level summaries.

Results

On average, 54 and 79% of clinical assessment tasks were observed to be completed by HCWs in the control and intervention districts respectively (cluster-level mean difference = 29.9%; P-value = 0.002). The proportion of children for whom the validation nurses and the HCWs recorded the same classifications (ignoring the severity) was 73 and 79% in the control and intervention districts respectively (cluster-level mean difference = 10.1%; P-value = 0.004). The proportion of children who received correct prescriptions in accordance with HCWs’ classifications were similar across arms, 78% in the control arm and 77% in the intervention arm (cluster-level mean difference = − 1.1%; P-value = 0.788).

Conclusion

The IeDA intervention improved substantially HCWs’ adherence to IMCI’s clinical assessment tasks, leading to some overall increase in correct classifications but to no overall improvement in correct prescriptions. The largest improvements tended to be observed for less common conditions. For more common conditions, HCWs in the control districts performed relatively well, thus limiting the scope to detect an overall impact.

Trial registration

ClinicalTrials.gov NCT02341469; First submitted August 272,014, posted January 19, 2015.

Background

Currently, more than 75 low- and middle-income countries (LMIC) are implementing the Integrated Management of Childhood Illness (IMCI) strategy on a large scale. However, poor adherence of health care workers (HCWs) to guidelines has often been reported [1, 2], likely due to health system limitations, such as lack of training, coordination and supervision, or low availability of essential medicines and equipment [3,4,5,6]. In Burkina Faso, the IMCI strategy was introduced in 2003, but an evaluation conducted in 2011 reported a low coverage of training and poor performance in terms of adherence to guidelines [7].

Recent advances in Information and Communication Technologies (ICT) and the advent of electronic Clinical Decision Support System (eCDSS) could potentially transform health care services in LMICs, for instance by helping HCWs to correctly follow relatively complex charts. However, several reviews reveal the lack of evidence for a scalable and sustainable impact on health indicators [8,9,10,11,12]. In particular, the experience with using such technology to improve adherence to the IMCI guidelines is limited [13,14,15,16,17].

From 2014, Terre des hommes foundation (Tdh), in partnership with the Burkinabe Ministry of Health (MoH), implemented, in primary health facilities of two regions of Burkina Faso, the Integrated eDIagnosis Approach (IeDA), a complex intervention centred on an eCDSS developed in line with national IMCI guidelines, with the objective of improving HCWs’ adherence to the IMCI guidelines. Between 2014 and 2017, an evaluation was performed using a stepped-wedge cluster randomised design by an independent team from the London School of Hygiene and Tropical Medicine (LSHTM), United Kingdom, and Centre Muraz, Burkina Faso. The aim of the evaluation was to determine whether the IeDA intervention increased adherence to the IMCI guidelines and improved clinical assessment, classification, prescription, referral and counselling during under-five child consultations in primary health facilities.

Methods

Setting

In Burkina Faso, coverage of key effective interventions for preventing child deaths has steadily increased following the adoption of successive public health policies (e.g. free anenatal care, subsidies for child birth and emergency obstetric care, national distribution of insecticide treated nets, Artemisinin-based Combination Therapy (ACT) for treating uncomplicated malaria at facility and community level, expanded program for vaccination). Consequenttly, in 2015, the under-five mortality rate had declined by 56% compared to 1990, from an estimated 202 deaths per 1000 live births in 1990 to 89 deaths per 1000 live births in 2015 [18]. The government is the main health service provider and managed 83% of facilities within the country in 2014 [19]. The country is divided into 13 regions further subdivided into 63 health districts each with one district or regional hospital. In rural areas, primary health facilities, usually run by one or more nurses with the support of health assistants, are the most common point of care and provide a basic package of outpatient services. In 2014, there were 1824 primary health facilities, corresponding to about one facility per 10,000 inhabitants.

The evaluation took place in the Boucle du Mouhoun and Nord regions from September 2014 to November 2017. Of the 11 districts in these two regions, three districts were selected by the implementing agencies to pilot the first versions of the eCDSS in 2010 and were therefore excluded from the evaluation, which was restricted to the eight remaining districts (Fig. 1). In addition to IeDA, a performance-based financing (PBF) intervention was independently implemented in four trial districts (Nouna, Solenzo, Gourcy and Ouahigouya districts). From April 2016, free care for under-five children was also introduced by the MoH in all public facilities [20].

Fig. 1

Eight health districts included in the trial. Blue and red circles indicate control and intervention districts respectively. Source: Burkina Faso, Map No. 4230, November 2004, UNITED NATIONS

The IeDA intervention

The IeDA intervention comprised five components: 1. An eCDSS provided on tablets to primary health facilities for the management of under-five connsultations: Based on the information recorded by HCWs from the clinical assessment of the child (e.g. body temperature), the eCDSS displays the relevant charts on the screen to guide HCWs through the IMCI national protocol, from the classification (e.g. uncomplicated malaria), through prescription (e.g. first line antimalarial), referral and counselling. During the trial period, several versions were deployed following feedback from users and stakeholders; 2. A six-day training course provided to HCWs on IMCI guidelines and the use of the eCDSS. During the last year of the trial, learning modules with short videos were also available on the eCDSS to support continuous training; 3. A quality assurance coaching system involving team meetings two to four times a year through which health district authorities and HCWs discussed solutions to their local issues (e.g. organisation of care); 4. A supervision system including monthly visits to primary health facilities; 5. A health information system based on data collected through the eCDSS. During the last year of the trial, descriptive dashboards on under-five consultations were developed and shared with the health district authorities and HCWs.

Evaluation design

Since some components of the intervention could only be delivered at the district level, and rolling out the intervention in a phased manner was more practical for the implementing agencies, the evaluation used a stepped-wedge cluster randomised design, with health districts (“clusters”) receiving the intervention at different time points in a randomised order.

Nine steps, one every 4 months, were initially planned, with the first step used as baseline (Fig. 2a). However, funding and logistic issues resulted into delayed roll-out and only four out of eight districts with the intervention implemented. The baseline phase included the first two steps, and during each of the next four steps, from step 3 to step 6, a new district implemented the intervention (Fig. 2b). For the purposes of data collection, ten primary facilities with staff trained in IMCI were randomly selected in each district with stratification on the 2013 annual under-five consultations caseload [21]. Eight rounds of data collection were conducted in total (Fig. 2b).

Fig. 2

a Stepped-wedge design: planned roll-out of the IeDA intervention. b Stepped-wedge design: actual roll-out of the IeDA intervention. Districts shaded in dark green had full implementation of the IeDA intervention. Districts shaded in light green had partial implementation of the IeDA intervention (“contaminated” control districts)

Full implementation in a district was considered to have been achieved when the eCDSS was provided to all primary facilities and when all HCWs had been trained in its use and IMCI guidelines. In some control districts, data were collected after implementation started but before the full implementation was completed, resulting in some “contamination” of these control districts (Fig. 2b).

Randomisation and masking

Randomisation was restricted to ensure intervention and control clusters were balanced with respect to region and the PBF intervention. Details of the randomisation procedure used to allocate districts to receive the intervention have been published elsewhere [21]. Randomization was performed by JL, independently of Tdh. The nature of the intervention precluded formal masking of fieldworkers.

The allocation of the intervention to each district was gradually communicated by the research team to the implementing agencies and the list of surveyed facilities was not communicated to reduce the likelihood that more intensive support was provided to those facilities.

Sample size

The sample size was determined using the method described by Hussey and Hughes [22], assuming a design effect of 2 due to clustering within facilities and a between cluster coefficient of variation of 0.3. With a harmonic mean of ten children seen at each of the ten selected health facility of the eight districts per step (and therefore 100 children per district and 800 children per step), the trial would provide 90% power to detect an increase in any of the primary outcome from 25 to 33% %. With a harmonic mean of only four children seen per facility at each step, the trial would have 98% power to detect an increase from 25 to 40% [21].

Data collection

Data collection was conducted by two teams, each comprising two trained nurses. At each step, all ten selected primary facilities in each of the eight districts, were visited once for data collection. Data were collected for all consultations of children aged 2 months to 5 years old occurring during the research team’s visit to the facility. Each visit lasted 2 days or less if the required minimum sample size of children observed per facility was achieved. At each step, the newest intervention district was visited last to maximise the chances that HCWs had learnt how to use the new technology. Each visit was notified, by the data collection team, to the facility the day before the visit.

One independent trained nurse observed the consultation and recorded, using a structured and pre-tested observation form programmed into a tablet, the HCW’s clinical practices, illness classifications and prescriptions given to the child. Observations were passive, and the observer never intervened during the consultation. Validation data were collected by the second independent trained nurse, who conducted a repeat consultation with the child, using the eCDSS. These validation data were intended to provide a “gold standard” classification for each child. When there were discrepancies between the HCW and the validation nurse, the final management of the child was agreed by discussion between the two of them.

In addition, at each visit, a shortened version of the WHO Service Availability and Readiness Assessment (SARA) questionnaire [23] was completed to document the availability of essential medicines and equipment required by IMCI guidelines.

The four nurses recruited for data collection had previously been trained in IMCI by the MoH. The two nurses responsible for observation of consultations had at least 5 years of experience working in a health centre. The two validation nurses had at least 10 years of experience working in a health centre and were also IMCI trainers. In addition, all underwent 2 weeks of training, provided by the main investigators, on the study methods and tools prior to the trial, and benefited from two refresher trainings, provided by Tdh, on IMCI and the eCDSS during the trial.

Outcomes

The evaluation focussed on the adherence to IMCI charts designed for new consultations of children aged 2 months to 5 years old to assess, classify and treat danger signs, cough/difficult breathing, diarrhoea, fever and nutritional status.The evaluation did not consider IMCI charts designed for children who return after an intial consultation. We excluded charts related to HIV and ear problems due to their very low prevalence during the trial period (across all steps and according to the validation nurses, only 0.9% of children classified with HIV infection, and 2.7% of children classified with ear problems). We also excluded the charts related to vitamin A supplementation and vaccination as coverage was high in Burkina Faso. Upon the advice of the trial’s scientific advisory committee, for anaemia, only adherence to the clinical assessment task was evaluated due to the difficulty of assessing anaemia reliably when laboratory testing was locally unavailable.

Primary and secondary outcomes are defined in the Additional file 1. Briefly, the primary outcomes included: 1. overall adherence to clinical assessment tasks; 2. overall correct classification ignoring the severity of the classifications (upon the advice of the trial’s scientific advisory committee); and 3. overall correct prescription according to HCWs’ classifications. The secondary outcomes included: 1. adherence to assessment of danger signs; 2. correct identification of at least one danger sign; 3. overall correct classification accounting for the severity of the classifications; 4. overall correct prescription according to validation nurses’ classifications; 5 & 6. overall correct referral or hospitalisation according to HCWs’ assessment and to validation nurses’ assessment; and 7. overall correct treatment counselling.

Other reported outcomes are: sensitivity and specificity of the HCWs’ classifications; over-prescription of antibiotics and antimalarials; overall availability index of essential oral medicines and equipment (Additional file 2).

Analyses

Analyses were performed using Stata version 14. Analysis included all new consultations of children aged 2 months to 5 years old and excluded children who return after an intial consultation for a follow-up consultation. Primary analyses included “contaminated” control districts as control districts based on the intention-to-treat (ITT) principle.

Secondary analyses excluded these districts for the period when they were contaminated.

Descriptiive analyses were performed using individual-level data and point estimates and confidence intervals for all outcomes were computed accounting for the clustering of observations within districts and facilities using the svy family of commands in Stata.

Comparisons between trial arms and statistical tests to investigate evidence of an intervention effect were performed on cluster/step-level summaries as recommended by Hayes and Moulton [24] for trials with fewer than about 15 clusters per arm to account for the clustered nature of the data. A “vertical” stepped wedge analysis was performed with permutation test using the swpermute command in Stata [25]. This approach analyses each step as a parallel arm trial or, in other words, computes, for each step, one cluster summary per district and one effect estimate and then combines these step-level effect estimates into a weighted average (with the weights proportional to the harmonic mean of the number of clusters in each arm and step). This approach, recommended by Thomson et al. [26], preserves the randomisation and accounts for secular trends. “Horizontal” comparisons, i.e. comparison within a cluster over time (which are non-randomised), do not contribute to the analysis. Applied to our design, across the six steps and the eight clusters, 46 cluster/step summaries were computed (two cluster/step-level summaries were excluded from the analysis due to data lost in two districts at step 6 and 7 respectively) giving six effect estimates which were then combined into a weighted average for each of our outcome.

The above approach was used for all primary and secondaty outomes with the exception of correct identification of at least one danger sign and overall correct referral/hospitalisation. Given the very small number of children with danger signs or severe classifications warranting referral/hospitalisation who contributed to these two outcomes, Fisher’s exact test, performed on individual level data and ignoring clustering, was used to test for an intervention effect.

Statistical tests to investigate evidence of a difference between trial arms were only performed on the primary and secondary outcomes to reduce the problem of multiple testing. No formal adjustment was made for multiple testing. Because our ten endpoints are not all independent to each other, applying the Bonferronni correction would be overly conservative (as it assumes that all hypotheses being tested are independent of each other).

Results

After excluding 189 follow-up consultations, data were recorded for 2724 new consultations of children aged 2 months to 5 years old: 686 consultations at baseline, 1343 consultations in control districts and 695 consultations in intervention districts (Fig. 3, Additional file 4).

Fig. 3

Trial flow diagram (number of consultations of children aged 2 to 60 months). * Eight districts randomised but only 4 actually received the IeDA intervention

About half of consultations at baseline and in the control and intervention districts (46%, 317/686, 49%, 658/1343, and 48%, 335/695 respectively) were conducted in districts where the concurrent PBF intervention was implemented.

While the IMCI paper-form was used for 70% (479/686) and 68% (918/1343) of the consultations at baseline and in control clusters respectively, the eCDSS was used in nearly all consultations (97%, 674/694) in intervention clusters. The occasional use of the eCDSS at baseline (1%, 8/686) or in the control districts (9%, 120/1343) reflects instances of early roll-out of the eCDSS prior to training.

Gender and age distributions were similar at baseline and by trial arm (Table 1). Based on validation nurses’ assessment, the most common classification given to children was malaria (between 53 and 69% of children across baseline and trial arms) (Table 2). Other common classifications included: diarrhoea with no dehydration (about 27%) and pneumonia (between 16 and 27%). About 45% of children had one classification only and between 33 and 48% had two or more classifications (Table 3).

Table 1 Child’s gender and age at baseline and by trial arm

Table 2 Validation nurses’ classifications at baseline and by trial arm

Table 3 Number of classifications according to validation nurses at baseline and by trial arm

Adherence to clinical assessment

Across the six IMCI charts, the average percentage of tasks completed by the HCWs was 48% at baseline, 54% in the control districts and 79% in the intervention districts with evidence for a difference between trial arms (cluster-level mean difference = 30%; P-value = 0.002) (Table 4). For all IMCI charts, HCWs in the intervention districts completed more of the recommended tasks compared to HCWs in the control districts (Table 5). In particular, more of the recommended tasks were completed for assessing danger signs: 95% versus 34% in the intervention and control districts respectively (cluster-level mean difference = 71%; P-value = 0.002) (Table 4).

Table 4 Primary and secondary outcomes

Table 5 Adherence to clinical assessment by IMCI chart

Identification of danger signs

The proportion of children correctly identified, by the HCWs, with at least one danger sign was 67% (16/24) at baseline and 56% (14/25) in the control districts. It appeared to be somewhat higher (75%, 12/16) in the intervention districts but the small number of children with danger signs preclude firm conclusion (cluster-level mean difference = 19%; P-value = 0.322) (Table 4).

Classification

Ignoring the severity of the classifications, the proportion of children for whom the validation nurses and the HCWs recorded the same classifications was 75% (457/609) at baseline, 73% (767/1049) in the control districts and 79% (450/572) in the intervention districts with evidence for a difference between trial arms (cluster-level mean difference = 10%; P-value = 0.004) (Table 4). Accounting for the severity of the classifications slightly lowered the proportions of correct classifications (cluster-level mean difference = 9%; P-value = 0.038) (Table 4).

By IMCI chart, HCWs in the intervention districts correctly classified children having diarrhoea with no dehydration, dysentery and acute malnutrition (severe or moderate) more often than those in the control districts (Table 6). Although based on a small number of children, HCWs in intervention districts also appeared to correctly classify children with severe malaria or severe febrile illness more often than those in control districts.

Table 6 Sensitivity of the HCW’s classification: Proportion of children correctly classified in a given classification

HCWs in the intervention districts were also less likely to wrongly diagnose pneumonia as being present when it was not: 7% (38/521) versus 19% (209/1113) (Table 7). For other conditions, false positive diagnoses were rare (< 5%) in both arms.

Table 7 Specificity of the HCW’s classification: Proportion of children correctly not classified in a given classification

Prescription

Overall, the proportion of children who received all the recommended prescriptions in accordance with the HCWs’ classifications was 76% (465/614) at baseline, 78% (836/1074) in the control districts and 77% (437/567) in the intervention districts with no evidence for a difference between trial arms (cluster-level mean difference = − 1%; P-value = 0.788) (Table 4). According to the validation nurses’ classifications, these proportions were 65% (398/610) at baseline, 66% (693/1049) in the control districts and 69% (392/572) in the intervention districts (cluster-level mean difference = 7%; P-value = 0.226).

By IMCI chart, correct prescriptions for dysentery were much more common in the intervention districts than in the control districts, as were correct prescriptions for acute malnutrition (severe without complications or moderate) and severe malaria or severe febrile illness, although still infrequent (Tables 8 and 9). Correct prescriptions for diarrhoea with no dehydration were also higher in the intervention districts compared to the control districts (Table 9).

Table 8 Correct prescription according to the HCWs’ classifications: Proportion of children who received at least all the recommended prescriptions

Table 9 Correct prescription according to the validation nurses’ classifications: Proportion of children who received at least all the recommended prescriptions

Over-prescription

According to the HCWs’ classifications, the proportion of children who were not in need of an antibiotic but who were actually prescribed one was 11% (77/681) at baseline, 14% (187/1341) in the control districts and 8% (56/694) in the intervention districts (Table 10). According to validation nurses’ classifications, these proportions were 18% (123/668) at baseline, 23% (289/1252) in the control districts and 10% (69/676) in the intervention districts (Table 11). With respect to antimalarials, a Rapid Diagnostic Test (RDT) was performed for about 90% of febrile children in both arms (Table 5) and over-prescription was low and similar at baseline and between trial arms: around 2 to 4%.

Table 10 Over-prescription according to the HCWs’ classifications: Proportion of children who were not in need of a given medicine but who were actually prescribed it

Table 11 Over-prescription according to the validation nurses’ classifications: Proportion of children who were not in need of a given medicine but who were actually prescribed it

Referral/ hospitalisation

Overall, the proportion of children in need of referral or hospitalisation according to the HCWs’ assessment who were actually referred or hospitalised by the HCWs was 60% (21/35) at baseline, 52% (22/42) in the control districts and 61% (25/41) in the intervention districts with no evidence for a difference between trial arms (individual-level mean difference = 9%; P-value = 0.509) (Table 4). According to the validation nurses’ assessment, these proportions were 55% (16/29) at baseline, 53% (17/32) in the control districts and 68% (15/22) in the intervention districts (individual-level mean difference = 15%; P-value = 0.398).

Treatment counselling

The proportion of caretakers to whom the HCWs mentioned both the number of doses a day and the number of days for all the relevant oral medicines prescribed for treating the child at home was 77% (473/612) at baseline, 92% (1046/1143) in the control districts and 88% (506/576) in the intervention districts with no evidence for a difference between trial arms (cluster-level mean difference = − 4.1%; P-value = 0.355) (Table 4). For all oral medicines, both the number of doses per day and the number of days were mentioned by the HCWs to a high proportion of caretakers at baseline and in both trial arms (Table 12).

Table 12 Correct treatment counselling: Proportion of caretakers to whom the HCWs mentioned both the number of doses a day and the number of days for home-based treatment

Availability of essential oral medicines and equipment

The average proportion of essential oral medicines that were observed to be available at the health facilities was high: 98% at baseline, 94% in the control districts and 89% in the intervention districts (Table 13). However, deworming treatments, amoxicillin, ORS and multivitamins were less frequently available in the intervention districts compared to the control districts.

Table 13 Availability of essential oral medicines and equipment

With respect to essential equipment, availability at the health facilities was high: 87% at baseline, 87% in the control districts and 91% in the intervention districts. Better availability of electricity and equipment to administer ORS was observed in the intervention districts compared to the control districts.

Explanatory analyses

Comparison of HCWs’ performance with and without use of IMCI paper-forms in the control districts

In order to assess whether the frequent use of IMCI paper-based form in the control districts had an effect on HCWs performance, primary and secondary outcomes in the control districts were compared between HCWs who were observed to use an IMCI paper-form and those who did not.

Surprisingly, HCWs who did not use an IMCI paper-form in the control districts seem to have better assessed danger signs than those who used a form: on average they performed 45% versus 22% of the recommended tasks respectively (Additional file 5). For all other outcomes, HCWs’ performance was similar between the two groups.

Agreement between HCWs and validation nurses’ clinical assessment

The square root of the mean square errors (RMSE) for the differences in child’s weight, height and temperature measurements between HCWs and validation nurses indicate differences of a small magnitude (< 1 kg, < 3 cm or < 1 °C) at baseline and in the trial arms (Additional file 6a). Higher RMSE were observed between HCWs and validation nurses’ measurements of mid-upper arm circumference (MUAC) (around 5 mm) and respiratory count (around 9 counts). All differences were fairly balanced between trial arms.

With respect to RDT results and caretakers’ answers about children’s key symptoms, actual agreement between HCWs and validation nurses were high (> 90%) at baseline and in the trial arms (Additional file 6b). The Kappa coefficients indicate that 90% or more of RDT results were in agreement beyond that expected by chance. The Kappa coefficients for caretakers’ answers range from 0.60 to 0.88.

Secondary analyses

Excluding “contaminated” control districts for the period when they were contaminated removed a total of 173 consultations from the analysis and made little or no difference to the results (Additional file 7).

Discussion

The IeDA intervention improved substantially HCW’s adherence to IMCI’s clinical assessment tasks (30% point increase on average across the intervention districts compared to the control districts), including the assessment of danger signs, which led to some overall increase in the proportion of children being correctly classified (around 10% point increase on average across the intervention districts compared to the control districts) but to no improvement in overall proportion of children receiving correct prescriptions. The intervention, however, appeared to have reduced over-prescription of antibiotics by 6 to 13% points.

Achieving correct classification depends, at least in part, on the clinical skills of the HCWs, which may be more difficult to improve than task adherence itself and may have limited the effect of the intervention on correct classification. Recent more advanced clinical charts, also built on electronic tools, such as electronic point-of-care tests (ePOCT) integrating malaria RDT, haemoglobin, pulse oximetry in all febrile patients and other tests (e.g. glucometer, C-reactive protein) in subgroups of them, have led to major improvements in febrile disease classification and a considerable reduction of antibiotic prescription [27].

In addition, using the eIMCI in Burkina Faso, improvements in classifications and prescriptions tended to be observed for less common conditions, such as dysentery and malnutrition, for which HCWs in the control districts performed relatively poorly. The data were also consistent with an improvement in danger sign identification, correct referrals/hospitalisations and management of severe malaria or severe febrile illness, although small numbers of such children preclude firm conclusions. For other, more common, conditions (e.g. malaria or pneumonia), HCWs in the control districts performed relatively well in classifying and prescribing the correct medicines, thus limiting the scope to detect an overall impact.

There were some notable differences between findings at baseline and in the control arm with respect to prevalence of pneumonia (27 and 16% respectively), malaria (69 and 55% respectively) and anaemia (13 and 7% respectively). At baseline and in the control arm, 33 and 18% of observations respectively occurred from January to March, during the peak of the pneumonia season. Observations during the malaria season (July to November) were less frequent at baseline (49%) compared to the control arm (61%). However, the higher prevalence at baseline is consistent with the higher proportion of positive RDT: 82% of RDTs were positive at baseline compared to 66% during the control steps. These results may reflect a more intense malaria season during the baseline steps. This could also explain the difference in anaemia prevalence, which is associated with malaria.

Our findings are broadly consistent with the limited evidence available on the effectiveness of eCDSS for improving adherence to IMCI (eIMCI). In 18 primary facilities in four districts of Tanzania, only 21% of children had all ten critical IMCI tasks assessed under paper-based IMCI compared to 71% under eIMCI (p < 0.001) [14]. In two basic health centres in the Kabul province of Afghanistan, only 24% of children underwent a physical examination in line with IMCI at baseline compared to 84% after 1 year of implementation (p < 0.05) [17]. Comparison of HCWs classifications with classifications given by an independent nurse in Tanzania showed that the electronic protocol improved overall correct classification: 83% under paper-based IMCI compared to 91% under eIMCI (p < 0.001) [14]. In Afghanistan, only 35% of children received a treatment in line with HCWs’ classifications at baseline compared to 99% after 1 year of implementation [17]. Reduction in over-prescriptions of antibiotic have also been reported using eIMCI in Afghanistan [17] and Tanzania [15].

In Burkina Faso, interviews with HCWs indicated that IeDA was well accepted, in particular with respect to the usefulness of the eCDSS in guiding through the clinical assessment (Blanchet K et al.: Realist evaluation of the Integrated electronic Diagnostic Approach (IeDA) for the management of childhood illness at primary health facilities in Burkina Faso, submitted). In Ghana, South Africa and Tanzania, HCWs reported similar opinions [13, 16]. Nevertheless, our realistic evaluation in Burkina Faso also revealed contextual factors that may have limited the effect of the IeDA intervention. First, staff turnover was reported to be common by district managers, in particular in remote rural facilities where most HCWs do not want to spend more than a few years. A visit in July 2017 in all intervention facilities revealed that around a third of HCWs (36%) had been changed within the last 12 months and that a relatively large proportion (36%) of HCWs had not benefited from the eIMCI training (Blanchet K et al.: Realist evaluation of the Integrated electronic Diagnostic Approach (IeDA) for the management of childhood illness at primary health facilities in Burkina Faso, submitted). Second, while supervision and audit with feedback can be effective in improving performance [28,29,30], monthly supervision visits planned under the IeDA intervention in Burkina Faso faced challenges. The district management teams reported limited budget, access to vehicles and time to dedicate to these visits (Blanchet K et al.: Realist evaluation of the Integrated electronic Diagnostic Approach (IeDA) for the management of childhood illness at primary health facilities in Burkina Faso, submitted).

In addition to incomplete coverage of the IeDA intervention, while pressure from children’s caretakers, sometimes reported during interviews with HCWs (Blanchet K et al.: Realist evaluation of the Integrated electronic Diagnostic Approach (IeDA) for the management of childhood illness at primary health facilities in Burkina Faso, submitted), may have limited the reduction in over-prescription of antibiotics, the relatively lower availability of some essential medicines, such as amoxicillin and ORS, in the intervention facilities compared to the control facilities may have limited improvement in correct prescriptions for pneumonia, severe acute malnutrition without complications and diarrhoea. Multiple conditions may also have influenced the medicines prescribed. Across baseline and trial arms, about a third or more of children were diagnosed with two or more classifications. In Tanzania, a large know-do gap was observed, and a lack of knowledge was not the only constraint identified for improved performance. HCWs’ weak belief in the importance of following guidelines and confidence in their own experience, lack of intrinsic motivation, and physical or cognitive “overload” were also reported, with poor remuneration contributing to several of these factors [31].

Limitations

Some limitations of our evaluation should be acknowledged. First, the “gold standard” classifications were provided by a repeat consultation after the initial consultation and it is possible that the clinical status of some children (e.g. respiratory rate, temperature, current convulsions) may have changed in the interval between the two. Therefore, we should not expect full agreement between HCWs and validation nurses. Our “gold standard” is certainly less than perfect, and this would tend to reduce the apparent magnitude of any improvement in classifications.

Second, it is likely that the behaviour of HCWs was impacted by the fact that they were observed [32]. The high proportion of HCWs observed using IMCI paper-forms in the control districts (68% overall) compared to routine practice (less than 8% of under-five consultations in 2012 [33]) suggests that HCWs in this arm were motivated to perform better than usual. Even if HCWs in the control districts who used IMCI paper-forms did not seem to have performed better compared to those who did not use IMCI paper-forms, repeated observations might explain improvements in some indicators from baseline to control steps, for instance adherence to assessment of danger signs (18% at baseline compared to 34% during control steps). Nevertheless, the behaviour of HCWs in the intervention districts may also have been affected by the presence of observers. Therefore, our findings may over-estimate how well HCWs perform in the absence of an observer, but it is difficult to assert whether or in which direction this may have affected the comparison of intervention and control districts.

Third, the initial evaluation design was not followed. In particular, rolling out the intervention to all districts as planned would have led to more data in the intervention arm, which could have strengthen our findings. In addition, the evaluation design could not address the multi-faceted nature of the intervention and evolving version of the eCDSS. It is therefore not possible to distinguish which component of the intervention led to observed improvements or whether improvements were the result of the combination of components.

Lastly, with respect to statistical analyses, multiple comparisons between arms were performed and can increase the overall error in hypothesis testing, so that P-values should be interpreted with caution. The small number of clusters per trial arm precluded using random effects models on individual level data, thus limiting our ability to control for individual child-level factors.

Conclusion

To conclude, the IeDA intervention was well accepted and improved substantially HCW’s adherence to IMCI clinical assessment which led to some improvements in overall correct classifications but little or no improvement in overall correct prescriptions. Nevertheless, substantial improvements were observed in correct classifications and prescriptions for dysentery and malnutrition. To some degree, we also observed an improvement in danger sign identification, correct referrals/hospitalisations and management of severe malaria, although small numbers prevent firm conclusions. For the most common conditions, HCWs in the control districts, who may have been influenced by a Hawthorne effect, performed relatively well, limiting the scope to detect an overall impact.

HCWs’ practices are complex behaviours that have many potential contextual and intrinsic influences. Lower availability of some essential medicines in the intervention districts was observed and our realistic evaluation concurrently reported staff turnover and incomplete coverage of training and supervision which may have limited the effect of the IeDA intervention on correct classification and prescription. Task adherence may be easier to achieve than correct classifications which require clinical skills. In the context of national scaling up, disparities between regions exist in terms of structures, staff and resources. Nevertheless, complete coverage of the eIMCI training could be achieved by its integration into the initial nursing curriculum. Supervision will inevitably require resources but also management capacity to deal with relationships, organisation culture and HCWs’ professional norms, experiences and motivation (Blanchet K et al.: Realist evaluation of the Integrated electronic Diagnostic Approach (IeDA) for the management of childhood illness at primary health facilities in Burkina Faso, submitted).