Given the growing emphasis on monitoring and evaluation of health systems strengthening programs in developing countries, understanding how to interpret and compare findings from various health facility assessments has taken on added importance [1, 13]. This study, a secondary analysis of SPA conducted in five sub-Saharan African countries, systematically compared estimates of medicine availability using five definitions of availability (four of current availability and one six-month availability). Our results show that estimates of current availability vary substantially depending on how it is defined. Compared to ‘observed availability of at least one valid unit’, the reference definition used in this study, availability of medicines based solely on reported response was on average 6% higher across the five countries, while observed availability where all units were valid was 11% lower. In addition, availability during the six-month period preceding the survey was 14% lower than the reference. The pattern and magnitude of relative differences in various current availability estimates were comparable between public and non-public facilities in most countries. However, in Kenya, Tanzania, and Uganda, relative differences between the six-month availability and reference values were significantly larger among public facilities than non-public facilities.
Definitions of current availability used in our study provide rather simple snapshots of medicine availability. The reference definition of ‘observed availability of at least one valid unit on the day of assessment’ does not reveal the presence of expired medicines in the storage area, which would indicate poor commodity supply management practices. Neither the reference nor the more strict definition of current availability (i.e., observed availability in which all units are valid) provide any information on whether facilities currently have a sufficient amount of medicines on-hand to meet client needs. Also, current availability may not be a good proxy for availability over an extended period, as our results showed a wide range of variation in six-month period to current availability ratios. Finally, affordability and rational use of medicines, as well as the presence of falsified and substandard medicines, are important aspects of pharmaceutical systems that cannot be assessed from availability alone.
Nevertheless, in a large scale health facility assessment such as SPA – with a large sample size to provide sub-national estimates of indicators across a number of services, we believe ‘observed availability of at least one valid unit’ is the most appropriate definition to measure current medicine availability. The reference definition has practical advantages compared to the other definitions assessed in this analysis. First, compared to the two more inclusive definitions of current availability (i.e., reported availability, and observed availability without verifying validity), it provides more accurate data, while requiring minimal additional costs in fieldwork. One of the most important factors determining implementation cost is the total number facilities selected and, especially in low-resource settings where transportation can be limited, the number of facilities that can be visited per day. Once the survey team is at a sampled facility, the additional time required to observe medicines and verify the expiration date of one unit per medicine is typically not long enough to affect implementation costs. And, compared to the more strict definition of current availability (verification of the expiration date of every unit), the reference definition can reduce surveyor fatigue substantially – especially in large facilities, which is critical for achieving high data quality. Finally, the six-month period availability definition has limited value, since it is based on reported responses on ever having stock-out. In order to measure period availability with minimum errors and bias, medicine registers need to be reviewed for all medicines to obtain the number of clinic days during the period and presence of medicines on each clinic day. While it is important for facilities to be able to provide such detailed information for supply chain management purposes, reviewing such data would not be feasible for a large-scale health facility assessment.
Balancing these practical advantages and limitations, the SPA questionnaires have been recently revised to include only questions that are necessary to calculate estimates based on the reference current availability defition . Questions regarding verification of all units for selected medicines and reported six-month stock out were eliminated. The revised questions to assess medicine availability have been adopted in the World Health Organization’s latest health facility assessment tool, the Service Availability and Readiness Assessment (SARA) .
There are limitations in our analysis. First, considering potential recall errors in reporting retrospective six-month stock-out, the relative difference of 11% between the current and the six-month period availability estimates might have been underestimated. Second, relative differences across estimates in our study, the fairly small difference between the reported and the observed current availability in particular, may be limited to surveys collecting both reported and verified responses. In SPA, surveyors inform the respondents that they will validate expiration dates of available units for each medicine. Thus, respondents have little incentive to provide systematically biased responses; therefore, any difference between reported and observed availability is likely due to random reporting errors. However, in assessments that rely on reported responses only, there may be more reporting bias in addition to random errors, depending on the objective and purpose of the assessment. For example, if a health facility assessment is conducted for monitoring and evaluation of performance based financing programs, there may be increased incentives to underreport stock-outs. Finally, in comparison of results by managing authority and facility level, the small number of medicines could have contributed to lack of statistical power in spite of relatively large differences between sub-groups in some cases. Also, if all or a portion of the 32 medicines are supplied by the same distributor, availability among them might be correlated, violating an assumption of independence among observations for T-test.