Improving data quality and supervision of antiretroviral therapy sites in Malawi: an application of Lot Quality Assurance Sampling

  • Bethany L Hedt-Gauthier1, 2Email author,

    Affiliated with

    • Lyson Tenthani3,

      Affiliated with

      • Shira Mitchell1,

        Affiliated with

        • Frank M Chimbwandira3,

          Affiliated with

          • Simon Makombe3,

            Affiliated with

            • Zengani Chirwa3, 4,

              Affiliated with

              • Erik J Schouten3, 5,

                Affiliated with

                • Marcello Pagano1 and

                  Affiliated with

                  • Andreas Jahn3, 4

                    Affiliated with

                    BMC Health Services Research201212:196

                    DOI: 10.1186/1472-6963-12-196

                    Received: 12 January 2012

                    Accepted: 9 July 2012

                    Published: 9 July 2012

                    Abstract

                    Background

                    High quality program data is critical for managing, monitoring, and evaluating national HIV treatment programs. By 2009, the Malawi Ministry of Health had initiated more than 270,000 patients on HIV treatment at 377 sites. Quarterly supervision of these antiretroviral therapy (ART) sites ensures high quality care, but the time currently dedicated to exhaustive record review and data cleaning detracts from other critical components. The exhaustive record review is unlikely to be sustainable long term because of the resources required and increasing number of patients on ART. This study quantifies the current levels of data quality and evaluates Lot Quality Assurance Sampling (LQAS) as a tool to prioritize sites with low data quality, thus lowering costs while maintaining sufficient quality for program monitoring and patient care.

                    Methods

                    In January 2010, a study team joined supervision teams at 19 sites purposely selected to reflect the variety of ART sites. During the exhaustive data review, the time allocated to data cleaning and data discrepancies were documented. The team then randomly sampled 76 records from each site, recording secondary outcomes and the time required for sampling.

                    Results

                    At the 19 sites, only 1.2% of records had discrepancies in patient outcomes and 0.4% in treatment regimen. However, data cleaning took 28.5 hours in total, suggesting that data cleaning for all 377 ART sites would require over 350 supervision-hours quarterly. The LQAS tool accurately identified the sites with the low data quality, reduced the time for data cleaning by 70%, and allowed for reporting on secondary outcomes.

                    Conclusions

                    Most sites maintained high quality records. In spite of this, data cleaning required significant amounts of time with little effect on program estimates of patient outcomes. LQAS conserves resources while maintaining sufficient data quality for program assessment and management to allow for quality patient care.

                    Keywords

                    HIV treatment programs Malawi Africa Supervision Data quality

                    Background

                    In 2004, the Malawi Ministry of Health (MoH) began the rapid expansion of the national HIV treatment program, initiating 271,105 and retaining 198,846 patients on antiretroviral therapy (ART) at 377 sites (279 static and 98 mobile/outreach) by October 2009 [1]. The successful scale-up of the Malawi ART program was made possible through a public health approach [2]. ART eligibility is determined by WHO clinical staging or a low CD4 count. In 2009, these criteria were Stage III or IV disease or CD4 < 250 cells/mm3 (CD4 < 350 cells/mm3 for pregnant women). One standard, fixed-dose first-line regimen (stavudine, lamivudine, nevirapine in 2009) is used for >90% of patients and standard alternative regimens are used for patients who develop side effects to the first-line regimen [3]. Experienced sites (generally district hospitals that have been offering ART for at least three years) also provide standard second-line regimens for eligible patients. Patients visit ART clinics monthly until stable on therapy (usually after six to ten months). Two- or three-monthly appointments are scheduled thereafter.

                    Two critical components of the national ART program are 1) a standardized system for registering and monitoring patients, including individual patient treatment cards with registration and follow-up data and a clinic register summarizing critical data for each patient [3] and 2) quarterly supervision at all ART sites in Malawi [4]. Every quarter, ART clinic personnel perform a standardized patient cohort analysis that includes aggregation of case-finding details of patients registered during the previous quarter and since ART was first begun, as well as treatment outcomes for the cumulative cohort. The latter analysis requires a review of all treatment cards in order to update the clinic register and then a tally-score on current patient outcomes. The primary outcome is patient status, namely, alive on ART, died, stopped ART (with clinician’s knowledge), transferred to another ART clinic, or lost to follow-up. Secondary outcomes for patients alive on ART include 1) ART regimen, 2) drug adherence, 3) ART side effects, and 4) current tuberculosis status. A rolling cohort survival analysis is also performed by counting primary outcomes of patients who registered during specified previous quarters [2].

                    During quarterly supervision, teams of at least two individuals (including a representative from the MoH national or regional office along with a MoH clinical officer or nurse) visit each site for up to a full day. Additional representatives from the health sector, including other MoH staff or non-governmental organization partners, join teams on occasion. Currently, quarterly supervision requires ten teams to be in the field for two to three weeks each at a cost of approximately $60,000 per quarter (excluding staff salaries). The quarterly supervision activity is coordinated by the monitoring and evaluation (M&E) team at the Department for HIV and AIDS in the MoH and funded through the U.S. President's Emergency Plan for AIDS Relief (PEPFAR) and the Global Fund to Fight AIDS, Tuberculosis and Malaria. These visits provide an opportunity to identify any problems at the sites, to discuss difficult cases with medical staff, and to monitor drug stocks, thus improving the overall quality of care [5].

                    The supervision teams also review the patient cohort analyses produced by the ART staff and clean patient data and aggregate quarterly and cumulative reports on 44 data elements. These data are used for drug forecasting, monitoring overall program performance, identifying gaps in care, and as early warning indicators for HIV drug resistance [618], and therefore maintaining high quality data is of utmost importance. Like other national treatment programs, Malawi has strived since its inception to provide complete, accurate and up-to-date data. Common errors include incomplete registers, failure to update follow-up outcomes, duplicate entries, and incorrect/missing records [5]. In 2008, Makombe et al reported that unchecked site-produced reports resulted in a 12% undercount of patients on ART, affecting the programs ability to precisely forecast future drug needs [19].

                    Currently every patient treatment card is reviewed and compared against the entries in the register during supervision visits to ensure up-to-date and accurate data in the register prior to data extraction. Inconsistencies or incomplete data are updated as appropriate. This process can take many hours at each visit, competing with other important aspects of supervision and mentoring. Further, this full audit of patient records by the supervision team may not yield any changes, especially at sites where experienced clinic staff have correctly updated treatment cards and clinic registers in preparation for the supervision visit.

                    While the quality of recording for certain key elements (such as outcome status or regimen) seem to have improved with repetition of this intensive data cleaning process, there remain significant challenges in quality of secondary elements, such as pill counts and side effects. The goal of this study was to analyze the types and frequency of data errors currently observed in the national HIV treatment program and to evaluate their impact on management and monitoring. Based on these findings we evaluated the performance of a lot quality assurance sampling (LQAS) algorithm to prioritize ART sites with poor data quality for a full review of their records before aggregation of quarterly reports. We then estimated the potential amount of time saved by omitting an exhaustive review of all treatment cards at sites that pass the LQAS assessment. Further, we estimated secondary outcomes from treatment cards during the sampling process.

                    Methods

                    Site Visits

                    From 11 to 22 January 2010, a study team joined supervision teams for the 4th Quarter (Q4) 2009 supervision. Twenty-three sites were chosen. Selection was not random, but purposely sampled to achieve diverse representations in terms of geography, size, length of providing treatment, and public/private sectors. As per MoH standard, the clinics were informed of the visit by the supervision team but were not aware that the team would be accompanied by a study team. Although the study team visited all 23 clinics; four were excluded because they either had not initiated any patients on treatment or the patient treatment card filing system had not yet been created. At each site, the study team recorded every treatment card/register discrepancy identified by the supervision team including the type of discrepancy, the information recorded in the treatment card and register, and the correct (updated) information. The study team randomly sampled from the list of all patients enrolled at the site, found the corresponding treatment cards, and recorded the time required for sampling and selected secondary outcomes of patients still alive and on treatment (side effects and treatment adherence; the latter was defined by MoH as eight or fewer pills remaining since last visit). If a selected treatment card was missing, then the next card was included in the sample. All data were recorded in real time in an electronic spreadsheet.

                    LQAS analysis

                    We evaluated the impact of an LQAS system that classified each site either as high or as low data quality. The proposed system randomly sampled treatment cards and compared the sample of treatment cards to the register. Based on the results from these samples, sites were classified and only sites classified as low data quality would receive an exhaustive review of treatment cards and registers during supervision.

                    The sample size required was determined by LQAS methodology [20, 21]. Thresholds for classification as determined by the M&E team at the Department for HIV and AIDS at the MoH, were: high data quality if there was 95% or better concordance between the treatment cards and register (with an allowable 10% risk of misclassification) and low data quality if concordance between the treatment card and register was 85% or less (with an allowable 5% risk of misclassification). Concordance between treatment card and register is defined as complete agreement on primary outcomes (current status and treatment regimen, if alive). Based on the binomial distribution, the minimum sample size required to meet these constraints was 76 (n = 76), with a corresponding decision rule of 7: i.e. a site with less than 7 discrepancies in the 76 sampled treatment cards would be classified as high data quality and a site with 7 or more discrepancies classified as low data quality (see Figure 1a). We also investigated a more conservative rule for site classification: with “high data quality” defined as 99% or more concordance between the treatment card and register and “low data quality” defined as 89% or less concordance. Fixing the sample size at 76, based on the binomial distribution we arrived at a decision rule of 3 for this sample size (see Figure 1b).
                    http://static-content.springer.com/image/art%3A10.1186%2F1472-6963-12-196/MediaObjects/12913_2012_2042_Fig1_HTML.jpg
                    Figure 1

                    Operating Characteristic Curves for the two LQAS classifications. The curves plot the probability of site being classified as ‘high data quality’ based on the LQAS procedure for a given level of concordance between treatment cards and the register. Figure 1a is the plot for the 85%/95% classification. Figure 1b is the plot for the 89%/99% classification, superimposed on top of the 85%/95% classification. Each point represents a clinic, plotted at their true level of concordance between the treatment cards and register based on exhaustive review at the site. Circles indicate sites that are classified as ‘low data quality’ based on the LQAS classification. Triangles indicate sites that are classified as ‘high data quality’.

                    To evaluate the LQAS tool, random sampling of 76 records was done in parallel to the standard supervision process, where complete review and comparisons of treatment cards to registers was performed by the ART supervisors.

                    Analysis

                    We compared basic site characteristics (patient burden, public versus private, length of time providing treatment, certificates of excellence) using frequencies. We also used counts to compare updated versus unchanged patient outcomes and treatment regimens, if alive. Secondary data outcomes (pill count and adherence) were restricted to patients alive and on treatment. These outcomes are reported with binomial confidence intervals at the site level, and aggregated using stratified sampling analysis, weighting by site size. Analyses and graphs were produced in Stata v.11 and R v.2.0.1.

                    Ethical approval

                    All data were collected as part of standard clinical supervision and monitoring practices. Data were extracted from patient cards and registers and there was no direct contact with patients. This activity was an evaluation and quality improvement activity for the Department for HIV and AIDS. The whole process was reviewed and cleared by the Harvard School of Public Health and the Harvard Medical School IRBs.

                    Results

                    Table 1 summarizes characteristics of the 19 sites included in the study. Four (21%) had initiated fewer than 100 patients at the site, six (32%) 101–500 patients, three (16%) 501–1000 patients, and six (32%) more than 1000 patients, a similar distribution to 249 out of 279 static ART clinics in the national treatment program that had initiated patients by the end of Q4 2009. The sample included fewer public clinics, and on average, the sampled sites had provided treatment longer than those in the general national treatment program. Only five (33%) of the sites received a certificate for excellent performance (with “performance” including measures of data quality and record keeping) at the previous supervision visit, a lower rate than in the national program.
                    Table 1

                    Site Characteristics

                    Characteristics of sites

                    Sites included

                    National ART program

                      

                    n = 19

                    n = 249

                    Patient burden

                        
                     

                    100 patients or fewer

                    4

                    21%

                    52

                    21%

                     

                    101 - 500 patients

                    6

                    32%

                    72

                    29%

                     

                    501 - 1000 patients

                    3

                    16%

                    39

                    16%

                     

                    more than 1000 patients

                    6

                    32%

                    86

                    35%

                    Public clinics

                    13

                    68%

                    199

                    80%

                    Length of time providing treatment

                     

                    3 months or less

                    3

                    16%

                    13

                    5%

                     

                    3 months to 2 years

                    3

                    16%

                    73

                    29%

                     

                    2 to 4 years

                    3

                    16%

                    103

                    41%

                     

                    more than 4 years

                    10

                    53%

                    60

                    24%

                    Certificate given at last supervision visit

                    (n = 15)

                    (n = 204)

                      

                    5

                    33%

                    98

                    48%

                    Restricted to public sector sites. Originally, private and public sector sites were supervised separately. As a result, the national ART program records do not document length of time providing treatment for all private sector sites and these were excluded.

                    A total of 16,561 individuals had ever registered for ART at all sites. The exhaustive comparisons and cleaning of the treatment cards to the register took 28.5 hours total for the 19 sites. Extrapolating to all sites, we estimate 374 team-hours are dedicated to data cleaning at the 249 sites every quarter. During the exhaustive review, the supervision team identified few discrepancies in outcomes and treatment regimens when comparing the treatment card and register (median percent of records with discrepancies, 1.0%; IQR: 0.5%-3.0%). Of the 210 discrepancies between records (1.2%) in outcomes, a majority (58%) were incorrect in both the treatment card and the register, often as a result of a patient who had defaulted treatment and neither record had been updated. For 34% of the outcome errors, the treatment card was correct (with the register not updated), and in the remaining 8% the register was correct.

                    For the 11,034 patients alive and on treatment at the sampled sites, only 50 (0.4%) had discrepancies in treatment regimens recorded in their treatment card and register. Most often (94%), regimen changes had been recorded in the treatment card, but not updated in the register; the remaining were correct in the register (4%) or neither the treatment card nor the register (2%). For the 2,605 patients who initiated treatment in Q4 2009, 378 (15%) had discrepancies in their registration data (gender, age group, transfer status, TB history). The treatment card was determined correct in 71% of these errors, the register in 21% and neither in 8%. Usually this resulted from incorrect WHO clinical staging determined from the documented clinical conditions (where a person with a Stage IV condition would be listed as Stage III or vice versa), or TB status was incorrectly recorded as current instead of TB in the previous two years.

                    Since summary measures on outcomes and treatment regimens are extracted from the register, we explored the impact of relying on the registers in their original state versus the registers after a full review and update. Using the register without updates led to a 1.2% overcount in the number alive and on treatment, and a 5.6% undercount in the number of defaulters (Table 2). The uncorrected register data also overcounted by 0.4% the number on first-line treatment, and undercounted those on a zidovudine substitute and efavirenz substitute regimens by 3.6% and 9.7%, respectively. None of the site factors identified previously [19] were significant predictors of low data quality, possibly due to low power since so few sites were visited (results not shown).
                    Table 2

                    Impact of LQAS classification systems

                     

                    System without any review of data quality

                    LQAS Triaging

                    LQAS Triaging

                     

                    Upper threshold = 95%

                    Upper threshold = 99%

                     

                    Lower threshold = 85%

                    Lower threshold = 89%

                    n (sample size)

                    N/A

                    76

                    76

                    d (decision rule)

                    N/A

                    7

                    3

                    Number of sites above the upper threshold

                    N/A

                    16

                    9

                    Number classified as high data quality

                    N/A

                    16

                    9

                    Number of sites below the lower threshold

                    N/A

                    1

                    1

                    Number classified as low data quality

                    N/A

                    1

                    1

                    Differences between updated and non-updated data

                      

                    Alive

                    +1.2%

                    +1.2%

                    +0.7%

                    Default

                    −5.6%

                    −5.8%

                    −2.8%

                    1st Line Rx

                    +0.4%

                    +0.3%

                    +0.1%

                    Substitute, AZT

                    −3.6%

                    −2.8%

                    −1.5%

                    Substitute, EFV

                    −9.7%

                    −9.7%

                    −3.1%

                    Time Required for Review

                    N/A

                    8 hours;5 minutes

                    13 hours;19 minutes

                    Design and error rates in reporting under systems with a) no review process, b) LQAS classification system with pu = 0.95 and pl = 0.85, and c) LQAS classification system with pu = 0.99 and pl = 0.89. AZT = zidovudine. EFV = efavirenz.

                    At 16 sites, we randomly sampled 76 patient treatment cards. The remaining three had fewer than 76 cards inspected because fewer than 76 patients had ever initiated treatment at that site. At these three sites, the classification was made on the exhaustive review. LQAS classification of sites with the 85%/95% thresholds would have reduced the time needed for record review from 28.5 to 8 hours (see Table 2). Based on the exhaustive review, 16 sites had “high data quality” (95% or more agreement between treatment cards and register entries) and all of these sites would have been correctly classified using the LQAS system (Table 2 and Figure 1). “One site had less than 85% concordance between records, and this site would have been classified as low data quality via LQAS. For the two sites with concordance between 85-95%, one site would have been classified as low data quality. Because only two sites were classified as “low data quality” and thus would have received exhaustive review, there were very few updates to the aggregated data extracted from the register. The only marked improvement was a reduced undercount of numbers on the substituted zidovudine regimen (from 3.6% to 2.8%).

                    We also investigated the more conservative rules for site classification, with “high data quality” defined as 99% or more concordance between the treatment card and register and “low data quality” defined as 89% or less concordance. The nine sites with greater than 99% concordance based on the exhaustive inspection were correctly classified as “high data quality” by LQAS. The one site with “low data quality” was also correctly classified. The subsequent exhaustive review at the one additional site identified with this more conservative classification rule improved the accuracy of the aggregate measures extracted from the register, halving the percent errors on nearly all measures and reducing the undercount of efavirenz substitutes to only one third of its previous size (from 9.7% undercount to only 3.1% undercount). This increased accuracy required half of the time currently required for chart review at sites, though it increased the time by 50% over the less conservative (85%/95%) LQAS approach.

                    Table 3 reports the site specific estimates of side effects and adherence in patients alive and on treatment for the 16 sites where sampling was employed (sites with fewer than 76 patients were excluded). At these 16 sites, 41% to 97% of the sampled treatment cards were from patients who were alive and on treatment. Of those alive, between 62-100% had presence or absence of a side effect recorded. For those with recorded side effects, percent with side effects range from 0-27%, with a median of 3.3%. Based on the sample, the estimated proportion of patients alive with side effects was 7.2% (95% CI: 4.8-9.6%). For adherence, pill counts are only conducted on individuals alive and on 1st line treatment, 39-92% of the original sample. For each site, 60-100% of eligible individuals had pill counts recorded and amongst these individuals, patient adherence ranges from 68-100%, with a median of 85.3%. The estimated proportion of patients alive that were adherent was 86.3% (95% CI: 82.8-89.8%).
                    Table 3

                    Secondary analysis of side effects and pill counts

                      

                    Side effects

                    Adherence

                     

                    Number alive in sample

                    Number with side effects recorded

                    Percent with adverse side effects

                    95% Confidence interval

                    Number alive and on 1st line in sample

                    Number with PC recorded

                    Percent with good adherence

                    95% Confidence interval

                    Site 1

                    49

                    47

                    2.1%

                    0.1%

                    11.3%

                    49

                    46

                    69.6%

                    54.2%

                    82.3%

                    Site 2

                    42

                    26

                    7.7%

                    0.9%

                    25.1%

                    39

                    29

                    89.7%

                    72.6%

                    97.8%

                    Site 3

                    47

                    41

                    14.6%

                    5.6%

                    29.2%

                    43

                    39

                    94.9%

                    82.7%

                    99.4%

                    Site 4

                    59

                    55

                    1.8%

                    0.0%

                    9.7%

                    57

                    47

                    87.2%

                    74.3%

                    95.2%

                    Site 5

                    58

                    55

                    0.0%

                    0.0%

                    6.5%

                    56

                    53

                    77.4%

                    63.8%

                    87.7%

                    Site 6

                    45

                    36

                    5.6%

                    0.7%

                    18.7%

                    39

                    31

                    80.6%

                    62.5%

                    92.5%

                    Site 7

                    44

                    44

                    0.0%

                    0.0%

                    8.0%

                    44

                    43

                    100.0%

                    91.8%

                    100.0%

                    Site 8

                    32

                    28

                    7.1%

                    0.9%

                    23.5%

                    30

                    28

                    85.7%

                    67.3%

                    96.0%

                    Site 9

                    44

                    33

                    15.2%

                    5.1%

                    31.9%

                    32

                    13

                    92.3%

                    64.0%

                    99.8%

                    Site 10

                    39

                    38

                    2.6%

                    0.1%

                    13.8%

                    39

                    39

                    74.4%

                    57.9%

                    87.0%

                    Site 11

                    31

                    27

                    3.7%

                    0.1%

                    19.0%

                    30

                    26

                    92.3%

                    74.9%

                    99.1%

                    Site 12

                    73

                    69

                    2.9%

                    0.4%

                    10.1%

                    70

                    65

                    67.7%

                    54.9%

                    78.8%

                    Site 13

                    41

                    34

                    2.9%

                    0.1%

                    15.3%

                    38

                    33

                    84.8%

                    68.1%

                    94.9%

                    Site 14

                    74

                    59

                    0.0%

                    0.0%

                    6.1%

                    65

                    56

                    85.7%

                    73.8%

                    93.6%

                    Site 15

                    57

                    49

                    26.5%

                    14.9%

                    41.1%

                    39

                    32

                    78.1%

                    60.0%

                    90.7%

                    Site 16

                    48

                    45

                    6.7%

                    1.4%

                    18.3%

                    41

                    34

                    76.5%

                    58.8%

                    89.3%

                    Total

                      

                    7.2%

                    4.8%

                    9.6%

                      

                    86.3%

                    82.8%

                    89.8%

                    Discussion

                    Because of the extensive use of the national ART program data for management and analysis of patient care, ensuring the quality of this data is critical. Poor data can lead to imprecise budgeting and procurement of drugs, incorrect inference on program performance, and inappropriate adaptations to program implementation and, ultimately, poor patient care. Like other countries, Malawi has strived to improve and maintain data quality. As evident in these results, the overall level of quality is currently high for critical elements, but the system in place to produce this quality is expensive, time consuming, and can be redundant as many sites have already reviewed and cleaned data prior to supervision visits. Excluding travel time, nearly 750 team-hours every three months are spent at sites, and this time burden will only increase as more patients initiate treatment. We attribute half of this time solely to the data cleaning process and reducing the time for data cleaning can translate into budget savings for the ART program or, perhaps more importantly, into reallocation of time during supervision visits to more crucial activities such as mentoring or discussing difficult cases. With the LQAS system, we save time while maintaining the current standards for data quality. Other research highlights the challenges and associated consequences of poor data quality in health programs and each provides a method for measuring data quality, including some that rely on LQAS [19, 2234]. However, one unique benefit of this LQAS-based system is that the assessment is integrated into routine supervision. The simplicity of the procedure allows the classification to happen on site and immediately links to actions that improve data quality, such as the exhaustive data review in this case.

                    Based on these results, we recommend the following data cleaning strategy for the ART program in Malawi.
                    1. 1.

                      Complete an exhaustive data cleaning process at sites with fewer than 500 patients ever initiated. These sites should have an exhaustive review because error rates are often higher at smaller, less experienced sites such as these, because they have less practice with forms and are unlikely to have a designated data clerk. Further, the review at small sites is less time consuming, reducing the benefit of a sampling process.

                       
                    2. 2.

                      Exhaustive review of all patients initiating in the last quarter at every site. Basic patient characteristics are collected once at registration and had high error rates (15%). Correcting these errors at the first supervision visit after initiation prevents them from recurring across future reports.

                       
                    3. 3.

                      For the public sites with more than 500 patients, sample 76 treatment cards from patients who initiated treatment before the last quarter and compare these against the register. Only complete an exhaustive review if 7 or more inconsistencies are identified. The LQAS system with 85%/95% thresholds maintains data quality sufficient for program purposes and minimizes time required for data review/cleaning. We also recommend collecting information on secondary outcomes such as pill count and side effects during the sampling process.

                       

                    Because time required for the exhaustive review was not separated by new and formerly registered patients, we could not estimate the exact time savings of this system. However, we expect the overall savings to remain significant since less than 16% of records were from new initiates. Further, the estimated time savings attributable to LQAS reviews at the small sites was small relative to the overall time savings (3 of the 20.5 hour reduction). The extrapolation regarding number of team hours taken to clean data each quarter assumed the same average quality of data across all clinics in Malawi, which given the purposive nature of our sampling may not be the case. Another limitation of this study, and perhaps the LQAS system for supervision, is the skipping of the patient treatment cards if the card was not available during the sampling procedure. Out of the records sampled, an average of 1.4 records per site were missing and skipped at the time of the visit. While some of these records were simply lost, often the records were not available because the patient was in the clinic receiving care during the supervision visit. Since most discrepancies were for patients who are no longer alive and on treatment, or who have defaulted, excluding these records likely increases the chance of classifying sites as having poor data quality and following up with a exhaustive data review. An additional limitation of this study (and the proposed supervision system) is the missing data on secondary outcomes from patient records. This likely biases the estimates of secondary outcomes, especially side effects where no record of side effects often suggests none are detected. We believe that active recording of secondary outcomes through the proposed supervision system could actually lead to improved documentation, as clerks and health workers will receive feedback during the process if the data is not recorded. Finally, while we believe the effect to be minimal since sites were not informed of the study team ahead of time, there could be some Hawthorne effect on the rigor of the supervision process. The Hawthorn effect could also affect the speed and accuracy of the ART supervision team, which would impact our estimates of time required for data cleaning as well as time saved using the LQAS system.

                    Additional concerns are raised when proposing a sampling approach for monitoring data quality. One is that failure to identify low data quality sites translates into inaccurate aggregated reports. The proposed algorithm has been designed to minimize this risk – sites with 85% or lower concordance between the treatment card and register have less than a 5% risk of being classified as high data quality at a single visit. As the data quality gets worse the chance of misclassification becomes even smaller, decreasing to less than 1% if the concordance goes down to 80%. This thus results in a very small potential for bias in the aggregated measures. Another concern is that sites may accumulate errors over time or that the site clerks may become less diligent if the exhaustive review is not completed. However, because the LQAS process is repeated independently each quarter, sites with degrading quality, if not captured at one visit, will likely be captured at future visits. Further, during previous rounds of supervision, clerks and health workers have expressed dissatisfaction at the automatic exhaustive review noting the duplication of their efforts and the impression that the supervision teams do not trust their work. The LQAS system addresses these concerns by prioritizing and conducting exhaustive reviews only at sites with poor data.

                    Conclusion

                    In this paper, we describe the development of a simple LQAS system to assess data quality of sites. The LQAS classification is conducted independently at each visit; however the method could be extended so that performance at previous data quality assessments can be incorporated into the decision to conduct exhaustive review at a subsequent visit. Further, the Department for HIV and AIDS may still opt for exhaustive review at sites under special circumstances, such as immediately following guideline changes or changes in personnel at a site. Before any LQAS-based system can be implemented, forms to support the LQAS process must be developed and supervision teams need to be trained. The LQAS system, although straightforward to implement, is more complicated than the current methods that are standardized across all sites. However, supervisors would most likely welcome the system due to the substantial time saved. The LQAS system is unlikely to be a long-term solution for rapidly expanding ART programs such as in Malawi, it would provide a mid-term solution to ensure adequate quality of data until a reliable and sustainable system, such as an electronic record system, could be introduced nationwide [35]. Further, the system illustrated here can be readily adapted by other national programs using paper-based systems to ensure high quality data.

                    Abbreviations

                    ART: 

                    Antiretroviral therapy

                    LQAS: 

                    Lot quality assurance sampling

                    MoH: 

                    Ministry of Health

                    M&E: 

                    Monitoring and evaluation

                    PEPFAR: 

                    President's Emergency Plan for AIDS Relief

                    Q4: 

                    Quarter 4.

                    Declarations

                    Acknowledgements

                    BH and SM received funding from National Institutes of Health Training Grant T32 AI007358. MP received funding from NIH R01 EB006195 and AI097015.

                    The supervision of the national HIV treatment program is supported by the U.S. President's Emergency Plan for AIDS Relief (PEPFAR) and the Global Fund to Fight AIDS, Tuberculosis and Malaria.

                    The authors thank all individuals supporting the supervision of the national treatment program, especially Lucy Chalunda, Thom Chaweza, Absalom Kaunda, Eustice Mhango, Josheph Njala, Sabina Phiri, and MacCloud Pirigu. We also thank Anne Ben-Smith for her assistance in reviewing the paper.

                    Authors’ Affiliations

                    (1)
                    Department of Biostatistics, Harvard School of Public Health
                    (2)
                    Department of Global Health and Social Medicine, Harvard Medical School
                    (3)
                    Department for HIV and AIDS, Ministry of Health
                    (4)
                    I-TECH, Malawi and University of Washington
                    (5)
                    Management Sciences for Health

                    References

                    1. Malawi Ministry of Health HIV Department: Malawi Antiretroviral Treatment Programme Quarterly Report: Results up to 31st December 2009. 2010.
                    2. Harries AD, Gomani P, Teck R, de Teck OA, Bakali E, Zachariah R, Libamba E, Mwansambo A, Salaniponi F, Mpazanje R: Monitoring the response to antiretroviral therapy in resource-poor settings: the Malawi model. Trans R Soc Trop Med Hyg 2004, 98:695–701.PubMedView Article
                    3. Ministry of Health, Malawi: Treatment of AIDS: Guidelines for use of antiretroviral therapy in Malawi. 2008.
                    4. Libamba E, Makombe SD, Harries AD, Schouten EJ, Yu JK, Pasulani O, Mhango E, Aberle-Grasse J, Hochgesang M, Limbambala E, Lungu D: Malawi’s contribution to “3 by 5”: achievements and challenges. Bull World Health Organ 2007,85(2):156–160.PubMedView Article
                    5. Libamba E, Makombe S, Mhango E, Teck O, Limbambala E, Schouten E, Harries AD: Supervision, monitoring and evaluation of nationwide scale-up of antiretroviral therapy in Malawi. Bull World Health Organ 2006,84(4):320–326.PubMed
                    6. Harries AD, Schouten E, Libamba E: Scaling up antiretroviral treatment in resource-poor settings. Lancet 2006, 367:1870–1872.PubMedView Article
                    7. Van Oosterhout JJG, Kumwenda JK, Hartung T, Mhango B, Zijlstra EE: Can the initial success of the Malawi ART scale-up programme be sustained? The example of Queen Elizabeth Central Hospital, Blantyre. AIDS Care 2007,19(10):1241–1246.PubMedView Article
                    8. Makombe SD, Jahn A, Tweya H, Chuka S, Yu JK, Hochgesang M, Aberle-Grasse J, Pasulani O, Schouten EJ, Kamoto K, Harries AD: A national survey of the impact of rapid scale-up of antiretroviral therapy on health-care workers in Malawi: effects on human resources and survival. Bull World Health Organ 2007,85(4):310–314.
                    9. Makombe SD, Jahn A, Tweya H, Chuka S, Yu JK, Hochgesang M, Aberle-Grasse J, Thambo L, Schouten EJ, Kamoto K, Harries AD: A national survey of teachers on antiretroviral therapy in Malawi: Access, retention in therapy, survival. PLoS One 2007,2(7):e620.PubMedView Article
                    10. Makombe SD, Jahn A, Tweya H, Thambo L, Yu JK, Hedt B, Weigel R, Nkata A, Schouten EJ, Kamoto K, Harried AD: A national survey of prisoners on antiretroviral therapy in Malawi: access to treatment and outcomes on therapy. J Infect Dev Countries 2007,1(3):303–307.
                    11. Makombe SD, Harries AD, Yu JK, Hochgesang M, Mhango E, Weigel R, Pasulani O, Fitzgerald M, Schouten EJ, Libamba E: Outcomes of patients with Kaposi’s sarcoma who start antiretroviral therapy under routine program conditions in Malawi. Trop Doct 2008, 38:5–7.PubMedView Article
                    12. Makombe SD, Jahn A, Tweya H, Chuka S, Yu JK, Hedt B, Weigel R, Nkhata A, Schouten EJ, Kamoto K, Harries AD: Antiretroviral therapy in the Malawi police force: access to therapy and treatment outcomes. Malawi Med J 2008,20(1):23–27.PubMed
                    13. Lowrance D, Filler S, Makombe S, Harries A, Aberle-Grasse J, Hochgesang M, Libamba E: Assessment of a national monitoring and evaluation system for rapid expansion of antiretroviral treatment in Malawi. Trop Med Int Health 2007,12(3):377–381.PubMedView Article
                    14. Lowrance DW, Makombe S, Harries AD, Shiraishi RW, Hochgesang M, Aberle-Grasse J, Libamba E, Schouten E, Ellerbrock T, Kamoto K: A public health approach to rapid scale-up of antiretroviral treatment in Malawi during 2004–2006. JAIDS 2008, 49:287–293.PubMedView Article
                    15. Hedt BL, Wadonda-Kabondo N, Makombe S, Harries AD, Schouten EJ, Limbambala E, Hochgesang M, Aberle-Grosse J, Kamoto K: Early warning indicators for HIV drug resistance in Malawi. Antivir Ther 2008,13(Suppl 2):69–75.PubMed
                    16. Banda AC, Makombe SD, Jahn A, Tweya H, Chuka S, Yu JK, Hedt B, Weigel R, Nkhata A, Schouten EJ, Kamoto K, Harries AD: Antiretroviral therapy in the Malawi defence force: access, treatment outcomes and impact on mortality. PLoS One 2008,3(1):e1445.PubMedView Article
                    17. Makwiza I, Nyirenda L, Bongololo G, Banda T, Chimzizi R, Theobald S: Who has access to counseling and testing and anti-retroviral therapy in Malawi – an equity analysis. Int J Equity Health 2009, 8:13.PubMedView Article
                    18. Teferra TB, Hochgesang M, Makombe SD, Kamoto K, Harries AD: Who is accessing antiretroviral therapy in Malawi? A study of the Southern Region on the occupation category “other”. Malawi Med J 2007,19(4):138–139.
                    19. Makombe SD, Hochgesang M, Jahn A, Tweya H, Hedt B, Chuka S, Yu JK, Aberle-Grasse J, Pasulani O, Bailey C, Kamoto K, Schouten EJ, Harries AD: Assessing the quality of data aggregated by antiretroviral treatment clinics in Malawi. Bull World Health Organ 2008,86(11):851–857.
                    20. Lwanga S, Lemeshow S: Sample Size Determination in Health Studies: A Practical Manual. World Health Organization, Geneva; 1991.
                    21. Valadez JJ: Assessing child survival programs in developing countries: testing lot quality assurance sampling. Harvard University Press, Cambridge, MA; 1991.
                    22. Kiragga AN, Castelnuovo B, Schaefer P, Muwonge T, Easterbrook PJ: Quality of data collection in a large HIV observational clinic database in sub-Saharan Africa: implications for clinical research and audit of care. J Int AIDS Soc 2011, 14:3.PubMedView Article
                    23. Doherty TM, McCoy D, Donohue S: Health system constraints to optimal coverage of the prevention of mother-to-child HIV transmission programme in South Africa: lessons from the implementation of the national pilot program. Afr Health Sci 2005,5(3):213–218.PubMed
                    24. Loveday M, Zweigenthal V: TB and HIV integration: obstacles and possible solutions to implementation in South Africa. Trop Med Int Health 2011,16(4):431–438.PubMedView Article
                    25. Garrib A, Stoops N, McKenzie A, Dlamini L, Govender T, Rhode J, Herbst K: An evaluation of the District Health Information System in rural South Africa. S Afr Med J 2008, 98:549–552.PubMed
                    26. Heunis C, Wouters E, Kigozi G, Engelbrecht M, Tsibolane Y, Merwe S, Motlhanke S: Accuracy of Tuberculosis routine data and nurses’ view of the TB-HIV information system in the Fress State, South Africa. J Assoc Nurses AIDS Care 2011,22(1):67–73.PubMedView Article
                    27. Mate KS, Bennett B, Mphatswe W, Barker P, Rollins N: Challenges for routine health systems data management in a large public programme to prevent Mother-to-Child HIV transmission in South Africa. PLoS One 2009,4(5):e5483.PubMedView Article
                    28. Lowrance DW, Ndamage F, Kayirangwa E, Ndagije F, Lo W, Hoover DR, Hanson J, Elul B, Ayaba A, Ellerbrock T, Rukundo A, Shumbusho F, Nash D, Mugabo J, Assimwe A: Adult clinical and immunologic outcomes of the national antiretroviral treatment program in Rwanda during 2004–2005. J Acquir Immune Defic Syndr 2009, 52:49–55.PubMedView Article
                    29. Hoskins S, Weller I, Jahn A, Kaleebu P, Malyuta R, Kirungi W, Fakoya A, Porter K: An appraisal of indicators used to monitor the treated population in antiretroviral programmes in low-income countries. AIDS 2010, 24:2603–2607.PubMedView Article
                    30. Helleringer S, Frimpong JA, Phillips JF, Awooner JK, Yeji F: Operational study of the quality of health data aggregated by community health workers in the Upper East Region of Ghana. American Public Health Association Annual Meeting 2010. Abstract number 219301
                    31. Mahmood S, Ayub M: Accuracy of primary health care statistics reported by community based lady health workers in district Lahore. J Pak Med Assoc 2010,60(8):649–653.PubMed
                    32. Ronveaux O, Rickert D, Hadler S, Groom H, Lloyd J, Bchir A, Birmingham M: The immunization data quality audit: verifying the quality and consistency of immunization monitoring systems. Bull World Health Organ 2005, 83:503–510.PubMed
                    33. Stewart JC, Schroeder DG, Marsh DR, Allhasane S, Kone D: Assessing a computerized routine health information system in Mali using LQAS. Health Policy Plan 2001, 16:248–255.PubMedView Article
                    34. Galvao L, Kaye K: Using lot quality assessment techniques to evaluate quality of data in a community-based health information system. Trop Doct 1994, 24:149–151.PubMed
                    35. Douglas GP, Gadabu OJ, Joukes S, Mumba S, McKay MV, Ben-Smith A, Jahn A, Schouten EJ, Landis Lewsi Z, van Oosterhout JJ, Allain TJ, Zachariah R, Berger S, Harries AD, Chimbwandira F: Using touchscreen electronic medical record systems to support and monitor national scale-up of antiretroviral therapy in Malawi. PLoS Med 2010,7(8):e1000319.PubMedView Article
                    36. Pre-publication history

                      1. The pre-publication history for this paper can be accessed here:http://​www.​biomedcentral.​com/​1472-6963/​12/​196/​prepub

                    Copyright

                    © Hedt-Gauthier et al.; licensee BioMed Central Ltd. 2012