The role of AFB microscopy training in improving the performance of laboratory professionals: analysis of pre and post training evaluation scores

Background Tuberculosis (TB) remains major cause of morbidity and mortality due to any one of infectious agent worldwide. In low income countries, Ziehl-Neelsen sputum smear microscopy is the only cost-effective tool for diagnosis and monitoring of patients on treatment. In order to have efficient AFB microscopy centers, it is imperative to have continuous refresher training for laboratory professionals and strong External Quality Assessment (EQA) system). However, very little data exists as to the effect of in-service training on performance of laboratory personnel in Ethiopia. The objective of this study was to investigate the role of AFB microscopy refresher training on the performance of laboratory professionals. Methods A cross-sectional retrospective study was conducted to appraise theoretical and practical performance of laboratory professionals before and after AFB microscopy training. Theoretical assessment was based on standard questions while practical assessment was based on smear reading of 10 standard slides. Data on eight rounds of a five days training at Adama regional laboratory on AFB microscopy in 2009 was obtained and analyzed using SPSS 16.0 statistical software. Result The pre-training mean score of the theoretical knowledge and practical skills were 61.8% and 75.7%, respectively. The post training mean scores were 84.2% and 89.2% for theoretical knowledge and practical skills, respectively. The increase in mean score of both theoretical and practical assessment was statistically significant (p < 0.0001). Post training mean score of theoretical knowledge was higher among diploma holders trainees than the BSc degree counter parts (p = 0.001). The mean scores on practice before and after training was dependent on participation in previous AFB microscopy trainings (p < 0.0001). Proportions of trainees with both major and minor errors were found to decrease after they were trained. Trainees who have had previous training were found to commit less errors than those who were not participated in previous training (p < 0.0001). Conclusion Training has improved theoretical and practical performance of laboratory professionals. Pre-placement and continuous training irrespective of lab professionals qualification and service year and sustainable EQA are highly recommended to ensure quality of AFB microscopy service.


Background
Tuberculosis remains major cause of morbidity and mortality due to any one of infectious agent worldwide. It is estimated that one third of the worlds population is infected with M. tuberculosis and this result in an estimated eight million new cases annually [1]. Among these new cases 2 to 3 million deaths occur annually [2]. Most of the tuberculosis cases and deaths are reported in developing countries [1]. As per the World Health Organization (WHO) report of 2011, Ethiopia ranks seventh among the 22 tuberculosis high burden countries. According to the report, the estimated prevalence and incidence of all forms TB in Ethiopia was 394 and 261/ 100,000 population, respectively [3].
Case detection through quality assured bacteriology is an essential element of the WHO STOP TB Strategy [4]. In low income countries, Ziehl-Neelsen sputum smear microscopy is the only cost-effective tool for diagnosing patients with infectious tuberculosis and to monitor their progress in treatment [5]. It yields timely results but the sensitivity is low as compared fluorescent microscopy and culture [6,7].
In order to have an efficient TB microscopy centers, it is imperative to have strong External Quality Assessment (EQA) in which laboratory results are checked by an external agency. Moreover, refresher training for laboratory professionals involved in Acid Fast Bacilli (AFB) microscopy at peripheral health facilities is important [8][9][10]. EQA which consists of blind rechecking, panel/proficiency testing and onsite supervision is important to identify errors so that corrective actions can be taken to improve the overall performances of microscopy centers [11]. Training of professionals along with sustainable EQA is important to improve the technical competency of laboratory professionals in every aspects of AFB microscopy [12,13].
Studies in Africa and other different parts of the world have shown that effective and sustainable EQA and training programs are significant in improving the performance of AFB microscopy centers [10,14,15]. In Ethiopia multiple trainings have been conducted but most of the training data were not analyzed to see the overall effect of training on performance of the professionals that can be measured in terms of scores on theoretical and practical assessments. Therefore, the aim of present study was to appraise the performance of laboratory professionals before and after they were trained in AFB microscopy in Oromia region.

Significance of the study
Despite many AFB microscopy trainings, investigation on training data is not commonly practiced in Ethiopia and other countries. Investigation of training data is important as it can provide information on performance of trainees so that appropriate interventions can be planed. Therefore the finding of this study will alert trainers to plan for enhanced quality of AFB microscopy trainings and policy makers in tuberculosis control program to give attention to continuous on job AFB microscopy training towards enhanced case detection and better control of tuberculosis.

Ethical considerations
Before the commencement of the investigation, official approval was obtained from the ethical clearance committee of Oromia region health bureau (IRB number BEFO/ BTFH/1-8/2066). Adama regional laboratory was also officially communicated and permission was obtained to get trainees' information from the training data base.

Methods
Retrospective investigation was conducted in December 2011 to assess effect of AFB microscopy training on performance of laboratory professionals. Data on eight rounds of a five days training for 316 trainees on AFB microscopy in 2009 was obtained and analyzed. All these trainees were enrolled at different times in the eight rounds of training and their evaluation was based on their score on pre and post training assessment on theory and practice. Both theoretical and practical assessments were corrected by the trainers who were senior medical laboratory professionals with at least Bachelors degree in medical laboratory technology and minimum of 5 years of service. All of them were certified with training of trainers (TOT) in AFB microscopy.
Theoretical evaluation was based on standard multiple choice questions corrected out of hundred. Practical assessment was based on smear reading on a set of 10 stained panel slides that were prepared and graded as per the standard procedure [11]. Each trainee was provided with 10 stained slides and the reading was corrected out of 100 in which 10 point was given for correct reading, 5 point for Quantification Error (QE) and 0 point for any type of false reading (Low False Positive, High False Positive, Low False Negative or High False Negative) as per the scoring system of proficiency in reading [16]. The overall evaluation system was based on the following table (Table 1).
Using data collection format, trainees' score on pre and post training theoretical and practical assessments as well as information on their characteristics including sex, qualification, service year, participation in previous training and their facility type was collected from the training data base of Adama regional laboratory. Data was entered and analyzed using SPSS statistical soft ware (version 16) at a statistical significance of p < 0.05. The mean theoretical and practical scores as well as error types with their rates on smear reading before and after training were determined. Mean score before and after training was compared using paired T test. The effect of trainees' characteristics on their theoretical and practical scores as well as error rates before and after training was statistically tested using logistic regression analysis.

Characteristics of trainees
Out of 316 trainees, 259 (82%) were males, 238 (75.3%) with qualification of diploma, 270 (65.5%) with service year ranging from 0 to 3 years, 169 (53.5%) did not participate in similar previous trainings and 236 (74.3%) were from government health institutions. No trainee with BSc degree had service year of eight or more. Analysis of training data has shown that more than half of the trainees enrolled in 2009 have not been participated in previous trainings ( Table 2). Out of total trainees with qualification of BSc degree, 97.4% and 2.6% had service year of 0-3 and 4-7, respectively ( Figure 1).

Theoretical and practical scores of trainees
Data on pre training evaluation have shown that the mean score of the trainees in the theoretical assessment was 61.8% with minimum score of 20%, maximum score of 100%. Pre training practical assessment has shown that the mean, minimum and maximum scores of trainees were 75.7%, 20%, 100%, respectively. Analysis of post training evaluation has shown that the mean, minimum and maximum scores of trainees in theory were 84.2%, 40%, and 100%, respectively. Data on post training evaluation revealed that the mean, minimum and the maximum, scores were 89.3%, 50% and 100%, respectively in practical performance. Post training mean score of trainees in theory and practice was significantly increased (P < 0.0001).

Theoretical and practical scores by characteristics of trainees
Trainees' characteristics were investigated to identify their effect on mean score of theoretical knowledge before and after training. The pre training theoretical mean score was not affected by any of the investigated trainees' characteristics but post training assessment has shown that the mean score of theory was higher among diploma holder trainees than the degree counter parts (p = 0.001) ( Table 3). Pre and post training mean score in practice was not dependent on trainee's sex, qualification and service year (P > 0.05). But it was observed that both pre and post training mean scores in practice was higher among trainees who have had previous training than those who have had no such training (P < 0.0001). It was also found that the mean score of practice after training was higher among participants from private health institutions than the government counterparts (P < 0.011) ( Table 4).

Type of errors committed by trainees
Pre and post training score analysis on type and error rates has shown that majority of trainees committed quantification error but the percentage of trainees with this type of error substantially decrease after they were trained (p = 0.02). Before the training 12.4% of trainees were found to commit minor error of low false negative but after training, the percentage of trainees with this error were found to decrease (p < 0.0001) ( Table 4). Similarly, the microscopic reading result of 5.5% of the trainees  was found to be high false positive before trainings however after the training only 1% of participants were found to commit high false positive result (p = 0.031). Before and after the whole rounds of trainings, it was also observed that 9% and 35% of trainees respectively, have correctly read all slides (P < 0.0001) ( Table 5).

Error rates by trainee's characteristics
Analysis of effect of trainees' characteristics on error rates was carried out after categorizing all types of errors to one group (error), and correct reading in to no error. Accordingly, more males (95%) than females (92.1%) were found to commit at least one error in microscopic reading before training, but the difference was not statistically significant (p = 0.208). Except participation in previous training, other trainees' characteristics were also not associated with smear reading errors. Trainees who have participated in previous training than those who have not, were found to read all slides without error (p < 0.0001) ( Table 6).

Discussion
In 2009 Figure 1 Distribution of trainees by service year and qualification, Adama regional laboratory, 2009. AFB microscopy service in Oromia region. Data on those trainings were retrospectively investigated to measure the effect of training on performance of trainees in the form of post test assessment. Most of the trainees were diploma graduates with maximum service year of 20 years. On the contrary, those participants with qualification of BSc degree have minimal service years and this may be due to the fact that the BSc degree program in Ethiopia has started recently. Nearly half of the trainees enrolled in the current investigation have also had similar training before. Training along with other interventions is very important to strength AFB microscopy centers [13]. The purpose of training is to improve the performance of professionals and this requires analysis of training data as it is important for planning. Our investigation on training data has revealed that trainees have significantly improved their performance both in theoretical knowledge and practical skills. The average pre training proficiency of trainees on smear reading was found to be 75.6% but it was increased to 89% after the training. In routine panel testing, participants are expected to score a proficiency of 80% in smear reading [16]. Proficiency of the trainees was lower than the standard before they were trained but due to the effect of training they were able to attain the standard score. Studies in other countries have also reported similar findings [9,10,14]. Besides improving the proficiency, training can also motivate, update on new information and facilitate ways to share experience so that laboratory professionals can thrive for better services.
In routine AFB microscopy set ups, laboratory professionals have different back grounds in terms of qualification, training, experience and others. Therefore it is important to consider all these characteristics for possible interventions during training as well as analysis of training data. In the current investigation, post training theoretical performance of laboratory professionals with qualification of BSc degree was lower than the diploma graduates. In principle, the performance of BSc degree graduates is expected to be better than diploma graduates. The observed difference may be due to lack of attention resulting from over confidence by BSc degree graduates during trainings.
In pre and post practical assessment, trainees who have participated in previous similar training have shown better performance than those who have not been trained previously. This finding indicates that training and retraining is important to improve practical performance of laboratory professionals. Other studies have also reported Table 4 Pre and post training practical mean score on smear reading by different characteristics of trainees, Adama regional laboratory, 2009  the importance of on job training in improving performance of laboratory professionals involved in AFB microscopy [9,13,15]. In addition to training, it is also important to implement sustainable EQA program to have efficient AFB microscopy centers [13,17]. In the post training practical assessment, it was also observed that participants from private health facilities have performed better than those from government (p = 0.011). This could be due to the fact that trainees from private health facilities might have participated with more attention to achieve better to cope up with strict rules and regulations in private health facilities. In routine AFB microscopy, both minor (QE, LFN, and LFP) and major (HFN, HFP) errors occur at a different rates as a result patient management as well as TB control program can be affected depending on the magnitude and type of error. Our investigation has identified higher rate of QE which is comparable with other study in Mexico [13] but study in India has reported higher rate of LFN than QE [15]. In our study, significant number of participants had previous similar training, but participants in the study of India were fresh graduates. This could be the possible reason for the observed variation.
Proportion of trainees who committed false positive (LFP and HFP) and negative (LFN and HFN) errors were significantly reduced after they were trained, but these errors are not totally avoided. Data from other studies [9,10,15] have also shown similar finding. This could be due to the inherent low sensitivity of Ziehl-Neelsen AFB microscopy, technical problems which can be tackled through continuous refresher training and other interventions. In routine EQA, any major error (HFP or HFN) or any HFP with more than three LFN is not acceptable performance, but lower rates of minor error can be acceptable if the numbers do not demonstrate trends. Unlike QE, false negative and false positive errors have significant impact on patient management as well as the TB control program [16]. Hence, improving the competency of professionals' thorough refresher training, implementation of EQA and other interventions are critically important to reduce or avoid these types of errors.
In our investigation, it was found that participants who have participated in previous similar trainings were found to commit fewer errors than those who had no previous training. This finding may indicate the actual performance in facilities. Ideally, it is important to evaluate the impact of training on performance of microscopy centers. A post training evaluation on impact of AFB microscopy training in Ghana has reported better performance of AFB microscopy centers [9]. The successive improvement of case detection in Oromia region from previous 32% to current 39% (unpublished report) indicates the improved performance of AFB microscopy centers and this could be due continuous refresher trainings and other interventions.
Our study was not without limitations. The main pitfall of our study was the fact that it failed to conduct impact assessment at each health facility where the trainees were based after the training on case detection due to financial constraints.