Preventive medical care in remote Aboriginal communities in the Northern Territory: a follow-up study of the impact of clinical guidelines, computerised recall and reminder systems, and audit and feedback
BMC Health Services Research volume 3, Article number: 15 (2003)
Interventions to improve delivery of preventive medical services have been shown to be effective in North America and the UK. However, there are few studies of the extent to which the impact of such interventions has been sustained, or of the impact of such interventions in disadvantaged populations or remote settings. This paper describes the trends in delivery of preventive medical services following a multifaceted intervention in remote community health centres in the Northern Territory of Australia.
The intervention comprised the development and dissemination of best practice guidelines supported by an electronic client register, recall and reminder systems and associated staff training, and audit and feedback. Clinical records in seven community health centres were audited at regular intervals against best practice guidelines over a period of three years, with feedback of audit findings to health centre staff and management.
Levels of service delivery varied between services and between communities. There was an initial improvement in service levels for most services following the intervention, but improvements were in general not fully sustained over the three year period.
Improvements in service delivery are consistent with the international experience, although baseline and follow-up levels are in many cases higher than reported for comparable studies in North America and the UK. Sustainability of improvements may be achieved by institutionalisation of relevant work practices and enhanced health centre capacity.
A variety of interventions to improve the delivery of preventive services in primary care have been shown to improve coverage and quality of care in North America and the UK.  However, there is little information on the sustainability of the impacts of such systems, or of the impact of such systems in remote settings where poor socio-economic and environmental conditions and relative lack of infrastructure impose particular challenges. Previous relevant reports that address sustainability have been limited by relatively short follow-up periods [2, 3] or do not provide empirical data.  Recent health reform initiatives in two remote regions of the Northern Territory of Australia focussed in part on improving delivery of preventive medical services through the implementation of locally developed best practice guidelines supported by an electronic care planning, recall and reminder system. 
The guidelines were developed by an experienced multidisciplinary team of health professionals.  The incorporation of the guidelines into a computerised clinical information system enabled the generation of individualised care plans with a schedule of services specified according to one of four client age groups. The care plans are required to be updated through the entry of new clinical information at the time of each visit.
The implementation of the guidelines and information system took place in the context of a broader health reform initiative in these two regions. Other aspects of the so-called "coordinated care trials" included the establishment of a health board in each region with the responsibility for planning and purchasing a wide range of health services for the population in each region, and the associated transfer of government health department funds to these health boards. [6, 7] The establishment of the two health boards had to draw on different representative and administrative structures in each region. However, the boards have followed broadly similar trajectories: rather than remaining content to purchase services from the government and other providers, they have each moved to become providers in their own right, moving to take over the community health centres: three on the Tiwi Islands and eight in the Katherine West region.[6, 7] Each has developed community based programs in public health and other areas of prevention and health promotion.
The aim of this paper is to examine the levels of, and trends in, delivery of preventive medical services prior to and following implementation of the guidelines.
Study setting and population
The two trial settings differed in significant respects. The Tiwi Islands cover 7,900 sq km and are located approximately 60 km north of Darwin. The population in 1996 was estimated at 2,032 persons, 90% of whom were Indigenous. The two islands contain three main communities and a number of smaller settlements. The Katherine West region covers 162,000 sq km and includes several large Aboriginal communities and smaller outstations, a number of cattle stations, and a small township. The southernmost community, Lajamanu, is approximately 890 km from Darwin. The population in 1996 was estimated by the Australian Bureau of Statistics to be 2,800 persons although local government estimates are in excess of 3,000. Some 85% of the regional population are Aboriginal, drawn from ten language groups. By Australian standards the health and socio-economic profiles of these regions are very poor, and can be typified by the life expectancy of Indigenous Australians of about 20 years less than the rest of the Australian population.[8, 9]
Study design and sampling
Follow up with repeated measures of a panel sample over a three-year-period was conducted to assess changes in clinical practice following the health reforms described above. Panel samples for the audits were drawn from all consenting trial participants at the end of the first intensive recruitment phase of the trial (1205 consenters from a population of approximately 2000 for Tiwi, and 1340 consenters from a population of approximately 3000 for Katherine West). The target samples of 90 in the population care plan group (all age groups together) for each trial were designed to provide an estimate within 10% of the true population rate at a 95% confidence level for each trial site. The panel samples were followed across all five audit periods reported. A stratified random sample design was used to ensure the sample included participants from each community and care plan age group. At the time of the 3 year audit the samples were supplemented by further random sampling of consenting participants to make up for movement between age groups and loss-to-follow-up. The number of participants at each audit is detailed in Table 1.
The clinical guidelines, electronic systems to support these guidelines, and associated staff training were introduced progressively starting in late 1998. There were three aspects of the intervention that were expected to possibly impact directly on clinical practice. First, each health centre received evidence based clinical guidelines targeted at four categories of population (0–3, 4–15, 16–49, 50+ age groups).  Details of guidelines are in Figure 1 and Figure 2. These standard guidelines (care plans) were developed by a multidisciplinary team of health professionals using National Health and Medical Research Council "Guidelines for the Development and Implementation of Clinical Practice Guidelines" as well as adopting local prevalence data and relevant social and cultural information. Guidelines were incorporated into a computerised information system to assign preventive services to individuals within population groups with respect to basic measurements, clinical examination and screening, immunisations / deworming, and counselling.
Second, a computerised Coordinated Care Trial Information System (CCTIS) was installed in community health centres. This computer system maintained a database of all consented trail participants and their preventive services. CCTIS was designed to be used in 'real time' at client consultation, in order to serve as an effective reminder system in the delivery of health services according to the embedded age-group-specified guidelines. Two education officers were employed to provide on-site training to health centre staff in application of the software packages.
Third, findings from the audits were fed back to, and their interpretation discussed with, health centre clinicians and managers. This usually occurred within a few months of the conduct of each audit, and within a timeframe which allowed a response to feedback to influence findings of the next audit.
The audits were conducted on the records of a sample of the clients who had consented to make their health records available for the trial evaluation. The audits followed the detailed guidelines (in Figure 1 and 2) and were conducted on five occasions over a three year period: baseline; and at approximately 6, 12, 24 and 36 months after baseline. Baseline audits were conducted around the same time as the introduction of the guidelines, and recorded services delivered over the preceding 12 months – i.e. before the introduction of the guidelines could have had effect.
Both the paper record and the electronic record for each client were checked for the most recent record of delivery of each service according to a schedule specified in the best practice guidelines. A service was recorded as delivered if there was a record of delivery within one month of the due date for services scheduled before 12 months of age, within 2 months for services scheduled at 12 months and up to 4 years of age, within 6 months for services scheduled at 4 years and up to 16 years, and within the past year for services scheduled for 16 years and older, except where recommended service intervals were more than one year (pap smear, mammography, pneumococcal and tetanus/diphtheria vaccine).
Information on attendance at health centres within the last 6, 12, or 24 months for each client, depending on age group, was also collected at year 1, year 2 and year 3 audits in order to determine the proportion of the population attending the health centre within specified times, and the extent of possible missed opportunities for delivery of preventive services at times when clients were attending for other reasons.
The outcome of interest was adherence to guidelines in terms of delivery of scheduled services. The assessment of delivery of each specific service at each audit period was defined as a dichotomous variable (delivered/not delivered). The assessment of delivery of services for each client at each audit period was defined as a continuous variable (delivered services as a percentage of all scheduled services for that client). An example is given to demonstrate the calculation as follows: the scenario is an 11-month-old boy undergoing a baseline audit. According to the clinical guidelines, the most recently scheduled services for this infant are weight measurement, haemoglobin test, hearing examination, ear disease check, and deworming at the age of 9 months. If any one of above services was done within one month of due date (i.e. when he was 9 months old), the service was labelled as delivered, otherwise as undelivered (service based dichotomous variable). If 3 of these 5 scheduled services were carried out, the percentage of delivered services for the boy was 60% (3/5) (client based continuous variable).
To determine the effectiveness of interventions over the total follow up period, multilevel analysis was conducted.  In view of our panel data with 5 rounds of audits, inherent dependency structure existed in repeated measures from the same individual, and clients were nested within community health centres. Therefore, the client and the health centre were defined respectively as level one and level two units.
The parameter of interest was the regression coefficient of the time variable (value of 0, 0.5, 1, 2, 3 for each audit respectively), indicating the average magnitude of improvement in delivery of scheduled services per year from baseline. The p value of the regression coefficient was used to assess whether there was a significant increasing or decreasing trend in delivery of services from year to year across the full period of follow-up.
A logistic (random effects) model was used for the dichotomous outcome for each service in each care plan group and regression coefficients were transformed to odds ratios (OR). A linear random effects model was employed for client-based continuous outcome (client's percentage of delivered services). Separate analyses were conducted for each care plan group and each community. The means of the continuous outcome measures (and associated confidence intervals) were also calculated for each audit (see data tables for each figure).
All multilevel analyses were performed using Stata version 7.0 (commands xtlogit and xtreg with defined iis for level two and tis for level one identifiers) and were adjusted for recent attendance at the health centre.
Sample by age category and audit interval
The total sample at baseline included 265 clients (Table 1).
Loss to follow-up
Loss to follow-up was most marked between the year 2 and year 3 audits and in the 0–3 and 4–15 age groups (Table 1). In the 0–3 age group, and to a lesser extent in the 4–15 group, the main reason for loss to follow up was through aging of the cohort. In the older age groups the main reasons were movement out of the region, death, and lost records.
Attendance at health centres
Of clients in the 0–3 year sample, between 89 and 97% had a record of attendance at the health centre within the 6 months preceding each of the year 1, year 2 and year 3 audits. For the 4–15 year sample the corresponding figures for the 12 months preceding each audit were between 87 and 94%. The corresponding figures for the 16–49 year sample were consistently 96%, and for the 50+ sample between 96 and 98% for the 24 months preceding each audit. Data on attendance were not collected at the first two audits.
Trends in service delivery over study period
For each care plan age group there was a significant improvement in percentages of delivered services delivered between baseline and the audits at 1 year and/or 2 years (Figure 3 and Table 6). For the 0–3 age group the improvement in service delivery was sustained to the 3 year audit. However, for the other three age groups there was a decline in service delivery by year 3. In the case of the 16–49 year age group this decline had begun at the time of the 2 year audit, and by year 3 had dropped below baseline.
Proportion of scheduled services delivered to 0–3 age group, and trends
The increase in the proportion of services delivered to the 0–3 age group showed a highly significant trend for all services together across the study period (Figure 3 and Table 6). The most marked improvement occurred between the 6 month and year 1 audits, and improvement was sustained to year 3. Improving trends occurred between baseline and year 1 for almost all services (Table 2). There was wide variation in the level of delivery of different services at baseline and across the study period (Table 2). Those services with high baseline levels of coverage had less room for improvement and as expected were less likely to show improvement. Services starting from a lower baseline were more likely to show significant improvement. In general services requiring basic measurements or procedures were delivered at higher levels than those requiring more advanced clinical skills.
Proportion of scheduled services delivered to 4–15 year age group, and trends
The same general pattern of higher service delivery levels at years 1 and 2, with levels at 6 months well below baseline and at year 3 similar to baseline, was seen for almost all individual services (Table 3). No individual services showed a significant improving trend across the study period, except for deworming. There was wide variation in the level of delivery of different services at baseline and across the study period. Overall, there was a significant increasing trend for delivery of services across the study period (Figure 3 and Table 6).
Proportion of scheduled services delivered to 16–49 year age group, and trends
Almost all individual services followed the same general trend with higher levels of service delivery recorded at the year 1 and year 2 audits, with a decline at year 3 to levels close to or below baseline (Table 4). Notable exceptions are pap smears where the highest levels of delivery were recorded at the year 3 audit. As for other age groups, the levels of service delivery varied widely between different services, with counselling services being recorded at lower levels than basic measures. Overall, there was no significant trend for delivery of services across the study period (Figure 3 and Table 6).
Proportion of scheduled services delivered to 50+ year age group, and trends
Most individual services followed the same general trend with higher levels of service delivery recorded at the year 1 and year 2 audits, with a decline at year 3 (Table 5). Notable exceptions to this pattern included the adult immunisations where levels were highest at year 3. Counselling services tended to be reported at lower levels than basic measures, and immunisations at higher levels. No individual services showed a significantly improving trend across the study period other than BMI and pneumococcal vaccination. Overall, there was a significant increasing trend for delivery of services across the study period (Figure 3 and Table 6).
Service delivery and trends by trial region and community
Tiwi communities tended to have higher levels of service delivery at baseline and for the duration of follow-up (Figures 4 and 5, Table 7 and 8). The general pattern of improvement in levels of service delivery from baseline to year 1 and/or year 2, with a decline at year 3, is seen for most communities. However, two communities in Katherine West buck this trend (Figure 4). In both there is a decline in service delivery from year 1 to year 2, and a subsequent increase to year 3. Overall, 2 of the 3 Tiwi communities, and one of the Katherine West communities, showed a significantly increasing trend in service delivery across the study period.
The implementation of care coordination through the care planning process, best practice guidelines and computerised recall and reminder systems was associated with a general improvement in delivery of specified preventive services in the study population. However, the sustainability of the improvements was variable. While the overall improvement in service delivery at year 3 had dropped below levels achieved at years 1 and 2, service delivery was still significantly higher than at baseline for 3 out of the 4 population age groups.
In general the services for which the highest levels of service delivery were achieved were those supported by the best quality evidence and the strongest recommendations  (eg immunisation in children and older adults). The clearest exceptions to this trend were mammography in the 50+ women and counselling of adults on alcohol use, smoking, exercise, and home accidents, where there were low levels of service delivery in the face of particularly strong recommendations. Counselling services may be more likely to not be recorded when they are delivered, and this may partly explain low recorded levels of service. However, a low level of recording of services is a problem in itself, in that it impedes effective, efficient and coordinated service delivery.
It is remarkable that the delivery of preventive services in these remote health centres is in most cases as good as or better than reported by North American and UK studies with comparable data. [12–22] Possible explanations may be a recent wider acceptance of the value of evidence-based preventive care, and the strong public health orientation of health workers in the environment in which these trials were conducted.
The patterns of service delivery over time raise questions about the sustainability and nature of systems required to support a strategy that aims to enhance delivery of key prevention and health promotion services across all age groups. It is apparent from the data that these systems had some impact on improving delivery of care in the first one to two years following the introduction of the care coordination model, and that subsequently the systems to maintain the improvement in practice failed, or that the practitioners responded less effectively to these systems. Interviews with health centre staff and health service management and community specific patterns of service delivery suggest that the reasons include:
discontinuities in staffing
lack of ongoing training and support for the care coordination process
lack of development of relevant work practices
lack of clear priorities to guide delivery of care plan services
lack of capacity to monitor service activity through timely analysis of service data
reluctance on the part of patients to accept offered services when presenting for acute, non-scheduled services.
The implementation of guidelines and computerised recall and reminder systems will not of itself lead to sustained improvement in delivery of preventive services. Their impact clearly depends on effective use by health centre staff. The introduction of recall for preventative services resulted in many items listed as overdue and consequently required development of a strategy to integrate active and opportunistic recall for these services with acute care and other activities. It demanded attention to the recording of greater numbers of services in the systems of electronic and paper records. Health centres appear to have not been able to systematically develop work practices in such a way as to efficiently cope with the additional demands. Inadequate training and work practice support for the use of care plans beyond initial induction in their basic use appears therefore to have limited their impact.
The high levels of attendance at health centres for all age groups indicates that low service levels were not the result of poor access, but rather of clients not being offered, or not taking up preventive services when they attended the centres. Client behaviour, including an unwillingness to accept care plan services offered opportunistically, is almost certainly a factor in sustaining delivery of preventive services to adults. For those with health problems, there may be a great number of services due on any given occasion. Clients were reportedly often unhappy to wait for a second consultation, particularly for preventive activity unrelated to the initial reason for presentation. However, these difficulties may not be insurmountable, as the trends to improvement noted in the first and second years of the trials have indicated.
The low levels of services seen at six months for the 4–15 year age group was a result of school screening programs being conducted later than usual at some sites, and after the audit. Most services to this age group are provided as part of a school screening program in these communities. Most services scheduled on the care plans for this age group are also specific to children age 5, 10 or 15 years, and were therefore only relevant to a proportion of the sample. The numbers on which most estimates are based for this sample are therefore relatively small, and the estimates unstable.
Systematic differences between the original sample and the sample of additional participants recruited for the year 3 audit is unlikely to be an explanation for the declining trend between year 2 and year 3. The mean level of service delivery for those in the original sample was very close to that of the overall sample for both adult samples at the year 3 audit (see Table 6). While the corresponding difference for the 4–16 age group was larger, as stated above, the estimates for this group are relatively unstable.
Finally, sustained improvement in the delivery of preventive care services will depend on the institutionalisation of the processes of care at health centre level, and the development of capacity to support these processes at the level of management, policy and training supports.[23, 24] Institutionalisation could be supported by linking care plan protocols more firmly to work practices and work flow, and specifically supporting opportunistic delivery of preventive services. Increased involvement of clients in reviewing preventive care plans, and thereby enhancing their understanding of the value of preventive services, may also increase service uptake. However, sustainable improvement is unlikely to follow simply from practitioner and health centre response to a set of protocols. Management needs to actively support strategies for sustained delivery of preventive services.
Hulscher MEJL, Wensing M, van der Weijden T, Grol R: Interventions to implement prevention in primary care (Cochrane Review). In: The Cochrane Library. 2002, Oxford Issue 3: Update Software
Buntix F, Knottnerus JA, Essed GGM, Crebolder HFJM: Long-term effect of feedback and peer comparison on the sampling quality of cervical smears – a randomized controlled trial. European Journal of Cancer Prevention. 1995, 4: 153-157.
Carney PA, Dietrich AJ, Keller A, Landgraf J, O'Conner GT: Tools, teamwork, and tenacity: an office system for cancer prevention. J Fam Pract. 1992, 35: 388-394.
Dietrich AJ, Woodruff CB, Carney PA: Changing office routines to enhance preventive care. Arch Fam Med. 1994, 3: 196-183. 10.1001/archfami.3.2.176.
Weeramanthri TS, Connors C, O'Leary S, Yarmirr D, Wright J, Bell A: Chronic disease guidelines and the Indigenous Coordinated Care Trials. Australian Health Review. 2002, 25: 1-10.
Robinson G, Bailie R: Tiwi coordinated care trail final local evaluation report. Darwin: Centre for Social Research, Northern Territory University. 2000
d'Abbs P, Togni S, Fitz J, Bailie R, Adsett S, Del Fabbro L, Wales N, Deeble J: Katherine West coordinated care trail final report. Darwin:. 2000
Australian Institute of Health and Welfare: Australia's Health 2000: the seventh biannual health report of the Australian Institute of Health and Welfare. Canberra:. 2000
Australian Bureau of Statistics and Australian Institute of Health and Welfare: The health and welfare of Australia's Aboriginal and Torres Strait Islander people. Canberra:. 1997
Goldstein H: Multilevel statistical models. London: Edward Arnold. 1995
National Preventive and Community Medicine Committee of the Royal Australian College of General Practitioners: Guidelines for preventive activities in general practice. Australian Family Physician. 2002, 31.
Battista RN, Williams JI, Boucher J, Rosenberg E, Stachenko SJ, Adam J, Levinton C, Suissa S: Testing various methods of introducing health charts into medical records in family medicine units. CMAJ. 1991, 144: 1469-1474.
Becker DM, Gomez EB, Kaiser DL, Yoshihasi A, Hodge RH: Improving preventive care at a medical clinic: how can the patient help?. American Journal of Preventive Medicine. 1989, 5: 353-359.
Belcher DW: Implementing preventive services. Success and failure in an outpatient trial. Archives of Internal Medicine. 1990, 150: 2533-2541. 10.1001/archinte.150.12.2533.
Cheney C, Ramsdell JW: Effect of medical records' checklists on implementation of periodic health measures. American Journal of Medicine. 1987, 83: 129-136.
Frame PS, Zimmer JG, Werth PL, Hall WJ, Eberly SW: Computer-based vs manual health maintenance tracking. A controlled trial. Archives of Family Medicine. 1994, 3: 581-588. 10.1001/archfami.3.7.581.
Fullard E, Fowler G, Gray M: Promoting prevention in primary care: controlled trial of low technology, low cost approach. British Medical Journal Clinical Research Edition. 1987, 294: 1080-1082.
Litzelman DK, Dittus RS, Miller ME, Tierney WM: Requiring physicians to respond to computerized reminders improves their compliance with preventive care protocols. Journal of General Internal Medicine. 1993, 8: 311-317.
McDonald CJ, Hui SL, Smith DM, Tierney WM, Cohen SJ, Weinberger M, McCabe G: Reminders to physicians from an introspective computer medical record. A two-year randomized trial. Annals of Internal Medicine. 1984, 100: 130-138.
McPhee SJ, Bird JA, Fordham D, Rodnick JE, Osborn EH: Promoting cancer prevention activities by primary care physicians. Results of a randomized, controlled trial. JAMA. 1991, 266: 538-544. 10.1001/jama.266.4.538.
Ornstein SM, Garr DR, Jenkins RG, Rust PF, Arnon A: Computer-generated physician and patient reminders. Tools to improve population adherence to selected preventive services. Journal of Family Practice. 1991, 32: 82-90.
Szczepura A, Wilmot J, Davies C, Fletcher J: Effectiveness and cost of different strategies for information feedback in general practice. British Journal of General Practice. 1994, 44: 19-24.
Bossert T: Can they get along without us? Sustainability of donor-supported health projects in Central America and Africa. Soc Sci Med. 1990, 30: 1015-1023. 10.1016/0277-9536(90)90148-L.
Shediac-Rizkallah MC, Bone LR: Planning for the sustainability of community-based health programs: conceptual frameworks and future directions for research, practice and policy. Health Educ Res. 1998, 13: 87-108.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6963/3/15/prepub
The cooperation of the Katherine West and Tiwi Health Boards in the evaluation is appreciated.
The authors were contracted as independent evaluators of the two trials that are the subject of this report.
RB designed the clinical audit method, oversaw the audit data collection and analysis, and prepared the manuscript for publication.
ST assisted with audit design, took primary responsibility for data collection in one study region, and took a lead role in data analysis.
DS made a major contribution to the data analysis.
GR and Pd'A played a major role in project leadership and design, in data collection and interpretation of findings.
All authors contributed to, have read and approved the final manuscript.
About this article
Cite this article
Bailie, R.S., Togni, S.J., Si, D. et al. Preventive medical care in remote Aboriginal communities in the Northern Territory: a follow-up study of the impact of clinical guidelines, computerised recall and reminder systems, and audit and feedback. BMC Health Serv Res 3, 15 (2003). https://doi.org/10.1186/1472-6963-3-15