Skip to content


  • Study protocol
  • Open Access
  • Open Peer Review

Study protocol: Evaluating the impact of a rural Australian primary health care service on rural health

  • 1Email author,
  • 1,
  • 1,
  • 1,
  • 2 and
  • 2
BMC Health Services Research201111:52

  • Received: 3 February 2011
  • Accepted: 1 March 2011
  • Published:
Open Peer Review reports



Rural communities throughout Australia are experiencing demographic ageing, increasing burden of chronic diseases, and de-population. Many are struggling to maintain viable health care services due to lack of infrastructure and workforce shortages. Hence, they face significant health disadvantages compared with urban regions. Primary health care yields the best health outcomes in situations characterised by limited resources. However, few rigorous longitudinal evaluations have been conducted to systematise them; assess their transferability; or assess sustainability amidst dynamic health policy environments. This paper describes the study protocol of a comprehensive longitudinal evaluation of a successful primary health care service in a small rural Australian community to assess its performance, sustainability, and responsiveness to changing community needs and health system requirements.


The evaluation framework aims to examine the health service over a six-year period in terms of: (a) Structural domains (health service performance; sustainability; and quality of care); (b) Process domains (health service utilisation and satisfaction); and (c) Outcome domains (health behaviours, health outcomes and community viability). Significant international research guided the development of unambiguous reliable indicators for each domain that can be routinely and unobtrusively collected. Data are to be collected and analysed for trends from a range of sources: audits, community surveys, interviews and focus group discussions.


This iterative evaluation framework and methodology aims to ensure the ongoing monitoring of service activity and health outcomes that allows researchers, providers and administrators to assess the extent to which health service objectives are met; the factors that helped or hindered achievements; what worked or did not work well and why; what aspects of the service could be improved and how; what benefits have been realised and for whom; the level of community satisfaction with the service; and the impact of a health service on community viability. While the need to reduce the rural-urban health service disparity in Australia is pressing, the evidence regarding how to move forward is inadequate. This comprehensive evaluation will add significant new knowledge regarding the characteristics associated with a sustainable rural primary health care service.


  • Health Service Utilisation
  • Primary Health Care Service
  • Community Satisfaction
  • Rural Health Service
  • Comprehensive Primary Health Care


The Australian rural health service context

Australia is a vast continent with a population of approximately 22 million people of which 3.5 million are spread across 1500 small rural and remote communities and 7.5 million square kilometres. Isolated rural communities throughout Australia are experiencing demographic ageing, an increasing burden of chronic diseases[1], and de-population as families move to larger cities. In addition, many of these rural communities are struggling to maintain viable and comprehensive health care services due to lack of service infrastructure, transport difficulties[2], and health worker shortages due to high levels of staff turnover and difficulties in recruiting new health workers[3]. In many cases rural communities are either foregoing care, travelling to larger regional centres or depending on irregular visiting services. As a consequence, Australia's rural and remote communities face significant health service disadvantages compared with metropolitan regions[4].

The vast distances separating small communities throughout rural Australia provide enormous challenges for authorities responsible for servicing population health needs as there are conflicts between ensuring operational efficiency and cost-minimisation, whilst maintaining effective and equitable delivery of accessible health services. Traditional urban health service models are proving to be unsustainable. Undoubtedly, there is no "one-size-fits-all" solution to meeting the diverse health needs of rural Australian residents and the range of service models needed is likely to vary between communities. Hence it is necessary to investigate models of health service delivery to ensure equitable access to care and reduce the health differential between rural and metropolitan people[5].

Research has shown that a primary health care approach yields the best health outcomes in situations characterised by limited resources[6]. As a result, the need for sustainable comprehensive primary health care, characterised by multi-disciplinary team approaches, is urgent in rural communities. In response, a wide variety of innovative health care models have evolved and been trialled in rural and remote areas[3]. However, whilst research has identified the requirements for sustainable rural and remote primary health care services[7], few rigorous longitudinal evaluations have been conducted to systematise them over time; assess their transferability to other regions[8]; assess how health services continually evolve to address ongoing changes in the external environment; monitor the effect of comprehensive primary health care on health service utilisation behaviour and health literacy of their communities[9]; or better understand how or when primary health care services can redress the poorer rural health status.

Given these gaps in knowledge it is vital to understand which rural health services 'work well, where and why' and are sustainable over the long-term in order to inform rural health service policy, assist with planning sustainable health services in other rural communities[10] and contribute to the equitable delivery of health care services that are likely to bring about improved health outcomes.

This paper describes the study protocol of a comprehensive longitudinal evaluation of a successful primary health care service located in a small rural Australian community to assess its performance, sustainability over time, responsiveness to changing community needs and health system requirements, and its impact on community health behaviours, health outcomes and community viability.


Setting: The Elmore Primary Health Service (EPHS)

The EPHS is located at Elmore in central Victoria (Australia) 170 kilometres from the capital city, Melbourne. While the population in Elmore in 2006 was only 693[11], the EPHS provides care to nearly 3,000 patients from surrounding districts up to 125 kilometres from Elmore. The EPHS is a single-entry point, comprehensive primary health care model formed in 2001 by a partnership between the private medical practice and the local, publicly funded, community health services after the closure of the local hospital and loss of the local doctor in 1994. The model combines health care, community coordination and outreach services, is financed by public and private funding and is delivered by a multidisciplinary team of, amongst others, doctors, nurses, physiotherapists, psychologists and podiatrists[12]. This service has grown in contrast to the rural national trend of service closures[13] and can be an exemplar for other communities to emulate.

Evaluation framework: domains, components and indicators

The longitudinal evaluation framework evolved from Donabedian's quality of care paradigm linking structure, process and outcome[14] and a conceptual framework for primary health care performance assessment developed originally by Sibthorpe[15], as reported in another publication[16]. This evaluation aims to examine the health service over a six year period in terms of:
  1. a.

    Key structural domains:

  • Health service performance characteristics;

  • The sustainability of the health service organisation and function; and

  • Quality of care that the service provides across the health promotion, treatment and rehabilitation spectrum.

  1. b.

    Key process domain:

  • The effect of the service on health service utilisation and satisfaction.

  1. c.

    Key outcome domain:

  • The effect of the service on health behaviours, health outcomes and community viability.

A key aspect of the evaluation was the identification and development of unambiguous sentinel indicators for each component of these key domains that can be reliably and validly operationalised so that data can be routinely collected and the health service monitored over time. The recent health systems reform process being undertaken in Australia has highlighted the difficulties in developing appropriate and valid accountability and performance benchmarks[17]. Despite potential indicators being easily identified on the conceptual level, it can be extremely difficult to operationalise them in forms for which data can be routinely and unobtrusively collected whilst being beneficial to health outcomes and health services. Hence our indicator selection was guided by significant health services research conducted by the Canadian Institute for Health Information[18]; the Australian National Health Performance Committee[19]; the Australian National Health and Hospital Reform Commission[17]; the Royal Australian College of General Practitioners[20]; the Australian Institute of Health and Welfare [21]; and Wakerman et al[22].

Drawing on this evidence, sentinel indicators for assessing the structural domains: health service performance (Table 1); sustainability (Table 2); and quality of care (Table 3), were developed in accord with the evaluation framework. As described in each table, each domain is divided into components, for which sentinel indicators were selected based on their technical merits and validity as identified in the research literature and their applicability in the health service setting[23] - that is, the data can be routinely collected, validated, and extracted reliably from the data sources identified for this study (health service record audits, community surveys, interviews and focus group discussions).
Table 1

Health Service Performance Domain


Reason for selection

Sentinel indicator items selected

Data source



The importance of geographical proximity to services; and timely and affordable access to routine and emergency care.

• Distance to nearest GP


AIHW [21]


• Distance to usual GP



• Operating hours


Campbell S [26]


• After hours services


CIHI [18]


• Availability of emergency care


RACGP [20]


• Availability of bulk-billing


AIHW [21]


To assess the comprehensiveness of services in dealing with a whole of health approach to health care.

• Availability of disease prevention, health promotion, early identification, sub-acute, acute and rehabilitation services.


AIHW [21] CIHI [18] NHHRC [17] RACGP [20]


• Availability of female GPs


AIHW [21]


To assess extent to which a service's disease prevention interventions are achieving the desired results within an expected timeframe.

• Immunisation coverage


NHHRC [17]


• Cervical cancer screening coverage



• Use of chronic disease registries for diabetes mellitus, asthma and hypertension for timely planning and disease management.


Campbell S [26]; CIHI [18]


To assess the extent to which respectful care is provided that promotes dignity, privacy, safety and community empowerment.

• Service response to cultural and other specific needs of people utilising the service


RACGP [20]


• Community input into service planning

Interviews Focus group

NZRCGP [27], RACGP [20]


The importance of providing uninterrupted, coordinated care across programs, providers, and organisations over time

• Choice of GP or nurse


RACGP [20]


• Age-specific health assessments


CIHI [18]


• 45 year old health check


• Use of integrated care plans for diabetes, asthma and depression



• Use of recall and reminder systems


RACGP [20]


The importance of achieving desired results with the most cost-effective use of service resources.

• Electronic billing system


RACGP [20]


• Electronic medical records


RACGP [20]

Table 2

Sustainability Domain


Reason for selection

Sentinel indicator items

Data source



The importance of having a workforce that is appropriate in number, volume and distribution and is responsive to emerging needs.

• Staff profile- Numbers and FTE.

• Staff length of stay

• Succession planning

• CPD activities




AIHW [21] Wakerman[22] RACGP [20]


Efficient and effective co-ordination between providers and between services is essential for continuity of care and service sustainability.

• Centralised electronic medical records

• Care integrated with external agencies and mainstream programs


Wakerman et al [22]; CIHI[18];


Infrastructure and ICT needs to be appropriate to the service, its catchment population and monitoring and reporting requirements.

• Uptake of Information and Communication Technology (ICT)


CIHI [18]; Wakerman et al [22]


Financing and provider remuneration should be appropriate, sustainable and clearly identified within program budgets to maximise service efficiencies and adequate to meet identified community health needs.

• Funding sources: public, private, other

• Service providers' remuneration methods



Wakerman et al [22] CIHI [18]

Governance, management and leadership

Good governance and leadership have been identified to be integral to service sustainability

• Governance structure and processes and a risk management plan in relation to service sustainability need to be clearly defined, implemented and reviewed.


RACGP [20], Wakerman et al [22]


• Level of accreditation


NHPC [19]

Table 3

Quality of Care Domain


Reason for selection

Sentinel indicator items

Data source


Primary prevention


Cervical cancer screening

Cervical smear tests improve the early detection and treatment of cervical cancer and improve survival and quality of life.

Cervical cancer screening coverage


NHHRC [17]


Immunisation is a very important public health measure that can prevent the spread of common infectious diseases that cause significant morbidity and mortality.

Immunisation coverage:


AIHW [21], NHHRC [17]


   • Children


   • Older adults (65+years)


   • Aboriginal and Torres Strait Islanders


Secondary prevention


Recording of modifiable risk factors in medical records

Many common chronic diseases are preventable if the risk factors are identified and managed so that there can be improved health status and reduced health inequalities and need for health care.

   • Smoking status


AIHW [21] Broemeling [10]


   • Body Mass Index


   • Alcohol use


   • Blood pressure


Treatment goals and outcomes



Minimizing or eliminating inappropriate prescribing improves quality of care and health outcomes.

Safety - Risk management plan and use of medication alerts

Audit Audit

RACGP [20]

Diabetes mellitus

The percentage of patients with diabetes mellitus for whom the ideal treatment goal of HbA1c* < 7% is met.

Management of diabetes mellitus (HbA1c readings)


CIHI [18] NHHRC [17]

* HbA1c is a test that measures the amount of glycosylated haemoglobin in the blood - levels below 6% are normal; a person with diabetes mellitus should aim to keep their levels below 7% to reduce the risk of diabetic complications.

To evaluate the impact of the EPHS on health service utilisation, community viability and satisfaction in Elmore and its hinterland, sentinel indicators include: community participation within health service planning; community experience of and satisfaction with health services; indicators of population composition and growth, employment trends, and health service multiplier effects in the local economy[24].

Data sources

The quantitative data for indicators of health service performance, sustainability and quality are being obtained by an annual comprehensive health service record audit and biennial community surveys. These quantitative data will be analysed using descriptive statistics that monitor trends over the period of the study.

The comprehensive health service audit is examining medical record data, billing and financial data, human resources records, registry data (e.g. Australian Childhood Immunisation Register and the Victorian Cervical Cytology Register) and government reports to the practice (e.g. Practice Incentive Payments and Service Incentive Payments). All data are being extracted by an EPHS practice nurse and provided in a de-identified and aggregated format to the external university research team to ensure utmost privacy and confidentiality of patient and health service records.

Further quantitative data are to be obtained through a biennial community survey that is to be delivered to all residents in the catchment area and is designed to obtain information in relation to health service utilisation, satisfaction and need, in addition to health risk behaviours.

The qualitative data that explores sustainability of the service and the impact of the EPHS on the local community are being obtained by in-depth interviews with key stakeholders and document analysis. These data will be transcribed and thematically analysed and triangulated with the quantitative data for validation and clarification of the data. As new understandings emerge through this iterative and reflective process, new indicators will be developed or existing indicators may be modified.

Knowledge translation

Engagement with the local community is essential for this study as the current health service grew from community activism in collaboration with the local doctor and regional community health service. A detailed communication strategy, including community forums and articles in the local newspaper, will ensure that all stakeholders and community residents are kept informed of the purpose and conduct of the evaluation study. The conduct of the research and dissemination of findings will be guided by a Reference Group that will be comprised of representatives from the local community (leaders, businesses and community groups), regional health authorities, and general practice support agencies (e.g. Divisions of General Practice and general practice education and training services). The research team will provide information to the community through plain language summaries, oral presentations and presence at key community events.


The study has obtained ethics approvals from Monash University Human Research Ethics Committee (CF08/0419- 2008000176; CF08/0238 - 2008000089; CF08/2434 - 2008001256; CF10/2540 - 20100001423).


Ongoing monitoring and evaluation of successful rural health service models is needed to provide sound empirical evidence of what makes a primary health care service sustainable and effective in a rapidly changing health policy environment in order to provide an evidence base that can inform health service policy makers and other rural health services.

This iterative evaluation framework and methodology aims to ensure the ongoing monitoring of service activity and health outcomes associated with the service that allows researchers, health service providers and administrators to assess the extent to which health service objectives have been met; the factors that have helped or hindered achievements; what worked or did not work well and why; what aspects of the service could be improved and how; what benefits have been realised and for whom; the level of community satisfaction with the service; and the impact of a health service on community viability.

This evaluation focuses on health service performance and capacity rather than focussing on how well a particular disease is treated. This raises challenges in identifying appropriate indicators as primary health care organisation and function is complex, dynamic and composed of both measurable and unmeasurable elements[23] that are subject to many external influences that can impede performance, sustainability and quality of care. The scope and breadth of ideal indicators for the components of each domain in the research literature is more extensive than those selected for this study. The selection of indicators was limited principally by the types of data items that are stored in paper records and electronic databases that are reliable, valid and readily extracted.

This study also provides an opportunity to assess, over time, the effect of locally available health promotion, preventive and early intervention programs on health literacy and health status of a small rural community. As it is possible to relate the EPHS catchment to a geographically circumscribed hinterland, this evaluation may examine the associations between sustainable local health services and community viability and prosperity through examining community satisfaction, growth, employment, and multiplier effect indicators. Hence, building on previous studies that highlighted the significance of rural hospitals and health services to the local economy and fortune of rural communities [25], the findings from this study may assist health authorities and other small rural communities to benchmark what services are appropriate and successful.

The danger in not undertaking such comprehensive longitudinal research to identify "what works well, where and why" is the risk of widening health differential between urban and rural communities; increasing costs of health care treatment to individuals and to government; and reduced implementation of best practice care equitably to all communities.

The need to provide equitable access to quality primary health services for Australia's rural communities is essential and urgent. However, the evidence base for rural health service design remains sparse. This longitudinal evaluation will contribute significant knowledge to how a primary health service can be evaluated and play a role in reducing the Australian rural-metropolitan health disparity.



This study is being funded by Australian Rotary Health for the period 2008 - 2010 and the Australian Government Department of Health and Ageing for the period 2011-2013. The funding bodies had no role in the conception or development of the study design, the writing of the manuscript or the decision to submit it for publication. The authors wish to acknowledge the contributions of Karly Smith, Karen Riley and Judith Jones in the development of the study design and ethics applications.

Authors’ Affiliations

Monash University School of Rural Health, PO Box 666, Bendigo, Victoria, 3552, Australia
Elmore Primary Health Service, 46 - 48 Jeffrey Street, Elmore, Victoria, 3558, Australia


  1. National Health and Hospitals Reform Commission: A Healthier Future For All Australians - Final Report of the National Health and Hospitals Reform Commission - June 2009. 2009, Commonwealth of AustraliaGoogle Scholar
  2. Chenoweth L, Stehlik D: Using technology in rural practice - local area coordination in rural Australia. Rural Social Work. 2002, 7 (1): 14-21.Google Scholar
  3. Humphreys J: Health service models in rural and remote Australia. The New Rural Health: An Australian Text. Edited by: Wilkinson D, Blue I. 2002, Oxford University Press, 273-296.Google Scholar
  4. Australian Institute of Health and Welfare (AIHW): Whose health? How population groups vary. Australia's health 2010. 2010, Canberra: AIHW, 227-280.Google Scholar
  5. Starfield B: Primary care: is it essential?. Lancet. 1994, 344: 1129-1133. 10.1016/S0140-6736(94)90634-3.View ArticlePubMedGoogle Scholar
  6. Starfield B: Primary Care: Balancing health needs, services and technology. 1998, Oxford: Oxford University PressGoogle Scholar
  7. Wakerman J, Humphreys J, Wells R, Kuipers P, Jones J, Entwistle P, Kinsman L: Features of effective primary health care models in rural and remote Australia: a case-study analysis. Medical Journal of Australia. 2009, 191 (2): 88-91.PubMedGoogle Scholar
  8. Kuipers P, Humphreys J, Wakerman J, Wells R, Jones J, Entwistle P: Collaborative review of pilot projects to inform policy: A methodological remedy for pilotitis?. Australia and New Zealand Health Policy. 2008, 5 (1): 17-10.1186/1743-8462-5-17.View ArticlePubMedPubMed CentralGoogle Scholar
  9. Wakerman J: Innovative rural and remote primary health care models: What do we know and what are the research priorities?. Australian Journal of Rural Health. 2009, 17: 21-26. 10.1111/j.1440-1584.2008.01032.x.View ArticlePubMedGoogle Scholar
  10. Broemeling AM, Watson DE, Black C, Reid RJ: Measuring the performance of primary health care: Existing capacity and future information needs. 2006, Centre for Health Service and Policy Research University of British ColumbiaGoogle Scholar
  11. Australian Bureau of Statistics: ABS CDATA Online. 2010Google Scholar
  12. Asaid A, Riley K: From solo practice to partnering: the evolution of the Elmore model of primary health. Australian Family Physician. 2007, 36 (3): 167-169.PubMedGoogle Scholar
  13. Gregory A, Armstrong RM, Van Der Weyden MB: Rural and remote health in Australia: how to avert the deepening health care drought. Med J Aust. 2006, 185 (11/12): 654-660.PubMedGoogle Scholar
  14. Donabedian A: The quality of care. How can it be assessed?. Journal of the American Medical Association. 1988, 260: 1743-1748. 10.1001/jama.260.12.1743.View ArticlePubMedGoogle Scholar
  15. Sibthorpe B: A proposed conceptual framework for performance assessment in primary health care: a tool for policy and practice. 2004, Canberra: Australian National University. Australian Primary Health Care Research Institute, []Google Scholar
  16. Tham R, Humphreys J, Kinsman L, Buykx P, Asaid A, Tuohey K, Riley K: Evaluating the impact of sustainable comprehensive primary health care on rural health. Australian Journal of Rural Health. 2010, 18: 166-172. 10.1111/j.1440-1584.2010.01145.x.View ArticlePubMedGoogle Scholar
  17. National Health and Hospitals Reform Commission: Beyond the Blame Game - Accountability and performance benchmarks for the next Australian Health Care Agreements. 2008, Woden ACT AustraliaGoogle Scholar
  18. Canadian Institute for Health Information: Pan-Canadian Primary Health Care Indicators. 2006, Ottawa, Canada, 1.Google Scholar
  19. National Health Performance Committee: National Health Performance Framework. 2001, Brisbane: Queensland HealthGoogle Scholar
  20. The Royal Australian College of General Practitioners: Standards for general practices. 2010, South Melbourne, Australia, 4Google Scholar
  21. Australian Institute of Health and Welfare (AIHW): Rural, regional and remote health - information framework and indicators -Version 1b. Rural Health Series no 6. 2005, Canberra: AIHWGoogle Scholar
  22. Wakerman J, Humphreys J, Wells R, Kuipers P, Entwistle P, Jones J: A systematic review of primary health care delivery models in rural and remote Australia 1993-2006. 2006, Australian Primary Health Care Research Institute. Canberra: Australian Primary Health Care Research Institute, []Google Scholar
  23. Perera R, Dowell A, Crampton P, Kearns R: Panning for gold: An evidence-based tool for assessment of performance indicators in primary health care. Health Policy. 2007, 80: 314-327. 10.1016/j.healthpol.2006.03.011.View ArticlePubMedGoogle Scholar
  24. Miller WP: Economic multipliers: How communities can use them for planning. Community and Economic Development. 2007, Little Rock: University of Arkansas Division of AgricultureGoogle Scholar
  25. Doeksen GA, Schott V: Economic importance of the health-care sector in a rural economy. Rural and Remote Health. 2003, 3 (online).Google Scholar
  26. Campbell SM, Roland M, Quayle J, Buetow S, Shekelle P: Quality indicators for general practice: which ones can general practitioners and health authority managers agree are important and how useful are they?. Journal of Public Health Medicine. 1998, 20 (4): 414-421.View ArticlePubMedGoogle Scholar
  27. The Royal New Zealand College of General Practitioners: Aiming for Excellence - An assessment tool for New Zealand general practice. 2009, New Zealand, 3Google Scholar
  28. Pre-publication history

    1. The pre-publication history for this paper can be accessed here:


© Tham et al; licensee BioMed Central Ltd. 2011

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.