Study protocol: Evaluating the impact of a rural Australian primary health care service on rural health
BMC Health Services Research volume 11, Article number: 52 (2011)
Rural communities throughout Australia are experiencing demographic ageing, increasing burden of chronic diseases, and de-population. Many are struggling to maintain viable health care services due to lack of infrastructure and workforce shortages. Hence, they face significant health disadvantages compared with urban regions. Primary health care yields the best health outcomes in situations characterised by limited resources. However, few rigorous longitudinal evaluations have been conducted to systematise them; assess their transferability; or assess sustainability amidst dynamic health policy environments. This paper describes the study protocol of a comprehensive longitudinal evaluation of a successful primary health care service in a small rural Australian community to assess its performance, sustainability, and responsiveness to changing community needs and health system requirements.
The evaluation framework aims to examine the health service over a six-year period in terms of: (a) Structural domains (health service performance; sustainability; and quality of care); (b) Process domains (health service utilisation and satisfaction); and (c) Outcome domains (health behaviours, health outcomes and community viability). Significant international research guided the development of unambiguous reliable indicators for each domain that can be routinely and unobtrusively collected. Data are to be collected and analysed for trends from a range of sources: audits, community surveys, interviews and focus group discussions.
This iterative evaluation framework and methodology aims to ensure the ongoing monitoring of service activity and health outcomes that allows researchers, providers and administrators to assess the extent to which health service objectives are met; the factors that helped or hindered achievements; what worked or did not work well and why; what aspects of the service could be improved and how; what benefits have been realised and for whom; the level of community satisfaction with the service; and the impact of a health service on community viability. While the need to reduce the rural-urban health service disparity in Australia is pressing, the evidence regarding how to move forward is inadequate. This comprehensive evaluation will add significant new knowledge regarding the characteristics associated with a sustainable rural primary health care service.
The Australian rural health service context
Australia is a vast continent with a population of approximately 22 million people of which 3.5 million are spread across 1500 small rural and remote communities and 7.5 million square kilometres. Isolated rural communities throughout Australia are experiencing demographic ageing, an increasing burden of chronic diseases, and de-population as families move to larger cities. In addition, many of these rural communities are struggling to maintain viable and comprehensive health care services due to lack of service infrastructure, transport difficulties, and health worker shortages due to high levels of staff turnover and difficulties in recruiting new health workers. In many cases rural communities are either foregoing care, travelling to larger regional centres or depending on irregular visiting services. As a consequence, Australia's rural and remote communities face significant health service disadvantages compared with metropolitan regions.
The vast distances separating small communities throughout rural Australia provide enormous challenges for authorities responsible for servicing population health needs as there are conflicts between ensuring operational efficiency and cost-minimisation, whilst maintaining effective and equitable delivery of accessible health services. Traditional urban health service models are proving to be unsustainable. Undoubtedly, there is no "one-size-fits-all" solution to meeting the diverse health needs of rural Australian residents and the range of service models needed is likely to vary between communities. Hence it is necessary to investigate models of health service delivery to ensure equitable access to care and reduce the health differential between rural and metropolitan people.
Research has shown that a primary health care approach yields the best health outcomes in situations characterised by limited resources. As a result, the need for sustainable comprehensive primary health care, characterised by multi-disciplinary team approaches, is urgent in rural communities. In response, a wide variety of innovative health care models have evolved and been trialled in rural and remote areas. However, whilst research has identified the requirements for sustainable rural and remote primary health care services, few rigorous longitudinal evaluations have been conducted to systematise them over time; assess their transferability to other regions; assess how health services continually evolve to address ongoing changes in the external environment; monitor the effect of comprehensive primary health care on health service utilisation behaviour and health literacy of their communities; or better understand how or when primary health care services can redress the poorer rural health status.
Given these gaps in knowledge it is vital to understand which rural health services 'work well, where and why' and are sustainable over the long-term in order to inform rural health service policy, assist with planning sustainable health services in other rural communities and contribute to the equitable delivery of health care services that are likely to bring about improved health outcomes.
This paper describes the study protocol of a comprehensive longitudinal evaluation of a successful primary health care service located in a small rural Australian community to assess its performance, sustainability over time, responsiveness to changing community needs and health system requirements, and its impact on community health behaviours, health outcomes and community viability.
Setting: The Elmore Primary Health Service (EPHS)
The EPHS is located at Elmore in central Victoria (Australia) 170 kilometres from the capital city, Melbourne. While the population in Elmore in 2006 was only 693, the EPHS provides care to nearly 3,000 patients from surrounding districts up to 125 kilometres from Elmore. The EPHS is a single-entry point, comprehensive primary health care model formed in 2001 by a partnership between the private medical practice and the local, publicly funded, community health services after the closure of the local hospital and loss of the local doctor in 1994. The model combines health care, community coordination and outreach services, is financed by public and private funding and is delivered by a multidisciplinary team of, amongst others, doctors, nurses, physiotherapists, psychologists and podiatrists. This service has grown in contrast to the rural national trend of service closures and can be an exemplar for other communities to emulate.
Evaluation framework: domains, components and indicators
The longitudinal evaluation framework evolved from Donabedian's quality of care paradigm linking structure, process and outcome and a conceptual framework for primary health care performance assessment developed originally by Sibthorpe, as reported in another publication. This evaluation aims to examine the health service over a six year period in terms of:
Key structural domains:
Health service performance characteristics;
The sustainability of the health service organisation and function; and
Quality of care that the service provides across the health promotion, treatment and rehabilitation spectrum.
Key process domain:
The effect of the service on health service utilisation and satisfaction.
Key outcome domain:
The effect of the service on health behaviours, health outcomes and community viability.
A key aspect of the evaluation was the identification and development of unambiguous sentinel indicators for each component of these key domains that can be reliably and validly operationalised so that data can be routinely collected and the health service monitored over time. The recent health systems reform process being undertaken in Australia has highlighted the difficulties in developing appropriate and valid accountability and performance benchmarks. Despite potential indicators being easily identified on the conceptual level, it can be extremely difficult to operationalise them in forms for which data can be routinely and unobtrusively collected whilst being beneficial to health outcomes and health services. Hence our indicator selection was guided by significant health services research conducted by the Canadian Institute for Health Information; the Australian National Health Performance Committee; the Australian National Health and Hospital Reform Commission; the Royal Australian College of General Practitioners; the Australian Institute of Health and Welfare ; and Wakerman et al.
Drawing on this evidence, sentinel indicators for assessing the structural domains: health service performance (Table 1); sustainability (Table 2); and quality of care (Table 3), were developed in accord with the evaluation framework. As described in each table, each domain is divided into components, for which sentinel indicators were selected based on their technical merits and validity as identified in the research literature and their applicability in the health service setting - that is, the data can be routinely collected, validated, and extracted reliably from the data sources identified for this study (health service record audits, community surveys, interviews and focus group discussions).
To evaluate the impact of the EPHS on health service utilisation, community viability and satisfaction in Elmore and its hinterland, sentinel indicators include: community participation within health service planning; community experience of and satisfaction with health services; indicators of population composition and growth, employment trends, and health service multiplier effects in the local economy.
The quantitative data for indicators of health service performance, sustainability and quality are being obtained by an annual comprehensive health service record audit and biennial community surveys. These quantitative data will be analysed using descriptive statistics that monitor trends over the period of the study.
The comprehensive health service audit is examining medical record data, billing and financial data, human resources records, registry data (e.g. Australian Childhood Immunisation Register and the Victorian Cervical Cytology Register) and government reports to the practice (e.g. Practice Incentive Payments and Service Incentive Payments). All data are being extracted by an EPHS practice nurse and provided in a de-identified and aggregated format to the external university research team to ensure utmost privacy and confidentiality of patient and health service records.
Further quantitative data are to be obtained through a biennial community survey that is to be delivered to all residents in the catchment area and is designed to obtain information in relation to health service utilisation, satisfaction and need, in addition to health risk behaviours.
The qualitative data that explores sustainability of the service and the impact of the EPHS on the local community are being obtained by in-depth interviews with key stakeholders and document analysis. These data will be transcribed and thematically analysed and triangulated with the quantitative data for validation and clarification of the data. As new understandings emerge through this iterative and reflective process, new indicators will be developed or existing indicators may be modified.
Engagement with the local community is essential for this study as the current health service grew from community activism in collaboration with the local doctor and regional community health service. A detailed communication strategy, including community forums and articles in the local newspaper, will ensure that all stakeholders and community residents are kept informed of the purpose and conduct of the evaluation study. The conduct of the research and dissemination of findings will be guided by a Reference Group that will be comprised of representatives from the local community (leaders, businesses and community groups), regional health authorities, and general practice support agencies (e.g. Divisions of General Practice and general practice education and training services). The research team will provide information to the community through plain language summaries, oral presentations and presence at key community events.
The study has obtained ethics approvals from Monash University Human Research Ethics Committee (CF08/0419- 2008000176; CF08/0238 - 2008000089; CF08/2434 - 2008001256; CF10/2540 - 20100001423).
Ongoing monitoring and evaluation of successful rural health service models is needed to provide sound empirical evidence of what makes a primary health care service sustainable and effective in a rapidly changing health policy environment in order to provide an evidence base that can inform health service policy makers and other rural health services.
This iterative evaluation framework and methodology aims to ensure the ongoing monitoring of service activity and health outcomes associated with the service that allows researchers, health service providers and administrators to assess the extent to which health service objectives have been met; the factors that have helped or hindered achievements; what worked or did not work well and why; what aspects of the service could be improved and how; what benefits have been realised and for whom; the level of community satisfaction with the service; and the impact of a health service on community viability.
This evaluation focuses on health service performance and capacity rather than focussing on how well a particular disease is treated. This raises challenges in identifying appropriate indicators as primary health care organisation and function is complex, dynamic and composed of both measurable and unmeasurable elements that are subject to many external influences that can impede performance, sustainability and quality of care. The scope and breadth of ideal indicators for the components of each domain in the research literature is more extensive than those selected for this study. The selection of indicators was limited principally by the types of data items that are stored in paper records and electronic databases that are reliable, valid and readily extracted.
This study also provides an opportunity to assess, over time, the effect of locally available health promotion, preventive and early intervention programs on health literacy and health status of a small rural community. As it is possible to relate the EPHS catchment to a geographically circumscribed hinterland, this evaluation may examine the associations between sustainable local health services and community viability and prosperity through examining community satisfaction, growth, employment, and multiplier effect indicators. Hence, building on previous studies that highlighted the significance of rural hospitals and health services to the local economy and fortune of rural communities , the findings from this study may assist health authorities and other small rural communities to benchmark what services are appropriate and successful.
The danger in not undertaking such comprehensive longitudinal research to identify "what works well, where and why" is the risk of widening health differential between urban and rural communities; increasing costs of health care treatment to individuals and to government; and reduced implementation of best practice care equitably to all communities.
The need to provide equitable access to quality primary health services for Australia's rural communities is essential and urgent. However, the evidence base for rural health service design remains sparse. This longitudinal evaluation will contribute significant knowledge to how a primary health service can be evaluated and play a role in reducing the Australian rural-metropolitan health disparity.
National Health and Hospitals Reform Commission: A Healthier Future For All Australians - Final Report of the National Health and Hospitals Reform Commission - June 2009. 2009, Commonwealth of Australia
Chenoweth L, Stehlik D: Using technology in rural practice - local area coordination in rural Australia. Rural Social Work. 2002, 7 (1): 14-21.
Humphreys J: Health service models in rural and remote Australia. The New Rural Health: An Australian Text. Edited by: Wilkinson D, Blue I. 2002, Oxford University Press, 273-296.
Australian Institute of Health and Welfare (AIHW): Whose health? How population groups vary. Australia's health 2010. 2010, Canberra: AIHW, 227-280.
Starfield B: Primary care: is it essential?. Lancet. 1994, 344: 1129-1133. 10.1016/S0140-6736(94)90634-3.
Starfield B: Primary Care: Balancing health needs, services and technology. 1998, Oxford: Oxford University Press
Wakerman J, Humphreys J, Wells R, Kuipers P, Jones J, Entwistle P, Kinsman L: Features of effective primary health care models in rural and remote Australia: a case-study analysis. Medical Journal of Australia. 2009, 191 (2): 88-91.
Kuipers P, Humphreys J, Wakerman J, Wells R, Jones J, Entwistle P: Collaborative review of pilot projects to inform policy: A methodological remedy for pilotitis?. Australia and New Zealand Health Policy. 2008, 5 (1): 17-10.1186/1743-8462-5-17.
Wakerman J: Innovative rural and remote primary health care models: What do we know and what are the research priorities?. Australian Journal of Rural Health. 2009, 17: 21-26. 10.1111/j.1440-1584.2008.01032.x.
Broemeling AM, Watson DE, Black C, Reid RJ: Measuring the performance of primary health care: Existing capacity and future information needs. 2006, Centre for Health Service and Policy Research University of British Columbia
Australian Bureau of Statistics: ABS CDATA Online. 2010
Asaid A, Riley K: From solo practice to partnering: the evolution of the Elmore model of primary health. Australian Family Physician. 2007, 36 (3): 167-169.
Gregory A, Armstrong RM, Van Der Weyden MB: Rural and remote health in Australia: how to avert the deepening health care drought. Med J Aust. 2006, 185 (11/12): 654-660.
Donabedian A: The quality of care. How can it be assessed?. Journal of the American Medical Association. 1988, 260: 1743-1748. 10.1001/jama.260.12.1743.
Sibthorpe B: A proposed conceptual framework for performance assessment in primary health care: a tool for policy and practice. 2004, Canberra: Australian National University. Australian Primary Health Care Research Institute, [http://www.anu.edu.au/aphcri/Publications/conceptual_framework.pdf]
Tham R, Humphreys J, Kinsman L, Buykx P, Asaid A, Tuohey K, Riley K: Evaluating the impact of sustainable comprehensive primary health care on rural health. Australian Journal of Rural Health. 2010, 18: 166-172. 10.1111/j.1440-1584.2010.01145.x.
National Health and Hospitals Reform Commission: Beyond the Blame Game - Accountability and performance benchmarks for the next Australian Health Care Agreements. 2008, Woden ACT Australia
Canadian Institute for Health Information: Pan-Canadian Primary Health Care Indicators. 2006, Ottawa, Canada, 1.
National Health Performance Committee: National Health Performance Framework. 2001, Brisbane: Queensland Health
The Royal Australian College of General Practitioners: Standards for general practices. 2010, South Melbourne, Australia, 4
Australian Institute of Health and Welfare (AIHW): Rural, regional and remote health - information framework and indicators -Version 1b. Rural Health Series no 6. 2005, Canberra: AIHW
Wakerman J, Humphreys J, Wells R, Kuipers P, Entwistle P, Jones J: A systematic review of primary health care delivery models in rural and remote Australia 1993-2006. 2006, Australian Primary Health Care Research Institute. Canberra: Australian Primary Health Care Research Institute, [http://www.anu.edu.au/aphcri/Domain/RuralRemote/index.php]
Perera R, Dowell A, Crampton P, Kearns R: Panning for gold: An evidence-based tool for assessment of performance indicators in primary health care. Health Policy. 2007, 80: 314-327. 10.1016/j.healthpol.2006.03.011.
Miller WP: Economic multipliers: How communities can use them for planning. Community and Economic Development. 2007, Little Rock: University of Arkansas Division of Agriculture
Doeksen GA, Schott V: Economic importance of the health-care sector in a rural economy. Rural and Remote Health. 2003, 3 (online).
Campbell SM, Roland M, Quayle J, Buetow S, Shekelle P: Quality indicators for general practice: which ones can general practitioners and health authority managers agree are important and how useful are they?. Journal of Public Health Medicine. 1998, 20 (4): 414-421.
The Royal New Zealand College of General Practitioners: Aiming for Excellence - An assessment tool for New Zealand general practice. 2009, New Zealand, 3
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6963/11/52/prepub
This study is being funded by Australian Rotary Health for the period 2008 - 2010 and the Australian Government Department of Health and Ageing for the period 2011-2013. The funding bodies had no role in the conception or development of the study design, the writing of the manuscript or the decision to submit it for publication. The authors wish to acknowledge the contributions of Karly Smith, Karen Riley and Judith Jones in the development of the study design and ethics applications.
The authors declare that they have no competing interests.
RT, JSH and LK conceived and developed the design of the study, obtained the ethics applications and drafted the manuscript. PB obtained the ethics application and contributed to drafting and revising the manuscript. AA and KT contributed to the conception and design of the study and revising the manuscript. All authors read and approved the final manuscript.
About this article
Cite this article
Tham, R., Humphreys, J.S., Kinsman, L. et al. Study protocol: Evaluating the impact of a rural Australian primary health care service on rural health. BMC Health Serv Res 11, 52 (2011). https://doi.org/10.1186/1472-6963-11-52
- Health Service Utilisation
- Primary Health Care Service
- Community Satisfaction
- Rural Health Service
- Comprehensive Primary Health Care