Implementation and evaluation of a multisite drug usage evaluation program across Australian hospitals - a quality improvement initiative
© Pulver et al; licensee BioMed Central Ltd. 2011
Received: 8 November 2010
Accepted: 29 August 2011
Published: 29 August 2011
With the use of medicines being a broad and extensive part of health management, mechanisms to ensure quality use of medicines are essential. Drug usage evaluation (DUE) is an evidence-based quality improvement methodology, designed to improve the quality, safety and cost-effectiveness of drug use. The purpose of this paper is to describe a national DUE methodology used to improve health care delivery across the continuum through multi-faceted intervention involving audit and feedback, academic detailing and system change, and a qualitative assessment of the methodology, as illustrated by the Acute Postoperative Pain Management (APOP) project.
An established methodology, consisting of a baseline audit of inpatient medical records, structured patient interviews and general practitioner surveys, followed by an educational intervention and follow-up audit, is used. Australian hospitals, including private, public, metropolitan and regional, are invited to participate on a voluntary basis. De-identified data collected by hospitals are collated and evaluated nationally to provide descriptive comparative analyses. Hospitals benchmark their practices against state and national results to facilitate change. The educational intervention consists of academic detailing, group education, audit and feedback, point-of-prescribing prompts and system changes. A repeat data collection is undertaken to assess changes in practice.
An online qualitative survey was undertaken to evaluate the APOP program. Qualitative assessment of hospitals' perceptions of the effectiveness of the overall DUE methodology and changes in procedure/prescribing/policy/clinical practice which resulted from participation were elicited.
62 hospitals participated in the APOP project. Among 23 respondents to the evaluation survey, 18 (78%) reported improvements in the documentation of pain scores at their hospital. 15 (65%) strongly agreed or agreed that participation in APOP directly resulted in increased prescribing of multimodal analgesia for pain relief in postoperative patients.
This national DUE program has facilitated the engagement and participation of a number of acute health care facilities to address issues relating to quality use of medicine. This approach has been perceived to be effective in helping them achieve improvements in patient care.
Commencing in the late 1990s DUE practitioners in Australia formed state-based special interest groups that carried out multisite audits of drug use [5–7]. Participation in multisite DUE projects enabled hospitals to compare their practice with others and enabled smaller hospitals with limited resources/capacity to carry out DUE. Some of the state-based audits were sponsored by the NPS - Better choices, Better Health (NPS), an independent, non-profit organization, funded by the Australian Government Department of Health and Ageing. The NPS provides evidence-based information about medicines, and educational and behaviour change strategies for health professionals and consumers. In 2001 the NPS funded five state Therapeutic Advisory/DUE Groups to conduct separate multisite initiatives [8–12]. Topics for each state were based on local needs analysis, and the projects comprised audit and feedback and subsequent re-audit. Based on an assessment of the achievements, barriers and enablers of state-based projects, the NPS in 2003 funded and coordinated the state Therapeutic Advisory/DUE groups to work collaboratively to address quality use of medicine issues at a national level for hospitals.
Published DUEs mostly present an initial clinical audit only and tended to be localised in nature . A recent systematic review of the literature demonstrated a greater chance of success in changing behaviour with regards to medication use when a number of different interventions are used . The national DUE program aimed to implement the complete DUE cycle, including a baseline audit, multifaceted interventions and post-intervention audits. We describe a model of national multisite DUE and its evaluation, as illustrated by the Acute Postoperative Pain (APOP) project.
This model utilises an established quality improvement methodology, DUE, as described previously. This involves hospitals collecting baseline audit data, feedback of evaluated data, targeted educational interventions, a repeat audit and feedback on improvements in practice. At least one complete DUE cycle is implemented during the course of each 2-year project.
The state Therapeutic Advisory/DUE groups consult with their hospital members and affiliated academic units to identify a therapeutic area that may benefit from a quality improvement initiative. Key criteria for topic selection are outlined below.
The topic selected should have:
A disease state or therapy-based focus
Evidence-based national guidelines of best practice available
A quantitative/qualitative gap in evidence-based practice
Potential for engagement with hospital physicians in a shared quality agenda
Alignment with national health priorities
Potential for synergy with programs undertaken by other agencies
Potential to improve patient care across the continuum of care
Potential to link with primary healthcare providers
A significant positive impact on patient care
Support for sustainable quality improvement activity (capacity and skills in hospitals) and embed quality use of medicine practices into hospital systems
The project is managed by the national project team comprising NPS staff members (project lead, data analyst and database/website developers) and the project officer and lead from each of the state-based Therapeutic Advisory/DUE groups. State project committees are established and comprise at least one member from participating hospitals and selected key medical, nursing and pharmacy clinicians from the state with expertise in disease/therapeutic area.
Australian hospitals, private and public, metropolitan and regional, are invited to participate via the state groups on a voluntary basis. A quality assurance or full ethics application is made as required by individual hospital ethics committees using the National Ethics Application Form  where accepted. Hospitals are provided with some financial support from NPS to assist with implementation of the project. Participating hospitals appoint a local hospital coordinator, whose responsibilities are to: liaise with the state project officer, recruit a local hospital project team, coordinate and facilitate project activities including data collection, feedback of results within the hospital, and organise educational activities and system changes where necessary. A local opinion leader, who is seen as an expert within the therapeutic area, is sought to support and endorse local project activity. A project team consisting of the local hospital coordinator, local opinion leader, medical, nursing and pharmacy clinicians working in the therapeutic area and, where available, a clinical governance/quality manager is formed.
Patient inclusion criteria and the dataset are determined by the national project team. The dataset includes patient demographics, relevant clinical data and prescribed medicines, information on the patients' experience of care, and information from the General Practitioner (GP) on the quality of the discharge communication. Each hospital is responsible for collecting its own data, via the inpatient medical record review, structured patient interview and GP survey.
For APOP, the dataset included information on documentation of preoperative patient education and postoperative pain assessment, postoperative analgesic prescribing and administration, monitoring and documentation of adverse effects, documentation of pain management plans at discharge, the patient's experience of pain management in hospital and following discharge, and the adequacy of communication to the GP about on-going pain management.
Data are collected and then de-identified data are entered into a specifically designed web-based database (electronic-DUE audit tool). The electronic-DUE audit tool is a secure, password protected electronic database developed by the NPS . The tool generates automated feedback reports for each hospital (in real time), reporting key quality measures. Hospital de-identified data are submitted electronically via the electronic-DUE audit tool in an encrypted file to the NPS for central collation and analysis. This comparative analysis enables each hospital to compare their baseline practice and subsequent improvements to state and national data. In APOP the quality measures included documentation of pain assessment, postoperative analgesic prescribing, patient satisfaction with pain management and communication of the discharge pain management plan to the patient/carer and GP.
A multifaceted intervention strategy is used to deliver nationally-developed best practice guidance in the form of key messages for the project. Academic detailing (educational outreach, educational visiting) by trained detailers is the key intervention . Each hospital nominates up to two staff members (medical, nursing or pharmacy) from relevant specialty areas to be trained in academic detailing skills. Two-day workshops for approximately twenty participants each, conducted by the Drug and Therapeutic Information Service (Adelaide) and NPS, are held. Participants are trained in the theory and practice of academic detailing and the evidence behind the key messages, using didactic sessions, small group work and role plays with other health professionals. The target audience for the academic detailing is determined by the hospital and includes key opinion leaders, senior and junior medical officers, nursing staff, pharmacists and other allied health members.
For the APOP project, education was addressed to medical, nursing, pharmacy and allied health staff working in surgical and anaesthetic areas. For nursing staff the focus was the use of pain and sedation scores, and for medical staff education focused on the importance of communication to the patient and GP at discharge. The evidence for prescription of multimodal analgesia, multiple anti-emetic options and prophylactic laxatives for patients prescribed opioids was emphasised to all groups.
Other interventions include point-of-prescribing prompts (e.g. reminder bookmark), promotional material (e.g. wall posters) and a feedback presentation with comparative state and national data, provided for local adaptation by each hospital. System changes are implemented at the discretion of the hospital project team.
A short qualitative survey designed in consultation with state-based Therapeutic Advisory/DUE groups was undertaken by NPS between December 2009 and February 2010. The main aims of the survey were to establish: hospitals' perceptions of the effectiveness of the overall DUE methodology and the changes in procedure/prescribing/policy/clinical practice which resulted from participation in one of the three national DUE projects [18–20]. Questions were selected on the basis of eliciting quality assurance/improvement responses and to probe responses regarding the varying stages of the DUE cycle. The survey questions can be found in additional file 1. (Note: Results displayed below are for questions relating to APOP only).
Questions were embedded into the online surveying and data collection tool - Survey Monkey(R). A link to the survey was emailed to a list of participating staff at hospitals, in New South Wales, Victoria, Tasmania, Queensland and South Australia. Recipients on that list were those who had an active involvement in a DUE project. The list was provided by state coordinators of the DUE projects. Responses were coded and grouped into themes according to one investigator and refined by two investigators if necessary.
62 hospitals (31 principal referral, 10 large major city, 10 private, 5 large regional and remote, 3 medium major city/regional, 2 specialist women's/children's hospitals and 1 private regional hospital collaborative) from all states and territories participated in APOP. Results regarding APOP have been previously published .
The evaluation survey was sent to 72 nominated individuals. 37 recipients had participated in the APOP project. 23 (62%) of these 37 respondents to the evaluation survey reported being involved in the APOP project.
Changes in practice
78% (n = 18) of the 23 APOP respondents agreed or strongly agreed that documentation of pain scores increased at their facility as a result of APOP. Responses elaborating on these changes were grouped into two themes: awareness of best practice (via in-service education) resulting in improved local practice and system change to embed best practice in care processes. System changes included a bedside pain prompt, changes in observation forms, additional posters and patient education leaflets. Some hospitals reported that the changes to observation charts had occurred at a state level.
'I think pain score documentation increased simply because of an increased awareness among staff of the use of pain scores as the 5th vital sign. There was a number of ward in-services given re: APOP and the use of pain scores. The inclusion of a pain score was already part of the observation chart.'
'The Nurses on the Orthopaedic ward were actively involved and really did take this to town. They started recording pain at rest and pain at movement on their Ob's charts [observation chart] where previously they had only recorded 'pain at rest '. They had the key message of 'Please remember to record your pain score' on the outside cover of each pts [patients] bedside file'
'Prescription of multimodal analgesia was already practiced in this hospital, however staff became more aware of rationale behind this and reduced co-prescription of same/similar drugs e.g. Panadeine Forte + Oxycodone'
'We were already adopting the multimodal approach but it was great to reinforce this. If for some reason it was not being used the Pharmacists were educated to question say why the regular paracetamol was not prescribed and could quite easily back it up with the literature'
One hospital reported a mandatory Junior Medical Officer pain management orientation program devised by the Acute Pain Service to improve the prescribing of analgesics. Another hospital reported on the frequent changeover of staff and the need to repeat education to sustain improvements in the prescribing of multimodal analgesia.
'Patient information leaflets were developed detailing drugs given at discharge. These were also faxed to GP on discharge. GP survey indicated improvement'
'Basically the Drs [doctors] always had to be prompted to make headway with this element of the project. The medication discharge summaries that we use are practically illegible ie the copy that goes with the pt to take to their local Dr. However if that pt [patients] was seen by a Pharmacist this aspect would be done well and a Medilist would be supplied too'
Feedback from respondents on drug use evaluation methodology
Examples of comments
Improved analgesic prescribing by JMOs [Junior Medical Officers].
Improved and better documentation
'Increase in documentation of pain scores at rest and movement due to a change in hospital paper work.'
'Audits allow for identification of specific weaknesses in a process e.g. the lack of a pain score in the patient observation chart. It is these system changes that have the most lasting effects'
Improvement at discharge for patients
'Improved patient information process to understand analgesics prescribed at discharge'
Use of benchmarking for improvement
'It was great to have local data for feedback'
'Improved provision of information to patients.'
'The DUE project provided an infrastructure - data collection, analysis, intervention, and follow-up - which allows the hospital to focus on a particular area that may otherwise not happen. There is little time available in hospitals to develop all the necessary materials to conduct such quality improvement activities in isolation. Providing deadlines meant the work got done. Providing training in communication via academic detailing facilitated effective inter-professional communication around key messages. It became more about working together not about one person telling someone else what to do.'
Facilitators of change
'Benchmarking showed improvement with implementation of education programs and development of patient information. Projects have been actively endorsed by Pharmacy & Therapeutics Committee. Provision of education for staff (both medical, nursing and allied health) has facilitated change in the hospital setting with assisted use of DUE's program templates and checklists.'
'For a rural site like ours the state and national benchmarking was great'.
'Benchmarking very useful'
We describe a unique model of a national multisite DUE program. Our model protects the privacy of the data from individual hospitals while allowing for comparisons between the participating hospital, and aggregated state and national results. Participation in a nationally coordinated program, with education resources and training provided, enables hospitals to undertake the DUE, with efficient use of local resources. Hospitals are provided with support/assistance at all stages by state project officers.
A multifaceted educational intervention strategy is employed as part of the overall methodology, allowing hospitals to select and tailor the intervention tools according to local need, to influence clinician behavior. There is an increasing international recognition of academic detailing as an important evidence-based strategy for influencing behavior change among health professionals [17, 21–24]. A review by Wensing and Grol found positive effects, mainly on prescribing behavior when academic detailing was used as part of a multifaceted intervention including audit and feedback and reminders . Academic detailing has been used by NPS in Australia to improve quality use of medicine in the general practice setting. In the national DUE program hospital staff were trained in skills for professional communication to influence behaviour change. These skills may be used in other local initiatives and in everyday practice.
Our model gives participating hospitals immediate access to their own results as well as enabling benchmarking with state and national results. Respondents to the evaluation survey reflected that feedback of current practice and benchmarking was particularly useful. Participation may also enable hospitals to meet local, state and national standards/indicators/accreditation requirements. Factors that facilitate implementation of a multisite project are described below.
Factors that facilitate implementation of a multisite project
A national coordinating body
Experienced state project officers
Provision of a small financial incentive
Use of an evidence-based intervention method (academic detailing) as part of a multi-faceted intervention
Use of an electronic data collection tool with automated feedback in real-time
Engagement of enthusiastic hospital staff
The undertaking of a multisite DUE program has some challenges. Firstly national consensus is required in selecting a therapeutic area of interest and priority. Consensus amongst clinicians regarding appropriate management (best practice) within the selected area is also required. To facilitate national consensus, identification of best-practice guidance is required e.g. in Australia we have the Therapeutic Guidelines  and other national guidelines. With data being collected across many sites, the use of a manual to guide data collection is essential to ensure data integrity. In the current model submitted data are checked for obvious discrepancies; however potential misinterpretation of data elements by local data collectors is not assessed during the data collection process. This is a limitation of the multisite methodology, and could be overcome by random audits of data collection at the various sites by state project officers. The cost effectiveness of academic detailing has not been evaluated. It is unclear whether the impact of the multifaceted intervention is sustained over time .
In the current national DUE model financial support offered to hospitals is a contribution only, with the expectation that hospitals will provide support in kind towards project completion. Reliance on the funding from NPS to support all project related activity resulted in some hospitals being unable to complete the project as they were not able to retain project staff.
Other challenges for collaborative work include the coordination of activities across all sites within defined timeframes e.g. hospital recruitment, obtaining ethics approval and data entry/submission to enable timely reporting of state and national results. Delays in obtaining ethics approval, variation in rate of presentation of eligible patients across participating sites, management of project workload and staff leave may influence the ability of the project teams in meeting key milestones/deadlines.
A challenge for most quality improvement activities at a hospital level is the rotation of junior hospital staff to other clinical areas. Repeated education is therefore required as new staff are rotated through the targeted clinical area. This was highlighted by comments from the evaluation survey. Conversely rotating staff take the learnings with them to other clinical areas which may facilitate the spread of the educational messages to a wider audience.
To promote sustainability and spread of improvements achieved, the NPS provides the electronic DUE tool as a stand-alone version freely available online after the completion of the project . Hospitals are encouraged to undertake further audits of their practice and continue quality improvement.
This national program has facilitated the participation of a substantial number of hospitals to undertake DUE, and deliver standardized and consistent key educational messages across Australia. The model has been perceived by hospitals to be effective in helping hospitals achieve improvements in patient care.
The authors acknowledge the other members of the national project team: NPS - Better choices, Better health; Karen Kaye, Judith Mackson; School of Pharmacy, University of Queensland: Susan Tett, Donna Taylor; Victorian Department of Human Services: Kylie McIntosh; School of Pharmacy, University of Tasmania: Gregory Peterson, Anna Tompson; South Australian Therapeutic Advisory Group/Department of Health: Shwu Fen Loh, Anne Reeves; Drug and Therapeutic Information Service, South Australia: Debra Rowett; APOP project teams in Queensland, New South Wales, Victoria, South Australia, Tasmania and the NPS - Better choices, Better health, an independent public company funded by the Australian Government Department of Health and Ageing.
- National Health Survey. [http://www.abs.gov.au/ausstats/abs@.nsf/Latestproducts/4364.0Main%20Features32007-2008%20(Reissue)?opendocument&tabname=Summary&prodno%20=%204364.0&issue%20=%202007-2008%20(Reissue)&num=&view=]
- Chassin MR, Loeb JM: The Ongoing Quality Improvement Journey: Next Stop, High Reliability. Health Affairs. 2011, 30 (4): 559-568. 10.1377/hlthaff.2011.0076.View ArticlePubMedGoogle Scholar
- Society of Hospital Pharmacists of Australia Committee of Specialty Practice in Drug Use Evaluation: SHPA Standards of Practice for Drug Use Evaluation in Australian Hospitals. Journal of Pharmacy Practice and Research. 2004, 34 (3): 220-223.View ArticleGoogle Scholar
- Dartnell J: Understanding, influencing and evaluating drug use. 2001, Melbourne: Therapeutic Guidelines Limited, FirstGoogle Scholar
- Roberston MB, Dartnell JG, Korman TM, on behalf of the Victorian Drug Usage Evaluation Group: Vancomycin and teicoplanin use in Victorian hospitals. Med J Aust. 1999, 171 (3): 127-131.Google Scholar
- Robertson MB, Korman TM, Dartnell JG, Ioannides-Demos LL, Kirsa SW, Lord JAV, Munafo L, Byrnes GB, on behalf of the Victorian Drug Usage Evaluation Group: Ceftriaxone and cefotaxime use in Victorian hospitals. Med J Aust. 2002, 176 (11): 524-529.PubMedGoogle Scholar
- Kaye KI, Welch SA, Graudins LV, Graudins A, Rotem T, Davis SR, Day RO: Pethidine in emergency departments: promoting evidence-based prescribing. Med J Aust. 2005, 183 (3): 129-133.PubMedGoogle Scholar
- Victorian Drug Usage Evaluation Group: Antibiotic treatment of lower respiratory tract infection in emergency departments. Report to the National Prescribing Service and the Victorian Drug Usage Advisory Committee. 2003Google Scholar
- Therapeutic Advisory Group New South Wales: An investigation of hospital prescribing of proton pump inhibitors. 2001Google Scholar
- Matthews J, Poulier MV, Cardiff LM, Coombes J, Tett SE: Drug use evaluation of ipratropium bromide at two tertiary referral hospitals in Brisbane. Australian Journal of Hospital Pharmacy. 2001, 31 (4): 279-283.View ArticleGoogle Scholar
- Peterson GM, Boyles PJ, Bleasel MD, Vial JH: Ipratropium Treatment of Acute Airways Disease. Annals of Pharmacotherapy. 2003, 37 (3): 339-344.View ArticlePubMedGoogle Scholar
- Boyles PJ, Peterson GM, Bleasel MD, Vial JH: Under treatment of congestive heart failure in an Australian setting. Journal of Clinical Pharmacy and Therapeutics. 2004, 29 (1): 15-22. 10.1046/j.1365-2710.2003.00531.x.View ArticlePubMedGoogle Scholar
- Pulver LK, Tett SE: Drug utilization review across jurisdictions-a reality or still a distant dream?. Eur J Clin Pharmacol. 2006, 62 (2): 97-106. 10.1007/s00228-005-0087-6.View ArticlePubMedGoogle Scholar
- Lu CY, Ross-Degnan D, Soumerai SB, Pearson SA: Interventions designed to improve the quality and efficiency of medication use in managed care: A critical review of the literature - 2001-2007. BMC Health Services Research. 2008, 8 (75).
- National Ethics Application Form for submitting research protocols to Human Research Ethics Committees. [https://www.neaf.gov.au/]
- NPS acute postoperative pain (APOP) DUE toolkit. [http://www.nps.org.au/due_apop]
- Soumerai SB, Avorn J: Principles of educational outreach ('academic detailing') to improve clinical decision making. JAMA. 1990, 263 (4): 549-556. 10.1001/jama.263.4.549.View ArticlePubMedGoogle Scholar
- McIntosh KA, Maxwell DJ, Pulver LK, Horn F, Robertson MB, Kaye KI, Peterson GM, Dollman WB, Wai A, Tett SE: A quality improvement initiative to improve adherence to national guidelines for empiric management of community-acquired pneumonia in emergency departments. International Journal for Quality in Health Care. 2010, 23 (2): 142-150.View ArticlePubMedGoogle Scholar
- Taylor DR, Loh SF, Mulligan KT, Pulver LK, Tompson AJ, Wai A: Management of Acute Postoperative Pain in Australian Hospitals - Room for Improvement. Journal of the Australasian Association for Quality in Health Care. 2010, Summer: 29-36.Google Scholar
- Peterson GM, Thompson A, Pulver LK, Robertson MB, Brieger D, Wai A, Tett SE, for the DMACS Project Group: Management of Acute Coronary Syndromes at Hospital Discharge: Do Targeted Educational Interventions Improve Practice Quality?. Journal for Healthcare Quality. 2011Google Scholar
- Avorn J, Soumerai SB: Improving drug-therapy decisions through educational outreach. A randomized controlled trial of academically based "detailing". N Engl J Med. 1983, 308 (24): 1457-1463. 10.1056/NEJM198306163082406.View ArticlePubMedGoogle Scholar
- Bloom BS: Effects of continuing medical education on improving physician clinical care and patient health: A review of systematic reviews. International Journal of Technology Assessment in Health Care. 2005, 21 (3): 380-385.View ArticlePubMedGoogle Scholar
- O'Brien MA, Rogers S, Jamtvedt G, Oxman AD, Odgaard-Jensen J, Kristoffersen DT, Forsetlund L, Bainbridge D, Freemantle N, Davis DA, et al: Educational outreach visits: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2007, CD000409-4
- Shaw J, Harris P, Keogh G, Graudins L, Perks E, Thomas PS: Error reduction: academic detailing as a method to reduce incorrect prescriptions. Eur J Clin Pharmacol. 2003, 59 (8-9): 697-699. 10.1007/s00228-003-0683-2.View ArticlePubMedGoogle Scholar
- Wensing M, Grol R: "Multifaceted interventions", in Improving patient care: The implementation of change in clinical practice. 2005, Edinburgh: ElsevierGoogle Scholar
- eTG complete [Internet]. Therapeutic Guidelines Volume eTG complete [Internet] Melbourne: Therapeutic Guidelines Limited; 2006 Jan. 2009, Melbourne: Therapeutic Guidelines Limited
- Alvanzo AH, Cohen GM, Nettleman M: Changing physician behaviour: half-empty or half-full?. Clinical Governance: An International Journal. 2003, 8 (1): 69-78. 10.1108/14777270310459995.View ArticleGoogle Scholar
- NPS - Better choices, Better health. [http://www.nps.org.au]
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6963/11/206/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.