This is a mixed methods study with three interdependent research streams. The streams employ a range of quantitative and qualitative methods to address a number of research questions. Each stream is conducted consecutively and concurrently over the study time frame and outputs from each inform the others. This material is united into a final, fourth stream to address the overall aims of the study (Figure 1).
Methodological approach with respect to health inequalities
This research project has considerable potential to contribute to the health needs of Māori. Robust examination of both clinical outcomes and the policy implementation process can reveal critical insights on how this policy will affect Māori patients receiving acute emergency care. Research findings will provide empirical support for identifying both the positive and negative effects of the policy for Māori health, and potentiate Māori health gain. Understanding whether the setting of ED targets introduces disparities for Māori (or vice versa) is a specific area of investigation for this research.
Given this context, this research is being guided by a Kaupapa Māori Research (KMR) methodology [38, 39]. Within the context of this study, the KMR framework includes senior Māori research expertise throughout all stages and streams of the research process to: ensure utility and cultural safety of the research process for Māori; Māori research kaitiakitanga (guardianship and protection) of Māori data; review and approval of Māori data collection, analysis and interpretation; analysis and interpretation of Māori data and comparisons with non-Māori and review of any manuscripts involving Māori data prior to submission for publication. Such an approach requires the researchers to aim for equal explanatory and analytical power for Māori compared to non-Māori, a rigorous process to the collection, evaluation and reporting of ethnicity data from available datasets, and interpretation of the data from a non-victim blame or cultural deficit positioning [40, 41]. The research team has agreed to conduct all research activity in accordance with the Tōmaiora Māori Health Research Centre research protocols [42].
Stream one
Research aims
The aim of this research stream is to identify initiatives that have been implemented in response to the performance target across the 20 DHBs, to explore the impact of these initiatives on patient flow into and out of EDs, and to identify four case study sites for in-depth analysis. The selection of hospitals was based on initial target performance results from the first quarter of 2009, and by geographic location and population density.
Design/method and rationale
The core research question for this stream is 'What impact do different initiatives aimed at the ED performance target have on measures of patient flow? We used a dynamic system modelling approach similar to that used in other studies internationally that have attempted to model the dynamics of an ED [12, 43, 44] to frame this stream. This approach had four sequential steps [45].
-
1.
An initial mapping of patient flow.
-
2.
Refining the initial model with an expert reference group.
-
3.
Using the refined concept model to inform a survey of clinical directors (CD) and service managers within each ED that is subject to the national performance time-target. This survey was designed to collect information on the initiatives that have been developed in response to the target and where these initiatives sit on the concept model of patient flow (Additional file 1). Respondents were be asked to identify any changes in resource use (especially staff changes) associated with the interventions and to estimate any additional departmental expenditure associated with the intervention. Information from January 2006 onwards of measures of patient flow (ED admissions/discharges, bed occupancy, LOS elective admissions and elective surgical cancellations) were collected.
-
4.
Survey data collected from each site will subsequently be used to develop a working system dynamic model of patient flow into and out of ED, and illustrate the impact of initiatives put in place in response to the performance target.
Recruitment and data collection
Reference group
This was convened in December 2010 and consisted of the representatives of expert groups relevant to this study, including members of: the national ED advisory group to the Ministry of Health, the Australasian College for Emergency Medicine (ACEM), the College of Emergency Nurses of New Zealand (CENNZ), the Royal New Zealand College of General Practitioners (RNZCGP), Māori representatives, Pacific peoples representatives, representatives from inpatient specialties covering medicine, surgery and paediatrics, representatives of inpatient and community older people's health and the CD and service managers from each case site.
Survey
The CD and/or service manager in each ED completed the survey in mid 2011. Follow-up will occur at the end of 2012 to capture any further interventions.
Data analysis
Further analysis of the data will follow the usual conventions for system modelling [45]. The concept model will be entered into a specialised software package, iThink [46] and differential equations will be calculated to estimate the patient flow, and impact of the interventions. This will allow us to examine whether the intervention has simply moved the 'access block' to other parts of the hospital. It will also allow us to identify any cost-shifting to other services and to make some estimates of any changes in down-stream costs (such as changes in LOS). Survey results will also be analysed to determine the areas of cost related to the interventions, and the perceived impacts the interventions have had on performance.
Stream two
Research aims and questions
This research stream aims to investigate the impact of the target on objective markers of quality of care. The core research questions for this stream are; 'Was there a change in clinically relevant outcomes after the target was introduced?' and 'Were there differential impacts of the target in at risk ethnic, age and deprivation groups?'
Design/method
We will investigate whether the rate of change of quality of care markers differs over time, pre and post introduction of the ED time targets in July 2009. The study will compare outcomes of interest using data collected from three years prior to and after the introduction of the target (1st January 2006 to 31st December 2008 and 1st January 2010 to 31st December 2012). A one year 'target settling' period six months either side of the target introduction in July 2009 will be excluded from analysis. The primary outcomes will be:
-
ED and Hospital length of stay (LOS) [47]
-
Re-attendance rates within 48 hours of discharge [48, 49]
-
Differences in these outcomes in different ethnic, age and deprivation groups
The secondary outcomes are other clinical process measures and clinical outcomes that reflect the quality of care delivery both in ED and wider hospital. The candidate indicators were determined by a systematic literature review and the final set was confirmed by an expert reference group (see Stream one, above).
The secondary outcomes will be;
-
all cause mortality [2, 3, 50, 51]
-
time to reperfusion for ST elevation myocardial infarction (STEMI) [49]
-
time to analgesia in ED [52]
-
time to theatre for fractured neck of femur [53]
-
time to antibiotics for severe infections [54]
-
time to treatment in acute asthma [55, 56]
-
proportion of patients who leave without being seen [48, 49]
-
time to CT for traumatic brain injury [57, 58]
-
time to appendectomy for acute appendicitis [59, 60]
-
appropriateness of discharge information provided to General Practitioners from ED [61, 62]
Outcome measures: definition, rationale and data collection
Primary
LOS starts from date/time of arrival and ends at time/date of discharge. Lack of capacity for acute admissions is a key factor driving the length of time spent in ED [63–65]. Increasing capacity by reducing LOS may have drawbacks. NZ has a shorter average LOS than both Australia and the UK [66] and LOS for Māori is two days shorter than for non-Māori [67, 68], with personal experiences in hospital a contributing factor to this [67]. A small change in LOS, for example a decrease of 0.25 days, is important, as MOH data suggests this would result in approximately 125000 more bed-days available per year nationwide. The distribution of times that patients spend in ED should be a smooth curve with a right skew when plotted graphically (most patients leave within a short time, some stay much longer). We will examine the distribution of ED LOS times pre and post target introduction, looking for a 'spike' of admissions or discharges at or near the target time (which may reflect gaming or patients moved elsewhere prior to completion of their care) [69, 70].
Re-attendance is defined as re-admission to hospital or representation to ED within 48 hours. This outcome is considered an adverse event [71, 72] and may result from inappropriate early discharge. Data on LOS and re-attendance, along with age, ethnicity, gender and NZ deprivation score is available from routinely collected hospital data and will be collected for all 20 DHBs. Ethnicity will be sourced from the National Health Index (a unique patient identifier in NZ) and reported according to the Ethnicity Data Protocols for the Health and Disability Sector, with appropriate adjustments made to account for known discrepancies in NHI data [73].
Secondary
While hospital mortality will be available from national datasets for all patients within the time periods, the other secondary outcomes occur within a subgroup of patients. Using the four case study sites selected in Stream One, a series of chart audits from patient hospital records, each with specific sampling frame and sample size calculations will be conducted. These records will be chosen at random throughout the time periods of interest to gain an in-depth understanding of within hospital variation before and after the introduction of the time target policy. The records will be identified by ICD-10 codes appropriate to each clinical outcome.
Statistical analyses
The data for each of the outcomes will be recorded by hospital by twelve month periods (to avoid modeling seasonal change). For continuous outcomes, in order to compare the rate of change in the measures pre to post intervention and whether any change is influenced by age, ethnicity or hospital, a general linear mixed model will be used with outcome transformed as necessary to overcome correlation of mean and variance in variables such as LOS. A random coefficients model will be used to allow the change to be modeled within hospital. Explanatory variables will be year within pre- and post-intervention time periods, hospital, age category, ethnicity and pre or post intervention. Appropriate interactions of year, intervention, ethnicity, age and hospital will be investigated with higher order interactions being removed if not important and the analysis being split where non-ignorable interactions exist. Binary outcomes such as readmission will be analysed similarly using a generalised linear mixed model. Estimates of least square mean values of outcomes, with 95% confidence intervals, will be calculated, within important subgroups. Data will be analysed using STATA version 10, StataCorp LP, 4905 Lakeway Drive, College Station, TX 77845 USA.
Sample size calculations
All sample size estimations below are based on having 80% power to demonstrate a difference at the five percent level of significance using the models described above. Expected distributions used for estimating variance were based on one tertiary hospital's data, which was the only data available to the researchers at that stage of the study. Expected proportions in each ethnic and age category were based on New Zealand hospital admission data. Māori proportions ranged from five to 18% in the different age groups. For outcomes retrieved from routine electronic data there are very large numbers (almost one million ED attendances annually) and it is possible to look for disparities in change in Māori and other at risk ethnic and age groups and differences in change within hospitals.
The sample size calculations for the primary outcomes are based on detecting interactions. The real difference in LOS considered important to be powered to detect is 0.25 days. To detect a difference in change for Māori compared to European/other, within age groups (< 65 and ≥ 65) requires a total sample of 12,000 in those aged < 65, 76,000 in ≥ 65 (or 4,000 in ≥ 65 if there was no ethnic interaction). These numbers would be present within individual hospitals except for investigating ethnic differences in change in the older age group, where Māori numbers are less. This will be investigated at the national level.
The re-attendance rate at the index hospital was approximately five percent, somewhat higher than reported elsewhere [30, 63, 65, 66]. To detect a real difference in change of one percent in re-attendance in Māori, or other at risk ethnic groups, compared to European/other, within age groups would require approximately 10,000 in the smallest group. Nationwide, there are about 9,000 Māori people older than 65 years admitted annually. Therefore, with three years data after the target was introduced we will have greater than 80% power to investigate this difference.
For secondary outcomes that require manual data extraction, random samples of records of the appropriate size from the six years of the study will be drawn. A sample size calculation for time to reperfusion is given here as this represents the smallest number of clinical events, is clinically very important and health outcomes are associated with disparities by age and ethnicity [74]. Across NZ approximately 1,500 patients per year with STEMI receive reperfusion therapy [74]. In order to detect a five minute change in time to reperfusion, based on a mean (SD) of the log transformed time of 3.61(.58) would require 160 patients per year over the time period of the study or 50 per year to detect a 10 minute change. For each confirmed clinical outcome, a power calculation to determine an appropriate sample size for our research question will be undertaken.
Stream three
Research aim and question
The specific aim of this research stream is to explore how organisations respond to the target by investigating the perspectives, experiences and actions of front line clinical and management personnel in the ED and wider hospital. A second and important aim is to identify the variations in organisational responses between case study sites and different informants. The core research question for this steam is 'How is the ED time target policy implemented?'
Design/method and rationale
This stream involves qualitative research into the four case study hospitals. Key strengths of qualitative research in the field of health service and policy research include the ability to enhance understanding of complex phenomenon in dynamic organizational contexts, and interpretation of experience from a variety of actors [75]. Qualitative design and methods have been recently applied to investigate the effect of performance improvement initiatives in the UK from the perspective of front line personnel, for example the emergency nursing experience of the 4 hour ED time target [76], the experience of pay for performance in primary health care providers [77, 78] and factors affecting the adoption of a "see and treat" model in emergency care [79]. The design for this research stream is qualitative multiple case study, using semi-structured interviews (Additional file 2) and policy documents within the case study sites to collect data. Comparison and contrast of findings can be made within and across different cases and different informant groups (clinical and management, inside the ED and in the wider hospital).
Recruitment and data collection
The first of two rounds of semi-structured interviews across the four case study hospitals, was conducted in early 2011, the second will be in mid 2012. These interviews explore the organisational response to the ED target and associated interventions. Two rounds of data collection will ensure that any changes over time can be captured. The choice of four case studies is based on achieving richness of data, variance in context and comparison of performance, but also recognises resource and time constraints within the research project overall. Between ten and twelve research informants at each site were purposively selected to achieve theoretical saturation, ensure spread across informant groups and requisite experience of policy implementation. Semi-structured interviews ensure focus and structure in the interview, whilst enabling flexibility to probe informant responses for detail, clarification of meaning or examples. Interviews were conducted face to face and audio taped. Strategic, operational, service and policy documents relevant to policy implementation, such as memoranda, guidelines and procedures, will also be collected from case study sites.
Data analysis
Transcribed interview data will be analysed using a general thematic inductive approach to identify the range of perspectives, experiences and actions across the hospitals and the respective groups of policy implementers. Documentary data will also be analysed using thematic inductive techniques. Interview data analysis will be analysed using MAXQDA (version 10), Verbi GmbH, Berlin, Germany.
Stream four
As indicated in Figure 1 above, this stream will draw together information from the other streams. In essence, Stream Two will provide a picture of the patterns of clinical and organisational performance over time (2006-2012), while Streams One and Three will provide the means to interpret and explain such patterns and provide information on costs and resources used to meet the target.
Data analysis
Using data from Stream Two we will classify the case study organisations according to degree of success in meeting ED target with/without adverse clinical and hospital outcomes. Themes developed in the analysis of Stream One and Stream Three data will be mapped against Stream Two results to construct explanations of success/failure. For example, information collected in Stream Two may raise a suspicion of gaming in a particular site. A 'spike' of admissions or discharges at or near the target time may reflect gaming; or that patients are moved out of ED prior to completion of their care. Such consequences may show up in data on clinical outcomes. The data collected in Stream Three will provide a more detailed interpretation of this phenomenon and its broader consequences, especially when such information is incorporated into Stream Three interview schedules.
There are some important precedents for using a mixed methods approach in order to account for ED and hospital responses to targets. Mason et al. were able to attribute some variation in ED mean waiting times to the management style of lead clinicians, with data on management style initially collected through in-depth interviews [32]. Mannion et al., using a multiple case-study approach, identified key attributes of organisational culture and management style that accounted for differences between 'high performing' from 'low performing' trusts according to official performance measures [80].
The multi-case study approach should allow the research to distinguish common organisational factors affecting quality and performance from factors that are highly context-specific. As such, this research can help to distinguish between general strategies for quality and performance improvement (strategies that appear to have been successful in more than one setting), and future initiatives that will need to be highly tailored to specific organisational contexts.