Investigators
We assembled a team of experienced social and clinical researchers to design a program of research in order to investigate the ACHS accreditation process, outcomes and impact. In developing this program of research the team had four aims: to survey the literature; to test the initial design against a range of stakeholder groups for face validity; to incorporate into the design a comprehensive set of examinations which would provide data to inform and direct health policy regarding accreditation; and to articulate a design not only to suit an investigation of the ACHS accreditation process but which could be, with modification, used by international research groups seeking to examine other accreditation processes.
The research team comprised 12 senior investigators with expertise in organisational behaviour, organisational psychology, health quality and safety, social sciences, health informatics, health services research, statistics, health consumer needs, accreditation, and the chief areas of clinical practice (medicine, nursing and allied health). The emerging design was subject to numerous discussions, exposure drafts and re-drafts until all were satisfied as to its capacity to yield appropriate results. It was modified progressively in the light of consultations with several peak health care bodies, four industry partners in both private and public health care, the nine government jurisdictions of Australia (six States, two Territories and the Federal (Commonwealth) government, and various consumer representatives.
Research program
The research program sought to investigate five major variables central to the clinical and organisational performance of an organisation (organisational performance; clinical performance indicators; organisation culture; consumer participation; and accreditation performance on EQuIP) and the inter-relationships between these variables. In Figure 1 we present a simplified model of these complex inter-relationships.
This model suggests that these characteristics are associated, and that performance of one is related to performance of another. For example, organisational performance should be directly or indirectly affected by attainment of accreditation standards (in this case, performance on EQuIP), strong clinical performance, productive involvement of consumers [48] and an effective organisational culture [49]. At this stage the nature of the relationships between all these factors is unknown. For example, does poor performance on accreditation predict poor clinical and organisational performance? If not, is the dissonance explainable?
In view of the multi-dimensional nature of health care performance, a research strategy investigating these dimensions necessarily engages both quantitative and qualitative methodologies. Dimensions not readily captured through archetypal continuous measurement tools can be caught using an array of social research tools. No other research projects were identified which have tried to investigate the relationships between accreditation and other key organisational variables using a multi-method strategy, so we had little basis on which to build a design. Thus, the strategy proposed is novel and innovative. We are unable to show causation through a randomised controlled trial (that is, between accreditation and the other variables) due to potentially confounding variables; for example, previous exposure to accreditation processes and bias due to self-selection amongst the participating and non-participating health services. Hence the necessity to examine associations rather than causality, and to use qualitative and quantitative methods, to increase our understanding of these relationships. To be useful, any research findings will need to quantify the association between accreditation and organisational and individual performances, clarify the actual and potential role of accreditation in evaluating care, and provide an evidence-base for the future development of accreditation in health and other industries. Further, an important methodological outcome will be the trialling of the multi-method research design for future research programs.
Proposed aims and objectives
The proposed research program has two central aims addressed by six specific research objectives. We outline these in turn.
First research aim
The first research aim is to examine the relationships between accreditation status and processes, and the clinical performance and culture of health care organisations.
There are four proposed objectives relating to this first aim. They are as follows:
Research objective 1: To determine whether there is a relationship between accreditation status (as measured by EQuIP) and organisational cultural characteristics
We hypothesise that if the accreditation process is successful in improving the delivery of services through organisational change, then relative performance on EQuIP (based on standards criteria) will be associated with observable health service cultural characteristics. Thus, a health service with exemplary performance on EQuIP should exhibit positive organisational cultural features like sound relationships, positive practices, strong attitudes in favour of continuous improvement and a team-oriented approach to care.
Research objective 2: To assess the relationship between accreditation status and clinical performance
We hypothesise that if the accreditation process is successful in improving the standard of care, then relative performance on EQuIP should be positively associated with improvements in clinical performance. Thus an organisation with exemplary performance on the 19 mandatory EQuIP criteria should have demonstrated improvements in clinical performance, the number of care-related consumer complaints, and the number of sentinel and adverse events, or Coroner's cases which generate recommendations. We would also investigate the individual relationships between individual criterion and indicator levels, for example contrasting the infection control system with the hospital infection rate.
Research objective 3: To analyse the associations between consumer participation, accreditation status and organisational cultural characteristics
We hypothesise that if the accreditation process is successful in promoting participation of consumers, then relative performance on EQuIP should be positively associated with higher-level consumer participation both at individual care level and in broader governance structures. Consumers' participation in their own care has been linked with positive quality of care, treatment outcomes and reduced hospital and medical visits [50–53]. Most commentators assume that consumer participation is positively related to improved performance on standards [54, 55]. However, the relationship between participation of consumers at the care level and in broader system level processes, such as in quality improvement or advisory groups, and performance on standards, have not been effectively examined.
Research objective 4: To evaluate the relative performance, on quality of care measures, between health services participating in and not participating in accreditation
We hypothesise that if the accreditation process is successful in improving the standard and delivery of care, then health services participating in EQuIP should demonstrate better performance on quality care measures than those which do not. The answer to this question will provide comparative evidence vis à vis a sample of controls – that is, those who have never participated in accreditation.
Second research aim
The second research aim is to examine the influence of accreditation surveyors by assessing the reliability of the accreditation process and the effect of accreditation surveyors on their own health organisations. There are two proposed objectives related to this aim. They are as follows:
Research objective 5: To appraise the intra- and inter-rater reliability of EQuIP surveyors and survey teams
We hypothesise that if the EQuIP instrument is reliable, performance on EQuIP should be independent of the different surveying teams. Establishing the reliability of an instrument or process is critical to understanding its limitations. EQuIP is a document-technology that requires interpretation by the surveyors. The reliability of the EQuIP instrument is potentially affected by inconsistency between surveyors. A central question therefore is whether or not different surveyors and different teams of surveyors are reliable judges of health service performance using EQuIP. The results of this investigation would have implications for how surveyors are trained and the tools needed to improve intra- and inter-rater reliability across different settings.
Research objective 6: To examine the relationship between accreditation status, clinical performance, organisational cultural characteristics and the number, network influence and characteristics of surveyors
We hypothesise that the presence and influence of surveyors in an organisation has a positive association with its own health service performance on EQuIP, clinical performance indicators and organisational culture. In this light, a health service with multiple surveyors would presumably benefit in measurably greater ways compared with a health service which had few or no surveyors on staff.
Design
These research objectives require a project utilising a multi-method [56] multi-level [57] approach incorporating multi-layered data [58]. In conducting the research program, a wide range of evaluation techniques need to be applied including more objective measurements, for example clinical indicator data, as well as ethnographic observations. In this way the research will investigate performance in terms of empirical data, to compare what people record, and what people say occurs, and observations of what actually occurs. The strength of this design is that it allows triangulation of results. To this end, four inter-related studies, three prospective studies and one prospective and retrospective study have been designed to meet our aims and objectives (Figure 2). The Human Research Ethics Committee of the University of New South Wales approved the project on 25 May 2005 (HREC 05081). The design features are discussed below.
Study 1: Prospective study of the relationships between accreditation and clinical and organisation performances, and consumer participation profiles
A random stratified sample of 20 currently accredited health services would be prospectively studied at the time of EQuIP assessment. For the measurement of clinical performance, the ACHS clinical indictor data would be independently reviewed by researchers blind to the EQuIP outcome. The EQuIP assessment incorporates submission of clinical performance data collected by health service staff. Data includes operationally defined ratios and scores for clinical indicator performance across a range of clinical areas in a specified time period.
Concurrently, but independently from the EQuIP accreditation process, each health service will be subjected to a comprehensive prospective cultural assessment. This would include direct observation and interviews targeting organisational practices, communication processes, work standardisation, and consumer participation. Previous investigations of work standardisation [59] and cultural analysis [60] provide the basis for the tools and methods to do this. This study would be grounded in ethnography, involving observation of managerial work, interviews with relevant clinician-managers and lay managers, and a survey targeting perceptions of the relevance and effectiveness of accreditation measurements as defined under EQuIP [61, 62]. Other independent, standardised organisational performance data would be collected, for example number of sick days per employee, the rate of injuries to staff, staff turnover, and information about the organisational learning and development program.
Following accreditation the relationships between EQuIP performance and clinical performance and the cultural assessment would be examined. For this study, analysis would involve both quantitative and qualitative techniques. Quantitative analyses would include descriptive statistics and regression analyses. Simultaneously, and blinded to the quantitative analyses, qualitative analyses would be based on grounded theory [63] with both induction and deduction utilised to draw together the empirical data with the theoretical material.
Study 2: Prospective study of health services participating in and not participating in accreditation
All health services not participating in accreditation (EQuIP or otherwise) would be identified. These organisations will be matched with health services which participate in accreditation. These non-participating health services would be subjected to the same analyses as the participating health service (from study 1), that is, subjected to a comprehensive prospective cultural assessment, and subjected to the same review of performance measures. Comparison of the cultural assessments would seek to identify similarities and differences between the organisations.
Study 3: Prospective study of intra- and inter-rater reliability of EQuIP surveyors and survey teams
There would be three parts to this study. Firstly, an examination of survey teams in practice would be undertaken. A sample of health services currently accredited and requiring re-accreditation would be randomly selected for study. Two teams would be matched and undertake the EQuIP surveying process together for two health services. During the surveying process the two teams would independently undertake interviews with relevant health service staff. The genuine accrediting team would be concealed from both the surveyors and health service. The teams would be asked to keep team discussions separate from one another and not to interact at other times. Team ratings and comments on the health services' EQuIP performances would be compared. Observations and interviews with individual team members and the teams as a whole would be undertaken.
Secondly, inter- and intra-rater surveyor reliability will be examined using scenario-rating exercises. This would be done at ACHS surveyor training sessions. Surveyors would be asked to consider de-identified case studies individually and then as a member of an accreditation team, documenting their decisions at each point.
Thirdly, separate focus groups of surveyors would be held to explore their experiences of team-work and decision making processes. Such information would help explain any differences or similarities in reliability and consistency between individuals and teams. The groups would be conducted when the participants meet on a state by state basis around Australia for their yearly training conducted by the ACHS. Participants would be asked to volunteer for the focus groups.
Study 4: Prospective and retrospective study of the organisational influence of accreditation surveyors
Data relating to the ACHS accredited health services in Australia would be analysed to determine whether those with and without multiple, experienced accreditation surveyors have different performance profiles. Potential confounding factors such as health service size and casemix would be controlled for in the analyses. Up to four health services from study 1 would be randomly selected for in-depth prospective case study, involving fieldwork across the sites and qualitative analyses of surveyor influence using network influence theory [64].