Acting on audit & feedback: a qualitative instrumental case study in mental health services in Norway

Design

This was a qualitative, instrumental, multiple case study [33, 34]. The phenomenon we sought insight into was part of an improvement process: how QI teams discussed results from A&F and completed action forms in a “natural setting”.

Study site

The study took place in a DPC in South-Eastern Norway. The DPC is part of a larger hospital trust and consists of four units, two outpatient units and two inpatient units, each representing a case; where specialist mental health services are offered to approximately 72,000 inhabitants. Each unit is subject to various organisational and professional frameworks and is thus considered a separate case. Meanwhile, they belong to the same DPC, with the same management and the same organisational and legal conditions providing a common external conceptual framework connecting the cases and the activities involved.

The two inpatient units had similar staff in terms of number and type and length of education. In Norwegian inpatient units, nurses and social educators (minimum three-year bachelor’s degree) and assistant nurses are commonly referred to as milieu personnel, while psychologists and psychiatrists are often referred to as therapists. The units had somewhat different groups of patients; one had more patients with psychosis-related disorders and another had more patients within general psychiatry or personality disorders. The staff consisted mainly of milieu personnel and 1–2 therapists in both inpatient units. One outpatient unit was a crisis resolution team (CRT), mainly consisting of staff like nurses, social educators or others with similar relevant qualifications, such as social workers, and had a psychologist and psychiatrist attached to the unit. They reached out from the hospital setting to patients mostly referred from GPs or other hospital departments. The second outpatient unit was mainly staffed by therapists such as a psychiatrists, psychologists or specialist nurses, operating as a general psychiatric outpatient clinic with regular hours and booked appointments.

Recruitment and implementation process

An improvement process for the entire DPC was proposed by the leader group of the Hospital Trust in October 2013 and thereafter agreed upon by the DPC later the same year, with an anticipated start in the units in the beginning of 2014. The decision was to implement the National Guideline guided by the implementation process outlined by the National Advisory Unit. The DPC was selected since they were ready to start an improvement process and we were looking for a site to study the implementation process. The implementation process was owned and executed by the DPC and would have taken place without the research project.

The National Advisory Unit had developed tools to support implementation of the National Guideline as part of the development of the guideline. One was a brief description of an implementation process in stages adapted from the implementation of change model of Grol and colleagues [23]. This was accompanied by an electronic survey to audit practice in DPCs and a standardised action form with a template of how to use it. The action form contained columns for areas of improvement, goals, actions, progress plan, main responsibility, economic assessments and evaluation. The survey contained 46 questions about screening practices, assessment of target group, integrated treatment, collaboration, use of evidence-based methods and competence requirements, and was a self-report questionnaire. The audit survey was designed to be applicable to mental health services and was available with templates at the National Advisory Unit web site [19].

An audit of existing practice was conducted in February and March 2014. A project supervisor (MM) from the Health Trust, with special responsibility for concurrent substance abuse and mental health disorders, assisted in the execution of the audit. All four units of the DPC were included in the audit. Feedback of the results was given verbally and in writing by the project supervisor to each unit separately at meetings for the whole unit with unit leaders present. Results were presented at unit level, together with sessions on evidence-based practice, recommendations in the National Guideline and how to conduct an improvement process informed by the Knowledge-to-Action cycle [35]. The meetings were led by the project supervisor in March 2014.

The discussions in the QI meetings were based on the collected audit data and the participants’ professional and organisational experience. A joint action plan for the DPC was adopted in June 2014 with common areas for improvement, actions, work schedule, responsibilities, budget assessment and evaluation, based on the work conducted in the QI teams. Three improvement areas were selected: 1) Screening the use of substances with AUDIT (Alcohol Use Identification Test) and DUDIT (Drug Use Identification Test), 2) Enhancing knowledge of the treatment of concurrent substance abuse and mental illness, and 3) Strengthening integration between the DPC, substance abuse departments and community health care. The actual implementation phase was planned for autumn 2014 to spring 2015, with a re-audit in spring 2015.

One researcher (MSP) spent some time at the study site before the actual data collection started, attending management meetings and talking to the project supervisor, the unit leaders and the department manager. This was to gain an understanding of the general context of the cases.

Data collection

All seven QI teams, with eleven team meetings, were observed, resulting in 11 h of recordings. The meetings lasted between 36 min and 1 h and 52 min. All were held at the hospital, in familiar environments for the teams.

In this study, we mainly used open non-participant observation to describe and investigate an early stage of an improvement process: how and what the staff discussed when aiming to adopt the recommendations in the National Guideline by filling out the action forms as part of completing the audit cycle. To be able to record words rather than summaries, we used an audio recorder together with the observer’s reflection notes. These were always written on the same day as the meeting, to ensure fresh reflections on the atmosphere, surprising statements and agreements or disagreements in the teams.

One observer (MSP) was present at the meetings. The researcher was unfamiliar with the DPC and its staff before the process started. It was considered important to explain that the researcher was not the one to initiate or decide on the improvement process or to perform the A&F. At her first meeting with a team, the researcher introduced herself, shook hands with all the participants and presented the research project in detail. The researcher has a background as an information scientist in medicine and organisational learning, with a focus on evidence-based practice and implementation. The observer sat at the table together with the participants in the meeting. This enabled open observation without unduly disturbing the process.

Data analysis

The audio recordings from the meetings were transcribed verbatim and in detail. Together with the reflection notes, this formed the data base. Such an analysis relies largely on the spoken word, hence the use of an audio recorder. But the reflection notes were also part of the analysis, to make sense of the atmosphere in the room, the relationship between the participants, etc., and to observe what was not verbalised. The computer software QSR International’s NVivo 10 was used to help to organise the analytical process.

A short, but thorough description of the cases was written, together with necessary context.

The analysis was performed in six steps as a thematic analysis suitable for instrumental case studies [33, 37]. Each transcription and reflection notes was read and listened to several times. Codes were assigned to meaningful units in the text, rather generously to ensure that nothing interesting or important was lost at this stage. The codes identified semantic features of the data. Codes were assigned to the data base, case by case. A thorough review of the codes was undertaken, with uncoding and recoding while comparing with the text, to ensure a unified way of labelling and interpreting the text. The third phase was to start searching for themes, by sorting codes into potential themes. The codes could appear across the cases, but were sometimes apparent in just one case and gave a first glance at similarities and differences in the cases. We sought to ascertain whether the codes could form a theme, going back and forth between potential themes, the codes and the dataset. Phase four consisted of reviewing and refining the themes. At this phase, all the data was reread to ensure that the themes fitted with the dataset. In phase five, we defined and named the themes. Phase six was to produce the report. We made adjustments until the end to make sure the analysis and themes had captured the essence.

In a multiple instrumental case study, we are interested in understanding the phenomenon under study [38]. We sought to shed light on the research question by using multiple cases bound together by organisational belonging and the common activity undertaken. We needed to strike a balance between the particular and the common features of each of our four cases while looking for similarities and differences. We mainly reported on general perspectives where present, in order to form an idea of the use of an audit in a DPC as a whole [33, 38, 39].

Eye-opening and sharing practices

When the QI teams gathered, some began immediately to complete the action forms and came back to the audit results afterwards, but most devoted time to the actual process of reflection and discussion of their own unit’s results. They focused on the results showing the largest gaps between recommended and actual practice. It was a realisation of a research-practice gap, an eye-opener.

The staff seemed to become aware of their own practice in a different way through the discussions. Important events in some of the meetings were participants asking about and listening to details of colleagues’ practices. “How do you do this?”, “Are you using the recommended questionnaire or do you ask questions more randomly?” are examples of comments when they began to delve into their own practice. Sometimes they seemed somewhat surprised when listening to each other’s stories about how they perceived everyday tasks or actually performed them in detail. It became clear that practice could sometimes develop in different directions, and that the staff did not necessarily have the same knowledge base for their practice.

Many participants realised for the first time that there existed a standard, i.e. a national guideline; a typical statement was “I wasn’t aware there was a standard for this, I’ve only done what seemed best or what was closest to my previous knowledge or educational background”. Several times there were statements about the newly discovered gap, but they would also literally point at the audit results, holding them up like a proof of practice – or lack of best practice. This awareness seemed to be important in enabling the start of an implementation process, or choosing improvement areas and filling out the action forms as required, representing willingness to take action.

Lack of knowledge and unclear role expectations

Without exception, all QI teams acknowledged their lack of knowledge about the patients with concurrent substance use disorder and mental illness, although some members of staff had experience and expertise in the field. Several of the teams talked about lack of knowledge about substances, their impact and what signs to look for in patients in active intoxication, withdrawal phases or long-term effects.

QI teams from three of the units seemed to be uncertain about the expectations placed on them. What tasks were they expected to perform and what expertise was required in implementing recommendations from the National Guideline? A decision on screening for substance use as an improvement area arose fairly early in the discussions, followed by the question “What do we do when we know?”, i.e. “How do we deal with the awareness that a patient has a concurrent substance use disorder and mental illness?”

In all teams there were discussions related to uncertainty about the participants’ own competence, but they manifested themselves differently in the general outpatient clinic on the one hand, and in the three remaining units on the other. In the latter, the QI teams appeared mainly to be uncertain about what they were supposed to know and work expectations, and we found that the temperature in the discussions sometimes rose concerning expectations from the management and we could see signs of despondency and even anger. These statements usually came after recognition of the research-practice gap from the audit. Some of the employees seemed to feel taken by surprise by the gap, or felt that they should have known already and taken care of this and that this responsibility lay with the management.

We also found insecurity around whether they were actually giving the best treatment to the patients with co-occurring disorders. So when they discussed the audit results, their own practice, and implementation of new recommendations, we noted statements and reflections that what they were used to doing was no longer valid or good enough. We often found that the staff demanded in-service courses and seminars, and stated that they should have been offered this earlier. It was clear that the statements about lack of knowledge arising from the audit results led to a decision to gain enhanced knowledge of the treatment of concurrent substance abuse and mental illness.

The QI teams from the general outpatient clinic also recognised a lack of knowledge in the field of substance use disorders. Otherwise, they recognised themselves as professionals. Their uncertainty was related to whether their expertise was acknowledged by the management and taken into account as they felt it should be. They talked about “we, as professionals”, and we could trace dejection and slight defiance when this topic was discussed in both teams from the general outpatient clinic.

Where to seek answers to clinical questions

The psychiatrists and psychologists seemed to be the ones to “own” the knowledge. They were the ones to keep themselves updated by virtue of education or position. Then the nurses, assistant nurses and social educators could harvest from this knowledge. We found this phenomenon to be present when they were looking for someone to hold educational courses and when they addressed clinical questions to colleagues, and also when they referred to where or whom they got knowledge from. When discussing “best practice”, statements like “psychiatrists have different opinions” appeared. Milieu personnel, nurses and social educators seemed to be used to seeking answers from psychiatrists or psychologists, but found it frustrating when they had divergent opinions on the same matter. It seemed to give rise to confusion and professional insecurity.

None of the professions or QI teams talked about evidence-based practice, systematic reviews or the fact that knowledge might be found in the hospital’s medical library. Some of the teams talked about “building a library”, but less about what should be in it other than certain academic textbooks. There were, however, in some teams references to the National Guideline as a source of knowledge of the area, and it was also held up like a “proof of best practice” with statements like “This is the knowledge base” in a couple of teams. But there were also those who clearly questioned the National Guideline as trustworthy or as a source of best practice.

In the QI teams formed by the outpatient clinic, the discussions were somewhat different. They acknowledged a professional disagreement, and solved the disagreement by either accepting a professional discussion or by simply sweeping the whole disagreement under the rug and accepting differences in practice, mainly due to differences in specialities.

New practices in old systems

All QI teams talked about how the recommendations fitted in with their usual practice and that new ways of working and new practices often felt fragmented and not integrated with current practice. They used the A&F to discuss local practice and possible barriers to new practice, like resources, knowledge of substance abuse, organisation of daily tasks, and lack of time. They assumed that if they put the National Guideline recommendations into practice, it would be uncertain how this would fit in with their usual practice, the way the unit and its tasks were organised – and the way the DPC works. This was perceived as a clear barrier to implementing the recommendations from the Guideline.

At a system level, there were statements about the lack of connected and integrated treatment. Seven questions in the audit survey concerned cooperation and integrated treatment. These particular questions generated discussion in most of the QI teams. All the teams realised that they did not cooperate to any great extent with others, whether GPs, local authorities or substance abuse treatment units. They found that patients with the most severe co-occurring disorders fell between cracks also at a system level, and were thus not receiving appropriate treatment. The audit results were used to discuss how the system worked, especially for people with concurrent substance use disorder and mental illness. Several teams, particularly from the inpatient units, stated that they did not know the system they were working within well enough, and were therefore unable to offer integrated or optimal services, leading to interrupted and fragmented chains of treatment. This was clearly seen as a barrier, but not one they could easily find a solution to. It was considered a management or organisational problem.

Strengths and limitations

A major strength of this study was the natural setting where the A&F cycle took place. This allowed for an exploration of what happened in the QI teams after A&F when the process was organised by the DPC itself, without a research team being part of the project, and thus reflected real people in real settings. In many similar projects, the research team has had an important and visible presence in all parts of the A&F cycle, from setting the standard, auditing practice, giving feedback and facilitating the work of the QI teams and in that way possibly having an impact on the process itself. Case studies are useful for gaining an in-depth understanding of an issue and answering the “how” questions when the behaviour of those involved cannot be, or is not desired to be, manipulated [33]. They focus on contemporary events in real-life contexts, where the phenomenon of interest is interdependent with the context of study [39]. An instrumental case is used to illuminate the “problem”; the case itself is not the most important part [33]. We use the case to understand and answer the research question and it is thus instrumental to accomplishing this understanding [34].

Qualitative research and case studies may be limited in their generalisability. A common conclusion in implementation research is that the effect of the context makes the findings less relevant in other settings. However, we believe that our study has relevance to other settings in and outside mental health care services, and knowledge gained from this study may contribute to the planning and execution of A&F cycles under similar circumstances.

There are certain considerations involved in choosing a multiple case study design. One is vulnerability, in that that the supposedly common cases may turn out to be less common than originally thought in certain aspects, and thus do not shed light on the research question as expected. However, we believe that the cases in this study are quite representative for many Norwegian DPCs in an initial phase of the implementation process, enabling generalisation to other DPCs. We have described the case in as much detail as necessary to account for this risk and make it transparent. A strength of the study is the use of instrumental multiple cases, allowing for a holistic view of the phenomenon, less dependent on the individual context. It is nevertheless worth noting that the cases are linked to a general context, the DPCs being similar in all cases.

There are strengths of open non-participant observation considered important enough to choose it as a method. Open observation allows for conceptualisation of health professionals behaviour and interaction in their natural environment [64] and the direct evidence of observed outcomes and processes, as opposed to reported accounts. It may also capture the flavour of the setting and provide specific examples. Open observation, not driven by theory, was enabled by the relatively demarcated units of analysis, the QI team meetings.

There is no doubt that both the use of an audio recorder and the presence of an observer are very noticeable elements in a meeting room (“the researcher effect”). It is uncertain whether the observer or the audio recorder is the more significant element for the participants. The observer not being a health professional, but with extensive experience from administrative parts of health services, gives a possibility for a fresh look at a health care setting, allowing for new insight. We believe the use of an audio recorder and reflection notes immediately after meetings helped to ensure the verifiability and thus the reliability of the observations.