- Research article
- Open Access
Partners in Recovery: an early phase evaluation of an Australian mental health initiative using program logic and thematic analysis
BMC Health Services Research volume 19, Article number: 524 (2019)
Mental illness is a leading cause of illness and disability and around 75% of people suffering mental illness do not have access to adequate care. In Australia, nearly half the population experiences mental illness at some point in their life. The Australian Government developed a National program called Partners in Recovery (PIR) to support those with severe and persistent mental illness. The program was implemented through 48 consortia across Australia. One of these was led by the Nepean Blue Mountains Medicare Local who adapted the program according to its specific local needs.
We conducted an early evaluation of the PIR program in Nepean Blue Mountains (NBMPIR) using a program logic model (PLM) to frame the evaluation and complemented this with an additional thematic analysis. Participants (n = 73) included clients and carers, program management and staff of the Consortium and other partners and agencies, and clinical, allied health, and other service providers. Our PLM utilised multiple data sources that included document review, open and closed survey questions, and semi-structured interviews. Quantitative data received a descriptive analysis and qualitative data was analysed both in alignment with the PLM framework and inductively.
We aligned our results to PLM domains of inputs, activities, outputs, outcomes and impacts. The NBMPIR consortium implemented a recovery approach and provided greater access to services by enhancing healthcare provider partnerships. Our thematic analysis further described five key themes of collaboration; communication; functioning of PIR; structural/organisational challenges; and understanding of PIR approaches. Facilitators and barriers to the NBMPIR program centred on the alignment of vision and purpose; building an efficient system; getting the message out and sharing information; understanding roles and support and training of staff; building capacity and systems change; addressing service gaps; and engaging peers.
Our study provided helpful insights into the coordinated management of complex mental illness. The NBMPIR’s focus on partnerships and governance, service coordination, and systems change has relevance for others engaged in this work. This PLM effectively mapped the program, including its processes and resources, and is a useful framework providing a baseline for future evaluations.
Full report available at https://researchdirect.westernsydney.edu.au/islandora/object/uws:33977/
The World Health Organisation states that mental illness is the leading cause of ill health and disability worldwide . Moreover, health systems around the world have not responded adequately leaving 75% of those suffering mental illness without treatment [1, 2]. Almost half the Australian population (45.5%) experiences a mental health disorder at some point in their lifetime and 20% of the Australian population aged 16–85 years have experienced a mental disorder in the previous 12 months . There can be far-reaching impacts on those affected, their families and carers, and for the Australian community .
Mental illness is complex with many factors influencing its expression and impacting on management [5,6,7]. Successful management requires individual and systematic approaches that incorporate biological, social and psychological perspectives [8, 9]. These approaches are recommended to include recovery-based, consumer-driven, locally specific approaches focussed on empowerment [10,11,12,13], and coordination and integration of the multiple services required to meet consumer needs [5, 14]. However, barriers to recovery based care models have been identified including management conflicts , difficulties accurately measuring consumer and system outcomes [5, 10, 12, 14] and need for a commonly understood conceptual framework to guide practice [10, 12, 16], challenging the expertise of service providers [12, 17], and failure to value the experiences of street level workers [15, 18] and consumers [16, 19, 20].
Australia has a fragmented health system that is funded by federal, state and local governments [21, 22]. Medicare is Australia’s publically funded universal health care system and provides access to some health services at low or no cost. While public hospitals are managed by the state, most out of hospital and primary and allied health care services are delivered by private providers . Prior to the implementation of a nationally focussed approach, mental health services often failed to adequately address the needs of those with more complex and chronic problems and were not enabled to address social service needs or to “join up” the care across jurisdictions .
Following the launch of a national mental health policy , the Australian Government developed a national framework for recovery-oriented mental health services . As part of this framework, the Partners in Recovery (PIR) program was implemented nationally in late 2012 and aimed to provide better support to people with severe and persistent mental illness, including for their carers and families, using a collaborative, coordinated, and integrated approach .
Forty-eight regional PIR consortia were funded nationally. Each drew on the strengths and resources of the local consortium partners to provide services meeting the needs of the region [24, 27]. Some consortia recruited clients from hospitals and other mental health services while others recruited more from the community including from Aboriginal health services [28,29,30]. Partners in Recovery was defined by a facilitation approach intended to empower clients rather than a case management approach . Support Facilitators were engaged to provide client liaison and education and to coordinate client services .
Evaluations of other PIR programs have noted the central role of the support facilitator [22, 27] and how the role is adapted in different contexts. An effective organisational hierarchy was also considered important to allow the support facilitator role to develop within the program . Although PIR programs have reduced unmet needs and enhanced mental health recovery [29, 32], some have highlighted challenges in reaching certain populations such as Aboriginal people . Some needs such as daytime activities and accommodation were also not always met . Evaluation approaches were found to be inconsistent in some programs limiting the evidence base for these programs .
The Nepean Blue Mountains Medicare Local (NBMML), currently known as the Nepean Blue Mountains Primary Health Network, was the Lead Organisation responsible for implementing PIR in the Nepean Blue Mountains area, west of Sydney. The NBMML was a not for profit primary health care organisation funded by the Australian Government seeking to improve primary health care in its local region. The Nepean Blue Mountains region has a diverse population with wide ranging health needs [33, 34] including 3.7% who identify as Aboriginal and Torres Strait Islander and 22% from culturally and linguistically diverse (CALD) backgrounds, most of whom were born overseas and speak English as a second language . In establishing Nepean Blue Mountains Partners in Recovery (NBMPIR), the NBMML joined with partner organisations including the Nepean Blue Mountains Local Health District, Family and Community Services, RichmondPRA, Uniting Care Mental Health, and Aftercare, as well as consumer representatives to form a consortium. Most consortium organisations were already involved in mental health in the region. This consortium also collaborated with numerous government and non-government (NGO) non-consortium partners including community organisations and other service providers to refer clients to the program and draw on their services. These included Personal Helper and Mentor Support (PHaMS), Housing and Accommodation Support Initiative (HASI) and Centrelink (Australian welfare and employment agency). The key objectives of NBMPIR were focused on:
Partnerships and governance;
Service coordination; and
Our research examines achievement of these key objectives in a region with diverse population demographics. Unique to our research is use of a comprehensive evaluation approach including document review, survey and interviews, framed by a program logic model, and supplemented by an inductive thematic evaluation.
Research aims and objectives
The aim of our research was to conduct an early evaluation of the NBMPIR program, 2 years from commencement of funding.
Our local evaluation of NBMPIR complemented a national evaluation of Partners in Recovery  and reflected on the effectiveness of the locally implemented program at an early stage according to its key objectives, including its facilitators and barriers, in order to provide learning for future operations. In this paper we report the findings of our evaluation and reflect on the value of a program logic model to frame our evaluation. This research is presented in greater detail in the full report .
Dr. Steven Trankle (PhD) has a background in psychology and Professor Jenny Reath has clinical expertise in General Practice medicine. Both researchers have experience in health services research, and with multiple methods, and are engaged as researchers and educators.
A program logic model framework
We engaged a Reference Group of key stakeholder representatives from the NBMML to advise on the planning and implementation of the evaluation. After reviewing the literature, we consulted with the Reference Group and other key stakeholders to develop a program logic model (PLM) of the NBMPIR and assigned indicators for the program’s inputs, activities, outputs, outcomes and impacts (Fig. 1). Our evaluation aimed to measure achievement against each of the assigned indicators (Additional file 1).
Use of PLMs in evaluation is said to “provide learning opportunities, better documentation of outputs and outcomes, and shared knowledge about what works and why” . Program logic approaches to evaluation have been described as helpful in understanding and evaluating complicated and dynamic systems [39, 40] including in mental health domains [41, 42] and also in primary health systems [43,44,45]. A PLM approach was particularly suited to evaluating the NBMPIR in a demographically diverse local community with complex needs.
We received ethics approval from the following Human Research Ethics Committees:
Western Sydney University Human Research Ethics Committee (H10697); and
Nepean Blue Mountains Local Health District Human Research Ethics Committee (14/49).
This research also received approval from the Nepean Blue Mountains Local Health District Scientific Advisory Committee.
Data collection and analysis
Review of policy, operational and reporting documents of the NBMPIR;
Quantitative and qualitative survey data; and
Qualitative interview data.
Our reporting of qualitative data aligns with the COREQ guide .
Document identification and analysis
Information sources were identified in consultation with the Reference Group to measure indicators related to the NBMPIR and included a range of governance and organisational documents such as Memoranda of Understanding, Service Level Agreements, terms of reference, meeting minutes, and relevant mandated reporting documents to Department of Health such as quarterly performance reports, as well as other reports of activities and plans (Additional file 2).
The survey included both Likert-based questions and open-ended questions aligned to the PLM indicators (Additional file 3). Questions centred on participant roles and their experiences and perceptions of implementing the NBMPIR program. It was circulated in on-line and paper-based formats.
A semi-structured interview guide was developed to further assess the PLM indicators (Additional file 4). This provided in-depth responses to those issues canvassed in the survey, and an opportunity to explore perceptions particular to the individual . The guide was piloted in the first 5 interviews to ensure it adequately covered each stakeholder group. Interviews were between 30 and 60 min in duration and conducted one-on-one and face-to-face by ST at NBMML or health care provider offices. All interviews were audio-taped and transcribed verbatim by an independent transcription service. Transcripts were then checked by ST for accuracy. Participants were also offered their transcripts to check accuracy.
We used two approaches to analysis of the data, an initial deductive approach using the PLM as a theoretical framework for data analysis and a second thematic analysis of qualitative survey responses and interview data.
In the framework analysis we mapped our data sources against each element of the PLM [38, 49]. In analysing the data collected we used a descriptive approach to report responses to closed survey questions and integrated qualitative information from free text responses and interviews as well as data from our document review.
The second inductive approach, utilised a separate thematic analysis of qualitative responses according to established protocols [50,51,52] using N-Vivo 10®. We explored emergent themes across all survey and interview data and noted discrepant responses. Any unrepresentative quotes were identified as such. The initial five interview transcripts were coded independently by ST and JR and a third researcher (MK) and we then also cross-coded each other’s. We discussed our analysis and agreed on potential themes which provided a framework for further interview analysis. After interviewing concluded, we iteratively read and re-read each transcript and coded until we achieved saturation of the main themes. We then reviewed the coding frame again in order to confirm coding and the positioning of subthemes within main themes (Additional file 5). Lastly, we provided a selection of powerful and compelling quotes from all participants to represent the themes.
Participants were purposively selected across a variety of roles within PIR rather than on the basis of demographic characteristics. This ensured we adequately sampled from the available stakeholders with experience of the program. They included clients and carers, management and staff of the NBMML, Consortium and other partners and agencies, and a range of clinical and allied health, and other service providers. NBMPIR recruited participants and provided them with an ethics approved letter of invitation, information and consent sheet, and an online survey link or a mailed paper copy as preferred. To protect confidentiality NBMPIR did not know who completed the survey as this was administered by the researchers. Participants were recruited for interviews through an invitation in the survey requesting their contact information. Interview participants were then contacted directly by ST to schedule their interview. Signed consent was obtained for interviews and hard copy surveys. The information sheet provided by NBMPIR also included information about the researchers. Participants did not know the researchers prior to the study. No participants withdrew from the study.
A total of 73 participants completed the survey and 17 of those participated in an interview (Table 1). Community representatives worked in consumer advocacy and support roles, with some being employed by other mental health services, and included disability advocates, consumer representatives, and local government community development officers. Consortium partners and staff were involved in an administrative capacity and not as health care providers. The Support Facilitator role provided liaison between clients and services.
Our results are presented in two sections reflecting our use of two approaches to analysing the data we collected. We provide a PLM framework analysis first and then a separate thematic analysis.
Program logic model - framework analysis
In this analysis we describe how the documents, survey and interview responses address each of the PLM indicators in terms of program Inputs, Activities, and Outputs and, in a more limited way, Outcomes and Impacts (Additional file 1). Findings related to these longer term program effects will require later evaluation.
Program logic model inputs are the “human, financial, organizational, and community resources a program has available to direct toward doing the work” . Key inputs identified in our NBMPIR PLM were funding, management and governance structures, staffing, community and consumer stakeholders, and information technology.
The documentation we reviewed identified key areas of expenditure and amounts disbursed and all respondents agreed funding was adequate. At this early stage of implementation, difficulties were identified with providing partner organisations with access to this funding.
Clinical and corporate governance protocols had been established and were well documented, with most survey respondents agreeing that organisation and management of PIR was open and transparent. A client management information system (CMIS) called PENELOPE recorded client information and progress through the program.
Interviewees described a collaborative respectful approach with one interviewee commenting: “there is a strong and robust relationship within the Consortium working towards common goals for the people that we work with” (NBMML staff). However, one Support Facilitator noted some instability, commenting: “there’s been lots of chops and changes in the [Lead Organisation] management”.
Roles and responsibilities of key service personnel were identified in reviewed documents and these informed work practices Surveyed NBMML PIR staff and management agreed or strongly agreed that they had clear job descriptions (9/10) and that their practice matched their defined roles (8/10) and consortium staff similarly agreed their PIR roles were clear (6/8).
In terms of community and consumer engagement, community-based NGOs were engaged in the Consortium, and client and carer representative positions were filled. Community promotion of PIR was well attended through press releases, newsletters and community fora across all local government areas.
All clients (11/11), and most carer (3/4) and community representative (5/6) survey respondents agreed or strongly agreed that PIR sought consumers’ views and most agreed or strongly agreed that they had sufficient opportunity to provide feedback aimed at improving NBMPIR.
Information technology was a key input and training was provided for staff particularly in use of the CMIS - PENELOPE. Half (5/10) of the NBMML management and staff survey respondents agreed that IT and training met their requirements and most agreed that IT assisted communication, and was used efficiently. Interview data supported these findings with staff and relevant stakeholders reporting satisfaction with IT and the training provided: “I’m finding PENELOPE really helpful and I think it’s great. I think it’s really, really good and the [IT] support we’re getting now is really great” (Support Facilitator).
Program logic model activities are the “processes, tools, events, technology, and actions that are used to bring about the intended program changes or results” . Performance indicators in our PLM were related to program planning and development, client needs identification, and stakeholder education and support.
Program planning and development was attended through establishment of a commonly understood framework of language; consultation with service providers, community and researchers to inform ongoing program implementation; development of subprograms; and implementation of strategies to inform stakeholders of the program.
A clearly operationalised recovery focus informed key documents such as the PIR service manual and aligned strongly to national PIR policy documents and the recovery language guide from the Mental Health Coordinating Council. Half of the health care providers (8/16), most staff and management from the NBMML (7/10) and consortium partners (13/16), and all community representatives and other non-consortium partner staff agreed the framework of language assisted in building an understanding of PIR. A surveyed consortium staff member said: “Language is consistent and clear. Definitions and meanings are clear” while, in contrast, a NBMML manager noted: “a shared use of language has been established and is important but it isn’t necessarily a shared understanding”. Concepts of recovery appeared to be understood by most with descriptions provided by interviewees such as: “empowerment – having clients say what their definition of recovery is” (Community worker). However, implementing recovery based approaches was sometimes challenging: “it’s something that we’re really struggling to build in with that kind of overarching medical model” (NBMML staff).
Monthly Consortium and Support Facilitator Working Group meetings, which included consumer and carer representatives, fostered collaboration and facilitated consultation with key stakeholders. Most clients (3/5), carers (2/2) and community representatives (3/5) surveyed agreed PIR engaged them in planning programs. This contrasted with health care providers, most of whom disagreed with this statement (8/15). Though a communication strategy had been developed by NBMPIR, at times external stakeholders found the communications inadequate: “they first introduced themselves last year, maybe mid last year, but then we didn’t hear anything further from them” (Consortium staff). Most respondents noted strong common interests and goals among Consortium members through comments such as: “When we’re raising things at the Consortium, nearly everyone’s saying yes, that’s an initiative we have to fulfil, that’s a KPI for us - seems to be synergy there” (Consortium manager).
Subprograms focused on systems change were being developed such as Capacity Building, enhanced Consumer/Carer engagement, Smoking Cessation and Physical Fitness and Co-location of NBMML staff with Consortium and other provider organisations . Discussion about systems change occurred with senior management of all stakeholder organisations. Interviewees spoke about their understanding of systems change with collaboration a common theme: “Just getting people to work together. No longer that silo effect” (Support Facilitator).
Promotional strategies were tailored to suit different referrers and a website and quarterly newsletters provided information. Within 2 months a rapidly growing client waiting list required temporary suspension of promotion.
Client needs and eligibility were identified at intake through the Camberwell Assessment of Need Short Appraisal Schedule (CANSAS). Interviewees noted that CANSAS did not capture enough information and also described contractual constraints:
We didn’t have as much freedom as we might have had in relation to that because of the requirement to use CANSAS …it’s not really a tool with the consumer in mind. There’s nothing that says this is how people’s lives have changed (NBMML manager).
An individualised recovery plan was developed for coordinated support, and periodic assessments undertaken to provide information on client progress. Interviewees highlighted the importance of considering context: “I think that people are still finding out what the local needs and conditions are. I think we’re trying to be fluid enough to keep up” (NBMML staff).
Tailored stakeholder education and support was provided and a comprehensive Education Plan had been finalised at the time of our evaluation. Most of those surveyed agreed that support and education were satisfactory, although some health care providers disagreed (6/16) or responded neutrally (7/16) to this question. Survey respondents across stakeholder groups contrasted with interviewees who described the support and education provided to staff and other stakeholders as inadequate. Comments included: “I was thrown in at the deep end” (NBMML staff), and “I have received nil education and support from PIR workers” (Health care provider). Health care providers were particularly varied in their views about education and support with one reporting: “We also had quite a lot of written material around what Partners in Recovery does” whilst another suggested: “a flyer and referral information would probably be quite helpful”.
Program logic model outputs are the “direct products of activities including new resources, services and programs delivered by the NBMML” . Performance indicators in our PLM related to how the program was implemented and operating, including staff orientation and support, client intake, referral pathways, stakeholder engagement and satisfaction, and perceived efficiency of PIR operations and to evaluation of the program.
Reviewed documents revealed that most staff were appointed as planned and all relevant staff including Support Facilitators were trained in CMIS and the PIR Mental Health Care Coordination strategy. Some reported having existing skills: “I’ve been involved in homelessness provision for a long time. So when PIR came along, it felt natural with my own personal professional journey” (Consortium staff).
The majority of clients were referred from community based mental health services and other community services. Client intake recorded in the CMIS indicated that demand was higher than expected in all but one more geographically isolated area of NBMML. This illustrated that intake pathways were working. However, it was also noted that people who were homeless or destitute may not be accessing the program (NBMPIR Annual Activity Work Plan 2014–2015). By the end of 2015 all planned Support Facilitators had been engaged and the waiting list was reduced.
Interviewees commented on the effectiveness of referral pathways: “It’s quite easy. The referral pathways are something that, I think, the original information explains very well (Consortium manager)”. Support Facilitators considered these pathways a result of their work “the more Support Facilitators there’s been the more referral pathways that are being created”, although some health care providers were unsure of referral pathways: “I really don’t know, to be honest”.
Although interviewees reported limited stakeholder engagement in the program at this early stage, their experiences where this occurred were generally positive. One interviewee commenting on PIR staff noted: “people were really motivated and enthusiastic about getting everyone together. I think that was really good” (Carer). A health care provider with a client who was very reluctant to engage praised the Support Facilitator saying: “he’s great, really great. My client has a really good rapport with him and he’s quite aware of her needs, her limitations and things like that which has been really helpful”. One client highlighted the importance of engagement: “I think it depends on the amount of input I put in as well. So yeah, I’m satisfied with my engagement so far”.
Although most PIR staff and management in non-consortium partners agreed that implementation of PIR in the early stages was effective (4/5), responses from those more involved in the daily operations were less positive. At interview a consortium manager involved from the beginning commented: “It was effective in bringing the partners together in the one place, bringing senior figures together and engaging in each other’s strategies, having that sense of being in it together”. Whilst the partnerships were attended, provision of services appeared to take longer with another consortium manager commenting: “The services side probably took a bit longer to set up than expected”.
A key PLM indicator related to efficiency and cost effectiveness and reports indicated that the budget was not fully expended at the time of our evaluation. Some interviewees identified difficulties accessing funds from NBMML: “for the first eight or nine months we had no access to funding so it was really difficult” (Consortium manager). These early challenges with disbursement of funds may have impacted on survey responses indicating that only 50% of NBMML respondents believed PIR was efficient and cost effective.
Evaluation of the program was another key PLM indicator and included active monitoring of the program and use of the data to inform further development.
CMIS data provided detailed information about consumer recovery pathways and indication of where gaps were occurring such as for homeless people. However, there was little information about Aboriginal and Torres Strait Islander and CALD consumers use of the program (NBMPIR 6 Monthly Performance Report - Qualitative). Interviewees reported that evaluation activities were common within PIR and were often focused on specific stakeholder groups: “just about to implement a three-month consumer feedback form – to be given to all consumers” (NBMML staff).
Evaluation informed future program development with documents describing ongoing review of operations, including through use of CMIS data (NBMPIR 6 Monthly Performance Report - Qualitative). These reviews resulted in improvements such as revised referral forms and development of a waitlist policy. Interviewees described this program development: “With PENELOPE data as well, that would assist us in moving forward to tweak and - or change what we’re doing” (NBMML manager).
This evaluation focus was noted by survey respondents apart from health care providers and interview data reflected these views. One consortium manager stated: “All quality improvement initiatives are evaluated and measured” with a health care provider reporting: “I have not been asked for feedback by PIR workers or management before completing this survey”.
Most stakeholders were satisfied with their experience of evaluation. Commenting on the current Western Sydney University evaluation, one health care provider said: “like this evaluation process? Which is good, excellent, I think that’s great, I haven’t seen that happen before with any other service. That impresses me”. Similarly, a Support Facilitator commented: “I quite like the extensiveness of who is giving feedback and how the program is being evaluated”.
Program logic model outcomes are the “specific changes in program participants’ behaviour, knowledge, skills, status and level of functioning” . Although this was an early evaluation of PIR, some outcomes were described in accordance with PLM indicators including stakeholder knowledge of the program, consumer access to PIR services, program achieving its stated aims in terms of coordinated care, health status and functioning of consumers, and improvements in service provider knowledge, skills, functioning and collaboration.
Ten percent of referrals to NBMPIR were from clients and carers suggesting some level of awareness about the program amongst these stakeholder groups (NBMPIR 6 Monthly Performance Report - Qualitative) which was also supported by survey responses and in interviews. One carer said: “I think it’s good. I never knew anything about it. The mental health team rang me and asked me if I would go in it and I did. The information is good”, while a community worker similarly noted: “My knowledge of the program has changed from being kind of hypothetical to being practical - satisfied for sure”. Health care providers also indicated their increased awareness: “I’ve actively made referrals to PIR to try and address the identified need for the client. So that’s changed” although one health care provider commented: “I’ve been trying to find out more but the more I find out, the more confused I am. So it hasn’t changed, I guess”.
Consumers could better access PIR services according to all staff and management of stakeholder partners, and most health care providers (8/15). Clients (7/8) and carers (2/4) also agreed that PIR had assisted them in getting the right services although survey respondents described a need for widespread promotion of NBMPIR to increase consumer access to PIR services: “from what I have seen so far it’s about increasing the public awareness of PIR; other service providers and LHD’s [Local Health Districts] need to be advised on what PIR is about” (NBMML staff).
Interviews with consumers and carers revealed variation in access to PIR services. One client reported: “Directly through my case manager. And that’s pretty much through a phone call, whilst a carer had a contrasting experience: "I just found that Nepean [PIR] took a long time to get back to me”.
The program was widely observed to have achieved its stated aims with most respondents considering PIR effective in coordinating support for people with severe and persistent mental illness. Organisational staff commented “It’s certainly effective from the point of view of putting the person in the driving seat of their recovery” (Consortium manager), and “It is effective and that’s not just evidenced with how well our consumers are doing, it’s also evidenced within the relationship of the consortium and other NGOs - we’re able to build synergies with other organisations, it’s brilliant actually” (NBMML staff). A client also reported how PIR had helped him:
PIR funded additional sessions with my psychologist when I had a break down. Having that support kept me from self-harm or other destructive behaviours that have been a coping mechanism for me in the past. They also helped me find a new house, which is huge. My worker did all the stressful things which let me focus on the day to day. I am grateful.
Others however described reservations: “Inconsistent. I’ve had some fabulous experiences with Partners in Recovery but there’s been some not so good ones” (Health care provider).
Consumer health status and functioning had improved through engaging with PIR according to most surveyed clients (5/8) and carers (2/3). Interviewees generally commented positively on their interaction with PIR: “They have helped me significantly in many ways. I have felt supported and safe with my PIR worker” (Client). “My brother is already showing signs that Partners in Recovery have helped him greatly” (Carer). Carers also noted reduced burden: “It gives me a break from being a carer” (Carer).
Most clients (5/8) and carers (2/3) agreed they had more hope for their future health and functioning as a result of PIR. One client said: “PIR helped me gain independence by living alone. I now feel more confident about my abilities to live normally without support”, while another offered to engage in peer work by helping others: “…I’ve actually spoken with my case manager about trying to do some volunteer work myself”.
Provider knowledge, skills and level of functioning had improved for most surveyed consortium (8/13) and non-consortium partner staff (3/4). However, this was different for health care providers who equally disagreed (7/14) or were neutral (7/14) with this survey statement. Interviews contrasted with these responses with one health care provider describing self-improvement: “Modelling of good care coordination has been something that I’ve picked up personally; and every skill of mine - that’s improved through my contact with Partners in Recovery”.
New and more effective partnerships helped staff to meet the needs of consumers. Most surveyed consortium (10/13) and non-consortium partner (3/4) staff and management agreed that PIR had assisted them with partnership engagement. Although health care providers mostly disagreed (8/15) that PIR had assisted them in this way, qualitative survey feedback was sometimes positive:
It certainly extends your knowledge of all of the different services available - aged care providers, and disability services, and all kinds of other areas that we don’t get a lot of training and don’t necessarily have a lot of contact with (Health care provider).
Program logic model impacts refer to the “fundamental changes occurring in organizations, communities or systems as a result of program activities” . The indicators described for this domain related to improved community health and well-being, and better integration and coordination of health services. At the time of our evaluation, we did not expect to find evidence of impacts. However, survey and interview data suggested some early and potential program impacts.
To understand enhanced local community health and well-being, we asked about improvement in access to required services and supports. Most surveyed community representatives (3/5), clients (6/8) and all carers agreed that PIR has resulted in sustained improvement in access and non-consortium partner staff and management also agreed with this statement (3/4). Interviewees provided examples: “Being put in touch with the organisation has provided me with accommodation. They provided a few other services as well which I have occasionally accessed” (Client), and from a health care provider: “I can only speak about this one case. The client is now able to advocate for themselves, the client now has more agency and understands how to work with the services. Generally speaking, I think it’s working really well”.
Our survey also explored referral pathways for consumers of CALD and Indigenous backgrounds, with 50–60% participants across all stakeholder groups responding neutrally to this question. Qualitative feedback suggested this would be a later focus of the program: “CALD communities have not been a major focus as yet of NBMPIR probably due to the relatively small percentage of this group in regards to the rest of Sydney” (NBMML manager), and would require employment of CALD and Aboriginal staff: “We have NO CALD staff and NO Indigenous identified staff, how are we meant to engage with these communities without this” (NBMML staff)?
We looked for evidence of integrated and coordinated health services through improved team work and better care coordination, as well as improved client access to housing, employment, education and social activities, and evidence of clinical and community support services operating according to a community-based recovery model.
Improved team work was seen with most surveyed NBMML (8/10), consortium (7/13) and other non-consortium partner staff and management (3/4) agreeing that PIR had assisted them in networking with other stakeholders in order to respond to consumer needs. Health care providers also described improved team work in their interviews: “In the past it’s been hard to get everyone working together, it appears to be a huge improvement”.
However, at interview perceptions varied and some were critical about this aspect of the program: “We’re getting quite a lot of resistance from people saying, you know, it’s not your role, it’s not my role or you shouldn’t be doing that or in some cases they’ve said we’re not going to refer to you” (Support Facilitator). Interviewees also described how siloed and entrenched practices were difficult to change: “Changing the attitudes of people who have been in the industry for 20 or 30 years and have been trained and practiced with that medical model for so long, it’s very difficult to shift” (Community health worker).
Partners in recovery had improved consumer access to integrated services that addressed multiple needs according to most surveyed staff and management of the Lead Organisation (8/10), consortium (7/13) and other non-consortium partners (3/4), as well as clients (7/8) and carers (3/3). Community representatives provided a neutral response neither agreeing nor disagreeing as to any improvement in this area (5/5). Interviews revealed difficulties as described by one community worker: “We get a lot of homeless people come in here in crisis, so we’re dealing with crisis situations, and a carer said: "It’s social I need because he can’t work and there’s no way he can live by himself. They’ve got to have social interaction”.
The recovery model of care was understood well by most surveyed staff and management of the Consortium (12/13) and other non-consortium partners (4/4), and health care providers (11/14). However, interviews suggested that recovery based services were not universally accepted: “It’s mixed- there’ll be some pockets where the focus is more clinical than recovery focused” (NBMML Manager). One respondent described a pro-active response to this issue: “Part of our systems change is addressing that and at least providing modelling for what recovery looks like” (NBMML staff), and early signs of change were noted by others:
Health in the last couple of years has made a really big effort to kind of change the culture of the services that we provide, and really kind of try to build in that recovery-oriented focus. But at the same time health is also a beast that moves very, very slowly. And it is very difficult to change that culture (Health care provider).
To complement the PLM analysis, we conducted a separate thematic analysis on the 17 transcribed interviews and on the open ended survey responses provided . This enabled us to deepen our understanding of the NBMPIR program. Five key themes were identified, specifically: collaboration; communication; functioning of PIR; structural/organisational challenges; and understandings of PIR approaches. Each of the key themes was further elaborated by a range of subthemes which are described below.
We found that collaboration improved among different providers as the program became more established, although not all interviewees agreed this was the case. The NBMML was, however, considered instrumental in reducing siloing (Table 2).
Communication was identified as key to effective operation of PIR. Communication was improving across provider networks and with consumers who valued NBMPIR. Program staff were providing feedback to enhance the program and the consumer voice was regarded as crucial including in planning stages, however, promotion of NBMPIR did not reach all stakeholders initially (Table 3).
Functioning of PIR
Use of recovery language and practice using a recovery framework were becoming more common. Consumers were increasingly involved in the program’s implementation and considerable improvements were noted in their mental health status. Referrals into the program increased rapidly from a wide range of sources especially through Support Facilitators who were developing new service partnerships, however a range of service gaps were identified (Table 4).
Structure and organisational challenges were frequently mentioned by respondents. In this early phase, bureaucratic processes and lack of clear guidelines made accessing resources difficult while IT and communication systems were considered inefficient. To counter a perceived lack of provider education, NBMPIR co-located Support Facilitators across the provider network. The need for long term approaches was identified to achieve sustained improvements in consumer outcomes and to change entrenched attitudes averse to systems changes and recovery approaches (Table 5).
Understandings of PIR approaches
In the interviews, all respondents provided insights as to their understanding of PIR approaches. Concepts such as recovery and its associated language, consumer directed, coordinated care, and systems change appeared to be well understood and consistent with definitions found in the literature and documents provided by the funding body. However, a lack of understanding by some respondents as to the purpose of PIR was also identified (Table 6).
The NBMPIR is part of an Australia-wide mental health initiative implemented by 48 separate consortia . The principles underpinning PIR in Australia are the same key organising principles utilised by similar mental health initiatives in other countries [53, 54]. These principles emphasise a model that is client-driven, locally specific, focussed on empowerment [10,11,12,13], and that coordinates and integrates the multiple services required to meet client needs [5, 14]. Although this was an early evaluation of the NBMPIR, we found this program had established a consortium and local partnerships which, working together on a recovery approach, was facilitating access to a range of services to meet the needs of an increasing number of local clients with severe and persistent mental illness.
The following discussion is organised according to the aims of the NBMPIR and our evaluation. This includes the establishment of partnerships and governance, improving service coordination and creating systems change. Consideration is given to the facilitators and barriers in implementing NBMPIR and to program evaluation.
Partnerships and governance
A key objective of NBMPIR related to establishing partnerships and a strong and accountable organisational structure. Two years from receipt of funding, and 1 year since accepting the first client, we found NBMPIR had devoted much time to the governance and management of the Consortium and to establishing a reporting framework. Conflict both within and across organisations is described in the literature as an impediment for effective recovery based service provision [15, 55] that is addressed by clarifying roles and expectations . To this end NBMPIR signed Memoranda of Understanding and Service Level Agreements with all Consortium service provider partners.
Core to PIR is recovery focussed care facilitation, yet throughout the literature there is widespread inconsistency in the understanding and use of recovery language and practices [15, 16, 56,57,58,59,60,61,62]. Nepean Blue Mountains Partners in Recovery ensured that a clearly operationalised recovery focus, aligned strongly with the national PIR framework , informed its key documents and guided the implementation of its program. Our informants also confirm that PIR program implementation was consistent with care coordination and recovery support models described in the literature [57, 63, 64]. We identified a range of recovery oriented initiatives in the literature, a number of which were being implemented by NBMPIR. These include peer mentoring which has been reported as beneficial to clients in their recovery journey [65,66,67]. The NBMPIR have also used peer mentoring of service providers as a means of enhancing the understanding and practice of recovery by co-locating Support Facilitators within partner organisations in an educational and coordinating role. Co-location also helps to integrate care from the bottom up by breaking down barriers between services and clinicians .
Clients engaging with PIR are those with severe and persistent mental illness and often present in crisis, requiring immediate services that can be flexibly provided [13, 15]. Nepean Blue Mountains Partners in Recovery established a limited pool of flexible funding to address immediate needs such as crisis accommodation and medications [37, 68]. This allowed NBMPIR to assist clients in need despite client intake at the time of evaluation being at capacity [37, 68].
Systems change is acknowledged as long term , however, some evidence of early systemic changes emerged from our evaluation. We noted the increased use of recovery language and practices across the consortium and that siloing of health care providers was being addressed through greater collaboration. We identified improved access to services as a result of enhanced referral pathways and partnerships.
Facilitators and barriers to implementing PIR in the NBM area
Drawing on the findings of our evaluation including our literature review , we identified key facilitators and barriers for NBMPIR. As barriers often result from an absence of facilitators, we considered these together under the following headings related to the key areas our participants described: Alignment of vision and purpose; building an efficient system; getting the message out and sharing information; understanding roles and support and training of staff; building capacity and systems change; addressing service gaps and engaging peers. We consider these findings in relation to the literature, noting areas of common experience and also issues that were specific to NBMPIR. Finally, we reflect on the importance of evaluation.
Alignment of vision and purpose
Consortium partners of NBMPIR generally had a similar reform agenda - placing the client at the centre of a recovery based approach to service delivery. Partners also brought specific areas of expertise and unique resources. This enabled the Consortium to address the complex needs of those with severe and persistent mental illness. The importance of such a shared vision and aligned efforts among partner organisations is well recognised . However, barriers to successful collaboration in NBMPIR were observed with some partner approaches to service delivery less well aligned with the PIR model. When there are differences in ideologies, or organisational culture, and no shared philosophy or common goal, the beliefs and attitudes of individuals can adversely impact collaborative service delivery .
Building an efficient system
There was a marked change in NBMPIR system efficiency when funding was accessed to employ additional Support Facilitators, enabling greater consumer access. System capacity was also expanded by funding support projects and training for partner staff. Interviews further revealed that Support Facilitators supported this training and the development of service partnerships at ground level, an important consideration for integrated recovery-based care [15, 18].
Conversely, many interview and survey respondents commented on bureaucracy and red tape in the early program stages, which adversely impacted decision making. Respondents also commented on procedural guidelines which were unclear and caused confusion. Perceptions of risk with decisions can create avoidance [55, 71], and NBMPIR was observed to be slow in some of its decision making due to perceived risk. In clinical contexts these challenges are reportedly addressed through a shared decision making process which can also enhance efficiency and transparency .
Although the NBMPIR Consortium was operating within its means, its effectiveness may have been compromised in the early stages by a substantial amount of unspent funding. The lengthy waitlist after only 2 months of operations suggests that the system could not cope with the recruited client numbers. When funding was released and additional Support Facilitators recruited, the waiting list rapidly reduced.
Getting the message out and sharing information
Initial communication and promotion of NBMPIR strongly engaged local communities and was received enthusiastically. A risk noted in our evaluation was the relative lack of subsequent communication particularly with health care providers. They were often quite negative in describing their engagement with NBMPIR which was felt to be inconsistent, with contact and provision of information infrequent. Though the future communications strategy is targeted to specific stakeholders, especially service providers, it will be important to further strengthen and sustain processes for information sharing, as this underpins effective partnerships and integrated health care . It is also important that recovery principles receive continued promotion to sustain the PIR model. Indeed, the Support Facilitators in NBMPIR, like their counterparts in the Australian Capital Territory, were actively taking on this task and this contrasted with other PIR programs in western Sydney where promotion was considered inadequate [22, 31].
Understanding roles and supporting and training staff
Most respondents confirmed they understood their role well, although it was also reported that some Support Facilitators were still working in a case management rather than facilitator role. Respondents generally felt that understanding their roles and responsibilities led to effective communication both within the NBMML and across partner organisations. Clarity of individual and organisational roles and responsibilities is an important facilitator of partnerships, enhancing teamwork and job satisfaction  with inadequate knowledge or understanding of others’ roles suggested as causing unclear expectations or negative perceptions and non-collaboration . The Co-location project enables Support Facilitators to work in an educational capacity with partner organisations; however, it will be crucial that their roles are clearly defined and understood.
Well-resourced and evidence-based training is important for knowledge transfer and understanding and implementation of recovery practice [12, 73,74,75]. Survey and interview respondents described challenges including minimal training and a reliance on existing personal skills, though NBMPIR recently finalised an education plan tailored to staff of each stakeholder and provider group.
It is recognised that training must be aligned with the recovery model [63, 76] which can be a challenge in light of the lack of a commonly understood conceptual framework to guide practice [10, 12, 16]. Program documents were aligned with the recovery language guide from the Mental Health Coordinating Council - the peak body for community mental health organisations in New South Wales, and Support Facilitators further modelled recovery based practices . Survey and interview respondents noted a widespread understanding and use of recovery language and recovery-based practices.
Building capacity and systems change
The NBMPIR were developing and starting to rollout a number of projects aimed at strengthening recovery approaches and building service capacity in the region. Systems change projects such as co-location will enhance care coordination, whilst smoking cessation and physical fitness programs that include stakeholder training were designed to reduce hospital admissions and mortality . The NBMPIR recognised the importance of the consumer role, and a client/carer regional engagement program was intended to grow a resource of skilled advocates and engage them in leadership roles . The consumer voice at the organisational level is as important as consumer centeredness is at the service level. Indeed, ground level knowledge and information sharing has been identified as crucial [18, 70] and those who have had direct experience relating to mental health can provide powerful insights .
Addressing service gaps
Clearly, NBMPIR was making a positive difference to the lives of consumers and assisting the effective networking of providers, although, this was a work in progress. Given the short time that PIR had been operating, this program was progressing well.
The NBMPIR were aware of the challenges they face in service delivery especially given the geographical and population diversity of the region. The NBM population has a significant proportion of CALD and Aboriginal and Torres Strait Islander people and these groups have poorer access to mental health care and higher rates of suicidality [77, 78]. Our evaluation at an early stage of PIR did not identify any programs specifically targeting CALD and Indigenous needs. Furthermore, survey and interview respondents indicated a lack of focus on these populations and an absence of CALD and Aboriginal and Torres Strait Islander staff at this time who might facilitate PIR access for these communities. These are important considerations for the NBMPIR but also for similar programs with CALD and Indigenous populations in ensuring that unique needs can be understood and addressed in a culturally appropriate manner. Some groups in the community such as the homeless and those with dual diagnosis were also observed to be missing out on services. Other challenges in access included transport, housing, and social opportunities-the latter was also noted in other PIR programs as an unmet need .
Peer support is well evidenced to facilitate recovery approaches in areas of comorbid substance abuse and mental illness and is particularly effective when utilising peers who have been on similar recovery journeys themselves [65,66,67]. The Wellness Recovery Action Plan (WRAP) is a particularly effective way of utilising peers to help develop consumer strengths [79, 80]. Our interviews suggested that already at this early stage one client engaged with NBMPIR was ready and willing to assist in this way. Should this response be reproduced with other client/carers this will not only help with recovery of others, but will also increase the capacity and efficacy of a very relevant support network.
Evaluation is often a challenge in implementing a recovery approach [5, 10, 12, 14] including in other PIR programs . Nepean Blue Mountains Partners in Recovery used instruments such as CANSAS and developed databases such as PENELOPE which assist with evaluation. Evaluation activities were embedded within all PIR support programs with quality improvement measured against key performance indicators.
Importantly, the PIR evaluation reported in this paper provided a comprehensive understanding of the early implementation of the NBMPIR. Mapping the program using a PLM was helpful in ensuring all elements of this complex program were considered . Understanding was enhanced through collection of data through document review, surveys and interviews and later thematic analysis. We reported contrasting views and experiences of the program across different participant groups. We often noted that different roles and experiences in NBMPIR elicited different responses to our questions and we could gather more nuanced responses from those participants who answered open-ended survey and in-depth interview questions.
It is important to note that this was an early evaluation of a local program and findings are limited to that context despite their potential relevance for other similar programs. Other programs may have different populations and services that would influence program implementation. Although NBMPIR had developed relationships with a range of health care providers, at the time of evaluation few General Practitioners were engaged with the program. General Practitioners, as key providers of primary health care in the Australian setting, are a source of referral and also important in following up clients engaged with the program. Future evaluations could gain another important perspective on the operations of NBMPIR by collecting information from General Practitioners.
This study contributes much needed knowledge about the complexities of consumer-centred localised service delivery innovation in mental health care. Although this was an early evaluation of the NBMPIR, we found this program had established a consortium and local partnerships which worked together in a recovery approach to facilitate access to a range of services for an increasing number of clients in the region with severe and persistent mental illness. Support Facilitators were developing new collaborative partnerships with service providers and referral pathways that provided greater access to services. The use of recovery language and practices were becoming more common, especially as these were modelled at the consortium level and promoted by Support Facilitators on the ground. These achievements provide evidence of early systems change.
The PLM approach has proven a useful way of mapping the intended impacts of NBMPIR and the processes and resources required to achieve these. It has also provided a useful framework for this evaluation which provides a baseline for future evaluations conducted by NBMPIR.
The ultimate test for NBMPIR is whether their model of care will make a difference to the lives of consumers. Clearly this is a work in progress, and gaps were identified that need to be addressed, however our evaluation has documented positive changes in the lives of those with severe and persistent mental illness. This has been a remarkable achievement in the 12 months from commencement of PIR services until completion of our data collection.
Availability of data and materials
The data that support the findings of this study are available from Wentworth Healthcare Limited but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. Data are however available from the authors upon reasonable request and with permission of Wentworth Healthcare Limited.
Culturally and linguistically diverse
Camberwell Assessment of Need Short Appraisal Schedule
Australian Government employment and welfare service
Client Management Information System
Dialectical Behaviour Therapy
Family and Community Services
Housing and Accommodation Support Initiative
Nepean Blue Mountains Local Health District
Nepean Blue Mountains Medicare Local
Nepean Blue Mountains Partners in Recovery
The Client Information Management System (CMIS)
Personal Helper and Mentor Support
Partners in Recovery
Program logic model
Wellness Recovery Action Plan
World Health Organisation. Comprehensive mental health action plan 2013–2020. p. 2013. http://apps.who.int/gb/ebwha/pdf_files/WHA66/A66_R8-en.pdf. Accessed 20 Nov 2018
World Health Organisation. Executive summary: mental health atlas 2014. p. 2014. http://www.who.int/mental_health/evidence/atlas/executive_summary_en.pdf. Accessed 20 Nov 2018
Australian Bureau of Statistics. National Survey of Mental Health and Wellbeing. 2007. http://www.abs.gov.au/ausstats/abs@.nsf/mf/4326.0. Accessed 20 Nov 2018.
Australian Government. National Mental Health Policy 2008. 2014. http://www.health.gov.au/internet/main/publishing.nsf/Content/mental-pubs-n-pol08. Accessed 20 Nov 2018.
Webb L. The recovery model and complex health needs: what health psychology can learn from mental health and substance misuse service provision. J Health Psychology. 2012. https://doi.org/10.1177/1359105311425276.
Murray M, Poland B. Health psychology and social action. Health Psychol. 2006. https://doi.org/10.1177/1359105306063308.
Williams SJ, Birke L, Bendelow G, editors. Debating biology: sociological reflections on health, medicine and society. London: Routledge; 2003.
Pilgrim D, Rogers A. The politics of mental health: a three tier analytical framework. Policy Polit. 1999;27:13–24.
Adler HM. Toward a biopsychosocial understanding of the patient-physician relationship: an emerging dialogue. Soc Gen Int Med. 2007. https://doi.org/10.1007/sl1606-006-0037-8.
Federici MR. The importance of fidelity in peer-based programs: the case of the wellness recovery action plan. Psychiatric Rehab J. 2013. https://doi.org/10.1037/prj0000029.
Carpenter-Song EA, Holcombe BD, Torrey J, Hipolito MMS, Peterson LD. Recovery in a family context: experiences of mothers with serious mental illnesses. Psychiatric Rehab J. 2014. https://doi.org/10.1037/prj0000041.
Gilburt H, Slade M, Bird V, Oduola S, Craig TKJ. Promoting recovery-oriented practice in mental health services: a quasi-experimental mixed-methods study. BMC Psychiatry. 2013. https://doi.org/10.1186/1471-244X-13-167.
Kaplan T, Racussen L. A crisis recovery model for adolescents with severe mental health problems. Clin Child Psychol Psychiatry. 2013. https://doi.org/10.1177/1359104512449320.
Hungerford C. Recovery as a model of care? Insights from an Australian case study. Issues Ment Health Nurs. 2014. https://doi.org/10.3109/01612840.2013.855281.
Spitzmueller MC. Shifting practices of recovery under community mental health reform: a street-level organizational ethnography. Qual Soc Work. 2014. https://doi.org/10.1177/1473325013507472.
Le Boutillier C, Leamy M, Bird VJ, Davidson L, Williams J, Slade M. What does recovery mean in practice? A qualitative analysis of international recovery-oriented practice guidance. Psychiatr Serv. 2011. https://doi.org/10.1176/appi.ps.001312011.
Frese FJ, Stanley J, Kress K, Vogel-Scibilia S. Integrating evidence based practices and the recovery model. Psychiatr Serv. 2001;52:1462–8.
Ham C, Walsh N. Making integrated care happen at scale and pace. 2013. http://www.kingsfund.org.uk/publications/making-integrated-care-happen-scale-and-pace. Accessed 20 Nov 2018.
Fisher DB. Health care reform based on an empowerment model of recovery by people with psychiatric disabilities. Hosp Community Psychiatry. 1994;45:913–5.
Davidson L. Living outside mental illness: qualitative studies of recovery in schizophrenia. New York: New York University Press; 2003.
Robards F, Kang M, Steinbeck K, Hawke C, Jan S, Sanci L, et al. Health care equity and access for marginalised young people: a longitudinal qualitative study exploring health system navigation in Australia. Int J Equity Health. 2019. https://doi.org/10.1186/s12939-019-0941-2.
Smith-Merry J, Gillespie J, Hancock N, Yen I. Doing mental health care integration: a qualitative study of a new work role. Int J Ment Health Syst. 2015. https://doi.org/10.1186/s13033-015-0025-7.
Callander EJ, Fox H, Lindsay D. Out-of-pocket healthcare expenditure in Australia: trends, inequalities and the impact on household living standards in a high-income country with a universal health care system. Health Econ Rev. 2019. https://doi.org/10.1186/s13561-019-0227-9.
Partners in Recovery Organisation Reference Group. The success of Partners in Recovery: A collaborative approach to mental health 2015. https://humepir.org.au/wp-content/uploads/2015/06/the-success-of-partners-in-recovery.pdf. Accessed 23 July 2019.
Australian Government. National Framework for recovery-oriented mental health services. 2013. http://www.health.gov.au/internet/main/publishing.nsf/Content/mental-pubs-n-recovfra. Accessed 20 Nov 2018.
Australian Government. Partners in Recovery: coordinated support and flexible funding for people with severe and persistent mental illness with complex needs (PIR). 2016. http://www.health.gov.au/internet/main/publishing.nsf/Content/mental-pir#abt. Accessed 20 Nov 2018.
Gulliver A, Morse AR, Wilson N, Sargent G, Banfield M. Evaluation of a tailored care program for complex and persistent mental health problems: the Partners in Recovery Program. Eval Program Plann. 2018. https://doi.org/10.1016/j.evalprogplan.2018.03.001.
Hancock N, Smith-Merry J, Gillespie JA, Yen I. Is the Partners in Recovery program connecting with the intended population of people living with severe and persistent mental illness? What are their prioritised needs? Aust Health Rev. 2017. https://doi.org/10.1071/AH15248.
Isaacs AN, Sutton K, Dalziel K, Mayberry D. Outcomes of a care coordinated service model for persons with severe and persistent mental illness: a qualitative study. Int J Soc Psychiatry. 2017. https://doi.org/10.1177/0020764016678014.
Cheverton J, Janamiam T. The Partners in Recovery program: mental health commissioning using value co-creation. Med J Aust. 2016. https://doi.org/10.5694/mja16.00124.
Banfield M, Forbes O. Health and social care coordination for severe and persistent mental illness in Australia: a mixed methods evaluation of experiences with the Partners in Recovery Program. Int J Ment Health Syst. 2018. https://doi.org/10.1186/s13033-018-0194-2.
Hancock N, Scanlan JN, Gillespie JA, Smith-Merry J, Yen I. Partners in recovery program evaluation: changes in unmet needs and recovery. Aust Health Rev. 2018. https://doi.org/10.1071/AH17004.
Nepean Blue Mountains Medicare Local. 2012–13 Annual Plan Nepean-Blue Mountains Medicare Local. 2013. https://www.nbmphn.com.au/About/Plans-Reports.aspx. Accessed 20 Nov 2018.
NSW Government. Nepean Blue Mountains local Health District strategic plan 2012–2017. 2012. https://www.nbmlhd.health.nsw.gov.au/about-us/nbmlhd-strategic-plans-and-reports. Accessed 15 Nov 2018.
Wentworth Healthcare. Needs assessment 2016–2017. Kingswood, Australia: Wentworth Healthcare; 2017. https://www.nbmphn.com.au/Resources/About/130_0217-Needs-Assessment-A4_FINAL_WEB.aspx. Accessed 4 Apr 2019.
Australian Government Department of Health. PIR Evaluation Framework. 2015. http://www.health.gov.au/internet/main/publishing.nsf/content/mental-pir-eval. Accessed 20 Nov 2018.
Trankle SA, Reath J. The Nepean Blue Mountains Partners in Recovery Evaluation. 2015. http://researchdirect.westernsydney.edu.au/islandora/object/uws%3A33977. Accessed 20 Nov 2018.
W. K. Kellogg Foundation. Using logic models to bring together planning, evaluation, and action. Logic Model Development Guide; 2004. p. 1–71. http://www.wkkf.org/resource-directory/resource/2006/02/wk-kellogg-foundation-logic-model-development-guide. Accessed 21 Nov 2018
Australian Government Department of Health. A manual to guide the development of local evaluation plans: evaluating initiatives within the LIFE framework using a program logic approach. 2003. https://www.health.gov.au/internet/main/publishing.nsf/content/4FD5C304C536BDE3CA257CAF0017ADE9/$File/12.pdf. Accessed 21 Nov 2018.
National Center for Chronic Disease Prevention and Health Promotion. Developing an effective evaluation plan. Setting the course for effective program evaluation. 2011. http://www.cdc.gov/obesity/downloads/cdc-evaluation-workbook-508.pdf. Accessed 22 Nov 2018.
Ross E, Vingilis E, Osuch E. Early Intervention in the Real World. An engagement and access model for healthcare delivery to adolescents with mood and anxiety concerns. Early Interv Psychiatry. 2012. https://doi.org/10.1111/j.1751-7893.2011.00312.x.
Australian Government Department of Health and Aging. Life: Living is for everyone. A framework for prevention of suicide in Australia. 2007. https://www.lifeinmindaustralia.com.au/about-us/the-life-framework. Accessed 2 Nov 2018.
Agency for Health Care Research and Quality. The Logic Model: The foundation to implement, study, and refine patient-centered medical home models. 2013. https://pcmh.ahrq.gov/page/logic-model-foundation-implement-study-and-refine-patient-centered-medical-home-models. Accessed 2 Nov 2018.
Israel BA, Schulz AJ, Parker EA, Becker AB. Review of community-based research: assessing partnership approaches to improve public health. Ann Rev Public Health. 1998;19:173–202.
Lawless A, Freeman T, Baum F, Bently M, Jolley G, Labonte R, et al. Researching comprehensive primary health care in Australia: the development of a program logic model and evaluation framework to assess the effectiveness of services. Flinders University. 2013; http://www.flinders.edu.au/medicine/fms/sites/SACHRU/documents/CPHC%20presentations/Montreaux%20Posters/HSR%20Poster%20-%20Lawless%20et%20al.pdf. Accessed 22 Nov 2018.
Andrew S, Halcomb EJ, editors. Mixed methods research for nursing and the health sciences. West Sussex, UK: Wiley-Blackwell; 2009.
Unrau YA. Using client interviews to illuminate outcome in program logic models: a case example. Eval Program Plann. 2001;24:353–61.
Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19:349–57.
Gale NK, Heath G, Cameron E, Rashid S, Redwood S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol. 2013. https://doi.org/10.1186/1471-2288-13-117.
Patton MQ. Qualitative evaluation and research methods. 2nd ed. London: Sage; 1990.
Silverman D. Doing qualitative research. 3rd ed. London: Sage; 2010.
Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006. https://doi.org/10.1191/1478088706qp063oa.
Mental Health Commission of Canada. Toward recovery and well-being. A framework for a mental health strategy for Canada. 2009. http://www.mentalhealthcommission.ca/English/document/241/toward-recovery-and-well-being. Accessed 12 Nov 2018.
Mental Health Commission of New Zealand. Blueprint II - Improving mental health and wellbeing for all New Zealanders. How things need to be. 2012. https://www.mentalhealth.org.nz/assets/ResourceFinder/mhc3722-making-change-happen-web-pdf.pdf. Accessed 15 Nov 2018.
Garside P. Organisational context for quality: lessons from the fields of organisational development and change management. Qual Health Care. 1998;7:S8–S15.
Ng RMK, Pearson V, Chen EEY, Law CW. What does recovery from schizophrenia mean? Perceptions of medical students and trainee psychiatrists. Int J Social Psychiatry. 2011. https://doi.org/10.1177/0020764009354833.
Turner-Crowson J, Wallcraft J. The recovery vision for mental health services and research: a British perspective. Psychiatric Rehab J. 2002;25:245–54.
Andreasen R, Caputi R, Oades L. Stages of recovery instrument: development of a measure of recovery from serious mental illness. Aust N Z J Psychiatry. 2006;40:972–80.
Andreasen R, Caputi R, Oades L. Do clinical outcome measures assess consumer defined recovery? Psychiatry Res. 2010;177:309–17.
Bellack AS. Scientific and consumer models of recovery in schizophrenia: concordance, contrasts, and implications. Schizophr Bull. 2006;32:432–42.
Frese FJ. Advocacy, recovery, and the challenges of consumerism for schizophrenia. Psychiatr Clin N Am. 1998;21:233–49.
Roe D, Mashiach-Eizenberg M, Lysaker PH. The relation between objective and subjective domains of recovery among persons with schizophrenia-related disorders. Schizophr Res. 2011;131:133–8.
Leamy M, Bird V, Le Boutillier C, Williams J, Slade M. Conceptual framework for personal recovery in mental health: systematic review and narrative synthesis. British J Psychiatry. 2011;199:445–52.
Oades L, Deane F, Crowe T, Lambert WG, Kavanagh D, Lloyd C. Collaborative recovery: an integrative model for working with individuals who experience chronic and recurring mental illness. Australas Psychiatry. 2005;13:279–84.
Bernardy NC, Hamblen JL, Friedman MJ, Ruzek JI, McFall ME. Implementation of a posttraumatic stress disorder mentoring program to improve treatment services. Psychol Trauma Theory Res Pract Policy. 2011. https://doi.org/10.1037/a0024847.
Best DW, Groshkova T, Sadler J, Day E, White WL. What is recovery? Functioning and recovery stories of self-identified people in recovery in a services user group and their peer networks in Birmingham England. Alcohol Treat Q. 2011. https://doi.org/10.1080/07347324.2011.586270.
Laudet A, Harris K, Kimball T, Winters KC, Moberg DP. Collegiate recovery communities programs: what do we know and what do we need to know? J Soc Work Pract Addict. 2014. https://doi.org/10.1080/1533256X.2014.872015.
Nepean Blue Mountains Medicare Local. Partners in recovery client activity report - September 2014. Penrith, Australia: Wentworth Healthcare; 2014.
Jackson C, Nicholson C. Making integrated healthcare delivery happen - a framework for success. Asia Pac J Health Manag. 2008;3:19–24.
Martin-Misener R, Valaitis R, Wong ST, MacDonald M, Meagher-Stewart D, Kaczorowski J, et al. A scoping literature review of collaboration between primary care and public health. Prim Health Care Res Dev. 2012. https://doi.org/10.1017/S1463423611000491.
Tickle A, Brown D, Hayward M. Can we risk recovery? A grounded theory of clinical psychologists' perceptions of risk and recovery oriented mental health services. Psychol Psychother Theory Res Pract. 2014;87:96–110.
Brown L, Tucker C, Domokos T. Evaluating the impact of integrated health and social care teams on older people living in the community. Health Soc Care Community. 2003;11:85–94.
Piat M, Lal S. Service providers’ experiences and perspectives on recovery-oriented mental health system reform. Psychiatr Rehabil J. 2012. https://doi.org/10.2975/35.4.2012.289.296.
Peebles SA, Mabe PA, Fenley G, Buckley PF, Bruce TO, Narasimhan M, et al. Immersing practitioners in the recovery model: an educational program evaluation. J Community Ment Health. 2009;45:239–45.
Salgado JD, Deane FP, Crowe TP, Oades LG. Hope and improvements in mental health service providers’ recovery attitudes following training. J Mental Health. 2010;19:243–8.
Whitley R, Drake RE. Recovery: a dimensional approach. Psychiatr Serv. 2010;61:1248–50.
Australian Government Department of Health. Supporting communities to reduce the risk of suicide (Aboriginal and Torres Strait Islander component): grant program guidelines. 2012. http://www.health.gov.au/internet/main/publishing.nsf/Content/mental-scrrsab-guide. Accessed 12 Nov 2018.
Australian Government Department of Health. Mental health services for people of culturally and linguistically diverse (CALD) backgrounds. 2014. http://www.health.gov.au/internet/main/publishing.nsf/Content/mental-multi-fact. Accessed 20 Nov 2018.
National Registry of Evidence Based Programs. Wellness action recovery plan 2013. https://www.samhsa.gov/ebp-resource-center. Accessed 24 July 2019.
Copeland ME. Wellness action recovery plan: a system for monitoring, reducing and eliminating uncomfortable or dangerous physical symptoms and emotional feelings. Occup Therapy Ment Health; 2002. https://doi.org/10.1300/J004v17n03_09.
Watson DE, Broemeling A, Wong ST. A results-based logic model for primary healthcare: a conceptual foundation for population-based information systems. Healthc Policy. 2009;5:33–46.
We would like to thank:
● Wentworth Healthcare Limited who funded the research;
● The participants who gave their time and valuable insights in this study;
● The Project Reference Group - Mr. James Bishop, Ms. Johanna Dennis, Mr. John Downie, Mr. Brian Haisman, Mr. Ian Hoffman, and Ms. Lizz Reay who oversaw the research;
● Dr. Vlasios Brakoulias who assisted with recruitment of participants at Nepean Blue Mountains Local Health District; and
● Dr. Marlee King (PhD) from Western Sydney University who assisted with qualitative data coding
This work was supported by a grant from Wentworth Healthcare Limited as provider of the Nepean Blue Mountains Medicare Local. The funding body approved the decision to publish this research after reviewing the final draft. They had a consultancy role in the study design, survey and interview content and participant recruitment, but no role in data collection, analysis and interpretation, or manuscript writing.
Ethics approval and consent to participate
Ethics approval for the study was granted by:
● Western Sydney University Human Research Ethics Committee (H10697); and
● Nepean Blue Mountains Local Health District Human Research Ethics Committee (14/49). All participants provided signed informed consent.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Nepean Blue Mountains Partners in Recovery Evaluation Framework (DOCX 111 kb)
Documents Reviewed (DOCX 16 kb)
Online survey (DOCX 234 kb)
Semi structured interview guide (DOCX 207 kb)
Qualitative coding frame (DOCX 61 kb)
About this article
Cite this article
Trankle, S.A., Reath, J. Partners in Recovery: an early phase evaluation of an Australian mental health initiative using program logic and thematic analysis. BMC Health Serv Res 19, 524 (2019). https://doi.org/10.1186/s12913-019-4360-2
- Partners in Recovery
- Mental health
- Program logic
- Mixed methods