How is return on investment from quality improvement programmes conceptualised by mental healthcare leaders and why: a qualitative study
BMC Health Services Research volume 23, Article number: 1009 (2023)
Return on Investment (ROI), whereby the ratio of costs to benefits is assessed, is encouraged in-order to justify the value of Quality Improvement (QI) programmes. We previously performed a literature review to develop a ROI conceptual framework for QI programmes. We concluded that, QI-ROI is conceptualised as any monetary and non-monetary benefit. In the current study, we explored if this finding is shared by mental healthcare leaders. We also investigated the stability of this conceptualisation against influencing factors and potential for disinvestment.
We performed qualitative interviews with leaders in an NHS mental health organisation. There were 16 participants: nine board members and seven senior leaders. The interviews were held online via Microsoft Teams and lasted an hour on average. We performed deductive-inductive analysis to seek data from our initial ROI framework and any new data.
We found that in mental healthcare, QI-ROI is also conceptualised as any valued monetary and non-monetary benefits. There was a strong emphasis on benefits to external partners and a de-emphasis of benefit monetisation. This conceptualisation was influenced by the 1) perceived mandates to improve quality and manage scarce resources, 2) expectations from QI, 3) health and social care values, 4) ambiguity over expectations, and 5) uncertainty over outcomes. Uncertainty, ambiguity, and potential for disinvestment posed a threat to the stability of this conceptualisation but did not ultimately change it. Health and social care values supported maintaining the QI-ROI as any benefit, with a focus on patients and staff outcomes. Socio-political desires to improve quality were strong drivers for QI investment.
Mental healthcare leaders primarily conceptualise QI-ROI as any valued benefit. The inclusion of externalised outcomes which are hard to attribute may be challenging. However, mental healthcare services do collaborate with external partners. The de-emphases of benefit monetisation may also be controversial due to the need for financial accountability. Mental healthcare leaders recognise the importance of efficiency savings. However, they raised concerns over the legitimacy and utility of traditional ROI as a tool for assessing QI value. Further research is needed to bring more clarity on these aspects of the QI-ROI concept.
The need for higher quantity and quality of mental health services has always been paramount [1, 2]. The COVID-19 pandemic has added to this need due to the escalated global medical, social, and financial distress [3, 4] at a time when mental health services are grappling with chronic funding gaps [5,6,7]. In 2021, the United Kingdom (UK) National Health Service (NHS) ring-fenced over £2.3 billion for mental health services . Some of this will likely be spent on Quality Improvement (QI) programmes. Unlike small, localised QI projects, QI programmes combine a variety of strategies in local, national or international efforts to systematically improve quality and efficiency of organisations and or healthcare systems as a whole [9, 10]. As healthcare resources are limited, leaders must make and justify investment allocation decisions. The World Health Organisation (WHO) has recommended Return on Investment (ROI) as a tool to justify investments in mental healthcare programmes .
ROI can forecast financial value (e.g., in business cases), or evaluate a programme ex-post . Outcomes are converted to money (monetised) so as to assess the financial gains that a programme can or has yielded . To do this, a financial value of a benefit must be directly or indirectly determined using financial approximates (proxies) [12, 13]. Traditionally from accounting, little is known about ROI in healthcare. As such, there is a need for an operational definition of ROI in mental healthcare QI . We recently performed a systematic literature review where we studied ROI as a concept of returns or benefits from QI. This led to two studies; the first  describes our ROI concept analysis, development of the QI-ROI concept and its framework. We then further developed the framework by adding benefits that make up QI-ROI . In the current study, we explored if this conceptualisation is shared by mental healthcare leaders. We also explored the factors that influence their conceptualisation of ROI.
Currently, there is no standardised QI governance tool to guide QI investment allocation decisions . This hinders effective assessment and communication of QI value in mental healthcare and healthcare as a whole [18, 19]. However, before these challenges can be addressed, it is crucial to learn what the meaning of ROI in mental healthcare organisations is and why. This understanding is needed to help provide practical support to leaders in their QI investment/disinvestment decision-making processes. This is important as quality of care and cost of care are central to the goals of both healthcare provision and QI programmes [20, 21].
ROI is closely related to QI effectiveness (goal achievement) as achievement of stated QI programme goals is part of QI value. QI effectiveness has so far been found to be inconclusive [22,23,24,25,26,27,28]. However, the review preceding this study found that most healthcare leaders and other stakeholders saw QI benefits regardless of its effectiveness, for example in lessons learnt from failed attempts . QI cost-effectiveness and cost-efficiency (achieving the best outcome for the least cost, and without waste) is also a part but not a complete picture of QI value . A programme may be cost effective and efficient but fail to provide value in other important organisational aspects . Therefore, a more comprehensive view of ROI is needed. This requires a deeper understanding of the meaning of ROI in mental healthcare. To this effect, we employed institutional theory as our study’s underpinning theory.
Institutional Theory is the overriding theoretical base for this study, due to its explanatory power regarding organisational behaviour and reasoning . Institutional theory encompasses various connected, sometimes contradicting theories. Traditionally, institutional theory focused on organisational outcomes and their causes [32, 33]. The newer neo-institutional theory asserted that organisations are governed by taken-for-granted norms and values, often to economic detriment . Faced with institutional pressures, organisations are said to either blindly follow norms, be coerced to complying, or mimic other organisations. This then leads to isomorphism (homogeneity) and promote social legitimacy rather than efficiency . However, others argued that organisational actors do have influence [35, 36].
Through language in discourse (discourse theory) or rhetoric (rhetoric theory), organisational actors (re) create, communicate, and legitimise new meanings [37, 38]. Discourse is a system of knowledge, meaning-making, and communication which is greatly influenced by power relations in a context . Rhetoric is related to the use of language to manipulate a message towards a desired end . Discourse and rhetoric are related to a branch of institutional theory called Institutional Logic . An institutional logic is a set of organising rules and norms that influence meaning-making and reasoning in a context [38, 40]. Institutional ‘entrepreneurs’ are said to engage in strategies to create and change prevailing logics . These theories align with Institutional Work, a theory which highlights role dependent actions and meanings .
Related to institutional theories are the Stakeholder , and Stewardship theories . The stakeholder theory denotes that leaders strategically engage stakeholders (individuals that impact or are impacted by an organisation) to incorporate broad social values . The stewardship theory states that autonomous leaders are more likely to be intrinsically motivated, geared towards mutual gain and collaboration . All these theories acknowledge the institutional context within which meaning is socially constructed. We assess which of these theories are reflected in our findings. This deepened our understanding of why QI-ROI was conceptualised a certain way, and hence the antecedents of the QI-ROI concept.
The main aim of this study was to ascertain the extent to which the conceptualisation of QI-ROI as any benefit is shared by mental healthcare leaders. Our second aim was to further develop the QI-ROI concept and its conceptual framework for mental healthcare. The study addressed the following research questions:
How is ROI from QI programmes conceptualised by mental healthcare leaders?
What influences that conceptualisation?
How is that conceptualisation impacted by a potential for QI disinvestment?
All participants were employed by an NHS Trust, a mental health care provider in London. This setting has been engaged in quality improvement using QI methodologies since 2016.
We chose an interpretive qualitative design as this approach helps provide insights into a complex world where participants’ reality is assumed to be socially constructed . This is a stakeholder-centred methodological approach, where ‘reality construction’ occurs amongst the participants, and between the researcher and participants [44, 45]. This invokes the concept of double hermeneutics where multiple views reflect context-focused collective meaning-making .
Eligible participants were the board members as QI investors (e.g., Chief Financial Officer) or senior managers as influential leaders (e.g., Clinical Directors). To be included, participants had to have experience in QI activities (investment, implementation, evaluation), and have internet access. Leaders not influential in QI implementation, evaluation and investment were to be excluded. Our target sample was 15, which we deemed both feasible and sufficient to gain insights on the QI-ROI concept at high leadership level. We employed purposive sampling to target suitable informants. Potential participants were identified by and initially approached by the Trust’s QI Director, and co-author CH before being engaged by first author, ST. Twenty four potential participants were approached.
The following procedures were carried out by ST. Potential participants were sent formal invitations and participant information sheets (PIS) which detailed the study aims and objectives. Participants were informed about the study as part of ST’s PhD project. They were given an average of two weeks to consider the study and ask questions. Written consent from each participant was obtained prior to their interview. Individuals interviews were performed online via Microsoft (MS) Teams . The topic guide (Supplementary file 2) contained semi-structured questions to help explore the results from the systematic review and collect new data . Interviews took place between December 2021 and January 2022 and lasted about an hour each. Interviews were recorded and transcribed using the MS Teams. During transcription, participants personal details were deleted for anonymity. Data were moved to NVivo Release 1.6 , securely stored and managed in encrypted, and password protected King’s College London computer. All data from MS Teams were deleted. A COREQ  checklist for our study can be found in Supplementary file 1.
We analysed the data using Framework Analysis, a deductive approach that uses an existing framework to deduct specific data from a dataset . Combined with thematic analysis, this approach allowed us flexibility to induct new data [51, 52]. Our analysis had two phases: Phase I (the deductive phase), which included steps 1) developing a codebook based on the conceptual framework, and 2) testing the codebook against the existing framework and literature and adjusting codes as needed. Phase II (the inductive phase) entailed step 3) re-familiarising with interview data, 4) generating initial codes, 5) applying the codebook and identifying any additional codes, 6) connecting codes, 7) identifying themes, 8) re-checking themes, and 9) reporting study findings. Our data analysis framework can be found in Supplementary file 3. Our codebook and additional exemplary quotes can be found in Supplementary file 4.
Of the 24 potential participants approached, 16 took part in the study: nine board members and seven non-board directors were interviewed. Eight leaders were unable to partake due to their work schedules. We arranged themes from the data according to our three research questions: ROI conceptualisation (question 1), Influencing factors (question 2), and Disinvestment potential (question 3). ROI conceptualisation is the leader’s mental abstractions of QI-ROI as represented by how the leaders’ defined and described ROI. Influencing factors were those that directly influenced the consistency of ROI conceptualisation. Disinvestment potential relates to how all factors affect QI investment decisions, and how in-turn the potential for disinvestment impacts the stability of the conceptualisation. This indirectly reflects on the consistency of the QI-ROI concept as collective investment decisions may be based on the Trust’s prevailing ROI conceptualisation.
Mental healthcare leaders predominantly described ROI as any valued benefit that directly or indirectly contributes to the fulfilment of their organisational strategy. Here, ROI was associated with quality where the improvement in quality as demonstrated by desired outcomes was seen as a return (on investment). Valued QI benefits included patient outcomes, staff outcomes, financial outcomes, organisational development, and external outcomes for healthcare systems and societies. Financial benefits were seen as secondary to other outcomes.
“[we are] not looking at the money but actually we're driving the money from the quality of the service…high quality, ultimately costing us less and, giving us more scope for investment and innovation”. Participant 7.
A few participants associated ROI with the cost of care. In this description, the leaders focused more on costs (and investment) and less on the benefits or returns. This suggested ROI conceptualised more as a cost-saving or cost-management tool. For example, some leaders discussed using QI to prevent future high-cost care, maximise benefit, and thus save money.
“…saving money, achieving a sustainable organisation because we're working in an era of … severe constraints…Because it's public money, we should make sure that it's used to maximise the benefit of the people that we serve”. Participant 3.
Influencing factors consist of five interlinking subthemes. These are: healthcare mandates, values, expectations, ambiguity, and uncertainty. These factors reflected the complex nature of QI programmes as well as the complexity of mental healthcare.
Participants strongly suggested that their conceptualisation of ROI was anchored on what they saw as their mandates as mental healthcare leaders. Mandates were often external in their nature, for example through national quality frameworks (e.g., restraint reduction) or fiscal targets. The main QI mandate was seen as improving the quality of services. Indirectly, managing scarce resources was an important QI mandate that can eventually lead to saving costs. In addition, were perceived obligations towards patients, staff, societies, the organisation, and partners. Leaders also expressed internal aspirations such being the best service provider.
“I would like to see reduction in terms of money, but I think the quality aspects of supporting people in their lives and their recovery journey is a really valid way to show that that investment’s been worthwhile. Participant 10.
A few leaders referenced the organisation’s overall strategy on environmental sustainability. However, the main concern for most participants was the sustainability of QI programmes and outcomes. Sustainability was seen to be related to supporting and embedding new practices. Some participants were concerned about the organisation’s ability to cater for future patients. These participants felt that this required the organisation to be self-sufficient and sustainable. In this regard, QI was used to improve organisational efficiency and productivity rather than generate profits. This was expected to help free-up and redeploy resources where most needed.
“…when I talk about financial benefits, I talk about how we then reinvest that to make us a more sustainable service for the future, knowing that we've got increased demands often in decreasing capacity or capacity that's unsustainable …”. Participant 2.
Managing scarce resources, improving quality, and sustainability meant seeking cultural transformation. Through QI, internal and external cohesions or collaborations were encouraged. Internally, this entailed improving team-working and engaging patients, whilst externally cohesions and collaborations entailed relationships with external partners and communities. Cohesion and collaboration were sought through co-production, enhanced communication using a shared language, and sharing leadership. There was a desire to disseminate insights both internally and externally to the organisation. The assumptions were that through collaboration, benefits could be maximised, spread more quickly, and sustained. QI collaboration was expected to improve efficiency and avoid “reinvention of the wheel”.
“…the benefits of that for the Trust of doing it this way, is that where improvements are made in a pilot, they can then be rolled out … so that we're not reinventing the wheel, which is resource heavy”. Participant 9.
Leaders often made links between different outcomes. Particularly, improved staff skills were seen as central to achieving both patient and financial outcomes, as well as system-wide outcomes. Therefore, some QI investment was also keenly directed towards staff outcomes.
Mandates determined QI’s main goals and objectives and provided rationales for pursuing QI. These mandates were often explicitly and implicitly expressed through values. Some values were extrinsic, for example, QI was a means to manage economic pressures from rising service costs and demands. However, leaders predominantly expressed intrinsic motivation to improve service outcomes. Participants primarily viewed ROI through personal, professional, and or organisational values. Extrinsic values were applied within the framework of intrinsic values.
“I think when you see things like quality adjusted life years and monetised outcomes that can be used within health, I think that makes the hairs on lot of clinicians’ backs go up. …I think people can feel very uncomfortable with those monetised outcomes”. Participant 11.
Health and social care perspectives drove the predominant intrinsic values. Such values prioritised clinical and social agendas such as upholding human rights and justice. This included value-based healthcare where any outcome that matter to patients is favoured. Here, outcomes included helping individuals improve their personal, social, and work lives.
“By reducing restrictive practices, we're respecting people’s human rights. We are improving their wellbeing. We are increasing the chances of their recovery. There's a very human quality there that you can't monetise”. Participant 9.
There was recognition that the traditional ROI is meant to encourage fiscal responsibility. However, leaders collectively rejected the notion that only monetisable benefits should count as ROI in mental healthcare. Even though some named benefits were monetisable, most leaders’ ROI concept was focused on the benefit itself, and not the monetised version.
“…we could get an ROI actually to pick it apart and get what the cost is, and cost saving is. It's not really worth it for the investment…It doesn't matter what we're getting in return [on investment]. The important thing is the, the actual outcome”. Participant 8.
For some, their role meant conceiving ROI as primarily measurable outcomes. This meant treading a fine line between a broad based ROI and the econometric traditional ROI. This modification indicated a lean towards financial ROI (which can be measured) and or a tension between economic and healthcare values. Framing messages ‘correctly’ was important in managing this tension and fulfilling multiple obligations.
“We're running healthcare delivery organisation using the best of business practice and principles. It's a very different way of describing it. And it's a fundamental tension when you have a board where people talk about ROI and cash releasing, savings and flow…I mean, it's blindingly obvious if our patients don't end up in deep poverty, they are not going to relapse as much”. Participant 2.
Quality of care was not described in financial terms. There was a desire to avoid financial focus at the expense of desired goals like staff and patient outcomes. For many, a balanced ROI approach that combines both financial and non-financial outcomes was crucial.
“CQC (Care Quality Commission) will rate on the quality of services. But if their finances were in a mess, they wouldn't be able to be an outstanding [organisation]. It is about making sure that you're getting a good return on investment, both in terms of being financially sustainable but delivering first class services”. Participant 5.
Participants’ expectations were driven by mandates, perceived obligations, and values. The QI outcomes described as ROI aligned with what QI was expected to do or enable. All participants described QI as a systematic way to improve the quality of healthcare. QI was seen as a mechanism for staff empowerment, diagnosing problems, understanding systems, identifying, and testing solutions, as well as avoiding excessive waste by abandoning failing attempts early.
“…it's starting from a position of somewhat kind of helplessness about what they [staff] could do about the situation…” Participant 13.
The aim was to then embed, roll-out, and sustain improvements. In both instances (failure or success), lessons were to be taken forward and shared with other teams within the organisation or with partners. The ultimate objectives were sustained improvements in desired outcomes.
“We pursue quality improvement programs to embed in the organisation, approaches which deliver better quality care”. Participant 3.
Expectations often varied amongst participants, depending on their QI function knowledge and or buy-in, as well as influence of others.
QI function knowledge and buy-in
QI function knowledge was strongly related to a leader’s role, QI training, experience, and or proximity to QI programmes. The closer in proximity to QI programmes, the more QI training and experience, the broader the view of the QI-ROI concept. This was accentuated in those that were ‘bought-in’ into the QI methodology. Those not bought-in were sceptical about function, outcomes, and causality. Some were concerned about assumptions by others that QI can solve any problem. Negative experience limited expectations or forced adjustments to be more ‘realistic’. Personal experience with QI enabled more nuanced understanding of QI benefits beyond achieving programme goals in terms of organisational and system-wide outcomes. This enabled clearer expression of both immediate and long-term QI outcomes, and or broadened views on what QI is most suited for.
“I think some people, especially in a healthcare setting, would see quality as patient care, for an individual patient. Some might see as your ability to treat your population, so that's more of a performance element of quality, and I would include how you utilise your resources sustainably to maximise quality” Participant 1.
QI Success vs QI failure
Some saw QI effectiveness as encompassing outputs e.g., diagnosing process issues, or hard outcomes e.g., achieving set goals. Including softer benefits broadened the view of QI-ROI. Perceiving QI as a continuous incremental methodology also broadened the QI-ROI concept. Here, different benefits were perceived throughout what was seen as a “QI journey”. The assumption was that QI projects can be aggregated to unlock organisation level outcomes that improve overall performance. The collective perception was that not all QI is successful.
“Rather than spend a year setting something up and then failing, can we set up in a week and fail quickly so that we know, what's not working quickly. But in doing so, not discounting it, giving the chance to properly fail”. Participant 15.
Intervention vs implementation failure
QI-ROI was seen to be related to intervention and implementation failures. Intervention outcomes for some programmes were perceived to have been mostly positive. However, poor initial implementation, embedding, rolling-out, scaling-up, spread, dissemination, and sustainability were most frequently associated with failure to obtain QI-ROI. Deciphering the exact failure or cause, or even it was indeed a failure however seemed challenging.
“One of my frustrations would be that I think it's hard to see the return on investment from QI at the moment. …there are promising elements of what we have been trying to do, …It looks it looks like it works…, but never rolled out. So, is that a success of QI, or is that a failure of QI? Hard to say!”. Participant 1.
QI evaluation; how, what, and when to measure
Some desired outcomes were deemed neither measurable nor monetisable (e.g., well-being). This then created a dilemma of how QI-ROI should be assessed. Participants stated that a compromise can be reached through using proxies and or adding narratives to detail qualitative benefits. QI and thus ROI measurability (and monetisability) was deemed necessary by some. However, the lack of skill and or infrastructure was a challenge. The measurement challenges also included the ROI methodology itself. These concerns co-existed within some participants.
“I think we're fairly unsophisticated when it comes to thinking through the stuff that's harder [to measure], partly because we probably strapped for resource, and we don't have the people who have the time to think through bit more of the sophisticated proxy measures that we might like to use”. Participant 4.
I think that social impact is really important, and I don't believe that financial only ROI is sophisticated enough”. Participant 4.
There was differences of opinions as to when to measure QI outcomes. Some felt there was a misalignment between expectations of immediate results and the ability of organisational and QI processes to deliver immediate results. Most participants asserted that QI benefits do not show themselves in the immediate period, they come in phases. Some outcomes such as problem identification and diagnosis may be immediate, others such as patient outcomes may be intermediate, whilst sustainability and cost-saving were seen as long-term outcomes.
“If QI really is working, then then those outcomes should be being delivered. So, year on year…but I think if QI was really working you should be able to identify the next problem much more easily too”. Participant 8.
For some, QI was expected to be continuous and incremental and then aggregate to transformation (a complete change). However, some participants did not see QI as a tool for transformation. They felt QI can contribute to transformation, and top-down measures were needed for organisational transformation. This ambiguity co-existed within some participants.
“…[QI] it's a way of being able to know what it is we want to strategically deliver in the medium to long term and then use our methodology to start to make incremental changes that we know will aggregate up to that big change.” Participant 12.
“…all you do is just make those incremental changes, but never changed the system for the future. So, you've already failed before you started, because yes, you can improve processes to a particular percent or degree actually without really changing the whole value chain…you won't make systemic longstanding change”. Participant 12.
Influence of others
Some scepticism was related to the influence of others within and outside the organisation. Internally, trusted colleagues’ negative perception of QI or QI effectiveness narrowed others’ concept of QI-ROI due to the limited perceived benefits. There were also concerns regarding what was seen as faulty assumptions by others regarding what to expect from QI. External influential sources included literature, health economists, and politics. Lack of evidence of QI effectiveness in literature limited expectations and created scepticism. Awareness of ROI as monetised benefits from health economics literature was taken into consideration. Political and economic expectations caused some to frame their views of ROI as monetisable benefits, without fundamentally changing their conceptualisation of ROI as any valued benefit. Some modified their QI-ROI concept to fit context (e.g., economic, political contexts). There was a belief that economists also value intangible outcomes (that cannot be measured or monetised).
“I would say if I was talking to an economist who believed in intangible assets, which I think many of them do, then I would kind of take that position. But I think someone who purely wants to look at ROI where there would be something that stacks up that I can count, then I probably [would] deviate a little bit from that position”. Participant 16.
QI philosophy; theory and practice
There appeared to be various interpretations of the same QI concepts, for example the QI trial-and-error principle and QI effectiveness above. Ambiguity was also seen as resulting from competing goals. However, some indicated that the issue was less of ambiguity, but more of a dissociation between QI theory and practice.
“There's a difference between the concept, which absolutely remains fundamental, and the mechanism for delivery. You have to hold them separately…”. Participant 1.
The apparent contradictions appeared to have the effect of QI working against itself by producing results that are contrary to the QI philosophy and principles being promoted. For example, although QI is by principle a bottom-up approach, some as experienced QI as a top-down approach. Others found that QI could create silos rather promote collaboration. Others found that a shared language and communication was not as desired. Others felt discomfort at the ‘insistence of the use of 'Guru’ concepts. Structuring of QI was seen by some as a potential threat and counterintuitive to innovative thinking, as well as flexibility and adaptability.
“I've observed in the organisation, for example, as we've been running kind of three-monthly improvement cycles and testing interventions, and it's become sort of almost psychotic with no real thinking as to what's the big overall issue”. Participant 16.
Lack of dedicated ROI evaluation tool
There were uncertainties and scepticism regarding whether QI does live up to expectations. The board was said to have the responsibility to manage uncertainty and assessment of ROI. At the board level, QI-ROI was expected to be demonstrated by improvement in key performance measures within the organisation’s integrated quality framework. Some leaders provided specific data on ROI links to integrated quality framework. For example, staff and patient surveys as part of understanding the QI-ROI from collective QI programmes. For others, the ‘sense’ of QI-ROI was through intuition, first-hand knowledge, or dissemination.
“How can we measure the improvement compared with the investment that it's taken, and we have seen significant moves forward in some areas …”. Participant 7.
There was recognition that the ideal of perfect information was impossible. What leaders sought, was enough information to make a decision. Sometimes that meant accepting the reality of inability to provide definitive proof. Some information was seen as better than none.
“…some things you know we have to accept; we just can't measure”. Participant 5.
Uncertainty due to poor communication
Some saw ambiguities and dilemmas as opportunities to engage in communication over QI-ROI. Some participants commented on the poor communication within the organisation regarding what QI has or has not achieved. This appeared to worsen overall uncertainty over QI outcomes and therefore QI-ROI. Some participants noted that the focus on COVID-19 had, however, significantly impacted routine communications. At times poor communication was related to avoidance of the subject altogether due to the discomfort surrounding ROI.
“…the challenge is we are really bad on return on investment articulation and measurement, and I think we are deliberately bad on it. And the reason we are deliberately bad on it is that it’s an uncomfortable place to be. [A practitioner] very rarely wants to sign up to it, because then they actually have to deliver on the aspirations”. Participant 12.
Uncertainty over causality
Most participants indicated that organisational complexity challenged what can be realistically expected or causally linked to QI programmes. Participants expressed that in mental healthcare, costs and benefits may be shared with external partners, making ROI harder to assess.
“…we're now doing much more collaborative work with our partners and… so you know in terms of being able to assess what our role was and financially say, that this is a bit we did is much more tricky to do”. Participant 10.
Some felt that as the QI investment is fixed, and as such, QI-ROI should be based on actual outcomes of QI programmes (e.g., improved safety), rather than monetised outcomes (ROI).
“We don't think it's worth us going away and working that part out…that's not quite the way we think about it, so we wouldn't do the financial or economic analysis on and every program. Absolutely not. We're much more focused on the program outcomes for each piece of work [because] we have a fixed investment into QI”. Participant 8.
Some participants indicated that QI was already embedded and part of everyday business. However, some appeared unable to disentangle QI principles or philosophy from everyday or other ways of innovative working. It appeared difficult to tell when QI is embedded, and or when QI was not a factor in how things are done or thought of.
“I don't think I once thought about QI in that time or anybody did, we did use data quite a lot and most organisations now use run charts so that I wouldn't say that's particularly QI it was a QI-centric decision”. Participant 14.
This was also apparent when discussing QI’s performance during the COVID-19 pandemic. There were differences in opinions as to how QI performed during the pandemic. Others explained this to be a result of QI being already embedded within the organisation. Some thought that an indication of this embedding could be the speed of implementation or problem diagnosis. Some participants felt that this was achieved during this pandemic.
“…when COVID hit [they] were absolutely phenomenal, they've never done this before…and boy, did they make it happen! You know, it's incredibly impressive what they did, and it did happen quickly, so you know, that's the point”. Participant 9.
The influencing factors above also affected attitudes towards QI investment and disinvestment. The view of most participants was that QI methodology is known to be effective, as demonstrated by other industries or other mental healthcare organisations. As such, QI was generally supported. Although some questioned the QI methodology, investment into improving quality was seen as a necessity, an obligation, and a fundamental philosophical organisational position. In the face of ambiguities, dilemmas, and uncertainties; patience, compromise, and tolerance were exercised. Investment decisions were tailored to expectations based on what was seen as the reality.
“…sometimes you make decisions based on the fact that you won't get any of this. You know you won't get any financial returns. Sometimes you make it based on the fact that actually you’ll save”. Participant 4.
For some participants, financial pressures meant that some form of proof was essential to continue QI investment and support. Failure to provide proof created reluctance towards future QI investment. Some were concerned that this potentially creates a vicious cycle where uncertainty over ROI may lead to disinvestment, and resource disinvestment may lead to more failure and or further uncertainty. Thus, QI was viewed as a risky investment by some. There were also concerns that unwarranted investment into QI could result in a ‘locked-in’ state.
“I think there's sometimes there is a risk in not doing that that you get locked into something that feels right, and for the general good…”. Participant 7.
A participant described this as an issue with investing into the concept, and not the actual daily QI realities. Most participants expressed that QI failure was strongly linked to teams and organisations failing to implement and support QI. There was an awareness that organisational-level challenges such as board governance, resources, as well as QI team governance determined QI success. QI was seen as everybody’s business.
“I've seen some things where people don't implement it right. They don't provide the support and training for staff to do it…, you know you can implement it really badly. Just make it a process that people feel like they're going through. They don't feel any benefits for themselves….and you know you'll fail on it…” Participant 6.
Despite uncertainties, leaders appeared unwilling to disinvest from QI. Instead, they preferred re-examination and redesigning of improvement efforts using different tools or approaches within the QI methodology.
Summary of the themes
The above findings reflect the three themes based on research questions: the conceptualisation of QI-ROI, its influencing factors, and the impact of potential for disinvestment on the concept. Influencing factors had five subthemes: the perceived QI mandates, values, expectations, ambiguity, and uncertainty. There appeared to be a tension and mutual dependency between improving quality and managing scarce resources; the finances and other resources helped improve quality, and improved quality helped manage scarce resources. Ambiguity and uncertainty appeared intricately linked in a self-reinforcing cycle. Different ambiguities led to uncertainty, and uncertainty caused ambiguity. Institutional complexity added to the sense of uncertainty over both costs and benefits from QI. Ambiguity over expectations and uncertainty over QI outcomes posed a threat to the stability of the QI-ROI concept.
The inability to measure and or monetise some valued benefits constituted lack of QI-ROI objective proof. For some, this caused a rejection of non-financial benefits as a legitimate part of ROI, although they remained legitimate QI benefits. The rejection of non-financial benefits appeared to be based on what the traditional ROI warrants. Alternatively, uncertainty due to the inability to measure and or monetise valued benefits emerged as a strong factor in rejecting traditional ROI. Most leaders saw it inconceivable not to include certain valued benefits due to their non-monetisability. In this context, a valued benefit was any that contributes to achieving desired strategic goals. Crucially, leaders were more concerned about measurability (and ‘attributability’) of outcomes than monetisability of outcomes. The attitudes towards QI disinvestment in the face of uncertainty indicated a stability of QI-ROI as any valued benefit.
The net effect was the shifting of the ‘dial’ within the monetary-nonmonetary QI-ROI dimension, with a bias for non-monetary benefits. Some benefits would be included, excluded, or not even considered. Nonetheless, the QI-ROI concept was maintained as any valued monetary and non-monetary benefit, and the QI investment was also set to continue. The perceived mandates and values played a significant role in defining the QI-ROI concept as any benefit. Particularly, the focus on quality had a strong positive influence on maintaining a comprehensive QI-ROI concept. Concerns about scarce resources caused modifications to include financial benefits and view of ROI as a cost-saving tool. The themes are summarised in Fig. 1. Additional quotes, codes, and themes can be found in Supplementary file 4.
The main objective of this study was to explore the extent to which the conceptualisation of QI-ROI indicated by our prior systematic literature review , was shared by mental healthcare leaders. This study found that participants also conceptualised QI-ROI as any valued benefit that contributes to the fulfilment of their organisational strategy. The QI-ROI concept was represented by improved quality in certain internal and external aspects of the organisation. In conceptualising QI-ROI this way, the leaders followed a health and social care logic that prioritises patients’ clinical and social outcomes over monetary outcomes. This differentiates QI-ROI from traditional ROI . Also supported in this study, was a de-emphasis of benefit monetisation, valuing hard to measure and comprehensive internal and external benefits. Thus, this conceptualisation of QI-ROI has challenges, including ambiguities and uncertainties.
The were differences in opinions that gave an organisational level ambiguity over the specifics of QI-ROI, e.g., definitions of QI success and failure. Expectations from QI provided a framework for how ROI from QI was conceptualised. Ambiguities over expectations appeared to lead to ambiguity over QI success and failure. For example, this study indicated that QI can be valued regardless of failing to achieve its intended goals. Further, success at unit or project level was associated with achievement of goals (intervention effectiveness), whilst failure at programme or organisational level was portrayed as implementation failure. In the following section, we discuss our updates on the QI-ROI conceptual framework, and these controversies surrounding this QI-ROI concept.
The QI-ROI conceptual framework
Conceptually, the current study has indicated identical QI-ROI components with the previous literature review. QI contribution to organisational strategies e.g., value-based healthcare (VHBC), transformation, and resilience was also found to be important here. As such, only minor changes were made on the QI-ROI conceptual framework (Fig. 2). The main goals were predominantly presented as patient and staff outcomes. However, other internal and external benefits were also valued. Two main new outcomes emerged as important, an interest in organisational sustainability, and the speed of diagnosis of problems and solutions. QI was said to be driven by two main mandates, to improve quality and manage scarce resources. Thus, new additions in the QI-ROI conceptual framework are the QI mandates as the starting points alongside investment, organisational sustainability as the ultimate objective, and speed as an emerging valued implementation outcome.
The revised framework therefore illustrates the following: at initial implementation, change ensues. Either as a specific strategy or part of doing QI, development occurs (e.g., staff development), and collaboration with partners (e.g., other teams) may occur. This could lead to improved productivity, effectiveness, and efficiency. This may lead to improvements and good practice which must then be embedded, spread, and or disseminated. In the process, QI may contribute to other strategies such as VBHC and transformation. Eventually, organisations may save costs, become financially stable, self-sufficient, resilient, and sustainable. In addition, the framework also indicates that the inherent ambiguities and uncertainties surrounding QI-ROI may be minimised through focused contextualisation, decision analysis, and value judgements [53,54,55,56].
This process (the QI journey) occurs through phases, and gains are re-invested back to the organisation and its stakeholders. Initial investment occurs in phase 1, immediate outcomes maybe gained in phase 2 (ROI 1), intermediate outcomes in phase 3 (ROI 2), longer-term outcomes in phase 4 (ROI 3), and some outcomes may be sustained (end ROI or ROI 4). These phases correspond with four main QI-ROI components: development and improvement (ROI 1–2), cost-saving and sustainability (ROI 3–4). Organisational development is foundational to organisational effectiveness and resilience [57, 58]. Thus, savings are a late return (on investment), that occur after or during sustainment. These four QI-ROI components indicate a logical progression in a QI journey. Connected to them are related concepts (e.g., productivity and efficiency), which act as mechanisms for higher level returns . We consider sustainability an end ROI as it appears to be the ultimate outcome sought. However, QI programmes may continue to provide other benefits beyond this point through legacies.
In our literature review , over 70% of the articles highlighted sustainability as an important outcome. This was linked to adoption, spread, and dissemination of improved practices as well as QI legacies. In this study, sustainability also included the organisation itself. Sustainability is seen as holding current gains whilst continuing to evolve [59,60,61]. Evolution in QI is through trial-and-error. This makes stability difficult to achieve. Sustainability can be difficult to achieve for a variety of reasons [62,63,64]. Sustainability requires time and other resources to help embed and spread effective practices [60, 65]. Thus, sustainability includes widespread adoption of improved practices, not just the sum of multiple initiatives . Sustained improvements can enhance the speed of future programmes.
As stated earlier, the speed of problem and solution identification emerged as important in this study. This is supported by how QI effectiveness is viewed in large programmes. In large programmes, effectiveness includes systemic improvements, efficient problem identification and solving [66, 67]. The speed of implementation is thought to support organisational change . Paradoxically, developing the abilities for efficient problem identification and solving takes time. It follows continual learning, sustained success, and culture change [66, 67, 69]. In this process, QI programmes build a ‘QI legacy’ (retained relationships, capacities, and capabilities) and ‘QI intelligence’ (accumulated QI knowledge) that promote a ‘QI logic’ (a way of thinking about quality issues). All this can support an improvement culture where a QI philosophy is directly or indirectly infused in how new quality problems are managed. Over time, teams can more efficiently identify problems as well as identify and implement potential solutions e.g., the response to COVID-19 described by participants in this study.
In the following sections, we will discuss further some of the challenges highlighted in this study, starting with the ambiguity and uncertainty associated with QI-ROI conceptualisation.
Ambiguity refers to the simultaneous presence of equally plausible but mutually contradictory explanations of a situation or concept . This is common where multiple views co-exist [54, 71]. In this study, ambiguities presented in different mandates, values, and expectations from QI. Further some of the ambiguities appeared to be related to QI philosophy and practice. Ambiguity in QI may be a result of the amalgamation of multidisciplinary and multi-industry concepts e.g., lean and efficiency [71, 72]. When QI was introduced in healthcare, rhetoric was used to promote the QI discourse in efforts to improve communication and facilitate change [71, 72]. In this context, ambiguity can be tolerated and even desired, in-order to accommodate new ideas, multiple and diverse perspectives . However, tensions and conflicts can also occur as a result. As seen in this study, QI buy-in, scepticisms, and different expectations are the remnants of this process. These factors appear to further influence how QI effectiveness and QI benefits are defined and measured, and in-turn, how QI-ROI is conceptualised.
The QI trial-and-error philosophy was one prominent example of QI philosophy ambiguity in this study. The trial-and-error mentality appear to enable conflicting meanings of success, given that QI can still be valued even if ineffective in achieving intended goals. This mentality fosters a learning environment where even negative results are seen valuable . However, this also implies a paradox where failure must be seen as success. Trial-and-error aligns with the well known, ‘fail-fast philosophy that supports innovation and progress in companies such as Amazon and IBM. The fails-fast mindset is associated with Agile, an iterative software-based project management approach .
Although it has its criticisms, e.g., concerns about the neglect of deep-seated challenges, Agile has its advantages. The Agile philosophy empowers staff by fostering a blame-free culture [74, 75]. This can be a valuable philosophy where healthcare staff need psychological safety to experiment and innovate. It can also be an effective strategy where uncertainty about problems and solutions exist [54, 75]. As such, Lange et al.,  suggested embracing the flexible fail-fast mentality rather that push for unattainable success. However, as seen in this study, this can also cause uncertainty about ROI where success is seen as ROI.
Uncertainty can present in different ways depending on it’s a specific source. However, all uncertainty result from a lack of adequate information needed to make decisions [53, 76, 77]. Uncertainty impacts how QI-ROI is conceptualised by limiting perceptions about what QI is or can be beneficial for. A lack of knowledge about QI benefits, causes lack data and scientific evidence about QI-ROI. This may be compounded by differing values and assumptions about the existence, the nature or extent of QI benefits. Some uncertainty can be reduced e.g., through additional information. For example, scientific decision theory uses specific methods to reduce uncertainty depending on the source . Within decision theory methods, sensitivity analysis is used in ROI analysis to assess different hypothetical scenarios . However, healthcare complexity often prevents abilities to reduce uncertainty . There may always be unknown unknowns or unknowable unknowns [76, 80] than require value judgements.
In traditional rational modes of thinking, value judgments may be seen as unacceptable alternatives to objective data . Equally, the appropriateness of classical rational decision theories are challenged. Authors have argued that it should not be assumed that ethical and subjective decision-making is irrational [41, 55, 56, 82,83,84,85]. There is now an understanding that sensemaking is an ongoing social process, driven by plausibility rather that accuracy [56, 85, 86]. Today, managing ambiguity and uncertainty is seen as part of flexible effective leadership . A lack of full scientific certainty is not seen as a reason to postpone or discontinue cost-effective interventions [76, 87]. Thus, a future QI-ROI tool must acknowledge that science alone may be insufficient to guide how to make sound reasonable judgments. Uncertainty is also compounded by the challenges in measuring and attributing externalised costs and benefits.
Due to the psycho-societal causes and impacts of some of the mental health illnesses [5, 88,89,90], mental healthcare frequently engages with external partners. Collaboration with external partners improves patient outcomes, and enable efficient use of resources at both organisational and systems level [91, 92]. In the future, the success of NHS trusts will be judged against their contribution to integrated care systems (ICSs), as well as their internal performance . However, the tension between external and internal needs is a challenge . In financial and political accounting, a societal perspective is only justified in decisions about social welfare . Thus, healthcare leaders are put in conflicting stewardship positions with little guidance on how to manage their role conflict. This leads to a paradox where procedures may take precedence over accountability for performance . Accountability models are however being developed , and the Health and Social Care Act is being updated to support integrated partnerships . External costs and benefits are also hard to measure and attribute.
Immeasurability of valued outcomes
QI’s alignment with biomedical empirical reasoning means that QI largely treats quality as a measurable property . The appropriateness of quantitative approaches for judging QI effects has however been questioned [98, 99]. Today, QI evaluations can also assess qualitative implementation factors such as context and staff engagement . Further, the Healthcare Financial Management Association (HFMA) and NHS Improvement provide guidance on how QI data can be collected and used to support efficiency strategies [92, 100, 101]. This intelligent data in the future may support both internal and externalised value measurement. Various measurement methods are used in mental healthcare , including indicators and proxies [103, 104]. However, some items remain impossible to measure, and the use of proxies and indicators is disputed for the very reason of lacking certainty . Further, valid financial proxies are also lacking . These challenges compound the apprehension over QI benefit monetisation.
De-emphasis of monetisation
Healthcare has been accused of arguing ‘exceptionalism’ in relation to performance indicators used in other industries . Organisations can and do measure overall financial health based on composite financial performance indicators e.g., profit, loss, cash flow, capital, ROI etc. . These can then be matched against patient safety and quality indicators as well as value-based payments . However, increasingly, healthcare investments are being treated as discrete and expected to produce their own ROI, e.g., ROI of IT , research and development , and leadership programmes . Public services’ leaders do object to monetisation. Authors argue that the ‘bottom-line’ need not mean monetisation [105, 110, 111]. However, they are not unique in their concerns . ROI is known to be one aspect of QI value [12, 30].
An economic or markets logic is blamed for being narrow and for failing to acknowledge long-term financial neglect such as that in mental healthcare . Healthcare needs more resources and support to deliver both quality and save costs . Opportunities to improve quality and reduce costs exist, particularly in care overuse . However, the relationship between quality and cost is complex [20, 115]. The iron triangle (time, cost, quality) reflects these tensions where focusing on one aspect may compromise another [86, 116]. This tension sometimes leads to the separation of quality improvement from cost or value improvement campaigns . What appears certain is that profit making is not the focus in healthcare. The NHS can however generate profit to benefit the public, and does so increasingly through commercial and private work [117, 118]. Financial outcomes do matter , however, increasingly non-monetary benefits are also linked to other organisational benefits, such as sustainability .
Healthcare is expected to engage with modern agendas such as environmental, and organisational sustainability [100, 115, 121]. In 2007, Coiera & Hovenga  predicted that the healthcare system will fail if it did not transform itself substantially by 2020. Such predictions spurred efforts to create sustainable healthcare organisations. Organisations must develop dynamic capabilities and capacities, use resources sustainably, and make efficiency savings [91, 100, 122]. A sustainable organisation continually meets the needs of its stakeholders . This requires resilience; the ability to absorb and recover from shocks [59, 124]. However, organisational sustainability is primarily viewed in terms of financial sustainability [124, 125]. As such, a de-emphasis of monetisation in QI-ROI may be problematic. It may appear as a lack or avoidance of financial accountability.
QI-ROI institutional logic
As noted earlier, leaders in this study conceptualised QI-ROI from a health and social care logic. It is worth noting that although the findings here appear to support the institutional theory’s assertion that organisations are driven by norms and values at the expense of efficiency and economic benefit, this is not the complete picture here. Participants in this study saw themselves as stewards of multiple stakeholders and obligations. Efficiency and financial outcomes were seen as important, but not at the expense of patients and staff outcomes. The leaders here questioned the legitimacy of the traditional ROI in their context. These concerns are shared by others, in and outside healthcare [14, 78, 84, 126,127,128]. The overriding logic here appeared to be driven by health and social care values. Different institutional logics may co-exist, how each logic survives depends on its centrality and compatibility with the main logic . The leaders here appeared to be willing to accommodate the economic logic to a certain extent. However, more clarity is needed on the issues raised here before a conclusion is drawn.
ST’s engagement with this study and participants was preceded by prior exploration of the subject of ROI in healthcare and other industries through various literatures. This guided the choice of study design and methods. As such, the discussion on reflexivity is not to add to the rationales for the study design, but only to highlight ST’s experience and concerns during the interviews . Although ST had done background reading on ROI, she had no real-life experience of what was to be discussed with the participants. As such, ST deliberately chose a curiosity-led ‘conversational’ approach due to her student status and ignorance about their lived experiences on the subject. Her intention was not merely to ‘extract information’ but to learn from participants experiences and insights.
To our knowledge, none of the participants knew ST’s background before the interviews. However, her prior knowledge did at times come through. In one interview, a participant asked ST towards the end of the interview ‘how others out there describe ROI’. ST responded by giving a brief description of ROI as ‘seen as financial returns’. In another interview, a participant remarked “…you seem to know a fair amount about ROI already”. This remark was also made at the end an interview as ST thanked the participant for what she had learnt. This also made ST self-conscious as she became concerned that she may be biasing the data she was obtaining. From that point, ST made more efforts not to appear ‘knowing’ in any way.
ST found this exercise stressful as she tried to watch her words and questions more closely. ST then became concerned that her guardedness would or was affecting the flow of the conversations. ST ultimately returned to being less guarded and allowed the conversations to flow. From there, ST’s decided to be less guarded again when and if asked about her background knowledge. In the end ST felt she had to trust that the piloted and iteratively developed topic guide was fit for purpose. That is, it will help maintain the balance between’ knowing and ignorance’ enough to permit an easy conversational flow as well as honest data gathering. That restored ST’s confidence.
Semi-structured questions can lead to confirmation bias resulting from phrasing or rephrasing questions as means to get specific information from participants. However, our research design acknowledged that knowledge is socially constructed. Therefore, by this virtue, we cannot rule out our influence in the data obtained. Similarly, the use of framework analysis may have limited the focus on the emergent nature of qualitative data. However, the deductive-inductive approach minimises this effect. Additionally, the sample was identified and recruited by co-author and the Trust’s QI director. This purposeful sampling method can result in sampling bias. However, the main target sample was the finite number of top-level decision-makers, who were all approached. Finally, the participants were from same Trust, which limits the diversity of views. Although views from different Trusts would provide more depth on the QI-ROI concept, the views from these participants were sufficient at this initial exploratory phase.
Recommendations for research
Given the controversial nature of the features of ROI in mental healthcare, further clarity is needed before a conclusion can be drawn on the QI-ROI concept as it stands currently. Specifically, more data is needed on the collective view on monetisation of QI outcomes, immeasurability of valued QI benefits, the inclusion of external QI benefits, and what QI effectiveness at organisational level means. These questions may be best posed to a wider group of mental healthcare leaders to assess the prevalence and strength of these views.
Mental healthcare leaders primarily conceptualise ROI as any valued benefit. Some also conceptualise ROI as a cost-saving tool. This was a result of needing to manage both the improving quality and managing scarce resources mandates. Overall, leaders sought to compromise in areas of ambiguity and uncertainty so as find a more comfortable medium to service both scarcity and quality. The strong health and social care values, as well as flexible expectations were the strong factors in maintaining the QI-ROI as a broad concept. Political pressures, health, and social care values were strong drivers for QI mandates and investment, in-spite of ambiguity and uncertainty. The unwillingness to disinvest may also be an indicator of a consistent QI-ROI concept, driven by strong health and social care philosophies.
Availability of data and materials
The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request. Some data has been included in this published article as its supplementary documents.
Return on Investment
Return on Investment from healthcare quality improvement
Knapp M, Wong G. Economics and mental health: the current scenario. World Psychiatry. 2020;19(1):3–14.
McDaid D, Park AL, Wahlbeck K. The economic case for the prevention of mental illness. Annu Rev Public Health. 2019;40(1):373–89.
Pfefferbaum B, North CS. Mental health and the Covid-19 pandemic. N Engl J Med. 2020;383(6):510–2.
Kumar A, Nayar KR. COVID 19 and its mental health consequences. J Ment Health. 2021;30(1):1–2. https://doi.org/10.1080/09638237.2020.1757052.
Boland B. Quality improvement in mental health services. BJPsych Bull. 2020;44(1):30–5.
Docherty M, Thornicroft G. Specialist mental health services in England in 2014: overview of funding, access and levels of care. Int J Ment Heal Syst. 2015;9(1):34.
Ross S, Naylor C. Quality improvement in mental health. King's Fund, editor. 2017.
National Health Service (NHS). NHS Mental Health Implementation Plan 2019/20 – 2023/24. In: Service NH, editor. 2019.
Øvretveit J, Gustafson D. Evaluation of quality improvement programmes. Qual Saf Health Care. 2002;11(3):270–5.
Mery G, Dobrow MJ, Baker GR, Im J, Brown A. Evaluating investment in quality improvement capacity building: A systematic review. BMJ Open. 2017;7(2): e012431.
World Health Organization (WHO) Making the investment case for mental health: a WHO/UNDP methodological guidance note. Geneva: World Health Organization; 2019 Contract No.: WHO/UHC/CD-NCD/19.97.
Solid CA. Return on Investment for Healthcare Quality Improvement: Springer; 2020.
Bellucci M, Nitti C, Franchi S, Testi E, Bagnoli L. Accounting for social return on investment (SROI): The costs and benefits of family-centred care by the Ronald McDonald House Charities. Soc Enterp J. 2019;15(1):46–75. https://doi.org/10.1108/SEJ-05-2018-0044.
Glen T, Hostetter G, Roebuck-Spencer TM, Garmoe WS, Scott JG, Hilsabeck RC, Arnett P, Espe-Pfeifer P. Return on investment and value research in neuropsychology: a call to arms. Arch Clin Neuropsychol. 2020;35(5):459–68.
Thusini S, Milenova M, Nahabedian N, Grey B, Soukup T, Chua K-C, Henderson C. The development of the concept of return-on-investment from large-scale quality improvement programmes in healthcare: an integrative systematic literature review. BMC Health Serv Res. 2022;22(1):1492.
Thusini S, Milenova M, Nahabedian N, Grey B, Soukup T, Henderson C. Identifying and understanding benefits associated with return-on-investment from large-scale healthcare Quality Improvement programmes: an integrative systematic literature review. BMC Health Serv Res. 2022;22(1):1083. https://doi.org/10.1186/s12913-022-08171-3.
Chua K-C, Henderson C, Grey B, Holland M, Sevdalis N. Evaluating quality improvement at scale: routine reporting for executive board governance in a UK National Health Service organisation. medRxiv. 2021:2020.02.13.20022475.
Anderson JA. Evolution of the health care quality journey: from cost reduction to facilitating patient safety. J Leg Med. 2010;31(1):59–72.
Fischer HR, Duncan SD. The business case for quality improvement. J Perinatol. 2020;40(6):972–9.
Hussey PS, Wertheimer S, Mehrotra A. The association between health care quality and cost: a systematic review. Ann Intern Med. 2013;158(1):27–34.
Dopp AR, Narcisse M-R, Mundey P, Silovsky JF, Smith AB, Mandell D, et al. A scoping review of strategies for financing the implementation of evidence-based practices in behavioral health systems: State of the literature and future directions. Implementation Research and Practice. 2020;1:2633489520939980.
Baines R, Langelaan M, de Bruijne M, Spreeuwenberg P, Wagner C. How effective are patient safety initiatives? A retrospective patient record review study of changes to patient safety over time. BMJ Qual Saf. 2015;24(9):561–71.
Clay-Williams R, Nosrati H, Cunningham FC, Hillman K, Braithwaite J. Do large-scale hospital- and system-wide interventions improve patient outcomes: a systematic review. BMC Health Serv Res. 2014;14(1):369.
Dixon-Woods M, Martin GP. Does quality improvement improve quality? Future Hospital J. 2016;3(3):191–4.
Ellsbury DL, Clark RH. Does quality improvement work in neonatology improve clinical outcomes? Curr Opin Pediatr. 2017;29(2):129–34.
Savage C, Parke L, von Knorring M, Mazzocato P. Does lean muddy the quality improvement waters? A qualitative study of how a hospital management team understands lean in the context of quality improvement. BMC Health Serv Res. 2016;16(1):588.
Stavropoulou C, Doherty C, Tosey P. How effective are incident-reporting systems for improving patient safety? A systematic literature review. Milbank Q. 2015;93(4):826–66.
Taylor MJ, McNicholas C, Nicolay C, Darzi A, Bell D, Reed JE. Systematic review of the application of the plan–do–study–act method to improve quality in healthcare. BMJ Qual Saf. 2014;23(4):290.
Hjeltnes TA, Hansson B. Cost effectiveness and cost efficiency in e-learning. QUIS-Quality, Interoperability and Standards in e-learning, Norway. 2005.
Al-Raisi AN, Al-Khouri AM. Public value and ROI in the government sector. Advances in Management. 2010;3(2):33–8.
Aksom H, Tymchenko I. How institutional theories explain and fail to explain organizations. J Organ Chang Manag. 2020;33(7):1223–52.
Scott WR. Approaching adulthood: the maturing of institutional theory. Theory Soc. 2008;37(5):427–42.
Zucker LG. The role of institutionalization in cultural persistence. Am Soc Rev. 1977;42(5):726–43.
DiMaggio PJ, Powell WW. The iron cage revisited: Institutional isomorphism and collective rationality in organizational fields. Am Sociol Rev. 1983;48(2):147–60.
Green SE Jr, Li Y. Rhetorical institutionalism: Language, agency, and structure in institutional theory since Alvesson 1993. J Manage Stud. 2011;48(7):1662–97.
Lawrence T, Suddaby R, Leca B. Institutional work: refocusing institutional studies of organization. J Manag Inq. 2011;20(1):52–8.
Grant D, Marshak RJ. Toward a discourse-centered understanding of organizational change. J Appl Behav Sci. 2011;47(2):204–35.
Suddaby R, Greenwood R. Rhetorical strategies of legitimacy. Adm Sci Q. 2005;50(1):35–67.
Thornton PH, Ocasio W. Institutional logics. The Sage handbook of organizational institutionalism. 2008;2008(840):99–128.
Besharov ML, Smith WK. Multiple institutional logics in organizations: explaining their varied nature and implications. Acad Manag Rev. 2014;39(3):364–81.
Suddaby R. Challenges for Institutional Theory. J Manag Inq. 2010;19(1):14–20.
Gasparski WW. The stakeholder organisation theory and its systemic foundation. Int J Gen Syst. 2009;38(6):659–65.
Donaldson L, Davis JH. Stewardship theory or agency theory: CEO governance and shareholder returns. Aust J Manag. 1991;16(1):49–64.
Koch T. Interpretive approaches in nursing research: the influence of Husserl and Heidegger. J Adv Nurs. 1995;21(5):827–36.
Kinsella EA, editor Hermeneutics and critical hermeneutics: Exploring possibilities within the art of interpretation. Forum Qualitative Sozialforschung/Forum: Qualitative Social Research; 2006.
Microsoft. Microsoft Teams United States. 2021. Available from: https://support.microsoft.com/en-us/office/what-s-new-in-microsoft-teams-d7092a6d-c896-424c-b362-a472d5f105de?ui=en-us&rs=en-us&ad=us. Accessed 13 July 2023.
McIntosh MJ, Morse JM. Situating and constructing diversity in semi-structured interviews. Glob Qual Nurs Res. 2015;2. https://doi.org/10.1177/2333393615597674.
QSR International Pty Ltd. NVivo. 2020. Available from: https://www.qsrinternational.com/nvivo-qualitative-data-analysis-software/home.
Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349–57. https://doi.org/10.1093/intqhc/mzm042. Accessed 13 July 2023.
Srivastava A, Thomson SB. Framework analysis: a qualitative methodology for applied policy research. 2009.
Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101.
Fereday J, Muir-Cochrane E. Demonstrating rigor using thematic analysis: a hybrid approach of inductive and deductive coding and theme development. Int J Qual Methods. 2006;5(1):80–92.
Broekhuizen H, Groothuis-Oudshoorn CGM, van Til JA, Hummel JM, Ijzerman MJ. A review and classification of approaches for dealing with uncertainty in multi-criteria decision analysis for healthcare decisions. Pharmacoeconomics. 2015;33(5):445–55.
Hobbs V, Pigott D, Gammack J. Inaccuracy, ambiguity and irrelevance: an analysis of the nature of quality in knowledge management using the noetic prism. 2002.
Cribb A, Entwistle V, Mitchell P. What does ‘quality’ add? Towards an ethics of healthcare improvement. J Med Ethics. 2020;46(2):118.
Schillemans T, Bjurstrøm KH. Trust and verification: balancing agency and stewardship theory in the governance of agencies. Int Public Manag J. 2020;23(5):650–76.
Lalonde C. Managing crises through organisational development: a conceptual framework. Disasters. 2011;35(2):443–64.
Sharma N, Singh RK. A unified model of organizational effectiveness. J Organ Effect People Perform. 2019;6(2):114–28.
James HM, Papoutsi C, Wherton J, Greenhalgh T, Shaw SE. Spread, scale-up, and sustainability of video consulting in health care: systematic review and synthesis guided by the NASSS framework. J Med Internet Res. 2021;23(1): e23775.
Dückers ML, Wagner C, Vos L, Groenewegen PP. Understanding organisational development, sustainability, and diffusion of innovations within hospitals participating in a multilevel quality collaborative. Implement Sci. 2011;6(1):1–10.
Braithwaite J, Ludlow K, Testa L, Herkes J, Augustsson H, Lamprell G, et al. Built to last? The sustainability of healthcare system improvements, programmes and interventions: a systematic integrative review. BMJ Open. 2020;10(6): e036453.
Duclay E, Paille-Ricolleau C, Abbey H, Lombrail P, Moret L. Appropriateness of Medical Prescriptions. French Experience Therapie. 2010;65(4):387–95.
Lennox L, Maher L, Reed J. Navigating the sustainability landscape: a systematic review of sustainability approaches in healthcare. Implement Sci. 2018;13(1):1–17.
Robert G, Sarre S, Maben J, Griffiths P, Chable R. Exploring the sustainability of quality improvement interventions in healthcare organisations: a multiple methods study of the 10-year impact of the ‘Productive Ward: Releasing Time to Care’ programme in English acute hospitals. BMJ Qual Saf. 2020;29(1):31.
Greenhalgh T, Wherton J, Papoutsi C, Lynch J, Hughes G, Hinder S, et al. Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. J Med Internet Res. 2017;19(11): e8775.
Latino RJ. How is the effectiveness of root cause analysis measured in healthcare? J Healthc Risk Manag. 2015;35(2):21–30.
Reed JE, Card AJ. The problem with Plan-Do-Study-Act cycles. BMJ Qual Saf. 2016;25(3):147–52.
Coiera E, Hovenga ES. Building a sustainable health system. Yearb Med Inform. 2007;16(01):11–8.
Curry LA, Spatz E, Cherlin E, Thompson JW, Berg D, Ting HH, et al. What distinguishes top-performing hospitals in acute myocardial infarction mortality rates? A qualitative study. Ann Intern Med. 2011;154(6):384–90.
Mukherjee AS, Lapre MA, Van Wassenhove LN. Knowledge driven quality improvement. Manag Sci. 1998;44(11-part-2):S35-S49.
Giroux H. ‘It Was Such a Handy Term’: Management Fashions and Pragmatic Ambiguity*. J Manage Stud. 2006;43(6):1227–60.
Vakkuri J. Struggling with ambiguity: Public managers as users of NPM-oriented management instruments. Public Administration. 2010;88(4):999–1024.
Hagen M, Park S. Ambiguity acceptance as a function of project management: a new critical success factor. Proj Manag J. 2013;44(2):52–66.
Veneziano V, Rainer AW, Haider S. When Agile Is Not Good Enough: an initial attempt at understanding how to make the right decision. arXiv preprint arXiv:14025557. 2014.
Lange IL, Nalwadda CK, Kiguli J, Penn-Kekana L. The ambiguity imperative:“Success” in a maternal health program in Uganda. Med Anthropol. 2021;40(5):458–72.
Tallacchini M. Before and beyond the precautionary principle: epistemology of uncertainty in science and law. Toxicol Appl Pharmacol. 2005;207(2):645–51.
Rutz SI, Adams SA, Buitendijk SE, Robben PB, de Bont AA. Hiding complexity, uncertainty and ambiguity: how inspectorates simplify issues to create enforceable action. Health Risk Soc. 2013;15(4):363–79.
Chalutz B-G. An ROI-based review of HR analytics: practical implementation tools. Pers Rev. 2019;48(6):1429–48.
Gurses A, Seidl K, Vaidya V, Bochicchio G, Harris A, Hebden J, et al. Systems ambiguity and guideline compliance: a qualitative study of how intensive care units follow evidence-based guidelines to reduce healthcare-associated infections. BMJ Qual Saf. 2008;17(5):351–9.
Stigler GJ. The development of utility theory I. J Pol Eco. 1950;58(4):307–27.
Stanford Encyclopaedia of philosophy. Bounded rationality. 2018. Available from: https://plato.stanford.edu/entries/bounded-rationality/. Accessed 13 July 2023.
Covaleski MA, Dirsmith MW, Samuel S. Managerial accounting research: the contributions of organizational and sociological theories. J Manag Account Res. 1996;8:1–36. https://www.researchgate.net/profile/Sajay-Samuel/publication/284594334.
Eisenberg EM. Ambiguity as strategy in organizational communication. Commun Monogr. 1984;51(3):227–42.
Grant AM. ROI is a poor measure of coaching success: towards a more holistic approach using a well-being and engagement framework. Coaching. 2012;5(2):74–85.
Walshe K, Smith J. Healthcare management: McGraw-Hill Education (UK); 2011.
Storkholm MH, Mazzocato P, Savage M, Savage C. Money’s (not) on my mind: a qualitative study of how staff and managers understand health care’s triple Aim. BMC Health Serv Res. 2017;17(1):98.
Khan S, Vandermorris A, Shepherd J, Begun JW, Lanham HJ, Uhl-Bien M, et al. Embracing uncertainty, managing complexity: applying complexity thinking principles to transformation efforts in healthcare systems. BMC Health Serv Res. 2018;18(1):1–8.
Kennedy-Hendricks A, Huskamp HA, Rutkow L, Barry CL. Improving access to care and reducing involvement in the criminal justice system for people with mental illness. Health Aff. 2016;35(6):1076–83.
Senior M, Fazel S, Tsiachristas A. The economic impact of violence perpetration in severe mental illness: a retrospective, prevalence-based analysis in England and Wales. Lancet Public Health. 2020;5(2):e99–106.
Thornicroft G. People with severe mental illness as the perpetrators and victims of violence: time for a new public health approach. Lancet Public Health. 2020;5(2):e72–3.
National Health Srvice (NHS). Report of the Review of the Quality and Outcomes Framework in England. In: England N, editor. 2018.
National Health Service (NHS). Integrated Care Systems: design framework. NHS; 2021.
National Health Service (NHS). Code of governance for NHS provider trusts. 2022.
Anderson JE, Robert G, Nunes F, Bal R, Burnett S, Karltun A, et al. Translating research on quality improvement in five European countries into a reflective guide for hospital leaders: the “QUASER Hospital Guide.” Int J Qual Health Care. 2019;31(8):G87–96.
National Institute for Health and Care Excellence (NICE). Supporting investment in public health: Review of methods for assessing cost effectiveness, cost impact and return on investment. 2011.
Improvement NEaN. Working together at scale: guidance on provider collaboratives. NHS England and NHS Improvement; 2021.
Government U. Health and Social Care Act 2012. In: Care DoHaS, editor. London: UK Government; 2012.
Portela MC, Pronovost PJ, Woodcock T, Carter P, Dixon-Woods M. How to study improvement interventions: a brief overview of possible study types. BMJ Qual Saf. 2015;24(5):325.
Burke RE, Shojania KG. Rigorous evaluations of evolving interventions: can we have our cake and eat it too? BMJ Qual Saf. 2018;27(4):254–7.
Improvement N. NHS efficiency map. In: Improvement N, editor.: The Healthcare Financial Management Association (HFMA) and NHS Improvement and NHS England; 2019, 2022.
The HFMA and Grant Thornton UK LLP. Improving data quality for costing community and mental health services. 2021.
Margison FR, Barkham M, Evans C, McGrath G, Clark JM, Audin K, et al. Measurement and psychotherapy: Evidence-based practice and practice-based evidence. Br J Psychiatry. 2000;177(2):123–30.
Hubbard DW. How to Measure Anything Workbook : Finding the Value of Intangibles in Business. New York, UNITED STATES: John Wiley & Sons, Incorporated; 2014.
Boudreau JW, Ramstad PM. Measuring intellectual capital: Learning from financial history. Human Resource Manag. 1997;36(3):343-56
Krlev G, Münscher R, Mülbert K. Social Return on Investment (SROI): state-of-the-art and perspectives-a meta-analysis of practice in Social Return on Investment (SROI) studies published 2002–2012. 2013.
Akinleye DD, McNutt L-A, Lazariu V, McLaughlin CC. Correlation between hospital finances and quality and safety of patient care. PLoS One. 2019;14(8).
Czerwinski DS. Return on investment as an indicator of success: an analysis of information technology project management outcomes [Ph.D.]. Ann Arbor: Capella University; 2008.
Frontiers Economics. Rates of return to investment in science and innovation. 2014.
Jeyaraman Maya M, Qadar Sheikh Muhammad Z, Wierzbowski A, Farshidfar F, Lys J, Dickson G, et al. Return on investment in healthcare leadership development programs. Leadersh Health Serv. 2018;31(1):77–97.
Adcroft A, Willis R. The (un)intended outcome of public sector performance measurement. Int J Public Sect Manag. 2005;18(5):386–400.
Marx K, Engels F. TH E COMMUNIST MANIFESTO. Selected Works bu Karl Marx and Frederick Engels Neu York: International Publishers; 1848. p. 1363.
Boyd J, Epanchin-Niell R, Siikamäki J. Conservation planning: a review of return on investment analysis. Rev Environ Eco Policy. 2015;9(1):23–42.
Appleby J, Galea A, Murray R. The NHS productivity challenge. Experience from the front line London: The King’s Fund. 2014.
Øvretveit J. Does improving quality save money. A review of evidence of which improvements to quality reduce costs to health service providers London: The Health Foundation. 2009;95.
Alderwick H, Jones Bryan, Charles Anna, Will W. Making the case for quality improvement: lessons for NHS boards and leaders. King's Fund. 2017.
Pollack J, Helm J, Adler D. What is the Iron Triangle, and how has it changed? 2018.
Damian E. Hodgson SB, Mark Exworthy, Mike Bresnen, John Hassard, and Paula Hyde. LSE British Politics and Policy. 2021. [cited 2022]. Available from: https://blogs.lse.ac.uk/politicsandpolicy/corporatization-commercialization-nhs/. Accessed 13 July 2023.
Hunt P. Income generation in the NHS. J Manag Med. 1989;4(1):56-9. https://doi.org/10.1108/eb060526.
Leatherman S, Sutherland K. The quest for quality in the NHS: refining the NHS reforms, vol. 4. London: Nuffield Trust and the Stationery Office; 2008. p. 1–280.
Ali IM, Abd Rahim N, Abd Shukor SS, Abd Rashid HM, editors. The relationship between intangible assets and firm value. The International Conference on Business and Economic Research; 2010.
Braithwaite J. Changing how we think about healthcare improvement. BMJ. 2018;361: k2014.
Cohen WM, Levinthal DA. Absorptive capacity: a new perspective on learning and innovation. Adm Sci Q. 1990;35(1):128–52.
Dangelico RM, Pujari D, Pontrandolfo P. Green product innovation in manufacturing firms: a sustainability-oriented dynamic capability perspective. Bus Strateg Environ. 2017;26(4):490–506.
Rostkowski T, Strzemiński J, Szymański K. The concepts of sustainable organisation, sustainable human capital management and organisational resilience as guidelines for modernisation of healthcare units in Poland. Edukacja Ekonomistów i Menedżerów. 2020;57(3):9-23.
Lozano R. Proposing a definition and a framework of organisational sustainability: a review of efforts and a survey of approaches to change. Sustainability. 2018;10(4):1157.
Ashton K, Schröder-Bäck P, Clemens T, Dyakova M, Stielke A, Bellis MA. The social value of investing in public health across the life course: a systematic scoping review. BMC Public Health. 2020;20(1):597.
Ozawa S, Clark S, Portnoy A, Grewal S, Brenzel L, Walker DG. Return on investment from childhood immunization in low- and middle-income countries, 2011–20. Health Aff. 2016;35(2):199–207.
Ozminkowski RJ, Serxner S, Marlo K, Kichlu R, Ratelis E, Van de Meulebroecke J. Beyond ROI: Using value of investment to measure employee health and wellness. Popul Health Manag. 2016;19(4):227–9.
Lynch M. Against reflexivity as an academic virtue and source of privileged knowledge. Theory Cult Soc. 2000;17(3):26–54.
The authors would like to thank the South London and Maudsley NHS Foundation Trust’s QI Director and QI Lead for their help in identifying potential participants and piloting of the topic guide. The authors would also like to thank all the participants for their time and insightful views on the ROI concept from QI programmes.
This work is supported by the Economic and Social Research Council, grant number ES/P000703/1.
Ethics approval and consent to participate
This study is registered by King’s College London Research Ethics; MRSP-21/22–25739. All experimental protocols were approved by ethics committee of King’s College London Research Ethics Committee. All methods were carried out in accordance with relevant guidelines and regulations. Each participant received an information leaflet which explained the purpose of the study before they participated. Informed written consent was obtained from each study participants before their interview. Participants were informed that their participation in this study was voluntary and they had the right to withdraw from the study without requiring to provide any rationale. Confidentiality and anonymity of participants were maintained at all stages of the research process.
Consent for publication
TS received funding from Cancer Alliance and Health Education England for training cancer multidisciplinary teams (MDTs) in assessment and quality improvement methods in the United Kingdom. TS received consultancy fees from Roche Diagnostics. Infrastructure support for this research for TS was provided by the NIHR London Medtech and In vitro diagnostic Co-operative. The other authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Thusini, S., Soukup, T., Chua, KC. et al. How is return on investment from quality improvement programmes conceptualised by mental healthcare leaders and why: a qualitative study. BMC Health Serv Res 23, 1009 (2023). https://doi.org/10.1186/s12913-023-09911-9