Skip to main content
Download PDF

Modelling in economic evaluation of mental health prevention: current status and quality of studies

BMC Health Services Research volume 22, Article number: 906 (2022) Cite this article

Abstract

Background

The present study aimed to identify and critically appraise the quality of model-based economic evaluation studies in mental health prevention.

Methods

A systematic search was performed on MEDLINE, EMBASE, EconLit, PsycINFO, and Web of Science. Two reviewers independently screened for eligible records using predefined criteria and extracted data using a pre-piloted data extraction form. The 61-item Philips Checklist was used to critically appraise the studies. Systematic review registration number: CRD42020184519.

Results

Forty-nine studies were eligible to be included. Thirty studies (61.2%) were published in 2015–2021. Forty-seven studies were conducted for higher-income countries. There were mainly cost-utility analyses (n = 31) with the dominant primary outcome of quality-adjusted life year. The most common model was Markov (n = 26). Most of the studies were conducted from a societal or health care perspective (n = 37). Only ten models used a 50-year time horizon (n = 2) or lifetime horizon (n = 8). A wide range of mental health prevention strategies was evaluated with the dominance of selective/indicate strategy and focusing on common mental health problems (e.g., depression, suicide). The percentage of the Philip checkilst’s criteria fulfilled by included studies was 69.3% on average and ranged from 43.3 to 90%. Among three domains of the Philip checklist, criteria on the model structure were fulfilled the most (72.1% on average, ranging from 50.0% to 91.7%), followed by the data domain (69.5% on average, ranging from 28.9% to 94.0%) and the consistency domain (54.6% on average, ranging from 20.0% to 100%). The practice of identification of ‘relevant’ evidence to inform model structure and inputs was inadequately performed. The model validation practice was rarely reported.

Conclusions

There is an increasing number of model-based economic evaluations of mental health prevention available to decision-makers, but evidence has been limited to the higher-income countries and the short-term horizon. Despite a high level of heterogeneity in study scope and model structure among included studies, almost all mental health prevention interventions were either cost-saving or cost-effective. Future models should make efforts to conduct in the low-resource context setting, expand the time horizon, improve the evidence identification to inform model structure and inputs, and promote the practice of model validation.

Peer Review reports

Introduction

Mental disorders have posed a significant burden on health and wellbeing for individuals, families and communities worldwide. It is estimated that the burden of mental health disorders accounted for 14.4% of years lived with disability (YLDs) and 4.9% of disability-adjusted life years (DALYs) in 2017 [1]. An increasing body of literature discusses the benefits of interventions to promote better mental health and well-being and prevent mental illness from early childhood and adolescence until older age [2,3,4]. Even in high-income countries, mental health prevention interventions have not received adequate investment despite their profound benefit [2]. In the context of scarce resources, evidence on the burden of mental health and the effectiveness of mental health prevention is not adequate to advocate for the investment in mental health prevention [3, 5]. Economic evaluation tools play a more critical role in informing investment decision making both for mental health in particular and for health care in general [3].

Some systematic reviews of economic evaluations related to mental health prevention [5,6,7,8,9] were published, but none of them was dedicated to a model-based design. In general, the trial-based approach was the dominant study design in the previous systematic reviews. Trial-based economic evaluation might have several limitations, such as having inadequate patient follow-up and not capturing the final health outcome. Meanwhile, preventive interventions are expected to have a beneficial impact on mental health outcomes for some considerable period after the end of the trial [10]. Thus, model-based design is fundamental in an economic evaluation of mental health prevention due to its advantages, including the ability to: (1) consider all relevant alternatives required by policy makers; (2) make the results applicable to the decision-making context; (3) reflect all relevant evidence that not often collected in trials; (4) ability to reflect the final outcomes rather than intermediate outcome; (5) ability to extrapolate over medium- and long-term horizon of the evaluation. Model-based economic evaluation is also less costly than its counterpart employing trial-based design. However, poor practice in economic evaluation modelling of mental health prevention might deliver unreliable results and create barriers in disseminating the results to policymakers.

Thus, the primary objective of this study is to identify and critically appraise all model-based economic evaluations of mental health prevention interventions. This study will reveal the current situation of applying modelling techniques in the economic evaluations of mental health preventions. It will support practice and policy with evidence on the medium and long-term cost-effectiveness of mental health prevention along with the quality of evidence. This study also helps to make recommendations about future models in the field.

Methods

We followed the Cochrane Collaboration guideline of conducting a systematic review for economic evidence [11] and consulted with other recommendations [12,13,14] (See Table S1-Online Supplementary file for the Prefered Reporting Items for Systematic reviews and Meta-Analysis (PRISMA) checklist). We registered the review protocol on the International Prospective Register of Systematic Reviews (CRD42020184519).

Inclusion and exclusion criteria

The studies were included if meeting the following criteria presented in Table 1. There are many definitions relating to mental health prevention activities. This review considered the definition used by WHO [15]. Prevention of mental disorders could be categorised as universal prevention (i.e., targeting the general public or a whole population group); selective prevention (i.e., targeting subgroups of the population whose risk of developing a mental disorder is significantly higher than that of the rest of the population) and indicated prevention (i.e., targeting persons at high-risk for mental disorders). We included interventions that addressed mental disorders, such as depression, anxiety disorder, bipolar disorder, schizophrenia and other psychoses, based on ICD-10 classification [16]; or well-known mental health risks behaviours, including bullying victimisation, intimate partner violence, childhood sexual abuse and suicide. Due to the differences in the nature of prevention for mental health disorders resulting from substance abuse, dementia and other neurocognitive disorders, we excluded interventions addressing the above mental disorders.

Table 1 Inclusion and exclusion criteria
Full size table

We only included full economic evaluations, which addressed the identification, measurement, valuation and comparison of both costs and consequences of at least two alternatives [17]. We only included studies employing model-based design, which compares the expected costs and consequences of decision options by synthesising information from multiple sources and applying mathematical techniques [17, 18] (i.e., including any study beyond the direct application of observed data).

Information sources

The following electronic bibliographic databases of published studies were searched: MEDLINE (via Pubmed), EMBASE (via http://www.embase.com), EconLit, PsycINFO and Web of Science. We also identified potential additional studies by citation tracking in Google Scholar and systematic scanning of the reference lists of eligible studies and relevant review articles. We re-performed the search on 8th November 2021.

Search strategy and data management

The search query referred to terms covering the core concept of the research question, including mental health AND prevention/promotion intervention AND economic evaluation. We consulted the search strategy developed in a recent systematic review [8] to finalize our search strategy. Full details are available in Online Supplementary File (Table S2). The literature search results were managed using Endnote X9.

Selection process

Two reviewers (NTH and NQA) independently screened titles and abstracts against the selection criteria. Then, all potential full-text papers were reviewed. Any disagreement or conflicting views between the two reviewers were resolved by discussion with a third reviewer (NTHg). To aid the study selection and analysis of non-English language articles, translation, either in part or in whole, will be undertaken by an appropriately qualified person.

Data extraction

All recommended items [14], including general background, method and results of the studies, were recorded using Excel in a pre-piloted data extraction form. Two reviewers (NTH and NQA) extracted the data. Any discrepancies between the reviewers over the data extraction process were identified and resolved by discussion or the final judgement of a third reviewer (NTHg). The CCEMG-EPPI-Centre Cost Converter [19], a web-based tool, was used to adjust cost estimation into 2021 USD dollars (using International Monetary Fund World Economic Outlook Database for Purchasing Power Parities values).

Quality assessment of included studies

Since this review focuses on modelling studies, the Philips Checklist [20] was used as recommended [21, 22]. The 61-item Philips Checklist was completed by two reviewers (NTH and NQA). Any disagreements were discussed until a consensus was reached. Responses for the checklist items included yes (Y), no (N), not applicable (N/A, for items that were not relevant to the study), and partial (P, for items that had multiple elements and were not fully satisfied by the study). To summarize the quality assessment results, we calculated the percentage of criteria fulfilled as applied by other researchers. A “Y”, “N”, “P”, and “N/A” responses were counted as one, zero or half of a point and discounted from the calculation, respectively.

Data synthesis

Following guidance on narrative synthesis in systematic reviews [23], we employed textual descriptions, tabulation, groupings and vote-counting to synthesise the findings. Due to the heterogeneity, we used the dominance ranking matrix [24] to summarize cost-effectiveness results.

Results

Study selection

The systematic search returned 8,453 records. After removing duplicates and initial screening, 86 full texts were accessed. Thirty-seven full texts were excluded (See detailed reasons for exclusion in Table S3-Online Supplementary File). Forty-nine studies were included in the review (See Fig. 1 for the selection process).

Fig. 1
figure 1

PRISMA flow diagram of study identification and selection process

Full size image

Study characteristics

Table 2 summarises the characteristics of included studies. A wide range of mental health disorders and risk factors were evaluated in 49 included studies. Depression was the most common topic (n = 14), follow by suicide (n = 12), eating disorder (n = 4), anxiety (n = 4), bullying (n = 4), violence (n = 4), behavior disorder (n = 3), abuse (n = 3), and one exceptional study [25] on prevention of psychotic disorders for ultra-high risk population. The most common prevention approach across the studies was the indicated strategy, i.e., that targets high-risk populations (n = 31), followed by universal preventions (n = 15) and selective preventions (n = 10). Comparators were mainly “no intervention” or “usual care”.

Table 2 Summary of included studies
Full size table

The included studies were published from 2001 to 2021. Only three [26,27,28] studies were published before 2010, with the earliest one on depression published in 2001 [27]. From 2010 until 2014, 17 studies were published. Almost double this number of studies (n = 29) were published in 2015–2021. The majority of models (n = 47) were conducted for higher-income countries. Meanwhile, only one study was conducted in Sri Lanka [29], a lower-middle-income country, and another study [30] was performed in multiple countries, including both higher-income and lower-income countries. Regarding the type of economic evaluation, there were 26 CUAs, nine CEAs, six CBAs and three ROIs and the remaining studies were a combination of CEA and CUA (n = 4) or CUA and CBA (n = 1). For the CUAs, Quality-Adjusted Life Year (QALY) was most commonly used (n = 21). In ten studies, Disability-Adjusted Life Year (DALY) and its variant (Healthy-Life Year Gained, HLYG) were used. The clinical outcomes measuring in the CEAs included life-year (LY) gained [29, 31, 32], life year with a mental health problem (i.e., eating disorder) avoided [33], victim-free year (for bullying) [34, 35], cases (i.e., cases with behaviour disorder [26], eating disorder [36], depression [37], and suicide [38]) or cases with meaningful change on symptom scale [39].

A societal perspective was taken in 22 studies, followed by 15 studies that took the health sector perspective. Three studies did not state the perspective used [28, 31, 40]. Markov models were the most common modelling approach, used in 26 studies (52.0%). Other six studies employed decision tree [35, 38, 39, 41,42,43], and one study employed a combination of Markov and decision tree [44]. The remaining 16 studies did not explicitly describe their model type. They simply applied mathematic formulations without figures presenting their model structure. Their so-called modelling approach could not be classified under any paradigm (i.e. cohort-bassed like Markov, decision tree, system dynamics model or individual-based like discrete event simulation, agent-based model).

Quality assessment

The detailed quality assessment results using Philips Checklist for each study are presented in Table 3. As proposed in the method part, we applied a scoring system to estimate the percentage of the number of Philips Checklist’s items fulfilled (i.e., applied one, zero, half of a point and discounted from the calculation for the “Y”, “N”, “P”, and “N/A” responses, respectively). As a result, the scores from this calculation were 69.3% on average and ranged from 43.3% to 90.0% for overall study performance. Among three domains of the Philip checklist, criteria on model structure were fulfilled the most (72,1% on average, ranging from 50,0% to 91,7%), followed by the data domain (69,5% on average, ranging from 28,9% to 94,0%) and the consistency domain (54,6% on average, ranging from 20,0% to 100%). The following parts present the results of quality appraisal in terms of three domains of the Phillips Checklist, i.e., model structure, data and consistency.

Table 3 Quality assessment results using the Phillips Checklist
Full size table

Model structure

Detailed information on some key structural aspects of the included models is presented in Table 4. Almost all studies demonstrated a clear statement of the decision problem and objectives of the model. However, the primary decision-maker was only specified in 33 studies (67.3%). Although the statement of scope and perspective of the models were commonly stated clearly, there were four remaining studies [28, 31, 40, 49] that did not explicitly state the studies’ perspectives.

Table 4 General characteristic of included studies
Full size table

Less than half of the included studies (n = 23) provided sufficient explanation for selecting the structure of the decision-analytic model. Only five studies were informed by systematic reviews [41, 43, 44] or literature reviews [28, 33]. Other five studies stated that the models were based on intervention clinical evidence (e.g., RCTs) [29, 31, 55], a disease classification [25] or evidence from cohort data [47]. The remaining 13 studies stated that the models were built based on previous models [30, 45, 49, 50, 52, 56, 59, 61, 67, 69, 70, 72, 73]. It is also worth noting that none of the included studies mentioned any competing theories regarding model structure.

Several structural assumptions were made for the purpose of modelling. The key assumptions included efficacy of interventions over a long term period, assumptions to simplify the model structure, assumptions relating to transition probabilities and treatment pathway, etc. To extrapolate the long-term intervention effectiveness, 29 studies assumed the intervention effect lasted over time. Of 29 studies, almost all did not mention whether these assumptions were validated. The authors often assumed that the intervention effect remained over time (i.e., for one year [40, 45, 47, 48, 59, 64], two years [73], four years [52], five years [30] or even a lifetime [28, 56, 70, 72]. They also assumed that the intervention effect gradually decreased with a specified decay rate. A decay rate of 50% was commonly used in included studies [46, 52, 58, 73]. Another common assumption to extrapolate the long term intervention effectiveness was that considering the interventions run over the time horizon [33,34,35, 50, 67,68,69].

However, the above structural assumptions, and the model structure in general, were rarely validated. In only eight models, expert opinions were stated to be used to conduct face validation [25, 30, 43, 59] or to provide justification on interventions [33, 45, 47] and time horizon [29]. Even in the mentioned models, the authors often provided little explanation [25, 33, 43, 45, 59] or no explanation [29, 30, 47] for the methods of employing experts in providing justifications for the model.

Although almost all studies evaluated all feasible and practical options relating to the stated decision problem, only 12 models provided detailed justification and criteria for excluding feasible options [25, 31, 35, 43, 45,46,47, 52, 54, 66, 71, 73].

The model's time horizon was considered sufficient to reflect all important differences between options in 30 studies (61.2%). Only ten models used a 50-year time horizon [50, 59] or lifetime horizon [27, 28, 30, 32, 56, 66, 70, 72]. In models with a shorter time horizon, only 22 studies (44.9%) justified the use of a shorter time horizon. In 27 Markov models, three studies (accounted for 11.0% of all Markov models) did not explicitly state the cycle length [47, 50, 52] and 11 studies (accounted for 40.7% of all Markov models) did not provide any justification for the chosen cycle length [31, 34, 44,45,46, 48, 57, 58, 60, 61, 73].

Data

Generally, methods for identifying data were evaluated as transparent and appropriate in all included studies. However, only 25 studies (51.0%) stated to use a systematic review to inform the selection of key parameters. For example, in terms of measuring intervention effect, 16 studies (32.7%) employed systematic review to identify intervention effect [27, 30, 34,35,36,37, 41, 44, 45, 47, 52, 55, 56, 59, 64, 73]. Meanwhile, 26 studies (53.1%) used evidence from a single trial. Other remaining studies identified key parameters of intervention effect from surveys [33, 62], longitudinal data [63] or pre-post intervention study [43, 49, 60].

In 13 studies, expert opinions were stated to be used to estimate particular parameter [29,30,31, 41, 42, 45, 47, 52, 55, 56, 64, 66]. Although the remaining studies did not report the use of expert opinion, they employed many authors’ own opinions in parameter estimations [26,27,28, 31, 32, 43, 65, 71]. Besides, it is worth noting that only four out of 13 studies that stated the use of expert opinions described the methods of getting expert opinions [25, 30, 45, 47].

Relating half-cycle correction, only six studies applied [25, 44, 59, 61, 64, 69]. The remaining models did not state the application of half-cycle correction and the reasons for the omission.

Regarding uncertainty assessment, three studies [36, 49, 63] did not perform any kind of uncertainty assessment. Only nine studies [26, 30, 41, 44,45,46, 58, 60, 73] performed all four principle types of uncertainty assessment (i.e., parameter uncertainty, structure uncertainty, methodology uncertainty and heterogeneity). Heterogeneity was the most common type of uncertainty being omitted (n = 40), followed by methodology uncertainty (n = 17) and structural uncertainty (n = 16).

Among 46 models that performed parameter uncertainty analysis, 12 studies only addressed univariate sensitivity analysis [26, 29, 32, 39, 40, 43, 56, 62, 64, 66, 68, 71]. Nine studies only performed probabilistic sensitivity analysis [25, 41, 44, 47, 48, 54, 55, 67, 72]. The remaining 26 studies performed both univariate sensitivity analysis and probabilistic sensitivity analysis. Although it is recommended that the ranges used for sensitivity analysis be stated clearly and justified, many models did not specify the value ranges and their reasons [36, 39, 40, 49, 54, 55, 57, 58, 60, 61, 63, 71, 72]. Besides, only 12 studies clearly described and justified the choice of distribution for each parameter [25, 30, 33, 35, 37, 38, 42, 47, 50, 53, 57, 67].

Consistency

There was limited evidence that the mathematical logic of the models in included studies had been tested thoroughly before use. Only one study [25] mentioned that the model was validated based on the Assessment of the Validation Status of Health Economics decision models (AdViSHe) questionnaire [74]. Indeed, the mathematical logic of the model was validated by extreme value testing and by checking whether the relative number of patients in each cycle and state was consistent with empirical evidence [25].

Only six studies [25, 56, 64, 67,68,69] (12.2%) mentioned the application of model calibration for transition probabilities [25, 64, 67,68,69], epidemiological outcomes [25] and cost outcomes [56].

More than half of the studies (n = 29, 59.2%) compared their results with other models’ results and explained the reasons for any differences. The remaining 20 studies did not mention any earlier models for reference.

Cost-effectiveness

As mentioned in the analysis method, we used the dominance ranking metrics for the qualitative synthesis of the cost-effectiveness results of included studies (See Table 5). More detailed information on the cost-effectiveness of included studies could be found in Online Supplementary File (Table S4).

Table 5 The dominance ranking matrix
Full size table

Among 61 interventions that were analyzed in 49 included studies, no intervention was dominated (i.e., less effective but more costly). Twenty-one interventions (34.4% of interventions) were classified as “favour” because they were more effective but less costly. Most of them were selective or indicated prevention interventions (17 out of 21 interventions), were modelled from a time horizon of five years and above (14 out of 21 interventions), were targeted for the prevention of depression (n = 4), behavioural disorder (n = 4), suicide (n = 4), violence (n = 3), anxiety (n = 2), eating disorder (n = 2), abuse (n = 1), and psychosis (n = 1).

The remaining 40 interventions (65.6%) delivered better health outcomes but at a higher cost. Based on the authors’ conclusions and the thresholds provided, almost all of them (34 out of 40 interventions) were “value for money”, given that the ICER remained under corresponding thresholds (typically US$50,000 – US$100,000 in the US, AU$50,000 in Australia, £20,000-£30,000 in the UK) or ROI was greater than 1. Only six interventions, which four prevented depression in the adult population [27, 41, 42, 44], one intervention focused on eating disorders [52], and one intervention that prevented bullying in the children and adolescent population [34] were considered to be not cost-effective since the ICERs were above the thresholds.

Discussion

This systematic review has shown the current situation in published decision-analytic models for mental health prevention interventions. Although there were similar systematic reviews on economic evaluations of mental health prevention interventions, they did not focus on model-based studies. Thus, this systematic review is the first to try to summarise and critically appraise all model-based economic evaluations in the field. The results of this review will provide more evidence to support practice and policy with evidence on medium and long term cost-effectiveness of mental health prevention and aid researchers in improving the quality of future decision-analytic models.

There has been a rapid increase in the number of economic evaluation models in this field, with more than half of included models being published in the last five years (i.e., 2015 to 2020). However, almost all included models were conducted for higher-income countries rather than lower-income countries despite the fact that the burden of mental health problems (in terms of DALYs) is increasing more rapidly in lower-income countries than in their higher-income counterparts [75]. The most common type of economic evaluation was CUA, with the dominant use of QALY as the primary outcome and the application of the Markov model from the societal or health sector perspective. A wide range of prevention strategies was evaluated in the included studies, with the dominance of selective or indicated prevention. It is easy to understand since universal prevention intervention is believed to be more costly than its alternatives. Interventions in included studies also targeted a wide range of mental health problems and risk factors, in which interventions targeted depression and suicide were dominant. This review calls for more decision-analytic models in the future that diversify the topic of mental health problems being addressed, the type of prevention strategies (that focus more on universal prevention intervention) being evaluated and the context of intervention (that focus more on lower-income countries).

Despite a high level of heterogeneity relating to study scope and model structure among included decision-analytic models, almost all mental health prevention interventions were cost-saving (21 interventions, accounting for 34.4%) or cost-effective (34 interventions, accounting for 55.7%). This review identified a large number of interventions for mental health prevention that are cost-saving. All cost-saving interventions have characteristics of indicated or selective prevention strategies, except for one anti-suicide multicomponent program (which had a universal component along with indicated and selective component) [62]. The target population in the cost-saving interventions were often adults (80.9% of cost-saving interventions). They also tended to be analyzed in a longer time horizon (i.e., 12 out of 21 cost-saving interventions were captured in a time horizon of ten years or more). None of the included interventions was less effective but more costly. It is different from the findings of a similar review [9], in which two interventions on depression prevention (which were assessed in a trial-based economic evaluation) were less effective but more costly.

Quality of decision-analytic models

Critically appraising the quality of the included studies revealed several significant limitations of included decision-analytic models. Firstly, a large number of papers reported little or no details of the model structures and the rationale for choosing the models. Only in five studies, the model structures were informed by the systematic reviews or literature reviews. Secondly, although one of the advantages of applying modelling is that it allows estimating interventions’ cost and outcome over a sufficient time horizon outside RCTs, many included models in this review were only modelled for one year or less. Thirdly, the structural assumptions, notably those assumptions needed to extrapolate the short-term outcome of intervention into long-term outcome, were rarely validated. Even in the studies that mentioned the use of expert opinions to validate the assumptions, the report of the method used was insufficient. Fourthly, systematic reviews were not used to identify the key parameters such as intervention effect in many included studies. Fifthly, there was limited evidence that the mathematical logic of the models in included studies had been tested thoroughly before use. Internal validation techniques such as extreme value testing or model calibration were only mentioned in a minimal number of studies. Sixthly, many studies skipped performing at least one in four principal types of uncertainty analysis, i.e., parameter uncertainty, structure uncertainty, methodology uncertainty and heterogeneity. Notably, three studies did not perform any kind of uncertainty analysis despite the crucial role of uncertainty analysis in modelling studies. Lastly, many studies remained to be lack details and transparency in reporting their model structures (e.g., specified primary decision-makers, perspectives) and in the data selection/incorporation process (e.g., quality of data, justification for the choice of distribution, reason for the omission of half-cycle correction).

This review also calls for future decision-analytic models to improve their quality to better inform the policy-making process. The model structure should be sufficiently described, and evidence to inform the model structure should also be better provided. Similar to recommendations by other authors [3, 9], our review continues to call for the application of a longer time horizon to fully capture the costs and outcomes of mental health prevention interventions. To do so, the structural assumptions, notably those assumptions needed to extrapolate the short-term outcomes of intervention into long-term outcomes, were inevitable and necessary to be better reported and validated. Authors of future models should make efforts to validate the model, especially for model structure, model assumptions, and the mathematical logic of the models. Authors might consult the Assessment of the Validation Status of Health-Economic decision models (AdViSHe) questionnaire for this purpose [74]. Other methodological limitations should also be improved, such as applying a more systematic method for identifying key model parameters, addressing not only parameter uncertainty but also structure uncertainty, methodology uncertainty and heterogeneity. The quality of the reporting decision-analytic model should also be improved by applying a guideline or checklist specialised in modelling techniques, such as the Philips checklist [20] or the ISPOR checklist [76].

Strengths and limitations

This review is the first to focus on model-based economic evaluations of mental health prevention. Previous systematic reviews [9, 77, 78] commonly addressed trial-based economic evaluation studies, examined short-term costs and consequences and did not reflect real-life practice. Thus, our search strategy was more sensitive in detecting model-based economic evaluations. Our review comprehensively covers a wide range of mental health problems and well-known related issues such as suicide, violence, bullying or abuse. We also did not apply any restriction on beneficences age, economic evaluation type and publication year. Our review also critically appraised the quality of the included studies by the Philips Checklist, which is recommended for addressing model-based economic evaluations.

Our review has some limitations. Firstly, our search strategy only used English keywords to search for relevant records from proposed electronic databases and other sources. The study selection also included only records that their full texts were available in English. Thus, potentially relevant studies could be missed. Secondly, since many studies did not have a clear model structure, it was challenging to apply some items of the Philips Checklist, for example, the appraisal items related to transition probabilities or cycle length. Lastly, a wide range of mental health issues was covered in our review. We excluded studies that could not distinguish between mental health outcomes and other outcomes, e.g. physical outcomes, educational outcomes, and development outcomes. Besides, although it was not initially suggested to quantify the responses to the Philips Checklist, we applied a scoring approach to estimate the percentage of items fulfilled. By doing so, we must assume equal weighting to all criteria, even though some criteria might be more critical than others.

Conclusions

This review is the first to focus on decision-analytic models for mental health prevention. There is an increasing number of decision-analytic models. Still, evidence has limited to higher-income countries, in the most common mental health problems (e.g., depression and suicide), and still limited to the short-term horizon. Despite a high level of heterogeneity relating to study scope and model structure among included decision-analytic models, almost all mental health prevention interventions were cost-saving or cost-effective to invest in. Researchers should develop more models in the low-resource context, expand the time horizon, improve the evidence identification to inform model structure and inputs, and improve the practice of model validation.

Availability of data and materials

All data generated or analysed during this study are included in this published article [and its supplementary information files].

References

  1. Global Burden of Disease Collaborative Network. Global Burden of Disease Study 2019 (GBD 2019) Results. Seattle, United States: Institute for Health Metrics and Evaluation (IHME), 2020. Available from https://vizhub.healthdata.org/gbd-results/.

  2. McDaid D, Hewlett E, Park AL. Understanding effective approaches to promoting mental health and preventing mental illness. 2017.

    Google Scholar 

  3. McDaid D, Park A-L, Wahlbeck K. The economic case for the prevention of mental illness. Annual Review of Public Health. 2019;40:373–89.

    Article  PubMed  Google Scholar 

  4. Thomas S, Jenkins R, Burch T, Calamos Nasir L, Fisher B, Giotaki G, et al. Promoting mental health and preventing mental illness in general practice. London J Prim Care (Abingdon). 2016;8(1):3–9.

    Article  Google Scholar 

  5. Mihalopoulos C, Chatterton ML. Economic evaluations of interventions designed to prevent mental disorders: a systematic review. Early Intervention in Psychiatry. 2015;9(2):85–92.

    Article  PubMed  Google Scholar 

  6. Zechmeister I, Kilian R, McDaid D. Is it worth investing in mental health promotion and prevention of mental illness? A systematic review of the evidence from economic evaluations. BMC Public Health. 2008;8(1):20.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Mihalopoulos C, Vos T, Pirkis J, Carter R. The economic analysis of prevention in mental health programs. Annual Review of Clinical Psychology. 2011;7:169–201.

    Article  PubMed  Google Scholar 

  8. Schmidt M, Werbrouck A, Verhaeghe N, Putman K, Simoens S, Annemans L. Universal mental health interventions for children and adolescents: a systematic review of health economic evaluations. Appl Health Econ Health Pol. 2020;18(2):155-75.

  9. Le LKD, Esturas AC, Mihalopoulos C, Chiotelis O, Bucholc J, Chatterton ML, et al. Cost-effectiveness evidence of mental health prevention and promotion interventions: a systematic review of economic evaluations. PLoS Medicine. 2021;18(5):e1003606.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Colizzi M, Lasalvia A, Ruggeri M. Prevention and early intervention in youth mental health: is it time for a multidisciplinary and trans-diagnostic model for care? International Journal of Mental Health Systems. 2020;14(1):23.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). Cochrane, 2022. Available from www.training.cochrane.org/handbook.

  12. Thielen FW, Van Mastrigt G, Burgers LT, Bramer WM, Majoie HJM, Evers S, et al. How to prepare a systematic review of economic evaluations for clinical practice guidelines: database selection and search strategy development (part 2/3). Expert Review of Pharmacoeconomics & Outcomes Research. 2016;16(6):705–21.

    Article  CAS  Google Scholar 

  13. van Mastrigt GAPG, Hiligsmann M, Arts JJC, Broos PH, Kleijnen J, Evers SMAA, et al. How to prepare a systematic review of economic evaluations for informing evidence-based healthcare decisions: a five-step approach (part 1/3). Expert Review of Pharmacoeconomics & Outcomes Research. 2016;16(6):689–704.

    Article  Google Scholar 

  14. Wijnen BFM, Van Mastrigt G, Redekop WK, Majoie HJM, De Kinderen RJA, Evers S. How to prepare a systematic review of economic evaluations for informing evidence-based healthcare decisions: data extraction, risk of bias, and transferability (part 3/3). Expert Review of Pharmacoeconomics & Outcomes Research. 2016;16(6):723–32.

    Article  Google Scholar 

  15. World Health Organization. Prevention and promotion in mental health. Geneva: World Health Organization; 2002. Available from: https://apps.who.int/iris/bitstream/handle/10665/42539/9241562161.pdf.

  16. World Health Organization. The ICD-10 classification of mental and behavioural disorders: clinical descriptions and diagnostic guidelines. World Health Organization; 1992.

  17. Drummond MF, Sculpher MJ, Claxton K, Stoddart GL, Torrance GW. Chapter 9: Economic evaluation using decision-analytic modelling. Methods for the economic evaluation of health care programmes. Oxford: Oxford University Press; 2015.

  18. Briggs A, Sculpher M, Claxton K. Chapter 1: Introduction. Decision modelling for health economic evaluation. Oxford: Oxford University Press; 2006.

  19. Shemilt I, Thomas J, Morciano M. A web-based tool for adjusting costs to a specific target currency and price year. Evid Policy: J Res Debate Pract. 2010;6(1):51–9.

    Article  Google Scholar 

  20. Philips Z, Bojke L, Sculpher M, Claxton K, Golder S. Good practice guidelines for decision-analytic modelling in health technology assessment. PharmacoEconomics. 2006;24(4):355–71.

    Article  PubMed  Google Scholar 

  21. Aluko P, Graybill E, Craig D, Henderson C, Drummond M, Wilson E, et al. Chapter 20: Economic evidence. 2020. In: Cochrane Handbook for Systematic Reviews of Interventions version 61 (updated September 2020). Cochrane. Available from: https://training.cochrane.org/handbook/current/chapter-20.

  22. Wijnen BF, Van Mastrigt G, Redekop W, Majoie H, De Kinderen R, Evers S. How to prepare a systematic review of economic evaluations for informing evidence-based healthcare decisions: data extraction, risk of bias, and transferability (part 3/3). Expert Review of Pharmacoeconomics & Outcomes Research. 2016;16(6):723–32.

    Article  Google Scholar 

  23. Popay J, Roberts H, Sowden A, Petticrew M, Arai L, Rodgers M, et al. Guidance on the conduct of narrative synthesis in systematic reviews. A product from the ESRC methods programme Version. 2006;1:b92.

    Google Scholar 

  24. Gomersall JS, Jadotte YT, Xue Y, Lockwood S, Riddle D, Preda A. Conducting systematic reviews of economic evaluations. JBI Evid Implement. 2015;13(3):170–8.

    Google Scholar 

  25. Wijnen BFM, Thielen FW, Konings S, Feenstra T, Van Der Gaag M, Veling W, et al. Designing and testing of a health-economic markov model for prevention and treatment of early psychosis. Expert Review of Pharmacoeconomics & Outcomes Research. 2020;20(3):269–79.

    Article  Google Scholar 

  26. Mihalopoulos C, Sanders MR, Turner KM, Murphy-Brennan M, Carter R. Does the triple P-Positive parenting Program provide value for money? The Australian and New Zealand Journal of Psychiatry. 2007;41(3):239–46.

    Article  PubMed  Google Scholar 

  27. Valenstein M, Vijan S, Zeber JE, Boehm K, Buttar A. The cost-utility of screening for depression in primary care. Annals of Internal Medicine. 2001;134(5):345–60.

    Article  CAS  PubMed  Google Scholar 

  28. Hummel S, Naylor P, Chilcott J, Guillaume L, Wilkinson A, Blank L, et al. Cost effectiveness of universal interventions which aim to promote emotional and social wellbeing in secondary schools. Sheffield, UK: University of Sheffield; 2009.

    Google Scholar 

  29. Damerow SM, Weerasinghe M, Madsen LB, Hansen KS, Pearson M, Eddleston M, et al. Using ex-ante economic evaluation to inform research priorities in pesticide self-poisoning prevention: the case of a shop-based gatekeeper training programme in rural Sri Lanka. Tropical Medicine & International Health. 2020;25(10):1205–13.

    Article  Google Scholar 

  30. Lee YY, Chisholm D, Eddleston M, Gunnell D, Fleischmann A, Konradsen F, et al. The cost-effectiveness of banning highly hazardous pesticides to prevent suicides due to pesticide self-ingestion across 14 countries: an economic modelling study. The Lancet Global Health. 2021;9(3):e291–300.

    Article  CAS  PubMed  Google Scholar 

  31. Denchev P, Pearson JL, Allen MH, Claassen CA, Currier GW, Zatzick DF, et al. Modeling the cost-effectiveness of interventions to reduce suicide risk among hospital emergency department patients. Psychiatric Services (Washington, D C). 2018;69(1):23–31.

    Article  Google Scholar 

  32. Vasiliadis HM, Lesage A, Latimer E, Seguin M. Implementing suicide prevention programs: costs and potential life years saved in Canada. The Journal of Mental Health Policy and Economics. 2015;18(3):147–55.

    PubMed  Google Scholar 

  33. Wright D, Austin SB, Noh H, Jiang Y, Sonneville K. The cost-effectiveness of school-based eating disorder screening. American Journal of Public Health. 2014;104:e1–9.

    Article  Google Scholar 

  34. Persson M, Wennberg L, Beckman L, Salmivalli C, Svensson M. The cost-effectiveness of the kiva antibullying program: results from a decision-analytic model. Prevention Sci. 2018;19(6):728–37.

    Article  Google Scholar 

  35. Beckman L, Svensson M. The cost-effectiveness of the Olweus bullying prevention program: results from a modelling study. Journal of Adolescence. 2015;45:127–37.

    Article  PubMed  Google Scholar 

  36. Kass AE, Balantekin KN, Fitzsimmons-Craft EE, Jacobi C, Wilfley DE, Taylor CB. The economic case for digital interventions for eating disorders among United States college students. The International Journal of Eating Disorders. 2017;50(3):250–8.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Ssegonja R, Sampaio F, Alaie I, Philipson A, Hagberg L, Murray K, Sarkadi A, Langenskiöld S, Jonsson U, Feldman I. Cost-effectiveness of an indicated preventive intervention for depression in adolescents: a model to support decision making. J Affect Disord. 2020;277:789-99.

  38. Martínez-Alés G, Cruz Rodríguez JB, Lázaro P, Domingo-Relloso A, Barrigón ML, Angora R, et al. Cost-effectiveness of a contact intervention and a psychotherapeutic program for post-discharge suicide prevention. Canadian Journal of Psychiatry. 2021;66(8):737–46.

    Article  PubMed  Google Scholar 

  39. Simon E, Dirksen CD, Bögels SM. An explorative cost-effectiveness analysis of school-based screening for child anxiety using a decision analytic model. European Child and Adolescent Psychiatry. 2013;22(10):619–30.

    Article  PubMed  Google Scholar 

  40. Lintvedt OK, Griffiths KM, Eisemann M, Waterloo K. Evaluating the translation process of an Internet-based self-help intervention for prevention of depression: a cost-effectiveness analysis. Journal of Medical Internet Research. 2013;15(1):18.

    Article  Google Scholar 

  41. Paulden M, Palmer S, Hewitt C, Gilbody S. Screening for postnatal depression in primary care: Cost effectiveness analysis. BMJ. 2010.

  42. Premji S, McDonald SW, McNeil DA, Spackman E. Maximizing maternal health and value for money in postpartum depression screening: a cost-effectiveness analysis using the All Our Families cohort and administrative data in Alberta, Canada. J Affect Disord. 2021;281:839-46.

  43. Mallender J, Venkatachalam M, Onwude O, Jhita T. Economic analysis of interventions to reduce incidence and harm of domestic violence. London: National Institute for Health and Care Excellence; 2013.

    Google Scholar 

  44. Hunter RM, Nazareth I, Morris S, King M. Modelling the cost-effectiveness of preventing major depression in general practice patients. Psychol Med. 2014;44(7):1381-90.

  45. Lee YY, Barendregt JJ, Stockings EA, Ferrari AJ, Whiteford HA, Patton GA, et al. The population cost-effectiveness of delivering universal and indicated school-based interventions to prevent the onset of major depression among youth in Australia. Epidemiol Psychiatr Sci. 2017;26(5):545–64.

    Article  CAS  PubMed  Google Scholar 

  46. Mihalopoulos C, Vos T, Pirkis J, Smit F, Carter R. Do indicated preventive interventions for depression represent good value for money? The Australian and New Zealand Journal of Psychiatry. 2011;45(1):36–44.

    Article  PubMed  Google Scholar 

  47. Lokkerbol J, Adema D, Cuijpers P, Reynolds III CF, Schulz R, Weehuizen R, Smit F. Improving the cost-effectiveness of a healthcare system for depressive disorders by implementing telemedicine: a health economic modeling study. Am J Geriatr Psychiatr. 2014;22(3):253-62.

  48. van den Berg M, Smit F, Vos T, van Baal PH. Cost-effectiveness of opportunistic screening and minimal contact psychotherapy to prevent depression in primary care patients. PLoS One. 2011;6(8):e22884.

  49. Goetzel RZ, Tabrizi M, Henke RM, Benevent R, Brockbank CV, Stinson K, et al. Estimating the return on investment from a health risk management program offered to small Colorado-based employers. Journal of Occupational and Environmental Medicine. 2014;56(5):554–60.

    Article  PubMed  PubMed Central  Google Scholar 

  50. Jiao B, Rosen Z, Bellanger M, Belkin G, Muennig P. The cost-effectiveness of PHQ screening and collaborative care for depression in New York City. PLoS One1. 2017;12(8):e0184210.

    Article  CAS  Google Scholar 

  51. Feldman I, Fridman M. Mathematical evaluation model of cost-effectiveness due to indicative intervention for adolescent depression. Современная математика и концепции инновационного математического образования. 2020;7(1):157-65.

  52. Le LK, Barendregt JJ, Hay P, Sawyer SM, Paxton SJ, Mihalopoulos C. The modelled cost-effectiveness of cognitive dissonance for the prevention of anorexia nervosa and bulimia nervosa in adolescent girls in Australia. The International Journal of Eating Disorders. 2017;50(7):834–41.

    Article  PubMed  Google Scholar 

  53. Wang L, Nichols L, Austin SB. The economic effect of planet health on preventing Bulimia Nervosa. Archives of Pediatrics and Adolescent Medicine. 2011;165:756–62.

    Article  PubMed  Google Scholar 

  54. Mihalopoulos C, Vos T, Rapee RM, Pirkis J, Chatterton ML, Lee YC, et al. The population cost-effectiveness of a parenting intervention designed to prevent anxiety disorders in children. Journal of Child Psychology and Psychiatry and Allied Disciplines. 2015;56(9):1026–33.

    Article  PubMed  Google Scholar 

  55. Ophuis RH, Lokkerbol J, Hiligsmann M, Evers SMAA. Early intervention for subthreshold panic disorder in the Netherlands: A model-based economic evaluation from a societal perspective. PLoS One1. 2018;13(2):e0193338.

    Article  CAS  Google Scholar 

  56. Kumar S, Bell MJ, Juusola JL. Mobile and traditional cognitive behavioral therapy programs for generalized anxiety disorder: a cost-effectiveness analysis. PLoS One1. 2018;13(1):e0190554.

    Article  CAS  Google Scholar 

  57. Nystrand C, Hultkrantz L, Vimefall E, Feldman I. Economic return on investment of parent training programmes for the prevention of child externalising behaviour problems. Administration and Policy in Mental Health. 2020;47(2):300–15.

    Article  PubMed  Google Scholar 

  58. Nystrand C, Feldman I, Enebrink P, Sampaio F. Cost-effectiveness analysis of parenting interventions for the prevention of behaviour problems in children. PLoS One1. 2019;14(12):e0225503.

    Article  CAS  Google Scholar 

  59. Lebenbaum M, Cheng J, de Oliveira C, Kurdyak P, Zaheer J, Hancock-Howard R. Evaluating the cost effectiveness of a suicide prevention campaign implemented in Ontario, Canada. Applied Health Economics and Health Policy. 2020;18(2):189–201.

    Article  PubMed  Google Scholar 

  60. Pil L, Pauwels K, Muijzers E, Portzky G, Annemans L. Cost-effectiveness of a helpline for suicide prevention. US: Sage Publications; 2013. p. 273–81.

    Google Scholar 

  61. Comans T, Visser V, Scuffham P. Cost effectiveness of a community-based crisis intervention program for people bereaved by suicide. Crisis. 2013;34(6):390–7.

    Article  PubMed  Google Scholar 

  62. Godoy Garraza L, Peart Boyce S, Walrath C, Goldston DB, McKeon R. An economic evaluation of the Garrett Lee Smith memorial suicide prevention program. Suicide and Lifethreatening Behavior. 2018;48(1):3–11.

    Article  Google Scholar 

  63. Atkins Whitmer D, Woods DL. Analysis of the cost effectiveness of a suicide barrier on the golden gate bridge. Crisis. 2013;34(2):98–106.

    Article  PubMed  Google Scholar 

  64. Kinchin I, Russell AMT, Petrie D, Mifsud A, Manning L, Doran CM. Program evaluation and decision analytic modelling of universal suicide prevention training (safeTALK) in secondary schools. Applied Health Economics and Health Policy. 2020;18(2):311–24.

    Article  PubMed  Google Scholar 

  65. Richardson JS, Mark TL, McKeon R. The return on investment of postdischarge follow-up calls for suicidal ideation or deliberate self-harm. Psychiatric Services (Washington, D C). 2014;65(8):1012–9.

    Article  Google Scholar 

  66. Huitsing G, Barends SI, Lokkerbol J. Cost-benefit Analysis of the KiVa Anti-bullying Program in the Netherlands. Int J Bullying Prev. 2020;2(3):215-24.

  67. Devine A, Spencer A, Eldridge S, Norman R, Feder G. Cost-effectiveness of Identification and Referral to Improve Safety (IRIS), a domestic violence training and support programme for primary care: a modelling study based on a randomised controlled trial. British Medical Journal Open. 2012;2(3):e001008.

    Google Scholar 

  68. Norman R, Spencer A, Eldridge S, Feder G. Cost-effectiveness of a programme to detect and provide better care for female victims of intimate partner violence. Journal of Health Services Research & Policy. 2010;15(3):143–9.

    Article  Google Scholar 

  69. Barbosa EC, Verhoef TI, Morris S, Solmi F, Johnson M, Sohal A, et al. Cost-effectiveness of a domestic violence and abuse training and support programme in primary care in the real world: updated modelling based on an MRC phase IV observational pragmatic implementation study. British Medical Journal Open. 2018;8(8).

    Google Scholar 

  70. Dopp AR, Schaeffer CM, Swenson CC, Powell JS. Economic impact of multisystemic therapy for child abuse and neglect. Administration and Policy in Mental Health. 2018;45(6):876–87.

    Article  PubMed  PubMed Central  Google Scholar 

  71. Peterson C, Florence C, Thomas R, Klevens J. Cost-benefit analysis of two child abuse and neglect primary prevention programs for US States. Prevention Science. 2018;19(6):705–15.

    Article  PubMed  PubMed Central  Google Scholar 

  72. Kuklinski MR, Oxford ML, Spieker SJ, Lohr MJ, Fleming CB. Benefit-cost analysis of Promoting First Relationships (R): Implications of victim benefits assumptions for return on investment. Child Abuse & Neglect. 2020;106:104515.

    Article  Google Scholar 

  73. Mihalopoulos C, Vos T, Pirkis J, Carter R. The population cost-effectiveness of interventions designed to prevent childhood depression. Pediatrics. 2012;129(3):e723-30.

  74. Vemer P, Corro Ramos I, van Voorn GAK, Al MJ, Feenstra TL. AdViSHE: A validation-assessment tool of health-economic models for decision makers and model users. PharmacoEconomics. 2016;34(4):349–61.

    Article  CAS  PubMed  Google Scholar 

  75. Metrics IfH. Evaluation. GBD compare data visualization. 2017.

    Google Scholar 

  76. Jaime Caro J, Eddy DM, Kan H, Kaltz C, Patel B, Eldessouki R, et al. Questionnaire to assess relevance and credibility of modeling studies for informing health care decision making: an ISPOR-AMCP-NPC good practice task force report. Value Health. 2014;17(2):174–82.

    Article  CAS  PubMed  Google Scholar 

  77. Schmidt M, Werbrouck A, Verhaeghe N, Putman K, Simoens S, Annemans L. Universal mental health interventions for children and adolescents: a systematic review of health economic evaluations. Applied Health Economics and Health Policy. 2020;18(2):155–75.

    Article  PubMed  Google Scholar 

  78. Feldman I, Gebreslassie M, Sampaio F, Nystrand C, Ssegonja R. Economic evaluations of public health interventions to improve mental health and prevent suicidal thoughts and behaviours: a systematic literature review. Administration and Policy in Mental Health. 2021;48(2):299–315.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

This review is conducted within a research project funded by Vietnam National Foundation for Science and Technology Development (NAFOSTED)—Australian National Health and Medical Research Council Joint Call for Collaborative Research Projects (NHMRC.108.01-2018.02).

Funding

Not applicable.

Author information

Authors and Affiliations

  1. Department of Health Policy and Economics, Hanoi University of Public Health, Hanoi, Vietnam

    Nguyen Thu Ha & Nguyen Quynh Anh

  2. Department of Health Education and Promotion, Hanoi University of Public Health, Hanoi, Vietnam

    Nguyen Thanh Huong

  3. Hanoi University of Public Health, Hanoi, Vietnam

    Vu Nguyen Anh

Authors
  1. Nguyen Thu Ha
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nguyen Thanh Huong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Vu Nguyen Anh
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nguyen Quynh Anh
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

NTH, NQA, NTHg developed the protocol and completed PROSPERO trial registration. NTH and VNA performed the electronic and manual search; and initial title and abstract screening. NTH, NQA, NTHg performed the full-text screening, extracted the data from the included studies and applied the Philips Checklist. All authors contributed to the analysis of the results and the writing of the manuscripts. The author(s) read and approved the final manuscript.

Corresponding author

Correspondence to Nguyen Thanh Huong.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: Table S1.

The PRISMA 2020 Checklist. Table S2. Search Strategy. Table S3. Reasons for Fultext Exclusion. Table S4. Summary of cost-effectiveness results in the included studies.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ha, N.T., Huong, N.T., Anh, V.N. et al. Modelling in economic evaluation of mental health prevention: current status and quality of studies. BMC Health Serv Res 22, 906 (2022). https://doi.org/10.1186/s12913-022-08206-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12913-022-08206-9

Keywords

BMC Health Services Research

ISSN: 1472-6963

Contact us