Skip to main content

Meta-analysis of economic evaluation studies: data harmonisation and methodological issues



In the context of ever-growing health expenditure and limited resources, economic evaluations aid in making evidence-informed policy decisions. Cost-utility analysis (CUA) is often used, and CUA data synthesis is also desirable, but methodological issues are challenged. Hence, we aim to provide a step-by-step process to prepare the CUA data for meta-analysis.


Data harmonisation methods were constructed specifically considering CUA methodology, including inconsistent reports, economic parameters, heterogeneity (i.e., country’s income, time horizon, perspective, modelling approaches, currency, willingness to pay). An incremental net benefit (INB) and its variance were estimated and pooled across studies using a basic meta-analysis by COMER.


Five scenarios show how to obtain INB and variance with various reported data: Study reports the mean and variance (Scenario 1) or 95% confidence interval (Scenario 2) of ΔC, ΔE, and ICER for INB/variance calculations. Scenario 3: ΔC, ΔE, and variances are available, but not for the ICER; a Monte Carlo was used to simulate ΔC and ΔE data, variance and covariance can be then estimated leading INB calculation. Scenario-4: Only the CE plane was available, ΔC and ΔE data can be extracted; means of ΔC, ΔE, and variance/covariance can be estimated accordingly, leading to INB/variance estimates. Scenario-5: Only mean cost/outcomes and ICER are available but not for variance and the CE-plane. A variance INB can be borrowed from other studies which are similar characteristics, including country income, ICERs, intervention-comparator, time period, country region, and model type and inputs (i.e., discounting, time horizon).


Out data harmonisation and meta-analytic methods should be useful for researchers for the synthesis of economic evidence to aid policymakers in decision making.

Peer Review reports


In the context of ever-growing health expenditure and limited resources, identifying healthcare services that yield the highest benefit at the lowest cost is a priority. Economic evaluation studies (EES) provide a framework to systematize both clinical and economic outcomes [1] Cost-utility analysis (CUA) is commonly applied to compare clinical and economic outcomes by estimating an incremental cost-effectiveness ratio (ICER). The costs are usually measured in a specific country currency, while the health benefit is usually measured as a quality adjusted life year (QALY), i.e., the product of years lived and health utility score ranging from 0 (death) to 1 (perfect health), or disability adjusted life years (DALY) [2, 3]. The ICER, (Costintervention—Costcomparator)/(QALYintervention—QALYcomparator), is under the willingness to pay (WTP) threshold (measured in monetary cost per QALY gained), the health intervention is considered to be cost-effective[4]. The guidelines from Joanna Briggs Institute, Cochrane group [5, 6], mainly provide guidelines towards qualitative synthesis or only systematic review of all sorts of economic evaluation (e.g., cost–benefit analysis, cost minimisation analysis, cost-effective analysis, cost-utility analysis) [7,8,9,10,11]. However, these guidelines have a limited focus on data extraction and data harmonisation process to prepare the data for the meta-analysis [7,8,9,10,11].

Further, many methodological issues in the data synthesis of EESs are more challenging than clinical studies because there are many sources of heterogeneity, including study characteristics (e.g., setting, WTP, country, country income), methodology (time horizon, perspective, data source, model type, input parameters, and assumptions) [8]. This is perhaps why most previous systematic reviews of EESs have performed only descriptive analyses and reported only qualitative findings without applying a meta-analysis (MA) to estimate pooled effect measures.

Although Crespo et al. [8] have described a MA for pooling EES (known as the COMparative Efficiency Research, COMER), it has yet been widely adopted as such MA for clinical outcomes. This might be due to EESs being too heterogeneous to pool or choosing the lesser-known parameter “incremental net benefit” (INB) as the effect measure rather than the more commonly used ICER. However, we believe the choice for pooling INB was justified due to the limitations of the ICER [12]. For instance, a negative ICER may indicate a lower cost compared with higher effectiveness or higher costs along with lower effectiveness of interventions, thus introducing ambiguity in interpretation [8, 13]. In contrast, positive and negative INBs directly indicate cost-effectiveness and non-cost-effectiveness of interventions, respectively, which is the information required by policymakers [14, 15].

Furthermore, the COMER method mainly focused on the statistical methods for pooling but did not suggest a detailed step-by-step method of data extraction and data harmonisation as for various styles and suboptimal quality for reporting of economic evaluations [16]. In addition, assessing heterogeneity and exploring the source of it and publication bias have not been described. Therefore, our primary focus in this manuscript is to provide a methodological approach for meta-analysis of cost-utility studies; we have specifically detailed the step-by-step process to extract data from “Cost-utility studies” and to make it ready to be taken for meta-analysis. Data for the cost-effectiveness of diabetic drug controls are used as a demonstration.


Basic methods of MA in EESs, including identifying and selecting relevant studies, are similar to other systematic reviews and MAs [5, 17] and should follow the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [18] guideline when reported. This methodological study was a part of previous MAs, in which some additional specific issues apply to EESs are as follows [19,20,21,22]; the relevant protocol was registered in Prospero (PROSPERO 2018 CRD42018105193).

Step 1: Data extraction

All relevant data for comparative EESs (e.g., CUA) should be extracted as follows utilising the Population, Intervention, Comparator, Outcome, and Study type (PICOS) framework:

  • General characteristics of EESs including study setting/country, study design (e.g., CUA with model-based, primary CUA alongside RCT/cohort), study perspective, time horizon, discount rate for cost and utility, currency/currency-year, type of EESs [e.g., CUA or cost-effectiveness analysis (CEA)], willingness to pay [WTP; country-specific or gross domestic products (GDP)-base] or country-level cost-effectiveness threshold [23] where appropriate if WTP was not available, and type of economic models.

  • Characteristics of patients (P) including indication for treatment, sample size, type of patients (e.g., children/adult, general/specific disease, etc.), mean age, percent male/female, mean body mass index (BMI), etc.

  • Type of interventions and comparators (I & C) (with the duration of treatment/dosage/day, etc.)

  • Data needed to estimate INB and its variance (O); this includes currency and year, source of cost data (actual data cost collected from patients, central hospital/country costs, etc.), type of cost (e.g., direct/non-direct medical cost, indirect medical cost), and effectiveness outcomes (e.g., life year, QALY, improvement, success, adverse events etc.).

Specific data required for pooling include costs or incremental cost (ΔC), and incremental effectiveness (ΔE) along with their standard deviation (SD), standard error (SE), or 95% confidence interval (CI) along with covariance between ΔC and ΔE. Some studies may report ICER along with probabilistic sensitivity analysis (PSA). To calculate the INB and its variance, mean and variance of the costs and effectiveness of interventions and comparators along with WTP thresholds are required. In the model-based CUA, studies usually report point estimates of deterministic and/or probabilistic costs and outcomes. We suggest using primarily the measures of central tendency and dispersion measures from PSA results for pooling, as it could better represent a real-life situation considering the distribution of all input variables. Further, to conduct sensitivity analyses using point estimates from the deterministic analysis to see the robustness of results. The WTP threshold was initiated by the Commission on Macroeconomics and Health in 2002 by the World Health Organization CHOosing Interventions that are Cost-Effective (WHO CHOICE) [24]. The WTP threshold in each country usually refers to the standard country guideline based on a fixed value or per capita GDP with returns on investments in health to define whether a health intervention would be (very) cost-effective. [25, 26]. We suggest using the same WTP threshold in monetary units used in the study with further adjustment as per currency conversions as mentioned below. If studies have not reported WTP, one per capita GDP of that study’s country and year can be used as WTP along with a sensitivity analysis of three times per capita GDP.

We strongly recommend constructing data extraction forms in advance, a pilot should be performed to make sure that the forms work well and contain all important data specific to that topic.

Step 2: Data harmonisation

Currency conversions

We need to standardize money units usually reported in different currencies (i.e., US $, €, £, ¥) and years by converting to purchasing power parity (PPP) adjusted to US$ for the latest year of analysis[8]. For example, if a study reported cost, ICER, and thresholds in Euros for 2012 and we plan to pool for the current year (e.g., 2022), this currency is firstly converted to 2022 Euros using the historical consumer price index (CPI) of that country (IMF database: Then, the Euro 2022 value to be converted to PPP adjusted US$ rate using conversion rates from the International Monetary Fund [27]. In addition, GDP-based WTP threshold (K) values also need to be corrected for the current CPI 2022 year and PPP; however, standard/country-specific or fixed WTP values only need PPP correction. The variance is calculated as follows:


Estimation of INB and its variance

After currency conversions for cost and K, the INB can be further estimated as follows [8]:

$$INB=Kx\Delta E-\Delta C$$


$$INB=\Delta E(K-ICER)$$

Where K is the WTP, and ΔC and ΔE are incremental cost and incremental effectiveness, respectively.

A positive INB favours treatment, i.e., intervention is cost-effective, whereas a negative INB favours the comparator, i.e., intervention is not cost-effective [8, 14, 15].

The variance of INB [8] can be estimated as follows:

$$Var\left(INB\right)={K}^{2}{\sigma }_{\Delta E}^{2}+{\sigma }_{\Delta C}^{2}-2K{\sigma }_{\Delta E\Delta C}$$


$$Var(INB)={K}^{2}{\sigma }_{\Delta E}^{2}+{\sigma }_{ICER}^{2}$$

Where \({\sigma }_{\Delta C}^{2}, {\sigma }_{\Delta E}^{2}, {\upsigma }_{\Delta E\Delta C}\) are variances of ΔC and ΔE and their covariance, and \({\sigma }_{ICER}^{2}\) is variance of ICER. However, economic studies can report many different parameters; the five scenarios below, along with a flow in Fig. 1, shows how to obtain INB and variance starting with different reported data [28].

Fig. 1
figure 1

Selection flow chart of scenarios


The primary EES ideally reports the point estimates and variances for every parameter required for the calculation of INB and its variance. The INB can be calculated accordingly to Eqs. (2) to (5).


The study reports the means and measures of dispersion (95% CIs) of incremental costs & outcomes and ICER. The variance of the ICER can be calculated using the following formulas:

$$\begin{array}{c}95\mathrm{\%}CI of {\mu }_{ICER}={\widehat{\mu }}_{ICER}\pm {Z}_{a/2}xSE\\ {UL}_{ICER}={\widehat{\mu }}_{ICER}+{Z}_{a/2}xSE\\ \begin{array}{c}SE=\frac{{UL}_{ICER}-{\widehat{\mu }}_{ICER}}{{Z}_{a/2}}\\ \begin{array}{c}{\widehat{\sigma }}_{ICER}^{2}={SE}^{2}\\ {UL}_{ICER}=\mathrm{Upper limit of ICER}\\ \begin{array}{c}{Z}_{a/2}=\mathrm{Standardize normal}=1.96\mathrm{ if alpha}=0.05\\ {\widehat{\mu }}_{ICER}=\mathrm{mean ICER}\end{array}\end{array}\end{array}\end{array}$$

Once we know the variance of the ICER, the variance of the INB can be estimated using Eq. (5).


The study reports means, and 95% CI, SD/SE of costs/outcomes, or ΔC/ΔE, but does not provide the ICER or its variance. Data for Costs/ΔC and QALYs/ΔE are then used to simulate Costs/ΔC and QALYs/ΔE with 1000 replications using Monte Carlo methods with gamma and normal distributions for Costs/ΔC and QALYs/ΔE, respectively. We suggest simulating costs and QALYs of intervention and comparator for 1000 replications separately; this then leads to calculating the ΔC, ΔE, and covariance between ΔC & ΔE. Data for ΔC and ΔE are assumed to be gamma and normal distributions[29, 30].A sensitivity analysis can be performed using different distributions (e.g., log-normal, exponential for both costs and effective, etc.) to see the robustness of pooling results. The covariance (\({\upsigma }_{\Delta E\Delta C}\)) between ΔC and ΔE as well as \({\widehat{\sigma }}_{\Delta C}^{2}\) & \({\widehat{\sigma }}_{\Delta E}^{2}\) can be then estimated. If the 95% CI is provided, this is converted to SE using Eq. (6) above and used to simulate data. The INB and its variance can be further calculated using Eqs. (2) and (5).


The study does not report any dispersion but does provide the CE plane graphs, a scatter plot of ΔC on the Y-axis and ΔE on the X-axis, in which individual values of ΔC and ΔE data can be manually extracted from the CE plane using Web-Plot-Digitizer software[31]. Then, means of ΔC, ΔE, and their variances and covariances can be estimated accordingly. Finally, the INB and its variance can be estimated using Eqs. (2) and (5).


The study reports neither any dispersion nor the CE-plane graph but only provides the deterministic analysis means (or point estimates) of costs, outcomes, and ICER. In such situations, the measures of dispersions can be borrowed from another similar study if they fulfill the following criteria:

  • They are in the same stratum of country income,

  • Their ICERs are not much different, e.g., ± 50% to 75%

  • They are similar in intervention, comparator, time period, country region

  • Similar model type and inputs (i.e., discounting, time horizon).

If there is more than one study that meets the criteria, the average of the variances of those studies can be used.

Step 3: Pooling INB

When pooling INBs from many studies, we strongly recommend stratifying by the level of country income, model type, time horizon, and perspective in order to reduce heterogeneity. The country income should be classified as low (LIC), lower-middle (LMIC), upper-middle (UMIC), and high (HIC) as per the World Bank classification8. Economic models can include Markov, decision tree, discrete event simulation, or others. Study perspectives should include societal, third-party payer, and patient perspectives. Time horizon should be lifetime (e.g., ≥ 20 or 30 years depending on the disease context) and non-lifetime (e.g., < 5, < 10-years, etc.).

The INB can be pooled across studies using a fixed-effect or a random-effect model depending on the degree of heterogeneity [5, 8, 19, 20, 28].

  1. A)

    Fixed-effects model

  2. B)

    Random-effects model

    $${w}_{i}^{*}=\frac{1}{Var({INB}_{i})+{\tau }^{2}}$$
    $${\tau }^{2}=\frac{Q-(S-1)}{\sum {w}_{i}-\frac{\sum {w}_{i}^{2}}{\sum {w}_{i}}}$$

where Q is the Cochrane Q test, which has Chi-square distribution; S is a number of included studies/comparisons; τ2 is a between-study variation.

Similar to MA in other areas, heterogeneity needs to be assessed before pooling INB. Heterogeneity can be visualized by inspection of the forest plot and quantitated using the Cochrane-Q test and the I2 statistic[5].

$${\mathrm{I}}^{2}=100\mathrm{\%x }\frac{\mathrm{Q}-(\mathrm{S}-1)}{\mathrm{Q}}$$

If heterogeneity is present, i.e., the I2 ≥ 25% or p-value of Q test is less than 0.1; the INBs can be pooled using a random-effects model; otherwise, a fixed-effect model can be applied [8, 19, 20, 28]. Exploring source/s of heterogeneity is strongly recommended. This can be done using a meta-regression to fit each potential source (e.g., time horizon, percent discount rate, threshold values, source of effectiveness measure, risk of bias, economic structure, etc.) one-by-one [8, 19, 20, 28]. If that potential factor explains some proportion of the heterogeneity, including it in the meta-regression model should reduce the I2 accordingly. There are no established criteria for how much I2 should be decreased to consider that factor as a significant source of heterogeneity. In our experience, if the I2 is reduced by about 50% or more from the baseline model (i.e., the model without any factor), such factor/s may be source/s of heterogeneity. A post-hoc subgroup analysis by that factor should be performed accordingly. In addition, sensitivity analyses excluding a few studies with very different characteristics compared to the rest can be used to see if heterogeneity of INBs can be reduced.

Similar to general MA, publication bias should be assessed using a funnel plot and Egger's test. A funnel plot graphs INB estimates on the x-axis against their precision on the y-axis. If all studies are estimating the same true INB, their INBs should be randomly scattered around the true value and form a funnel shape. Egger’s test formally tests if the funnel is symmetrical; if this is significant, it usually indicates that there is heterogeneity or missing studies (publication bias) or both. A contour-enhanced funnel plot is further recommended [32]. This plot will contour the area of the funnel into non-significant (P-value > 0.05- < 0.1) and significant areas (P-value < 0.01 and < 0.05), which will help to differentiate the cause of the asymmetry. For instance, if missing studies fall into the non-significant area, asymmetry might be due to missing studies or publication bias. Conversely, if missing studies are in significant areas, heterogeneity is more likely to be the explanation.


We used data from a MA of CUA of glucagon-like peptide 1 agonists (GLP1) for treatment of type 2 diabetic (T2D) patients who failed to achieve control with metformin monotherapy [19]. A total of 56 studies with 82 comparisons were eligible for pooling INBs. We included comparisons of GLP-1 and dipeptidyl peptidase-4 inhibitors (DPP4i) (N = 10); study characteristics are described in Table 1. All studies were from HICs and used country-specific WTP threshold; 9/10 studies used the third-party payer perspective, and 7/10 used a lifetime horizon.

Table 1 Selected studies and its analysis scenario

In terms of preparing the data for pooling, 7, 1, and 2 studies provided data matching scenarios 3, 4, and 5, respectively (Table 1). Data for mean cost, QALY, and their incremental values are described in Table 2. Costs and WTP thresholds from each study were converted to $US currency using PPP adjusted for the year 2019 using formula (1).

Table 2 Descriptive of the mean cost and QALY along with their incremental data of comparison between GLP1a vs DPP4i

For the seven studies matching scenario 3 (where mean C and E data along with SDs were reported) (Table 1 and Supl Table 1), the Monte-Carlo method was used to simulate 1000 replicated data based on gamma and normal distributions for cost and QALY data, respectively. Then, ΔC and ΔE along with variance and covariance (\({\upsigma }_{\Delta E\Delta C})\)) were calculated. The INB and variance were then calculated following formulas (2) and (4).

The study matching scenario 4 provided CE-plane graphs (Table 1). Data for ΔC and ΔE were directly extracted from the CE plane using Web-Plot-Digitizer [31]. Then, variance and covariance (\({\upsigma }_{\Delta E\Delta C}\)) were calculated, leading to estimation of the INB and its variance using formulas (2) and (4).

For the two studies [33, 35] matching scenario 5, the INB variance was adopted from other studies following the steps outlined above. Of the ten included studies, two other studies [36, 40] were conducted in the USA. For the selection of variance values for the Guillermin et al. study[35], the study period, time-horizon, study perspective, ICER values, drug comparison (Sitagliptin) were most similar to Lee et al. [36] (Table 1 and Table 2). Hence, the INB variance value of the latter was used to estimate the former. The values for Lee et al. also matched the second study [33] most closely.

INB data along with variances are shown in Table 3. The forest plot was constructed by plotting point estimated INBs along with 95% CIs for individual studies (see Fig. 2a); the intervention is cost-effective if the estimated INB falls in the right side of a vertical line of zero and it is not cost-effective if it lines in the left side. These INBs were then pooled across studies using a fixed-effects (inverse variance) model yielding a pooled INB (95% CI) of US$ 4012.21 (-571.43, 8595.85) with I2 of 0% (see Fig. 2a), this corresponded to individual 95% CIs of INBs which are very much overlapped indicating less likely to be heterogeneous. In the presence of heterogeneity, as indicated by I2 ≥ 25% or Cochrane-Q p < 0.1, a random-effects model (DerSimonian and Laird model) could be used [43]. The pooled INB value is positive but its 95% CI covers 0, i.e., GLP1 agonists might be cost-effective as compared to DPP4 inhibitors but the results did not reach statistical significance.

Table 3 Describe incremental net benefit comparing GLP1i with DPP4i along with variance
Fig. 2
figure 2

a Forest plot of pooling INBs of GLP1 vs. DPP4i; b Funnel plot of pooling INB of GLP1 vs. DPP4i

The robustness of the pooled INB, as well as heterogeneity, can be assessed using various sensitivity and subgroup analyses. Sensitivity analyses omitting the study that used a societal perspective [38] and the study that did not use discounting [40] yielded a pooled INB of US$ 4,032.07 (US$ − 554.48, US$8,618.61) and US$4,068.19 (US$ − 650.66, US$8787.04), respectively.

The WTP threshold used for these comparisons ranged from US$ 29,382 to US$ 58,024, with a median of US $49,325. Subgroup analyses by median WTP threshold (< vs ≥ US $49,325), time horizon, and source of effectiveness measure were performed (see Table 4), indicating GLP1s were not significantly cost-effective compared with DPP4i in any subgroup. In all these sensitivity and subgroup analyses, the results were similar to the overall pooled INB, indicating that the results are robust.

Table 4 The subgroup analysis results of pooing INB between the GLP1 agonists and DPP4 inhibitors

As in general MA, publication bias was assessed using a funnel plot and Egger’s test. There was no evidence of publication bias, seen either by asymmetry on the funnel plot (Fig. 2b) or an Egger’s test (coefficient = 0.32, SE = 0.73, p = 0.672).


We have extended the COMER MA methods for EESs, focusing on data harmonisation; methodological issues include currency, time, discount, perspective, time horizon, and model used to aid in applying a MA for evidence synthesis in EESs. INB and its variance are estimated based on five scenarios. MA is then applied to pool INBs across studies providing a summary estimated CE of treatment relative to control. This evidence should be useful for policymakers in making decisions regarding reimbursement of treatments to a population in countries where resources are limited.

Despite the existence of several guidelines for reporting EESs, studies still vary in how they report the results[44]. This data harmonisation process reported here under the five scenarios can help prepare data to calculate and pool INB values. The different monetary units and year can all be converted to a common standard currency.

We used INB instead of ICER as the economic effect measure following COMER methods because of limitations of the ICER in the ambiguity of interpretation for negative ICER as mentioned above [8, 13]. On the other hand, positive INBs indicate cost-effectiveness, while negative INBs show non-cost-effectiveness. This information will be required by policymakers [14, 15] in making a decision from both resource-rich and resource-poor countries.

A few challenges should be highlighted when applying a MA in EESs. First, EEs are heterogeneous, which can be caused by model type, population, country income, GDP, perspective, time horizon, and discount rate. We applied the CPI and PPP to harmonise different economic backgrounds as well as the time-lag across the studies [45, 46]. However, it should be noted that using CPI and/or PPP may have some limitations as for the estimation method of price indices, which are calculated from individual prices of only selected commodities rather than all commodities in each country [47]. Considering not only country income but also model type, time horizon, and perspective in stratified analyses may also reduce heterogeneity, if there are sufficient data for stratifying. Furthermore, sub-group and/or sensitivity analyses should be performed to identify specific types of studies/country income where treatments show cost-effectiveness. Economic factors should be considered for subgroups, including WTP, discount rate, type of EES (e.g., within-trial EES versus model-based EES), quality of EESs or risk of bias, the structure of economic model, type of health state, and percent herd immunity for the vaccine, etc.). Different subgroups of these factors may result in different cost-effectiveness findings within HICs and UMICs.

Second, the health EESs are context-specific, usually conducted in individual country settings. However, not all countries have EESs that fit their context because conducting well-designed EESs is very resource-intensive and requires specialised expertise in economic evaluation. Therefore, there will be an even greater need for some systematic synthesis of evidence where resources are limited. Evidence from a MA of EESs will be useful if it is performed with sensitivity to country contexts (e.g., country income, type of model, lifetime, perspective, etc.).

In conclusion, we have described a tutorial of MA in EESs by applying the general methods of MA, additional with specific issues for EESs. The step-by-step approach of data harmonization is demonstrated for facilitating the process of MA. Although evidence of CE is context-specific for each country, conducting such specific individual study is challenging as similar to CE studies due to various practical limitations (e.g., trained manpower, time, resources, etc.). Thus, the MA of EESs should be encouraged; evidence synthesis would be of immense value for the policy decision-making process as well as aid in the comparability of such evidences across countries with similar contexts.

Availability of data and materials

All the relevant data is included in the manuscript.



Incremental cost


Incremental effectiveness


Body mass index


Cost effectiveness


Cost effectiveness plane


Confidence interval


Consumer price index


Cost effectiveness analysis


COMparative Efficiency Research


Cost utility study


Disability adjusted life year


Dipeptidyl peptidase 4 inhibitors


Economic evaluation studies


Gross domestic product


Glucagon-like peptide-1


High income countries


Incremental cost effectiveness ratio


Incremental net benefit


Low income countries


Meta analysis


Purchasing power parity


Preferred Reporting Items for Systematic Reviews and Meta-Analyses protocol


Probabilistic sensitivity analysis


Standard deviation


Standard error


Quality adjusted life years


Upper middle-income countries


United states


World health organisation


Willingness to pay


  1. Drummond MF, Sculpher MJ, Torrance GW, O’Brien BJ, Stoddart GL. Methods for the Economic Evaluation of Health Care Programmes, 3. edition. Oxford: Oxford University Press; 2005.

    Google Scholar 

  2. Torrance GW. Measurement of health state utilities for economic appraisal. J Health Econ. 1986;5(1):1–30.

    Article  CAS  Google Scholar 

  3. Wisloff T, Hagen G, Hamidi V, Movik E, Klemp M, Olsen JA. Estimating QALY gains in applied studies: a review of cost-utility analyses published in 2010. Pharmacoeconomics. 2014;32(4):367–75.

    Article  Google Scholar 

  4. Drummond M, Stoddart G, Labelle R, Cushman R. Health economics: an introduction for clinicians. Ann Intern Med. 1987;107(1):88–92.

    Article  CAS  Google Scholar 

  5. Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). Cochrane, 2021. Available from

  6. Gomersall JS, Jadotte YT, Xue Y, Lockwood S, Riddle D, Preda A. Conducting systematic reviews of economic evaluations. Int J Evid Based Healthc. 2015;13(3):170–8.

    Article  Google Scholar 

  7. Shemilt I, McDaid D, Marsh K, Henderson C, Bertranou E, Mallander J, Drummond M, Mugford M, Vale L. Issues in the incorporation of economic perspectives and evidence into Cochrane reviews. Syst Rev. 2013;2:83.

    Article  Google Scholar 

  8. Crespo C, Monleon A, Diaz W, Rios M. Comparative efficiency research (COMER): meta-analysis of cost-effectiveness studies. BMC Med Res Methodol. 2014;14:139.

    Article  Google Scholar 

  9. Rehfuess EA, Stratil JM, Scheel IB, Portela A, Norris SL, Baltussen R. The WHO-INTEGRATE evidence to decision framework version 1.0: integrating WHO norms and values and a complexity perspective. BMJ Glob Health. 2019;4(Suppl 1):e000844.

    Article  Google Scholar 

  10. Movsisyan A, Rehfuess E, Norris SL. When complexity matters: a step-by-step guide to incorporating a complexity perspective in guideline development for public health and health system interventions. BMC Med Res Methodol. 2020;20(1):245.

    Article  CAS  Google Scholar 

  11. Shields GE, Elvidge J. Challenges in synthesising cost-effectiveness estimates. Syst Rev. 2020;9(1):289.

    Article  Google Scholar 

  12. O’Mahony JF. The Limitations of Icers In Screening Interventions and The Relative Net Benefit Alternative. Value Health. 2015;18(7):A705.

    Article  Google Scholar 

  13. Hoch JS, Dewa CS. Advantages of the net benefit regression framework for economic evaluations of interventions in the workplace: a case study of the cost-effectiveness of a collaborative mental health care program for people receiving short-term disability benefits for psychiatric disorders. J Occup Environ Med. 2014;56(4):441–5.

    Article  Google Scholar 

  14. Willan AR. Incremental net benefit in the analysis of economic data from clinical trials, with application to the CADET-Hp trial. Eur J Gastroenterol Hepatol. 2004;16(6):543–9.

    Article  Google Scholar 

  15. Willan AR, Lin DY. Incremental net benefit in randomized clinical trials. Stat Med. 2001;20(11):1563–74.

    Article  CAS  Google Scholar 

  16. Miroshnychenko A, Uhlman K, Malone J, Waltho D, Thoma A. Systematic review of reporting quality of economic evaluations in plastic surgery based on the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement. J Plast Reconstr Aesthet Surg. 2021;74(10):2458–66.

    Article  Google Scholar 

  17. Normand SL. Meta-analysis: formulating, evaluating, combining, and reporting. Stat Med. 1999;18(3):321–59.

    Article  CAS  Google Scholar 

  18. Shamseer L, Moher D, Clarke M, Ghersi D, Liberati A, Petticrew M, Shekelle P, Stewart LA, Group P-P. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ. 2015;350:g7647.

    Article  Google Scholar 

  19. Bagepally BS, Chaikledkaew U, Gurav YK, Anothaisintawee T, Youngkong S, Chaiyakunapruk N, McEvoy M, Attia J, Thakkinstian A. Glucagon-like peptide 1 agonists for treatment of patients with type 2 diabetes who fail metformin monotherapy: systematic review and meta-analysis of economic evaluation studies. BMJ Open Diabetes Res Care. 2020;8(1):e001020.

  20. Bagepally BS, Gurav YK, Anothaisintawee T, Youngkong S, Chaikledkaew U, Thakkinstian A. Cost Utility of Sodium-Glucose Cotransporter 2 Inhibitors in the Treatment of Metformin Monotherapy Failed Type 2 Diabetes Patients: A Systematic Review and Meta-Analysis. Value Health. 2019;22(12):1458–69.

    Article  Google Scholar 

  21. Haider S, Chaikledkaew U, Thavorncharoensap M, Youngkong S, Islam MA, Thakkinstian A. Systematic review and meta-analysis of cost-effectiveness of rotavirus vaccine in low-income and lower-middle-income countries. Open Forum Infect Dis. 2019;6(4):ofz117.

    Article  PubMed  PubMed Central  Google Scholar 

  22. IVIR-AC. Meta-analysis of economic evaluations. Wkly Epidemiol Rec. 2021;96(17):138–40.

  23. Woods B, Revill P, Sculpher M, Claxton K. Country-Level Cost-Effectiveness Thresholds: Initial Estimates and the Need for Further Research. Value Health. 2016;19(8):929–35.

  24. Hutubessy R, Chisholm D, Edejer TT. Generalized cost-effectiveness analysis for national-level priority-setting in the health sector. Cost Eff Resour Alloc. 2003;1(1):8.

    Article  Google Scholar 

  25. Robinson LA, Hammitt JK, Chang AY, Resch S. Understanding and improving the one and three times GDP per capita cost-effectiveness thresholds. Health Policy Plan. 2016;32(1):141–5.

    Article  Google Scholar 

  26. Marseille E, Larson B, Kazi DS, Kahn JG, Rosen S. Thresholds for the cost–effectiveness of interventions: alternative approaches. Bull World Health Organ. 2015;93(2):118–24.

    Article  Google Scholar 

  27. IMF database. Available from:

  28. Haider S, Chaikledkaew U, Thavorncharoensap M, Youngkong S, Islam MA, Thakkinstian A. Systematic Review and Meta-Analysis of Cost-effectiveness of Rotavirus Vaccine in Low-Income and Lower-Middle-Income Countries. Open Forum Infect Dis. 2019 Mar 8;6(4):ofz117. PMID: 31049363; PMCID: PMC6488528.

  29. Polgreen LA, Brooks JM. Estimating incremental costs with skew: a cautionary note. Appl Health Econ Health Policy. 2012;10(5):319–29.

    Article  Google Scholar 

  30. Wang H, Zhao H. A study on confidence intervals for incremental cost-effectiveness ratios. Biom J. 2008;50(4):505–14.

    Article  Google Scholar 

  31. Rohatgi A. Web Plot Digitizer ver 4.5; 2021. Available from:

  32. Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry. J Clin Epidemiol. 2008;61(10):991–6.

    Article  Google Scholar 

  33. Sinha A, Rajan M, Hoerger T, Pogach L. Costs and consequences associated with newer medications for glycemic control in type 2 diabetes. Diabetes Care. 2010;33(4):695–700.

    Article  Google Scholar 

  34. Davies MJ, Chubb BD, Smith IC, Valentine WJ. Cost-utility analysis of liraglutide compared with sulphonylurea or sitagliptin, all as add-on to metformin monotherapy in Type 2 diabetes mellitus. Diabet Med. 2012;29(3):313–20.

    Article  CAS  Google Scholar 

  35. Guillermin AL, Lloyd A, Best JH, DeYoung MB, Samyshkin Y, Gaebler JA. Long-term cost-consequence analysis of exenatide once weekly vs sitagliptin or pioglitazone for the treatment of type 2 diabetes patients in the United States. J Med Econ. 2012;15(4):654–63.

    Article  Google Scholar 

  36. Lee WC, Samyshkin Y, Langer J, Palmer JL. Long-term clinical and economic outcomes associated with liraglutide versus sitagliptin therapy when added to metformin in the treatment of type 2 diabetes: a CORE Diabetes Model analysis. J Med Econ. 2012;15(Suppl 2):28–37.

    Article  CAS  Google Scholar 

  37. Mezquita Raya P, Perez A. Ramirez de Arellano A, Briones T, Hunt B, Valentine WJ: Incretin therapy for type 2 diabetes in Spain: a cost-effectiveness analysis of liraglutide versus sitagliptin. Diabetes Ther. 2013;4(2):417–30.

    Article  Google Scholar 

  38. Steen Carlsson K, Persson U. Cost-effectiveness of add-on treatments to metformin in a Swedish setting: liraglutide vs sulphonylurea or sitagplitin. J Med Econ. 2014;17(9):658–69.

    Article  Google Scholar 

  39. Perez A, Mezquita Raya P. Ramirez de Arellano A, Briones T, Hunt B, Valentine WJ. Cost-Effectiveness Analysis of Incretin Therapy for Type 2 Diabetes in Spain: 1.8 mg Liraglutide Versus Sitagliptin. Diabetes Ther. 2015;6(1):61–74.

    Article  CAS  Google Scholar 

  40. Bruhn D, Martin AA, Tavares R, Hunt B, Pollock RF. Cost-utility of albiglutide versus insulin lispro, insulin glargine, and sitagliptin for the treatment of type 2 diabetes in the US. J Med Econ. 2016;19(7):672–83.

    Article  Google Scholar 

  41. Roussel R, Martinez L, Vandebrouck T, Douik H, Emiel P, Guery M, Hunt B, Valentine WJ. Evaluation of the long-term cost-effectiveness of liraglutide therapy for patients with type 2 diabetes in France. J Med Econ. 2016;19(2):121–34.

    Article  Google Scholar 

  42. Barnett AH, Arnoldini S, Hunt B, Subramanian G, Hoxer CS. Switching from sitagliptin to liraglutide to manage patients with type 2 diabetes in the UK: A long-term cost-effectiveness analysis. Diabetes Obes Metab. 2018;20(8):1921–7.

    Article  CAS  Google Scholar 

  43. DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986;7(3):177–88.

    Article  CAS  Google Scholar 

  44. Husereau D, Drummond M, Petrou S, Carswell C, Moher D, Greenberg D, Augustovski F, Briggs AH, Mauskopf J, Loder E, et al. Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement. BMJ. 2013;346:f1049.

    Article  Google Scholar 

  45. Problems and Extensions of PPP, International Finance Theory and Policy - Chapter 30–4: Last Updated on 7/10/99 []

  46. Burdick C, Fisher L. Social Security cost-of-living adjustments and the Consumer Price Index. Soc Secur Bull. 2007;67(3):73–88.

    PubMed  Google Scholar 

  47. Pigou AC. The Foreign Exchanges. Q J Econ. 1922;37(1):52–74.

    Article  Google Scholar 

Download references


BSB acknowledged the financial support through a fellowship by the department of health research, government of India under Human Resource Development program, Indian Council of Medical Research. This article is an important part of the training of BSB, who is a master’s student in health technology assessment at Mahidol University, Bangkok, Thailand.


This work is a part of training in Health Technology Assessment's master’s degree, in which scholarship is provided by Mahidol University and the International Decision Support Initiative (iDSI). This work was produced as part of the International Decision Support Initiative ( www. idsihealth. org), which supports countries to get the best value for money from health spending. iDSI receives funding support from the Bill and Melinda Gates Foundation, the UK Department for International Development, and the Rockefeller Foundation.

Author information

Authors and Affiliations



Conceived and designed the experiments: BSB, AT. Performed the experiments: BSB, AT. Analyzed and interpreted the data: BSB, UC, AT. Contributed reagents, materials, analysis tools or data: BSB, UC, NC, JA, AT. Wrote the paper: BSB, UC, NC, JA, AT. BSB: Conceptualization; Data curation, Formal analysis, Writing—original draft. UC: Supervision, Writing—review & editing. NC: Writing—review & editing. JA: Writing—review & editing. AT: Conceptualization, Methodology, Supervision, Writing—review & editing. The author(s) read and approved the final manuscript.

Corresponding author

Correspondence to Ammarin Thakkinstian.

Ethics declarations

Ethics approval and consent to participate

Since the work is methods work on Systematic review and meta-analysis of published literature which is exempted from formal ethical review as per national ICMR Guidelines.

Consent for publication

Not Applicable.

Competing interests


Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bagepally, B.S., Chaikledkaew, U., Chaiyakunapruk, N. et al. Meta-analysis of economic evaluation studies: data harmonisation and methodological issues. BMC Health Serv Res 22, 202 (2022).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: