We assessed NIH-funded research projects in the 10th MP, from 2011 to 2015 [18], to inform priority setting for the 11th MP. Projects were included if the research objectives fulfilled research areas under the cluster of UAQH research [7]. A total of 71 projects were identified, and verified using administrative data from NIH and the National Medical Research Register database [19]. From these 71 projects, 67 were included and 4 excluded, as these did not receive direct NIH funding (Fig. 1).
Evaluation of research funding and impact
Projects identified under the cluster of UAQH in the 10th MP were categorised according to the domains of the World Health Organization (WHO) framework [20]: Governance, Health Economics (HE), Human Resources for Health (HRH), Information & Technology (IT) and Service Delivery (SD). Distribution of research funds were analysed based on these domains.
Stakeholder survey
Stakeholders consisted of researchers, decision makers, healthcare providers, research funders and academicians, from government and private sectors. Stakeholders’ perspectives and expectations on each of the five domains’ importance were elicited in the form of percentages. These values were considered as the intended fund allocation. If all domains had equal importance, each would be allocated 20%. However, if some domains were more important than others, the percentage allocated to each domain would differ. The feedback scores from stakeholders were used to compare against the 10th MP NIH research fund allocation. In November 2016, approximately 200 questionnaires were distributed to all participants of the NIH Research Week and a research priority setting workshop, as well as emailed to 85 stakeholders who were invited to the workshop but could not attend. To maintain anonymity, participants were requested to drop off completed questionnaires into collection boxes available at the venue or to respond online.
Key informant interview
We conducted document review and multiple semi-structured interviews with the principal investigator and/or project team member(s) in a workshop setting for project impact assessment (Additional file 1). Out of the 67 projects that were reviewed, 50 were included for impact analysis as the remaining 17 were ongoing projects. The analysis focused on three impact categories, which were informing policy, knowledge production and benefits to health and health sector, adopting the modified Payback Framework by Kwan et al. [11, 21] and the Canadian Institutes of Health Research [9, 16, 22]. The impact category of benefits to health and health sector was set out to show benefits that could or were expected to result in improved service delivery, cost savings, improved health or increased equity [21].
Data on funds awarded to each project were obtained from NIH administrative records, and divided into tertiles [21], namely low (<RM32,600 (€6936)), moderate (RM32,601–87,000 (€6936-18,510)) and high (>RM87,000 (€18,510)), using the exchange rate in April 2017 of €1 = RM4.70 [23].
Health research priority setting process
Identification of research gaps for the 11th Malaysia Plan
Health research priority areas of the 10th MP [7], national and global health plans [6, 18, 24, 25] were reviewed to identify national level research needs. We reviewed literature for conceptual frameworks to guide the identification of research gaps, and to increase comprehensiveness and inclusivity for each domain. Research gaps were grouped according to the domains of the WHO framework [20].
Research priority setting
We adopted and adapted the CHNRI methodology [1], a frequently used method to gain consensus on research areas. This involved setting criteria for prioritisation, incorporating stakeholders’ values through the application of weights, scoring research gaps, and calculating final scores for initial prioritisation. Stakeholders deliberated on this and identified the revised prioritised areas for the 11th MP.
Setting criteria for prioritisation
In setting criteria for prioritisation, we reviewed literature [1, 3, 26,27,28] and selected three (i.e. answerability/feasibility, importance/potential impact and magnitude/severity), in which each consisted of three questions (Additional file 2) chosen based on applicability, prior use in previous HRPS [7] and consensus from the project team. These criteria were applied in the stakeholders’ values instrument and research gaps assessment.
Incorporating stakeholders’ values
The stakeholder survey instrument contained a section on stakeholders’ values regarding prioritisation (Additional file 3). We incorporated these values as weights for the criteria, adopting the method of allocating monetary value [1, 3]. Stakeholders were asked to rate the importance of each criterion by allocating RM100 across the three criteria. For example, if all criteria had equal importance, each criterion should be allocated RM33.33. However, if one was more important, the allocation could be different. For example, the allocation could be as follows: answerability/feasibility criterion = RM65, importance/potential impact criterion = RM15 and magnitude/severity criterion = RM20. Regardless of research area, the total amount allocated to all criteria should add up to RM100.
Scoring of research gaps
In scoring each research area/gap, we asked the project team, predominantly researchers, to assess each area/gap by answering the questions for the criteria (see Additional file 2). We awarded 1 point to answers indicating agreement, 0.5 points for neither agreement nor disagreement, and 0 points for disagreement.
Calculation of scores for prioritisation
Summation of scores from the three criteria resulted in an intermediate score for each research area/gap. The stakeholders’ values in the form of monetary allocation for each criterion were incorporated as weighted mean score, representing the values of non-research expert groups. Thus, each research area’s final score consisted of the collective inputs of both researchers and stakeholders, similar to the method described by Rudan et al. [1] and was used for ranking. Further information on the calculation of weights to reflect stakeholders’ values can be found in Additional file 4.
Statistical analysis
All statistical analyses were conducted using Statistical Package for the Social Science Version 21. The One-Sample Wilcoxon Signed Rank Test was conducted to test the association between stakeholders’ values for the three criteria and hypothetical values.