Skip to main content
Download PDF

An evaluation index system for regional mobile SARS-CoV-2 virus nucleic acid testing capacity in China: a modified Delphi consensus study

BMC Health Services Research volume 22, Article number: 1080 (2022) Cite this article

Abstract

Background

Large-scale detection has great potential to bring benefits for containing the COVID-19 epidemic and supporting the government in reopening economic activities. Evaluating the true regional mobile severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) virus nucleic acid testing capacity is essential to improve the overall fighting performance against this epidemic and maintain economic development. However, such a tool is not available in this issue. We aimed to establish an evaluation index system for assessing the regional mobile SARS-CoV-2 virus nucleic acid testing capacity and provide suggestions for improving the capacity level.

Methods

The initial version of the evaluation index system was identified based on massive literature and expert interviews. The Delphi method questionnaire was designed and 30 experts were consulted in two rounds of questionnaire to select and revise indexes at all three levels. The Analytic Hierarchy Process method was used to calculate the weight of indexes at all three levels.

Results

The evaluation index system for assessing the regional mobile SARS-CoV-2 virus nucleic acid testing capacity, including 5 first-level indexes, 17 second-level indexes, and 90 third-level indexes. The response rates of questionnaires delivered in the two rounds of consultation were 100 and 96.7%. Furthermore, the authority coefficient of 30 experts was 0.71. Kendall’s coordination coefficient differences were statistically significant (P < 0.001). The weighted values of capacity indexes were established at all levels according to the consistency test, demonstrating that ‘Personnel team construction’ (0.2046) came first amongst the five first-level indexes, followed by ‘Laboratory performance building and maintenance’ (0.2023), ‘Emergency response guarantee’ (0.1989), ‘Information management system for nucleic acid testing resources’ (0.1982) and ‘Regional mobile nucleic acid testing emergency response system construction’ (0.1959).

Conclusion

The evaluation system for assessing the regional mobile SARS-CoV-2 virus nucleic acid testing capacity puts forward a specific, objective, and quantifiable evaluation criterion. The evaluation system can act as a tool for diversified subjects to find the weak links and loopholes. It also provides a measurable basis for authorities to improve nucleic acid testing capabilities.

Peer Review reports

Background

Coronavirus disease 2019 (COVID-19) [1] is an infectious disease caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) [2]. Since the disease was first reported, the virus has rapidly spread globally [3], leading to the coronavirus pandemic [4] and a severe global recession [5]. The global spread of the virus has been drastic; in particular, more than 243 million cases have been confirmed in more than 200 countries, including 4.9 million deaths, as of 26 October 2021 [6]. The infection can be spread by asymptomatic, presymptomatic, and symptomatic infectors [7]. The Chinese government implemented unprecedented nonpharmaceutical public health measures in the early stages of the COVID-19 outbreak to control the local spread of COVID-19 and stabilise the epidemic. The SARS-Cov-2 virus has undergone several mutations, whilst the epidemic has been expanding worldwide, resulting in virulence alterations that impact illness severity around the world [8]. The introduction of vaccines against COVID-19 globally and in China has undoubtedly improved epidemiological situations. Nevertheless, extensive data suggests that the immune protection conferred by vaccines declines over time, allowing for the emergence of new diseases [9, 10].

Given the above background, the possibility that an epidemic will occur in vaccinated populations remains high because of the emergence of new SARS-CoV-2 variants and postvaccination infection [11]. Imported SARS-CoV-2 infectors and/or contaminated commodities from abroad would bring the potential risk of the local COVID-19 epidemic in China. Analysing respiratory discharges using real-time quantitative polymerase chain reaction is the most reliable method for virus detection [12]. The goal of ‘early discovery, early reporting, early isolation, and early treatment’ necessitates rapid and precise testing to respond effectively to the epidemic. Early data from the China epidemic supported the idea that effective and prompt testing would reduce the time between the start of symptoms and diagnosis, thereby lowering the number of severe and critical cases [13]. Besides, data from Brazil also emphasized that identifying interval time could favour efficiently carrying out prevention actions to contain the COVID-19 pandemic [14]. In summary, the case for mass, community-wide polymerase chain reaction (PCR) testing for COVID-19 of the right individuals at the right time remains strong.

Since the containment strategy against COVID-19 has been adopted in China, the measure of large-scale PCR-based testing of SARS-CoV-2 in throat swab samples is critical for sustaining containment in mainland China [15], particularly for controlling numerous local outbreaks caused by imported viruses. For example, between 11 June and 14 July, 2020, 11.9 million persons were tested at Beijing’s Xinfadi market during the outbreak [16], contributing to an optimal balance between epidemic containment and economic protection in Beijing [17]. In another two cases of COVID-19 epidemic control, 4.5 million and 10.9 million people received nucleic acid testing for SARS-CoV-2 virus within 5 days in Dalian [18] and Qingdao [19], respectively. Guangzhou performed a large-scale nucleic acid testing amongst 18.7 million people within 3 days to control the spread of COVID-19 [20]. Between 8 and 21 August 2020, a comprehensive community testing approach was performed in Vietnam, together with an innovative sample pooling mechanism, contributing to the ultimate success of COVID-19 control in Da Nang City [21]. New Zealand’s efficient mass testing contributed to the government’s success in curbing the spread of the coronavirus [22]. The significant number of asymptomatic individuals and evidence of substantial presymptomatic transmission highlighted the efficacy of mass testing in controlling the illness [22]. Timely and effective large-scale detection has great potential to bring benefits for containing the epidemic and supporting the government in reopening economic activities [23, 24].

Comprehensive, active and innovative PCR testing strategies need heightened requirements for nucleic acid detection capability to respond effectively to the epidemic that may be caused by new SARS-Cov-2 virus variants. On 2 September 2020, the State Council’s interagency task force released the Plan on Advancing Coronavirus Nucleic Acid Testing Capacity Building (the Plan) [25]. According to the Plan, regional mobile nucleic acid testing capacity is needed to ensure a highly responsive mobilisation mechanism; thus, all individuals residing in corresponding locations can receive nucleic acid testing within a short period in cases when local COVID-19 patients (either symptomatic or asymptomatic ones) are discovered in routine screening measures under the government’s regular epidemic prevention strategies. Finding the weak links and loopholes is critical in building nucleic acid testing capacity against COVID-19. However, no comprehensive and practical evaluation tools are available for relevant departments to find and strengthen the weak links of testing capacity in different regions.

Previous studies reported several challenges encountered in a massive nucleic acid testing [26,27,28]. Firstly, the time-consuming detection period and expensive testing reagents are the main limitations of a massive testing. In particular, the high cost of the detection reagent needed in a massive testing causes stress to a region’s economic budget [26]. The equipment for nucleic acid testing is in short because of the concurrent increase in global requirements. Secondly, a vast workforce is needed to maintain the program. The effectiveness and scope of this program to contain the outbreak also depend on a comprehensive national strategy [27]. In addition, the close contact between medical staff and COVID-19 infectors in throat swab collection increases the risk of virus infection for the staff [28]. Compared with first-tier cities, such as Beijing and Guangzhou, third- and fourth-tier cities have a relatively poor response and reaction ability in handling public health emergencies. Cross-infection of SARS-CoV-2 occurred in the several rounds of massive testing against the COVID-19 outbreak in Yangzhou, China, because of the lack of experience, poor planning and management [29]. Given the evidence mentioned above, the primary goal of this study was to provide a scientific and comprehensive evaluation index system of regional mobile nucleic acid testing capacity. This index system will help optimise relevant issues in the whole process of massive testing, ensure the safety of working staff and subjects, and avoid cross-infection. Additionally, our study provides evidence for large-scale nucleic acid detection management in other countries.

Methods

Overview of the research process

The detailed research process was shown in Fig. 1. In this work, we employed a modified Delphi technique [30, 31] to solicit input from experts on what should be used and prioritised as assessment indices of testing capacity in the execution of large-scale COVID-19 infection screening at the community level. We formed a research group that included an epidemiologist, a health management expert, three doctorate students, and three master’s degree applicants. The research team was also in charge of creating an initial draft of the evaluation index system for the regional mobile testing capacity of SARS-CoV-2 virus nucleic acid, recruiting experts and supplying associated materials, reviewing expert suggestions and making necessary changes. We conducted the whole research in three steps: (1) Producing an initial draft for subsequent consultations: our research team advocated and drafted the initial version of the evaluation index system. We built a framework referring to related standards obtained via expert interviews and literature review; (2) Expert consultations using the modified Delphi method: a Delphi questionnaire was designed according to the above framework. The two-round Delphi survey was conducted via email between 18 March and 1 June, 2021; and (3) calculation of the weight of each index using the Analytic Hierarchy Process (AHP): the weight was calculated to understand the importance of each index in the regional mobile nucleic acid testing capacity evaluation of the whole system.

Fig. 1
figure 1

Flow diagram of modified Delphi study process

Full size image

Expert panel composition

We selected the experts by purposive sampling of those engaged in occupations/scientific fields related to infectious disease (e.g. COVID-19) prevention and control policies, strategies and measures. According to the nucleic acid testing needs, experts from different cities with strict epidemic control requirements were selected to ensure that they represent a wide array of approaches, practices, and backgrounds. No agreement existed on the number of panelists needed for a Delphi [32]. The typical Delphi panel size was 15–60 [31]. Evidence suggested that a panel size of 23 participants was necessary to stabilise the response characteristics in Delphi surveys [33]. Thus, we invited 30 experts based on the inclusion criteria and our resources to allow attrition. This number also guarantees that the study could continue smoothly and release financial pressure and labour cost. We recruited 30 experts based on the following criteria: (1) willingness to participate in this study, (2) a bachelor’s degree or higher, (3) work experience in epidemic prevention and control as a senior professional title or intermediate and above professional title and (4) engagement in health management, infectious/chronic disease prevention, hygienic detection, or health emergency for more than 5 years.

Delphi questionnaire preparation

A semi-open questionnaire was prepared for expert consultation. It consists of three parts: (1) the general information about the panelists, (2) the experts’ self-evaluation table and (3) the main text of the evaluation index system. The first part was to collect information about the time length of working experience in infectious diseases prevention and control, educational background, and professional title. The second phase involved gathering information on the experts’ decision-making processes and knowledge of the research topic. The third step involved gathering information on the indexes’ relevance, operability, and sensitivity and the retention and deletion of relevant items and expert comments on the items. The ‘Importance’ dimension indicates the role of this index in reflecting regional mobile nucleic acid testing capacity. The ‘Operability’ dimension refers to the content covered by the index that can be implemented smoothly under actual conditions. Moreover, the ‘Sensitivity’ dimension refers to the content covered by the index that can distinguish the mobile nucleic acid testing capacity between regions.

Evaluation index system draft construction

This study produced a complete and practicable evaluation index system for assessing regional mobile nucleic acid testing capacity to create a realistic and operational benchmark for future use at provincial, municipal and district-level government offices. We constructed the evaluation index system draft, considering three main aspects. Firstly, the experts from areas where large-scale nucleic acid testing was previously performed were consulted to assess the feasibility and usefulness of the established evaluation index method. Secondly, the Plan [25], the Protocol for Prevention and Control of COVID-19 (the Protocol) [34] and the Guidelines for Organization and Practice of Novel Coronavirus Nucleic Acid Mass Testing (the Guidelines) were referred to as the basis to ensure the scientific and authoritative integrity of the evaluation index system. Thirdly, the complete guiding opinions offered by experts and researchers in the published literature were employed as a reference base to maintain the scientific integrity of the evaluation index.

Two-round Delphi consultation

In the two-round Delphi consultation, the panelists who met the inclusion criteria were consulted with the questionnaire delivered by email and were required to respond and send the filled questionnaire in 2 weeks. If needed, video calls were made to provide necessary explanations to the experts who had questions about the project or indexes.

Upon receiving the email, the panelists read a brief introduction to the study processes and definitions of the dimensions of ‘Importance’, ‘Operability’ and ‘Sensitivity’. First, they were required to assign scores to the three dimensions, adding up to 100. Then, they provided their contact information (name and email address), educational background and familiarity (e.g. working years) with the research content in the questionnaire. Then, the importance, operability and sensitivity judgments were assessed on a 10-point scale, with 1 indicating ‘absolutely disagree’ (not a relevant or appropriate index) and 10 indicating ‘certainly agree’ (relevant and appropriate index) to the assertions. The experts were also encouraged to provide any advice or comments on each index in the questionnaire, particularly if they disagreed with the drafted indexes or suggested additional indexes.

Indexes were considered to achieve consensus if the mean values of the importance, operability and sensitivity scores were all equal to or more than 7 with a coefficient of variation (CV) < 0.25. The indexes were removed directly in the case of < 7 mean scores of either two of the ‘importance’, ‘operability’, and ‘sensitivity’ dimensions or a CV of ≥0.25. We also deleted the indexes that were suggested to be moved by four or more experts. In other situations of different opinions, the indexes were further discussed. Additionally, the research group further discussed expert opinions and then increased, merged, or modified some indexes. Finally, the group summarised and illustrated all modifications and sent the revised evaluation index system for the next round of expert consultation. After the consultation, the indexes achieving consensus were included in the final script of the evaluation index system (with minor amendments for sense only).

Reliability of the Delphi method

The positive coefficient was employed to show the enthusiasm and collaboration of specialists in the research regarding the response rate of the questionnaire. The questionnaire response rate was computed as the ratio of the number of completed questionnaires returned to the total number of questionnaires sent out. A response rate of 70% or above implied a high level of positivity amongst specialists [35].

The expert authority coefficient (Cr) was used to assess the validity of consultation results. It was calculated as the average of the sum of the scores indicating the expert’s familiarity with the consulting field (Cs) and the category of the basis for the expert’s consulting answers (Ca). The experts’ familiarity was evaluated on a five-point scale, with scores of 0.2, 0.4, 0.6, 0.8 and 1.0 from the lowest to the highest level of familiarity with the consulting field. The experts’ basis of the consultation was scored 0.8 in theoretical analysis, 0.6 in case of work experience, 0.4 in case of literature at home and abroad, and 0.2 in case of subjective judgment.

Kendall’s Concordance Coefficient (ω) was used to assess the level of coordination between expert viewpoints, with a range of 0 to 1 indicating low to high levels of coordination [31]. The Chi-square test was used to analyse the significance of the coordination coefficient. Statistically significant results indicated that expert opinions are well coordinated and the outcome is trustworthy.

Weight assignment of evaluation index system

Building the model

The hierarchical structure was built according to the Delphi method’s requirements, including the target, criteria and scheme layers [36]. The target layer in this study was the testing capacity evaluation system of regional mobile nucleic acid. The criteria layers were the first-level indexes established in this study. The subcriteria layers were the second-level indexes established in this study. The program layers were the third-level indexes established in this study.

Constructing expert judgment matrix

Judgment matrix assignment is a crucial part of AHP [37]. It ensures the consistency of judgmental thinking. For each index, we calculated the mean scores for the ‘importance’, ‘operability’ and ‘sensitivity’ dimensions. Then we calculated an aggregated weighted score [30] reflecting the combination of ratings for importance, operability, and sensitivity, using the formula below.

$$\mathrm{aggregated}\ \mathrm{weighted}\ \mathrm{score}=0.39\times \mathrm{mean}\ \mathrm{importance}+0.36\times \mathrm{mean}\ \mathrm{operability}+0.25\times \mathrm{mean}\ \mathrm{sensitivity}$$

The weights of importance, operability, and sensitivity were calculated based on the experts’ decision-making and familiarity with the research content. The experts assigned probability scores to importance, operability, and sensitivity that added up to 100%. Then, the mean probability scores of importance, operability, and sensitivity were calculated as the corresponding weights. The indexes in each level were compared in pairs according to aggregated weighted scores. Based on the judgment of indexes in each level, the weights of these indexes were calculated using the APH method [37].

Weight assignment of the index system

The above hierarchical structure pattern and judgment matrix were fed into the Yaahp software for analysis [38]. The weights of the first-, second- and third-level indexes were calculated using the weights of the first-, second- and third-level indexes. The combined weight signified that the weight distribution of the superior indices should be considered when determining the weight. The combined weight was calculated as the continuous production of each level’s initial index weight.

Statistical analyses

Each item was described using descriptive statistics such as mean, standard deviation, and CV; Expert opinion consensus and the calculation of the positive coefficient, authority coefficient, and coordination coefficients were used to test the Delphi method’s reliability and validity. The consistency rate (CR) was utilised to determine whether the matrix is consistent. All quantitative analyses used IBM SPSS Version 24. The hierarchical analysis was constructed based on the Yaahp software [38].

This study was classified as a service evaluation and did not require national research ethics committee approval (as advised by the Ethics Committee of Tongji Medical College of Huazhong University of Science and Technology). We certified that all applicable institutional and governmental regulations concerning the ethical use of human volunteers were followed during this research.

Results

Panel characteristics

In this study, we adopted two rounds of expert consultation, enrolling 30 experts in seven cities, including Beijing, Nanjing, Shenzhen, Ningbo, Zhengzhou, Yichang and Wuhan. Amongst the 30 experts, 20 (66.7%) had a doctorate, 8 (26.7%) had a master’s degree and 2 (6.7%) had a bachelor’s degree. Eighteen experts held a senior title, and 12 held a vice-senior title. The mean working years was 22.23 years (standard deviation: 10.54, median: 21.5, rang: 5–46) (Table 1).

Table 1 Characteristics of consulting experts using the modified Delphi method [n (%)]
Full size table

Reliability of the two-rounded Delphi consultations

The response rates of the two-round consultation were 100 and 96.7%. Moreover, all filled questionnaires collected were valid for subsequent analysis. The Cr was 0.71 with 0.81 Cs and 0.60 Ca. The experts involved in this study had high authority.

The overall coordination coefficients of importance, operability and sensitivity were 0.243, 0.158, and 0.129, respectively, in the first round (Table 2). In the second round, the overall coordination coefficients of importance, operability, and sensitivity were 0.249, 0.217 and 0.171. Kendall’s coordination coefficients were larger in the second round than those in the first round (all P < 0.001 by chi-square test).

Table 2 The result of expert opinions’ coordination degree
Full size table

Initial draft of the evaluation index system

Based on massive literature and expert interviews, the research generated a list of potential factors (‘indexes’) to cover the actual common situation related to detection capability. These factors were refined in a series of meetings to develop the initial draft of the evaluation index system, including 5 first-level indexes, 17 second-level indexes, and 93 third-level indexes.

Revisions to the draft after the first-round Delphi consultation

All of the panelists completed the expert consultation in the First Round. Nineteen (63.33%) experts suggested modifying the indexes and providing scores for each index at the three dimensions as ‘importance’, ‘operability’, and ‘sensitivity’. In particular, the modification suggestions included: (1) modifying the first-level index ‘Testing capacity building’ to ‘Laboratory performance building and maintenance’; (2) merging the second-level indexes of ‘Daily testing personnel’, ‘Reserve testing personnel’ and ‘Mobile response testing personnel’ into one, i.e. ‘Testing personnel’; (3) adding relevant second-level indexes about sampling personnel; (4) adding a third-level index, i.e. ‘Whether to establish an incentive and reward system for emergency task work’.

The means of importance, operability and sensitivity scores ranged from 6.83 to 9.50, 7.27 to 9.03 and 6.90 to 8.50, respectively (Table 3). The CVs of importance, operability and sensitivity scores ranged from 0.09 to 0.32, 0.11 to 0.28 and 0.16 to 0.26, respectively (Table 3). The minimum mean and maximum CVs were 6.83 and 0.32, respectively, in the third-level indexes’ importance scores (Table 3).

Table 3 The result of expert opinions’ concentration degree
Full size table

Considering the experts’ comments, the research team modified 12 indexes (1 first-level, 6 second-level and 5 third-level indexes), deleted 22 indexes (2 second-level and 20 third-level indexes), added 21 indexes (2 second-level and 19 third-level indexes) and merged 3 third-level indexes into 1. The revised version of the evaluation index system, consisting of 5 first-level, 17 second-level and 90 third-level indexes, was sent to the experts for a second-round consultation.

Further revisions after the second-round Delphi consultation

One expert dropped out in the second-round consultation; thus, 29 experts completed the consultation. Six (20.69%) experts suggested minor amendments to the present version. All means of scores for the importance, operability and sensitivity were ≥ 7, and all CVs were < 0.25 (Table 3). Such results demonstrated a consensus of experts’ opinions on the scores of the indexes at the three dimensions.

Eight third-level indicators were with minor changes based on the results of the questionnaires and group talks. The evaluation system for assessing the regional mobile nucleic acid testing capacity was constructed with 5 first-level, 17 second-level, and 90 third-level indexes (Table 4).

Table 4 The weight coefficient of each index in the evaluation system
Full size table

Capacity evaluation system weight distribution

In this study, 23 judgment matrices were constructed. These matrices had less than 0.1 CR values in the consistency test for all indexes, showing that the matrix’s degrees of inconsistency were scientifically acceptable. The weight coefficient for each of the evaluation indexes was calculated referring to the results of the consistency test, suggesting that ‘Personnel team construction’ with a weight coefficient of 0.2046 came first amongst the five first-level indexes, followed by ‘Laboratory performance building and maintenance’ (weight coefficient = 0.2023), ‘Emergency response guarantee’ (0.1989), ‘Information management system for nucleic acid testing resources’ (weight coefficient = 0.1982) and ‘Regional mobile nucleic acid testing emergency response system construction’ (weight coefficient = 0.1959) (Table 4).

Discussion

The testing capability evaluation index system was built using a modified Delphi approach in this study, including 5 first-level indexes, i.e. ‘Personnel team construction’, ‘Laboratory performance building and maintenance’, ‘Emergency response guarantee’, ‘Information management system for nucleic acid testing resources’ and ‘Regional mobile nucleic acid testing emergency response system construction’, 17 second-level indexes and 90 third-level indexes. The AHP approach was used to calculate the weight coefficient for the three-level evaluation indexes. Moreover, the weight coefficient was reliable with consistency ratios of all less than 0.10. As policymakers worldwide seek to improve COVID-19 prevention, detection and response amid a flare-up of cases driven by the highly contagious Delta variant, the introduction of new nucleic acid testing programmers appears likely to continue. However, comprehensive and practical evaluation tools have not been available to monitor and evaluate nucleic acid testing capacity within and across jurisdictions. For the first time, the outcomes of this research provided robust references to benchmark mass testing capacity in China, which may then be refined for comparison with the outcomes from other countries developing and offering mass testing.

The evaluation index system for assessing regional mobile nucleic acid testing capacity was scientific, comprehensive and diversified with the following characteristics. Firstly, the theoretical basis was efficient and reliable. The initial draft of the evaluation index system was constructed based on the technical specifications and laws and regulations promulgated by the state and literature reviews, with reference to the Plan [25], the Protocol [34] and the Guidelines. In this study, relevant panelists were selected based on specific features (e.g. age, profession title and working year). These panellists were influential and active in COVID-19 epidemic prevention and control. Our team implemented revisions provided by specialists from many departments and perspectives, thereby ensuring that the system was suited for usage in various places with varying epidemic risk levels. Secondly, the authority coefficient (0.71) and questionnaire response rate (100 and 96.7%) were within the acceptable limits (more than or equal to 0.70) [35, 39]. Thirdly, this evaluation index system was systematic and comprehensive because we considered not only the whole process of nucleic acid detection but also the nucleic acid testing preparation, nucleic acid detection implementation and testing result presentation. Moreover, timely financial and material support and the construction of the emergency response system were considered. Fourthly, the diversified subject could use the evaluation index system to adjust and perfect the regional mobile novel coronavirus nucleic acid test activities. Each provincial or city-level government can conduct a self-assessment according to its situation. Besides, the government can evaluate the capacity of different areas according to the index. Diversified evaluation might assure fairness and impartiality in the overall nucleic acid testing capacity assessment.

The evaluation system followed the principles of integrity, emphasis, hierarchy, comparability and operability to achieve the purpose of ‘early detection, early reporting, early isolation, and early treatment’ of COVID-19 patients [40], and reduced the risk of large-scale spread of epidemics, by making arrangements in five aspects: regional mobile nucleic acid testing emergency response system construction, personnel team construction, emergency response guarantee, laboratory performance building and maintenance, information management system for nucleic acid testing resources. ‘Personnel team construction’ and ‘laboratory performance building and maintenance’ had the highest weights, indicating that they were important in strengthening the nucleic acid sampling and testing capacity in the personnel and laboratory techniques. Cross-contamination and other diagnostic mistakes, including those caused by an increase in specimens, a shortage of laboratory personnel, and a lack of quality control, may occur in a laboratory setting [41]. Stratigraphically speaking, our evaluation index system was efficient and reliable to some extent. COVID-19 testing on a large scale necessitates many people, many of whom may be untrained and working in a stressful environment. The regional mobile nucleic acid testing task necessitates a dynamic deployment of staff on duty in each part and a standby shift. For the emergency circumstance of COVID-19 mass testing, an emergency human resource allocation plan and a reserve echelon rotation mechanism must be developed, and proper rotation and collocation must be carried out [42]. As a result, the study group argued that long-term and standard human resource management procedures should be devised to ensure the smooth progression of the mass testing activity.

As for testing capacity, ‘Laboratory performance building and maintenance’ integrated the specimen reception, testing and biosafety management into a network that can respond effectively and efficiently to emergencies. Sufficient qualified personnel and efficient laboratory performance support provinces, cities and districts to provide increased sample analyses in epidemic outbreaks or other large-scale emergency events requiring surge capacity testing of samples and products. As reported, two air-inflated COVID-19 test laboratories, namely Huo-Yan Laboratory [43] and Falcon laboratory [44], were built in 1 day to meet the surge of testing demands in Guangzhou. Saudi Arabia, Brunei, and Kazakhstan are amongst the countries and territories that have started or built Huo-Yan labs to combat the COVID-19 virus [43]. New sampling and testing approaches have been described in recent studies [45,46,47], indicating that further training with drills and assessments is required to assure standard and uniform sample collection and testing by all employees.

The weight of ‘Emergency response guarantee’ in the first-level indexes was the third largest. The guarantee of emergency nucleic acid testing supplies was essential for medical personnel to implement emergency testing work and safety protection [48]. Therefore, the procurement and use of emergency supplies and other management work were directly related to the speed of response to the outbreak and the final effect of emergency testing work [49]. Perfecting the procurement, reserve and use management system of emergency nucleic acid testing supplies is the key to improving emergency management and handling capacity. The weight of ‘Information management system for nucleic acid testing resources’ in the first-level indexes followed ‘Emergency response guarantee’. Establishing an information management system for nucleic acid testing resources could comprehensively, accurately and dynamically grasp the information of nucleic acid testing institutions, sampling and testing personnel, thereby achieving efficient and accurate management of their deployment. The first-level index ‘Nucleic acid mobile detection emergency system construction’ had the lowest weight; however, this finding did not mean it was unimportant. Studies have found that mass gatherings organized in places with high population density can easily lead to widespread and cluster outbreaks of epidemic diseases, such as the cluster epidemic of Buddhist gatherings in Ningbo [50] and the outbreak epidemic in the health training centre of Jilin Province [51]. Once an outbreak occurs, the difficulty of epidemic prevention and control increases if all cases cannot be identified quickly. Therefore, when COVID-19 patients were reported, the nucleic acid mobile detection emergency system should be activated as soon as possible to prevent the epidemic from spreading.

For the second-level indexes, we adopted expert opinions to merge the ‘Daily testing personnel’, ‘Reserve testing personnel’, and ‘Mobile response testing personnel’ to ‘testing personnel’. We also added ‘Sampling personnel’ because proper specimen collection is the most important step in the laboratory diagnosis of infectious diseases. A specimen not collected correctly may lead to false or inconclusive test results. The novel coronavirus testing specimens shall be collected by qualified technicians who have received biosafety training (who have passed the training) and are equipped with the corresponding laboratory skills.

The time spent in a large-scale nucleic acid program is an important factor for taking further interventions to control infection spread. We considered this issue along two lines (both authorities and residents). For authorities, the main aim was to find the virus infectors and block transmission at the fastest speed. Thus, several periods (e.g. the detection time of nucleic acid) are critical for authorities to take decisive action, which is needed as soon as possible. For residents, the major concern is the time from throat swab collection to the release of the test result. Thus, we defined indexes for the ‘Testing results presentation and feedback’ (Table 4). The results’ reporting time for residents/patients was defined as the time from throat swab collection to the release of the test result. It can represent the total turnaround time (time from throat swab collection or arrival of the sample at the laboratory until the result is communicated to the patient/person or authorities). Consequently, the total turnaround time and phased time can be used to assess the regional nucleic acid testing capacity.

Some experts believed that the reward for emergency work should be quantified according to the workload of different areas, and the epidemic’s severity should be considered. The study group argued that the flexible index should be set with the goal of protecting the health of all employees in mind, and that it should be updated dynamically as the epidemic progresses.

In the setting of the global COVID-19 pandemic, our findings supported the application of evaluation index system in China. Moreover, generalising the evaluation index system to other countries was prudent. China is a special country in terms of health care because of its large population size and unbalanced health care development in different areas [52]. The Chinese government has been spending a lot of manpower and resources to overcome this public health issue. The large-scale nucleic acid testing strategies support China’s sustained containment of COVID-19, regardless of backward and developed regions in China. Consequently, according to China’s experience, this evaluation index system may be prudently extrapolated to many of other low- and middle-income countries, each one with its particularities.

Several strengths characterized our study. Firstly, this study was the first to construct the evaluation index system of regional mobile nucleic acid testing capacity. Thus, this study provided a reference for nucleic acid mobile detection capability evaluation. It also has guiding relevance for ensuring nucleic acid detection capability. Secondly, Delphi method allows the development of defensible, valid, and reasonable solutions based on expert opinion [31]. The Delphi method is regarded as a structured approach for evaluating and combining human judgment. Rowe et al stated that the Delphi method can be used when the researcher is convinced that the technique can generate more accurate assessments and judgments than that provided by individuals [53]. Our objective was to develop a comprehensive tool for evaluating the regional mobile SARS-CoV-2 virus nucleic acid testing capacity. Thus, we resorted to the Delphi technique. Considering the issues of seniority, interfering or inhibiting personality traits that easily occur in a face-to-face meeting, the Delphi method is reliable in obtaining an objective and unbiased point of view. The experts from regions with strict epidemic control requirements in different provinces and cities were selected to avoid bias in the judgment of approaches, practices, and backgrounds. Thirdly, we ensured anonymity between participants, avoiding the interplay effects. Fourthly, the items’ importance, operability and sensitivity were comprehensively considered in this study, indicating that the representativeness of weights was multidimensional and practically meaningful. However, this study also had limitations. Firstly, we realised that the decision-making process inevitably involved subjectivity and judgmental inputs in terms of panel selection, item selection and dispute resolution. Secondly, the initial outbreak of COVID-19 was in Wuhan; hence, a limitation may exist because approximately one-third of the experts were from Hubei. Furthermore, more than a third of the specialists invited were from research institutes. Their perspectives could not be more comprehensive than frontline healthcare staff.

Conclusions

We established an evaluation index system for regional mobile nucleic acid testing capacity based on a scientifically designed Delphi process. Several important points obtained from this study. Firstly, the evaluation index system proposed specific, objective, and quantitative evaluation criteria that aid the government in containing the pandemic and resuming economic operations. In particular, the evaluation index system may help relevant departments to find and strengthen weak links of testing capacity in different regions. Secondly, the index weights informed the functional departments about the priority in managing a massive nucleic acid test. This evaluation index system may be generalised to other countries with prudence considering the shared, different, and complicated conditions in different countries. International consideration and feedback about the suitability of the evaluation index system are encouraged to develop an international consensus for virus nucleic acid testing against COVID-19.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request. The data are not publicly available due to privacy or ethical restrictions.

References

  1. Vannabouathong C, Devji T, Ekhtiari S, Chang Y, Phillips SA, Zhu M, et al. Novel coronavirus COVID-19: current evidence and evolving strategies. J Bone Joint Surg Am. 2020;102(9):734–44.

    Article  Google Scholar 

  2. Coronaviridae Study Group of the International Committee on Taxonomy of Viruses. The species Severe acute respiratory syndrome-related coronavirus: classifying 2019-nCoV and naming it SARS-CoV-2. Nat Microbiol. 2020;5(4):536.

    Article  Google Scholar 

  3. Zhou P, Yang XL, Wang XG, Hu B, Zhang L, Zhang W, et al. A pneumonia outbreak associated with a new coronavirus of probable bat origin. Nature. 2020;579(7798):270–3.

    Article  CAS  Google Scholar 

  4. WHO Director-General’s Opening Remarks at the Media Briefing on COVID-19. https://www.who.int/director-general/speeches/detail/who-director-general-s-opening-remarks-at-the-media-briefing-on-covid-19-11-march-2020.

  5. Barrett P, Das S, Magistretti G, Pugacheva E, Wingender MP. After-effects of the COVID-19 pandemic. Prospects Medium Term Econ Damage. 2021;21(203):1–23.

    Google Scholar 

  6. WHO Coronavirus (COVID-19) Dashboard https://covid19.who.int/.

  7. Wiersinga WJ, Rhodes A, Cheng AC, Peacock SJ, Prescott HC. Pathophysiology, transmission, diagnosis, and treatment of coronavirus disease 2019 (COVID-19): a review. JAMA. 2020;324(8):782–93.

    Article  CAS  Google Scholar 

  8. Khan WH, Hashmi Z, Goel A, Ahmad R, Gupta K, Khan N, et al. COVID-19 pandemic and vaccines update on challenges and resolutions. Front Cell Infect Microbiol. 2021;11:690621.

    Article  CAS  Google Scholar 

  9. Notarte KI, Ver AT, Velasco JV, Pastrana A, Catahay JA, Salvagno GL, et al. Effects of age, sex, serostatus, and underlying comorbidities on humoral response post-SARS-CoV-2 Pfizer-BioNTech mRNA vaccination: a systematic review. Crit Rev Clin Lab Sci. 2022;59(6):373–90.

  10. Notarte KI, Guerrero-Arguero I, Velasco JV, Ver AT, Santos de Oliveira MH, Catahay JA, et al. Characterization of the significant decline in humoral immune response six months post-SARS-CoV-2 mRNA vaccination: a systematic review. J Med Virol. 2022;94(7):2939–61.

    Article  CAS  Google Scholar 

  11. Brown CM, Vostok J, Johnson H, Burns M, Gharpure R, Sami S, et al. Outbreak of SARS-CoV-2 infections, including COVID-19 vaccine breakthrough infections, associated with large public gatherings - Barnstable County, Massachusetts, July 2021. MMWR Morb Mortal Wkly Rep. 2021;70(31):1059–62.

    Article  CAS  Google Scholar 

  12. Corman VM, Landt O, Kaiser M, Molenkamp R, Meijer A, Chu DK, et al. Detection of 2019 novel coronavirus (2019-nCoV) by real-time RT-PCR [published correction appears in Euro Surveill. 2020 Apr;25(14):] [published correction appears in Euro Surveill. 2020 Jul;25(30):] [published correction appears in Euro Surveill. 2021 Feb;26(5):]. Euro Surveill. 2020;25(3):2000045.

  13. Pan A, Liu L, Wang C, Guo H, Hao X, Wang Q, et al. Association of Public Health Interventions with the epidemiology of the COVID-19 outbreak in Wuhan, China. JAMA. 2020;323(19):1915–23.

    Article  CAS  Google Scholar 

  14. Lima FET, Albuquerque NLS, Florencio SSG, Fontenele MGM, Queiroz APO, Lima GA, et al. Time interval between onset of symptoms and COVID-19 testing in Brazilian state capitals, august 2020. Epidemiologia e servicos de saude : revista do Sistema Unico de Saude do Brasil. 2020;30(1):e2020788.

    Article  Google Scholar 

  15. Li Z, Liu F, Cui J, Peng Z, Chang Z, Lai S, et al. Comprehensive large-scale nucleic acid-testing strategies support China’s sustained containment of COVID-19. Nat Med. 2021;27(5):740–2.

    Article  CAS  Google Scholar 

  16. Tan W, Niu P, Zhao X, Pan Y, Zhang Y, Chen L, et al. Reemergent cases of COVID-19 - Xinfadi wholesales market, Beijing municipality, China, June 11, 2020. China CDC Weekly. 2020;2(27):502–4.

    Article  Google Scholar 

  17. Cui X, Zhao L, Zhou Y, Lin X, Ye R, Ma K, et al. Transmission dynamics and the effects of non-pharmaceutical interventions in the COVID-19 outbreak resurged in Beijing, China: a descriptive and modelling study. BMJ Open. 2021;11(9):e047227.

    Article  Google Scholar 

  18. Ma H, Zhang J, Wang J, Qin Y, Chen C, Song Y, et al. COVID-19 outbreak caused by contaminated packaging of imported cold-chain products - Liaoning Province, China, July 2020. China CDC Weekly. 2021;3(21):441–7.

    Article  Google Scholar 

  19. Xing Y, Wong GWK, Ni W, Hu X, Xing Q. Rapid response to an outbreak in Qingdao, China. N Engl J Med. 2020;383(23):e129.

    Article  CAS  Google Scholar 

  20. Cheng W, Zhao N, Qin Y, Tian J, Huang Y. Mass testing to support sustained containment of COVID-19. J Glob Health. 2021;11:03114.

    Article  Google Scholar 

  21. Thanh TT, Nhan NTT, Mai HK, Trieu NB, Huy LX, Thuy HTT, et al. The application of sample pooling for mass screening of SARS-CoV-2 in an outbreak of COVID-19 in Vietnam. Am J Trop Med Hyg. 2021;104(4):1531–4.

    Article  CAS  Google Scholar 

  22. Efficient mass testing, contact tracing New Zealand’s secret in containing COVID-19, says Filipino-Kiwi parliamentarian https://www.cnnphilippines.com/news/2020/6/25/mass-testing-contact-tracing-new-zealand-secret.html.

  23. Wang CJ, Ng CY, Brook RH. Response to COVID-19 in Taiwan: big data analytics, new technology, and proactive testing. JAMA. 2020;323(14):1341–2.

    Article  CAS  Google Scholar 

  24. Tangcharoensathien V, Bassett MT, Meng Q, Mills A. Are overwhelmed health systems an inevitable consequence of covid-19? Experiences from China, Thailand, and New York state. BMJ (Clinical research ed). 2021;372:n83.

    Google Scholar 

  25. State Council Inter-agency Task Force Issues Plan on Further Advancing Coronavirus Nucleic Acid Testing Capacity Building https://www.fmprc.gov.cn/mfa_eng/topics_665678/kjgzbdfyyq/t1811235.shtml.

  26. Zhu N, Wong PK. Advances in viral diagnostic Technologies for Combating COVID-19 and future pandemics. SLAS Technol. 2020;25(6):513–21.

    Article  CAS  Google Scholar 

  27. Kameda K, Barbeitas MM, Caetano R, Löwy I, Oliveira ACD, Corrêa M, et al. Testing COVID-19 in Brazil: fragmented efforts and challenges to expand diagnostic capacity at the Brazilian Unified National Health System. Cadernos de saude publica. 2021;37(3):e00277420.

    Article  Google Scholar 

  28. Chan JF, Yuan S, Kok KH, To KK, Chu H, Yang J, et al. A familial cluster of pneumonia associated with the 2019 novel coronavirus indicating person-to-person transmission: a study of a family cluster. Lancet (London, England). 2020;395(10223):514–23.

    Article  CAS  Google Scholar 

  29. Yangzhou improves testing moves http://english.www.gov.cn/news/topnews/202108/14/content_WS61171ae0c6d0df57f98de6d9.html.

  30. Cash RE, Leggio WJ, Powell JR, McKenna KD, Rosenberger P, Carhart E, et al. Emergency medical services education research priorities during COVID-19: a modified Delphi study. J Am Coll Emerg Phys Open. 2021;2(4):e12543.

    Google Scholar 

  31. Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000;32(4):1008–15.

    CAS  PubMed  Google Scholar 

  32. Feyissa GT, Lockwood C, Woldie M, Munn Z. Evaluation of a guideline developed to reduce HIV-related stigma and discrimination in healthcare settings and establishing consensus. PLoS One. 2018;13(7):e0198781.

    Article  Google Scholar 

  33. Akins RB, Tolson H, Cole BR. Stability of response characteristics of a Delphi panel: application of bootstrap data expansion. BMC Med Res Methodol. 2005;5:37.

    Article  Google Scholar 

  34. Liu F, Zheng C, Wang L, Geng M, Chen H, Zhou S, et al. Interpretation of the protocol for prevention and control of COVID-19 in China (edition 8). China CDC Weekly. 2021;3(25):527–30.

    Article  Google Scholar 

  35. Shi C, Zhang Y, Li C, Li P, Zhu H. Using the Delphi method to identify risk factors contributing to adverse events in residential aged care facilities. Risk Manag Healthc Policy. 2020;13:523–37.

    Article  Google Scholar 

  36. Ben-Assuli O, Kumar N, Arazy O, Shabtai I. The use of analytic hierarchy process for measuring the complexity of medical diagnosis. Health Inform J. 2020;26(1):218–32.

    Article  Google Scholar 

  37. Gao Z, Jiang Y, He J, Wu J, Xu J, Christakos G. An AHP-based regional COVID-19 vulnerability model and its application in China. Model Earth Syst Environ. 2022;8(2):2525–38.

  38. Xu M, Chen C, Deng X. Systematic analysis of the coordination degree of China's economy-ecological environment system and its influencing factor. Environ Sci Pollut Res Int. 2019;26(29):29722–35.

    Article  Google Scholar 

  39. Goodman CM. The Delphi technique: a critique. J Adv Nurs. 1987;12(6):729–34.

    Article  CAS  Google Scholar 

  40. Chen W, Wang Q, Li YQ, Yu HL, Xia YY, Zhang ML, et al. Early containment strategies and core measures for prevention and control of novel coronavirus pneumonia in China. Zhonghua yu fang yi xue za zhi [Chin J Prev Med]. 2020;54(3):239–44.

    CAS  Google Scholar 

  41. Albano PM, Notarte KI, Macaranas I, Maralit B. Cross-contamination in molecular diagnostic laboratories in low-and middle-income countries. PJP. 2020;5(2):7–11%@ 2507–8364.

    Google Scholar 

  42. HE H-y YQ, Li J-l, Xu G-y, Kuang Y-y. Organization and nursing management of mass nucleic acid testing of COVID-19. J Nurs (China). 2021;28:52–5.

    Google Scholar 

  43. Guangzhou builds the most capable COVID-19 testing lab in 10 hours. 2021. https://news.cgtn.com/news/2021-06-05/China-builds-nucleic-acid-testing-lab-in-10-hours-in-Guangzhou-10QpS4W6IXm/index.html.

  44. Four groups of air-inflated COVID-19 test labs put into use in Guangzhou. https://www.globaltimes.cn/page/202106/1225450.shtml.

  45. Boutros J, Benzaquen J, Marquette CH, Ilié M, Labaky M, Benchetrit D, et al. Salivary detection of COVID‑19: clinical performance of oral sponge sampling for SARS‑CoV‑2 testing. ERJ Open Res. 2021;7(4):00396–2021.

  46. Park I, Lim J, You S, Hwang MT, Kwon J, Koprowski K, et al. Detection of SARSCoV-2 Virus Amplification Using a Crumpled Graphene Field-Effect Transistor Biosensor. ACS Sens. 2021;6(12):4461–70.

  47. Wang X, Huang Z, Song J, Zhao R, Xiao Y, Wang H. Novel pooling strategy with sample concentration for screening of SARS-CoV-2. J Clin Pathol. 2021. https://doi.org/10.1136/jclinpath-2021-207579.

  48. Xiong Fan WY, Jiale W, Tao L, Hongwen W. Assurance and Management of Medical Supplies during the pandemic of COVID-19. Med Equip. 2021;34:55–7.

    Google Scholar 

  49. Wei G, Guang Y, Wenting W, Tao W, Lu D, Guixian T. Logistic service of emergency supplies during epidemic of COVID‑19. Modern Hospitals. 2020;20(08):1180–1182+1186

  50. Ye LX, Wang HB, Lu HC. Investigation of a cluster epidemic of COVID-19 in Ningbo. China J Epidemiol. 2029;2020:41.

    Google Scholar 

  51. Jilin Province Notification. The chain of transmission of the epidemic is clear 102 cases of secondary infection are transmitted by Lin of the same imported case https://wap.peopleapp.com/article/6111599/6020985.

  52. Lu X, Zhang R, Wu W, Shang X, Liu M. Relationship between internet health information and patient compliance based on trust: empirical study. J Med Internet Res. 2018;20(8):e253.

    Article  Google Scholar 

  53. Shewade HD, Jeyashree K, Kalaiselvi S, Palanivel C, Panigrahi KC. Competency-based tool for evaluation of community-based training in undergraduate medical education in India - a Delphi approach. Adv Med Educ Pract. 2017;8:277–86.

    Article  Google Scholar 

Download references

Acknowledgements

We would like to acknowledge the support of National Natural Science Foundation and the China Association for Science and Technology. We are also grateful to the panel members for their time and contributions. Finally, Dongsheng Di thanks the inimitable care and support of Lina Du over the past years.

Funding

The work was funded by the National Natural Science Foundation of China [Grant no. 81573235], the China Association for Science and Technology [Grant no. 20200608CG111315].

Author information

Authors and Affiliations

  1. MOE Key Lab of Environment and Health, Department of Epidemiology and Biostatistics, School of Public Health, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, 430030, China

    Dong-sheng Di, Jian-li Zhang, Mu-hong Wei, Hao-long Zhou, Yuan Cui, Ru-yi Zhang & Qi Wang

  2. Hubei Provincial Center for Disease Control and Prevention, Wuhan, 430079, Hubei, China

    Ye-qing Tong

  3. Department of Social Medicine and Health Management, School of Public Health, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, 430030, China

    Jun-an Liu

Authors
  1. Dong-sheng Di
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jian-li Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mu-hong Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hao-long Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yuan Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ru-yi Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Ye-qing Tong
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Jun-an Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Qi Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Qi Wang and Jun-an Liu are joint corresponding authors. Qi wang and Dong-sheng Di had full access to the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Qi wang and Dong-sheng Di were responsible for conception of the study and drafted the manuscript. Qi Wang was responsible for design of the study. Jian-li Zhang, Mu-hong Wei, Hao-long Zhou, Yuan Cui, Ru-yi Zhang and Ye-qing Tong contributed to preparation and data analysis. Qi wang, Jun-an Liu and Dong-sheng Di contributed to revision of the manuscript. All the authors contributed to the interpretation of the data and critically reviewed the manuscript for publication. All authors read and approved the final manuscript.

Corresponding authors

Correspondence to Jun-an Liu or Qi Wang.

Ethics declarations

Ethics approval and consent to participate

This study was classified as a service evaluation and did not require national research ethics committee approval (as advised by the Ethics Committee of Tongji Medical College of Huazhong University of Science and Technology). We certify that all applicable institutional and governmental regulations concerning the ethical use of human volunteers were followed during this research.

Consent for publication

Informed consent was obtained from all experts before enrollment.

Competing interests

The authors declare no competing interests. The findings and conclusions in this report are those of the author(s) and do not necessarily represent the views of the funding agency.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Di, Ds., Zhang, Jl., Wei, Mh. et al. An evaluation index system for regional mobile SARS-CoV-2 virus nucleic acid testing capacity in China: a modified Delphi consensus study. BMC Health Serv Res 22, 1080 (2022). https://doi.org/10.1186/s12913-022-08446-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12913-022-08446-9

Keywords

BMC Health Services Research

ISSN: 1472-6963

Contact us