A systematic review: the dimensions to evaluate health care performance and an implication during the pandemic
BMC Health Services Research volume 22, Article number: 621 (2022)
The balanced scorecard (BSC) has been implemented to evaluate the performance of health care organizations (HCOs). BSC proved to be effective in improving financial performance and patient satisfaction.
This systematic review aims to identify all the perspectives, dimensions, and KPIs that are vital and most frequently used by health care managers in BSC implementations.
This systematic review adheres to PRISMA guidelines. The PubMed, Embase, Cochrane, and Google Scholar databases and Google search engine were inspected to find all implementations of BSC at HCO. The risk of bias was assessed using the nonrandomized intervention studies (ROBINS-I) tool to evaluate the quality of observational and quasi-experimental studies and the Cochrane (RoB 2) tool for randomized controlled trials (RCTs).
There were 33 eligible studies, of which we identified 36 BSC implementations. The categorization and regrouping of the 797 KPIs resulted in 45 subdimensions. The reassembly of these subdimensions resulted in 13 major dimensions: financial, efficiency and effectiveness, availability and quality of supplies and services, managerial tasks, health care workers' (HCWs) scientific development error-free and safety, time, HCW-centeredness, patient-centeredness, technology, and information systems, community care and reputation, HCO building, and communication. On the other hand, this review detected that BSC design modification to include external and managerial perspectives was necessary for many BSC implementations.
This review solves the KPI categorization dilemma. It also guides researchers and health care managers in choosing dimensions for future BSC implementations and performance evaluations in general. Consequently, dimension uniformity will improve the data sharing and comparability among studies. Additionally, despite the pandemic negatively influencing many dimensions, the researchers observed a lack of comprehensive HCO performance evaluations. In the same vein, although some resulting dimensions were assessed separately during the pandemic, other dimensions still lack investigation. Last, BSC dimensions may play an essential role in tackling the COVID-19 pandemic. However, further research is required to investigate the BSC implementation effect in mitigating the pandemic consequences on HCO.
Evaluating the health care sector is quite challenging and complex. Unsatisfactory performance can result from long waiting times (WTs), inefficiency, dissatisfactory patients, and health care workers' (HCWs) burnout [1, 2]. Coronavirus disease 2019 (COVID-19) imposed further burdens on the health care system worldwide due to the limited capacity of hospital beds and the increased psychological stress of HCWs during the COVID-19 pandemic [3, 4]. There is still a lack of information that would help health care managers and policymakers in the era of COVID-19 to improve the delivery of health care quality and to learn for the future . Higher pandemic burdens, such as HCW burnout and stress, will rise when health care organizations (HCO) lack plans and preparedness to strengthen their surge capacity and HCW resilience [6, 7].
Researchers have employed different tools for the performance evaluation (PE) of HCO. The most utilized PE tools were the International Organization for Standardization (ISO standards), Malcolm Baldrige National Excellence Model (MBNQA), European Foundation for Quality Management (EFQM) Excellence Model, Singapore Quality Award (SQA), Six Sigma, Data Envelopment Analysis (DEA), Pabon Lasso Model, and Balanced Scorecard (BSC) [8,9,10,11,12].
The World Health Organization (WHO) initiated the Performance Assessment Tool for Quality Improvement in Hospitals (PATH) in 2003. It aimed to develop a framework for the assessment of hospital performance. The resulting dimensions from this project were clinical effectiveness, efficiency, HCW orientation, responsive governance, safety, and patient-centeredness. However, studies have shown that there are still some gaps in this model and issues concerning the dimensions investigated [13, 14]. Additionally, the Organization for Economic Co-operation and Development (OECD) launched the Health Care Quality Indicator (HCQI) project in 2006; it aimed to develop key performance indicators (KPIs) to compare quality in health care at the international level and achieve international benchmarking. This project concluded that health care must be safe, effective, patient-centered, timely, efficient, equitable, acceptable, and accessible [15, 16].
Most of the abovementioned managerial tools mainly focused on the KPIs related to quality, efficiency, productivity, and timeliness dimensions [8,9,10,11,12, 17]. Each of these dimensions is considered a dimension at the internal perspective of the BSC, which consists of four perspectives: the internal process, customer, knowledge and growth perspectives, and financial perspectives . Dimensions are described as collections of homogeneous or related KPIs. They are also referred to as diagnostic related groups (DRGs) , which have been proven to allow performance comparisons across hospitals and positively impact efficiency improvement .
The use of KPIs in the health care system before the pandemic has been beneficial for many reasons. First, the satisfaction rates of patients and HCWs were increased. Second, they lead to better efficiency, effectiveness, and financial performance and adapt to new technologies and ideas. Third, they lead to higher productivity and profitability [20,21,22]. In the pandemic, it is also crucial for HCO to track the performance of KPIs, which could draw faster attention to areas that require rapid responses and strengthening .
Most of the available PE models mainly focus on the internal perspective but lack coverage of the other dimensions or perspectives that are also important. BSC was considered different from the other managerial tools for two reasons. First, it offers a holistic approach to PE since it allows managers to highlight both financial and nonfinancial metrics. Second, the BSC is not only a planning or a PE tool. It is also a strategic managerial tool that assigns KPIs compatible with the HCO strategy [23, 24]. However, other PE tools, such as total quality management (TQM), lack these comprehensive properties . The first generation of the BSC, unveiled by Kaplan and Norton in 1992, involved four perspectives: the financial, customer, internal process, and knowledge and growth perspectives, steered by the organizational vision and strategy . See Fig. 1.
Later, the second generation of BSCs was developed to include strategic maps, in which cause-effect cascades between perspectives or KPIs were inspected . In the third generation of BSCs, a destination statement was incorporated, which evokes where the organization plans to go within a time horizon and the action plans to achieve each targeted objective . In health care, Duke Children's Hospital in the United States of America (USA) was the first to implement the BSC in 1997. Figure 2 represents the strategic map of Duke University's health system. As a result, the hospital converted 11 million American dollar losses into four million profits after four years of implementation . Since then, the BSC has gained increasing attention, and many HCOs in high-income countries and low- and middle-income countries have strategically utilized the BSC to develop their organizations [26,27,28,29,30].
A recent systematic review  proved that BSC implementations were effective in improving the financial performance of HCOs, elevating patient satisfaction rates, and to a lesser extent improving HCW satisfaction rates. Another review  revealed that there had been a lack of engaging stakeholders in BSC implementations, such as engaging patients and HCWs. However, researchers have pointed to the importance of patient and HCW engagement in the process of PE and delivery improvement [27,28,29]. The rest of the BSC reviews [26, 30, 33,34,35,36,37,38,39,40,41,42] focused only on the general narration of the BSC perspectives and subdimensions used. Moreover, none of them summarized the perspectives or dimensions of BSC based on their importance or frequency of use by health care managers. In other words, all the previous systematic reviews lack a systematic methodological categorization of perspectives, dimensions, and KPIs.
In correspondence with this research gap, this review aims at a) finding and recategorizing all the perspectives, dimensions, and KPIs that were employed in BSC implementations for unification purposes, b) ranking dimensions according to their frequency of use by HCO worldwide, and c) ranking dimensions according to their importance from the health care managers perspective.
This systematic review is part of broad research. After assessing the impact of the BSC on stakeholder satisfaction  and before developing instruments to engage stakeholders in BSC implementations, we sought to accomplish the previously mentioned aims to summarize which dimensions were the most frequently used and essential as per health care managers in implementing the BSC. This review was conducted according to the 27-point checklist of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist ; see Appendix (S1).
Data sources, search strategy, and study selection
First, a search strategy was developed for the PubMed database (see Table 1). Then, the strategy design was adapted for the Embase, Cochrane CENTRAL, and Google Scholar databases. Furthermore, an additional search in the Google engine was performed to find gray literature or unpublished papers, including theses and conference abstracts. Additionally, the reference lists of all the eligible articles were checked. The databases were searched from inception until October 2020.
The search strategy was developed by the first, second, and fourth authors; the first two were experts in health care management and BSC, while the fourth author was an expert in systematic reviews and meta-analysis. It was initially developed for the PubMed database. Moreover, the search strategy was developed to include both Medical Subject Headings (MeSH) terms and keywords to widen the search frame and obtain more results. Then, appropriate truncation and relevant indexing terms were used. See Appendix (S2), which shows search strategies in all databases. Afterward, the first author conducted an electronic database search and removed duplicates using the EndNote X9.2 program.
The selection of eligible studies was performed independently by the first and second authors in all steps. Disagreements were resolved by discussion after each step or, if necessary, through arbitration by the fourth author. First, the articles' titles and abstracts were examined to eliminate irrelevant papers between November 2020 and February 2021. Then, full texts were carefully inspected to decide the final papers' inclusion list between February and June 2021. If different KPIs were used in more than one implementation in the same study, each was counted as a different implementation. In comparison, implementations using the same KPIs in other locations or times in the same research were considered one implementation. The authors of studies with no available full texts or with partially reported results were contacted for missing data.
Data extraction process
Data extraction was performed between June and July 2021 and then compared to discuss differences. The following data were extracted from the eligible studies: 1) author/s, 2) year of publication, 3) country of origin, 4) data collection duration, 5) data collection tool, 6) the number of perspectives, 7) the number of KPIs, 8) availability of weights/importance for perspectives or KPIs, and 9) outcome, which is represented in the KPIs that have been used and their weights/importance. The frequency of each KPI used at each implementation was plotted on Microsoft Excel, and the sum was calculated. In addition, the weight/importance assigned for each KPI at each implementation was reported on a scale of 100%. In the case of studies that did not give weights/importance explicitly, each KPI weight/importance was calculated by dividing one by the number of KPIs used in that study to assign an equal weight/importance for each KPI.
Consequently, we computed an average of the weights/importance assigned for each KPI. Next, we performed regrouping and coding for the KPIs to find the frequency of use and the set weights/importance percentages for each dimension. Then, the resulting major and subdimensions were listed and described between August and September 2021.
The research design of eligible studies was extracted directly from the studies. However, if the research design was not explicitly mentioned, we determined it based on the role of the investigator in that study. Specifically, the study was considered observational if the BSC exposures were naturally determined and the investigator had no part. On the other hand, the study was considered experimental if the investigator actively assigned the BSC intervention.
The risk of bias (RoB) assessment was accomplished between August and September 2021 to assess the quality of the included studies. The Risk of Bias in Nonrandomized Intervention Studies (ROBINS-I) tool was used to evaluate the observational and quasi-experimental studies . In comparison, the Cochrane (RoB 2) tool was used for the assessment of randomized controlled trials (RCTs), as per the Cochrane collaboration's guidelines . The RoB was analyzed at the study level and across studies since authors should avoid summarizing the overall RoB as per the Cochrane Handbook guidelines [47, 48].
In the RoB 2 tool, five types of bias were assessed: bias arising from the randomization processes, bias due to deviations from intended interventions, bias due to missing outcome data, bias in the measurement of outcomes, and bias in the selection of the reported results. On the other hand, in the ROBINS-I tool, seven types of bias were assessed: bias due to confounding, bias in the selection of participants in a study, bias in the measurement/classification of interventions/exposures, bias due to deviations from intended interventions/exposures, bias due to missing data, bias in the measurement of the outcomes, and bias in the selection of the reported results.
Using the RoB 2 tool, each type of bias was assessed as low, high, or unclear. While using the ROBINS-I tool, each type of bias was evaluated into five categories: low, moderate, serious, critical, or no information. Figures for RoB were prepared using the ROBVIS (Risk Of Bias VISualization) tool . Last, it was recommended not to advocate quality appraisal as a criterion for inclusion in reviews . Therefore, the authors decided to include all studies in this systematic review regardless of their quality assessment. See Quality assessment below.
A total of 4028 studies resulted from running the search strategy in the four databases. In addition, another three studies were identified through a Google search. Therefore, a total of 4031 studies were included. Duplicates were removed (n = 1046) using the EndNote program, and then the remaining articles were screened based on their titles and abstracts (n = 2985). Irrelevant papers were excluded (n = 2794).
Consequently, the remaining 191 studies were examined by reading the full texts. Among these papers, 22 papers were written in non-English languages, including Spanish, German, French, Chinese, and Persian. A full-text translation was performed for each study to decide whether to include or exclude any of them. As a result of reading the full texts, 158 studies were excluded, and only 33 were eligible for this review, in which 36 full implementations of different BSC designs were actually applied. Table 2 shows a summary of the 36 implementations. Details of the study selection process are shown in the PRISMA flowchart . See Fig. 3.
Language and location
Asia [59,60,61, 64, 65, 67,68,69,70,71, 73,74,75, 78,79,80, 84], seven in North America [51, 53,54,55,56, 58, 62], six in Europe [52, 57, 66, 77, 82], three in Africa [72, 76, 81], and one without location information .
Twenty-one implementations were performed in hospitals (secondary and tertiary HCO) [41, 51,52,53, 55,56,57,58,59,60,61,62, 64, 66, 67, 71, 77, 79, 80, 82] and 15 in medical centers or health facilities (primary HCO) [54, 65, 68,69,70, 72,73,74,75,76, 81, 83, 84].
Two studies [66, 74] included three and two implementations, respectively, with different KPIs per implementation. Thus, the 33 resulting studies contained 36 unique implementations. No BSC implementation in the COVID-19 era was found.
The 36 BSC implementations varied in their designs. However, most studies did not explicitly report their study design. We categorized the 36 implementations based on the active role of the investigator in BSC implementation and the time of data collection. Consequently, one sole study design was an RCT . Moreover, 14 implementation designs were uncontrolled quasi-experiments. Specifically, six implementations had a posttest-only design [52, 60, 71, 78, 81, 82]. Five implementations in four studies had pretest–posttest designs [61, 66, 77, 79]. Finally, three implementations interrupted the time series design [54, 67, 85]. On the other hand, 20 implementations were observational; six implementations in four studies were cross-sectional [53, 73,74,75], one implementation was prospective , ten implementations were retrospective [58, 59, 62, 64, 65, 68,69,70, 76, 84], and two implementations were prospective and retrospective [55, 56]. Finally, one implementation did not have sufficient information or reported the study design .
Some of the resulting studies integrated multiple-criteria decision analysis (MCDA) with BSC. One study  combined BSC with simulation and MCDA techniques with what was referred to as S-MEDUTA. Another study  integrated the BSC with fuzzy analysis. Two studies [67, 78] combined BSC with AHP, and one  used the TOPSIS technique. Studies explained that using these methodologies with the BSC would help them arrive at more informed and better decisions.
Perspectives frequency of use and importance
A total of 797 KPIs were extracted from the resulting implementations. These KPIs were categorized in the studies under 15 perspectives. The average number of perspectives used per study was 4.5, and for the KPIs, it was 22. The most frequently used perspectives were the internal, financial, patient, learning and growth, HCW, managerial, community, and stakeholder perspectives. The total use frequencies of these perspectives at the implementations were 29.6%, 17%, 12.6%, 12.6%, 9.4%, 6.3%, 5%, and 3.1%, respectively. On the other hand, the topmost important perspectives from the health managers' viewpoint were the internal, financial, learning and growth, patient, HCW, community, managerial, and stakeholder perspectives with a total weight/importance of 37.9%, 15.4%, 12%, 11.3%, 7.8%, 7.7%, 3.6%, and 2.8%, respectively.
Categorization and regrouping of KPIs into Dimensions/Subdimensions
The 797 extracted KPIs were plotted according to their frequencies and weights/importance in the categorization process. See Appendix (S3). Grouping and recategorizing KPIs resulted in Figs. 4 & 5 below, showing 13 major dimensions and 45 subdimensions based on their.
Frequency of use and importance, respectively. After regrouping these KPIs into homogenous major dimensions and subdimensions, 13 major dimensions resulted, with 45 subdimensions. A summary of the resulting perspectives and their major and subdimensions contents are illustrated in Fig. 6. The description of each major and subdimensions is described further in Appendix (S4).
Each study was evaluated in terms of RoB, as illustrated in Appendix (S5). The RoB 2 tool was utilized to assess the ROB in the sole RCT study , for which the assessment showed fair evaluation, except for performance bias. On the other hand, utilizing the ROBINS-I tool for assessing the RoB in observational and quasi-experimental studies revealed no information about confounders' adjustment methods except in three studies [51, 59, 67]. The confounding agents were apparent in the three studies; one study  performed confounders adjustments. On the other hand, another  adjusted for patient severity but not for the LOS and mortality rate. Last, one study  did not perform adjustment at all, which may have affected measurement precision.
The selection bias across studies reflected a serious RoB in five studies [59, 63, 67, 73, 80]. Therefore, the intervention and the follow-up did not coincide, and a potentially substantial amount of follow-up was missing in their analysis. Studies with a moderate risk of intervention/exposure measurement bias reflected a well-defined intervention status, but some aspects of the assignments of intervention status were determined retrospectively. Furthermore, bias in selecting the reported results was serious in one study that partially reported the results . Studies that reported all results but did not have a preregistered protocol or whose outcome measurements were not defined in an initial plan were given a moderate risk. See Appendix (S5).
Discussion of the main results
All the perspectives, dimensions, and KPIs employed in BSC implementations were collected to fulfill the research aims. Categorization and regrouping of the KPIs into major and subdimensions were performed. Then, the dimensions were ranked according to their frequency of use and their importance. The BSC tool can offer comprehensive planning, monitoring, evaluation, and improvement of HCO KPIs. Hence, their performance should be improved in the short and long term.
In general, studies had either no information or low or moderate ROB. At the same time, only a few of them had serious or critical ROB. However, studies that had only fully reported BSC indicator measurements were included. Many of them did not have a preregistered protocol or predefined measures in their plan. No information was found regarding confounders or deviation from unintended interventions.
Overall completeness and applicability of evidence
Analyzing the results shows that BSC implementations typically utilized four fundamental perspectives: financial, customer, internal, and knowledge and growth. However, the analysis of Fig. 5 revealed the frequent employment and the importance of other BSC perspectives in BSC implementations. Specifically, the external and managerial perspectives. This reflects the need for slight modifications of BSC design and corresponds with the findings of another study , which referred to the sustainability perspective of the BSC as the fifth pillar. Additionally, our findings reveal that focusing on both internal and external customers from the customer perspective is essential.
The variation among BSC implementations in the categorization of the same KPIs reflects the need for data standardization. HCW training-related KPIs, for example, were categorized under the learning and growth perspective in almost half of the resulting studies [58,59,60, 62, 64,65,66,67,68,69,70,71,72,73,74,75,76,77,78, 84]. Meanwhile, the rest of the studies categorized them under the perspectives of HCW [72, 76], quality , service capacity, provision/service capacity [64, 68, 70, 73, 76, 84], and healthcare facility functionality . These results are consistent with a study  that referred to the lack of defining measures and the lack of data standardization. The differences in categorization prove our assumptions in the calculation imprecision in the previous reviews. Specifically, in the use frequency or the importance of the perspectives and KPIs. Our systematic review solved this calculation bias by uniformly forming the 797 KPI categorizations. Regrouping similar or semisimilar KPIs under the same category resulted in more precise results. Unification of dimensions can guide uniform future implementations of PE or BSC at HCO, allowing data sharing and comparability. Dimension unification can be why our findings are different from another systematic review  that did not consider unifying the classification of KPIs. According to HCO management, the average LOS, HAIs, patient satisfaction, bed occupancy, and bed turnover rate were the most useful KPIs.
Analyzing the results also shows a lack of BSC utilization in HCO during the pandemic. Additionally, there has been a lack of studies comprehensively examining the impact of COVID-19 on KPIs.
This review can guide healthcare managers and researchers since the resulting dimensions can be utilized to synthesize future BSC measurements. Specifically, the dimensions can direct the creation of new instruments to engage stakeholders in future BSC implementations. Moreover, this review can provide a road map for healthcare managers to perform a comprehensive PE of HCO during the COVID-19 pandemic. Since the COVID-19 pandemic may influence the BSC dimensions positively or negatively compared to prepandemic, analyzing the effect of the pandemic on the performance of the major- and subdimensions will allow HCO’s managers to better understand where to focus on their action plans to improve the overall performance of HCOs.
Practical assessment implications of the resulting dimensions in the COVID-19 era
Although this systematic review included ten months after the initiation of the COVID-19 pandemic, no research on BSC utilization in COVID-19 was found. Moreover, health policy experts stated that insufficient standardization of quality measurement approaches in the COVID-19 era challenged sharing purposes. As a result, the comparison between the performance of healthcare systems is disrupted . Comparison is critical in cases where the optimal performance is not fully understood as in pandemics, and a comparison with other health systems would be informative and necessary . Therefore, addressing the lack of data standardization was suggested to be overcomed by quickly defining measures, which could allow health systems, at least in the short term, to use standardized methods to better understand their performance .
We pursued further analysis in this paper based on independent studies per resulting dimension during the COVID-19 era to highlight how these dimensions can be utilized to monitor and improve HCO's performance during the COVID-19 pandemic.
The Financial major dimension
Due to COVID-19 hospitalizations at the beginning of the pandemic, health policy experts suggested that HCO in some regions will have more significant revenue and greater costs related to additional HCW and resources. In contrast, other hospitals will experience mostly sharp reductions in elective and outpatient payments, which will create unprecedented financial challenges for HCO . However, in addition to the higher costs of HCWs and resources, researchers found higher costs of treatment due to extra diagnostic tests and isolation costs .
In the United Kingdom (UK), the total expenditure on the National Health System (NHS) has increased significantly during the pandemic . The NHS made funding upgrades to expand waiting areas and treatment cubicles . Some studies have focused on cost-effectiveness calculations. A study in South Africa indicated that purchasing intensive care unit (ICU) capacity from the private sector during COVID-19 surges may not be a cost-effective investment . To date, there is still a lack of studies that handle the financial dimension or develop cost-saving strategies at the health organization level in COVID-19.
The efficiency and effectiveness major dimension
Analyzing the number of patient visits and admissions in the USA,  revealed a decrease in ER visits, increasing hospital admissions. However, another study in Alberta  perceived decreased admissions and ER visits to the hospital, despite the low volume of COVID-19 hospital admissions.
Many studies have been performed to analyze the efficiency, utilization, and productivity of HCO during the pandemic. A study  indicated that efficient hospitals under normal conditions lost their efficiency during COVID-19 and had to adapt to the new criteria. A systematic review  showed that healthcare utilization decreased by approximately one-third during the pandemic, with more significant reductions among people with less severe illnesses.
A study at an isolation hospital in Egypt  utilized the DEA tool to improve efficiency. This confirmed that the number of nurses and the number of beds impacted the operational efficiency of COVID-19, while the number of physicians had no significant effect on the efficiency. These results are compatible with a study in Mauritius  that found that nurses and beds are the most critical factors in hospital production; that is, a 1% increase in the number of beds and nurses resulted in an increase in hospital outputs by 0.73 and 0.51%, respectively.
The availability and quality of the supplies and services major dimensions
The supply and logistics management dimension was considered an important KPI in tackling COVID-19 . This dimension includes evaluating the availability and quality of COVID-related medications, masks, personal protective equipment (PPE), detergents, medical services, supportive services, etc. Additionally, researchers viewed the availability of both clinical and supportive services at hospitals as essential in responding to the COVID-19 pandemic and the flow of COVID-positive patients . The spectrum of supportive services to a hospital encompasses linen and laundry, diet, central sterile supply department, transport, consumables in large quantities at hospital stores, mortuary, and engineering services . Some of the essential items were filtering face-piece respirators or N95 respirators and the availability of PPE kits . The global challenge during this pandemic in terms of inadequate availability of PPE in HCO highlighted the vital role of CSSD. Centers for Disease Control and Prevention (CDC) suggested a method of decontamination, and reuse of filtering face respirators to overcome the shortage of these respirators is their extended use or reuse .
However, researchers have referred to the lack of studies on the quality of supplies and services at HCO in COVID-19 . Lack of studies can be referred to as data lag in pandemics: the time between care provision and quality measurement reporting . Policymakers suggested that measures should be less reliant on claims data, which by nature have a time lag, and focus on actions that can be generated from the electronic health record (EHR)  investigated during the pandemic.
The HCW scientific development major dimension
Due to its importance, many studies have aimed to evaluate HCW knowledge, attitudes, and practices (KAP) at the beginning of the pandemic . HCW adherence to IC measures is affected by their KAP toward COVID-19 . Some studies referred to insufficient knowledge about COVID-19 among nurses . Surgeons were worried about losing their skills after months of lockdown due to paused practice . However, HCWs were obliged to learn digital health skills and effectively communicate with patients during the pandemic .
A study  found that COVID-19 and non-COVID-19 publication productivity correlates with some factors. For example, epidemiologic, healthcare system-related, and pre-COVID publication expertise factors. Therefore, countries with a stable scientific infrastructure appear to maintain non-COVID-19 publication productivity nearly per year. More incentives must be drawn by HCOs to their HCWs to encourage research and scientific productivity related to COVID-19.
The managerial tasks and PE dimension
Standard policies, procedures, the availability of written standardized guidelines, and delivery in full and on time were considered essential in tackling COVID-19 . A lack of standardization capability and conflicting or irrational managerial decisions were deemed dissatisfactory factors for HCWs in the pandemic .
Planning and preparedness are also crucial managerial tasks. The CDC developed a checklist to help hospitals assess and improve their preparedness for responding to COVID-19 . Hospitals utilized a collection of some of the previously explained KPIs and dimensions to perform planning and internal assessment of their performance [106, 107].
Few studies [108, 109] have examined centralized governance's impact on HCO during the pandemic, which positively affected reactive strategies. Learning from past pandemics also positively influences proactive and reactive strategies. However, the role of PE internally, such as using BSC or MBNQA tools, or external assessments, such as Joint Commission International (JCI) accreditations, ISO certification, auditing, or peer review on HCO during the pandemic, still requires more investigation.
The HCW-centredness major dimension
Physicians referred to the importance of reliable acknowledgment and motivation both emotionally and financially, considering the sacrifices they provide every day . In parallel, staffing and recruitment of an adequate number of medical and nonmedical HCWs were considered important KPIs for the PE of HCO at COVID-19 . In the UK, the NHS employed strategies to facilitate the staffing process due to the shortage of HCWs. First, newly qualified/final year medicine and nursing students were deployed. Second, the return of the former HCW was made .
The HCW satisfaction rate and burnout have been evaluated in many studies during the pandemic. A study  showed that the physicians' burnout prevalence was 57.7% during the pandemic, which is considered high. HCWs who lack PPE reported lower occupational satisfaction than those who did not [103, 111]. HCW accomplishments during the pandemic were positively associated with higher occupational satisfaction rates . Therefore, emphasizing HCW accomplishments leads to increased satisfaction rates.
Moreover, as mentioned earlier, better performance of the communication dimension, including psychological support, will raise HCW job satisfaction and lower the rates of burnout and stress [111, 112]. Some HCWs felt anxiety and fear mainly due to the possibility of transmitting the virus to their family members and the elderly living in their house . A study in Canada  showed that HCW training and counseling services were perceived as helpful in reducing HCW stress. Despite that, they were underutilized in HCO.
On the other hand, although most nurses had to increase their workload due to staff shortages, a study  found that the elevation of the workload was not associated with lower occupational satisfaction. Additionally, another study in Singapore  found that HCW burnout was similar to the prepandemic rates. Nevertheless, the HCW vaccination, engagement, motivation, teamwork, and loyalty subdimensions and their impact are still not good.
The time major dimension
An "extra layer of processes" was added due to the donning and doffing protocols and cleaning requirements, which slowed all the operational processes down and increased the time required to accomplish serving the medical care to patients . Patient WT was also influenced. In the UK, WT reached high levels in studies with a notable impact on elective surgery. The number of patients who waited for more than a year to receive NHS treatment in July 2020 was 81-fold greater than the previous year's number .
Moreover, the patient length of stage (LOS) also increased for another 2–3 days. A reason for this was the delays in COVID-19 testing results . The LOS in the USA was two days more than that in Italy and five days less than that in Germany . A systematic review for patient LOS in COVID-19  concluded that LOS in China was longer than that in any other country, referring to differences in criteria for admission and discharge and different timings within the pandemic. Another study  found a negative association between the LOS and the case fatality rate. Therefore, LOS estimation can be introduced as a KPI to scale the success of the countries fighting the ongoing pandemic.
Moreover, LOS provides insights into when hospitals will reach capacity and predicts associated HCW or equipment requirements . Discharge status should be considered when analyzing LOS since patients who are discharged alive have a longer LOS than those who died during their admission . Hospitals reported that health insurance plans resisted paying for additional patient days in the hospital while awaiting COVID-19 test results .
However, complying with the CDC guidance on testing and disposition of patients was suggested to reduce the patient LOS, freeing up hospital beds for incoming COVID-19 patients . Another study in the UK  indicated that due to the complexity and partiality of different data sources and the rapidly evolving nature of the COVID-19 pandemic, it is most recommended to use multiple LOS analysis method approaches on various datasets.
A combination of an accelerated failure time survival model and a truncation corrected method with the multistate survival model was found to be helpful in epidemic planning and management. Finally, the findings of a cohort study  concluded that a multimechanism approach effectively decreased the average LOS in the ICU by 5.4 days and up to nine days in older patients. This finding suggests that implementing this treatment protocol could allow a healthcare system to manage 60% more COVID-19 patients with the same number of ICU beds.
Major error-free and safety major dimension
This dimension includes monitoring, analyzing, and comparing mortality rates and investigating its determinants in HCO. Although mortality may not be directly related to errors, mortality rates higher than the average can reveal an underlying mistake. A cohort study in Mexico City  found that the mortality rates at the hospital's ICU and non-ICU departments were similar. The reason behind this finding was the ICU bed's unavailability. Approximately 45% of the patients who did not survive did not receive an ICU bed, which raised the mortality rate in the non-ICU admitted patients. However, this study revealed that the leading cause of non-ICU admission was acute respiratory distress syndrome. The leading cause of mortality for admitted patients was septic shock, followed by ARDS and multiorgan failure.
The WHO has provided clear guidelines for infection control (IC) during healthcare when COVID-19 is suspected or confirmed . Patient safety was investigated in a systematic review of Indian-related studies . Patient safety was negatively impacted during the COVID-19 pandemic due to inadequate preparation of the healthcare system, such as infrastructure and human and material resources. Additionally, researchers categorized diagnostic errors that could occur in the COVID-19 pandemic into eight types and suggested how to reduce them .
However, many studies have shown improvements in this dimension during the pandemic. A study in the UK  found a significant increase in the safety attitude questionnaire scores of doctors and other clinical HCWs and no change in the nursing group. It also showed a significant decrease in error reporting after the onset of the COVID-19 pandemic. Another study in Iran  found that health-associated infections (HAIs) during the pandemic were reduced, which could be referred to as the proper implementation of IC protocols. This finding is supported by a study in Ghana , which found that HCW compliance with IC measures was high during the pandemic.
The health waste (HW) management subdimension was intensively investigated due to the tremendous increase in HW volume during the pandemic . A study in Iran  indicated that infectious waste increased by 121% compared with before the pandemic. Direct exposure of HCWs to virus-contaminated waste with inadequate safety measures and mismanagement of HWs may lead to their infection and facilitate the transmission of COVID-19 [124, 126]. The WHO has provided clear guidelines for managing healthcare waste during the pandemic . Nevertheless, many studies worldwide [126, 128, 129] illustrated the existence of gaps and a flawed system for handling HWs during the pandemic.
A mini-review  of HWs during the pandemic showed that disinfecting waste, followed by proper segregation and on-site treatment, can also provide better and healthier HW management. It also revealed that surplus HW accommodation, mobile treatment, and temporary storage strategies might aid the sustainable management of healthcare waste without further spreading the virus. Another study in Brazil  proposed a model for the proper management of HWs. It focused not only on the operational management KPIs of HWs but also on environmental management, such as sustainable practices. Moreover, it highlighted the importance of employee training on HW guidelines since HW management was not considered an essential competence or a priority for every HCO.
The patient-centredness major dimension
Many studies have been conducted to evaluate patient satisfaction. A study  indicated no difference in patient satisfaction during the period spent in the emergency room before and during the pandemic. Another study  showed positive patient experience and satisfaction rates in Saudi Arabia's largest institutions during the pandemic. Moreover, many studies have focused on the psychological assessment of the impact of COVID-19 on the population in general. However, few studies have focused on specifically assessing the psychological effects on patients. For example, a study  found that COVID-19 patients with low education levels and females who have undergone divorce or bereavement tended to have a high prevalence of adverse psychological events. Another study  found that the psychological consequences of the pandemic were better handled by cancer patients 65 years of age or older, while younger cancer patients were more psychologically affected. Early psychological status identification and intervention should be conducted to avoid extreme events such as self-mutilating or suicidal impulsivity for patients . Patient complaints and loyalty assessment during the pandemic and the psychological impact of COVID-19 on non-COVID-19 patients still need more investigation.
The responsiveness and communication major dimension
The main goal of HCO was considered to provide high-quality care to patients and meet their needs and expectations during an outbreak such as COVID-19 . Moreover, dialog and listening to health demands in COVID-19-suspected patients was highlighted as the foremost step in the flows of care and guidance .
Communication among HCWs was also highlighted. A study  considered HCW reception of family support, colleagues, support, clear communication, and COVID-19 information as the most valuable resources in the pandemic. Lower HCW psychological distress symptoms, burnout, and intentions to quit were perceived when these communication resources were more available. Another study  indicated that gratitude in communication could reduce depression in HCWs by promoting social support and hope.
In addition, communication between HCWs and patients was also investigated during the pandemic. A study in Jordan  found that physician–patient communication positively affected the patient's psychological status in COVID-19. It recommended avoiding communication errors using jargon, not being available to patients, and not showing empathy in communication. Additionally, it emphasized the benefit of physicians as excellent listeners to patients. However, HCW-patient communication faced few obstacles during the pandemic. The protective equipment used by HCWs in the pandemic could have imposed a barrier to effective communication or eye contact with them . Some pediatricians reported difficulty communicating with families and following up with patients, especially newly discharged neonates and infants, using the telephone .
However, more research is still needed to improve and evaluate patient education programs, patient guidelines, counseling and consultation services, and communication skills between HCWs and patients during the pandemic.
Community care and reputation major dimension
The external major dimension, including the social sustainability indicators for HCOs facing a crisis, can be ambiguous to define and apply, so SSIs have been organized under the broad categorical concerns of well-being, values, agency, and inequality . Despite the doctor–patient confidentiality clause and the protection law for patient data privacy, the Department of Health and Social Care for England has relaxed the rules on sharing confidential patient data. It required HCO and the NHS to exchange patient information to help fight COVID-19 . Moreover, COVID-19 patient data led to society breaching patient privacy in some countries [103, 138], which may have stigmatized those patients .
As mentioned earlier, a study  emphasized the importance of sustainable environmental practices for better HW management. The political situation was also considered an external influence during the pandemic. It was highlighted in a study  in the Palestinian territories, which referred to the COVID-19 situation in the presence of the Israeli military occupation to have a double epidemic effect, which eventually impacted the performance of the Palestinian health system HCO during the pandemic.
However, factors as exemptions offered by HCO for poor patients, social responsibility, patient privacy concerns, and HCO market shares in COVID-19 are still poorly investigated.
The HCO building major dimension
Design and infrastructure preparation were considered essential dimensions in some HCOs during the pandemic . Healthcare systems made adaptations in HCO buildings after the COVID-19 pandemic. Examples include expanding waiting areas, increasing ICU capacity, establishing isolation areas, and building new hospitals . In the UK, the NHS temporarily used private hospitals to provide public care, increasing the number of beds, ventilators, and all HCW categories. Moreover, nonhospital sites were temporarily turned into hospitals . However, researchers did not sufficiently investigate the ease of access to HCO during the pandemic.
The technology and information system major dimension
Experts emphasized the role of technology and information (TI) in tackling COVID-19 as inevitable due to its importance in the response, prevention, preparedness, and recovery phases [140, 141]. TI system application varies from allowing HCO to maintain and share studies to producing different reports and follow-up with pandemic analysis. Telehealth is another example that proved helpful in the pandemic. It allowed HCWs to provide care for patients without direct physical contact, especially to patients at quarantine, while keeping them safe .
Researchers summarized the emerging technologies used to mitigate the threats of COVID-19 in the following categories: artificial intelligence/deep learning, big data analytics, high-performance computing infrastructures, robots, 3D printing technology, digital contact tracing technology, blockchain , bioinformatics systems, telemedicine, mobile phone, decision support system, the IC system in HCO, online interactive dashboard/geographic information system, Internet of Things, virtual reality, surveillance systems, and internet search queries [140, 141]
Governments, healthcare systems, and HCO need to keep updated with the emerging technologies in this field, allocate resources to invest in them, and develop the required skills in HCWs to utilize them properly.
Strengths and weaknesses
We believe that this paper has several strengths. First, this systematic review includes all types of studies with BSC implementations, such as books, theses, conference papers, and letters to the editor. Second, this review contains all implementations despite the country, language, or HCO administrative type, which gives an advantage of generalizing HCO results worldwide. Third, unlike other BSC reviews [35, 37], which included definitions of biobanks, pharmacies, laboratories, radiology, and medical colleges in HCO, this review limited the report to primary, secondary, or tertiary healthcare organizations. However, an initial assessment by top management to evaluate the importance of each dimension and KPI based on the health organizations' strategy could be needed. This strategy leads to the homogeneity of the resulting studies and leads to more valid comparisons among the results. Fourth, this review calculates the use frequency of perspectives and the weights/importance assigned to them. Fifth, the first review has uniform KPIs in homogenous major dimensions and subdimensions despite the categorization differences among implementations, yielding more precise results. The resulting KPIs and dimensions in this review can be generalized or replicable to other HCOs and hospitals. Finally, this study is the first to analyze the implications of BSCs in HCO during the pandemic based on the literature. This implication provides a guide for future theoretical implications, such as performing systematic reviews for each major dimension during the pandemic. It also provides a guide for practical implications of BSC dimensions to assess HCOs’ performance.
On the other hand, this systematic review has some limitations. First, unlike previous studies, it excludes some HCOs, such as laboratories, pharmacies, radiology departments, and biobanks, as specified in the inclusion/exclusion criteria. Therefore, our results cannot be generalized to such HCO types. However, we excluded them to arrive at more homogenous KPIs and dimensions that are directly related to HCO that offer primary, secondary, and tertiary medical services. Second, it includes only the articles that report the complete implementations of BSC while excluding studies that display only the BSC design without reporting the full implementation results. Third, we extracted the KPIs from all resulting implementations despite their RoB. However, we included an ROB assessment for each implementation.
In conclusion, our review shows that the most frequently used perspectives in BSC papers were internal, financial, patient, learning and growth, HCW, managerial, community, and stakeholder perspectives. The perspectives that had the highest importance were internal, financial, learning and growth, patient, HCW, community, managerial, and stakeholder.
Moreover, this review solves the dilemma of the KPI categorization difference between BSC implementations by dimension unification into 13 major dimensions. The financial, efficiency and effectiveness, availability and quality of supplies and services, managerial tasks, HCW scientific development, error-free and safety, time, HCW-centeredness, patient-centeredness, technology and information system, community care and reputation, HCO building, and communication. The proper utilization of the 13 major dimensions and the 45 subdimensions will serve as a planning, monitoring, evaluation, and continuous improvement tool for HCO, resulting in performance augmentation.
This research showed a lack of BSC utilization and any holistic PE approach in HCO during the COVID-19 pandemic. Additionally, some dimensions that are essential for PE are still poorly investigated. Our analysis reflects that most KPIs were negatively affected during the pandemic, except IC and safety measures, which improved in some cases. However, a comprehensive PE of HCO during the COVID-19 pandemic worldwide is still needed. Therefore, we recommend that future researchers perform a comprehensive practical PE for HCO during COVID-19 using the measurements of the resulting dimensions. This analysis will provide a better understanding of the causal relationships between dimensions. It will also allow comparability of the interventions' outcomes, which will boost the performance and mitigate the consequences of the pandemic on HCO. Moreover, researchers are encouraged to perform systematic reviews for each dimension, especially those that are already well investigated and the investigation of dimensions that are still poorly investigated but essential for PE. This theoretical implication will lead to performance enhancement and mitigate the consequences of the pandemic on HCO.
Availability of data and materials
All data generated or analyzed during this study are included in this published article [and its supplementary information files].
Centers for Disease Control and Prevention
Coronavirus Disease 2019
Data Envelopment Analysis
Diagnostic Related Groups
European Foundation for Quality Management
Health Care Organizations
Health Care Quality Indicator
Health Care Workers
Health Management Information System
Intensive Care Unit
International Organization for Standardization
Joint Commission International
Knowledge, Attitude, and Practices
Key Performance Indicators
Length Of Stay
Malcolm Baldrige National excellence model
Management by Objectives
Multiple-criteria decision analysis
Medical Subject Headings
National Health System
Organization for Economic Co-operation and Development
Performance Assessment Tool for quality improvement in hospital
Personal protective equipment
Preferred Reporting Items for Systematic Reviews and Meta-Analyses
Return on Assets
The Risk of Bias
Return On Investment
Safety Attitude Questionnaire
Singapore Quality Award
Technology and information
Total Quality Management
United States of America
World Health Organization
Meena K, Thakkar J. Development of balanced scorecard for healthcare using interpretive structural modeling and analytic network process. J Adv Manag Res. 2014;11:232–56.
Buathong S, Bangchokdee S. The use of the performance measures in Thai public hospitals. Asian Rev Account. 2017;25:472–85.
Amer F, Hammoud S, Farran B, Boncz I, Endrei D. Assessment of countries’ preparedness and lockdown effectiveness in fighting COVID-19. Disaster Med Public Health Prep. 2021;15:e15-22. https://doi.org/10.1017/dmp.2020.217.
Lou NM, Montreuil T, Feldman LS, Fried GM, Lavoie-Tremblay M, Bhanji F, et al. Evaluations of healthcare providers’ perceived support from personal, hospital, and system resources: implications for well-being and management in healthcare in Montreal, Quebec, during COVID-19. Eval Health Prof. 2021;44:319–22. https://doi.org/10.1177/01632787211012742.
Austin JM, Kachalia A. The state of health care quality measurement in the era of COVID-19. JAMA. 2020;324:333. https://doi.org/10.1001/jama.2020.11461.
Ali A, Owied MM, Virginia P, Ruth E, Abdulellah AT. Using logic model and key performance indicators to construct a field hospital for Coronavirus pandemic cases. BMC Emergency Medicine. 2020;:12–4. doi:https://doi.org/10.21203/rs.3.rs-102321/v1.
Afulani PA, Gyamerah AO, Nutor JJ, Laar A, Aborigo RA, Malechi H, et al. Inadequate preparedness for response to COVID-19 is associated with stress and burnout among healthcare workers in Ghana. PLoS One. 2021;16 4 April:1–20. doi:https://doi.org/10.1371/journal.pone.0250294.
Andersen HV, Lawrie G, Savič N. Effective quality management through third-generation balanced scorecard. Int J Product Perform Manag. 2004;53:634–45. https://doi.org/10.1108/17410400410561259.
Ruiz U, Simon J. Quality management in health care: a 20-year journey. Int J Health Care Qual Assur. 2004;17:323–33. https://doi.org/10.1108/09526860410557570.
Hojabri R, Manafi M, Eftekhar F, Ghassemzadeh H. Effective methods for health care organizations: an evaluation of excellence models. African J Bus Manag. 2013;7:2665–75.
Klassen A, Miller A, Anderson N, Shen J, Schiariti V, O’Donnell M. Performance measurement and improvement frameworks in health, education and social services systems: a systematic review. Int J Qual Heal Care. 2010;22:44–69. https://doi.org/10.1093/intqhc/mzp057.
Sheykholeslam S, Sachin V. A study of performance evaluation models in health care sector with special reference to EFQM and BSC models. Int J Multidiscip Res Dev. 2015;2:378–85.
Veillard J, Champagne F, Klazinga N, Kazandjian V, Arah OA, Guisset AL. A performance assessment framework for hospitals: The WHO regional office for Europe PATH project. Int J Qual Heal Care. 2005;17:487–96.
Carini E, Gabutti I, Frisicale EM, Di Pilla A, Pezzullo AM, de Waure C, et al. Assessing hospital performance indicators. What dimensions? Evidence from an umbrella review. BMC Health Serv Res. 2020;20:1–13.
Busse R, Klazinga N, Panteli D, Quentin W, World Health Organisation. Improving healthcare quality in Europe: Characteristics, effectiveness and implementation of different strategies. Improving Healthcare Quality in Europe. 2019;:419. https://apps.who.int/iris/bitstream/handle/10665/327356/9789289051750-eng.pdf?sequence=1&isAllowed=y. Accessed 25 Jun 2021.
Kelley E. Hurst J. Health care quality indicators project: Conceptual framework paper. Oecd Heal Work Pap; 2006. p. 1–37.
Goshtasebi A, Vahdaninia M, Gorgipour R, Samanpour A, Maftoon F, Farzadi F, et al. Assessing hospital performance by the Pabon Lasso model. Iran J Public Health. 2009;38:119–24.
Kaplan R, Norton D. The balanced scorecard–measures that drive performance. Harv Bus Rev. 1992;70:71–9.
Nundoochan A. Improving public hospital efficiency and fiscal space implications: the case of Mauritius. Int J Equity Health. 2020;19:1–16.
Aghaei Hashjin A, Kringos DS, Manoochehri J, Aryankhesal A, Klazinga NS. Development and impact of the Iranian hospital performance measurement program. BMC Health Serv Res. 2014;31:280–8.
De Vos M, Graafmans W, Kooistra M, Meijboom B, Van der Voort P, Westert G. Using quality indicators to improve hospital care: a review of the literature. Int J Qual Heal Care. 2009;21:119–29.
Shaw C, Bruneau C, Baskia K, Jong DG, Sunol R. How can hospital performance be measured and monitored? Copenhagen, WHO Regional Office for Europe (Health Evidence Network report; http://www.euro.who.int/document/e82975.pdf, Accessed 9 Jan 2021).
Olve, Nils-Göran; Roy, J.; Wetter M. Performance drivers : a practical guide to using the balanced scorecard. New York: John Wiley and Sons; 1999.
Speckbacher G, Bischof J, Pfeiffer T. A descriptive analysis on the implementation of Balanced Scorecards in German-speaking countries. Manag Account Res. 2003;14:361–88.
Meliones J. Saving money, saving lives. Harv Bus Rev. 2000;78:57–62 64, 66–7.
Bohm V, Lacaille D, Spencer N, Barber CE. Scoping review of balanced scorecards for use in healthcare settings: development and implementation. BMJ open Qual. 2021;10:1–10. https://doi.org/10.1136/bmjoq-2020-001293.
Gagliardi AR, Lemieux-Charles L, Brown AD, Sullivan T, Goel V. Barriers to patient involvement in health service planning and evaluation: an exploratory study. Patient Educ Couns. 2008;70:234–41. https://doi.org/10.1016/j.pec.2007.09.009.
Anderson NN, Baker GR, Moody L, Scane K, Urquhart R, Wodchis WP, et al. Approaches to optimize patient and family engagement in hospital planning and improvement: qualitative interviews. Heal Expect. 2021;24:967–77. https://doi.org/10.1111/hex.13239.
Korlén S, Richter A, Amer-Wåhlin I, Lindgren P, von Thiele SU. The development and validation of a scale to explore staff experience of governance of economic efficiency and quality (GOV-EQ) of health care. BMC Health Serv Res. 2018;18:963.
Zelman WN, Pink GH, Matthias CB. Use of the balanced scorecard in health care. J Health Care Finance. 2003;29:1–16.
Duke-Children’s-Hospital. Duke Children ’ s Hospital Case Abstract. Balanced Scorecard Collaborative Hall of Fame. :1–13. https://thepalladiumgroup.com/download?file=AED_1551102784_116attachment_dubai_boot_camp_-_march_2019_opt.pdf%23Dubai_Boot_Camp_-_March_2019_opt.pdf. Accessed 15 May 2021.
Amer F, Hammoud S, Khatatbeh H, Lohner S, Boncz I, Endrei D. The deployment of balanced scorecard in health care organizations: is it beneficial? A systematic review. BMC Health Serv Res. 2022;22:1–14. https://doi.org/10.1186/s12913-021-07452-7.
Rouis S, Abdelaziz AB, Nouira H, Khelil M, Zoghlami C, Abdelaziz AB. Elaboration d’un tableau de bord stratégique pour le monitorage de la performance hospitalière dans les pays du gran. Tunisie Medicale. 2018;96:774–88.
Colbran R, Ramsden R, Stagnitti K, Toumbourou JW. Advancing towards contemporary practice: a systematic review of organisational performance measures for non-acute health charities. BMC Health Serv Res. 2019;19:132.
Rabbani F, Jafri SMW, AbbasPappas FG, Brommels M, Tomson G. Reviewing the application of the balanced scorecard with implications for low-income health settings. J Healthc Qual. 2007;29:21–34.
Gurd B, Gao T. Lives in the balance: an analysis of the balanced scorecard (BSC) in healthcare organizations. Int J Product Perform Manag. 2007;57:6–21.
Mcdonald B. A review of the use of the balanced scorecard in healthcare. BMcD Consult. 2012;2012:1–32.
Trotta A, Cardamone E, Cavallaro G, Mauro M. Applying the Balanced Scorecard approach in teaching hospitals: a Literature review and conceptual framework. Int J Health Plann Manage. 2013;28:181–201.
Behrouzi F, Shaharoun AM, Ma’Aram A. Applications of the balanced scorecard for strategic management and performance measurement in the health sector. Aust Heal Rev. 2014;38:208–17.
Broccardo L. The balance scorecard implementation in the Italian health care system: some evidences from literature and a case study analysis. J Health Manag. 2015;17:25–41.
Gao T, Gurd B. Meeting the challenge in performance management: the diffusion and implementation of the balanced scorecard in Chinese hospitals. Health Policy Plan. 2014;30:234–41.
Gonzalez-Sanchez MB, Broccardo L, Martins Pires AM. The use and design of the BSC in the health care sector: a systematic literature review for Italy, Spain, and Portugal. Int J Health Plann Manage. 2018;33:6–30.
Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JPA, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009;6:e1000100.
Methley AM, Campbell S, Chew-Graham C, McNally R, Cheraghi-Sohi S. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res. 2014;14:579.
Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919. https://doi.org/10.1136/bmj.i4919.
Higgins JP, Savović J, Page MJ, Elbers RG SJ. Chapter 8: Assessing risk of bias in a randomized trial. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated July 2019). Cochrane. 2019.
Higgins JP, Altman DG. Assessing risk of bias in included studies. In: Cochrane handbook for systematic reviews of interventions. Chichester, UK: John Wiley & Sons, Ltd; 2008. p. 187–241. doi:https://doi.org/10.1002/9780470712184.ch8.
Babic A, Vuka I, Saric F, Proloscic I, Slapnicar E, Cavar J, et al. Overall bias methods and their use in sensitivity analysis of Cochrane reviews were not consistent. J Clin Epidemiol. 2020;119:57–64. https://doi.org/10.1016/j.jclinepi.2019.11.008.
McGuinness LA, Higgins JPT. Risk-of-bias VISualization (robvis): An R package and Shiny web app for visualizing risk-of-bias assessments. Res Synth Methods. 2021;12:55–61. https://doi.org/10.1002/jrsm.1411.
Whittemore R, Knafl K. The integrative review: updated methodology. J Adv Nurs. 2005;52:546–53. https://doi.org/10.1111/j.1365-2648.2005.03621.x.
Pink GH, McKillop I, Schraa EG, Preyra C, Montgomery C, Baker GR. Creating a balanced scorecard for a hospital system. J Health Care Finance. 2001;27:1–20.
Zbinden AM. Introducing a balanced scorecard management system in a university anesthesiology department. Anesth Analg. 2002;95:1731–8.
Griffith RJ, Alexander AJ. Measuring comparative hospital performance. J Healthc Manag. 2002;47:41–57.
Biro LA, Moreland ME, Cowgill DE. Achieving excellence in veterans healthcare- a balanced scorecard approach. J Healthc Qual. 2003;25:33–9.
Smith H, Kim I. Balanced scorecard at Summa health system. J Corp Account Financ. 2005;16:65–72.
Devitt R, Klassen W, Martalog J. Strategic management system in a healthcare setting–moving from strategy to results. Healthc Q. 2005;8:58–65.
Martínez-Pillado M, Seco JM, Suárez A, González-Sanjuán R, Vázquez C. Hacia el cuadro de mando integral por el cuadro de mando posible. Rev Calid Asist. 2006;21:13–9.
Goodspeed SW. Metrics help rural hospitals achieve world-class performance. J Healthc Qual. 2006;28:28–33.
Yang M-C, Tung Y-C. Using path analysis to examine causal relationships among balanced scorecard performance indicators for general hospitals. Health Care Manage Rev. 2006;31:280–8. https://doi.org/10.1097/00004010-200610000-00003.
Lin QL, Liu L, Liu HC, Wang DJ. Integrating hierarchical balanced scorecard with fuzzy linguistic for evaluating operating room performance in hospitals. Expert Syst Appl. 2013;40:1917–24.
Chen XY, Yamauchi K, Kato K, Nishimura A, Ito K. Using the balanced scorecard to measure Chinese and Japanese hospital performance. Int J Health Care Qual Assur. 2006;19:339–50.
Josey C, Kim I. Implementation of the balanced scorecard at Barberton citizens hospital. J Corp Account Financ. 2008;19:57–63.
Chang W, Tung Y, Huang C. Performance improvement after implementing the balanced scorecard: a large hospital’s experience in Taiwan. Total Qual Manag Bus Excell. 2008;19:1143–54.
Chang W-C, Tung Y-C, Huang C-H, Yang M-C. Performance improvement after implementing the balanced scorecard: a large hospital’s experience in Taiwan. Total Qual Manag Bus Excell. 2008;19:1143–54. https://doi.org/10.1080/14783360802323560.
Hansen PM, Peters DH, Niayesh H, Singh LP, Dwivedi V, Burnham G. Measuring and managing progress in the establishment of basic health services: the Afghanistan Health Sector Balanced Scorecard. Int J Health Plann Manage. 2008;23:107–17.
Lupi S, Verzola A, Carandina G, Salani M, Antonioli P, Gregorio P. Multidimensional evaluation of performance with experimental application of balanced scorecard: a two year experience. Cost Eff Resour Alloc. 2011;9:1–5.
Chu H, Wang C. A study of a nursing department using the balanced scorecard and the performance measurement system: analytic hierarchy process. Nurs Econ. 2009;27:401–8.
Edward A, Kumar B, Kakar F, Salehi AS, Burnham G, Peters DH. Configuring balanced scorecards for measuring health system performance: evidence from 5 years’ evaluation in Afghanistan. PLoS Med. 2011;8:e1001066.
Chen HF, Hou YH, Chang RE. Application of the balanced scorecard to an academic medical center in Taiwan: the effect of warning systems on improvement of hospital performance. J Chinese Med Assoc. 2012;75:530–5.
Khan MM, Hotchkiss RD, Dmytraczenko T, Zunaid AK. Use of a balanced scorecard in strengthening health systems in developing countries: an analysis based on nationally representative Bangladesh health facility survey. Int J Health Plann Manage. 2013;28:202–15.
Ajami S, Ebadsichani A, Tofighi S, Tavakoli N. Medical records department and balanced scorecard approach. J Educ Health Promot. 2013;2:1–6.
Mutale W, Stringer J, Chintu N, Chilengi R, Mwanamwenge MT, Kasese N, et al. Application of balanced scorecard in the evaluation of a complex health system intervention: 12 months post intervention findings from the BHOMA Intervention : a cluster randomised trial in Zambia. PLoS ONE. 2014;9:1–10.
Rowe JS, Natiq K, Alonge O, Gupta S, Agarwal A, Peters DH. Evaluating the use of locally-based health facility assessments in Afghanistan: a pilot study of a novel research method. Confl Health. 2014;8:1–9.
Edward A, Osei-bonsu K, Branchini C, Yarghal T, Arwal SH, Naeem AJ. Enhancing governance and health system accountability for people centered healthcare: an exploratory study of community scorecards in Afghanistan. BMC Health Serv Res. 2015;15:1–15.
Rabbani F, Pradhan NA, Zaidi S, Azam SI, Yousuf F. Service quality in contracted facilities. Int J Health Care Qual Assur. 2015;28:520–31.
Teklehaimanot HD, Teklehaimanot A, Tedella AA, Abdella M. Use of balanced scorecard methodology for performance measurement of the health extension program in Ethiopia. Am J Trop Med Hyg. 2016;94:1157–69. https://doi.org/10.4269/ajtmh.15-0192.
Catuogno S, Arena C, Saggese S, Sarto F. Balanced performance measurement in research hospitals: the participative case study of a haematology department. BMC Health Serv Res. 2017;17:1–11.
Gao H, Chen H, Feng J, Qin X, Wang X, Liang S, et al. Balanced scorecard-based performance evaluation of Chinese county hospitals in underdeveloped areas. J Int Med Res. 2018;46:1947–62. https://doi.org/10.1177/0300060518757606.
Ebrahimpour A, Alimohammadzadeh K, Maher A. Evaluating the health reform plan in Ayatollah Taleghani Hospital, Tehran in 2010–2017 using the balanced scorecard. Hakim Heal Sys Res. 2019;22:100–11.
Widyasari NLG, Adi NR. Balanced scorecard implementation in the government hospital. Russ J Agric Socio-Economic Sci. 2019;93:285–91.
Mabuchi S, Alonge O, Tsugawa Y, Bennett S, Bennett S. Measuring management practices in primary health care facilities – development and validation of management practices scorecard in Nigeria. Glob Health Action. 2020;13:1763078.
Manolitzas P, Grigoroudis E, Christodoulou J, Matsatsinis N. S-MEDUTA: Combining balanced scorecard with simulation and MCDA techniques for the evaluation of the strategic performance of an emergency department. In: Vlamos P, editor. GeNeDis 2018 computational biology and bioinformatics. Switzerland: Springer; 2020. p. 1–22.
Gonzalez J, Jaber J, Barker PC, Markham MJ, Kittelson SM. Palliative care scorecard: an update for the outpatient setting. J Palliat Med. 2020;23:306–7.
Peters DH, Noor AA, Singh LP, Kakar FK, Hansen PM, Burnhama G. A balanced scorecard for health services in Afghanistan. Bull World Health Organ. 2007;85:146–51.
Olivas-Martínez A, Cárdenas-Fragoso JL, Jiménez JV, Lozano-Cruz OA, Ortiz-Brizuela E, Tovar-Méndez VH, et al. In-hospital mortality from severe COVID-19 in a tertiary care center in Mexico City; causes of death, risk factors and the impact of hospital saturation. PLoS One. 2021;16(2):1–17.
Baker GR, Pink GH. A Balanced Scorecard for Canadian Hospitals. Healthc Manag Forum. 1995;8:7–13.
Kalender ZT, Vayvay Ö. The fifth pillar of the balanced scorecard: sustainability. Procedia - Soc Behav Sci. 2016;235:76–83.
Rahimi H, Khammar-nia M, Kavosi Z, Eslahi M. Indicators of hospital performance evaluation : a systematic review. Int J Hosp Res. 2014;3:199–208.
Khullar D, Bond AM, Schpero WL. COVID-19 and the Financial Health of US Hospitals. JAMA. 2020;323:2127–8. https://doi.org/10.1001/jama.2020.6269.
Raghuvanshi VP, Raghuvanshi SP. Implications and future strategies on cost management for hospitals during and after COVID-19. Int J Community Med Public Heal. 2020;7:2405.
Coyle D, Manley A. Productivity in UK healthcare during and after the Covid-19 pandemic. 2021.
Griffin S. Covid-19: waiting times in England reach record highs. BMJ. 2020;370:m3557.
Cleary SM, Wilkinson T, Tamandjou Tchuem CR, Docrat S, Solanki GC. Cost-effectiveness of intensive care for hospitalized COVID-19 patients: experience from South Africa. BMC Health Serv Res. 2021;21:1–10.
Jeffery MM, D’Onofrio G, Paek H, Platts-Mills TF, Soares WE, Hoppe JA, et al. Trends in emergency department visits and hospital admissions in health care systems in 5 states in the first months of the COVID-19 Pandemic in the US. JAMA Intern Med. 2020;180:1328–33.
Rennert-May E, Leal J, Thanh NX, Lang E, Dowling S, Manns B, et al. The impact of COVID-19 on hospital admissions and emergency department visits: a population-based study. PLoS ONE. 2021;16:e0252441. https://doi.org/10.1371/journal.pone.0252441.
Shirazi H, Kia R, Ghasemi P. Ranking of hospitals in the case of COVID-19 outbreak: a new integrated approach using patient satisfaction criteria. Int J Healthc Manag. 2020;13:312–24. https://doi.org/10.1080/20479700.2020.1803622.
Moynihan R, Sanders S, Michaleff ZA, Scott AM, Clark J, To EJ, et al. Impact of COVID-19 pandemic on utilisation of healthcare services: a systematic review. BMJ Open. 2021;11:11–7.
Kamel MA, Mousa ME-S. Measuring operational efficiency of isolation hospitals during COVID-19 pandemic using data envelopment analysis: a case of Egypt. Benchmarking An Int J. 2021;ahead-of-p ahead-of-print. doi:https://doi.org/10.1108/BIJ-09-2020-0481.
Sodhi J, Satpathy S, Arora P. Role of hospital supportive services in COVID-19. Int J Infect Control. 2020;16:19–20.
Wen X, Wang F, Li X, Gu H. Study on the Knowledge, Attitude, and Practice (KAP) of nursing staff and influencing factors on COVID-19. Front Public Heal. 2021;8:1–6.
Limbu DK, Piryani RM, Sunny AK. Healthcare workers’ knowledge, attitude and practices during the COVID-19 pandemic response in a tertiary care hospital of Nepal. PLoS One. 2020;15 11 November:7–13. doi:https://doi.org/10.1371/journal.pone.0242126.
Huang H, Zhao WJ, Li GR. Knowledge and psychological stress related to COVID-19 among nursing staff in a hospital in China: Cross-sectional survey study. JMIR Form Res. 2020;4:1–8.
Alrawashdeh HM, Al-Tammemi AB, Alzawahreh MK, Al-Tamimi A, Elkholy M, Al Sarireh F, et al. Occupational burnout and job satisfaction among physicians in times of COVID-19 crisis: a convergent parallel mixed-method study. BMC Public Health. 2021;21:811.
Müller SM, Mueller GF, Navarini AA, Brandt O. National publication productivity during the COVID-19 Pandemic—A preliminary exploratory analysis of the 30 countries most affected. Biology (Basel). 2020;9:271.
The Centers for Disease Control and Prevention (CDC). Comprehensive Hospital Preparedness Checklist for Coronavirus Disease 2019 (COVID-19). https://www.cdc.gov/coronavirus/2019-ncov/hcp/hcp-hospital-checklist.html. Accessed 25 Jun 2021.
Members in Action. Four stages of planning and implementation during Covid-19: One rural Hospital’s preparations. American Hospital Association. 2020; April:1–3. https://www.aha.org/system/files/media/file/2020/04/rural-members-in-action-case-study-four-stages-planning-implementation-during-covid-19-one-rural-hospital-preparations-marshall-medical-center.pdf. Accessed 25 Jun 2021.
Burlea-Schiopoiu A, Ferhati K. The managerial implications of the key performance indicators in healthcare sector: a cluster analysis. Healthc. 2021;9:1–20.
Martin N. Finding a new normal: hospital governance best practices during COVID-19. Healthc Q. 2020;23:24–8. https://doi.org/10.12927/hcq.2020.26337.
Sharma A, Borah SB, Moses AC. Responses to COVID-19: The role of governance, healthcare infrastructure, and learning from past pandemics. J Bus Res. 2021;122 January:597–607. doi:https://doi.org/10.1016/j.jbusres.2020.09.011.
McCabe R, Schmit N, Christen P, D’Aeth JC, Løchen A, Rizmie D, et al. Adapting hospital capacity to meet changing demands during the COVID-19 pandemic. BMC Med. 2020;18:1–12.
Savitsky B, Radomislensky I, Hendel T. Nurses’ occupational satisfaction during Covid-19 pandemic. Appl Nurs Res. 2021;59 January:151416. doi:https://doi.org/10.1016/j.apnr.2021.151416.
Feng L, Yin R. Social support and hope mediate the relationship between gratitude and depression among front-line medical staff during the pandemic of COVID-19. Front Psychol. 2021;12:1–7.
Ng KYY, Zhou S, Tan SH, Ishak NDB, Goh ZZS, Chua ZY, et al. Understanding the psychological impact of COVID-19 pandemic on patients with cancer, their caregivers, and health care workers in Singapore. JCO Glob Oncol. 2020;:1494–509. doi:https://doi.org/10.1200/GO.20.00374.
Olanipekun T. The impact of COVID-19 testing on length of hospital stay and patient flow in hospitals. J Community Hosp Intern Med Perspect. 2021;11:180–3. https://doi.org/10.1080/20009666.2020.1866249.
Jamshidi B, Zargaran SJ, Bekrizadeh H, Rezaei M, Najafi F. Comparing length of hospital stay during COVID-19 pandemic in the USA, Italy and Germany. Int J Qual Heal Care. 2021;33:1–11. https://doi.org/10.1093/intqhc/mzab050.
Rees EM, Nightingale ES, Jafari Y, Waterlow NR, Clifford S, Carl CA, et al. COVID-19 length of hospital stay: a systematic review and data synthesis. BMC Med. 2020;18:270.
Pascua FV, Diaz O, Medina R, Contreras B, Mistroff J, Espinosa D, et al. A multi-mechanism approach reduces length of stay in the ICU for severe COVID-19 patients. PLoS One. 2021;16 1 January:1–15. doi:https://doi.org/10.1371/journal.pone.0245025.
World Health Organisation (WHO). Infection prevention and control during health care when COVID-19 is suspected or confirmed: Interim guidance. Interim Guidance. 2020; 29 June 2020:1–5. https://apps.who.int/iris/rest/bitstreams/1272420/retrieve. Accessed 25 Jun 2021.
Narwal S, Jain S. Building resilient health systems: patient safety during COVID-19 and lessons for the future. J Health Manag. 2021;23:166–81.
Gandhi TK, Singh H. Reducing the risk of diagnostic error in the COVID-19 era. J Hosp Med. 2020;15:363–6.
Denning M, Goh ET, Scott A, Martin G, Markar S, Flott K, et al. What has been the impact of COVID-19 on safety culture? A case study from a large metropolitan healthcare trust. Int J Environ Res Public Health. 2020;17:1–12.
Jabarpour M, Dehghan M, Afsharipour G, HajipourAbaee E, MangolianShahrbabaki P, Ahmadinejad M, et al. The impact of COVID-19 outbreak on nosocomial infection rate: a case of Iran. Can J Infect Dis Med Microbiol. 2021;2021:Cdc.
Ashinyo ME, Dubik SD, Duti V, Amegah KE, Ashinyo A, Asare BA, et al. Infection prevention and control compliance among exposed healthcare workers in COVID-19 treatment centers in Ghana: A descriptive cross-sectional study. PLoS One. 2021;16 3 March:1–13. doi:https://doi.org/10.1371/journal.pone.0248282.
Das AK, Islam MN, Billah MM, Sarker A. COVID-19 pandemic and healthcare solid waste management strategy – A mini-review. Sci Total Environ. 2021;778: 146220. https://doi.org/10.1016/j.scitotenv.2021.146220.
Kalantary RR, Jamshidi A, Mofrad MMG, Jafari AJ, Heidari N, Fallahizadeh S, et al. Effect of COVID-19 pandemic on medical waste management: a case study. J Environ Heal Sci Eng. 2021;19:831–6. https://doi.org/10.1007/s40201-021-00650-9.
Mekonnen B, Solomon N, Wondimu W. Healthcare waste status and handling practices during COVID-19 Pandemic in Tepi General Hospital. Ethiopia J Environ Public Health. 2021;2021:6614565.
World Health Organisation (WHO). Water, sanitation, hygiene, and waste management for SARS-CoV-2, the virus that causes COVID-19. Interim guidance. 2020; 29 July:1–11. https://www.who.int/publications/i/item/water-sanitation-hygiene-and-waste-management-for-the-covid-19-virus-interim-guidance. Accessed 25 Jun 2021.
de Aguiar Hugo A, Lima R da S. Healthcare waste management assessment: challenges for hospitals in COVID-19 pandemic times. Waste Manag Res J a Sustain Circ Econ. 2021;:0734242X2110103. doi:https://doi.org/10.1177/0734242X211010362.
Maalouf A, Maalouf H. Impact of COVID-19 pandemic on medical waste management in Lebanon. Waste Manag Res. 2021;39:45–55. https://doi.org/10.1177/0734242X211003970.
Abdelhadi A. Patients’ satisfactions on the waiting period at the emergency units. Comparison study before and during covid-19 pandemic. J Public health Res. 2021;10:15.
Bin Traiki TA, AlShammari SA, AlAli MN, Aljomah NA, Alhassan NS, Alkhayal KA, et al. Impact of COVID-19 pandemic on patient satisfaction and surgical outcomes: a retrospective and cross sectional study. Ann Med Surg. 2020;58:14–9. https://doi.org/10.1016/j.amsu.2020.08.020.
Wang M, Hu C, Zhao Q, Feng R, Wang Q, Cai H, et al. Acute psychological impact on COVID-19 patients in Hubei: a multicenter observational study. Transl Psychiatry. 2021;11:133.
Pigozzi E, Tregnago D, Costa L, Insolda J, Turati E, Rimondini M, et al. Psychological impact of Covid-19 pandemic on oncological patients: a survey in Northern Italy. PLoS One. 2021;16(3):1–13.
Rodrigues MENG, Belarmino A da C, Lopes Custódio L, Lima Verde Gomes I, Rodrigues Ferreira Júnior A. Communication in health work during the COVID-19 pandemic. Investig y Educ en Enfermería. 2020;38:1–12.
Al-Zyoud W, Oweis T, Al-Thawabih H, Al-Saqqar F, Al-Kazwini A, Al-Hammouri F. The psychological effects of physicians’ communication skills on covid-19 patients. Patient Prefer Adherence. 2021;15:677–90.
Samarasekara K. ‘Masking’ emotions: doctor–patient communication in the era of COVID-19. Postgrad Med J. 2021;97:406. https://doi.org/10.1136/postgradmedj-2020-138444.
Dyer C. Covid-19: Rules on sharing confidential patient information are relaxed in England. BMJ. 2020;369:m1378. https://doi.org/10.1136/bmj.m1378.
Zlobina A. Personal data of thousands of Covid-19 patients leaked in Moscow. Human Rights Watch. https://www.hrw.org/news/2020/12/15/personal-data-thousands-covid-19-patients-leaked-moscow. Accessed 25 Jun 2021.
Hammoudeh W, Kienzler H, Meagher K, Giacaman R. Social and political determinants of health in the Occupied Palestine Territory (OPT) during the COVID-19 pandemic: who is responsible? BMJ Glob Heal. 2020;5:e003683.
He W, Zhang Z (Justin), Li W. Information technology solutions, challenges, and suggestions for tackling the COVID-19 pandemic. Int J Inf Manage. 2021;57:102287. https://doi.org/10.1016/j.ijinfomgt.2020.102287.
Asadzadeh A, Pakkhoo S, Saeidabad MM, Khezri H, Ferdousi R. Information technology in emergency management of COVID-19 outbreak. Informatics Med Unlocked. 2020;21:100475. https://doi.org/10.1016/j.imu.2020.100475.
Monaghesh E, Hajizadeh A. The role of telehealth during COVID-19 outbreak: a systematic review based on current evidence. BMC Public Health. 2020;20(1):1193. https://doi.org/10.1186/s12889-020-09301-4.
We want to thank Dr. Vahideh Moghaddam, Diêgo Andrade, and Wang Zhe for their help in translating and data extraction of Persian, Spanish, and Chinese articles, respectively. Additionally, we would like to thank Dr. Duha Shellah for contributing to revising the manuscript. Last, special thanks to Mr. Amjad Amer for his supports in figures preparation.
The research was financed by the Thematic Excellence Program 2021 Health Sub-programme of the Ministry for Innovation and Technology in Hungary, within the framework of the EGA-10 project of the University of Pécs. Open access funding provided by University of Pécs.
Ethics approval and consent to participate
Not applicable since it is a systematic review.
Consent for publication
The authors have declared that no competing interests exist.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Amer, F., Hammoud, S., Khatatbeh, H. et al. A systematic review: the dimensions to evaluate health care performance and an implication during the pandemic. BMC Health Serv Res 22, 621 (2022). https://doi.org/10.1186/s12913-022-07863-0
- Balanced scorecard