- Open Access
Reliability and validity of an innovative high performing healthcare system assessment tool
BMC Health Services Research volume 23, Article number: 242 (2023)
Universal Health coverage (UHC) is the mantra of the twenty-first century yet knowing when it has been achieved or how to best influence its progression remains elusive. An innovative framework for High Performing Healthcare (HPHC) attempts to address these issues. It focuses on measuring four constructs of Accountable, Affordable, Accessible, and Reliable (AAAR) healthcare that contribute to better health outcomes and impact. The HPHC tool collects information on the perceived functionality of health system processes and provides real-time data analysis on the AAAR constructs, and on processes for health system resilience, responsiveness, and quality, that include roles of community, private sector, as well as both demand, and supply factors affecting health system performance. The tool attempts to capture the multidimensionality of UHC measurement and evidence that links health system strengthening activities to outcomes. This paper provides evidence on the reliability and validity of the tool.
Internet survey with non-probability sampling was used for testing reliability and validity of the HPHC tool. The volunteers were recruited using international networks and listservs. Two hundred and thirteen people from public, private, civil society and international organizations volunteered from 35 low-and-middle-income countries. Analyses involved testing reliability and validity and validation from other international sources of information as well as applicability in different setting and contexts.
The HPHC tool’s AAAR constructs, and their sub-domains showed high internal consistency (Cronbach alpha >.80) and construct validity. The tool scores normal distribution displayed variations among respondents. In addition, the tool demonstrated its precision and relevance in different contexts/countries. The triangulation of HPHC findings with other international data sources further confirmed the tool’s validity.
Besides being reliable and valid, the HPHC tool adds value to the state of health system measurement by focusing on linkages between AAAR processes and health outcomes. It ensures that health system stakeholders take responsibility and are accountable for better system performance, and the community is empowered to participate in decision-making process. The HPHC tool collects and analyzes data in real time with minimum costs, supports monitoring, and promotes adaptive management, policy, and program development for better health outcomes.
Emerging threats from infectious disease, ongoing demographic shifts characterized by continued growth of both young and ageing populations and urbanization, climate change and its associated extreme weather aftermath, as well as increasing social and economic disparities, reemphasize the importance of strengthening health systems to meet the health consequences of these challenges. The World Health Report  on improving health systems brought attention to the health system as vehicle to improve health of the people and the need to make health systems improvements a donor priority. Subsequent donors’ focus on health systems strengthening along with the development of multiple heath system strengthening (HSS) frameworks [2,3,4,5,6,7,8,9,10,11,12,13,14,15,16], have contributed to a better understanding of health system actors (public, private, community and outside of health system), functions, resources, services, and outcomes. However, development of the evidence base has not paid sufficient attention to the interactions among system components. The result is that evidence related to the impact of HSS interventions is often difficult to replicate and correlation to health outcomes are weak, leading some to question the overall effectiveness of HSS interventions and investments.
Limited investments in HSS not only weaken evidence generation but also have repercussion on achieving the United Nations  Sustainable Development Goal (SGD-3) target (3.8) to “Achieve universal health coverage (UHC), including financial risk protection, access to quality essential health-care services and access to safe, effective, quality, and affordable essential medicines and vaccines for all”. The measurement of UHC through SDG indicators 3.8.1 and 3.8.2 includes essential services coverage and financial hardship . Tracking progress towards UHC using these outcomes indicators is important. However, UHC is a multidimensional concept that is reflective of a system’s capabilities, society’s cultural values , priorities and power relationships [20, 21], and therefore requires more nuanced diagnostics to enable programmatic response to the measurement. For example, access is more than simply the availability of services, they must also be affordable, and people must have confidence in their effectiveness . Accountability is required to ensure this confidence and to guard against corruptive influences; to incentivize consistent (and improved) performance overtime [23,24,25,26,27,28,29,30,31,32,33]. Reliable care processes also underpin access and performance as they are critical to ensuring client use of the system. Achievement of UHC is grounded in social values like solidarity, equity, efficiency; it also involves contributions of individuals and communities in taking care of their own health. The challenge in tracking UHC progress is how to incorporate the processes that underscore the dynamic relationship between social values and the unique roles of systems actors in a measurement tool that also serves to help assess healthcare system performance.
The purpose of this paper is to describe a HPHC conceptual framework and tool to address the need for assessing essential processes that lead to AAAR care and their linkages to health system performance outcomes including UHC; creating testable hypotheses for measuring changes in a health system that are reflective of HSS interventions, bridge the evidence base between HSS and UHC, and can support increases to domestic and donors’ investment in HSS. This paper will answer the research question, “is the HPHC tool reliable and valid?”
The paper is organized in different sections starting with a literature review around HPHC AAAR constructs followed by methods and results on reliability and validity of the tool. In the discussion section, the paper comments on the tool’s added value, complementarity to other HSS tools and on how the tool could be used to improve the performance of a health system including its ability to achieve health outcomes and UHC. Lastly limitations and conclusions are presented.
The United States Agency for International Development (USAID) developed the HPHC framework  to further a collective understanding of the characteristics of strong health systems. By articulating a set of characteristics across four AAAR dimensions, the framework seeks to describe the role those different stakeholders across society play in a strong health system and to demonstrate that both supply and demand side actions contribute to a health system’s functionality. The HPHC framework presumes that to deliver needed health care reliably to clients/communities in ways they trust, and can afford, requires that the people/communities have a voice in shaping that care, and therefore, that healthcare needs to be simultaneously accountable, affordable, accessible, and reliable (AAAR; Fig. 1), irrespective of whether it is delivered through the public or private sector/non-government organizations (NGOs).
The HPHC framework postulates that the AAAR dimensions of healthcare must be present and functional to truly achieve UHC because addressing processes of care is the responsibility of everyone . The framework recognizes that a health system requires collective action by multiple actors from national to local levels and across community, public and private entities (Fig. 1). The accountable and reliable attributes of healthcare not only make the institutions included in the health system as the owners and organizers of health care, but also make them collectively responsible for assuring health care is available when people need it and can afford it without undue burden. Therefore, despite being distinct constructs, AAAR are interrelated. The HPHC framework acknowledges a well-functioning health system’s dual role as both supplier and demand generator for health services. The framework also recognizes the community’s critical and dynamic relationship with health system as consumer, advocate, overseer, and overall partner in ensuring that community health needs are met and UHC is achieved. The framework identifies under AAAR domains, the major salient health system processes, six building blocks/functions, stakeholders, and context comprehensively and acknowledges their interrelationships (Fig. 1), which typically have been limited in other HSS/UHC measurements related to outcomes.
Measuring the functionality of system processes/building blocks adds value to this space because these processes are critical to improve health system performance, outcomes, and impact (Fig. 1). The relationships among processes, outputs, outcomes, and impact are not linear in the framework, but dynamic; they interact and affect one another to drive toward systemwide changes (two-ways arrow). Achieving better UHC outcomes/impact requires HSS interventions that improve the weak or non-functional system processes. As specific processes get improved other weakness may be exposed or become more important thus, frequent measurement of such processes complemented by progress at the outcome level can create a cycle of continuous improvement. Measurement of the perceived functionality of processes when combined with data from other sources on health system performance provide richer insights into where additional action might drive better performance.
The HPHC framework and its tool are grounded in theories of system practice, process/quality improvement [35, 36] (to holistically oversee, reform, and measure system components/processes interactions and impact), organizational development [37,38,39] (to manage organizational change and learning), behavioral sciences [40,41,42,43] (to change practices and social norms for better performance), and empowerment [44,45,46] (for greater clients’ autonomy and community’ voice in shaping health systems).
The HPHC framework’s emphasis on effective processes adds value by: identifying major HPHC processes and their functionality; situating the locus of control for change within the health system (broadly defined to reflect institutions across society); empowering communities and fostering a dynamic relationship for joint actions and mutual accountability for better health outcomes; creating opportunities for continuous improvement as systems work toward these ideals; enabling future evidence generation related to testing hypotheses about AAAR and their sub-attributes’ relative contribution to health system outcomes; supporting innovative ways for designing, managing, monitoring, and evaluating health system strengthening and contributing to the state of the art knowledge.
This section provides further elaboration of literature on the AAAR dimensions of the HPHC framework, and their operational definitions and boundaries. The section next addresses the rationale of HPHC tool using perceptions, and how it counters potential biases. Lastly, this section describes different modalities to use the tool.
AAAR literature and operational definitions
The HPHC tool is based on the “Access to Universal Health Coverage through High Performing Health Care” fact sheet  which provided operational definition of AAAR and its subdomains (Table 1). These operational definitions were used to select five indicators that were important and relevant to measure the specific AAAR sub-domains. Further details on selected indicators for subdomains are provided in the face and content validity section. The complete tool is available upon request (Additional file 1).
Murray  defined health system responsiveness as meeting the non-medical needs of the population and as an outcome of the governance function. Accountability to the target population is implicit in responsiveness construct but never explicitly stated or measured. Lewis  has drawn attention to linkages between minimal funding levels, illegal payments in publicly financed and delivered care and limited accountability. Peters  noted that accountability measures in health systems, as a part of regulatory or oversight efforts, are relatively neglected in comparison to organization and financing of services. Cleary et al.  building on the concepts of bureaucratic (internal) and external accountability  showed that there is a gap in the power relationships among mechanisms of accountability. The relatively powerful bureaucracy with an emphasis on supervision and management systems and focused on compliance to centrally defined outputs and targets can constrain front line managers and providers from responding to patient and population priorities. Thus, accountability is best understood as refereeing the dynamics in two-way relationships, often between unequal partners .
Recognition of this unequal partnership became the rally point for the social accountability movement. This movement made important contributions by emphasizing citizens’ demand for and role in developing accountability processes such as assessment, demand articulation, feedback and negotiation with providers, changes in providers’ behavior and facility practices [20, 50]. Social accountability also contributed to accountability measurement through development of social audit, community score card and Citizen Charter [51,52,53]. Informal social accountability  in the context of community-based workers also highlights the social pressures borne by health workers and how informal social networks create incentives/disincentives for health worker accountability and relationships to formal accountability structures. A robust need for data for accountability is undeniable but evidence of the weakness of information systems in low-and-middle-income countries (LMIC) is well documented .
Building on existing literature and recognizing that progress toward the achievement of UHC requires explicit accountability structures within country health systems. Under HPHC framework, accountable health care means society as a whole works together to ensure care meets people’s needs, which brings in the WHO  concept that the health system is everyone’s responsibility. In the HPHC framework ‘everyone” is implied by the system’s stakeholders (clients, communities, public, private, NGOs, donors, international organizations); and each of their roles are considered in promoting accountability. Accountability is measured through six subdomains (Table 1) comprising of Public-private/civil society organizations (CSO) partnership for management, funding, and oversights of health institutions; patient privacy and satisfaction, transparency in information sharing on financing, service outcomes, sustaining clients’ health behaviors and outcomes, regulatory processes for quality, and recourse options available to dissatisfied clients. An overall accountable score provides level where health system stands in providing Accountable care. The Accountable sub-domains scores highlight strengths and weaknesses under each sub-domain.
Adequate financing of the health system across public, private and Non-Government Organization (NGO) sectors, to make healthcare affordable has remained problematic. Stenberg et al.  estimated an additional $274 billion global spending on health is needed per year by 2026–30 to make progress towards the SDG3 targets where health system absorptive capacity is constrained and US$371 billion would be needed by 2026–30 to achieve SDG3, necessitating more revenue generation and multisectoral collaboration for addressing social determinants of health. However, revenue generation needs to be fair and progressive to support the overall affordability of the health system because, for example, simply instituting user fees would in turn affect access to care [57,58,59]. A range of financial barriers affect access to healthcare, and the financial health burden of accessing care is a major cause of household’s impoverishment. Financial protection schemes such as health insurance , cash transfer , public health expenditures  have been shown to reduce out of pocket expenditure and improve overall affordability. Affordability is also affected by the cost of medicine, laboratory tests, x-ray, and the costs to individuals and families of seeking care such as cost of transport or childcare. Additionally, there is an economic cost to both individuals and populations of substandard and falsified medicines and [63,64,65] substandard medicines have been noted as a contributing cause of increased inequity in health outcomes in Uganda .
Under HPHC framework, affordable health care means money spent on care provides the best value possible. It is assessed through five sub-domains (Table 1) reflecting that costs do not impoverish people, that people consider the total costs before seeking care, and that availability and use of pre-payment schemes help to protect from financial hardship. Affordability also requires governments to allocate resources effectively and build public-private partnerships to increase domestic funding and ensure availability and rational use of safe and quality medicines without financial burden. A summary affordable score provides information on the overall level of affordability of care in and within the health system. The affordable sub-domains scores identify strengths and challenges under each sub-domain.
Tanahashi  provided a pathway for access to care that started with availability of health services, geographic and financial access, followed by a need for acceptability and actual receipt of care while adding a final step of effective coverage, meaning that care is of such quality that it has produced the intended positive effects. Thus, he created a linkage among access (those who can use the service), coverage (those who utilize the service), and quality to make effective service coverage (=quality*coverage). Perceptions of quality also affect access to care . Acceptability broadens the concept of access to include other sociocultural factors affecting access to health care such as gender norms, respectful care, and empowerment [69,70,71]. Thus, access to health care is a multidimensional, and complex social construct . It reflects not only how many people have access to services but also reveals inequities in access within social and vulnerable groups/communities, and within and between countries.
Accessible healthcare, per the HPHC framework exists when care is available when and where people need it and can use it and meets quality standards. It has six subdomains (Table 1) that cover geographical access to facility, staff and medicine, alternative options including eHealth to extend reach of care, emergency care, quality, cultural access through respectful care and achievement of equitable outcomes. An overall accessible score provides level of access to health care. The accessible sub-domains scores highlight strengths and gaps in dealing with different access issues.
In health literature, reliable care is mostly treated as an aspirational goal, a dimension of quality but not as a system characteristic. However, it is a distinct element of HPHC framework, which is not featured in other health system frameworks. Reliability includes emergent concepts of resilience, which has gained increasing relevance due to COVID-19 pandemic. Dependable quality and safety remain a concern globally causing human suffering and of waste of resources . High Reliable Organization (HRO) theory and practice [73,74,75] provides context and justification for use of this construct and has made inroads in healthcare literature ) with respect to systems processes, interdependencies among units and levels, and to create a high degree of accountability. To be a reliable organization  requires preoccupation with failure, reluctance to simplify interpretations, sensitivity to operations, commitment to resilience and deference to expertise in time of crisis, which implies a focus on interdependence, redundancy, and culture . The Joint Commission  in United States of America (USA) has made creating and maintaining a culture of safety a requirement for the healthcare organizations accreditation . The greater use of reliability-enhancing work practices such as respectful interaction and mindful organizing processes were associated with better performance (fewer medication errors and patient falls) compared to organizational citizenship behavior  while failure to convert periodic mindful practices as habit lead to unreliable healthcare .
HPHC framework incorporates the latest insights from the existing literature on reliable care. It defines Reliable health care as high-quality care, delivered in a timely manner that promotes dignity and respect for all patients and providers. This operational definition has seven characteristics (Table 1). Reliable health care assumes not only client safety through error free care but includes a time dimension that such care is replicable and consistent. Reliable care also entails resilience that in times crisis processes are inbuilt to continue providing care. An overall reliable score provides information on to what extent health care is reliable. The reliable sub-domains scores highlight strengths and weaknesses under each sub-domain.
HPHC tool boundaries
HPHC tool is confined to measuring functionality of AAAR processes for multiple reasons: to create a holistic picture of major processes that not only reflect direct processes of care but include the health system functions/building blocks, engagement of diverse stakeholders such as clients/communities, public/private/NGOs organizations and other multi-sector factors where changes can be made to affect health system outcomes/UHC and performance; to avoid participants’ fatigue with a long questionnaire; and to foster enhanced use of the existing sources of information that gather health system outcomes data, especially the routine information system. AAAR functionality scores should be combined with health system performance and outcomes data for understanding the linkages. Non-alignment of perceptions about functionality of healthcare processes with measures of performance through routine or survey data can provide information on where to focus efforts for improvement and the development of effective interventions. Building on the quality/performance improvement literature, outcome is the byproduct of effective processes, therefore, the tool creates opportunities to test the hypothesis that higher functionality of AAAR processes is associated with better outcomes.
Use of perceptions
The tool assesses the functionality of AAAR processes through individuals’ perceptions because people base their actions on their perceptions, impressions, and views . Objective reality does not always reflect actual practice, for example, policy and regulatory standards might be available on paper but actual practice might differ [84, 85]. Perceived health status is predictive of mortality and its association with objective health status is well documented [86,87,88]. These illustrated examples show that perceptions are a good proxy of objective reality. Therefore, we have used perceptions to assess level of overall performance and whether care is accountable, accessible, affordable, and reliable. The tool accounts for potential perception bias by including anonymity, instructions to be truthful and adding respondents’ characteristics to check and control for biases during analyses [89,90,91,92].
Modality of use
HPHC is a web-based tool (https://hphctool.org), which can be shared with many people in different locations. The tool improves representation and solves issues of cost and frequency of data collection, facilitating monitoring progress at low cost. In addition, HPHC can be applied through different modalities such as key informants, or the Delphi technique if all stakeholders are represented. The tool could be used in a survey using small sample size lot quality assurance sampling (LQAS) for overall estimate of HPHC scores. LQAS could also be used to assess the country/region/district is meeting HPHC predetermined level of performance.
Data collection instrument
The HPHC tool has 120 questions items spread under AAAR subdomains.
Sample size for reliability and validity
The sample size of 200 is calculated based on literature recommendations for testing reliability and validity considering how many respondents needed based on number of questions in the tool [93,94,95,96,97,98]. Sample size calculation for reliability accounts for expected Cronbach alpha = 0.75, type 1 error = 0.05, number of items = 120, and power = 80; and alpha = .8 requires lower sample size.
The survey design was cross-sectional.
Method of data collection
We used internet survey because it is cost-efficient and easier to reach eligible participants in LMIC.
All those age 18 years or more, who interacted with health system as part of the health system in some capacity or a client were eligible to participate in the survey to have broader representation from in and outside the system. It is assumed that if the health system has strong linkages with clients/communities, clients will be able to expresses their opinions about health system processes functionality. The survey was only available in English and thus required respondents to have English language skills. No personal information was collected except information about gender, age, types of organizations, and years of work experience. The research was conducted in compliance with the United States Federal Codes of Regulations (CFR) on protection of human subjects, including informed consent (22 CFR 225). The research protocol and ethical statement were approved by Kelly Saldana, Director, Office of Health Systems, USAID, who has the authority and determined that the research protocol met the criteria for exemptions from the review by Institutional Review Board committee (22 CFR 225.104).
We used non-probability sampling to recruit volunteers from LMIC using different global networks (Health System Strengthening Network, Routine Health Information system network, Christian Connections for International Health hospital network) and a consulting firm in Pakistan to reach public, private, NGOs, and international organizations in health/development sector and people with different socio-demographic characteristics such as age, education and employment for broader representations. Given no access to the networks’ memberships, it was not possible to randomly select specific number of participants from different countries. The use of non-probability sampling remains the dominant method for testing reliability and validity of new constructs for the last 40 years [99,100,101].
Convenience sampling is a dominant method for collecting reliable and valid information in development field such as World Bank Governance indicators  and Country Policy and Institutional Assessment tool  or Transparency International Corruption Perception Index . The results of non-probability sampling are comparable to probability sampling when all stakeholders in the target populations are represented, and respondents’ socio-demographic characteristics do not influence the responses [104,105,106,107,108]. We addressed representation in non-probability sampling by involving health and development professionals in multiple countries and to account for survey participants’ socio-demographic characteristics and types of organizations (public, private, non-government, and international organizations) influence on their perceptions of AAAR processes functionality.
Types of analyses
Out of 219 respondents [from 37 countries in Africa (Cameroon, Côte D′ Ivoire, Ethiopia, Ghana, Guinea, Kenya, Lesotho, Madagascar, Malawi, Mali, Nigeria, Senegal, Sierra Leone, South Africa, South Sudan, Tanzania, Uganda, Zambia, Zimbabwe); Asia (Bangladesh, India, Indonesia, Jordan, Kazakhstan, Myanmar, Nepal, Pakistan, Philippines, Uzbekistan, Yemen); Latin America/Caribbean (Antigua & Barbuda, Colombia, Guyana, Haiti, Peru)], we excluded six respondents from USA and Canada from analysis because we wanted to test the tool in LMIC. To contextualize where these countries stand on their health system performance, we have used the universal health coverage index (UHCI)  for comparison. UHCI is a good proxy for health system performance because it is comprised of service coverage and other indicators. The overall participants’ countries UHCI ranges from 28% (Madagascar) to 77% (Peru). However, in most of the African and Asian countries, UHCI is below 50%, indicating that these countries have a long way to achieve UHC.
Social desirability was tested by assessing whether the responses to negatively worded statements were in alignment with positive statements to show same direction of responses as well as spread of the responses. Cronbach’s Alpha was used to test the internal consistency of the overall tools and its four domains. Construct validity was tested using correlations among the four AAAR construct. Lastly, validation was conducted by comparing tool results with similar data from other sources of information.
The results are based on 213 people from 35 countries in Africa, Asia, and Latin America. Pakistan, Nigeria, India, and Philippines had 20 or more people respondents making it possible to answer whether the tool could collect reliable and valid information in different contexts/settings. Thus, the results are presented overall and by these selected four countries.
Respondents’ socio-demographic characteristics
Overall, 67% respondents were males and countries with 20 or more respondents showed similar distribution except for Nigeria where slightly more women responded (Table 2). The respondents were almost equally distributed among public sector, private sector/NGOs, and international organizations. However, in Pakistan more respondents were working in public sector, while in Nigeria and Philippines the dominant category was private sector and majority respondents in India were from NGOs. The respondents overall and in selected countries had more than 15 years of education and professional experience. There were no significant differences between overall HPHC score and: a) types of organizations (SS = 557, df = 3, F = 1.52, p = .211), b) respondents’ experience (SS = 129, df = 3, F = .35, p = .79), c) education (SS = 357, df = 2, F = 1.45, p = .23), and gender (SS = 258, df = 1, F = .211, p = .14) using ANalysis of VAriance (ANOVA) test, indicating that these characteristics did not influence the HPHC scores.
Social desirability bias test
The analysis showed that respondents’ rating on positive and negatively worded statements were aligned showing same direction and no social desirability bias. Descriptive analyses of data showed wide spread of overall HPHC scores and its domains (Fig. 2) indicating that the respondents were expressing their opinions freely without any social desirability bias, which if present could have titled score in positive direction. This trend was also observed in the selected countries data (Fig. 3). The respondents’ characteristics were not found to be associated with overall high performing and AAAR domains scores, indicating their no contributions affecting responses.
Reliability of the tool
Reliability of the tool is tested in two ways. Through internal consistency test and whether the internal consistency is repeatable in different contexts/settings.
The tool has an overall Cronbach Alpha of .92, indicating that the items and responses were internally consistent, showing reliability. Similarly, the major domains; accountable, accessible, and reliable sub-scales have alpha above .9 and for affordability 0.8 (Table 1). It also showed alpha for all four subdomains. Each subdomain has five items (Additional file 1. tool questionnaire) and have high internal consistency. Nunnally [110, 111] recommend that low reliability coefficients 0.5–0.6 are tolerable when the exploration of constructs is at the initial stage of construct exploration, otherwise alpha of 0.7 or more is a reliability standard of acceptability. The Table 1 shows that reliability standard was met by almost all subdomains except two affordability subdomains. However, these coefficients are in acceptable range given the initial stage of tool test.
Reliability in different contexts
We also tested repeatability of the alpha in Pakistan, Nigeria, Philippines, and India that met the sample size requirement of 9 or more respondents for scale 25–35 items for testing alpha at .8 or higher level [112,113,114]. We found an alpha >.80 in all four countries, which reflects that the tool reliability is repeatable in different countries and settings.
Validation of the tool
The tool was validated testing face, content, and construct validity, and confirming validity in different contexts and through triangulating tools findings from other sources of information.
Face and content validity
Face and content validity was checked through four steps. First, the HPHC tool was shared with 15 public health experts to check face and content validity and add/delete/revise question statements related to AAAR subdomains. Second, the tool was circulated within lead authors’ organization to broaden assessment of face and content validity and to solicit suggestions for improvement. Third, the revised tool was circulated to field staff of the lead authors’ organization with expertise in public health around the world for their review and comments. The country field staff confirmed tool items appropriateness, specificity, and relevance. Fourth, the final edited tool was again shared with a select group of public health advisors who concurred with earlier respondents group findings of tool face and content validity.
Tool construct validation
The HPHC framework hypothesizes that its four domain constructs: accountable, accessible, affordable, and reliable are interrelated. A correlation test among these constructs (Table 3) confirmed significant associations, showing construct validity. The associations among the four constructs were also found in the Pakistan, Nigeria, Philippines, and India data indicating that the tool is also valid in different contexts and settings for measuring HPHC AAAR domains. The interdependence among the health system processes/functions under AAAR domains and subdomains makes factor and principal component analyses irrelevant for discriminant and convergent validity. The new network model approaches [115,116,117] where characteristics are distinct but influence each other to contribute to underlying constructs are more appropriate and promising for future analyses.
Validation of tool findings from other sources
Triangulation of information from different sources is another method of validation of collected information. HPHC tool is unique and therefore, the overall findings are not comparable to other HS assessment tools. However, there are many indicators within AAAR subdomains that are close approximation to data collected by other sources and can be used for partial validation of HPHC tool findings. We have used this comparison with other sources of data and three examples are presented. The Fig. 4 shows that the mean perceived functionality of health facility which has staff and medicines and is a proxy for access is closely matched with mean MCH coverage estimated through demographic and health survey (DHS) , indicating that level of perceived functionality of access processes is comparable to actual services coverage, validating collected information.
The perceived functionality of cost containment processes (assessed by costs of care, impoverishment protection and opting for insurance) is compared with out-of-pocket expenditure (OPP), Fig. 5, available through WHO data repository . This finding validates that 50% perceived functionality of the cost containment processes is in alignment with high OPP as reported by WHO data. Limited functionality of cost containment processes is further supported by absence of national insurance schemes in 98% of the countries captured in DHS data .
Another example of validating the tool’s findings was done by comparing mean perceived surveillance functionality score with WHO International Health Regulation surveillance capacity score . There is close correspondence between perceived functionality of surveillance and IHR capacity score in selected countries (Fig. 6) validating that the tool is comparable in assessing functionality of surveillance processes.
Tool responsiveness and precision
Responsiveness of the tool addresses whether the tool could identify change over time in the respondents and in different contexts, while precision is about identifying changes in the phenomenon of interest . The assessment showed that the tool was able to identify variations in HPHC scores and its domains in selected four countries (Figs. 2 and 3) showing its responsiveness and precision and validating the tool utility in different contexts. We also tested whether tool was able to find differences in overall HPHC and AAAR domains scores among selected countries (Table 4), which confirmed the tool’s ability to detect significant differences in diverse contexts. Philippine overall HPHC score is significantly different from Pakistan, India, and Nigeria. There is no significant difference between Pakistan and India overall HPHC scores, but their overall HPHC scores significantly differ from Nigeria. Due to use of non-probability sampling and small sample sizes, HPHC mean percentile performance scores of India, Nigeria, and Philippines should not be considered reflective of country performance, whereas Pakistan’s sample size is adequate and representative of all provinces, thus HPHC mean percentile performance score is reflective of Pakistan’s health system performance.
Besides being highly reliable and valid, the tool’s uniqueness lies in its ability to assess not only performance in processes of accountable, accessible, affordable, and reliable care but also the processes dealing with the role of system stakeholders, clients/communities, public/private/NGOs, multi-sector entities in a way that opens opportunities for action at different levels and across the sectors. The tool also adds value by collecting information through use of a technology platform and: provides real time data collection and analysis; facilitates monitoring change over time; and limits costs. The tool identifies strengths and weaknesses in health system to facilitate adaptive management, policy, program, and research development.
An important contribution and implication of the HPHC tool is that accountability is beyond an elusive concept [122,123,124,125] and is being part and parcel of everyday health system performance. In addition, affordability is not confined to financial risk protection but recognizes all the factors that contribute to the cost of care, and reliable care subsumes everyday resilience [126,127,128] and is not limited to sporadic crisis. Access is broadened to cultural, organizational and equity issues. By measuring functionality of illustrated health system processes regularly, HPHC framework ensures transparency for two major purposes: the policy makers, managers and organizational members develop appropriate policies, incentive processes, and take management actions to curb corruption and be accountable for health system performance and outcomes for their internal and external (community, donors) clients; communities are empowered with information to have voice in decision-making process for better health status.
Limiting the tool’s boundaries to assess level of process performance is intentional to encourage use of other existing sources of information that collect information on system outcomes, and to avoid additional costs for data collection. HPHC findings could complement and explain performance gaps identified by other sources and especially reflected in routine information systems data.
There are other complimentary health system performance assessment tools available for use. Many of these tools requires observation, record reviews, and/or interviews to collect information, and often necessitate consultant labor and stakeholders’ time to implement, implying significant costs. Due to these high costs, frequency of data collection is limited, and these tools do not work for routine monitoring purposes. The Health System Assessment (HSA) tool  based on the health system building blocks was developed in 2007 to assess systems strengths and weaknesses. The HSA approach synthesizes existing information—from document reviews, in- country stakeholder interviews, and site visits and provides comprehensive insights across the system or specific functions of the system. Went  health system rapid diagnostic tool is like HSA tool but differs with heavy emphasis on a country-led design and planning process rather than a using a prescriptive (and usually very long) list of indicators and capacity building of the partners. Primary Health Care Performance Initiative (PHCPI)  takes a narrower approach to health system performance assessment in LMIC by focusing on PHC and filling a performance measurement gap at this level. The tool links PHC related higher level system determinants with input, processes, and outcome indicators. The tool collects information through existing information system, conducts a survey and uses key informant interview/focus groups for missing information.
Rohrer-Herold  reviewed the major seven health system assessment tools, excluding PHCPI, and showed that these tools are based on prescriptive diagnostics of health system building blocks/functions. There is neither a common set of indicators nor is the selection of indicators based on theoretical or strategic frameworks. In addition, there is no concrete guidance on how to adapt the tools to different cultural/health systems contexts, despite the authors stressing the flexibility and adaptability of the tool. The review concluded that a need exists to harmonize these tools, but a political will is lacking. The reviewer’s conclusion assumes that if these indicators are harmonized and adapted, changes in performance could be measured; but this view does not account for the reasons why these tools have not been adapted to date. Further, it is critical to acknowledge that rarely do changes in health systems indicators follow a linear path, even as some aspects of the system performance may be improving, thus a different approach is warranted.
The HPHC tool brings a new measurement approach based on systems practice [8, 133]. Unlike many tools requiring outside technical experts, HPHC shifts the responsibility of measurement internally through a user-friendly tool that foster empowerment to periodically review their system’s accountable, accessible, affordable, and reliable processes of care and make necessary changes for continuous improvement in processes for better outcomes.
The tool could be used by the government, private sector, NGOs, and international organizations at national or subnational levels. The tool could be used as rapid assessment by bringing major system stakeholders and key informants with representation from public, private/NGOs and international organizations. The findings could be used for priority setting, and policy, program, and research development. Implementation of the survey using lot quality assurance sampling can provide an overall estimate of HPHC performance and disaggregation by lots (districts/regions/facilities) or to test whether certain district/regions meet predetermined (or previous) HPHC performance standards. The interpretation of findings will depend on how representative the participants are and unit of analysis. The use guidance will be provided in the web-based tool.
HPHC tool identifies strengths and weaknesses in functionalities of AAAR processes dealing with health system functions, clients/communities, public/private/NGOs, stakeholders at different levels, and contextual factors including multi-sectoral collaboration and assumes that they affect health system outcomes (Fig. 1). Although the results are based on non-probability sample, selection bias is minimized by inclusion of a wide representation of respondents/stakeholders in the health system from thirty-five low-and-middle-income countries. Some caution and distinction are warranted that despite the high reliability and validity of the tool, India, Nigeria, and Philippines HPHC performance scores are not reflective of country performance due to small and non-probability sample, while Pakistan’s data are reflective of country performance because they are a representation all provinces and an adequate sample size. When using the HPHC tool, it is important to pay attention to sample size, sampling, and wider representation of all stakeholders in the health system to get a reliable and valid performance estimate as well as to test the reliability and validity of tool findings with statistical analyses and triangulation with other sources of data. More research is needed to: generate empirical evidence between HPHC findings and health system outcomes; understanding magnitude of change in strengthening processes of care; and to investigate network model of assessing and validating AAAR subdomains interdependence in influencing AAAR constructs scores and outcomes. Lastly, the length of the tool needs to be reduced to avoid respondents’ fatigue.
High performing healthcare tool is an important and innovative contribution to the measurement of health system performance because it identifies the level of performance within specific AAAR processes’ that contribute to system outcomes. It also fosters the system stakeholders’ ownership and responsibility for improving and designing policy, program, processes intended to improve the healthcare system’s outcomes/impact. The framework and tool will evolve over time and will help refine research, analysis, system practice, and contribute to better health system performance with more country applications.
Availability of data and materials
The datasets generated and/or analyzed during the current study are not publicly available due to privacy and confidentiality of the respondents but are available from the corresponding author on reasonable request.
World Health Organization
Health System Strengthening
- UN :
Sustainable Development Goal
Universal Health Coverage
High Performing Health Care
Accountable, Affordable Accessible, and Reliable
- USAID :
United States Agency for International Development
- LMIC :
Civil Society Organization
High Reliable Organization
United States of America
Demographic and Health Survey
International Health Regulation
Health System Assessment
Primary Health Care Performance Initiative
Code of Federal Regulations
Universal Health Coverage Index
World Health Organization. The world health report 2000: Health systems: improving performance: World Health Organization; 2000. https://apps.who.int/iris/handle/10665/42281.
van Olmen J, Criel B, Bhojani U, Marchal B, Chenge F, Van Damme W, et al. The health systems dynamics framework. Health Cult Soc. 2012;2(1):1–12.
Gruskin S, Ahmed S, Bogecho D, Ferguson L, Hanefeld J, MacCarthy S, et al. Human rights in health systems frameworks: What is there, what is missing and why does it matter? Glob Public Health. 2012; Retrieved from: https://www.ncbi.nlm.nih.gov/pubmed/22263700.
Mikkelsen-Lopez I, Wyss K, Des D. An approach to addressing governance from a health system framework perspective. BMC Int Health Hum Rights. 2011;11:13 Retrieved from https://www.ncbi.nlm.nih.gov/pubmed/22136318.
Shakarishvili G, Atun R, Berman P, Hsiao W, Burgess C, Lansang M. Converging health systems frameworks: Towards a concepts-to-actions roadmap for health systems strengthening in slow- and middle-income countries. Glob Health Govern. 2010;III:2 Retrieved from http://www.ghgj.org/Shakarishvili_Converging%20Health%20Systems%20Frameworks.pdf.
Shakarishvili G, Lansang M, Mitta V, Bornemisza O, Blakley M, Kley N, et al. Health systems strengthening: A common classification and framework for investment analysis. Health Policy Plann. 2011;26(4):316–26 Retrieved from: https://academic.oup.com/heapol/article/26/4/316/659551/Health-systems-strengthening-a-common.
Atun R, de Jongh T, Secci F, Ohiri K, Adeyi O. Integration of targeted health interventions into health systems: A conceptual framework for analysis. Health Policy Plann. 2009;25:104–11 Retrieved from https://academic.oup.com/heapol/article/25/2/104/641536/Integration-of-targeted-health-interventions-into.
de Savigny D, Adam T, editors. Systems thinking for health systems strengthening. Geneva: World Health Organization; 2009. Retrieved from: https://www.who.int/alliance-hpsr/resources/9789241563895/en/.
Kruk M, Freedman L. Assessing health system performance in developing countries: A review of the literature. Health Policy. 2008;85(3):263–76 Retrieved from: https://www.ncbi.nlm.nih.gov/pubmed/17931736.
World Health Organization (WHO). Everybody business: strengthening health systems to improve health outcomes: WHO’s framework for action; 2007. ISBN 978 92 4 159607 7.
Mossialos E, Allin S, Figueras J. Health systems in transition: Template for analysis. Copenhagen: World Health Organization Regional Office for Europe on behalf of the European Observatory on Health Systems and Policies; 2007.
Roberts M, Hsiao W, Berman P, Reich M, Getting health reform right. A guide to improving performance and equity. New York: Oxford University Press; 2004.
Murray C, Frenk J. A framework for assessing the performance of health systems. Bull World Health Org. 2000;78(6):717–31 Retrieved from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2560787/.
Roemer M. National health systems throughout the world. Ann Rev Public Health. 1993;14:335–53. https://doi.org/10.1146/annurev.pu.14.050193.002003.
Kleczkowski B, Roemer M, Van Der Werff A. National health systems and their reorientation towards health for all. Guidance for policy-making. 77th ed. Geneva: World Health Organization; 1984.
Evans R. Incomplete vertical integration: The distinctive structure of the health-care industry. In: Van der Graag JJ, editor. Health, economics, and health economics; 1981.
United Nation (2015) Sustainable Development Goals; https://sdgs.un.org, Accessed Jul 2020.
United Nations (2019) Sustainable Development Goals: SDG indicator meta repository. https://unstats.un.org/sdgs/metadata/; https://unstats.un.org/sdgs/metadata/files/Metadata-03-08-02.pdf; Accessed Jun 2020.
van Olmen J, Criel B, Damme WV, Marchal B, Belle SV, Dormael MV, et al. Analysing health systems to make them stronger. Stud Health Serv Org Policy. 2010;27 https://www.itg.be/Files/docs/shsop/shsop27.pdf; Accessed Jan 2017.
George A. Using accountability to improve reproductive health care. Reprod Health Matters. 2003;11(21):161–70. https://doi.org/10.1016/s0968-8080(03)02164-5.
Cleary SM, Molyneux S, Gilson L. Resources, attitudes, and culture: an understanding of the factors that influence the functioning of accountability mechanisms in primary health care settings. BMC Health Serv Res. 2013 Aug;16(13):320. https://doi.org/10.1186/1472-6963-13-320.
Cabieses B, Bird P. Glossary of access to health care and related concepts for low- and middle-income countries (LMICs): a critical review of international literature. Int J Health Serv. 2014;44(4):845–61. https://doi.org/10.2190/HS.44.4.j.
Transparency International. The ignored pandemic behind COVID-19: the impact of corruption on healthcare; 2020 http://ti-health.org/wp-content/uploads/2020/12/The-Ignored-Pandemic-Behind-COVID-19-the-impact-of-corruption-on-healthcare.pdf; Accessed Jan 2021.
Berger D. Corruption ruins the doctor-patient relationship in India. BMJ. 2014;348:g3169.
Hanf M, Van-Melle A, Fraisse F, Roger A, Carme B, Nacher M. Corruption kills: estimating the global impact of corruption on children deaths. PLoS One. 2011;6(11):e26990. https://doi.org/10.1371/journal.pone.0026990.
Kirya MT. Promoting anti-corruption, transparency and accountability in the recruitment and promotion of health workers to safeguard health outcomes. Glob Health Act. 2020;13:1. https://doi.org/10.1080/16549716.2019.1701326.
Jacobs E, Baez CC. Local health governance in Tajikistan: accountability and power relations at the district level. Int J Equity Health. 2020;19(1):30. https://doi.org/10.1186/s12939-020-1143-7 PMID: 32122333; PMCID: PMC7053113.
Mukinda FK, Van Belle S, Schneider H. Perceptions and experiences of frontline health managers and providers on accountability in a south African health district. Int J Equity Health. 2020;19:110. https://doi.org/10.1186/s12939-020-01229-w.
Bold T, Molina E, Safir A. Clientelism in the public sector: why public service reforms may not succeed and what to do about it. World Bank. 2017. https://doi.org/10.1596/26257 Accessed Sep 2020.
Hutchinson E, Balabanova D, McKee M. We need to talk about corruption in health systems. Int J Health Policy Manag. 2019;8(4):191–4. https://doi.org/10.15171/ijhpm.2018.123 PMID: 31050963; PMCID: PMC6499907.
García PJ. Corruption in global health: the open secret. Lancet. 2019;394(10214):2119–24. https://doi.org/10.1016/S0140-6736(19)32527-9 Epub 2019 Nov 27. PMID: 31785827.
Transparency International. Corruption perception index 2020; https://images.transparencycdn.org/images/CPI2020_Report_EN_0802-WEB-1.pdf; Accessed Jan 2021.
Mackey TK, Kohler J, Lewis M, Vian T. Combating corruption in global health. Sci Transl Med. 2017;9:eaaf9547.
USAID, Access to universal health coverage through high-performing health care: fact sheet; 2019; https://www.usaid.gov/documents/1864/access-universal-health-coverage-through-high-performing-health-care.
Deming WE. The New Economics for Industry, Government, and Education. Boston: MIT Press; 1993. p. 132.
Donabedian A. The quality of medical care. Science. 1978;200(4344):856–64. https://doi.org/10.1126/science.417400 PMID: 417400.
Rothwell WJ, Sullivan R, McLean GN. Practicing organizational development: a guide for consultant: Publisher Jossy-Bass/Pfieffer; 1995.
Ramanujam R, Rousseau DM. Organizational behavior in healthcare—the challenges are organizational, not just clinical. J Organ Behav. 2006;27(7):809–1029. https://doi.org/10.1002/job.412.
Lyman B, Moore C. The learning history: A research method to advance the science and practice of organizational learning in healthcare. J Adv Nurs. 2019;75(2):472–81. https://doi.org/10.1111/jan.13858 Epub 2018 Nov 12.
Bandura A. Social Foundation of thoughts and actions. Englewood Cliffs: Prentice-Hall; 1986.
Ewart CK. Social action theory for a public health psychology. Am Psychol. 1991;46(9):931–46. https://doi.org/10.1037//0003-066x.46.9.931 PMID: 1958012.
Kahneman D, Miller DT. Norm theory: comparing reality to its alternatives. Psychol Rev. 1986;93(2):136.
Feldman G, Albarracín D. Norm theory and the action-effect: The role of social norms in regret following action and inaction. J Exp Soc Psychol. 2017;69:111–20. https://doi.org/10.1016/j.jesp.2016.07.009.
Zimmerman MA. Psychological empowerment: issues and illustrations. Am J Community Psychol. 1995;23:581–99.
Aiyer SM, Zimmerman MA, Morrel-Samuels S, Reischl TM. From broken windows to busy streets a community empowerment perspective. Health Educ Behav. 2015;42:137–47.
Rupp LA, Zimmerman MA, Sly KW, Reischl TM, Thulin EJ, Wyatt TA, et al. Community-engaged neighborhood revitalization and empowerment: busy streets theory in action. Am J Commun Psychol. 2020;65(1–2):90–106. https://doi.org/10.1002/ajcp.12358 Epub 2019 Jul 16. PMID: 31313329.
Lewis M. Informal payments, and the financing of health care in developing and transition countries. Health Aff (Millwood). 2007;26(4):984–97. https://doi.org/10.1377/hlthaff.26.4.984.
Peters DH. The role of oversight in the health sector: the example of sexual and reproductive health services in India. Reprod Health Matters. 2002;10(20):82–94.
Brinkerhoff DW, Bossert TJ. Health governance: concepts, experience, and programming options. Health Syst. 2008;2020:20.
Lodenstein E, Dieleman M, Gerretsen B, Broerse JEW. Health provider responsiveness to social accountability initiatives in low- and middle-income countries: a realist review. Health Policy Plann. 2017;32(1):125–40. https://doi.org/10.1093/heapol/czw089 PMID: 27375128.
Joshi A, Houtzager PP. Widgets or watchdogs? Public Manag Rev. 2012;14(2):145–62.
Gullo S, Kuhlmann AS, Galavotti C, Msiska T, Nathan MC, Hastings P. Creating spaces for dialogue: cluster-randomized evaluation of CARE's community score card on health governance outcomes. BMC Health Serv Res. 2018;18(1):858. https://doi.org/10.1186/s12913-018-3651-3.
Gurung G, Gauld R, Hill PC, Derrett S. Citizen's charter in a primary healthcare setting of Nepal: an accountability tool or a "mere wall poster"? Health Expect. 2018;21(1):149–58. https://doi.org/10.1111/hex.12596 Epub 2017 Jul 21.
Lodenstein E, Ingemann C, Molenaar JM, Dieleman M, Broerse JEW. Informal social accountability in maternal health service delivery: a study in northern Malawi. PLoS One. 2018;13(4):e0195671. https://doi.org/10.1371/journal.pone.0195671 eCollection 2018.
Aqil A, Lippeveld T, Hozumi D. PRISM framework: a paradigm shift for designing, strengthening, and evaluating routine health information systems. Health Policy Plan. 2009;24(3):217–28. https://doi.org/10.1093/heapol/czp010 Epub 2009 Mar 20. PMID: 19304786; PMCID: PMC2670976.
Stenberg K, Hanssen O, Edejer TT, Bertram M, Brindley C, Meshreky A, et al. Financing transformative health systems towards achievement of the health sustainable development goals: a model for projected resource needs in 67 low-income and middle-income countries. Lancet Glob Health. 2017;5(9):e875–87. https://doi.org/10.1016/S2214-109X(17)30263-2 Epub 2017 Jul 17.
Li Z, Li M, Fink G, Bourne P, Bärnighausen T, Atun R. User-fee-removal improves equity of children's health care utilization and reduces families' financial burden: evidence from Jamaica. J Glob Health. 2017;7(1):010502. https://doi.org/10.7189/jogh.07.010502 PMID: 28685038; PMCID: PMC5481893.
Walque DD, Fernald L, Gertler P, Hidrobo M. Cash transfers and child and adolescent development. In: Bundy DAP, Silva ND, Horton S, Jamison DT, Patton GC, editors. Child and adolescent health and development. 3rd ed. Washington (DC): The International Bank for Reconstruction and Development / The World Bank; 2017. Chapter 23. PMID: 30212143.
Obare F, Abuya T, Matanda D, Bellows B. Assessing the community-level impact of a decade of user fee policy shifts on health facility deliveries in Kenya, 2003-2014. Int J Equity Health. 2018;17(1):65. https://doi.org/10.1186/s12939-018-0774-4 PMID: 29801485; PMCID: PMC5970478.
van Hees SGM, O'Fallon T, Hofker M, Dekker M, Polack S, Banks LM, et al. Leaving no one behind? Social inclusion of health insurance in low- and middle-income countries: a systematic review. Int J Equity Health. 2019;18(1):134. https://doi.org/10.1186/s12939-019-1040-0.
Rudgard WE, Evans CA, Sweeney S, Wingfield T, Lönnroth K, Barreira D, et al. Comparison of two cash transfer strategies to prevent catastrophic costs for poor tuberculosis-affected households in low- and middle-income countries: An economic modelling study. PLoS Med. 2017;14(11):e1002418. https://doi.org/10.1371/journalpmed1002418 eCollection 2017 Nov.
Kim TJ, Vonneilich N, Lüdecke D, von dem Knesebeck O. Income, financial barriers to health care and public health expenditure: a multilevel analysis of 28 countries. Soc Sci Med. 2017;176:158–65. https://doi.org/10.1016/j.socscimed.2017.01.044 Epub 2017 Jan 24.
Mori AT, Meena E, Kaale EA. Economic cost of substandard and falsified human medicines and cosmetics with banned ingredients in Tanzania from 2005 to 2015: a retrospective review of data from the regulatory authority. BMJ Open. 2018;8(6):e021825. https://doi.org/10.1136/bmjopen-2018-02182.
Beargie SM, Higgins CR, Evans DR, Laing SK, Erim D, Ozawa S. The economic impact of substandard and falsified antimalarial medications in Nigeria. LoS One. 2019;14(8):e0217910. https://doi.org/10.1371/journal.pone.0217910 eCollection 2019.
Ozawa S, Haynie DG, Bessias S, Laing SK, Ngamasana EL, Yemeke TT, et al. Modeling the economic impact of substandard and falsified Antimalarials in the Democratic Republic of the Congo. Am J Trop Med Hyg. 2019;100(5):1149–57. https://doi.org/10.4269/ajtmh.18-0334.
Evans DR, Higgins CR, Laing SK, Awor P, Ozawa S. Poor-quality antimalarials further health inequities in Uganda. Health Policy Plan. 2019;34(Supplement_3):iii36–47. https://doi.org/10.1093/heapol/czz012.
Tanahashi T. Health service coverage and its evaluation. Bull World Health Organ. 1978;56(2):295–303.
Baltussen R, et al. Perceived quality of care of primary health care services in Burkina Faso. Health Policy Plann. 2002;17(1):42–8.
Rosen HE, Lynam PF, Carr C, Reis V, Ricca J, Bazant ES, et al. Direct observation of respectful maternity care in five countries: a cross-sectional study of health facilities in east and southern Africa. BMC Pregnancy Childbirth. 2015;23(15):306. https://doi.org/10.1186/s12884-015-0728-4.
Thiede M, Akweongo R, McIntyre D. Exploring the dimensions of access. In: McIntyre D, Mooney G, editors. The economics of health equity. New York: Cambridge University Press; 2007.
Chuma J, Okungu V, Molyneux C. Barriers to prompt and effective malaria treatment among the poorest population in Kenya. Malaria J. 2010;9(1):144.
Rafter N, Hickey A, Condell S, et al. Adverse events in healthcare: learning from mistakes. QJM. 2014;108:273–7.
Roberts KH, Rousseau DM. Research in nearly failure free, high reliability organizations: having the bubble. IEEE Trans Eng Manag. 1989;36(2):132–9.
Rochlin GI. Reliable organizations: present research and future directions. J Contingencies Crisis Manag. 1996;2(4):55–9.
Perrow C. Normal accidents, living with high-risk technologies. Princeton, NJ: Princeton University press. 1999; first published in 1984, Basic Press.
Marrow R. Leading high-reliability organizations in healthcare; Boca Raton, Florida; London, England; New York. New York: CRC Press; 2016.
Weick KE, Sutcliffe KM, Obstfeld D. Organizing for high reliability: processes of collective mindfulness. In: Sutton RI, Staw BM, editors. Research in organizational behavior, vol. 21: Elsevier Science/JAI Press; 1999. p. 81–123.
Tolk JN, Cantu J, Beruvides MG. High reliability organization research: Transforming theory into practice, A literature review for healthcare. Minneapolis: Paper presented at the American Society for Engineering Management International Annual Conference; 2013.
Joint Commission Center for Transforming Healthcare. (2009). Joint Commission Center for Transforming Healthcare takes aim at patient safety failures. Retrieved from: http://www.centerfortransforminghealthcare.org/.
Chassin MR, Loeb JM. High-reliability health care: Getting there from here. Milbank Q. 2013;91(3):459–90.
Vogus T, Iacobucci D. Creating highly reliable health care: how reliability-enhancing work practices affect patient safety in hospitals. Indust Labor Relat Rev. 2016;69:4.
Vogus TJ, Hilligoss B. The underappreciated role of habit in highly reliable healthcare. BMJ Qual Saf. 2016;25(3):141–6. https://doi.org/10.1136/bmjqs-2015-004512 Epub 2015 Sep 2. PMID: 26337423.
Kaufmann D, Kraay A, Mastruzzi M. Governance Matters IV: Governance Indicators for 1996–2004. Washington, D.C: World Bank Policy Research Working Paper No. 3630; 2005.
Kaufmann D, Kraay A, Mastruzzi M, The worldwide governance indicators: methodology and analytical issues (September 2010). World Bank policy research working paper no. 5430. Available at SSRN https://ssrn.com/abstract=1682130; Accessed Jul 2019.
Hallward-Driemeier M, Khun-Jush G, Pritchett L, “Deals versus rules: policy implementation uncertainty and why firms hate it”. NBER Working Paper 16001. (2010).
Idler EL, Benyamini Y. Self-rated health and mortality: a re- view of twenty-seven community studies. J Health Soc Behav. 1997;38(1):21–37.
Vuorisalmi M, Lintonen T, Jylha M. Global self-rated health data from a longitudinal study predicted mortality better than comparative self-rated health in old age. J Clin Epidemiol. 2005;58(7):680–7.
McHugh JE, Lawlor BA. Perceived health status is associated with hours of exercise per week in older adults independent of physical health. J Phys Act Health. 2013;10:1102–8.
Bryan CJ, Adams GS, Monin B. When cheating would make you a cheater: implicating the self prevents unethical behavior. J Exp Psychol Gen. 2013;142:1001–5.
Gittelman S. Lange V, cook WA, accounting for social-desirability Bias in survey sampling. J Advert Res. 2015;55(3):242.
Larson RB. Controlling social desirability bias. Int J Mark Res. 2019;61(5):534–47 10.177/147008531805305.
Kwak DH, Philipp H, Sung SK. Measuring and Controlling Social Desirability Bias: Applications in Information Systems Research. J Assoc Inf Syst. 2019;20(4):317.
Everit BS. Multivariate analysis: the need for data, and other problems. Br J Psychiatry. 1975;126:237–40.
Cattell RB. The scientific use of factor analysis in behavioral and life sciences. New-York: Springer; 1978. p. 618.
Kline P. Psychometrics and psychology. London: Academic Press; 1979. p. 381.
Gorsuch RL. Factor analysis. 2nd ed. Hillsdale: Lawrence Erlbaum Associates; 1983. p. 448.
Hair JE, Anderson RE, Tatham RL, Black WC. Multivariate data analysis: with readings. Englewood Cliffs: Prentice -Hall; 1995. p. 757.
Bonett DG, Wright TA. Cronbach’s alpha reliability: interval estimation, hypothesis testing, and sample size planning; journal of organizational behavior. J Organiz Behav. 2015;36:3–15. https://doi.org/10.1002/job.1960.
Sears DO. College sophomores in the laboratory: influences of a narrow data base on social psychology's view of human nature. J Pers Soc Psychol. 1986;51(3):515.
Henry PJ. College sophomores in the laboratory redux: influences of a narrow data base on social psychology's view of the nature of prejudice. Psychol Inq. 2008;19(2):49–71.
Anderson CA, Allen JJ, Plante C, Quigley-McBride A, Lovett A, Rokkum JN. The MTurkification of social and personality psychology. Personal Soc Psychol Bull. 2018;0146167218798821:1.
Kaufmann D, Kraay A, Mastruzz M. Response to “What Do the Worldwide Governance Indicators Measure?”. Forthcoming Eur J Dev Res. 2009; https://info.worldbank.org/governance/wgi/Home/Documents; Accessed Jul 2020.
Independent Evaluation Group. The World Bank's country policy and institutional assessment: an IEG evaluation. Washington, DC: World Bank; 2010. © World Bank. https://openknowledge.worldbank.org/handle/10986/13547 License: CC BY 3.0 IGO accessed in Oct 2020
Berinsky AJ, Huber GA, Lenz GS. Evaluating online labor markets for experimental research: Amazon. com's mechanical Turk. Polit Anal. 2012;20(3):351–68.
Owen JE, Bantum EO, Criswell K, Bazzo J, Gorlick A, Stanton AL. Representativeness of two sampling procedures for an internet intervention targeting cancer-related distress: a comparison of convenience and registry samples. J Behav Med. 2014;37(4):630–41. https://doi.org/10.1007/s10865-013-9509-6 Epub 2013 May 4. PMID: 23645145; PMCID: PMC3842405.
Weinberg JD, Freese J, McElhattan D. Comparing data characteristics and results of an online factorial survey between a population-based and a Crowdsource- recruited sample. Sociol Sci. 2014;1:1.
Catania JA, Margaret Dolcini M, Orellana R, Narayanan V. Nonprobability and probability-based sampling strategies in sexual science. J Sex Res. 2015;52(4):396–411. https://doi.org/10.1080/00224499.2015.1016476.
Goldberg M, van der Linden S, Ballew M, Rosenthal S, Leiserowitz A. Convenient but biased? The reliability of convenience samples in research about attitudes toward climate change. https://doi.org/10.31219/osf.io/2h7ashttps://www.researchgate.net/publication/335370073 downloaded Nov 202.
WHO, 2019-uhc-report.pdf (who.int).
Nunnally J. Psychometrics theory. New York: McGraw-Hill; 1967.
Nunnally J. Psychometrics theory. 2nd ed. New York: McGraw-Hill; 1978.
Bujang MA, Omar ED, Baharum NA. A Review on Sample Size Determination for Cronbach's Alpha Test: A Simple Guide for Researchers. Malays J Med Sci. 2018;25(6):85–99. https://doi.org/10.21315/mjms2018.25.6.9 Epub 2018 Dec 28. PMID: 30914882; PMCID: PMC6422571.
Bonett DG. Sample size requirements for testing and estimating coefficient alpha. J Educ Behav Stat. 2002;27(4):335–40. https://doi.org/10.3102/10769986027004335.
Bonett DG. Sample size requirements for comparing two alpha reliability coefficients. Appl Psychol Meas. 2003;27(1):235–40. https://doi.org/10.1177/0146621602239477.
Fried EI. What are psychological constructs? On the nature and statistical modelling of emotions, intelligence, personality traits and mental disorders. Health Psychol Rev. 11(2):130–4. https://doi.org/10.1080/17437199.2017.1306718 Accessed in Dec 2020.
Van Der Maas HLJ, Kan KJ, Marsman M, Stevenson CE. Network models for cognitive development and intelligence. J Intell. 2017;5:16. https://doi.org/10.3390/jintelligence5020016 Accessed in Dec 2020.
Mõttus R, Allerhand M. Why do traits come together? The underlying trait and network approaches. In: Zeigler-Hill IV, Shackelford TK, editors. The sage handbook of personality and individual differences: SAGE Publications Ltd; 2018. p. 130–50. https://doi.org/10.4135/9781526451163.n6. Accessed in Dec 2020.
STATcompiler: The DHS program, USAID - https://www.statcompiler.com/en/ accessed August 2020. WHO - Global Health Observatory https://www.who.int/data/gho/data/indicators/indicator-details/GHO/out-of-pocket-expenditure-as-a-percentage-of-total-expenditure-on-health; Accessed Aug 2020.
WHO - Global Health Observatory https://www.who.int/data/gho/data/indicators/indicator-details/GHO/out-of-pocket-expenditure-as-a-percentage-of-total-expenditure-on-health; accessed August 2020.
World Health Organization (WHO) International Health Regulations 2019 https://extranet.who.int/e-spar/#capacity-score; Accessed Aug 2020.
Bamberger M, Rugh J, Marby L. Real world evaluation: working under budget, time, data, and political constraints: Sage Publication; 2006.
Mulgan R. Accountability: an ever-expanding concept? Public Adm. 2000;78:555–73.
Schedler A. Conceptualizing accountability. In: Schedler A, Diamond L, Plattner MF, editors. The self-restraining state: power and accountability in new democracies. Boulder: ynne Rienner publishers; 1999. p. 13–29.
Brinkerhoff DW. Accountability and health systems: toward conceptual clarity and policy relevance. Health Policy Plan. 2004;19(6):371–9. https://doi.org/10.1093/heapol/czh052 PMID: 15459162.
Vian T. Anti-corruption, transparency and accountability in health: concepts, frameworks, and approaches. Glob Health Action. 2020;13(sup1):1694744. https://doi.org/10.1080/16549716.2019.1694744 PMID: 32194010; PMCID: PMC7170369.
Barasa EW, Cloete K, Gilson L. From bouncing back, to nurturing emergence: reframing the concept of resilience in health systems strengthening. Health Policy Plan. 2017;32(suppl_3):iii91–4. https://doi.org/10.1093/heapol/czx118 Retrieved from https://academic.oup.com/heapol/article-lookup/.
Braithwaite J, Wears RL, Hollnagel E. Resilient health care: turning patient safety on its head. Int J Qual Health Care. 2015;27(5):418–20. https://doi.org/10.1093/intqhc/mzv063 Retrieved from https://academic.oup.com/intqhc/articlelookup/.
Gilson L, Barasa E, Nonhlanhla N, Cleary S, Goudge J, Molyneux S, et al. Everyday resilience in district health systems: emerging insights from the front lines in Kenya and South Africa. BMJ Glob Health. 2017;2(e000224) Retrieved from: https://gh.bmj.com/content/2/2/e000224.
Health Finance & Governance (HFG) Project. Health systems assessment approach a how-to manual. Version 3.0. Bethesda: Health Finance & Governance Project, Abt Associates Inc.; 2017.
Went, D. Health System Rapid Diagnostic Tool: Framework, operational Guide, and metrics to measure the strengths of priority health system Functions; (2012). Family Health International-FHI360.
Veillard J, Cowling K, Bitton A, Ratcliffe H, Kimball M, Barkley S, et al. Better Measurement for Performance Improvement in Low- and Middle-Income Countries: The Primary Health Care Performance Initiative (PHCPI) Experience of Conceptual Framework Development and Indicator Selection. Milbank Q. 95(4):2017, 836–883.
Rohrer-Herold K. UHC 2030 technical working group on health systems assessments: review of health system assessment tools and approaches; (2017) https://www.uhc2030.org/fileadmin/uploads/uhc2030/Documents/About_UHC2030/UHC2030_Working_Groups/2017_Health_Systems_Assess_Working_Group/First_face_to_face_meeting_2017/D1_Rohrer_Consultant.pdf; Accessed Aug 2020.
Aqil A, Silvestre E, Hotchkiss D, Maniscalco L, Health Systems Strengthening Monitoring, Evaluation and Learning Guide; MEASURE Evaluation, USAID. 2017 https://www.measureevaluation.org/resources/publications/tr-17-167c; Accessed Jan 2020.
We highly appreciate the support from Mariam Malik, Sangita Patel, Anuradha Jain, Berhan Hagos, Joseph Lachica, HSS network, RHINO, Christian Hospitals network for distributing the HPHC tool in different parts of the world.
The authors’ views expressed in this publication are their own and do not necessarily reflect the views of the USAID or the United States Government.
The authors did not receive any external funding.
Ethics approval and consent to participate
The research study was implemented in compliance with the United States Federal Codes of Regulations (CFR) on protection of human subjects (22 CFR 225) and exemption for Institutional Review Board (22 CFR 225.104). The ethical statement including informed consent alteration and research approval statement by relevant authority, Kelly Saldana, Director, Office of Health Systems, USAID, are attached.
We also confirm that all methods were performed in accordance with relevant guidelines and regulations of the journal.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
Aqil, A., Saldana, K., Mian, N.u. et al. Reliability and validity of an innovative high performing healthcare system assessment tool. BMC Health Serv Res 23, 242 (2023). https://doi.org/10.1186/s12913-022-08852-z
- Universal health coverage
- Health systems
- System practice