Skip to main content

Balancing adaptability and standardisation: insights from 27 routinely implemented ICHOM standard sets

Abstract

Background

Healthcare systems around the world experience increasing pressure to control future growth of healthcare expenditures. Among other initiatives, quality and value-based benchmarking has become an important field to inform clinical evaluation and reimbursement questions. The International Consortium for Health Outcomes Measurement (ICHOM) has become one of the driving forces to translate scientific evidence into standardized assessments that are routinely applicable in day-to-day care settings. These aim to provide a benchmarking tool that allows the comparison and competition of health care delivery on the basis of value-based health care principles.

Methods

This work focuses on the consolidation of the ICHOM methodology and presents insights from 27 routinely implemented Standard Sets. The analysis is based on a literature review of the ICHOM literature repository, a process document review and key informant interviews with ICHOM’s outcomes research and development team.

Results

Key findings are that the scope of ICHOM Standard Sets shifted from a more static focus on burden of disease and poorly standardized care pathways to a more dynamic approach that also takes into account questions about the setting of care, feasibility of implementing a benchmarking tool and compatibility of different Standard Sets. Although certain overlaps exist with other initiatives in the field of patient reported outcomes (PRO), their scopes differ significantly and they hence rather complement each other. ICHOM pursues a pragmatic approach to enable the benchmarking and the analysis of healthcare delivery following the principles of value-based healthcare.

Conclusion

The ICHOM Standard Sets complement other initiatives in the field of patient-reported outcomes (PRO) and functional reporting by placing a particular focus on healthcare delivery, while other initiatives primarily focus on evaluation of academic endpoints. Although ICHOM promotes a pragmatic approach towards developing and devising its Standard Sets, the definition of standardized decision making processes emerged as one of the key challenges. Furthermore, the consolidation of core metrics across number of disease areas to enable the parallel implementation of different Standard Sets in the same care setting is an important goal that will enable the widespread implementation of patient-reported outcome measures (PROM).

Peer Review reports

Introduction

Healthcare expenditures have been exceeding real growth of GDP for decades and more and more healthcare systems are under pressure to control further growth of their spending [1]. The U.S. reached $ 3.8 trillion in healthcare spending in 2019, another substantial increase of 4.6% from the previous year, and healthcare spending now accounts for 17.7% of the total GDP [2]. Although the U.S. is not alone in confronting this challenge [3], its annual per-capita spending is currently almost twice that of other high-income countries [4]. Despite high healthcare expenditures, the U.S. performs poorly on key health performance indicators such as life expectancy, preventable and amenable mortality, and population coverage for a core set of services [5].

Given this backdrop, the U.S. government has been reorienting its approach towards care delivery and now emphasizes quality and value as new core metrics used to generate evidence, determine reimbursement and empower patients to make informed decisions about healthcare service utilisation. While early efforts for the introduction of performance-based payment systems date back to the 1990s [6, 7], major legislative efforts (i.e. the Affordable Care Act and the Medicare Access and CHIP Reauthorization Act) are now greatly promoting the incorporation of quality- and value-based principles into research [8] and reimbursement schemes [9], which have moved the aspects of patient-centered outcome (PRO) reporting, quality and value up on the public agenda. How best to deliver, and measure delivery of, value-based healthcare remains greatly debated.

In this evolving field, the International Consortium for Health Outcomes Measurement (ICHOM) has emerged as one of the organisations advancing value-based healthcare, with a particular emphasis on patient-centered outcome measurement and quality. ICHOM’s mission is rooted in Porter’s and Teisberg’s seminal work on value-based healthcare [10], which defines value as improvement in health outcomes per dollar spent. It proposes to shift competition from zero-sum (i.e. any advantage that is gained by one party is lost by another party) to increased value for patients [11, 12]. ICHOM’s work driven by this mission includes the development of condition-specific measures of health outcome (Standard Sets), their respective validation, and the facilitation of the implementation of these Standard Sets. These efforts aim towards fostering benchmarking and collaborative learning between healthcare contexts on a national and global level. Since its founding in 2012, ICHOM has developed 40 Standard Sets in consultation with more than 900 experts and, along with partners, has supported implementation in selected health care settings in 44 countries [13]. In total, the Standard Sets available cover almost 50% of the global burden of disease [14]. This article focuses on the articles that have been published as the primary reports about the respective Standard Sets, examines the spectrum of conditions covered and consolidates the methodology applied.

Methodology

This was a review of published literature describing development of ICHOM Standard Sets. Of the 40 available ICHOM Standard Sets, 27 have associated publications in peer-reviewed journals which describe the Sets development process (Table 1, Fig. 1). The remaining 13 Standard Sets are, at the time of our literature research, in the process of publication. Table 1 lists the names of the 27 Standard Sets for which publications were reviewed. Publications were from 2015 to 2021. The publications reviewed were identified using ICHOM’s online repository ‘Connect’, and through consultation with ICHOM’s outcomes research and development team. The first author read the articles in full and collated findings regarding scope, methodology, consensus finding process, domains covered and measures used. All findings were discussed with members of ICHOM’s team and reviewed in context of all published Standard Sets.

Table 1 Peer-reviewed ICHOM Standard Sets, conditions addressed, disease spectrum and estimated burden of disease
Fig. 1
figure 1

Distribution of conditions addressed by published ICHOM Standard Sets

Results

The spectrum of available ICHOM standard sets

The published available 27 ICHOM Standard Sets cover a vast range of the global burden of disease and address 12 of the 25 leading causes of disability globally in 2019 [42]. In addition to covering high burden conditions, we also observed that a subset of Standard Sets focused on conditions that represented challenges with managing them. Examples include the Standard Sets for overactive bladder syndrome (OAB) [29], care in the context of a cleft palate [21] and overall oral health in adults [40]. Additionally, some Standard Sets do not address medical conditions in the narrow sense, but rather focus on living status and wellbeing across the life course. Examples are Older Persons [32], overall Pediatric Health [39], and Pregnancy and Childbirth [33]. One Standard Set focuses on the highly prevalent condition of hypertension, but with a particular focus of measurement and care appraisal in low and middle income countries (LMIC) [35].

Nonetheless, ICHOM Standard Sets are developed from an international perspective and their wide applicability is generally a distinctive feature that sets ICHOM Standard Sets apart from other initiatives in the field of PRO and functional outcome reporting, which often focus on a narrow group of conditions [43,44,45]. Yet, the broad scope and a decentralized process of Standard Set development require a flexible methodological approach that allows adaptation to the clinical setting of each condition of interest.

The process of the ICHOM standard set development

Key steps of the development are illustrated in Fig. 2. The methods of development for each Set are outlined in detail in the referenced publications included in Table 1. Below, we summarise these steps highlighting patterns in the development process across Standard Sets and over time.

Fig. 2
figure 2

Integrated framework for the composition of an ICHOM Standard Set. The cycle of care concept (black) is adapted from Porter [46]. Domains are generic (red) and span across the entire cycle of care, but are not necessarily included in all tiers (dashed). Measures (green) are selected as they are considered to meet the criteria for inclusion. The decision for inclusion or exclusion into a Standard Set is based on the Delphi process outlined below. Case-mix variables (yellow) are included to allow for comparisons between sub-groups and stratifications

Initiation of a standard set development project

Since the establishment of ICHOM, the decision to develop a specific Standard Set has primarily been driven to address conditions with a particularly high burden of disease and, secondarily, those with poorly standardized clinical management for the given condition (Table 1). In this context, the available evidence suggests that a structured framework for capturing patient-centered outcomes in clinical practice may reduce costs and drive up quality of care and would hence result in a higher healthcare value [47], which addresses the very principle of VBHC, as outlined before [12, 46].

ICHOM has leveraged its stakeholder network to monitor conditions and clinical care settings for which a new Standard Set might be beneficial. Initially more statically focused on disease prevalence and variability in quality and care delivery, the selection of a condition has evolved to a more dynamic shared decision-making process between medical experts with experience in the field of VBHC, the burden of disease and the feasibility of devising a comprehensive, yet condensed Standard Set for the respective condition.

After selecting an appropriate condition, ICHOM composes an international and interdisciplinary working group following a purposive sampling mechanism [48], as exemplified by Obbarius and colleagues [27]. This approach enhances scientific rigor and independence of the ICHOM Standard Set development and aims to include representation across key expertise and perspectives a) clinical, b) methodological (i.e. PRO and health system evaluation), c) geographical, d) socioeconomic terms and e) and patients. The inclusion of patients or patient representatives is mandatory and these members account for up to 25% of all working group members [36]. A typical working group consists of 20–25 members, ranging in size from 12 [30] to 31 members [32].

To coordinate the efforts of the working group, ICHOM assigns a project team that performs literature research and devises proposals for the working group meetings. The project team includes therefore an ICHOM-based project leader with significant experience in the field of PRO and the composition of Standard Sets and a research fellow, typically seconded from an external academic institution and with condition-specific expertise (e.g. in clinical practice and/or epidemiology). The project team organizes meetings, polls and other administrative tasks, but is set up to not exert influence on the decision-making of the working groups that is informed by technical (evidence-based) and lived (experience-based) expertise. Additionally, a project chair, typically a senior clinician with a strong scientific background in the respective field, supports the work with subject-matter expertise and clinical advice. Typically, these clinicians pilot selected PROMs and drafted Standard Sets in their clinical work to provide advisory support for the ongoing development and implementation of the respective Standard Set.

Process of evidence collection and structuring

The project team’s core task is to conduct a systematic literature review following the PRISMA guidelines for systematic reviews [49] and the compilation of relevant outcomes, which provides the basis for all Standard Set developments. Typically, case registries are included if possible and working group members are encouraged to identify additional relevant resources. Outcome domains of interest center around a) disease-specific survival and disease control, b) symptoms, c) functioning and quality of life, d) disutility of care and e) healthcare utilization. These domains have been exemplarily outlined by Kim and colleagues [28]. Yet, some working groups adjust the domains to be clinically appropriate [21] for the given condition under review or define different domains for different periods of the cycle of care [17, 18].

To ensure comparability between patient cohorts, however, a standardized set of case-mix variables is included in every Standard Set. Case-mix variables typically cover a) demographics, b) baseline clinical status (e.g. disease stage), c) baseline functional status (e.g. assessment of physical activity or functional scores) and d) treatment factors (e.g. history of surgical treatments for condition of interest).

To reflect the full cycle of care of each condition in the outcomes captured by the respective Standard Set, ICHOM follows a three-tier framework. The principle of the three-tiered approach on covering the entire cycle of care has been discussed by Porter [12]. Generally, the lower tier domains can only be addressed when the higher tier domains are met. The structure of the three-tier framework is, for universal applicability, generic and the primary tier includes domains on the health status achieved or retained, with a focus on survival and measures of the degree of health or recovery at the initial stage. The second tier represents the process of recovery and incorporates measures on functional outcomes and indicators of disutility of care. Lastly, the third tier addresses the sustainability of health and includes measures on the sustainability of the recovery and potential long-term consequences related to the care provided. Notably, the framework suggests a feedback loop back to the first tier to ensure that eventual consequences of the process of care are represented in the outcome measures in the primary (i.e. highest) tier.

Respective measures are selected on the basis of their a) psychometric properties, b) frequency among the patient population of interest to ensure a sufficient coverage of all patients, c) potential impact/relevance of the selected measure on the patient, d) clinical interpretability, and, lastly, e) the feasibility of capturing the measure of interest in a clinical setting. A detailed account on the last criterion is presented in the article by de Roos and colleagues [30]. More recently, working groups (12 out of 27 in total) also performed focus group interviews with patients on the appropriateness of the domains and measures selected in a structured and explicit way before moving to the consensus-building stage of the development process, as outlined below. The integration of the approach is depicted in Fig. 2.

Delphi process and the composition of a standard set

Upon completing the collection of available evidence and an optional focus group interview, the working groups convene iteratively for six to eleven remote meetings. After each meeting, they submit anonymous follow-up surveys to build consensus for the domains covered and measures included. A quorum of 75% attendance was defined to account for time zone differences and other organizational issues (e.g. Allori and colleagues [21]). The agenda of the discussion panels is decided jointly by the project team and the working group and evaluates the proposed scope, included domains and specific measures. A modified Delphi process is employed to form consensus among the working group participants to define the core list of outcomes. Although different Delphi approaches have been referenced for the various Standard Sets that have been developed [50,51,52,53], the general pattern is similar, requiring agreement on a) the scope, b) the identification of domains and measures, c) their respective inclusion and d) the selection of measurement timepoints. Consensus is achieved when a threshold of 66 to 80% of positive votes is surpassed. To consolidate the core process of decision-making, ICHOM recently proposed a more formalized approach for the Delphi process (Table 2). Votes are casted in the form of an anonymous survey based on a 9-point Likert scale (1 = not important, 5 = somewhat important, 9 = most important), where scores between seven and nine typically represent the equivalent of agreement. Agreement rates between the threshold and 50% lead to a second debate about the item of interest during the next meeting of the working group. All working group participants must endorse the draft in order for it to proceed to the final step of the Standard Set development. Lastly, after internally agreeing on the composition of the Standard Set, some working groups (14 out of 27) submitted their draft for open or expert review. After the successful completion of the review process, the Standard Set is considered finalized and is prepared for dissemination.

Table 2 Consolidated Delphi process to be employed in future Standard Sets to homogenize decision making process

Discussion

The evolution of scope and methodology

Since its establishment in 2012, ICHOM’s scope has evolved from exclusively focusing on developing Standard Sets for conditions with a high disease burden and poorly standardised care towards a more multi-faceted selection approach. This new approach now includes aspects of the clinical care setting and the feasibility of developing a comprehensive, yet condensed Standard Set to facilitate implementation in day-to-day care settings. This underscores ICHOM’s focus on improving the quality of care. Yet, this comes at the cost of conceptual and methodological homogeneity (e.g. a separate Standard Set for different stages of the same condition [15, 16], instead of only one Standard Set for reasonably similar conditions [20, 23]), which requires even closer and more frequent consultations with stakeholders as part of the consensus-based decision-making process. This challenge has been acknowledged elsewhere [13], but constitutes one of the key levers to provide flexibility and broad applicability of the ICHOM methodology.

Similarly, the methodology has evolved from a strong and static emphasis of the tier-based approach by Porter [12, 46] to a more dynamic and flexibly applicable approach: Although the principles of Porter’s work continue to provide the foundation for ICHOM’s Standard Sets development approach, some working groups have developed different approaches to account for the nature of the specific condition. Mahmud and colleagues, for example, assess the domains outlined above along the specific surgical procedure of the condition addressed [17], whereas McNamara and colleagues distinguish between longitudinal and procedure-specific outcome measures when discussing coronary artery disease [18]. Salinas and colleagues, lastly, assess the patient-reported health status mostly via the PROMIS-10 questionnaire [54] and, hence, rely on a validated core set of patient-centered outcomes that is only complemented by selected individual parameters to accentuate the scope of the Standard Set accurately [25]. This emphasizes that ICHOM does not aim to follow a methodological dogma, but promotes a pragmatic approach towards selecting an appropriate methodology that is guided by principles rather than rules. To sustain this approach, however, a core concern of ICHOM is a reliable and reproducible decision-making process for the composition of its Standard Sets and, hence, a more precise outline of the Delphi process methodology has been proposed (Table 2), which is set to become the standard in all future Standard Sets.

Validation and implementation

For a thorough validation of the proposed Standard Sets, external input, review and feedback proved to be critical. However, the later Standard Sets in particular faced significant challenges in recruiting patient focus groups ex ante. An internal focus group, composed of patients and patient representatives of members of the working group, eventually proved to be helpful and is now becoming the standard approach for early patient input prior to the working group meetings.

Additionally, an open review process was enabled through the distribution of Standard Set drafts to patients and provider organisations. Additionally, post-implementation studies have been published in peer-reviewed scientific journals involving numerous Standard Sets that provide a valuable source of feedback for the continuous improvement of existing Standard Sets and guidelines for future projects: Although the benefits of PRO and the structure provided by the Standard Sets are much acknowledged [55,56,57], heterogeneity among measures included and increased organizational work pose challenges to the implementation [58, 59]. ICHOM itself conducts ongoing internal reviews and identifies a lack of clinical resources, challenges around the implementation of Standard Sets in existing clinical routines and cost-related issues for the use of proprietary or licensed PROMs as challenges that need to be addressed. Many of these issues have also been highlighted by other researchers who propose the implementation generic sets of PROMs instead of condition-specific sets of PROMs to a) enable the assessment of PRO across different conditions, b) the use in the context of multimorbidity and c) the easy-to-use implementation at the point of care, e.g. through short forms or device-based computerized adaptive tests (CAT) [60]. More recent work has also focused on the compatibility of ICHOM Standard Sets to already established frameworks and highlights the importance of a further homogenisation between proposed PROMs and established patient-reported outcomes measurement information systems (PROMIS) to facilitate the collection of PROMs at the point of care [61]. ICHOM has been acting upon this valuable feedback and has launched a harmonisation initiative that focuses on the consolidation of semantic meaning across core measures, which can be applied across different spectra of conditions without a condition-specific adaptation [14]. This initiative provided the basis for the establishment of an ICHOM taxonomy that eliminates redundancy from existing Standard Sets and enables the interoperable use of Standards Sets in the existing, machine-readable formats SNOMED CT and LOINC [14]. Further efforts also aim at the realignment of PROMs across different countries and healthcare systems, respectively.

ICHOM’s position in relation to other PRO / functional outcome reporting initiatives

ICHOM considers itself a complement in the growing field of initiatives working towards the patient-centered improvement of outcome reporting and puts a particular emphasis on the measurement of quality in care for a broad spectrum of different conditions. In contrast, other well established initiatives (e.g. OMERACT [45, 62], SONG [43]) cover a more narrow spectrum and apply a much more condition-specific methodology, specifically developed to meet the requirements of rheumatological conditions or clinical manifestations of end-stage renal disease (ESRD), respectively. Additionally, although their focus has been shifting towards care-related issues (e.g. SONG requires outcomes to be measured “in a meaningful, appropriate, and easy way as accurately as possible” [63]), both initiatives greatly emphasize endpoint homogenisation for scientific reporting. Due to the different focus, we do not consider the ICHOM Standard Sets on ESRD by Verberne and colleagues [31] and on inflammatory arthritis by Oude Voshaar and colleagues [36] as redundant, but as a complementing perspective on the care delivered for these specific conditions, providing a quality monitoring tool as compared to a trial reporting tool. We are, however, aware of the fact that the parallel implementation of multiple Standard Sets and with different foci is challenging and most likely not practical in a clinical setting. Additionally, without any particular scope, but with a strong focus on methodology, the COMET initiative [64] has also become a key player in the field of PRO / functional outcome reporting. We see that all of the initiatives above use aspects of the COMET methodology in their own approaches and two ICHOM Standard Sets explicitly state the COMET guidelines as a core component of their methodology [26, 41]. Other governmental initiatives, like the US-based National Quality Forum (NQF) or the Patient-Centered Outcomes Research Institute (PCORI) and the Dutch Programme for Outcome Based Healthcare [65] support efforts for the development of PRO / functional outcome reporting through endorsements of applicable measures and funding, respectively. The widespread interest in PROMs has furthermore led to the exploration of adjacent fields like quality improvement and health technology assessment (HTA) [66]. These efforts underscore, from our perspective, the importance of value-based principles for future developments in patient care, performance assessment and quality benchmarking in modern healthcare systems. Yet, they also highlight the complexity of the field and the demand for further alignment and a focus on the compatibility between the different initiatives and their purposes.

Conclusion

In the context of unsustainable healthcare spending and sub-optimal value of delivered care, ICHOM has become one of the shaping forces in the field of PRO / functional outcome measurement coupled with clinical outcomes. With its aspiration to provide a broadly applicable approach for a diverse spectrum of medical conditions and care settings, its methodology has evolved over the past decade from a theoretical framework to a repeatedly validated, dynamic methodological approach that facilitates the development of a scientifically valid Standard Sets. Yet, further homogenisation – along the dimensions of conditions covered and measures collected - is required to facilitate the compatibility with other initiatives and the widespread implementation in clinical settings.

Availability of data and materials

All primary literature used is available as referenced, yet internal process documents are only available upon reasonable request from the corresponding author (L.B., leo.benning@uniklinik-freiburg.de).

Abbreviations

CAT:

Computerized Adaptive Test

CHIP:

Children’s Health Insurance Program

COMET:

Core outcome measures in effectiveness trials

ESRD:

End-stage renal disease

GDP:

Gross domestic product

HTA:

Health technology assessment

ICHOM:

International Consortium for Health Outcomes Measurement

LMIC:

Low and middle income countries

NQF:

National quality forum

OAB:

Overactive bladder syndrome

OMERACT:

Outcome measures in Rheumatology

PCORI:

Patient-centered outcomes research institute

PRISMA:

Preferred reporting items for systematic reviews and meta-analyses

PRO:

Patient-reported outcome(s)

PROMIS:

Patient-reported outcomes measurement information system

SONG:

Standardised outcomes in Nephrology

VBHC:

Value-based health care

References

  1. Keehan SP, Cuckler GA, Poisal JA, et al. National Health Expenditure Projections, 2019-28: expected rebound in prices drives rising spending growth. Health Aff (Millwood). 2020;39(4):704–14.

    Article  PubMed  Google Scholar 

  2. Martin AB, Hartman M, Lassman D, Catlin A, National Health Expenditure Accounts T. National Health Care Spending in 2019: steady growth for the fourth consecutive year. Health Aff (Millwood). 2021;40(1):14–24.

    Article  PubMed  Google Scholar 

  3. Papanicolas I, Woskie LR, Jha AK. Health care spending in the United States and other high-income countries. JAMA. 2018;319(10):1024–39.

    Article  PubMed  Google Scholar 

  4. Papanicolas I, Marino A, Lorenzoni L, Jha A. Comparison of Health care spending by age in 8 high-income countries. JAMA Netw Open. 2020;3(8):e2014688.

    Article  PubMed Central  PubMed  Google Scholar 

  5. OECD. Health at a Glance. p. 2019.

  6. Kouides RW, Lewis B, Bennett NM, et al. A performance-based incentive program for influenza immunization in the elderly. Am J Prev Med. 1993;9(4):250–5.

    Article  CAS  PubMed  Google Scholar 

  7. Morrow RW, Gooding AD, Clark C. Improving physicians’ preventive health care behavior through peer review and financial incentives. Arch Fam Med. 1995;4(2):165–9.

    Article  CAS  PubMed  Google Scholar 

  8. Esmail LC, Barasky R, Mittman BS, Hickam DH. Improving comparative effectiveness research of complex Health interventions: standards from the Patient-Centered Outcomes Research Institute (PCORI). J Gen Intern Med. 2020;35(Suppl 2):875–81.

    Article  PubMed Central  PubMed  Google Scholar 

  9. Burwell SM. Setting value-based payment goals--HHS efforts to improve U.S. health care. N Engl J Med. 2015;372(10):897–9.

    Article  CAS  PubMed  Google Scholar 

  10. Porter M, Teisberg E. Redefining competition in Health care. Harv Bus Rev. 2004;82:64–76 136.

    PubMed  Google Scholar 

  11. Langel SJ. Solving the value equation in health care. Health Aff (Millwood). 2013;32(3):526.

    Article  PubMed  Google Scholar 

  12. Porter ME. What is value in health care? N Engl J Med. 2010;363(26):2477–81.

    Article  CAS  PubMed  Google Scholar 

  13. Seligman WH, Salt M, la Torre RA, Das-Gupta Z. Unlocking the potential of value-based health care by defining global standard sets of outcome measures that matter to patients with cardiovascular diseases. Eur Heart J Qual Care Clin Outcomes. 2019;5(2):92–5.

    Article  PubMed  Google Scholar 

  14. Blom MC, Khalid M, Van-Lettow B, et al. Harmonization of the ICHOM quality measures to enable Health outcomes measurement in multimorbid patients. Front Digit Health. 2020;2:606246.

    Article  PubMed Central  PubMed  Google Scholar 

  15. Morgans AK, van Bommel AC, Stowell C, et al. Development of a standardized set of patient-centered outcomes for advanced prostate Cancer: an international effort for a unified approach. Eur Urol. 2015;68(5):891–8.

  16. Martin NE, Massey L, Stowell C, et al. Defining a standard set of patient-centered outcomes for men with localized prostate cancer. Eur Urol. 2015;67(3):460–7.

    Article  PubMed  Google Scholar 

  17. Mahmud I, Kelley T, Stowell C, et al. A proposed minimum standard set of outcome measures for cataract surgery. JAMA Ophthalmol. 2015;133(11):1247–52.

    Article  PubMed  Google Scholar 

  18. McNamara RL, Spatz ES, Kelley TA, et al. Standardized outcome measurement for patients with coronary artery disease: consensus from the international consortium for Health outcomes measurement (ICHOM). J Am Heart Assoc. 2015;4(5).

  19. Clement RC, Welander A, Stowell C, et al. A proposed set of metrics for standardized outcome reporting in the management of low back pain. Acta Orthop. 2015;86(5):523–33.

    Article  PubMed Central  PubMed  Google Scholar 

  20. Ong WL, Schouwenburg MG, van Bommel ACM, et al. A standard set of value-based patient-centered outcomes for breast Cancer: the international consortium for Health outcomes measurement (ICHOM) initiative. JAMA Oncol. 2017;3(5):677–85.

    Article  PubMed  Google Scholar 

  21. Allori AC, Kelley T, Meara JG, et al. A standard set of outcome measures for the comprehensive appraisal of cleft care. Cleft Palate Craniofac J. 2017;54(5):540–54.

    Article  PubMed  Google Scholar 

  22. Rolfson O, Wissig S, van Maasakkers L, et al. Defining an international standard set of outcome measures for patients with hip or knee osteoarthritis: consensus of the international consortium for Health outcomes measurement hip and knee osteoarthritis working group. Arthritis Care Res (Hoboken). 2016;68(11):1631–9.

    Article  PubMed  Google Scholar 

  23. Mak KS, van Bommel AC, Stowell C, et al. Defining a standard set of patient-centred outcomes for lung cancer. Eur Respir J. 2016;48(3):852–60.

    Article  PubMed Central  PubMed  Google Scholar 

  24. Rodrigues IA, Sprinkhuizen SM, Barthelmes D, et al. Defining a minimum set of standardized patient-centered outcome measures for macular degeneration. Am J Ophthalmol. 2016;168:1–12.

    Article  PubMed  Google Scholar 

  25. Salinas J, Sprinkhuizen SM, Ackerson T, et al. An international standard set of patient-centered outcome measures after stroke. Stroke. 2016;47(1):180–6.

    Article  PubMed  Google Scholar 

  26. Zerillo JA, Schouwenburg MG, van Bommel ACM, et al. An international collaborative standardizing a comprehensive patient-centered outcomes measurement set for colorectal Cancer. JAMA Oncol. 2017;3(5):686–94.

    Article  PubMed  Google Scholar 

  27. Obbarius A, van Maasakkers L, Baer L, et al. Standardization of health outcomes assessment for depression and anxiety: recommendations from the ICHOM depression and anxiety working group. Qual Life Res. 2017;26(12):3211–25.

    Article  PubMed Central  PubMed  Google Scholar 

  28. Kim AH, Roberts C, Feagan BG, et al. Developing a standard set of patient-Centred outcomes for inflammatory bowel disease-an international, cross-disciplinary consensus. J Crohns Colitis. 2018;12(4):408–18.

    Article  PubMed  Google Scholar 

  29. Foust-Wright C, Wissig S, Stowell C, et al. Development of a core set of outcome measures for OAB treatment. Int Urogynecol J. 2017;28(12):1785–93.

    Article  PubMed Central  PubMed  Google Scholar 

  30. de Roos P, Bloem BR, Kelley TA, et al. A consensus set of outcomes for Parkinson's disease from the international consortium for Health outcomes measurement. J Parkinsons Dis. 2017;7(3):533–43.

    Article  PubMed Central  PubMed  Google Scholar 

  31. Verberne WR, Das-Gupta Z, Allegretti AS, et al. Development of an international standard set of value-based outcome measures for patients with chronic kidney disease: a report of the international consortium for Health outcomes measurement (ICHOM) CKD working group. Am J Kidney Dis. 2019;73(3):372–84.

    Article  PubMed  Google Scholar 

  32. Akpan A, Roberts C, Bandeen-Roche K, et al. Standard set of health outcome measures for older persons. BMC Geriatr. 2018;18(1):36.

    Article  PubMed Central  PubMed  Google Scholar 

  33. Nijagal MA, Wissig S, Stowell C, et al. Standardized outcome measures for pregnancy and childbirth, an ICHOM proposal. BMC Health Serv Res. 2018;18(1):953.

    Article  PubMed Central  PubMed  Google Scholar 

  34. Burns DJP, Arora J, Okunade O, et al. International consortium for Health outcomes measurement (ICHOM): standardized patient-centered outcomes measurement set for heart failure patients. JACC Heart Fail. 2020;8(3):212–22.

    Article  PubMed Central  PubMed  Google Scholar 

  35. Zack R, Okunade O, Olson E, et al. Improving hypertension outcome measurement in low- and middle-income countries. Hypertension. 2019;73(5):990–7.

    Article  CAS  PubMed  Google Scholar 

  36. Oude Voshaar MAH, Das Gupta Z, Bijlsma JWJ, et al. International consortium for Health outcome measurement set of outcomes that matter to people living with inflammatory arthritis: consensus from an international working group. Arthritis Care Res (Hoboken). 2019;71(12):1556–65.

    Article  PubMed  Google Scholar 

  37. Seligman WH, Das-Gupta Z, Jobi-Odeneye AO, et al. Development of an international standard set of outcome measures for patients with atrial fibrillation: a report of the international consortium for Health outcomes measurement (ICHOM) atrial fibrillation working group. Eur Heart J. 2020;41(10):1132–40.

    Article  PubMed Central  PubMed  Google Scholar 

  38. Nano J, Carinci F, Okunade O, et al. A standard set of person-centred outcomes for diabetes mellitus: results of an international and unified approach. Diabet Med. 2020;37(12):2009–18.

    Article  CAS  PubMed  Google Scholar 

  39. Alguren B, Ramirez JP, Salt M, et al. Development of an international standard set of patient-centred outcome measures for overall paediatric health: a consensus process. Arch Dis Child. 2020;106(9):868–76.

  40. Ni Riordain R, Glick M, Al Mashhadani SSA, et al. Developing a standard set of patient-centred outcomes for adult oral health - an international, cross-disciplinary consensus. Int Dent J. 2020;71(1):40–52.

  41. Krause KR, Chung S, Adewuya AO, et al. International consensus on a standard set of outcome measures for child and youth anxiety, depression, obsessive-compulsive disorder, and post-traumatic stress disorder. Lancet Psychiatry. 2021;8(1):76–86.

    Article  PubMed  Google Scholar 

  42. Vos T, Lim SS, Abbafati C, et al. Global burden of 369 diseases and injuries in 204 countries and territories, 1990–2019: a systematic analysis for the global burden of disease study 2019. Lancet. 2020;396(10258):1204–22.

    Article  Google Scholar 

  43. Tong A, Craig JC, Nagler EV, et al. Composing a new song for trials: the standardized outcomes in nephrology (SONG) initiative. Nephrol Dial Transplant. 2017;32(12):1963–6.

    Article  PubMed  Google Scholar 

  44. Tong A, Manns B, Wang AYM, et al. Implementing core outcomes in kidney disease: report of the standardized outcomes in nephrology (SONG) implementation workshop. Kidney Int. 2018;94(6):1053–68.

    Article  PubMed Central  PubMed  Google Scholar 

  45. Boers M, Kirwan JR, Wells G, et al. Developing core outcome measurement sets for clinical trials: OMERACT filter 2.0. J Clin Epidemiol. 2014;67(7):745–53.

    Article  PubMed  Google Scholar 

  46. Porter ME, Larsson S, Lee TH. Standardizing patient outcomes measurement. N Engl J Med. 2016;374(6):504–6.

    Article  CAS  PubMed  Google Scholar 

  47. Fisher ES, Wennberg DE, Stukel TA, Gottlieb DJ, Lucas FL, Pinder EL. The implications of regional variations in Medicare spending. Part 1: the content, quality, and accessibility of care. Ann Intern Med. 2003;138(4):273–87.

    Article  PubMed  Google Scholar 

  48. Polit-O’Hara D, Hungler BP. Essentials of nursing research: methods, appraisal, and utilization Denise F Polit essentials of nursing research: methods, appraisal, and utilization. Nurse Res. 2006;13(4):91–2.

    Article  Google Scholar 

  49. Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Ann Intern Med. 2009;151(4):W65–94.

    Article  PubMed  Google Scholar 

  50. Pill J. The Delphi method: substance, context, a critique and an annotated bibliography. Socio Econ Plan Sci. 1971;5(1):57–71.

    Article  Google Scholar 

  51. Fitch K, Bernstein SJ, Aguilar MD, et al. The RAND/UCLA appropriateness method User's manual. Santa Monica: RAND Corporation; 2001.

  52. Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. Plos One. 2011;6(6):e20476.

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  53. Keeney S, Hasson F, McKenna H. The Delphi technique. In: The Delphi Technique in Nursing and Health Research; 2011. p. 1–17.

    Chapter  Google Scholar 

  54. Cella D, Yount S, Rothrock N, et al. The patient-reported outcomes measurement information system (PROMIS): progress of an NIH roadmap cooperative group during its first two years. Med Care. 2007;45(5 Suppl 1):S3–S11.

    Article  PubMed Central  PubMed  Google Scholar 

  55. Lee WJ, Peng LN, Lin CH, et al. First insights on value-based healthcare of elders using ICHOM older person standard set reporting. BMC Geriatr. 2020;20(1):335.

    Article  PubMed Central  PubMed  Google Scholar 

  56. Lagendijk M, van Egdom LSE, Richel C, et al. Patient reported outcome measures in breast cancer patients. Eur J Surg Oncol. 2018;44(7):963–8.

    Article  CAS  PubMed  Google Scholar 

  57. Ackerman IN, Cavka B, Lippa J, Bucknill A. The feasibility of implementing the ICHOM standard set for hip and knee osteoarthritis: a mixed-methods evaluation in public and private hospital settings. J Patient Rep Outcomes. 2017;2:32.

    Article  PubMed  Google Scholar 

  58. Depla AL, Crombag NM, Franx A, Bekker MN. Implementation of a standard outcome set in perinatal care: a qualitative analysis of barriers and facilitators from all stakeholder perspectives. BMC Health Serv Res. 2021;21(1):113.

    Article  PubMed Central  PubMed  Google Scholar 

  59. Bittar PG, Carlson AR, Mabie-DeRuyter A, Marcus JR, Allori AC. Implementation of a standardized data-collection system for comprehensive appraisal of cleft care. Cleft Palate Craniofac J. 2018;55(10):1382–90.

    Article  PubMed  Google Scholar 

  60. Cella D, Gershon R, Lai JS, Choi S. The future of outcomes measurement: item banking, tailored short-forms, and computerized adaptive assessment. Qual Life Res. 2007;16(Suppl 1):133–41.

    Article  PubMed  Google Scholar 

  61. Terwee CB, Zuidgeest M, Vonkeman HE, Cella D, Haverman L, Roorda LD. Common patient-reported outcomes across ICHOM standard sets: the potential contribution of PROMIS(R). BMC Med Inform Decis Mak. 2021;21(1):259.

    Article  PubMed Central  PubMed  Google Scholar 

  62. Tugwell P, Boers M, Brooks P, Simon L, Strand V, Idzerda L. OMERACT: an international initiative to improve outcome measurement in rheumatology. Trials. 2007;8:38.

    Article  PubMed Central  PubMed  Google Scholar 

  63. SONG. The SONG handbook, vol. 1. Sydney: Standardized Outcomes in Nephrology (SONG) Initiative; 2017.

  64. Prinsen CA, Vohra S, Rose MR, et al. Core outcome measures in effectiveness trials (COMET) initiative: protocol for an international Delphi study to achieve consensus on how to select outcome measurement instruments for outcomes included in a ‘core outcome set’. Trials. 2014;15:247.

    Article  PubMed Central  PubMed  Google Scholar 

  65. Netherlands Got. Outcome-based Healthcare 2018–2022. https://www.government.nl/topics/quality-of-healthcare/information-on-the-quality-of-care. Accessed 21 Sept /2022, 2022.

    Google Scholar 

  66. Kalf R, Vreman RA, Delnoij D, Bouvy ML, Goettsch WG. Bridging the gap: Can International Consortium of Health Outcomes Measurement standard sets align outcomes accepted for regulatory and health technology assessment decision-making of oncology medicines. Pharmacol Res Perspect. 2021;9(2):e00742.

Download references

Acknowledgements

Not applicable.

Funding

Open Access funding enabled and organized by Projekt DEAL. The work presented did not require external funding. Yet, L.B. received funding through the Berta-Ottenstein-Programme for Clinician Scientists from the Faculty of Medicine, University of Freiburg.

Author information

Authors and Affiliations

Authors

Contributions

L.B. and Z.D.G. developed the concept of the analysis presented. L.B. reviewed all peer-reviewed articles that contained proprietary ICHOM Standard Sets. Findings were discussed with all authors listed. Z.D.G., L.F. and S.W. provided and commented on process documents. N.T. and S.G. contributed as key informant interviewees. L.B. prepared the first draft of the manuscript, all authors revised respective draft and contributed to the composition of the final version of the manuscript. The author(s) read and approved the final manuscript.

Corresponding author

Correspondence to Leo Benning.

Ethics declarations

Ethics approval and consent to participate

Not applicable. As this work is exclusively based on literature review of the ICHOM literature repository, a process document review and key informant interviews with the authors listed, no ethics approval was required. The key informant interviewees are among the authors of this work and agree to the publication of the insights they provided.

Consent for publication

Not applicable.

Competing interests

Z.D.G., L.F., S.W., N.T. and S.G. are affiliated with the International Consortium for Health Outcomes Measurement (ICHOM). Beyond their affiliation, they did not declare any competing interests. L.B. declared no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Benning, L., Das-Gupta, Z., Sousa Fialho, L. et al. Balancing adaptability and standardisation: insights from 27 routinely implemented ICHOM standard sets. BMC Health Serv Res 22, 1424 (2022). https://doi.org/10.1186/s12913-022-08694-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12913-022-08694-9

Keywords