- Open Access
Theoretical and methodological considerations in evaluating large-scale health information technology change programmes
BMC Health Services Research volume 20, Article number: 477 (2020)
Attempts to achieve digital transformation across the health service have stimulated increasingly large-scale and more complex change programmes. These encompass a growing range of functions in multiple locations across the system and may take place over extended timeframes. This calls for new approaches to evaluate these programmes.
Drawing on over a decade of conducting formative and summative evaluations of health information technologies, we here build on previous work detailing evaluation challenges and ways to tackle these. Important considerations include changing organisational, economic, political, vendor and markets necessitating tracing of evolving networks, relationships, and processes; exploring mechanisms of spread; and studying selected settings in depth to understand local tensions and priorities.
Decision-makers need to recognise that formative evaluations, if built on solid theoretical and methodological foundations, can help to mitigate risks and help to ensure that programmes have maximum chances of success.
Many countries worldwide see large-scale system-wide health information technology (HIT) programmes as a means to tackle existing health and care challenges [1,2,3]. For example, the United States (US) federal government’s estimated $30 billion national stimulus package promotes the adoption of electronic health records (EHRs) through the Health Information Technology for Economic and Clinical Health (HITECH) Act . Similarly, the English National Health Service (NHS) has invested £4 billion in a national digitisation fund . Digitisation strategies and funding schemes reflect national circumstances, but such programmes face common challenges. These include for example tensions in reconciling national and local requirements. While some standardisation of data transactions and formats is essential to ensure interoperability and information exchange, there is also a need to cater for local exigencies, practices and priorities .
Summative evaluations that seek to capture the eventual outcomes of large national programmes appear to answer questions about the effectiveness of public investments. However, funders and administrators are under pressure to demonstrate outcomes quickly - often within the lifetime of programmes, whilst the full benefits of major change programmes can take a long time to materialise. Premature summative evaluation can generate unwarranted narratives of “failure” with damaging political consequences .
The success or failure of HIT projects involves many different dimensions and at times incommensurable factors [8, 9]. The political context may change within the medium- to long-term timeframes of a major change programme, [6, 8] as seen with some aspects of the English National Programme for Information Technology (NPfIT) [10, 11]. A formative evaluation approach cannot avoid these issues, but can help to better navigate the associated complexities. It can identify apparently productive processes, emerging unintended consequences, and inform the programme’s delivery strategy in real time [12, 13]. It seeks to capture perceptions of actors involved about what is, and is not, working well and feed back findings into programme management. Such evaluations often involve gathering qualitative and quantitative data from various stakeholders and then feeding back emerging issues to implementers and decision-makers so that strategies can be put in place to mitigate risks and maximise benefits.
Our team has conducted several formative evaluations of large HIT programmes and developed significant expertise over the years [14,15,16]. In doing so, we have encountered numerous theoretical and methodological challenges. We here build on a previous paper discussing the use of formative approaches for the evaluation of specific technology implementations in the context of shifting political and economic landscapes [10, 14]. In this previous work (Table 1), we described the complex processes of major HIT implementation and configuration. We argued that evaluation requires a sociotechnical approach and advocated multi-site studies exploring processes over extended timeframes, as such processes are not amenable to conventional positivist evaluation methodologies.
We here offer an extension of this work to explore not only implementations of specific functionality (such as electronic health records (EHRs)), but their programmatic integration with ancillary systems (e.g. electronic prescribing and medicines administration, radiology). This can help to gain insights into the emergence and evolution of information infrastructures (systems of systems) that are increasingly salient as we see functional integration within hospitals and across care settings. We also consider mechanisms of spread, evolving networks/processes, and vendor markets .
The difficulty of attributing outcomes
The first challenge concerns the difficulty of attributing outcomes (i.e. exploring what caused a specific outcome) for major changes in HIT. Although often required to justify investments, the direct effects of complex HIT such as EHRs are difficult to track and measure . This is particularly true for large-scale transformative and systemic upgrades in infrastructures, which are not one-off events, but occur through multiple iterations and interlinkages with existing systems. Such systems tend to have distributed effects with hard-to-establish gradually emerging baselines (when compared to local discrete technologies implemented in specific settings, although the effects of these can also be hard to measure) . Infrastructure renewal is a long term process where current achievements rest on earlier upgrades over long timeframes as systems are incrementally extended and optimised . An example may be the implementation of EHRs and their integration with ancillary systems. Here, decision-makers are championing not just one, but multiple implementations of various transformative systems.
Theoretically informed formative evaluations that draw on science and technology studies and acknowledge the interrelationship between social and technological factors can help to address this issue . A particularly effective methodology is exploring selected settings in depth to understand local complexities, while also monitoring a wider number of settings in less detail to understand general trends. Complex research designs drawing on case study methods and a range of sociotechnical approaches can help to explore how technological and social factors shape each other over time . They can therefore provide an insight into local changes and potential mechanisms leading to outcomes . In our current work on evaluating the Global Digital Exemplar Programme, for example, we are conducting 12 in-depth case studies of purposefully recruited hospitals. In addition, we are collecting more limited longitudinal qualitative data across all 33 hospitals participating in the Programme . This research design offers a balance between depth (achieved through the case studies) and breadth (achieved through testing emerging findings across the broader sample).
Balancing local diversity and autonomy with national aims
Decision-makers cannot simply roll out standard solutions across the health service as sites vary in terms of clinical practices, existing information systems and data structures, size and organisational structures, contexts and local demographics. A key challenge for evaluation of large programmes is reconciling tensions between bringing specific sites up to international best practice, and levelling up the local ecosystem . Organisational settings differ in their local contexts, structures and (emerging) service configurations. They are often separate autonomous entities that may be in competition . Various groups of clinical staff and decision-makers may have different priorities (e.g. between decision-makers and various groups of clinical staff). Programme visions may be differently interpreted by local stakeholders, which can lead to unanticipated outcomes and deviation from central aims. In the US, the Meaningful Use criteria have for instance resulted in increasing implementation of EHRs, but the impact on quality and safety is still unknown and concern has been expressed that they may have stifled local innovation .
There is a tension between local and national priorities – and there is no stable way to reconcile these. Instead, strategies constantly shift between these poles, never standing still, pulled by a network of stakeholder groups with conflicting interests in a process that has been conceptualised as a swinging pendulum (Fig. 1) . For example the UK NPfIT exemplified a strong pull towards national priorities, with a strategy that focused on concerted procurement and interoperability. In the period that followed, organisations were responsible for the procurement of locally selected systems. The pendulum swung the other way.
To explore this process and associated tensions, evaluators need to study evolving networks, relationships, and processes to understand how various stakeholders are mobilised nationally and locally as part of the change programme and what the perceived effects of these mobilisations are. This may involve working closely with national Programme Leads to identify current policy directions and intended national strategy, whilst also exploring local experiences of this strategy. From our experience, it can be helpful to move from arms-length critical analyst to constructive engagement with different stakeholders groups. Establishing long-term personal relationships with senior decision-makers whilst retaining independence is important is this respect. These need to be characterised by mutual trust and frank discussion where evaluators play the role of a ‘critical friend’ at times delivering painful truths.
A recurring example of tensions in our work relates to progress measures. National measures of progress designed to provide justification for programme resources are liable to clash with local priorities and circumstances. Participating local organisations may negatively perceive achieving these as requiring large amounts of resources in comparison to limited local benefits and driven by the need to satisfy reporting demands. Some agreement over a limited core set of measures to satisfy both local and national demands may be helpful.
The evolving nature of HIT programmes over time
Takian and colleagues noted how the policy context changed in the course of a single long-term change programme . These factors may result in various stakeholders chasing moving targets and scope creep. For example, the economic recession of 2008–13 heavily influenced the English NPfIT, which led to a lack of sustained funding .
Although important, shifting socio-political environments only constitute part of the picture. A long-term view of nurturing evolving infrastructures highlights that visions of best practices will inevitably change over time . They also often have no definite end point and there is at times no consensus about strategic direction. We have previously discussed this in the context of digital maturity, which is a somewhat contested concept . Different kinds of programme management and evaluation tools may be needed that give cognisance to this kind of evolution. These may include an emphasis on flexibility and reflexivity, where decision-makers can adjust strategies and roadmaps in line with emerging needs and changing environments. This approach will also require learning historical lessons and drawing on the wealth of experience of those who have experienced similar initiatives first hand.
Changes in medical techniques and diagnosis, models for care delivery, and vendor offerings affect available technologies (and vice versa). The market may not immediately be able to respond to new policy-driven models, and therefore evaluations and policies need to consider these dimensions . This may involve exploring evolving vendor-user relationships, the emergence and mobilisation of user groups, procurement frameworks, and market diversity . Our work, for instance, shows that, reinforced by the English NPfIT, multi-national mega-suite solutions revolving around core EHR systems increasingly dominate the UK market. These offer a relatively well-established and reliable pathway to achieving digital maturity and interoperability. The alternative pathway involves knitting together EHRs with a range of other functionality provided by diverse vendors. This may offer advantages in allowing an adopter to achieve a Best-of-Breed (BoB) solution unique to each local setting, and potentially better suited to local organisations . However, there are difficulties for vendors of modular solutions designed for BoB to enter the market and develop interfaces. Existing EHR vendors are struggling to upgrade their systems to become mega-packages. Implementers must carefully consider interoperability challenges and innovation opportunities afforded by various systems. Programmes must ensure procurement approaches stimulate (or at least to not inhibit) a vibrant marketplace.
Scaling of change through developing a self-sustaining learning ecosystem
Large HIT change programmes are often concerned with not only stimulating local changes but also with promoting ongoing change ensuring that efforts are sustained and scaled beyond the life of the programme . But this is not straightforward, partly due to lack of agreement over suitable metrics of success and partly due to limited understanding of the innovation process .
Studies of the emergence and evolution of information infrastructures have in turn helped articulate new strategies for promoting/sustaining such change [35,36,37]. However, the notion of scaling-up tacitly implies that innovation stops when diffusion starts. A more nuanced perspective flags that innovations evolve as they scale (‘innofusion’), requiring strong learning channels between adopter communities and vendors .
Evaluators can explore success factors and barriers to scaling qualitatively and formatively feed these back to decision-makers who can then adjust their strategies accordingly. Evaluation needs to address local change in tandem with evolving networks at ecosystem level. By studying a range of adopter sites and their relationships with each other, as well as other stakeholders that are part of the developing ecosystem, evaluators can identify mechanisms that promote digital transformation and spread. Understanding these dynamics can also help decision makers focus strategy on achieving programme objectives. By addressing networks and relationships, evaluators can, for example, explore how knowledge spreads throughout the wider health and care ecosystem in which the change programme is embedded, and how stakeholders were motivated to exchange and trade knowledge .
We are now entering an era that emphasises patient-centred care and data integration across primary, secondary and social care. This is linked to a shift from discrete technological changes to systemic long-term infrastructural change associated with large national/regional HIT change programmes. There are some attempts to characterise and study these changes including our own . However, these provide only a partial picture, which we have built on here based on our ongoing experiences reflecting our current thinking (see Table 2).
We now need new methods of programme management geared towards developing learning in ecosystems of adopters and vendors. These evolutionary perspectives also call for broader approaches to complex formative evaluations that can support the success of programmes and help to mitigate potential risks.
Although there is no prescriptive way to conduct such work, we hope that this paper helps decision-makers to commission work that is well suited to the subject of study, and implementers embarking on the evaluative journey to navigate this complex landscape.
Availability of data and materials
Electronic Health Record
Health Information Technology
Health Information Technology for Economic and Clinical Health
National Health Service
National Programme for Information Technology
O’Malley AS. Tapping the unmet potential of health information technology. N Engl J Med. 2011;364(12):1090–1.
Greenhalgh T, Stones R. Theorising big IT programmes in healthcare: strong structuration theory meets actor-network theory. Soc Sci Med. 2010;70(9):1285–94.
Coiera E. Building a national health IT system from the middle out. J Am Med Inform Assoc. 2009;16(3):271–3.
Thornton M. The “meaningful use” regulation for electronic health records. N Engl J Med. 2010;2010(363):501–4.
NHS £4 billion digitisation fund: Industry reaction. Available from: https://www.itproportal.com/2016/02/08/nhs-4-billion-digitisation-fund-industry-reaction/. Accessed 13 Dec 2019.
Greenhalgh T, Potts HW, Wong G, Bark P, Swinglehurst D. Tensions and paradoxes in electronic patient record research: a systematic literature review using the meta-narrative method. Milbank Q. 2009;87(4):729–88.
Hendy J, Reeves BC, Fulop N, Hutchings A, Masseria C. Challenges to implementing the national programme for information technology (NPfIT): a qualitative study. BMJ. 2005;331(7512):331–6.
Berg M. Implementing information systems in health care organizations: myths and challenges. Int J Med Inform. 2001;64(2–3):143–56.
Wilson M, Howcroft D. Power, politics and persuasion: a social shaping perspective on IS evaluation. In Proceedings of the 23rd IRIS Conference; 2000. p. 725–39.
Greenhalgh T, Russell J, Ashcroft RE, Parsons W. Why national eHealth programs need dead philosophers: Wittgensteinian reflections on policymakers’ reluctance to learn from history. Milbank Q. 2011;89(4):533–63.
Greenhalgh T, Stramer K, Bratan T, Byrne E, Russell J, Potts HW. Adoption and non-adoption of a shared electronic summary record in England: a mixed-method case study. BMJ. 2010;340:c3111.
McGowan JJ, Cusack CM, Poon EG. Formative evaluation: a critical component in EHR implementation. J Am Med Inform Assoc. 2008;15(3):297–301.
Catwell L, Sheikh A. Evaluating eHealth interventions: the need for continuous systemic evaluation. PLoS Med. 2009;6(8):e1000126.
Sheikh A, Cornford T, Barber N, Avery A, Takian A, Lichtner V, Petrakaki D, Crowe S, Marsden K, Robertson A, Morrison Z. Implementation and adoption of nationwide electronic health records in secondary care in England: final qualitative results from prospective national evaluation in “early adopter” hospitals. BMJ. 2011;343:d6054.
Cresswell KM, Bates DW, Williams R, Morrison Z, Slee A, Coleman J, Robertson A, Sheikh A. Evaluation of medium-term consequences of implementing commercial computerized physician order entry and clinical decision support prescribing systems in two ‘early adopter’ hospitals. J Am Med Inform Assoc. 2014;21(e2):e194–202.
Cresswell K, Callaghan M, Mozaffar H, Sheikh A. NHS Scotland’s decision support platform: a formative qualitative evaluation. BMJ Health Care Informatics. 2019;26(1):e100022.
Takian A, Petrakaki D, Cornford T, Sheikh A, Barber N. Building a house on shifting sand: methodological considerations when evaluating the implementation and adoption of national electronic health record systems. BMC Health Serv Res. 2012;12(1):105.
Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E, Morton SC, Shekelle PG. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med. 2006;144(10):742–52.
Clamp S, Keen J. Electronic health records: is the evidence base any use? Med Inform Internet Med. 2007;32(1):5–10.
Wiegel V, King A, Mozaffar H, Cresswell K, Williams R, Sheik A. A systematic analysis of the optimization of computerized physician order entry and clinical decision support systems: a qualitative study in English hospitals. Health Informatics J. 2019 Sep;30:1460458219868650.
Sismondo S. An introduction to science and technology studies. Chichester: Wiley-Blackwell; 2010.
Cresswell KM, Sheikh A. Undertaking sociotechnical evaluations of health information technologies. J Innov Health Informatics. 2014 Mar 18;21(2):78–83.
Global Digital Exemplar (GDE) Programme Evaluation. Available from: https://www.ed.ac.uk/usher/research/projects/global-digital-exemplar-evaluation. Accessed 13 Dec 2019.
Gunter TD, Terry NP. The emergence of national electronic health record architectures in the United States and Australia: models, costs, and questions. J Med Internet Res. 2005;7(1):e3.
Cresswell K, Sheikh A. Organizational issues in the implementation and adoption of health information technology innovations: an interpretative review. Int J Med Inform. 2013;82(5):e73–86.
Slight SP, Berner ES, Galanter W, Huff S, Lambert BL, Lannon C, Lehmann CU, McCourt BJ, McNamara M, Menachemi N, Payne TH. Meaningful use of electronic health records: experiences from the field and future opportunities. JMIR Med Inform. 2015;3(3):e30.
The Devil’s in the Detail. Available from: https://www.birmingham.ac.uk/Documents/college-mds/haps/projects/cfhep/projects/007/CFHEP007SCRIEFINALReport2010.pdf. Accessed 13 Dec 2019.
Aanestad M, Grisot M, Hanseth O, Vassilakopoulou P. Information infrastructures within European health care: working with the installed base. Cham: Springer; 2017.
Cresswell K, Sheikh A, Krasuska M, Heeney C, Franklin BD, Lane W, Mozaffar H, Mason K, Eason S, Hinder S, Potts HW. Reconceptualising the digital maturity of health systems. Lancet Digit Health. 2019;1(5):e200–1.
Mozaffar H, Williams R, Cresswell K, Morrison Z, Bates DW, Sheikh A. The evolution of the market for commercial computerized physician order entry and computerized decision support systems for prescribing. J Am Med Inform Assoc. 2015;23(2):349–55.
Mozaffar H. User communities as multi-functional spaces: innovation, collective voice, demand articulation, peer informing and professional identity (and more). In the new production of users. New York: Routledge; 2016. p. 219–46.
Light B, Holland CP, Wills K. ERP and best of breed: a comparative analysis. Bus Process Manag J. 2001;7(3):216–24.
Mandl KD, Kohane IS. No small change for the health information economy. N Engl J Med. 2009;360(13):1278–81.
Charif AB, Zomahoun HT, LeBlanc A, Langlois L, Wolfenden L, Yoong SL, Williams CM, Lépine R, Légaré F. Effective strategies for scaling up evidence-based practices in primary care: a systematic review. Implement Sci. 2017;12(1):139.
Making IT work: harnessing the power of health information technology to improve care in England. Available from: https://www.gov.uk/government/publications/using-information-technology-to-improve-the-nhs/making-it-work-harnessing-the-power-of-health-information-technology-to-improve-care-in-england. Accessed 8 Jan 2020.
Monteiro E, Hanseth O. Social shaping of information infrastructure: on being specific about the technology. InInformation technology and changes in organizational work. Boston: Springer; 1996. p. 325–43.
Williams R, Stewart J, Slack R. Social learning in technological innovation: experimenting with information and communication technologies. Cheltenham: Edward Elgar Publishing; 2005.
Fleck J. Innofusion: feedback in the innovation process. InSystems science. Boston: Springer; 1993. p. 169–74.
Gorman ME. Trading zones and interactional expertise: Creating new kinds of collaboration. Cambridge: MIT Press; 2010.
We gratefully acknowledge the input of the wider GDE Evaluation team and the Steering Group of this evaluation.
This article has drawn on a programme of independent research funded by NHS England. The views expressed are those of the author(s) and not necessarily those of the NHS, NHS England, or NHS Digital. This work was also supported by the National Institute for Health Research (NIHR) Imperial Patient Safety Translational Research Centre. The views expressed in this publication are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health and Care. The funders had no input in this manuscript.
Ethics approval and consent to participate
Consent for publication
All authors are investigators on the evaluation of the GDE programme (https://www.ed.ac.uk/usher/digital-exemplars). AS was a member of the Working Group that produced Making IT Work, and was an assessor in selecting GDE sites. BDF supervises a PhD student partly funded by Cerner, unrelated to this paper.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Cresswell, K., Sheikh, A., Franklin, B.D. et al. Theoretical and methodological considerations in evaluating large-scale health information technology change programmes. BMC Health Serv Res 20, 477 (2020). https://doi.org/10.1186/s12913-020-05355-7
- Health information technology