Development of modelling method selection tool for health services management: From problem structuring methods to modelling and simulation methods

Background There is an increasing recognition that modelling and simulation can assist in the process of designing health care policies, strategies and operations. However, the current use is limited and answers to questions such as what methods to use and when remain somewhat underdeveloped. Aim The aim of this study is to provide a mechanism for decision makers in health services planning and management to compare a broad range of modelling and simulation methods so that they can better select and use them or better commission relevant modelling and simulation work. Methods This paper proposes a modelling and simulation method comparison and selection tool developed from a comprehensive literature review, the research team's extensive expertise and inputs from potential users. Twenty-eight different methods were identified, characterised by their relevance to different application areas, project life cycle stages, types of output and levels of insight, and four input resources required (time, money, knowledge and data). Results The characterisation is presented in matrix forms to allow quick comparison and selection. This paper also highlights significant knowledge gaps in the existing literature when assessing the applicability of particular approaches to health services management, where modelling and simulation skills are scarce let alone money and time. Conclusions A modelling and simulation method comparison and selection tool is developed to assist with the selection of methods appropriate to supporting specific decision making processes. In particular it addresses the issue of which method is most appropriate to which specific health services management problem, what the user might expect to be obtained from the method, and what is required to use the method. In summary, we believe the tool adds value to the scarce existing literature on methods comparison and selection.


Results:
The characterisation is presented in matrix forms to allow quick comparison and selection. This paper also highlights significant knowledge gaps in the existing literature when assessing the applicability of particular approaches to health services management, where modelling and simulation skills are scarce let alone money and time.
Conclusions: A modelling and simulation method comparison and selection tool is developed to assist with the selection of methods appropriate to supporting specific decision making processes. In particular it addresses the issue of which method is most appropriate to which specific health services management problem, what the user might expect to be obtained from the method, and what is required to use the method. In summary, we believe the tool adds value to the scarce existing literature on methods comparison and selection.

Background
There is an increasing recognition that modelling and simulation can assist in the process of redesigning health services to reconcile expanding demands for health care with cost-containment [1][2][3][4][5][6][7][8][9][10]. Policymakers are also keen to capture the benefits of modelling and simulation to healthcare managers [1,2,[11][12][13][14][15][16]. In the English National Health Service (NHS), for example, Primary Care Trusts (PCTs) which purchase health care on behalf of their citizens face a mandatory requirement to undertake some "predictive modelling" to ensure they factor health need into their decisions [10]. Various tools are being developed to assist healthcare managers to model need [17,18]. One example is the Scenario Generator, designed to help PCT managers assess the needs of their population and plan care accordingly [19]. The tool allows commissioners to make changes to service provision "virtually" and assess the impact on costs, waits, etc.
Whilst it is not possible to fully gauge the use of modelling and simulation in health services management, it is apparent to expert modellers that the use of modelling and simulation is currently not widespread in the UK [20]. The lack of desire or skill of health care managers to use modelling and simulation, particularly mathematical methods has been highlighted as one of the practical challenges [3,7,11,12,21]. In addition, a need for better awareness and use of a broader set of methods has been recognised to deal with complex and very often political healthcare services management [22][23][24][25]. The most appropriate method may be that which fits the problem and the experience of the client and the circumstances in which the modelling and simulation takes place [11][12][13].
There is value therefore in finding a means to assist healthcare managers to know what methods are available, how they are different, what methods to use and when. The aim of this study is to propose a tool for decision makers in health services planning and management to compare a broad range of modelling and simulation methods so that they can better select and use them or better commission relevant modelling and simulation work. This paper reports on the tool development process and results, specific outcomes are published in a form of a workbook [26] as a part of the results of the RIGHT project (Research Into Global Healthcare Tools); a team of researchers of five UK universities investigated the use of modelling and simulation in health care with a grant from the UK Engineering and Physical Sciences Research Council (EPSRC).
The next section presents the methodology followed for developing the tool and its steps. This is followed by detailed description of results and the paper ends with a discussion and conclusions including lessons learned and future research plans. Figure 1 shows three levels of the tool development process and research methods used in each stage. The tool development was primarily based on an extensive review of the literature on the application of modelling and simulation to health care, as well as manufacturing, aerospace and military. The method-related information unobtainable from the literature review was complemented by the research team's extensive expertise in modelling and simulation. Inputs from potential users (health care managers and modelling practitioners) were obtained to capture requirements for the tool, and co-develop and validate it mainly through workshops.

Methods
This section explains in further details the three phases of the research process: a comprehensive literature review; use of expert knowledge; and inputs from potential users.

Literature review
The aim of undertaking the literature review in the RIGHT project was twofold. First, it provided an upto-date review of simulation and modelling application in health care and other industries. Second, it fed information on the methods (what, where and how they are used) into the comparison and selection tool development. This paper is intended to show the literature review from the perspective of the tool development. The literature review methodology is briefly covered in this paper but more detailed methodology and findings can be found in separate papers [20,27,28].
In summary, the following four topics were searched to find relevant papers using refined search criteria for inclusion and exclusion: Topic 1. Simulation and modelling in health care; Topic 2. Simulation and modelling in manufacturing industry; Topic 3. Simulation and modelling in aerospace and military; Topic 4. Management and planning methods in health care.
In order to provide a base for tool development, each paper was summarised using a review template which consists of the three categories and sub-fields: method (name, purposes, strength and limitations); problem (specific issue, type, functional area, layer, scenario and setting); resources (time, data, people, expertise, others).

Experts' knowledge
Despite the comprehensive literature reviews carried out in the study, articles did not always provide all the information required to develop the tool, especially regarding what is required to use the methods in terms of resources such as time, money, data and knowledge. Expertise of the research team was utilised to complement the information elicited from the literature. The experts' knowledge was continuously captured according to their specialties and cross-checked through communication and group discussions throughout the tool development. The research team consisted of nine academics and seven researchers from five UK universities in the field of knowledge management, operational research & management science, information & communication engineering and systems engineering. Methods each of the research team members particularly specialise in include, but are not limited to, soft OR approaches, conceptual systems modelling, mathematical modelling and simulation techniques. They have extensively applied these methods for planning and management of health care and their experience ranges from 5 years to 30 years.

Inputs from potential users
Three user engagement workshops in combination with in-depth interviews were carried out to get inputs from potential users. They were by no means a formal test evaluating the effectiveness of the tool, but were meant to provide valuable insights into users' requirements for the tool.
At the early stage of the project, the potential users' requirements for the tool were captured in a one-day workshop, where a dozen health care managers attended. The research team introduced the research objectives and project plans. Then, the health care managers shared their previous modelling experience and expectation for the project through group discussion.
In the middle of the project, i.e. after developing a prototype tool, the research team organized another one and a half day workshop. Sixteen delegates (nine national or local-level health care managers and seven professional modellers) attended and were asked to first review and then redesign the prototype tool. Various suggestions were made from specific wording changes to new overall comparison and selection mechanism. The prototype tool was, then, revised to reflect some of the suggestions from the workshop and printed in the form of a workbook. The workbook was redistributed by post to the health care managers for another review. Then, two one-hour following-up interviews were carried out with local and national level health care managers to investigate their responses in depth.
At the end of the project (after iterative revision of the tool), the national level health care manager at the NHS Institute of Innovation and Improvement (UK' national institute for supporting the National Health Service transformation by developing and spreading new ways of working) wanted to use the workbook [26] for its workshop, 'Building capability in modelling and simulation for commissioning and strategic planning.' The purpose of the workshop was to increase awareness of various modelling and simulation methods and have discussion on how to build capability in these methods. Approximately sixty health care managers (mostly commissioners) attended a full one-day workshop. The tool in the form of the workbook was distributed to each delegate and used as a reference point for group discussion. One of the project team researchers (G. Jun) helped lead group discussion and captured the comments on the utility and usability of the tool throughout the workshop (mostly group feedback session).
The potential users' inputs at these various occasions have been reflected in the tool presented in this paper. Besides, their inputs for the future development of the tool are summarized in the results section. Twenty-eight methods, ranging from problem structuring methods, conceptual modelling methods, mathematical modelling methods to simulation methods, were identified from a commonly applied method list in the literature and through the iterative discussions between the research team members.  In addition to these commonly applied methods, the research team with expertise in different methods agreed to include additional methods. Qualitative modelling approaches such as various problem structuring methods [29] and conceptual modelling methods [30] were especially expanded based on the expertise of the research team since these types of methods had not been extensively searched in the literature review. It was agreed to identify a broad range of indicative modelling and simulation methods in this project, rather than a full list of comprehensive methods. The method list defined in this project was also agreed to remain open to the possibility of adding or removing at the later stage.

Method Identification
In the end, twenty eight methods were agreed and categorised into four different groups as shown Table 2: five problem structuring methods; eight conceptual modelling methods; seven mathematical modelling methods; eight simulation methods.

Method Characterisation by Application Area and Project Life Cycle Stage
First, the list of the eight application areas, drawn from MeSH Terms (Medical Subject Headings), was used for characterising the twenty eight methods (the first column of Table 3). This list was considered most suitable since it covers a broad range of application area without too much overlapping and presumably uses terminology familiar to health care professionals. The lists from the different review topics were found less suitable owing to the industry-specific nature of them e.g. aerospace, military and manufacturing.
Second, the eight project life cycle stages, which were drawn from Royston [31], were used for characterising the twenty eight methods (the second column of Table 3).
The matches between the twenty eight methods and application area/project life cycle stage were initially made based on the literature and additionally complemented by the experts' knowledge of the research team. Figure 2 shows the matches of which methods are suitable for different combinations of project life cycle stage and application area using 8 × 8 matrix. Each cell in this matrix consists of a smaller matrix (4 × 8) to   Figure 2 shows that twenty methods are potentially suitable: five problem structuring methods (1~5); eight conceptual modelling methods (6~13); six mathematical modelling methods (14, 15, 16, 18 and 20); two simulation methods (23 and 28).

Methods Characterisation by Level of Insight and Type of Output
The twenty eight methods were also characterised in terms of two different output parameters, level of insight    and type of output. The definitions of the five attributes for each output parameter are summarised in Appendix 1. Figure 3 shows the matches of which methods are suitable for different combination of level of insight and type of output using 5 × 5 matrix in the same way with Figure 2. For example, you expect outputs at managerial level of insight (third column: 3. managerial) and want a relatively well-characterised view of the system and how it interacts with the rest of the health care system (third column: 3. system interaction). Then the thick black line box in Figure 3 shows that eight methods are potentially suitable: four problem structuring methods (1, 3, 4 and 5); two conceptual modelling methods (9 and 11); two simulation methods (24 and 28).

Method Characterisation by Four Input Resources
The twenty eight methods were characterised by four different input resource parameters such as time, money, knowledge and data. A five scale index was used to show the ranges of the requirements for each parameter and help users promptly compare them between different methods. Table 4 shows the five scale indices for each parameter, which were determined through internal discussion between the research team and consultation with health care professionals [32]. Our intention was to determine input requirement ranges for conventional application rather than quick-and-dirty application.
Time is the amount of time required with expertise available, whereas money is the amount of money required to purchase hardware, software and expertise.
Knowledge is not knowledge about specific methods, but qualitative knowledge about problems. Data refers to quantitative data required. The definitions of the former two input parameters (time and money) are straightforward as shown in Table 4, but the definitions of the latter two input parameters (knowledge and data) are summarised in detail in Appendix 2. Table 5 shows the ranges of the requirements for each method and identifies constraints on the use of candidate methods. For example, one of the simulation methods, 28. System Dynamics is indexed to require the following ranges of the resources: from as short as days to more than a year; from as small as £10s to £10,000s; from moderate to complete knowledge about problems; from no quantitative data to good statistics. Figure 4 shows a two-stage method comparison and selection mechanism using two matrices (Figures 2 and 3) and    Table 5). The first stage is to filter potential methods using two matrices (Figures 2 and 3) and the second stage is to compare the filtered methods in terms of the four resource requirements (time, money, knowledge and data).

Method Comparison and Selection Mechanism Building
The tool is designed to assist selection and comparison of methods appropriate to supporting particular problem situation. For example, a team of health care professionals face the challenge of identifying potential hazards and adverse occurrences (application area: risk management) when planning a new service development (project life cycle stage: new service development). They want to understand how their new services would interact with the rest of the services (type of output: system   28. System dynamics interaction) at a managerial level (level of insight: managerial). Then, the corresponding common set from two matrices (Figures 2 and 3) shows seven potential methods (1, 3, 4, 5, 9, 11 and 28) as in Figure 4. The team has not much quantitative data, but they think they know the problems relatively well. Taking into account these constraints, influence diagrams or issue maps can be applied with moderate investment of time (hours~months) and money (£tens~hundreds). If data, time and money more available, system dynamics can be applied, which can additionally support trend analysis. This comparison and selection process enables the selection of methods most suited to the needs and constraints of the particular decision process.

Tool Validation
Many of the inputs from the co-development workshop were reflected in the current tool, but in-depth interviews and additional workshop provided insight into what is this tool for and how to improve it. What echoed around between the potential users was that the tool was very informative rather than prescriptive. The tool that put a broad range of modelling and simulation methods together with a consistent structure was considered very instrumental in increasing awareness of various methods and their differences. However, many considered that the tool needs to be further improved to convince them to use for the selection of methods. The comments on the further tool development are summarised into the following three main aspects.
First, it was observed that the tool, although originally intended to help identify appropriate methods to problem solving, needs to aid more in problem structuring. A more systematic and phased approach was suggested to understand/formulate problems first before deciding whether or which modelling and simulation may apply. To do that, more exhaustive questions on the symptoms of the problems or the use of the problem structuring methods were proposed to be applied as a part of the tool.
Second, additional information on each method was suggested crucial to the intended use of the tool. In the    (Table 5)  workbook presentation of the tool [26], each method was additionally described briefly in a page per method regarding its typical functions, purposes and example application areas. However, more specific case examples describing how and where each method was applied were considered essential to convince users to push forward with selected methods. In addition, information proving the reliability/authenticity of selected methods such as reference to literature or experts was considered important. Information on practical supports for method application was also suggested needed such as modelling tools and modelling expertise in commercial and academic communities. Third, there were divergent opinions about the scope of methods. One group of the participants suggested that the current aim for a generic tool across health care may be overambitious. They suggested that the tool be more specific to certain problems and target users, e.g. commissioning, waiting-time target. On the other hand, the other group suggested including general change management methods as well so that they can have a better understanding how the modelling and simulation methods fit with their existing management tools.

Contributions of the research
We are still far from a definitive tool, if such a tool is indeed possible, but believe that the tool makes a contribution in two major ways. The first is to fill a research practice gap in evidence-based health care management [22] by providing a practical support for the method comparison and selection. Not only can the tool help health care professionals commission more appropriate modelling work, but may also assist health care modelling consultants and researchers to expand their modelling repertoire in order to meet the diverse needs of their health care clients. Research shows that modelling practitioners and researchers tend to select their approaches based on previous experience and competences, despite awareness of other methods [33][34][35]. They also tend to select from within either a 'hard (quantitative)' or 'soft (qualitative)' paradigm, when they use a number of approaches [36]. Whether the tool helps health care managers find better methods to aid decision-making remains to be seen, but the early feedback from the potential users has been positive and gave valuable insight into the further development.
The second contribution is that the development of the tool has also highlighted significant gaps in knowledge which could be usefully filled. Building the tool from the extant literature proved challenging because studies were often vague when reporting their modelling and simulation process, meaning that information on input resource requirements, such as time, money and data, was often missing. These are important gaps in knowledge when assessing the applicability of particular approaches to health care, where modelling and simulation skills are scarce let alone money and time.

Extensions to the research
In spite of many debates and discussions throughout the tool development process, we identified the following two research areas to be further addressed.
First, during the development of the tool it became clear that more efforts need to be made in defining the "problem space." Figure 2 shows that the application area does not effectively differentiate the methods, whereas the project life cycle stage better differentiates the methods. It means that the application area used in the tool clearly do not capture the complexity and variety faced by health care managers from the perspective of modelling and simulation application, and nor is there an existing literature which provides adequate insight. Identifying effective parameters and developing meaningful categories requires further characterisation and differentiation by different aspects of problem types. Taxonomy of more specific problem types was suggested to be further developed from the current application area categories and added to the future tool so that candidate methods can be mapped to more specific problem situations. It can also help specify the input resources required and the outputs expected for each method in specific problem situation.
Second, whether and how a comparison and selection tool of this type will actually encourage them increase the range of methods they apply to a problem is an empirical question which we hope to further investigate in time. In the same context, the interface design of this tool, i.e. how different interfaces interact with users during the method comparison and selection, remains an important question that need to be further addressed. In general, health care managers we engaged showed more interest in informative interactions rather than definitive. For example, the two matrices could be informative since they allow users to effectively explore not only methods filtered from their problem and output definitions, but also alternative methods around their definitions. The provision of such overall information was considered to help users redefine their problem and output and learn about capabilities of different methods. Further research in understanding usability and utility of different information visualisation/presentation is needed for the tool to be genuinely accepted by health care users.

Conclusions
The modelling and simulation method comparison and selection tool is developed to assist health care professionals to commission more appropriate modelling work and health care modelling practitioners or researchers to broaden their selection of methods appropriate to supporting specific decision making processes. In particular it addresses the issue of which method is most appropriate to which specific health care management problem, what the user might expect to be obtained from the method, and what is required to use the method. In summary, we believe the tool adds value to the scarce existing literature on methods comparison and selection. However, we also recognise the limitations of the tool, many of which are reflected in the feedback from the potential users: a more structured problem formulation/structuring stage; and more detailed case examples. Further research is proposed to help address these issues by evaluating and refining the tool closely with healthcare professionals.

Appendices
Appendix 1 Definitions of the five attributes of two output parameters • Level of insight: what level of insight do you require from the modelling? 1) Policy: decisions made at national or regional level, e.g. design of public health initiatives with long-term impact such as cancer screening programmes 2) Strategy: major decisions with medium-term impact, e.g. permanently closing a hospital ward, buying an MRI scanner or opening a Walk-in Centre 3) Managerial: e.g. determining nursing staff levels across different specialties in a hospital 4) Operational: e.g. deciding how many fracture clinics to run per week, or how many ICU beds to staff 5) Detailed: e.g. nurse rostering or operating theatre list scheduling