The ecometric properties of a measurement instrument for prospective risk analysis in hospital departments
© van Schoten et al.; licensee BioMed Central Ltd. 2014
Received: 24 June 2013
Accepted: 7 February 2014
Published: 3 March 2014
Safety management systems have been set up in healthcare institutions to reduce the number of adverse events. Safety management systems use a combination of activities, such as identifying and assessing safety risks in the organizational processes through retrospective and prospective risk assessments. A complementary method to already existing prospective risk analysis methods is Tripod, which measures latent risk factors in organizations through staff questionnaires. The purpose of this study is to investigate whether Tripod can be used as a method for prospective risk analysis in hospitals and whether it can assess differences in risk factors between hospital departments.
Tripod measures risk factors in five organizational domains: (1) Procedures, (2) Training, (3) Communication, (4) Incompatible Goals and (5) Organization. Each domain is covered by 15 items in the questionnaire. A total of thirteen departments from two hospitals participated in this study. All healthcare staff working in the participating departments were approached. The multilevel method ecometrics was used to evaluate the validity and reliability of Tripod. Ecometrics was needed to ensure that the differences between departments were attributable to differences in risk at the departmental level and not to differences between individual perceptions of the healthcare staff.
A total of 626 healthcare staff completed the questionnaire, resulting in a response rate of 61.7%. Reliability coefficients were calculated for the individual level and department level. At the individual level, reliability coefficients ranged from 0.78 to 0.87, at the departmental level they ranged from 0.55 to 0.73. Intraclass correlations at the departmental level ranged from 3.7% to 8.5%, which indicate sufficient clustering of answers within departments. At both levels the domains from the questionnaire were positively interrelated and all significant.
The results of this study show that Tripod can be used as a method for prospective risk analysis in hospitals. Results of the questionnaire provide information about latent risk factors in hospital departments. However, this study also shows that there are indications that the method is not sensitive enough to detect differences between hospital departments. Therefore, it is important to be careful when interpreting differences in potential risks between departments when using Tripod.
KeywordsProspective risk Prospective risk analysis Risk factors Ecometric Patient safety Adverse event Hospitals
Patient safety is an important aspect of the quality of care in hospitals. Patient safety can be defined as the reduction of the risk of unnecessary harm associated with healthcare to an acceptable minimum . Previous studies have shown that between 2.9% and 16.6% of hospital admissions lead to adverse events , of which approximately 50% are potentially preventable . Adverse events can be defined as harm to patients that was not caused by the underlying disease but by medical management, leading to prolonged hospitalization, re-hospitalization, disability or death . In order to reduce the number of adverse events, safety management systems have been set up in healthcare institutions [4–6]. Safety management systems aim to prevent undesired outcomes in healthcare by a combination of activities, such as improvement projects, incident reporting and analyses, and risk assessments to identify and assess safety risks in the organizational processes .
Risk assessments can be performed retrospectively or prospectively. To date, retrospective risk management has been most common in healthcare . Several methods for retrospective risk assessments in healthcare are currently used; the key element in all these methods is the analysis of the causes of incidents, near-misses and unsafe situations in order to prevent them from happening again in the future. Examples of these methods are the Prevention and Recovery Information System for Monitoring and Analysis (PRISMA) and Root Cause Analysis (RCA) [7, 8]. These methods have two disadvantages. First, they require an open incident reporting culture since they rely on reporting by healthcare staff. Second, the analysis can only take place after an unsafe situation has been revealed, with or without consequences . It therefore makes sense to analyze risks in a prospective manner, complementary to retrospective methods, to prevent unsafe environments that could potentially lead to adverse events .
In prospective risk analysis, processes are analyzed for potential risks in order to prevent errors from happening in the first place. Some well-known methods are the Healthcare Failure Mode (HFMEA) and Bow-Tie [9–13]. Both methods focus on analysis of care processes, and are often organized around a specific disease and therefore bound to one particular medical specialty. In these methods, a group of professionals meets several times to systematically map out the care process that was chosen for the risk analysis. The potential risks and their consequences are determined, and ways of preventing these situations are considered. Although these methods can create awareness about potential risks, they are time-consuming and only focus on one care process at a time.
A complementary method to existing prospective risk analysis methods is Tripod, which takes account of the key organizational processes. Tripod measures latent risk factors categorized into what are termed Basic Risk Factors (BRFs) at the departmental level by means of staff questionnaires. Latent risk factors are risk factors that are present within departments but are not always clearly visible. Tripod uses the individual risk perception of staff to determine the BRFs for an individual department. BRFs are used to determine potential risks in five general organizational domains: Procedures, Training, Communication, Incompatible Goals, and Organization. The method has its origins in the petrochemical sector [14–16] but could also be applied in the healthcare setting . Tripod has the potential to be broader and less time-consuming than existing methods. The method can be used as a starting point to obtain a broad picture of the level of control over the organizational processes at the departmental level, and the results allow prioritization of further in-depth prospective risk analyses. The purpose of this study is to investigate whether Tripod is appropriate as a method for prospective risk analysis within hospital departments. This study is based on the adjusted Tripod for prospective risk analysis in healthcare, known as Tripod Delta HC. We will discuss the changes to Tripod to create Tripod Delta HC in more detail in the Methods section. The research questions addressed in this study are:
Can Tripod Delta HC be used as a measurement instrument for prospective risk analysis in hospitals?
Can Tripod Delta HC be used to assess differences in latent risk factors between hospital departments?
Measurement instrument: theory and development
Tripod is founded on the idea that human error can be prevented or mitigated by controlling the environment people work in [14, 16, 18–20]. According to Tripod, there are latent failures in every work environment and these can be categorized into BRFs. Each of the BRFs may contribute to adverse events in different ways, and when a combination of different undesirable situations emerge at the same time, this will lead to disturbances in the operational process, with or without consequences [21, 22]. BRFs are controllable in the sense that they can be influenced by changing the organization and management of processes.
Tripod was originally developed for use in the petrochemical industry but is also seen as a promising option for patient safety [5, 20]. However, the original questionnaire could not be applied in healthcare without some modifications to create ownership of the users in a healthcare setting . In this study, we took the shortened Tripod questionnaire, known as Tripod Delta Lite, as a starting point for the development of Tripod Delta HC. This shortened questionnaire, which contained five general BRFs and consisted of 75 items, was extracted from the full version that contained ten further specific risk factors and a total of 150 items.
Modifications to Tripod Delta Lite were made by the authors, who consulted a group of 14 experts. The group consisted of registered nurses and researchers with experience in healthcare who assessed the content validity of the questionnaire by reviewing the modified questionnaire. Experts were asked to evaluate the importance and content of each item in the questionnaire and state whether they thought any important aspects were missing. Comments and suggestions were summarized. Nurses generally suggested changes to improve the applicability of the item in a healthcare setting. Researchers commented on the structure of the questionnaire and response categories. This led to the use of five-point scales rather than dichotomous (yes/no) response categories that were used in Tripod Delta Lite. The modifications did not alter the underlying assumptions of the items or the underlying constructs of the questionnaire and led to a revised questionnaire that was used in this study and is called Tripod Delta HC. In the remainder of this article we will only refer to this modified healthcare version.
Measurement instrument: structure
Definitions of the five basic risk factors of Tripod Delta HC
Basic Risk Factors
Insufficient quality or availability of procedures, guidelines, instructions, and manuals (specifications, administration, use in practice).
“Because procedures are insufficiently clear, I sometimes have to act according to my own discretion”.
No or insufficient competence or experience among healthcare staff (not sufficiently suited to their tasks, inadequately trained).
“There are always sufficiently experienced healthcare staff present in the department”.
No or ineffective communication between the various sites, departments or healthcare staff of an organization or with the official bodies.
“Important information is often sent to the wrong department in the hospital”.
BRF Incompatible Goals
The situation in which healthcare staff must choose between optimal working methods according to the established rules on one hand, and the pursuit of production, financial, political, social or individual goals on the other.
“Necessary maintenance work has been postponed due to high costs”.
Shortcomings in the organization’s structure, organization’s philosophy, organizational processes or management strategies, resulting in inadequate or ineffective management of the organization.
“The tasks are not properly coordinated between departments so that work is carried out twice”.
In total the questionnaire contains 75 items that are measured on a five- point Likert scale ranging from ‘totally disagree’ to ‘totally agree’. Respondents were also offered the response option of ‘not applicable’: this box could be checked if the participant was unfamiliar with the content of the item or had no opinion. Respondents were asked to answer the questions with the last six months in mind. In addition, some background characteristics were measured (i.e. age, gender, years working in hospital). See Additional file 1 for the items used.
This study was carried out at one academic hospital and one general hospital, based on a convenience sample. Three departments in the academic hospital took part and ten departments in the general hospital, making thirteen departments in total. There was a wide variety of departments. All healthcare staff working in the participating departments at the time of the study were approached, regardless of whether they had daily patient interaction. The Dutch Medical Research Involving Human Subjects Act does not apply to this research. Therefore, no approval was needed from the Medical Ethics Committee.
Hospital A- academic
Hospital B- general
Overview of the study population
Intensive care department
Lung diseases & cardio department
Short-stay nursing department
Other (interns, operation assistants etc.)
First, negatively worded items were recoded so that a lower score reflected a lower potential risk for all items. In general, respondents were more likely to agree that a positive situation applied than a negative situation. Data was checked for completeness. All items in the questionnaire were mandatory and respondents were not able to skip questions, although they could exit the questionnaire before the end. A total of 32 respondents (5.1% of the total sample) did not answer at least 50% of the questions, meaning that they had stopped partway through the questionnaire; these respondents were excluded from further analyses. The remaining dataset did not contain any missing data. When respondents checked the category ‘not applicable', the score was replaced by a missing value. The analyses were performed on the remaining 594 respondents.
Ecometrics is a multilevel method to evaluate the validity and reliability of imperfect measures of contextual properties . An ecometrics approach was used to ensure that the differences between departments are attributable to differences in potential risks at the departmental level and not to differences between the individual perceptions of the healthcare staff who responded to the questionnaire. The aim of this method is to measure the latent characteristics of ecological units (in this research, the ecological unit is the hospital department). Furthermore, the method aims to combine multiple observations into one scale to analyze the reliability and validity of the scale. The data structure is as follows: the items are at the lowest level, nested within the healthcare staff member, and healthcare staff are nested within the departments, which are at the highest level. There were 13 departments in our sample, which were treated as separate units in the analyses. A weighted item average for all healthcare staff was calculated for each item to calculate an average scale value. This was done by using the item weights for the fixed effects. The item variance, which is an indication of the measurement error, was taken into account in this analysis.
Aspects of the reliability of the scales were assessed in terms of internal consistency using a reliability coefficient. Reliability indicates how well the individual items of a scale measure the underlying concept of the scale. The interpretation is the same at the individual level as at the departmental level, and comparable to Cronbach’s alpha coefficient [24, 25]: values range between 0 and 1, with a higher score representing a more reliable scale. If the reliability coefficient for a scale is at least 0.70, this suggests that the items in a scale are measuring the same concept. Values above 0.80 indicate high internal consistency. Descriptive analyses were used to calculate means, standard deviations and the range of scores for the different scales in the questionnaire at the individual and departmental levels. Variance and intraclass correlations were calculated to assess the clustering of answers at the individual and departmental levels . An intraclass correlation of 20% is seen as moderate . Correlations between scales were calculated at the individual and departmental levels to check that the scales were measuring different concepts. High correlations between scales indicate that the scales are measuring similar concepts. The minimum required sample size to assure adequate reliability for comparing results between departments was estimated for each of the scales, based on the number of healthcare staff in each department and reliability coefficients in the current study. This gives the minimum number of respondents per BRF that is needed to make reliable inferences about differences between departments. The descriptive analyses were conducted using STATA version 11.0 and the multilevel analyses were performed using MlwiN version 2.24.
Reliability of scales at the individual level and departmental level
BRF Incompatible Goals
Descriptives of the scales
Descriptive statistics of the basic risk factors at the individual and departmental levels on a scale from 1 to 5
Individual level (N = 588)
Departmental level (N = 13)
1.42 - 4.11
2.68 - 3.09
1.14 - 4.05
1.92 - 2.28
1.51 - 4.01
2.29 - 2.59
BRF Incompatible Goals
0.99 - 3.42
2.01 - 2.24
1.49 - 3.83
2.38 - 2.82
Variance and intraclass correlations
Variance at the individual and departmental levels and intraclass correlation coefficients (ICCs)
BRF Incompatible Goals
Correlations between scales
Correlations at the individual level (left of the diagonal) and departmental level (right of the diagonal)
BRF Incompatible Goals
BRF Incompatible Goals
Minimum required sample size for acceptable reliability at the departmental level
Minimum required sample size for acceptable reliability at the departmental level*
Number of respondents
BRF Incompatible Goals
This study assessed whether Tripod Delta HC can be used as a measurement instrument for prospective risk analysis in healthcare, and whether it can detect differences in risk factors between hospital departments. Most studies in healthcare use classical psychometric methods originating from psychology for assessing the reliability and validity of questionnaires, for example research into patient safety climate [27, 28]. However, in healthcare these methods are only suitable for assessing differences in individual perceptions or attitudes. This study is mainly interested in the characteristics of the ecological construct (the department), and less in individual attitudes. Using individuals to assess the characteristics of departments results in imperfect measures because it is an indirect way to assess a department characteristic . Therefore, this study used ecometrics to filter out this individual component. The individual variance is split from the variance at the departmental level and the remaining variance at the departmental level can be used as an indication for differences between departments. The results of this study show that the variance at the departmental level is small, although significant for some BRFs. This could have several causes. First, there might actually be only small differences between departments in the study population. Our study population consisted of only two hospitals; it could be that the departments within the hospitals were made very similar by organization-wide policies. In terms of patient safety, this is positive as it indicates that departments are similar and obtain low scores for the different BRFs. Second, there could be a large spread in the individual scores which would increase the variance at the individual level. However, we found no indication for this in our data. And lastly, Tripod Delta HC might not be sensitive enough to detect differences between departments. Given the small number of hospitals in this study, we are unable to determine the precise cause. However, this study does point out that it is important to take account of the variance at the higher level when measuring ecological units. Good psychometric properties are needed to describe differences in individual scores (healthcare staff), but when the intention of a measurement instrument is to describe differences in compositional scores (departments), good ecometric properties are needed.
The results of this study show that the reliability of Tripod Delta HC is acceptable and that the questionnaire can be used to assess potential risks in hospitals. However, some of the BRFs exhibit lower reliability and little variance at the departmental level. For these BRFs it is important to be careful when drawing conclusions about differences between departments in potential risks. The minimum required sample size for acceptable reliability should be considered when using Tripod Delta HC in healthcare. This can be used as a tool to improve the reliability of the scales but also to prevent the unnecessary inclusion of healthcare staff in a study. As can be seen from Table 8, a larger number of respondents is needed for the BRF Incompatible Goals to assure good reliability than for the other BRFs. In practice, hospital departments are generally not large enough to deliver these numbers of healthcare staff. In these cases, the scores should not be used to compare departments with respect to this BRF. Furthermore, when developing interventions to improve patient safety, it is important to consider whether they need to be implemented at the individual level or the departmental level. As all BRFs show low variance at the departmental level, it would be more effective to implement interventions at the individual level throughout the hospitals.
We acknowledge several limitations to this study. First, only two hospitals were included in this study. The results cannot be generalized to the wider hospital population without caution. Future research, for instance in a larger group of hospitals, is needed to validate the findings of the current study. Second, the recruitment of departments was different in the two participating hospitals. However, the questionnaires were filled out by the healthcare staff and the decision to participate in both hospitals was individual and voluntary. Furthermore, the aim of this study was not to compare the two hospitals, but to validate the questionnaire that was used. Therefore, we are confident that the difference in numbers of participating departments between the hospitals did not affect the results of this study. Third, the largest group of respondents was registered nurses (47.4%; see Table 3). The group of physicians in our study was too small for separate statistical analyses. Fourth, questions in the questionnaire were grouped per BRF. This was meant to make it more convenient for the respondent to fill in the questionnaire, not having to switch mentally between concepts. It would be interesting to test whether a mix of questions from different concepts can improve the sensitivity of reliability measures. Although improving Tripod Delta HC was not the purpose of this study, future research could study the practical usage, for example by excluding items with low sensitivity or study the meaning of differences in correlations between departments with similar specialties from different hospitals.
This research assessed whether Tripod Delta HC can be used as a method for prospective risk analysis in hospitals and whether it can measure differences between departments in their assessed risks, using a multilevel ecometric approach. Tripod Delta HC measures five BRFs at the hospital departmental level. In general, the ecometric properties of the modified healthcare version of Tripod Delta HC are satisfactory. The results show that the reliability of the instrument, as measured by the internal consistency of the items, is good at both the individual and departmental levels for most of the BRFs. This indicates that Tripod Delta HC can be used as a measurement instrument for prospective risk analysis in hospitals. However, two BRFs did not show much clustering at the departmental level or acceptable internal consistency at the departmental level. It was not possible to detect differences between departments for these BRFs. Overall, the results show that Tripod Delta HC is a useful instrument for assessing latent risks at the individual level, and for some BRFs at the departmental level as well. It might not be possible to measure differences between departments with Tripod Delta HC but further research in larger settings is needed to confirm these findings.
We wish to thank the participating hospitals, their contact persons and all the healthcare staff who participated in our study. NIVEL acknowledges the Stichting Tripod Foundation for their permission to use a subset of the Tripod Delta questions for this study and paper. Tripod Delta is copyrighted to Stichting Tripod Foundation.
- Brennan TA, Leape LL, Laird NM, Hebert L, Localio AR, Lawthers AG, Newhouse JP, Weiler PC, Hiatt HH: Incidence of adverse events and negligence in hospitalized patients. N Engl J Med. 1991, 324: 370-376. 10.1056/NEJM199102073240604.View ArticlePubMed
- Zegers M, De Bruijne MC, Wagner C, Hoonhout LHF, Waaijman R, Smits M, Hout FA, Zwaan L, Christiaans-Dingelhoff I, Timmermans DR, Groenewegen PP, van der Wal G: Adverse events and potentially preventable deaths in Dutch hospitals: results of a retrospective patient record review study. Qual Saf Health Care. 2009, 18: 297-302. 10.1136/qshc.2007.025924.View ArticlePubMed
- De Vries EN, Ramrattan MA, Smorenburg SM, Gouma DJ, Boermeester MA: The incidence and nature of in-hospital adverse events: a systematic review. Qual Saf Health Care. 2008, 17: 216-223. 10.1136/qshc.2007.023622.PubMed CentralView ArticlePubMed
- van der Schaaf TW: Medical applications of industrial safety science. Qual Saf in Health Care. 2002, 11: 205-206. 10.1136/qhc.11.3.205.View Article
- Vincent C: Patient safety. 2011, Chichester: Wiley-Blackwell
- Kohn LT, Corrigan JM, Donaldson MS: To err is human: building a safer health system. 2000, Washington DC: National Academies Press, 627
- Smith A, Boult M, Woods I, Johnson S: Promoting patient safety through prospective risk identification: example from peri-operative care. Qual Saf Health Care. 2010, 19: 69-73. 10.1136/qshc.2008.028050.View ArticlePubMed
- Nicolini D, Waring J, Mengis J: The challenges of undertaking root cause analysis in health care: a qualitative study. J Health Ser Res Policy. 2011, 16: 34-41. 10.1258/jhsrp.2010.010092.View Article
- Habraken MMP, Van der Schaaf TW, Leistikow IP, Reijnders-Thijssen PMJ: Prospective risk analysis of health care processes: A systematic evaluation of the use of HFMEA in Dutch health care. Ergonomics. 2009, 52: 809-819. 10.1080/00140130802578563.View ArticlePubMed
- Wierenga PC, Lie A, Huen L, De Rooij SE, Klazinga NS, Guchelaar HJ, Smorenburg SM: Application of the bow-tie model in medication safety risk analysis consecutive experience in two hospitals in the Netherlands. Drug Saf. 2009, 32: 663-673. 10.2165/00002018-200932080-00005.View ArticlePubMed
- Bonnabry P, Cingria L, Ackermann M, Sadeghipour F, Bigler L, Mach N: Use of a prospective risk analysis method to improve the safety of the cancer chemotherapy process. Int J Qual Health Care. 2006, 18: 9-16. 10.1093/intqhc/mzi082.View ArticlePubMed
- Joint commission on accreditation of healthcare organizations: Failure Mode and Effects Analysis in Health Care. 2005, Oakbrook Terrace, IL: Ref Type: Report
- Institute for healthcare improvement: Failure modes and effects analysis (FMEA). 2013, Cambridge, Massachusetts, USA: Ref Type: Online Source
- Reason JT: Managing the risks of organizational accidents. 1997, Aldershot: Ashgate, 6
- Groeneweg J: Controlling the controllable: preventing business upsets. 2002, Leiden, The Netherlands: Global Safety Group
- Reason J: Human error. 1990, Cambridge: Cambridge University PressView Article
- Parker D, Lawton R: Recent Advances in Anaesthesia and Intensive Care. Managing medical mishaps: learning lessons from industry. 2003, London: Greenwich Medical Media, 209-222.
- Croskerry P, Cosby KS, Schenkel SM, Wears RL: Patient safety in emergency medicine. 2009, Philadelphia: Lippincott Williams & Wilkins
- Van Beuzekom M, Boer F, Akerboom S, Hudson P: Patient safety: latent risk factors. Br j anaesth. 2010, 105: 52-59. 10.1093/bja/aeq135.View ArticlePubMed
- Armitage G: Human error theory: relevance to nurse management. J Nurs Manag. 2009, 17: 193-202. 10.1111/j.1365-2834.2009.00970.x.View ArticlePubMed
- Carayon P: Handbook of human factors and ergonomics in health care and patient safety. 2011, Boca Raton, CA: CRC PressI LlcView Article
- Cambon J, Guarnieri F, Groeneweg J: Learning from Diversity: Model-Based Evaluation of Opportunities for Process (Re)-Design and Increasing Company Resilience. Towards a new tool for measuring Safety Management Systems performance. 2006, 53.
- Sujan MA: A novel tool for organisational learning and its impact on safety culture in a hospital dispensary. Reliab Eng System Safety. 2012, 101: 21-34.View Article
- Raudenbush SW: The quantitative assessment of neighborhood social environments. Neighborhoods and Health. Edited by: Kawachi I, Berkman L. 2003, Oxford University Press, 112-131.View Article
- Cronbach LJ: Coefficient alpha and the internal structure of tests. Psychometrika. 1951, 16: 297-334. 10.1007/BF02310555.View Article
- Twisk JW: Applied multilevel analysis: a practical guide. 2006, Cambridge: Cambridge University PressView Article
- Colla JB, Bracken AC, Kinney LM, Week WB: Measuring patient safety climate: a review of surveys. Qual Saf Health Care. 2005, 14: 364-366. 10.1136/qshc.2005.014217.PubMed CentralView ArticlePubMed
- Smits M, Christiaans-Dingelhoff I, Wagner C, Van der Wal G, Groenewegen PP: The psychometric properties of the 'Hospital Survey on Patient Safety Culture' in Dutch hospitals. BMC Health Serv Res. 2008, 8: 230-10.1186/1472-6963-8-230.PubMed CentralView ArticlePubMed
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6963/14/103/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.