- Research article
- Open Access
- Open Peer Review
The safety attitudes questionnaire – ambulatory version: psychometric properties of the Slovenian version for the out-of-hours primary care setting
BMC Health Services Research volume 17, Article number: 36 (2017)
Several tools have been developed to measure safety attitudes of health care providers, out of which the Safety Attitudes Questionnaire (SAQ) is regarded as one of the most appropriate ones. In 2007, it was adapted to outpatient (primary health care) settings and in 2014 it was tested in out-of-hours health care settings in Norway. The purpose of this study was to translate the English version of the SAQ-Ambulatory Version (SAQ-AV) to Slovenian language; to test its reliability; and to explore its factor structure.
This was a cross-sectional study that took place in Slovenian out-of-hours primary care clinics in March-May 2015 as a part of an international study entitled Patient Safety Culture in European Out-of-hours services. The questionnaire consisted of the Slovenian version of the SAQ-AV. The link to the questionnaire was emailed to health care workers in the out-of-hours clinics. A total of 438 participants were invited. We performed exploratory factor analysis.
Out of 438 invited participants, 250 answered the questionnaire (response rate 57.1%). Exploratory factor analysis put forward five factors: 1) Perceptions of management, 2) Job satisfaction, 3) Safety climate, 4) Teamwork climate, and 5) Communication. Cronbach’s alpha of the whole SAQ-AV was 0.922. Cronbach’s alpha of the five factors ranged from 0.587 to 0.791. Mean total score of the SAQ-AV was 56.6 ± 16.0 points. The factor with the highest average score was Teamwork climate and the factor with the lowest average was Job satisfaction.
Based on the results in our study, we cannot state that the SAQ-AV is a reliable tool for measuring safety culture in the Slovenian out-of-hours care setting. Our study also showed that there might be other safety culture factors in out-of-hours care not recognised before. We therefore recommend larger studies aiming to identify an alternative factor structure.
Patient safety is an important aspect of quality assurance and improvement in health care. In recent years, a concept of patient safety culture has been developed. It describes leader and staff interaction, attitudes, routines, awareness, and practices, which impinge on the risk of patient adverse events . Safety culture is regarded as a group phenomenon rather than that of individuals . In organizational psychology research, safety culture is described by both qualitative and quantitative methods . Quantitative surveys have concentrated on measuring staff perceptions, which are referred to as organizational climates. Organizational climates are mathematical expressions of how members in natural social units perceive that cultural norms are enacted by leadership and members in the unit. These climates are measured both according to level of mean and the degree to which staff share the perceptions, which is the organizational climate strength . Organizational climates with diverging perceptions amongst staff are regarded as weak with limited power to predict staff practices .
Primary health care differs from hospitals in terms of organisational structure, administrative and clinical processes and the reasons for encounters . Therefore, also patient safety culture dimensions could differ when compared to secondary or tertiary health care.
So far, several studies have addressed the issue of patient safety culture in primary health care [7–10]. The revealed dimensions were similar but not identical. The most comprehensive view was provided by Kirk et al.  who conducted a literature review of patient safety dimensions and put forward nine dimensions that patient safety culture in primary care was expressed by: overall commitment to quality, priority given to patient safety, perceptions of the causes of patient safety incidents and their identification, investigating patient safety incidents, organisational learning following a patient safety incident, communication about safety issues, personnel management and safety issues, staff education and training about safety issues, and teamworking around safety issues.
Out-of-hours care (OOHC) services are primary health care services enabling access to primary health care services during out-of-hours working hours (weeknights and weekends) . There are many different models for organizing OOHC services  and yet no clear definition of OOHC has been developed in the literature .
In Slovenia, OOHC and emergency medical services (EMS) are combined and are available both at the same place and time. Professionals that work in OOHC are family physicians, emergency physicians, and emergency nurses. Sometimes, there are also laboratory technicians and radiology technicians. Usually, emergency nurses work only in OOHC settings while family physicians work in their practice and in OOHC on the basis of rotation. One or two teams are on call at the same time on 8-hours rotation. An OOHC team consists of a family- or emergency physician and two emergency nurses. These teams are located in primary health care centres and available 24/7 enabling free access of patients on their own demand. If there is an on-the-field emergency, the team leaves the OOHC surgery in order to respond to the call. This is a major difference between Slovenian and most European OOHC centres. In a majority of European countries OOHC centres are organized separately from EMS . An OOHC centre in Slovenia has thus a unified leadership and can be seen as a “natural social unit”; which is a validation criteria for organizational climate measurements . Patient safety in OOHC has been studied only in few European countries [9, 13–16]. These studies dealt with several aspects of patient safety, i.e. patient safety incidents  and triage .
Several tools have been developed to measure safety attitudes of health care providers [7, 17–22]. The Safety Attitudes Questionnaire (SAQ) is one of the widely used and also one of the most appropriate instruments for measuring patient safety culture [22–24]. It measures six factors: Teamwork climate, Safety climate, Job satisfaction, Perceptions of management, Working conditions and Stress recognition . In 2007, the SAQ was adapted to outpatient (primary health care) settings . This SAQ – Ambulatory Version (SAQ-AV) proved to be a reliable tool for comparing attitudes across different professional groups of health care providers outside hospitals . In 2014, Bondevik et al.  tested the Norwegian version of SAQ-AV and proved it to be a reliable tool for measuring patient safety culture in OOHC in Norway.
In Slovenia, patient safety culture in OOHC has not been studied and no tools for measuring it have yet been developed or adapted. The purpose of this study was to translate the SAQ-AV to Slovenian language and to adapt it to the Slovenian OOHC settings. We had the following aims of the study: 1) to translate the English version of SAQ-AV to Slovenian language; 2) to test the reliability of the Slovenian version of SAQ-AV in OOHC settings; and 3) to determine the factor structure of the Slovenian version of SAQ-AV in OOHC settings.
Type of study and settings
This was a cross-sectional study that took place in Slovenian out-of-hours primary care clinics from March 16th to May 1st 2015. The study was a part of an international study entitled Patient Safety Culture in European Out-of-hours services (SAFE-EUR-OOH), which was led by a coordinating research group from Norway. It was a project of the European research network for out-of-hours primary health care (EurOOHnet) .
In Slovenia, there are 60 out-of-hours clinics. All were invited, and 37 (61.7%) agreed to participate. These OOHC clinics were of variable size; the smallest covered 35,000 inhabitants and the largest 300,000 inhabitants. On average 30 professionals work in these clinics.
We used the SAQ-AV which was translated from English to Slovenian according to modified principles adapted from Beaton et al. . Initially, the original English version was translated into Slovenian using a professional translation bureau. Next, an expert committee with Slovenian clinicians and researchers adapted the initial translated version to the OOHC setting in Slovenia. This adapted version of the questionnaire was translated back into English by a second independent translation bureau being blinded to the original version. Based on this back-translated version, the national expert committee made the necessary adjustments in order to clarify possible misunderstandings. The pre-final version was evaluated by six employees in different OOHC clinics. They were asked to give feedback about the comprehensibility of the Slovenian version of SAQ-AV. Minimal changes to the questionnaire regarding Slovenian wording were made according to their suggestions. These six employees were later invited to participate in the study.
The SAQ-AV consists of 62 items. Each item should be answered on a 5-point Likert scale by which the respondents indicated their level of agreement with the statement (1 = disagree strongly, 2 = disagree slightly, 3 = neutral, 4 = agree slightly, 5 = agree strongly) . In the analysis, scores of negatively worded items were reversed, so that higher scores in the data set always indicate a more positive evaluation of the unit’s patient safety culture.
There were also some demographic questions (sex, age, function, working experiences, shifts, type of employment).
In each Slovenian OOHC clinic that agreed to participate, a person in charge for the data collection was selected. This person asked all employees (physicians, graduate nurses, nurse managers, trainees, nurses, radiology technicians, and office managers) working in the OOHC clinic to participate. Using this procedure, a total of 438 people agreed to participate. The participation was voluntary.
The national key researcher for Slovenia (ZKK) collected the e-mail addresses of all willing employees in participating OOHC clinics. On March 16th 2015, the link to an electronic version of the Slovenian SAQ-AV was mailed to all participants from the coordinating research group in Norway, using the data collection program Qualtrics. An automatic reminder was sent by Qualtrics to those who had not responded after two weeks.
The Qualtrics file with SAQ-AV data from Slovenia was converted into SPSS for further analysis. The researchers were provided with anonymous data files where possible identifiers like e-mail and IP addresses had been removed by the administrative coordinator in the project. It was not possible for the researchers to link participants to their responses.
We performed the explorative factor analysis (EFA) with the aim of data reduction and therefore simplification of a large number of intercorrelated measures of safety attitude to a few representative constructs or factors. Out of 62 items of the SAQ, 31 items were entered into the EFA. The 31 items corresponds to the measurement model of SAQ which was tested and validated in a previous study . Further, the principal components analysis (PCA) with the Varimax rotation method was applied to discover the main patterns of variation among respondents in accordance with the dimensions identified by the EFA. First, mean scores were calculated from the scale’s items to generate the composite scores for the safety attitude dimensions. The newly created composite variables were subject of the PCA. Scale reliability was tested by Cronbach’s alpha. We also determined the K-M-O and Bartlett statistic. Additionally, we performed corrected item-total correlations (CITCs) in order to strengthen validity and reliability results.
A free software environment for statistical computing and graphics R was applied using the prcomp() function in the stats package with the purpose of applying PCA and visualizing the data by using the biplot.
We decided to retain the five factors which had eigenvalues greater than one.
Demographic characteristics of the sample
Out of 438 invited participants, 250 answered the questionnaire (response rate 57.1%). The demographic characteristics of the sample are presented in Table 1.
Factor structure of the SAQ
Due to incomplete SAQ-AV responses, 119 (47.6%) cases were excluded from the factor analysis. EFA put forward five factors: 1) Perceptions of management, 2) Job satisfaction, 3) Safety climate, 4) Teamwork climate, and 5) Communication. Perceptions of Management included six items, Job Satisfaction six items, Safety Climate four items, Teamwork Climate three items, and Communication three items (Table 2).
Reliability of the SAQ and its factors
Cronbach’s alpha of the five factors ranged from 0.587 to 0.791. Those five factors accounted for 52.7% of the variance (K-M-O statistic 0.897; Bartlett statistic 2378.201 (p < 0.001). The corrected item-total correlation scores of the factors ranged from 0.39 to 0.64 (Table 2).
The bi-plot (the bivariate plot) in Fig. 1 shows a strong relationship between the Perceptions of management, Job satisfaction and Safety climate. It should be noted that all variables point to the same direction, which indicate that variables are positively associated. The eigenvalues indicate that two components provide a reasonable summary of the data, accounting for 69.65% of the total variance. From the PCA bi-plot (Fig. 1) it can be observed that the variance along the Comp. 2 axis is higher than along Comp. 1 axis, especially if the account outliers (e.g. 161, 166 and 133) are taken into account. For example, in this case, outliers have significantly lower values corresponding to the Communication (F5) dimension of the safety attitude construct. In contrast, the respondents who are plotted in upper part of the biplot (positive loadings on Comp. 1 and Comp. 2) expressed higher level of agreement with the Safety climate, Teamwork climate and Communication. Moreover, the density of the respondents is a bit higher around the centre of the bi-plot (at the starting point of the vectors). It appears that those respondents expressed moderate agreement with the Perceptions of management, Job satisfaction and Safety climate.
The Slovenian version of SAQ-AV, EFA revealed five factors, some of which were similar, but not identical to previous versions of the SAQ-AV [7, 16]. This study was a part of the international study lead by the Norwegian researchers who had adapted the SAQ-AV for Norwegian primary care settings. The Norwegian factor structure  provided five factors: Teamwork climate, Safety climate, Job satisfaction, Working conditions, and Perception of management. The original SAQ, developed at the University of Texas at Austin , described six factors: Teamwork climate, Safety climate, Working conditions, Job satisfaction, Perceptions of management and Stress recognition.
Two of the factors in our study had Cronbach’s alphas lower than the recommended 0.7 . Considering cultural differences between the countries this is not surprising. Although content of four factors had similarities in the Norwegian and Slovenian factor structure, several items loaded on different factors in Slovenia compared to Norway.
This indicates that the perceptions of safety culture in Norway and Slovenia may be different. Items related to perceived problems voicing concerns regarding patient safety, emerged as an independent factor in our study. We named this factor Communication. Items included in this factor in Slovenia belong to several factors found in other studies [25, 29]: Teamwork climate, Safety climate, and Perceptions of management. Actually, Communication is a part of all these fields but it seems that it was so important to the Slovenian participants that it emerged as an individual factor.
There is some overlap between Communication and the concept of psychological safety. The term psychological safety refers to which extent team members feel comfortable seeking feedback, sharing information, asking for help, talking about errors, and experimenting [30–32]. Psychological safety is a requirement for effective communication regarding risk in patient care , and is complementary to a good safety culture . Psychological safety has been recognised as safety relevant in other health care settings [30, 33] but not in previous analyses of SAQ-AV data nor in OOHC settings. Our results may indicate that Communication and psychological safety are perceived more important and perhaps cultural traits more at stake to participants in Slovenia compared to other countries, since they possibly emerge as individual factors without the study seeking to map them explicitly. The finding supports that Communication and Psychological safety are considered possible independent factors in future safety culture surveys.
Our study has some methodological limitations which need to be addressed. The first one is the process of factor analysis. We chose not to proceed by confirmatory factor analysis despite the fact that it is usually a key step in psychometric evaluation of survey tools. Our decision was based on small sample size which was a consequence of specific study settings.
The second one is the low value of Cronbach’s alpha in two of the factors. These are the same factors that have only three items. As the alpha value is related to the number of items in the scale , the low values of Cronbach’s alpha could be a consequence of the inclusion of a limited number of items. These two factor were also problematic in terms of item-to-total correlations, which were below the recommended 0.5 . However, some authors advocate the cut-off value of 0.3 . These factors were not removed from the model and the decision was based on content consideration and acceptable alpha values. The exclusion of items did not improve the model.
Another limitation is the limited response rate and the relatively high number of incomplete questionnaires. However, only two clinics out of 37 failed to provide at least one complete questionnaire. After the exclusion of 119 incomplete questionnaires, 131 cases were available for factor analysis of the selected 22 variables. As noted by Mundfrom et al.  there are no clear criteria for deciding the appropriate factor analysis sample size. Suggestions for minimum absolute sample size range from 100 to over 1000, and suggested minimum ratio of number of variables to number of factors range from 3 to 20, depending, among other things, upon the number of factors extracted and the level of communality . The sample size in our study was limited, and we recommend further studies with a higher number of participants.
An EFA always produces a solution, but does not assess the risk that the EFA solution only describes the data set, and may not be generalizable. Due to these limitations, the results of our study may not be generalised to the whole population.
Based on the results in our study, we cannot state that the SAQ-AV is a reliable tool for measuring safety culture in the Slovenian out-of-hours care setting. This might be due to methodological limitations. Our study also showed that there might be other safety culture factors in OOHC not recognised before, such as psychological safety. We therefore recommend larger studies aiming to identify an alternative factor structure and other concepts of safety culture in OOHC.
Confirmatory factor analysis
Corrected item-total correlations
Explorative factor analysis
Emergency medical services
European research network for out-of-hours primary health care
Out-of-hours Health Care
Safety attitudes questionnaire
Safety attitudes questionnaire-ambulatory version
Deilkas ET. Patient safety culture - opportunities for healthcare management. Oslo: University of Oslo, Norway; 2010.
Zohar D. Safety climate: conceptual and measurement issues. In: Campbell Quick J, Tetrick L, editors. Handbook of occupational health psychology. Washington, D. C: American Psychological Association; 2003. p. 123–42.
Guldenmund FW. The nature of safety culture: a review of theory and research. Safety Sci. 2000;34(1-3):215–57.
Lawrence J, Demaree R, Wolf G. Estimating within-group interrater reliability with and without response bias. J Appl Psychol. 1984;69(1):85–98.
Zohar D, Livne Y, Orly T, Admi H, Donchin Y. Healthcare climate: A framework for measuring and improving patient safety. Crit Care Med. 2007;35(5):1312–7.
Palacios-Derflingher L, O’Beirne M, Sterling P, Zwicker K, Harding BK, Casebeer A. Dimensions of patient safety culture in family practice. Healthcare quarterly (Toronto, Ont). 2010;13 Spec No:121–7.
Modak I, Sexton JB, Lux TR, Helmreich RL, Thomas EJ. Measuring safety culture in the ambulatory setting: the safety attitudes questionnaire - ambulatory version. J Gen Intern Med. 2007;22:1–5.
Schutz AL, Counte MA, Meurer S. Development of a patient safety culture measurement tool for ambulatory health care settings: analysis of content validity. Health Care Manag Sci. 2007;10(2):139–49.
Bondevik GT, Hofoss D, Holm Hansen E, Deilkas ET. Patient safety culture in Norwegian primary care: a study in out-of-hours casualty clinics and GP practices. Scand J Prim Health Care. 2014;32:132–8.
Kirk S, Parker D, Claridge T, Esmail A, Marshall M. Patient safety culture in primary care: developing a theoretical framework for practical use. Qual Saf Health Care. 2007;16(4):313–20.
Huibers L, Giesen P, Wensing M, Grol R. Out-of-hours care in western countries: assessment of different organizational models. BMC Health Serv Res. 2009;9:105.
Leutgeb R, Walker N, Remmen R, Klemenc-Ketis Z, Szecsenyi J, Laux G. On a European collaboration to identify organizational models, potential shortcomings and improvement options in out-of-hours primary health care. Eur J Gen Pract. 2014;20(3):233–7.
Phillips H, Van Bergen J, Huibers L, Colliers A, Bartholomeeusen S, Coenen S, Remmen R. Agreement on urgency assessment between secretaries and general practitioners: an observational study in out-of-hours general practice service in Belgium. Acta Clin Belg. 2015;70(5):309–14.
Huibers LAMJ, Moth G, Bondevik GT, Kersnik J, Huber CA, Christensen MB, Leutgeb R, Casado AM, Remmen R, Wensing M. Diagnostic scope in out-of-hours primary care services in eight European countries: an observational study. BMC Fam Pract. 2011;12:30.
Smits M, Huibers L, Kerssemeijer B, de Feijter E, Wensing M, Giesen P. Patient safety in out-of-hours primary care: a review of patient records. BMC Health Serv Res. 2010;10:335.
Bondevik GT, Hofoss D, Hansen EH, Deilkas EC. The safety attitudes questionnaire - ambulatory version: psychometric properties of the Norwegian translated version for the primary care setting. BMC Health Serv Res. 2014;14:139.
Singer SJ, Gaba DM, Geppert JJ, Sinaiko AD, Howard SK, Park KC. The culture of safety: results of an organization-wide survey in 15 California hospitals. Qual Saf Health Care. 2003;12:112–8.
Weingart SN, Farbstein K, Davis RB, Phillips RS. Using a multihospital survey to examine the safety culture. Jt Comm J Qual Saf. 2004;30(3):125–32.
Sorra JS, Nieva VF. Hospital survey on patient safety culture. Agency for Healthcare Research and Quality: Rockville; 2004.
Colla JB, Bracken AC, Kinney LM, Weeks WB. Measuring patient safety climate: a review of surveys. Qual Saf Health Care. 2005;14(5):364–6.
Flin R, Burns C, Mearns K, Yule S, Robertson EM. Measuring safety climate in health care. Qual Saf Health Care. 2006;15(2):109–15.
Parker D, Wensing M, Esmail A, Valderas JM. Measurement tools and process indicators of patient safety culture in primary care. A mixed methods study by the LINNEAUS collaboration on patient safety in primary care. Eur J Gen Pract. 2015;21(Suppl):26–30.
Mannion R, Davies H, Konteh F, Jung T, Scott T, Bower P, Whalley D, McNally R, McMurray R. Measuring and Assessing Organisational Culture in the NHS (OC1). In: York. 2008.
Use of Patient Safety Culture Instruments and Recommendations. In. Aarhus: European Society fr Quality in Healthcare; 2010.
Sexton JB, Helmreich RL, Neilands TB, Rowan K, Vella K, Boyden J, Roberts PR, Thomas EJ. The safety attitudes questionnaire: Psychometric properties, benchmarking data, and emerging research. BMC Health Serv Res. 2006;3:44.
Huibers L, Phillips H, Giesen P, Remmen R, Christensen MB, Bondevik GT. EurOOHnet-the European research network for out-of-hours primary health care. Eur J Gen Pract. 2014;20(3):229–32.
Beaton DE, Bombardier C, Guillemin F, Ferraz MB. Guidelines for the Process of Cross-Cultural Adaptation of Self-Report Measures. Spine. 2000;25(24):3186–91.
Hair JF, Black WC, Babin BJ, Anderson RE. Multivariate data analysis. London: Pearson Prentice Hall; 2010.
Deilkas ET, Hofoss D. Psychometric properties of the norwegian version of the safety attitudes questionnaire (SAQ), generic version (short form 2006). BMC Health Serv Res. 2008;8:191.
Edmondson AC. Psychological safety and learning behavior in work teams. Adm Sci Q. 1999;44(2):350–83.
Edmondson AC. Speaking up in the operating room: How team leaders promote learning in interdisciplinary action teams. J Manag Stud. 2003;40(6):1419–1452(1434).
Edmondson AC. Learning from failure in health care: frequent opportunities, pervasive barriers. Qual Saf Health Care. 2004;13(suppl_2):ii3–9.
Gadd S, Collins AM. Safety Culture: A review of the literature. Sheffield: Health & Safety Laboratory; 2002.
Koufteros XA. Testing a model of pull production: A paradigm for manufacturing research using structural equation modeling. J Oper Manag. 1999;17:467–88.
Nunnally J, Bernstein I. Psychometric theory. New York: McGraw-Hill; 1994.
Mundfrom DJ, Shaw DG, Ke TL. Minimum sample size recommendations for conducting factor analyses. Int J Test. 2005;5(2):159–68.
MacCallum RC, Widaman KF, Zhang S, Hong S. Sample size in factor analysis. Psychol Methods. 1999;4(1):84–99.
We thank all the participants in the study. We also thank the late Professor Janko Kersnik for his support in this study.
The National Centre for Emergency Primary Health Care, Uni Research Health, Bergen, Norway, covered the expenses in connection with the translation/back-translation of the Safety Attitudes Questionnaire and the data collection.
This study was partially supported by unrestricted programme grant from National Research Agency P3 0339.
Availability of data and materials
The dataset upon which conclusions are made is available by the authors.
GTB conceived the international study and prepared its methodology. ZKK and VS lead and performed the study in Slovenia. MM, ETD and DH performed the statistical analysis, interpreted the data and reviewed the manuscript critically. ZKK and MM drafted the manuscript. All authors reviewed and revised the manuscript critically and approved the final version of the manuscript.
The authors declare no competing interests.
Consent for publication
Ethics approval and consent to participate
The study was approved by the National Ethics Committee of the Republic of Slovenia (No. 25/11/14). All the participants gave an oral informed consent to participate in the study. Their participation was voluntary and data were analysed anonymously.