Skip to main content

A mixed-methods evaluation of the MOREOB program in Ontario hospitals: participant knowledge, organizational culture, and experiences



MOREOB (Managing Obstetrical Risk Efficiently) is a patient safety program for health care providers and administrators in hospital obstetric units. MOREOB has been implemented widely in Canada and gradually spread to the United States. The main goal of MOREOB is to build a patient safety culture and improve clinical outcomes. In 2013, 26 Ontario hospitals voluntarily accepted provincial funding to participate in MOREOB. The purpose of our study was to assess the effect of MOREOB on participant knowledge, organizational culture, and experiences implementing and participating in the program at these 26 Ontario hospitals.


A convergent parallel mixed-methods study in Ontario, Canada, with MOREOB participants from 26 hospitals. The quantitative component used a descriptive pre-post repeated measures design to assess participant knowledge and perception of culture, administered pre-MOREOB and after each of the three MOREOB modules. Changes in mean scores were assessed using mixed-effects regression. The qualitative component used a qualitative descriptive design with individual semi-structured interviews. We used content analysis to code, categorize, and thematically describe data. A convergent parallel design was used to triangulate findings from data sources.


308 participants completed the knowledge test, and 329 completed the culture assessment at all four time points. Between baseline and post-Module 3, statistically significant increases on both scores were observed, with an increase of 7.9% (95% CI: 7.1 to 8.8) on the knowledge test and an increase of 0.45 (on a scale of 1–5, 95% CI: 0.38 to 0.52) on the culture assessment. Interview participants (n = 15) described improvements in knowledge, interprofessional communication, ability to provide safe care, and confidence in skills. Facilitators and barriers to program implementation and sustainability were identified.


Participants were satisfied with their participation in the MOREOB program and perceived that it increased health care provider knowledge and confidence, improved safety for patients, and improved communication between team members. Additionally, mean scores on knowledge tests for obstetric content and culture assessment improved. The MOREOB program can help organizations and individuals improve care by concentrating on the human and organizational aspects of patient safety. Further work to improve program implementation and sustainability is required.

Peer Review reports


There are over 380,000 births in Canada each year [1] and the majority occur in hospitals [2]. Improvements in women’s health and health care, as well as advancements in prenatal and obstetric care have led to decreases in maternal and neonatal mortality rates [3]. However, recent trends in maternal characteristics, including older age of mothers and higher rates of obesity, have resulted in increased rates of some maternal and neonatal adverse events [4]. In 1999, the Institute of Medicine’s landmark report To Err Is Human identified adverse events, many of which are preventable, as a common cause of morbidity and mortality in hospitals and called for a coordinated strategy to improve patient safety [5]. Although the report set a goal of reducing adverse events by 50% over five years, there remains room for improvement. For example, in Canada in 2014–2015, 1 in 18 hospital stays (or 138,000 hospitalizations) involved at least one occurrence of potentially preventable harm [6]. While obstetric and newborn patients have a lower rate of harm compared to other patient groups (with a harm rate of 4.2 per 100 patients, and 1.0 per 100 patients, respectively) [6], obstetrical departments are often a focus for patient safety and quality improvement. There is a need to prioritize patient safety in obstetrics due to the unique challenges of working in labour and delivery [7], the fact that an adverse outcome may affect more than one patient, and due to the high cost of liability insurance and claims [8, 9], with obstetrics accounting for nearly half of the Healthcare Insurance Reciprocal of Canada’s (HIROC) liability claims and patient compensation payments [10].

The MOREOB (Managing Obstetrical Risk Efficiently) program was developed by the Society of Obstetricians and Gynaecologists of Canada (SOGC), and is currently administered by Salus Global, a healthcare consulting and implementation firm established by the SOGC and HIROC in 2007 [11]. The MOREOB program aims to create a culture of patient safety in obstetrical units by using high reliability organization (HRO) principles [12],including awareness of systems that influence patient care and outcomes, focusing on near-misses as opportunities to improve processes, a culture that promotes open communication and teamwork, and a commitment to ongoing training and learning [13]. The program contains three evidence-informed modules: Learning Together, Working Together, and Changing the Culture [14]. The program requires that all obstetric team members at enrolled hospitals (e.g., nurses, midwives, family physicians, obstetricians, administrators) jointly participate in the three modules, each of which is implemented over approximately one year. MOREOB uses a “train-the-trainer” approach whereby the hospital recruits a “core team” of interdisciplinary team members, who are then trained and supported by Salus Global to implement the program at their respective hospitals. Standardization of implementation across sites is facilitated by detailed program manuals and implementation responsibilities for the core team, orientation sessions by MOREOB facilitators prior to implementation of each module, and ongoing support by program consultants [14]. Additional information about the MOREOB program goals is provided in Additional file 1.

To date, over 300 North American hospitals have participated in the MOREOB program [12], but there are limited studies and variable outcomes. In Canada, the province of Alberta implemented the MOREOB program in all hospitals in 2004, and a subsequent evaluation found statistically significant reductions in perineal tears, length of stay, and neonatal severe morbidity rates [15]. No significant changes were observed for postpartum infection, postpartum hemorrhage with caesarean section, or fetal mortality [15]. One Ontario study reported that participation in the MOREOB program resulted in a significant decrease in the number of reportable events and associated costs for the healthcare liability insurer [16]. Another Ontario study found that participation in the MOREOB program did not translate to an observable decrease in adverse maternal or neonatal birth outcomes [17]. However, data collected from Canadian sites showed increases in participant knowledge and organizational safety culture [18].

In 2002 in the province of Ontario, hospitals providing obstetrical services began to voluntarily implement the program, and by 2012, 67 hospitals were involved with MOREOB. In 2013, the Ontario Ministry of Health and Long-Term Care (MOHLTC) offered to fund the program for the remaining 34 provincial hospitals that were not yet engaged with MOREOB, and 26 hospitals accepted and completed the program. The implementation of the MOREOB program at these Ontario hospitals provided an opportunity to conduct a mixed-methods evaluation of the program implementation. This evaluation aimed to assess the outcomes and local implementation process for the MOREOB program in the Ontario context, provide data to facilitate comparison between provincial settings, and contribute qualitative data for more in-depth information on participant experiences with the program.


This study used a convergent parallel mixed-methods design [19, 20]. Simultaneous data collection and subsequent analysis and integration of data from quantitative (i.e., surveys) and qualitative (i.e., semi-structured interviews) sources allowed us to compare and corroborate our findings from different data sources (Fig. 1). Our research team was not involved in implementing the MOREOB program in any of the sites but conducted the comprehensive evaluation of the program with two main goals, specifically to evaluate (1) the program outcomes and (2) the implementation process (Table 1).

Fig. 1
figure 1

Quantitative and qualitative data sources

Table 1 Overall evaluation of MOREOB program in Ontario

This paper reports the methods and results for objectives 2a, 2b, and 2c only. The outcomes of knowledge and culture were selected as they were aligned with the MOREOB program goals (Additional file 1), and the data available from existing surveys administered routinely as part of participation in the MOREOB program. The evaluation of the implementation process aimed to identify barriers and facilitators to implementing and participating in the program to provide insight into why the program may or may not have worked in different settings and to identify areas for improvement.



We used a descriptive pre-post repeated measures design to assess the effect of the MOREOB program on participants’ knowledge about obstetrical core content contained in the MOREOB program and to assess the effect of the MOREOB program on participants’ perceptions of their organizational culture.


We used data from two existing surveys administered routinely to all participants by Salus Global as part of involvement in the MOREOB program:

  1. 1.

    An online, standardized, 75-item multiple-choice knowledge test previously developed by the MOREOB program which assesses core clinical knowledge covered in the program. The knowledge test had a maximum score of 93, however for ease of interpretation, we transformed the score to a percentage (D Walker, 2015, unpublished; [18]).

  2. 2.

    A previously developed and validated online 54-item Culture Assessment Survey (CAS) [21] to assess participants’ perceptions of the culture of patient safety in their organization. As designed, the CAS is divided into six categories (patient safety is everyone’s priority, teamwork, valuing individuals, open communication, learning, and empowering people), each of which measures aspects of patient safety in obstetrics. Nine questions are included in each category, with each question scored on a scale of 1 to 5. The mean response to the nine questions on the 1 to 5 scale is used as the score for each category. The overall score for the CAS is then the mean score on the 1 to 5 scale for the six data elements, and therefore the overall score for the CAS is presented on a scale from 1 to 5 with a maximum score of 5 [21]. The total CAS score as a whole, as well as the scores on the six categories were found to have good internal consistency [21].

The knowledge test and CAS were administered at four time points; pre-MOREOB, post-Module 1, post-Module 2, and post-Module 3 (Fig. 1).


As a standard component of participation in the MOREOB program, all MOREOB participants from the 26 hospitals were asked to complete a knowledge test and the CAS at four time points while progressing through the program. In addition to participant scores, the database from Salus Global contained information about job title, hospital site, and a unique identifier which allowed for participant responses to be linked over time. Only those participants who completed the knowledge test and/or CAS at all four time points were included in the final analysis.

Data collection

MOREOB participants at the 26 sites completed the standardized knowledge-based assessment tool and CAS at each of the four time points as they progressed through the program. Data were collected by Salus Global on their MOREOB secure on-line platform and released to the research team for analysis.

Data analysis

We summarized items from the baseline questionnaire and after each of the three modules using descriptive statistics (means and standard deviations, or frequencies and percentages, as appropriate). Changes in mean scores across the four time points were assessed using linear mixed-effects regression analysis. Repeated measures were accounted for using an unstructured covariance pattern matrix, and clustering at hospital sites was accounted for using a random intercept. Least square means and pairwise comparisons were assessed at the 1% level. All analyses were carried out using SAS v. 9.4.



We used a qualitative descriptive design [22, 23]. This design aligned with our evaluation objectives to provide an accurate and comprehensive summary of participants’ perceptions of the effect of the MOREOB program on health care provider knowledge, organizational culture, and barriers and facilitators to implementing and participating in the MOREOB program [22, 24].


A 14-question semi-structured interview guide (Additional file 2) was used to gather in-depth information about interprofessional team experiences with the MOREOB program. The research team reviewed the initial draft interview guide, with modifications made to increase comprehensiveness and clarity (e.g., adding additional probes to elicit information on more concepts, changing wording to make the questions more easily understood). Minor modifications to the guide were made throughout the qualitative data collection period to reflect the timing of the interview in relation to program implementation and to probe new themes emerging from previously conducted interviews.


We used a purposeful sampling approach to recruit interview participants [20]. Specifically, individuals were eligible to participate in an interview if they were (1) health care providers (i.e., nurse, family physician, obstetrician, midwife) and/or administrators/leaders (i.e., nursing or medicine); and (2) participants in the MOREOB program and/or involved in implementing the program as a member of the core team at one of the 26 sites during the two interview recruitment periods (January–April 2015 and April–July 2016).

Permission from each of the 26 hospitals was obtained for Salus Global to release the names and contact information of the MOREOB core team chairs to the research team for the purpose of interview participant recruitment. Interviews were conducted at two time points (post-Module 2 and post-Module 3) to capture the variety of experiences as the MOREOB program progressed. At each of the two time points, an email with an attached recruitment poster was sent by the research team to the MOREOB core team chair at each of the 26 hospitals, with instructions to circulate the invitation to all team members participating in the MOREOB program at their site. Interested participants were instructed to contact the principal investigator to learn more about the study, ascertain eligibility, and schedule a time for an interview. Reminders were sent by the research team to the MOREOB core team chairs to maximize responses. Individuals could participate in interviews at one or both time points. We welcomed participants who volunteered to be interviewed a second time as an opportunity to learn about differences they experienced between modules.

While in qualitative research there are no standardized rules for sample size, we used the concept of information power for guidance [25]. For example, our study aims were relatively narrow (we aimed to explore key dimensions and implementation of the MOREOB program), our purposive sample of participants were specific to our study aims (all had experience participating in and/or implementing the MOREOB program), and we anticipated rich dialogue with participants due to the use of interviewers experienced in both qualitative research and the clinical practice area. These factors, in combination with our experience from past provincial program evaluations, led us to predict that we would need up to 20 participants to provide sufficient information power to answer our research questions.

Data collection

Two research staff (one midwife, one nurse) experienced in qualitative interviewing conducted the telephone interviews. Informed written consent was obtained prior to each interview. Interviews lasted an average of 34 min. The first round of interviews (n = 9) was conducted between March and April 2015 to learn about participants’ experience with modules 1 and 2 of the MOREOB program. The purpose of conducting the first round of interviews after module 2 was to allow sufficient time for participants to be able to experience and report on any perceived changes to culture and clinical processes in their setting. The second round of interviews (n = 10) was conducted between April and July 2016 to learn about participants’ cumulative experiences after completion of module 3 of the MOREOB program and plans for sustainability. Four health care providers were interviewed during both round one and two, meaning that there were a total of 15 unique interviewees. Interviewing, transcription, and analysis proceeded concurrently to permit follow-up of issues emerging from the data, and allow probing of emerging themes in subsequent interviews [22, 23, 26].

Data analysis

Digital recordings of the semi-structured interviews were transcribed verbatim. Data were imported into NVivo 11™ [27]. In alignment with our qualitative descriptive study design, we used conventional content analysis to code and analyze our data [22, 28, 29]. Initially, interviews were analysed independently by two study team members. We used an inductive process whereby the codes emerged from the data itself, rather than a pre-existing framework. Specifically, the two study team members independently read each transcript, followed by assigning preliminary codes (i.e., labels that represent the content of words and excerpts in the transcripts [30]). The two coders met every three to five transcripts to compare their coding, reach consensus, and develop and revise the coding scheme. All discrepancies were resolved through discussion between the two coders. As analysis progressed, codes were continually reviewed to identify similarities and differences, and conceptually similar codes were grouped into broader categories [30]. Lastly, we looked for common underlying meaning between categories and created broader themes and sub-themes, which contributed to answering “how” participants experienced the MOREOB program [30]. The analysis of data collected in phase 1 was used to inform revisions to the interview guide for phase 2. For example, questions were added to elicit feedback on perceived differences between modules and on plans for sustainability of the program, and additional probes on key emergent themes such as health care provider confidence were included. Research team members met to discuss the coding scheme, emerging themes and sub-themes, and to build consensus regarding study findings.


In total, we analyzed data from 308 respondents for the knowledge test, 329 respondents for the CAS, and 19 semi-structured interviews (15 participants) (Fig. 2).

Fig. 2
figure 2

Participant flow diagram


The number of respondents at each time point is shown in Table 2. Of the 962 respondents who completed the knowledge test at any time point, 308 (32%) completed it at all four time points. Of the 988 respondents who completed the CAS at any time point, 329 (33.3%) completed it at all four time points.

Table 2 Number of respondents at each time point for the knowledge test and Culture Assessment Survey

Knowledge scores increased by a mean of 7.9 percentage points (95% CI: 7.1 to 8.8) from baseline to post-Module 3 and CAS scores increased by a mean of 0.45 (on a scale of 1 to 5, 95% CI: 0.38 to 0.52). Both increases were statistically significant. Least square mean estimates with standard errors across the four time points are presented in Fig. 3, for the knowledge test and CAS scores.

Fig. 3
figure 3

Least square mean estimates and standard errors for Knowledge Tests (a), Culture Assessment Survey (b). *Pairwise statistical significance at the 1% level. **Differences in Knowledge Test are assessed as mean difference on score from 0 to 100, differences in Culture Assessment Survey scores are assessed as mean differences on score from 1 to 5

Increases in knowledge test and CAS scores for each professional group are presented in Table 3 The one respondent in the knowledge test group and two in the CAS group who selected “other” for profession were excluded from this analysis due to low numbers. Increases in both scores were noted for the three professional groups presented, with nurses experiencing the largest increase in knowledge scores, with a mean increase of 9.06 percentage points (95% CI 8.05 to 10.07). Physicians and residents had the greatest increase in CAS scores, with a mean increase of 0.72 on a scale of 1 to 5 (95% CI: 0.57 to 0.88).

Table 3 Increases in knowledge test and Culture Assessment Survey scores by professional group, from baseline to post-Module 3a


Between March 2015 and July 2016, 19 telephone interviews were conducted with a total of 15 health care providers from 11 sites (Table 4).

Table 4 Qualitative interview participant demographics

Overall, participants reported being satisfied with the MOREOB program. Five main themes and their respective sub-themes were identified: communication, knowledge, safer care, increased confidence, and program implementation (Fig. 4). Table 5 provides participant quotes to support our findings [29].

Fig. 4
figure 4

Organization of themes and sub-themes

Table 5 Themes and sub-themes with example quotes

Theme 1: communication

An important focus of the MOREOB program was improving communication strategies. Participants identified specific communication strategies learned through the MOREOB program implemented in practice, including ‘Take Five’ and ‘SBAR’ (Situation, Background, Assessment, Recommendation), which were implemented during debriefing about cases.

Participants perceived increased interprofessional communication, resulting from bringing the different professions together to participate in the program, sometimes for the first time. These opportunities for interprofessional communication and teamwork led to participants perceiving a more familiar and collegial work environment.

Theme 2: knowledge

Participants valued the knowledge itself gained from the MOREOB program. MOREOB was described as a comprehensive, time efficient, and accessible resource as all information is contained in one place, both in-print and online. In addition, participants perceived that the content of MOREOB had been well-researched, was evidence-based, and relevant to the local context.

Participants described how participating in the MOREOB program allowed all team members to share the same knowledge and language, put everyone “on the same page,” and gave everyone a common reference point.

All participants described how MOREOB increased their knowledge base and that of their colleagues. For some, this increase in knowledge was a reminder of the importance of ongoing training to maintain current knowledge and skills. The MOREOB program was described as being particularly helpful in a small hospital setting with low birth volumes where it can be difficult for new obstetrical nurses to gain sufficient experience in obstetrics.

Theme 3: safer care

Participation in MOREOB resulted in the perception that the health care team was providing safer care to patients. Participants identified concrete changes made on their units due to their participation in MOREOB. Examples of concrete changes included development or updating of policies and procedures, creating disaster carts with appropriate equipment, changes to clinical practices, posting algorithms on walls for clinicians to view, conducting case audits, and utilizing debriefing methods.

Participants described how MOREOB changed the way they delivered care. First, participants described improved collaboration with their colleagues, which resulted in better ‘team work, openness, and transparency’, which ultimately improves safety for patients. For those sites who described pre-existing hierarchical relationships between obstetrical care providers, participants described a decrease in hierarchy within their team. However, a couple of sites described still struggling with hierarchy and were still working to change the culture at their sites. In addition, participants identified that participation in the MOREOB program improved the patients’ experience by improving their collaboration with women and emphasizing the importance of patient centred care.

Generally, participants could not speak definitively as to whether their hospital’s participation in the MOREOB program had led to improved outcomes such as decreased maternal and infant morbidity and mortality, as they did not have data to refer to during the interview. One site had these data and stated they observed a reduction in critical events since participating in MOREOB. Overall, participants believed that the increased knowledge and concrete changes were ultimately improving care, outcomes, and patient experience.

Theme 4: increased confidence

Participants described the impact MOREOB had on their self-confidence, particularly nurses who were described as being empowered through the MOREOB program by increasing their knowledge and skills and improving their team communication. In addition, participants described an increase in their confidence in the knowledge, skills, and judgments of their team members. As MOREOB participants improved communication, increased their knowledge base, and improved their ability to provide safer care, their confidence in themselves and others increased.

Theme 5: program implementation

Participants described different strategies used for implementing the MOREOB program, including use of huddle boards, staff emails and staff newsletters. In general, participants described following the MOREOB activities as intended, with sites modifying the activities to make it more relevant for their setting or to fit it into limited time frames as needed. Some participants stated that the MOREOB program plan was not clear, even after attending the orientation, and felt that additional detail on upcoming activities would be beneficial.

Participants described several influencing factors that affected how easy or difficult it was to implement the MOREOB program in their context. For example, buy-in was increased when team members were interested in the content of the module and “hungry for new knowledge.” Participants also identified organizational-level barriers including limited time of staff and lack of funding to pay staff for their time to participate in the program, and a baseline organizational culture that was more resistant to change. Organizational-level facilitators included engaged hospital administration and the presence of MOREOB champions who were highly committed to the implementation of the program.

Participants described varying levels of awareness regarding the sustainability of the MOREOB program at their hospital once their three-year funding ended. Participants were keen to keep MOREOB active at their hospital, and many sites were pursuing continued involvement, either formally with Salus Global, or when funding was not available, through internally-led education programs and skills drills. Regardless of the specific strategy, sustainability of the knowledge and skills gained through MOREOB was identified as a priority.

Staff turnover and new hospital priorities were identified as ongoing challenges to sustainability of the program. As new people joined the unit, they had not participated in the initial MOREOB program modules with the remainder of the team which presented challenges as the team was “storming and forming” all over again. A key challenge for sustainability of the program was cost, with participants perceiving that the program should be kept at a reasonable price to ensure institutions and providers can benefit from it.


Our mixed-methods study evaluated the effect of the MOREOB program on participant knowledge, perception of organizational culture, and experiences implementing and participating in the program. Findings from both the quantitative and qualitative components of this study demonstrated increases in knowledge and positive perceptions of organizational culture. Findings from the qualitative component revealed that participants spoke positively of the program and the benefits of participating. In addition, barriers and facilitators to implementation and sustainability were identified.

Participant knowledge

The increases in knowledge scores occurring for all professional groups resonate with findings from our qualitative analysis, with participants describing MOREOB benefits such as increased knowledge and sharing a common knowledge, as well as increased confidence in themselves and in their teammates as a result of this increased knowledge.

Data collected in a previous study revealed similar results on the knowledge tests in Canadian hospitals after MOREOB implementation, with all professional groups showing an increase in knowledge from baseline to post-Module 3 [18]. As in our evaluation, nurses were the professional group with the largest increase in knowledge test scores from baseline to post-Module 3 [18]. This may be partially explained by the small size of the hospitals included in our sample, with approximately 80% providing level 1 (low-risk) care and approximately 70% having a birth volume of less than 500 per year. At such small sites, many nurses practice in multiple areas (e.g., medical/surgical) and are cross-trained to provide obstetrical care as needed. Conversely, physicians and midwives practicing at these sites specialize in obstetrics and thus may have had a greater depth of baseline knowledge in this area.

Participant perception of organizational culture

Our study showed an overall significant increase in the safety culture from baseline to post-Module 3, with scores increasing almost half a point out of 5 from pre-MOREOB levels. Likewise, in the previous Canadian evaluation, there was a significant increase on all six elements of the CAS [18]. Our qualitative findings were aligned with these results, with participants giving examples of improvements related to the safety culture in their obstetric units, including strengthened teamwork and communication.

Team work, which is one of six categories on the CAS, has been identified as an essential element for reducing maternal and infant morbidity and mortality, as well as improving patient experience in obstetrics [31]. Characteristics of team training to improve obstetrical outcomes include use of in-house rehearsals and debriefing, interprofessional training, and access to and training on evidence-based clinical guidelines [31], all of which are consistent with features of the MOREOB program and may have facilitated the positive culture changes we saw in our evaluation.

From the qualitative data, we learned that despite participation in the MOREOB program, some sites still struggled to overcome issues with hierarchy on the interdisciplinary team, which is important as it can prevent health care providers from speaking up [32]. The ongoing challenges with hierarchy we observed in our interviews may not be surprising, given that hierarchal gradients continue to exist in many health care settings, despite attempts to reduce these hierarchies within and between professions [33]. One industry that has successfully implemented interventions to reduce hierarchy between team members and improve culture and safety is aviation [33]. Specifically, the use of Crew Resource Management (CRM) training helps to flatten the hierarchy, minimize power differentials, and improve communication by training all crewmembers, regardless of role, seniority, or experience, to speak up with safety concerns [33, 34]. Parallels have been drawn between aviation and healthcare related to safety management [35, 36] and the application of CRM training in health care was recommended in the 1999 To Err is Human Report [5]. Examples of CRM in the field of obstetrics specifically can be found in the literature [7, 37,38,39,40]. While the MOREOB program uses HRO principles [12] applied in the aviation industry, additional strategies and ongoing training opportunities beyond the 3-year MOREOB program may be required for sites to continue to decrease interprofessional hierarchy and improve their ability to work and communicate as a team.

The varying experiences in culture change found in our qualitative interviews speak to the importance of the context in which the MOREOB program is being implemented. The influence of site context on culture change has been observed in other healthcare settings, with a longitudinal mixed-methods study of 10 hospitals participating in a quality collaborative intervention reporting distinct differences between the hospitals with substantial culture change versus those without. Hospitals with positive culture change employed an interprofessional and inter-hierarchal approach, reported high levels of participation, and utilized strategies for managing conflict and maintaining engagement [41]. Although the MOREOB program aims to improve the safety culture in birthing units, the extent to which this can be improved may be influenced by specific hospital and team characteristics.

While our quantitative and qualitative findings support the effectiveness of the MOREOB program in improving several dimensions of a culture of safety, it is important to acknowledge that other patient safety initiatives in obstetrics, such as CRM team training, the Comprehensive Unit-based Safety Program (CUSP), and other locally developed quality improvement initiatives, have also demonstrated improvements in health care provider safety attitudes [37, 38, 42, 43]. Examples of key elements of these other initiatives included implementing an obstetric patient safety nurse [37, 42] and an obstetrics patient safety committee [37, 42, 43], using interdisciplinary team training [37, 38, 42, 43], and implementing improved communication systems and strategies [37, 38, 42, 43]. All reported improvements in safety culture after implementation of their respective patient safety initiatives, with significant improvements in teamwork culture [37, 42], safety culture [37, 42], job satisfaction [42, 43], working conditions [43], and perceptions of management [37, 42, 43].

Lastly, there is evidence to suggest that positive workplace and organizational culture is associated with improved patient outcomes including decreased mortality rates, adverse events, readmission rates, and increased patient satisfaction across a variety of health care settings [44]. While our study found a significant increase in the CAS which was largely supported by our qualitative findings, it remains unknown how this positive change in culture translates to clinical outcomes in our setting.

Barriers and facilitators to implementing and participating in MOREOB program

Overall, interview participants expressed satisfaction with their participation in the MOREOB program. Several individual- and organizational-level facilitators and barriers to implementation and maintenance of the MOREOB program were identified, some of which are consistent with factors reported in the literature, and align with existing theoretical frameworks [45]. For example, participants indicated that social influences [45] such as the effect of the core team and champions, influenced their team’s success implementing the MOREOB program. This was similar to lessons learned in another Canadian evaluation, where the hospitals’ MOREOB core team was identified as essential to driving the participation of members [18]. Second, our participants described the effect of the environmental context and resources [45] such as the availability of funding to support the MOREOB program and baseline organizational culture. With limited financial, physical, and human resources within health care organizations, running and sustaining the MOREOB program can present a challenge.

Previous research suggests potential strategies to facilitate implementation and uptake of the MOREOB program include increased leadership engagement, conducting a baseline barriers assessment, and use of ongoing monitoring and evaluation at a local level. First, leadership has been identified as critical for developing a culture of safety [46], and in the case of MOREOB, the support of senior administration is key to successful implementation of the program, through direct participation on the core team, and through supporting their work [18]. Next, an assessment prior to MOREOB program implementation may assist hospitals to identify individual- and organization-level barriers, and identify existing strengths and resources that may be used to facilitate implementation [47]. Lastly, participants in our study were largely unable to definitively state whether their team’s participation in the MOREOB program had translated to improved patient outcomes at their hospital. Evaluating key outcomes using local data is an important component of the knowledge translation process [47] to improve patient safety. Ongoing monitoring and evaluation of hospital-based patient safety outcomes may assist with keeping teams engaged with the MOREOB program by showing participants where they are (or are not) making a difference, and identifying areas of improvement for the MOREOB core team related to program implementation and participation.


While our evaluation showed positive findings from the participant surveys and qualitative interviews, understanding the actual impact on perinatal outcomes is important, because a patient safety program should really translate into improved outcomes. As part of the larger evaluation we reported that these findings did not translate into a statistically significant reduction in adverse maternal and neonatal outcomes at the provincial level [17]. Despite the lack of reduction in adverse outcomes, it seems clear from the results we present in this paper that the MOREOB program increased knowledge, changed the patient safety culture, and improved the work environment at many participating hospitals. For example, participants in the interview component of our study identified other improvements that were not within the scope of this current study to assess quantitatively. Specifically, interview participants spoke of perceived improvements from the MOREOB program related to patient experience, health care provider confidence, and interprofessional communication. These additional benefits are important to individuals and health systems because: (1) improved patient experience is an important goal for health care systems [48], and (2) health care provider confidence and communication are important individual-level factors influencing “speaking up behavior” [49], an essential component of a patient safety culture [50]. Given the potential importance of these additional health care provider and patient outcomes, future work to comprehensively assess the effect of the MOREOB program on these constructs using validated measures may be warranted.

Considering the overall positive results from the surveys and interviews, it is not surprising that most participants expressed a desire to keep the program active at their centers. Future work to create a program model that is affordable and sustainable for participating hospitals is needed. Since the completion of this evaluation, Salus Global has launched MOREOB 2.0 and MOREEX, which can support health care teams in acute settings beyond obstetrics [51]. Future evaluations of the revised MOREOB 2.0 program may consider our lessons learned from the Alberta [15] and Ontario provincial evaluations as well as the Canadian evaluation [18], and eventually provide comparative data.

Strengths and limitations

The use of a mixed-methods approach is an important strength of this study. We used data from two surveys, carried out at four time points as participants progressed through the MOREOB program. As well, we included data from qualitative interviews carried out with 15 participants, including an administrator, nurses, midwives, and physicians, providing rich in-depth information on the MOREOB program. The fact that findings from both methods were convergent regarding improvements in knowledge and safety culture supports the effect of the program on these outcomes.

To our knowledge, ours is only the second paper to publish results of the knowledge test and CAS at baseline and after implementation of the three MOREOB modules. Our findings are similar to those published previously, and are an important addition to the literature due to the limited methodologic and sampling information provided in the earlier publication [18]. Further, we have used a more robust analytic approach for analyzing repeated measures survey data.

A limitation is the inclusion of only those 26 hospitals who received provincial funding to implement the MOREOB program in 2013. These sites likely differ from the 67 hospitals that voluntarily initiated participation in the MOREOB program prior to 2013, independently of any funding. As well, for the knowledge test and CAS, all of the respondents were participants in the MOREOB program, meaning that we do not have data on non-participants. This is problematic, in that one would expect knowledge scores to increase over a four-year time period, with increased time in the workplace. Due to this limitation, we cannot attribute the increases we observed solely to participation in the MOREOB program. Additionally, the majority of respondents did not complete the CAS and knowledge test at all four time points. This is not surprising given staff turnover rates on obstetric units, however, our analyses were therefore limited to the one third of respondents who completed all of the assessments. Furthermore, it is difficult to know the clinical effect of the increases in knowledge and culture that were observed. Another limitation is the lack of data available on survey respondents, for example, we do not know their age or number of years in the workforce, which could influence survey scores. While our qualitative interviews included participants from diverse hospitals and professional backgrounds, only one obstetrician participated in an interview.

Lastly, it is important to acknowledge that our sample of 15 unique interviewees only included individuals from 11 out of the 26 potential hospitals. While efforts to maximize recruitment were made by communicating through the chair of the MOREOB core team at each site, by sending multiple email reminders, and by accommodating all participant schedules for interviews, we still only managed to capture participants from about 40% of eligible hospitals. Nonetheless, the information we obtained from these interviews provided valuable insight into the MOREOB program and its implementation to answer our research questions, although these findings may not be transferable to individuals and teams at the remaining sites.


Results of this mixed-methods study suggest that participants were satisfied with their participation in the MOREOB program. Barriers and facilitators to implementation and sustainability of the MOREOB program were identified. Strategies to address barriers, maximize facilitators, and enhance sustainability of the MOREOB program are needed. Our results also suggest that the program may have a positive effect on participant knowledge and organizational safety culture. Further work to understand how these improvements translate to outcomes for patients and health care providers is needed.

Availability of data and materials

The qualitative data analyzed during this study (i.e., interview transcripts) are not publicly available due to them containing information that could compromise research participant privacy/consent.

The data that support the findings from the survey component of this study are available from Salus Global but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. Data are only available with permission from Salus Global and the participating hospitals.



The American College of Obstetricians and Gynecologists


Advances in Labour and Risk Management


Culture Assessment Survey


Crew Resource Management


Healthcare Insurance Reciprocal of Canada


High Reliability Organization


Ministry of Health and Long-Term Care


Managing Obstetrical Risk Efficiently


Situation, Background, Assessment, Recommendation


Society of Obstetricians and Gynaecologists of Canada


  1. Statistics Canada. Births, 2015 and 2016. Ottawa; 2018. Accessed 12 June 2019.

  2. Canadian Institute of Health Information (CIHI). Giving birth in Canada: a regional profile. Ottawa; 2004. Accessed 8 Apr 2019

  3. Public Health Agency of Canada. Perinatal health indicators for Canada 2017. Ottawa; 2017. Accessed 8 Apr 2019.

  4. Liu S, Joseph KS, Bartholomew S, Fahey J, Lee L, Allen AC, et al. Temporal trends and regional variations in severe maternal morbidity in Canada, 2003 to 2007. J Obs Gynaecol Can. 2010;32:847–55.

    Article  CAS  Google Scholar 

  5. Kohn LT, Corrigan JM, Donaldson MS. To err is human: building a safer health system. Washington, DC: National Academy Press; 1999.

    Google Scholar 

  6. Canadian Institute for Health Information (CIHI) and Canadian Patient Safety Institute (CPSI). Measuring patient harm in Canadian hospitals. Ottawa, 2016. Accessed 8 Apr 2019.

  7. Nielsen PE, Goldman MB, Mann S, Shapiro DE, Marcus RG, Pratt SD, et al. Effects of teamwork training on adverse outcomes and process of care in labor and delivery: a randomized controlled trial. Obstet Gynecol. 2007;109:48–55.

    Article  Google Scholar 

  8. Pettker CM, Thung SF, Norwitz ER, Buhimschi CS, Raab CA, Copel JA, et al. Impact of a comprehensive patient safety strategy on obstetric adverse events. Am J Obstet Gynecol. 2009;200:492.e1–8.

    Article  Google Scholar 

  9. Pettker CM, Thung SF, Lipkind HS, Illuzzi JL, Buhimschi CS, Raab CA, et al. A comprehensive obstetric patient safety program reduces liability claims and payments. Am J Obstet Gynecol. 2014;211:319–25.

    Article  Google Scholar 

  10. The Healthcare Insurance Reciprocal of Canada (HIROC) and the Canadian Medical Protective Association (CMPA). Delivery in focus. Strengthening obstetrical care in Canada: Ottawa; 2018. Accessed 8 Apr 2019

  11. Salus Global Corporation. History. 2019. Accessed 6 May 2019.

    Google Scholar 

  12. Salus Global Corporation. MOREOB. 2018. Accessed 21 Sep 2018.

    Google Scholar 

  13. Hines S, Luna K, Lofthus J, Lewin group. Becoming a high reliability organization: operational advice for hospital leaders. Rockville, MD; 2008. Accessed 8 April 2019.

    Google Scholar 

  14. Salus Global Corporation. The MOREOB program overview 2012–2013. London; 2013.

  15. Thanh NX, Jacobs P, Wanke MI, Hense A, Sauve R. Outcomes of the introduction of the MOREOB continuing education program in Alberta. J Obstet Gynaecol Can. 2010;32:749–55.

    Article  Google Scholar 

  16. Geary M, Ruiter PJA, Yasseen AS. Examining the effects of an obstetrics interprofessional programme on reductions to reportable events and their related costs. J Interprof Care. 2018;1(9). epub ahead of print.

  17. Weiss D, Fell DB, Sprague AE, Walker MC, Dunn S, Reszel J, et al. Effect of implementation of the MOREOB program on adverse maternal and neonatal birth outcomes in Ontario, Canada: a retrospective cohort study. BMC Pregnancy Childbirth. 2019;19:151.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Milne J, Walker D, Vlahaki D, Milne J, Walker D, Vlahaki D. Reflections on the Canadian MORE (OB) obstetrical risk management programme. Best Pr Res Clin Obs Gynaecol. 2013;27:563–9.

    Article  CAS  Google Scholar 

  19. Fetters MD, Curry LA, Creswell JW. Achieving integration in mixed methods designs-principles and practices. Health Serv Res. 2013;48:2134–56.

    Article  Google Scholar 

  20. Creswell JW. Research design - qualitative, quantitative, and mixed methods approaches. 4th ed. Thousand Oaks, California: SAGE Publications, Inc; 2014.

    Google Scholar 

  21. Milne JK, Bendaly N, Bendaly L, Worsley J, Fitzgerald J, Nisker J. A measurement tool to assess culture change regarding patient safety in hospital obstetrical units. J Obstet Gynaecol Can. 2010;32:590–7.

    Article  Google Scholar 

  22. Sandelowski M. Whatever happened to qualitative description? Res Nurs Health. 2000;23:334–40.

    Article  CAS  Google Scholar 

  23. Sandelowski M. What’s in a name? Qualitative description revisited. Res Nurs Health. 2010;33:77–84.

    PubMed  Google Scholar 

  24. Kim H, Sefcik JS, Bradway C. Characteristics of qualitative descriptive studies: a systematic review. Res Nurs Health. 2017;40:23–42.

    Article  CAS  Google Scholar 

  25. Malterud K, Siersma VD, Guassora AD. Sample size in qualitative interview studies. Qual Health Res. 2016;26:1753–60.

    Article  Google Scholar 

  26. Thorne S. Interpretive description: qualitative research for applied practice. 2nd ed. New York: Routledge Taylor & Francis Group; 2016.

    Book  Google Scholar 

  27. QSR International Pty Ltd. NVivo 11 for Windows. 2015.

  28. Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15:1277–88.

    Article  Google Scholar 

  29. Bradshaw C, Atkinson S, Doody O. Employing a qualitative description approach in health care research. Glob Qual Nurs Res. 2017;4:1–8.

    Google Scholar 

  30. Graneheim U, Lundman B. Qualitative content analysis in nursing research: concepts, procedures and measures to achieve trustworthiness. Nurse Educ Today. 2004;24:105–12.

    Article  CAS  Google Scholar 

  31. Cornthwaite K, Edwards S, Siassakos D. Reducing risk in maternity by optimising teamwork and leadership: an evidence-based approach to save mothers and babies. Best Pract Res Clin Obstet Gynaecol. 2013;27:571–81.

    Article  Google Scholar 

  32. Guise J-M, Segel S. Teamwork in obstetric critical care. Best Pract Res Clin Obstet Gynaecol. 2008;22:937–51.

    Article  Google Scholar 

  33. Green B, Oeppen RS, Smith DW, Brennan PA. Challenging hierarchy in healthcare teams – ways to flatten gradients to improve teamwork and patient care. Br J Oral Maxillofac Surg. 2017;55:449–53.

    Article  CAS  Google Scholar 

  34. Government of Canada. Chapter 7: Crew Resource Management (CRM). In: Development and implementation of an Advanced Qualification Program (AQP). Ottawa; 2010. Accessed 8 April 2019.

  35. Kapur N, Parand A, Soukup T, Reader T, Sevdalis N. Aviation and healthcare: a comparative review with implications for patient safety. JRSM Open. 2015;7:1–10.

    Google Scholar 

  36. Skinner L, Tripp TR, Scouler D, Pechacek JM. Partnerships with aviation: promoting a culture of safety in health care. Creat Nurs. 2015;21:179–85.

    Article  Google Scholar 

  37. Raab CA, Will SEB, Richards SL, O’Mara E. The effect of collaboration on obstetric patient safety in three academic facilities. J Obstet Gynecol Neonatal Nurs. 2013;42:606–16.

    Article  Google Scholar 

  38. Pratt SD, Mann S, Salisbury M, Greenberg P, Marcus R, Stabile B, et al. Impact of CRM–based team training on obstetric outcomes and clinicians’ patient safety attitudes. Jt Comm J Qual Patient Saf. 2007;33:720–5.

    Article  Google Scholar 

  39. Haller G, Morales M, Pfister R, Garnerin P, Chipp P, Guillemot V, et al. Improving interprofessional teamwork in obstetrics: a crew resource management based training programme. J Interprof Care. 2008;22:545–8.

    Article  Google Scholar 

  40. Mancuso MP, Dziadkowiec O, Kleiner C, Halverson-Carpenter K, Link T, Barry J. Crew resource management for obstetric and neonatal teams to improve communication during cesarean births. J Obstet Gynecol Neonatal Nurs. 2016;45:502–14.

    Article  Google Scholar 

  41. Bradley EH, Brewster AL, McNatt Z, Linnander EL, Cherlin E, Fosburgh H, et al. How guiding coalitions promote positive culture change in hospitals: a longitudinal mixed methods interventional study. BMJ Qual Saf. 2018;27:218–25.

    Article  Google Scholar 

  42. Pettker CM, Thung SF, Raab CA, Donohue KP, Copel JA, Lockwood CJ, et al. A comprehensive obstetrics patient safety program improves safety climate and culture. Am J Obstet Gynecol. 2011;204:216.e1–6.

    Article  Google Scholar 

  43. Simpson KR, Knox GE, Martin M, George C, Watson SR. Michigan health & hospital association keystone obstetrics: a statewide collaborative for perinatal patient safety in Michigan. Jt Comm J Qual Patient Saf. 2011;37:544–52.

    Article  Google Scholar 

  44. Braithwaite J, Herkes J, Ludlow K, Testa L, Lamprell G. Association between organisational and workplace cultures, and patient outcomes: systematic review. BMJ Open. 2017;7:e017708.

    Article  Google Scholar 

  45. Cane J, O’Connor D, Michie S. Validation of the theoretical domains framework for use in behaviour change and implementation research. Implement Sci. 2012;7:37.

    Article  Google Scholar 

  46. Sammer CE, Lykens K, Singh KP, Mains DA, Lackan NA. What is patient safety culture? A review of the literature. J Nurs Scholarsh. 2010;42:156–65.

    Article  Google Scholar 

  47. Graham I, Logan J, Harrison M, Straus S, Tetroe J, Caswell W, et al. Lost in knowledge translation: time for a map? J Contin Educ Heal Prof. 2006;26:13–24.

    Article  Google Scholar 

  48. Health Quality Ontario. Ontario patient experience measurement strategy: Toronto; 2016. Accessed 8 Apr 2019

  49. Okuyama A, Wagner C, Bijnen B. Speaking up for patient safety by hospital-based health care professionals: a literature review. BMC Health Serv Res. 2014;14:61.

    Article  Google Scholar 

  50. Canadian Patient Safety Institute. Patient safety culture. 2016. Accessed 8 April 2019.

    Google Scholar 

  51. Salus Global Corporation. New look, more content. 2018. Accessed 8 Apr 2019.

    Google Scholar 

Download references


The research group would like to acknowledge the contributions of the hospital teams and health care providers who volunteered to share their experiences with the implementation of the MOREOB program.


This study was funded by the Ontario Ministry of Health and Long-Term Care. The funding body had no role in the design of the study, nor did they have a role in data collection, analysis, interpretation of data, or publications.

Author information

Authors and Affiliations



AES and DBF are co-principal investigators for the described study. AES, DBF, SD, MCW, MT, and WEP contributed to the conceptualization and the design of the study. JR and DS collected all interview data; JR and WEP coded and analyzed all interview data; DW, DBF, and MT analyzed the survey data; JR and DW drafted the initial manuscript. All authors participated in the writing of this manuscript and reviewed and approved the final manuscript.

Authors’ information

JR is a Research Coordinator at the Better Outcomes Registry & Network (BORN) Ontario and the Children’s Hospital of Eastern Ontario (CHEO) Research Institute; DW is an Epidemiologist at BORN Ontario; AES is the past Scientific Manager and a current Project Advisor at BORN Ontario; DBF is an Assistant Professor in the School of Epidemiology and Public Health at the University of Ottawa and a Scientist at the CHEO Research Institute; SD is a Clinical Investigator at the CHEO Research Institute and the Knowledge Translation Specialist and Research Team Lead at BORN Ontario; MCW is a Maternal-Fetal Medicine Specialist and the Chief of Obstetrics at The Ottawa Hospital, an Epidemiologist and Senior Scientist at the Ottawa Hospital Research Institute (OHRI), a co-founder of the OMNI Research Group, and the Scientific Director for BORN Ontario; DS is the former Midwifery Coordinator at BORN Ontario and is currently a Family Nurse Practitioner working with a primarily refugee population in Ottawa, Canada; MT is a Biostatistician and Senior Scientist in the Clinical Epidemiology Program, OHRI, and Associate Professor in the School of Epidemiology and Public Health at the University of Ottawa; WEP is an Associate Professor in the School of Nursing at the University of Ottawa.

Corresponding author

Correspondence to Jessica Reszel.

Ethics declarations

Ethics approval and consent to participate

Ethical approval for this study was obtained from the Children’s Hospital of Eastern Ontario Research Ethics Board (13/135X). All individuals who participated in an interview signed an informed consent form prior to their participation. Written consent was obtained from administration at all participating hospitals for their de-identified MOREOB survey data to be released from Salus Global Corporation to BORN Ontario.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional files

Additional file 1:

MOREOB program goals. This file provides the seven main goals of the MOREOB program we evaluated in this study. (DOCX 15 kb)

Additional file 2:

Semi-structured interview guide. This file provides the 14 questions that were used during the semi-structured interviews. (DOCX 14 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Reszel, J., Weiss, D., Sprague, A.E. et al. A mixed-methods evaluation of the MOREOB program in Ontario hospitals: participant knowledge, organizational culture, and experiences. BMC Health Serv Res 19, 460 (2019).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: