Skip to main content

Implementation, spread and impact of the Patient Oriented Discharge Summary (PODS) across Ontario hospitals: a mixed methods evaluation

Abstract

Background

Traditional discharge processes lack a patient-centred focus. This project studied the implementation and effectiveness of an individualized discharge tool across Ontario hospitals. The Patient Oriented Discharge Summary (PODS) is an individualized discharge tool with guidelines that was co-designed with patients and families to enable a patient-centred process.

Methods

Twenty one acute-care and rehabilitation hospitals in Ontario, Canada engaged in a community of practice and worked over a period of 18 months to implement PODS. An effectiveness-implementation hybrid design using a triangulation approach was used with hospital-collected data, patient and provider surveys, and interviews of project teams. Key outcomes included: penetration and fidelity of the intervention, change in patient-centred processes, patient and provider satisfaction and experience, and healthcare utilization. Statistical methods included linear mixed effects models and generalized estimating equations.

Results

Of 65,221 discharges across hospitals, 41,884 patients (64%) received a PODS. There was variation in reach and implementation pattern between sites, though none of the between site covariates was significantly associated with implementation success. Both high participation in the community of practice and high fidelity were associated with higher penetration. PODS improved family involvement during discharge teaching (7% increase, p = 0.026), use of teach-back (11% increase, p < 0.001) and discussion of help needed (6% increase, p = 0.041). Although unscheduled healthcare utilization decreased with PODS implementation, it was not statistically significant.

Conclusions

This project highlighted the system-wide adaptability and ease of implementing PODS across multiple patient groups and hospital settings. PODS demonstrated an improvement in patient-centred discharge processes linked to quality standards and health outcomes. A community of practice and high quality content may be needed for successful implementation.

Peer Review reports

Background

Traditional discharge summaries are laden with clinical information meant for the primary care provider. Instructions for patients are often not patient-centered or not included [1]. Moreover, while approximately 80–90% of hospitals in Ontario have a discharge planning process, previous reports suggest it is not standardized throughout the province, where involvement of the patient in this process and education provided is not consistent [2]. Patient-centered discharge tools, unlike traditional discharge summaries, engage the patient and family during the hospital admission with an individualized discharge plan and are associated with improved patient outcomes [3,4,5,6]. While improving care transitions is a focus of health systems globally, no Canadian tools exist to our knowledge. Very few tools from other countries have been co-designed with patients and families to address barriers to understanding and use of their discharge instructions [7, 8].

The Patient Oriented Discharge Summary (PODS) is an individualized discharge tool designed with patient and family engagement that contains five sections of information that are actionable and useful for patients and families; 1) medications, 2) changes to daily activities and diet, 3) follow-up appointments, 4) resources for patients and families, and 5) expected and worrisome symptoms to watch out for after leaving hospital (Fig. 1) [1]. PODS utilises design features such as plain language, large fonts, pictograms, and white space for patients to take notes that enhance retention and understanding of information [9]. PODS includes accompanying process guidelines which highlight key patient-centered processes proven to increase adherence to instructions [10, 11]. PODS is designed to be modular. Every organization that implements it is tasked with adapting the tool and guidelines to meet the needs of their patients and internal processes for discharge using a quality improvement process.

Fig. 1
figure1

Image of the basic PODS template. Content and design features are identified and associated local practice change guidelines are listed

Other complex discharge interventions with some overlapping components to those in the PODS such as Project RED in the United States [12, 13] and the TRANSITION tool in Australia [14] have been successfully implemented and show promising improvements in patient experience, patient understanding, and even hospitalizations. However, studies evaluating the system-wide implementation of these complex interventions are lacking. This is especially true of evaluations of these interventions at larger scales inclusive of multiple diverse organizations, making it difficult to assess whether they are adaptable and transferable to other settings, described as phase IV by Campbell et al. [15] in their framework for evaluation of complex interventions in healthcare. While a pilot study in Toronto by our group demonstrated that PODS was both usable and feasible and suggested improvements in patient understanding of discharge instructions and patient experience, implementation and evaluation were limited to short-term pilot analysis of patient understanding and satisfaction [16]. The present study, of a large provincial implementation enables the study of a combination of different measures at the system level.

Aims

In the present work, we aimed to understand 1) implementation of PODS in different settings and factors influencing increased penetration and 2) the effectiveness of PODS on patient-centred discharge processes, patient and provider experience, and 30-day emergency department (ED) visits and readmissions.

Methods

Setting and context

Ontario is Canada’s largest province with a population of more than 14 million people and 123 acute care and 58 rehabilitation hospitals, spread across 14 Local Health Integration Networks (LHINs), designated geographical areas. The hospitals are commonly grouped as being rehabilitation or acute care and as being academic, community, or small hospitals [17, 18]. This project was the result of an Accelerated Research to Improve Care (ARTIC) grant, government funding for province-wide quality improvement (QI) implementation of evidence based initiatives [19]. Following the consolidated framework for implementation research (CFIR), [20] factors known to facilitate successful implementation were built into both the processes of identifying participating hospitals and supporting the implementation. As part of the project application process, hospitals were selected based on the presence of an executive sponsor and a clear rationale for implementing PODS. Hospitals were chosen from all 14 LHINs and from within all hospital categories to be representative of the province as a whole.

Of 42 hospitals that applied, 21 were invited to participate in a community of practice (COP) and received a stipend to support their implementation and data collection. The hospitals involved all had a high level of organizational commitment and readiness as defined by the identification of an executive sponsor, clear rationale for implementing PODS in line with strategic goals, previous quality improvement experience, and sufficient resource capacity for project management. However, the hospitals ranged widely in size, geographic area, target patient population, discharge process (i.e. what members of the healthcare team were involved in providing patient education at discharge), whether PODS was implemented in isolation or as part of broader discharge process improvements, and whether the process was supported through the electronic medical record (EMR). There were eighteen acute care and three rehabilitation hospitals, eight which were considered academic hospitals, five large community hospitals and eight small community hospitals with under 100 beds (Table 1).

Table 1 Description of hospital and implementation characteristics of hospitals implementing PODS (n = 21 hospitals)

Our team supported the implementation of PODS at the 21 hospitals in three stages: (1) Start Up: hospitals were guided through the process of adapting the PODS tool and process together with stakeholders; (2) Plan Do Study Act (PDSA): hospitals “went live” with either a pilot group or their full target group and then iteratively tested, refined, and evaluated the tool and process while implementing; and (3) Scale Up: hospitals spread the use of the tool and mentored others who were interested. Using a community of practice model, [21] where all the project teams came together for collective learning and sharing while implementing PODS, our team hosted regular meetings for education, shared knowledge and mentorship. A website was developed to house central resources and collective knowledge [22]. Project teams were encouraged to engage with patients and families throughout the project. A project advisory group was formed to guide the project with representation from the hospital, community, and patients including authors (SHG, TH, AC, HA, CB, KO). The group provided advice on project implementation, evaluation, and interpretation of results.

The project started in April 2017, with hospitals beginning implementation of PODS between April 2017 and March 2018. Majority of hospitals “went live” in October and November of 2017. Evaluation data was collected between April 2017 and December 2018 and spanned all three stages of start up, when hospitals had yet to discharge any patients with the PODS, and PDSA and scale up, when hospitals expanded the number of patients receiving the intervention. Implementation generally took between 3 and 6 months and required the support of a project lead with some amount of dedicated time. Once the PODS is implemented, although quality improvement techniques are recommended to refine and evaluate the intervention, no additional resources were required.

Study design

Due to variation in adaptation and implementation across organizations, we chose to use an effectiveness-implementation hybrid design that uses both quantitative and qualitative methods for evaluating complex interventions [23]. Specifically, we employed mixed methods to achieve our first aim of understanding implementation while also addressing our second aim, of assessing the effectiveness of PODS. According to the categories of outcomes outlined by Proctor et al. [24], implementation was measured using implementation outcomes of acceptability, penetration, and fidelity and effectiveness was measured by assessing whether there were improvements in both service (process) and client (patient, family, and provider) outcomes (see Table 2). Service outcomes included measures of efficiency and patient centeredness and client outcomes included measures of satisfaction, function, and symptomatology.

Table 2 Characteristics of outcome measures categorized within the framework presented by Proctor et al. [23] of implementation, service, and client outcomes. Information includes description of measures and datasets and variables used for each category

Data collection

Data was collected via four data sets: (1) Hospital submissions of quarterly data describing their target population and intervention implementation; (2) pre-post surveys (Additional File 1) with providers involved in the discharge process; (3) pre-post surveys (Additional File 1) with a random sample of patients and families at each hospital (a different sample was collected after discharge for pre- and post-implementation of the PODS tool); and (4) end-of project interviews with project leads from each hospital.

The surveys were created by the project team with specific questions related to the intervention and outcomes of interest. Whenever possible, questions were taken from the Canadian Institute for Health Information Patient Experience Survey for Inpatient Care, a validated survey used after discharge at many Canadian hospitals [25,26,27]. In total, three questions were taken from that survey, one of which is also questions used in the US HCAHPS survey by the Centers for Medicare and Medicaid Services and the Agency for Healthcare Research and Quality, which are validated and used extensively in the United States. The questions used have a Cronbach alpha above the cut-off of 0.65 (unpublished data, Canadian Institute for health Information and L Russell, 2018).

Qualitative data was collected throughout the project through multiple sources which included: surveys, presentations at community of practice meetings and open-ended responses in both provider and patient and family surveys. The themes that emerged through the data were used to create a semi-structured interview guide developed specifically for this project (see Additional File 1) that was then used to conduct end-of-project interviews over the phone with project teams from each of the 21 hospitals. Interviews were conducted by authors SHG (PhD) and CM, who were coordinating the CoP and are trained in qualitative research methods. The purpose of collecting the qualitative data was known to the project teams and was collected to inform the implementation outcomes related to each hospital’s implementation process and barriers and facilitators impacting the implementation guided by the CFIR framework [20]. Interviews were between 30 and 60 min. They were recorded and transcribed for analysis.

Implementation outcomes

Implementation data was collected through quarterly submissions of penetration within the target population. Context and explanations of the experience of implementation through the community of practice model was collected during end-of-project interviews. Additionally, during the interviews, project teams were asked to rate the quality of their implementation using measures of consistency, completeness, quality of the content of their PODS, and quality of their process of providing PODS to patients in order to measure fidelity.

Factors impacting implementation were collected through demographic data on the target populations reported quarterly by hospitals and through information about the organization and their discharge process collected during end-of-project interviews with each project team (see Additional File 1 for the interview guide). Factors were chosen based on whether they were thought of as important by the project advisory team, whether they were present in the literature on implementation success, or whether they were found as a theme that emerged through qualitative data collected throughout the project.

Service (process) outcomes

Provider survey measures included the use of teach-back methods and involvement of family during discharge education, in order to see if using PODS effected important and evidence-based patient-centred processes. Additionally, providers reported whether the intervention added to workload as a measure of efficiency. Patient and family surveys included a question on whether patients had a discussion before discharge about the help they would need at home, another measure of patient-centeredness. Several questions within the surveys had open comment fields so that participants could add context and explanations to their responses.

Client (patient, family, and provider level) outcomes

Providers and patients were both surveyed on whether the intervention added value to the discharge experience. Patient and family survey measures also included a rating of understanding of medications, and a rating of understanding of what to do if they are worried about their condition, t. These specific questions, along with the service question about the discussion about help needed at home were chosen because they are relevant questions related to transitions in care selected by the Canadian Institute for Health Information Patient Experience Survey for Inpatient Care, and endorsed by provincial, Canadian and international quality agencies, based on their use in the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey to measure quality of care transitions. Quarterly submissions from hospitals included data on all-cause 30-day return ED visits and readmissions for the target population.

Analysis

Penetration was evaluated by measuring the proportion of patients in the target population who received a PODS in each quarter relative to implementation. To this end, a generalized linear mixed effects model was used, with fixed effects for quarter relative to implementation and between site covariates, and random slopes and intercepts for each site to model within site temporal trends. The mean age, percent male, and percent of the population with language barriers were included in the model as fixed effects.

Exploratory analysis using a non-parametric Wilcox Rank Sum test was conducted to examine any association between patient, hospital, implementation process, fidelity, and provider-reported outcome factors aggregated to the hospital level with whether a hospital had high penetration within the target population (defined as 75% or more of target patients receiving a PODS by Q4 after implementation). Factors were chosen as supported by literature and the qualitative findings.

Numerical summaries of provider, patient, and family reported data on patient-centred processes, patient understanding, and discharge experience were assessed by comparing means before and after implementation using generalized estimating equations (GEE) to account for within site clustering of responses [28, 29]. Change in healthcare utilization was analysed by plotting change over time. Linear mixed effects regression was used to assess percentage of 30-day return ED visits and readmissions, with percent PODS, within site, as the predictor of interest.

The analyses and data visualizations were performed using R version 3.6.2 by John Matelski and supervised by George Tomlinson (Biostatistics Research Unit, UHN).

Qualitative data was analysed using an iterative constant comparative process involving descriptive and interpretive analyses, open-coding, and identifying themes in the data. Study leads and members of the research team (SHG, TH, CM), read transcripts, then met to discuss initial codes and develop a preliminary coding framework. Interpretation of themes was discussed among team members to achieve consensus. Theoretical saturation, constant comparative analysis, trustworthiness, and validity checks provided assurance of data quality and rigor.

Using a triangulation approach, qualitative and quantitative results were interpreted together, towards the understanding of factors that may influence successful implementation and to draw inferences from the data.

This study was approved by the University Health Network Research Ethics Board.

Results

Implementation outcomes

The mean age of the target population across all hospitals was 66.4 ± 13.5 years, 48.7% ± 7.8% were male, and 7.8% ± 8.8% had a language barrier. Rehabilitation hospitals had the lowest (average of 80) target population discharges per quarter and academic hospitals had the largest number of discharges with an average of 813 per quarter.

All hospitals felt successful in their implementation. The most common barriers to implementation cited were competing priorities within the hospital, limited resources, and changes in project staff. Hospitals felt supported in working to overcome these barriers through the community of practice and by working together with their executive sponsor.

Between April 2017 and December 2018, there were 65,221 discharges within the target populations and 41,884 patients received a PODS (64.2%). Penetration as a percentage of target discharges increased over time, reaching 78% of the target population by the end of the study. The odds of penetration in each quarter relative to implementation also increased according to the generalized linear mixed effects model, from 2.90 in the second quarter to 24.29 in the fifth quarter. In addition, 12 sites had unplanned spread with a total of 25,190 patients outside the target groups receiving a PODS.

Penetration patterns varied widely from site to site (Fig. 2) with three common patterns: (1) Seven hospitals reached nearly 100% of the target population by the second quarter after implementation with majority reaching high penetration almost immediately following implementation, (2) seven hospitals slowly increased coverage within the target population, and (3) seven hospitals had a dip in implementation as their original target group was increased to a larger group.

Fig. 2
figure2

Individual hospital penetration via a generalized linear mixed effects model presented as percent of target by quarter after implementation (i.e. Q1 is the first quarter after implementation). Q0 refers to hospitals who began implementation in the middle of a quarter (i.e. a portion of the quarter includes data from before implementation began)

Hospitals varied in their implementation patterns for several reasons. Findings showed that hospitals identified that 100% reach early was either due to size, as it was easier for smaller hospitals to implement a change in practice across all patient groups simultaneously, or due to implementation within an EMR, as some EMRs required mandatory PODS completion, enforcing a completed PODS for every patient discharged. A majority of hospitals that had a slower increase or dip in their implementation pattern identified initially utilizing PDSA cycles with small groups then slowly expanding to their full target group. Many hospitals identified trialing PODS with small groups at first facilitated implementation because it increased staff engagement and integration of PODS into the frontline process.

Measurement of fidelity was high across institutions. Average quality scores ranged from 8.1 for both acute and community hospital categories to 9.3 for small hospitals (Table 3).

Table 3 Fidelity as measured by ratings out of 10 provided during end-of-project interviews by project leads from each hospital

All hospitals identified a need for continual work to improve implementation quality. Even for hospitals that implemented with nearly 100% of their target population early on, they continued to work on improving quality, as can be seen from the following quote:

“We have to watch and train on using it properly and the use of plain language and ongoing PDSA is needed. We do audits and are providing feedback to the providers via practice hints. This is one of the ways that we augment the PDSA cycle.” (Project lead)

Very few patient, hospital, implementation process, fidelity or provider-reported factors showed association with a high level of penetration as defined as penetration within the target group of 75% or more by the fourth quarter after implementation (Table 4). Variables were all aggregated to the hospital level to assess association with hospital penetration. Trends showed association between higher penetration with higher ratings of quality (fidelity) and high participation in the community of practice.

Table 4 Results of exploratory analysis using a non-parametric Wilcox Rank Sum test to uncover group differences/associations based on levels of penetration of the intervention (high penetration is defined as 75% or more by Q4 after implementation)

Hospitals reported many factors that facilitated implementation. In general, these fell into three categories: (1) the intervention, (2) the implementation process, and (3) the organization.

Factors about the intervention that facilitated implementation were adaptability of the tool to local context of various patient groups and unique processes within the same hospital, co-design of PODS included patient and family input, and unplanned and natural spread. Local adaptability was described as being particularly important, as is highlighted in the following quote:

“We adapted because our patients have a high amount of dementia. We wrote it in a way that a caregiver or loved one could read it. That way it’s not given directly to the patient and the caregiver is looped into the discussion.”(Project lead)

Factors about the process of implementation that facilitated implementation were strong engagement with multi-disciplinary stakeholders, integration with frontline processes via a thorough current state assessment including removing alternative workflows and building the tool into the EMR if applicable, and participation in the community of practice, as is highlighted in the following quote:

We benefited greatly from having a platform for collaboration and learning from other sites during the implementation phase which was hosted by the lead project team. Being part of the group allowed us to secure strong leadership support and dedicate resources to patient and family engagement.”(Project lead)

Organizational-level factors that facilitated implementation were having a commitment to the implementation, smaller size hospitals, and integrating implementation into other QI initiatives. Smaller hospitals identified that it was easier to coordinate the project and engage and educate staff. Senior management in smaller hospitals, who wear “multiple hats”, found it easier to be more involved. Larger hospitals found it helpful to create small pilot projects and then expand to other departments. Integrating PODS into other QI initiatives was identified as both a facilitator and a consequence of implementation, as described in the following quote:

“PODS was part of a bigger discharge push. We leveraged our Nursing Workflow Optimization project (an informatics project) because PODS rollout and nursing workflow are well aligned. We also developed several new inputs that helped with implementing PODS [...] All of these new inputs were piggybacked off of PODS [...] Instead of being a product of optimization PODS helped drive optimization.”(Project lead)

Service and client outcomes

Hospitals surveyed 627 providers and 324 patients/families pre-implementation. 348 providers and 173 patient/families completed post-implementation surveys. A majority (64.1%) of providers felt that PODS added to their workload, however 86.2% found that it added value to the experience of providing discharge instructions. These measures of efficiency and satisfaction were not correlated with high penetration (Table 4). 98% of patients and families reported that PODS added value to their discharge experience.

Many patients and families added explanations and comments within their surveys on the usefulness of PODS, noting that they referred to the form at home and found having all the information in one place valuable. Some patients felt that it was not necessary for them, but that it would be especially useful if “you don’t have a lot of healthcare experience.” Those with previous experience in the same hospital commented on the improvement from previous forms. Several patients said it would have been more useful if provided earlier during their stay.

Many providers added comments on the usefulness of PODS for patients and families. Many also identified that it was only useful when the whole care team collaborated on completing the document and several providers commented that it enhanced collaboration.

Caregiver presence, use of teach-back and discussion about help needed at home improved with PODS implementation (Table 5). Over time, as sites increased penetration and fidelity, readmissions and ED visits decreased (Fig. 3). However, there was no significant change in readmissions or ED visits as a function of penetration (Table 5).

Table 5 Service and client outcome results. Efficiency and satisfaction outcomes are presented as the overall mean. Patient centeredness and symptomology outcomes are calculated using GEEs. Function outcomes are modeled using linear mixed effects regression
Fig. 3
figure3

Quarterly all-cause 30-day readmissions and ED visits within the target groups plotted together with PODS penetration

Within their added comments, many providers specifically noted the value of the teach-back process. The majority of providers who identified increased workload associated with PODS also noted qualitatively that the increase was small but worthwhile given the value added, as supported by the following quote:

“It increases workload a little, but it is worthwhile because it helps me to feel more confident that the patient know what to do and expect when they get home.” (Nurse)

Discussion

Over the course of the project, many diverse Ontario hospitals succeeded in implementing the PODS intervention with variation in penetration and implementation pattern between sites. High participation in the community of practice and high fidelity ratings were associated with higher penetration. PODS met a need to improve standardization and quality of patient-centred discharge processes, which was supported by a growing patient-centred focus across hospitals and policy circles [8].

Implementation outcomes

As expected, consistent with the CFIR framework, [20] having organizational commitment and a structured community of practice with a process of iteration and stakeholder involvement, were important to increased penetration. This was also reflected through qualitative reports identifying how the community of practice and executive sponsor helped in overcoming common barriers. To this end, our team, together with the hospitals involved in this project, developed an online community of practice open to anyone interested in implementing PODS. Future hospitals interested in implementing PODS should ensure executive commitment and consider joining the online community of practice or developing their own communities with other hospitals or within their hospital.

Local adaptation of the PODS tool was found to be a facilitator in this project and is known to play an important role in the diffusion of innovations [30, 31]. Likewise, local adaptation is considered the most important intervention characteristic within the CFIR [20]. The majority of individualized discharge tools are not developed through a process of co-design with patients and families [7]. The co-design process used to develop PODS was an important facilitator to implementation because organizations valued the source of the intervention and felt confident in the strength and quality of the tool. This also occurs when hospitals feel confident in the quality of their PODS, consistent with the association between high fidelity and penetration. These factors are intervention characteristics identified in the CFIR [20].

Small hospital size was an identified facilitator. This is inconsistent with implementation frameworks, [20, 32] which find larger organizations more likely to be successful predominantly due to stability of the organization and having sufficient resources to run the project. Interestingly, the CFIR notes several features of an organization’s internal setting as important for implementation success that are consistent with the smaller organizations in our cohort, namely having intertwined networks and good communication. Additionally, all hospitals, regardless of size, had to ensure sufficient resources for implementation before joining the project. Although our data did not show association between hospital size and penetration, smaller hospitals had higher fidelity highlighting the importance of combining penetration and fidelity when evaluating implementation success.

There were many factors studied that did not have any association with penetration including patient age, patient sex, whether the patient had a language barrier, whether the hospital was acute or rehabilitation, and whether the providers found that it added to their workload. These were factors that we would have thought might make the PODS more difficult to implement, however seem to be offset by the value added by the tool and by adapting the tool to make it part of the frontline process, meeting the needs of patients, families, and healthcare providers.

Service and client outcomes

The PODS intervention was found to be effective in improving patient-centred processes, specifically involving the family, using teach-back, and having conversations about help needed at home. These findings are especially important in light of a recent Ontario-wide study, [33] which found that family involvement in discharge planning and knowing who to call with questions about your condition, both key features of PODS, were two of the top five priorities identified to help improve discharge experience. The remaining top responses in the study focused on lack of available homecare supports. Although PODS does not address homecare capacity, it facilitates a discussion of the help available or needed by patients at home prior to discharge.

As a balancing measure and in line with evidence, [34] we know that provider workload and perceived value are important measures for adoptability of QI initiatives such as PODS. We found that providers indicated that PODS did increase their workload, yet added value to the discharge experience and neither factor on its own was correlated with increased penetration. This was confirmed with qualitative data, which showed that the added value was enough to offset the minimal addition to workload. This also fits with the CFIR, [20] which identifies the perceived advantage to implementing a tool as a factor supporting implementation.

Unscheduled healthcare utilization decreased over the course of the project, however no connection was found between the percentage of the target population receiving a PODS and healthcare utilization. The lack of connection can be partially explained by the earlier identified need to look at both penetration and fidelity to assess success. Over time, with both implementation reach and improved quality due to improving PODS quality and other QI initiatives, there was a decrease in resource utilization. Similarly, a review of interventions using the ideal transitions in care framework, [35] found that although the majority of interventions designed to improve transitions do not result in reduced utilization, having multiple interventions concurrently addressing different elements of transitions results in the best chances of reducing readmissions.

Strengths and limitations

This project has strengths as a large-scale implementation was evaluated that included many diverse hospitals representative of the Ontario-wide landscape. Large scale quality improvement implementations are rarely described in the literature [36]. By studying the implementation of PODS, we are able to understand how the community of practice was helpful and why organizations implemented the intervention in different ways. It provided insight into how future quality improvement communities of practice might be structured to support the needs of all participants. It also provided insight into interpretation of effectiveness results.

There were limitations with how hospital data on resource utilization was reported. Data was collected on whole target populations and not collected specific to patients who received PODS. Logically, this data still provides a useful estimate of resource utilization because the percentage of the target population who received a PODS is known and consistently increased over time. For survey data collected from providers, patients, and families, there was clustering because of variation in the number of responses each hospital was able to obtain and clustering in the actual site level means of the response variables. This is a limitation in terms of predicting effectiveness for particular sites, yet is a strength in that the estimated effects pool over a diversity of sites. For our analysis, we used this strength and aggregated the data and did not use it to infer organization-specific conclusions. Additionally, there was approximately 5% missing data in survey responses. At this level, we would expect to find significant effects using the GEE methods used. The addition of qualitative methods to triangulate and analyze results was also used as a method of minimizing these limitations. Lastly, this project was not designed to study causal inferences and designs with individual patient or cluster randomization are needed to study a causal link between PODS and health outcomes including adherence to discharge advice, healthcare utilization, and whether there are specific patient groups that are more likely to benefit.

Conclusions

This project confirmed the adaptability and ease of implementation of the PODS tool and highlighted the usefulness of implementing PODS to improve patient-centred discharge processes, a need and growing focus among hospitals internationally. In particular, we found that quality of PODS content and participation in a community of practice were factors that should be considered by hospitals implementing PODS and similar interventions in the future. Future research in the field of implementation of healthcare QI should consider studying hospital size, which was found to be a facilitator to implementation that differs from general implementation frameworks. Our findings can help inform policy related to hospital transitions as well as, more generally, future system-wide collaborative implementation projects centered on patient-centered tools where different implementation patterns and practices are prevalent.

Availability of data and materials

The datasets used during the current study are available from the corresponding author on reasonable request.

Abbreviations

PODS:

Patient Oriented Discharge Summary

ED:

Emergency Department

ARTIC:

Accelerating Research to Improve Care

QI:

Quality Improvement

CFIR:

Consolidated Framework for Implementation Research

EMR:

Electronic Medical Record

COP:

Community of Practice

PDSA:

Plan Do Study Act

GEE:

Generalized Estimating Equations

UHN:

University Health Network

References

  1. 1.

    Hahn-Goldberg S, Okrainec K, Huynh T, Zahr N, Abrams H. Co-creating patient-oriented discharge instructions with patients, caregivers, and healthcare providers. J Hosp Med. 2015;10(12):804–7. https://doi.org/10.1002/jhm.2444.

    Article  PubMed  Google Scholar 

  2. 2.

    McMartin K. Discharge planning in chronic conditions: an evidence-based analysis. Ont Health Technol Assess Ser. 2013;13(4):1–72.

    PubMed  PubMed Central  Google Scholar 

  3. 3.

    Centers for Medicare & Medicaid Services (CMS). Medicare program: hospital inpatient value-based purchasing program. Fed Regist. 2011;76(88):26490–547.

    Google Scholar 

  4. 4.

    Lapointe-Shaw L, Mamdani M, Luo J, Austin PC, Ivers NM, Redelmeier DA, et al. Effectiveness of a financial incentive to physicians for timely follow-up after hospital discharge: a population-based time series analysis. CMAJ. 2017;189(39):E1224–9. https://doi.org/10.1503/cmaj.170092.

    Article  PubMed  PubMed Central  Google Scholar 

  5. 5.

    Norwegian Ministry of Health and Care Services. The coordination reform–proper treatment–at the right place and right time. Summary in English. Report No. 47 (2008–2009) to the Storting. https://www.regjeringen.no/en/dokumenter/report.no.-47-to-the-storting-2008-2009/id567201/. Accessed 15 Feb 2020.

  6. 6.

    World Health Organization. Transitions of care. World Health Organization. 2016.https://apps.who.int/iris/handle/10665/252272. Accessed 15 Feb 2020.

  7. 7.

    Okrainec K, Lau D, Abrams HB, et al. Impact of patient-centered discharge tools: A systematic review. J Hosp Med. 2017;12(2):110–7. https://doi.org/10.12788/jhm.2692.

    Article  PubMed  Google Scholar 

  8. 8.

    Health Quality Ontario. Transitions Between Hospital and Home: Care for people of all ages. Quality Standards. 2020. Accessed on February 13, 2020 at https://www.hqontario.ca/Portals/0/documents/evidence/quality-standards/qs-transitions-between-hospital-and-home-quality-standard-en.pdf. Accessed 13 Feb 2020.

  9. 9.

    Mueller P, Oppeneimer D. The pen is mightier than the keyboard: advantages of longhand over laptop notetaking. Psychol Sci. 2014;25(6):1159–68. https://doi.org/10.1177/0956797614524581.

    Article  PubMed  Google Scholar 

  10. 10.

    Ha Dinh T, Bonner A, Clark R, et al. The effectiveness of the teach-back method on adherence and self-management in health education for people with chronic disease: A systematic review. JBI Database Syst Rev Implement Rep. 2016;14(1):210–47. https://doi.org/10.11124/jbisrir-2016-2296.

    Article  Google Scholar 

  11. 11.

    Rodakowski J, Rocco P, Ortiz M, et al. Caregiver integration during discharge planning for older adults to reduce resource use: a meta-analysis. J Am Geriatr Soc. 2017;65(8):1748–55. https://doi.org/10.1111/jgs.14873.

    Article  PubMed  PubMed Central  Google Scholar 

  12. 12.

    Cancino RS, Manasseh C, Kwong L, Mitchell SE, Martin J, Jack BW. Project RED Impacts Patient Experience. J Patient Exp. 2017;4(4):185–90. https://doi.org/10.1177/2374373517714454 Epub 2017 Jun 16. PMID: 29276765; PMCID: PMC5734517.

    Article  PubMed  PubMed Central  Google Scholar 

  13. 13.

    Jack BW, Chetty VK, Anthony D, Greenwald JL, Sanchez GM, Johnson AE, et al. A reengineered hospital discharge program to decrease rehospitalization: a randomized trial. Ann Intern Med. 2009;150(3):178–87. https://doi.org/10.7326/0003-4819-150-3-200902030-00007 PMID: 19189907; PMCID: PMC2738592.

    Article  PubMed  PubMed Central  Google Scholar 

  14. 14.

    Allen J, Hutchinson AM, Brown R, Livingston PM. Evaluation of the TRANSITION tool to improve communication during older patients' care transitions: healthcare practitioners' perspectives. J Clin Nurs. 2020;29(13–14):2275–84. https://doi.org/10.1111/jocn.15236 Epub 2020 Mar 31. PMID: 32129530.

    Article  PubMed  Google Scholar 

  15. 15.

    Campbell M, Fitzpatrick R, Haines A, Kinmonth AL, Sandercock P, Spiegelhalter D, et al. Framework for design and evaluation of complex interventions to improve health. BMJ. 2000;321(7262):694–6. https://doi.org/10.1136/bmj.321.7262.694 PMID: 10987780; PMCID: PMC1118564.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  16. 16.

    Hahn-Goldberg S, Okrainec K, Damba C, Huynh T, Lau D, Maxwell J, et al. Implementing patient-oriented discharge summaries (PODS): a multisite pilot across early adopter hospitals. Healthcare Q. 2016;19(1):42–8. https://doi.org/10.12927/hcq.2016.24610.

    Article  Google Scholar 

  17. 17.

    Canadian Institute for Health Information, the Government of Ontario, the Ontario Hospital Association and the Hospital Report Research Collaborative. 2007 Hospital Report: Acute Care. https://secure.cihi.ca/free_products/OHA_Acute07_EN_final_secure.pdf. Accessed 13 Feb 2020.

  18. 18.

    Canadian Institute for Health Information, the Government of Ontario, the Ontario Hospital Association and the Hospital Report Research Collaborative. 2007 Hospital Report: Rehabilitation. https://secure.cihi.ca/free_products/OHA_Rehab_07_EN_final_secure.pdf. Accessed 13 Feb 2020.

  19. 19.

    Health Quality Ontario and the Council of Academic Hospitals of Ontario. The ARTIC Program: Reflecting on five years of spread and scale 2019. https://www.hqontario.ca/Portals/0/documents/qi/artic/the-artic-program-reflecting-on-five-years-of-spread-and-scale-en.pdf. Accessed 1 Dec 2019.

  20. 20.

    Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4, 4(50, 1). https://doi.org/10.1186/1748-5908-4-50.

  21. 21.

    Wenger E. Communities of practice: A brief introduction. 2011. https://scholarsbank.uoregon.edu/xmlui/handle/1794/11736. Accessed 6 Apr 2021.

  22. 22.

    OpenLab. PODS-toolkit website. 2019. http://pods-toolkit.uhnopenlab.ca/. Accessed 1 Dec 2019.

  23. 23.

    Curran G, Bauer M, Mittman B, et al. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012;50(3):217–26. https://doi.org/10.1097/MLR.0b013e3182408812.

    Article  PubMed  PubMed Central  Google Scholar 

  24. 24.

    Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Admin Pol Ment Health. 2011;38(2):65–76. https://doi.org/10.1007/s10488-010-0319-7.

    Article  Google Scholar 

  25. 25.

    Canadian Institute for Health Information. Canadian Patient Experiences Survey — Inpatient Care. https://www.cihi.ca/sites/default/files/document/cpes-prem-preliminary-measures-table-april2019-en-web.pdf. Accessed 7 Feb 2019.

  26. 26.

    Canadian Institute for Health Information. Canadian Patient Experiences Survey – Inpatient Care Procedure Manual. https://www.cihi.ca/sites/default/files/document/cpes-ic-procedure-manual-2019-en-web.pdf. Accessed 14 Dec 2020.

  27. 27.

    Hadibhai S, LacroixJ LK. Developing the first pan-Canadian acute care patient experiences survey. Patient Exp J. 2018;5(3):25–33. https://doi.org/10.35680/2372-0247.1227.

    Article  Google Scholar 

  28. 28.

    Bell ML, Horton NJ, Dhillon HM, Bray VJ, Vardy J. Using generalized estimating equations and extensions in randomized trials with missing longitudinal patient reported outcome data. Psycho-oncology. 2018;27(9):2125–31. https://doi.org/10.1002/pon.4777.

    Article  PubMed  Google Scholar 

  29. 29.

    Akter T, Sarker EB, Rahman S. A tutorial on GEE with applications to diabetes and hypertension data from a complex survey. J Biomed Anal. 2018;1(1):37–50. https://doi.org/10.30577/jba.2018.v1n1.10.

    Article  Google Scholar 

  30. 30.

    Rogers EM. Diffusion of innovations. 5th ed. New York: Free Press; 2003.

    Google Scholar 

  31. 31.

    Berwick DM. Disseminating innovations in health care. JAMA. 2014;289(15):1969–75.

    Article  Google Scholar 

  32. 32.

    Greenhalgh T, Robert G, Macfarlane F, et al. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82(4):581–629. https://doi.org/10.1111/j.0887-378X.2004.00325.x.

    Article  PubMed  PubMed Central  Google Scholar 

  33. 33.

    Kiran T, Wells D, Okrainec K, Kennedy C, Devotta K, Mabaya G, et al. Patient and caregiver priorities in the transition from hospital to home: results from province-wide group concept mapping. BMJ Qual Saf. 2020;29(5):390–400. https://doi.org/10.1136/bmjqs-2019-009993.

    Article  PubMed  Google Scholar 

  34. 34.

    Hayes CW, Goldmann D. Highly adoptable improvement: a practical model and toolkit to address adoptability and sustainability of quality improvement initiatives. Jt Comm J Qual Patient Saf. 2018;44(3):155–63. https://doi.org/10.1016/j.jcjq.2017.09.005.

    Article  PubMed  Google Scholar 

  35. 35.

    Burke RE, Guo R, Prochazka AV, Misky GJ. Identifying keys to success in reducing readmissions using the ideal transitions in care framework. BMC Health Serv Res. 2014;14(1):423. https://doi.org/10.1186/1472-6963-14-423.

    Article  PubMed  PubMed Central  Google Scholar 

  36. 36.

    Rohweder C, Wangen M, Black M, Dolinger H, Wolf M, O'Reilly C, et al. Understanding quality improvement collaboratives through an implementation science lens. Prev Med. 2019;129:105859. https://doi.org/10.1016/j.ypmed.2019.105859.

    Article  Google Scholar 

Download references

Acknowledgements

The authors acknowledge the input and advice of our project advisory group as well as the critical work and support of all the individual organizations along with their project teams. Their implementation evaluation is included in this report for the work they completed while implementing PODS to improve the transition experience for patients and families (Bruyere Continuing Care, The Centre for Addiction and Mental Health, Chatham-Kent Health Alliance, Campbellford Memorial Hospital, Georgian bay General Hospital, Geraldton District Hospital, Grand River Hospital and the Waterloo Wellington LHIN, Kingston Health Sciences Centre, London Health Sciences Centre, Markham Stouffville Hospital, Oakville Trafalgar Hospital, Perth and Smith Falls District Hospitals, Ross Memorial Hospital, Renfrew Victoria and St. Frances memorial, St. Joseph General Elliot Lake, St. Joseph’s healthcare Hamilton, Thunder Bay Regional Health Sciences Centre, Temiskaming Hospital, Toronto Rehabilitation Hospital, Toronto Western Hospital, and William Osler Health System).

Funding

The implementation and evaluation of PODS across Ontario was funded by Health Quality Ontario and the Council of Academic Hospitals of Ontario through the ARTIC program. The ARTIC program was critical in our ability to implement and spread the PODS initiative and they had membership as knowledge users on the project advisory group and as such were involved in the project design. The evaluation reported here presents the first phase of a CIHR funded Transitions in Care Evaluation grant held by Drs. Karen Okrainec and Shoshana Hahn-Goldberg. Dr. Karen Okrainec is supported by an Early Research Award (ERA) which supports members of her research team and the research described in this manuscript. Both CIHR and the ERA body played no role in the design of the study or any data collection, analysis, data interpretation, or manuscript writing.

Author information

Affiliations

Authors

Contributions

SHG, TH, AC, MK, VR, GT, JM, HA, CB, and KO contributed to the conception and design. SHG, CF1, AT, CF2, SH, BN, and BO contributed to acquisition. SHG, MK, VR, GT, JM, and KO contributed to analysis. SHG, TH, AC, MK, VR, GT, JM, HA, CB, CM, CF1, AT, CF2, SH, BN, BO, KO contributed to interpretation of data. SHG drafted the work and TH, AC, MK, VR, GT, JM, HA, CB, CM, CF1, AT, CF2, SH, BN, BO, KO contributed to revision, approved the submitted version, and have agreed both to be personally accountable for their own contributions and to ensure any questions related to the work are appropriately investigated and resolved. The authors read and approved the final manuscript.

Corresponding author

Correspondence to Shoshana Hahn-Goldberg.

Ethics declarations

Ethics approval and consent to participate

This study was approved by the University Health Network Research Ethics Board as Project Number 17–6272. The UHN REB provided permissions to access and use the data described in this study. As only aggregate and non-identifiable data was being used for this quality improvement initiative that was ongoing, the UHN REB deemed informed consent as unnecessary for this project.

Consent for publication

Not Applicable.

Competing interests

The Authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1.

Interview Guides and Surveys Developed for this Project. The file includes one interview guide and two sets of survey questions with descriptions of response options and the sources of the survey questions. The interview guide was developed and used to interview project leads at the end of the project. The first survey, Patient/Caregiver Survey Question Information, was provided to patients and caregivers at two points in time, pre- and post-intervention. The second survey, Provider Survey Information, was provided to healthcare providers at two points in time, pre- and post-intervention. These materials were used to collect some of the qualitative and quantitative data presented in the manuscript.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Hahn-Goldberg, S., Huynh, T., Chaput, A. et al. Implementation, spread and impact of the Patient Oriented Discharge Summary (PODS) across Ontario hospitals: a mixed methods evaluation. BMC Health Serv Res 21, 361 (2021). https://doi.org/10.1186/s12913-021-06374-8

Download citation

Keywords

  • Implementation
  • Discharge
  • Patient Centred
  • Transitions in care
  • Local adaptability
  • Triangulation
  • Patient experience
  • Quality improvement
  • Hospital