- Open Access
Development and psychometric properties of surveys to assess patient and family caregiver experience with care transitions
BMC Health Services Research volume 21, Article number: 785 (2021)
The purpose of this study was to develop and administer surveys that assess patient and family caregiver experiences with care transitions and examine the psychometric properties of the surveys. The surveys were designed to ask about 1) the transitional care services that matter most to patients and their caregivers and 2) care outcomes, including the overall quality of transitional care they received, patient self-reported health, and caregiver effort/stress.
Survey items were developed based on a review of the literature, existing surveys, focus groups, site visits, stakeholder and expert input, and patient and caregiver cognitive interviews. We administered mail surveys with telephone follow up to patients recently discharged from 43 U.S. hospitals. Patients identified the caregivers who helped them during their hospital stay (Time 1 caregiver) and when they were home (Time 2 caregiver). Time 1 and Time 2 caregivers were surveyed by telephone only. The psychometric properties of the survey items and outcome composite measures were examined for each of the three surveys. Items that performed poorly across multiple analyses, including those with low variability and/or a high missing data, were dropped except when they were conceptually important.
The analysis datasets included responses from 9282 patients, 1245 Time 1 caregivers and 1749 Time 2 caregivers. The construct validity of the three proposed outcome composite measures—Overall Quality of Transitional Care (patient and caregiver surveys), Patient Overall Health (patient survey) and Caregiver Effort/Stress (caregiver surveys) —was supported by acceptable exploratory factor analysis results and acceptable internal consistency reliability. Site-level reliability was acceptable for the two patient outcome composite measures, but was low for Caregiver Effort/Stress (< 0.70). In all surveys, the Overall Quality of Transitional Care outcome composite measure was significantly correlated with other outcome composite measures and most of the single-item measures.
Overall, the final patient and caregiver surveys are psychometrically sound and can be used by health systems, hospitals, and researchers to assess the quality of care transitions and related outcomes. Results from these surveys can be used to improve care transitions, focusing on what matters most to patients and their family caregivers.
Patient transitions in care from the hospital to post-acute settings or home continue to be fraught with potential gaps in care and services that can put patients at risk and overburden family caregivers [1, 2]. It is therefore important to identify the critical transitional care services or groups of services that matter most to patients and family caregivers and that yield the best outcomes. In a literature review on patient experience with healthcare, Wolf et al.  found that studies about patient experience focus on individualized care and tailoring of services to meet patients’ needs and engage them as partners in their care, which is integral to the principles and practice of patient- and family- centered care. Other research has found that better patient care experiences are associated with better clinical outcomes [4, 5], better patient safety culture within hospitals , and lower 30-day hospital readmission rates for acute myocardial infarction, heart failure, and pneumonia .
While patients’ experiences are increasingly recognized as a critical component of the overall quality of care, much less attention has been given to the experiences of family caregivers, who often bear much responsibility in caring for and managing the patient’s care during transitions across health settings. When it comes to care transitions, family caregivers need more information and knowledge about how to care for the patient, need to be more involved in identifying patient needs , want to feel cared for and about by medical providers, and want to feel prepared and capable of implementing patient care plans . Family caregivers need information during every step of the process, both before and after hospital discharge, which underscores the importance of communication with healthcare professionals across the continuum of care .
Understanding patient and family caregiver experience with healthcare is critical in moving toward care that is more patient-and-family-centered. While there are numerous measurement instruments designed to assess patient experience with healthcare, only a subset of these include a focus on care coordination across the continuum of care or family caregiver experiences with care transitions from hospital to home. Therefore, the purpose of this study was to develop and administer survey instruments that assess patient and family caregiver experiences with care transitions from hospital to home and examine the psychometric properties of the surveys. The surveys were designed to ask about 1) the transitional care services that matter most to patients and their family caregivers and 2) care outcomes, including the overall quality of transitional care they received, patient self-reported health, and caregiver effort/stress.
This study was undertaken as one of the specific aims of a much larger project funded by the Patient-Centered Outcomes Research Institute (PCORI) called Project ACHIEVE (Achieving Patient-Centered Care and Optimized Health In Care Transitions by Evaluating the Value of Evidence). The overall aims of Project ACHIEVE were to identify the transitional care services and outcomes that matter most to patients and family caregivers, and to identify which combinations of transitional care strategies, or groups of services, yield desired outcomes among a large and diverse cohort of United States (U.S.) hospitals [11, 12].
All study procedures for our survey study were approved by the Institutional Review Boards at the University of Kentucky, Kaiser Permanente Southern California, and Westat. The study protocol was carried out in accordance with relevant guidelines and regulations.
Survey item development
To develop the content for the surveys, we first conducted a literature review to identify the types of transitional care services or components of care that are important from both the patient and family caregiver perspectives. We reviewed articles that assessed transitional care services and expected care outcomes [13,14,15,16,17,18,19] We identified existing relevant surveys, including Consumer Assessment of Healthcare Providers and Systems (CAHPS) surveys [20, 21] and other validated measures, such as Patient-Reported Outcomes Measurement Information System (PROMIS®) measures [22–24]. We also reviewed research and surveys on family caregiver burden and stress [25–27]. Because most existing surveys focused on either patients or caregivers, and did not comprehensively assess the range of services provided by both hospitals and outpatient providers, we sought to develop surveys to fill these important gaps. We designed surveys to ask patients and caregivers about the transitional care components and services they each received across the continuum of care from the hospital to outpatient providers once the patient returned home. We also designed the surveys to assess outcome measures of the overall quality of transitional care, patient self-reported health, and caregiver effort/stress.
The Project ACHIEVE research team conducted background research through qualitative focus groups and individual interviews with a total of 138 patients and 110 family caregivers across the U.S. . This qualitative data collection elicited descriptions of patient and caregiver experiences around care transitions and identified the outcomes that were most important from their perspectives. Survey content was also informed by site visits to hospitals included in the study to better understand the facilitators and barriers of effective care transitions . In addition, a Stakeholder Advisory Group (SAG), which included patient and caregiver representatives, and Scientific Advisory Council (SAC) provided important input on survey content, item wording, and survey length throughout survey development and analysis.
Synthesizing these diverse sources of input, the ACHIEVE research team identified main content areas for inclusion in the patient and caregiver surveys. We identified content areas and critical components of care provided in the hospital and at home. These components included communication with providers; communication about medications; receiving supplies or equipment; and home visits, among other critical components of care. We then drafted survey items to assess those content areas and components of care. Some survey items were adapted from existing surveys and other items were developed to assess content areas where existing items were not available or sufficient for our measurement goals.
The patient survey was designed for patients recently discharged from the hospital to assess their experiences with care both in the hospital and once they got home. The caregiver surveys were developed in parallel with the patient survey to assess similar questions about care in the hospital and at home, but from the perspective of caregivers. The caregiver surveys were designed for the family member or friend who was identified by the patient as the person who provided the most help while the patient was in the hospital and/or once they got home. The caregiver surveys had two versions intended for caregivers at two different points in time: a Time 1 (T1) caregiver who provided most of the support during the patient’s hospitalization, and a Time 2 (T2) caregiver who was most involved with the patient’s care at home. A patient could have both a T1 and T2 caregiver, or at only one of these points in time. In addition, the T1 and T2 caregiver could be the same family member or friend, or a different one.
The research team conducted cognitive interviews with patients and caregivers to pretest the draft survey items. The goal of the cognitive interviews was to assess item comprehension, relevance, and ease of responding. Sixty-eight cognitive interviews (34 patients and 34 caregivers) were conducted to iteratively test variations of the patient and caregiver survey items. Both patients and caregivers were recruited to vary in age, gender, and race/ethnicity. Results of cognitive testing were used to refine survey items prior to data collection.
Table 1 describes the measures included in the patient and caregiver surveys, showing the number of items in each section. The patient survey included 60 items and the caregiver surveys each included 56 items. The majority of items were similar in the patient and caregiver surveys, with wording customized as needed. However, some items were unique. For example, the patient survey included items about patient-reported overall health, whereas the caregiver surveys asked about caregiver effort/stress. The surveys also included background questions about respondent characteristics. The patient and caregiver surveys were also translated into Spanish.
We developed multiple survey items to assess each of the three outcomes: the Overall Quality of Transitional Care from the patient and caregiver perspectives, Patient Overall Health, and Caregiver Effort/Stress. These outcomes were proposed as composite measures or groups of two or more survey items designed to measure an underlying construct. All other items were either single-item measures or filter questions.
Forty-three hospitalsFootnote 1 across the U.S. were recruited into the ACHIEVE Study using a purposive sampling strategy to ensure representation of the following characteristics: 1) urbanicity; 2) safety-net; 3) critical access; 4) integrated delivery system (including Kaiser Permanente hospitals); 5) participation in alternative payment models (e.g., Accountable Care Organizations); and /or 6) participation in a formal evidence-based TC program (e.g., Project RED) or community-based transitional care program (e.g., CMS Community-based Care Transitions Program (CCTP)).
Patient and caregiver recruitment
Medicare beneficiaries or dual-eligible patients that were discharged from the medical or surgical units at the participating hospitals were eligible to participate. Hospital staff recruited patients and Time 1 (T1—in the hospital) family member or friend caregivers of the patient. Hospital staff approached patients before discharge to obtain HIPAA authorization, consent to be contacted to complete a mail or phone survey, and contact information for a T1 caregiver who helped them during their hospitalization, if applicable. On a weekly basis throughout the data collection field period, hospital staff provided the research team with contact information for consenting discharged patients and their T1 caregivers. Contact information for Time 2 (T2—since the patient has been home) caregivers was requested from patients that completed the patient survey. Over 44 weeks of patient and caregiver recruitment from June 2017 to April 2018, 43 hospitals recruited 17,638 patients; and 41 hospitals recruited 5031 T1 caregivers (two hospitals did not recruit T1 caregivers).
Patient survey administration
Patients were contacted beginning 51 days after discharge per the Centers for Medicare and Medicaid Services (CMS) guidelines to avoid conflicts with Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) data collection. Patient survey administration included a two-wave mail survey with phone follow-up for nonrespondents. Patient data collection was conducted over 49 weeks from August 2017 through July 2018. Patients received an initial mail survey packet which included a cover letter explaining the project, the survey, and a $5 prepaid cash incentive. The exception to the prepaid cash incentive was for patients from an integrated health system, who received a $5 promised incentive upon completion of the survey, as preferred by the system’s IRB protocol.
Seven days after the initial mailing, all patients were mailed a reminder postcard and a second survey was mailed to non-respondents. After the second survey mailing, up to five follow-up phone calls were made. A Spanish language survey and materials were mailed by request; however, patients were able to complete the phone interview in English or Spanish based on their preference. On average, the patient survey was completed and/or returned 75 days after discharge.
Time 1 (T1) and time 2 (T2) caregiver survey administration
Both the T1 and T2 caregiver surveys were administered only by phone, with up to five phone call attempts. Caregivers were promised a $5 incentive upon completion of the survey, and interviews were conducted in both English and Spanish. Interviewers contacted the T1 caregiver 14 to 28 days after patient discharge; on average, the T1 caregiver survey was completed 18 days after patient discharge. Data were collected from T1 caregivers from July 2017 through May 2018 (about 42 weeks).
T2 caregivers were contacted at least 51 days after patient discharge, after patients completed their survey and provided the T2 caregiver name and phone number. On average, the T2 caregiver survey was completed 85 days after patient discharge. Interviewers collected data from T2 caregivers from August 2017 through July 2018 (about 47 weeks).
Creating the analysis datasets
To create the analysis datasets, we combined patient mail and phone survey responses and then cleaned the combined patient survey and caregiver surveys. To include a response as “complete,” a respondent had to respond to at least 50% of the applicable-to-all questionsFootnote 2 .
Several psychometric analyses were conducted with the goal of identifying conceptually meaningful and reliable outcome composite measures in the patient and caregiver surveys. We also examined the psychometric properties of the single-item measures. Psychometric analyses included (1) item response variability and missing data patterns, (2) exploratory factor analysis, (3) internal consistency and site-level reliability, and (4) correlations among the proposed outcome composite measures and other survey items.
Item variability and missing data
As a first step, we examined item frequencies to evaluate the variability of responses. Items with little response variability may not be helpful in differentiating higher-scoring from lower-scoring individuals and hospitals. To assess item variability, we examined either top-box scores or percent positive scores. The top-box scores (the top, most positive responses) were calculated for yes/no items with three or fewer response options (e.g. percent of respondents who answered “Yes” for items with Yes/No response options, or percent “Yes, definitely” for items with Yes, definitely/Yes, somewhat/No response options ). For items with four or more response options using Likert-type scales, we calculated percent positive scores (percent of respondents who answered using the top two most positive responses , e.g., percent “Very well/Moderately well or Excellent/Very good). For patient-reported bodily pain, lower frequency was considered to be positive (i.e., percent Not at all/Once in the past week) as well as for caregiver effort/stress (percent No effort/A little effort and Not at all/Somewhat stressful). To indicate low item variability, we flagged items that were extremely positive with top box scores or percent positive scores greater than 95%.
Next, we identified items with high percentages of missing data. High missingness might indicate that items are not relevant to a large portion of respondents. Sources of missing data in the patient, T1 caregiver, and T2 caregiver surveys included tailored inapplicable responses (e.g., “I already knew what to do”), valid skips (based on filter questions), and other types of missing (not answered, don’t know, or refused). Items were flagged as having high missingness if all missing responses combined (tailored inapplicable, valid skips, and all other missing) exceeded 65%.
Exploratory factor analysis (EFA)
We conducted EFA to examine the construct validity of the three proposed outcome composite measures in the patient, T1, and T2 caregiver surveys. A separate EFA was conducted for the patient, T1, and T2 datasets. Each EFA included all items comprising the proposed outcome composite measures in each survey: Overall Quality of Transitional Care (for the patient and caregiver surveys), Patient Overall Health (patient survey), and Caregiver Effort/Stress (caregiver surveys). We used iterated principal axis factors as the method of extraction, with varimax (orthogonal) rotation to maximize the dispersion of factor loadings within factors (i.e., the number of factor loadings close to one and close to zero). Factor loadings, or correlations between items and factors, range from − 1.00 to 1.00. In general, factor loadings with absolute values above 0.40 (which explain around 16% of the variance in the item) are considered acceptable .
Outcome composite measure internal consistency reliability and site-level reliability
Next, we examined Cronbach’s alpha (α) to determine the internal consistency reliability of the items within each of the three outcome composite measures to assess whether respondents answered the items in a similar way. Cronbach’s alpha ranges from 0 to 1, with higher alphas indicating better reliability. The minimum criterion for acceptable reliability is an alpha of 0.70 .
To examine the variability of the outcome composite measures and single-item measures within hospitals compared to between hospitals, we computed site-level reliability. Site-level reliability, which is directly related to the standard error of measurement, captures the extent to which responses from patients and caregivers within the same hospital are more similar to each other than they are to responses from other hospitals. In other words, site-level reliability helps to assess how well a measure differentiates hospitals. It does so by comparing between-site variability to within-site variability, while adjusting for the average number of respondents within each hospital. Similar to internal consistency reliability, values of 0.70 or higher are considered acceptable for site-level reliability .
Outcome composite measure correlations
Finally, as another indicator of construct validity, we examined individual-level Spearman’s rank-order correlations among and between the outcome composite measures and the single-item measures. Since the surveys were designed to assess different, but related aspects of transitional care in hospitals, the outcome composite measures and single-item measures should be correlated, showing a correspondence or convergence that would result in moderate or moderately high correlations (e.g., +/− 0.50 to +/− 0.80). However, correlations that are very high (e.g., +/− 0.90 to +/− 1.00) may indicate a significant amount of overlap, implying that the composite measures or items may be measuring the same or very similar concepts . On the other hand, correlations that are very low, close to zero, may indicate that the composite measures or items are not related to one another, potentially measuring unrelated concepts.
Criteria for evaluation item performance
Items that performed poorly across multiple analyses and/or in two or more surveys were dropped from the final instruments. When considering which items to drop from the surveys, we placed most emphasis on item analysis, as items with low variability and/or a high percentage of missing data would not be very useful to hospitals looking to measure and improve care transitions. Exceptions to dropping included items that were considered conceptually important to measuring care transitions and items for which a large percentage of missing data was expected (e.g., Q11_A. Hospital: Written information in Spanish). Demographic/background items were excluded from psychometric analysis.
Overall response rates for the patient, T1 caregiver, and T2 caregiver surveys across the 43 participating hospitals were 57% (9450/16,573), 28% (1262/4455), and 35% (1788/5106), respectively. After data cleaning, the final analysis datasets consisted of 9282 patient responses, 1245 T1 caregiver responses, and 1749 T2 caregiver responses representing 43 hospitals. Supplemental Table 1 presents the characteristics of the 43 participating hospitals and provides the comparison of study hospitals to the 2015 American Hospital Association (AHA) registered hospitals on selected characteristics. The study hospitals were more likely than AHA hospitals to be from the Northeast and West, to be large (≥300 beds), and have nongovernment/non-for-profit ownership. In addition, the study hospitals were more likely to be large urban and teaching compared to 2019 CMS Impact hospitals (Supplemental Table 2).
Table 2 presents patient respondent characteristics. The majority of patient respondents were female (53%), White (78%), and Non-Hispanic (86%). Twenty-seven percent of patient respondents had at least a 4-year college degree. Most patients (80%) had a family member or friend who helped to take care of them at home. The most common category of informal caregiver was husband/wife (53%), followed by son/daughter (including in-laws) (27%).
T1 and T2 caregiver respondent characteristics are presented in Table 3. The majority of both T1 and T2 caregiver respondents were female (72 and 70%, respectively). Approximately one-third both T1 and T2 caregivers had at least a 4-year college degree. Most of the caregivers were not working or were retired (64% of T1 and 70% of T2), and identified as the husband or wife of the patient (58% of T1 and 61% of T2). Approximately one-quarter of both T1 and T2 caregivers identified as sons or daughters of the patient (including in-laws). The majority of caregivers had been caring for the patient for 12 months or more (58% of T1 and 56% of T2), lived with the patient (78% of T1 and 84% of T2), and identified as the patient’s sole caregiver (51% of T1 and 60% of T2).
Item variability and missingness
Table 4 presents percent positive and top-box responses for all survey items identified as having low variability and/or high percentage of missing values in the patient, T1 caregiver, and/or T2 caregiver surveys. Percent positive responses, top-box responses and missingness for all items relevant to psychometric analyses (i.e., non-background, non-demographic items) are shown in Supplemental Table 3. Percent positive/top box responses ranged from 4 to 96% in the patient survey, from 1 to 96% in the T1 caregiver survey, and from 4 to 96% in the T2 caregiver survey. Two items had percent positive scores greater than 95% (Q20. Home: How well been able to use supplies/equipment? 96% for patients; and Q22. Home: How well been able to take care of wound/surgical site? 96% for T1 and T2 caregivers). Items with excessively high or low percent positive/top box scores were flagged as having low variability.
The percentages shown in Table 4 combine missingness due to tailored inapplicable responses (e.g., “I already knew what to do”), valid skips, and other sources (not answered, don’t know, or refused). Seven survey items had greater than 65% missing values in the patient, T1 caregiver, and T2 caregiver surveys, indicating that the majority of respondents in all three surveys did not answer these questions. For six of the seven items (all except Q11_A [Hospital: Written information in Spanish]), the high missingness was due to valid skips. Three of the seven items also had very low percentages of affirmative (“yes”) responses across all three surveys (Q19_B through Q19_D, reasons patients did not take medicine as directed), indicating that these items were not applicable to most respondents. Therefore, these three items were dropped from the final surveys. Because Q19_A (forgot to take medicine) was incomplete as a standalone item and had a high percentage of missing values in all three surveys, it was also dropped from the final surveys. Finally, Q22 was dropped from the final caregiver surveys because of its low variability and high missingness. Despite excessive missingness, Q11_A (Hospital: Written information in Spanish) was not considered problematic because very few respondents took the survey in Spanish, so the item was retained. In summary, four items were dropped from the final patient and caregiver surveys (Q19_A through Q19_D), and one additional item was dropped from the final caregiver surveys but kept in the final patient survey (Q22).
Exploratory factor analysis (EFA) for proposed outcome composite measures
Results of the EFA revealed that Q38 (Home: Rate ability to take care of self/patient) in the Overall Quality of Transitional Care outcome composite measure had a factor loading above 0.40 on two factors in the patient survey and did not load above 0.40 on either factor in the T1 caregiver survey. We therefore removed Q38 and repeated the EFA. However, given the importance of the content measured by Q38, it was retained for all subsequent analyses as a single-item measure. Results of the final EFA for the patient survey are presented in Table 5. The EFA retained two factors. All factor loadings for items on their respective composite measures were above 0.40 (range 0.51 to 0.85).
The EFA for both the T1 and T2 caregiver surveys also retained two factors (Table 6). All factor loadings for items on their respective composite measures were above 0.40 for both T1 and T2 caregivers (range 0.60 to 0.92 and 0.65 to 0.78, respectively).
The pattern and magnitudes of the factor loadings in all three surveys indicated a clear differentiation between the factors, reflecting the proposed measurement structure.
Outcome composite measure internal consistency reliability
Table 7 presents Cronbach’s alpha (α) measuring internal consistency reliability for each outcome composite measure, as well as alpha if an item were to be deleted. The two outcome composite measures in the patient survey—Overall Quality of Transitional Care and Patient Overall Health—had internal consistency reliability above the criterion of at least 0.70 (α = 0.79 for both composite measures). Similarly, the two outcome composite measures in the T1 and T2 caregiver surveys—Overall Quality of Transitional Care and Caregiver Effort/Stress—had internal consistency reliabilities of at least 0.70 (T1 α = 0.70, 0.73 and T2 α = 0.75, 0.72, respectively). For all three surveys, deleting any items would not improve reliability.
Outcome composite measure site-level reliability
The site-level reliability for the outcome composite measures in the patient survey was 0.82 for Overall Quality of Transitional Care and 0.78 for Patient Overall Health. The site-level reliability for Overall Quality of Transitional Care was 0.72 in the T1 caregiver survey and 0.62 for the T2 caregiver survey (below the criterion of 0.70). The Caregiver Effort/Stress outcome composite measure had site-level reliability below the criterion for both T1 (0.63) and T2 (0.64) caregivers. Site-level reliability for all final survey items is shown in Supplemental Table 4.
Outcome composite measure correlations
In both the patient and caregiver surveys, the Overall Quality of Transitional Care outcome composite measure was significantly correlated with the other outcome composite measures and most of the single-item measures. In the patient survey, higher Overall Quality of Transitional Care was significantly correlated with better self-reported Patient Overall Health (rs = 0.40, p < .05). In the caregiver surveys, higher Overall Quality of Transitional Care was significantly related to lower Caregiver Effort/StressFootnote 3 for T1 caregivers (rs = 0.19, p < .05,) and for T2 caregivers (rs = 0.18, p < .05). We also examined correlations between the outcome composite measures and the single-item measures (in Supplemental Table 5). Out of 40 possible associations between the single-item measures and each of the two outcome composite measures on the patient survey, the majority were statistically significant (p < .05) (37 correlations with Overall Quality of Transitional Care; 39 correlations with Patient Overall Health). Out of 37 possible associations between the single-item measures and each of the two outcome composite measures on the caregiver surveys, the majority were also statistically significant (p < .05) (32 correlations for T1 caregivers and 30 for T2 caregivers with Overall Quality of Transitional Care; 30 correlations for T1 caregivers and 33 for T2 caregivers with Caregiver Effort/Stress).
Final survey items
After reviewing the performance of the single-item measures, we identified and removed items with multiple analytic issues from the final patient and caregiver surveys (four items from the patient survey, five from the T1 caregiver survey, and five from the T2 caregiver survey, with dropped items shown in Supplemental Table 6). The final patient, T1 caregiver, and T2 caregiver surveys (shown in Additional Files 7, 8, and 9-- Appendixes A, B, and C) have 56, 51, and 51 items respectively.
Lack of appropriate, well-organized transitions can lead to unplanned hospital readmissions and poor patient outcomes. While hospitals continue to focus efforts to improve care transitions and reduce readmissions, it is important to understand the care transitions experience from the perspective of patients and their family caregivers when deciding where to invest finite resources. Our study developed and tested patient and caregiver surveys designed to assess 1) the transitional care services or components of care that are provided in the hospital and at home, and 2) care outcomes including the overall quality of transitional care they received, patient self-reported health, and caregiver effort/stress. Importantly, the transitional care services included in the survey were based on multiple sources of research and input about what matters most to patients and family caregivers in transitional care [9, 13–17].
Across the patient and caregiver surveys, most items had reasonable response variability and missingness, even though overall responses tended to be positive. Percent positive or top box scores were similar between T1 and T2 caregivers. Only four items were dropped from the final patient survey and five items were dropped from the final caregiver surveys because of low variability and/or a high percentage of missingness.
When examining the factor structure of the three proposed outcome composite measures—Overall Quality of Transitional Care (patient and caregiver surveys), Patient Overall Health (patient survey), and Caregiver Effort/Stress (caregiver surveys)—the final factor analyses yielded good item factor loadings (above 0.40) that supported the construct validity of the three final outcome composite measures. The final outcome composite measures also demonstrated good internal consistency reliability (above 0.70).
Site level reliability of the composite measures was also good for the patient survey, but did not reach acceptable levels for T2 caregivers on Overall Quality of Transitional Care, and for T1 and T2 caregivers on Caregiver Effort/Stress. Because T1 and T2 caregivers were recruited through patients and response rates were much lower for caregivers than for patients (28% for T1 and 35% for T2 caregivers), the average number of caregiver respondents associated with a specific hospital was low (for the items in the final survey the average number of caregivers ranged from 8 to 30 for T1s and 7 to 41 for T2s). Therefore, it is likely that the small number of respondents affected the site level reliability of the caregiver composite measure scores.
In the patient survey, higher Overall Quality of Transitional Care was significantly correlated with better Patient Health Outcomes, demonstrating a positive relationship between the quality of transitional care and patient-reported physical, mental/emotional health, sleep, pain and mobility. Higher Overall Quality of Transitional Care for T1 and T2 caregivers was also significantly correlated with lower Caregiver Effort/Stress, again demonstrating an important relationship among these key outcomes. In addition, the majority of the single-item measures within each survey were also related to the outcome composite measures. These significant correlations support the construct validity of the outcome composite measures in relation to the single-item measures. Including built-in “outcome” composite measures within the surveys enables analyses to examine how receiving certain transitional care services or components relates to the overall quality of transitional care, patient-reported health outcomes, and caregiver effort/stress, which are important patient and family-centered outcomes of care.
Because the survey items ask about many different types of services or components of care, including communication, providing education and information, medical supplies or equipment, transportation assistance, meals, and physical or occupational therapy, most of the items are not grouped into composite measures. Instead, the single-item measures that assess the type or number of services or components of care can be used to identify which groups or clusters of services result in more positive patient and caregiver experiences and outcomes. Another study by the Project ACHIEVE team  conducted analyses of survey data from hospitals and data from the patient survey and identified five groups of transitional care components or strategies that were most likely to co-occur and be delivered by hospitals. The strategies patients reported receiving were more important in predicting 30-day hospital readmissions than strategies that hospitals reported delivering, highlighting the importance of patient experience as a driver of outcomes.
Our study included a comprehensive survey development and testing process that included extensive involvement and input from patients, caregivers, and researchers throughout the entire process. In addition, our study’s large-scale data collection spanned almost an entire year with thousands of patients and caregivers. While there are numerous existing measures that assess patient experience with healthcare delivery in various settings of care, there are a limited number of measures that specifically focus on transitions of care, and even fewer that obtain a family caregiver perspective. In addition, few surveys focus on the transitional care services provided both in the hospital and in the outpatient setting after patients return home.
In a systematic review of research on the quality of transitional care interventions, Allen et al.  concluded that there was a need for improved understanding and evidence about the quality of transitional care for older patients and their caregivers. In particular, they called for more research on patient and caregiver experiences, caregiver burden and support, and emotional support for older patients and their caregivers during care transitions. Our survey study involving Medicare beneficiaries and dual-eligible patients and their caregivers provides much-needed measurement tools to enable future research in these areas to fill these important gaps, especially in the measurement of family caregiver experiences with care transitions .
The patient and caregiver surveys developed by our study can be used independently to assess patient and caregiver experiences with care transitions, but the parallel focus of surveys also allows for more nuanced comparisons. Patients can identify their family caregivers so patient and caregiver experiences on equivalent items can be compared, as well as comparisons between caregivers that provided assistance during the patient’s hospital stay and those that assisted the patient at home. The corresponding nature of the outcome composite measures and single-item measures in the patient and caregiver surveys adds new possibilities for examining associations among these critical perspectives on care transitions.
Strengths and limitations
Our study’s strengths include broad input from a Stakeholder Advisory Group (SAG), which included patient and caregiver representatives, a Scientific Advisory Council (SAC), and the Project ACHIEVE research team, at key steps in the survey development and testing process which enabled us to incorporate comprehensive content about transitional care services received both in the hospital and at home. In addition, our iterative development process involved extensive cognitive testing prior to main data collection. Furthermore, caregivers were identified by and linked to patients rather than using unrelated samples of patients and caregivers. Finally, we conducted our study with a large sample of hospitals, patients, and caregivers over an extended period of data collection of 49 weeks.
There were also several limitations of the study. The 51-day wait time constraint before we could begin patient data collection, that was imposed to avoid overlap with Hospital CAHPS patient experience data collection, is very likely to have affected patient recall of their hospital experiences. While we tried to overcome that limitation by surveying T1 caregivers shortly after the patient’s discharge, the caregivers’ responses are not a proxy for patients since they are from a different perspective. Although we had high response rates for patients, caregiver response rates were lower and because patients did not always identify caregivers, the actual number of caregiver responses was low relative to patient responses. In addition, although we had a large sample of hospitals, it is possible that those hospitals that agreed to participate in the study may already have been more engaged in transitional care than hospitals not included in the study and therefore led to more positive survey results. Finally, the majority of patient respondents were white (78%) and high-school educated or higher (85%) so the results obtained in our study may not be as generalizable to more diverse patient populations.
Psychometric analyses provided overall support for the three outcome composite measures and single-item measures in the final patient and caregiver surveys. The final patient, Time 1, and T2 caregiver surveys are psychometrically sound and can be used by health systems, hospitals and healthcare researchers to assess care transitions and related outcomes. Results from these surveys can be used to improve care transitions and outcomes, focusing on what matters most to patients and their family caregivers.
Availability of data and materials
The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.
The 43 hospitals include two locations of the same hospital, which were counted as separate entities for the purposes of psychometric analysis, but considered a single site in the larger Project ACHIEVE analytic study.
“Applicable to all” indicates if a question was applicable to all respondents, excluding questions skipped based on previous responses.
Caregiver Effort/Stress was reverse-coded so “No/A little effort” and “Not at all/Somewhat stressful” were higher/more positive scores.
Achieving patient-centered care and optimized health in care transitions by evaluating the value of evidence
American hospital association
Consumer assessment of healthcare providers and systems
Community-based care transitions program
U.S. centers for medicare and medicaid services
Exploratory factor analysis
Hospital consumer assessment of healthcare providers and systems
Institutional review board
Over the counter
Patient-centered outcomes research institute
Patient-reported outcomes measurement information system
Scientific advisory council
Stakeholder advisory group
Naylor M, Keating S. Transitional Care: Moving patients from one care setting to another. Am J Nurs. 2008;108(9 Suppl):58–63. https://doi.org/10.1097/01.NAJ.0000336420.34946.3a.
Mansukhani RP, Bridgeman MB, Candelario D, Eckert LJ. Exploring Transitional Care: Evidence-Based Strategies for Improving Provider Communication and Reducing Readmissions. Pharm Ther. 2015;40(10):690–4. PMID: 26535025; PMCID: PMC4606859. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4606859/.
Wolf J, Niederhauser V, Marshburn D, LaVela S. Defining patient experience. Patient Exp J. 2014;1(1):7–14. https://doi.org/10.35680/2372-0247.1004 Available at: https://pxjournal.org/journal/vol1/iss1/3.
Glickman SW, Boulding W, Manary M, Staelin R, Roe MT, Wolosin RJ, et al. Patient satisfaction and its relationship with clinical quality and inpatient mortality in acute myocardial infarction. Circ Cardiovasc Qual Outcomes. 2010;3:188–95.
Meterko M, Wright S, Lin H, Lowy E, Cleary PD. Mortality among patients with acute myocardial infarction: The influences of patient-centered care and evidence based medicine. Health Serv Res. 2010;45(5 Pt 1):1188–204.
Sorra J, Khanna K, Dyer N, Mardon R, Famolaro T. Exploring relationships between patient safety culture and patients’ assessments of hospital care. J Patient Saf. 2012;8(3):131–9. https://doi.org/10.1097/PTS.0b013e318258ca46.
Boulding W, Glickman SW, Manary MP, Schulman KA, Staelin R. Relationship between patient satisfaction with inpatient care and hospital readmission within 30 days. Am J Manag Care. 2011;17(1):41–8.
Almborg AH, Ulander K, Thulin A, Berg S. Discharge planning of stroke patients: the relatives’ perceptions of participation. J Clin Nurs. 2009;18(6):857–65. https://doi.org/10.1111/j.1365-2702.2008.02600.x.
Mitchell SE, et al. Care transitions from patient and caregiver perspectives. Ann Fam Med. 2018;16(3):225–31. https://doi.org/10.1370/afm.2222.
Plank A, Mazzoni V, Cavada L. Becoming a caregiver: new family carers’ experience during the transition from hospital to home. J Clin Nurs. 2012;21(13-14):2072–82. https://doi.org/10.1111/j.1365-2702.2011.04025.x.
Li J, Brock J, Jack B, Mittman B, Naylor M, Sorra J, et al. Project ACHIEVE—using implementation research to guide the evaluation of transitional care effectiveness. BMC Health Serv Res. 2016;16(1):70. https://doi.org/10.1186/s12913-016-1312-y.
Li J, Du G, Clouser J, Stromberg A, Mays G, Sorra J, Brock J, Davis T, Mitchell S, Nguyen H, Williams M. Improving evidence-based grouping of transitional care strategies in hospital implementation using statistical tools and expert review. BMC HSR. 2021;21(35). https://doi.org/10.1186/s12913-020-06020-9.
Naylor MD, Shaid EC, Carpenter DC, Gass B, Levine C, Li J, et al. Components of Comprehensive and Effective Transitional Care. J Am Geriatr Soc. 2017;65(6):1119–25. https://doi.org/10.1111/jgs.14782. PMID: 28369722 PMCID: PMC5497308.
Hansen LO, Greenwald JL, Budnitz T, Howell E, Halasyamani L, Maynard G, et al. Project BOOST: effectiveness of a multihospital effort to reduce rehospitalization. J Hosp Med. 2013;8(8):421–7. https://doi.org/10.1002/jhm.2054.
Jack BW, Chetty VK, Anthony D, Greenwald JL, Sanchez GM, Johnson AE, et al. A reengineered hospital discharge program to decrease rehospitalization: a randomized trial. Ann Intern Med. 2009;150(3):178–87. https://doi.org/10.7326/0003-4819-150-3-200902030-00007.
Coleman EA, Parry C, Chalmers S, Min SJ. The care transitions intervention: results of a randomized controlled trial. Arch Intern Med. 2006;166(17):1822–8. https://doi.org/10.1001/archinte.166.17.1822.
Naylor MD, Brooten D, Campbell R, Jacobsen BS, Mezey MD, Pauly MV, et al. Comprehensive discharge planning and home follow-up of hospitalized elders: a randomized clinical trial. JAMA. 1999;281(7):613–20. https://doi.org/10.1001/jama.281.7.613.
Coleman EA, Mahoney E, Parry C. Assessing the quality of preparation for posthospital care from the patient's perspective: the care transitions measure. Medical Care. 2005;43(3):246–55. Care transitions measure® (CTM-15) available at www.caretransitions.org.
Hibbard JH, Stockard J, Mahoney ER, Tusler M. Development of the patient activation measure (PAM): conceptualizing and measuring activation in patients and consumers. Health Serv Res. 2004;39(4):1005–26.
Consumer Assessment of Healthcare Providers and Systems (CAHPS®) Clinician & Group Survey 3.0. Available at https://www.ahrq.gov/cahps/surveys-guidance/cg/index.html. Accessed June 2021.
Consumer Assessment of Healthcare Providers and Systems (CAHPS®) Home Health Care Survey. Available at https://homehealthcahps.org/Survey-and-Protocols/Survey-Materials. Accessed June 2021.
Cella D, Riley W, Stone A, Rothrock N, Reeve B, Yount S, et al. The patient-reported outcomes measurement information system (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005–2008. J Clin Epidemiol. 2010;63(11):1179–94. https://doi.org/10.1016/j.jclinepi.2010.04.011.
Hays RD, Bjorner JB, Revicki DA, Spritzer KL, Cella D. Development of physical and mental health summary scores from the patient-reported outcomes measurement information system (PROMIS) global items. Qual Life Res. 2009;18(7):873–80 whereas the sleep outcome was based on the PROMIS Sleep Disturbance and Sleep-Related Impairment Item Banks Adult (PROMIS http://www.healthmeasures.net/explore-measurement-systems/promis/intro-to-promis/list-of-adult-measures).
Yu L, Buysse DJ, Germain A, Moul DE, Stover A, Dodds NE, et al. Development of short forms from the PROMIS™ sleep disturbance and sleep-related impairment item banks. Behav Sleep Med. 2012;10(1):6–24. https://doi.org/10.1080/15402002.2012.636266.
Elmstahl S, Malmberg B, Annerstedt L. Caregiver's burden of patients 3 years after stroke assessed by a novel caregiver burden scale. Arch Phys Med Rehabil. 1996;77(2):177–82. https://doi.org/10.1016/S0003-9993(96)90164-1.
Pearlin LI, Mullan JT, Semple SJ, Skaff MM. Caregiving and the stress process: an overview of concepts and their measures. Gerontologist. 1990;30(5):583–94. https://doi.org/10.1093/geront/30.5.583.
Zarit SH, Reever KE, Bach-Peterson J. Relatives of the impaired elderly: correlates of feelings of burden. Gerontologist. 1980;20(6):649–55. https://doi.org/10.1093/geront/20.6.649.
Scott AM, Li J, Oyewole-Eletu S, Nguyen HQ, Gass B, Hirschman KB, et al. Understanding facilitators and barriers to care transitions: insights from project ACHIEVE site visits. Jt Comm J Qual Patient Saf. 2017;43(9):433–47. https://doi.org/10.1016/j.jcjq.2017.02.012.
Centers for Medicare & Medicaid Services. CAHPS® Hospital Survey (HCAHPS) Quality Assurance Guidelines V12.0. 2017. p 152–158. http://www.hcahpsonline.org/globalassets/hcahps/quality-assurance/qag_v12.0_2017.pdf.
Centers for Medicare & Medicaid Services. Hospital Consumer Assessment of Healthcare Providers and Systems (CAHPS) Tables on HCAHPS On-Line. A note about HCAHPS “Boxes.” Available at https://www.hcahpsonline.org/en/summary-analyses/. Accessed June 2021.
Famolaro T, Hare R, Yount ND, Fan L, Liu H, Sorra J. Surveys on Patient Safety CultureTM (SOPS®) Hospital Survey 2.0: 2021 User Database Report. Prepared by Westat, Rockville, MD, under Contract No. HHSP233201500026I/HHSP23337004T. Rockville: Agency for Healthcare Research and Quality; 2021. AHRQ Publication No. 21-0017. Available at https://www.ahrq.gov/sites/default/files/wysiwyg/sops/quality-patient-safety/patientsafetyculture/hsops2-database-report-part-I.pdf. Accessed June 2021.
Stevens JP. Applied multivariate statistics for the social sciences. 4th ed. Mahwah: Lawrence Erlbaum; 2002.
Nunnally JC, Bernstein IH. Psychometric theory (3rd ed.). New York: McGraw-Hill; 1994.
Hinkle DE, Wiersma W, Jurs SG. Applied statistics for the behavioral sciences. 5th ed. Boston: Houghton Mifflin; 2003.
Allen J, Hutchinson AM, Brown R, Livingston PM. Quality care outcomes following transitional care interventions for older people from hospital to home: a systematic review. BMC Health Serv Res. 2014;14:346. https://doi.org/10.1186/1472-6963-14-346.
Gillick MR. The critical role of caregivers in achieving patient-centered care. JAMA. 2013;310(6):575–6. https://doi.org/10.1001/jama.2013.7310.
We would like to acknowledge PCORI for funding the project, the entire Project ACHIEVE research team, Stakeholder Advisory Board, Scientific Advisory Council, the 43 participating hospitals for their efforts in recruiting patients, and the patient and family caregivers who participated in the study.
Research reported in this manuscript was funded through a Patient-Centered Outcomes Research Institute (PCORI) Award (TC-1403-14049). The statements in this manuscript are solely the responsibility of the authors and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute (PCORI), its Board of Governors or Methodology Committee.
Ethics approval and consent to participate
Institutional review boards at the University of Kentucky (14–0789-F3R, 15–0298-F3R, 47657), Kaiser Permanente Southern California (10594), and Westat (FWA 00005551) reviewed and approved the study protocol. HIPPA authorization was obtained from all patients when recruited from hospitals. Informed consent was obtained from all patient and caregiver telephone survey participants. Mail-based patient surveys were prefaced with a cover sheet containing the elements of informed consent.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Sorra, J., Zebrak, K., Carpenter, D. et al. Development and psychometric properties of surveys to assess patient and family caregiver experience with care transitions. BMC Health Serv Res 21, 785 (2021). https://doi.org/10.1186/s12913-021-06766-w
- Transitional care
- Patient experience
- Surveys and questionnaires
- Quality of healthcare