KPNC is an integrated healthcare delivery system that includes 21 hospitals and 257 medical offices [17, 20,21,22,23,24] serving ~ 4.5 million Kaiser Foundation Health Plan members in Northern California. The program completed deployment of the Epic EHR (www.epicsystems.com) in all its hospitals and clinics in mid-2010. For this study, we recruited patients at 3 KPNC hospitals (Oakland, San Leandro, Walnut Creek). The study was approved by the KPNC Institutional Review Board for the Protection of Human Subjects.
Patients were required to meet the following eligibility criteria: age ≥ 18 years; English speaker; insurance coverage other than Medicaid (Medi-Cal); inpatient or admitted for observation, other than Labor & Delivery service; current hospitalization began inside a KPNC hospital; discharge without further hospitalization elsewhere; not on infection isolation precautions; no “Comfort Care Only” order in effect at the time of discharge; and cognitively and functionally able to provide informed consent and answer questionnaires. The latter was determined in several ways: no diagnosis of dementia in EHR; ability to answer questions and provide informed consent without a proxy; and was alert, oriented, and approachable as verbally confirmed by the patient’s nurse or other hospital care provider. Since we were trying to interview patients as close to discharge date as possible, only patients with planned discharge that day or the following day were approached.
All hospitalized adults in KPNC are assigned an automated daily Transition Support Level (TSL) score every morning at 0600. The TSL score is based on a patient’s admission acute severity of illness, longitudinal comorbidity score, whether the patient experienced any hospitalizations in the 7 and 30 days preceding the index hospitalization, length of stay (truncated at 30 days), and discharge care directive (“full code” or not) . Patients with a TSL of ≥ 25% on the day of discharge are automatically enrolled in the KPNC Transitions Program, and they receive additional in-person discharge services and extra follow-up calls.
We oversampled patients whose predicted risk was between 15–44% (low-middle range) for two reasons. First, based on published [17, 24] and internal analyses, we knew that most outcomes occurred above the 15% threshold (i.e., higher risk patients are more likely to experience adverse outcomes). Second, we reasoned that the greatest potential benefit would be to reclassify mid-level risk patients into either a higher or lower risk band, since very high-risk patients would be unlikely to have their risk estimates changed with new information and lower risk patients may not require additional intervention or follow-up.
Recruitment and interview procedures (Fig. 1 and Appendix 2)
Prior to recruitment, we obtained permission to approach patients from the study hospitals’ leadership and individual hospital attending physicians. Physician approvals permitted research staff to approach any patient who was under their care for the duration of the study, given that all eligibility and screening criteria were met. Research staff obtained confirmation from the patient’s nurse that the patient was alert, oriented, and approachable. Recruitment was conducted Monday-Friday between 9:00 AM and 5:00 PM from May 7, 2018 to October 31, 2019.
Research staff approached eligible patients and described the study. Patients who agreed to participate confirmed their identity using an authentication process, provided informed consent, signed an authorization form for use and disclosure of their information, and signed an acknowledgment of receipt of the KPNC Research Participants’ Bill of Rights. Patients then completed a staff-administered interview covering basic demographics, social risk factors, and physical and cognitive functioning. Staff used tablets to enter responses into a secure online data management tool separate from the EHR system, and data for the Patient Reported Outcomes Measurement Information System (PROMIS) questionnaires were entered directly into the PROMIS online Assessment Center (www.assessmentcenter.net). Participants received a $10 gift card/code as a token of our appreciation.
Research staff administered interview questionnaires
Our interview questionnaire included the National Institutes of Health’s PROMIS Physical Function and Cognitive Function Abilities Subset questionnaires  and social factors items derived from the KP YCLS item bank. PROMIS measures have a mean of 50 and a standard deviation (SD) of 10 in a referent population. Means above 50 and means within 0.5 SDs below 50 are considered within normal limits for function. For our final analyses, we created six social factor predictor variables based on the YCLS items: (1) relationship/marital status (married or living with partner); (2) food insecure (had worried about running out of food sometimes in the prior 3 months or anticipates having trouble paying for food in next 3 months); (3) housing-related concerns (in a temporary housing situation or homeless, concerned about housing conditions, or anticipates having trouble paying for housing or utilities in next 3 months); (4) financial strain (anticipates having trouble paying for ≥ 1 of 9 basic expenses in next 3 months); (5) transportation difficulties (anticipates problems with transportation during next 3 months); and (6) disability and help status (3-level variable: no health problem or disability that limits normal daily activities, limited but has ready access to help with medical needs and daily activities; limited but lacks ready access to help) (see Appendix 1: Interview Instruments).
Electronic health record data
In addition to the EHR information needed to create the composite outcome (non-elective rehospitalization or 30-day mortality post-discharge), we confirmed that patients had Kaiser Foundation Health Plan membership, and extracted the TSL and its individual components (patients’ longitudinal comorbidity burden, severity of illness, length of stay, past healthcare utilization, and code status [17, 23]). As previously described , all adults with a KPNC medical record number are assigned a monthly COmorbidity Point Score, version 2 (COPS2, based on Centers for Medicare and Medicaid Services Hierarchical Condition Categories), with increasing COPS2 scores associated with increasing mortality risk . Additionally, patients were assigned a Laboratory-based Acute Physiology Score, version 2 (LAPS2)  on admission and every hour after hospitalization. Increasing LAPS2 scores reflect worsening instability – for example, in July 2018, the median hourly LAPS2 among all KPNC intensive care unit patients was 110, whereas the median ward score was 52. It is not possible to admit a patient to KPNC hospitals without specifying code status, which can be subsequently updated. We classified each patient’s care directive as “full code” or “other” (which included “partial code,” “do not resuscitate,” and “comfort care only”) . To compare, we assigned each hospitalization a Charlson Comorbidity Index score (CCI) using the methodology of Deyo et al. .
We extracted age at hospitalization; sex; self-reported race, hospitalization venue (via the ED or not); total index hospital length of stay (LOS); whether a patient experienced any overnight inpatient hospitalization in the first 7 days and separately in the 8 to 30 days preceding the index hospitalization ; discharge disposition (home; regular or custodial skilled nursing facility, SNF; and Home Health services); and referral to hospice. We classified the principal diagnosis codes using the Health Care Utilization Project (www.ahrq.gov/data/hcup) single-level diagnosis clinical classification software categories and clustered them into 30 groups called Primary Conditions [17, 23].
All analyses were done in SAS GRID, version 9.04.01M5P091317 and RStudio, version 1.3.1073. We performed bivariate comparisons using Chi-square, t-tests, and Wilcoxon rank-sum tests, as appropriate. We assessed correlations between continuous variables using Spearman’s rho statistic.
For our principal outcome analyses, we employed logistic regression. The dependent outcome was the same composite outcome (non-elective rehospitalization and/or death within 30 days of hospital discharge) that was used to calibrate the TSL score . The independent variables included age, sex, the 6 above-mentioned social factors predictors, the 2 PROMIS scores, the aggregate TSL score, and all individual components of the TSL. In addition to testing various specific combinations (e.g., TSL + social factors items), we tested a random forest model that included all the individual above-mentioned predictors. We assessed model performance by measuring the area under the receiver operator characteristic curve (c statistic), Nagelkerke pseudo-R , and Brier score [28,29,30]. For all models, the results we report are those obtained after fivefold cross validation.