Data collection
Predictors, hospital outcomes, 7-day follow-up calls
All study nurses will be trained and well versed on the standard operating procedures to facilitate standardization of all measurements. Study nurses will collect data using a custom-built Android application on a Samsung Galaxy Tablet A8. The list of predictors to be collected include clinical signs and symptoms, demographic/sociodemographic data, and pregnancy/birth information (see Additional file 1). Similarly, designated study nurses will obtain hospital outcomes (see Additional file 1) from patient records and enter them into the application on the Samsung Galaxy Tablet. Study nurses will conduct 7-day follow up calls in accordance with the standard operating procedures and enter the data into the application on the Samsung Galaxy Tablet.
Time outcomes
Timing tracking will be automated using customised RFID (see RFID system section for details).
Usability testing data
Think Aloud transcriptions and observer checklists will be entered into a computer and uploaded to REDCap. The Triage Tool Training Questionnaire (Additional file 2) will be captured on paper and stored in a locked cabinet, in a locked room in our research spaces nearby the study sites.
Post-study healthcare worker satisfaction survey
After completion of Phase III of the study, healthcare workers that participated in the usability testing initiative will be invited to complete a Healthcare Worker Satisfaction Survey (Additional file 3). This will be used to generate a report that provides insight on the overall perception of health worker’s experiences with the digital triage tool.
Data management
Data collection infrastructure
Study Nurses will collect data using a custom-built Android application, created using LambdaNative (lambdanative.org), the open-source cross-platform toolkit developed internally at BC Children’s Hospital Research institute. All data entered into the mobile application is stored in an encrypted database using the encryption cipher Rabbit. Access to the tablet and application is secured by passwords; without using the application, the encrypted files are not readable. The Masimo iSpO2® Pulse Oximeter with Micro USB Connector will be used to collect pulse oximetry and heart rate (including 30 s of raw plethysmographic data) and the Masimo Caregiver™ non-contact thermometer will be used to measure core temperature. The data collection application also contains complex error checking specific to the survey questions such as date inconsistency checks and ensures only relevant data items are collected, by dynamically hiding redundant questions.
Due to the complex nature of a large multi-center study, data will be uploaded directly from the Android tablets to REDCap (Research Electronic Data Capture, https://projectredcap.org/). REDCap is a secure web-based application designed to support data capture for research studies and it has been used for over 300,000 projects, in over 100 countries, including prior studies in Uganda [13]. Encrypted data will be stored for less than 14 days after completion of data collection on the tablets. In Kenya, the data will be directly uploaded weekly (depending on internet availability) over a secured internet connection to KTWRP servers, where it will be stored. A deidentified copy will be sent to the central study server at the BC Children’s Hospital Research institute where data will be checked for completeness and consistency with data definitions. In Uganda, the data will be sent to the central study server at the BC Children’s Hospital Research Institute. After this upload, the data on the tablets will be deleted. Each subject will be given a unique number and all data will be connected to this unique number. Using REDCap limits the amount of paper-based data, further ensuring data integrity and safety. The uploaded data will be accessible to only study team members with secure access to the server.
Data collected during follow-up interviews conducted by phone or in person will also be collected electronically and shared in the same secured manner. Personal identifiers are required for the collection of admission data and follow-up data. The data collection application contains several forms. All identifiers are collected on a single form, separate from the other forms containing non-identifying information, and stored in a separate and restricted REDCap form. Access to identifiers will be limited to those requiring this data for follow-up (i.e. only study personnel involved in follow-up or data verification). No analysts, co-investigators or principal investigators not directly involved in the follow-up or data verification will have access to this data. Access to REDCap will require 2-way authentication: in addition to the normal password process, a secure code (sent via SMS to the user) will be required for access to this data.
Paper based data collection items include consent forms and research assistant field notes (which do not contain identifiers). These will be stored in a locked cabinet, in a locked at our research spaces next to each study site.
Health intervention infrastructure
The bundle of care during the intervention will include triage in the OPD using an additional custom Android web-app, running locally on the device with no required Internet connection. As with the data collection app, this application will be password protected and data will be stored encrypted on the device. Following each triage, the triage data will be sent to a local low-cost Unix server Soekris box based in a secure room at each site. Data will be sent through encrypted HTTPS requests to server-side PHP scripts, which insert it within MySQL tables.
The clinician dashboard that will be used to clinically manage all children in the OPD who have been triaged will be implemented as a password protected website accessible by registered medical staff on any computer or tablet on the local network. As with the triage app, this website runs completely locally, independent of an outside Internet connection and is not accessible from off-site. The dashboard is implemented using the Laravel (https://laravel.com), a PHP web framework, which queries the MySQL server tables securely. Personal identifiers will be collected in the triage app and sent to the server for display on the dashboard as is necessary for correct identification of patients, but such information will never leave the hospital site. As the study hospitals already have computers in their OPDs, use of the clinical dashboard will not be extra work for the hospital staff or interfere with other tasks.
Statistical analysis
Model development
The predictive model for severe infection and sepsis in children will be developed based on the need for hospital admission of 24 h or longer. To find the best performing, and most parsimonious model, a variety of model building techniques will be compared. These will include but are not limited to: stepwise regression based on Akaike’s Information Criterion (AIC), penalized regression via elastic net, and random forest [21]. Models will be compared based on the area under the receiver operating curve, and the specificity achieved at high (80–90%) sensitivity thresholds. Models within 10% of the best performing model will be considered. Final model selection will depend on parsimony, availability of predictors across sites (based on resources, cost and feasibility of collection), discrimination and calibration. All model comparisons will take place within an internal validation resampling framework such as cross validation or bootstrapping. Risk thresholds to stratify participants into triage categories (emergency, priority, and queue) be selected based on sensitivity analysis and expert opinion. To account for geographical differences and disparities in disease prevalence (high malaria prevalence in Jinja, Uganda), we will use net reclassification improvement to optimize and re-calibrate (if required) the prediction model [22].
RFID system validation
We will validate the RFID technology against a human timekeeper control for the time to bundle of care outcome by investigating bias and accuracy using a Bland Altman analysis. The RFID technology will be considered as interchangeable with a human timekeeper if the 95% limits of agreement fall within +/− 5 min. We chose this cut off as delays in bundle of care delivery below this threshold is unlikely to be of clinical significance. We will use the first 100 pairs of RFID and human measurements to calculate a mean difference and standard deviation of difference between the two measurements. Using these two values and a clinically acceptable limit of 5 min, with an alpha of 0.05 and beta of 0.2, we will calculate the sample size required for the Bland Altmann analysis by methods previously described [23]. If the 95% limits of agreement fall outside of 5 min, the same Bland Altman analysis will be done for the patient arrival time and time of bundle of care delivery; these comprise of the start and end time components of the time to bundle of care outcome. These secondary analyses serve to ascertain which of the two components contains the larger discrepancy between RFID and human measurement. Should these secondary analyses be necessary, implementation issues in the RFID technology will be brainstormed in focus groups with the technology team and addressed on the ground, and we will repeat the same validation process as described. After successful logistical improvements and validation of the RFID technology to within 5 min of a human timekeeper, we will utilize the RFID technology for data collection with a human timekeeper as backup as needed.
Outcomes analysis
We will be conducting our primary analysis using a difference in differences method. For our primary analysis, a logistic regression model for receipt of bundle of care within 1 h will be fit including: 1) phase of study (pre vs. post PRST) and 2) group (intervention versus control) as independent dummy variables and 3) their interaction. The interaction term corresponds to the difference in log-odds between groups in baseline vs intervention and it’s exponent as the ratio of odds ratios. We will summarise this difference by estimating the marginal effect (risk difference) of phase in both the control and intervention groups. The difference of these marginal effects is the so-called ‘difference in difference’ for a binary outcome [24]. Confidence intervals will be based on the delta-method, or bootstrapping, where appropriate. A similar approach will be used for binary secondary outcomes (mortality, readmission), or quantile regression for median time to bundle of care, a non-parametric outcome.
Subgroup analyses specified a priori include Ugandan vs. Kenyan intervention site, young infant (under 2 months old) vs. children under 5 vs. older children, and presentation during day-shift hours versus night-shift hours (if applicable). We will also perform subgroup analyses on each triage category (emergency, priority, queue); patients from the pre-intervention phase will be retroactively categorized into these subgroups for this analysis. These subgroup analyses will be conducted with the inclusion of a three-way interaction between said variable, phase, and group. Statistical significance will be based on a likelihood ratio test comparing models with and without this term. This three-way interaction term would represent potential interactions between different subgroups and our difference in odds ratio obtained in our primary analysis, and results will again be presented on the risk difference scale based on marginal effects. This method of subgroup analysis sometimes termed a difference in difference in differences method.
Although none are expected, if there are any time varying confounders identified during the study that differ between intervention and control (health system or policy changes, natural events, changes in patient populations etc.) they will be included in above models as sensitivity analyses.
As we are continually collecting data regarding predictors for admission alongside the clinical team, there is the possibility that our research activities may interact with the efficiency of delivering the sepsis bundle of care. Although the possibility of such interaction is minimized by our use of dedicated research personnel who would not approach patients who are actively receiving clinical care, any such interactions could potentially limit the external validity of our study to non-research-controlled settings. Therefore, to assess this potential interaction, we will pause the collection of predictor data 1 month before and after implementation of the PRST program. This pause will also occur in the control group in order to replicate any bias caused by changes in our primary outcome attributable to the pause rather than the PRST program. To investigate the potential for this bias, we will treat the collection or non-collection of predictors as a dummy variable for subgroup analysis using the difference in difference in differences method described above.
All analyses will be conducted using R statistical software [25], and an alpha level of 0.05 will be considered statistically significant for all outcome analyses.