Study design
This study forms part of a research project focused on the development and implementation of ‘surgical patient’s safety checklist’ (PASC). We have previously interviewed surgical patients and perioperative healthcare professionals and identified risk areas before and after surgery as well as how these risks can be reduced by patient participation [3]. The PASC content is based on the findings from our previous qualitative study, together with the development and validation process that we report here.
This was a prospective study consisting of the development and validation of PASC, and a reliability analysis of the checklist items. PASC was developed and validated in Norwegian, however for this publication it was translated into English to enable reporting and wider sharing globally. The English translation was performed by a person fluent in both languages and back translated into Norwegian by a healthcare professional and a surgical patient and only minor word differences were detected in the Norwegian back translation (which means the checklist as reported here is an accurate representation of the checklist evaluated in the study).
The development consisted of a consensus process including patients’ representatives and multi-professional healthcare personnel. In the validation process, elective surgical patients from six surgical wards received PASC two to six weeks before surgery. The patients were asked to use the checklist as well as to score the importance of each checklist item; these scores were subsequently used to produce an item content validation index (I-CVI). A small number (n = 10) of surgical patients were also interviewed in focus groups to investigate the face validity of PASC. The finalisation of PASC is based on the patients’ I-CVI scores, risk assessment of items with low I-CVI score using Healthcare Failure Mode and Effect Analysis (HFMEA) hazard scoring and a consensus process [24]. The study followed the consolidated criteria guideline for reporting of intervention development studies (GUIDED) [25].
Setting and participants
Study participants were recruited from two Norwegian hospitals; one tertiary teaching hospital and one central community hospital, which cover populations of 1.1 million and 110,000 inhabitants, respectively. Among all eligible surgical wards at the two hospitals, six surgical specialties were invited to take part in the study. The selection of surgical wards was based on a randomization for an upcoming trial of the clinical effectiveness of PASC. This included Ear, Neck, Throat (ENT)/Maxillo-Facial; Cardio-thoracic; Neuro-; Breast- and Endocrine-; Gastrointestinal; and General surgery. All departments agreed to participate.
Healthcare personnel included in the development consensus process were service managers, surgeons, ward doctors, ward nurses, and patients’ representatives from the surgical wards included in the study. Additionally, anesthesiologists, nurse anesthetists, intensive care nurses, specialist dietitians, pharmacists and general practitioners were included. A safety expert from the aviation industry, and hospital communication advisors were also consulted on the wording, layout and design of the checklist.
In the content validation process, elective surgical patients having surgery in the same six surgical specialties were invited to participate. Inviting surgical patients as lay experts ensured that they were an integral part of the checklist development [26]. Inclusion criteria for the participants were: elective surgical patients aged 18 years or older, cognitively able to complete the checklist, living at home, able to give informed consent and fluent in Norwegian. Participants were recruited within a time period of two to twelve weeks before surgery, in cooperation with the nurses and surgeons at each ward. The time for when patients received the checklist before surgery depended on their severity of their disease and urgency of surgery. Patients returned the completed checklists before hospital discharge in collaboration with ward nurses and secretaries. If participants had forgotten to return PASC at the time of discharge, a reminder letter was sent to their home address with an enclosed prepaid envelope to return their completed PASC.
Checklist development
In a previous study, focus group interviews of patients and healthcare workers were utilised to identify risk areas for complications before and after surgery [3]. Subsequent PASC item development focused on the risk areas identified in that study.
To develop the checklist, we applied the recommended guidance for developing and validating checklists for patients [18]. The development process included a consensus-based process and a validation process with statistical testing of content validity and reliability. The checklist development and consensus process before the checklist validation lasted from December 2018 through June 2019. The steps of the PASC development and validation process are described in Fig. 1.
Content of preoperative PASC before content validation
The preoperative PASC included 32 items covering issues patients should consult their general practitioner for prior to surgery, such as medication usage, medical history, need for multi-resistant bacteria testing after overseas treatments and/or hospitalisation and lifestyle issues. This checklist also encouraged patients who have not seen a dentist in the previous 12 months to do so and to read all information given to them related to their surgery. Further, it included information and preparations for patients need to be aware of two weeks prior to surgery. Lastly, the preoperative PASC included issues patients need to be aware of the day before, and immediately before surgery.
Content of postoperative PASC before content validation
The postoperative PASC contained 26 items that included information about risk factors and complications that may arise, and what patients or families/relatives should do if such complications occur. Secondly, items relating to the importance of physical activity after surgery, and reminders to patients to adhere to important restrictions. Thirdly, this checklist included medication safety information before discharge from hospital and other information, like gastro-intestinal function, after surgery. Lastly, this checklist also covered further treatment plans and follow up after surgery.
Depending on a patient’s answer (yes/no) to each item on PASC, they receive clear instructions on what actions should be taken if needed. Due to the checklists’ large number of items, the items were structured into sections of no more than eight items for ease of completion. Each section had a heading that described the item content in each section, as recommended by the guideline for developing and validating checklists for patients [18]. The checklists were designed to follow the patient surgical pathway and to be used over 2-6 weeks before surgery and also before hospital discharge.
Content validity and reliability
After establishing the PASC content, elective surgical patients used the checklist (Norwegian version) and scored each item to content-validate the checklist [27]. The data on patients, checklist usage and I-CVI were collected over a period of 14 months (August 2019 to September 2020). Descriptive statistics were used to describe patient demographic information and a chi-squared test was performed to investigate any demographic differences between responders and non-responders. Participants were given PASC, consisting of two parts; one prior surgery and one before discharge. While using the two checklist parts (a total of 58 items) the patients rated each item from not relevant to very relevant on a four-point scale (1 = not relevant, 2 = somewhat relevant, 3 = quite relevant, 4 = highly relevant) [27]. For each item, a final I-CVI was calculated by including the number of patients who rated the item 3 or 4 and dividing that number by the total number of experts rating each item [28]. I-CVI scores ≥0.78 were considered satisfactory; items that reached this score were kept unchanged in the final version of the checklists [29].
Items with scores < 0.78 were subsequently risk-assessed by the research team using the HFMEA Hazard Scoring Matrix [24]. The risk of possible complications related to each reviewed item was estimated based on the rated frequency and the potential severity of the hazard. Hazard scores can range from 1 to 16, where 1- 4 indicates low frequency/impact and 8 to 16 indicates high frequency/impact, as described by the standard HFMEA Hazard Scoring Matrix [24]. Lastly, a final consensus and revision process on PASC items that received I-CVI > 0.78 and hazard scores < 8 was performed as recommended by Polit and Beck [29]. Tables 3 and 4 describe which items were kept unchanged; which items were revised due to hazard scoring and consensus; and which items were ultimately removed as a result of this development and scoring process.
To investigate the total content validity of the checklist, the Averaging Scale-level Content Validity Index (S-CVI/Ave) of items scoring 3-4 was calculated for both parts of the checklist before and after revision based on I-CVI. S-CVI/Ave was calculated by summing all I-CVI scores and then dividing by the total numbers of items [29] (Tables 3 and 4). Descriptive analyses of the I-CVI and S-CVI/Ave were performed in STATA version SE 16.1. (StataCorp. 2019. College Station, TX: StataCorp LLC).
Intraclass Correlation Coefficients (ICC) were used to assess the PASC checklist reliability (internal consistency). ICC estimates and 95% confidence intervals were calculated using SPSS Statistical Package Version 26 (SPSS, Inc., Chicago, IL) based on mean-rating, two-way random-effects model [30]. Variables with missing values > 50% were removed from the ICC analyses (n = 23), and variables with missing values < 50% were replaced with mean values based on multiple imputation [31].
Face validity
After PASC was used and validated by surgical patients, ten patients were invited to attend small focus group interviews (two to five surgical patients in each group). The focus groups interviews lasted for up to 60 min and performed by one interviewer and one moderator (MR and KH). These consisted of a purposive sample of surgical patients three to eight weeks post-surgery from: Ear, Neck, Throat (ENT)/Maxillo-Facial; Cardio-thoracic; Neuro-; Breast- and Endocrine surgery. The focus group interviews were carried out in hospitals according to COVID-19 regulations. Focus group interviews were semi-structured, driven by a topic guide based on the checklist items, which was first piloted on patient representatives. They were recorded and transcribed verbatim for analysis. Qualitative content analysis was used to identify codes and categories from condensed patients meaning units to assess face validity as described in Fig. 2 under results [32].