Skip to main content

Table 1 Studies utilizing the Delphi technique to develop content for checklists used in clinical settings

From: Developing content for a process-of-care checklist for use in intensive care units: a dual-method approach to establishing construct validity

Study Sample Purpose / Method Findings* / Critique
Setting n / cohort
Huang, Lin & Lin. (Taiwan) [13] College of Nursing 14 / 20 invited panel members accepted; 10 scholars in relevant fields of expertise, 4 clinical nurses. To develop content for a fall-risk checklist
Framework presented to panel who were asked to review a 4-point Likert scale checklist (from strong agreement to strong disagreement), submit comments & provide revision suggestions
Likert scale used to calculate content validity index (CVI) score for each item, rated along 3 dimensions i.e. content importance, appropriateness and discreteness
Scoring calculation method detailed
70% of potential panel members accepted, 3 rounds required, completed over 4-month period
Response rates: round 1, 78.5% (3 withdrew); 2, 91% (1 withdrew); 3, 100%
Results of each round reported in summarized format
Key suggestions & resulting refinements for each round provided
Changes to domains and checklist processes documented
CVI scores for each domain along the 3 dimensions and total score (range 0.84 – 1.00) in last review round provided
Information not provided: complete checklist, criteria for deleting items, variation in responses & scores to individual items (results summarized by domain)
Morgan et al. (Canada) [19] 2 independent academic centers 5 anesthesiologists To develop a simulation performance checklist to evaluate performance of practicing anesthesiologists, using a computer-based Delphi technique
Checklist items generated by participants after reading 2 pre-prepared scenarios, error weighting assigned to each item based on risk level
Responses collated anonymously & emailed back to participants asking them to check off items to retain or delete & to (re)assign weightings
Process repeated until no further items added, deleted or changes to weightings
A-priori decision to delete responses endorsed by ≤ 20% respondents
100% response rate
Required four rounds to reach consensus
Participants generated 104 items for scenario 1 & 99 items for scenario 2
Final percentage weightings for checklist items provided
Small sample size
Information not provided: variation in error weighting to individual items, key study timeframes e.g. time from survey distribution to response
Hart & Owen. (Australia) [17] Anesthesia Department at a tertiary hospital Not reported - consultants with special interest in obstetric anesthesia To generate checklist items for use prior to commencing non-emergency Cesarean delivery under general anesthesia
Participants contacted via email and remained anonymous to other participants
Two questionnaires were circulated
Two questionnaires were circulated
Results of 2 questionnaires informed construction of checklist items
Items were later divided into four sub-categories
Key information not reported: sample size; contents of questionnaires; response rates; how responses were used to inform 2nd round questionnaire & construct final checklist items e.g. not known whether pre-defined consensus methods were used, how checklist items were grouped & ordered
Ursprung et al. (USA) [16] 20-bed tertiary care medical-surgical neonatal ICU Not reported - experts in neonatology, pediatrics, health services research, systems engineering, infection control, advanced practice nursing To develop a patient safety audit checklist for PICUs
Questions formatted into a checklist and refined iteratively by consensus
Participants responses based on potential clinical impact of mistakes, system failures, perceived frequency
Checklist reviewed and refined by physicians and nursing staff from study NICU to ensure relevance locally
36 audit questions representing a broad range of errors associated with NICU patient care generated
Questions later divided into 2 categories
Information not reported: sample size and participant designations; contents of questionnaire; number of rounds required; method of obtaining consensus; how checklist items were further reviewed and refined for relevance by local PICU staff after consensus was reached; method of categorization
Pronovost et al. (USA) [18] 13 adult medical & surgical ICUs in urban teaching & community hospitals Interviews: 8 nurses & 5 ICU physicians
Focus group: not reported
Development and pilot testing of daily goals form
Validity of measures: obtaining agreement from ICU physicians and quality experts who developed the measures; semi-structured interviews with nurses & physicians who piloted the measures
Face validity: focus group of physicians and nurses from 13 participating ICUs
Validity of measures: ICU physicians and quality experts unanimously agreed process measures addressed important aspects of ICU quality
Focus group: participants believed measures ‘evaluated the domain of quality they intended to measure and identified important opportunities to improve quality’ [18], p.154
Information not provided: sample sizes for development of measures and focus group; content for focus group discussion & semi-structured interviews; how qualitative data analyzed and interpreted