Skip to main content

Table 1 Studies utilizing the Delphi technique to develop content for checklists used in clinical settings

From: Developing content for a process-of-care checklist for use in intensive care units: a dual-method approach to establishing construct validity

Study

Sample

Purpose / Method

Findings* / Critique

Setting

n / cohort

Huang, Lin & Lin. (Taiwan) [13]

College of Nursing

14 / 20 invited panel members accepted; 10 scholars in relevant fields of expertise, 4 clinical nurses.

To develop content for a fall-risk checklist

Framework presented to panel who were asked to review a 4-point Likert scale checklist (from strong agreement to strong disagreement), submit comments & provide revision suggestions

Likert scale used to calculate content validity index (CVI) score for each item, rated along 3 dimensions i.e. content importance, appropriateness and discreteness

Scoring calculation method detailed

70% of potential panel members accepted, 3 rounds required, completed over 4-month period

Response rates: round 1, 78.5% (3 withdrew); 2, 91% (1 withdrew); 3, 100%

Results of each round reported in summarized format

Key suggestions & resulting refinements for each round provided

Changes to domains and checklist processes documented

CVI scores for each domain along the 3 dimensions and total score (range 0.84 – 1.00) in last review round provided

Information not provided: complete checklist, criteria for deleting items, variation in responses & scores to individual items (results summarized by domain)

Morgan et al. (Canada) [19]

2 independent academic centers

5 anesthesiologists

To develop a simulation performance checklist to evaluate performance of practicing anesthesiologists, using a computer-based Delphi technique

Checklist items generated by participants after reading 2 pre-prepared scenarios, error weighting assigned to each item based on risk level

Responses collated anonymously & emailed back to participants asking them to check off items to retain or delete & to (re)assign weightings

Process repeated until no further items added, deleted or changes to weightings

A-priori decision to delete responses endorsed by ≤ 20% respondents

100% response rate

Required four rounds to reach consensus

Participants generated 104 items for scenario 1 & 99 items for scenario 2

Final percentage weightings for checklist items provided

Small sample size

Information not provided: variation in error weighting to individual items, key study timeframes e.g. time from survey distribution to response

Hart & Owen. (Australia) [17]

Anesthesia Department at a tertiary hospital

Not reported - consultants with special interest in obstetric anesthesia

To generate checklist items for use prior to commencing non-emergency Cesarean delivery under general anesthesia

Participants contacted via email and remained anonymous to other participants

Two questionnaires were circulated

Two questionnaires were circulated

Results of 2 questionnaires informed construction of checklist items

Items were later divided into four sub-categories

Key information not reported: sample size; contents of questionnaires; response rates; how responses were used to inform 2nd round questionnaire & construct final checklist items e.g. not known whether pre-defined consensus methods were used, how checklist items were grouped & ordered

Ursprung et al. (USA) [16]

20-bed tertiary care medical-surgical neonatal ICU

Not reported - experts in neonatology, pediatrics, health services research, systems engineering, infection control, advanced practice nursing

To develop a patient safety audit checklist for PICUs

Questions formatted into a checklist and refined iteratively by consensus

Participants responses based on potential clinical impact of mistakes, system failures, perceived frequency

Checklist reviewed and refined by physicians and nursing staff from study NICU to ensure relevance locally

36 audit questions representing a broad range of errors associated with NICU patient care generated

Questions later divided into 2 categories

Information not reported: sample size and participant designations; contents of questionnaire; number of rounds required; method of obtaining consensus; how checklist items were further reviewed and refined for relevance by local PICU staff after consensus was reached; method of categorization

Pronovost et al. (USA) [18]

13 adult medical & surgical ICUs in urban teaching & community hospitals

Interviews: 8 nurses & 5 ICU physicians

Focus group: not reported

Development and pilot testing of daily goals form

Validity of measures: obtaining agreement from ICU physicians and quality experts who developed the measures; semi-structured interviews with nurses & physicians who piloted the measures

Face validity: focus group of physicians and nurses from 13 participating ICUs

Validity of measures: ICU physicians and quality experts unanimously agreed process measures addressed important aspects of ICU quality

Focus group: participants believed measures ‘evaluated the domain of quality they intended to measure and identified important opportunities to improve quality’ [18], p.154

Information not provided: sample sizes for development of measures and focus group; content for focus group discussion & semi-structured interviews; how qualitative data analyzed and interpreted