The key outcomes of this study were the development and validation of a suite of clear and concise statements on nine essential processes of care, to be used as a checklist for supporting practice during daily rounds in an ICU. Study findings added evidence in support of the content validity of the checklist items - particularly the relevance, adequacy, and clarity of checklist statements.
Interviews with local ICU clinicians confirmed the adequacy of content covered by the process-of-care checklist as well as providing initial information pertaining to the practice relevance of each individual statement. These informants also offered important information on the local context, which supported the refinement of checklist statements for inclusion in the first round Delphi survey. These initial revisions provided additional credibility to the Delphi process by ensuring the preliminary statements were relevant to the local ICU.
The modified-Delphi technique used was developed in line with contemporary research guidelines [12, 32] to address the limitations of other research in this field and enhance rigor in this type of study. This was exemplified by the methods used (i.e. incorporation of information obtained from a literature review , a point-prevalence study , and a criterion-related validation study) prior to and during the pre-Delphi interviews that consequently informed revision of the checklist items. This preliminary information was then incorporated into the first Delphi round, as this approach may be more reliable than an open-first round Delphi survey .
Only two rounds were required to reach consensus. When viewed collectively, the findings from both Delphi rounds demonstrated the “stability” of responses, suggesting a reasonable indicator of consensus [25, 32]. Despite almost gaining consensus after the first round, several suggestions were made to improve the clarity for all but one of the statements. After second round responses were collated, all statements had either been accepted without further changes, or suggestions for changes had been integrated into the final statements. This is evidence of: 1) previous work on developing checklist content was a sufficient starting point for this modified-Delphi study; 2) only refinements to the existing statements were required to generate the most clear, concise and instructive statements. It is likely that the preliminary work also ensured quick replies from panel respondents and a shorter time to reach consensus. Other studies, particularly those that generated content from scratch reported much longer study periods [13, 26].
Although there is contention pertaining to acceptable consensus levels, recent recommendations suggested that levels be: established prior to data collection; based on the importance of the research topic; and supported by rational justification . The decision to accept second-round statements with at least 51% agreement was based on the following: 1) majority agreement was more practicable than 100% consensus given there could be countless minor variations of the same statement that met the criteria of being clear, concise and instructive statements; 2) there was near-consensus after the first survey round; and 3) to minimize respondent burden and exhaustion from busy ICU clinicians, and managers, which has also been reported .
Purposive sampling for the Delphi study allowed the selection of experts best able to provide advice on statement development. Similar to a previous study , the use of two expert panels strengthened the validation process. The participation of IC quality committee members lent support to the external validity of the checklist statements i.e. they can be used in all general ICUs as a starting point from which local clinician input can be obtained. The panel of intensivists provided the desired local ICU input, ensuring that terminology was applicable for use in that ICU. Their involvement enabled an opportunity for input into tool development that would be used by themselves or their colleagues in routine practice, and also facilitated engagement in planned future studies.
The Delphi panel size of at least 8 respondents was in line with recommendations that the membership number be relevant to the purpose of the study, the selected design, and data collection time frame [12, 13]. The panel size was also large enough to obtain a substantial amount of useful feedback, and proved adequate for reaching consensus on the wording of checklist statements. A larger sample size may have generated more variations that still met the criteria of being clear, concise and instructive, but this may have prolonged the process unnecessarily, and may have diminished applicability of the statements to the local setting. Similar to other studies using the Delphi technique, treatment of data obtained from panel members was de-identified (i.e. individual responses were not made available to other participants), removing any risk of influence on group conformity, power, and the effect of others on responses .
Unlike previous studies [16–18], results of the Delphi technique were reported for each round, with key suggestions for improvements to the statements reflected in the second round Delphi questionnaire and the final checklist statements. The importance of describing the sampling process in detail has also been emphasized in the literature , and as such, detailed information pertaining to the selection processes and characteristics of the panel members has been reported. This level of data collection and reporting allows for increased transparency for the purposes of study replication and provides evidence of sufficient methodological rigor in developing the checklist statements.
There were limitations to this study. First, the response rate to the Delphi survey was moderate (56% and 50% in the two rounds, respectively). Non-responders were not followed-up further and therefore their reasons for non-participation were unknown. A previous paper  reported a range of response rates from two Delphi studies – the highest response rate was 73% for a second-round Delphi survey of 22 international panelists; the lowest was 39% for a second-round survey of 18 regional panelists. They partly attributed the higher response to pre-selecting panelists that indicated their willingness to participate (which was not the case with other panels they used), resulting in a motivated, committed panel of experts. Response rates to our study were similar to a first-round survey of one of these previous studies  (i.e. 56% of 16 national panelists), supporting the notion of pre-selecting willing participants as a possible solution to improve response rates.
Where possible, other suggestions made in the literature for obtaining an optimal response rate were followed, including: making personal contact and building rapport by informing participants to enhance personal ownership of the project ; and planned follow-up  in the form of a reminder email. Similar to earlier studies , we opted not to pursue non-responders further as we did not wish to pressure already busy clinicians who we needed to be supportive of planned studies that required their contribution. It was possible however, that non-responders were not interested in checklist development, did not have anything to add to the process, or were not able to make study participation a priority given their primary role was in clinical and teaching responsibilities.
Participants within each of the panels were known to each other and all participants were known to the researcher. The risk of potential bias was minimized by allowing respondents to complete the questionnaires in their own time, ensuring responses remained strictly anonymous within the Delphi process, and providing synthesized feedback during the second-round survey. It has been suggested that this kind of ‘quasi-anonymity’ could actually motivate panelists to participate, discourage ill-considered hasty judgments, and ensure some level of accountability for the responses given .
Due to practical constraints of creating a parallel-form measure we did not test reliability by comparing the final checklist statements generated using the Delphi technique with statements generated by another method of developing the tool – for example via focus groups or consensus meetings with experts . Coordinating a single meeting time to suit all experts would have been difficult, particularly since the majority had clinical duties. Even if one had been arranged, it is questionable whether a group meeting can produce reliable results given the risk of bias with group conformity .
Recommendations for research
There are a few key areas that require evaluation in future studies. First, it is important to verify these findings with further research conducted in clinical settings. The checklist items generated should be evaluated for their practical use, interpretation and clinical utility. Second, testing the reliability of items should be undertaken to establish whether the items produce consistent results. Third, the methods used for checklist development and validation also have applicability beyond the ICU and can be tested as a model for improvement in other clinical areas irrespective of geographical location.