This paper has compared the labour costs and the feasibility of different methods of barrier identification used within three theory-based implementation projects to identify the most feasible method/s.
Conducting a literature review to identify barriers from published papers was found to be more feasible than use of a questionnaire, interviews or focus group. The literature review method removed the challenge of recruiting busy health professionals, which was experienced in project one (which used a questionnaire and interviews) and project two (which used a questionnaire and focus group). Conducting the literature review was, therefore, quicker than the other methods (for example, taking two months compared with five months for interviews, or three months for the focus group). The literature review method was used partly to reduce the resources required but also to experiment with a briefer form of barrier identification. Importantly, our NHS partners’ desired to speed up the process of barrier identification, with pressure to be seen to be intervening quickly to address low adoption rates once they had been identified. However, whilst this method was found to be the most feasible, it is acknowledged that for some clinical topics, identifying published barriers papers could be more difficult than it was in this particular study, with either a high volume of papers (which would be more time consuming, and, therefore, more costly in terms of labour time), or very few papers available.
Using combined methods for barrier identification is recommended, increasing robustness through comparing barriers collected from multiple sources [12]. However, this study drawn attention to the potential for increased costs when combining methods; for example, with project two costing £59, 834.31, compared with £34,725.65 for project three which used a single method. When considering combining methods, it is, therefore, particularly important to consider the cost and feasibility of the methods to be combined, as well as the order in which the methods will be used. For example, if qualitative methods are to be used to identify barriers to then inform the design of a questionnaire, the methods cannot be run concurrently, compared with if the qualitative methods are to be used purely to enable a richer exploration of barriers alongside use of a questionnaire.
Questionnaires were found to be the least feasible method. They were time consuming to develop (requiring multiple iterations and piloting to ensure optimal design), and despite this effort, response rates were low (15–19 %). Project costs were then increased through also providing paper copies of the electronic questionnaire in project one (incurring costs of printing and administration: an additional £4352.44 in project one), and feasibility reduced through having to use multiple reminders to try and bolster response rates, which introduced time delays. Similar challenges to the use of questionnaires were also reported by Kraus et al. [5] in their assessment of the feasibility of different methods of barrier identification. For project one, the questionnaire method took approximately eight months, and in project two it took approximately four months. Comparing projects one and two, the feasibility of questionnaires may be increased through opting for only electronic administration to keep costs lower, and through modifying ‘off the shelf’ questionnaires where they are available. For example, if using the theoretical domains framework [13] to underpin barrier identification, a survey has been developed that would require minimal input to adapt it to different settings [14], and if using the theory of planned behaviour [15] (as in project two), questionnaire development is facilitated through making use of guidance on to how to operationalise the constructs [16]. However, for use in the NHS on a rolling basis across a number of implementation studies, the need to consistently achieve satisfactory response rates is unlikely to be feasible. Further challenges with the use of questionnaires are also likely to arise once the data is collected, with low response rates limiting the robustness and generalizability of the findings.
Qualitative methods are recommended in the early stages of intervention development in the revised Medical Research Council Framework for the development and evaluation of complex interventions [1]. Comparing projects one and two which used qualitative methods alongside a questionnaire, interviews were found to be a less feasible qualitative method of barrier identification than use of a single focus group, with the interviews taking five months from start to finish, compared with the focus group which took three months. This was largely due to the challenge experienced in recruiting health professionals to take part in interviews. No responses were received to the initial round of interview recruitment letters sent to a randomly drawn sample. Subsequently, we had to go through practice managers who acted as gatekeepers to recruitment, which caused time delays. Time and, consequently, labour costs were further increased through having to conduct, transcribe and analyse individual interview data which took longer for the interviews than the focus group. For example, it was necessary to conduct each of the seven interviews on different days and in different locations to facilitate recruitment. The focus group, by comparison collected the views of 10 general practitioners, three practice nurses and three practice managers in a single meeting. However, the feasibility of this method was increased by the opportunity to conduct the focus group during a pre-arranged meeting which was brought to our attention via the collaborating quality improvement team. Without this opportunity, it is likely that the same recruitment challenges experienced for the interviews would have occurred, with the added complexity of trying to coordinate busy health professionals’ diaries. This highlights the importance of having well established collaborations in place with local quality improvement colleagues in the collaborating organisation: these team members were able to use existing links with practice managers to facilitate recruitment for the interviews, and to advise regarding the opportunity to conduct the focus group at a pre-arranged meeting.
This study has several limitations. The method used to estimate time spent on each project was based on retrospective self-reports by each team member. For each project, estimation was provided on completion of the project (on average, completion being approximately 12 months after the period of barrier identification) and, therefore, will be subject to some unreliability, despite team members being prompted to use calendar templates to facilitate recall and reduce the risk of recall bias. Additionally, due to the overarching research programme being organised into phases: a ‘developmental phase (topic selection and barrier identification), followed by an implementation and evaluation phase, costs were estimated for topic selection work with stakeholders and barrier identification work combined, rather than per method used. This limits our ability to make direct comparisons of the costs of the different methods, especially since both projects one and two used combined methods. In basing the comparison of barrier identification methods on those employed across three studies within an overarching research programme, the full array of different methods and combination of methods had not been attempted, nor was possible. The final limitation of this study is the absence of an assessment of the ‘comprehensiveness’ or robustness of the barriers identified (whether all important barriers have been identified); consideration of this would also have enabled evaluation of the costs in relation to effectiveness of the methods employed. However, there is no established standard for assessing this; in their review of barrier identification methods, Krause et al. counted the number and type of barriers identified using each method. The number of barriers is, however, only a rough yardstick measure, and the more barriers identified, the greater the need for, and the greater the challenge becomes of tailoring the intervention to those most important barriers. Each project used a different theory-based to underpin the exploration of barriers, with the identified barriers coded against the theoretical constructs in the subsequent phase of the research programme. However, whilst this may enable a rough gauge as to the variety of barriers identified, it gives little confidence as to whether the most influential barriers have actually been identified. Using qualitative methods in projects one and two alongside the questionnaire, and engagement with stakeholders in project three, however, provided some assurance that the main barriers had been identified using each method/combination of methods.