Development of the prototype concise safe systems checklist tool for general practice

Abstract

Background

In the course of producing a patient safety toolkit for primary care, we identified the need for a concise safe-systems checklist designed to address areas of patient safety which are under-represented in mandatory requirements and existing tools. This paper describes the development of a prototype checklist designed to be used in busy general practice environments to provide an overview of key patient safety related processes and prompt practice wide-discussion.

Methods

An extensive narrative review and a survey of world-wide general practice organisations were used to identify existing primary care patient safety issues and tools. A RAND panel of international experts rated the results, summarising the findings for importance and relevance. The checklist was created to include areas that are not part of established patient safety tools or mandatory and legal requirements. Four main themes were identified: information flow, practice safety information, prescribing, and use of IT systems from which a 13 item checklist was trialled in 16 practices resulting in a nine item prototype checklist, which was tested in eight practices. Qualitative data on the utility and usability of the prototype was collected through a series of semi-structured interviews.

Results

In testing the prototype four of nine items on the checklist were achieved by all eight practices. Three items were achieved by seven of eight practices and two items by six of eight practices. Participants welcomed the brevity and ease of use of the prototype, that it might be used within time scales at their discretion and its ability to engage a range of practice staff in relevant discussions on the safety of existing processes. The items relating to prescribing safety were considered particularly useful.

Conclusions

As a result of this work the concise patient safety checklist tool, specifically designed for general practice, has now been made available as part of an online Patient Safety Toolkit hosted by the Royal College of General Practitioners. Senior practice staff such as practice managers and GP partners should find it a useful tool to understand the safety of less explored yet important safety processes within the practice.

Background

The importance of patient safety continues to be recognised yet progress on improvement has been modest and patients everywhere continue to experience avoidable harm and substandard care [1]. One setting in the UK where patients experience increasing risk is primary care; where the diversity, scope and variation in infrastructure is combined with unprecedented demand from an ageing and chronically ill population [2]. To help meet this need the National Institute for Health Research School for Primary Care Research (UK) (NIHR-SPCR) funded the development of a multi-strand Patient Safety Toolkit (PST) comprising a number of tools that would equip practices to independently address a range of patient safety issues [3]. Our steering panel of experts felt that checklists are an important patient safety tool that remains under-utilised in primary care settings. Originally used in highly ordered environments such as the aviation industry [4] they have been effectively adopted in secondary care where patient safety has been improved in a range of specialities such as surgery, haemodialysis, anaesthesiology and other highly protocol driven areas of medicine [4,5,6,7].

Limited attempts have been made to transfer the success of checklists to the diverse environment of primary care. For example, the NHS Education for Scotland (NES) checklist [8] is a 78 item tool designed to meet regulatory compliance, or the checklists designed by UK indemnity organisations and made available to general practices to assist in medico-legal risk assessments [9, 10]. However these examples are either lengthy [8], not specific to patient safety [8, 9] or require external facilitation [9]. An approach that had not yet been taken was the development of a concise or ‘short-form’ checklist, specific to patient safety within UK general practice and designed to be completed by practice staff without external support.

The aim of this work was to produce a precise checklist that would minimise the impact on staff time and frame constructive discussions that reconsider and if needed improve existing safety related processes [3]. It would also have the potential to address areas of the patient safety taxonomy developed for the PST not covered by other tools [11]. The items included on the checklist were to be informed by international expert consensus [12], avoiding issues already being met by mandatory health and safety requirements [13, 14] and complementing the content of other tools in the Patient Safety Toolkit for UK General Practices [12, 15, 16]. Here we describe the development of the prototype safe-systems checklist, (PSC) including the rationale of the final design and the quantitative and qualitative testing of the prototype within UK general practice.

Methods

The National Institute of Health Research (NIHR) School for Primary Care Research (SPCR) Patient Safety Toolkit for general practices involved multiple academic centres (Birmingham, Exeter, Keele, Manchester, Nottingham, Oxford and Southampton) [10]. All participants were consented in line with the procedures of the Patient Safety Toolkit project [3]. Here we present the two discrete phases of the development of the prototype safe-systems checklist (PSC); Phase One consisted of the development of the (PSC which was conducted in two parts by senior members of the research team BB, RS, AA, and SC; In Phase Two the checklist was tested by practice staff at eight practices within North Staffordshire as part of the broader implementation of the patient safety toolkit [3]. All The overall study design is summarised in Fig. 1.

Fig. 1

Flow chart showing study design

Phase One: Development

Part A of the development phase consisted of the identification of items for the draft list and Part B the refinement of the draft list integrating user feedback to produce the final prototype checklist.

Part A

An extensive narrative review of tools for general practice patient safety was conducted by two of the authors (RS, SC) [11] and existing tools, ideas and items were considered by a Research and Development Corporation (RAND) consensus panel of nine international experts [12] who produced a score relating to the priority of each of the items with the nascent checklist formed from items deemed ‘high priority’. These were then compared to Care Quality Commission (CQC) requirements [13] and the Canadian Quality Book of Tools (CQBT) ‘safe theme’ [17] and items already included in these or other UK mandatory requirements [13, 14] were removed from the checklist in order to prevent duplication. Finally the wording of the draft list of items was reviewed by the study team for clarity and precision and those judged to be similar in theme were combined leaving 13 items.

Part B

The 13-item checklist was used by an opportunistic sample of staff from 16 practices recruited as part of the wider patient safety toolkit (PST) project [3, 15]. The checklist was completed by staff alongside a questionnaire which explored their perspectives on the items of the checklist primarily relating to their utility. Staff that took part in this initial testing were practice managers, and nurses reflecting the intention that it could be used by a range of clinical and non-clinical staff. Individuals were interviewed by KM using a topic guide which asked whether they felt individual items addressed important aspects of patient safety, those already being routinely addressed within the practice, and if they had made any changes to practice processes as a result of completing the checklist. Staff were also asked about the clarity of the items presented and if they were able to answer the questions posed as either ‘yes’ or ‘no’ as intended.

Phase Two: Testing of the prototype safe-systems checklist

Phase Two involved the distribution and use of the PSC across a number of practices. Convenience sampling [18] was used to recruit eight practices previously involved in the PST project. Each participating practice was visited by the researcher (KM) who described how the checklist should be used and by whom i.e. that it could be completed by either clinical or non-clinical staff. The checklist was completed once by a member of staff chosen by the individual practice and the summary statistics of the compliance with each item by participating practices was produced.

Semi-structured interviews [19] were then conducted individually with the one member of practice staff at each of the eight practices that had used the checklist. The interviews were completed by an experienced qualitative researcher with expertise in primary care, patient safety and health service research (KM) who had not previously met any of the participants using a topic guide that asked questions on their expectations and experience of using the PSC including its usability and applicability, and how it might be improved. All were digitally recorded before being transcribed verbatim and nvivo used to manage the data. We used a post hoc deductive analysis based on the relevant domains and constructs of the consolidated framework for implementation research (CFIR) [20] and emergent sub-themes noted. To enable this a sample of three transcripts underwent an independent analysis by two of the authors (IL & SG). These analyses and any discrepancies were then discussed by both before the analysis of the remaining transcripts was conducted by IL. Our intention was not to reach data saturation but to gain a more structured understanding of user perceptions of the PSC. The ultimate interpretation of the data was approved by all authors.

Results

Phase One: Development

Part A – Identification of the items for inclusion on the draft checklist

A total of 205 items were identified from the literature review for consideration by the RAND consensus panel [12] who judged 37 to be of “high priority”. Following comparison with the CQC, CQBT and UK mandatory requirements and the review of the study team for repetition and clarity a total of 13 items were eventually included on the draft checklist.

Part B – Refinement of the draft checklist to produce the final prototype

A total of 10 East Midlands (EM) and six Greater Manchester (GM) practices tested the draft checklist. A representation of the characteristics of the practices participating in the toolkit project can be found in Table 1. Within these practices a total of 38 respondents (23 from EM and 15 from GM) completed the pilot version of the checklist and accompanying questionnaire (23 GPs, four nurses, nine practice managers, and two administrative staff). An additional 25 staff from participating practices were interviewed with regard to the applicability of the PST (three GP registrars, three practice nurses, six practice managers and 13 GPs) interviews lasted between 10 and 33 min.

Table 1 Summative characteristics of participating practices [10]

At this stage the checklist was divided into four sections (information flow, safety information about the practice, prescribing, and use of IT systems), each with an introductory statement that was taken directly from our project taxonomy of patient safety [11].

A summary of the results from the development of the PSC can be found in Table 2 relating to one of the domains. Within each we define the domain, present the draft items, the rationale and evidence for each, the changes made as a result of the feedback collated from the questionnaires and semi-structured interviews and the related item as it appeared on the PSC which numbered nine in total.

Table 2 Development of Items for inclusion in prototype checklist

Phase Two: Testing of the prototype safe-systems checklist

Eight participating practices within North Staffordshire (NS) agreed to test the checklist.

Quantitative data

Table 3 shows the percentage of practices which answered yes to final checklist items. Items with a response of ‘No’ indicate where as a practice they feel they have not addressed a checklist item and might need to make a change to its systems. The two items with the lowest percentage of ‘Yes’ responses (25% of practices did not think achieve these safety goals) were item 6 regarding the failure to monitor the non-collection of prescriptions and item 7 relating to follow-up of vulnerable patients following discharge from hospital. Several items were met by all participating practices including the appropriate handling of incoming clinical information and the timely follow-up of abnormal results.

Table 3 Percentage of “yes” answers across practices

Qualitative data

We interviewed eight participants each from one of the practices that trialled the prototype checklist. Of these three were practice managers, three were general practitioners, one was healthcare assistant and one was a practice nurse manager. Interviews lasted between and 13 and 37 min [10].. The practices they represented were situated within a variety of socio-economic backgrounds represented using the Index of Multiple Deprivation [21] and patient list sizes from 4000 to just over 12,000 these characteristics are summarised in Table 4.

Table 4 Participant job role and practice characteristics

Two domains within the CFIR were relevant to our data set. The first was Intervention Characteristics and themes emerged within constructs relating to the relative advantage of using the checklist its adaptability and overall design quality. The second domain was Outer setting and within the construct of Patient needs and resources the theme relating to a lack of capacity emerged. These are summarised in Table 5. Below we describe these emergent themes alongside exemplar quotes.

Table 5 Summary of CFIR Domains, Constructs and Emergent themes

Intervention characteristics

This domain relates to the overall design, utility and usability of an intervention [20]. A number of constructs were identified in our data namely its Relative advantage, Adaptability and Design quality.

Relative advantage

This construct describes the stakeholder’s perception of the advantage of implementing the Prototype Safe-Systems Checklist (PSC) as opposed to maintaining existing practice. Any tool or instrument designed to improve safety of care can also improve aspects of care in other respects as patient safety and quality of care are so intrinsically linked. In terms of the advantages of using the PSC a number of themes emerged; staff described how they improved patient safety directly in terms of prescribing safety and enabling the review of existing systems, but also indirectly by using it to provide a framework to discuss patient safety with the broader practice team.

Staff engagement

Participants described how using the PSC indirectly benefitted patient safety by helping engage a range of staff. Although the tool was designed to be used by a single individual frequently, its completion would or could rely on other members of the practice team, helping raise awareness of patient safety.

“So we found it on several levels a really useful tool and not least, of course, patient safety, but in terms of actually being another vehicle to encourage cross-team understanding within the practice, as well.” Practice Manager, P01.

One Practice Manager felt that the document could be used to frame a discussion with GPs on whether policies and procedures were implemented as expected.

“…it’s quite straightforward, I’ll just run through everything with the GPs instead of saying ‘yes, we do this’… I mean you can have policy and procedure and no-one can follow it.” Practice Manager, P06.

High level approach

The benefits of the high level approach adapted by the checklist as a way of immunising specific items against local or sporadic change were described.

“I think one of the things that’s hard … with the checklist, is… keeping it up to date as things change so fast in practice, but a lot of your sentences are quite high-level, so it means that it lasts…” Practice Manager, P03.

Prescribing safety

A number of participants commented on the benefits of using the tool to improve medication safety and one practice manager felt the section on medications was the most useful.

“The medication things I thought was probably the most useful section… they say the most errors in a general practice are made on medicines…” Practice Manager, P03.

The other area that the PSC appeared to be effective was at highlighting the non-collection of repeat prescriptions. One GP acknowledged how this item had raised awareness of the issue and a practice manager how it had encouraged them to discuss the issue with other members of the team.

“Non-collection of prescriptions, that’s the one that we found that we weren’t doing very well… because we’re moving to electronic prescribing in a couple of weeks’ time, we’ll look into that, that way…” GP, P02.

“The non-collection of prescriptions was good and that did encourage me to talk to the dispensing team – “what did they do with those?”” Practice Manager, P07.

Review existing systems

It was noted, how as a whole, the PSC provided the opportunity to look again at the safety of existing systems that due to familiarity might otherwise be overlooked.

“Actually, it gives you the chance to reflect that some of the things [we do] are a system and to think, ‘Oh, yes!’ Something like mail-handling is, like so embedded …we take 500 letters in… every day, scan them in, pass them round and whatever - that, you know, you can almost forget that that is a safe system.” Practice Manager P03.

Training staff

Another way in which the PSC may indirectly benefit patient safety is by its use as a training tool for clinicians in the early part of their career. One practice manager described how it presented a useful overview for inexperienced clinicians.

“One thing I thought it would be …a good training tool for, like, an overview...These things would be good for, like, GP registrars and things, like in training… it’s a good overview position.” Practice Manager, P03.

Adaptability

The construct of Adaptability describes the degree to which an intervention can be tailored, refined or reinvented to better meet local needs [20]. The flexibility of the PSC in terms of how frequently it could be used emerged.

Frequency of use

There was no prescribed time interval in between using the PSC, meaning that practices could decide how often it could be used. One practice manager described how they might use the tool monthly..

“…If you’re doing it monthly, you’re more aware of the questions in your head, aren’t you, so it’ll become more of a routine. So, yes, I think it would [be monthly], in the long term.” Practice Manager, P04.

Another practice manager felt it would be usefully applied every 12 months to ensure systems were operating as safely as expected.

“I think once you’ve checked through it, it might be worth just going through it on an annual basis, just to make sure that you are doing these things….” Practice Manager, P06.

Design quality

The construct of Design quality describes the perceptions of users of the quality of its design [20]. The primary design element which participants commented on was how straightforward it was to use.

Ease of use

The PSC was considered well-structured and easy to follow, which meant that it was quick and easy to use.

“I think because it is quite brief it’s quite a useful thing, just a pointer to go through it and make sure that these things are still being done as they should.” Practice Manager, P06.

Outer setting

The domain of Outer setting relates to the influence of factors external to the design of the tool and the organisation [20], and the relevant construct in our analysis related to patient needs and resources.

Patient needs and resources

This describes the extent to which the practice understands and is able to meet the needs of its patients [20]. Within this construct the emergent theme concerned the lack of capacity of practices to absorb additional work streams.

Lack of capacity

One factor that may inhibit its further use was the limited capacity, in terms of time and workload in primary care. Despite not knowing the length of time it would take to use the tool, a GP at one practice asked a part-time member of staff to be responsible for the tool because of concerns over their own lack of time.

“Because we were just totally snowed under, so I knew I wouldn’t have time to do this so I asked my colleague who only works part time and did that for me. So he’s… done the Safe Systems questionnaire.” – GP P02.

One practice manager was positive towards the PSC but cautioned that its future implementation might depend on the ability of practices to meet the twin pressures of time and resource.

“As much as I am a big fan of this tool, I think the two key issues are finding time and, if it involves any resources, is actually finding support for those resources because that’s always challenging in this day and age.” Practice Manager, P01.

Discussion

Main findings

We have described the creation of the prototype safe systems checklist (PSC) a new tool in patient safety developed for inclusion in the Royal College of General Practitioners hosted Patient Safety Toolkit [3] (Please see Additional file 1 for a final version of the PSC). The development involved gaining a structured consensus of priority issues gathered from an international panel of experts underpinned by published evidence. These became the draft list of 13 items from which nine items were selected and edited for clarity based on user feedback. The 9-item prototype was tested in general practice, and participating practices reported that they met the majority of the items; seven out of nine checklist items were answered ‘yes’ by seven of nine practices with only two items (follow-up of vulnerable patients after discharge and the non-collection of prescriptions) being answered no by two practices. When asked about their experience of the checklist participants spoke favourably of its brevity, the relevance of its content, and its ability to frame broader safety-based discussions across the practice team. However that limitations of capacity familiar across much of UK general practice might hinder its wider implementation.

Strengths and limitations

The PSC has fulfilled the requirement for a tool that can quickly and easily provide valuable information on previously overlooked yet significant systems influencing primary care safety. Though the instrument described here is a prototype and yet to be fully validated its prima facie value has subsequently seen its successful incorporation into the patient safety toolkit [3]. The practices testing this instrument were located in one geographical area (North Staffordshire) yet their characteristics were reflective of national averages [32]. Drawing interviewees from this small pool of potential candidates that had used the checklist may have served to reduce response bias [33]. The ‘ceiling effect’ observed for some of the items [34] may be explained at least in part by the commitment of practices involved in the development of the Patient Safety Toolkit to improving patient safety.

Specific findings

Relative advantage

Staff engagement

For any intervention to be adopted it is necessary that it is perceived to offer significant advantages over existing methods of work [20]. There is a tendency to see checklists as a summative ‘tick-box’ exercise in which a list of all ‘Yes’ answers is more important than any process of system maturation, yet checklists, like other quality or safety improvement interventions, can also be used for formative purposes [35, 36]. The PSC is designed to be used as part of formative learning and development exercises that take the needs and experiences of each practice as the starting point [37]. Successful quality improvement is facilitated by close team working within practices [38, 39] and by strategies that include the wider practice team [40], particularly where they enable practice staff to learn, work, and plan together with clear objectives [41, 42]. In relation to this staff described a number of areas where the PSC might be used and one of the most significant was its use in framing conversations across the practice team that reflect on the safety of practice systems and participants.

Prescribing safety

Each checklist item was precisely described to provide the opportunity to reflect on not only whether but how each item was achieved inhibiting the tendency to tick an item for the sake of compliance [34]. Of the items selected for inclusion in the PSC participants described how those relating to the non-collection of repeat prescriptions were of significant value acknowledging that existing systems lacked clarity in this area. The importance of the safety of medicines and prescribing in the era of polypharmacy has been widely described [42, 43] and is recognised as a key area of focus for the future of primary care safety [44].

Review of existing systems

Participants described how the PSC enables senior practice staff e.g. practice managers or GP partners to review existing systems equally it might support an audit of a particular area or process for example around the communication of results or non-collection of prescriptions, again promoting intra-practice discussion and collective learning.

Design quality

Participants described the ease of use of the PSC and the speed with which it could be completed. The PSC was designed to be quick and easy to use, requiring neither facilitation nor specialist training. Though suitable as a framework for wider practice discussion, the checklist does not require the practice team to meet simultaneously to be effective. This offers a complementary set of outputs to the other GP checklist tool in the public domain, the NHS Education for Scotland (NES) checklist which consists of 78 items designed to help practices prepare for CQC or similar inspections [45].

Adaptability

The frequency with which the PSC can be used is dependent on the preferences of individual practices with our participants expecting benefits whether used monthly or annually. It’s been recognised previously that in healthcare checklists should be monitored to avoid over-burdening staff [46] and general practice in particular is a field in which practitioners have been shown to be over-worked [47]. This adaptability may also be increased by creating a digital version which was not available at the time of testing yet electronic versions of similar general practice tools has resulted in enhanced utility [30].

Outer setting

Despite the ease and flexibility of its use the longer term implementation of the PCS remained subject to constraints on resources which could impact on additional work streams [48]. This reflects the tension that persists in UK primary care between improving quality and safety amidst a sustained increase in demand [47] being met by a workforce that continues to lose experienced staff [49].

Conclusions

Successfully assessing quality and safety requires a mixture of subjective and objective approaches, [50, 51] and the PSC appears to offer a valuable objective lens through which to reflect on subjective procedures and actions within individual practices. It is important that checklists produced for general practice have their origin in safety factors relevant to their environment and are designed for use in the real world. The PSC meets our goal of highlighting relevant yet less obvious safety factors in general practice processes and has now become an integral part of the RCGP Patient Safety Toolkit.