Consensus on quality indicators to assess the organisation of palliative cancer and dementia care applicable across national healthcare systems and selected by international experts

Background

Large numbers of vulnerable patients are in need of palliative cancer and dementia care. However, a wide gap exists between the knowledge of best practices in palliative care and their use in everyday clinical practice. As part of a European policy improvement program, quality indicators (QIs) have been developed to monitor and improve the organisation of palliative care for patients with cancer and those with dementia in various settings in different European countries.

Method

A multidisciplinary, international panel of professionals participated in a modified RAND Delphi procedure to compose a set of palliative care QIs based on existing sets of QIs on the organisation of palliative care. Panellists participated in three written rounds, one feedback round and one meeting. The panel’s median votes were used to identify the final set of QIs.

Results

The Delphi procedure resulted in 23 useful QIs. These QIs represent key elements of the organisation of good clinical practice, such as the availability of palliative care teams, the availability of special facilities to provide palliative care for patients and their relatives, and the presence of educational interventions for professionals. The final set also includes QIs that are related to the process of palliative care, such as documentation of pain and other symptoms, communication with patients in need of palliative care and their relatives, and end-of-life decisions.

Conclusion

International experts selected a set of 23 QIs for the organisation of palliative care. Although we particularly focused on the organisation of cancer and dementia palliative care, most QIs are generic and are applicable for other types of diseases as well.

Europe faces a huge challenge with a population that is rapidly aging in the coming decades. It is estimated that the incidence and prevalence of cancer will increase by about 20% and the prevalence of dementia will double before 2050 [1–4]. Although it concerns two totally different diseases with different care needs and disease trajectories, they do have a lot in common: they are often unnecessarily hospitalized, [5] have a high need for a multidisciplinary approach [6] and many suffer from symptoms which are partly the same, like pain [7, 8]. Higher survival rates of people with life-threatening and progressive chronic diseases will result in a larger number of patients that have multiple and complex health-threatening problems. Therefore, a growing number of patients will be in need of palliative care. However, a wide gap exists between the knowledge of best practice in palliative cancer and dementia care and its application in every day clinical practice [9].

As a first step in bridging this gap, it is important to assess current performance of palliative care in relation to its desired performance. Such an assessment of health care can be achieved by using quality indicators (QIs). QIs are evidence based, explicitly defined and measurable items that evaluate and describe structures, processes and outcomes of health care [10]. As such, they reflect the core elements of good clinical care. In day-to-day terms QIs can, for example, show whether pain is regularly being assessed using a validated tool; or whether a general practitioner is timely informed about a patient’s situation before or directly after discharge from hospital [11, 12]. QIs can help trace potential problems or confirm good quality of care and can be used to guide quality improvement processes [10]. They have been used effectively to assess and improve hospital care, [13] primary care, [14] and dementia care [15, 16]. Several studies have also developed QIs to improve the structures and process needed for the delivery of good quality palliative cancer or dementia care [11, 12, 17, 18]. However, these studies were performed five or more years ago, developed large sets of QIs, ranging from 56 to 142 QIs. Furthermore, none of these sets were widely implemented in everyday clinical practice.

The objective of this study was to integrate existing sets of QIs into one generic set that can be used to assess and improve the organisation of palliative care in different services and countries. The study was conducted within the framework of the European IMPACT project (IMplementation of quality indicators for PAlliative Care sTudy) [19].

A modified RAND Delphi procedure was used to develop a set of QIs, [20] which is considered an accepted methodology to develop QIs [10]. Typically, a RAND Delphi procedure consists of a written and a face-to-face round [20]. In this study, four written rounds and one face-to-face round were performed to reach consensus about the essential aspects regarding the organisation of palliative care.

Panellists

The IMPACT consortium consists of experts of the pan-European research group on detection and timely INTERvention in DEMentia (Interdem) and the European Association for Palliative Care (EAPC), all of whom are stakeholders in their country in palliative cancer and/or dementia care. The research team was invited to use their networks to purposefully select panellists for the modified RAND Delphi procedure. A key selection criteria was that the expert had to have extensive knowledge about palliative care, cancer care or dementia care. Additionally, experts had to be able to communicate in English (both verbally and non-verbally) and were planning to attend the 2012 Congress of the European Association of Palliative Care (EAPC) in Trondheim, Norway. Project partners nominated national and international experts in palliative cancer and dementia care. All nominated experts were approached via email (n = 50), forty experts from twelve countries agreed to participate in the modified RAND Delphi procedure (Table 1). All participants provided written informed consent. About half of them were experts in palliative cancer care and the other half in dementia care. Twenty-two panellists were active clinicians in this field (e.g. physician, nurse, psychologist, etc. currently involved in direct-patient care), the others were researchers.

Selecting a preliminary set of QIs

A search for existing sets of QIs was conducted in PubMed. The search strategy was limited to English literature and consisted of various search terms that referred to subject-specific keywords describing palliative care (combined using “or”), as well as (“and”) the assessment of care using QIs (combined using “or”). Synonyms and medical subheading terms were used to fully include relevant literature (see Table 2).

Subsequently, references of key papers describing sets of QIs were hand searched. Additionally, consortium members of the IMPACT project were asked to nominate national and international sets of QIs on palliative cancer and dementia care they considered important. Two researchers (YE & JvRP) subsequently reviewed all of the identified QIs independently to determine if the QIs assessed the structure and process of palliative care and to structure them according to the domains of the recommended framework for the organisation of palliative care of the Council of Europe [21].

First written Delphi-round

For the first written Delphi-round (April 2012), panellists received a personal invitation for an online questionnaire. To reduce the large number of identified QIs, panellists were asked to nominate one QI per domain of palliative care [22]. Those QIs that were nominated by the panellists were included in the second round of the modified RAND Delphi procedure.

Second written Delphi-round

In the second round (May 2012), also via an online questionnaire, panellists were asked to rate QIs on a 9-point Likert scale for clarity (1 = not clear at all; 9 = very clear), usefulness (1 = not useful at all; 9 = very useful), to rephrase unclear and to add missing QIs. They were instructed to rate a QI high on usefulness if it: 1) corresponded with a basic quality level; 2) referred to a higher quality level that would be met only in very good practices; or 3) was associated with an innovative quality level which is exceptional at the moment, but could become the optimal quality level in the near future [23]. They were asked to give a low rating on usefulness if a proposed QI: 1) was too ambiguous or represented an unrealistically high quality level; 2) did not correspond with the material, social or cultural conditions of the situation in their country; or 3) was not in accordance with the regulations of palliative cancer and dementia care in their country [23].

Third interactive Delphi-round

A consensus meeting was organised during the EAPC Congress June 7, 2012 in Trondheim, Norway. The meeting was chaired by an independent researcher with the aim to reach consensus on the QIs on which there was disagreement or where the median score was between 4 and 6 in the second Delphi round. Participants received a rating sheet on which the median ratings of the second Delphi-round of all experts were visible. Participants were given 30 minutes to rate the adapted QIs for usefulness. Next, per QI, participants were asked to raise their hand if they had rated usefulness 6 or less. If at least nine (30%) participants raised their hand, [20] the QI was discussed until consensus was reached.

Fourth written Delphi-round

After the consensus-round, the remaining indicators were fed back to the panellist with the purpose to validate the changes that were made (September 2012).

Fifth written Delphi round

In the final step of the QI development process, QIs were operationalised by the research team into questions that could be used by healthcare professionals to assess their organisation of palliative care and identify areas for improvement. During this process, it appeared that some QIs were inappropriate or not measurable (e.g. too time consuming to answer them appropriately). All QIs were therefore rated for necessity by the IMPACT research team (October 2012), representing both clinicians and researchers that also took part in the modified RAND Delphi procedure. QIs that were considered not necessary after this round, were omitted from the list.

Analysis

QIs with a median rating on the usefulness scale of 7, 8 or 9 without disagreement were considered to have face validity. Disagreement was defined as: 30% or more of the panellists rated a single QI in the 1–3 tertile and more than 30% in the 7–9 tertile. QIs scored with a median of 1–3 without disagreement were not considered to have face validity. Because panellists had rated QIs high on usefulness, only QIs with median ratings of 8 or 9 were considered face valid for the second Delphi-round. Only QIs that were rated valid by all panellists were included in the final set [20].

Ethical considerations

The Medical Ethics Committee of the district Arnhem-Nijmegen has declared that this study doesn’t fall within the remit of the Medical Research Involving Human Subjects Act (WMO) (registration number 2012/075). This means that this study can be carried out without an approval by an accredited medical ethics committee.

650 QIs were selected from literature [11, 16, 18, 22, 24–54]. After having assessed these QIs, 554 were excluded because they were not about the organisation of palliative care or because of overlap; the remaining 96 QIs were included in a preliminary set of QIs (Figure 1). Of the 40 experts invited as panellists, 25 (63%) participated. In the first Delphi round, 65 of the 96 QIs were selected and 13 missing QIs were suggested in an open question in which panellists were asked if they missed any relevant QI. This resulted in an adapted list of 78 QIs, which were included in the second Delphi round.

Modified RAND Delphi procedure. ¹Equivocal is defined as all QIs on which there was no agreement: e.g. QIs with 30% or more of ratings in both the 1–3 tertile and the 7–9 tertile and all indicators with a median rating in the 4–6 tertile. ²At the end of round three, panelists agreed that 11 QIs could be merged.

Full size image

In the second Delphi-round, 27 (67,5%) of the 40 invited panellists participated. Sixteen QIs were considered to have face validity, 28 were invalid, and six QIs were added to the list based on suggestions made by the panellists. The 40 QIs on which there was no agreement on, were included into the third Delphi round.

In the third Delphi-round, a consensus meeting, 29 (72,5%) of the 40 panellists participated. After having rated 40 QIs, one was excluded and 10 were discussed. Of the QIs that were discussed, six were accepted and four were excluded. Panellists also agreed to merge 11 QIs. Round three therefore resulted in 24 accepted QIs. The total list of QIs (QIs considered to have face validity in round two and three) comprised 40 QIs.

In the fourth Delphi-round, panellist provided feedback to the remaining set of QIs. This resulted in minor linguistic changes and the exclusion of three QIs because they were considered inappropriate by the majority of panellists. The resulting set of QIs, therefore, consisted of 37 QIs. This list was critically assessed by members of the IMPACT consortium for their necessity (round five). Fourteen QIs were considered to be overlapping, inappropriate, or not measurable in palliative care. The final list of QIs, therefore, consisted of 23 QIs, covering seven domains (Table 3). Key findings can be summarized as follows:

Access to palliative care

The availability of a dedicated palliative care team was considered important by almost all panellists in the Delphi procedure. They explicitly stated that palliative care services should not only be available during office hours, but at all times (day, evening, night and weekends). Furthermore, specific elements of palliative care were considered important, such as the availability of opioids and anticipatory medications for symptom control, as well as the availability of bereavement support.

Almost all panellists also rated important the accessibility of the medical record to health care professionals, timely transfer of information between settings (including when transferring or discharging patients). Panellists also considered an assigned contact person, who maintains regular contact with patients and their families, useful.

Infrastructure

The infrastructure of the place where palliative care is provided, such as a single bed hospital room, was rated important. Access to equipment (such as anti-decubitus mattresses, suction equipment, etc.), required to provide palliative care, was considered important. Panellists also rated high consensus for facilities for relatives to visit, stay overnight, and a private area for saying goodbye to the deceased. There was no agreement on QIs that aimed to control waiting time or waiting list, i.e. these aspects were not considered to be unique for palliative care and therefore not important.

Assessment tools

Regular assessment of pain and other symptoms was rated as a valid quality criterion, though it was commented that a validated instrument might not always be available, particularly for specific patient groups (e.g. for persons with advanced dementia).

Personnel

There was agreement on the need for a multidisciplinary team, which should consist of at least a physician and nurse, and have access to a range of supporting disciplines, such as: physiotherapist, psychologist, occupational therapist, social worker, dietician, and chaplain. Panellists also rated a regular multidisciplinary team meeting important.

Documentation of clinical data

Panellists recognized the importance of having a well-structured medical record. However, a QI about the structure of the medical record was not considered important by the panellists. Panellists only considered the inclusion of a medication regimen in the medical record important. They also considered a timely assessment (within 48 hours) of pain and other symptoms, psychosocial and spiritual needs, patient preferences, wishes and needs, and the patient's capacity to be involved in the decision making process as important.

Furthermore, almost all panellists rated the documentation of communication on the medical condition, goals of treatment, physical, psychosocial, and spiritual needs of the patient and their relatives, intention to return home, advanced directive, and end-of-life decisions as important.

Quality and safety

Panellists rated the QI about assessing the experiences of care givers with the palliative care service important. Secondly, a QI about the quality of care, assessing the use of an end-of-life care pathway within the last three days of life, was also considered useful.

Education

QIs about the staff’s learning objectives and a program for specialised and/or continuing medical education about the physical, psychosocial, and spiritual needs of a patient in need for palliative care were not rated important. There was also no agreement on disease-specific education for staff members, but panellists considered palliative care training specified to the professiona’ls background important.

With the help of a modified five-round RAND Delphi-procedure, we were able to develop an internationally validated set of QIs for the organisation of palliative care with high face validity as judged by experts in the field of cancer and dementia care. The final set provides 23 quality aspects regarding the accessibility of the service, its infrastructure, the use of symptom assessment tools, management of personnel, documentation of clinical data, quality of care, and education. Of these 23 QIs, one was identical to the original one (Family members and friends are able to visit the dying person without restrictions of visiting hours), [12] two were new (Family and caregiver experiences of the palliative care service are assessed/evaluated/recorded and An end-of-life care pathway (such as the Liverpool Care Pathway) was used for the last 3 days of life of a person in need of palliative care) and 20 were rephrased QIs. Panellist agreed not to formulate disease-specific QIs for the organisation of palliative care, since our set of QIs provide information about the organisation of services and not about the care provision itself. This might explain why so many QIs were rated face-valid for as well the organisation of cancer as dementia palliative care: regarding aspects as access to and 24 h availability of specialist palliative care or transferring information between settings, the specific condition of the patient (advanced cancer, dementia, or even COPD or heart failure) is not relevant, making our set of QIs much more broadly applicable. Therefore, these QIs are generically applicable and can be used in different settings. Thereby, they can also be used for (cross-)national comparisons and to identify best practices regarding the organisation of palliative care in other services and countries.

In the recent literature several sets of QIs for palliative care have been identified [11, 12, 17, 38, 55]. For example, Pastrana et al. used a nominal group technique to identify indicators for the assessment and evaluation of palliative care [38]. However, they primarily focused on the German health care system, which makes this set difficult to apply in an international context [38]. Pasman et al. conducted a literature review, and identified 142 QIs in 16 studies [11]. However, this set also has not been developed within an international context and it does not focus on the organisation of palliative care [11]. An update of this review, published in 2013, included a further 187 QIs, bringing the total to 326 QIs, with still few QIs about the organisation of palliative care [55]. Around the same time, Woitha et al. developed a set of 56 QIs [12, 17]. Woitha et al. conducted two written Delphi rounds, leaving little room for discussion, while we conducted a consensus round with extensive opportunity to discuss the QIs. Secondly, they included professionals from different European countries only, while we also included professionals from countries such as Canada, Australia and the USA, making the set of QIs presented here globally applicable. Thirdly, they focused on the organisation of palliative care in general and did not specifically consider the organisation of palliative care for patients with dementia, like was done in the present study.

A recently published White paper defining optimal palliative care in older people with dementia, [56] described several recommendations on palliative care treatment for persons with dementia. All of their recommendations that can be translated to the organisation of care, like the use of assessment tools, multidisciplinary meetings, bereavement support and about specialist palliative care teams, are represented in our QI set.

Another quality indicator, suggested by the World Health Organization as part of the framework programme on non-communicable diseases has been the focus of attention recently. This QI is being proposed to describe access to palliative care by assessing morphine equivalent consumption per death from cancer [57, 58]. However, this QI has been criticized as it might have provided flawed information due to inaccuracies in the underlying data base and the unavailability of national cancer registries [59]. Instead of assessing palliative care on a global level, we aim to assess whether palliative care services meet a basic quality level or higher quality level that would be met only in very good practices. Our set of QIs can therefore be used as internal QIs by health care providers (professionals and managers) to monitor and improve their service. They can also be used to describe and rank services according to performance, but this should not lead to a quality rating, as there may be good reasons for the differences in performance with the QI (e.g. different organisational structure).

Using our QIs as an external quality assessment tool will therefore make them unfit for their task [60]. Berwick et al. summarized this as ‘measuring for improvement is not measuring for judgement’ [61]. An ongoing intervention in 40 palliative care services in Europe, including hospitals, hospices, nursing homes and primary care settings, performed as part of the IMPACT project, in which this set of QIs is used as starting point to assess the organisation of palliative care, will evaluate the feasibility and discriminatory power of the QIs in relation to improving the organisation of palliative care in the participating services.

Strengths and limitations

Strengths of this study were that we used a large international group of panellists for our Delphi procedure, who were actively involved in palliative care (such as members of the European Association for Palliative Care). Secondly, by organising our consensus meeting during the EAPC Congress in Trondheim, Norway, key persons active in the fields of palliative care and dementia care were able to contribute extensively to the discussion of the Delphi procedure. Thirdly, the multidisciplinary character of palliative care was represented by the panellists (e.g. physicians, nurses, psychologists, etc.) involved in the Delphi procedure. Furthermore, half of them were professionals active in dementia care. Fourth, combining QIs for the organisation of services that provide care to palliative patients with cancer and those that provide such care to patients with dementia is unique. Our QIs can therefore be used in different settings.

A limitation of this study is that this set of QIs might not be comprehensive. Because an international, generic set of QIs was developed, some QIs that were important in only one or a few countries were excluded from the list. For that reason, important national or setting-specific QIs must be added when the set is used in a specific country. Secondly, this set of QIs is only related to the organisation of palliative care. Outcome and patient-related outcome measures were not included because they address a distinct purpose in measuring quality of palliative care. Thirdly, participants of the modified RAND Delphi procedure were selected because of their knowledge about palliative care, cancer care or dementia care. Because some experts of two large European networks (EAPC and InterDem) are part of the IMPACT consortium, they were also selected as participant for the modified RAND Delphi procedure (n = 18). Although not all countries were represented (like France) and others were overrepresented (like the Netherlands), the experts covered 12 countries from three continents, covering different health care systems and types of organisation of palliative care. Pilot testing the set of QIs in those countries and continents that were not represented in this study will reveal whether they are applicable in these countries too. Fourthly, unfortunately, there were no patient representatives involved as panellist. Testing the final set of QIs will therefore also have to incorporate their views on the basic quality level or higher quality level that would be met only in very good services.

International experts selected a set of 23 QIs for the organisation of palliative care that can be implemented in daily practice in order to demonstrate that organisations are providing high quality and effective palliative care or to identify areas for improvement.

This study is part of the larger IMPACT project (IMplementation of quality indicators in PAlliative Care sTudy). The research leading to these results has received funding from the European Union's Seventh Framework Programme FP7/2007-2013 under grant agreement n°258883.

The set of palliative care QIs could not have been established without the support and extensive feedback received from a large international panel of professionals in palliative care for patients with cancer and dementia, both in research as in clinical practice. Therefore, we would like to thank (in alphabetical order): Sam Ahmedzai, Marcel Arcand, Lieve van den Block, Rabih Chattat, Jacqueline Crowther, Rose-Marie Dröes, Yvonne Engels, Anneke Francke, Manuel A. Franco, Marie-José Gijsberts, Dagny F. Haugen, Marianne Hjermstad, Josephine Hockley, Steve Iliffe, Birgit Jaspers, Stein Kaasa, Raymond Koopmans, Carlo Leget, Wojciech Leppert, Jill Manthorpe, Franka Meiland, Johan Menten, Esme Moniz-Cook, Nele van den Noortgate, Christophe Ostgathe, Deborah Parker, Tania Pastrana, Sophie Pautex, Lukas Radbruch, Daniel Ramos, Claud Regnard, Julia Riley, Roman Rolke, Elizabeth Sampson, Rob van de Sande, Jenny van der Steen, Florian Strasser, Joan Teno, Myrra Vernooij-Dassen and Kris Vissers. We would especially like to acknowledge Marieke Groot for her contributions to the Delphi procedure by chairing the consensus meeting. Furthermore, we thank all participating researchers involved in the IMPACT consortium for their support.

Affiliations

1. Scientific Institute for Quality of Healthcare (IQ healthcare), Radboud university medical center, P.O. Box 9101, 6500, HB, Nijmegen, The Netherlands

   • Jasper van Riet Paap
   •  & Myrra Vernooij-Dassen

2. Kalorama Foundation, Nijmegen, The Netherlands

   • Myrra Vernooij-Dassen

3. Department of General practice & Elderly care medicine and Department of Psychiatry, VU University Medical Centre, P.O. Box 7057, 1081, BT, Amsterdam, The Netherlands

   • Rose-Marie Dröes

4. Department of Palliative Medicine, Universitätsklinikum Bonn, and Department of Palliative Care, Malteser Hospital Bonn, Sigmund-Freud-Street 25, 53127, Bonn, Rhein-Sieg, Germany

   • Lukas Radbruch

5. Department of Anaesthesiology, Pain and Palliative Medicine, Radboud university medical center, P.O. Box 9101, 6500, HB, Nijmegen, The Netherlands

   • Kris Vissers
   •  & Yvonne Engels

6. the IMPACT research team, Spain

   • Jasper van Riet Paap

Corresponding author

Correspondence to Jasper van Riet Paap.

Competing interests

The authors declare that they have no competing interests.

Authors’ contributions

YE and MVD are responsible for the design of the study and provided extensive feedback to multiple versions of the manuscript. JvRP is responsible for the analysis and manuscript. KV was involved in the organisation of the modified RAND Delphi procedure and provided extensive feedback to multiple versions of the manuscript. LR is work package leader in the IMPACT project. As such, LR is responsible for the intervention study in which the QIs are tested for their feasibility. Therefore, LR provided feedback to multiple versions of the manuscript. RMD provided feedback to multiple versions of the manuscript. All authors approved the final manuscript.

Reprints and Permissions