Skip to main content
Download PDF

Quality of care in the last year of life: adaptation and validation of the German “Views of Informal Carers’ Evaluation of Services – Last Year of Life – Cologne”

BMC Health Services Research volume 22, Article number: 1433 (2022) Cite this article

Abstract

Background

To inform quality improvement and strengthen services provided in the last year of life, measuring quality of care is essential. For Germany, data on care experiences in the last year of life that go beyond diagnoses and care settings are still rare. The aim of this study was to develop and validate a German version of the ‘Views of Informal Carers’ Evaluation of Services – Short Form (VOICES-SF)’ suitable to assess the quality of care and services received across settings and healthcare providers in the German setting in the last year of life (VOICES-LYOL-Cologne).

Methods

VOICES-SF was adapted and translated following the ‘TRAPD’ team approach. Data collected in a retrospective cross-sectional survey with bereaved relatives in the region of Cologne, Germany were used to assess validity and reliability.

Results

Data from 351 bereaved relatives of adult decedents were analysed. The VOICES-LYOL-Cologne demonstrated construct validity in performing according to expected patterns, i.e. correlation of scores to care experiences and significant variability based on care settings. It further correlated with the PACIC-S9 Proxy, indicating good criterion validity. The newly added scale “subjective experiences of process and outcome of care in the last year of life” showed good internal consistency for each given care setting, except for the homecare setting. Test-retest analyses revealed no significant differences in satisfaction ratings according to the length of time since the patient’s death. Overall, our data demonstrated the feasibility of collecting patient care experiences reported by proxy-respondents across multiple care settings.

Conclusion

VOICES-LYOL-Cologne is the first German instrument to analyse care experiences in the last year of life in a comprehensive manner and encourages further research in German-speaking countries. This instrument enables the comparison of quality of care between settings and may be used to inform local and national quality improvement activities.

Trial registration

This study was registered in the German Clinical Trials Register (DRKS00011925; Date of registration: 13/06/2017).

Peer Review reports

Background

The last year of life constitutes a particularly emotional and vulnerable period for patients as well as for their relatives. It may be characterized by the experience of physical and mental decline, by symptom burden, and the need of support from others [1]. Especially older adults are confronted with multiple illnesses and higher care and medical service utilisation. Health-care needs are complex and admission to a hospital, nursing home or hospice is common [2, 3]. Health care systems are challenged to respond effectively to the intense needs of these patients [4]. In 2015 Germany adopted a law to improve and extend palliative and hospice care [5]. Depending on their needs, patients may receive generalist palliative care or specialist palliative care provided by health care professionals with expert knowledge, skills and attitudes [6]. Initial studies concluded that available services do not yet meet patient needs, since the provision of palliative care often starts too late or is restricted to cancer patients [7]. Thus, there is a need to systematize approaches to assess performance and quality of care provided in the last year of life.

Measuring the quality of care is a core priority to strengthen these services [8]. To conceptualize quality of care, Donabedian developed a three-part approach by assessing elements of (1) “structure” – the attributes of the settings in which care occurs, (2) “process” – the activities in giving (practitioner) and receiving (patient) care, and (3) “outcome” – the effects of care on the health status of patients and populations [9]. Key aspects to measure and assure quality are patient preferences. User perspectives are particularly important in end-of-life care research because commonly used endpoints, such as morbidity and mortality, are not useful for this patient population [10]. Recently, there has been increased development and use of patient reported experience measures (PREM) in order to obtain this information. PREM assess the way in which patients experience the process of care including satisfaction (e.g. satisfaction with information given), subjective experiences (e.g. pain control), objective experiences (e.g. waiting time) and observations of healthcare providers’ behaviour (e.g. whether or not a patient was given discharge information by a nurse). PREM allow patients to provide direct feedback on their care to drive improvement in services [11].

Yet, evaluating care experiences at the end of life presents unique challenges, as many patients are too ill to participate or are not identified as dying. Bereaved relatives are therefore often used as proxy to investigate patients’ views about their care [12]. In addition, care in the last year of life comprises a wide range of services important to patients, which requires a multidimensional assessment approach. Furthermore, measures need to enable the respondent to differentiate between care received by different providers. Several instruments have been developed to measure patient care experience comprising a wide variation in content areas. Many focus on the dying phase or the last weeks to months of life, or they are limited to a single care setting [4]. There is one survey, the “Views of Informal Carers – Evaluation of Services (VOICES)” from England that assesses patient and relative experiences of care across different settings and providers in the last three months of life [13]. A short form, VOICESSF, was developed following extensive research and consultation with patients, relatives and healthcare professionals to be applied in the national representative survey in England [14, 15]. This validated instrument was successfully used in several studies in England as well as internationally to examine the quality of care and services received and compare them across care settings [3, 13, 16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49].

So far, little is known on the care experiences of patients in the last year of life that go beyond diagnoses and care settings in Germany. The aim of this study was to develop and validate a German version of the VOICES-SF suitable to assess the quality of care and services received across settings and healthcare providers in the context of the German health care system in the last year of life.

Methods

Study design and participants

Data were collected in a retrospective cross-sectional survey as part of the ‘Last Year of Life Study-Cologne (LYOLC)’ [50]. LYOLC is a mixed methods study composed of four steps (1) claims data analysis, (2) postbereavement survey with next-of-kin, (3) qualitative interviews with next-of-kin, (4) qualitative focus groups with healthcare professionals) to examine care trajectories and transitions in the last year of life until death. Sample and data collection of the survey has been described in detail elsewhere [51]. Briefly, we included relatives, friends and volunteer workers (all will be referred to as ‘informants’ hereafter) of deceased adult persons in the Cologne area. Accidental and suspicious deaths were not included. Informants had to be aged 18 years and older and proficient in German.

Instrument development

The VOICES-SF survey is a questionnaire about experiences of end-of-life care in the last three months of life, focusing on quality of care and services received. It uses the postbereavement method to gather information from bereaved relatives, friends or carers acting as proxies. It is a survey instrument rather than a psychometric scale, questions may be added or deleted depending on a survey’s objective [14]. Using the VOICESSF (Version VOICES-SF 2014) as foundation [52], we developed a culturally adapted German version and called it ‘VOICES-Last Year of Life-Cologne (VOICES-LYOL-Cologne)’. This version evaluates care received in the last year of life, different to the original VOICESSF survey, which captures the last three months of life. This modification results from evidence showing that adding palliative care in the last 12 to 24 months of life has proven to be greatly beneficial [53, 54]. The longer period of observation enables the assessment of the introduction of palliative care services.

Figure 1 shows the translation and adaptation process. Procedures were based on the team translation approach ‘TRAPD’ (translation, review, adjudication, pre-testing, documentation) [50]. While back translation has become a controversial assessment method for questionnaire translations, ‘TRAPD’ is one of the most widely acknowledged frameworks for best practice in survey translation [51]. Two native German speakers (GD, NS) produced independent parallel translations, which were discussed with one reviewer. All of them were proficient in English. Three adjudicators (RV, JS, CR) familiar with the research project and the survey design, went through the final review version to decide on further modifications of the translation. Adaptation needs were addressed in different stages. Question adaptations were made to the content, response scales, and visual presentation of parts of the questionnaire. To produce a questionnaire that is culturally appropriate to the German health care system we conducted one group discussion with German healthcare professionals (n = 7) who work in palliative care. The resulting questionnaire was then tested in cognitive pre-tests with think-aloud technique (n = 8) to check whether (1) translated items as well as response categories were clearly understandable, (2) the questionnaire covered all important aspects of healthcare at the end of life, (3) wording and length were considered acceptable for bereaved relatives [52]. Pretesting again resulted in refinement before the adjudicators signed off on the version for final fielding. Documentation of various translation-related aspects followed the “Documenting Survey Translation” guidelines published by the - Leibniz Institute for the Social Sciences [53].

Fig. 1
figure 1

Development of the VOICES-LYOL-Cologne following the ‘TRAPD’ team approach

Full size image

The following modifications were made:

  • Revised observation period from “last three months of life” to “last year of life”.

  • Added questions about the communication of the diagnosis of a life-limiting disease to assess if, by whom, when and how conversations about dying were initiated.

  • Added the scale “subjective experiences of process and outcome of care in the last year of life” which comprises four key domains (relief of pain, relief of other symptoms, coordination of care, respect and dignity) in all settings of care.

  • Revised terminology to align with German sample.

  • Added sections related to healthcare providers additional available in the German healthcare system.

  • Added single items about informal carers, place of care in the last two days of life, unsolved problems, financial problems, and demographic and clinical data.

  • Harmonised response options and added “do not know” where appropriate.

The VOICES-LYOL-Cologne comprises a total of 106 items and contains skip logic so that informants only answer questions relevant to the care the patient received. In detail, it assesses care provided at home (by a nursing service, specialist palliative home care team, hospice volunteer services, general practitioner (GP), outpatient specialist physicians), care provided in a care home, hospital care (last admission), hospice care (in-patient), care provided in the last two days of life across all care settings, circumstances surrounding the death (communication of a life-limiting disease, carer support, unmet needs, financial needs, preferences and decision-making, communication of imminent death, place of death, bereavement support), and disease specific and sociodemographic data.

For each specified setting/health care provider informants could rate their subjective experiences of process and outcome of care in the last year of life on a four-point scale. Domains assessed are relief of pain, relief of other symptoms, coordination of care (1 = yes, 2 = rather yes, 3 = rather no, 4 = no), respect and dignity (1 = always, 2 = most of the time, 3 = some of the time, 4 = never). The total score is obtained by calculating the average of item scores. In addition, overall satisfaction with care in the last year of life by specified setting as well as taking all services into account is assessed on a four-point scale (1 = good, 2 = rather good, 3 = rather bad, 4 = bad). The questionnaire concludes with one free-text item to make overall comments about the care provided to the patient, which can be evaluated, for example, with qualitative content analysis [54].

Other instruments administered

The survey also comprised the ‘German Patient Assessment of Chronic Illness Care (PACIC) short form for bereaved persons as proxies (PACIC-S9-proxy)’ [55] in order to assess criterion validity. This validated instrument assesses patient-centred care during the last year of life retrospectively and by bereaved relatives.

Data collection

Informants were recruited in cooperation with healthcare practitioners from Cologne between November 2017 and August 2018. Applied strategies (postal distribution, personal distribution, and self-selection) to identify potential participants have been described in detail elsewhere [56]. Study procedures were approved by the Ethics Commission of the Faculty of Medicine of the University of Cologne, Germany (#17–188).

Data analysis

Descriptive statistics were calculated using SPSS Statistics version 26 (IBM Corp., Armonk, NY, USA). Results are presented as mean ± standard deviation (SD) and count (percentage), respectively. The presented p-values are two-sided and considered significant if p < 0.05. To assesses the validity and reliability of VOICES-LYOL-Cologne we followed the procedures applied in the validation of the modified Canadian VOICES survey [46]. This study was informed by the guidelines of the Scientific Advisory Committee of the Medical Outcomes Trust for assessing healthcare surveys [57].

Validity

Construct validity was investigated by generating and testing hypotheses about expected patterns of scores for groups known to differ on relevant variables (nomological network) [57, 58].

  1. 1.

    We expected the subjective experiences of process and outcome of care in the last year of life to be different among settings and health care providers. In previous studies informants rated hospice care more positively than homecare or hospital care [16, 18, 46, 59]. Accordingly, we hypothesised that each of the domains within hospice care would be ranked higher than care provided at home or in hospital and tested this using the Wilcoxon signed-rank test for paired samples.

  2. 2.

    Satisfaction with care has been found to correlate to place of death. As shown in previous VOICES studies we hypothesised that informants of people who died in hospital rated overall satisfaction with care (taking all services into account) significantly worse than in any other place of death [46, 59]. We tested this hypothesis using the Mann-Whitney U-Test.

  3. 3.

    The possibility of providing specialist palliative care at home was introduced in Germany in 2007 [60]. First evaluations showed that this care service enables more patients with complex symptoms and intensive care needs to die in their own home [61, 62]. We therefore expected a higher rate of home deaths in patients who received care by a specialist palliative home care team. This hypothesis was tested using the chi-square test.

  4. 4.

    The provision of palliative care has been found to correlate with cancer indication [63, 64]. Accordingly, we hypothesized that cancer patients received care provided by hospice, a hospital palliative care unit or by a specialist palliative home care team more often than patients with non-cancer indications and tested this using the chi-square test.

Criterion (concurrent) validity is the correlation of a scale with some other measure of the trait [65]. The PACIC-S9-proxy served as external criteria to assess criterion validity. We calculated Spearman’s rank correlation between the overall satisfaction rating on the VOICES-LYOL-Cologne and the PACIC-S9-proxy mean score, both taking all services into account. We proceeded analogously with the overall satisfaction rating by specified healthcare provider, i.e., GP, specialist physician and hospital doctor on the VOICES-LYOL-Cologne and the PACIC-S9-proxy (mean score).

Reliability

Cronbach’s alpha was used to assess the internal consistency of the scale “subjective experiences of process and outcome of care in the last year of life”. Cronbach’s alpha indicates whether an item of a scale is appropriate for assessing the underlying concept of its scale [66]. Commonly accepted are values above 0.70 for group comparisons [57].

As a surrogate for test-retest-reliability, we assessed whether overall satisfaction rating of care differed according to the length of time since the patient’s death (reproducibility) and used the Kruskal-Wallis test. This approach was applied by Seow et al. to evaluate the Canadian VOICES survey since retesting bereaved relatives was not feasible [46]. We expected stability of the instrument over time if informants who respond closer to the patient’s death do not differ systematically from those responding longer after a patient’s death. We defined four equally sized groups (≤ 111 days, 112 to 215 days, 216 to 331 days, ≥ 332 days) to perform this test.

Results

Characteristics of the sample

365 informants returned the questionnaire. The response rate depended on the recruitment strategy and differed between 10.3% for personal distribution, 21.1% for postal distribution and 74.9% for self-selection. 14 questionnaires were excluded due to a lack of inclusion criteria. The remaining 351 questionnaires were mainly answered by a spouse or a child (81.8%), the majority was female (71.5%). The decedents were 76.5 ± 13.0 years old and 52.1% were female. The main illnesses reported were cancer (59.5%) and cardiovascular diseases (40.5%). Most of the decedents (97.2%, n = 341) had received care from multiple, at least two, settings or health care providers in the last year of life. Characteristics of the deceased patients and their informants are presented in Table 1.

Table 1 Demographics and characteristics of deceased patients and informants (N = 351)
Full size table

Validity

Construct validity

  1. 1.

    Subjective experiences of process and outcome of care in the last year of life differed between health care providers. As expected, hospice care was rated best in every single domain (Table 2). Significant differences between ratings of hospice care and homecare were found for relief of pain (Z = 2.105, p = 0.035, n = 45), relief of other symptoms (Z = 3.273, p = 0.001, n = 47), coordination of care (Z = 3.214, p = 0.001, n = 41), respect and dignity (Z = 3.243, p = 0.001, n = 59). Between hospice care and hospital care significant differences were found for relief of pain (Z = 3.622, p < 0.001, n = 39), relief of other symptoms (Z = 2.586, p = 0.010, n = 34), coordination of care (Z = 2.759, p = 0.006, n = 22), respect and dignity (Z = 3.968, p < 0.001, n = 39).

  2. 2.

    Informants’ overall satisfaction with care in the last year of life varied depending on the place of death (Table 3). Dying in hospital corresponded to a lower overall satisfaction rating (mean = 2.08 (SD = 0.84), 26.2% rated care as “rather bad” or “bad”) compared to dying in any other place (mean = 1.6 (0.63), 5.3% rated care as “rather bad” or “bad”)). Differences were statistically significant (U = 5892.5, p = < 0.001, n = 330).

  3. 3.

    As hypothesized the integration of a specialist palliative home care team was associated with the place of death (Table 4). While half of the patients treated by specialist palliative home care died at home, this was only fulfilled for 14% of patients who did not receive this service. The results show that not including a specialist palliative home care team is associated with more people dying elsewhere than at home. Differences were statistically significant (χ2 = 49.056, df = 1, p < 0.001, n = 345).

  4. 4.

    As hypothesized palliative care was more often provided to cancer patients than to other indications. The majority of cancer patients (86%) received care either provided by hospice, a hospital palliative care unit or a specialist palliative home care team (Table 5). On the other hand, only one third of patients with non-cancer indications received specialist palliative care. Differences were statistically significant (χ2 = 88.221, df = 1, p < 0.001, n = 320).

Table 2 Subjective experiences of process and outcome of care in the last year of life
Full size table
Table 3 Overall satisfaction with care in the last year of life by place of death
Full size table
Table 4 Association of specialist palliative home care and place of death
Full size table
Table 5 Association of cancer indication and the provision of specialist palliative care
Full size table

Criterion validity

Ratings of overall satisfaction on the VOICES-LYOL-Cologne were found to highly correlate with the mean scores of PACIC-S9 Proxy. A significant correlation was found for taking all services into account (rs = 0.400, p < 0.001, n = 225) as well as for specified healthcare providers: care by a GP (rs = 0.522, p < 0.001, n = 93), care by a specialist physician (rs = 0.491, p < 0.001, n = 69), care by a hospital doctor (rs = 0.481, p = 0.001, n = 43).

Reliability

Internal consistency

Cronbach’s alpha was calculated for each given setting (hospital: α = 0.829, care home: α = 0.821, hospice: α = 0.814, home: α = 0.652).

Reproducibility

Regarding overall satisfaction with care, there was no statistical difference between relatives who responded closer to the patient’s death and those responding longer after a patient’s death (n = 325, Kruskal-Wallis H = 1.59, p = 0.662).

Discussion

This study describes the development of the VOICES-LYOL-Cologne – an adapted German version of the English VOICES-SF – and evaluates its validity and reliability. We used the team translation approach ‘TRAPD’ and included healthcare professionals as experts to produce a questionnaire that is culturally appropriate to the German health care system. The evaluation followed procedures applied in the validation study of the Canadian VOICES survey including a multitude of statistical tests [46]. Hypotheses about expected patterns of scores for groups known to differ on relevant variables were tested and generated results that corresponded with expectations. Criterion validity was also evident against the PACIC-S9 Proxy, an instrument to evaluate the level of patient-centeredness in the care during the last year of life [55]. Good internal consistency was found for the scale “subjective experiences of process and outcome of care in the last year of life” for each given care setting, except for the homecare setting. Test-retest analyses revealed no significant differences in satisfaction ratings according to the length of time since the patient’s death. Overall, our data demonstrate the feasibility of collecting patient care experiences reported by proxy-respondents across multiple care settings.

Comparison with previous research

VOICES’ content is based on patients’, bereaved relatives’ and healthcare professionals’ views about what is important at the end of life and has been developed in England [14]. Further versions are already available and have been published for the use in Sweden, Denmark, New Zealand, Canada, Thailand, Italy, and a Bengali-speaking community in London [19, 30, 31, 39, 44, 46, 68]. This study used the most recent version of the VOICES-SF from 2014 [69] as foundation to develop a culturally adapted German version. The adaptation process comprised alterations independent of unavoidable translation change. The main objective was to render questions culturally and linguistically appropriate. Since adapted questions should be treated as new questions, we do not recommend to compare questions of the VOICES-LYOL-Cologne with the original VOICES-SF or the other translations and their performance.

VOICES-LYOL-Cologne evaluates care received in the last year of life and incorporates the communication of the diagnosis of a life-limiting disease, different to the original VOICESSF survey, which captures the last three months of life. This modification results from evidence showing that adding palliative care in the last 12 to 24 months of life has proven to be greatly beneficial [70, 71]. Therefore, while evaluating experiences of care at the end of life, consideration could now also be given to the introduction of these services [51]. Furthermore, key domains of the subjective experiences of process and outcome of care in the last year of life with individual health care providers were added. This scale enables a more detailed assessment and comparison of care experiences across care settings.

Strengths and limitations

Experience measures are used to capture what really matters to patients to identify gaps in care and to make service improvements [11]. The instruments available so far focus on single care settings, are restricted to the dying phase or examine care in case of a concrete diagnosis. The VOICES-LYOL-Cologne assesses care in the entire last year of life. It proceeds irrespective of the underlying diagnosis and includes all healthcare providers involved in patients’ care. This allows us to gain a broad overview of care processes in the last year of life, to analyse potential correlations, and to inform quality improvements.

We conducted a retrospective survey with bereaved relatives as proxies for deceased patients – as intended by the original VOICES survey [13]. Postbereavement studies may be influenced by participant’s memory, feelings or the level of agreement with patient’s views [72, 73]. Our data did not show any significant differences in overall satisfaction with care regarding time since patient’s death to survey completion. Evidence shows that salient events may be recalled more accurately and that the level of agreement may be good on service evaluations and observable symptoms [74,75,76].

This study was part of the mixed methods ‘Last Year of Life Study-Cologne (LYOLC)’ conducted in the city of Cologne, Germany [50]. Data of the postbereavment survey were based on a purposive sample since a population-based survey was not feasible. Patients from palliative care services were overrepresented, which can be attributed to the recruitment strategy. Nevertheless, the sample was in line with the age and gender distribution of people who died in the city of Cologne [77, 78].

Conclusion

VOICES-LYOL-Cologne is the first German instrument to analyse care experiences in the last year of life in a comprehensive manner. The reliability of one core scale to assess “subjective experiences of process and outcome of care in the last year of life” was tested and showed satisfactory psychometric properties. In addition to recording the provision of services and circumstances surrounding death, VOICES-LYOL-Cologne might enable the comparison of care quality between settings. The instrument will be available on request and encourages further research in German-speaking countries. Future studies should use the survey in broader, more representative populations. VOICES-LYOL-Cologne may also be used to benchmark individual providers, who may benefit from feedback and to inform local as well as national quality improvement activities. In England the VOICES-SF survey was used annually as part of the “National Survey of Bereaved People” to achieve the goals of the “End of Life Care Strategy” [79]. VOICES-LYOL-Cologne may also be used to improve regional care in the last year of life. In Cologne for example, a PDSA cycle has been set up by analysing patient care experiences [56]. A working group to improve regional care and further research projects have already resulted [80, 81]. To gain in-depth insight into the reasons for transitions and the effects on the quality of life of the patients and their relatives, future studies may use a mixed-methods design as we did in the project LYOL-C [82]. Qualitative interviews with bereaved informal caregivers after completing the questionnaire may gain in-depth insight into patient trajectories [67]. Furthermore, the views of healthcare professionals may be important to analyse challenges associated with transitions in the last year of life and identify possibilities for improvements [83].

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author upon reasonable request and following application to CoRe-Net (Core-Net@uk-koeln.de).

Abbreviations

CoRe-Net:

Cologne Research and Development Network

GP:

general practitioner

PACIC-S9-proxy:

Patient Assessment of Chronic Illness Care- short form containing nine items for completion by proxy informants

PREM:

patient reported experience measures

TRAPD:

translation, review, adjudication, pre-testing, documentation

VOICES:

Views of Informal Carers’ Evaluation of Services

VOICES-LYOL-Cologne:

Views of Informal Carers’ Evaluation of Services-Last Year of Life-Cologne

References

  1. Jürges H, Hank K. The last year of life in europe: initial findings from the SHARE study. 2009. https://ssrn.com/abstract=1443992. Accessed 20 Jan 2022.

  2. Barrio-Cortes J, Soria-Ruiz-Ogarrio M, Martínez-Cuevas M, Castaño-Reguillo A, Bandeira-de Oliveira M, Beca-Martínez MT, et al. Use of primary and hospital care health services by chronic patients according to risk level by adjusted morbidity groups. BMC Health Serv Res. 2021;21:1046.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Hanratty B, Lowson E, Grande G, Payne S, Addington-Hall J, Valtorta N, Seymour J. Transitions at the end of life for older adults – patient, carer and professional perspectives: a mixed-methods study. Health Serv Deliv Res. 2014. https://doi.org/10.3310/hsdr02170.

    Article  Google Scholar 

  4. Lendon JP, Ahluwalia SC, Walling AM, Lorenz KA, Oluwatola OA, Anhang Price R, et al. Measuring experience with end-of-life care: a systematic literature review. J Pain Symptom Manage. 2015. https://doi.org/10.1016/j.jpainsymman.2014.10.018.

    Article  PubMed  Google Scholar 

  5. Federal Ministry of Justice, Germany. Act to improve hospice and palliative care. https://www.ilo.org/dyn/natlex/docs/ELECTRONIC/101621/122527/F228719032/bgbl115s2114_134875.pdf. Accessed 8 Feb 2022.

  6. German Cancer Society, German Cancer Aid (German Guideline Program in Oncology). Palliative care for patients with incurable cancer, Short version 2.2. 2020. https://www.leitlinienprogramm-onkologie.de/fileadmin/user_upload/Downloads/Leitlinien/Palliativmedizin/Version_2/GGPO_Palliative_Care_ShortVersion_2.2.pdf. Accessed 20 Jan 2022.

  7. Bertelsmann Stiftung. Palliative care. Gütersloh: Bertelsmann Stiftung; 2015.

    Google Scholar 

  8. World Health Organization. Better palliative care for older people. 2004. https://www.euro.who.int/__data/assets/pdf_file/0009/98235/E82933.pdf. Accessed 20 Jan 2020.

  9. Donabedian A. The quality of care. How can it be assessed? JAMA. 1988;260:1743–8.

    CAS  PubMed  Google Scholar 

  10. Axelsson B, Sjödén PO. Assessment of quality of life in palliative care - psychometric properties of a short questionnaire. Acta Oncol. 1999;38:229–37.

    Article  CAS  PubMed  Google Scholar 

  11. Desomer A, van den Heede K, Triemstra M, Paget J, Boer D de, Kohn L, Cleemput I. Use of patient-reported outcome and experience measures in patient care and policy. Health Services Research Brussels: Belgian Health Care Knowledge Centre (KCE); 2018.

    Google Scholar 

  12. Teno JM. Measuring end-of-life care outcomes retrospectively. J Palliat Med. 2005;8(Suppl 1):42-9.

    Google Scholar 

  13. Addington-Hall J, Walker L, Jones C, Karlsen S, McCarthy M. A randomised controlled trial of postal versus interviewer administration of a questionnaire measuring satisfaction with, and use of, services received in the year before death. JECH. 1998;52:802–7.

    CAS  Google Scholar 

  14. Hunt KJ, Richardson A, Darlington A-SE, Addington-Hall JM. Developing the methods and questionnaire (VOICES-SF) for a national retrospective mortality follow-back survey of palliative and end-of-life care in England. BMJ Support Palliat Care. 2017. https://doi.org/10.1136/bmjspcare-2016-001288.

    Article  PubMed  Google Scholar 

  15. Hunt KJ, Shlomo N, Richardson A, Addington-Hall JM. VOICES Redesign and Testing to Inform a National End of Life Care Survey. Final Report for the Department of Health; 2011.

  16. Addington-Hall JM, O’Callaghan AC. A comparison of the quality of care provided to cancer patients in the UK in the last three months of life in in-patient hospices compared with hospitals, from the perspective of bereaved relatives: results from a survey using the VOICES questionnaire. Palliat Med. 2009;23:190–7.

    Article  CAS  PubMed  Google Scholar 

  17. Andersson S, Lindqvist O, Fürst C-J, Brännström M. End-of-life care in residential care homes: a retrospective study of the perspectives of family members using the VOICES questionnaire. Scand J Caring Sci. 2017;31:72–84.

    Article  PubMed  Google Scholar 

  18. Bainbridge D, Seow H. Palliative Care Experience in the Last 3 Months of Life: A Quantitative Comparison of Care Provided in Residential Hospices, Hospitals, and the Home From the Perspectives of Bereaved Caregivers. Am J Hosp Palliat Care. 2018;35:456–63.

    Article  PubMed  Google Scholar 

  19. Beccaro M, Caraceni A, Costantini M. End-of-life care in Italian hospitals: quality of and satisfaction with care from the caregivers’ point of view—results from the Italian Survey of the Dying of Cancer. J Pain Symptom Manage. 2010;39:1003–15.

    Article  PubMed  Google Scholar 

  20. Beccaro M, Costantini M, Merlo DF. Inequity in the provision of and access to palliative care for cancer patients. Results from the Italian survey of the dying of cancer (ISDOC). BMC Public Health. 2007;7:66.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Brännström M, Fürst CJ, Tishelman C, Petzold M, Lindqvist O. Effectiveness of the Liverpool care pathway for the dying in residential care homes: An exploratory, controlled before-and-after study. Palliat Med. 2016;30:54–63.

    Article  PubMed  Google Scholar 

  22. Burt J, Shipman C, Richardson A, Ream E, Addington-Hall J. The experiences of older adults in the community dying from cancer and non-cancer causes: a national survey of bereaved relatives. Age Ageing. 2010;39:86–91.

    Article  PubMed  Google Scholar 

  23. Chini F, Giorgi Rossi P, Costantini M, Beccaro M, Borgia P. Validity of caregiver-reported hospital admission in a study on the quality of care received by terminally ill cancer patients. J Clin Epidemiol. 2010;63:103–8.

    Article  CAS  PubMed  Google Scholar 

  24. Costantini M, Beccaro M, Merlo F. The last three months of life of Italian cancer patients. Methods, sample characteristics and response rate of the Italian Survey of the Dying of Cancer (ISDOC). Palliat Med. 2005;19:628–38.

    Article  PubMed  Google Scholar 

  25. Costantini M, Pellegrini F, Di Leo S, Beccaro M, Rossi C, Flego G, et al. The Liverpool Care Pathway for cancer patients dying in hospital medical wards: a before-after cluster phase II trial of outcomes reported by family members. Palliat Med. 2014;28:10–7.

    Article  PubMed  Google Scholar 

  26. Donnelly S, Prizeman G, Coimín D, Korn B, Hynes G. Voices that matter: end-of-life care in two acute hospitals from the perspective of bereaved relatives. BMC Palliat Care. 2018;17:117.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Elkington H, White P, Addington-Hall J, Higgs R, Edmonds P. The healthcare needs of chronic obstructive pulmonary disease patients in the last year of life. Palliat Med. 2005;19:485–91.

    Article  PubMed  Google Scholar 

  28. ElMokhallalati Y, Woodhouse N, Farragher T, Bennett MI. Specialist palliative care support is associated with improved pain relief at home during the last 3 months of life in patients with advanced disease: analysis of 5-year data from the national survey of bereaved people (VOICES). BMC Med. 2019;17:50.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Frey R, Robinson J, Old A, Raphael D, Gott M. Factors associated with overall satisfaction with care at the end-of-life: Caregiver voices in New Zealand. Health Soc Care Community. 2020;28:2320–30.

    Article  PubMed  Google Scholar 

  30. Frey R, Williams L, Trussardi G, Black S, Robinson J, Moeke-Maxwell T, Gott M. The Views of Informal Carers’ Evaluation of Services (VOICES): Toward an adaptation for the New Zealand bicultural context. Palliat Support Care. 2017;15:67–76.

    Article  PubMed  Google Scholar 

  31. Hughes R, Saleem T, Addington-Hall J. Towards a culturally acceptable end-of-life survey questionnaire: a Bengali translation of VOICES. Int J Palliat Nurs. 2005;11:116–23.

    Article  PubMed  Google Scholar 

  32. Hunt KJ, Shlomo N, Addington-Hall J. End-of-life care and achieving preferences for place of death in England: results of a population-based survey using the VOICES-SF questionnaire. Palliat Med. 2014;28:412–21.

    Article  PubMed  Google Scholar 

  33. Hunt KJ, Shlomo N, Addington-Hall J. End-of-life care and preferences for place of death among the oldest old: results of a population-based survey using VOICES-Short Form. J Palliat Med. 2014;17:176–82.

    Article  PubMed  Google Scholar 

  34. Ingleton C, Morgan J, Hughes P, Noble B, Evans A, Clark D. Carer satisfaction with end-of-life care in Powys, Wales: a cross-sectional survey. Health Soc Care Community. 2004;12:43–52.

    Article  PubMed  Google Scholar 

  35. Johnston B, Patterson A, Bird L, Wilson E, Almack K, Mathews G, Seymour J. Impact of the Macmillan specialist Care at Home service: a mixed methods evaluation across six sites. BMC Palliat Care. 2018;17:36.

    Article  PubMed  PubMed Central  Google Scholar 

  36. Karlsen S, Addington-Hall JM. How do cancer patients who die at home differ from those who die elsewhere? Palliat Med. 1998;12:279–86.

    Article  CAS  PubMed  Google Scholar 

  37. Ó. Coimín D, Prizeman G, Korn B, Donnelly S, Hynes G. Dying in acute hospitals: voices of bereaved relatives. BMC Palliat Care. 2019;18:91.

    Article  Google Scholar 

  38. O’Sullivan A, Alvariza A, Öhlen J, Håkanson C. Bereaved Family Members’ Satisfaction with Care during the Last Three Months of Life for People with Advanced Illness. Healthc (Basel). 2018;6:E130.

    Article  Google Scholar 

  39. O’Sullivan A, Öhlen J, Alvariza A, Håkanson C. Adaptation and validation of the VOICES (SF) questionnaire - for evaluation of end-of-life care in Sweden. Scand J Caring Sci. 2017. https://doi.org/10.1111/scs.12542.

    Article  PubMed  Google Scholar 

  40. Pungchompoo W, Suwan P, Kunapun S, Pungchompo S, Tungpunkom P. Experiences of symptoms and health service preferences among older people living with chronic diseases during the last year of life. Int J Palliat Nurs. 2019;25:129–41.

    Article  PubMed  Google Scholar 

  41. Reid K, Kirk R, Barnett P, Richardson A, Ahuriri-Driscoll A. VOICES: South Island pilot survey of bereaved people. N Z Med J. 2020;133:64–72.

    PubMed  Google Scholar 

  42. Robinson J, Frey R, Raphael D, Old A, Gott M. Difficulties in navigating the intersection of generalist and specialist palliative care services: A cross-sectional study of bereaved family’s experiences of care at home in New Zealand. Health Soc Care Community. 2021. https://doi.org/10.1111/hsc.13381.

    Article  PubMed  Google Scholar 

  43. Rogers A, Karlsen S, Addington-Hall J. ‘All the services were excellent. It is when the human element comes in that things go wrong’: dissatisfaction with hospital care in the last year of life. J Adv Nurs. 2000;31:768–74.

    Article  CAS  PubMed  Google Scholar 

  44. Ross L, Neergaard MA, Petersen MA, Groenvold M. Measuring the quality of end-of-life care: Development, testing, and cultural validation of the Danish version of Views of Informal Carers’ Evaluation of Services-Short Form. Palliat Med. 2017. https://doi.org/10.1177/0269216317740274.

    Article  PubMed  Google Scholar 

  45. Seow H, Bainbridge D, Bryant D, Guthrie D, Urowitz S, Zwicker V, Marshall D. The CaregiverVoice Survey: A Pilot Study Surveying Bereaved Caregivers To Measure the Caregiver and Patient Experience at End of Life. J Palliat Med. 2016;19:712–9.

    Article  PubMed  Google Scholar 

  46. Seow H, Bainbridge D, Brouwers M, Pond G, Cairney J. Validation of a modified VOICES survey to measure end-of-life care quality: the CaregiverVoice survey. BMC Palliat Care. 2017;16:44.

    Article  PubMed  PubMed Central  Google Scholar 

  47. Williams L, Trussardi G, Black S, Moeke-Maxwell T, Frey R, Robinson J, Gott M. Complex contradictions in conceptualisations of ‘dignity’ in palliative care. Int J Palliat Nurs. 2018;24:12–21.

    Article  PubMed  Google Scholar 

  48. Young AJ, Rogers A, Addington-Hall JM. The quality and adequacy of care received at home in the last 3 months of life by people who died following a stroke: a retrospective survey of surviving family and friends using the Views of Informal Carers Evaluation of Services questionnaire. Health Soc Care Community. 2008;16:419–28.

    Article  PubMed  Google Scholar 

  49. Young AJ, Rogers A, Dent L, Addington-Hall JM. Experiences of hospital care reported by bereaved relatives of patients after a stroke: a retrospective survey using the VOICES questionnaire. J Adv Nurs. 2009;65:2161–74.

    Article  PubMed  Google Scholar 

  50. Harkness JA, van de Vijver FJR, Mohler PP, editors. Cross-cultural survey methods. Hoboken: Wiley; 2003.

    Google Scholar 

  51. Behr D. Assessing the use of back translation: the shortcomings of back translation as a quality testing method. Int J Soc Res Methodol. 2017;20:573–84.

    Article  Google Scholar 

  52. Eccles DW, Arsal G. The think aloud method: what is it and how do I use it? Qualitative Research in Sport. Exerc Health. 2017;9:514–31.

    Article  Google Scholar 

  53. Behr D, Zabal A. Documenting Survey Translation. Mannheim: GESIS - Leibniz Institute for the Social Sciences (GESIS- Survey Guidelines); 2020.

    Google Scholar 

  54. Knop J, Dust G, Kasdorf A, Schippel N, Rietz C, Strupp J, Voltz R. Unsolved problems and unwanted decision-making in the last year of life: A qualitative analysis of comments from bereaved caregivers. Pall Supp Care. 2022:1–9. https://doi.org/10.1017/S1478951522000165.

  55. Vennedey V, Dust G, Schippel N, Shukri A, Strupp J, Rietz C, et al. Patient-centered care during the last year of life: adaptation and validation of the German PACIC short form for bereaved persons as proxies (PACIC-S9-proxy). BMC Palliat Care. 2020;19:177.

    Article  PubMed  PubMed Central  Google Scholar 

  56. Voltz R, Dust G, Schippel N, Hamacher S, Payne S, Scholten N, et al. Improving regional care in the last year of life by setting up a pragmatic evidence-based Plan-Do-Study-Act cycle: results from a cross-sectional survey. BMJ Open. 2020;10:e035988.

    Article  PubMed  PubMed Central  Google Scholar 

  57. Aaronson N, Alonso J, Burnam A, Lohr KN, Patrick DL, Perrin E, Stein RE. Assessing health status and quality-of-life instruments: attributes and review criteria. Qual Life Res. 2002;11:193–205.

    Article  PubMed  Google Scholar 

  58. Cronbach LJ, Meehl PE. Construct validity in psychological tests. Psychol Bull. 1955;52:281–302.

    Article  CAS  PubMed  Google Scholar 

  59. Office for National Statistics. National Survey of Bereaved People (VOICES): England. 2015. 2016. https://www.ons.gov.uk/peoplepopulationandcommunity/healthandsocialcare/healthcaresystem/bulletins/nationalsurveyofbereavedpeoplevoices/england2015. Accessed 8 Feb 2022.

  60. Federal Joint Committee. Directive on the regulation of specialised outpatient palliative care (§ 37b, paragraph 1, Social Code Book V). 2008. https://www.g-ba.de/richtlinien/64/. Accessed 22 Sep 2022.

  61. Schneider W, Eichner E, Thoms U, Stadelbacher S, Kopitzsch F. Zur Praxis von SAPV in Bayern: Wirksamkeit, Struktur-/prozesseffekte und ländliche Versorgung. [Specialised out-patient palliative care (SAPV) in Bavaria: efficiency, structural and process-related effects and rural care]. Gesundheitswesen. 2015;77:219–24.

    CAS  PubMed  Google Scholar 

  62. Blaschke K, Hermes-Moll K, Lappe V, Ihle P, Baumann W, Schubert I. Krebserkrankung und das letzte Lebensjahr: Palliative Leistungen, Krankenhausaufenthalte und Sterbeort. [Cancer Patients and the Last Year of Life: Palliative Care, Hospitalization and Place of Death]. Gesundheitswesen. 2019. https://doi.org/10.1055/a-0829-6604.

    Article  PubMed  Google Scholar 

  63. Moens K, Higginson IJ, Harding R. Are there differences in the prevalence of palliative care-related problems in people living with advanced cancer and eight non-cancer conditions? A systematic review. J Pain Symptom Manage. 2014;48:660–77.

    Article  PubMed  Google Scholar 

  64. Murray SA, Sheikh A. Palliative Care Beyond Cancer: Care for all at the end of life. BMJ. 2008;336:958–9.

    Article  PubMed  PubMed Central  Google Scholar 

  65. Streiner DL, Norman GR. Health measurement scales: A practical guide to their development and use. 4th ed. Oxford: Oxford Univ. Pr; 2008.

    Book  Google Scholar 

  66. Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16:297–334.

    Article  Google Scholar 

  67. Schippel N, Dust G, Reeken C von, Voltz R, Strupp J, Rietz C. Can we determine burdensome transitions in the last year of life based on time of occurrence and frequency? An explanatory mixed-methods study. Pall Supp Care. 2021. https://doi.org/10.1017/S1478951521001395.

    Article  Google Scholar 

  68. Pungchompoo W. Experiences and health care needs of older people with End Stage Renal Disease managed without dialysis in Thailand during the last year of life. Southampton: University of Southampton, Faculty of Health Sciences; 2013.

    Google Scholar 

  69. Office for National Statistics. National Survey of Bereaved People (VOICES): Questionnaire changes in 2014. 2015. https://webarchive.nationalarchives.gov.uk/ukgwa/20160106062912/http://www.ons.gov.uk/ons/publications/re-reference-tables.html?edition=tcm%3A77-407293. Accessed 6 Sep 2022.

  70. Temel JS, Greer JA, Muzikansky A, Gallagher ER, Admane S, Jackson VA, et al. Early Palliative Care for Patients with Metastatic Non–Small-Cell. Lung Cancer. 2010;363:733–42.

    CAS  Google Scholar 

  71. Smith TJ, Temin S, Alesi ER, Abernethy AP, Balboni TA, Basch EM, et al. American Society of Clinical Oncology provisional clinical opinion: the integration of palliative care into standard oncology care. J Clin Oncol. 2012;30:880–7.

    Article  PubMed  Google Scholar 

  72. Baddeley AD. Human memory: Theory and practice. Boston: Allyn and Bacon; 1990.

    Google Scholar 

  73. Bower GH. Mood and memory. Am Psychol. 1981;36:129–48.

    Article  CAS  PubMed  Google Scholar 

  74. Fakhoury W, McCarthy M, Addington-Hall J. Determinants of informal caregivers’ satisfaction with services for dying cancer patients. Soc Sci Med. 1996;42:721–31.

    Article  CAS  PubMed  Google Scholar 

  75. McPherson CJ, Addington-Hall JM. Judging the quality of care at the end of life: can proxies provide reliable information? Soc Sci Med. 2003;56:95–109.

    Article  CAS  PubMed  Google Scholar 

  76. Mathiowetz NA. Forgetting Events in autobiographical memory: Findings from a health care survey. 1988. http://www.asasrms.org/Proceedings/papers/1988_029.pdf. Accessed 22 Sep 2022.

  77. Stadt. Köln - Amt für Stadtentwicklung und Statistik. Kölner Zahlenspiegel. Köln: Stadt Köln; 2018.

    Google Scholar 

  78. Stadt Köln. - Amt für Stadtentwicklung und Statistik. Kölner Statistische Nachrichten – 1/2019: Statistisches Jahrbuch Köln 2018. Köln: Stadt Köln; 2019.

    Google Scholar 

  79. Department of Health. End of Life Care Strategy: Promoting high quality care for all adults at the end of life. London: DH; 2008.

    Google Scholar 

  80. Strupp J, Kasdorf A, Dust G, Hower KI, Seibert M, Werner B, et al. Last Year of Life Study-Cologne (LYOL-C) (Part II): study protocol of a prospective interventional mixed-methods study in acute hospitals to analyse the implementation of a trigger question and patient question prompt sheets to optimise patient-centred care. BMJ Open. 2021;11:e048681.

    Article  PubMed  PubMed Central  Google Scholar 

  81. Kremeike K, Ullrich A, Schulz H, Rosendahl C, Boström K, Kaur S, et al. Dying in hospital in Germany - optimising care in the dying phase: study protocol for a multi-centre bottom-up intervention on ward level. BMC Palliat Care. 2022;21:67.

    Article  PubMed  PubMed Central  Google Scholar 

  82. Strupp J, Hanke G, Schippel N, Pfaff H, Karbach U, Rietz C, Voltz R. Last Year of Life Study Cologne (LYOL-C): protocol for a cross-sectional mixed methods study to examine care trajectories and transitions in the last year of life until death. BMJ Open. 2018;8:e021211.

    Article  PubMed  PubMed Central  Google Scholar 

  83. Kasdorf A, Dust G, Vennedey V, Rietz C, Polidori MC, Voltz R, Strupp J. What are the risk factors for avoidable transitions in the last year of life? A qualitative exploration of professionals’ perspectives for improving care in Germany. BMC Health Serv Res. 2021;21:147.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

This study was made possible by the collaborative efforts of health and social care practitioners in Cologne. We further thank everyone who contributed their time and expertise, in particular the study participants.

This study was conducted within the Cologne Research and Development Network (CoRe-Net) in cooperation with Christian Albus, Lena Ansmann, Frank Jessen, Ute Karbach, Ludwig Kuntz, Holger Pfaff, Ingrid Schubert, Frank Schulz-Nieswandt, and Nadine Scholten.

Funding

This work was supported by the German Federal Ministry of Education and Research, grant number #01GY1606.

Open Access funding enabled and organized by Projekt DEAL. We acknowledge support for the Article Processing Charge from the DFG (German Research Foundation, 491454339).

Author information

Authors and Affiliations

  1. Faculty of Medicine and University Hospital, Department of Palliative Medicine, University of Cologne, 50924, Cologne, Germany

    Gloria Dust, Nicolas Schippel, Julia Strupp & Raymond Voltz

  2. Faculty of Medicine and University Hospital, Institute of Health Economics and Clinical Epidemiology, University of Cologne, Cologne, Germany

    Stephanie Stock

  3. Faculty of Medicine and University Hospital, Center for Integrated Oncology Aachen, University of Cologne, Bonn Cologne Duesseldorf (CIO ABCD), Cologne, Germany

    Raymond Voltz

  4. Faculty of Medicine and University Hospital, Center for Health Services Research, University of Cologne, Cologne, Germany

    Raymond Voltz

  5. Faculty of Educational and Social Sciences, Department of Educational Science, University of Education Heidelberg, Heidelberg, Germany

    Christian Rietz

Authors
  1. Gloria Dust
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nicolas Schippel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stephanie Stock
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Julia Strupp
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Raymond Voltz
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Christian Rietz
    View author publications

    You can also search for this author in PubMed Google Scholar

Consortia

On behalf of the CoRe-Net Co-applicants

Contributions

GD and NS designed the survey instrument and collected data. GD analysed data and drafted the manuscript. CR, JS, and RV are principal investigators and responsible for the study design, project management, data analysis and dissemination. SS contributed to study design and data analysis. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Gloria Dust.

Ethics declarations

Ethics approval and consent to participate

The study was approved by the Ethics Commission of the Faculty of Medicine of Cologne University (#17–188), all methods were carried out in accordance with relevant guidelines and regulations (declaration of helsinki). Relatives received written information about the study and data protection and were required to give written informed consent for participation. For the deceased person’s data the informed consent was obtained from the next of the kin.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dust, G., Schippel, N., Stock, S. et al. Quality of care in the last year of life: adaptation and validation of the German “Views of Informal Carers’ Evaluation of Services – Last Year of Life – Cologne”. BMC Health Serv Res 22, 1433 (2022). https://doi.org/10.1186/s12913-022-08700-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12913-022-08700-0

Keywords

BMC Health Services Research

ISSN: 1472-6963

Contact us