The Prostate Care Questionnaire for Patients (PCQ-P): Reliability, validity and acceptability
© Tarrant et al; licensee BioMed Central Ltd. 2009
Received: 21 July 2009
Accepted: 04 November 2009
Published: 04 November 2009
In England, prostate cancer patients report worse experience of care than patients with other cancers. However, no standard measure of patient experience of prostate cancer care is currently available. This paper describes an evaluation of the reliability, validity and acceptability of the PCQ-P, a newly developed instrument designed to measure patient experience of prostate cancer care.
The reliability, acceptability and validity of the PCQ-P were tested through a postal survey and interviews with patients. The PCQ-P was posted to 1087 prostate cancer patients varying in age, occupation, and overall health status, sampled from five hospitals in England. Nonresponders received one reminder. To assess criterion validity, 935 patients were also sent sections of the National Centre for Social Research Shortened Questionnaire; and to assess test-retest reliability, 296 patients who responded to the questionnaire were resent it a second time three weeks later. A subsample of 20 prostate cancer patients from one hospital took part in qualitative interviews to assess validity and acceptability of the PCQ-P. Acceptability to service providers was evaluated based on four hospitals' experiences of running a survey using the PCQ-P.
Questionnaires were returned by 865 patients (69.2%). Missing data was low across the sections, with the proportion of patients completing less than 50% of each section ranging from 4.5% to 6.9%. Across the sections of the questionnaire, internal consistency was moderate to high (Cronbach's alpha ranging from 0.63 to 0.80), and test-retest stability was acceptable (intraclass correlation coefficients ranging from 0.57 to 0.73). Findings on criterion validity were significant. Patient interviews indicated that the PCQ-P had high face validity and acceptability. Feedback from hospitals indicated that they found the questionnaire useful, and highlighted important considerations for its future use as part of quality improvement initiatives.
The PCQ-P has been found to be acceptable to patients and service providers, and is ready for use for the measurement of patient experience in routine practice, service improvement programmes, and research.
In this paper we report the reliability and validity of a questionnaire to measure the experience of prostate cancer care. We also report the acceptability of the questionnaire to patients and to hospitals providing care for patients with prostate cancer.
In England, patients with prostate cancer report worse experience of care than patients with other cancers [1, 2], and the provision of readily usable measures was one element of a wider initiative designed to improve services [3, 4]. While care outcomes can be assessed through standard measures of health-related quality of life , which include reports of general health and impact on functioning, no standard measures of patient experience of the process and delivery of prostate cancer care are available . In recent years, patient surveys have tended to focus on patient experience of care rather than on satisfaction with care. Satisfaction is a complex concept incorporating patients' expectations, their experience of what happened in care, their feelings about the care they received, and the importance to the patient of the aspect of care concerned. Measures of patients' experience - which comprise patients' reports of what happened in specific aspects of the delivery of care - are simpler to interpret and act upon than measures of satisfaction . Surveys of experience are now generally preferred by providers, and considered helpful in evaluating and improving services.
In the wake of the NHS next stage review, all NHS organisations will be required to gather and publish information about patient experience of care, as part of a process of working towards high quality of care for all . The availability of valid and acceptable measures of patient experience is crucial to ensuring that meaningful data are collected. To be suitable for wide use, measures of experience need to be developed systematically to address the issues that are important to patients, to be readily understood and acceptable by patients, and to meet standards of reliability and validity. They should also be acceptable to providers, enabling them to assess specific stages of care, depending on the provider's focus for quality improvement. Our aim was to develop a robust and acceptable measure suitable for use in routine practice and research. This paper describes such a measure of patient experience of prostate cancer care - the Prostate Care Questionnaire for Patients (PCQ-P) - and reports on the formal evaluation of the measure's reliability, validity and acceptability to patients and service providers. A companion measure for use with carers of men with prostate cancer has also been developed .
Development and characteristics of the PCQ-P
The PCQ-P is a measure developed from preliminary research designed to determine the issues most important to prostate cancer patients, including a literature review of the experiences of patients of prostate cancer care, and interviews with patients and service providers [10–12]. Key issues included: information and explanations, involvement in decision making, provision of advice and support, delays in care, choice, coordination of care, and practical issues (such as travel, and facilities at the hospital). Questions on these issues were developed, phrased in the words used by patients. Thorough piloting was undertaken . The measure, along with a user guide, is available online for download and use . The questionnaire is divided into five sections: GP visits and referral (Section A, 17 questions); tests at the hospital (Section B, 19 questions); diagnosis and treatment decision (Section C, 30 questions); treatment and discharge (Section D, 25 questions); and monitoring (Section E, 15 questions). There is a sixth section to collect health and sociodemographic information (Section F, 10 questions). Sections can be administered separately, or in appropriate combinations (e.g. it may be useful to give sections A, B, and C in combination to patients who have recently received a diagnosis of prostate cancer). A short version of the questionnaire, comprising 24 questions chosen to include the most important issues to patients, is also available [10, 13].
Features of hospitals: testing the questionnaire for reliability, validity, and acceptability to patients
Location in England
Measures of reliability, validity and acceptability to patients
To assess criterion validity, 935 patients (in hospitals 1 to 4) were also sent sections of the National Centre for Social Research Shortened Questionnaire (NCSRSQ), a questionnaire designed to measure patients' experience of care for several different forms of cancer . To assess test-retest reliability, 296 patients from two hospital sites (hospitals 1 and 2) were posted the PCQ-P questionnaire again three weeks later.
In addition, 20 patients from hospital 5, who had completed all sections of the PCQ-P, took part in semi-structured interviews to explore acceptability and face validity. Eleven face-toface and nine telephone interviews were carried out. Interviews were not transcribed, but notes were taken by the interviewer during the interview.
Key properties of the questionnaire sections, including aspects of validity, reliability, and acceptability to patients were analysed. For the purpose of testing the properties of the questionnaire, overall scores were calculated for each section of the questionnaire, by summing scores across questions and converting to a score out of 100, with higher scores indicating more positive experiences of care . All statistical analyses were conducted using SPSS version 16.0.
Acceptability to patients
Acceptability was evaluated by examining completion rates for individual questions and questionnaire sections, and by analysing distributions of responses for individual questions. Acceptability was also assessed in patient interviews, by asking patients how they felt about the experience of completing the questionnaire.
Criterion validity was assessed by examining Pearson correlations between scores on sections B to E of the PCQ-P and the NCSRSQ. The NCSRSQ did not contain comparable questions for section A of the PCQ-P. The NCSRSQ is a measure of problems in care, where in contrast to the PCQ-P a higher score indicates a less positive experience of care, therefore a negative correlation between the scores on the two questionnaires was expected. The NCSRQ is a generic questionnaire rather than being specific to prostate cancer, and a medium-sized correlation co-efficient (-0.3 to -0.5) was expected as an indicator of validity [15, 16]. Face validity was investigated through patient interviews, in which patients were asked to describe their experiences of care in conjunction with inspection of their responses to the questionnaire, as well as being asked directly whether there were any important aspects of care that were not included in the questionnaire. Content validity was assessed through comparing the results of exploratory principal components analysis (PCA) with themes identified through the preliminary research [11, 12].
Internal consistency reliability for each section was measured using Cronbach's alpha . Stability reliability was assessed using intraclass correlation coefficients (ICCs) between scores on the first and second completion of the questionnaire. Stability was also assessed by calculating the percentages of patients answering each question the same way on the first and second completion of the questionnaire.
Usability and acceptability to service providers
Features of hospitals: testing the questionnaire for usability and acceptability to service providers
Location in England
Acceptability to patients
Demographic characteristics and overall health status of patient sample: reliability and validity testing
Current or most recent occupation
Descriptive statistics of overall scores from the five sections of the questionnaire
% with lowest possible score
% with highest possible score
In the patient interviews, patients described the questions as easy to understand, and the majority did not report any problems with filling in the questionnaire. One patient commented that his case did not fit well with the sequence of sections as they were presented in the full questionnaire (compiled from all five subsections), and two patients felt that the full questionnaire was too long. Three patients emphasised that each section should be administered as soon as possible after the relevant phase of treatment or care, as they found it difficult to remember in detail events from several months previously. Patients suggested that the sections should be divided and used as separate questionnaires, or with just two to three sections in combination, with appropriate sections being administered as soon as possible after the relevant stage of care.
Out of 935 patients, 592 (63.3%) completed both a PCQ-P and a NCSR questionnaire. Response rates were 224/431 (52%) for Sections A, B, C, and F, and 368/504 (73%) for Sections D, E, and F. Pearson's correlation coefficient between Sections B and C and the first half NCSRSQ was -0.23 (p = 0.002, N = 175), and between Sections D and E and the second half NCSRSQ was -0.46 (p < 0.001, N = 201). These correlations are in the expected direction, the first is small and the second is medium [15, 16], and both are significant at p < 0.005.
Face and content validity
All interview participants indicated that the questionnaires covered important aspects of care. However, several patients highlighted gaps in the section on discharge. The particular issues identified as important were: knowing what to expect in terms of recovery time and side effects; and knowing how to obtain appropriate supplies after discharge (such as continence pads). This led to two questions on these issues being added to the questionnaire.
Content validity was assessed through exploratory principal components analysis. Analysis identified three to four components for each section of the questionnaire (see additional file 1: exploratory PCA). For example in section A (GP visits and referral), three components emerged, and inspection of questions within each component suggested that the components related to: 'explanation', 'taking the problem seriously' and 'experience of referral'. Comparison of these components with themes from the initial patient interviews and literature review [11, 12] confirmed that the key aspects of care identified in the preliminary research were satisfactorily incorporated in the PCQ-P.
Internal consistency reliability
Reliability: Internal consistency and stability of the five sections of the PCQ-P
Stability: Test-retest reliability
1st mailing mean
2nd mailing mean
19.6, 11.5-100 (60)
18.7, 16.7-100 (62)
15.3, 28.5-100 (73)
10.6, 39.7-99.4 (72)
12.0, 55.0-100 (62)
14.5, 41.4-100 (49)
21.9, 16.7-100 (61)
19.7, 18.8-100 (60)
Out of 296 patients, 148 (50%) completed the retest questionnaire; 79/125 (63.2%) completed Sections A, B, C, and F, and 69/171 (40.4%) completed Sections D, E, and F. Patients completing retest questionnaires did not differ significantly from other patients in terms of age, health status, ethnic group, or employment status (p > 0.05 in each case). The test-retest ICCs for the five sections were between 0.57 and 0.73, and all were significant at p < 0.001 (Table 5), indicating acceptable reliability . The consistency of responses to individual questions was high, between 52.6% and 100% of patients answering identically on the first and second mailing. Most questions (94, 88.7%), were answered perfectly consistently by over 70% of responders. The questions where responses were less consistent were those with a higher number of response options, and for these questions the difference between responses on the first and second completion of the questionnaire tended to be a shift to the neighbouring response option, for example, from 'good' to 'very good'.
Acceptability to service providers
Three of the four participating hospitals chose to administer the questionnaire by post, and one used a combination of postal administration and face-to-face administration in urology clinics. None of the hospitals reported major difficulties with running the survey. However, the administration of the survey was seen as time consuming, and hospitals found this problematic; one hospital in particular had difficulty finding staff with time available to run the survey. Hospitals felt that the provision of support to run the survey would be helpful, and one hospital suggested that administering the survey prospectively to patients over a longer period of time would be more manageable and would produce valuable data.
Agreement to conduct the survey had initially been obtained from a consultant in each hospital, and some staff felt that an essential first stage would be a meeting involving all staff who would have a role in the administration of the survey. This would promote ownership of the survey, ensure local support, and provide a setting where input of time could be negotiated. There were mixed feelings about the involvement of an external organisation to coordinate the survey, possibly on a national basis; staff described a preference for using surveys as part of their own internal quality improvement process, and were resistant to the idea of surveys being used as an "external stick" (Hospital B) to raise performance, or being linked to financial incentives.
The evaluation of the PCQ-P demonstrates that the instrument incorporates issues that are important to prostate cancer patients, has good reliability and validity, and is acceptable to patients and service providers. The PCQ-P can be used flexibly - single sections can be used independently or in combination with other relevant sections, and a short version  covering the whole patient journey is also available.
Some limitations should be noted. The evaluation of criterion validity was hampered by the absence of any satisfactory criterion. No other suitable measure of patient experience of prostate cancer care was available, hence a generic patient experience questionnaire (NCSRSQ) was used as an indicative criterion. Nevertheless, the findings of the criterion validity check show small to medium sized, but significant, correlations between scores on the PCQ-P and the NCSRSQ. Taken in conjunction with other tests of the PCQ-P's properties, this provides support for the validity of the PCQ-P as a measure of patient experience of prostate cancer care. The findings of this study confirm that the PCQ-P is suitable for use in service improvement programmes. The study also highlights the approach to using the questionnaire preferred by patients (sections of the questionnaire administered separately, as soon as possible after the corresponding care stage), and indicates important considerations for hospitals planning to use the questionnaire, including ensuring that staff time is made available to administer the questionnaire. The development of a version of the questionnaire that could be self-administered via computer would be valuable in addressing this problem.
Attempts to develop and improve services should include assessment of patient experience when evaluating the impact of changes. In the UK, the greater involvement of commissioners in the future will present an additional opportunity to monitor and improve patient experience, and patient experience surveys should play a role in this . The PCQ-P could be used in future national patient surveys. The use of a robust and acceptable questionnaire, specifically designed to collect experiences of prostate cancer care, would allow GPs and hospitals to evaluate their performance on the aspects of care that are most important to prostate cancer patients, and to compare their performance to other, similar GP practices and hospitals. The establishment of benchmark scores for the questionnaire would be of value in this context, and could be established via a national survey.
This study has demonstrated how a sophisticated patient experience questionnaire can be developed and evaluated systematically. Although the development process is prolonged, thoroughness in development and testing provides confidence in the data produced by the measure, and increases the value of the data for use in service evaluation and research. The PCQ-P could provide a starting point for the development of instruments for other cancer groups, as many of the issues that it covers are relevant to other cancers. For example, the need for information and support throughout care is pertinent to the other common cancers, including lung, colorectal, and breast cancer. New measures would require careful testing before use. Nonetheless, the detailed process undertaken to develop the prostate care questionnaire provides a springboard for developing instruments for other cancers.
We wish to thank all the patients who participated in the study, and the hospital staff who helped with the administration of the postal surveys. We also thank Kerry Sproston, National Centre for Social Research, for her comments on the early drafts of the questionnaire. This project was funded by the National Institute for Health Research Service Delivery and Organisation Programme (project number SDO/77/2004).). Ethics approval for the study was granted by the Leicestershire, Northamptonshire & Rutland Research Ethics Committee 1 (reference 7248; acting MREC). The views and opinions expressed herein are those of the authors and do not necessarily reflect those of the Department of Health.
- Airey C, Becher H, Erens B, Fuller E: National Survey of NHS Patients - Cancer: National Overview 1999/2000. 2002, London: Department of HealthGoogle Scholar
- National Audit Office: National Patient Survey (Cancer). 2004, Oxford: Picker InstituteGoogle Scholar
- Department of Health: The NHS Cancer Plan and the new NHS: Providing a patientcentred service. 2004, London: Department of HealthGoogle Scholar
- Department of Health: Cancer Reform Strategy. 2007, London: Department of HealthGoogle Scholar
- Ware JE, Sherbourne CD: The MOS 36-item short-form health survey (SF-36): I. Conceptual framework and item selection. Med Care. 1992, 30 (6): 473-483. 10.1097/00005650-199206000-00002.View ArticlePubMedGoogle Scholar
- McNaughton M, Collins M, Walker-Corkery E, Barry MJ: Health-related quality of life, satisfaction and economic outcome measures in studies of prostate cancer screening and treatment 1990-2000. J Natl Cancer I Monogr. 2004, 33: 78-101. 10.1093/jncimonographs/lgh016.View ArticleGoogle Scholar
- Coulter A: Opinion and Experience: Do They Concur?. 2005, Oxford: Picker Institute Europe, [http://www.pickereurope.org/Filestore/Articles/2005/Populus_What_patients_really_want.pdf]Google Scholar
- Darzi A: High Quality Care for All. 2008, London, Department of HealthGoogle Scholar
- Sinfield P, Baker R, Tarrant C, Agarwal S, Colman AM, Steward W, Kockelbergh R, Mellon JK: Development of measures of patient and carer experience of prostate cancer care. 3. Reliability, validity and acceptability of the Prostate Care Questionnaire for Carers (PCQ-C). BMC Health Serv Res.Google Scholar
- Baker R, Sinfield P, Agarwal S, Tarrant C, Mellon K, Steward W, Colman AM, Kockelbergh R, Sproston K: Prostate Cancer Care: Improving Measures of the Patient Experience Report for the National Co-ordinating Centre for NHS Service Delivery and Organisation R&D. London: NCCSDO. 2007, [http://www.sdo.nihr.ac.uk/sdo772004.html]Google Scholar
- Sinfield P, Baker R, Agarwal S, Tarrant C: Patient-centred care: what are the experiences of prostate cancer patients and their partners?. Patient Educ Couns. 2008, 73: 91-6. 10.1016/j.pec.2008.05.001.View ArticlePubMedGoogle Scholar
- Sinfield P, Baker R, Camosso-Stefinovic J, Colman AM, Tarrant C, Mellon JK, Steward W, Kockelbergh R, Agarwal S: Men's and carers' experiences of care for prostate cancer: a narrative literature review. Health Expect. 2009, 12 (3): 301-312. 10.1111/j.1369-7625.2009.00546.x.View ArticlePubMedGoogle Scholar
- Prostate cancer care questionnaires and user guide. [http://www.sdo.nihr.ac.uk/files/adhoc/77-questionaires2.pdf]
- Prescott A: National Survey of NHS Patients - Cancer: Analysis of Themes. 2004, London: Department of HealthGoogle Scholar
- Cohen J: A power primer. Psych Bull. 1992, 112: 155-9. 10.1037/0033-2909.112.1.155.View ArticleGoogle Scholar
- Cohen J: Statistical power analysis for the behavioral sciences. Edited by: Hillsdale NJ. 1988, Country: Lawrence Erlbaum Associates, 2Google Scholar
- Cronbach LJ: Coefficient alpha and the internal structure of tests. Psychometrika. 1951, 16: 297-333. 10.1007/BF02310555.View ArticleGoogle Scholar
- Department of Health: Methodology of the national survey of NHS cancer patients. [http://www.dh.gov.uk/en/Publicationsandstatistics/PublishedSurvey/NationalsurveyofNHSpatients/Nationalsurveycancer/DH_4001299]
- Pasta DJ, Suhr D: Creating scales from questionnaires: PROC VARCLUS vs. factor analysis. Proceedings of the Twenty-Ninth Annual SAS Users Group International Conference (Paper 205-29). 2004, Cary, NC: SAS Institute IncGoogle Scholar
- Department of Health: World Class Commissioning: Competencies. 2007, London: Department of HealthGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6963/9/199/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.