The National Adult Inpatient Survey conducted in the English National Health Service from 2002 to 2009: how have the data been used and what do we know as a result?

Questionnaire and Survey Background

There are numerous patient experience surveys conducted annually in Europe, the USA and Canada [7]; Delnoij (2009) states that these endeavour to obtain "detailed reports of what actually happened to patients during a hospital stay" not just an overall measure of satisfaction. In England, the adult inpatient survey seeks to collect the views of recent patients about "how good the hospitals are and how they can be improved" [8] as well as some socio-demographic information about each respondent.

The two main current uses of the inpatient survey are to aid the regulatory functions of the Care Quality Commission (CQC) and to assist local improvement. The patient survey comprises over 20 per cent of the items in the CQC's Quality and Risk Profiles (QRP) [9], which assess compliance with Essential Standards of Care. It is unclear to what extent acute NHS trusts use data locally; there appears to be no systematic collection of information about how these data are used within hospital Trusts.

Responsibility for the programme began with the Department of Health (DoH) and transferred to the Commission for Health Improvement (CHI) in 2003. In April 2004 authority passed to the Healthcare Commission and from October 2008 the survey has been overseen by the CQC. Official results and key findings have been reported annually, with trust level reports made available from inception [10].

The survey was designed by and is conducted through the Picker Institute Europe on behalf of the regulatory body. Questions for the survey were developed through pilot studies that sought to understand the issues that are of importance to patients [11].

The survey questions are organised under thirteen headings that cover the journey of the inpatient from arrival at the hospital to discharge. Most ask respondents to select one option from a set of pre-defined responses. Patients may also include their own comments in a section at the end.

Using detailed written guidance [12, 13], the surveys are conducted according to a standard protocol by an approved contractor on the trust's behalf or, in a small proportion, by the trust itself. For each trust, the sample consists of 850 consecutively discharged patients aged 16 and over. Questionnaires are posted to selected patients, and up to two reminders are sent to non-responders at 2-weekly intervals. For responders who are unable to read English, a free telephone line is provided that aims to connect responders to a translator who would complete the questionnaire on their behalf over the telephone.

The highest response rate (64 per cent) was achieved in 2002 [14]. Response rates have declined over the years to 50 per cent in 2010 [15].

Search strategy

Websites were selected by their relevance to the NHS, the national inpatient survey, following consultation with experts in the field [see Acknowledgements] and websites of organisations approved by the CQC to conduct the inpatient survey. Journals and databases were interrogated and a literature search completed by the Royal College of Nursing (RCN). For details of the search strategy and the RCN outcome, please see Additional file 1.

A scoping search was conducted in July 2010, followed by a full systematic investigation conducted by two independent researchers in October. The search was also updated by one researcher in January 2011. Search results were screened by the researchers based on title and date of publication and if these suggested that the paper fit the inclusion criteria, the abstract was read. Full papers were obtained for all unique research articles identified in the search as well as a group of papers that were recommended by experts in the area. These were read by the two reviewers who each formed a view as to whether they met the inclusion criteria. Where they disagreed, a third reader was consulted. Papers were read in full and further exclusions made.

Search terms

To ensure a consistent and replicable approach, PICO (population, intervention, comparison and outcome) methodology was adopted to create an extensive list of search terms, highlighting differences in phraseology, spelling and acronyms. The search terms were combined with lessons learnt from performing the preliminary search and adapted according to search engine capability. For a breakdown of search terms and mechanisms used to identify papers for review, please see Additional file 2.

Eligibility criteria

Identified papers made reference to a minimum of one year of national inpatient survey data. Papers reporting response rates only were excluded. Research comprising direct analysis of raw figures, or extracts from annual regulator-led findings, were both included. Articles reporting results at trust level that included national comparisons were accepted. Papers reporting results for one trust only, with no national referencing, were excluded. Research that made use of models which used the inpatient survey as one of numerous indicators, such as the star ratings system or DoH toolkit for analysts [17], were excluded as the inpatient survey was not the main focus. Papers that discussed the survey in general without including or analysing data were excluded, as were articles based on another paper without reporting independent analyses.

No exclusions were applied regarding ethnicity, gender, language [18] or country of origin. However, the search was limited to publications no earlier than January 2002 because the first survey was conducted in 2001/02. Research based solely on another survey, such as United States' National Inpatient Survey (NIS) or Patient Experiences Questionnaire, were excluded. Grey literature was beyond the scope of the search and consequently not included.

Controlling for bias

To ensure the search was comprehensive and to reduce the impact of researcher selection bias, two research fellows systematically and independently conducted searches of the literature and their findings were amalgamated. A third reader assisted with the process of deciding which papers should be included. Search terms, sources and eligibility criteria were pre-agreed to ensure consistency and make certain the requirements were fully understood. A wide variety of research studies were included and experts were consulted [see Acknowledgements], to ensure a fair and wide search.

Excluded papers

Of the 23 papers excluded, five addressed the surveys' impact in general without analysing survey data, these detailed:

Four articles discussed the findings of other papers that analysed data but did not report independent analyses of survey data [24–27]. Four more papers made no reference to the acute trusts adult inpatient survey:

Shipton et al. [32], Stevens et al. [33] and Davies et al. [34] also did not use data, although authors did adopt the star ratings system as a measure of performance. Peytremann-Bridevaux et al. [35] used an adaptation of the Picker inpatient survey questionnaire for a study in University Hospital of Geneva, but did not make use of the English national inpatient survey data. A paper by Crow et al. [36] used the survey data for one trust but did not refer to national data.

Two reports discussed ethnicity in terms of response rates and increasing representation. One reported the demographic profile of responders, but not their reported experiences of care [37]. The second reviewed literature outlining strategies to increase the response rate of minority groups [38]. Likewise, Jumaa [39] referred to the response rate of the inpatient survey with no experience data or reference to year.

Finally, two articles outlined local projects that made use of trust level survey results to improve local services and measure change. The first reviewed results alongside Patient Advice and Liaison Services (PALS) [40]. The second used survey results as a measure for the success of pain improvement techniques [41].

Annual survey reports

Annual key finding reports were identified for 2002 [14], 2004 [42], 2005 [43], 2006 [44], 2007 [45] and 2008 [46]. The Picker Institute Europe produced all of these on behalf of the regulatory body responsible for the survey at the time. In 2008, the key findings paper was accompanied by an historical comparisons document [47] and a web-based information page, which included a briefing report and trust level reports. In 2009 the historical comparisons document [48] and website [15] replaced the key findings report.

The first report in 2002 covered every question in the survey through descriptive explanations, bar charts and tables. Respondent demographic characteristics such as age, ethnicity and health status, were reported alongside Trust characteristics such as size and location. Each demographic variable was related to the questions. Older, male and white patients reported better experiences. Patients with poorer self-rated health status and those living in London reported less favourable experiences. Overall, 74 per cent rated the care they received as excellent (38 per cent) or very good (36 per cent).

In 2004, the report was reduced to descriptive national findings for key domains such as Patient Care and Treatment and Leaving Hospital. Results for each question were tabulated and also cross-tabulated to show percentage change between 2002 and 2004. All reported findings were statistically significant. Those with poorer health were more negative about the care they had received. Black, minority and ethnic (BME) groups, particularly people of South Asian origin, were more negative, though the differences were quite small relative to other factors. As in the previous survey, older people, men, elective patients, people living outside London and those who were treated in specialist Hospital Trusts reported more positive experiences. The most significant factor in explaining variations in patients' experiences was self-reported health status; those who reported their own health to be good, very good or excellent tended to report a more positive experience of care.

The 2005 report was more comprehensive than previous years. Cross-tabulated results included demographic data on respondents and non-respondents, and displayed 'all respondents' beside '18 and over' to allow for comparison to 2004. Only significant findings were included and the Bonferroni correction method applied. Due to the "differences in the sample" (16 and 17 year olds not included), results for the 2004 survey were not included in the annual reports from 2006 onwards [44].

The 2006 to 2008 surveys followed the same structure as 2005, with increasingly sophisticated analyses. Comparisons between years were tested for significance using Z-tests and the Bonferroni method. From 2008, tabulated results for the relevant years were simplified. Results were presented in full for all response options, with no further grouping or comparisons. Significant changes over time were sought by comparing the last two years (2007 and 2008) with the survey results from the first year (2002).

In 2004 selected 'free text' comments were included to support key findings. It was not until 2007 that the full free text comments were included in an additional summary report [49] which showed that nearly 60 per cent of respondents wrote at least one comment suggesting that these free text comments are also a rich source of data which have not been analysed in any other way apart from this one study conducted by the Picker Institute Europe in 2007.

Evidence based articles and reports

A number of reports and articles identified through the review referenced the inpatient survey as supporting evidence. The papers below quoted figures from the annual survey reports:

One article solely reported on the findings of the inpatient survey for 2009 [60], highlighting differences from year to year and concludedthat while some improvements have been made, there are "still key areas of concern" such as 45 per cent of patients reporting a lack of information regarding potential side effects of new medication.

The authors of one study adapted the Picker Institute questionnaire to conduct an in-house pilot to test how a hospice was performing when compared to national hospital results for 2007 [61]. Unfortunately the sample size was too small to draw any conclusions.

Finally, an internal publication produced by the Department of Health used factor analysis to determine a model that linked patient experience to satisfaction, based on 2001/02 inpatient survey data [62]. The paper also reported analysis produced by the University of Sheffield using inpatient survey data for 2003/04, which sought to explain the variation between patients' experiences in the five core domains identified in the NHS Outcomes Framework 2011/2012 [5] as set out above. In sum, there is a substantial body of work that has made use of the national inpatient survey. The fact that so many authors have drawn on these data for diverse purposes does suggest that it is an important national resource and this should be considered in debates about the future of the survey programme.

Multiple survey comparison reports

Nine papers made use of the national inpatient survey to both enhance an argument and to draw conclusions with other national NHS surveys. One such paper, A High-performing NHS? [63], reviewed the evidence on whether increased Government NHS funding had improved eight core domains from 1997 to 2010. The domains comprised access, patient safety, promoting health and managing long-term conditions, clinical effectiveness, patient experience, equity, efficiency and accountability. Key points, the situation in 1997, progress since then and future plans were discussed for each domain. The study included inpatient survey results for 2008 with healthcare surveys from different years and specialties, such as mental health. The inpatient survey results were reported briefly; patients ability to choose the hospital where they were treated had increased from 28 per cent to 33 per cent from 2007 to 2008, 21 per cent of patients reported not being given enough information about their condition or treatment, and there had been a slight increase in patients reporting that they had been asked their views on quality of care while in hospital (6 per cent to 9 per cent from 2002 to 2008). Similarly, the NHS Confederation's Lost in translation [64] briefly reported inpatient survey results for 2005. They concluded that patients report "high levels of satisfaction with the NHS and care they [patients] receive", with 92 per cent describing their care as good, very good or excellent (the other options being 'fair' or 'poor'). The findings were used to support the contention that patients' evaluation of their care is more positive than the predominantly negative perceptions communicated by the media and the general public.

Two reports provided a national overview of the NHS. One gave a snap shot of "patient and public expectations, experiences and evaluations" [65]. The Chartbook concluded that though some areas have improved, such as access to care, important variables such as patient engagement in decision making have not. The second report briefly discussed the inpatient survey from 2002 to 2007 as part of An Economic Health Check providing an overview of the NHS [66]. The report described a "largely static picture" in five key areas: access and waiting; safe, high quality, coordinated care; better information, more choice; building closer relationships and clean, friendly, comfortable place to be.

A further six reports were produced by the Healthcare Commission or Picker Institute Europe. The first report, State of Healthcare 2007 [67], highlighted areas that had improved as well as areas where further improvement was needed. The Picker Institute Europe sought to answer the question "Is the NHS getting better or worse?" in an overview of 15 national surveys between 1998 and 2004 [68]. One of the reported conclusions was that government targets and coordinated action have had a positive effect on patients' experience. A similar report examining trends of inpatients' experiences between 2002 and 2004 [69] also concluded that coordinated action facilitated improvement. A report from the Picker Institute, Is the NHS becoming more patient centred? [70], covered 26 surveys from 2002 to 2007, including inpatient data from 2005 and 2006 and also concluded that "many aspects of care targeted by the government have improved significantly". The report also highlighted significant improvements, such as being treated with dignity and respect, and areas of continued concern, such as cleanliness and food.

Finally, the Healthcare Commission produced a report that analysed the national patient survey programme for 2003/04 [71]. The report covered inpatient survey data from 2004 and used multiple linear regression analysis to show that self-reported health was the most significant variable for explaining variations in patients' experience. The analysis also revealed that elective patients, those admitted to specialist trusts, those admitted outside London, older people and men were likely to respond more positively which supports the conclusion that there are important socio-demographic variables that need to be taken into account when interpreting the survey results.

Ethnicity, age and patient experience

One study investigated whether self-reported ethnic group influenced patient experiences [72] using inpatient survey data from 2006 as well as other healthcare surveys such as the outpatient survey (2004/05). The authors caution against a too literal interpretation of their findings, however, their analysis suggested that compared to the largest category, "white British", most BME groups in England were more negative about their experiences as patients, particularly in responses to questions about access and waiting times, involvement in decisions about their care and treatment, and the quality of information given to them. They were also more likely to say that staff talked about them "as if they were not there". Only those respondents who described themselves as "white Irish" were more likely to report more positive experiences than the baseline category of "white British". Ethnic differences were least marked in the inpatient survey compared to surveys from different healthcare settings, such as the community. This study was updated in 2009 [73] and once again, the results showed that apart from white Irish, BME populations were less positive about their experiences of care, particularly with regard to access and waiting and relationships with staff.

In 2001, the UK government declared its intention to "root out" age discrimination in the health service through the National Services Framework for Older People [74]. Lievesley et al. [75] used inpatient survey data for 2004 along with other healthcare surveys to assess whether ageism and age discrimination were still apparent in the NHS. The report revealed that the oldest (81+) and youngest (16-35) hospital patients were most likely to feel that doctors and nurses talked about them "as if they are not there". Patients aged over 81 were also less likely than those aged between 51 and 80 to rate their overall care as "excellent". The authors quoted results from the 2006 inpatient survey as described in the National Services Framework for Older People: older patients were particularly affected by hospital management issues, such as privacy, continence management, single sex accommodation and provision of nutritious food. A number of reports have documented that older patients tend to be less critical [57, 71], making the more negative evaluation of the very oldest patients in this report all the more striking.

Sociological studies

Several studies have attempted more complex and theoretically informed analyses using the national adult inpatient survey. These include studies that correlate other variables with overall evaluation of care, studies that compare the inpatients survey results to the NHS staff survey and one study that investigated how workforce and community characteristics affect levels of "civility" in acute Trusts.

A report by Ipsos MORI, Frontiers of performance in the NHS II [76], analysed inpatient survey data from 2006 along with survey data gathered from patients treated in the community from 2005. The variables correlating most strongly with "overall rating of care" were with being treated with dignity and respect, cleanliness of the room and ward, and being included in treatment decisions. The "relative importance" of these factors was calculated, with dignity and respect being the most important (59 per cent), followed by being involved in decisions (28 per cent), and then cleanliness (13 per cent). The authors used stepwise regression to identify these "most important" factors, but did not address the well-known criticisms of the stepwise approach [77]. Among these problems are that standard errors and p-values are biased toward zero and parameter estimates are biased away from zero. These problems are known to be especially severe in the presence of collinearity. The method by which they calculated the "percentage importance" of each factor was also not explained.

Ipsos MORI also investigated the association between inpatient satisfaction and characteristics of the geographical area in which the hospital was located. The authors based their characterisation of a trust's "local population" on the closest 2,100 output areas (as defined by the Census [78]) and found only relatively weak associations between patients' satisfaction and ethnic divisions and age, and no association with the index of multiple deprivation. They identified two external factors as the most important, again using stepwise regression. These were whether the trust was in London and the percentage of the local population aged under 16. They then compared "predicted" (using these two factors in a regression) and observed ratings of overall inpatient care.

Doyle et al. [79] conducted a similar analysis to demonstrate a way that trusts could identify the "key drivers" of quality. The paper compared strength of association between overall satisfaction and various components by generating 3 × 2 contingency tables, with three categories of overall satisfaction, Positive, Fair, or Poor. Data from 2006 and 2007 were analysed separately. Chi-square tests were then performed on the contingency tables to test for a statistically significant association, and finally Cramer's V was calculated to assess the bivariate "strength of association" (though it is noted that since one of the variables has two categories, it is actually the phi statistic [see Figure 2]). No correction was made for the multiple pairwise comparisons that were carried out. The analysis was at the patient level, but no allowance was made for clustering within trusts. The authors report that being treated with respect and dignity was the most important component of overall satisfaction, followed by how well doctors and nurses worked together, and confidence and trust in nurses.

Sizmur et al. [80] investigated whether patients' considered certain aspects of their experience of care to be more important than others or if all were felt to be equal. The author tested which elements of the 2008 inpatient survey were most highly correlated with 'overall satisfaction' as measured by "Overall, how would you rate the care you received?" To ascertain the survey items with the strongest relationship to overall satisfaction, basic correlation analyses were applied to individual questions and overall satisfaction, at the patient level. Composite scores were created that revealed consistency and coordination of care, nurses, and patient involvement to be most strongly correlated with overall satisfaction. Multivariate stepwise regression was applied to account for confounding factors, such as gender or admission to hospital, and to identify specific factors that might predict satisfaction independently. The method resulted in similar predictors of overall satisfaction as the previous method, such as consistency and coordination of care. The author also performed analysis at the trust level, but suggested that as the variance accounted for implied high levels of multicollinearity (95 per cent), the results were "likely to be unstable".

Two papers investigated whether there was a homology between the responses on the staff and the inpatient surveys. Raleigh et al. [81] used twenty eight staff survey items as explanatory variables in regressions involving four inpatient survey responses from 2006 data. Results were aggregated to the trust level to allow the two data sets to be merged. Dummy variables for whether the trust was inside or outside London and trust type (general, teaching or specialist trust) were included in all these regressions. The selection of dependent variables was carried out using stepwise regression. The authors acknowledge some of these problems in their discussion, but do not explain why they did not use alternative, more robust methods such as LASSO or LAR [82]. They argue that their results "...show some significant patterns, indicating that the safety and quality of services and patient experience could improve if trusts acted on feedback from their staff."

Dawson (2009) asked whether the experience of staff working in the NHS was reflected in patients' experience of care [83]. Using data from 2007 to test for correlations between trust level aggregated inpatient and staff survey items he found a staggering 12,214 bivariate correlations of which 56 per cent were statistically significant. No level of significance was given, and there was no evidence of Bonferroni or alternative multiple test correction (a Bonferroni correction would imply a level of significance of 0.05/12214 ≈ .000004, and consequently fewer statistically significant correlations). Some of the key findings were intuitively plausible, for example, high levels of bullying, harassment and abuse against staff by outsiders were related to many negative patient experiences. Similarly, inpatients' perceptions of the adequacy of staffing levels and the amount of dignity and respect with which they were treated were correlated with employee's feelings of work pressure and staffing levels. However, some findings lacked any theoretical or intuitive support, such as higher numbers of staff who have had health and safety training leading to more patients perceiving staff as conscientious and available. This suggests that a stronger theoretical framework could be a useful guide to future analyses.

The Healthcare Commission's Acute Hospital Portfolio Review: Ward Staffing [84] also linked inpatient and staff survey data and evaluated changes in ward staffing since the Audit Commission's first investigations in 2000/01 [85]. They reported a very strong relationship between the use of temporary "bank" and agency staff, indicating high levels of vacancies and low levels of patient satisfaction. London hospitals had high vacancy rates and used temporary staff more frequently than hospitals outside the capital. This relationship was so strong that it made it difficult to investigate other staff variables related to patients' evaluations of care so separate analyses had to be conducted on Trusts inside and outside London. These analyses showed that higher proportions of registered, more experienced and skilled staff, greater numbers of staff who were satisfied with their jobs and had lower expressed intention to leave their current job were related to better patient evaluations of care. Consistent with the Audit Commission's earlier findings, the Healthcare Commission found no relationship between the total number of nurses or total expenditure and patient satisfaction. Together, these findings suggest that to improve patients' experiences, hospital trusts should invest in a richer skill mix (more experienced and skilled nurses), rather than just increasing the number of nursing staff. Although these results could be important in improving the quality of care in the NHS, it was difficult to critically appraise the research conducted by the Audit Commission as the report did not give detailed information about the data used and the ways in which they were analysed. Their report was originally backed up by a detailed document explaining the statistical analyses but a Freedom of Information (FOI) request to the CQC confirmed that this paper is no longer available.

The final article in this review used inpatient survey data to derive a trust measure of the "civility" of staff towards patients, King et al. [86]. The measure was a 14-item scale, using questions such as being treated with dignity and respect. Additional data sources were used to calculate the ethnic diversity of the medical and nursing workforce in the trust, the similarity of the ethnic makeup of the workforce and the local population, and the trust's performance. Civility was found to be negatively associated with ethnic diversity in the medical and nursing workforce. Low civility was in turn associated with poor trust performance. On the other hand, similarity of the ethnic makeup of workforce and local community was associated with increased "civility" and hence with better trust performance. The study had a strong theoretical framework from which hypotheses were derived and tested. A number of different datasets were used and data were analysed using appropriate statistics. One problem that remains is that it was not entirely clear that the 14-item scale actually measures civility and this should be tested in future research.