Pre-test of questions on health-related resource use and expenditure, using behaviour coding and cognitive interviewing techniques

Data on health care utilisation (e.g. hospitalisations, use of outpatient health services, medication use) is required for several reasons. For example, to analyse service use patterns, to identify “underuse” or “overuse” of health care services, to assess health care needs or health-related costs. Data on health care utilisation may be obtained from various sources: health insurance claims, disease registries, provider records, patient self-report, and expert opinion.

In principle, if detailed information is required, provider records may be a better source of utilisation data than burdening patients with a detailed data collection procedure by means of a diary, a written questionnaire or an interview [1]. In practice, however, provider records are often difficult to access or retrieve, because (i) it may be necessary to contact many different providers and (ii) the patient’s consent may be required but not forthcoming. Often, the same limitations to data availability also apply to health insurance data, in particular, if linking health insurance data to other data sources (e.g. data from a clinical trial or a survey) is necessary. Accuracy and completeness of administrative data are also a concern when providers are in a capitation fee system and have little financial incentive to record diagnoses and services accurately [2]. Moreover, information on utilisation of health services not covered by the health insurance and on the non-medical resource-use (e.g. patient’s or caregiver’s time) is usually not available from health insurance or provider records.

Consequently, researchers as well as public policymakers often have to rely – at least in part – on self-reported health care utilisation. Considerable work is usually undertaken in individual projects to develop self-report tools (i.e. questionnaires or diaries) collecting data on health-related resource use. Yet these tools and the results of their validation are seldom published and there seems to be a tendency to develop new questionnaires for new studies [3, 4]. Bertoldi et al. [3] conducted a review of the methodologies used in household surveys on medication use (61 studies published between January 1995 and June 2008 were included in the review). They showed that 70% of the studies did not publish the questions used to assess medication use; and 93.4% of the studies provided no information on the validity of the questionnaire employed to collect data on medication use. If methods of assessment of health care utilisation are not appropriate, results may be invalid. For example, Bhandari et al. [2] found that physician visits have been underreported by up to 50%, depending on the recall period covered by a questionnaire. Hence, publication of questionnaires, as well as results of their validation can contribute to the standardisation of data collection methods and comparability of results across different studies.

To analyse the current situation in Germany, we conducted a systematic search in MEDLINE and the Cochrane library (Method studies) to identify published validated or at least standardised self-report instruments appropriate to collect healthcare utilisation data in Germany (search strategy, inclusion and exclusion criteria and the list of identified instruments can be obtained from the first author). Despite extensive research on health care utilisation and costs per se, a validated or at least standardised set of questions on health care utilisation in German language was not identified by this search. Thus we developed, tested and subsequently refined a German questionnaire on health-related resource-use and expenditure. Patients with diabetes constituted our specific target group. However, it is often difficult to attribute resource utilisation and cost to a specific disease. Therefore, in addition to a disease-specific module, the instrument includes a generic module collecting data on a broad range of resource use categories. Hence the questionnaire can be used both in patients with diabetes and – omitting a diabetes-specific profile – also in the general population or in other patient populations. The questionnaire can be used in cross-sectional studies and for longitudinal data collection, for example, in conjunction with a clinical trial.

The questionnaire was tested by a combination of two established techniques – behaviour coding and cognitive interviewing [8–14]. Behaviour coding shows how often response behaviours defined a-priori, and suggestive of difficulties with a particular question, occur across interviews. The strength of this technique is its systematic and quantitative nature, allowing comparison across questions. However, it is somewhat arbitrary what rate of a problem indicator should be considered high enough to constitute a problem with the question. We used the guideline cut-off point of 15% proposed in the literature [8, 9].

Cognitive interviews were conducted to identify possible reasons for any problems observed and to allow for more qualitative analysis of difficulties with questions. Results of cognitive interviews suggest that in many cases difficulties reflected in response behaviour can be explained by inability to retrieve the information required, for example, due to recall problems. Often, extensive probing was needed to obtain an answer. Hence it seems that information retrieval may be improved by interviewer-administration or by additional probes following written administration of questions. However, this challenges the possibility to collect valid health care utilisation data by self-administered questionnaires.

Cognitive interviews can indicate the existence of problems with questions, for example, difficulties pertaining to a particular recall period or to the mode of administration. However, they cannot provide quantitative information on the quality of self-reported data. To obtain this type of information, studies comparing self-reported data to data from alternative sources are required.

As already mentioned in the introduction, it is often difficult to attribute resource utilisation and cost to a specific disease. A pragmatic approach to handling the issue of cost attribution may be to apply a generic questionnaire collecting data on a broad range of resource-use categories and subsequently to attempt to determine their attribution. To enable meaningful analysis following this approach, detailed data on comorbidity should also be collected. Hence more research with regard to validity of self-reported comorbidity is needed.

We developed the questionnaire on health-related resource use and expenditure for use in diabetes research in Germany. In addition to a disease-specific module, the instrument includes a generic module collecting data on a broad range of resource-use categories. Hence the questionnaire can be used both in diabetic patients and – omitting a diabetic-specific profile – also in the general population or other patient populations.

Efforts to further standardise the questionnaire should be based on validation studies. Ideally, the questions should be externally validated according to a range of criteria (e.g. for different recall periods and modes of administration) and within a variety of populations, in order to meaningfully interpret health care utilisation reported in single studies and to compare findings across studies. The questionnaire is currently being validated for two different recall periods applying an experimental design.

ⁱThe English version of the questionnaire presented in the Additional file 2 is a translation of the refined German version presented in the Additional file 1. The translational process used forward and backward translation techniques. However, further adaptation of the English version would be required to take the peculiarities of the service provision in other countries into account.