Development of a questionnaire on health-related resource use and expenditure for use in diabetes research
Potentially relevant questions on health care utilisation were identified by reviewing German questionnaires obtained via Internet and personal communication with different research groups – mainly, instruments developed for large scale German health surveys and epidemiologic studies, by Robert Koch institute, by TNS Health care for the Health Care Monitor project, as well as instruments developed for the Cooperative Health Research in the Region of Augsburg (KORA) study  and the Heinz Nixdorf Recall study . Alternative approaches to asking questions covering similar content were grouped together, and a set of questions on general health care utilisation (primary care visits, visits to emergency departments, visits to specialists, hospital stays, and other therapies or paramedical care) was assembled either by adopting wording from those candidate questions directly or by adapting them to fit a question form seemingly more appropriate to the new instrument.
Specific diabetes-related questions were elaborated in close cooperation with clinical experts in the field of diabetology from two large specialized diabetes centres in Germany – the German Diabetes Centre in Düsseldorf and the research institute of the Diabetes Academy Mergentheim.
Particular attention was paid to the development of questions regarding medication use (medication name, strength, prescribed frequency, and duration of use). Items were developed that captured (i) current use of diabetes-specific medication, (ii) changes in diabetes-specific medication profile within a defined reference period, i.e. changes with regard to the number of medications taken, the dosage level for one or more medications, and the pharmacologic class of the medications being taken; (iii) current use of regularly-taken medications for other conditions and (iv) occasionally-taken medication within a defined reference period.
The initial questionnaire collected data on (i) a wide range of health care services utilisation during a specified recall period (number of visits to the general practitioner, including home visits, number of visits to emergency room or departments, number of outpatient visits to various specialists and therapists, utilisation of hospital outpatient services, diagnostic tests and procedures carried out ambulatorily, hospital inpatient admissions and their duration; (ii) time spent obtaining these healthcare services, (iii) use of diabetes-specific and other medication; (iv) out-of-pocket expenditure on medication; (v) comorbidity; (vi) disability days and days off work; and (vii) unpaid or paid help received by the patient because of a limited ability to do household chores (the time for which help was needed and corresponding cost, if applicable); (viii) other variables possibly required to analyse the data (e.g. participation in disease management programmes and employment status).
Test of the questionnaire
Study setting and participants
To test the developed questionnaire, patients with diabetes mellitus were recruited from a general practitioner’s office and an outpatient centre specialising in diabetes treatment in two cities in North Rhine-Westfalia in Germany. We aimed to recruit respondents covering the range of individuals with diabetes, e.g. those with type 1 and type 2 diabetes, younger and older patients and those with different socioeconomic status, who may all be sampled in future surveys or recruited to RCTs, that is, a sampling approach aiming to explore diversity by maximising variance was applied. The questionnaire was tested by a combination of two techniques – behaviour coding and cognitive interviewing (see below). Since it is recommended that behaviour coding studies contain at least 30 respondents , we recruited more patients than would be needed for cognitive interviewing alone.
As a means of systematically identifying questions which needs revision, behaviour coding  was carried out. A standardized behaviour coding form was employed to document problem indicators, i.e. behaviour suggestive of problems with a particular question. The following problem indicators were registered for each question: (i) request for clarification; (ii) answer with comments, i.e. answer appears to meet question objective, but comments indicate uncertainty, misunderstanding, etc.; (iii) inadequate answer, i.e. answer that does not meet question objective; (iv) “don’t know” answer; (v) item non-response; and (vi) “no improvement after probing” if the answer was inadequate even after feedback from the interviewer.
In order to explain problems registered by the behaviour coding and to reveal problems not evident in response behaviour such as silent misinterpretation, cognitive interviewing  was added to the testing procedure. To this end, both scripted and unscripted probes were used. Scripted probes aimed to (i) provide some standardisation of analyses across interviews and (ii) to ensure that the survey questions of greatest concern were probed appropriately within the limited cognitive interview time. Three various types of scripted probes were employed: (i) comprehension probes asking respondents to explain their understanding of particular concepts or terms; (ii) information retrieval probes asking them to explain how respondents arrived at an answer and (iii) confidence rating asking respondents to evaluate the degree of confidence in their answers. Unscripted probes were usually applied if the respondent gave an inadequate answer or if item non-response occurred. The protocol concluded with a series of general questions allowing the respondent to provide additional feedback on particular items or the questionnaire as a whole.
The questionnaire was tested both as self-administered and as an interviewer-administered tool. In the self-administration group, participants were instructed to report any difficulties or problems with questions while completing the questionnaire, in order to enable behaviour coding. Problem indicators “inadequate answer“ and “item non-response” (i.e. missing) were assigned during a follow-up interview and by a subsequent analysis of the completed questionnaires. In this group, a retrospective approach to cognitive interviews was adopted. Once participants had completed the entire questionnaire, the interviewer went back through it, asking follow-up questions (probes).
In the interviewer-administered group, behaviour was coded during the interview. A concurrent approach to cognitive interviews was implemented, i.e. the interviewer read the survey questions aloud and probed immediately after the respondent had answered a particular question. The concurrent strategy aimed to avoid retrospection problems, which might occur when probing is carried out at the end of the questionnaire. However, concurrent probing can influence responses to subsequent questions and it was important to take this possibility into account by implementing retrospective probing in the self-administration group.
The study was conducted in September – October 2010 by two researchers acting as interviewer (CE) and observer (NC). Both received training in cognitive interviewing techniques prior to conducting interviews. Test-interviews supervised by a psychologist were conducted in the diabetes outpatient department of the University Clinical Centre in Düsseldorf. Test interviews were also used to test and modify problem indicators for behaviour coding and to finalise the cognitive interview protocol.
The interviews were tape-recorded with the permission of the respondents. Both researchers (interviewer and observer) listened to the audiotapes and independently performed behaviour coding from the tape-recordings. Behaviour coding performed during the interviews was also reviewed. Disagreement with regard to behaviour coding was solved by consensus between the researchers. Coding summaries reflecting the relative frequency of problem indicators across all interviews were produced for each question. According to a guideline cut-off point proposed in the literature [8, 9], questions were classified as problematic if 15% or more of responders had problem(s) with a question, i.e. at least one problem indicator was assigned to the question in 15% or more of the interviews.
Qualitative analysis of cognitive interviews was also performed by both researchers. Summaries highlighting problems pertaining to particular questions were created. These summaries served as a basis for revision of the questionnaire.
The study was approved by the Ethics Committee of the Medical Faculty, University of Düsseldorf, on April 30, 2010 (Reference number: 3370). Informed consent, which is required in accordance with the Declaration of Helsinki, was obtained from each individual participating in the study.