This article has Open Peer Review reports available.
Empirical evidence of recall bias for primary health care visits
© Brusco and Watts. 2015
Received: 6 February 2015
Accepted: 4 September 2015
Published: 15 September 2015
While it is common for an economic evaluation of health care to rely on trial participants for self-reported health service utilisation, there is variability in the accuracy of this data due to potential recall bias. The aim of this study was to quantify the level of recall bias in self-reported primary health care general practitioner (GP) visits following inpatient rehabilitation over a 12 month period.
This report is a secondary analysis from a larger randomised control trial of an economic evaluation of additional Saturday inpatient rehabilitation. Participants were adults who had been discharged into the community following admission to an acute general rehabilitation hospital. Participants were asked to recall primary health care visits, including community GP visits, via a telephone questionnaire which was administered at 6 and 12 months following discharge from inpatient rehabilitation. Participants were asked to recall health service utilisation over each preceding 6 month period. The self-reported data were compared to equivalent claims data from the national insurer, over the same period.
751 participants (75 % of the full trial) with a mean age of 74 years (SD 13) were included in this analysis. Over the 12 month period following discharge from rehabilitation there was an under-reporting of 14 % in self-reported health service utilisation for GP visits compared to national insurer claims data over the same period. From 0 to 6 months following discharge from rehabilitation, there was an over-reporting of self-reported GP visits of 35 % and from 7 to 12 months there was an under-reporting of self-reported GP visits of 36 %, compared to national insurer claims data over the same period. 46 % of patients reported the same or one number difference in self-reported GP visits between the 0 to 6 and the 7 to 12 month periods.
Based on these findings we recommend that an economic evaluation alongside a clinical trial for an elderly adult rehabilitation population include a sensitivity analysis that inflates self-reported GP visits by 16 % over 12 months. However caution is required when utilising self-reported GP visits as the data may contain periods of both over and under reporting. Where general practitioner visits are expected to vary significantly between intervention and control groups we recommend that administrative data be included in the trial to accurately capture resources for an economic evaluation.
It is common for an economic evaluation of health care to rely on trial participants for self-reported health service utilisation [1, 2]. Often self-reported health care visits are collected by a questionnaire designed specifically for the individual study to capture the health care resource of interest and this can be administered face to face, by mail out or over the telephone. However, with any self-reported data there is risk of recall bias. Recall bias arises when there is error in the recall of information that may include forgetting an event or recalling an event that did not occur  and the reason for this can be multi-factorial . An event may also be recalled with an incorrect timeline, that is, recalling an event prior to the specified period that actually occurred within the specified period (backward telescoping) or recalling an event inside the specified period that actually occurred prior to the specified period (forward telescoping) . In the context of health service utilisation, poor memory may substantially contribute to recall bias with an inability to recall when, and the precise number of times, an event took place . Accuracy of health service self-reported resource utilisation can be influenced by different demographic and methodological factors such as patient population, recall time frame, questionnaire design and mode of data collection .
This study involved follow up for 12 months of 996 adults post-discharge from an acute adult rehabilitation facility. The trial intervention was an additional inpatient rehabilitation service on a Saturday. The primary aim of the study was to determine the effect of the additional Saturday service with respect to clinical outcomes and costs . A secondary aim of the research and the aim of the current study, was to quantify the level of recall bias in self-reported primary health care visits, specifically community medical general practitioner (GP) visits, by comparing primary health care visits from a the self-reported questionnaire to those recorded by the national insurer over a 12 month period.
This study was a sub-group analysis from an economic evaluation alongside a randomised controlled trial that compared usual care Monday to Friday rehabilitation to Monday to Saturday rehabilitation for a general mixed adult population across two Australian inpatient rehabilitation facilities. The rehabilitation facilities serviced a general mixed adult population and patients were typically admitted to rehabilitation following an acute hospital admission. Diagnoses included routine orthopaedic lower limb joint replacement, falls and fracture management, neurological diagnosis, deconditioning, lower limb amputation, as well as other rehabilitation conditions. Many of the patients admitted were older adults, although age ranged from 18 to 100+ and the average admission was three weeks . Full details of the protocol , the clinical outcomes  and economic evaluations of the clinical trial [7, 11] have been published elsewhere. The trial was registered with the Australian and New Zealand Clinical Trials Registry (ACTRN12609000973213). The trial was approved by the Eastern Health Research and Ethics Committee (E58 09/10) and La Trobe University Human Research Ethics Committee (FHEC10/14). All patients provided written informed consent for the clinical trial and follow up data collection period, as well as a separate consent to meet requirements for Medicare Australia for Medicare Benefits Schedule and Pharmaceutical Benefit Scheme claims data.
To investigate if there was any recall bias associated with self-reported health service utilisation we compared community general practitioner (GP) claims recorded by the national insurer, Medicare Australia, to self-reported data over the same 12 month period. GP claims were the only variable that was collected by both Medicare and included on the health service utilisation questionnaire. A sub-group of participants was selected from the 996 participants in the full clinical trial and these were the participants who were alive at 12 months, had completed both the 6 and 12 month questionnaires and for whom we had 12 months of national insurer data. A copy of the self-reported questionnaire is available as an Additional File from the publication of the economic evaluation of the inpatient rehabilitation admission including the 12 month follow up period .
During the 12 months following discharge from rehabilitation both administrative and self-reported data were collected. Administrative data were collected at 12 months following discharge from rehabilitation and these included data from the primary rehabilitation health service, as well as from the national insurer, Medicare Australia. Data from the primary rehabilitation service included hospital admissions and outpatient services and Medicare data included pharmaceuticals, medical specialist and community GP claims. A minimum six month lag was employed between the end point of data collection to the time of data extraction. This allowed for the Medicare claim data to be processed.
In addition, a health service utilisation questionnaire was administered by telephone at 6 and 12 months following discharge from rehabilitation to supplement the administrative data. The health service utilisation questionnaire was administered by assessors blinded to group allocation in the wider clinical trial. Participants were asked to recall health service utilisation over the preceding 6 month period. Participants reported admissions and length of stay of acute or rehabilitation hospital admissions in facilities outside of the primary rehabilitation health service, community GP visits, non-Medicare health care visits, over-the-counter medications, and the assistance of carers. Assessors completing the follow up telephone calls prompted the participant to include both GP visits in the GP’s consulting rooms as well as GP visits where the GP travelled to the participant’s home environment. Home was defined as place of residence, for example this could include the family home or a residential care facility.
Mean difference between self-reported data and Medicare data were determined using a paired samples t-test. Correlation was reported using the Spearman coefficient. The Kappa coefficient (ĸ) was calculated to measure beyond chance agreement among measurements of GP visits between self-reported and Medicare data, as well as between the 6 and 12 month data. The self-reported data were also examined to determine the degree of similarity for reporting the same number of GP visits in the two time periods. Likewise, Medicare data were examined to determine the degree of similarity for GP utilisation in the two time periods. All data sets were checked for assumptions of normality using Kolmogorov-Smirnov. IBM SPSS Statistics Version 21 was used for the analyses and to produce the figures . Data were analysed and compared over the full 12 months and for each 6 month period.
Characteristics of patients from the randomised controlled trial who were and were not included in the recall bias analyses
Patients who were included in the recall bias sample (n = 751)
Patients who were not included in the recall bias sample (n = 245)
Age, mean (SD)
Gender, n male (%)
Diagnosis category, n orthopaedic (%)
Charlson comorbidity index, mean (SD)
Intervention group of the randomized controlled trial, n (%)
Rehabilitation length of stay, mean days (SD)
Therapy per week during rehabilitation admission, mean (SD)
Functional independence (FIM) on admission, mean (SD)
Functional independence (FIM) on discharge, mean (SD)
Health-related quality of life EQ-5D utility index on admission, mean (SD)
0.34 (0.35) (n = 721)
0.37 (0.36) (n = 222)
Health-related quality of life EQ-5D utility index on discharge, mean (SD)
0.65 (0.26) (n = 699)
0.63 (0.29) (n = 205)
Living independently in the community prior to admission, n (%)
713 (97) (n = 737)
218 (93) (n = 235)*
Living independently in the community following discharge, n (%)
616 (86) (n = 720)
200 (92) (n = 221)
There were 36 different Medicare item numbers pertaining to a GP community visit that were included in this analysis, from the Medicare administrative data set. The three most common Medicare item categories were Consultation at Consulting Rooms – Level B (14 %); Initiation of a Patient Episode (8 %); and Prothrombin time (including INR) (7 %).
Medicare general practitioner claims (national insurer) compared to self-reported general practitioner visits in the twelve months following discharge from rehabilitation (n = 751)
0-12 Months (combined)
Medicare GP claims
Self-reported GP visits
Difference (Medicare – Self-reported)
Medicare GP claims
Self-reported GP visits
Difference (Medicare – Self-reported)
Medicare GP claims
Self-reported GP visits
Difference (Medicare – Self-reported)
Number of GP visits
(0 to 29)
(0 to 52)
(−43 to 22)
(0 to 65)
(0 to 70)
(−26 to 56)
(0 to 76)
(0 to 100)
(−52 to 50)
(CI −2.1 to −1.1)
(CI 3.0 to 4.1)
(CI 1.2 to 2.7)
p = <0.01
p = <0.01
p = <0.01
Correlation of Medicare general practitioner claims (national insurer) and self-reported general practitioner visits (n = 751)
Medicare GP claims compared to self-reported GP visits 0 to 6 months
Medicare GP claims compared to self-reported GP visits 7 to 12 months
Medicare GP claims compared to self-reported GP visits 0 to 12 months
Self-report GP visits 0 to 6 months compared to 7 to 12 months
Medicare GP claims 0 to 6 months compared to 7 to 12 months
p < 0.001
p < 0.001
p < 0.001
p < 0.001
p = 0.28
Measure of agreement, Kappa
Number of occasions the data matches for the individual participant n (%)
From 0 to 6 months following discharge from rehabilitation, there was an over-reporting of self-reported GP visits by 1.6 sessions (95 % CI 2.1 to 1.1; p < 0.01) with very low levels of correlation or agreement (Tables 2, 3 and Fig. 1b). This indicates over-reporting of 35.2 % for self-reported data, alternately, self-reported data would need to be deflated by 26.0 % to equate to data from the national insurer. In the second 6 month period from 7 to 12 months following discharge from rehabilitation, there was an under-reporting of self-reported GP visits by 3.6 visits (95 % CI 3.0 to 4.1; p < 0.01) with very low levels of correlation or agreement (Table 3 and Fig. 1c). This indicates under-reporting of 36.1 % for self-reported data, alternately, self-reported data would need to be inflated by 56.4 % to equate to data from the national insurer.
Data collection forms for collecting resource utilisation are commonly used in an economic evaluation alongside a randomised controlled trial . Although recall bias is a potential issue in this form of data collection, the impact of recall bias is not known for a mixed adult population following a rehabilitation admission over a 12 month period. Our finding of under-reporting over 12 months is consistent with a literature review that found that under-reporting is more common than over-reporting for recall of primary health care GP visits . This contrasts with recall for hospitalisation where the number of nights in hospital were found to be over-reported by 17 % . In addition to different service types other factors influencing recall bias include the population surveyed, the length of the recall period, the presence of a salient event and in the administration of the questionnaire . We tested the recall of people discharged from an Australian rehabilitation hospital for visits to a community GP, over two recall periods of 6 months in a survey conducted via the telephone.
While overall the recall bias was 13.5 % under-reporting, when the results were broken down into two different time periods (0–6 months and 7–12 months post discharge from rehabilitation), we found mixed results with over-reporting of 35.2 % in the first 6 months following discharge and under-reporting of 36.1 %, in the second 6 month period after discharge. This unexpected result may be influenced by multiple factors and this is further explored, although it is noted that it is not uncommon for data sets to contain both under and over-reporting [6, 2].
For the 0–6 month period it is assumed that the recall of services outside the time point of interest, known as forward telescoping, may have been less of an issue as there was a salient event, the rehabilitation admission. The rehabilitation admission could be used as a memory aid to assist recall to the correct point in time . Yet there was significant over-reporting for this time period. There are two potential explanations. The first is based on the premise that a population, who had just been discharged from rehabilitation, would typically receive a suite of health care services to support the transition from the hospital to the community and to continue rehabilitation. Other types of health care utilisation were high in this group of participants in the recall analysis, with around 17 non-Medicare health care visits self-reported in the same 6 months following discharge from rehabilitation. Recall from 0 to 6 months may have been influenced by an inability to distinguish GP visits from these additional health care services. As such, a possible explanation for over-reporting may be that when participants were asked to recall the different types of visits under separate questions, there may have been inaccuracy when differentiating GP visits from other non-admitted health services in the first 6 months following discharge.
The second explanation is that for these patients, a hospital admission for rehabilitation is not a significant salient event to frame the commencement of the recall period. The participants in the recall analysis had on average 8 days additional hospitalisation in the 6 month period following discharge from rehabilitation. This suggests that using the rehabilitation admission as a memory aid may be a strategy with limitations in this patient population who are exposed to high levels of admitted and non-admitted health care services. It is noted that this is in contrast to previous literature that report a salient event, such as a hospital admission, improves the ability to distinguish events occurring within the recall period and therefore improves accuracy .
Recall from the 7 to 12 month period following discharge, which was significantly under-reported, may have been influenced by memory decay (forgetting visits) or forward telescoping (assuming that events are outside of the timeframe) as this is common in recall periods of 6 months or greater . This recall period did not have the salient event of the rehabilitation admission to define the start of the recall period. An explanation is that while there was an increase in the frequency of Medicare reported visits in the period from 7 to 12 months compared to 0–6 months, patients may have recalled a lower average number over the whole 12 months and applied this lower number to the period from 7 to 12 months.
Comparing the results from the first and second recall periods in the recall analysis, it was noted that patients reported a mean of six visits for both periods. From the Medicare data there were a mean of five visits in the 0–6 month period and a mean of 10 visits in the 7–12 month period, a substantial increase in the latter period. However there was a reversed pattern of self-reported utilisation for non-Medicare admitted and non-admitted health care services. In the 7–12 month period there were 10 non-Medicare health care visits compared to 17 non-Medicare health care visits in the 0–6 month period. The inability to differentiate GP services from other health care services may have resulted in patients reporting a mean of six visits for both periods.
Economic analyses alongside clinical trials relying on self-reported data should account for potential recall bias, particularly when self-reported data provides a significant contribution to the final resource utilisation included in an economic evaluation. This can be done via a sensitivity analysis of the included self-reported data that may consider inflating, deflating or removing the self-reported health services. Within trial economic analyses should also consider measuring potential recall bias by comparing a health services that are captured in both administrative and self-reported data. Strengths of this study include a large sample size across two 6 month periods of data collection, providing a combined recall period of 12 months , noting that recent literature suggests a total recall period of 12 months may improve accuracy of recall .
This study is limited to a mixed adult population that had been discharged from inpatient rehabilitation. Other limitations include multiple assessors and therefore potential variation in the wording or prompting provided to study participants in the recall of health service utilisation. A telephone interview was used to administer the questionnaire which does not allow the same visual cues to prompt recall of information compared to face-to-face interviews. There is potential for bias in the sample of participants included in this sub-group analysis. It has been noted that compared to participants excluded from the analysis, those included were younger, had less comorbidities, and higher functional status on admission and discharge from rehabilitation. Patients in poorer health who are older and have reduced cognition are more likely to under-report , therefore the under-reporting in this study may be a conservative estimate for a mixed adult population following a rehabilitation admission. In addition, while Medicare has been shown to have a high correlation of accuracy when compared to the Cancer Registry in Australia , it is acknowledged that there are limitations to this data set and there is limited literature reporting the accuracy and external validity of this data .
The salient event of the rehabilitation admission is common to all patients at the start of the data collection period. What is varying in the study is whether there is a landmark event to bind the start of the two six month recall periods (0 to 6 months and 7 to 12 months post discharge). It is noted that these results are in contrast to previous literature of how such a memory cue would affect recall. The recall bias examined in this analysis only pertained to community GP visits, not more salient events such as an inpatient admissions that are reported to have a greater degree of recall accuracy . Therefore the results of this study are only generalizable to community health care visits, not hospital admissions. Future consideration may be given to resource data collection forms that further probe as to the nature of the health care visit to improve accuracy.
Self-reported health service utilisation is a commonly used method for collecting resource use data when conducting an economic evaluation alongside a clinical trial. Our findings give direction and magnitude for recall bias associated with self-reported general practitioner visits compared to Medicare data over 12 months following an acute rehabilitation admission. Based on these findings we recommend that an economic evaluation alongside a clinical trial for an elderly adult rehabilitation population include a sensitivity analysis that inflates self-reported GP visits by 16 % over 12 months. However caution is required when utilising self-reported GP visits as the data may contain periods of both over and under reporting. Where general practitioner visits are expected to vary significantly between intervention and control groups we recommend that administrative data be included in the trial to accurately capture resources for an economic evaluation.
The authors thank and acknowledge fellow Chief Investigators Professor Nicholas Taylor and Professor Nora Shields, as well as the team who were involved in the randomised controlled trial and economic evaluation of this project. This work was supported by a NHMRC PhD scholarship for author NKB (Natasha Kareem Brusco) (Grant Number APP1017877) and by a NHMRC Partnership Grant (Grant Number 541958) between Eastern Health and La Trobe University. These funding sources did not have a role in the data collection, analysis or interpretation; in the writing of the report; or in the decision to submit the article for publication. The researchers operated independent of the funding sources. The research was conducted via the Allied Health Clinical Research Office, Eastern Health, Level 2, 5 Arnold Street, Box Hill, Victoria, Australia, 3128.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Clarke PM, Fiebig DG, Gerdtham UG. Optimal recall length in survey design. J Health Econ. 2008;27:1275–84.View ArticlePubMedGoogle Scholar
- Kjellsson G, Clarke P, Gerdtham U-G. Forgetting to remember or remembering to forget: a study of the recall period length in health care survey questions. J Health Econ. 2013;35:34–46.View ArticleGoogle Scholar
- Hassan E. Recall bias can be a threat to retrospective and prospective research designs. Internet J Epidemiol. 2006;3(2):339–412.Google Scholar
- Gaskell GD, Wright DB, O’Muircheartaigh CA. Telescoping of landmark events: Implications for survey research. Public Opin Q. 2000;64:77–89.View ArticlePubMedGoogle Scholar
- Mistry H, Buxton M, Longworth L, et al. Comparison of general practitioner records and patient self-report questionnaires for estimation of costs. Eur J Health Econ. 2005;6:261–6.View ArticlePubMedGoogle Scholar
- Bhandari A, Wagner T. Self-reported utilization of health care services: improving measurement and accuracy. Med Care Res Rev. 2006;63(2):217–35.View ArticlePubMedGoogle Scholar
- Brusco NK, Watts JJ, Shields N, Taylor NF. Is cost effectiveness sustained after weekend inpatient rehabilitation? 12 month follow up from a randomized controlled trial. BMC Health Serv Res. 2015;15(1):165.View ArticlePubMedPubMed CentralGoogle Scholar
- Brusco NK, Shields N, Taylor NF, Paratz J. A Saturday physiotherapy service may decrease length of stay in patients undergoing rehabilitation in hospital: a randomised controlled trial. Aust J Physiother. 2007;53(2):75–81.View ArticlePubMedGoogle Scholar
- Taylor NF, Brusco NK, Watts JJ, Shields N, Peiris C, Sullivan N, et al. A study protocol of a randomised controlled trial incorporating a health economic analysis to investigate if additional allied health services for rehabilitation reduce length of stay without compromising patient outcomes. BMC Health Serv Res. 2010;10:308.View ArticlePubMedPubMed CentralGoogle Scholar
- Peiris C, Shields N, Brusco NK, Watts JJ, Taylor NF. Additional Saturday rehabilitation improves functional independence and quality of life and reduces length of stay: a randomized controlled trial. BMC Med. 2013;11:198.View ArticlePubMedPubMed CentralGoogle Scholar
- Brusco NK, Watts JJ, Shields N, Taylor NF. Are weekend inpatient rehabilitation services value for money? An economic evaluation alongside a randomized controlled trial with a 30 day follow up. BMC Med. 2014;12:89. doi:10.1186/1741-7015-12-89.View ArticlePubMedPubMed CentralGoogle Scholar
- IBM Corp. Released 2012. IBM SPSS Statistics for Windows, Version 21.0. Armonk, NY: IBM Corp.Google Scholar
- Goldsbury DE, Smith DP, Armstrong BK, O’Connell DL. Using linked routinely collected health data to describe prostate cancer treatment in New South Wales, Australia: a validation study. BMC Health Serv Res. 2011;11(1):253.View ArticlePubMedPubMed CentralGoogle Scholar
- Trevena JA, Rogers KD, Jorm LR, Churches T, Armstrong B. Quantifying under-reporting of pathology tests in Medical Benefits Schedule claims data. Aust Health Rev. 2013;37(5):649–53.View ArticlePubMedGoogle Scholar