Impact of a referral management “gateway” on the quality of referral letters; a retrospective time series cross sectional review
© Xiang et al.; licensee BioMed Central Ltd. 2013
Received: 21 December 2012
Accepted: 8 August 2013
Published: 14 August 2013
Referral management centres (RMC) for elective referrals are designed to facilitate the primary to secondary care referral path, by improving quality of referrals and easing pressures on finite secondary care services, without inadvertently compromising patient care.
This study aimed to evaluate whether the introduction of a RMC which includes triage and feedback improved the quality of elective outpatient referral letters.
Retrospective, time-series, cross-sectional review involving 47 general practices in one primary care trust (PCT) in South-East England. Comparison of a random sample of referral letters at baseline (n = 301) and after seven months of referral management (n = 280). Letters were assessed for inclusion of four core pieces of information which are used locally to monitor referral quality (blood pressure, body mass index, past medical history, medication history) and against research-based quality criteria for referral letters (provision of clinical information and clarity of reason for referral).
Following introduction of the RMC, the proportion of letters containing each of the core items increased compared to baseline. Statistically significant increases in the recording of ‘past medical history’ (from 71% to 84%, p < 0.001) and ‘medication history’ (78% to 87%, p = 0.006) were observed. Forty four percent of letters met the research-based quality criteria at baseline but there was no significant change in quality of referral letters judged on these criteria across the two time periods.
Introduction of RMC has improved the inclusion of past medical history and medication history in referral letters, but not other measures of quality. In approximately half of letters there remains room for further improvement.
KeywordsGeneral practice Referral letters Quality improvement Peer review Referral management
Referral letters are a key means of communication between general practitioners (GPs) and hospital specialists. GPs write more than 10 million elective referral letters each year for routine management of conditions . Accumulated evidence suggests that GP referral letters often lack essential clinical information, for example, 38% of specialists in outpatients reported that referral letters contain inadequate information fairly often or very often [2, 3]. Letters may fail to explain clearly why the patient is being referred or what is being asked of the specialist team (diagnosis, investigation, treatment or reassurance) . Consequently, GPs, consultants and patients may each have a very different understanding of the purpose of referral. Specialists have said that, in order to better address the patient’s current problem, they would like GPs to provide details that the patient is unable or unlikely to provide themselves, such as detailed past medical history, medications, examination findings, investigation results and details of treatments tried already, as well as more information about the patient’s presenting complaint, the problem to be addressed and clinical questions to be answered .
Various methods have been tried to improve letter writing and to produce more informative letters, for example guidelines or standardised referral templates which highlight information preferred by the specialist [5, 6]. Used passively and in isolation, guidelines and proformas have not always succeeded in improving the content of referral letters [7, 8]. However, multi-faceted interventions such as peer or specialist feedback together with guidelines, proformas or risk factor check-lists have been shown to improve the referral process, including improved letter content [6, 9–17]. A Cochrane review of educational, organisational and financial interventions to improve out-patient referral rates and appropriateness has noted that simple interventions and passive dissemination are ineffective in changing clinicians’ behaviour .
Referral management schemes (RMS) are recent developments. There is a hierarchy of RMS, from low level interventions, which encourage compliance with guidelines such as the Map of Medicine , through to Referral Management Centres (RMCs), where all GP referrals undergo clinical triage. The primary aim of these schemes is to control demand and reduce unnecessary and inappropriate referral of patients to specialist services. However, they also have the potential to influence other aspects of the referral decision and referral process, including the quality of the referral letter.
A 2010 King’s Fund report highlighted a lack of research on the impact of referral management centres, particularly those with centralised models that cover referrals to all specialities. It noted that existing research has not explored the impact of RMC on referral quality .
The concerns around poor referral letter quality nationally, and the lack of research on the impact of RMCs prompted this current study. Our aim was to retrospectively assess the introduction of a RMC on the quality of referral letters.
The data extracted included the destination speciality of each referral and the letter’s information format (letter only, letter plus selected download of data from electronic patient record, and letter plus unedited additional data).
Two researchers (PH, KM) assessed the content of the referral letters for “quality” using two methods:
Assessing the presence or absence of four items of information that had been agreed and implemented by the local Practice Based Commissioning board to monitor referral letter quality prior to the introduction of the RMC. These four items considered ‘core’ were blood pressure, body mass index, past medical history and medication history. All GPs had been briefed about the necessity for these items to be included in out-patient referral letters when first introduced as proxy indicators of quality and the triaging GPS were also aware of these indicators.
Application of the two quality criteria for referral letters developed by Grol and colleagues from national guidelines, international literature and surveys . The referral letter is judged to meet the first criterion (provision of clinical information) when four or more of the following clinical items of information are present: patient symptoms; previous examination findings; whether or not investigations were performed; whether or not treatment had been given; current medication. The second criterion assesses clarity of the reason for referral. To meet this, the letter must contain at least one specific request for the specialist (request for diagnosis, treatment, and/or a management plan). Neither the referring GPs nor the triaging GPs were aware that these criteria would be used to assess referral quality when they wrote the letters.
Letters were reviewed independently and then scores were compared. Where there was non-concordance in the letter quality assessment, a third “expert” assessor (HS) reviewed the letter and the majority decision (two out of three) was used in analysis (Figure 1).
The assessment process and proforma were piloted (n = 20) before commencing the audit and the Kappa values achieved ( >0.7) represented substantial agreement. In this preliminary review, it was recognised that the utility of the data being collected could be improved by capturing some additional variables. Firstly, inclusion of all four items of information (blood pressure, body mass index, past medical history and medication history) was found not to be essential for every referral reviewed. For example, a referral to dermatology about a fast-growing keratising skin lesion that does not include body mass index or blood pressure in the letter may still have sufficient essential information. Therefore the option to comment on relevance of missing data was introduced and reviewers were able to record “not relevant” if data was missing but deemed non-essential to that particular case, or “relevant” if an item was missing but needed.
When piloting the Grol criteria we found several examples where the reason for referral was implied but not stated specifically. In order to explore this poor articulation of the reason for referral further, we categorised each request for the specialist as either ‘explicit’ or ‘implicit’. If a request was judged implicit, the exact phrasing used was noted. As above, we also enabled assessors to make a subjective override of the quality decision based on the assessment criteria. Where there was a majority decision to override the objective decision, these cases were identified in subsequent analyses.
We performed statistical analysis using PASW Statistics 18 (SPSS Inc). Data were nominal and independent, so non-parametric statistics were used. Chi-square tests were used to compare the quality of referral letters between time 0 and 1. Because of the exploratory nature of this study, all analyses were two-tailed, with p ≤ 0.05 considered significant. As this review was a before and after service evaluation using routinely collected data it did not require ethics committee approval.
Characteristics of referral letters (target specialty, format of correspondence) at time 0 (n = 301) and at time 1 (n = 280)
Specialty referred to
Ear, nose and throat
Format of referral
Letter plus selected download of data from electronic patient record
Letter plus unedited additional data
Other: examples included referral letters with forwarded radiology result, care plan, copies of previous correspondence, another health professional’s letter, other department investigation results and private clinic letter
Inclusion of core information
Percentage of referral letters including core information
Body mass index
Past medical history
For all core information items, the proportion of letters containing them increased at time 1 compared to baseline. Increased frequencies in the recording of ‘past medical history’ and ‘medication history’ were statistically significant, but the improvements in inclusion ‘blood pressure’ and ‘body mass index’ did not achieve significance (Table 2).
Of the letters missing one or more items of “core information”, the missing item(s) were judged not essential in 45.1% (152/337) of referrals.
Grol quality criteria
Percentage of referral letters meeting the Grol quality criteria
% Meeting criteria
After subjective over-ride
Number and percentage of referral letters meeting individual sections of the first Grol quality criterion
Time 0 number (%)
Time 1 number (%)
Findings of previous examinations
Investigations were performed
Treatment had been given
Percentage and number of different types of requests contained in referral letters categorised by the second Grol  quality criterion
Time 0 (n = 301)
Time 1 (n = 280)
Specific requests regarding
Summary of main findings
Our study has demonstrated that since the introduction of a RMC, there has been a significant improvement in some aspects of the quality of the referral letters written by GPs to their specialist colleagues (the inclusion of past medical history and medication history). No improvements were observed using the more demanding Grol criteria which assess the inclusion of more contextual clinical information (whether or not investigations were performed, and treatments had been given) together with formulation of a specific request to the specialist (the provision of a diagnosis and/or treatment and/or management plan).
The limited improvement in referral letter quality found in this study is disappointing. A total of 7% referral letters received peer to peer feedback. Unfortunately the way data were routinely captured did not allow identification of the proportion of feedback that related to the quality of the referral letters. (Feedback could be provided for several other issues, including the appropriateness of referral or target specialty selected). Knowing that at least 25% letters were missing core information relevant to the referral request, higher rates of feedback should have been expected and might have improved referral letter quality over the observation period. Anecdotally, triage staff were reluctant to feedback when they recognised a letter had been written by a locum, as they knew there was little chance of being able to contact that doctor. Better processes are needed to ensure all referrers, irrespective of their employment status, can benefit from peer to peer feedback.
Traditionally an outpatient referral has been considered to represent the transfer of responsibility for some aspect of the patient’s care from primary to secondary care . In referral management a third party is introduced into this referral process. This third party is employed to assess risk, to triage, and to select the most appropriate destination for each patient based on the GP referral letter. Without knowing the patient or having access to their full primary care record, the triaging GP has to depend solely on the contents of the referral letter to inform their decision making. In this situation the quality of the referral letter acquires even greater importance as the absence of key information may lead to erroneous diversion of a referral which would have been judged appropriate if all the relevant information had been available. In future audits, consideration should be given to possible harm to patients that may result from the delay when letters are intercepted and re-routed. We have no evidence to suggest that any patients suffered harm, but our study was not designed to detect this.
Strengths and limitations of the study
This study tackled an appropriate and important question, addressing the paucity of research identified by the King’s Fund. Furthermore the study has good internal validity, provided by a large, randomly selected sample of letters which are representative of the local referral patterns. The tools used to judge letter quality were piloted before the study to ensure consistency and had acceptable inter-rater reliability (Kappa values >0.7 representing substantial agreement). The risk of bias was further reduced by the use of a third person who independently reviewed any letters where there was a lack of concordance between the two researchers’ assessment. The study has good external validity, as the setting has socio-economically and ethnically diverse population, a variety of practice sizes and a wide range of secondary care services; study findings are thus generalisable to areas within the UK where RMCs may be introduced.
This study has some limitations. It was retrospective, but the risk of this influencing results was reduced by blinding the reviewers to the period from which each letter had been sampled. The credibility of the quality criteria used differed, whilst the Grol quality criteria had been formally developed the four ‘core’ items of information did not have a theoretical base and were adopted as proxy measures of quality. The utility of the four items of information as an outcome measure may be further threatened by GPs’ awareness of these ‘proxy’ quality criteria prior to the introduction of the RMC. The study also lacked any assessment by the receiving specialists of the adequacy of the referral letters or whether they felt the RMC process resulted in any improvements. A further limitation is the relatively short time period between the two assessment periods (seven months) which may not have been long enough for the feedback process to change the practice of individual GPs. That said, there is no research precedent for how long changes based on triage feedback may take and seven months seems a reasonable period for individual GPs to make changes to their practice.
Comparison with existing literature
We cannot conclude causality, but the introduction of triage and referral management of the referral letters was associated with moderate improvements in some aspects of referral letter quality. Greater shifts in quality have been achieved with more active and multifaceted interventions, for example weekly practice-level referral review meetings and six-weekly cluster meetings including consultant feedback. After one year, that study achieved improved awareness and use of referral guidelines, referral letter content and high acceptability amongst GPs .
Nevertheless, it still remains difficult to change the behaviour of an established GP. For example, a qualitative study of GP referrals to psychiatric services concluded that GPs did not use a technical framework to describe patient needs. Instead they described patients’ needs in holistic and social terms, following “reasoned recipes of action indicating how to bring forth typical results in typical situations by typical means”. The letters reflected the strength of practitioner’s sense of acting as a gateway to secondary services, the degree of suffering, disruption or perceived urgency. Interestingly, GPs expressed unwillingness to make clear, explicit requests for a particular form of support or treatment, which was attributed to a traditional deference to “expert help” rather than true deference . These observations may help explain our observation that GPs frequently failed to make explicit requests regarding diagnosis, treatment or management plans.
Implications for future research or clinical practice
The poor quality of out-patient referral letters is a persistent problem and whilst this study suggests a potential role for a referral management gateway to contribute to improvement there remains a need for further research. Evaluation over a longer period is necessary to assess its full potential and whether impact can be sustained. This preliminary evaluation has highlighted the importance of refining the intervention. For example, the future formal training of triaging GPs in peer to peer feedback may enhance their impact. The documentation on feedback could provide both qualitative data to inform training needs but also the design of more inclusive feedback processes that enable all referrers, irrespective of their employment status, to benefit.
Previous studies have observed an unwillingness amongst GPs to make a clear explicit statement about the reason for referral, this evaluation also confirms this reluctance. We observed an unfortunate trend with the introduction of referral management towards fewer letters including an explicit statement about the reason for referral. This greater vagueness about reason for referral may be an unintended consequence of referral management, arising as referrers learn to ‘hedge their bets’ to minimise the risk of a referral being deflected or refused by the RMC. To overcome this behaviour change, a within-practice intervention may be helpful, such as internal peer review prior to sending the letter, which uses a checklist based on the Grol quality criteria for explicit requests.
GP training, continuous professional training, and revalidation should all include development of letter writing skills. For example, paediatric specialty training in the UK utilises the Sheffield Assessment Instrument for Letters (SAIL) . This is an 18-point check-list covering areas such as problem list, history, examination, overall assessment, management, follow-up and clarity, and a 10-point scale to record a global rating of the letter against gold standard, which was defined as ‘This letter clearly conveys the information I would like to have about the patient if I were the next doctor to see him/her’. Individualised feedback is then relayed to the writer to enable reflection and self improvement. SAIL has been shown to be effective in improving the quality of outpatient letters to GPs .
This study begins to address the evidence gap highlighted by the 2010 King’s Fund report. With the increasing importance of RMCs, more research is needed to address their efficacy and usefulness. By improving the quality of referral letters, decisions on the appropriateness of referrals are likely to become more accurate and specialists will be provided with the information they need to complement the patient’s own history and reach a better management plan, making their part of the process more efficient and effective.
Predictive analytics SoftWare
Primary care trust
Referral management centre
Referral management scheme
Sheffield assessment instrument for letters
Statistical package for the social sciences.
Liz Cheek, Dr Carrie Llewellyn and Dr Christina Jones for their advice on data entry and analysis.
- ospital Episode Statistics. 2009, Available at: http://www.hesonline.nhs.uk/Ease/servlet/ContentServer?siteID=1937&categoryID=898 (date last accessed 18.12.2011)
- Bowling A, Redfern J: The process of outpatient referral and care: the experiences and views of patients, their general practitioners, and specialists. Br J Gen Pract. 2000, 50: 116-120.PubMedPubMed CentralGoogle Scholar
- Berendsen AJ, Kuiken A, Benneker WHGM, Meyboom-de Jong B, Voorn TB, Schuling J: How do general practitioners and specialists value their mutual communication? a survey. BMC Health Serv Res. 2009, 9: 1-10.1186/1472-6963-9-1.View ArticleGoogle Scholar
- Allan AN: Effective communication from general practice: the referral letter. Health Informatics J. 1997, 3 (1): 37-40. 10.1177/146045829700300107.View ArticleGoogle Scholar
- Piterman L, Koritsas S: Part II. General practitioner – specialist referral process. Intern Med J. 2005, 35 (8): 491-496. 10.1111/j.1445-5994.2005.00860.x.View ArticlePubMedGoogle Scholar
- Navarro CM, Miranda IAN, Onofre MA, Sposto MR: Referral letters in oral medicine: standard versus non-standard letters. Int J Oral Maxillofac Surg. 2002, 31: 537-543. 10.1054/ijom.2002.0277.View ArticlePubMedGoogle Scholar
- Shaw I, Smith KM, Middleton H, Woodward L: A letter of consequence: referral letters from general practitioners to secondary mental health services. Qualitative Health Res. 2005, 15 (1): 116-128. 10.1177/1049732304270725.View ArticleGoogle Scholar
- West YR, Kendrick BJL, Williamson DM: Evaluation of the impact of orthopaedic guidelines on referrals from primary care to a specialist department. Qual Prim Care. 2007, 15: 27-31.Google Scholar
- Kerry S, Oakeshott P, Dundas D, Williams J: Influence of postal distribution of the royal college of Radiologists’ guidelines, together with feedback on radiological referral rates, on X-ray referrals from general practice: a randomized controlled trial. Fam Pract. 2000, 17 (1): 46-52. 10.1093/fampra/17.1.46.View ArticlePubMedGoogle Scholar
- Lucassen A, Watson E, Harcourt J, Rose P, O’Grady J: Guidelines for referral to a regional genetics service: GPs respond by referring more appropriate cases. Fam Pract. 2001, 18 (2): 135-140. 10.1093/fampra/18.2.135.View ArticlePubMedGoogle Scholar
- Griffiths CEM, Taylor H, Collins SI, Hobson JE, Collier PA, Chalmers RJG, Stewart EJC, Dey P: The impact of psoriasis guidelines on appropriateness of referral from primary to secondary care: a randomized controlled trial. Epidemiology and Health Serv Res. 2006, 155: 393-400.Google Scholar
- Jiwa M, Skinner P, Coker AO, Shaw L, Campbell MJ, Thompson J: Implementing referral guidelines: lessons from a negative outcome cluster randomised factorial trial in general practice. BMC Fam Pract. 2006, 7: 65-10.1186/1471-2296-7-65.View ArticlePubMedPubMed CentralGoogle Scholar
- Wright J, Harrison S, McGeorge M, Patterson C, Russell I, Russell D, Small N, Taylor M, Walsh M, Warren E, Young J: Improving the management and referral of patients with transient ischaemic attacks: a change strategy for a health community. Qual Saf Health Care. 2006, 15: 9-12. 10.1136/qshc.2005.014704.View ArticlePubMedPubMed CentralGoogle Scholar
- Junghans C, Feder G, Timmis AD, Eldridge S, Sekhri N, Black N, Shekelle P, Hemingway H: Effect of patient-specific ratings vs conventional guidelines on investigation decisions in angina appropriateness of referral and investigation in angina (ARIA) trial. Arch Intern Med. 2007, 167: 195-202. 10.1001/archinte.167.2.195.View ArticlePubMedGoogle Scholar
- McRobbie H, Hajek P, Feder G, Eldridge S: A cluster-randomised controlled trial of a brief training session to facilitate general practitioner referral to smoking cessation treatment. Tob Control. 2008, 17 (3): 173-176. 10.1136/tc.2008.024802.View ArticlePubMedGoogle Scholar
- Kourkouta S, Darbar U: An audit of the quality and content of periodontal referrals and the effect of implementing referral criteria. Prim Dental Care. 2006, 13 (3): 99-106. 10.1308/135576106777795590.View ArticleGoogle Scholar
- Bennett K, Haggard M, Churchill R, Wood S: Improving referrals for glue ear from primary care: are multiple interventions better than one alone?. J Health Serv Res Policy. 2001, 6 (3): 139-144. 10.1258/1355819011927387.View ArticlePubMedGoogle Scholar
- Akbari A, Mayhew A, Alawi Al-Alawi M, Grimshaw J, Winkens R, Glidewell E, Pritchard C, Thomas R, Fraser C: Interventions toimprove outpatient referrals from primary to secondary care. Cochrane database of Sytenmatic Reviews. 2009, 10.1002/14651858.CD005471.pub2.Google Scholar
- Map of Medicine. Available at http://www.mapofmedicine.com/solution/referralmanagement (last accessed 08.05.12)
- Imison C, Naylor C: The King’s Fund. Referral management Lessons for success. 2010, Accessed via http://www.kingsfund.org.uk/publications/referral_management.html (last accessed 09.05.12)Google Scholar
- Grol R, Rooijackers-Lemmers N, van Kaathoven L, Wollersheim H, Mokkink H: Communication at the interface: do better referral letters produce better consultant replies?. Br J Gen Pract. 2003, 53 (488): 217-219.PubMedPubMed CentralGoogle Scholar
- McWhinney IR: A textbook of family medicine. 1989, Oxford: Oxford University PressGoogle Scholar
- Evans E: The Torfaen referral evaluation project. Qual Prim Care. 2009, 17 (6): 423-429.PubMedGoogle Scholar
- Crossley JGM, Howe A, Newble D, Jolly B, Davies HA: Sheffield assessment instrument for letters (SAIL): performance assessment using outpatient letters. Med Educ. 2001, 35: 1115-1124. 10.1046/j.1365-2923.2001.01065.x.View ArticlePubMedGoogle Scholar
- Fox AT, Palmer RD, Crossley JGM, Sekaran D, Trewavas ES, Davies HA: Improving the quality of outpatient clinic letters using the Sheffield assessment instrument for letters (SAIL). Med Educ. 2004, 38: 852-858. 10.1111/j.1365-2929.2004.01898.x.View ArticlePubMedGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6963/13/310/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.