- Open Access
Understanding the factors that affect the appropriateness of rheumatology referrals
BMC Health Services Research volume 21, Article number: 1124 (2021)
Reducing inappropriate referrals to specialists is a challenge for the healthcare system as it seeks to transition from volume to value-based healthcare. Given the projection of a severe shortage of rheumatologists in the near future, innovative strategies to decrease demand for rheumatology services may prove more fruitful than increasing the supply of rheumatologists. Efforts to increase appropriate utilization through reductions in capacity may have the unintended consequence of reducing appropriate care as well. This highlights the challenges in increasing the appropriate use of high cost services as the health system transitions to value based care. The objective of this study was to analyze factors affecting appropriateness of rheumatology services.
This was a cross-sectional study of patients receiving Rheumatology services between November 2013 and October 2019. We used a proxy for “appropriateness”: whether or not there was any follow-up care after the first appointment. Results from regression analysis and physicians’ chart reviews were compared using an inter-rater reliability measure (kappa). Data was drawn from the EHR 2013–2019.
We found that inappropriate referrals increased 14.3% when a new rheumatologist was hired, which increased to 14.8% after wash-out period of 6 months; 15.7% after 12 months; 15.5% after 18 months and 16.7% after 18 months. Other factors influencing appropriateness of referrals included severity of disease, gender and insurance type, but not specialty of referring provider.
Given the projection of a severe shortage of rheumatologists in the near future, innovative strategies to decrease demand for rheumatology services may prove more fruitful than increasing the supply of rheumatologists. Innovative strategies to decrease demand for rheumatology services may prove more fruitful than increasing the supply of rheumatologists. These findings may apply to other specialties as well. This study is relevant for health care systems that are implementing value-based payment models aimed at reducing inappropriate care.
Using specialist-trained physicians appropriately is a challenge for the healthcare system. Specialists have greater knowledge of particular diseases and conditions and can, for some patients, provide superior care to primary care physicians . But specialists are also more expensive [2, 3], use more ancillary services , and often provide care outside their narrow provider specialty . And, beyond those issues, the availability of many specialists is limited, with projections of shortages for many specialist types [6, 7].
The appropriate use of specialists is not a primary problem for many healthcare systems in a volume-based system. Under Fee-For-Service (FFS), reimbursement rates were higher for specialists than primary care providers were, so “over” utilization of specialists generated higher revenues for the system. But as the healthcare system begins to transition away from volume and toward value based care [8,9,10,11,12], the financial incentives for the use of specialty care are also transitioning [13,14,15], leading to a need for more effective targeting of patients to specialists.
Yet most specialists actually have limited control over the demand for their services. Patients do sometime self-refer to specialists [16,17,18], but many specialists receive referrals from primary care providers . To effectively target patients and resolve potential or actual shortages of services, it is therefore necessary to understand what patient, provider and clinic factors effect patient referrals.
One additional complication is the possibility of “supplier induced demand” [20,21,22]. This is the idea that the number of referrals may be endogenous to the number of providers. Given that there is discretion in when a referral is “needed”, the availability of capacity may induce inappropriate utilization.
One specialty where the referral challenge is particularly acute is rheumatology. An aging population and declining work force are creating shortages of providers in some areas, with demand for rheumatology services already exceeding supply by about 13%, or approximately 700 full-time equivalent (FTE) rheumatologists . By 2030, demand is expected to outpace supply by 4133 FTEs (102%).
The ability to increase the supply of rheumatologists is limited by a number of factors, including restrictions on fellowship program positions, inadequate fellowship program fill-rates, a trend toward part-time work, and challenges surrounding international medical graduates’ ability to remain in the US . Given the projection of a severe shortage of rheumatologists in the near future, innovative strategies to decrease the demand for rheumatology services by targeting services to patients who will benefit the most may prove more fruitful than increasing the supply of rheumatologists.
A review of the literature on interventions that address specialty referral management found that about one-third of referrals to rheumatologists are unnecessary or inappropriate . Several studies have evaluated interventions to improve the quality of referrals to rheumatologists. The most effective interventions combine iterative rheumatologist feedback to the referring provider, along with clear referral criteria or evidence-based guidelines [24,25,26]. A study by Lohr et al. compared the quality of referrals from physicians, physician assistants and nurse practitioners and found that referrals from physicians were of better quality (measured by a number of factors) and were less likely to be unnecessary . Little is known regarding other (physician) factors that affect the appropriateness or quality of referrals.
The objective of this study was to identify factors that affect the appropriateness of rheumatology referrals from primary care providers. We used a quantitative analysis using quasi-experimental design and regression analysis. To understand whether induced demand for rheumatology care leads to an increase in inappropriate referrals, we looked at the effect of the start of an additional rheumatologist on the appropriateness of referrals. We looked at this effect 2 months after the start of the rheumatologist (assuming that there would be at least 2 months between a referral and a first appointment) and during a washout period of 6 months, 12, 18 and 24 months, enabling us to analyze the referrals pattern over a period of time.
All methods were carried out in accordance with relevant guidelines and regulations. The institutional review boards of the Central Vermont Medical Center and the University of Vermont approved this study. We used de-identified data from electronic health records; no consent to participate was needed or received for this study.
Our primary source of data for this study was Electronic Health Records (EHRs). Our sample included all patients who visited the Central Vermont Medical Center (CVMC) between November 1, 2013 and October 31, 2019 where the EHR was used to record all data. Our sample inclusion criteria was an appointment in CVMC Rheumatology, providing an initial sample size of 3387 referrals. Exclusion criteria included: patients who did not complete their initial Rheumatology appointment and those with no-shows and cancellations. We also limited the final data set to be one occurrence per unique patient during the study period (n = 2765). Approximately a third of referrals were internal providers (n = 916) and two thirds were external (n = 1849). There were three rheumatology specialists handling all patients in our study and none of the referrals were denied. The three rheumatologists were all experienced, at least 10 years post-fellowship. They were all full-time clinicians, two female and one male and all employed by the hospital.
Design and study setting: defining appropriate referral
To determine whether a referral was appropriate, three Rheumatology rheumatologists at the medical center performed chart reviews (n = 1020 and made independent assessments about appropriateness. We had a time period for collecting the data for the chart review and we reviewed every new referral request over a select time period. That allowed us to manually track the outcomes of the referral much more accurately than using the EHR. We had to make a judgment of appropriate or not appropriate prior to the referral being scheduled. We would not have been able to do this if we just relied on the EHR data – the referral would have already been placed.
The inter-rater reliability of the comparison was calculated using Cohen’s kappa (K). Kappa measures the concordance between two different measures of two raters. It adjusts for the rate of agreement expected by random chance using equation (1):
where P0 is the observed agreement among raters, and Pe is the hypothetical probability of chance agreement, which is defined by using the observed data to calculate the probabilities of each observer randomly seeing each referral. If the raters are in complete agreement, then K = 1.
The results of the chart reviews, indicating which referrals were “appropriate” according to clinician’s judgment, were also compared to the proxy measure from the EHR data using an inter-measure reliability measure.
We also used proxy for “appropriateness” of referrals: whether or not there was any follow-up care for the patient after the first rheumatologist visit. The logic of this measure is that if the patient required continuing care from the rheumatologist, the referral was appropriate. If no continuing care was offered, then the referral was not appropriate.
Analytically, we used logit models to identify factors affecting the appropriateness of referral. Control variables included patient characteristics (age, gender, CCI and county), insurance status and provider characteristics (whether the referring person was a physician or Advanced Practice Provider (APP) and service line (family practice, internal medicine, other specialty or naturopath). In addition, we used 10 years of discharge diagnoses for patients with Rheumatology appointments to develop a Charlson Comorbidity Index (CCI) based on their ICD-10 codes reported in the EHR during this 10-year period. The dependent variable in the analysis was whether or not the referral was identified as appropriate.
We used a Difference-in-Differences (DiD) model to analyze if there were changes in referral-appropriateness when the new rheumatologist started, hypothesizing that increased supply may lead to induced demand.
The difference-in-differences model is estimated as follows in equation (2):
where IR = 1 if the patient was internally referred, according to the EHR, and 0 if a patient was identified as externally referred, and rheumatologist = 1 after the date the extra rheumatologist started and 0 otherwise. The coefficient of the treatment variable, β1, is the estimated mean difference in inappropriate referrals between the internally and externally referred patients. It represents whatever “baseline” differences existed between the groups before the new rheumatologist was added. β2 is the expected mean change in outcome from before to after the start of the new rheumatologist. The rheumatologist effect on inappropriate referrals is measured by β3, the coefficient on the interaction term (IR x rheumatologist) which measures the “difference-in-differences”. It tells us whether the expected mean change in the probability of having an inappropriate referral from before to after the rheumatologist started was different in the two groups. We used marginal effects for all models, which measure the discrete change.
Descriptive statistics for the sample are provided in Table 1. Of the additional scheduled appointments in the EHR during the research period, 70% were completed, 9% were cancelled by the patient, 13% were rescheduled, 4% of patients did not show up for the appointment and 4% were office cancellations. After the non-completed referrals were excluded, the total sample size was 2765 patients, of which about a third internally referred (n = 916) and two thirds were externally referred (n = 1849).
After comparing rheumatologist’s chart reviews (n = 102) for the appropriateness of referrals, we found that there was 84.3% actual agreement and 72.2% expected agreement. The inter-rater reliability measure kappa, therefore, was 0.68, which represents “substantial agreement” . We found that there was 88% agreement between the reviewers and the follow-up proxy measure in the EHR data. The inter-measure kappa (comparing the reviewer rating and whether a follow-up was scheduled) is 0.65, which again represents substantial agreement.
There were more females with inappropriate referrals (73.5%) than with appropriate referrals (69.4%). A higher comorbidity index was associated with a greater likelihood of an appropriate referral (i.e., for CCI = 1, appropriate referrals 31.7% versus 24.4% for inappropriate referrals; 13.1% versus 6.8% for CCI = 2 and 15.9% versus 12.8% for CCI > 2). There were fewer Medicare enrollees (10.2% versus 14.1%) and more Medicaid enrollees (45.2% versus 41.1%) in the group with appropriate referrals. The number of patients with private or other insurance did not significantly differ between groups (44.6% versus 44.8%). The rate of inappropriate referrals was higher for external referrals compared to internal referrals (72% vs. 28%) chi-sq, p = .03). There was a significantly higher percentage of patients under the age of 40 in the group that had inappropriate referrals (19.1%) than in the group with appropriate referrals (13.7) as well as in the age-group 40 to 50 (15.2% versus 13.8%).
We analyzed factors affecting appropriateness of referrals and the effect of introducing an additional rheumatologist (Table 2). We found that patient gender, severity of disease (comorbidities represented by the CCI) insurance status, and where patients live/were referred were predictive of inappropriate referrals. Females had 4.2% higher probability of inappropriate referrals than men did (p = 0.03). Patients with a CCI of 1 had a 15.1% lower probability (p < 0.01) of getting an inappropriate referral compared to those with a CCI of 0. This relationship remained for the other comparisons of non-zero CCI compared with CCI = 0. Specifically, the probability of an inappropriate referral was 16.7% lower for CCI = 2 compared to CCI-0 (p < .01); 13.6% lower for CCI = 3 (p < .01) and 18.8% for CCI = 4 and 13.9 for patients with CCI = 5 or higher (p < 0.01).
Compared to Medicare enrollees, those with private or other insurance had a 4.7% lower probability of getting an inappropriate referral (p = 0.05). We also found that patients living in New Hampshire had a 24.2% higher probability of having an inappropriate referral (p = 0.02) compared to patients living in Washington county, where the hospital is based. We also found that those living in New York had a 30.4% lower probability of inappropriate referral (p = 0.04) compared to Washington county residents. Those living in Chittenden county, where the biggest medical center of the state is based, have a 10.1% lower probability of having an inappropriate referral (p = 0.02).
The addition of another rheumatologist increased the rate of inappropriate referrals. Patients who were referred when the rheumatologist started were 14.8% more likely to have an inappropriate referral compared to patients referred before the new rheumatologist started (p < .001). In this model, we also found that patients who were sicker and those with private insurance (compared to Medicare enrollees) were less likely to have inappropriate referrals.
Figure 1 shows the effect of additional supply affecting the appropriateness of referrals by time period. Inappropriate referrals increased from 14.8 to 15.0% after 6 months after the start of the rheumatologist, to 15.7% after 12 months to 15.3% after 18 months and to 16.4% after 24 months.
Table 3 shows the results of the difference-in-differences (DiD) model. The mean change in the probability of having an inappropriate referral from before to after the additional rheumatologist started was 8.6% higher among patients who were internally referred (p = 0.03), suggesting that the bulk of the additional inappropriate referrals were internally referred.
As in the previous models, we found no significant effect for whether or not the referring provider was a physician versus APP or other type of rheumatologist.
In this paper, we asked what proportion of rheumatology referrals are appropriate and which factors predict the appropriateness of referrals, such as insurance type, socio-demographics, and specialty capacity (i.e., wait time). We found moderately strong agreement among the rheumatologists about which cases should and should not have been referred, and the clinician’s judgment agreed largely with the proxy EHR measure of inappropriateness.
Our findings suggest that there are high levels of inappropriate referrals and that there are systematic, modifiable factors that predict whether a referral is appropriate or not. We found that patient characteristics such as gender, age and comorbidities affect the probability of an inappropriate referral. Those who are privately insured were less likely to have an inappropriate referral than Medicare enrollees, the latter who may seek care without a referral from their primary care provider. We found no significant difference among referring physicians versus APPs and inappropriate referrals; we also did not find differences across services lines. We did find that internal referrals drive a large portion of the difference in the change in unnecessary referrals after adding another rheumatologist to the practice. This might occur because internal providers are more likely to be aware of the additional capacity than external providers are.
The decision to not offer a follow-up appointment may be due to other factors, such as lack of appointment slots due to limited resources. While we acknowledge that the use of a proxy of “If no continuing care was offered, then the referral was not appropriate” does not capture all inappropriate referrals, as often one appointment is all that is necessary to exclude a rheumatologic diagnosis, the study does use a new and innovative approach to evaluate referral appropriateness by comparing the results of a quantitative analysis using EHR data with the “gold standard” of chart reviews. We found there there was a strong correlation between the quantitative analysis using the proxy and the more traditional chart revi ews, suggesting that this approach should be used more often in future research.
In this study, we initially also wanted to analyze the effect of the new MD on no-show rates and cancellations, as we expected wait times to decrease and therefore no-shows and cancellations to decrease. However, we did not have enough data for this analysis and the no-show rates were generally lower than we initially expected.
Reducing inappropriate referrals can happen in a multitude of ways. The challenge moving forward will be to prospectively identify the inappropriate referrals and then reduce their volume. Moving toward adopting a system where we are able to provide additional training and guidance to primary care providers may help. eConsult is a structure from which this teaching may occur . Specifically, consult questions are sent to the specialist and cases are then reviewed based on the available information. Back and forth, communication can then occur between the PCP and specialist until either the issue is resolved or there is a decision to have an in-person visit. Over time, possible benefits include increased PCP knowledge, reduced wait times , and reduction in inappropriate in-person visits. Symptom checkers (SCs) are another potential option to accelerate diagnosis, reduce misdiagnoses, and guide patients more effectively through the health care system .
We also find that capacity is strongly associated with inappropriate referrals. This suggests a dilemma for the health system moving forward. As capacity was added, the vast majority of the additional referrals were appropriate, for both internal and external referrals. Yet the proportion of inappropriate referrals increased. This suggests that efforts to increase appropriate utilization through reductions in capacity may have the unintended consequence of reducing appropriate care as well. This highlights the challenges in increasing the appropriate use of high cost services as the health system transitions to value based care.
Availability of data and materials
The datasets used and/or analyzed during the current study available from the corresponding author on reasonable request.
Harrold LR, Field TS, Gurwitz JH. Knowledge, patterns of care, and outcomes of care for generalists and specialists. J Gen Intern Med. 1999;14(8):499–511.
Emanuel EJ, Fuchs VR. The perfect storm of overutilization. JAMA. 2008;299(23):2789–91.
Gabriel SE, Wagner JL, Zinsmeister AR, Scott CG, Luthra HS. Is rheumatoid arthritis care more costly when provided by rheumatologists compared with generalists? Arthritis Rheum. 2001;44(7):1504–14.
Schofield D. Ancillary and specialist health services: the relationship between income, user rates and equity of access. Aust J Soc Issues. 1999;34(1):79–96.
Lacaille D, Anis AH, Guh DP, Esdaile JM. Gaps in care for rheumatoid arthritis: a population study. Arthritis Care Res. 2005;53(2):241–8.
Cooper RA. There’s a shortage of specialists: is anyone listening? Acad Med. 2002;77(8):761–6.
Glabman M. Specialist shortage shakes emergency rooms; more hospitals forced to pay for specialist care. Physician Exec. 2005;31(3):6.
Halvorson SA, Tanski ME, Yackel TR. Transitioning from volume to value: one academic medical center’s approach to improving population health. Acad Med. 2017;92(5):666–70.
Mayes R. Moving (realistically) from volume-based to value-based health care payment in the USA: starting with Medicare payment policy. J Health Serv Res Policy. 2011;16(4):249–51.
Miller HD. From volume to value: better ways to pay for health care. Health Aff. 2009;28(5):1418–28.
Porter ME, Lee TH. From volume to value in health care: the work begins. JAMA. 2016;316(10):1047–8.
VanLare JM, Conway PH. Value-based purchasing—national programs to move from volume to value. N Engl J Med. 2012;367(4):292–5.
Conrad DA. The theory of value-based payment incentives and their application to health care. Health Serv Res. 2015;50:2057–89.
Lansky D, Nwachukwu BU, Bozic KJ. Using financial incentives to improve value in orthopaedics. Clin Orthop Relat Res. 2012;470(4):1027–37.
Scott A, Liu M, Yong J. Financial incentives to encourage value-based health care. Med Care Res Rev. 2018;75(1):3–32.
Chan BT, Austin PC. Patient, physician, and community factors affecting referrals to specialists in Ontario, Canada: a population-based, multi-level modelling approach. Med Care. 2003;1:500–11.
Kulu-Glasgow I, Delnoij D, de Bakker D. Self-referral in a gatekeeping system: patients’ reasons for skipping the general-practitioner. Health Policy. 1998;45(3):221–38.
Rasoulinezhad S. Patient views for self-referral to specialists; 2007.
Faulkner A, Mills N, Bainton D, Baxter K, Kinnersley P, Peters TJ, et al. A systematic review of the effect of primary care-based service innovations on quality and patterns of referral to specialist secondary care. Br J Gen Pract. 2003;53(496):878–84.
Karimi S, Khorasani E, Keyvanara M, Afshari S. Factors affecting physicians' behaviors in induced demand for health services. Int J Educ Psychol Res. 2015;1(1):43.
Mulley AG. Inconvenient truths about supplier induced demand and unwarranted variation in medical practice. BMJ. 2009;339.
Xirasagar S, Lin HC. Physician supply, supplier-induced demand and competition: empirical evidence from a single-payer system. Int J Health Plann Manag. 2006;21(2):117–31.
Battafarano DF, Ditmyer M, Bolster MB, Fitzgerald JD, Deal C, Bass AR, et al. 2015 American College of Rheumatology Workforce Study: supply and demand projections of adult rheumatology workforce, 2015â 2030. 2018.
Imison C, Naylor C. Referral management: Lessons for success. London: Kings Fund; 2010.
Akbari A, Mayhew A, Al-Alawi MA, Grimshaw J, Winkens R, Glidewell E, et al. Interventions to improve outpatient referrals from primary care to secondary care. Cochrane Database Syst Rev. 2005;3:CD005471.
Roland M, Porter R, Matthews J, Redden J, Simonds G, Bewley B. Improving care: a study of orthopaedic outpatient referrals. Br Med J. 1991;302(6785):1124–8.
Lohr RH, West CP, Beliveau M, Daniels PR, Nyman MA, Mundell WC, Schwenk NM, Mandrekar JN, Naessens JM, Beckman TJ. Comparison of the quality of patient referrals from physicians, physician assistants, and nurse practitioners. Mayo Clinic proceedings. 2013;88(11):1266-71. https://doi.org/10.1016/j.mayocp.2013.08.013.
Rostom K, Smith CD, Liddy C, Afkham A, Keely E. Improving access to rheumatologists: use and benefits of an electronic consultation service. J Rheumatol. 2018;45(1):137–40.
Patel V, Stewart D, Horstman MJ. E-consults: an effective way to decrease clinic wait times in rheumatology. BMC Rheumatol. 2020;4(1):1–6.
Knitza J, Mohn J, Bergmann C, Kampylafka E, Hagen M, Bohr D, et al. Accuracy, patient-perceived usability, and acceptance of two symptom checkers (Ada and Rheport) in rheumatology: interim results from a randomized controlled crossover trial. Arthritis Res Ther. 2021;23(1):1–7.
No funding was received for this study.
Ethics approval and consent to participate
The institutional review boards of the Central Vermont Medical Center and the University of Vermont approved this study, CHRMS (Medical): STUDY00000717 (Approval Date - January 8, 2020). We used de-identified data from electronic health records; the CHRMS waived the need for informed consent according to Sec.__.104(d)(4): Secondary research for which consent is not required.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
van den Broek-Altenburg, E., Atherly, A., Cheney, N. et al. Understanding the factors that affect the appropriateness of rheumatology referrals. BMC Health Serv Res 21, 1124 (2021). https://doi.org/10.1186/s12913-021-07036-5
- Value-based healthcare
- Access to care
- Electronic health records data
- Intermeasure reliability