Low agreement among patients and clinicians about urgency and safety to wait for assessment in primary care after hours medical care: results of cross-sectional matched surveys
BMC Health Services Research volume 23, Article number: 422 (2023)
Discordance between patient and clinician perceived urgency may drive “inappropriate” presentations to after-hours medical services. This paper investigates the level of agreement between patient and clinicians’ perceptions of urgency and safety to wait for an assessment at after-hours primary care services in the ACT.
Cross-sectional survey voluntarily completed by patients and then clinicians at after-hours medical services in May/June, 2019. Agreement between patients and clinicians is measured by Fleiss kappa. Agreement is presented overall, within specific categories of urgency and safety to wait, and by after-hours service type.
888 matched records were available from the dataset. Overall inter-observer agreement between patients and clinicians on the urgency of presentations was slight (Fleiss kappa = 0.166; 95% CI 0.117–0.215, p < 0.001). Agreement within specific ratings of urgency ranged from very poor to fair. Overall inter-rater agreement on how long it would be safe to wait for assessment was fair (Fleiss kappa = 0.209; 95% CI 0.165–0.253, p < 0.001). Agreement within specific ratings ranged from poor to fair. By site type, agreement between patients and clinicians on urgency ranged from not significant to fair and agreement for safety to wait ranged from very poor to slight. Agreement on urgency of issue was more often reported among patients attending their usual health service or seeing their usual clinician compared to patients attending an unfamiliar health service or clinician (χ2(1) = 7.283, p = 0.007 and χ2(1) = 16.268, p < 0.001, respectively).
Low levels of agreement between patients and clinicians on perceived urgency and safety to wait for issues to be assessed indicate potential inefficiency in primary care use after-hours. Agreement on urgency of issues was more common among patients attending a familiar health service or familiar clinician. Improving health literacy, particularly health system literacy, and supporting continuity of care may help to support patients to engage with the most appropriate level of care at the most appropriate time.
Accessible, cost effective, and efficient after-hours medical care in Australia continues to be a focus and challenge for policy makers and health system managers. Currently, a hybrid model of Emergency Departments (ED) and community-based medical services exist outside of routine working hours (9am – 5pm) and during weekends. Emergency Department (ED) presentations are high cost and increasing, with an estimated 40% labelled as “potentially avoidable” [1, 2]. As such, Australian government initiatives plan to reduce “potentially avoidable” ED presentations, particularly in the after-hours periods, with the current focus on GP Urgent Care Centres [1,2,3,4,5,6].
The most common patient reasons for presenting to after-hours medical services are perceived need and sense of urgency [7,8,9,10,11,12]. However, clinicians rarely assess presentations as urgent as patients do [7, 9, 10, 12, 13]. Current understanding of Australian patient and clinician sense of urgency are only from ED settings; with only one international study set in community-based after-hours services . While after-hours primary care in Australia manages a mixture of urgent and non-urgent presentations [11, 14,15,16]; studies report unmatched proportions of patient perceived urgency and clinician reported appropriateness of presentations [7, 9, 10, 12], with only two papers considering case-by-case basis [12, 13]. As such discordance between patient perceived and clinician perceived urgency of a presenting problem may underpin the concept of “potentially avoidable” presentations to after-hours medical services. Matched assessments of perceived urgency between patients and clinicians may help understand current after-hours Australian primary care services, and inform strategies to enhance efficiencies in after-hours access[7, 13].
This paper investigates the level of agreement between matched patient and clinician perceptions of urgency and safety to wait for an assessment for medical encounters at after-hours primary care services in Australia. Findings can inform after-hours models of care and public health messages to direct patients to the most appropriate service at the most appropriate time.
This study examines data collected from a broader project that explored presentations in after-hours primary care [11, 14]. A cross-sectional survey voluntarily completed by practitioners and patients was undertaken in May/June, 2019 from 6pm on Thursday to 8am Monday; covering one weeknight and weekend. Data was not collected Friday 8am-6pm. The project received ethical approval through the ACT Health Human Research Ethics Committee and Calvary Public Hospital Bruce Human Research Ethics Committee (2019/LRE/00003 and CPHB-HREC 10-2019). All methods were conducted in accordance with the declaration of Helsinki.
Participants and recruitment processes have been reported [11, 14]. Briefly, potential participants included (i) any person presenting to a study site during the study period, without obvious barriers to consent, and (ii) any clinician providing care at any medical service in the ACT, open after 6pm Thursday or Friday for at least two hours, or anytime Saturday or Sunday. Data was collected from 50 General Practices with varied billing practices, three Medical Deputising Services (MDS) staffed by GPs generally incurring a co- payment; three Nurse led Walk in Centres (WICs) offering services free at point-of-care; and two public EDs, with services free at point-of-care. Patients can choose to attend any of the services, though are dependent on opening hours and availability of clinicians . GP services are site specific with some providing “walk-in” options, and most open until 6pm weekdays and at least 4 h on Saturday morning; WICs are a “walk in” service only, open 7am-10pm every day; MDS is an appointment and “walk in” service available after 6pm weekdays and anytime on weekends; and ED is a “walk in” service open at all times.
Participants were invited by a research assistant in the waiting room. Patients < 18 years of age, were included if a carer or parent was able to provide informed consent. Practitioners from the EDs were unable to participate. As such, matched data from ED was unavailable. Not all practitioners consented to take part. Patients seeing non-consenting practitioners were still eligible to take part in the patient survey, but could not contribute matched patient data. Unmatched patient data are reported elsewhere .
The research team developed the patient questionnaire, which was informed by literature review and consultation with health care clinicians and consumers. The questionnaire contained 26 questions including a mixture of multiple choice, Likert scale and short answer questions. Patients completed the survey prior to their consultation. Patients could give their health care clinician, at their discretion, a unique code number and consent card to assist data matching.
The research team developed the clinician questionnaire, which was modelled on the Bettering the Evaluation of Care and Health (BEACH) survey . The clinician recorded the patient’s unique code to support paired data analysis. Clinicians completed the questionnaire during or after the patient consultation, with access to their usual diagnostic resources. Details of both patient and clinician survey tools have been previously published [11, 14].
This paper focusses on survey questions: “How urgent is this issue?” and “How long could this issue wait to be safely assessed?” from both patient and clinician questionnaires.
For urgency, patients and clinicians were asked, “How urgent is the problem you are having assessed today?” with answer options including: “An urgent issue (e.g. I have a new pain or my asthma is no longer under control)”, ‘A non-urgent issue (e.g. I need a repeat script or a routine pap smear); or ‘Uncertain about urgency’. For safety to wait, patients and clinicians were asked, ‘How long do you think it would be safe to wait to have this problem assessed?’, For analysis, data was condensed into: ‘Immediately (within 1 hour)’, ‘Within 12 hours’, ‘Within 12–48 hours’ and ‘Within the next week or so’.
Only matched patient and clinician data were used in this analysis. Frequencies, percentages, median and IQR (where required) were used to describe patient characteristics with a matched record. Matched record patient characteristics were compared to ACT population data from 2019 , and the whole patient dataset  using Chi square goodness of fit.
Cross tabulations were generated for patient versus clinician perceptions of safety to wait and urgency. Percentages within tables were calculated as the proportion of patient responses that were agreed upon by the clinician. Proportion of patient responses was considered the most appropriate as this paper focusses on patient perceptions as a driving factor in presentation. For example, if 100 patients reported their problem as ‘Urgent’, and only 20 of their respective clinicians’ responses matched, the agreement is reported as 20%. Total columns for all patient responses and clinician responses are reported.
Fleiss’ kappa was used to indicate inter-rater agreement  for overall ratings of urgency and safety to wait, as well as within specific categories (e.g. agreement within urgent, uncertain, and non-urgent ratings) and is reported alongside 95% Confidence Intervals and p values. Unlike Cohen’s kappa which relies on the same two raters for all cases, Fleiss kappa accounts for each case being rated by a different clinician and patient pair . Table 1 indicates interpretation of kappa scores . Two new bivariate outcomes were calculated to identify when patients and clinicians rated urgency or safety the same (agreement) or different. Pearson chi square was used to identify if agreement was associated with using a familiar health service and with seeing a familiar clinician.
Study population including sites and total patient population have been previously reported [11, 14]. Matched patient questionnaires and consultation records were identified for 888 participants (31.2% of all presentations, and 53.5% of all study participants). Table 2 describes the matched patient population.
Patients with matched records differ from the general ACT population, with over representation of females (51.3% vs. 50.6% p < 0.001) and underrepresentation of males (47.0% vs. 49.4%, p < 0.001) . Patients with matched records did not differ significantly from the total study sample with respect to demographic or characteristic variables .
Of 315 clinicians working during the study period, 86 participated (27.3%). Most clinicians were General Practitioners (n = 56, 65.1%), of whom six were registrars. Other clinicians included Registered Nurses (n = 25, 29%) and Nurse Practitioners (n = 5, 5.8%).
Matched ratings of presentation urgency were available for 828 participants. Matched and total patient and clinician ratings of urgency are shown in Table 3, by site and overall.
Overall, the majority of patients were uncertain of the urgency for their presentation (n = 335; 40.6%), while the majority of clinicians rated presentations as non-urgent (n = 444; 53.6%). Of the patient rated urgent presentations (n = 180), clinicians rated 58.9% as urgent, 31.7% as non-urgent and 9.4% as uncertain. Both patients and clinicians agreed on urgency of presentations in 46.4% of cases (n = 384). Overall inter-observer agreement between patients and clinicians on the urgency of presentations was slight. Agreement within specific ratings of urgency ranged from very poor to fair.
Agreement on the urgency of presentations between patients and GPs within GP services was slight. Patients and GP agreement within specific categories of urgency ranged from fair to very poor. Agreement between patients and Locum doctors through the CALMS services was slight, though not significant. Patient and locum GP agreement within specific categories of urgency ranged from very poor to slight. Agreement on the urgency of presentations between patients and Nurses within WIC services was slight, though not significant. Patients and Nurse agreement within specific categories ranged from very poor to slight. Patients attending their usual health service, or seeing their usual health professional, more often reported agreement with their clinician on urgency of their issue compared to patients not attending their usual health service or not seeing their usual health clinician (χ2(1) = 7.283, p = 0.007 and χ2(1) = 16.268, p < 0.001, respectively) .
Matched ratings of how long it would be safe to wait for assessment was available for 845 participants. Matched and total patient and clinician ratings for safety to wait are shown in Table 4, by site and overall.
The majority of patients rated their presentations as safe to wait for assessment > 12-hours (n = 521; 62.1%). Clinicians rated most presentations as being safe to wait > 12-hours (n = 615, 72%). Both patients and clinicians reported the same ratings in 49.2% of cases (n = 413). Overall inter-rater agreement on how long it would be safe to wait for assessment was fair. Agreement within specific ratings ranged from poor to fair.
Agreement between patients and GPs within GP services about safety to wait for assessment was fair. Patient and GP agreement within specific categories of safety to wait ranged from slight to fair. Agreement between patients and locum GPs within MDS services about safety to wait for assessment was slight. Patient and locum GP agreement within specific categories of safety to wait ranged from very poor and not significant to fair. Agreement between patients and Nurses within WIC services about safety to wait for assessment was slight. Patient and Nurse agreement within specific categories ranged from very poor and not significant to slight. No observed correlation occurred between agreement on safety to wait and patients attending a familiar health service or familiar health professional.
This paper reports low levels of agreement between patients and clinicians on perceived urgency and safety to wait for assessment, indicating potential inefficiencies in primary care use after-hours. Low agreement is not unexpected as patients and clinicians approach medical problems from different perspectives . Identifying where disagreements occur may help highlight strategies to improve efficiency in primary care use and access after-hours.
Slight inter-rater agreement was observed between patients and clinicians about the urgency of presentations. Low levels of overall agreement may be due to the very poor agreement within the “uncertain” category. Clinicians less often rated the urgency of presentations as uncertain compared to patients, which is expected given clinicians training and role in assessing and managing health conditions. Low levels of overall agreement may also occur from both over- and under-estimations of urgency from patients compared to their clinicians. One-third of patients (31.5%) who felt their problem was urgent were rated as non-urgent by clinicians, and nearly a fifth of patients (17%) who believed their problem was non-urgent were rated as urgent by the clinicians. Patient under-estimation may occur when patients present with a benign symptom but the consultation unearths more complex issues (e.g., a patient presents for a repeat pain relief script to manage their “usual headache” but the clinician discovers malignant hypertension). Patient over-estimation may occur where a patient presents with what they fear is life-threatening, but proves to be benign (e.g., a patient presents with abdominal pain fearing acute appendicitis that the clinician finds to be constipation). The frequency of both over- and under-estimation of urgency by patients highlights the complexity of patient health-seeking behaviours. While strategies aimed at improving individual health literacy and self-management may support patients to select the most appropriate health service at the most appropriate time [21, 22]; it could also have an unintended consequence of increasing patients’ general health concerns and therefore drive increased presentations of the “worried well” . Clearly, multiple strategies are needed to support patients to select the most appropriate service at the most appropriate time.
Fair inter-relator agreement was observed between patients and clinicians on how long presentations could safely wait to be seen. Agreement was poorest in the group where patients perceived the need to be seen immediately, with only 6% of clinicians making the same assessment. Low proportions of over- and under-estimation of safety to wait were promising andmay indicate that the patient judgments on when to be seen were mostly safe.
Most presentations in the after-hours period could have safely waited until usual hours to be seen. Patients and clinicians agreed in 37.5% of cases that the assessment could wait > 12 h. If this 37.5% of patients had waited > 12 h to be seen, many would have moved from presenting after-hours to presenting during usual hours. Compared to care during usual hours, after-hours care has higher staffing costs, fewer staff available, and minimal access to ancillary services such as pathology and radiology [3, 5, 23]. While we recognise that presentations are also driven by patient preference and prioritisation around other life commitments ; this proportion of non-urgent patients being seen in the after-hours period is significant, and likely reduces the availability of after-hours primary care for urgent “GP type” presentations. Urgent “GP type” presentations after-hours may instead present to other services, such as Emergency Departments.
Greater agreement between patients and clinicians on urgency and safety to wait for assessment was reported within GP compared to other services, and greater agreement between patients and familiar clinicians. Greater agreement in the GP setting may be underpinned by the relationship-based and ongoing care provided by GPs, where the clinician better understands a patients psycho-social context alongside the presenting issue and can make a more contextually informed assessment of urgency for each patient . In contrast, the models of care provided by WICs and MDS are much more transactional and the clinicians are unlikely to have supporting knowledge of the patient’s social situation or their prior responses to illness episodes. Therefore the clinicians in WIC and MDS can only judge urgency from a medical perspective, without the benefit of greater contextual background. These results highlight the importance of clinician-patient relationships and continuity of care for patient assessment and understanding. However, comparing agreement across services should note the differences in purpose and problems managed:: WICs are advertised as providing care for minor non-urgent issues and often manage wounds; GPs offer usual care on weekends and often manage chronic conditions, infections, and preventive health issues; MDS provide GP-type care that cannot wait for the availability of a usual GP and often manage infections . Findings from this paper aligns with previous studies that show the majority of patients are using services as advertised, though some patients remain unclear on where to present [11, 14].
Strategies to improve efficiencies in after-hours medical services in Australia are currently focussed on diverting patients away from ED [1,2,3,4,5,6]. However, this study shows that more effort is needed to direct patients in selecting the most appropriate alternate ED service for their needs, at the most appropriate time. Improved health literacy could be supported by increased advertising via public health and individual services, to both members of the public and health service counterparts . Advertisement should focus on availability and scope of services, out-of-pocket cost of accessing services after-hours and during usual hours, as well as the benefits of continuity of care. Advertisements could encourage seeing a familiar clinician whenever possible, and promote accessing services during usual hours whenever possible, even if there may be a short delay in care for non-urgent issues. More broadly, patients should be supported to access care during usual hours when possible, such as encouraged use of sick leave for healthcare appointments .
This paper has several strengths and limitations. This is the first study to use matched patient and clinician data from primary care in the after-hours periods, providing a good indication of patient health-seeking behaviours. While number of matched records available was high, the data is representative of a third of presentations to primary care during the study period, and only half of participants in the wider study. High dropout rate from the wider study to matched questionnaires (47%), may partly be due to patients not sharing their study details with their clinician, and partly due to multiple clinicians choosing not to take part. High dropout may indicate discomfort with collecting personal medical data, and may mean that patients with common and non-serious issues were more likely to take part. As such, our results may underestimate the perceived urgency from both patient and clinician viewpoints. However, patients may have experienced a desirability bias where they over reported the urgency of their presentation, which may have increased observed discordance with clinician ratings. Anonymous surveys and data collection being separate from reception or point of care were attempts to mitigate desirability bias. The study sample had a slight overrepresentation of females compared to the ACT population and may not be reflective of the broader Australian population; however, this is reflective of the gender balance of people interacting with the health system . Furthermore, the ACT health system and population may have different structures (e.g., Nurse led WIC) and health seeking behaviours compared with other health districts. However this data may be useful to inform possible interventions to improve efficiencies in after-hours care, and may provide useful comparative data. Finally, data were collected before the COVID-19 pandemic, and it is unclear if and how COVID-19 and related health system changes have altered patient health-seeking behaviours.
Recent efforts to improve health outcomes whilst reducing the burden of care in the formal health system have focussed on the need to improve patients’ health literacy and self-management skills. This study indicates that improving health system literacy and supporting continuity of care may be important components of this overall strategy to help promote utilisation of the appropriate level of care at the appropriate time. Ideally, patients will develop an understanding of not only what their health problem may be and how to manage it, but also where within the health system their problem can be managed effectively and efficiently, as well as how and when to engage with each service.
The data that support this study cannot be shared publicly due to ethical or privacy reasons. De-identified data may be shared upon reasonable request to the corresponding author and approval from the original institutional ethics committee.
Australian Institute of Health and Wellfare. Use of emergency departments for lower urgency care: 2015–16 to 2018–19 2020 [Jan, 2021]. Available from: https://www.aihw.gov.au/reports/primary-health-care/use-of-ed-for-lower-urgency-care-2018-19/contents/lower-urgency-care/summary.
Deloitte Access Economics. Analysis of after hours primary care pathways Canberra. ACT; 2016.
Health Policy Analysis. Evaluation of PHN after hours Program, final report volume 2 Main Report. Canberra: Commonwealth Department of Health; 2020.
Hong M, Thind A, Zaric GS, Sarma S. The impact of improved access to after-hours primary care on emergency department and primary care utilization: a systematic review. Health Policy. 2020;124(8):812–8.
Jackson C. Review of after hours primary health care: Report to the Minister for Health and Minister for Sport. Canberra, Australia; 2014.
Wilkins F. Urgent Care Clinics Kick Off ALP Health Campaign: Medical Republic; 13 April 2022 [Available from: https://medicalrepublic.com.au/urgent-care-clinics-kick-off-alp-health-campaign/67216.
Masso M, Bezzina AJ, Siminski P, Middleton R, Eagar K. Why patients attend emergency departments for conditions potentially appropriate for primary care: reasons given by patients and clinicians differ. Emerg Med Australasia. 2007;19(4):333–40.
Unwin M, Kinsman L, Rigby S. Why are we waiting? Patients’ perspectives for accessing emergency department services with non-urgent complaints. Int Emerg Nurs. 2016;29:3–8.
Callen JL, Blundell L, Prgomet M. Emergency department use in a rural australian setting: are the factors prompting attendance appropriate? Aust Health Rev. 2008;32(4):710–20.
Payne K, Dutton T, Weal K, Earle M, Wilson R, Bailey J. An after hours gp clinic in regional Australia: appropriateness of presentations and impact on local emergency department presentations. BMC Fam Pract. 2017;18(1):86.
Barnes K, Ceramidas D. Why patients attend Afterhours Medical Services: a cross-sectional survey of patients across the australian Capital Territory. Aust J Prim Health. 2022;28(6):549–55.
Keizer E, Smits M, Peters Y, Huibers L, Giesen P, Wensing M. Contacts with out-of-hours primary care for nonurgent problems: patients’ beliefs or deficiencies in healthcare? BMC Fam Pract. 2015;16(1):1–8.
Toloo G-S, Aitken P, Crilly J, FitzGerald G. Agreement between triage category and patient’s perception of priority in emergency departments. Scand J Trauma Resusc Emerg Med. 2016;24(1):126.
Barnes K, Ceramidas D, Agostino J. K. D. After hours presentations to community-based primary care in the australian Capital Territory. Aust J Prim Health. 2022;28(3):232–8.
Turner LR, Pearce C, Borg M, McLeod A, Shearer M, Mazza D. Characteristics of patients presenting to an after-hours clinic: results of a MAGNET analysis. Aust J Prim Health. 2017;23(3):294–9.
Baker J, Britt H, Harrison C. GP services in Australia: presentation profiles during usual practice hours and after-hours periods. Aust J Prim Health. 2020;26(2):117–23.
Britt H, Miller GC, Bayram C, Henderson J, Valenti L, Harrison C, et al. A decade of australian general practice activity 2006–07 to 2015–16. Sydney University Press; 2016.
Australian Bureau of Statistics. ‘Population by age and sex tables’ 2019 [July, 2021]. Available from: https://www.abs.gov.au/statistics/people/population/national-state-and-territory-population/dec-2019.
Fleiss JL. Measuring nominal scale agreement among many raters. Psychol Bull. 1971;76(5):378–82.
Sim J, Wright CC. The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther. 2005;85(3):257–68.
World Health Organization. WHO global strategy on people-centred and integrated health services: interim report. World Health Organization; 2015.
Sørensen K, Van den Broucke S, Fullam J, Doyle G, Pelikan J, Slonska Z, et al. Health literacy and public health: a systematic review and integration of definitions and models. BMC Public Health. 2012;12(1):80.
Australian Government Department of Health. After hours primary health care: Australian Government; 2021 [July, 2021]. Available from: https://www1.health.gov.au/internet/main/publishing.nsf/Content/primary-ahphc#:~:text=The%20PIP%20After%20Hours%20Incentive,to%20provide%20after%20hours%20access.
Hansen E, Walters JHowes F. Whole person care, patient-centred care and clinical practice guidelines in general practice. Health Sociol Rev. 2016;25(2):157–70.
Australian Bureau of Statistics. Patient Experiences in Australia: Summary of Findings: Australian Bureau of Statistics; 2019 [Available from: https://www.abs.gov.au/statistics/health/health-services/patient-experiences-australia-summary-findings/2018-19.
This study would not have been possible without collaboration from multiple groups. The authors would like to acknowledge and express deep gratitude for the contribution of: local GP clinicians in design and interpretation of this study through the ACT Practice Based Research Network (PracNet); feedback on survey design from Health Care Consumers Association (ACT); assistance from the Canberra After-hours Locum Medical Service and Canberra Health Services in implementation of the study in their health services; dedication of all patients, clinicians and managers undertaking the study; and commitment of paid ANU Medical Student research assistants in collecting data.
This study was funded by ACT Health.
ACT Health also fund salaries for KB, and KD. The funder had no input into study design, data collection, analysis, or interpretation. The authors declare no other conflicts of interest.
Ethical approval and consent to participate
This study was conducted in accordance with the Helsinki Declaration, and approved by the ACT Health Human Research Ethics Committee (2019/LRE/00003) and Calvary Public Hospital Human Research Ethics Committee (CPHB HREC 10-2019). All participants provided informed consent via return of completed questionnaire. Patients < 18 years of age, were included if a carer or parent was able to provide informed consent.
Consent for publication
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Barnes, K., Arpel, C. & Douglas, K. Low agreement among patients and clinicians about urgency and safety to wait for assessment in primary care after hours medical care: results of cross-sectional matched surveys. BMC Health Serv Res 23, 422 (2023). https://doi.org/10.1186/s12913-023-09399-3