Skip to main content

Objective risk assessment vs standard care for acute coronary syndromes—The Australian GRACE Risk tool Implementation Study (AGRIS): a process evaluation



Structured risk-stratification to guide clinician assessment and engagement with evidence-based therapies may reduce care variance and improve patient outcomes for Acute Coronary Syndrome (ACS). The Australian Grace Risk score Intervention Study (AGRIS) explored the impact of the GRACE Risk Tool for stratification of ischaemic and bleeding risk in ACS. While hospitals in the active arm had a higher overall rate of invasive ACS management, there was neutral impact on important secondary prevention prescriptions/referrals, hospital performance measures, myocardial infarction and 12-month mortality leading to early trial cessation. Given the Grace Risk Tool is under investigation internationally, this process evaluation study provides important insights into the possible contribution of implementation fidelity on the AGRIS study findings.


Using maximum variation sampling, five hospitals were selected from the 12 centres enrolled in the active arm of AGRIS. From these facilities, 16 local implementation stakeholders (Cardiology advanced practice nurses, junior and senior doctors, study coordinators) consented to a semi-structured interview guided by the Theoretical Domains Framework. Directed Content Analysis of qualitative data was structured using the Capability/Opportunity/Motivation-Behaviour (COM-B) model.


Physical capability was enhanced by tool usability. While local stakeholders supported educating frontline clinicians, non-cardiology clinicians struggled with specialist terminology. Physical opportunity was enhanced by the paper-based format but was hampered when busy clinicians viewed risk-stratification as one more thing to do, or when form visibility was neglected. Social opportunity was supported by a culture of research/evidence yet challenged by clinical workflow and rotating medical officers. Automatic motivation was strengthened by positive reinforcement. Reflective motivation revealed the GRACE Risk Tool as supporting but potentially overriding clinical judgment. Divergent professional roles and identity were a major barrier to integration of risk-stratification into routine Emergency Department practice. The cumulative result revealed poor form completion behaviors and a failure to embed risk-stratification into routine patient assessment, communication, documentation, and clinical practice behaviors.


Numerous factors negatively influenced AGRIS implementation fidelity. Given the prominence of risk assessment recommendations in United States, European and Australian guidelines, strategies that strengthen collaboration with Emergency Departments and integrate automated processes for risk-stratification may improve future translation internationally.

Peer Review reports


Acute coronary syndromes (ACS) are a major cause of morbidity, hospital admission, and mortality placing considerable burden on health systems. Evidence-based therapies informed by timely, structured risk-stratification may improve patient outcomes. Paradoxically, however, lower-risk patients more often receive guideline-recommended, in-hospital care including early invasive strategies and more aggressive drug therapy [1]. This inconsistency is attributed to subjective risk-assessment that underestimates coronary risk and over-estimates risk from cardiac procedures [1,2,3].

As care variance is associated with higher cardiac event and death rate [1, 3], United States, European and Australian clinical guidelines recommend routine use of validated ischaemic/bleeding risk-assessment instruments to assist decisions about invasive strategies for ACS [4,5,6]. Given the weak level of evidence [7] for this recommendation (IIIB), the Australian GRACE Risk score Intervention Study (AGRIS) aimed to test the evidence for routine risk-stratification using the GRACE Risk Tool in patients admitted with ACS [8].

AGRIS was a prospective cluster [hospital-level] randomised, open-label, blinded, endpoint (PROBE) trial of 24 Australian hospitals comparing use of an evidence-based ischaemic/bleeding risk-stratification tool to usual care. The AGRIS rationale and design is published elsewhere [8]. Briefly, the key aims were to: 1) reduce mismatch between individual patient risk, ACS management and clinical outcomes by embedding ischaemic/bleeding risk assessment using the GRACE Risk Tool (GRT) and evidence-based treatment recommendations into routine clinical practice, and 2) comparing patient outcomes between active (n = 12) and control (n = 12) arms. Outcomes of interest were hospital performance (receipt of early invasive strategies, prescription of 4 of 5 guideline-recommended medications, cardiac rehabilitation referral) and all-cause mortality, myocardial infarction (MI), and a composite of mortality or MI at 12-months [8].

The GRT which underpins this practice change combines two validated risk scores to guide ACS treatment decisions [8]: the GRACE Risk Score (GRS) is a compilation of clinical risk-stratification measures (age, heart rate, systolic blood pressure, creatinine, Killip class, ST-segment deviation on ECG, troponin and cardiac arrest event) to assess risk for in-hospital and 6-month mortality [9]; the CRUSADE score uses a compilation of factors to assess major bleeding risk in patients presenting with Non-ST segment Elevation Myocardial Infarction (NSTEMI), before commencement of treatment (heart rate, systolic blood pressure, haematocrit, creatinine clearance, sex, clinical signs of congestive heart failure, peripheral vascular disease and diabetes mellitus) [10].

The analysis of AGRIS ran parallel to, and independent of, this implementation process evaluation. The AGRIS trial found that among 2,318 patient participants, 63% were at high ischaemic risk. The overall rate of invasive management for ACS was higher among hospitals in the active arm, 92% versus 84% (adjusted odds ratio 2.26 [95% C.I.: 1.30–3.96], p = 0.004), and invasive therapy was higher for younger patients at lower ischaemic risk. However, there was no overall difference in: prescription of 4/5 guideline-recommended medications; cardiac rehabilitation referral; provision of all three hospital performance measures for high ischaemic-risk patients; MI; nor, mortality at 12-months [11]. After 4.5 years, interim analysis led to early cessation of AGRIS based on futility to identify a difference between groups [12].

There are several possible contributors to this result. Larger sample sizes are often necessary to address variability in complex interventions [13]: while no difference in secondary prevention, hospital performance or clinical outcomes were observed, this was in the context of a trial stopped before recruitment completion, due to futility. Across groups, clinical performance was better than expected and event rates were low; the hospitals recruited for AGRIS demonstrated higher rates of guideline-indicated therapies than was anticipated, based on historical data from the CONCORDANCE Registry. These factors may have reduced the power to detect meaningful differences [14]. A limitation of the original trial may be the application of the GRT to STEMI patients (one-third of participants), given that in STEMI, early angiography is not influenced by risk-stratification tools.

A further explanation for neutral results may be the quality of implementation fidelity; the degree to which an intervention is delivered as intended [15]. Poor implementation fidelity can shape the relationship between the intervention and its outcomes, leading to erroneous conclusions about the intervention’s effectiveness [13, 16]. This paper aims to understand implementation fidelity (the barriers and enablers to implementation of the GRT into routine clinical workflow) and in doing so, to allow a more nuanced reflection on AGRIS findings [14].



This multi-centre, cross-sectional, descriptive qualitative study used semi-structured interview data. Directed Content Analysis [17] guided by Michie et al. [18] Capability/Opportunity/ Motivation-Behaviour (COM-B) model structured qualitative analysis (Fig. 1). The COnsdolidated criteria for REporting Qualitative research (COREQ) [19] guided reporting rigour.

Fig. 1
figure 1

The COM-B model (reproduced with permission [20])

The AGRIS implementation strategy

While strategies for implementation were expected to vary between hospitals according to local structure, resources and needs, two important implementation principles were pivotal: 1) Adjustment of ACS admission process to include GRT risk-stratification 2) Communication of the clinical utility and relevance of the GRT and treatment recommendations to the entire multi-disciplinary team, specific to their roles.

Key tasks were that staff calculate both the GRS and CRUSADE Score to estimate risk of recurrent MI or death, and bleeding respectively; classify the patient as low, intermediate or high-risk; use this classification to communicate treatment recommendations using the GRT nomogram; and documentation of subsequent recommended treatment as either “indicated”, “not indicated, or “contraindicated” by admitting medical or nursing staff [8]. Stages of implementation appear in Table 1.

Table 1 Stages of implementation

Sample and setting

The process evaluation sought participation from five hospitals from the active arm of AGRIS. Seven hospitals from three Australian states were invited using purposive sampling seeking maximum variation [21] for geographical setting (city versus rural/regional), level and complexity of clinical services, and a diversity of implementation success based on the number of GRT forms completed (including the least and most successful available sites)(Table 2). Two sites in the lowest range of success declined participation: one site did not provide a reason, and a second declined, citing the burden of local submission of a revised ethics protocol.

Table 2 Characteristics of the hospitals participating in AGRIS process evaluation

Participant characteristics

Participants (n = 16) were clinicians and local study coordinators considered key informants in the local implementation of AGRIS. Key stakeholders across active sites were aware, prior to implementation, of the AGRIS process evaluation phase and of possible contact for interview. Once the five participating sites were confirmed, local study coordinators were invited to participate and emailed a participant information and consent form. They then contributed to snowball sampling by inviting other local stakeholders who were forwarded recruitment documentation and invited to contact a researcher (JG) to discuss participation.

Participants included cardiologists (n = 3) (including two heads of department); cardiology registrars (basic physician trainees [n = 2], advanced cardiology trainees [n = 2]); advanced practice cardiology nurses [22] (n = 3); and study coordinators (n = 5) who had clinical backgrounds or concurrent practice in cardiology (predominantly nursing) (Table 3).

Table 3 Role descriptors of participants

Data collection

An interview guide was constructed using the Theoretical Domains Framework; a validated list of expert consensus-derived theoretical constructs known to influence behaviour change [23]. This framework easily intersects with the Behaviour Change Wheel, enabling reflection upon Capability, Opportunity, and Motivation alongside policy categories and implementation functions to design/explain new Behaviours (Table 4) [18, 23]. Once implementation was considered well-established, interviews were conducted by a nurse academic (JG) with experience as an advanced practice cardiology nurse and expertise in qualitative research, including ACS systems of care. No author involved in participant recruitment, nor interviewing, had a pre-existing relationship with any participant. Participants provided a single, semi-structured interview either face-to-face (n = 4), or by phone (n = 12) in a private space in the clinical setting. Median interview duration was 26-min. Interviews were audio-recorded and transcribed verbatim with supporting field notes taken immediately after interview. Analysts were blinded to AGRIS trial results.

Table 4 Mapping the Behaviour Change Wheel’s COM-B system to the Theoretical Domains Framework (reproduced with permission [24])

Data analysis

We drew on the methods of Directed Content Analysis [17], guided by the COM-B model [18] to analyse verbatim interview data. The COM-B model is a theoretical construct, developed through synthesis of 19 frameworks of behaviour change identified through systematic literature review. Inter-rater reliability (79–88%) and concurrent validity (75–85%) of the COM-B as a coding structure was confirmed by the framework developers [18], and the model has since been recommended for use by government agencies [25] and the World Health Organisation [26], with published application to a broad range of clinical contexts [27,28,29,30].

Operational definitions were established for the theoretical categories of the COM-B (Fig. 1): Capability (physical/psychological) is the knowledge and skills that enable people to enact a behaviour; Opportunity (physical/social) refers to external mechanisms that activate or inhibit a behaviour; and Motivation (automatic/reflective), influenced by both capability and opportunity, energises and drives behaviours based on habit or contemplative reasoning (e.g., values/beliefs). Subsequent Behaviours were then explored as outcomes arising from the COM components. Data were coded using these predetermined categories by two doctorally-prepared researchers not otherwise involved in AGRIS (JG, JW). Within these categories, data were further themed inductively to produce a detailed description of the factors that enabled or obstructed implementation of AGRIS for risk-stratification, documentation and subsequent clinical reasoning and action. The resulting description was ‘rich’ with a well-rounded explanation resulting from diverse cases within key themes, and ‘thick’, with multiple exemplars available for theme construction and confirmation [31].

Finally, we discuss the implications of our findings with reference to the Behaviour Change Wheel [18]. Designed to intersect with the COM-B, the Behaviour Change Wheel proposes a range of possible intervention functions (orange band, Fig. 2) and policy categories (grey band, Fig. 2) providing structure for reflection on local implementation.

Fig. 2
figure 2

The Behaviour Change Wheel (Reproduced with Permission [32])


In the reporting of findings, participants are indicated by pseudonym, followed by the superscript number of their employing hospital (Table 2), and their role descriptor (Table 3).

Physical Capability

Physical capability refers to the skills required to complete the target behaviour, specifically, completion of the GRT [18].

GRT usability

Most participants raised the ease of the GRT format. Thea1(BPT) noted this simplicity: “It's all on one sheet… a few numbers, you circle and add them… anyone could do it.” Beth2(APN) noted the form format presented occasional difficulty integrating two risk scores on the one document: “Staff didn't know… if they had a high GRACE score, but… an intermediate CRUSADE score, which colour they should [prioritise]”.

Psychological Capability

Psychological capability refers to the knowledge and thought processes required to make application of the new skill possible [18].

Clinician capability

Regardless of their role in the health system and AGRIS, all participants confirmed their own psychological capability, explaining in detail, the purpose and potential clinical utility of AGRIS. Jared2(Card) believed AGRIS was most useful for junior or non-cardiology doctors: “Cardiologists and trainees do [risk-stratification] automatically. But general medical rotators… don’t see the wood from the trees… if you give them a format… it’s in front of their face.” However, other non-cardiology clinicians had difficulty calculating and documenting the GRT scores: Beth2(APN) explained: "It baulks people… because we're not an interventional hospital. It talks about… primary PCI… medications we don't use… If you're not familiar with specialty terminology, it puts up barriers.”

Education of frontline clinicians

The solid education and support provided by the central AGRIS team were noted across participating hospitals, while information sessions led by influential cardiologists lent credibility. These introductory sessions were then followed by connecting central AGRIS coordinators with local study coordinators and implementation leads to ensure familiarity with study processes. After initial facility training, local study personnel arranged education for rotating medical clinicians with varying levels of engagement. Fran5(CT Coord) explained: “We had all the evidence available… as we trained JMOs… ‘Here's the background… the paperwork’. Some looked at it. Most didn't, but some said, ‘Okay, now I'd like to know this’…”.

Medical trainees Seb3(AT) and Chloe5(Int) felt they missed some initial and ongoing training: “…it's probably issue of our availability… our busyness and not being around.” Sam1(HoD) provided a caveat to the impact of education: “I think we've educated our trainees… The difficult thing is getting the patients’ GRACE score into the conversation and into routine practice.”

Because some frontline clinicians’ had competing demands and priorities, local champions ‘selling the message’ became an important part of psychological capability. Abigail2(CT Coord/APN) explained: “I did a bit of reading… so I could throw some numbers at people… When you say ‘Well, it’s reducing mortality and we’re all here for our patients…’, some people are like, ‘Oh, yeah, I see that. Show me again’.”

Physical opportunity

Physical opportunity requires access to the necessary materials, and time to enact the target behaviour [18].

One more thing to do

Across institutions, more than half the participants described a wide perception of risk-stratification as “one more thing to do”. This was particularly, but not exclusively the case when engaging Emergency Department (ED) clinicians. Beth2(APN) explained: "ED is at saturation point… ‘It's just another form’. They see it as more work and they're not coping with what they’ve got… they’re drowning”. This was confirmed by ED physicians. Jerome2(AT) explained: “When I asked some ED doctors why they weren't doing it routinely the most common response was that they didn't have time”.

GRT visibility

Physical opportunity depended on the GRT’s visibility and availability: it was much more likely to be incorporated into daily workflow when it was ready-to-hand. Jerome2(AT) explained how the lack of form visibility was a barrier: “…it would get lost easily… When I went to ED to find the AGRIS scoring sheet… they were under a mannequin”. Fran5(CT Coord) described how the set-up of AGRIS as a research study hampered GRT visibility and access, with lack of clarity about whether to file the GRT with the progress notes or research files: “We were instructed… to keep it with the patient files, but it became hard… duplicating things, because [research coordinators] have to have copies.”

Paper versus electronic format

Participating hospitals had varying degrees of EMR establishment and none used online ACS order sets. The growing prominence of the electronic medical record (eMR) stimulated discussion about the paper-based GRT format and its impact on physical opportunity. Maree5(APN) appreciated the tangible nature of the paper-based format for fostering clinician engagement: “We’d say, "We'll put [the GRT] in a folder for you… Go through these and use them for your decision-making." Whereas, if they're electronic, you need to go into each patient record to see that.” Jared,2(Card) saw a future in automated curation of GRT data in the eMR to enhance physical capability: “Most of the… score should be self-populating… any decent computer programmer [should enable automatic calculation] so it hits you in the face”.

Social Opportunity

Social opportunity refers to how conducive the socio-cultural milieu is to the target behaviour [18].

A culture of research and evidence

The high initial acceptance of AGRIS in Cardiology was attributed to a strong research culture. Meredith3(BPT) reflected: “Consultants would like us to contribute [to research]… that’s the biggest enabler.”

Clinical workflow

Sometimes, completion of the GRT required information that was not available because of clinical workflow factors. Thea,1(BPT) revealed: “…country patients arrive who we don't yet have details [for]… So we have to chase their presentation heart rate, blood pressure… so [the GRT] can't be done on the fly in the Cath Lab”.

Sandra3(APN) explained further: “It’s workflow… the way the doctors round is a pretty big barrier, it’s rushed… they're thinking of a hundred things, so the [GRT] falls way behind in priority”. This was compounded when complex patients were managed in non-Cardiology areas meaning some risk assessment was missed. A further workflow impact was explained by Oliver(HoD): in his institution, it was the seasonal variation in ACS presentations that dictated angiogram capacity, reducing the relevance of the GRT during the winter peak.

Rotating medical officers

Despite ongoing systematic training, the routine rotation of junior medical officers challenged social opportunity due to a loss of project momentum. Jerome2(AT) explained: “Initially, it had an uptake of about 75%… as the first set of BPTs were leaving, it dropped to about 30%. Second set of BPTs, maximum 20%.” Across participating hospitals, addressing this decline was difficult a when new medical staff already felt overwhelmed by new information.

Automatic motivation

Automatic motivation refers to the elements that enable the target behaviour to become habitual [18].

Reinforcement as an enabler

Automatic motivation to complete the GRT required constant reinforcement through incentives and reminders, and this was frequently led by local study coordinators. Fran5(CT Coord) used incentives with new staff:

…developing relationships as [medical officers] changed…. we'd have coffee vouchers... a nice spread of food… For them to be able to… sit, have a coffee… listen to what we had to say, then go to their next job… lowered [our impact on] their workload.

Reinforcement from senior medical staff was also valued. Jerome2(AT) explained: “If I came across someone in the hallway I’d say, ‘Just remember to use the AGRIS criteria’.” One facility had introduced an innovative reinforcement system where the GRACE/CRUSADE scores were visible on an eMR inpatient list.

Reflective motivation

Reflective motivation is dependent upon people’s intentions to perform, and positive or negative beliefs about the value of the target behaviour [18].

Support for clinical judgement

For regional hospitals, reflective motivation was strengthened by the GRT’s capacity to support clinical judgement, informing triage and transfer decisions. Beth2(APN) explained: “…it's very useful to figure, ‘Hang on, this patient is higher-risk than we thought… let's get them to an interventional hospital sooner.”

Fran5(CT Coord) raised the value of the GRT in discharge-planning: “From a medico-legal perspective, we want to be sure. A lot of patients, we're sending back home… 600-800 km from here”. Jared2(Card) expanded: “It’s not just calculating the score, it’s when… it says, ‘According to this score, you should be doing this’.” Even in tertiary centres, doctors were surprised at the level of some patients’ risk, while others found it captured best practice by prompting a rationale when recommended treatment was omitted.

Equally, some senior physicians were concerned that the GRT could override good clinical judgement. Jerome2(AT) revealed: “…if someone's 85-years-old… Their co-morbidities might not be covered by the GRT… It’s a tool. It shouldn't take the place of a clinician’s decision”. Sam1(HoD) believed the GRT merely replicated good clinical judgement: “If people practice good clinical medicine, this study shouldn't make any difference.”

Divergent professional roles and identity

There were differing approaches to who completed the GRT according to professional roles and identity: (ED versus Cardiology clinicians; and medical versus nursing staff versus study coordinators). The AGRIS investigators intended that ideally, an admitting doctor would complete the form, calculate the risk score and use it to inform and communicate clinical judgement. Jared2(Card) believed that ED involvement was vital to the success of AGRIS: “In our situation [regional hospital]… We ‘drip and ship’: thrombolyse and transfer, so you need ED to be involved.” While some facilities planned for ED to be the primary site for form completion, this was universally unsuccessful. Beth2(APN) explained: “It was to be started in ED but… We got a lot of resistance, so… cardiology teams do it because we've got 24-h to get the form done”.

Difficulty engaging leadership from senior ED clinicians impeded reflective motivation. Abigail2(CT Coord/APN) recalled: “The Director of Emergency [is] like, ‘Nah, not doing it’. It's really hard then. Everyone's like, ‘Well if he thinks it's… not important’, then it's just some nurse saying it”. Participants from three facilities suggested the divergence between departmental approaches arose from a systematic lack of structured risk-documentation in ED, and all raised the Chest Pain Pathway as a similar tool that remained unembedded. Sandra3(APN) explained: “Our [ED] Chest Pain Pathway compliance rate is 2% or something ridiculous. So if that isn't being done…".

The differing evidence bases between ED and Cardiology also seemed to underpin clinicians’ professional identity and feed into their motivation and intention to harmonise clinical practices. Jared2(Card) explained: “If you look at [evidence] within cardiology and ED literature, they’re often diametrically opposed. It’s bizarre. There’s all sorts of things we wouldn’t agree with.” Two participants felt the GRT’s clinical fit for ED was not optimal. Maree5(APN) explained: “[GRT] is not as easily applied in an emergency setting… they would prefer a HEART or EDACS score. They tend to follow their own cohort of experts… So, GRACE is fairly distinctly for cardiology users”.

Due to resistance from ED clinicians across sites, Cardiology teams took primary ownership of GRT completion. Role modelling and leadership from Heads of Department were key in promoting reflective motivation. Sam1(HoD) explained his clear expectations as a leader, and the resulting compliance and impact on clinical reasoning and autonomy. “The registrars have to do it … because I told them to do it… and I think that gave them more motivation to think ‘I should push for this patient having an angiogram’.” This leadership was appreciated at all levels and, along with adequate human resources, seemed to explain why some departments found implementation easier. However, leadership needed to be sustained and engaged. Beth2(APN) explained: “[HoDS] assumed their doctors were doing it, but it's not something they policed.” At one hospital, the head of the Cardiology department was unaware that implementation had begun.

In most hospitals, effective leadership also came from advanced practice cardiology nurses (APNs) who brought skills in education and systems improvement. Despite the research protocol stipulating that implementation be led by clinical staff, study coordinators also completed the forms or recruited other nursing staff to do so to increase completion rates. In fact, most facilities took a pragmatic approach and eventually accepted any reasonable person to complete the GRT to meet the minimum number required per-month (nine) under the AGRIS protocol. One limitation of APN and research coordinator leadership was that there were fewer APN roles in regional hospitals, and many APNs and study coordinators worked part-time, impacting the 24-h completion goal. This led to retrospective form completion, with obvious issues for the GRT’s wider application in early clinical risk-stratification and care-planning.

Resulting behaviours

Form completion behaviours

All facilities reported difficulty maintaining form completion behaviours. This was corroborated by a mean GRT non-completion rate of 33% across the five process evaluation sites, and a mean non-completion rate of 41% across the 12 active arm sites (for the subset of patients also enrolled in the CONCORDANCE Registry). The frequency of non-completion was distributed evenly across tertiary, metropolitan and regional hospitals. One reason for the lack of embedded scoring behaviours was that, despite intentions for embedded practice, AGRIS was perceived as a temporary research project. Thea1(BPT) explained: “Embed is probably a strong word [laughs]… it's still a trial in terms of its perception”.

The minimum of nine forms per-month of CONCORDANCE participants, (rather than all eligible admitted patients) seemed the overall target for several sites. Yvette1(CT Coord) explained: “I collected all the GRACE scales… but they said, ‘Oh no, we only do the first ten per-month’.” Alice4(CT Coord) struggled for that target: “They needed nine consecutive admissions … we were struggling, so… [the team] decided… get two or three consecutive patients, for a total nine each month.”

Communication behaviours

No hospital reported the GRT as an embedded element in routine clinical conversations. Regional hospitals described the greatest utility for communication. Jerome2(AT) revealed: “Having objective measures increases your ability to make an argument… Particularly when we can’t get patients up to bigger hospitals easily.” From the same facility, Abigail2(CT Coord/APN) recalled: “I've seen them mention the score in a couple of transfer letters…”, and “Some… notes will say, ‘Therefore, we've started [X] treatment’… but not a lot”.

Cardiology APNs Maree5 and Beth2 used the GRT in bedside rounds during implementation. Beth2 explained: “You can point out… ‘This patient has a high [GRT]. If we can't get them to Hospital X, can we get them somewhere else?'” However, few participants recalled its application in this context. The problem seemed to be its unsystematic use. Sam1(HoD) explained, “We don't have the culture of someone saying, ‘Mrs. Smith's arrived with ACS and her GRACE score is this.’ So it might be calculated, but it's not integrated into handover or clinical pathways”.

Clinical practice behaviours

While participants saw value in the GRT, not many believed it changed clinical practice. Thea1(BPT) explained: “I don't think it's changed our [angiography] practice… checking optimal guideline medical therapy on discharge… that's probably improved… and… cardiac rehab referrals”. Sandra3(APN) agreed: “I don't think it's changed what we do. I think it's validated what we do.”


The COM-B model, underpinned by the Theoretical Domains Framework, was a valuable analytic tool to collect and analyse qualitative data on implementation fidelity. The sources of behaviours (capability; opportunity; motivation) are integrated by its theorists into the “Behaviour Change Wheel” (Fig. 2), a useful vehicle to discuss implementation strategies described by participants. In future GRT implementation initiatives, local implementation strategies could, perhaps, be strengthened by longitudinal stakeholder interviews to inform dynamic, tailored application of the intervention functions and policy categories of the Behaviour Change Wheel in a continuous cycle of practice development. While often seen as one more thing to do, the extra work generated by the GRS tool apparently fed into care processes and quality indicator alignment for some participants. Using ongoing stakeholder feedback during implementation may address issues early, raise awareness among junior clinicians and shape their cognitive approach to ongoing patient assessment. This manuscript, through the use of this theoretical structure, provides insight into how to improve and promote future implementation of risk assessment into daily practice, and how this ongoing, timely feedback may complete the quality improvement cycle.

Perhaps the intervention functions most evident were the positive elements of training, education, and persuasion (eg staff meetings, education sessions and incentives), and modelling and leadership (through enablement eg making forms visible, and key clinical advocates supporting/encouraging form completion). These strategic foci likely reflect the investigators’ reasonable assumption that specialist clinicians concerned for patient welfare, presented with compelling evidence, appropriate tools and support would engage with an intervention. Study coordinators were integral to GRT completion and this likely reflects their passion and commitment to the study’s momentum. This was, however, at odds with the study protocol which reserved implementation activities for clinicians. This may explain the wider perception of GRT risk-stratification as a time-limited research activity, rather than an intention for usual care.

The AGRIS steering committee deliberately avoided coercing or legislating change (i.e., mandating), with treatment decisions remaining at the discretion of clinicians. Such flexibility is known to enhance implementation success [33, 34]. Local adaption of GRT layout and treatment recommendations were permissible, providing these remained aligned to process measures for high-risk patients (coronary angiography, evidence-based medications, cardiac rehabilitation). This discretion reflected understanding that risk-assessment tools are designed to complement rather than replace clinical judgement [34]. Regardless, some medical participants described a persisting perception that the GRT (and perhaps decision support tools in general) “mandate” a specific approach rather than guiding/complementing clinical judgement.

Despite this implementation science strategy, it was difficult to embed behaviour change, particularly in EDs where this risk-stratification would ideally occur. This mirrors findings of another Australian process evaluation, the T3 trial [35] that aimed to improve triage, treatment and transfer of stroke patients in the ED. Notwithstanding a rigorously-designed implementation program and initial high-level clinician buy-in, contextual factors including beliefs about supporting evidence, and poor medical staff engagement were significant barriers. Perceived professional boundaries created tension between the roles and responsibilities of general ED versus specialist stroke practice [35]. This resonates with identified barriers in our study, with a perception of differing evidence bases underpinning cardiology and ED practice, and preferred tools such as the EDAC score for the emergency setting.

Another Australian evaluation, the multi-centre Accelerated Chest pain Risk Evaluation (ACRE) project [36] was initiated by Queensland Emergency clinicians. Using the Theoretical Domains Framework, they also identified professional roles and boundaries as a barrier, although with the reverse of our problem: the ED initiators found it difficult to engage Cardiology clinicians, despite obvious overlap in clinical ACS care. They noted higher engagement in facilities with a strong existing collaborative culture between Cardiology and ED.

The perceived impact on workflow during medical rounds, and the sense that formal risk-stratification was ‘one more thing to do’ resonates with past research. ED participants in the T3 trial [35] saw the extra clinical activities as time-consuming distractions to more urgent priorities, and the Canadian Head CT rule study [37] found institutional context and resources shaped engagement, citing poor rule compliance when clinicians were particularly busy. In the process evaluation of computerised asthma and angina decision-support [38], it was the complexity and comorbidities of patients that challenged tool integration during busy clinical encounters. This study was used as an exemplar in the UK’s Medical Research Council guideline on developing/evaluating complex interventions [25]. They acknowledged the difficulty gaining traction in such programs, despite a well-conceived implementation and initial enthusiasm and optimism of stakeholders.

Several of our participants cited Chest Pain Pathways as exemplifying a systemic lack of engagement with standardised ACS assessment/decision-support. Despite Chest Pain Pathways mandated as the Australian minimum standard for chest pain evaluation, a “cultural aversion to pathways” and disparities between ED and cardiology commitment are two reported explanations for inconsistent pathway use and significant adverse events in Root Cause Analyses and coronial investigations [39].

Furthermore, a tool considered widely adopted at systems level, but with low levels of use when audited, reveals the dichotomy between care‐as‐practiced and care‐as‐documented [40]: failing to enrol key actors in a tool’s use gives an outside appearance of transparency and standardisation, while preserving medical autonomy that results in process variance with evidence-based care [25]. Without the logical structure of a risk-assessment tool, assessment findings and associated evidence-based decision-making is hard to teach, document and communicate [41] and is difficult to audit for quality review and benchmarking.

Perhaps the greatest potential for formalised risk-assessment tools is the visual nomogram designed to link risk-stratification to clinical decision-making, encouraging systematic deliberation. This may reduce conceptual biases in memory, improving behavioral intentions among less-skilled individuals [42]. If the data required for these common risk stratification tools were built into eMR admission systems as threshold activities (i.e. electronic order sets), feeding into automated risk-calculations and digitally-displayed, evidence-based treatment plans, it could reduce perceived ED workload, increase risk score visibility, and capture important data for quality monitoring/improvement. These should be developed with ED/Cardiology workflow in mind [43]. The groundwork for such technologies is rapidly developing [44, 45].

A limitation of this study is the nature of the sample. Qualitative research most commonly uses a non-probability strategy guided by the characteristics of a population, and the study objectives, and in our case, we did seek maximum variation for discipline and level of experience. In the initial design, the process evaluation was intended to focus on key professionals involved in the local AGRIS implementation. While we planned for implementation to be well-established by the time of recruitment, several trainee physicians and nursing staff involved in initial implementation had moved from their departments/hospitals and were unavailable for interview. A snowball strategy was used because the implementation approach, by design, differed at each site, as did the characteristics of staff involved. While we sought inclusion of people in key roles such as Heads of Department, APNs and medical trainees at each site, the ethical principle of arms-length recruitment meant these invitations were extended via the local study coordinators. This snowball sampling may have introduced selection bias [46]: coordinators likely approached people most motivated by, and committed to, the trial; senior clinicians who felt accountable for an unsuccessful implementation may have been less willing to provide an interview; and, while one APN worked in the emergency setting, the ED clinicians who were difficult to engage in the trial (and therefore would be most informative for this evaluation) were not recruited. Furthermore, in making an initial decision not to engage, they may not have viewed themselves as stakeholders. Finally, the opinions of rotating medical officers in control arm facilities without access to decision-support would be useful to access, given the GRT is considered most valuable for less experienced physicians.


Despite implementation leadership from local AGRIS champions, poor wider implementation fidelity reflected a failure to embed risk-stratification processes into routine admission procedures, clinical conversations and a documented rationale for practice throughout the ED and early cardiology patient journey. This understanding may, in part, explain the neutral AGRIS results, while providing valuable insights into future implementation initiatives. Strategies that strengthen collaboration with EDs and integrate automated processes for risk-stratification data may improve future translation. The English (UKGRIS) and planned Canadian (CanGRIS) studies will provide larger samples, and more conclusive understandings of the potential for routine risk-stratification in improving ACS care [14].

Availability of data and materials

The datasets generated and analysed during the current study are not publicly available. Raw qualitative data contains unique stories that if shared, threatens participant confidentiality. As this is workplace data, many participants are known to each other. We do not have participant consent to share raw data. Additionally, the context-bond nature of qualitative data limits its use for secondary analysis when removed from its context.



Acute Coronary Syndrome


Australian Grace Risk Score Intervention Study


Advanced Practice Nurse


Advanced Physician Trainee


Basic Physician Trainee

COM-B Model:

Capability, Opportunity, Motivation – Behaviour Model

CT Coord:

Clinical Trials Coordinator


Emergency Department


Global Registry of Acute Coronary Events


Grace Risk Tool


ST Elevation Myocardial Infarction


Myocardial infarction


Non-ST Elevation Myocardial Infarction

T3 trial:

Triage, treatment and transfer of patients with stroke in emergency departments trial


United Kingdom GRACE Risk Intervention Study


Canadian GRACE Risk Intervention Study


  1. Saar A, Marandi T, Ainla T, Fischer K, Blöndal M, Eha J. The risk-treatment paradox in non-ST-elevation myocardial infarction patients according to their estimated GRACE risk. Int J Cardio. 2018;1(272):26–32.

    Google Scholar 

  2. Engel J, van der Wulp I, de Bruine M, Wagner C. A cross-sectional multicentre study of cardiac risk score use in the management of unstable angina and non-ST-elevation myocardial infarction. BMJ Open. 2015;5(11).

    PubMed  PubMed Central  Google Scholar 

  3. Lee C, Tan M, Yan A, Yan R, Fitchettt D, Grima E, et al. Use of cardiac catheterization for non-ST-segment elevation acute coronary syndromes according to initial risk: reasons why physicians choose not to refer their patients. Arch Intern Med. 2008;168(3):291–6.

    PubMed  Google Scholar 

  4. Roffi M, Patrono C, Collet J-P, Mueller C, Valgimigli M, Andreotti F, et al. 2015 ESC guidelines for the management of acute coronary syndromes in patients presenting without persistent ST-segment elevation. Eur Heart J. 2016;37(3):267–315.

    CAS  PubMed  Google Scholar 

  5. Amsterdam E, Wenger N, Brindis R, Casey D, Ganiats T, Holmes D, et al. 2014 AHA/ACC Guideline for the Management of Patients With Non–ST-Elevation Acute Coronary Syndromes. Circulation. 2014;130(25):e344–426.

    PubMed  Google Scholar 

  6. NHF/CSANZ, National Heart Foundation of Australia & Cardiac Society of Australia and New Zealand. Australian Clinical Guidelines for the management of Acute Coronary Syndromes. Heart Lung and Circulation. 2016;25(9):895–951.

    Google Scholar 

  7. National Health and Medical Research Council. NHMRC levels of evidence and grades for recommendations for developers of guidelines. In: NHMRC, editor. Canberra: Australian Government; 2009.

  8. Chew D, Astley C, Luker H, Aliprandi-Costa B, Hillis G, Chow C, Brieger D. A cluster randomized trial of objective risk assessment versus standard care for acute coronary syndromes: Rationale and design of the Australian GRACE Risk score Intervention Study (AGRIS). Am Heart J. 2015;170(5):995–1004.

    PubMed  Google Scholar 

  9. Fox K, Dabbous O, Goldberg R, Pieper K, Eagle K, Van de Werf F, et al. Prediction of risk of death and myocardial infarction in the six months after presentation with acute coronary syndrome: prospective multinational observational study. BMJ. 2006;333(7678):1091–4.

    PubMed  PubMed Central  Google Scholar 

  10. Subherwal S, Bach R, Chen A, Gage B, Rao S, Newby K, et al. Baseline Risk of Major Bleeding in Non–ST-Segment–Elevation Myocardial Infarction: The CRUSADE (Can Rapid risk stratification of Unstable angina patients Suppress ADverse outcomes with Early implementation of the ACC/AHA guidelines) Bleeding Score. Circulation. 2009;119(14):1873–82.

    PubMed  PubMed Central  Google Scholar 

  11. Ariza-Solé A, Sánchez-Elvira G, Sánchez-Salado JT, Lorente-Tordera V, Salazar-Mendiguchía J, Sánchez-Prieto R, Romaguera-Torres R, et al. CRUSADE bleeding risk score validation for ST-segment-elevation myocardial infarction undergoing primary percutaneous coronary intervention. Thrombosis Res. 2013;132(6):652–8.

    Google Scholar 

  12. Bavry A, Bhatt D. Australian GRACE Risk Intervention Study - AGRIS. Am J Cardio. 2019.  Available from: Accessed 4 May 2021.

  13. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: The new Medical Research Council guidance. Int J Nurs Stud. 2013;50(5):587-92.

  14. Chew D, Hyun K, Morton E, Horsfall M, Hillis G G, Chow C, et al. Objective Risk Assessment vs Standard Care for Acute Coronary Syndromes: A Randomized Clinical Trial. JAMA Cardiol. 2021;6(3):304–13.

    PubMed  Google Scholar 

  15. Toomey E, Hardeman W, Hankonen N, Byrne M, McSharry J, Matvienko-Sikar K, et al. Focusing on fidelity: narrative review and recommendations for improving intervention fidelity within trials of health behaviour change interventions. Health Psychol Behav Med. 2020;8(1):132–51.

    CAS  PubMed  PubMed Central  Google Scholar 

  16. Carroll C, Patterson M, Wood S, Booth A, Rick J, Balain S. A conceptual framework for implementation fidelity. Implementation Sci. 2007;2:40.

    Google Scholar 

  17. Hsieh H-F, Shannon S. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277–88.

    PubMed  Google Scholar 

  18. Michie S, van Stralen M, West R. The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Science. 2011;6:42.

    Google Scholar 

  19. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349–57.

    PubMed  Google Scholar 

  20. Waltz T, Powell B, Fernández M, Abadie B, Damschroder L. Choosing implementation strategies to address contextual barriers: diversity in recommendations and future directions. Implementation Sci. 2019;14:42.

    Google Scholar 

  21. Sandelowski M. Whatever happened to qualitative description? Res Nurs Health. 2000;23(4):334–40.

    CAS  PubMed  Google Scholar 

  22. Nursing and Midwifery Board of Australia, editor. Fact sheet: Advanced nursing practice and specialty areas within nursing. Melbourne: Australian Health Professionals Regulation Authority; 2019.

  23. Cane J, O’Connor D, Michie S. Validation of the theoretical domains framework for use in behaviour change and implementation research. Implement Sci. 2012;7:37.

    PubMed  PubMed Central  Google Scholar 

  24. Michie S, Johnston M, Abraham C, Lawton R, Parker D, Walker A. Making psychological theory useful for implementing evidence based practice: a consensus approach. Qual Saf Health Care. 2005;14(1):26–33.

    CAS  PubMed  PubMed Central  Google Scholar 

  25. West R, Michie S, Chadwick P, Atkins L, Lorenccatto F, for Public Health England. Achieving behaviour change A guide for national government. London: Department of Health and Social Care; 2020.

    Google Scholar 

  26. Dodson S, Heggen A, Solomon A, Sarah VW, Woods G, Wohlgemuth L. Behavioural change interventions for sustained trachoma elimination. Bull World Health Organ. 2018;96(10):723–5.

    PubMed  PubMed Central  Google Scholar 

  27. West R, Michie S, Rubin G, Amlôt R. Applying principles of behaviour change to reduce SARS-CoV-2 transmission. Nat Hum Behav. 2020;4(5):451–9.

    PubMed  Google Scholar 

  28. Barker F, Atkins L, de Lusignan S. Applying the COM-B behaviour model and behaviour change wheel to develop an intervention to improve hearing-aid use in adult auditory rehabilitation. Int J Audiol. 2016;55:S90–8.

    PubMed  Google Scholar 

  29. McDonagh L, Saunders J, Cassell J, Curtis T, Bastaki H, Hartney T, et al. Application of the COM-B model to barriers and facilitators to chlamydia testing in general practice for young people and primary care practitioners: a systematic review. Implement Sci. 2018;13(1):130.

    PubMed  PubMed Central  Google Scholar 

  30. Boyd J, McMillan B, Easton K, Delanet B, Mitchell C. Utility of the COM-B model in identifying facilitators and barriers to maintaining a healthy postnatal lifestyle following a diagnosis of gestational diabetes: a qualitative study. BMJ Open. 2020;10:e037318.

    PubMed  PubMed Central  Google Scholar 

  31. Fusch P, Ness L. Are We There Yet? Data Saturation in Qualitative Research. The Qualitative Report. 2015;20(9):1408–16.

    Google Scholar 

  32. Michie S, West R. Sustained behavior change is key to preventing and tackling future pandemics. Nat Med. 2021;27:749–52.

    CAS  PubMed  Google Scholar 

  33. Damschroder L, Aron D, Keith R, Kirsh S, Alexander J, Lowery J, et al. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4(1):50.

    PubMed  PubMed Central  Google Scholar 

  34. Kirsh S, Lawrence R, Aron D. Tailoring an intervention to the context and system redesign related to the intervention: a case study of implementing shared medical appointments for diabetes. Implement Sci. 2008;3:34.

    PubMed  PubMed Central  Google Scholar 

  35. McInnes E, Dale S, Craig L, Phillips R, Fasugba O, Schadewaldt V, et al. Process evaluation of an implementation trial to improve the triage, treatment and transfer of stroke patients in emergency departments (T3 trial): a qualitative study. Implement Sci. 2020;15:99.

    PubMed  PubMed Central  Google Scholar 

  36. Skoien W, Page K, Parsonage WA, Ashover S, Milburn T, Cullen L. Use of the Theoretical Domains Framework to evaluate factors driving successful implementation of the Accelerated Chest pain Risk Evaluation (ACRE) project. Implement Sci. 2016;11(1):136.

    PubMed  PubMed Central  Google Scholar 

  37. Curran J, Brehaut J, Patey A, Osmond M, Stiell I, Grimshaw J. Understanding the Canadian adult CT head rule trial: use of the theoretical domains framework for process evaluation. Implementation Sci. 2013;8:25.

    Google Scholar 

  38. Eccles M, McColl E, Steen N, Rousseau N, Grimshaw J, Parkin D, et al. Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. BMJ. 2002;325(7370):941–8.

    PubMed  PubMed Central  Google Scholar 

  39. NSW Health. Minimum standards for chest pain evaluation: Implementation support guide – clinicians summary. Agency for Clinical Innovation Cardiac Services Network. Chatswood; 2011. Accessed 25 June 2021.

  40. Martin G, Kocman D, Stephens T, Peden C, Pearse M. Pathways to professionalism? Quality improvement, care pathways, and the interplay of standardisation and clinical autonomy. Sociol Health Illn. 2017;39(8):1314–29.

    PubMed  Google Scholar 

  41. Fletcher J, Cooper P, Pearson B, Fenwick K. Identifying risk: clinical judgement versus risk assessment tools (RATs). Wounds UK debate. 2010;6(4):180–93.

    Google Scholar 

  42. Garcia-Retamero R, Cokely E. Designing Visual Aids That Promote Risk Literacy: A Systematic Review of Health Research and Evidence-Based Design Heuristics. Hum Factors. 2017;59(4):582–627.

    PubMed  Google Scholar 

  43. Li R, Wang J, Sharp C, Chen J. When order sets do not align with clinician workflow: assessing practice patterns in the electronic health record. BMJ Qual Saf. 2019;28(12):987–96.

    PubMed  PubMed Central  Google Scholar 

  44. Tam C, Gullick J, Saavedra A, Vernon S, Figtree G, Chow C, et al. Combining structured and unstructured data in EMRs to create clinically defined EMR-derived cohorts. BMC Med Inform Decis Mak. 2021;21(1):91.

    PubMed  PubMed Central  Google Scholar 

  45. Tiberio A, Brown A, Lee A, Femia G, Fetahovic T. Electronic Chest Pain Pathway (eCCP). Heart Lung Circ. 2015;24:S118.

    Google Scholar 

  46. Cohen N, Arieli T. Field research in conflict envirinments: methodological challenges and snowball sampling. J Peace Res. 2011;48(4):423–35.

    Google Scholar 

Download references


Professor Keith Fox, University of Edinburgh, is acknowledged for support with funding acquisition and critical comment on the final manuscript.


This process evaluation phase of AGRIS was funded through the grant scheme “ACS Risk Award” by the University Court of the University of Edinburgh, with financial support through an educational grant from AstraZeneca AB: Project name—“Process Evaluation of implementation of the GRACE Risk Score in hospitals participating in the AGRIS study”. Project ref 008; Award Recipient David Brieger, Donna Waters and Bernadette Aliprandi-Costa. The funder had no role in design of the study, nor collection, analysis nor interpretation of data, nor in manuscript development. KH’s salary is funded by a National Health and Medical Research Council Emerging Leadership 1 Fellowship [APP1196724].

Author information

Authors and Affiliations



DC, DB, and DW conceived the work as the final phase of the substantive AGRIS project; JG and DW designed the methodological approach; JG collected the qualitative interview data; JG, and JW analysed the qualitative data; KH provided descriptive statistics in relation to site engagement and form completion; JG interpreted the results; JG, DC, CG, AT, SG, DW, KH and DB made a substantial contribution to critical revision of the manuscript. All authors have approved the final version of the manuscript.

Corresponding author

Correspondence to Janice Gullick.

Ethics declarations

Ethics approval and consent to participate

Participants provided written consent. Study approval was obtained from the Sydney Local Health District (CRGH Zone) Human Research Ethics Committee. CH62/2013–154, HREC/13/CRGH/220. All methods were performed in accordance with the guidelines and regulations of the National Health and Medical Research Council Statement on Ethical Conduct in Research [Australian Government] and are in line with the Declaration of Helsinki. Participants provided informed consent.

Consent for publication

Not applicable.

Competing interests

The main AGRIS study, and this process evaluation study had funding support by AstraZeneca Australia, and DC, AY, and DB have received other grant support from Astra Zeneca outside of this work. Other authors declare they have no competing interests. The funders of the study had no role in the study design, data collection, data analysis, data interpretation or preparation of the manuscript.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gullick, J., Wu, J., Chew, D. et al. Objective risk assessment vs standard care for acute coronary syndromes—The Australian GRACE Risk tool Implementation Study (AGRIS): a process evaluation. BMC Health Serv Res 22, 380 (2022).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: