Table 1 External validation of prediction tools

 Campbell et al. (2009) (version 8.0) [41]

1065 patients ≤85 years with T1–2, N0, M0 tumors diagnosed between 1986 and 1996 at Churchill Hospital in Oxford (UK).

For the whole cohort at 10 years, Adjuvant!Online significantly overestimated OS (by 5.54%, P = 0.001), BCSS (by 4.53%, P = 0.001), and EFS (by 3.51%, P = 0.001).

OS significantly overestimated for the following parameters: age, menopausal status, Grade 2 + 3, nodal involvement, tumor size 1–2 cm, ER+, local therapy, no systemic therapy and hormone therapy only.

de Glas et al. (2014) (version 8.0) [42]

2012 patients ≥65 years with early breast cancer diagnosed between 1997 and 2004 in the western Netherlands.

Adjuvant!Online significantly overestimated 10-year OS. The difference between observed and predicted 10-year overall survival was 9.8% ([95% CI 5.9–13.7], p < 0.0001, c-index 0.75). 10-year cumulative recurrence was overestimated by 8.7% ([95% CI 6.7–10.7], p < 0.0001, c-index 0.67) when comorbidity was defined as “average for age”. Definition of comorbidity by an expert panel resulted in significant underestimation of 10-year OS by − 17.1% ([95% CI − 21.0 to − 13.2], p < 0.0001, c-index 0.70) but accurate prediction of cumulative recurrence (− 0.7% [95% CI − 2.7–1.3], p = 0.48, c-index 0.62).

 Hajage et al. (2011) (version 8.0) [43]

I. 456 French patients with N0 M0 tumors diagnosed between 1995 and 1996.

II. 295 Dutch patients with T1–2 N0 M0 tumors ≤52 years diagnosed between 1984 and 1995.

I. No significant difference between predicted and observed survival, but survival overestimated for women receiving chemotherapy only.

II. 10-year OS was significantly overestimated by 13% (p = 0.00001).

 Jung et al. (2013) (version 8.0) [44]

699 Korean patients with T1–3, N0–3, M0 treated between 1986 and 1999.

Adjuvant!Online significantly overestimated 10-year OS by 11.1%, BCSS by 11.6% and EFS by 9.3% (all p< 0.001).

 Yao-Lung (2012) (version 8.0) [45]

559 Taiwanese patients treated between 1992 and 2001with N0–3, M0

No significant differences in predicted OS in low-risk patients but overestimation of survival in high risk patients (predicted:observed risk = 1.26; p = 0.016)

 Mook et al. (2009) (version 8.0) [46]

5380 patients with T1–3, M0 tumors diagnosed between 1987 and 1998 at the Netherlands Cancer Institute.

For the whole cohort, there were no significant differences between predicted and observed 10-year OS and BCSS. OS was significantly overestimated for women < 40 years and ≥ 70 years. BCSS was significantly underestimated for women with mastectomy, DCIS, and ER-. Tumor size and age resulted in overestimation as well as underestimation of BCSS.

C-index was 0.70 for OS and 0.71 for BCSS.

 Olivotto et al. (2005) (version 5.0) [47]

4083 patients with T1–2, N0–1, M0 tumors diagnosed between 1989 and 1993 in British Columbia (Canada).

No significant differences between predicted and observed 10-year OS, BCSS and EFS. OS was significantly overestimated for women < 35 years, with positive nodes and a combination of hormones and chemotherapy. OS was underestimated for women with negative nodes, without systemic therapy and BCS + RT.

 Candido Dos Reis (2017) (version 2, refitted) [57]

5738 patients diagnosed between 1999 and 2003 in the UK (ECRIC dataset)

1944 patients diagnosed between 1989 and 1998 from the Nottingham/Tenovus Breast Cancer Study (NTBCS)

981 patients < 50 years from the Breast Cancer Outcome Study of Mutation Carriers (BCOS) diagnosed between 1990 and 2000 with stage I-III breast cancer in the Netherlands.

2609 patients diagnosed between 2000 and 2008 in the UK (POSH dataset)

PREDICT significantly overestimated ACM in the POSH dataset by 12% (p = 0.00) and BCSM by 9% (p = 0.018). Non-breast cancer mortality was significantly overestimated by 57% (p < 0.001) in the POSH dataset and significantly underestimated by 19% (p = 0.039) in the NTBCS dataset.

Across all datasets, PREDICT significantly overestimated BCSM in ER+ women aged 20–29 years by 40% (p = 0.0047) and in ER+ and ER- women with tumor size ≥5 mm by 35 and 33%, respectively (p = 0.04 and p = 0.00). BCSM was significantly underestimated in ER+ with tumor size 0-9 mm by 35% (p = 0.024).

Discrimination was better for ER+ than ER- in all datasets (ER+: AUC from 0.741 in BCOS to 0.796 in ECRIC, ER-: AUC from 0.632 in BCOS to 0.726 in ECRIC).

 de Glas et al. (2016) (version 2) [53]

2012 patients ≥65 years with early breast cancer diagnosed between 1997 and 2004 in the western Netherlands.

5-year OS was underestimated in patients without comorbidity (predicted:observed OS = − 3.7%, [95% CI = − 7.2 to − 0.2], P = 0.040), and overestimated in patients with 4 or more comorbidities (predicted:observed OS 11.8%, [95% CI = 6.9–16.7], p< 0.0001).

10-year OS was overestimated in patients with 4 or more comorbidities (predicted:observed OS = 20.7%, [95% CI = 15.8–25.6]).

Overall, c-index of the predicted 5-year OS was 0.73, [95% CI = 0.70–0.75], and for 10-year OS 0.74, [95% CI = 0.72–0.76].

 Maishman et al. (2015) (version 2) [54]

3000 patients ≤40 years diagnosed in the UK between 2000 and 2008.

PREDICT provided accurate long-term (8- and 10-year) survival estimates for younger women.

Five-year estimates were less accurate, with the tool significantly overestimating survival by 5% overall, and in subgroups of patients with ER+ tumors, grade 2, tumors ≥1 cm or patients receiving a combination of hormone and chemotherapy. OS was also overestimated for patients receiving second and third generation chemotherapy. PREDICT significantly underestimated 5-year survival by 25% among patients with ER- tumors and patients receiving trastuzumab.

PREDICT significantly underestimated 10-year OS in patients with ER- tumors, grade 3, tumors > 5 cm, and in patients receiving chemotherapy alone. C-index was 0.72 vs 0.69 for ER+ vs ER- at 10 years.

 Wishart et al. (version 1) (2010) [31]

5468 patients diagnosed between 1999 and 2003 in the UK

5-year OS was significantly underestimated by 1.6% (p = 0.004) but no difference between predicted and observed survival at 8 years. C-index was 0.81 for ER+ and 0.75 for ER-.

 Wishart et al. (version 3) (2014) [55]

1726 patients diagnosed between 1989 and 1998 in Nottingham (UK).

No significant differences between predicted and observed breast cancer deaths. C-index was 0.77.

 Wong et al. (2015) [56]

1480 Chinese, Malay and Indian patients treated between 1998 and 2006 with stage I-III

No significant differences between predicted and observed breast cancer deaths but overestimated OS for patients < 40 years (5-year OS by 6.8% and 10-year OS by 17.2%.). 5-year OS was underestimated for women without nodal involvement by 3.2%, for ER- by 6% and for Her2+ by 6.6%. 10-year OS was overestimated for Her2-negative by 9.9%.

C-index for 5-year OS 0.78 [95% CI: 0.74–0.81] and for 10-year OS 0.73 [95% CI: 0.68–0.78].

 Engelhardt et al. (2017) [48] Adjuvant!Online (version 8.0) PREDICT (version 1.3)

2710 women < 50 years from the Netherlands with unilateral breast cancer diagnosed between 1990 and 2000

ACM: Adjuvant!Online significantly underestimated ACM by − 2% [95% CI: − 3.7 to − 0.3; p = 0.02], PREDICT tends to underestimate ACM but not significantly (only significant for women ≤35 years, good prognosis (stage 1, T1, N0)). PREDICT overestimated ACM for poor prognosis by 2.6–9.4% (stage 3, T3, N1) and 2.2% for Her 2 positive patients. C-index PREDICT = 0.70, Adjuvant!Online =0.69.

BCSM: PREDICT significantly overestimated BCSM by 3.2% (95% CI: 0.8 to 5.6; p = 0.007). With Adjuvant!Online, there is no difference between predicted and observed BCSM but it significantly overestimated BCSM in various subgroups. C-index PREDICT = 0.73, Adjuvant!Online =0.72.

For both tools, calibration curves were accurate for women with predicted 20–40% mortality probability.

 Hearne et al. (2015) [49] Adjuvant!Online (and NPI)

92 women < 40 years treated in the UK between 1998 and 2007.

No significant difference between predicted and observed survival.

 Quintyne et al. (2013) [50] Adjuvant!Online (and NPI)

77 women with early breast cancer treated in Ireland in 2002.

Predicted 10-year OS was 72.9%, while observed OS was 81.8%. NPI

prognostic groups did not separate as well (P > .05), and the Adjuvant!Online groups separated better (P < .05).

 Plakhins et al. (2013) [51] Adjuvant!Online (version 8.0) and PREDICT

71 Latvian BRCA-1 patients treated between 2000 and 2008.

Both tools significantly underestimated OS. Adjuvant!Online underestimated 10-year OS (predicted:observed − 9.75%; [95% CI = − 13.93 to − 5.57]; p < 0.0001) and BCSS (predicted:observed − 8.64%; [95% CI = − 12.88 to − 4.39]; p < 0.0001). PREDICT underestimated 5-year OS (predicted:observed − 6.67% [95% CI = − 10.14 to − 3.19]; p < 0.0001) and 10-year OS (predicted:observed − 10.21%; [95% CI = − 14.93 to − 5.47]; p < 0.0001).

 Wishart et al. (2011) [52]

PREDICT (version 1)

Adjuvant!Online (version 8.0)

3140 patients with stage I or II tumors diagnosed between 1989 and 1993 in British Columbia (Canada).

No significant differences in 10-year OS or BCSS.

C-index for PREDICT and Adjuvant!Online for OS was 0.709 vs 0.712 and for BCSS 0.723 vs. 0.727 respectively.

 Wishart et al. (2012) [30]

PREDICT (version 2, “PREDICT+”)

Adjuvant!Online (version 8.0)

1653 patients with stage I or II tumors and known Her2 status diagnosed between 1989 and 1993 in British Columbia (Canada).

No statistically significant differences in 10-year OS for Adjuvant!Online, but OS was underestimated for PREDICT by 8.8% (p = 0.04) and PREDICT+ by 8.4% (p = 0.05). In women aged 20–35 years, all models underestimated OS by 32%. 10-year BCSS was underestimated by Adjuvant!Online by 14% (p = 0.01) but no significantly differences for PREDICT or PREDICT+. In women aged 20–35 years, all models underpredicted survival by 32%.

In HER2-positive women, there were no significant differences in predicted and observed OS. There were no significant differences in breast cancer specific deaths with PREDICT and PREDICT+. Adjuvant!Online underestimated survival by 29% (53 vs 75, p = 0.01).

Across all risk categories, calibration was good for Adjuvant!Online (goodness-of-fit, p = 0.51), and reasonable for PREDICT+ (goodness-of-fit, p = 0.042) and Predict (goodness-of-fit, P = 0.032)

 Chen et al. (2009) [58]

362,491 patients from SEER dataset

Predicted and observed survival agreed within 1% for patients with a chance of death up to 48%, which comprised 97% of the study population. For the remaining 3%, predicted and observed survival rates agreed within 7%.

 Michaelsson (2011) [32]

293,576 patients from SEER dataset diagnosed after 1987

24,771 patients diagnosed at the Massachusetts General and Brigham and Women’s Hospitals (Partners dataset) (1968–2007).

Predicted and observed survival agreed within 2% for the 97% of patients with up to a 48% risk of death, while for the remaining 3% of patients with greater than a 48% chance of death, the expected and observed survival values for each group agreed within 7%.

Partners dataset: data not reported.