Research · Updated 6 Oct, 09:30 am IST
Reference choice can shift multilingual benchmark F1 by up to 7.55 points
Why it matters for readers: This shows evaluation scores can change because of human annotation quirks, so numbers alone may not tell the whole story.
- Exchanging the reference annotation (keeping outputs fixed) changed one output's F1 by 4.95 points and the other by 2.00 points with reported 95% CIs.1
- The worst-performing language experienced a shift of 7.55 F1 when the reference was changed.1
- The interaction between reference choice and system changed pairwise comparison margins by +2.95 F1, a result that survived multiple-testing correction and flipped one language's margin.1
- A root cause was an undocumented aggregation default that often retained one annotator's labels when adjudication did not occur, making the shipped gold partially copy a system output.1
- The paper proposes computing a reference-sensitivity band from retained annotation records and argues this quantity should accompany reported scores.1
Get a brief like this every morning
Uzha reads hundreds of sources and gives you the stories that matter for your work, with every source linked. Free.
Get started