Topic
annotation aggregation default
Latest news
- 6 OctReference choice can shift multilingual benchmark F1 by up to 7.55 points
The paper measures how changing the reference annotation moves benchmark scores while keeping system outputs fixed, finding shifts up to 7.55 F1 in one language and average changes of 4.95 and 2.00 F1 for two outputs. The authors trace a major cause to an undocumented aggregation default that left one annotator's labels in the shipped gold, creating a partial copy of a system output and altering comparisons between systems.
Research · 1 source
Articles
No articles about annotation aggregation default yet.