Skip to content
Vol. I · No. 251
Mon · 8 Jun
A Daily Lexicon of Trustworthy Data
The Lexicon

006·319

inter-annotator agreement

/ˌɪntərˈænəteɪtər əˈɡriːmənt/ - n.

1 [colloq.] The metric that quietly reports how undefined the term was all along.Keep. Punchy.This is the problem.

Working definition

2. A measure of how consistently independent labelers assign the same label to the same example.

Evidence
See also
  • data labelingThe step where the team discovers it never agreed on the categories.
  • ground truthOne analyst's spreadsheet, promoted to truth because no one else volunteered.
  • label driftDefinition drift, now with a confidence interval.
  • model evaluationThe benchmark the model passes, chosen after seeing which benchmark it passes.