Address Annotator Bias Part 1: Evaluation #10
Description
Pronunciation perception is very subjective. Thus annotators with different language backgrounds will naturally disagree on which IPA symbols to label a given speech sound with. In our and Menzel et al.'s experience (from email discussions), this is a large portion of annotations and is not something that can necessarily be directly resolved through discussion between the annotators, visual inspection in Praat, nor even has a "right" answer. We need to evaluate annotator consistency (like in Menzel et al.'s paper) for all our datasets.