CrowdTruth 2.0: Quality Metrics for Crowdsourcing with Disagreement

18 Aug 2018  ·  Anca Dumitrache, Oana Inel, Lora Aroyo, Benjamin Timmermans, Chris Welty ·

Typically crowdsourcing-based approaches to gather annotated data use inter-annotator agreement as a measure of quality. However, in many domains, there is ambiguity in the data, as well as a multitude of perspectives of the information examples. In this paper, we present ongoing work into the CrowdTruth metrics, that capture and interpret inter-annotator disagreement in crowdsourcing. The CrowdTruth metrics model the inter-dependency between the three main components of a crowdsourcing system -- worker, input data, and annotation. The goal of the metrics is to capture the degree of ambiguity in each of these three components. The metrics are available online at .

PDF Abstract


  Add Datasets introduced or used in this paper