Handwritten Text Recognition from Crowdsourced Annotations
In this paper, we explore different ways of training a model for handwritten text recognition when multiple imperfect or noisy transcriptions are available. We consider various training configurations, such as selecting a single transcription, retaining all transcriptions, or computing an aggregated transcription from all available annotations. In addition, we evaluate the impact of quality-based data selection, where samples with low agreement are removed from the training set. Our experiments are carried out on municipal registers of the city of Belfort (France) written between 1790 and 1946. % results The results show that computing a consensus transcription or training on multiple transcriptions are good alternatives. However, selecting training samples based on the degree of agreement between annotators introduces a bias in the training data and does not improve the results. Our dataset is publicly available on Zenodo: https://zenodo.org/record/8041668.
PDF Abstract International Workshop 2023 PDFDatasets
Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Benchmark |
---|---|---|---|---|---|---|
Handwritten Text Recognition | Belfort | PyLaia (human transcriptions + random split) | CER (%) | 10.54 | # 4 | |
WER (%) | 28.11 | # 4 | ||||
Handwritten Text Recognition | Belfort | PyLaia (human transcriptions + agreement-based split) | CER (%) | 5.57 | # 3 | |
WER (%) | 19.12 | # 3 | ||||
Handwritten Text Recognition | Belfort | PyLaia (all transcriptions + agreement-based split) | CER (%) | 4.34 | # 1 | |
WER (%) | 15.14 | # 1 | ||||
Handwritten Text Recognition | Belfort | PyLaia (rover consensus + agreement-based split) | CER (%) | 4.95 | # 2 | |
WER (%) | 17.08 | # 2 |