FarsTail: A Persian Natural Language Inference Dataset

Natural language inference (NLI) is known as one of the central tasks in natural language processing (NLP) which encapsulates many fundamental aspects of language understanding. With the considerable achievements of data-hungry deep learning methods in NLP tasks, a great amount of effort has been devoted to develop more diverse datasets for different languages. In this paper, we present a new dataset for the NLI task in the Persian language, also known as Farsi, which is one of the dominant languages in the Middle East. This dataset, named FarsTail, includes 10,367 samples which are provided in both the Persian language as well as the indexed format to be useful for non-Persian researchers. The samples are generated from 3,539 multiple-choice questions with the least amount of annotator interventions in a way similar to the SciTail dataset. A carefully designed multi-step process is adopted to ensure the quality of the dataset. We also present the results of traditional and state-of-the-art methods on FarsTail including different embedding methods such as word2vec, fastText, ELMo, BERT, and LASER, as well as different modeling approaches such as DecompAtt, ESIM, HBMP, and ULMFiT to provide a solid baseline for the future research. The best obtained test accuracy is 83.38% which shows that there is a big room for improving the current methods to be useful for real-world NLP applications in different languages. We also investigate the extent to which the models exploit superficial clues, also known as dataset biases, in FarsTail, and partition the test set into easy and hard subsets according to the success of biased models. The dataset is available at https://github.com/dml-qom/FarsTail.

PDF Abstract

Datasets


Introduced in the Paper:

FarsTail

Used in the Paper:

MultiNLI SNLI ParsiNLU
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Natural Language Inference FarsTail mBERT % Test Accuracy 83.38 # 1
Natural Language Inference FarsTail Translate-Target + fastText % Test Accuracy 70.46 # 8
Natural Language Inference FarsTail Decomposable Attention Model + word2vec % Test Accuracy 66.62 # 9
Natural Language Inference FarsTail Translate-Source + fastText % Test Accuracy 78.13 # 3
Natural Language Inference FarsTail ULMFiT % Test Accuracy 72.44 # 6
Natural Language Inference FarsTail HBMP + word2vec % Test Accuracy 66.04 # 10
Natural Language Inference FarsTail ESIM + fastText % Test Accuracy 71.16 # 7
Natural Language Inference FarsTail ParsBERT % Test Accuracy 82.99 # 2
Natural Language Inference FarsTail LSTM + BERT (concat) % Test Accuracy 75.83 # 4
Natural Language Inference FarsTail ESIM + BERT (FarsTail, MultiNLI) % Test Accuracy 74.62 # 5

Methods