TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK
Toxic Comment Classification	Civil Comments	RoBERTa Focal Loss	AUROC	0.9818	# 1
Toxic Comment Classification	Civil Comments	RoBERTa Focal Loss	Macro F1	0.4648	# 2
Toxic Comment Classification	Civil Comments	RoBERTa Focal Loss	Micro F1	0.5524	# 3
Toxic Comment Classification	Civil Comments	RoBERTa Focal Loss	Precision	0.4017	# 3
Toxic Comment Classification	Civil Comments	RoBERTa Focal Loss	Recall	0.8839	# 8
Toxic Comment Classification	Civil Comments	RoBERTa Focal Loss	GMB Subgroup	0.8807	# 1
Toxic Comment Classification	Civil Comments	RoBERTa Focal Loss	GMB BPSN	0.901	# 1
Toxic Comment Classification	Civil Comments	RoBERTa Focal Loss	GMB BNSP	0.9581	# 6
Toxic Comment Classification	Civil Comments	AlBERT	AUROC	0.979	# 5
Toxic Comment Classification	Civil Comments	AlBERT	Macro F1	0.3541	# 9
Toxic Comment Classification	Civil Comments	AlBERT	Micro F1	0.4845	# 9
Toxic Comment Classification	Civil Comments	AlBERT	Precision	0.3247	# 10
Toxic Comment Classification	Civil Comments	AlBERT	Recall	0.9104	# 5
Toxic Comment Classification	Civil Comments	AlBERT	GMB Subgroup	0.8734	# 6
Toxic Comment Classification	Civil Comments	AlBERT	GMB BPSN	0.8982	# 2
Toxic Comment Classification	Civil Comments	AlBERT	GMB BNSP	0.9499	# 7
Toxic Comment Classification	Civil Comments	BiGRU	GMB BPSN	0.8616	# 9
Toxic Comment Classification	Civil Comments	Unfreeze Glove ResNet 56	AUROC	0.9639	# 9
Toxic Comment Classification	Civil Comments	Unfreeze Glove ResNet 56	Macro F1	0.3778	# 6
Toxic Comment Classification	Civil Comments	Unfreeze Glove ResNet 56	Recall	0.8707	# 9
Toxic Comment Classification	Civil Comments	Unfreeze Glove ResNet 56	GMB Subgroup	0.8487	# 9
Toxic Comment Classification	Civil Comments	Unfreeze Glove ResNet 56	GMB BPSN	0.8445	# 11
Toxic Comment Classification	Civil Comments	DistilBERT	AUROC	0.9804	# 3
Toxic Comment Classification	Civil Comments	DistilBERT	Macro F1	0.3879	# 5
Toxic Comment Classification	Civil Comments	DistilBERT	Micro F1	0.5115	# 5
Toxic Comment Classification	Civil Comments	DistilBERT	Precision	0.3572	# 5
Toxic Comment Classification	Civil Comments	DistilBERT	Recall	0.9001	# 6
Toxic Comment Classification	Civil Comments	DistilBERT	GMB Subgroup	0.8762	# 4
Toxic Comment Classification	Civil Comments	DistilBERT	GMB BPSN	0.874	# 8
Toxic Comment Classification	Civil Comments	DistilBERT	GMB BNSP	0.9644	# 1
Toxic Comment Classification	Civil Comments	Freeze Glove ResNet 44	Macro F1	0.4189	# 4
Toxic Comment Classification	Civil Comments	Freeze Glove ResNet 44	Micro F1	0.5591	# 2
Toxic Comment Classification	Civil Comments	Freeze Glove ResNet 44	Precision	0.4631	# 2
Toxic Comment Classification	Civil Comments	Freeze Glove ResNet 44	Recall	0.7053	# 12
Toxic Comment Classification	Civil Comments	Freeze Glove ResNet 44	GMB Subgroup	0.8219	# 11
Toxic Comment Classification	Civil Comments	Freeze Glove ResNet 44	GMB BPSN	0.7876	# 13
Toxic Comment Classification	Civil Comments	BiLSTM	Micro F1	0.5115	# 5
Toxic Comment Classification	Civil Comments	BiLSTM	Precision	0.3572	# 5
Toxic Comment Classification	Civil Comments	BiLSTM	GMB Subgroup	0.8636	# 8
Toxic Comment Classification	Civil Comments	BERTweet	AUROC	0.979	# 5
Toxic Comment Classification	Civil Comments	BERTweet	Macro F1	0.3612	# 8
Toxic Comment Classification	Civil Comments	BERTweet	Micro F1	0.4928	# 7
Toxic Comment Classification	Civil Comments	BERTweet	Precision	0.3363	# 8
Toxic Comment Classification	Civil Comments	BERTweet	Recall	0.9216	# 3
Toxic Comment Classification	Civil Comments	BERTweet	GMB Subgroup	0.878	# 3
Toxic Comment Classification	Civil Comments	BERTweet	GMB BPSN	0.8945	# 3
Toxic Comment Classification	Civil Comments	BERTweet	GMB BNSP	0.9603	# 3
Toxic Comment Classification	Civil Comments	XLNet	Macro F1	0.3336	# 11
Toxic Comment Classification	Civil Comments	XLNet	Micro F1	0.4586	# 12
Toxic Comment Classification	Civil Comments	XLNet	Precision	0.3045	# 12
Toxic Comment Classification	Civil Comments	XLNet	Recall	0.9254	# 1
Toxic Comment Classification	Civil Comments	XLNet	GMB Subgroup	0.8689	# 7
Toxic Comment Classification	Civil Comments	XLNet	GMB BPSN	0.8834	# 7
Toxic Comment Classification	Civil Comments	XLNet	GMB BNSP	0.9597	# 4
Toxic Comment Classification	Civil Comments	HateBERT	AUROC	0.9791	# 4
Toxic Comment Classification	Civil Comments	HateBERT	Macro F1	0.3679	# 7
Toxic Comment Classification	Civil Comments	HateBERT	Micro F1	0.4844	# 10
Toxic Comment Classification	Civil Comments	HateBERT	Precision	0.3297	# 9
Toxic Comment Classification	Civil Comments	HateBERT	Recall	0.9165	# 4
Toxic Comment Classification	Civil Comments	HateBERT	GMB Subgroup	0.8744	# 5
Toxic Comment Classification	Civil Comments	HateBERT	GMB BPSN	0.8915	# 4
Toxic Comment Classification	Civil Comments	HateBERT	GMB BNSP	0.9589	# 5
Toxic Comment Classification	Civil Comments	RoBERTa BCE	AUROC	0.9813	# 2
Toxic Comment Classification	Civil Comments	RoBERTa BCE	Macro F1	0.4749	# 1
Toxic Comment Classification	Civil Comments	RoBERTa BCE	Micro F1	0.5359	# 4
Toxic Comment Classification	Civil Comments	RoBERTa BCE	Precision	0.3836	# 4
Toxic Comment Classification	Civil Comments	RoBERTa BCE	Recall	0.8891	# 7
Toxic Comment Classification	Civil Comments	RoBERTa BCE	GMB Subgroup	0.88	# 2
Toxic Comment Classification	Civil Comments	RoBERTa BCE	GMB BPSN	0.8901	# 5
Toxic Comment Classification	Civil Comments	RoBERTa BCE	GMB BNSP	0.9616	# 2
Toxic Comment Classification	Civil Comments	XLM RoBERTa	Micro F1	0.468	# 11
Toxic Comment Classification	Civil Comments	XLM RoBERTa	Precision	0.3135	# 11
Toxic Comment Classification	Civil Comments	XLM RoBERTa	Recall	0.923	# 2
Toxic Comment Classification	Civil Comments	XLM RoBERTa	GMB BPSN	0.8859	# 6
Toxic Comment Classification	Civil Comments	Unfreeze Glove ResNet 44	AUROC	0.966	# 8
Toxic Comment Classification	Civil Comments	Unfreeze Glove ResNet 44	Macro F1	0.4648	# 2
Toxic Comment Classification	Civil Comments	Unfreeze Glove ResNet 44	Micro F1	0.5958	# 1
Toxic Comment Classification	Civil Comments	Unfreeze Glove ResNet 44	Precision	0.4835	# 1
Toxic Comment Classification	Civil Comments	Unfreeze Glove ResNet 44	Recall	0.7759	# 11
Toxic Comment Classification	Civil Comments	Unfreeze Glove ResNet 44	GMB Subgroup	0.8421	# 10
Toxic Comment Classification	Civil Comments	Unfreeze Glove ResNet 44	GMB BPSN	0.8493	# 10
Toxic Comment Classification	Civil Comments	Compact Convolutional Transformer (CCT)	AUROC	0.9526	# 10
Toxic Comment Classification	Civil Comments	Compact Convolutional Transformer (CCT)	Macro F1	0.3428	# 10
Toxic Comment Classification	Civil Comments	Compact Convolutional Transformer (CCT)	Micro F1	0.4874	# 8
Toxic Comment Classification	Civil Comments	Compact Convolutional Transformer (CCT)	Precision	0.3507	# 7
Toxic Comment Classification	Civil Comments	Compact Convolutional Transformer (CCT)	Recall	0.7983	# 10
Toxic Comment Classification	Civil Comments	Compact Convolutional Transformer (CCT)	GMB Subgroup	0.8133	# 12
Toxic Comment Classification	Civil Comments	Compact Convolutional Transformer (CCT)	GMB BPSN	0.8307	# 12
Toxic Comment Classification	Civil Comments	Compact Convolutional Transformer (CCT)	GMB BNSP	0.9447	# 8

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/a-benchmark-for-toxic-comment-classification/toxic-comment-classification-on-civil)](https://paperswithcode.com/sota/toxic-comment-classification-on-civil?p=a-benchmark-for-toxic-comment-classification)`

A benchmark for toxic comment classification on Civil Comments dataset

26 Jan 2023 · Corentin Duchene, Henri Jamet, Pierre Guillaume, Reda Dehak ·

Toxic comment detection on social media has proven to be essential for content moderation. This paper compares a wide set of different models on a highly skewed multi-label hate speech dataset. We consider inference time and several metrics to measure performance and bias in our comparison. We show that all BERTs have similar performance regardless of the size, optimizations or language used to pre-train the models. RNNs are much faster at inference than any of the BERT. BiLSTM remains a good compromise between performance and inference time. RoBERTa with Focal Loss offers the best performance on biases and AUROC. However, DistilBERT combines both good AUROC and a low inference time. All models are affected by the bias of associating identities. BERT, RNN, and XLNet are less sensitive than the CNN and Compact Convolutional Transformers.

PDF Abstract

Code

Add Remove Mark official

Nigiva/hatespeech-detection-models official

Tasks

Add Remove

Toxic Comment Classification

Datasets

Civil Comments

Results from the Paper

Edit

Ranked #1 on Toxic Comment Classification on Civil Comments

Get a GitHub badge

Task	Dataset	Model	Metric Name	Metric Value	Global Rank	Benchmark
Toxic Comment Classification	Civil Comments	RoBERTa Focal Loss	AUROC	0.9818	# 1	Compare
			Macro F1	0.4648	# 2	Compare
			Micro F1	0.5524	# 3	Compare
			Precision	0.4017	# 3	Compare
			Recall	0.8839	# 8	Compare
			GMB Subgroup	0.8807	# 1	Compare
			GMB BPSN	0.901	# 1	Compare
			GMB BNSP	0.9581	# 6	Compare
Toxic Comment Classification	Civil Comments	AlBERT	AUROC	0.979	# 5	Compare
			Macro F1	0.3541	# 9	Compare
			Micro F1	0.4845	# 9	Compare
			Precision	0.3247	# 10	Compare
			Recall	0.9104	# 5	Compare
			GMB Subgroup	0.8734	# 6	Compare
			GMB BPSN	0.8982	# 2	Compare
			GMB BNSP	0.9499	# 7	Compare
Toxic Comment Classification	Civil Comments	BiGRU	GMB BPSN	0.8616	# 9	Compare
Toxic Comment Classification	Civil Comments	Unfreeze Glove ResNet 56	AUROC	0.9639	# 9	Compare
			Macro F1	0.3778	# 6	Compare
			Recall	0.8707	# 9	Compare
			GMB Subgroup	0.8487	# 9	Compare
			GMB BPSN	0.8445	# 11	Compare
Toxic Comment Classification	Civil Comments	DistilBERT	AUROC	0.9804	# 3	Compare
			Macro F1	0.3879	# 5	Compare
			Micro F1	0.5115	# 5	Compare
			Precision	0.3572	# 5	Compare
			Recall	0.9001	# 6	Compare
			GMB Subgroup	0.8762	# 4	Compare
			GMB BPSN	0.874	# 8	Compare
			GMB BNSP	0.9644	# 1	Compare
Toxic Comment Classification	Civil Comments	Freeze Glove ResNet 44	Macro F1	0.4189	# 4	Compare
			Micro F1	0.5591	# 2	Compare
			Precision	0.4631	# 2	Compare
			Recall	0.7053	# 12	Compare
			GMB Subgroup	0.8219	# 11	Compare
			GMB BPSN	0.7876	# 13	Compare
Toxic Comment Classification	Civil Comments	BiLSTM	Micro F1	0.5115	# 5	Compare
			Precision	0.3572	# 5	Compare
			GMB Subgroup	0.8636	# 8	Compare
Toxic Comment Classification	Civil Comments	BERTweet	AUROC	0.979	# 5	Compare
			Macro F1	0.3612	# 8	Compare
			Micro F1	0.4928	# 7	Compare
			Precision	0.3363	# 8	Compare
			Recall	0.9216	# 3	Compare
			GMB Subgroup	0.878	# 3	Compare
			GMB BPSN	0.8945	# 3	Compare
			GMB BNSP	0.9603	# 3	Compare
Toxic Comment Classification	Civil Comments	XLNet	Macro F1	0.3336	# 11	Compare
			Micro F1	0.4586	# 12	Compare
			Precision	0.3045	# 12	Compare
			Recall	0.9254	# 1	Compare
			GMB Subgroup	0.8689	# 7	Compare
			GMB BPSN	0.8834	# 7	Compare
			GMB BNSP	0.9597	# 4	Compare
Toxic Comment Classification	Civil Comments	HateBERT	AUROC	0.9791	# 4	Compare
			Macro F1	0.3679	# 7	Compare
			Micro F1	0.4844	# 10	Compare
			Precision	0.3297	# 9	Compare
			Recall	0.9165	# 4	Compare
			GMB Subgroup	0.8744	# 5	Compare
			GMB BPSN	0.8915	# 4	Compare
			GMB BNSP	0.9589	# 5	Compare
Toxic Comment Classification	Civil Comments	RoBERTa BCE	AUROC	0.9813	# 2	Compare
			Macro F1	0.4749	# 1	Compare
			Micro F1	0.5359	# 4	Compare
			Precision	0.3836	# 4	Compare
			Recall	0.8891	# 7	Compare
			GMB Subgroup	0.88	# 2	Compare
			GMB BPSN	0.8901	# 5	Compare
			GMB BNSP	0.9616	# 2	Compare
Toxic Comment Classification	Civil Comments	XLM RoBERTa	Micro F1	0.468	# 11	Compare
			Precision	0.3135	# 11	Compare
			Recall	0.923	# 2	Compare
			GMB BPSN	0.8859	# 6	Compare
Toxic Comment Classification	Civil Comments	Unfreeze Glove ResNet 44	AUROC	0.966	# 8	Compare
			Macro F1	0.4648	# 2	Compare
			Micro F1	0.5958	# 1	Compare
			Precision	0.4835	# 1	Compare
			Recall	0.7759	# 11	Compare
			GMB Subgroup	0.8421	# 10	Compare
			GMB BPSN	0.8493	# 10	Compare
Toxic Comment Classification	Civil Comments	Compact Convolutional Transformer (CCT)	AUROC	0.9526	# 10	Compare
			Macro F1	0.3428	# 10	Compare
			Micro F1	0.4874	# 8	Compare
			Precision	0.3507	# 7	Compare
			Recall	0.7983	# 10	Compare
			GMB Subgroup	0.8133	# 12	Compare
			GMB BPSN	0.8307	# 12	Compare
			GMB BNSP	0.9447	# 8	Compare

Methods

Add Remove

Adam • Attention Dropout • BERT • BiLSTM • BPE • Dense Connections • DistilBERT • Dropout • Focal Loss • GELU • Layer Normalization • Linear Layer • Linear Warmup With Linear Decay • LSTM • Multi-Head Attention • Residual Connection • RoBERTa • Scaled Dot-Product Attention • SentencePiece • Sigmoid Activation • Softmax • Tanh Activation • Weight Decay • WordPiece • XLNet

Edit Social Preview

A benchmark for toxic comment classification on Civil Comments dataset

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods Edit Add Remove

Code

Add Remove Mark official

Tasks

Add Remove

Datasets

Results from the Paper

Edit

Methods

Add Remove