Regularizing Transformers With Deep Probabilistic Layers
Language models (LM) have grown with non-stop in the last decade, from sequence-to-sequence architectures to the state-of-the-art and utter attention-based Transformers. In this work, we demonstrate how the inclusion of deep generative models within BERT can bring more versatile models, able to impute missing/noisy words with richer text or even improve BLEU score. More precisely, we use a Gaussian Mixture Variational Autoencoder (GMVAE) as a regularizer layer and prove its effectiveness not only in Transformers but also in the most relevant encoder-decoder based LM, seq2seq with and without attention.
PDF AbstractTasks
Datasets
Fashion-MNIST
SNLI
Results from the Paper
Submit
results from this paper
to get state-of-the-art GitHub badges and help the
community compare results to other papers.
Methods
Adam •
Attention Dropout •
AutoEncoder •
BERT •
Dense Connections •
Dropout •
GELU •
GMVAE •
Layer Normalization •
Linear Layer •
Linear Warmup With Linear Decay •
LSTM •
Multi-Head Attention •
Residual Connection •
Scaled Dot-Product Attention •
Seq2Seq •
Sigmoid Activation •
Softmax •
Tanh Activation •
Weight Decay •
WordPiece
