Transformers

Transformer

Introduced by Vaswani et al. in Attention Is All You Need

A Transformer is a model architecture that eschews recurrence and instead relies entirely on an attention mechanism to draw global dependencies between input and output. Before Transformers, the dominant sequence transduction models were based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The Transformer also employs an encoder and decoder, but removing recurrence in favor of attention mechanisms allows for significantly more parallelization than methods like RNNs and CNNs.

Source: Attention Is All You Need

Papers


Paper Code Results Date Stars

Tasks


Task Papers Share
Language Modelling 58 7.39%
Large Language Model 30 3.82%
Question Answering 22 2.80%
In-Context Learning 22 2.80%
Semantic Segmentation 18 2.29%
Machine Translation 17 2.17%
Retrieval 16 2.04%
Decision Making 15 1.91%
Image Classification 14 1.78%

Categories