LATTE: Low-Precision Approximate Attention with Head-wise Trainable Threshold for Efficient Transformer

no code implementations • 11 Apr 2024 • Jiing-Ping Wang, Ming-Guang Lin, An-Yeu, Wu

With the rise of Transformer models in NLP and CV domain, Multi-Head Attention has been proven to be a game-changer.