Normalization

Edit

General • 38 methods

Normalization layers in deep learning are used to make optimization easier by smoothing the loss surface of the network. Below you will find a continuously updating list of normalization methods.

Methods

Add a Method

Method	Year	Papers
Layer Normalization Layer Normalization	2016	17101
Batch Normalization Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift	2015	5593
Instance Normalization Instance Normalization: The Missing Ingredient for Fast Stylization	2016	479
Local Response Normalization ImageNet Classification with Deep Convolutional Neural Networks	2012	402
Adaptive Instance Normalization Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization	2017	374
Spectral Normalization Spectral Normalization for Generative Adversarial Networks	2018	208
Weight Demodulation Analyzing and Improving the Image Quality of StyleGAN	2019	169
Conditional Batch Normalization Modulating early visual processing by language	2017	142
Weight Normalization Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks	2016	77
Activation Normalization Glow: Generative Flow with Invertible 1x1 Convolutions	2018	53
Group Normalization Group Normalization	2018	52
SPADE Semantic Image Synthesis with Spatially-Adaptive Normalization	2019	32
SRN Stable Rank Normalization for Improved Generalization in Neural Networks and GANs	2019	12
MPN Is Second-order Information Helpful for Large-scale Visual Recognition?	2017	11
Weight Standardization Micro-Batch Training with Batch-Channel Normalization and Weight Standardization	2019	10
LayerScale Going deeper with Image Transformers	2021	10
Switchable Normalization Differentiable Learning-to-Normalize via Switchable Normalization	2018	9
Gradient Normalization Gradient Normalization for Generative Adversarial Networks	2021	8
Local Contrast Normalization	2009	7
SyncBN Context Encoding for Semantic Segmentation	2018	5
ReZero ReZero is All You Need: Fast Convergence at Large Depth	2020	5
Decorrelated Batch Normalization Decorrelated Batch Normalization	2018	4
RMSNorm Root Mean Square Layer Normalization	2019	4
Attentive Normalization Attentive Normalization	2019	3
Conditional Instance Normalization A Learned Representation For Artistic Style	2016	3
Online Normalization Online Normalization for Training Neural Networks	2019	3
Cosine Normalization Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks	2017	2
Mixture Normalization Training Faster by Separating Modes of Variation in Batch-normalized Models	2018	2
Filter Response Normalization Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks	2019	2
InPlace-ABN In-Place Activated BatchNorm for Memory-Optimized Training of DNNs	2017	2
BatchChannel Normalization Rethinking Normalization and Elimination Singularity in Neural Networks	2019	1
Instance-Level Meta Normalization Instance-Level Meta Normalization	2019	1
Mode Normalization Mode Normalization	2018	1
Sparse Switchable Normalization SSN: Learning Sparse Switchable Normalization via SparsestMax	2019	1
Virtual Batch Normalization Improved Techniques for Training GANs	2016	1
SaBN Sandwich Batch Normalization: A Drop-In Replacement for Feature Distribution Heterogeneity	2021	1
EvoNorms Evolving Normalization-Activation Layers	2020	1
CInC Flow CInC Flow: Characterizable Invertible 3x3 Convolution	2021	1

Normalization Edit

Methods Add a Method

Normalization

Edit

Methods

Add a Method