Analysis and Comparison of Two-Level KFAC Methods for Training Deep Neural Networks

no code implementations • 31 Mar 2023 • Abdoulaye Koroko, Ani Anciaux-Sedrakian, Ibtihel Ben Gharbia, Valérie Garès, Mounir Haddou, Quang Huy Tran

As a second-order method, the Natural Gradient Descent (NGD) has the ability to accelerate training of neural networks.

Efficient Approximations of the Fisher Matrix in Neural Networks using Kronecker Product Singular Value Decomposition

no code implementations • 25 Jan 2022 • Abdoulaye Koroko, Ani Anciaux-Sedrakian, Ibtihel Ben Gharbia, Valérie Garès, Mounir Haddou, Quang Huy Tran

Several studies have shown the ability of natural gradient descent to minimize the objective function more efficiently than ordinary gradient descent based methods.