Gradient Descent for Deep Matrix Factorization: Dynamics and Implicit Bias towards Low Rank

no code implementations • 27 Nov 2020 • Hung-Hsu Chou, Carsten Gieshoff, Johannes Maly, Holger Rauhut

This suggests that deep learning prefers trajectories whose complexity (measuredin terms of effective rank) is monotonically increasing, which we believe is a fundamental concept for thetheoretical understanding of deep learning.

Denoising