Search Results for author:
Found 3 papers, 0 papers with code
The High Line: Exact Risk and Learning Rate Curves of Stochastic Adaptive Learning Rate Algorithms
For noiseless targets, we further demonstrate that the AdaGrad-Norm learning rate converges to a deterministic constant inversely proportional to the average eigenvalue of the data covariance matrix, and identify a phase transition when the covariance density of eigenvalues follows a power law distribution.
Hitting the High-Dimensional Notes: An ODE for SGD learning dynamics on GLMs and multi-index models
In addition to the deterministic equivalent, we introduce an SDE with a simplified diffusion coefficient (homogenized SGD) which allows us to analyze the dynamics of general statistics of SGD iterates.
Overlap of a spherical spin glass model with microscopic external field
Previous research has noted that, at low temperature, this overlap exhibits dramatically different behavior in the presence of an external field as compared to the model with no external field.
Probability Mathematical Physics Mathematical Physics
