Distributed Least Squares in Small Space via Sketching and Bias Reduction

no code implementations • 8 May 2024 • Sachin Garg, Kevin Tan, Michał Dereziński

Matrix sketching is a powerful tool for reducing the size of large data matrices.

Second-order Information Promotes Mini-Batch Robustness in Variance-Reduced Gradients

no code implementations • 23 Apr 2024 • Sachin Garg, Albert S. Berahas, Michał Dereziński

We show that, for finite-sum minimization problems, incorporating partial second-order information of the objective function can dramatically improve the robustness to mini-batch size of variance-reduced stochastic gradient methods, making them more scalable while retaining their benefits over traditional Newton-type approaches.