no code implementations • 15 Apr 2024 • Nachuan Xiao, Kuangyu Ding, Xiaoyin Hu, Kim-Chuan Toh
Preliminary numerical experiments on deep learning tasks illustrate that our proposed framework yields efficient variants of Lagrangian-based methods with convergence guarantees for nonconvex nonsmooth constrained optimization problems.
no code implementations • 18 Mar 2024 • Siyuan Zhang, Nachuan Xiao, Xin Liu
Furthermore, we establish that our proposed framework encompasses a wide range of existing efficient decentralized subgradient methods, including decentralized stochastic subgradient descent (DSGD), DSGD with gradient-tracking technique (DSGD-T), and DSGD with momentum (DSGDm).
no code implementations • 13 Oct 2023 • Kuangyu Ding, Nachuan Xiao, Kim-Chuan Toh
As a practical application of our proposed framework, we propose a novel Adam-family method named Adam with Decoupled Weight Decay (AdamD), and establish its convergence properties under mild conditions.
no code implementations • 19 Jul 2023 • Nachuan Xiao, Xiaoyin Hu, Kim-Chuan Toh
We further illustrate that our scheme yields variants of SGD-type methods, which enjoy guaranteed convergence in training nonsmooth neural networks.
no code implementations • 6 May 2023 • Nachuan Xiao, Xiaoyin Hu, Xin Liu, Kim-Chuan Toh
In this paper, we present a comprehensive study on the convergence properties of Adam-family methods for nonsmooth optimization, especially in the training of nonsmooth neural networks.
