Search Results for author:
Found 22 papers, 4 papers with code
Convex and Bilevel Optimization for Neuro-Symbolic Inference and Learning
We address a key challenge for neuro-symbolic (NeSy) systems by leveraging convex and bilevel optimization techniques to develop a general gradient-based framework for end-to-end neural and symbolic parameter learning.
Optimally Teaching a Linear Behavior Cloning Agent
The goal of the teacher is to teach a realizable target policy to the learner using minimum number of state demonstrations.
On Penalty Methods for Nonconvex Bilevel Optimization and First-Order Stochastic Approximation
When the perturbed lower-level problem uniformly satisfies the small-error proximal error-bound (EB) condition, we propose a first-order algorithm that converges to an $\epsilon$-stationary point of the penalty function, using in total $O(\epsilon^{-3})$ and $O(\epsilon^{-7})$ accesses to first-order (stochastic) gradient oracles when the oracle is deterministic and oracles are noisy, respectively.
Cut your Losses with Squentropy
We provide an extensive set of experiments on multi-class classification problems showing that the squentropy loss outperforms both the pure cross entropy and rescaled square losses in terms of the classification accuracy.
A Fully First-Order Method for Stochastic Bilevel Optimization
Specifically, we show that F2SA converges to an $\epsilon$-stationary solution of the bilevel problem after $\epsilon^{-7/2}, \epsilon^{-5/2}$, and $\epsilon^{-3/2}$ iterations (each iteration using $O(1)$ samples) when stochastic noises are in both level objectives, only in the upper-level objective, and not present (deterministic settings), respectively.
BOME! Bilevel Optimization Made Easy: A Simple First-Order Approach
Bilevel optimization (BO) is useful for solving a variety of important machine learning problems including but not limited to hyperparameter optimization, meta-learning, continual learning, and reinforcement learning.
On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regime
Finding the optimal configuration of parameters in ResNet is a nonconvex minimization problem, but first-order methods nevertheless find the global optimum in the overparameterized regime.
Overparameterization of deep ResNet: zero loss and mean-field analysis
Finding parameters in a deep neural network (NN) that fit training data is a nonconvex optimization problem, but a basic first-order optimization method (gradient descent) finds a global optimizer with perfect fit (zero-loss) in many practical situations.
Stochastic Learning for Sparse Discrete Markov Random Fields with Controlled Gradient Approximation Error
We study the $L_1$-regularized maximum likelihood estimator/estimation (MLE) problem for discrete Markov random fields (MRFs), where efficient and scalable learning requires both sparse regularization and approximate inference.
Computing Estimators of Dantzig Selector type via Column and Constraint Generation
We consider a class of linear-programming based estimators in reconstructing a sparse signal from linear measurements.
Convergence and Margin of Adversarial Training on Separable Data
Our results are derived by showing that adversarial training with gradient updates minimizes a robust version of the empirical risk at a $\mathcal{O}(\ln(t)^2/t)$ rate, despite non-smoothness.
Bilinear Bandits with Low-rank Structure
We introduce the bilinear bandit problem with low-rank structure in which an action takes the form of a pair of arms from two different entity types, and the reward is a bilinear function of the known feature vectors of the arms.
ATOMO: Communication-efficient Learning via Atomic Sparsification
We present ATOMO, a general framework for atomic sparsification of stochastic gradients.
Dissipativity Theory for Accelerating Stochastic Variance Reduction: A Unified Analysis of SVRG and Katyusha Using Semidefinite Programs
Our combination of perspectives leads to a better understanding of accelerated variance-reduced stochastic methods for finite-sum problems.
Blended Conditional Gradients: the unconditioning of conditional gradients
We present a blended conditional gradient approach for minimizing a smooth convex function over a polytope P, combining the Frank--Wolfe algorithm (also called conditional gradient) with gradient-based steps, different from away steps and pairwise steps, but still achieving linear convergence for strongly convex functions, along with good practical performance.
k-Support and Ordered Weighted Sparsity for Overlapping Groups: Hardness and Algorithms
We study the norms obtained from extending the k-support norm and OWL norms to the setting in which there are overlapping groups.
Online Learning for Changing Environments using Coin Betting
A key challenge in online learning is that classical algorithms can be slow to adapt to changing environments.
Improved Strongly Adaptive Online Learning using Coin Betting
This paper describes a new parameter-free online learning algorithm for changing environments.
Beyond the Birkhoff Polytope: Convex Relaxations for Vector Permutation Problems
Using a recent construction of Goemans (2010), we show that when optimizing over the convex hull of the permutation vectors (the permutahedron), we can reduce the number of variables and constraints to $\Theta(n \log n)$ in theory and $\Theta(n \log^2 n)$ in practice.
Forward - Backward Greedy Algorithms for Atomic Norm Regularization
CoGEnT combines a greedy selection scheme based on the conditional gradient approach with a backward (or "truncation") step that exploits the quadratic nature of the objective to reduce the basis size.
An Approximate, Efficient LP Solver for LP Rounding
Many problems in machine learning can be solved by rounding the solution of an appropriate linear program.
Hogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
Stochastic Gradient Descent (SGD) is a popular algorithm that can achieve state-of-the-art performance on a variety of machine learning tasks.
