Exploiting Strong Convexity from Data with Primal-Dual First-Order Algorithms
We consider primal-dual first-order algorithms for solving convex-concave saddle-point problems where the coupling between the primal and dual variables are bilinear. They have become powerful tools for modeling and solving machine learning problems. In particular, for empirical risk minimization of linear predictors, the coupling matrix consists of all the data. In order to obtain fast linear rates of convergence, usually we need to assume both strong convexity in the primal and strong concavity in the dual. In this talk, we show that such an assumption can be relaxed if the data matrix is full rank, which can effectively transfer the strong convexity or concavity from one side to the other. We show how to exploit strong convexity from data by setting the step sizes appropriately for the Chambolle-Pock batch algorithm, as well as for randomized primal-dual algorithms. Since the strong convexity parameter from data is hard to estimate, we propose simple heuristics for adaptively tuning the step sizes and demonstrate their effectiveness in numerical experiments.