Stochastic Modified Flows for Riemannian Stochastic Gradient Descent

Benjamin Gess; Sebastian Kassing; Nimit Rana

doi:10.48550/arXiv.2402.03467

Stochastic Modified Flows for Riemannian Stochastic Gradient Descent

Benjamin Gess, Sebastian Kassing, Nimit Rana

Mathematics

Research output: Contribution to journal › Article › peer-review

Abstract

We give quantitative estimates for the rate of convergence of Riemannian stochastic gradient descent (RSGD) to Riemannian gradient flow and to a diffusion process, the so-called Riemannian stochastic modified flow (RSMF). Using tools from stochastic differential geometry, we show that, in the small learning rate regime, RSGD can be approximated by the solution to the RSMF driven by an infinite-dimensional Wiener process. The RSMF accounts for the random fluctuations of RSGD and, thereby, increases the order of approximation compared to the deterministic Riemannian gradient flow. The RSGD is built using the concept of a retraction map, that is, a cost-efficient approximation of the exponential map, and we prove quantitative bounds for the weak error of the diffusion approximation under assumptions on the retraction map, the geometry of the manifold, and the random estimators of the gradient.

Original language	English
Pages (from-to)	3288-3314
Journal	SIAM Journal on Control and Optimization
Volume	62
Issue number	6
DOIs	https://doi.org/10.48550/arXiv.2402.03467 https://doi.org/10.1137/24M163863X
Publication status	Published - 13 Dec 2024

Bibliographical note

This is an author-produced version of the published paper. Uploaded in accordance with the University’s Research Publications and Open Access policy.

Access to Document

Stochastic modified flows for Riemannian stochastic gradient descent-arXiv2402.03467v2
This is an author-produced version of the published paper. Uploaded in accordance with the University’s Research Publications and Open Access policy.
Accepted author manuscript, 357 KBLicence: CC BY

Cite this

@article{7dd6bff6755c471b9742aced02039010,

title = "Stochastic Modified Flows for Riemannian Stochastic Gradient Descent",

abstract = "We give quantitative estimates for the rate of convergence of Riemannian stochastic gradient descent (RSGD) to Riemannian gradient flow and to a diffusion process, the so-called Riemannian stochastic modified flow (RSMF). Using tools from stochastic differential geometry, we show that, in the small learning rate regime, RSGD can be approximated by the solution to the RSMF driven by an infinite-dimensional Wiener process. The RSMF accounts for the random fluctuations of RSGD and, thereby, increases the order of approximation compared to the deterministic Riemannian gradient flow. The RSGD is built using the concept of a retraction map, that is, a cost-efficient approximation of the exponential map, and we prove quantitative bounds for the weak error of the diffusion approximation under assumptions on the retraction map, the geometry of the manifold, and the random estimators of the gradient.",

author = "Benjamin Gess and Sebastian Kassing and Nimit Rana",

note = "This is an author-produced version of the published paper. Uploaded in accordance with the University{\textquoteright}s Research Publications and Open Access policy. ",

year = "2024",

month = dec,

day = "13",

doi = "10.48550/arXiv.2402.03467",

language = "English",

volume = "62",

pages = "3288--3314",

journal = "SIAM Journal on Control and Optimization",

issn = "0363-0129",

publisher = "Society for Industrial and Applied Mathematics Publications",

number = "6",

}

TY - JOUR

T1 - Stochastic Modified Flows for Riemannian Stochastic Gradient Descent

AU - Gess, Benjamin

AU - Kassing, Sebastian

AU - Rana, Nimit

N1 - This is an author-produced version of the published paper. Uploaded in accordance with the University’s Research Publications and Open Access policy.

PY - 2024/12/13

Y1 - 2024/12/13

N2 - We give quantitative estimates for the rate of convergence of Riemannian stochastic gradient descent (RSGD) to Riemannian gradient flow and to a diffusion process, the so-called Riemannian stochastic modified flow (RSMF). Using tools from stochastic differential geometry, we show that, in the small learning rate regime, RSGD can be approximated by the solution to the RSMF driven by an infinite-dimensional Wiener process. The RSMF accounts for the random fluctuations of RSGD and, thereby, increases the order of approximation compared to the deterministic Riemannian gradient flow. The RSGD is built using the concept of a retraction map, that is, a cost-efficient approximation of the exponential map, and we prove quantitative bounds for the weak error of the diffusion approximation under assumptions on the retraction map, the geometry of the manifold, and the random estimators of the gradient.

AB - We give quantitative estimates for the rate of convergence of Riemannian stochastic gradient descent (RSGD) to Riemannian gradient flow and to a diffusion process, the so-called Riemannian stochastic modified flow (RSMF). Using tools from stochastic differential geometry, we show that, in the small learning rate regime, RSGD can be approximated by the solution to the RSMF driven by an infinite-dimensional Wiener process. The RSMF accounts for the random fluctuations of RSGD and, thereby, increases the order of approximation compared to the deterministic Riemannian gradient flow. The RSGD is built using the concept of a retraction map, that is, a cost-efficient approximation of the exponential map, and we prove quantitative bounds for the weak error of the diffusion approximation under assumptions on the retraction map, the geometry of the manifold, and the random estimators of the gradient.

U2 - 10.48550/arXiv.2402.03467

DO - 10.48550/arXiv.2402.03467

M3 - Article

SN - 0363-0129

VL - 62

SP - 3288

EP - 3314

JO - SIAM Journal on Control and Optimization

JF - SIAM Journal on Control and Optimization

IS - 6

ER -