Date: Fri. October 22, 2021
Organized by: FAU DCN-AvH, Chair for Dynamics, Control and Numerics – Alexander von Humboldt Professorship at FAU Erlangen-Nürnberg (Germany)
Title: Analysis of gradient descent on wide two-layer ReLU neural networks

Speaker: Dr. Lénaïc Chizat
Affiliation: EPFL, École Polytechnique Fédérale de Lausanne (Switzerland)

Abstract. In this talk, we propose an analysis of gradient descent on wide two-layer ReLU neural networks that leads to sharp characterizations of the learned predictor. The main idea is to study the training dynamics when the width of the hidden layer goes to infinity, which is a Wasserstein gradient flow. While this dynamics evolves on a non-convex landscape, we show that for appropriate initializations, its limit, when it exists, is a global minimizer. We also study the implicit regularization of this algorithm when the objective is the unregularized logistic loss, which leads to a max-margin classifier in a certain functional space. We finally discuss what these results tell us about the generalization performance, and in particular how these models compare to kernel methods.

Recording/Video:


_

Don’t miss out our Upcoming events!

|| Subscribe to our FAU DCN-AvH newsletter

Tags:

Don't miss out our posts on Math & Research!

Transition Layers in Elliptic Equations By Maicon Sônego   Stable transition layers in an unbalanced bistable equation Consider the following semi-linear problem where are positive functions in ; is a positive parameter and We assume that the functions satisfy ; for all ; there is a […]
Randomized time-splitting in linear-quadratic optimal control By Daniël Veldman   Introduction Solving an optimal control problem for a large-scale dynamical system can be computationally demanding. This problem appears in numerous applications. One example is Model Predictive Control (MPC), which requires the solution of several optimal control […]
Felix Klein: A Legacy of Innovation in Mathematics and Education By Roberto Rodríguez del Río, Complutense University of Madrid | IES San Mateo, Madrid   Felix Christian Klein lived in a period of history of science in which Mathematics were involved in a process of transformation, […]
Our last Publications
[cris show="publications" persID="223281397,105092142,229344528,239343629,241149469,243434665,105514816,242263337,104776092,236754096,243266999,243266999" year="2020" type="beitrag_fachzeitschrift" sortby="updated" quotation="apa" items="5"]
© 2019-2021 Chair for Dynamics, Control and Numerics - Alexander von Humboldt Professorship at FAU Erlangen-Nürnberg, Germany | Imprint | Contact