+ SCIENCE Stochastic gradient descent (SGD), one of the most fundamental optimization algorithms in machine learning (M…+ LIFESTYLE With the increasing interest in glamping, the demand for more functional camping cookware has grown, driving…− MEDIA & ARTS Ai Weiwei is talking me through the decision-making process before his first visit to China in over a decade.…+ EDUCATION Anil Seth is right to point out that to overestimate artificial intelligence is to underestimate ourselves (O…− POLICY Content generated by artificial intelligence models can ‘defy basic physics, design principles or operational…+ MEDIA & ARTS Hook users can remix, mash up and share their content using fully licensed and cleared songs from UMG, Downto…− POLICY Pulitzer prize-winning US historian Jill Lepore’s new book addresses the threat posed to liberal democracy by…± EDUCATION Since its maiden release into the public domain on November 30, 2022, ChatGPT garnered more than one million…
TruaceTracing the truth around AIFriday, August 21, 2026
Subscribe
Science·G Space·Evidence-backed gain·Published 2026-08-21

Optimal Initialization Scale for Neural Networks With Locally Quadratic Loss Landscapes: An SGD Dynamics Perspective

Abstract: Stochastic gradient descent (SGD), one of the most fundamental optimization algorithms in machine learning (ML), can be recast through a continuous-time approximation as a Fokker-Planck equation for Langevin dynamics, a viewpoint that has motivated many theoretical studies. Within this framework, we study the relationship between the quasi-stationary distribution derived from this equation and the initial distribution through the Kullback-Leibler (KL) divergence. As the quasi-steady-state distribution depends on…

TRV-2026-0842Peer-reviewedPermanent record — cite & verify
Optimal Initialization Scale for Neural Networks With Locally Quadratic Loss Landscapes: An SGD Dynamics Perspective

An artificial neural network control system for spacecraft attitude stabilization by Segura, Clement M.. Public domain

The quick read

Stochastic gradient descent (SGD), one of the most fundamental optimization algorithms in machine learning (ML), can be recast through a continuous-time approximation as a Fokker-Planck equation for Langevin dynamics, a viewpoint that has motivated many theoretical studies. Within this framework, we study the relationship between the quasi-stationary distribution derived from this equation and the initial distribution through the Kullback-Leibler (KL) divergence.

In addition, we experimentally confirm our theoretical results by using classical SGD to train shallow fully connected neural networks on the MNIST dataset and a two-layer CNN model on the CIFAR-10 dataset. Our work thus supplies a mathematically grounded indicator for choosing the initialization variance and clarifies its physical meaning in terms of the parameter dynamics in neural network models.

Main points
  • Stochastic gradient descent (SGD), one of the most fundamental optimization algorithms in machine learning (ML), can be recast through a continuous-time approximation as a Fokker-Planck equation for Langevin dynamics, a viewpoint that has motivated many theoretical studies.
  • Within this framework, we study the relationship between the quasi-stationary distribution derived from this equation and the initial distribution through the Kullback-Leibler (KL) divergence.
  • As the quasi-steady-state distribution depends on the expected cost function, the KL divergence eventually reveals the connection between the expected cost function and the initialization distribution.
Gain

Optimal Initialization Scale for Neural Networks With Locally Quadratic Loss Landscapes: An SGD Dynamics Perspective: The results show that, for the simple network model, if the variance of the initialization distribution satisfies our theoretical optimal condition, then the corresponding network achieves lower final training loss and higher test accuracy than the conventional He-normal initialization.

The rundown

As the quasi-steady-state distribution depends on the expected cost function, the KL divergence eventually reveals the connection between the expected cost function and the initialization distribution. Under the assumptions of a locally quadratic neural-network loss landscape and a quasi-stationary SGD regime, we derive an explicit upper bound for the expected loss in terms of the initialization variance.

Sources

Reader signal

How should this claim be treated?

The debate