Researchers argue scientific theory of deep learning, called "learning mechanics," is emerging from five bodies of work on neural network training dynamics and universal behaviors.

There Will Be a Scientific Theory of Deep Learning

Authors:Jamie Simon, Daniel Kunin, Alexander Atanasov, Enric Boix-Adserà, Blake Bordelon, Jeremy Cohen, Nikhil Ghosh, Florentin Guth, Arthur Jacot, Mason Kamb, Dhruva Karkada, Eric J. Michaud, Berkan Ottlik, Joseph Turnbull View PDF HTML (experimental) Abstract:In this paper, we make the case that a scientific theory of deep learning is emerging. By this we mean a theory which characterizes important properties and statistics of the training process, hidden representations, final weights, and pe...