PIRSA:25040085 Conference

DOI10.48660/25040085

Conference/School: Theory + AI Workshop: Theoretical Physics for AI

Bordelon, B. (2025). Scaling Limits for Learning: Dynamics and Statics. Perimeter Institute. https://pirsa.org/25040085

Bordelon, Blake. Scaling Limits for Learning: Dynamics and Statics. Perimeter Institute, Apr. 09, 2025, https://pirsa.org/25040085 

@misc{ pirsa_PIRSA:25040085,
  doi = {10.48660/25040085},
  url = {https://pirsa.org/25040085},
  author = {Bordelon, Blake},
  keywords = {},
  language = {en},
  title = {Scaling Limits for Learning: Dynamics and Statics},
  publisher = {Perimeter Institute},
  year = {2025},
  month = {apr},
  note = {PIRSA:25040085 see, \url{https://pirsa.org}}
}

Abstract

In this talk, I will discuss how physics can help improve our understanding of deep learning systems and guide improvements to their scaling strategies. I will first discuss mathematical results based on mean-field techniques from statistical physics to analyze the feature learning dynamics of neural networks as well as posteriors of large Bayesian neural networks. This theory will provide insights to develop initialization and optimization schemes for neural networks that admit well defined infinite width and depth limits and behave consistently across model scales, providing practical advantages. These limits also enable a theoretical characterization of the types of learned solutions reached by deep networks, and provide a starting point to characterize generalization and neural scaling laws (see Cengiz Pehlevan's talk).

Resources

Next talk