PIRSA:23120027 Scientific Series Other

DOI10.48660/23120027

Series: Machine Learning Initiative

Dangel, F. (2023). Deep Learning Convolutions Through the Lens of Tensor Networks. Perimeter Institute. https://pirsa.org/23120027

Dangel, Felix. Deep Learning Convolutions Through the Lens of Tensor Networks. Perimeter Institute, Dec. 01, 2023, https://pirsa.org/23120027 

@misc{ pirsa_PIRSA:23120027,
  doi = {10.48660/23120027},
  url = {https://pirsa.org/23120027},
  author = {Dangel, Felix},
  keywords = {Other},
  language = {en},
  title = {Deep Learning Convolutions Through the Lens of Tensor Networks},
  publisher = {Perimeter Institute},
  year = {2023},
  month = {dec},
  note = {PIRSA:23120027 see, \url{https://pirsa.org}}
}

Abstract

Despite their simple intuition, convolutions are more tedious to analyze than dense layers, which complicates the transfer of theoretical and algorithmic ideas. We provide a simplifying perspective onto convolutions through tensor networks (TNs) which allow reasoning about the underlying tensor multiplications by drawing diagrams, and manipulating them to perform function transformations and sub-tensor access. We demonstrate this expressive power by deriving the diagrams of various autodiff operations and popular approximations of second-order information with full hyper-parameter support, batching, channel groups, and generalization to arbitrary convolution dimensions. Further, we provide convolution-specific transformations based on the connectivity pattern which allow to re-wire and simplify diagrams before evaluation. Finally, we probe computational performance, relying on established machinery for efficient TN contraction. Our TN implementation speeds up a recently-proposed KFAC variant up to 4.5x and enables new hardware-efficient tensor dropout for approximate backpropagation.

---

Zoom link https://pitp.zoom.us/j/99090845943?pwd=NHBNVTNnbDNSOGNSVzNGS21xcllFdz09

Resources

Next talk

Related Talks