Physical approaches to the extraction of relevant information

          @misc{ pirsa_PIRSA:16080006,
            doi = {10.48660/16080006},
            url = {https://pirsa.org/16080006},
            author = {Schwab, David},
            keywords = {Condensed Matter},
            language = {en},
            title = {Physical approaches to the extraction of relevant information},
            publisher = {Perimeter Institute},
            year = {2016},
            month = {aug},
            note = {PIRSA:16080006 see, \url{https://pirsa.org}}
          }
          

Abstract

In the first part of this talk, I will focus on the physics of deep learning, a popular subfield of machine learning where recent performance on tasks such as visual object recognition rivals human performance. I present work relating greedy training of deep belief networks to a form of variational real-space renormalization. This connection may help explain how deep networks automatically learn relevant features from data and extract independent factors of variation. Next, I turn to the information bottleneck (IB), an information theoretic approach to clustering and compression of relevant information that has been suggested as a framework for deep learning. I present a new variant of IB called the Deterministic Information Bottleneck, arguing that it better captures the notion of compression while retaining relevant information.