Unsupervised representation learning has been dominated by two paradigms. In the probabilistic paradigm, the aim is to learn a probability distribution that models the observed data. More recently, the contrastive or self-supervised paradigm, which instead aims to learn an embedding of the data that preserves distances between related data, or predicts one part of the data from another, has gained significant traction. In both of these paradigms, observations are usually projected into a flat vector space, throwing out information about the natural curvature of the data manifold. Here we propose a new approach to unsupervised learning, which uses the curvature as a learning signal rather than throwing away curvature information.

We show how this insight leads to a new approach to the problem of disentangling in unsupervised learning. Disentangling is usually defined in the probabilistic paradigm. However, it can be shown that in this formulation, fully unsupervised disentangling is not possible without further prior assumptions. We present an alternative formulation based on a group-theoretic definition, and an algorithm which aims to learn a representation which satisfies this definition - the Geometric Manifold Component Estimator (GEOMANCER). GEOMANCER Is a nonparametric algorithm in the model of Laplacian Eigenmaps, but generalized to discover subspaces of a latent space which are invariant under transport around a loop. We show that GEOMANCER is effective at discovering disentangled submanifolds from data when the true metric in the latent space is known. Critically, GEOMANCER is capable of learning disentangled submanifolds even when the data from different submanifolds are not sampled independently, showing that the geometric approach to unsupervised learning really is distinct from the probabilistic paradigm.

Bio: David Pfau is a staff research scientist at DeepMind, where he has been since 2015, and a visiting professor at Imperial College London in the Department of Physics. Prior to that, he was a PhD student at the Center for Theoretical Neuroscience at Columbia University, where he worked on Bayesian nonparametric time series modeling with Frank Wood and applications of machine learning to analyzing high dimensional neuroscience data with Liam Paninski. At DeepMind, in addition to his research on geometry and disentangling, he works on applications of deep learning to computational physics, in areas such as ab initio quantum chemistry and real-time control for magnetic confinement fusion.