In 2014, Saxe et al. provided empirical evidence showing that deep linear neural networks (DLNNs) and their non-linear counterparts have similar learning dynamics. Recently, Liao and Couillet (2018) showed that the analysis of DLNNs is amenable to random matrix theory techniques. In this talk we review some of these results and present a new take on the problem using free probability ideas. This is work in progress with Carlos Madrid (Universidad de Guanajuato) and Víctor Pérez-Abreu (CIMAT).