In this talk, we present some recent results on the approximation theory of deep learning architectures for sequence or time series analysis. In particular, we formulate a basic mathematical framework, under which different popular architectures such as recurrent neural networks, dilated convolutional networks (e.g. WaveNet), encoder-decoder structures can be rigorously compared. These analyses reveal some interesting connections between approximation, memory, sparsity and low rank phenomena that may guide the practical selection and design of these network architectures.