Systems of self-interacting particles/agents arise in multiple disciplines, such as particle systems in physics, flocking birds and swarming cells in biology, and opinion dynamics in social science. We present an efficient nonparametric regression algorithm to learn the distance-based interaction kernel between the particles/agents from data for three types of systems: ODEs, SDEs, and mean-field PDEs. Importantly, we provide a systematic learning theory addressing the fundamental issues such as identifiability and convergence of the estimators. In particular, the identifiability theory highlights a transition from well-posed to ill-posed inverse problems when the number of particles increases infinity and the need of regularization. We demonstrate our algorithm on various examples, including the opinion dynamics, the Lennard-Jones kernel, and aggregation-diffusions.