We develop a reinforcement learning framework for retail robo-advising. The machine does not know the investor's risk preferences, but dynamically learns them by observing the investor's responses to its proposed investment decisions. We show that the optimal investment problem can be cast to a hidden-goal Markov decision process, and develop an exploration-exploitation algorithm for the optimal portfolio selection. We show that in a number of steps that grows polynomially with the dimension of the state space, the machine will learn the investor's risk preferences with high accuracy. We present numerical examples to illustrate the impact of state-dependent risk-preferences, and investor's overriding costs on the investment value.