Utility-based shortfall risk measure is proposed by Föllmer and Schied and has received increasing attentions over the past few years for its potential to quantify more effectively the risk of large losses than conditional value at risk. In this paper, we consider the case when the true utility/loss function cannot be specified either because there is missing information about how the decision maker perceives risk, or because he is simply hesitant about it. We propose a preference robust shortfall risk model that exploits empirical data about subjective judgements to construct a set of plausible utility-based loss functions and suggest minimizing shortfall risk as measured using the worst loss function from this set. We develop tractable reformulations when the underlying probability distribution is discrete. In the case when the probability distribution is continuous, we propose a sample average approximation scheme and show that its optimal solution and value converges to the true ones as the sample size increases.