A fundamental invariant associated to a lattice polytope is its Ehrhart polynomial, which encodes the number of lattice points inside all the integer dilations of the polytope and much more arithmetic, algebraic and combinatorial information. One may associate to any matroid two polytopes called respectively the base polytope and the independence polytope; both of these polytopes can be seen as part of the larger family of generalized permutohedra. A conjecture of De Loera, Haws and Köppe asserted that the Ehrhart polynomials of base polytopes of matroids had positive coefficients only; more generally, Castillo and Liu conjectured this was true for all generalized permutohedra (in particular, also for independence polytopes). We will show how to construct counterexamples to these conjectures; we will exhibit examples of matroids whose base and independence polytopes attain negative Ehrhart coefficients. On the positive side, we will discuss about some families of matroids that satisfy Ehrhart positivity. Several open problems regarding Ehrhart polynomials of matroids will be stated.