Reducing theoretical uncertainties in Galactic dark matter (DM) searches is an important challenge as several experiments are now delving into the parameter space relevant to popular (particle or not) candidates. Since many DM signal predictions rely on the knowledge of the DM velocity distribution—direct searches, capture by stars, p-wave-suppressed or Sommerfeld-enhanced annihilation rate, microlensing of primordial black holes, etc—it is necessary to assess the accuracy of our current theoretical handle. Beyond Maxwellian approximations or ad-hoc extrapolations of fits on cosmological simulations, approaches have been proposed to self-consistently derive the DM phase-space distribution only from the detailed mass content of the Galaxy and some symmetry assumptions (e.g. the Eddington inversion and its anisotropic extensions). Although theoretically sound, these methods are still based on simplifying assumptions and their relevance to real galaxies can be questioned. In this paper, we use zoomed-in cosmological simulations to quantify the associated uncertainties. Assuming isotropy, we predict the speed distribution and its moments from the DM and baryonic content measured in simulations, and compare them with the true ones. {Taking as input galactic mass models fitted on full simulation data, we reach a predictivity down to ∼ 10% for some velocity-related observables, significantly better than some Maxwellian models.} This moderate theoretical error is particularly encouraging at a time when stellar surveys like the Gaia mission should allow us to improve constraints on Galactic mass models.