Particle shape variability is a key to understanding the rich behavior of granular materials. Polyhedra are among the most common particle shapes due to their ubiquitous origins in nature such as rock fragmentation and mineral crystallisation. Because of their faceted shape, polyhedral particles tend to assemble in jammed structures in which face-face and face-edge contacts between particles control the packing-level properties. In this paper, we use tri-periodic particle dynamics simulations to derive for the first time a generic analytical expression of the elastic moduli of polyhedral and spherical particle packings subjected to triaxial compression as a function of two contact network variables: (1) a “constraint number" that accounts for the face-face and edge-face contacts between polyhedra and is reduced to the coordination number in the case of spherical particles, and (2) the contact orientation anisotropy induced by compression. This expression accurately predicts the simulated evolution of elastic moduli during compression, revealing thereby the origins of the higher elastic moduli of polyhedral particle packings. We show that particle shape affects the elastic moduli through its impact on the contact network and the level of nonaffine particle displacements is the same for the simulated shapes. Its nearly constant value during compression underlies the constant values of our model parameters. By connecting the elastic moduli to the contact network through parameters that depend on particle shape, our model makes it possible to extract both the connectivity and anisotropy of granular materials from the knowledge of particle shape and measurements of elastic moduli.