Motivated by proving the loss of ergodicity in expanding systems of piecewise affine coupled maps with arbitrary number of units, all-to-all coupling and inversion symmetry, we provide ad-hoc substitutes - namely inversion-symmetric maps of the simplex with arbitrary number of vertices - that exhibit several asymmetric absolutely continuous invariant measures when their expanding rate is sufficiently small. In a preliminary study, we consider arbitrary maps of the multi-dimensional torus with permutation symmetries. Using these symmetries, we show that the existence of multiple invariant sets of such maps can be obtained from their analogues in some reduced maps of a smaller phase space. For the coupled maps, this reduction yields inversion-symmetric maps of the simplex. The subsequent analysis of these reduced maps show that their systematic dynamics is intractable because some essential features vary with the number of units; hence the substitutes which nonetheless capture the coupled maps common characteristics. The construction itself is based on a simple mechanism for the generation of asymmetric invariant union of polytopes, whose basic principles should extend to a broad range of maps with permutation and inversion symmetries.