(Ba, Ca)TiO3 and Ba(Ti, Zr)O3 solid solutions are the building blocks of lead-free piezoelectric materials that attract a renewed interest. We investigate the properties of these systems by means of first-principles calculations, with a focus on the lattice dynamics and the competition between different ferroelectric phases. We first analyze the four parent compounds in order to compare their properties and their different tendency towards ferroelectricity. The core of our study is systematic characterization of the binary systems (Ba, Ca)TiO3 and Ba(Ti, Zr)O3 within both the Virtual Crystal Approximation and direct supercell calculations. In the case of Ca doping, we find a gradual transformation from $A$-site to $B$-site ferroelectricity due to steric effects that largely determines the behavior of the system. In the case of Zr doping, in contrast, the behavior is eventually dominated by cooperative Zr-Ti motions and the local electrostatics. In addition, our comparative study reveals that the specific microscopic physics of these solid sets severe limits to the applicability of the Virtual Crystal Approximation for these systems.