We investigate the impact of confinement density (i.e the number of individuals in a group per unit area of available space) on transitions from polarized to milling state, using groups of rummy-nose tetra fish (Hemigrammus rhodostomus) under controlled experimental conditions. We demonstrate a continuous state transition controlled by confinement density in a group of live animals. During this transition, the school exhibits a bistable state, wherein both polarization and milling states coexist, with the group randomly alternating between them. A simple two-state Markov process describes the observed transition remarkably well. The confinement density influences the statistics of this bistability, shaping the distribution of transition times between states. Our findings suggest that confinement plays a crucial role in state transitions for moving animal groups. More generally, they provide an experimental benchmark for active matter models of macroscopic, self-propelled, confined agents.