Zebrafish larvae use stereotyped discrete swim bouts of various types to navigate their environment. Their temporal sequence displays a complex structure, whose characteristics are modulated by external factors, such as the water temperature. Here, we show that the use of Hidden Markov Models allows one to parse the exploratory kinematics of larval zebrafish in an agnostic fashion. Our approach thus provides a more robust method of bout classification than was previously proposed with standard Markov Models relying on ad hoc state identification hypothesis. We then unveil temporal persistence in the navigation at low water temperature that was previously overlooked or underestimated. We further show that the model is accurate enough to capture subtle differences in exploratory trajectories between individuals, and has thus potential application in behavioral phenotyping. The code used in this study is made available in a format specifically designed for educational purposes.