Machine learning algorithms thrive on large data sets of good quality. Here we show that they can also excel in a typical research setting with little data of limited quality, through an interplay of insights coming from machine, and human researchers. The question we address is the unsolved problem of ordering out of a spin-liquid phase described by an emergent rank-2 U(1) gauge theory, as described by [H. Yan it et al., Phys. Rev. Lett. 124, 127203 (2020)]. Published Monte Carlo simulations for this problem are consistent with a strong first-order phase transition, out of the R2-U1 spin liquid, but were too noisy for the form of low-temperature order to be identified. Using a highly-interpretable machine learning approach based on a support vector machine with a tensorial kernel (TKSVM), we re-analyze this Monte Carlo data, gaining new information about the form of order that could in turn be interpreted by traditionally-trained physicists. We find that the low-temperature ordered phase is a form of hybrid nematic order with emergent Z_2 symmetry, which allows for a sub-extensive set of domain walls at zero energy. This complex form of order arises due to a subtle thermal order-by-disorder mechanism, that can be understood from the fluctuations of the tensor electric field of the parent rank-2 gauge theory. These results were obtained by a back-and-forth process which closely resembles a collaboration between human researchers and machines. We argue that this "collaborative" approach may provide a blueprint for solving other problems that have not yielded to human insights alone.