A non-magnetic liquid with nematicity in the spin-1 $SU(3)$ Heisenberg model on the square lattice

We study the spin-1 $SU(3)$ Heisenberg model with the nearest-neighbor bilinear and biquadratic interactions on the square lattice by using the large-scale density matrix renormalization group. By calculating spin and quadrupolar order parameters on the cylinder geometry up to system width $L_y = 9$, we find many competing peaks of structure factor at different momenta including the three-sublattice magnetic order proposed by previous studies. However, through appropriate extrapolation on large system size, all the spin and quadrupolar orders are scaled to zero. Surprisingly, we also find a finite lattice nematicity that characterizes a spontaneous lattice $C_4$ symmetry breaking. Our results exclude the three-sublattice magnetic order, and reveal a non-magnetic liquid with nematicity in the vinicity of the highly competing $SU(3$) point. We further discuss this new quantum phase by analyzing the low-energy excitations and by considering different perturbations on the $SU(3)$ model.