Frustrated magnetism and quantum transitions of nematic phases in FeSe

The iron-based superconductivity has been known to develop near an antiferromagnetic order, but this paradigm apparently fails in the FeSe. The striking puzzle that FeSe displays a nematic order while being non-magnetic has led to competing proposals for the origin of the nematic order. Here we show that the phase diagram of FeSe can be fully described by a quantum spin model with highly frustrated interactions. We perform density matrix renormalization group calculations on a frustrated spin-1 bilinear-biquadratic model on the square lattice, and find three stable phases breaking $C_4$ rotational symmetry, including the antiferromagnetic states with wave vectors $(0,\pi)$ and $(\pi/2,\pi)$, and a $(\pi,0)$ antiferroquadrupolar state. Tuning the ratio of competing interactions, we show quantum transitions from the $(\pi,0)$ antiferroquadrupolar order to the $(\pi,0)$ antiferromagnetic state, either directly or through the $(\pi/2,\pi)$ antiferromagnetic order. Our findings explain the recent dramatic experimental observations of an orthorhombic antiferromagnetic order in the pressurized FeSe, and suggest that superconductivity in a wide range of iron-based materials has a common origin in the antiferromagnetic correlations of strongly correlated electrons.