Defect-driven Competition between the Interlayer Magnetic Interactions in MnSb₂Te₄

Isostructural to the first realized antiferromagnetic (AF) topological insulator (TI) MnBi₂Te₄ (MBT), MnSb₂Te₄ (MST) demonstrates a complex magnetic behavior due to the heavy anti-site mixing between the Mn and Sb atoms. Targeted sample preparations reveal that small variations in the anti-site defect concentration result in samples having either the ferromagnetic (FM) or the AF ground state. Here we compare the inelastic neutron scattering results of the spin dynamics in the FM and AF MST single-crystal samples to MBT. In MST, the anti-site mixing introduced disorder that disrupts the spin waves, and also promotes new energetic AF couplings, both of which are absent in MBT. Based on the quantitative analysis using linear spin wave theory and classical atomistic spin dynamics simulations, we propose a scheme of anti-site defect-driven competition between the FM and AF interlayer interactions in MST, and discuss the implication of anti-site mixing in Mn(Bi,Sb)₂Te₄ and Mn-doped Bi₂Te₃ magnetic TIs.