Chiral spin density wave order on frustrated honeycomb and bilayer triangular lattice Hubbard model at half-filling

We study the ground state properties of the Hubbard model on the honeycomb lattice with nearest-neighbor $t_1$ and second nearest-neighbor hopping $t_2$, which is isomorphic to the bilayer triangular lattice. We show that, at half-filling, chiral spin-density wave ($\chi$-SDW) order emerges due to on-site Coulomb interaction $U$ in a wide range of $t_2/t_1$ where both the two-sublattice antiferromagnetic order for small $t_2/t_1$ and the decoupled three-sublattice 120$^\circ$ magnetic order are strongly frustrated. For fixed $t_2/t_1$, we find that increasing $U$ leads to a continuous transition from a $\chi$-SDW semimetal with anomalous Hall effect to a topological chiral Chern insulator exhibiting quantum anomalous Hall effect, which undergoes a first order transition into a $\chi$-SDW insulator with zero total Chern number but anomalous AC Hall effect. We obtain the rich phase diagram and discuss the novel magnetic and topological properties.