Studying frustration with ultracold bosons in a triangular optical lattice

In the lowest band of the triangular lattice, the inherent geometric frustration of the lattice gives rise to two degenerate, inequivalent maxima with different wavevectors. When tunnelling is inverted, such that these maxima become the minima, the superfluid ground state becomes chiral and chooses one of the two minima, breaking a discrete symmetry. The nature of the quantum phase transition from this chiral superfluid to a Mott insulator (SF-MI) at strong interactions is not settled, as the lattice frustration hinders quantum Monte-Carlo simulation.

We experimentally study this transition in an optical lattice. Instead of actively changing the sign of the tunnelling, we utilize negative absolute temperatures where atoms predominantly occupy the highest-energy states. We find that due to the inherent geometric frustration, the critical interaction for the bosonic SF-MI transition is strongly reduced. Furthermore, by dynamically crossing the SF-MI transition in variable time we observe a smooth emergence of coherence, reminiscent of the Kibble-Zurek mechanism. This is consistent with a continuous phase transition and thereby suggests the existence of an intervening chiral Mott insulator phase.