Topological Phase Transition of the 5/2 Fractional Quantum Hall Effect

We study the effect of three-body interaction on the fractional quantum Hall effect at filling factor 5/2 using exact diagonization method with torus geometry in a finite-size system. The calculation of the wave function overlap suggests that a repulsive three-body interaction will induce a phase transition to a Pfaffian state. Its particle-hole conjugate (anti-Pfaffian) state is found only in a very narrow region of the attractive three-body interaction while a stronger attraction will push the system into a compressible state. The results from the low energy spectrum and the particle-hole parity evolution further indicate a continuous phase transition between the Pfaffian and anti-Pfaffian states, with the pure Coulomb system sitting at the critical point of the transition.