Observing precursors of the Dicke quantum phase transition

A Bose-Einstein condensate coupled to an optical high-finesse cavity realizes an effective version of the Dicke Hamiltonian. This model was predicted to show an intriguing quantum phase transition. In our experiment, a transverse pump field couples collective density waves of the condensate to the cavity light field. By controlling the transverse pump power, the system is driven from the normal to the super-radiant phase. We investigated the excitation spectrum of the coupled system in the normal phase by Bragg spectroscopy and identified a collective density mode which softens at the transition point. This vanishing energy scale significantly alters the spectrum of atomic density fluctuations: the barrier to incoherently populate the soft mode is continuously reduced. The openness of the cavity allows to extract in-situ information about the fluctuations of the system from the photons leaving the cavity. We could reveal increased noise at the frequency of the soft mode by analyzing time-correlations between the leaking cavity photons.