Observation of thickness-dependent topological phase transition in Sb films

Topological insulators are distinguished by their metallic boundary states, and topological phase transitions that can be tuned between trivial and nontrivial by several parameters, such as composition, strain, and electrical field. Here we demonstrate such phase transition by fine tuning the thickness of Sb films prepared by molecular beam epitaxy. Based on in situ STM observations, we find that Sb growth on 3D topological insulator Sb2Te3(111) initiates at step edges, leading to islands with various heights. Spatially resolved tunneling spectroscopy measurements reveal edge states for Sb films 4-8 bilayer thick, which are absent for 2 and 10 bilayer films. The edge states are confined within a few lattice spacing (1.5-2nm) from the step edge. Supported by density functional theory calculations, our findings indicate Sb(111) films undergo a phase transition from normal insulator to two-dimensional topological insulator at a critical thickness of 3 bilayers.