Toppling failure of unstable rocks was analyzed using fracture mechanics. Based on tensile-shear fracture tests of rock samples, fracture initiation and propagation mechanisms under loading were analyzed, and crack propagation during the fracture process was investigated. The effects of the crack length, width, and inclination angle as well as the loading point on the failure mode and stability of unstable rocks were discussed. An overhanging rock, the Taibaiyan unstable rock in Wanzhou District of Chongqing City was used as a case study. The stress state of crack tips under different cracking conditions was calculated using the finite element software ANSYS, the relationship between the complex breaking stress intensity factor and stress state was discussed, and the dynamic process of crack propagation was simulated. Results show that, during the processes of tension, shear, and toppling failures of unstable rocks, tensile stresses were concentrated at creak tips, unstable rock cracking was first induced by tension failures, and shear-compression failures occurred at the bottom. The sensibilities of the factors to the instability of unstable rocks were ranked in an descending order as follows: the loading position, crack length, crack inclination angle, and crack width.