The instability principles of density current are less studied at present, but they are significant to the life-span extension and the optimal operation of reservoirs. Mike3 was used to construct the three-dimensional numerical model for density current, which includes a hydrodynamic model, a turbulence model and the sediment transport diffusion equation. The model was validated by a flume experiment and the results show that the modeled sediment concentration agrees well with the measured value. Based on the model, the instability principles of turbidity current under the condition of rapidly varying topography is studied by using the Zipingpu Reservoir as an example. The influence of the bottom obstacle and the abrupt widening of the cross section on density current were analyzed and the discriminant conditions for the instability of density current are obtained for the two situations respectively. For the situation of bottom obstacle, the discriminant conditions are(1)the velocity of the density current decreases, (2)the sediment concentration decreases, and(3)the thickness of the front and the density current decreases. For the situation of abrupt widening at the cross section, they are(1)the velocity of the density current decreases, (2)the vertical profiles of sediment concentration change from parabolic distribution to logarithmic distribution, and(3)the thickness of the density current decreases.