Based on the Particle Swarm Optimization (PSO) and Least Square Support Vector Machine (LSSVM), a nonlinear mapping relationship is constructed. Combined with the Back Propagation Neural Network (BP) for machine learning on the database generated by the nonlinear mapping relationship, the PSO-LSSVM-BP model is built to determine the optimal mechanical parameters of rock mass. The PSO-LSSVM-BP model takes the monitored displacement data of high slope as input information, obtains the mechanical parameters of high slope rock mass through the inverse analysis, and applies them in the numerical calculations of displacement with FLAC3D. The results show that the simulation results are in good agreement with the monitoring data, which verifies the feasibility and effectiveness of the model. Utilizing the PSO-LSSVM-BP model, a three-dimensional stability evaluation is established on the intake slope of the Lianghekou Hydropower Station at different reservoir water levels. The results indicate that water level is the primary factor affecting the slope stability. As the water level rises, slope displacement gradually increases, with surface and fault damage increasing. While safety coefficients of local point decrease, the overall point safety coefficients remain above 1.30, indicating the intake slope has a certain level of safety margin.