In order to improve the problems of low training efficiency, easy over-fitting, and poor parameter sensitivity in traditional modeling methods, an extreme gradient boosting (XGBoost) algorithm is introduced, combined with a Bayesian optimization method based on Gaussian process (GP), to improve the learning efficiency and the prediction accuracy.The dam deformation prediction model based on the proposed method was established and the prediction effect was compared with the traditional statistical model and the neural network model. The results show that the monitoring model based on this method has high prediction accuracy and fast iteration speed. Overfitting can be effectively avoided by adjusting the regular term parameters.