In order to accurately and conveniently obtain the morphological information of cracks from embankment surface with a large scale of complex backgrounds, a crack detection method based on U²-ADSNet is proposed based on the improvement of U²-Net. This method combines depthwise separable convolution and atrous convolution in U²-Net, which expands the receptive field of the original model, enhances the learning ability of detailed features, and reduces model parameters. Based on a limit number of UAV measurable image data, the open source dataset of cracks for transfer learning is applied to reduce the training cost. Crack detection on a large range of UAV imagery is achieved by slicing prediction and possible false detections are removed using connected domain search. The effectiveness of the improved model has been verified by comparing U²-ADSNet with semantic segmentation models such as FCN, SegNet, U-Net and DeepCrack on the embankment crack dataset. The model intersection of union reaches 78.55% after migration learning, and the comprehensive evaluation index is 87.87%, which can be used for embankment crack detection.