Aiming at the difficulty of visual monitoring for ships in the safety inspection of river courses, a visual measurement method for tracking the ship motion in river model tests and a UAV-based trajectory tracking method in real rivers were proposed based on the high-resolution color image recognition technology. The experimental results show that the maximum relative error between the model ship trajectory measured by visual monitoring and the real trajectory is -3.284%, meeting the requirements of river model experiments. The SURF method was applied to rectify the UAV images during the ship trajectory tracking process, effectively eliminating the effects of UAV position drifting on the tracking results, and the simultaneous visual tracking and monitoring of multiple ship targets were realized.