To enhance the efficiency of container transfers between trains and ships under the direct transfer mode and to reduce operational costs, this paper proposes an optimized strategy for crane and tractor scheduling that fully utilizes the overlapping time windows of both modes. A multi-objective optimization model is constructed to minimize the operation time and cost per container, and an intelligent algorithm is designed for its resolution. An empirical analysis at the WuXue Port multimodal transport hub tests the optimization effects of different terminal layouts and advanced railway utilization scenarios. The results show that, for a specific direct transfer ratio (50%), the proposed optimization model effectively improves the coordination between cranes and tractors, enhancing container transfer efficiency and reducing operational costs across various scenarios, with a maximum comprehensive benefit increase of 11.7%. The optimization extent of generalized direct transfer in terms of time and overall efficiency is higher than that of the narrow sense. Moreover, the segregated yard layout achieves more significant optimization benefits than the shared yard layout for generalized direct transfer. Additionally, for different direct transfer ratios, the proposed strategy yields optimization benefits overall. In the medium to low direct transfer ratio range (＜70%), the generalized mode shows a more significant improvement after optimization than the narrow sense. Conversely, at high ratio conditions (≥70%), the narrow sense mode and shared yard layout achieve better optimization performance, indicating that adopting a suitable direct transfer layout mode is an effective way to improve the comprehensive efficiency of terminal operations in different scenarios.