Focusing on the stability problem of floating wind turbine subjected to various complex loads, by equipping a tuned mass damper(TMD) on the nacelle of the 5MW Barge wind turbine and using FAST to simulate the dynamic performance of the structure, this paper is to investigate the control effect of TMD on the motion characteristics, tower top displacement and tower root load of floating wind turbines. In addition, the TMD structure parameters were optimized based on the Genetic Algorithm. Meanwhile, motion responses of floating wind turbines without the control, with the nonoptimized TMD control and with the optimized TMD control were compared. Based on the measured sea state data, the rationality and reliability of optimized TMD were verified. The results show that compared with the nonoptimized TMD, the optimized TMD is more effective in controlling the roll, the sidetoside displacement at the tower top and the sidetoside load at the tower root. The optimized TMD can be applied to most sea states and can well control the amplitude of motion response in the high frequency interval of sea state. Besides, with the improvement of sea state level, the control effect of motion response for the floating wind turbine becomes more obvious. The results show the reliability and superiority of TMD control method and optimization algorithm.