The non-point source pollution and carbon emission hinder the improvement of green production efficiency of agriculture in China, and the promotion and realization of green development of agriculture is related to its high green production efficiency. Based on the non-point source pollution and carbon emission, an index system for economy-energy-environment-society of green production efficiency of agriculture is established. Taking the panel data of agriculture in the Yangtze River Economic Belt from 2010 to 2019 as the demonstration belt of ecological civilization construction as an example, the improvement paths of efficiency are studied by combining the internal growth and the external drive. The results show that during the study period, the eco-efficiency of agriculture in the upper, middle and lower reaches of the Yangtze River Economic Belt exhibits a U-shaped change to the right, and the efficiency improvement should be transformed from the increase mode to the optimization one. The regional differences are characterized by enlargement and convergence, and the high-efficiency areas indicate the effect of point-point-point-piece diffusion, with a certain trickle-down effect. In terms of the internal growth, attention should be paid to the influences of some factors including management and use of water resources, environmental governance, regional drive for scientific and technological progress, and regional cooperation chain. Among the external driving factors, the urbanization degree, industrial structure, scientific research level and efficiency have a positive and significant relationship, while the income level of employees, planting area of food crops and financial support intensity have a negative and significant correlation. The insignificant results of environmental regulation need to be highlighted.