One application challenge of the neural networks is the training, the essence of which is an iterative optimization process involving numerous data. The training process requires significant computing power and efficient searching methods for optimal solutions. To meet the requirement, a parallel BFGS quasi-Newton training algorithm based on GPU is proposed in this paper. To maximize the parallelism, the BFGS quasi-Newton algorithm is divided into different function modules, and each module is designed with a specific parallelism regarding its characteristics. In addition, the processing and memory resources of GPU are fully utilized to achieve a better parallelization. Experimental results show that the GPU implementation accelerates the neural network training by up to 80 times compared to the CPU implementation for the complicated neural network structure, while the speed up ratio is up to 430 for the modelling test of microwave device, where the training error is about 1%.