In order to study the nonlinear hardening problem of three-dimensional complex structural materials, the consistent tangent modulus was introduced into the constitutive model considering the Von Mises plastic yield criterion and isotropic/kinematic hardening model, and the constitutive model was solved based on the scaled boundary finite element method (SBFEM) and balanced octree algorithm. The Newton Raphson iteration method was used to solve the elastic-plastic increment of displacement and stress. In order to reduce the computational complexity, the variables of the iteration process were only executed and stored at the proportional center point of the subdomain. At the same time, a solution program was developed using Fortran language. Two application examples show that the calculation results of the model have high accuracy.