The paper combines the balanced octree with polyhedron mesh trimming based on the image octree method, and proposes a mesh generation algorithm of polyhedron in the 3D scaled boundary finite element method. According to the size of the structure, a cube grid is built to exactly include the whole structure, and then the pixel information of the structure image is formed. Then, according to the principle of 2∶1 balanced partition, the grid is divided recursively to realize the fast transition of mesh, and the balanced octree mesh can be generated. The internal elements are completely preserved while the external elements are removed. For the mesh of the boundary element, the surface-cube intersection method is proposed to screen the intersection surfaces of boundary element and structural boundary, and the intersection points between boundary element and structural boundary surface are searched to form a new element surface. The cut surface of the boundary element is formed by connecting the intersection points in order, and then combined with other surfaces of the boundary element to form the cut polyhedron element. Two numerical examples show that the calculated results of the scaled boundary finite element mesh generated by the proposed method have good accuracy and boundary adaptability.