Gears are the core components of transmission systems in industrial equipment, including air crafts, new energy vehicles, and wind turbines. Tooth surface wear is among the most prevalent modes of failure. Many existing research on the wear and dynamic characteristics of gears fails to consider the influence of friction and surface roughness changes induced by wear accumulation in the running-in process, resulting in the inaccurate prediction of surface wear and response. Based on the fractal theory, this paper simulates the exterior topography of tooth in the wear running-in process, and uses it for roughness calculation. Considering the impact of roughness variations on the friction coefficient and the progression of wear, an improved wear prediction model introducing dynamic contact loads and a six degree-of-freedom (DOF) dynamic model that encompasses both friction and alterations in tooth profile are established. Furthermore, the interconnected relationship between dynamics and surface wear is considered. Subsequently, the vibration responses of the gear system under different wear conditions are analyzed. This work provides a deeper insight into the wear behavior of gears and lays a foundation for the intelligent operation and maintenance of the worn gear system.