Overtopping-induced landslide dam failures as a severe geological hazard can generate catastrophic flooding and pose a great threat to downstream areas. In this study, large-scale landslide dam breach experiments were carried out in natural river channel for studying the breach mechanism. The overtopping failure of landslide dam was found to be divided into two stages based on different erosion characteristics: headward erosion and overall erosion. Besides, there are two modes of lateral slope failure, toplling failure and shear failure. Successive erosion of the breach can lead to intermittent instability of the slope, causing continuous fluctuations in the slope angle. Based on the failure mechanism of landslide dam combined with hydraulic module, soil erosion module, and sediment force analysis, a new simulation method was developed in this study to predict dam failure process. The model successfully predicted the change process of water level, breach discharge, and breach size in the large-scale experiment. The results show that the relative error of the simulated peak discharge and time to peak were 1.17% and 1.17%, respectively, which meet the accuracy requirements of hydrological forecasting. Additionally, the relative errors of the simulated final breach size, including top width, bottom width, and depth, were all less than 2%. Finally, sensitivity analyses show that coefficient of erodibility, spillway depth, and inflow rate have a significant influence on the breach process, and soil mechanical parameters have a great influence on the breach morphology.