Medium-manganese steel exhibits excellent strength and toughness, which are essential features in wear resistance applications. This study examines the impact of annealing temperature on impact abrasive wear. The results have indicated that samples annealed at different temperatures display plowing and fatigue wear effects. In the initial wear stage, the high-temperature annealed steel outperforms samples annealed at a lower temperature in terms of anti-plowing wear performance. This phenomenon is mainly due to the lower initial hardness of the samples subjected to low-temperature annealing. However, with prolonged wear time, the low-temperature annealed samples exhibit improved plowing wear performance, which is ascribed to a refinement of the lamellar microstructure and an increased residual austenite (RA), which enhances the work hardening effect, improving the hardness of the worn surface. The low-temperature annealed samples consistently delivered superior fatigue wear performance when compared with samples annealed at the higher temperature. The latter effect may be attributed to two factors. Firstly, the finer lamellar microstructure in the low-temperature annealed samples, coupled with greater RA, results in transformation-induced plasticity or twin-induced plasticity effect that hinders crack formation and propagation. Secondly, the low-temperature annealed samples form nanoscale equiaxed grains near the worn surface during the wear process. These grains can withstand crack driving forces in fine-grained regions, suppressing the formation and propagation of cracks.