Refractive changes are reportedly affected by age, sex, and current refractive error. To clarify the pattern of refractive changes in a Japanese population, we conducted a 5-year follow-up longitudinal analysis of spherical equivalent (SE) refractive changes with stratification by sex, age, and SE in 593,273 eyes from Japanese individuals ages 3–91 years. The 5-year SE change with myopic shift dramatically increased over time after age 4 years, and the largest change was observed in both males and females who were age 8 years at baseline [males: − 2.654 ± 0.048 diopters (D); females: − 3.110 ± 0.038 D]. During school age, the 5-year myopic change was greater in females than in males, and emmetropic and low-to-moderate myopic eyes underwent larger myopic changes than hyperopic and high-to-severe myopic eyes. After the peak at age 8 years, the 5-year myopic change gradually declined with age and fell below − 0.25 D at age 27 in males and age 26 years in females. The 5-year SE changes transitioned from a myopic to a hyperopic shift at age 51 in both sexes, and hyperopization advanced more quickly in hyperopic eyes. Our findings highlight the importance of myopia prevention in school-aged children.