Longitudinal studies of the relationship between hyperglycemia and brain health are rare and there is limited information on sex differences in associations. We investigated whether glycosylated hemoglobin (HbA1c) measured at ages of 53, 60-64 and 69 years, and cumulative glycemic index (CGI), a measure of cumulative glycemic burden, were associated with metrics of brain health in later life. Participants were from Insight 46, a substudy of the Medical Research Council National Survey of Health and Development (NSHD) who undertook volumetric MRI, florbetapir amyloid-PET imaging and cognitive assessments at ages of 69-71. Analyses were performed using linear and logistic regression as appropriate, with adjustment for potential confounders. We observed a sex interaction between HbA1c and whole brain volume (WBV) at all 3 time points. Following stratification of our sample, we observed that HbA1c at all ages, and CGI were positively associated with lower WBV exclusively in females. HbA1c (or CGI) was not associated with amyloid status, white matter hyperintensities (WMHs), hippocampal volumes (HV) or cognitive outcomes in either sex. Higher HbA1c in adulthood is associated with smaller WBV at 69-71 years in females but not in males. This suggests that there may be preferential target organ damage in the brain for females with hyperglycemia.