Imaging studies have reported conflicting findings on how brain structure differs with age and sex. This may be explained by discrepancies and limitations in study population and study design. We report a study on brain tissue volumes in one of the largest cohorts of individuals studied to date of subjects with high mean age (mean ± standard deviation (SD) 76 ± 6 years). These analyses are based on magnetic resonance imaging (MRI) scans acquired at baseline on 4303 non-demented elderly, and 367 who had a second MRI, on average 2.5 ± 0.2 years later. Tissue segmentation was performed with an automatic image analysis pipeline. Total brain parenchymal (TBP) volume decreased with increasing age while there was an increase in white matter hyperintensities (WMH) in both sexes. A reduction in both normal white matter (NWM)- and gray matter (GM) volume contributed to the brain shrinkage. After adjusting for intra-cranial volume, women had larger brain volumes compared to men (3.32%, p