This study examined the single-nucleotide polymorphism heritability and genetic correlations of cognitive abilities and brain structural measures (regional subcortical volume and cortical thickness) in middle-aged and elderly East Asians (Korean) from the Gwangju Alzheimer’s and Related Dementias cohort study. Significant heritability was found in memory function, caudate volume, thickness of the entorhinal cortices, pars opercularis, superior frontal gyri, and transverse temporal gyri. There were 3 significant genetic correlations between (i) the caudate volume and the thickness of the entorhinal cortices, (ii) the thickness of the superior frontal gyri and pars opercularis, and (iii) the thickness of the superior frontal and transverse temporal gyri. This is the first study to describe the heritability and genetic correlations of cognitive and neuroanatomical traits in middle-aged to elderly East Asians. Our results support the previous findings showing that genetic factors play a substantial role in the cognitive and neuroanatomical traits in middle to advanced age. Moreover, by demonstrating shared genetic effects on different brain regions, it gives us a genetic insight into understanding cognitive and brain changes with age, such as aging-related cognitive decline, cortical atrophy, and neural compensation.