Understanding the genomic basis of memory processes may help in combating neurodegenerative disorders. Hence, we examined the associations of common genetic variants with verbal short-term memory and verbal learning in adults without dementia or stroke (N = 53,637). We identified novel loci in the intronic region of CDH18, and at 13q21 and 3p21.1, as well as an expected signal in the APOE/APOC1/TOMM40 region. These results replicated in an independent sample. Functional and bioinformatic analyses supported many of these loci and further implicated POC1. We showed that polygenic score for verbal learning associated with brain activation in right parieto-occipital region during working memory task. Finally, we showed genetic correlations of these memory traits with several neurocognitive and health outcomes. Our findings suggest a role of several genomic loci in verbal memory processes.