Amyloid PET imaging has been crucial for detecting the accumulation of amyloid beta (Aβ) deposits in the brain and to study Alzheimer’s disease (AD). We performed a genome-wide association study on the largest collection of amyloid imaging data (N = 13,409) to date, across multiple ethnicities from multicenter cohorts to identify variants associated with brain amyloidosis and AD risk. We found a strong APOE signal on chr19q.13.32 (top SNP: APOE ɛ4; rs429358; β = 0.35, SE = 0.01, P = 6.2 × 10-311, MAF = 0.19), driven by APOE ɛ4, and five additional novel associations (APOE ε2/rs7412; rs73052335/rs5117, rs1081105, rs438811, and rs4420638) independent of APOE ɛ4. APOE ɛ4 and ε2 showed race specific effect with stronger association in Non-Hispanic Whites, with the lowest association in Asians. Besides the APOE, we also identified three other genome-wide loci: ABCA7 (rs12151021/chr19p.13.3; β = 0.07, SE = 0.01, P = 9.2 × 10-09, MAF = 0.32), CR1 (rs6656401/chr1q.32.2; β = 0.1, SE = 0.02, P = 2.4 × 10-10, MAF = 0.18) and FERMT2 locus (rs117834516/chr14q.22.1; β = 0.16, SE = 0.03, P = 1.1 × 10-09, MAF = 0.06) that all colocalized with AD risk. Sex-stratified analyses identified two novel female-specific signals on chr5p.14.1 (rs529007143, β = 0.79, SE = 0.14, P = 1.4 × 10-08, MAF = 0.006, sex-interaction P = 9.8 × 10-07) and chr11p.15.2 (rs192346166, β = 0.94, SE = 0.17, P = 3.7 × 10-08, MAF = 0.004, sex-interaction P = 1.3 × 10-03). We also demonstrated that the overall genetic architecture of brain amyloidosis overlaps with that of AD, Frontotemporal Dementia, stroke, and brain structure-related complex human traits. Overall, our results have important implications when estimating the individual risk to a population level, as race and sex will needed to be taken into account. This may affect participant selection for future clinical trials and therapies.