Brain lobar volumes are heritable but genetic studies are limited. We performed genome-wide association studies of frontal, occipital, parietal and temporal lobe volumes in 16,016 individuals, and replicated our findings in 8,789 individuals. We identified six genetic loci associated with specific lobar volumes independent of intracranial volume. Two loci, associated with occipital (6q22.32) and temporal lobe volume (12q14.3), were previously reported to associate with intracranial and hippocampal volume, respectively. We identified four loci previously unknown to affect brain volumes: 3q24 for parietal lobe volume, and 1q22, 4p16.3 and 14q23.1 for occipital lobe volume. The associated variants were located in regions enriched for histone modifications (DAAM1 and THBS3), or close to genes causing Mendelian brain-related diseases (ZIC4 and FGFRL1). No genetic overlap between lobar volumes and neurological or psychiatric diseases was observed. Our findings reveal part of the complex genetics underlying brain development and suggest a role for regulatory regions in determining brain volumes.