The impact of apolipoprotein E ε4 (APOE4), the strongest genetic risk factor for Alzheimer’s disease (AD), on human brain cellular function remains unclear. Here, we investigated the effects of APOE4 on brain cell types derived from population and isogenic human induced pluripotent stem cells, post-mortem brain, and APOE targeted replacement mice. Population and isogenic models demonstrate that APOE4 local haplotype, rather than a single risk allele, contributes to risk. Global transcriptomic analyses reveal human-specific, APOE4-driven lipid metabolic dysregulation in astrocytes and microglia. APOE4 enhances de novo cholesterol synthesis despite elevated intracellular cholesterol due to lysosomal cholesterol sequestration in astrocytes. Further, matrisome dysregulation is associated with upregulated chemotaxis, glial activation, and lipid biosynthesis in astrocytes co-cultured with neurons, which recapitulates altered astrocyte matrisome signaling in human brain. Thus, APOE4 initiates glia-specific cell and non-cell autonomous dysregulation that may contribute to increased AD risk.