Alzheimer disease (AD) is a chronic neurodegenerative disease with a multitude of contributing genetic factors, many of which are related to inflammation. The apolipoprotein E (APOE) ε4 allele is the most common genetic risk factor for AD and is related to a pro-inflammatory state. To test the hypothesis that microglia and AD-implicated cytokines were differentially associated with AD pathology based on the presence of APOE ε4, we examined the dorsolateral frontal cortex from deceased participants within a community-based aging cohort (n = 154). Cellular density of Iba1, a marker of microglia, was positively associated with tau pathology only in APOE ε4 positive participants (p = 0.001). The cytokines IL-10, IL-13, IL-4, and IL-1α were negatively associated with tau pathology, independent of Aβ1-42 levels, only in APOE ε4 negative participants. Overall, the association of mostly anti-inflammatory cytokines with less tau pathology suggests a protective effect in APOE ε4 negative participants. These associations are largely absent in the presence of APOE ε4 where tau pathology was significantly associated with increased microglial cell density. Taken together, these results suggest that APOE ε4 mediates an altered inflammatory response and increased tau pathology independent of Aβ1-42 pathology.