The drivers of sexual dimorphism in heart failure phenotypes are currently poorly understood. Divergent phenotypes may result from differences in heritability and genetic versus environmental influences on the interplay of cardiac structure and function. To assess sex-specific heritability and genetic versus environmental contributions to variation and inter-relations between echocardiography traits in a large community-based cohort. We studied Framingham Heart Study participants of Offspring Cohort examination 8 (2005-2008) and Third Generation Cohort examination 1 (2002-2005). Five cardiac traits and six functional traits were measured using standardized echocardiography. Sequential Oligogenic Linkage Analysis Routines (SOLAR) software was used to perform singular and bivariate quantitative trait linkage analysis. In our study of 5674 participants (age 49 ± 15 years; 54% women), heritability for all traits was significant for both men and women. There were no significant differences in traits between men and women. Within inter-trait correlations, there were two genetic, and four environmental trait pairs with sex-based differences. Within both significant genetic trait pairs, men had a positive relation, and women had no significant relation. We observed significant sex-based differences in inter-trait genetic and environmental correlations between cardiac structure and function. These findings highlight potential pathways of sex-based divergent heart failure phenotypes.