Understanding the causes and architecture of genetic differentiation between natural populations is of central importance in evolutionary biology. Crosses between natural populations can result in heterosis if recessive or nearly recessive deleterious mutations have become fixed within populations because of genetic drift. Divergence between populations can also result in outbreeding depression because of genetic incompatibilities. The net fitness consequences of between-population crosses will be a balance between heterosis and outbreeding depression. We estimated the magnitude of heterosis and outbreeding depression in the highly selfing model plant Arabidopsis thaliana, by crossing replicate line pairs from two sets of natural populations (C?R, B?S) separated by similar geographic distances (Italy?Sweden). We examined the contribution of different modes of gene action to overall differences in estimates of lifetime fitness and fitness components using joint scaling tests with parental, reciprocal F1 and F2, and backcross lines. One of these population pairs (C?R) was previously demonstrated to be locally adapted, but locally maladaptive quantitative trait loci were also found, suggesting a role for genetic drift in shaping adaptive variation. We found markedly different genetic architectures for fitness and fitness components in the two sets of populations. In one (C?R), there were consistently positive effects of dominance, indicating the masking of recessive or nearly recessive deleterious mutations that had become fixed by genetic drift. The other set (B?S) exhibited outbreeding depression because of negative dominance effects. Additional studies are needed to explore the molecular genetic basis of heterosis and outbreeding depression, and how their magnitudes vary across environments.