Reconstructions of the global mean annual temperature evolution during the Holocene yield conflicting results. One temperature reconstruction shows global cooling during the late Holocene. The other reconstruction reveals global warming. Here we show that both a global warming mode and a cooling mode emerge when performing a spatio-temporal analysis of annual temperature variability during the Holocene using data from a transient climate model simulation. The warming mode is most pronounced in the tropics. The simulated cooling mode is determined by changes in the seasonal cycle of Arctic sea-ice that are forced by orbital variations and volcanic eruptions. The warming mode dominates in the mid-Holocene, whereas the cooling mode takes over in the late Holocene. The weighted sum of the two modes yields the simulated global temperature trend evolution. Our findings have strong implications for the interpretation of proxy data and the selection of proxy locations to compute global mean temperatures.