It is commonly believed that the correlations between stock returns increase in high volatility periods. We investigate how much of these correlations can be explained within a simple non-Gaussian one-factor description with time independent correlations. Using surrogate data with the true market return as the dominant factor, we show that most of these correlations, measured by a variety of different indicators, can be accounted for. In particular, this one-factor model can explain the level and asymmetry of empirical exceedance correlations. However, more subtle effects require an extension of the one factor model, where the variance and skewness of the residuals also depend on the market return.