Simultaneously recorded neurons exhibit correlations whose underlying causes are not known. Here, we use a population of threshold neurons receiving correlated inputs to model neural population recordings. We show analytically that small changes in second-order correlations can lead to large changes in higher correlations, and that these higher-order correlations have a strong impact on the entropy, sparsity and statistical heat capacity of the population. Remarkably, our findings for this simple model may explain a couple of surprising effects recently observed in neural population recordings.