A Broad Study on the Transferability of Visual Representations with Contrastive Learning