On the non-universality of deep learning: quantifying the cost of symmetry