On the Robustness of Language Guidance for Low-Level Vision Tasks: Findings from Depth Estimation