Test-Time Selective Adaptation for Uni-Modal Distribution Shift in Multi-Modal Data