On the Robustness of Spectral Algorithms for Semirandom Stochastic Block Models