From Isolated Islands to Pangea: Unifying Semantic Space for Human Action Understanding