Intelligent assistants like Sirior the Google Assistantrely on ASR. Current ASR systems require users to manually specify the system’s correct input language to work properly. However, as a sensible pre-processing step we can infer the spoken language using an automatic LID system. Traditional LID systems utilize domain-specific expert knowledge in the field of audio signal processing for extracting hand-crafted features from the audio samples. Lately, deep learning and artificial neural networks have become the state-of-the-art for many pattern recognition problems. Deep Neural Networks (DNNs) have become the best performing method for a range of computer vision tasks, such as image classification [17,18], or object detection and recognition [14,15].

In this paper, we address the problem of language identification from a computer vision perspective. We extract the target language of a given audio sample by utilizing a hybrid network constructed of a Convolutional Neural Network (CNN) combined with an Recurrent Neural Network (RNN). Our contributions can be summarized as follows: (1) we propose a hybrid CRNN, combining the descriptive powers of CNNs with the ability to capture temporal features of RNNs. (2) We perform extensive experiments with our proposed network and show its applicability to a range of scenarios and its extensibility to new languages. (3) We release our code and a large scale training set for LID systems to the community.

The paper is structured in the following way: In section 2 we introduce related work in the field of LID systems. We showcase our system in section 3 and evaluate it on extensive experiments in section 4. We conclude our work in section 5.

Traditional language identification systems are based on identity vector systems for spoken-language processing tasks [4,11,13,19]. In the recent years systems using feature extractors solely based on neural networks, especially Long-Short Term Memory (LSTM) networks, became more popular [5,10,19]. These neural networks based systems are better suited to LID tasks, since they are both simpler in design and provide a higher accuracy than traditional approaches.

2.1 Identity Vector Systems

Identity vector systems (i-vectors) have been introduced by Dehak et al. [4] for the purpose of speaker verfication tasks. i-vectors are a special form of joined lowdimensional representations of speaker and channel factors, as found in earlier joint factor analysis supervectors.

Identity vectors are employed as a data representation in many systems and are fed as inputs to a classifier. Dehak et al. [4] used Support Vector Machines (SVMs) with cosine kernels. Other researchers used logistic regression [11] or neural networks with three to four layers [7,13]. Gelly et al. [6] proposed a complex system consisting of phonotactic components [20], identity vectors, a lexical system, and Bidirectional Long-Short Term Memory (BLSTM) networks for language identification. The extensive feature engineering with i-vectors results in very complex systems, with an increasing number of computational steps in their pipeline.

2.2 Neural Network Approaches

Approaches solely based on applying neural networks on input features like MelFrequency Cepstral Coefficients (MFCC) show that they reach state-of-the-art results, while being less complex.

Current research on language identification systems using DNNs mainly focuses on using different forms of LSTMs, working on input sequences of transformed audio data. Zazo et al. [19] use Mel Frequency Cepstral Coefficients with Shifted Delta Coefficients (MFCC-SDC) features as input to their unidirectional LSTM, which is directly connected to a softmax classifier. The last prediction of the softmax classifier contains the predicted language. Gelly et al. [5] use a BLSTM network to capture language information from the input (audio converted to Perceptual Linear Prediction (PLP) coefficients and their first and second order derivatives) in forward and backward direction. The resulting sequence features are fused together and used to classify the language of the input samples. Both approaches only consider sequences of features as input to their networks.

Lozano-Diez et al. [10] perform language identification with the help of CNNs. The authors transform the input data to an image containing MFCC-SDC features. The x-axis of that image represents the time-domain and the y-axis describes the individual frequency bins. Besides plain classification of the input languages with the CNN, they also use the CNN as feature extractor for identity vectors. The authors achieve better performance when combining both the CNN features and identity vectors.

Our research differs from the mentioned works in the following way: (1) We utilize a strong convolutional feature extractor based on the VGG [17] or In-ception-v3 [18] architecture. (2) We use the extracted convolutional features as input to a BLSTM and generate our predictions solely based on a deep model.

In our work, we utilize the power of CNNs to capture spatial information, and the power of RNNs to capture information through a sequence of time steps for identifying the language from a given audio snippet. We developed a DNN based on a sequence recognition network presented by Shi et al. [16]. In this section, we present the datasets we used for training the network, the audio representation used for training our models, and the structure of our proposed network in detail.

3.1 Datasets

Since there are no large-scale, freely available datasets for LID tasks (datasets such as the NIST Language Recognition Evaluation are only available behind a paywall), we resorted to creating our own datasets for our experiments. We collected our datasets from two different sources: (1) we processed speeches, press conferences and statements from the European Parliament, and (2) we sourced data from news broadcast channels hosted on YouTube. We chose to collect data for 6 different languages, while making sure that we include languages with similar phonetics. Following this idea, we collected data for two Germanic languages (English and German), two Romance languages (French and Spanish), Russian, and Mandarin Chinese.

EU Speech Repository The EU Speech Repositoryis a collection of video re- sources for interpretation students. This dataset is provided for free and consists of debates of the European Parliament, as well as press conferences, interviews, and dedicated training materials from EU interpreters. Each audio clip is recorded in the speaker’s native language and features only one speaker. The dataset consists of many different female and male speakers. From this dataset we collected 131 hours of speech data in four languages: English, German, French and Spanish.

YouTube News Collection We chose to use news broadcasts as a second data source to obtain audio snippets of similar quality to the EU Speech Repository (different speakers, mostly one speaker at a time and a single defined language). We gathered all data from YouTube channels such as the official BBC NewsYouTube channel.

The obtained audio data has many desired properties. The quality of the audio recordings is very high and hundreds of hours are available online. News programs often feature guests or remote correspondents resulting in a good mix of different speakers. Further, news programs feature noise one would expect from a real-world situation: music jingles, nonspeech audio from video clips and transitions between reports. All in all, we were able to gather 1508 hours of audio data for this dataset.

3.2 Audio Representation

To make our gathered data compatible with our LID system, we need to do some preprocessing. As a first step, we encode all audio files in the uncompressed, lossless WAVE format, as this format allows for future manipulations without any deterioration in signal quality. In order to treat our audio snippets as images, we need to transfer the data into the image domain. We convert our audio data to spectrogram representations for training our models. The spectrograms are discretized using a Hann [2] window and 129 frequency bins along the frequency axis (y-axis). As most phonemes in the English language do not exceed 3 kHz in conversational speech, we only included frequencies of up to 5kHz in the spectrograms. The time axis (x-axis) is rendered at 50 pixels per second. We split each audio sequence into nonoverlapping ten-second segments and discard all segments shorter than ten seconds, as we did not want to introduce padding, which might resemble unnatural pauses or silence. The resulting images are saved as grayscale, lossless 500129 PNG files, where frequency intensities are mapped to an eight-bit grayscale range.

3.3 Architecture

For our network architecture, we followed the overall structure of the network proposed by Shi et al. [16] in their work on scene text recognition. This network architecture consists of two parts. The first part is a convolutional feature extractor that takes a spectrogam image representation of the audio file

Fig. 1: Our proposed CRNN network architecture consists of two parts. A CNN extracts local visual features from our input images. The output of the final convolutional layer is sliced along the time axis into t time steps. Each time step represents the extracted frequency features, used as input to the LSTM. The final LSTM output is fed into a fully-connected layer for classification.

as input (see section 3.2). This feature extractor convolves the input image in several steps and produces a feature map with a height of one. This feature map is sliced along the x-axis and each slice is used as a time step for the subsequent BLSTM network. The design of the convolutional feature extractor is based on the well known VGG architecture [17]. Our network uses five convolutional layers, where each layer is followed by the ReLU activation function [12], BatchNormalization [8] and 2 2 max pooling with a stride of 2. The kernel sizes and number of filters for each convolutional layer are (716), (532), (364), (3128), (3256), respectively. The BLSTM consists of two single LSTMs with 256 outputs units each. We concatenate both outputs to a vector of 512 dimensions and feed this into a fully-connected layer with 4/6 output units serving as the classifier. Figure 1 provides a schematic overview of the network architecture.

Using our gathered dataset and the network architecture introduced in section 3, we conducted several experiments, to assess the performance of our proposed network architecture on several kinds of input data. While performing our experiments we had a range of different questions in mind:

combines a CNN with a LSTM, compared to a CNN-only approach? – Is the network able to reliably discriminate between languages? – Is the network robust against different forms of noise in the input data? – Can the network easily be extended to handle other languages as well?

First, we shortly introduce our experimental environment and the metrics used. Then, we show our results on the EU Speech Repository and YouTube News dataset. Following this, we show the results of our experiments on noise robustness. We conclude this section with a discussion about the extensibility of our model to new languages.

Fig. 2: Performance of the proposed network architectures on the EU Speech Repository dataset

Fig. 3: Confusion matrix for our best performing CRNN, trained on the YouTube News dataset.

4.1 Environment

We implemented our proposed model using Keras [3] with the Tensorflow [1] backend. We splitted the datasets into a training (70 %), a validation (20 %) and a testing set (10 %), and all files were distributed equally between the languages. The European Speech dataset yields a total of about 19 000 training images, which amounts to roughly 53 hours of speech audio. The YouTube News dataset yields a total of about 194 000 training images, or 540 hours of speech audio. For training our networks, we used the Adam [9] optimizer and resorted to using stochastic gradient descent during fine-tuning. We observed the following metrics: accuracy, recall, precision and F1 score. We indicate the used networks in the following way: (1) CNN - A network only consisting of the proposed convolutional feature extractor without the recurrent part. (2) CRNN - The proposed hybrid CRNN model from section 3.3.

4.2 EU Speech Dataset

In order to verify our idea of applying CNNs for classifying image representations of audio data, we established a baseline with the smaller EU Speech Repository dataset. Figure 2 shows the results for the two network architectures (CNN and CRNN). As can be seen, the CRNN architecture outperforms the plain network without the recurrent part significantly. This proves our assumption that combining a recurrent network with a convolutional feature extractor increases the accuracy for spoken language identification.

4.3 YouTube News

After achieving these promising results, we trained our networks on the considerably larger YouTube News dataset. We used only the same four languages that are available in both datasets in order to have comparable results between

CNN fine-tuned on EU dataset CNN CRNN Inception-v3 CNN Inception-v3 CRNN

Fig. 4: Performance of the trained networks on the YouTube News dataset

these. First, we took the pre-trained CNN from our experiments on the EU Speech Repository dataset and fine-tuned this model with the data from the YouTube dataset. However, with an accuracy of 79 %, our CNN did not perform as accurately as expected. We argue that this is the case, because the EU dataset does not feature such a diverse range of situations, as the YouTube News dataset. Hence, we were unable to take advantage of the already trained convolutional features and the network did not converge on the new dataset.

We trained the CNN again, this time from scratch, and were able to achieve an accuracy of 90 % on this dataset. Our CRNN, using the trained CNN as feature extractor, is only able to increase the accuracy to 91 %. We argue that the CNN already learns to capture some time-related information from the input images.

As a next step, we evaluate how our proposed model architecture compares to a model with a stronger convolutional feature extractor. Therefore, we trained a model using Google’s Inception-v3 layout [18]. This network architecture is considerably deeper than our proposed CRNN architecture and hence should be able to extract more general features. With Inception-v3 we were able to achieve an accuracy of 95 % and 96 % using the CNN and CRNN, respectively. These results show that the deeper model is able to catch more general features on our large YouTube dataset. However, this increase in accuracy comes with an increase of computational cost, as the Inception-v3 model uses six times more parameters, than our initially proposed CNN.

Figure 3 shows the confusion matrix when evaluating language family pairs on our best-performing CRNN. Spanish and French separate very well with hardly any wrong classifications. German and English are more likely to be confused, but English has a slighty stronger bias towards French. All in all, the learned representations of the model are quite distinctive for each language.

Table 1: Accuracy and F1 Score measurements for our experiments on the YouTube News dataset with additional noise.

4.4 Noise Robustness

In our next series of experiments, we added three different forms of noise to our test data. We evaluated our already trained models to judge their robustness to these changed conditions. First, we mixed the audio signal with randomly generated white noise, which has a strong audible presence, but still retains the identifiability of the language. Second, we added a periodic crackling noise, emulating analog telephony or a bad voice chat connection. For the last experiment, we added background music from different genres to our samples.

We performed all experiments using our initially proposed CRNN and the Inception-v3 CRNN architectures. As expected we observed a decrease in accuracy and F1 score for both models, accross all experiments. The decrease in accuracy and F1 score is significantly higher for the initially proposed CRNN model, but relatively small for the Inception-v3 CRNN. We argue that this is the case because the Inception-v3 CRNN, with its deeper and more complex structure, is able to capture the frequency features in a more robust manner. Table 1 shows the results of our experiments with additional noise on the YouTube dataset.

4.5 Extensibility of the network

In our last experiment, we evaluated how well our model is able to expand its capabilities to also include new languages, and how adding further languages affects the overall quality of the results produced by the network. We extended the existing set of four languages by two further languages spoken by millions around the globe: Mandarin Chinese and Russian. First, we fine-tuned our best performing CNN based on our proposed architecture. The resulting model served as the basis for training the CRNN as described earlier. Applied on the test set, we measure an accuracy of 92 % and an F1 score of 0.92. Both measurements match our previous evaluation with four languages on the YouTube News dataset, proving that the proposed CRNN architecture can indeed easily be extended to cover more languages. Figure 5 shows individual performance measures for each language.

We note that Mandarin Chinese outperforms every other language with a top Accuracy of 96 %, which is likely due to the fact that the sound of Mandarin Chinese is very distinct compared to western languages. We also find that English

Fig. 5: Individual performance measurements for each of our six target languages. Chinese performs best, while English performs worst. Overall, the model performance is consistent with the results of earlier experiments reported in section 4.3.

is now the worst performing language, which is in part due to a significant number of misclassifications as Russian samples.

In summary, we are content to find that the features learned by our model are universal in nature, as both new languages are rooted within their own language families and feature considerably different intonations. We believe that our proposed approach to language identification can be successfully applied to a wide variety of languages.

In this paper we proposed a language identification system, that solves the language identification problem in the image domain, rather than the audio domain. We proposed a hybrid CRNN that consists of a convolutional feature extractor and a RNN that combines the extracted features over time. Using this architecture, we performed several experiments on different datasets to show the wide applicability of our model to various scenarios and its extensibility to new languages. In order to compensate for the lack of freely available datasets for language identification, we gathered more than 1508 hours of audio data from the EU Speech Repository and YouTube and offer them to the research community.

1. Abadi, M., Agarwal, A., Barham, P., Brevdo, E., et al.: Tensorflow: Large-scale ma- chine learning on heterogeneous distributed systems. arXiv:1603.04467 [cs] (2016)

2. Blackman, R.B., Tukey, J.W.: The measurement of power spectra from the point of view of communications engineering-part i. Bell Labs Technical Journal 37(1), 185–282 (1958)

3. Chollet, F.: keras: Deep learning library for python. runs on tensorflow, theano or cntk (2017), https://github.com/fchollet/keras

4. Dehak, N., Kenny, P.J., Dehak, R., Dumouchel, P., Ouellet, P.: Front-end factor analysis for speaker verification. IEEE Transactions on Audio, Speech, and Language Processing 19(4), 788–798 (2011)

5. Gelly, G., Gauvain, J.L., Le, V., Messaoudi, A.: A divide-and-conquer approach for language identification based on recurrent neural networks. Interspeech 2016 pp. 3231–3235 (2016)

6. Gelly, G., Gauvain, J.L., Lamel, L., Laurent, A., Le, V.B., Messaoudi, A.: Language recognition for dialects and closely related languages. Odyssey, Bilbao, Spain (2016)

7. Gonzalez-Dominguez, J., Lopez-Moreno, I., Moreno, P.J., Gonzalez-Rodriguez, J.: Frame-by-frame language identification in short utterances using deep neural networks. Neural Networks 64, 49–58 (2015)

8. Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: Proceedings of The 32nd International Conference on Machine Learning. pp. 448–456 (2015)

9. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: Proceedings of the 3rd International Conference on Learning Represenations. San Diego (2015)

10. Lozano-Dez, A., Zazo Candil, R., Gonzlez Domnguez, J., Toledano, D.T., Gonzlez- Rodrguez, J.: An end-to-end approach to language identification in short utterances using convolutional neural networks. International Speech and Communication Association (2015)

11. Martnez, D., Plchot, O., Burget, L., Glembek, O., Matjka, P.: Language recogni- tion in ivectors space. In: Twelfth Annual Conference of the International Speech Communication Association (2011)

12. Nair, V., Hinton, G.E.: Rectified linear units improve restricted boltzmann ma- chines. In: Proceedings of the 27th international conference on machine learning (ICML-10). pp. 807–814 (2010)

13. Plchot, O., Matejka, P., Glembek, O., Fer, R., Novotny, O., Pesan, J., Burget, L., Brummer, N., Cumani, S.: Bat system description for nist lre 2015. Odyssey 2016 pp. 166–173 (2016)

14. Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 779–788 (2016)

15. Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object de- tection with region proposal networks. In: Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M., Garnett, R. (eds.) Advances in Neural Information Processing Systems 28. pp. 91–99. Curran Associates, Inc. (2015)

16. Shi, B., Bai, X., Yao, C.: An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (2016)

17. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale im- age recognition. In: International Conference on Learning Representations (2015)

18. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the incep- tion architecture for computer vision. pp. 2818–2826 (2016)

19. Zazo, R., Lozano-Diez, A., Gonzalez-Dominguez, J., Toledano, D.T., Gonzalez- Rodriguez, J.: Language identification in short utterances using long short-term memory (lstm) recurrent neural networks. PLOS ONE 11(1), e0146917 (2016)

20. Zissman, M.A., et al.: Comparison of four approaches to automatic language iden- tification of telephone speech. IEEE Transactions on speech and audio processing 4(1), 31 (1996)