AUTOMATIC speech processing systems drastically im-proved the past few years, especially Automatic Speech Recognition (ASR) systems. It is also the case for other speech processing tasks such as speaker identification, emotion classification, etc. This success was made possible by the large amount of annotated data available combined with the extensive use of deep learning techniques and the capacity of modern Graphics Processing Units. Some models are already deployed for everyday usage such as your personal assistants on your smartphones, your connected speakers and so on.

Nevertheless, challenges remain for automatic speech processing systems. They lack robustness against large vocabulary in real-world environment: this includes noises, distance from the speaker, reverberations and other alterations. Some challenges, such as CHIME [1], provide data to let the community try to handle some of these problems. It is being investigated to improve the generalization of modern models by avoiding the inclusion of other annotated data for every possible environment.

State-Of-The-Art (SOTA) techniques for most speech tasks require large datasets. Indeed, with modern DNN speech processing systems, having more data usually imply better performances. The TED-LIUM 3 from [2] (with 452 hours) provide more than twice the data of the TED-LIUM 2 dataset. Doing so, they obtain better results by training their model on TED-LIUM 3 than training their model over TED-LIUM 2 data. This improvement in performance for ASR systems is also observed with the LibriSpeech dataset (from [3]). V. Panayotov et al. obtain better results on the Wall Street Journal (WSJ) test set by training a model over LibriSpeech dataset (1000 hours) than training a model over the WSJ training set (82 hours) [3].

This phenomenon, of having more data imply better performances, is also observable with the VoxCeleb 2 dataset compare to the VoxCeleb dataset: [4] increase the number of sentences from 100,000 utterances to one million utterances and increase the number of identities from 1251 to 6112 compared to the previous version of VoxCeleb. Doing so, they obtain better performances compare to training their model with the previous VoxCeleb dataset.

With under-resourced languages (such as [5]) and/or tasks (pathological detection with speech signals), we lack large datasets. By under-resourced, we mean limited digital resources (limited acoustic and text corpora) and/or a lack of linguistic expertise. For a more precise definition and details of the problem you may look [6]. Non-conventional speech tasks such as disease detection (such as Parkinson, gravity of ENT cancer and others) using audio are examples of tasks under resourced. Train Deep Neural Network models in such context is a challenge for these under-resourced speech datasets. This is especially the case for large vocabulary tasks. M. Moore et al. showed that recent ASR systems are not well adapted for impaired speech [7] and M. B. Mustafa et al. showed the difficulties to adapt such models with limited amount of data [8]. Few-shot learning consists of training a model using k-shot (where shot means an example per class), where and k is a low number. Training an ASR system on a new language, adapting an ASR system on pathological speech or doing a speaker identification with few examples are still complicated tasks. We think that few-shot techniques may be useful to tackle these problems.

This survey will be focused on how to learn Deep Neural Network (DNN) models under low resources for speech data with non-overlapping mono signals. Therefore, we will first review SOTA ASR techniques that use a large amount of data (section II). Then we will review techniques and speech tasks (speaker identification, emotion recognition) requiring fewer data than SOTA techniques (section III). We will also look into pathological speech processing for ASR using adaptation techniques (subsection III-B). Finally, we will review few-shot techniques for audio (section IV) which is the focus of this survey.

In this section, we will review SOTA ASR systems using multi-models and end-to-end models. Here, we are focused on mono speech sequences where can be speech features or audio samples. ASR systems consist in matching x into a sequence of words (where ). The systems reviewed were evaluated using Word Error Rate (WER) measure.

A. Multi-models

A multi-model approach consists in solving a problem using multiple models. Those models are designed to solve either sub-tasks (related to the problem) and the targeted task. The minimum configuration is with two models (let say f and g) to solve a given task. Classically for the ASR task we can first learn an acoustic model (a phoneme classifier or equivalent sound units), then learn on top of it a language model that output the desired sequence of words. Hence, we have:

with f being the language model and g being the acoustic model. Both can be learned separately or conjointly. Usually, hybrid models are used as acoustic models.

Hybrid models consist in using probabilistic models with deterministic ones. Probabilistic models involve randomness using random variables combined with trained parameters. Hence, every prediction is sightly different on a given example x. Gaussian Mixture Models (GMMs) are an example of such models. Deterministic models do not involve randomness and every prediction are the same given an input x. DNNs are an example of such models. A popular and efficient hybrid model is the DNN-Hidden Markov Model (DNN-HMM). DNN-HMM consists in replacing the GMMs that estimate the probability density functions by DNNs. The DNNs can be learned as phone classifiers. They form the acoustic model. This acoustic model is combined with a Language Model (LM) that maps the phonemes into a sequence of words. C. L¨uscher et al. used DNN-HMMs combined with a Language Model to obtain SOTA on LibriSpeech test-other set (official augmented test set) [9]. This model process MFCC computed on the audio signals. Their best LM approach consisted in the use of Transformer from [10]. Transformers are autoregressive models (depending on the previous outputs of the models) using soft attention mechanisms. Soft attention consists in determining a glimpse g over all possible glimpses such as:

with x being the input data and a the attention parameters. Their best hybrid model got a Word Error Rate (WER) of 5.7% for the test-other set and a WER of 2.7% for test-clean set.

B. End-to-end systems

In end-to-end approaches, the goal is to determine a model f that can do the mapping:

It will be learned straightforward from the x to the desired y. Only supervised methods can be end-to-end to solve the speech tasks we are focused on.

In ASR systems, [11] got SOTA on LibriSpeech test-clean official set. Compared to [9] they used Vocal Tract Length Perturbation as the input of their end-to-end model. C. Kim et al. model is based on the Encoder-Decoder architecture using stacked LSTM for the encoder and LSTM combined with soft attention for the decoder [11]. They obtain a WER of 2.44% on test-clean and a WER of 8.29% on test-other. Those results are close to [9] (best hybrid model results) and show that end-to-end approaches are competitive compared to multi-model approaches.

Some techniques require fewer data than the techniques of the previous section. In this section we will enumerate the principal ways to leverage (to our best knowledge) the lack of large datasets like unimpaired speech. We will also look into tasks requiring fewer data (speaker identification and emotion recognition). We will not talk of semi-supervised techniques that use a large amount of unsupervised data.

A. Data augmentation

The first way to leverage the lack of data is to artificially augment the number of data. To do so, classic approach consists for example in adding noise or deformation. Such as in [12]. They obtain near SOTA on Librispeech (1000 hours from [3]) with an end-to-end models. Nevertheless, they obtain SOTA results on SwitchBoard (300 hours from [13]) with a WER of 6.8%/14.1% on the Switchboard/CallHome portion using shallow fusion and their data augmentation. But theses are handcrafted augmentations and some of them require additional audios (like adding noise).

Some other approaches use generative models to have new samples such as in [14], [15]. A. Chatziagapi et al. used conditional Generative Adversarial Networks (GAN) to generate new samples [14]. Conditioned GAN are GAN where we can control the mode of the generated samples. Doing so, they balanced their initial dataset and obtain better results. Y. Jiao et al. used Deep Convolutional GANs to generate dysarthric speech and improve their results [15].

B. Domain transposition

Another way to leverage the lack of data is to use domain transposition to avoid complex domain, here is some recent examples on speech:

• K. Wang et al. used GAN to dereverberate speech signal [16]. In their work, the generator is used as a mapping function of reverberated signals into dereverberated speech signals.

• L.-W. Chen et al. do vocal conversion using GAN with a controller mapping impaired speech to a representation space z [17]. z is then the input of the generator that is used as a mapping function to have unimpaired speech signals.

• S. Zhao et al. used Cycle GAN (framework designed for domain transfer) as an audio enhancer [18]. Their resulting model is SOTA on Chime-4 dataset.

C. Models requiring fewer parameters

Having fewer data disallow the use of many parameters for Neural Network models to avoid overfitting. This is why some techniques tried to have models requiring fewer parameters. Here, we highlight some recent techniques that we find interesting:

• The use of SincNet, from [19], layers to replace classic 1D convolutions over raw audio. Here, instead of requiring window size parameters (with window size being the window size of the 1D convolution) per filter, we only need two parameters per filter for every window size. Theses two parameters represent in a way (not directly) the values of the bandwidth at high and low energy.

• The use of LightGRU (LiGRU), from [20], based on the Gated Recurrent Unit (GRU) framework. LiGRU is a simplification of the GRU framework given some assumption in audio. They removed the reset gate of the GRU and used the ReLU activation function (combined with the Batch Normalization) instead of the tanh activation function.

• The use of quaternions Neural Networks, from [21], for speech processing. The quaternion formulation allows the fuse of 4 dimensions into one inducing a drastic reduction of required parameters in their experiments (near 4 times).

D. Multi-task approach

Multi-task models can be viewed as an extension of the Encoder-Decoder architecture where you have a decoder per task with a shared encoder (like in Figure 1). Then those tasks are trained conjointly with classic feed-forward algorithms. The goal of a multi-task learning is to have an encoder outputting sufficient information for every task. Doing so, it can potentially improve the performances of each task compared to mono task architectures. It is a way to have a more representative encoder given the same amount of data.

In emotion recognition, [22] got SOTA results over a modified version of the IEMOCAP database to have a fourclass problem. Those emotions are: angry, happy, neutral and sad. Y. Li et al. used an end-to-end multi-task system with only supervised tasks: gender identification and emotion identification [22]. The resulting model achieve an overall accuracy for the emotion task (which is the main target) of 81.6% and an average accuracy of each emotion category of 82.8%. Using such approach allows them to achieve balanced results over unbalanced data.

Nevertheless, using only supervised tasks requires multiple ground-truth for the targeted dataset. S. Pascual et al. used

Figure 1. Multi-task architecture illustration. The output of the encoder is given to each decoder to have the prediction for each

a combination of self-supervised tasks combined with unsupervised tasks to tackle this problem and used the resulting encoder for transfer learning [23]. They recently improved this work in [24] where they use more tasks, a recurrent unit on top of the encoder and denoising mechanisms using multiple data augmentation on their system.

E. Transfer Learning

Transfer learning techniques consist of using a pre-trained model and transfer its knowledge to solve a related problem/task. Usually we use the encoding part of the pre-trained model to initialize the model for the new problem/task.

Contrastive Predictive Coding (CPC from [25]) is an architecture to learn unsupervised audio representation using a 2-level architecture combined with a self-supervised loss. They achieved good results by transferring the obtained model for speaker identification and phone classification (on LibriSpeech dataset) compared to MFCC features.

This work inspired [23]. They developed an unsupervised multi-task model (with certain losses being self-supervised) to obtain better encoders for transfer learning. They applied it on multiple tasks and obtain decent results on speaker identifica-tion (using VTCK), emotion recognition (using INTERFACE) and ASR (using TIMIT).

The benefit of pre-trained network for transfer learning decrease as the target task diverges from the original task of the pre-trained network [26]. To tackle this, [25], [23] attempt to have generic tasks with their unsupervised approach, and they obtained promising results. Also, the benefit of transfer learning decrease when the dissimilarity between the datasets increase [26]. This problem can discourage the use of transfer learning for some pathological speech. Whereas, Dysarthric and Accented Speech seems similar to speech in librispeech dataset according to [27]. Where they successfully used transfer learning to improve their results over a 36.7 hours dataset.

Nevertheless, [8] showed that acoustic characteristics of unimpaired and impaired speech are very different. In the case of having few data such problems can be critical. It is why looking into few-shot techniques could be helpful.

In the previous sections, we reviewed models that require a large amount of data. This among of data is not always available such as for pathological speech. Google is trying to acquire more data of that nature1. But acquiring such data can be quite expensive and time consuming. M. B. Mustafa et al. recommend the use of adaptive techniques to tackle limited amount of data problem in such case [8]. But we think few-shot technique can be an other solution to this problem. Nevertheless, some non-common tasks such as pathological or dialect identification with few examples are still hard to train with SOTA techniques based on large speech datasets. This is why we investigate the following few-shot techniques and see the adaptations required for using them on speech datasets.

A. Few-shot Notations

Let consider a distribution P from which we draw Independent Identically Distributed (iid) episodes E, where E is composed of a support set S, unlabeled data and a query set Q. Support set correspond to the supervised samples the model has access to:

with being samples and being the corresponding labels such as being the number of classes appearing in P. The query set is composed of samples to classify with being the corresponding ground truth.

To summarize, episodes drawn from P have the following form:

with s, r and t fixed values that respectively represent the number of supervised samples for the support set, the number of unsupervised samples and the number of supervised samples for the query set.

In this survey, we will focus on Few-Shot Learning techniques where and s = kn, with n being the number of times each label appears for the support set and k the number of classes selected from P, such as . Hence, we have a n-shot with k ways (or classes) for each episode. One-shot learning is just a special case of few-shot learning where n = 1. In some few-shot framework, we only sample one episode from P and it represents our task.

B. Few-shot learning techniques

In this section we will review frameworks that impacted the few-shot learning field in image processing, frameworks with a formulation that seems adapted for speech processing and frameworks already successfully used by the speech community.

1) Siamese technique

Siamese Neural Networks are designed to be used per episode [28]. They consist of measuring the distance between two samples and tell if they are similar or not. Hence, Siamese network uses the samples from the support set S as references for each class. It is then trained using all the combinations of samples from which represent much more training than having only s+t samples in classical feedforward frameworks. Siamese Networks take two samples (and ) as input and compute a distance between them, as follows:

with Enc being a DNN encoder that represents the signal input, being the sigmoid function, learnable parameters that weight the importance of each component of the encoder and and sampled from either the support set nor the queries set.

To define the class of a new sample from Q or any new data, we have to compute the distance between each reference from S and the new sample. An example of comparison between a reference and a new example is shown in Figure 2. Then, the class of the reference with the lowest distance become the prediction of the model. To learn such model, [28] used this loss function:

with from S and Q. y(x) is a function that returns the label corresponding to the example x. Also, last layer should be a softmax.

Figure 2. Example of comparison between a reference () and a new example () from the query set. Where Enc is the same network applied to both and . The model output the distance between

R. Eloff et al. used a modified version of this framework for Multimodal Learning (framework that is out of scope for this survey) between speech and image signal [29]. The speech signals used consist of 11-digit number (zero to nine and oh) with the corresponding 10 images (oh and zero give the same images). The problem is to associate speech signals with the corresponding image. In their experiment, the model shows some invariances to speakers (accuracy of 70.12% 0.68) using only a one-shot configuration, which is promising results.

Siamese Neural Networks are not well adapted when the number of classes K or the number of shots q become too high. It increases the number of references to compare and the computation time to forward the model. It is mostly a problem for learning the model. After the model is learned, we can pre-calculate all representations for the support set to reduce this effect. Also, it drastically increases the number of combinations to do for training, this can be viewed as a positive point as we can truncate the number of combinations to use for training the model. This framework seems not adapted for end-to-end ASR with large vocabulary such as in the English speech (around 470,000 words). Maybe it will be sufficient for languages such as Esperanto language (around 16,780 words). The other way to use such a framework in ASR systems is to use it in hybrid models as an acoustic model. Where we can learn it on every phoneme (for example 44 phonemes/sounds in English) or more refined sound units.

Siamese framework seems interesting for tasks such as speaker identification. Indeed, this framework allows adding new speaker without retraining the model (supposing the model had generalized) or change the architecture of the model. We have to at least add one example of the new speaker to the references. Furthermore, Siamese formulation seems well adapted for speaker verification. Indeed, by replacing the pair (x, speaker id) by the pair we can do speaker verification with such technique. Where is a support set composed of signals from the 5 top predictions of the identification sub-task.

Nevertheless, this framework will be limited if the number of speakers to identify become too high. Even so, it is possible to use such techniques in an end-to-end ASR system when the vocabulary is limited, such as in [29] experiment.

2) Matching Network

Matching Networks from [30] is a few-shot framework designed to be trained on multiple episodes. This framework is composed of one model . This model is trained over a set of training episodes (with typically 5 to 25 ways). This model evaluates new examples given the support set S like in the Siamese framework:

In matching learning, is as follows:

with, a being the attention kernel.

In [30] this attention kernel is as follows:

where c is the cosine distance, f and g are embedding functions.

O. Vinyals et al. used a recurrent architecture to modulate the representation of f using the support set S [30]. The goal is to have f following the same type of representation of g. To do this, g function is as follows:

where and represent a bi-LSTM output over which is a DNN.

f function is as follows:

with, attLST M being an LSTM with a fixed number of recurrences to do (here m), g(S) represents the application of g to each from the S set. is a DNN with the same architecture as , but not necessarily share the parameter values. Hence, training this framework consists in the maximization of the log likelihood of given the parameters of g and f.

Figure 3 illustrates forward time of the Matching Network model. For forward time on new samples g(S) can be pre-calculated to gain computation time. Nevertheless, as for Siamese networks, Matching networks have the same disadvantages when q and/or K become too high. Furthermore, adding new classes to a trained Matching Network model is not as easy as for Siamese Network models. Indeed, it requires retraining the Matching Network model to add an element to the support set. Whereas, Matching learning showed better results than the Siamese framework on image datasets from [30] experiments. It is why it should be investigated in speech processing to see if it is still the case.

Figure 3. Illustration of the Matching Network model to predict class of a new example

3) Prototypical Networks

Prototypical Networks [31] are designed to work with multiple episodes. In the prototypical framework, the model does its predictions given the support set S of an episode such as the previously seen frameworks. This framework uses training episodes as mini-batches to obtain the final model. This model is formulated as follows:

where is the prototype of the class k, d being a Bregman divergence (for their useful properties in optimization, see [31] for more details) that also follow this property: .

J. Snell et al. used the Euclidean distance for d instead of the cosine distance used in Meta Learning and Matching Learning papers [31]. Doing so, they obtain better results in their experiments. Next, they go further by reducing the Euclidean to a linear function.

In the prototypical framework, there is only one prototype for each class k as illustred in Figure. 4. It is computed such as:

with f being a mapping function such as and being the samples with k of the support set.

Compared to Siamese and Matching Learning Networks, prototypical networks require only one comparison per class and not q per class for q-shot learning like in Siamese and Matching Learning Networks. It is why this framework is less subject to the high computation problem for prediction of new samples as it is only influenced by high K. It will certainly be insufficient for end to end ASR systems on English language but it is a step forward to it.

Figure 4. Illustration of the Prototypical Network model to predict class of a new example

4) Meta-Learning Meta-learning [32] are designed to be learned on multiple episodes (also called datasets). In this framework a trainee model (T ) with parameters is trained for every episode from the start of every episode. It usually has a classic DNN architecture. The support set and the query set in the episodes are considered as the training set and the test set for the trainee model. Along with this trainee model, a second model is learned: the meta model (M) with parameters . This meta model is the key of meta learning, it consists in monitoring the trainee model by updating parameters. To learn this meta model, sampling iid episodes from P to form the meta-dataset (D) is suggested in [32]. This meta-dataset is composed of a training set (), a validation set () and a testing set (). While the trainee model is training on an episode , the meta model is charged to update its parameters:

with being the loss function of the trainee model learned over the episode and are the parameters of the trainee model at step . Also, M has to guess initial weights of the trainee models at step ). The learning curve (loss) of the trainee model over is viewed in [32] as a sequence that can be the input of the meta model M. For simplicity, we will use the notation of T instead of for the next paragraphs. Figure 5 illustrate the learning steps of the trainee using the meta model.

a) Trainee parameters update: S. Ravi and H. Larochelle identify the learning process of T using classic feedforward update on episode to be similar with the update gate of the LSTM framework [32]. In the meta learning framework, the update gate of the LSTM framework is then used as the estimator, such as:

with being the update term of the parameters being the forget gate and the update gate. b) Parameters of the meta model: Both and are part of the Meta learner. In the meta-learning framework, the update gate is formulated as follows:

with the and being parameters of M. The update gate is used to control update term in 16 like the learning rate in classic feedforward approach.

Next, the forget gate in the meta-learning framework is formulated as follows:

with and parameters of M. This gate is here to decide whether the learning of the trainee should restart or not. This can be useful to get out of a sub-optimal local minimum. Note that this gate is not present in classic feedforward approaches (where this gate is equal to one).

Figure 5. Meta-Learning illustration for training over episode Here the Meta model M process the different training step of the trainee T as a sequence.

The trainee model (T ) of this framework can be any kind of model such as a Siamese Neural Network. Hence, it can have the advantages of this framework. It also can avoid the Siamese neural network disadvantages as it can use any other framework (usually classic DNN). This framework is interesting for speech processing to learn efficient models (in terms of learning speed) when we have multiple ASR tasks with different vocabulary. For example, let say we have these kinds of speech episodes: dialing numbers, commands to a robot A and commands to a robot B. The model can initialize good filters for the first layers (as it is still speech processing). Another example could be learning acoustic models for multiple languages (with each episode corresponding to a language).

5) Graph neural network

The use of Graph Neural Network (GNN) is used by V. Garcia and J. Bruna introduce the use of Graph Neural Network (GNN) in their few-shot framework [33]. This framework is designed to be used with multiple episodes they called tasks. In this framework, one model is used over a complete graph G. G = (V, E) where every node corresponds to an example. GNN for few-shot learning consists in applying Graph Convolutions Layers over the graph G.

Initial vertices construction to guess the ground truth of a query from the query set Q:

V Enc, h, . . . , (Enc, h, (Enc, u), . . . , (Enc, u)

where Enc is an embedding extraction function (a Neural Network or any classic feature extraction technique), h the one-hot encoding function and an uniform distribution for examples with unknown labels (the unsupervised ones from and/or from the query set Q).

From now the vertices at each layer l (with 0 being the initial vertices) will be denoted:

where n = s + r + 1 and .

Every layers in GNN are computed as follows:

with being the adjacency operators constructed from and Gc being the graph convolution. a) The adjacency operators construction: The adjacency operator us a set:

with being the adjacency matrix of .

For every (recall we have complete graphs), we compute the values of the adjacency matrix such as:

with f being a multi-layer perceptron with its parameter denoted . is then normalized using the softmax function over each line.

Figure 6. Illustration of the input of the first layer (or Graph Convolution) of a GNN. Here we have three samples (represented by vertices ) in the support set and one query (represented by the vertex

b) Graph convolution: The graph convolution requires the construction of the adjacency operators set and is computed as follows:

with B being an adjacency operator from learnable parameters and being a point wise linearity (usually leaky ReLU). c) Training the model: The output of the resulting GNN model is a mapping of the vertices to a K-simplex that give the probability of being in class k. V. Garcia and J. Bruna used the cross-entropy to learn the model other all examples in the query set Q [33]. Hence, the GNN few-shot framework consists in learning and parameters over all episodes. d) Few-shot GNN on audio: This framework was used by [34] on 5-way audio classification problems. The 5 ways episodes are randomly selected from the initial dataset: AudioSet [35] for creating the 5-ways training episodes and [36] data to create the 5-ways test episodes.

S. Zhang et al. compare the use of per class attention (or intra-class) and global attention which gave the best results [34]. They applied it for each layer. Their experiments were done for 1-shot, 5-shots and 10-shots with the respective accuracy of and . Such results really motivate us in the path of few-shot learning for speech signals. Nevertheless, this framework does not allow the use of many classes and shots per episode which increase the number of nodes and thus the computations in forward time. Hence, it is not suited for large vocabulary problems.

In this survey, we investigated few-shot techniques for speech usage. In order to do so, we started with state-of-the-art speech processing systems. These systems require a large amount of data and are not suited for under-resourced speech problems. We also looked into techniques requiring fewer data using data augmentation, domain transposition, models requiring fewer parameters, multi-task approach and transfer learning. Nevertheless, these techniques are less efficient in a data-limited context. Next, we studied few-shot techniques and how well the different frameworks are adapted for classical speech tasks.

The main drawback of the reviewed techniques is the amount of computation required for large datasets (like LibriSpeech from [3]) compared to SOTA models we reviewed in section II. Nevertheless, we considered some recent works already using few-shot techniques on speech with promising results. Such techniques seem useful for classical speech tasks on impaired speakers. Moreover, we think it can be useful for unconventional speech tasks like measuring the intelligibility of a person (with impaired or unimpaired speakers) to help the re-education process (by identifying the problems faster). Acquiring a large amount of data is painful for some patients (with severe pathologies). We believe that few-shot techniques may help the community to tackle this problem. To see the interest of such techniques we will work on a benchmark for different speech tasks. We will do some adaptations when necessary, but we think that we can use the different frameworks straightforward. After that, we plan to use the technique with the best results on this benchmark as a base for learning the concept of intelligibility.

[1] J. Barker, S. Watanabe, E. Vincent, and J. Trmal, “The Fifth ’CHiME’ Speech Separation and Recognition Challenge: Dataset, Task and Baselines,” in Interspeech 2018. ISCA, Sep. 2018, pp. 1561–1565.

[2] F. Hernandez, V. Nguyen, S. Ghannay, N. Tomashenko, and Y. Est`eve, “TED-LIUM 3: Twice as much data and corpus repartition for experiments on speaker adaptation,” in Speech and Computer - 20th International Conference, vol. 11096, Sep. 2018, pp. 198–208.

[3] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An ASR corpus based on public domain audio books,” in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). South Brisbane, Queensland, Australia: IEEE, Apr. 2015, pp. 5206–5210.

[4] J. S. Chung, A. Nagrani, and A. Zisserman, “VoxCeleb2: Deep Speaker Recognition,” in Interspeech 2018. ISCA, Sep. 2018, pp. 1086–1090.

[5] B. Deka, J. Chakraborty, A. Dey, S. Nath, P. Sarmah, S. R. Nirmala, and S. Vijaya, “Speech corpora of under resourced languages of northeast india,” in 2018 Oriental COCOSDA - International Conference on Speech Database and Assessments, Miyazaki, Japan, May 7-8, 2018, 2018, pp. 72–77.

[6] L. Besacier, E. Barnard, A. Karpov, and T. Schultz, “Automatic speech recognition for under-resourced languages: A survey,” Speech Communication, vol. 56, pp. 85–100, Jan. 2014.

[7] M. Moore, H. Venkateswara, and S. Panchanathan, “Whistle-blowing ASRs: Evaluating the Need for More Inclusive Speech Recognition Systems,” in Interspeech 2018. ISCA, Sep. 2018, pp. 466–470.

[8] M. B. Mustafa, S. S. Salim, N. Mohamed, B. Al-Qatab, and C. E. Siong, “Severity-Based Adaptation with Limited Data for ASR to Aid Dysarthric Speakers,” PLoS ONE, vol. 9, no. 1, p. e86285, Jan. 2014.

[9] C. L¨uscher, E. Beck, K. Irie, M. Kitza, W. Michel, A. Zeyer, R. Schl¨uter, and H. Ney, “RWTH ASR Systems for LibriSpeech: Hybrid vs Attention,” in Interspeech 2019. ISCA, Sep. 2019, pp. 231–235.

[10] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is All you Need,” in Advances in Neural Information Processing Systems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds. Curran Associates, Inc., 2017, pp. 5998–6008.

[11] C. Kim, M. Shin, A. Garg, and D. Gowda, “Improved Vocal Tract Length Perturbation for a State-of-the-Art End-to-End Speech Recognition System,” in Interspeech 2019. ISCA, Sep. 2019, pp. 739–743.

[12] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition,” Interspeech 2019, pp. 2613–2617, Sep. 2019.

[13] J. J. Godfrey, E. C. Holliman, and J. McDaniel, “SWITCHBOARD: Telephone speech corpus for research and development,” in [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 1. IEEE, 1992, pp. 517–520.

[14] A. Chatziagapi, G. Paraskevopoulos, D. Sgouropoulos, G. Pantazopou- los, M. Nikandrou, T. Giannakopoulos, A. Katsamanis, A. Potamianos, and S. Narayanan, “Data Augmentation Using GANs for Speech Emotion Recognition,” in Interspeech 2019. ISCA, Sep. 2019, pp. 171–175.

[15] Y. Jiao, M. Tu, V. Berisha, and J. Liss, “Simulating dysarthric speech for training data augmentation in clinical speech applications,” in IEEE International Conference on Acoustics, Speech and Signal Processing, Apr. 2018.

[16] K. Wang, J. Zhang, S. Sun, Y. Wang, F. Xiang, and L. Xie, “Investigating Generative Adversarial Networks based Speech Dereverberation for Robust Speech Recognition,” Interspeech 2018, pp. 1581–1585, Sep. 2018.

[17] L.-W. Chen, H.-Y. Lee, and Y. Tsao, “Generative Adversarial Networks for Unpaired Voice Transformation on Impaired Speech,” in Interspeech 2019. ISCA, Sep. 2019, pp. 719–723.

[18] S. Zhao, C. Ni, R. Tong, and B. Ma, “Multi-Task Multi-Network Joint- Learning of Deep Residual Networks and Cycle-Consistency Generative Adversarial Networks for Robust Speech Recognition,” in Interspeech 2019. ISCA, Sep. 2019, pp. 1238–1242.

[19] M. Ravanelli and Y. Bengio, “Interpretable Convolutional Filters with SincNet,” in NIPS 2018 Workshop IRASL, Nov. 2018.

[20] M. Ravanelli, P. Brakel, M. Omologo, and Y. Bengio, “Light Gated Re- current Units for Speech Recognition,” IEEE Transactions on Emerging Topics in Computational Intelligence, vol. 2, no. 2, pp. 92–102, Apr. 2018.

[21] T. Parcollet, M. Ravanelli, M. Morchid, G. Linar`es, and R. De Mori, “Speech recognition with quaternion neural networks,” in NeurIPS 2018 - IRASL, Nov. 2018.

[22] Y. Li, T. Zhao, and T. Kawahara, “Improved End-to-End Speech Emotion Recognition Using Self Attention Mechanism and Multitask Learning,” in Interspeech 2019. ISCA, Sep. 2019, pp. 2803–2807.

[23] S. Pascual, M. Ravanelli, J. Serr`a, A. Bonafonte, and Y. Bengio, “Learning Problem-Agnostic Speech Representations from Multiple Self-Supervised Tasks,” in Interspeech 2019. ISCA, Sep. 2019, pp. 161–165.

[24] M. Ravanelli, J. Zhong, S. Pascual, P. Swietojanski, J. Monteiro, J. Trmal, and Y. Bengio, “Multi-task self-supervised learning for Robust Speech Recognition,” arXiv:2001.09239 [cs, eess], Jan. 2020.

[25] A. van den Oord, Y. Li, and O. Vinyals, “Representation Learning with Contrastive Predictive Coding,” CoRR, Aug. 2018.

[26] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, “How transferable are features in deep neural networks?” in Advances in Neural Information Processing Systems 27, Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, Eds. Curran Associates, Inc., 2014, pp. 3320–3328.

[27] J. Shor, D. Emanuel, O. Lang, O. Tuval, M. Brenner, J. Cattiau, F. Vieira, M. McNally, T. Charbonneau, M. Nollstadt, A. Hassidim, and Y. Matias, “Personalizing ASR for Dysarthric and Accented Speech with Limited Data,” in Interspeech 2019. ISCA, Sep. 2019, pp. 784–788.

[28] G. Koch, R. Zemel, and R. Salakhutdinov, “Siamese Neural Networks for One-shot Image Recognition,” ICML Deep Learning Workshop, p. 8, 2015.

[29] R. Eloff, H. A. Engelbrecht, and H. Kamper, “Multimodal One-shot Learning of Speech and Images,” in ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2019, pp. 8623–8627.

[30] O. Vinyals, C. Blundell, T. Lillicrap, k. kavukcuoglu, and D. Wierstra, “Matching Networks for One Shot Learning,” in Advances in Neural Information Processing Systems 29, D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, Eds. Curran Associates, Inc., 2016, pp. 3630–3638.

[31] J. Snell, K. Swersky, and R. Zemel, “Prototypical Networks for Few- shot Learning,” in Advances in Neural Information Processing Systems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds. Curran Associates, Inc., 2017, pp. 4077–4087.

[32] S. Ravi and H. Larochelle, “Optimization as a Model for Few-Shot Learning,” in ICLR 2017, 2017, p. 11.

[33] V. Garcia and J. Bruna, “Few-Shot Learning with Graph Neural Net- works,” in ICLR 2018, 2018, p. 13.

[34] S. Zhang, Y. Qin, K. Sun, and Y. Lin, “Few-Shot Audio Classification with Attentional Graph Neural Networks,” in Interspeech 2019. ISCA, Sep. 2019, pp. 3649–3653.

[35] J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio Set: An ontology and human-labeled dataset for audio events,” in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). New Orleans, LA: IEEE, Mar. 2017, pp. 776–780.

[36] S. Zhang, H. Jiang, S. Zhang, and B. Xu, “Fast SVM Training Based on the Choice of Effective Samples for Audio Classification,” INTERSPEECH 2006 - ICSLP, p. 4, 2006.

Vincent Roger doctorate is founded by Federal University of Toulouse and Occitanie Region no2018-1290 (ALDOCT no500). This work is part of the ANR-18-CE45-0008 RUGBI project founded by French National Research Agency.