The human visual system is efficient at grasping the main concepts of any new image from only a single or a few images. Over the last few years, few-shot learning techniques have been developed by many researchers to achieve human-level performance on image recognition tasks. Generally, it is expected that a “good” few-shot learning technique should satisfy properties such as the following: (i) it is able to learn new tasks with few-shot examples efficiently, thus learning the new categories fast; (ii) the performance can be improved even as increasing numbers of input samples are given on a new task; (iii) performance on the initial tasks at training time is not sacrificed (without forgetting).
Although many few-shot classification algorithms are proposed, it is a tough task to organize the best unified framework for few-shot learning. Metric learning methods [Vinyals et al., 2016; Snell et al., 2017; Sung et al., 2018] aim to learn a data-dependent metric to reduce intraclass distance and increase inter-class distances. Gradientbased meta-learning [Kim et al., 2018; Finn et al., 2017; Snell et al., 2017] attempts to learn the commonalities among various tasks. MAML [Finn et al., 2017] is an effective meta-learning method that directly optimizes the gradient descent procedure for task-specific learners. In the amortized Bayesian inference framework, [Qiao et al., 2018; Ravi and Beatson, 2019] proposed a method for predicting the weights of classes from activations of a pre-trained network to transfer from a high-shot classification task to a separate lowshot classification task. Recently, [Gordon et al., 2019] proposed a general meta-learning framework (ML-PIP) with approximate probabilistic inference and its implementation to few shot learning tasks (VERSA). ML-PIP unified a number of important approaches on meta-learning, including both gradient- and metric-based meta-learning [Kim et al., 2018; Finn et al., 2017; Snell et al., 2017; Qiao et al., 2018; Ravi and Beatson, 2019] with amortized inference frameworks (neural processes) [Garnelo et al., 2018a; Garnelo et al., 2018b]. It is a general framework because of the end-to-end training and supports full multi-task learning by sharing information between many tasks. In particular, VERSA replaces the optimization at test time with efficient posterior inference by generating a distribution over the task-specific parameters in a single forward pass. Therefore, this framework can amortize the cost of inference and relieve the need for second derivatives for few-shot training during test time. It is also worth noting that their inference framework is focused on the posterior predictive distribution, i.e., it aims to minimize the KL-divergence between the true and approximate predictive distributions rather than maximizing the ELBO, which is generally utilized in VAE-based methods [Kingma and Welling, 2014].
In the state-of-the-art models on few-shot learning tasks, amortized inference distribution is practically utilized because it is efficient and scalable to large datasets, and it requires only the specified parameters of the neural network. However, to get proper amortized inference, we need to tackle the amortization gap problem and information preference problem as stated below. As analyzed in [Cremer et al., 2018], the inference mismatch between the true and approximate posterior which consists of two gaps (i) approximation gap and (ii) amortization gap. Their conclusions are that increasing the capacity of the encoder reduces the amortization error and when efficient test time inference is required, encoder generalization is important and expressive approximations in decoder are likely advantageous. Another example of the estimation difficulty of amortized inference is that cosine-similarity-based non-amortization models [Chen et al., 2019] achieved superior performance than those with amortization inference on few-shot learning. This implies that effective estimation methodology for amortization inference has still not been established.
Our contributions in this paper are as follows:
1. We show that one of the amortization gap problems comes from the information preference problem of the latent distribution.
2. We adapt both the annealing method and regularization of parameter estimation in the amortized inference network to avoid the information preference problem by applying cyclical annealing schedule and maximum mean discrepancy.
3. Our proposal meets the “good” properties of few-shot learning, get better performance on standard few-shot classification tasks.
Despite its simplicity of our proposed method, it can significantly improve the performance. Through several experimental analyses, we show that our methodology outperforms other state-of-the-art few-shot learning algorithms.
Along with the many few-shot learning methods, a number of measures for assessing their actual performance has also being proposed. The ML-PIP model unified a number of important approaches on meta-learning, including both gradient and metric based meta-learning [Kim et al., 2018; Finn et al., 2017; Snell et al., 2017; Qiao et al., 2018] with amortized inference framework [Garnelo et al., 2018a; Gar- nelo et al., 2018b]. Although their method is similar to these models, it is more general, employing end-to-end training and supporting full multi-task learning by sharing information between many tasks. In this section, we describe the multi-task meta-learning problem that we deal with in this paper, and we review the VERSA (implementation of ML-PIP for meta-learning) and neural processes (NPs) [Gordon et al., 2019; Garnelo et al., 2018b].
2.1 Meta-learning problem
In this paper, we mainly consider few-shot classification problems, in which we are given few-shot (say, k-shot) observations consisting of input-output pairs for each of the C-classes (we call C-way), and we perform C-class classification for an unseen test input data. We call this problem C-way k-shot meta-learning problem. One typical approach to tackle this problem is to construct an “encoder”
beforehand, estimate a weight vector
from the few-shot observations and apply the softmax operation for the linear discriminator (we call “decoder”)
. The encoder h is usually trained based on other training data (which typically does not contain the C-class few-shot observations) so that h extracts informative features that can distinguish the unseen classes. For the training phase, we are given training data for several tasks,
(t is the task index: t = 1, . . . , T) where
and
, the number of observations
for each task is supposed to be small. Based on the support dataset
, we train the encoder h and the network which produces the weight vector
. This procedure can be seen as a kind of learning a training procedure. In the test phase, we are given test data
(t is the task in- dex:
) where
and
of new unseen tasks. Based on the support dataset
, the encoder produces the new weight vector
. In the few-shot learning setting,
is a class label among C-classes and the total number of data is M.
2.2 Meta-Learning via amortized Bayesian inference
VERSA is a Bayesian meta-learning framework and is also used in the few-shot classification task. Its graphical model is shown in Figure 1 (a). VERSA consists of two parts: an encoder and an amortization network. The encoder h(x) = maps an input to a feature vector, and its parameter is denoted by
(we call global latent variable). We use the same
across all the tasks. As we have described in the previous section, this encoder is trained through the support dataset
. When a new task appears at test time, the same en- coder as the one estimated at the training time is used for test as well; that is, for a newly observed task t, shared statistical encoder h is fed
as input and it outputs
as a representation of the input
. The amortization network outputs the predictive distribution from the representation of the input
. It is characterized by the task specific parameter
which represents a network that maps the encoded input
to the parameters of the approximated posterior distribution of the parameters of the output label
. In VERSA model,
have to be trained with few-shot samples at training time using the training data
. In practice, as the amortized function, essentially a neural network, is estimated to take a representation variable as input, and outputs the mean and variance parameter for predictive distribution of each task.
Figure 1: Graphical models for meta-learning framework corresponding to our proposal method. The original graphical model ML-PIP [Gor- don et al., 2019] (a) is transformed into the center (b) after performing inference over . The graphical model (b) represents VERSA [Gor- don et al., 2019] and NPs [Garnelo et al., 2018b]. We can use all observables to obtain an inference. Therefore, we can derive an additional dependency of
. The graphical model (c) represents the additional
dependency on
. Dotted lines denote variational approximations. Grey node indicates “observed”. White node indicates “non-observed”. Purple node indicates latent variable. Green node indicates global latent variable or meta-parameter.
For the few-shot classification task, VERSA encodes the class by the average of the encoded-input
. This acts like the weight vector
for the classification. Basically, the predictive distribution for
is given by the softmax value of
. To obtain the approximated posterior predictive distribution, VERSA generates
as a stochastic version of
from the Gaussian distribution with mean
and variance
specified by the output of
, and sample the predictive distribution corresponding to
This framework approximates the posterior predictive distribution by an amortized distribution as follows. Here, the predictive distribution of the test output given the input
and the few-shot sample
is given as
where corresponds to softmax function. However, the posterior distribution of
is difficult to calculate. Therefore, VERSA approximates the predictive distribution by the amortized distribution
by utilizing the approximated posterior distribution
. VERSA employs a Gaussian distribution as the approximated posterior
which is characterized by the network output:
. Then, the amortized predictive distribution is given as
Since VERSA wants to approximate the predictive distribution as accurate as possible, the end-to-end stochastic training objective to be minimized for and
is given as follows:
However, in general, learning “good” latent code is difficult because even when a powerful prior can be meta-learned from a large number of prior tasks, a small dataset for a new task can simply be too ambiguous to acquire a single accurate model. Here, we consider a more general objective which includes the regularization term:
In the objective, the KL-divergence between the posterior distributions work as regularization. Unfortunately, the conditional prior
in the above expression is intractable. To resolve this issue, we instead use an approximated posterior
, which gives:
It is interesting to note that replacing the feature vector with
, we got the NPs objective from the above objective:
Figure 2: This figure describes implementation differences between VERSA and our model. We replace encoder1 for a more precise implementation with the graphical model (b) [see Figure 1]. Then, representations of each (x) replace representations of each (x, y) pair. This approach is in a similar manner to NPs [Garnelo et al., 2018a; Garnelo et al., 2018b].
where the function r is the neural network. NPs combines the strengths of neural networks and Gaussian processes to achieve both flexible learning and fast prediction in stochastic processes. While VERSA uses linear discriminator as a decoder, NPs uses neural network as it. Both models are represented in Figure 1 (b).
The point is that, in VERSA and NPs, the central (stochastic) function being learnt has a form , of an output
given an input
, a support dataset
and the encoder’s parameter (global latent variable)
.
2.3 Information preference property
As in VERSA and NPs, we consider the following generative process for ,
where is the prior and
is given by a generative model with parameter
. Under ideal conditions, optimizing the objective using sufficiently flexible model families for
and
over
will achieve both goals of correctly capturing
and performing correct amortized inference. However, this approach suffers from the following problem: the decoder tends to neglect the latent variables
altogether, that is, the mutual information between
and
conditioned on
becomes negligibly small. For example,
Therefore,
The above objective tells us that the more the learning procedure proceeds (that is, the left-hand side reduces), the more the mutual information between and
decreases. It follows that the mutual information between
and
decrease. Intuitively, the reason is because the distribution of
tends to shrink to a single point and
is almost uniquely identified by a given
and
. This is undesirable because such a shrunken posterior of
is far from the true posterior and severely lose variation of the posterior sampling for
. This effect, which we shall refer to as the information preference problem, was studied in the VAE framework with a coding efficiency argument [Kingma and Welling, 2014]. In the VAE framework, the issue causes two undesirable outcomes: (1) the learned features are almost identical to the uninformative Gaussian prior for all observed tasks; and (2) the decoder completely ignores the latent code, and the learned model reduces to a simpler model [Fu et al., 2019].
As seen in the previous section, a unified Bayesian inference framework ML-PIP and its implementation VERSA and NPs utilize the amortized inference distribution because it is effi-cient and scalable to large datasets, and it requires only the specified parameters of the neural network. However, as we have seen in the previous section, the regularization in the objective causes the information preference problem (which is a well-known issue in the VAE framework) and inaccurate estimation of amortized inference distributions.
One approach to remedy this issue is to introduce a hyperparameter to control the strength of regularization [Hig- gins et al., 2017]. Furthermore, [Fu et al., 2019] found that scheduling
during the model training highly improve the performance. In addition, [Zhao et al., 2018] reported an alternative approach: replacing the KL-divergence of the latent distributions in the objective with the alternative divergence.
However, these previous studies applied their methodologies to only single task learning framework. In contrast, this paper considers the cyclical annealing schedule for during multi-task learning (meta-training) and replacing the divergence with the maximum mean discrepancy criterion. This procedure leads high mutual information between
and
, which encourages the model to use the latent code and avoids the information preference problem.
3.1 Cyclical annealing schedule
Several attempts have been made to ameliorate the information preference problem in the VAE framework. Among them, the simplest solution is monotonic KL annealing, where the weight of the KL penalty term is scheduled to gradually increase during training (monotonic schedule) [Bowman et al., 2016] (see Figure 3 (a)). In the VAE framework, latent code
learned earlier can be viewed as the initialization; such latent variables are much more informative than random and are thus ready for the decoder to use. Therefore, to mitigate the information preference problem, it is key to have meaningful
at the beginning of training the decoder, so that
can be utilized. Furthermore, [Fu et al., 2019] found that simply repeating the monotonic schedule multiple times (cyclical annealing schedule) enables high improvement in the above method (see Figure 3 (b)).
We apply this type of regularization to the multi-task learning context. We basically consider the following objective that anneals the second term of the right-hand side in Eq. (6) by the factor with
:
We can see that by annealing the regularization term (second term), the effect of information preference property discussed in Section 2.3 is mitigated. However, it is experimentally observed that just utilizing a fixed does not produce a good result. Thus, we gradually change the penalty term
during the training. For that purpose, we decompose the objective into each task and control
depending on each task as
(8) where
is the loss corresponding to a task t and
is the regularization term for each training task. During the training, we randomly sample task t one after another and update the parameters where we apply different
at each update. There could be several possibility of scheduling
(Figure 3), but we employ the cyclical annealing schedule during the training phase. This realizes that each task can be trained with several different values of
throughout the training, which yields avoiding the information preference problem. Our experimental results show that this approach helps each task-specific learner to avoid falling into local minima. We call this approach as the meta cyclical annealing (MCA).
3.2 Maximum mean discrepancy
Unfortunately, the KL-term in the right-hand side in Eq. (8) is difficult to compute. Therefore, we employ the maximum mean discrepancy (MMD) [Gretton et al., 2007] as a discrepancy measure between the distributions, which enables us to compute the corresponding term. MMD is a framework to quantify the distance between two distributions by comparing all of their moments via a kernel technique. Letting be any positive definite kernel, such as Gaussian kernel
. MMD between p and q
Figure 3: Annealing with (a) monotonic schedule, (b) cyclical annealing schedule during multi-task learning (meta-training).
It is known that if the kernel k is characteristic,
if and only if
[Muandet et al., 2017]. A rough intuition of MMD is that difference of the moments of each distributions
and
are measured through the (characteristic) kernel to know how different those distributions are. MMD can accomplish this efficiently via the kernel embedding trick.
We propose to employ MMD as the alternative divergence because MMD is easy to calculate and is stable against the support mismatch between the two distributions. To do so, instead of optimizing the objective introduced in Eq.(8), we minimize the following objective:
In this section, we experimentally show that the information preference problem of the posterior distribution actually occurs in non-regularized amortized inference and our proposal which aims to restrict the parameters of amortized distributions with MCA and MMD significantly improves the performance compared with existing methods.
4.1 Omniglot
Omniglot [Lake et al., 2015] consists of 1623 characters from 50 different alphabets. Each of alphabets was hand drawn by 20 different people, thus 20 instances for each class (each character). We follow a pre-processing and training procedure by [Vinyals et al., 2016] and [Gordon et al., 2019].
The training, validation and test sets consist of a random split of 1100, 100, and 423 characters, respectively. Each training iteration consists of the number of the mini batch that consists of random tasks extracted from the training set. During training, k-shot samples are used as training and remained 15 are used as test inputs. Evaluation after training is conducted on 600 randomly selected tasks from the test set. At the test phase, k-shot instances are utilized as test inputs which is unseen task for trained model. We use the Adam [Kingma and Welling, 2014] optimizer with a constant learning rate of 0.0001 with 16 tasks per batch to train all models.
4.2 mini-Imagenet
The mini-ImageNet dataset consists of a subset of 100 classes from the ImageNet dataset [Deng et al., 2009] and contains 600 images for each class. Also 100 classes are divided into 64 train, 16 validation, and 20 test classes. This dataset is
Figure 4: Visualizing the learned latent code We examine 20-way 5-shot classication in the Omniglot dataset. We randomly sample and fix fifty such tasks. Latent space with (a) the ordinary NPs, (b) NPs + MMD regularization without MCA (MMD+NPs), (c) NPs + meta cyclical annealing (MCA+NPs) and (d) NPs + meta cyclical annealing + MMD regularization (MCA+MMD+NPs).
complex and difficult enough to evaluate few-shot classifica-tion tasks. Training proceeds in the same episodic manner as with Omniglot.
4.3 Effect of regularization
Here, we checked if our model estimates the latent code as expected via MCA and MMD. MCA and MMD regularizes the latent distribution close to standard Gaussian distribution to avoid information preference problem. Figure 4 shows the distributions of
. We can see that the distributions of MCA+NPs and MCA+MMD+NPs are well regulated and close to Gaussian distributions (see Figure 4 (c) and (d)). This is because of our MCA and MMD regularization. On the other hand, the distribution of NPs is far from Gaussians and the distribution of each class tends to degenerate to one-point (like a delta-distribution) which loses variation of
resulting in worse posterior approximation (see Figure 4 (a)). This supports our expectation that the MMD-regularization effectively avoids the information preference problem.
4.4 Few-shot classification
To compare with existing methods, we focus our method on standard few-shot classification tasks, 20-way classification for Omniglot and 5-way classification for mini-ImageNet. We do not evaluate 5-way classification for Omniglot because it is already set to more than 99% with the existing methods, which is too high for comparing accuracy.
The results of Omniglot are shown in Table 1. Our proposal, MCA+NPs and MCA+MMD+NPs set good results. For 20-way 1-shot classification of Omniglot, our model achieves a new state-of-the-art result (99.810.14) which is significantly improved comparing with exiting methods.
Table 1: Accuracy comparison of few-shot classification. The sign indicates the 95% confidence interval. Bold text indicates the highest scores that overlap in their confidence intervals.
Table 2: Accuracy comparison of few-shot classification. The sign indicates the 95% confidence interval. Bold text indicates the highest scores that overlap in their confidence intervals.
The result on mini-ImageNet is shown in Table 2. We see that, for mini-ImageNet, MCA+NPs achieves 77.371.67% for 5-way 1-shot classification, MCA+MMD+NPs achieves 91.78
0.89% for 5-way 5-shot classification. Both results are also new state-of-the-art. Furthermore, our experimental results demonstrate that our models surpass VERSA in terms of performance, which suggests that mitigating amortization error provides improvement.
In this paper, we proposed the MCA+NPs and MCA+MMD+NPs models to improve amortized inference distribution with regularization techniques based on the latest few-shot learning framework, VERSA [Gordon et al., 2019] and NPs [Garnelo et al., 2018b]. Through comparing methods on a common ground, our results show that the MCA+MMD+NPs model is comparable to state-of-the-art models under standard conditions, and the MCA+NPs model achieves comparable performance to recent state-of-the-art meta-learning algorithms on both the Omniglot and miniImageNet benchmark datasets. Additionally, our proposal seems to avoid the information preference problem by analysis.
We would like to thank Naonori Ogasahara and Iwato Amano for insightful discussions. Taiji Suzuki was partially supported by JSPS Kakenhi (26280009, 15H05707 and 18H03201), and JST-CREST.
[Bowman et al., 2016] Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. Proceedings of Conference on Computational Natural Language Learning SIGNLL, 2016.
[Chen et al., 2019] Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. The International Conference on Learning Representations (ICLR), 2019.
[Cremer et al., 2018] Chris Cremer, Xuechen Li, and David Duvenaud. Inference suboptimality in variational autoencoders. The International Conference on Learning Representations (ICLR), 2018.
[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, LiJia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
[Finn et al., 2017] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning (ICML), 2017.
[Fu et al., 2019] Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, and Lawrence Carin. Cyclical annealing schedule: A simple approach to mitigating KL vanishing. Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2019.
[Garnelo et al., 2018a] Marta Garnelo, Dan Rosenbaum, Chris J. Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo J. Rezende, and S. M. Ali Eslami. Conditional neural processes. In International Conference on Machine Learning (ICML), 2018.
[Garnelo et al., 2018b] Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J. Rezende, S.M. Ali Eslami, and Yee Whye Teh. Neural processes. In International Conference on Machine Learning (ICML), 2018.
[Gordon et al., 2019] Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard E. Turner. Meta-learning probabilistic inference for Prediction. The International Conference on Learning Representations (ICLR), 2019.
[Gretton et al., 2007] Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard, Schlkopf, and Alex J. Smola. A kernel method for the two-sample-problem. In Advances in Neural Information Processing Systems (NIPS), 2007.
[Higgins et al., 2017] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick,
Shakir Mohamed, and Alexander Lerchner. -vae: Learning basic visual concepts with a constrained variational framework. The International Conference on Learning Representations (ICLR), 2017.
[Kim et al., 2018] Yoon Kim, Sam Wiseman, Andrew C. Miller, David Sontag, and Alexander M. Rush. Semiamortized variational autoencoders. The International Conference on Learning Representations (ICLR), 2018.
[Kingma and Welling, 2014] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. The International Conference on Learning Representations (ICLR), 2014.
[Lake et al., 2015] Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 2015.
[Lee et al., 2019] Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. 2019.
[Li et al., 2017] Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few shot learning. 2017.
[Muandet et al., 2017] Krikamol Muandet, Kenji Fukumizu, Bharath Sriperumbudur, Bernhard Sch¨olkopf, et al. Kernel mean embedding of distributions: A review and beyond. Foundations and Trends in Machine Learning, 10(1-2):1– 141, 2017.
[Nichol et al., 2018] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. 2018.
[Qiao et al., 2018] Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan Yuille. Few-shot image recognition by predicting parameters from activations. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[Ravi and Beatson, 2019] Sachin Ravi and Alex Beatson. Amortized bayesian meta-learning. The International Conference on Learning Representations (ICLR), 2019.
[Rusu et al., 2019] Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. 2019.
[Snell et al., 2017] Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems (NIPS), 2017.
[Song et al., 2019] Liang Song, Jinlu Liu, and Yongqiang Qin. Fast and generalized adaptation for few-shot learning. 2019.
[Sung et al., 2018] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip H.S. Torr, and Timothy M. Hospedales. Learning to compare: Relation network for few-shot learning. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[Vinyals et al., 2016] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In Advances in Neural Information Processing Systems (NIPS), 2016.
[Zhao et al., 2018] Shengjia Zhao, Jiaming Song, and Stefano Ermon. InfoVAE: Information maximizing variational autoencoders. 2018.