Recently, Deep Learning attracts a lot of attention in methodology research of artificial intelligence such as machine learning [1]. The learning architecture has an advantage of not only multi-layered network structure but also actualization of pre-training. The pre-training realizes that the architecture of Deep Learning accumulates prior knowledge of the features for input patterns. The convolutional neural network (CNN) is well known to be feed forward neural network and the layers of CNN have neurons arranged in 3 dimensions. However, the building a new CNN takes much costs to find an optimal structure for the given input patterns and the CNN with the higher detection capability for the unknown patterns can not be constructed because the optimal set of many parameters is not found instantly. On the other hand, Restricted Boltzmann Machine (RBM) [2] and Deep Belief Network (DBN) [3] are focused as one of popular method of Deep Learning for unsupervised learning. RBM has the capability of representing a probability distribution of input data set, and it can represent an energybased statistical model. Moreover, the Contrastive Divergence

c2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

(CD) learning procedure can be often used as one of the learning methods of RBM [4], [5]. CD method is a faster algorithm of Gibbs sampling based on Markov chain Monte Carlo method. However, the problem of RBMs is also the definition of an optimal initial network structure including the optimal number of hidden neurons according to the features of input patterns.

DBN has a deep architecture that can represent multiple features of input patterns hierarchically with the pre-trained RBM. In other word, each layer of DBN employs RBM learning method to implement the pre-training and then DBN consists of the hierarchical two or more trained RBMs. RBM at lower layer can represent the abstract character, while one at higher layer shows the concrete object. If the energy function at each layer are good performance, the whole network can achieve the higher capability than the traditional neural network.

We proposed the adaptive learning method of RBM that can construct a RBM with an optimal number of hidden neurons according to the training situation by applying the neuron generation and annihilation algorithm [6], [7]. Moreover, we developed the hierarchical adaptive learning method of DBN that can determine the optimal number of hidden layers according to the given data set [8]. The layer generation of DBN is natural extension to develop the adaptive learning method with self-organizing mechanism.

Recently, the analysis of high dimensional input patterns such as images in video, sounds in music is required as a new data set of Big data. Such data are time series one which consist of two or more sequences of discrete time data. A general time series data are frequently plotted by using line charts such as signal processing, weather forecasting and mathematical finances. However, high dimensional data at each time step such as video and music have the multi modal conditional distributions. The active researchers pursue the developing model of such sequences to predict the conditional distribution at the next time by using the previous time step data.

In recurrent neural networks, there are two variants of RNNs for modeling slot sequences: the Elman-type RNN [9] and the Jordan-type RNN [10]. In the deeplearning field, tempral RBM and RNN-RBN are the current main stream researches. Recur-

Fig. 1. An overview of RBM

rent Temporal Restricted Boltzmann Machine (RTRBM) [11] is a probabilistic model for high dimensional sequences and a directed graphical model consisting of a sequence of RBMs. The learning consists of a conditional RBM at each time step. Despite its simplicity, this model successfully accounts for several interesting sequences. Recurrent Neural Networks RBM (RNN-RBM) [12] is a similar model of RTRBM and realizes the specification of recurrent neural networks (RNNs), which can incorporate an internal memory that can summarize the entire sequence history.

We propose the adaptive learning method in RNN-RBM to embed the neuron generation / annihilation. Moreover, we employ the hierarchical adaptive learning method of DBN that can determine the optimal number of hidden layers according to the given data set. The learning method can perform more accuracy to input patterns even if the pattern has noise. Our proposed adaptive learning method of RNN-DBN has a deep architecture that can represent multiple features of input patterns hierarchically with the pre-trained adaptive learning method of RNN-RBM. In order to verify the effectiveness of our proposed method, we investigate the prediction capability for some benchmark data sets. As a result, the adaptive learning method of RNN-DBN shows the higher score than the traditional methods.

A. An Overview of RBM

Fig. 1 shows a structure of RBM which has 2 kinds of layers. The lower layer is a visible layer for input data. The upper layer is a hidden layer for representing the features to train the input data space. The neuron in both layers is a binary neuron and their connections between neurons in same layer are removed. RBM learning method trains some parameters including weights and bias till the energy function reaches the expected value. The trained RBM can represent the model to be fit for the probability distribution of input data space.

We explain the general RBM model mathematically in the section. Let and be a binary variable for each neuron. I and J are the number of visible and hidden neurons, respectively. Eq.(1) is the energy function E(v, h) for visible vector and hidden vector . Eq.(2) is the probability distribution for v and h.

where and are the bias parameters for and , respectively. is the weight between and is the partition function by summing over all possible pairs of visible and hidden vectors. The RBM can learn the biases and weights according to the distribution of input data by Contrastive Divergence (CD) [4]. CD method is a faster algorithm of Gibbs sampling based on Markov chain Monte Carlo methods and can realize a good performance in smaller number of sampling steps. The RBM learning method by CD method should consider the convexity and continuous conditions for an objective function. Carlson et al. et al. derived that the RBM learning by CD method will be converged if the variance for 3 kinds of parameters falls into a certain range during the training under the Lipschitz continuous (please see Eq.(18)-(20) in [13] for details). Under the convexity and continuous conditions, we observed the oscillation of gradients of three biases b, c, and W during the learning phase. From some results on benchmark datasets, the change for the two kinds of parameters c and W is very large and then we found the regularity to avoid the repercussion of uniformity of impact data [7].

B. Neuron Generation and Annihilation Algorithm

We have proposed the adaptive learning method of RBM that the optimal number of hidden neurons can be self-organized according to the features of a given input data set in learning phase. The neuron generation and annihilation algorithm of RBM can measure the criterion of network stability with the fluctuation of weight vector [6], [7].

The basic concept of neuron generation and annihilation algorithm is as follows. A new neuron is generated in case of the lack of classification capability of hidden neurons. The weight will fluctuate greatly even after the training process, since some hidden neurons may not represent an ambiguous pattern. In such a case, the corresponding neuron is split into 2 neurons to represent the complex pattern by inheriting the attributes of the parent hidden neuron. Then we monitor the value of left term of Eq.(3) to monitor the condition of neuron generation.

The equation is the inner product of variance of monitoring both of 2 parameters c and and in Eq.(3) are the gradient vectors of the hidden neuron j and the weight vector between the neuron i and j, respectively. and are the constant values for the adjustment of the range of

Fig. 2. The neuron generation/annihilation algorithm

each parameter. is an appropriate threshold value. Fig. 2(a) shows that a new hidden neuron generates by insertion it into the neighborhood of the parent neuron.

After the neuron generation process, if the network with the inactivated neuron that does not work to the classification capability, the corresponding neuron should be removed in terms of the reduction of calculation. If Eq.(4) is satisfied in learning phase, the corresponding neuron will be annihilated as shown in Fig. 2(b).

where is a given input data, N is the number of samples of input data. means a conditional probability of under a given is an appropriate threshold value. The proposed method by the neuron generation and annihilation algorithm showed the good performance for some benchmark tests [7].

C. A Structural Learning Method with Forgetting

Although the optimal number of hidden neurons is determined by the neuron generation and annihilation algorithms, we may meet another difficulty that the trained network is still a black box, and then we cannot extract some explicit knowledge from the trained network. In order to solve the difficulty, we developed the Structural Learning Method with Forgetting (SLF) to discover the regularities of the network structure in RBM [6]. The basic concept is the structural learning with forgetting method [14] where three kinds of penalty terms are added into an original objective function J as shown in Eq.(5) - Eq.(7). Each equation is for ‘learning with forgetting,’ ‘hidden units clarification,’ and ‘learning with selective forgetting’, respectively.

Fig. 3. An overview of DBN

where the range of and are smaller than the pre-determined small value, which are negligible small value in the whole weight vector space. After the optimal number of hidden neurons is determined by neuron generation and annihilation algorithm, both Eq.(5) and Eq.(6) should be applied simultaneously during a certain learning period. Alternatively, Eq.(7) is used instead of Eq.(5) at final learning period so as to be the large objective function.

Distillation method of Deep learning has been proposed by Hinton [15]. The method can compress the knowledge in an ensemble large model of Deep Learning into a single smaller model. The ensemble implements a function from input patterns to output patterns. Instead of the models in the ensemble model of Deep Learning, they consider a way that the model is parameterized and they focus the obtained function of network. The forgetting method in this paper realizes a basic idea of distillation method in each RBM learning before the construction of DBN.

D. An Adaptive Learning Method of DBN

Fig. 3 shows the model of Deep Belief Network (DBN) [3]. The model has the hierarchical network structure where each layer is pre-trained by RBM. The hierarchical DBN network structure becomes to represent higher and multiple level features of input patterns by building up the pre-trained RBM.

In this paper, we propose the adaptive learning method of DBN that can determine the optimal number of hidden layers. The adaptive learning method of RBM is worked by the neuron generation and annihilation algorithm described in section II-B. In general, data representation of DBN performs the specified features from abstract to concrete at each layer in the direction to output layer. That is, the lower layer has the power of non figurative representation, and the higher layer constructs the object to figure out an image of input patterns. Adaptive DBN can automatically adjust self-organization of structured data representation.

In the learning process of adaptive DBN, we observe the total WD for only the variance of both c and W and energy function. If the energy function is still large and the overall WD is larger than the given threshold value, a new RBM is required to express the concrete pattern for the given input data. In other words, the large energy function and the large WD stays in the condition that the DBN has lack data representation capability to figure out an image of input patterns. Eq.(8) and Eq.(9) are the condition of layer generation with the total WD and the energy function.

where is the total variance of parameters c and W in l-th RBM. is the total energy function in l-th RBM. k is the top RBM in the current DBN structure. and are the constant values for the adjustment of deviant range of and and are the pre-determined threshold values. If Eq.(8) and Eq.(9) at layer k are satisfied simultaneously during learning, a new RBM k + 1 will be generated after the learning at layer k. The initial values of b, c and W at the generated layer k + 1 are given to be inherited from the parent(lower) RBM.

As far as the image benchmark data set CIFAR-10 and CIFAR-100 [16] as shown Table I and Table II, the classifica-tion rate for the training data set and the test dataset showed the highest capability [17], [7], [8]. The determined data sets for the training and test in benchmark test are given to verify the effectiveness of the propoed learning method in the competitive evaluation.

Because a Deep Neural Network can bring back the ever seen data into an existence, the generalization power is higher than the traditional neural network. Each RBM classify the detailed patterns to represent a little difference in given data. However, the reason of the misclassification of recurrent data does not depend on the generalization power of DBN. In fact, some sequential data have the same input pattern but different output pattern in both the training data and test data.

Moreover, RBM has a restricted hidden layer which each hidden neuron does not connect each other to train the subset of input signal patterns independently. In other word, there is no intersection of subsets in hidden layer. If the input pattern is complex, RBM needs more hidden neurons where a neuron works as nonlinear signal separator for the specified pattern. Therefore, the trained RBM can incarnate the detailed data representation to prevent over fitting situation.

As mentioned in section I, the active researcher pursues the developing model of such sequences to predict the conditional

TABLE I CLASSIFICATION ACCURACY ON CIFAR-10

distribution at the next time by using the previous time step data.

Fig. 4 and Fig. 5 show the overview of Temporal RBM and RTRBM, respectively. RTRBM [11] is a similar model of Temporal RBM. RTRBM is a probabilistic model for high dimensional sequences and forms a directed graphical model consisting of a sequence of RBMs. The learning method consists of a conditional RBM at each time step. Despite its simplicity, this model successfully accounts for several interesting sequences.

Recurrent Neural Networks RBM (RNN-RBM) [12] as shown in Fig. 6 is a similar model of RTRBM and realizes the specification of recurrent neural networks (RNNs). The network structure can incorporate an internal memory that can summarize the entire sequence history.

We propose the adaptive learning method in RNN-RBM to embed the neuron generation / annihilation described in subsection II-B. Moreover, we employ the hierarchical adaptive learning method of DBN that can determine the optimal number of hidden layers according to the given data set. The learning method can perform higher accuracy rate of classification to input patterns even if the input pattern has noise. Our proposed adaptive learning method of RNN-DBN as shown in Fig. 7 has a deep architecture that can represent multiple features of input patterns hierarchically with the pre-trained adaptive learning method of RNN-RBM.

The RNN-RBM model has the state of representing con-

Fig. 4. Temporal RBM[11]

Fig. 5. Recurrent Temporal RBM[11]

Fig. 6. Recurrent Neural Network RBM[12]

texts in time series data, , related to past sequences of time series data in addition to the visible neurons and the hidden neurons of the traditional RBM. Let the sequence of input sequence with the length T be . The parameters and

for the visible layer and the hidden layer, respectively are calculated from for time by using Eq.(10) and Eq.(11). The state at time t is updated by using Eq.(12).

Fig. 7. Recurrent Neural Network DBN

where is a sigmoid function. Fig.6 shows the flow of RNN-RBM. is the initial state which is given a random value. RNN-RBM is trained the weights between and by using at time t and weights W . After the error are calculated till time t, the gradients for are updated to trace from time T back to time t by BPTT( Back Propagation Through Time) method. Algorithm 1 shows the learning process of adaptive RNN-RBM.

In the similar way of the adaptive learning method of RBM, we monitor c and W in the learning. If Eq.(3) and Eq.(4) are satisfied, the neuron generation / annihilation algorithm works to find the optimal structure of RBM. Moreover, the structural learning mentioned in subsection II-C is applied to find the sparse structure.

The adaptive learning method of RNN-DBN is building the hierarchical network structure to pile up the pre-trained RNNRBM in stages. In RNN-RBM, the hidden neuron for the input signal at time t is calculated from at time deterministically. The output signals for the given input signals in RNN-RBM are determined uniquely and set them to the input signals in the subsequent layer in RNN-DBN are trained repeatedly. Algorithm 2 shows the learning process of adaptive RNN-DBN.

Fig. 8. Experimental Results on Nottingham (layer 1)

TABLE III PREDICTION ACCURACY ON NOTTINGHAM

Algorithm 2 Adaptive RNN-DBN

Set 1-th input and initial value of parameters . for do Make pre-training l-th RBM for given and . if the neuron layer conditions are satisfied with Eq.(8) and Eq.(9) during the learning then The layer l + 1 is generated. Calculate l-th input and set initial value of parameters . else The layer generation is stopped. end if end for

A. Data Sets

In this experiments, the benchmark data set ‘Nottingham’ [21] and ‘CMU’ [22] were used to verify the effectiveness of our proposed method. Nottingham is a classical piano MIDI archive included in about 694 training cases and about 170 test cases. On the other hand, CMU is a motion capture database collected by Carnegie Mellon University. There are 2,605 trials in 6 categories which are divided into 23 subcategories. We use the same Benchmark test used in [11] and [12] to compare the capability of our proposed method with their methods.

The parameters used in this paper are as follows. The training algorithm is Stochastic Gradient Descent (SGD) method,

Fig. 9. Experimental Results on CMU (layer 1)

TABLE IV PREDICTION ACCURACY ON CMU

the batch size is 100, the learning rate is 0.01, , , . We used the computer with the following specifi-cations: CPU = Intel(R) 24 Core Xeon E5-2670 v3 2.3GHz, GPU = Tesla K80 4992 24GB times 3, Memory = 64GB, OS = Centos 6.7 64 bit.

B. Experimental Results

Fig. 8 shows the learning situation of the RNN-RBM [12] and the adaptive RNN-RBM on Nottingham. Fig. 8(a) - Fig. 8(d) are the learning curve of energy function and error function, the variance of W , the variance of c, and the change of the number of hidden neurons. As shown in Fig. 8(c) and Fig. 8(d), the variance of the parameters was significantly fluctuated even after a certain period of the learning in the RNN-RBM. As a result, the energy function and the error function were not converged with smaller value as shown in Fig. 8(a) and Fig. 8(b). On the other hand, the neuron generation process in our proposed RNN-RBM was operated after 20th iterations. After the generation process, 52 additional hidden neurons were generated (total number is 62) and the network structure enough to the input patterns was obtained. As a result, the energy function and error function were converged into a small value. After the learning of RNN-RBM, a structural learning method with forgetting was applied to get the sparse structure of RBM with 42 hidden neurons. The RNN-DBN builds the hierarchical network structure to pile up the trained RNN-RBMs. The total CPU time to the end of computation was 529 and 413 minutes for the traditional RNN-DBN and the adaptive RNN-DBN, respectively.

Such an observation can be also seen on the training result on CMU as shown in Fig. 9. In the case of CMU, there are 100 hidden neurons in the beginning of learning and 100 additional hidden neurons were generated during the learning, then the total number of hidden neuron was 200. After a structural learning method with forgetting, there were 167 hidden neurons. The total CPU time to the end of computation was 151 and 125 minutes for the traditional RNN-DBN and the adaptive RNN-DBN, respectively.

Table III and Table IV show the prediction accuracy on Nottingham and CMU, respectively. Our proposed adaptive RNN-RBN and RNN-DBN can obtain a higher prediction accuracy for not only training set but also test set than traditional RNN-RBM and RNN-DBN model. The prediction accuracy rate are 76.4% for Nottingham test set, 73.2% for CMU test set, which the value are traditional DBN model. It is remarkable that adaptive learning method of RNN-RBM performs better than the traditional RNN-DBN, because the pre-training at lower layer in the traditional RNN-DBN cannot work well to represent abstract features for input patterns. The problem will be solved by finding the optimal setting of parameters, however, a try and error operation should be required. As a result, the adaptive RNN-DBN can obtain the highest prediction accuracy with smallest energy and error for test set.

The adaptive structural learning method of RBM is to selforganize the optimal network structure in terms of energy stability as well as clarification of knowledge according to the given input data during the learning phase. Moreover, we developed assemble method of pre-trained RBM by using the layer generation condition in hierarchical DBN. The proposed method is superior at the self-organized method to train the optimal network structure for given training data set during training process. Our proposed adaptive learning method of DBN records a great score as for image classification task. However, the previous method showed the classification capability of static image data enough, but the classification for the time-series data was not high. In this paper, we embed the adaptive learning method into the RNN-RBM model. Moreover, RNN-RBM is extended to realize the DBN model with self-generated layer algorithm and then the model shows the high prediction accuracy for the time-series data sets. We will apply our proposed method into another time-series data set such as medical image data such mammography data in future works.

This work was supported by JAPAN MIC SCOPE Grand Number 162308002.

[1] Y.Bengio, Learning Deep Architectures for AI. Foundations and Trends in Machine Learning archive, Vol.2, No.1, pp.1–127 (2009)

[2] G.E.Hinton, A Practical Guide to Training Restricted Boltzmann Machines. Neural Networks, Tricks of the Trade, Lecture Notes in Computer Science, Vol.7700, pp.599–619 (2012)

[3] G.E.Hinton, S.Osindero and Y.Teh, A fast learning algorithm for deep belief nets. Neural Computation, Vol.18, No.7, pp.1527–1554 (2006)

[4] G.E.Hinton, Training products of experts by minimizing contrastive divergence, Neural Computation, Vol.14, pp.1771-1800 (2002)

[5] T.Tieleman, Training restricted Boltzmann machines using approximations to the likelihood gradient, Proc. of the 25th international conference on Machine learning, pp.1064-1071 (2008)

[6] S.Kamada, T.Ichimura, ‘A Structural Learning Method of Restricted Boltzmann Machine by Neuron Generation and Annihilation Algorithm’, Neural Information Processing, Proc. of the 23rd International Conference on Neural Information Processing, Springer LNCS9950), pp.372-380 (2016)

[7] S.Kamada and T.Ichimura, An Adaptive Learning Method of Restricted Boltzmann Machine by Neuron Generation and Annihilation Algorithm. Proc. of 2016 IEEE SMC (SMC2016),pp.1273-1278 (2016)

[8] S.Kamada and T.Ichimura, An Adaptive Learning Method of Deep Belief Network by Layer Generation Algorithm, Proc. of IEEE TENCON2016, pp.2971-2974 (2016)

[9] J.Elman, Finding structure in time, Cognitive Science, Vol.14, No.2 (1990)

[10] M.Jordan, Serial order: A parallel distributed processing approach, Tech. Rep. No. 8604. San Diego: University of California, Institute for Cognitive Science (1986)

[11] I.Sutskever, G.E.Hinton, and G.W.Taylor, The Recurrent Temporal Restricted Boltzmann Machine, Proc. of Advances in Neural Information Processing Systems 21 (NIPS-2008) (2008)

[12] N.B.Lewandowski, Y.Bengio and P.Vincent, Modeling Temporal Dependencies in High-Dimensional Sequences:Application to Polyphonic Music Generation and Transcription, Proc. of the 29th International Conference on Machine Learning (ICML 2012), pp.1159-1166 (2012)

[13] D.Carlson, V.Cevher and L.Carin, Stochastic Spectral Descent for Restricted Boltzmann Machines. Proc. of the Eighteenth International Conference on Artificial Intelligence and Statistics, pp.111-119 (2015)

[14] M.Ishikawa, Structural Learning with Forgetting. Neural Networks, Vol.9, No.3, pp.509-521 (1996)

[15] G.Hinton, O.Vinyals, J.Dean, Distilling the Knowledge in a Neural Network, Proc. of NIPS 2014 Deep Learning Workshop (2014)

[16] A.Krizhevsky, ‘Learning Multiple Layers of Features from Tiny Images’, Master of thesis, University of Toronto (2009)

[17] Classification datasets results, http://rodrigob.github.io/arewe