Automating the feature extraction is currently an active research field aiming to learn enhanced representations directly from the raw data rather than handcrafting them. Neural Network based architectures have been used in this regard for image recognition [13] and sound [1]. The adoption of these architectures to sound recognition usually occurs after they gain wide acceptance in other application domains such as image recognition. For example, stacked Restricted Boltzmann Machines (RBM) [5] forming a Deep Belief Net (DBN)[9] to extract features were initially introduced to showcase the capability of these stacked generative layers to be used as a dimensionality reduction technique when applied on images of handwritten digits. Later, Hamel et al. [8] trained a DBN of three RBM layers over frames of a spectrogram to extract abstract representations from music files that were classified using a Support Vector Machine (SVM) [37] for a music genre classification task. Convolutional Neural Networks (CNN) as well were initially introduced in the work of LeCun et al. [14] for images, and later attempts followed to use it for sound [28, 11, 30]

Despite the success of these architectures for images, they are not designed to exploit the time-frequency representation of sound efficiently. For example, DBNs ignore the inter-frames relation by treating a spectrogram’s frame in isolation from neighboring frames, and CNNs depend on weight sharing, which does not preserve the spatial locality of the learned features.

The ConditionaL Neural Networks (CLNN) [20] and the Masked ConditionaL Neural Networks (MCLNN) [20] are designed to preserve the spatial locality of the learned features, where there is a dedicated link for every feature in a feature vector compared to the weight sharing using the CNN. The CLNN preserve the temporal relation between the frames by considering a window rather than the isolated frame used in the RBM, and the mask in the MCLNN enforces a systematic sparseness over the network’s links. The mask design follows a band-like pattern, which allows the network to be frequency shift-invariant mimicking a filterbank. Additionally, the mask explores several feature combinations concurrently analogous to handcrafting the optimum combination of features through a mix-and-match operation, while preserving the spatial locality of the features.

Fig. 1. Conditional RBM

Fig. 2. Convolutional Neural Network

The Conditional Restricted Boltzmann Machine (CRBM) [35] by Taylor et al. extended the RBM to the temporal dimension to allow an RBM to learn about a temporal window of frames rather than being trained on static bag-of-frames. To fulfill this aim, the CRBM adapted conditional links to capture the influence of the previous frames on the current one. Fig. 1 shows a CRBM layer, where the normal RBM is represented with the bidirectional connections ˆW going across the visible vector ˆv0 and the hidden nodes ˆh. The ˆB links in the figure represent the conditional links from the previous visible vectors (ˆvˆvˆv) to the hidden layer ˆh. Similarly, the ˆA links capture the autoregressive relation from the previous visible vectors to the current one ˆv0. Layers of a CRBM can be stacked over each other similar to a DBN, where Taylor et al. trained a CRBM to model the human motion over a multichannel signal of human joints activity. Mohamed et al. [22] extended the CRBM with the Interpolating Conditional Restricted Boltzmann Machine (ICRBM), which showed an enhanced performance by including the influence of the future frames in addition to the past ones for phoneme recognition. The work of Battenberg et al. [3] was another attempt to use the CRBM for sound, where they used the CRBM to analyze drum patterns.

Similar modifications were introduced to the CNN to fit the time-frequency representation. The CNN architecture, shown in Fig. 2, is based on the two primary operations: convolution and pooling. The convolution operation scans the 2-dimensional representation with a small weight matrix (or filter), e.g. 55, where a form of a weighted sum is generated from the element-wise multiplication between the filter and the region of the image being scanned. The output of each step of the filter is a scalar value positioned in a new representation of the image known as the feature map. The convolutional layer generates several feature maps. The number of feature maps matches the number of filters used. Mean or max pooling follows the convolution to reduce the resolution of the feature maps. These two operations are consecutively repeated to form a deep architecture of a CNN, where the output of the final layer is flattened to a single feature vector to be fed to a fully connected neural network for the final classification. CNN depends on weight sharing, which performs well in favor of large images without the need to have a dedicated weight going across each pixel and the network’s hidden layer. Weight sharing does not preserve the spatial locality of the learned features, which is practical for images, but not for time-frequency representation. This is related to the influence of the location of the detected feature at a specific frequency as a property to distinguish between sounds. The work of Abdel-Hamid et al. [1] approached this problem by redesigning the convolutional filters to operate over bands. Another attempt was in [28], where they proposed using separate filters to convolve each of the time and frequency dimensions separately combined in the same model.

The Masked ConditionaL Neural Network (MCLNN) was introduced in [20] with an analysis of the influence of the data split on model accuracy. In this work, we further evaluate the MCLNN performance on the music genre classification task.

The ConditionaL Neural Network (CLNN) [20] is a discriminative model that extends from the generative Conditional Restricted Boltzmann Machine (CRBM) [35] discussed earlier. The CLNN adapts the conditional previous visible to hidden links proposed in the CRBM, and it further extends the connections to the future frames as presented in the ICRBM [22].

where yjis the activation at node j of the hidden layer for the window’s middle frame at index t of the segment. The segment, discussed later in detail, is a chunk of frames of a minimum size equal to the window. f is the transfer function and bj is the bias at the jth node. xiis the ith feature of the feature vector x. u refers to the index within the window and t refers to the window’s middle frame (having u=0 in the window), which is at the same time the index of the middle frame in the input segment. Wiis the weight between the ith feature of the vector at position u in the window and the jth neuron in the hidden layer. u is the index of a frame in the window and also the index of its corresponding weight matrix in the weight tensor. The hidden layer activation can be reformulated in a vector form in (3).

where the hidden layer activation vector ˆy for the window’s middle frame xt conditioned on the n neighboring frames in either direction is given by the transfer function f, the bias vector ˆb and the vector matrix multiplication between the feature vector ˆxu at index u and its corresponding weight matrix ˆWu at the same index. The number of matrices in the weight tensor is equal to 2n + 1 matching the number of frames in the window, where each frame is processed by its dedicated matrix. The conditional distribution is formulated in p( ˆyt|ˆxˆxˆxt, ˆx1ˆxn, where is a logistic function such as a Sigmoid or the output layer Softmax.

Fig. 3. Two CLNN layers with n = 1.

Fig. 4. Masking patterns. a) Bandwidth = 5 and Overlap = 3, b) the active links following the masking pattern in a. c) Bandwidth

where q is the segment size, the order n is for the number of frames in a single direction (the 2 is for the past and future frames ), m is the number of layers, and k is for the extra frames to be pooled across beyond the CLNN layers. For example, at n = 4, m = 3 and k = 5, a segment of size 29 frames is presented at the input of the first CLNN layer. The second CLNN layer will receive 29 21 vectors at its input and consequently will generate 21 13 vectors as an output. Similarly, the third layer will generate 135 vectors, which undergo flattening or pooling to a single vector before the fully-connected layers.

Spectrograms represent the energy at different frequency bins as the signal progresses through time. Despite the usefulness of such representations for signal analysis, they are susceptible to the frequency shifts, which could provide different spectral representations for very similar sounds. Frequency shift involves a smearing in the energy of a frequency bin across nearby bins due to uncontrolled factors affecting the signal propagation. Filterbanks tackle the frequency shifts in raw spectrograms. A filterbank is a group of filters used to subdivide the spectrograms into frequency bands allowing the new representation to be frequency shift-invariant. They are the principal operating component of Mel-scaled transformations such as the MFCC. The Masked ConditionaL Neural Networks (MCLNN) [20] embed a filterbank-like behaviour within the network by enforcing a systematic sparseness over the network’s links that follows a band-like pattern.

where the linear index lx is given by the bandwidth bw, the overlap ov and the feature vector length l. a takes the values in [ 0, bw and g is in the interval ov. The mask plays another role of exploring a range of feature combinations analogous handcrafting the optimum feature combinations. This operation is applied in the MCLNN for several feature combinations concurrently as shown in Fig. 4.c., where the 2nd set of three columns holds a shifted version of the 1st three columns and similarly for the 3rd set. In a closer analysis, each hidden node (mapped to a column in the mask) will have a different input to observe. For example, the input at the 1st node is the first three features of the feature vector, the 4th node’s input is the first two features, and the 7th node is the first feature. The masking is applied through an element-wise multiplication following (6).

where ˆWu is the original weight matrix at index u, ˆM is the masking pattern and ˆZu is the masked weight matrix to replace the original one in (3).

Fig. 5. Single MCLNN step.

Fig. 5 shows a single MCLNN step, where a window of frames of size 2n + 1 is processed with a matching count of matrices. Each frame in the window has a corresponding matrix to process. The vector-matrix multiplication generates d new vectors, which are summed feature-wise before applying the nonlinearity by a transfer function. The output of a single step over the window is a resultant single frame. The highlighted cells in each matrix depict the active links enforced through the mask.

We evaluated the performance of the MCLNN using the Ballroom [6] and the Homburg [10] datasets widely adapted for Music Information Retrieval tasks including genre classification.

The Ballroom dataset is composed of 698 music clips of 30 seconds each, unevenly partitioned across 8 music genres: Cha Cha (CC), Jive (Ji), Quickstep (Qs), Rumba (Ru), Samba (Sa), Tango (Ta), Viennese Waltz (VW) and Slow Waltz (SW).

The Homburg dataset contains 1886 music clips of 10 seconds each, distributed across 9 classes: Alternative (Al), Blues (Bl), Electronic (El), FolkCountry (FC), FunkSoulRnb (FS), Jazz (Ja), Pop (Po), RapHiphop (RH) and Rock (Ro).

All files for both datasets were transformed to a logarithmic mel-scaled spectrogram of 256 bin using an FFT of 2048 and 1024 hop size. Segments were extracted following (4) and the z-score parameters of the training data were used to standardize the testing and validation sets. Experiments were carried out using a 10-folds cross-validation with the mean accuracy across the folds reported. The hyper-parameters used for the MCLNN are listed in Table 1.

Table 1. MCLNN hyper-parameters for the Ballroom and the Homburg

The two MCLNN layers are followed by a global single dimension pooling layer to pool feature-wise over the k extra frames. The global pooling emulates the aggregation over a musical texture window, which was studied by Bergstra et al. [4]. We used k = 11 and k = 2 for the Ballroom and the Homburg, respectively. Two densely connected layers of 50 and 10 nodes followed the global pooling layer, before the final Softmax. The model was trained using ADAM [12] to minimize the categorical cross-entropy between the predicted vector and the target label. Dropout [34] was used as a regularizer. The final decision of the clip’s category is decided using probability voting across the frames of the clip.

Table 2. Reported accuracies on Ballroom

Table 3. Reported accuracies on Homburg

As listed in Table 2 and Table 3, MCLNN achieved an accuracy of 90.4% and 61.45% on the Ballroom and the Homburg, respectively, which surpasses several neural network based architectures in addition to hand-crafted attempts on both datasets. MCLNN achieved the mentioned accuracies without a special design to exploit musical perceptual properties compared to other attempts. In the work of Peeters [26], he achieved 96.13% on the Ballroom using the Tempo annotations released with the dataset. Peeters reapplied his proposed handcrafted features without Tempo data, and the accuracy was 88%, which shows the influence of the tempo annotations. In a similar type of analysis, Gouyon et al.[7] used the Tempo annotations as a baseline to benchmark their proposed handcrafted features, where the Tempo annotations alone achieved 82.3% and their proposed features with the Tempo achieved 90.1%. The work of Marchand et al. [19] achieved 93.12% using multiple processing stages including on-set energy calculation, autocorrelation, modulation scale spectra and dimensionality reduction to exploit rhythmic pattern in a music clip. Seyerlehner et al. [32] achieved 92.44% using several features extracted from blocks of the spectrogram. A neural network based attempt in the work of Pons et al. [28] achieved 87.68% using a shallow CNN architecture with pre-trained filters convolving the time and spectral dimensions separately in the same model. Handcrafted features for the Homburg dataset has been explored as well. The work of Panagakis et al. [25] achieved 63.46% using the auditory cortical representations in combination with their introduced classifier. Their work reports the accuracy achieved on the Ballroom dataset using the same features (cortical representations) and the classifier used for the Homburg, where they achieved 81.93% on the Ballroom dataset. The work in [24] achieved 62.4% on the Homburg dataset using auditory cortical representations, MFCC and Chroma as features. A neural network based attempt on the Homburg dataset in the work of Schluter et al. [31] achieved 45.5% using mcRBM [29], a variant of the RBM, applied on a mel-spectrogram.

Fig. 6 and Fig. 7 show the confusion matrix for Ballroom and the Homburg datasets, respectively. High confusion is noticed for the Rumba and the Waltz genres with the Slow Waltz, which overlap with the findings in [18]. For the Homburg dataset, less confusion is noticed with the availability of more samples in the genre category.

Fig. 6. Ballroom confusion using the MCLNN.

Fig. 7. Homburg confusion using the MCLNN.

In this work, we have explored the applicability of the ConditionaL Neural Network (CLNN) and the Masked ConditionaL Neural Network (MCLNN) on the music genre classification task. The CLNN preserves the inter-frames relation of a temporal signal and the spatial locality of the features. The MCLNN extends the CLNN by enforcing a systematic sparseness over the network’s links following a band-like pattern, which mimics a filterbank. The filterbank-like pattern induces the network to learn in frequency bands. The mask also automates the exploration of several feature combinations concurrently, which is usually a manual process of handcrafting the optimum feature combinations. The MCLNN has achieved competitive accuracies on the Ballroom and the Homburg music datasets compared to several handcrafted attempts, in addition to state-of-the-art Convolutional Neural Networks. The MCLNN has achieved these accuracies without depending on any musical perceptual properties used in several hand-crafted attempts, which allow the MCLNN to generalize to other types of multi-dimensional temporal signals. Future work, we will consider using deeper MCLNN architectures with more optimization to the masking patterns used, in addition to using different orders across the layers. We will also explore applying the MCLNN to multi-dimensional temporal representations other than spectrograms.

Acknowledgments. This work is funded by the European Union’s Seventh Framework Programme for research, technological development and demonstration under grant agreement no. 608014 (CAPACITIE).

1. Abdel-Hamid, O., Mohamed, A.R., Jiang, H., Deng, L., Penn, G., Yu, D.: Convolutional neural networks for speech recognition. IEEE/ACM Transactions on Audio, Speech and Language Processing 22(10), 1533–1545 (2014)

2. Aryafar, K., Shokoufandeh, A.: Music genre classification using explicit semantic analysis. In: International ACM workshop on Music Information Retrieval With User-Centered and Multimodal Strategies (MIRUM) (2011)

3. Battenberg, E., Wessel, D.: Analyzing drum patterns using conditional deep belief networks. In: International Society for Music Information Retrieval, ISMIR (2012)

4. Bergstra, J., Casagrande, N., Erhan, D., Eck, D., Kgl, B.: Aggregate features and adaboost for music classification. Machine Learning 65(2-3), 473–484 (2006)

5. Fahlman, S.E., Hinton, G.E., Sejnowski, T.J.: Massively parallel architectures for al: Netl, thistle, and boltzmann machines. In: National Conference on Artificial Intelligence, AAAI (1983)

6. Gouyon, F., Klapuri, A., Dixon, S., Alonso, M., Tzanetakis, G., Uhle, C., Cano, P.: An ex- perimental comparison of audio tempo induction algorithms. IEEE Transactions on Audio, Speech and Language Processing 14(5), 1832–1844 (2006)

7. Gouyon, F., Dixon, S., Pampalk, E., Widmer, G.: Evaluating rhythmic descriptors for musical genre classification. In: International AES conference (2004)

8. Hamel, P., Eck, D.: Learning features from music audio with deep belief networks. In: Inter- national Society for Music Information Retrieval Conference, ISMIR (2010)

9. Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural net- works. Science 313(5786), 504–7 (2006)

10. Homburg, H., Mierswa, I., Moller, B., Morik, K., MichaelWurst: A benchmark dataset for audio classification and clustering. In: International Symposium on Music Information Retrieval (2005)

11. Kereliuk, C., Sturm, B.L., Larsen, J.: Deep learning and music adversaries. IEEE Transac- tions on Multimedia 17(11), 2059–2071 (2015)

12. Kingma, D., Ba, J.: Adam: A method for stochastic optimization. In: International Confer- ence for Learning Representations, ICLR (2015)

13. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Neural Information Processing Systems, NIPS (2012)

14. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proceedings of the IEEE 86(11), 2278–2324 (1998)

15. Lidy, T., Rauber, A.: Evaluation of feature extractors and psycho-acoustic transformations for music genre classification. In: International Conference on Music Information Retrieval, ISMIR (2005)

16. Lidy, T., Rauber, A., Pertusa, A., Inesta, J.M.: Improving genre classification by combination of audio and symbolic descriptors using a transcription system. In: International Conference on Music Information Retrieval (2007)

17. Lin, M., Chen, Q., Yan, S.: Network in network. In: International Conference on Learning Representations, ICLR (2014)

18. Lykartsis, A., Lerch, A.: Beat histogram features for rhythm-based musical genre classifica- tion using multiple novelty functions. In: Conference on Digital Audio Effects (DAFx-15) (2015)

19. Marchand, U., Peeters, G.: The modulation scale spectrum and its application to rhythm- content description. In: International Conference on Digital Audio Effects (DAFx) (2014)

20. Medhat, F., Chesmore, D., Robinson, J.: Masked conditional neural networks for audio clas- sification. In: International Conference on Artificial Neural Networks (ICANN) (2017)

21. Moerchen, F., Mierswa, I., Ultsch, A.: Understandable models of music collections based on exhaustive feature generation with temporal statistics. In: ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) (2006)

22. Mohamed, A.R., Hinton, G.: Phone recognition using restricted boltzmann machines. In: IEEE International Conference on Acoustics Speech and Signal Processing, ICASSP (2010)

23. Osendorfer, C., Schluter, J., Schmidhuber, J., Smagt, P.v.d.: Unsupervised learning of low- level audio features for music similarity estimation. In: Workshop on Speech and Visual Information Processing in conjunction with the International Conference on Machine Learning (ICML) (2011)

24. Panagakis, Y., Kotropoulos, C.: Music classification by low-rank semantic mappings. EURASIP Journal on Audio Speech and Music Processing 2013, 1–15 (2013)

25. Panagakis, Y., Kotropoulos, C.L., Arce, G.R.: Music genre classification via joint sparse low-rank representation of audio features. IEEE/ACM Transactions on Audio, Speech, and Language Processing 22(12), 1905–1917 (2014)

26. Peeters, G.: Spectral and temporal periodicity representations of rhythm for the automatic classification of music audio signal. IEEE Transactions on Audio, Speech, and Language Processing 19(5), 1242–1252 (2011)

27. Pohle, T., Schnitzer, D., Schedl, M., Knees, P., Widmer, G.: On rhythm and general music similarity. In: International Society for Music Information Retrieval, ISMIR (2009)

28. Pons, J., Lidy, T., Serra, X.: Experimenting with musically motivated convolutional neural networks. In: International Workshop on Content-based Multimedia Indexing, CBMI (2016)

29. Ranzato, M., Hinton, G.E.: Modeling pixel means and covariances using factorized third- order boltzmann machines. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2551–2558 (2010)

30. Salamon, J., Bello, J.P.: Deep convolutional neural networks and data augmentation for en- vironmental sound classification. IEEE Signal Processing Letters 24(3) (2017)

31. Schluter, J., Osendorfer, C.: Music similarity estimation with the mean-covariance restricted boltzmann machine. In: International Conference on Machine Learning and Applications, ICMLA. pp. 118–123 (2011)

32. Seyerlehner, K., Schedl, M., Pohle, T., Knees, P.: Using block-level features for genre clas- sification, tag classification and music similarity estimation. In: Music Information Retrieval eXchange, MIREX (2010)

33. Seyerlehner, K., Widmer, G.: Fusing block-level features for music similarity estimation. In: International Conference on Digital Audio Effects (DAFx-10) (2010)

34. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: A sim- ple way to prevent neural networks from overfitting. Journal of Machine Learning Research, JMLR 15, 1929–1958 (2014)

35. Taylor, G.W., Hinton, G.E., Roweis, S.: Modeling human motion using binary latent vari- ables. In: Advances in Neural Information Processing Systems, NIPS. pp. 1345–1352 (2006)

36. Tzanetakis, G., Cook, P.: Musical genre classification of audio signals. IEEE Transactions On Speech And Audio Processing 10(5) (2002)

37. Vapnik, V., Lerner, A.: Pattern recognition using generalized portrait method. Automation and Remote Control 24, 774780 (1963)