The power of neural networks [1] in classifying linearly non-separable data lies in the use of multiple (at least two) layers. We take inspiration from the recent study of Gidon et al. [2] and propose a simpler yet more effective approach: a new computational model of the neuron, termed pyramidal neuron with apical dendrite activations (PyNADA), along with a novel activation function, termed apical dendrite activation (ADA), allowing us to classify linearly non-separable data using an individual neuron.
Biological motivation. Recently, Gidon et al. [2] observed that the apical dendrites of pyramidal neurons in the human cerebral cortex have a different activation function than what was previously known from observations on rodents. The newly-discovered apical dendrite activation function produces maximal amplitudes for electrical currents close to threshold-level stimuli and dampened amplitudes for stronger electrical currents, as shown in Figure 1a. This new discovery indicates that an individual pyramidal neuron from the human cerebral cortex can classify linearly non-separable data, contrary to the conventional belief that nonlinear problems require multi-layer neural networks. This is the main reason that motivated us to propose ADA and PyNADA.
Psychological motivation. Remember the first time you ate your favorite dish. Was it better than the second or the last time you ate the same dish? According to Knutson et al. [3], our brains provide higher responses to novel stimuli than to known (repetitive) stimuli. This means that our brains get bored while eating the same dish over and over again, although the dish might be our favorite. If we were to model the brain response over time for a certain stimulus, we would obtain the function illustrated in Figure 1a. This is
Fig. 1: Original [2] and proposed versions of the apical dendrite activation. The input corresponds to the horizontal axis and the output to the vertical axis.
yet another reason to propose and experiment with ADA and PyNADA in a computational framework based on deep neural networks, which try to mimic the brain.
Mathematical motivation. Despite the recent significant advances brought by deep learning [4] in various application domains [5, 6], state-of-the-art deep neural networks rely on an old and simple mathematical model of the neuron introduced by Rosenblatt [7]. Minsky and Papert [8] argued that a single artificial neuron is incapable of learning nonlinear functions, such as the XOR function. In order to classify linearly non-separable data, standard artificial neurons are typically organized into multi-layer neural networks that are equipped with at least one hidden layer. Contrary to the common belief, we propose an activation function (ADA) that transforms a single artificial neuron into a nonlinear classifier. We also prove that the nonlinear neuron can learn the XOR logical function with 100% accuracy. Hence, the ADA function can increase the computational power of individual artificial neurons.
Empirical motivation. We provide empirical evidence in favor of replacing the commonly-used (e.g. Rectified Liner Units (ReLU) [9] and leaky ReLU [10]) or the recently-proposed (e.g. Swish [11]) activation functions [12, 13] with the ones proposed in this work, namely ADA and leaky ADA, in various neural network architectures ranging from one-hidden-layer or two-hidden-layer multi-layer perceptrons (MLPs) to convolutional neural networks (CNNs) such as LeNet [14], VGG [15], ResNet [16] and Character-level CNN [17]. We obtain accuracy improvements on several tasks: object class recognition on Fashion-MNIST [18], ImageNet [19] and Tiny ImageNet [19]; gender prediction and age estimation on UTKFace [20]; voice emotion recognition on CREMAD [21]; Romanian dialect identification on MOROCO [22]. We report further accuracy improvements when the standard artificial neurons are replaced with our pyramidal neurons with apical dendrite activations.
Contribution. In summary, our contribution is threefold:
• We propose a new artificial neuron called the pyramidal neuron with apical dendrite activation (PyNADA), along with a new activation function called the apical dendrite activation (ADA).
• We demonstrate that, due to the novel apical dendrite activation, a single neuron can learn the XOR logical function.
• We show that the proposed neural building blocks, ADA and PyNADA, provide superior results compared to standard neurons based on the notorious ReLU and leaky ReLU activations, for a broad range of tasks and neural architectures. In most cases, our improvements are statistically significant.
Organization. The remainder of this work is organized as follows. In Section 2, we discuss related articles presenting activation functions and artificial neuron models. In Section 3, we present our novel apical dendrite activation function and pyramidal neuron. In Section 4, we present a comprehensive set of experiments with multiple neural networks on several data sets. Finally, we draw our conclusions and point out future work directions in Section 5.
4 Nonlinear Neurons with Human-like Apical Dendrite Activations
Since our work introduces a novel activation function as well as a novel type of artificial neuron, we consider articles presenting activation functions and artificial neuron models as closely related to our work. Therefore, we discuss related articles on activation functions in Section 2.1, and papers presenting models of artificial neurons in Section 2.2.
2.1 Activation Functions
Since activation functions [12, 13] have a large impact on the performance of deep neural networks (DNNs), studying and proposing new activation functions is an interesting and important topic [23]. Nowadays, perhaps the most popular activation function is ReLU [9, 12, 13]. Formally, ReLU is defined as max(0, x), where x is a scalar input. Because ReLU is linear on the positive side (for x > 0), its derivative is 1, so it does not saturate like sigmoid and tanh. On the negative side of the domain, ReLU is constant, so the gradient is 0. Hence, a neuron that uses ReLU as activation function cannot update its weights via gradient-based methods on examples for which the neuron is inactive. To eliminate the problem caused by inactive neurons with ReLU activation, Maas et al. [10] introduced leaky ReLU. The leaky ReLU function is defined as:
where l is a number between 0 and 1 (typically very close to 0), allowing the gradient to pass even if x < 0. While in leaky ReLU the leak parameter l is kept fixed, He et al. [24] proposed Parametric Rectified Linear Units (PReLU), in which the leak parameter l is learned by back-propagation. Different from ReLU, the Exponential Linear Unit (ELU) [25] outputs negative values, while still avoiding the vanishing gradient problem on the positive side of the domain. This helps to bring the mean unit activation down to zero, enabling faster convergence times. Another generalization of ReLU is the Maxout unit [26], which, instead of applying an element-wise function, divides the input into k groups of values and then outputs the maximum value across all groups.
In contrast to most recent (ReLU, PReLU, ELU, etc.) and historicallymotivated (sign, sigmoid, tanh, etc.) activation functions [12, 13], we propose an activation function that transforms a single artificial neuron into a nonlinear classifier. To support our statement, we prove that a neuron followed by our apical dendrite activation function can learn the XOR logical function. We note that there are other activation functions, e.g. Swish [11] and Radial Basis Function (RBF), that generate nonlinear decision boundaries. Different from Swish and RBF, our activation function is supported by recent neuroscience discoveries [2]. In addition, our experiments show that ADA and leaky ADA generally provide superior performance levels.
2.2 Models of Artificial Neurons
To the best of our knowledge, one of the first mathematical models of the biological neuron is the perceptron [7]. The Rosenblatt’s perceptron was introduced along with a rule for updating the weights, which converges to a solution only if the data set is linearly separable. Although the perceptron is a simple and old model, it represents the foundation of modern DNNs. Different from the Rosenblatt’s perceptron, the Adaptive Linear Neuron (ADALINE) [27] updates its weights via stochastic gradient descent, back-propagating the error before applying the sign function. ADALINE has the same disadvantage as Rosenblatt’s perceptron, namely that it cannot produce nonlinear decision boundaries.
More recently, researchers proposed the artificial spiking neuron [28–30], a model where the current state of the neuron is determined by the membrane potential, which can raise or decline for a period of time due to electrical impulses. To date, spiking neural network models are computationally expensive, being commonly avoided in real-world applications because of this problem.
In contrast to existing models of artificial neurons [7, 27–30], we propose an artificial neuron that has two input branches, the basal branch and the apical tuft. The apical tuft is particularly novel because it uses a novel activation function [2] that can solve nonlinearly separable problems.
There are a few works that studied various aspects of the modeling of pyramidal neurons, e.g. segregated dendrites in the context of deep learning [31] or memorizing sequences with active dendrites and multiple integration zones [32]. Inspired by the recent discovery of Gidon et al. [2], to the best of our knowledge, we are the first to propose a human-like artificial pyramidal neuron. Different from previous studies, the apical tuft of our pyramidal neuron is equipped with the novel apical dendrite activation suggested by Gidon et al. [2]. Furthermore, we integrate our pyramidal neuron into various deep neural architectures, showing its benefits over standard artificial neurons. We note that Gidon et al. [2] have not presented the pyramidal neuron in a computational scenario. Hence, we are the first to model it computationally.
We would like to emphasize that the proposed apical dendrite activation simulates a particular feature of the human brain, namely a class of calciummediated dendritic action potentials observed by analyzing the dendrites of layer 2/3 pyramidal neurons of the human cerebral cortex, as detailed by Gidon et al. [2]. Based on their neuroscientific discovery, we propose the ADA and leaky ADA functions, as well as the pyramidal neuron with apical dendrite activations (PyNADA). All our contributions represent novel building blocks that can be used to construct more powerful neural network architectures.
We next describe the proposed activation functions and pyramidal neuron. Our presentation starts with an introduction of the activation function in
6 Nonlinear Neurons with Human-like Apical Dendrite Activations
Section 3.1, which includes a proof of our statement regarding the capability of solving nonlinear problems, in particular the XOR classification problem, with a single neuron activated by our function. Our presentation continues with a description of the proposed pyramidal neuron in Section 3.2. Finally, in Section 3.3, we provide examples demonstrating how to integrate ADA and PyNADA into a neural network.
3.1 ADA: Apical Dendrite Activation Function
The activation function illustrated in Figure 1a is deduced from the ex vivo experiments conducted by Gidon et al. [2] on the human cerebral cortex. It can be formally expressed as follows:
where is the input of the activation function and y is the output.
However, the function defined in Equation (2) is not directly useful in practice, because commonly-used deep learning frameworks such as TensorFlow [33] and PyTorch [34] do not cope well with functions containing if branches. Consequently, we propose a closed form definition that approximates the activation function defined in Equation (2), as follows:
where is the input of the activation function,
0 is a parameter that controls the width of the peak, c > 0 is a constant that controls the height of the peak and y is the output. The input x of the activation function is the result of summing the weighted inputs of an artificial neuron. In the remainder of this section, we use bold letters to denote vectors and matrices. Given an input
and the learnable weights
and
of an artificial neuron, the input of the ADA function is computed as follows:
From Equations (3) and (4), we can obtain the output of the neuron activated by ADA illustrated in Figure 2, as follows:
Similar to ReLU, our apical dendrite activation (ADA) is saturated on the negative side, i.e. its gradients are equal to zero for x < 0. Thus, a neural model trained with back-propagation [35] would not update the corresponding weights. We therefore propose leaky ADA, a more generic version that avoids
Fig. 2: An artificial neuron with apical dendrite activation (ADA). The neuron computes the dot product between the input and the weight vector
, and passes the result through the ADA function.
saturation on the negative side, just as leaky ReLU. We formally extend the definition of ADA from Equation (3) to leaky ADA as follows:
where 0 1 is the leak parameter controlling the function steepness on the negative side and the other parameters are the same as in Equation (3). By setting l = 0.005,
6), we obtain the activation function illustrated in Figure 1b. By comparing Figure 1a and Figure 1b, we observe that the (leaky) ADA function defined in Equation (6) has a similar shape to the transfer function defined in Equation (2). Indeed, both functions have no activation or almost no activation when x < 0. Then, there is a high activation peak for small but positive values of x. Finally, the activation dampens along the horizontal axis, as x gets larger and larger. In Algorithm 1, we demonstrate how a neuron activated by ADA works. It essentially involves sequentially applying Eq. (4) and Eq. (3). Note that a neuron activated by leaky ADA is formalized analogously, by replacing Eq. (3) with Eq. (6) in Algorithm 1.
We next demonstrate that the ADA function can solve a nonlinearly separable problem (XOR). Previously, this type of problems were solved by neural networks using two layers. ADA is more powerful than a typical activation function (ReLU, PReLU, ELU, sigmoid, tanh, etc.), enabling a single neuron to solve the XOR nonlinearly separable problem.
Lemma 1 There exists an artificial neuron followed by the apical dendrite activation from Equation (3) which can predict the labels for the XOR logical function, by rounding its output.
8 Nonlinear Neurons with Human-like Apical Dendrite Activations
Proof Given an input data sample represented as a row vector , the output
of an artificial neuron with ADA is obtained as follows:
where are defined as in Equation (3), w is the column weight vector and b is the bias term. The following equation shows how to obtain the rounded output:
where is the rounding function. Similarly, we can obtain the rounded outputs for an entire set of data samples represented as row vectors in an input matrix X:
Let X and T represent the data samples and the targets corresponding to the XOR logical function, i.e.:
Since the output Y computed in Equation (11) is equal to the target T defined in Equation (10), it results that Lemma 1 is true.
Our proof is intuitively explained in Figure 3. The four data points from the XOR data set are represented on a plane and the output of the neuron followed by ADA is represented on the axis perpendicular to the plane in which the XOR points reside. The output for the red points (labeled as class 0) is 0 or close to 0, while the output for the green points (labeled as class 1) is 1.
Fig. 3: The output of a neuron with apical dendrite activation, as defined in Equation (7), is able to classify the XOR logical function. The output is obtained by setting the weights to w =
parameters of the ADA function to (closer to 1) correspond to the green data points labeled as class 1, while low output values (closer to 0) correspond to the red data points labeled as class 0. Best viewed in color.
Applying the rounding function on top of the output depicted in Figure 3 is equivalent to setting a threshold equal to 0.5, labeling all points above the threshold with class 1 and all points below the threshold with class 0. This gives us the labels for the XOR logical function.
Not surprisingly, ADA is still able to solve linearly separable functions, such as OR and AND, without requiring any modification to its definition. We demonstrate this below.
Corollary 2 There exists an artificial neuron followed by the apical dendrite activation from Equation (3) which can predict the labels for the OR logical function, by rounding its output.
Proof We can trivially prove Corollary 2 by following the proof for Lemma 1. We just have to set to different values, e.g.
Corollary 3 There exists an artificial neuron followed by the apical dendrite activation from Equation (3) which can predict the labels for the AND logical function, by rounding its output.
10 Nonlinear Neurons with Human-like Apical Dendrite Activations
Fig. 4: A pyramidal neuron with apical dendrite activations (PyNADA). The input x goes through the basal dendrites followed by ReLU and through the apical tuft followed by ADA. The results are summed up and passed through the axon. Best viewed in color.
Proof We can trivially prove Corollary 3 by following the proof for Lemma 1. We just have to set the bias term to a different value, e.g.
From Lemma 1, Corollary 2 and Corollary 3, it results that an artificial neuron followed by ADA has more computational power than a standard artificial neuron followed by sigmoid, ReLU or other commonly-used activation functions. More precisely, the ADA function enables individual artificial neurons to classify both linearly (e.g. AND, OR) and nonlinearly (e.g. XOR) separable data.
3.2 PyNADA: Pyramidal Neurons with Apical Dendrite Activations
Pyramidal neurons have two types of dendrites: apical dendrites and basal dendrites. Electrical impulses are sent to the neuron through both kinds of dendrites and the impulse is passed down the axon, if an action potential occurs. Prior to Gidon et al. [2], it was thought that apical and basal dendrites had identical activation functions. This is because experiments were usually conducted on pyramidal neurons extracted from rodents. In this context, proposing an artificial pyramidal neuron would not make much sense, because its mathematical model would be identical to a standard artificial neuron. Gidon et al. [2] observed that the apical dendrites of pyramidal neurons in the human cerebral cortex have a different (previously unknown) activation function, while the basal dendrites exhibit the well-known hard-limit transfer function. This observation calls for a new model of artificial pyramidal neurons. We therefore propose pyramidal neurons with apical dendrite activations (PyNADA).
Given an input data sample , the output
of a PyNADA is obtained through the following equation:
where and c are defined as in Equation (3),
and
are column weight vectors and
and
are bias terms. A graphical representation of PyNADA is provided in Figure 4. In the proposed model, the input x is distributed to the basal dendrites represented by the weight vector
and the bias term
, and to the apical dendrites (apical tuft) represented by the weight vector
and the bias term
. Formally, the computation carried out by a pyramidal neuron with apical dendrite activation is detailed in Algorithm 2.
For practical reasons, we replace the hard-limit transfer function, suggested by Gidon et al. [2] for the basal dendrites, with the ReLU activation. This change ensures that we can optimize the weights and the bias
through back-propagation, i.e. we have at least some non-zero gradients. Since the intensity of electrical impulses is always positive, the biological model proposed by Gidon et al. [2] is defined for positive inputs and the thresholds for the activation functions are well above 0. However, an artificial neuron can also take as input negative values, i.e.
. Hence, the thresholds of the activation functions used in PyNADA are set to 0 (but the bias terms can shift these thresholds).
Corollary 4 There exists a pyramidal neuron with apical dendrite activation, as defined in Equation (12), which can predict the labels for the XOR logical function, by rounding its output.
12 Nonlinear Neurons with Human-like Apical Dendrite Activations
Fig. 5: An example explaining how ReLU can be replaced with ADA in a standard neural network. Besides changing the activations, the neural architecture remains unchanged. Best viewed in color.
Proof We can trivially prove Corollary 4 by following the proof for Lemma 1. For the apical tuft, we can simply set
just as in the proof for Lemma 1. The demonstration results immediately when we simply drop out the basal dendrites by setting
By proposing PyNADA, we can jointly leverage the capability of solving nonlinear problems of ADA, and the simplicity and speed of the ReLU activation function, creating more powerful models.
a A one-hidden-layer MLP with pyramidal neurons activated by ReLU on both basal and apical tufts, denoted as PyNReLU.
b A one-hidden-layer MLP with pyramidal neurons activated by ReLU and ADA, denoted as PyNADA.
Fig. 6: An example explaining how to use PyNReLU and PyNADA units in a standard neural network. Besides switching between ReLU and ADA for the apical tuft, the neural architectures are identical. Best viewed in color.
3.3 Neural Architectures with ADA and PyNADA
Both ADA and PyNADA are powerful neural building blocks that can increase a network’s nonlinear discrimination capacity. Furthermore, they can be easily integrated into any network architecture, without changing the overall structure of the respective neural network. In Figure 5, we illustrate how ADA
14 Nonlinear Neurons with Human-like Apical Dendrite Activations
can replace the ReLU activation in a one-hidden-layer MLP architecture with 4 hidden neurons and 2 output neurons. In Figure 5a, we depict the regular architecture based on ReLU activations, while in Figure 5b, we illustrate an analogous neural network where ReLU is replaced with ADA. It is fairly easy to observe that replacing ReLU with ADA implies no other changes.
Similarly, in Figure 6, we showcase how a one-hidden-layer MLP based on pyramidal neurons should look like. We present two configurations, one where ReLU activations are used for both branches of the pyramidal neurons, called PyNReLU (illustrated in Figure 6a), and one where we use ReLU for the basal branch and ADA for the apical branch of each neuron, called PyNADA (illustrated in Figure 6b). Employing pyramidal neurons as in Figure 6 doubles the number of learnable parameters of the network with respect to using standard neurons, as in Figure 5. To this end, we take PyNReLU as the baseline for PyNADA, since both of them share the same number of weights. We once again emphasize that the architectural changes are straightforward and minimal.
Finally, we underline that the architectural changes illustrated in Figure 5 and Figure 6 are directly transferable to any type of neural network architecture.
To demonstrate the practical abilities of the proposed activation function and artificial neuron, we conduct experiments on six data sets: MOROCO [22], UTKFace [20], CREMA-D [21], Fashion-MNIST [18], ImageNet [19] and Tiny ImageNet. In these experiments, we compare our contributions to state-of-the-art activation functions, alternatively inserting the respective activations and our activation into several neural network architectures. In Section 4.1, we provide details about the chosen data sets. In Section 4.2, we present the generic evaluation setup used across all six data sets. We discuss the results on the six data sets in Sections 4.3, 4.4, 4.5, 4.6, 4.7 and 4.8, respectively. We present a series of ablation results in Section 4.9. We demonstrate the capabilities of approximating nonlinear functions with neural networks based on various activation functions in Section 4.10. We conclude the experiments with the discussion provided in Section 4.11.
4.1 Data Sets
MOROCO. We conduct text classification experiments on MOROCO [22], a data set of 33,564 news articles that are written in Moldavian or Romanian. Each text sample has an average of 309 tokens, the total number of tokens being over 10 millions. The main task is to discriminate between the Moldavian and the Romanian dialects. The data set comes with a training set of 21,719 text samples, a validation set of 5,921 text samples and a test set of 5,924 text samples.
UTKFace. UTKFace [20] is a large data set of 23,708 high-resolution images. The images contain faces of people of various ages, genders and ethnic groups.
We use the unaligned cropped faces in our experiments. We randomly divide the data set into a training set of 16,595 images (70%), a validation set of 3,556 images (15%) and a test set of 3,557 images (15%). We consider two tasks on UTKFace: gender recognition (binary classification) and age estimation (regression).
CREMA-D. The CREMA-D multimodal database [21] contains 7,442 videoclips of 91 actors (48 male and 43 female) with different ethnic backgrounds. The actors were asked to convey particular emotions while producing, with different intonations, 12 particular sentences that evoke the target emotions. Six labels have been used to discriminate among different emotion categories: neutral, happy, anger, disgust, fear and sad. In our experiments, we consider only the audio modality. We split the audio samples into 70% for training, 15% for validation and 15% for testing.
Fashion-MNIST. Fashion-MNIST [18] is a recently introduced data set that shares the same structure as the more popular MNIST [14] data set, i.e. it contains 60,000 training images and 10,000 test images that belong to 10 classes of fashion items. We use a subset of 10,000 images from the training set for validation.
ImageNet. The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) [19] is based on a data set of 1000 object categories and about 1.2 million images. We use the official validation set of 50,000 images for comparing the models.
Tiny ImageNet. We present results with longer apical dendrites on Tiny ImageNet, a subset of ImageNet [19]. The data set provides 500 training images, 50 validation images and 50 test images for 200 object classes. The size of each image is 64 64 pixels.
4.2 Evaluation Setup
Evaluation metrics. For the classification tasks (object recognition, gender prediction, voice emotion recognition and dialect identification), we report the classification accuracy. For the regression task (age estimation), we report the mean absolute error (MAE).
Baselines. We consider several neural architectures ranging from shallow MLPs to deep CNNs: one-hidden-layer MLP, two-hidden-layer MLP, LeNet, VGG-9, ResNet-18, ResNet-50 and character-level CNNs with and without Squeeze-and-Excitation (SE) blocks [36]. The baseline architectures are based on ReLU, leaky ReLU, RBF or Swish. The latter two activation functions are added because they share one property with ADA, namely the capability to solve XOR when plugged into a single artificial neuron. The goal of our experiments is to study the effect of (i) replacing ReLU and leaky ReLU with ADA and leaky ADA, respectively, and (ii) replacing the standard neurons with our PyNADA. The number of weights in PyNADA is twice as high, but the size of the activation maps is the same as in standard neurons (due to summation of the two branches in PyNADA). For a fair comparison, we included three baseline pyramidal neurons: one with ReLU on both branches (PyNReLU), one
16 Nonlinear Neurons with Human-like Apical Dendrite Activations
with ReLU and RBF (PyNRBF) and one with ReLU and Swish (PyNSwish). All neural models are trained using the Adam optimizer [37]. With the exception of ResNet-18 and ResNet-50, which are implemented in PyTorch [34], all other neural networks are implemented in TensorFlow [33]. Since we employ different architectures in each task, we describe them in more detail in the corresponding subsections below.
In summary, we emphasize that our experiments aim to compare models within the following three groups: (i) ReLU, RBF, Swish versus ADA; (ii) leaky ReLU versus leaky ADA; (iii) PyNReLU, PyNRBF, PyNSwish versus PyNADA and leaky PyNADA. Note that, for PyNADA, we consider both standard and leaky activations. In leaky PyNADA, the basal dendrites are followed by leaky ReLU and the apical dendrites are followed by leaky ADA.
Significance testing [38] is performed within each group, considering the above organization. We also emphasize that the compared models within a group have the same number of parameters and the same computational complexity, e.g. we do not aim to compare ReLU with PyNADA directly.
We underline that we replace all activations or neurons inside each neural network. For example, when we compare ReLU, RBF, Swish and ADA using a certain neural network, all the activations of the respective network are of the same kind, as illustrated in Figure 5. The same applies to leaky ReLU and leaky ADA. The only case with two types of activation functions in the same network is when we use pyramidal neurons. Each pyramidal neuron has two branches (basal and apical), one with ReLU and the other with one of the following activations: ReLU, RBF, Swish and ADA. When experimenting with pyramidal neurons, we replace all neurons in a neural network with pyramidal neurons, as illustrated in Figure 6. Moreover, when changing the activation functions or the type of neurons (from conventional to pyramidal), we do not modify the depth of the underlying neural networks. Generic hyperparameter tuning. We tune the basic hyperparameters such as the learning rate, the mini-batch size and the number of epochs, for each neural architecture using grid search on the validation set of the corresponding task. We consider learning rates between 10and 10
, mini-batches of 10, 32, 64, 128 or 800 samples, and numbers of epochs between 10 and 300. Other parameters of the Adam optimizer are used with default values. We use Xavier initialization across all models, regardless of the activation function type or the depth of the network. The parameter tuning is performed using the baseline architectures based on ReLU. Once tuned, the same parameters are used for leaky ReLU, RBF, Swish, ADA, leaky ADA, PyNReLU, PyNRBF, PyNSwish, PyNADA and leaky PyNADA. Hence, we emphasize that the basic parameters are tuned in favor of ReLU. Following Maas et al. [10], the leak parameter for leaky ReLU and leaky ADA is set to l = 0.01 in all the experiments, without further tuning. We note that (leaky) ADA and (leaky) PyNADA have two additional hyperparameters that require tuning on validation. For the constant c, we consider two possible values, either 0 or 1. For the parameter
, we consider two options: (a) perform grid search in the range [0.1, 1] using a step
Fig. 7: Bar chart of the results obtained on the MOROCO data set by different versions of the character-level CNN [22] architecture with both standard and pyramidal neurons activated by various functions: ReLU, RBF, Swish, ADA, leaky ReLU and leaky ADA. Best viewed in color.
of 0.1, or (b) learn using gradient descent during training. As recommended in [11], we use a learnable
in Swish. We highlight once again that the same activation function is used in every layer of the entire network, regardless of the depth of the model. Thus, we establish the activation function hyperparameters at the architecture level.
To avoid accuracy variations due to weight initialization or stochastic training, we train each neural model in five consecutive trials, keeping the model with the highest validation performance. In the experiments, we report the performance level of the selected neural models on the held-out test set.
4.3 Results on MOROCO
Neural architectures. For dialect identification, we consider the character-level CNN models presented in [22], which follow closely the model of Zhang et al. [17]. The two CNNs share the same architecture, being composed of an embedding layer, followed by three convolutional and max-pooling blocks, two fully-connected layers with dropout 0.5, and the final softmax classification layer. The second architecture incorporates an attention mechanism in
18 Nonlinear Neurons with Human-like Apical Dendrite Activations
Table 1: Dialect identification accuracy rates (in %) for two character-level neural models (CNN and CNN+SE) on MOROCO. Results are reported with various activations (ReLU, leaky ReLU, RBF, Swish, ADA, leaky ADA) and artificial neurons (standard, PyNReLU, PyNRBF, PyNSwish and PyNADA). Results that are significantly better than the corresponding baselines, according to a paired McNemar’s test [38], are marked with or
for the significance levels 0.05 or 0.01, respectively. Training times are measured on a computer with Nvidia GeForce GTX 1080 GPU with 11GB of RAM. Best model within each group is highlighted in bold.
the form of Squeeze-and-Excitation (SE) blocks [36] inserted after every convolutional layer. For the SE blocks, we set the reduction ratio to 64. For both architectures, we keep the same size for the embedding (256) and the same number of convolutional filters (128) as Butnaru et al. [22]. In fact, our baseline architectures (CNN and CNN+SE) are identical to those of Butnaru et al. [22].
Fig. 8: Bar chart of the results obtained on the MOROCO data set by different versions of the character-level CNN+SE [22] architecture with both standard and pyramidal neurons activated by various functions: ReLU, RBF, Swish, ADA, leaky ReLU and leaky ADA. Best viewed in color.
Specific hyperparameter tuning. Since Butnaru et al. [22] already tuned the hyperparameters of the character-level CNNs on the MOROCO validation set, we decided to use the same parameters and skip the grid search. Hence, we set the learning rate to 5 10
and use mini-batches of 128 samples. Each CNN is trained for 50 epochs in 5 trials, keeping the model with the highest validation accuracy for evaluation on the test set. For (leaky) ADA and (leaky) PyNADA, we obtain optimal results with
is either validated or optimized during training.
Results. We present the dialect identification results on MOROCO in Table 1. In addition to Table 1, we illustrate the results for both standard and pyramidal neurons in Figure 7 (for the CNN [22] architecture) and Figure 8 (for the CNN+SE [22] architecture). First, we observe the baseline CNN and CNN+SE models confirm the results reported by Butnaru et al. [22]. For the character-level CNN (without SE blocks), we obtain the largest improvement on the test set when we replace ReLU (92.79%) with ADA (93.68%). Furthermore, the improvements of leaky ADA, PyNADA and leaky PyNADA are all higher than 0.6%, and the differences are statistically significant. The results are somewhat consistent among the two architectures, CNN and CNN+SE. For example, for
20 Nonlinear Neurons with Human-like Apical Dendrite Activations
the CNN+SE model, we report the largest improvement by replacing ReLU (92.99%) with ADA (93.99%), just as for the CNN without SE blocks. Our highest absolute gain on MOROCO is 1%. Overall, the results indicate that all variants of (leaky) ADA and (leaky) PyNADA obtain significantly better results than the corresponding baselines. We observe that neither character-level CNN model is able to converge using RBF as activation. For RBF, we reported the test accuracy corresponding to the last model before the gradients explode. Interestingly, PyNRBF is able to converge, probably due to the basal tuft based on ReLU, but its performance level is not adequate. The networks converge with Swish and PyNSwish, but the corresponding results are much lower than those with (leaky) ReLU or (leaky) ADA. Running times. Since ADA needs to compute the exponential function, it is more computational intensive than ReLU. Moreover, PyNADA has twice more weights than a standard neuron. Hence, in addition to the accuracy rates, we hereby report the training time (in seconds per epoch) in Table 1. With respect to (leaky) ReLU, it seems that (leaky) ADA requires 5 to 7 additional seconds per epoch, which means that the training times increases by 15% or 20%. The other activation functions that include the exponential function, e.g. RBF and Swish, have the same disadvantage as ADA.
Meanwhile, (leaky) PyNADA seems to need about 25 to 30 extra seconds compared to (leaky) ReLU, increasing the training time by 60% to 100%. We thus conclude that the accuracy improvements brought by (leaky) ADA and (leaky) PyNADA come with a non-negligible computational cost with respect to ReLU or leaky ReLU. At the same time, RBF and Swish are slower than ReLU, while also yielding inferior accuracy levels.
As the time measurements are consistent across all benchmarks, we refrain from reporting and commenting on the running times in the subsequent experiments.
4.4 Results on UTKFace
Neural architecture. For gender prediction and age estimation, we employ the ResNet-50 architecture [16]. Residual networks use batch normalization and skip connections to propagate information over convolutional layers, avoiding the vanishing or exploding gradient problem. This enables effective training of very deep models such as ResNet-50, which is formed of 50 layers.
Specific hyperparameter tuning. All ResNet-50 variants are trained on mini-batches of 10 samples using a learning rate of 10. The models are trained for 15 epochs on the gender prediction task, and for 100 epochs on the age estimation task. For (leaky) ADA and (leaky) PyNADA, we either validate or learn the parameter
, at the same time setting the parameter c to 0.
Results. We present the gender prediction and age estimation results in Table 2. In addition, we illustrate the results on gender prediction for both standard and pyramidal neurons in Figure 9. In the gender prediction task, we notice that ADA yields slightly lower results than ReLU, while leaky ADA attains slightly better results than leaky ReLU. Both RBF and Swish
Table 2: Gender prediction accuracy rates (in %) and age estimation MAEs for ResNet-50 on UTKFace. Results are reported with various activations (ReLU, leaky ReLU, RBF, Swish, ADA, leaky ADA) and artificial neurons (standard, PyNReLU, PyNRBF, PyNSwish and PyNADA). Results that are significantly better than the corresponding baselines, according to a paired McNemar’s test [38], are marked with for the significance level 0.01. Best model within each group is highlighted in bold.
provide lower results than ReLU and ADA. Nevertheless, we observe significant improvements with PyNADA over PyNReLU. Our highest absolute gain (3.15%) in the gender prediction task is obtained when ResNet-50 is equipped with PyNADA (91.65%) instead of the baseline PyNReLU (88.50%).
Aside from Table 2, we also present the results on age estimation for both standard and pyramidal neurons in Figure 10. In the age estimation task, we notice that all versions of (leaky) ADA and (leaky) PyNADA surpass the
22 Nonlinear Neurons with Human-like Apical Dendrite Activations
Fig. 9: Bar chart of the gender prediction results on the UTKFace data set obtained by different versions of the ResNet-50 architecture with both standard and pyramidal neurons activated by various functions: ReLU, RBF, Swish, ADA, leaky ReLU and leaky ADA. Best viewed in color.
corresponding baselines by significant margins. With an MAE of 5.74 on the test set, our PyNADA attains the best results in age estimation. With respect to the PyNReLU baseline, PyNADA reduces the average error by 1.61 years and the difference is statistically significant.
4.5 Results on CREMA-D
Neural architecture. For speech emotion recognition, we employ the ResNet-18 architecture [16]. We modify the number of input channels of ResNet-18 from 3 to 2, in order to feed the network with the Short Time Fourier Transform of the raw audio signals, where the real and the imaginary parts are considered as separate input channels.
Specific hyperparameter tuning. All ResNet-18 variants are trained on mini-batches of 16 samples using a learning rate of 5 10
. The models are trained for 70 epochs. For (leaky) ADA and (leaky) PyNADA, we either validate or learn the parameter
, at the same time setting the parameter c to 0. Moreover, we apply training data augmentation using time shifting,
Fig. 10: Bar chart of age estimation results on the UTKFace data set obtained by different versions of the ResNet-50 architecture with both standard and pyramidal neurons activated by various functions: ReLU, RBF, Swish, ADA, leaky ReLU and leaky ADA. Best viewed in color.
thus obtaining more training samples which improve the robustness to data variation of ResNet-18.
Results. We present the speech emotion recognition results in Table 3. In addition, we illustrate the results for the emotion recognition task in Figure 11. We observe that ADA and leaky ADA attain superior results compared with the other activation functions, when conventional neurons are employed in ResNet-18. The RBF activation function offers significantly lower results with respect to the most commonly-used activation function, ReLU, while the Swish function fails to converge for both standard and pyramidal neurons. We generally observe that the ResNet-18 based on pyramidal neurons outperforms the ResNet-18 based on conventional neurons, regardless of the activation function. This highlights the effectiveness of the pyramidal design. Our highest absolute gain (2.24%) is obtained for leaky ADA in comparison with leaky ReLU. Moreover, the performance gains brought by (leaky) ADA and (leaky) PyNADA are statistically significant with respect to the corresponding baselines. We notice that our accuracy rates for the audio modality on CREMA-D surpass the state-of-the-art accuracy levels reported in [39, 40]. Given that we
24 Nonlinear Neurons with Human-like Apical Dendrite Activations
Table 3: Voice emotion recognition accuracy rates (in %) for ResNet-18 on CREMA-D. Results are reported with various activations (ReLU, leaky ReLU, RBF, Swish, ADA, leaky ADA) and artificial neurons (standard, PyNReLU, PyNRBF, PyNSwish, PyNADA). Results that are significantly better than the corresponding baselines, according to a paired McNemar’s test [38], are marked with for the significance level 0.01. As reference, the state-of-the-art methods are also included. Best model within each group is highlighted in bold.
report significant improvements over baselines that are already superior to the state of the art, we consider that our results on CREMA-D are remarkable.
4.6 Results on Fashion-MNIST
Neural architectures. For the Fashion-MNIST data set, we consider two MLPs and two CNNs (LeNet, VGG-9). We kept the same design choices as in the original paper [15] for VGG-9, but for LeNet [14], we replaced the average-pooling layer with max-pooling. The first MLP architecture (MLP-1) is composed of one hidden layer with 100 units and one output layer with 10 units (the number of classes). The second MLP has two hidden layers with 100 units and 10 units, respectively, followed by the output layer with another 10 units. The considered MLP architectures are similar to those attaining better results among the MLP architectures evaluated by Xiao et al. [18].
Specific hyperparameter tuning. We train LeNet for 30 epochs, using a learning rate of 10for the first 15 epochs and 10
for the last 15 epochs. We train MLP-1 and MLP-2 in the same manner as LeNet. However, VGG-9 is trained for 100 epochs, starting with a learning rate of 10
in the first 50 epochs, decreasing it to 10
in the last 50 epochs. We train all models on
Fig. 11: Bar chart of the emotion classification results on the CREMA-D data set obtained by two state-of-the-art methods [39, 40], as well as different versions of the ResNet-18 architecture with both standard and pyramidal neurons activated by various functions: ReLU, RBF, Swish, ADA, leaky ReLU and leaky ADA. Best viewed in color.
mini-batches of 64 images. In all the experiments with (leaky) ADA or (leaky) PyNADA, we either validate or learn , and we set the parameter c to 0. Results. We present the Fashion-MNIST results with various neural architectures and activation functions in Table 4 and Table 5. In addition, we illustrate the results with both standard and pyramidal neurons in Figure 12 (for the one-hidden-layer MLP architecture), Figure 13 (for the two-hidden-layer MLP architecture), Figure 14 (for the LeNet architecture) and Figure 15 (for the VGG-9 architecture).
Our baseline MLP architectures obtain better accuracy rates than those of Xiao et al. [18], e.g. the difference obtained for MLP-1 with ReLU is 1.78% (we report 88.88%, while Xiao et al. [18] report 87.10%). We notice that leaky ReLU obtains slightly lower accuracy rates than ReLU, the only exception being the result with LeNet. Nonetheless, for both MLP and CNN architectures, the accuracy of ADA on the test set is superior (by up to 0.5%)
26 Nonlinear Neurons with Human-like Apical Dendrite Activations
Table 4: Object class recognition accuracy rates (in %) for two MLPs (one-hidden-layer MLP and two-hidden-layer MLP) on Fashion-MNIST. Results are reported with various activations (ReLU, leaky ReLU, RBF, Swish, ADA, leaky ADA) and artificial neurons (standard, PyNReLU, PyNRBF, PyNSwish and PyNADA). Results that are significantly better than the corresponding baselines, according to a paired McNemar’s test [38], are marked with for the significance levels 0.05 or 0.01, respectively. As reference, the results with one-hidden-layer and two-hidden-layer MLPs reported by Xiao et al. [18] are also included. Best model within each group is highlighted in bold.
compared to the accuracy of ReLU. We obtain statistically significant improvements for the replacement of ReLU with ADA, from 90.84% to 91.34% using LeNet, and from 93.39% to 93.84% using VGG-9, respectively. Interestingly, we also noticed that the value of the cross-entropy loss is always lower (on both validation and test sets) when we use ADA instead of ReLU.
We observe that RBF converges only for small networks (MLP-1, MLP-2 or LeNet). When it converges, the results are below the ReLU and leaky ReLU
Fig. 12: Bar chart of the classification results on the Fashion-MNIST data set obtained by different versions of the one-hidden-layer MLP [18] architecture with both standard and pyramidal neurons activated by various functions: ReLU, tanh, RBF, Swish, ADA, leaky ReLU and leaky ADA. Best viewed in color.
baselines. For VGG-9, the accuracy of RBF is equal to the random choice baseline. Swish attains better results than ReLU for MLP-1 and MLP-2, but below ADA and leaky ADA. For LeNet and VGG-9, Swish surpasses only RBF (all other activation functions are better). When we employ our PyNADA, the accuracy rates improve for all the architectures. The largest improvement of leaky PyNADA on the test set (with respect to the baseline PyNReLU) is 0.73%, obtained with LeNet. The reported difference is statistically significant.
Finally, we emphasize that the results presented in Table 4, based on the one-hidden-layer MLP and two-hidden-layer MLP architectures, confirm that the complexity of the network can be reduced by using ADA, leaky ADA,
PyNADA or leaky PyNADA. For example, the one-hidden-layer MLP based on ADA (88.98%) outperforms the deeper two-hidden-layer MLPs based on ReLU (88.71%), tanh (86.30%), RBF (87.84%) and Swish (88.92%). Similarly, the one-hidden-layer MLP based on leaky ADA (88.97%) outperforms the deeper two-hidden-layer MLP based on leaky ReLU (88.18%). Moreover, the one-hidden-layer MLPs based on PyNADA (89.45%) and leaky PyNADA (89.34%) outperform the deeper two-hidden-layer MLPs based on PyNReLU (88.98%), PyNRBF (88.43%) and PyNSwish (88.88%).
28 Nonlinear Neurons with Human-like Apical Dendrite Activations
Fig. 13: Bar chart of the classification results on the Fashion-MNIST data set obtained by different versions of the two-hidden-layer MLP [18] architecture with both standard and pyramidal neurons activated by various functions: ReLU, tanh, RBF, Swish, ADA, leaky ReLU and leaky ADA. Best viewed in color.
4.7 Results on Tiny ImageNet
We note that, in order to follow exactly the connectivity of pyramidal neurons, the apical dendrites should be longer, i.e. not connected to neurons in the immediately preceding layer 1, but to neurons in layers
3 or perhaps even further apart. With this design change, the apical dendrites will act as a kind of skip-connections. We hereby show some results in this direction, although this aspect can be studied considering various architecture configurations in future work. Our goal is to prove that the biological design of pyramidal neurons is viable for artificial neural networks as well.
Neural architecture. For object recognition on Tiny ImageNet, we consider a ResNet-18 architecture [16] without standard skip-connections. Instead of standard skip-connections, we use PyNADA with longer apical dendrites, closely modeling the biological pyramidal neurons. Using longer apical dendrites introduces new parameters (e.g., where to connect the apical dendrites) and constraints (the pyramidal design does not apply to every network architecture). The ResNet-18 model allows us to connect the apical dendrites to layer 3 instead of the immediately preceding layer
1. As a first baseline for this experiment, we use PyNReLU with equally-long apical dendrites, but
Table 5: Object class recognition accuracy rates (in %) for two neural models (LeNet and VGG-9) on Fashion-MNIST. Results are reported with various activations (ReLU, leaky ReLU, RBF, Swish, ADA, leaky ADA) and artificial neurons (standard, PyNReLU, PyNRBF, PyNSwish and PyNADA). Results that are significantly better than the corresponding baselines, according to a paired McNemar’s test [38], are marked with for the significance levels 0.05 or 0.01, respectively. Best model within each group is highlighted in bold.
Table 6: Object class recognition accuracy rates (in %) for ResNet-18 on Tiny ImageNet. Results are reported with two artificial neurons, PyNReLU and PyNADA, respectively. Results significantly better than the baseline, according to a paired McNemar’s test [38], are marked with for the significance level 0.01. Best model is highlighted in bold.
30 Nonlinear Neurons with Human-like Apical Dendrite Activations
Fig. 14: Bar chart of the classification results on the Fashion-MNIST data set obtained by different versions of the LeNet architecture with both standard and pyramidal neurons activated by various functions: ReLU, RBF, Swish, ADA, leaky ReLU and leaky ADA. Best viewed in color.
with ReLU instead of ADA for the apical tuft. The ResNet-18 with PyNReLU is similar to ResNet-18 with standard skip-connections, the difference being that the skip-connections have learnable weights and ReLU activations. We additionally consider two more baselines with similar design: PyNRBF and PyNSwish.
Specific hyperparameter tuning. The ResNet-18 models with PyNReLU, PyNRBF, PyNSwish and PyNADA are each trained for 120 epochs using a learning rate of 10and mini-batches of 200 samples. For PyNADA, we obtain optimal results with
1 (obtained through validation). For
Swish, the parameter is learnable.
Results. We present the object recognition results on Tiny ImageNet in Table 6. First of all, we note that, without pre-training on ImageNet, the accuracy rates on Tiny ImageNet reported in literature are typically around 50% or 60%. The baseline PyNReLU attains a fair accuracy of 51.54%. PyNADA brings a statistically significant improvement of 2.32% over the baseline. The accuracy rates reached by PyNRBF and PyNSwish are between the accuracy rates of PyNReLU and PyNADA. This experiment shows that it is useful to consider longer apical dendrites in conjunction with ADA, as observed in biology.
4.8 Results on ImageNet
Neural architecture. For object recognition on ImageNet, we employ the ResNet-50 architecture [16]. Since one experiment takes about two weeks
Fig. 15: Bar chart of the classification results on the Fashion-MNIST data set obtained by different versions of the VGG-9 architecture with both standard and pyramidal neurons activated by various functions: ReLU, RBF, Swish, ADA, leaky ReLU and leaky ADA. Best viewed in color.
Table 7: Object class recognition accuracy rates (in %) for top 5 predictions of a ResNet-50 on ImageNet. Results are reported with ReLU and ADA activations and two artificial neurons, PyNReLU and PyNADA, respectively. Results significantly better than the corresponding baseline, according to a paired McNemar’s test [38], are marked with for the significance level 0.01. Best model within each group is highlighted in bold.
on our machine (equipped with two Nvidia GeForce GTX 1080 GPUs with 11GB of RAM), we restrict our comparisons to ReLU versus ADA on the one hand, and PyNReLU versus PyNADA on the other hand. For both PyNReLU and PyNADA, we connect the apical dendrites to layer 3 instead of the immediately preceding layer
1, thus following the same design used in the experiments carried out on Tiny ImageNet.
32 Nonlinear Neurons with Human-like Apical Dendrite Activations
Specific hyperparameter tuning. The ResNet-50 models with ReLU, ADA, PyNReLU and PyNADA are each trained for 300 epochs using a learning rate of 10and mini-batches of 800 samples. For ADA and PyNADA, we obtain optimal results with
2 (determined through validation).
Results. We present the object recognition results on ImageNet in Table 7. We observe that the baseline ResNet-50 with ReLU activations attains an accuracy of 79.31%, while the same architecture with ADA leads to a higher accuracy of 79.52%. The same trend can be observed while comparing the pyramidal architectures, PyNReLU and PyNADA. More precisely, the ResNet-50 based on PyNADA surpasses the baseline ResNet-50 based on PyNReLU by 0.2%. These experiments shows once again that ADA and PyNADA outperform the counterparts based on ReLU and PyNReLU, respectively. Furthermore, the ImageNet experiments confirm the Tiny ImageNet experiments, indicating that longer apical dendrites (to layer 3) in pyramidal neurons are useful.
4.9 Ablation Results
Next, we present an ablation study to assess the influence of the hyperparameter on the training stability of ADA. We also present experiments on how input noise, labeling noise or the imbalanced nature of the data set affect the performance obtained by ADA and the other competing approaches.
The hyperparameter
defined in Equation (3) controls the width of the peak, meaning that it inflates or deflates the non-asymptotic region, which can alleviate the vanishing gradient problem. In order to demonstrate that the ADA function leads to stable training, we perform additional experiments on Fashion-MNIST with the MLP-2 and VGG-9 models, while varying the parameter
. More precisely, we vary
between 0.1 and 1.0 with a step of 0.1, reporting the performance for each value of
in Figure 16. When varying
, the performance of the two-hidden-layer MLP fluctuates between 87.3% and 88.9%, while the performance of VGG-9 fluctuates between 93.5% and 93.9%. These results indicate that the training converges to good optima, regardless of the value of
. We thus conclude that these experiments do not show slow convergence, demonstrating that the possibility of encountering vanishing gradients is not to be expected.
Effect of input noise. We study the behavior of ADA and PyNADA when training and testing on noisy images by performing experiments on the Fashion-MNIST data set with two different models, namely MLP-2 and VGG-9. Fashion-MNIST is a clean data set, meaning that the image samples do not contain noise. To simulate a data set of noisy images, we add Gaussian noise of 0 mean and different standard deviations, namely . We can visualize some randomly chosen images after applying the noise with different standard deviations in Figure 17. We carry out experiments to compare the ADA and leaky ADA functions with ReLU and leaky ReLU, respectively. Likewise, we compare PyNReLU with PyNADA and leaky PyNADA. The corresponding results are presented in Figure 18 and Figure 19. We observe that the performance of ADA surpasses the performance of ReLU, regardless
Fig. 16: Object class recognition accuracy rates (in %) for two neural models (MLP-2 and VGG-9) on Fashion-MNIST. Results are reported with the ADA function, while varying the hyperparameter defined in Equation (3). The training converges for all values of
, without showing signs of slow convergence or vanishing gradients.
Fig. 17: Examples of input images from Fashion-MNIST, with various levels of Gaussian noise. We apply Gaussian noise of 0 mean and different standard deviations (shown on different rows).
34 Nonlinear Neurons with Human-like Apical Dendrite Activations
Fig. 18: Object class recognition accuracy rates (in %) for a two-hidden-layer MLP on Fashion-MNIST, with various levels of Gaussian noise applied on the input images. Results are reported with various activations (ReLU, leaky ReLU, ADA, leaky ADA) and artificial neurons (standard, PyNReLU, PyNADA and leaky PyNADA). We add Gaussian noise of 0 mean and different standard deviations (illustrated on the horizontal axis). Best viewed in color.
Fig. 19: Object class recognition accuracy rates (in %) for a two-hidden-layer MLP on Fashion-MNIST, with various levels of Gaussian noise applied on the input images. Results are reported with various activations (ReLU, leaky ReLU, ADA, leaky ADA) and artificial neurons (standard, PyNReLU, PyNADA and leaky PyNADA). We add Gaussian noise of 0 mean and different standard deviations (illustrated on the horizontal axis). Best viewed in color.
Fig. 20: Object class recognition accuracy rates (in %) for a two-hidden-layer MLP on Fashion-MNIST, with various levels of labeling noise. Results are reported with various activations (ReLU, leaky ReLU, ADA, leaky ADA) and artificial neurons (standard, PyNReLU, PyNADA and leaky PyNADA). We simulate the labeling noise by randomly changing the labels for several ratios (illustrated on the horizontal axis) of the training set. Best viewed in color.
of the input noise level and the underlying architecture. The same applies to the comparison between PyNReLU and PyNADA. When it comes to classifying noisy input images, we conclude that ADA and PyNADA maintain their competitive edge over ReLU and PyNReLU.
Effect of labeling noise. To analyze the behavior of ADA and PyNADA when training on noisy labels, we carry out experiments on Fashion-MNIST with two models, MLP-2 and VGG-9. Fashion-MNIST is a well annotated data set. Therefore, we randomly change the labels of a certain fraction (0.1, 0.2 or 0.3) of the training data set to simulate noisy labeling. The corresponding results are presented in Figure 20 and Figure 21. We perform experiments with ReLU, leaky ReLU, ADA and leaky ADA as activation functions. We also carry out experiments with PyNReLU, PyNADA, and leaky PyNReLU. We observe that ADA and leaky ADA consistently outperform ReLU and leaky ReLU, regardless of the noise level, for both neural networks. The same observation applies when comparing PyNReLU with PyNADA and leaky PyNADA. We conclude that both ADA and PyNADA keep their competitive edge over ReLU and PyNReLU, in the presence of noisy labels.
Effect of data imbalance. We investigate the effect of data imbalance when using ADA and PyNADA by performing experiments on Fashion-MNIST with the same models as above, namely MLP-2 and VGG-9. Fashion-MNIST is a balanced data set, meaning that the samples are evenly distributed among classes. To simulate an imbalanced version of Fashion-MNIST, we randomly
36 Nonlinear Neurons with Human-like Apical Dendrite Activations
Fig. 21: Object class recognition accuracy rates (in %) for VGG-9 on FashionMNIST, with various levels of labeling noise. Results are reported with various activations (ReLU, leaky ReLU, ADA, leaky ADA) and artificial neurons (standard, PyNReLU, PyNADA and leaky PyNADA). We simulate the labeling noise by randomly changing the labels for several ratios (illustrated on the horizontal axis) of the training set. Best viewed in color.
choose 5 classes (out of 10) and keep only 10% of the data samples for the selected classes. We report the weighted accuracy for this experiment, to account for the imbalanced nature of the data set. We perform experiments with ReLU, leaky ReLU, ADA and leaky ADA as activation functions, and PyNReLU, PyNADA and leaky PyNADA as neurons, respectively. The corresponding results are presented in Figure 22. The evaluation shows that ADA and leaky ADA surpass ReLU and leaky ReLU, regardless of the underlying architecture (MLP-2 or VGG-9). Similarly, PyNADA and leaky PyNADA outperform PyNReLU. Overall, the results confirm that the proposed ADA and PyNADA keep their competitive edge over ReLU and PyNReLU, when it comes to solving classification tasks suffering from data imbalance.
4.10 Results on Nonlinear Function Approximation
In Figure 23 and Figure 24, we compare the nonlinear approximation capabilities of our ADA and leaky ADA functions against ReLU, leaky ReLU, RBF and Swish. More precisely, we vary the activation functions of a simple one-hidden-layer MLP with 20 hidden neurons and one output neuron, and employ the resulting models to approximate two nonlinear functions, sin(x) and sinc(, where x is taken between [
]. When it comes to approximating these functions, ADA and leaky ADA outperform RBF, ReLU and leaky ReLU. Swish is able to output the best approximation for sin(x), while ADA and leaky ADA are better approximators of the sinc(x)
Fig. 22: Weighted accuracy rates (in %) for two neural models (MLP-2 and VGG-9) on a data imbalanced version of Fashion-MNIST. Results are reported with various activations (ReLU, leaky ReLU, ADA, leaky ADA) and artificial neurons (standard, PyNReLU, PyNADA and leaky PyNADA). We simulate the imbalanced version of the data set by randomly dropping 90% of the samples for 5 out of 10 classes.
function. We underline that the chosen neural model is restricted to a basic shallow architecture with only one hidden layer of 20 neurons, thus emphasizing the approximation capabilities of the activation functions rather than the neural architecture itself. Certainly, all compared activation functions can provide much better approximations with deeper and wider architectures, but an empirical study in this direction goes beyond the scope of this work. All in all, our results highlight the intrinsic nonlinear property of the proposed ADA and leaky ADA functions.
4.11 Discussion
We demonstrated the applicability of our activation function (ADA) by integrating it into different CNN (LeNet, VGG, and ResNet) and MLP models and addressing a broad range of tasks: object recognition, age estimation and gender prediction in images; dialect identification in text; emotion recognition in speech. More precisely, we performed experiments on six benchmark data sets from computer vision, signal processing and natural language processing. Our experiments show that we can successfully replace the most used activation function, namely ReLU, with ADA and obtain better performance without requiring any other change to the neural network. In general, ADA can be applied to any neural network without requiring further modifications. In a similar way, we show the benefits of using pyramidal neurons, i.e. PyNADA.
38 Nonlinear Neurons with Human-like Apical Dendrite Activations
Fig. 23: Approximating the sin(x) function with a one-hidden-layer MLP based on various activation functions. The red graph represents the groundtruth, while the blue graph represents the approximation. Best viewed in color.
Fig. 24: Approximating the sinc(x) function with a one-hidden-layer MLP based on various activation functions. The red graph represents the groundtruth, while the blue graph represents the approximation. Best viewed in color.
Although we selected a broad range of tasks and domains to carry out our experiments, we strongly believe that ADA and PyNADA are applicable to many other tasks and domains. This advantage stems from the fact that ADA and PyNADA are basic neural building blocks, which can be immediately integrated into most neural network architectures.
By looking at Figure 1b, we can observe that ADA may lead to vanishing gradients, a potential drawback of the proposed function. However, we never encountered the vanishing gradient problem in practice. We performed many experiments (on six data sets, with multiple architectures) and none of the experiments showed slow convergence. Different from the well-known sigmoid and tanh activation functions, we can control the width of the peak via the parameter , increasing the non-asymptotic region of ADA as necessary to avoid the vanishing gradient problem. Hence, in general, it is sufficient to tune or learn the hyperparameter
of ADA to avoid vanishing gradients.
Another drawback of the proposed activation function is the slower computational speed with respect to ReLU, due to the use of the exponential function. Although our time measurements presented in Table 1 showed marginal slow downs, we recommend keeping an eye on the computational time when integrating ADA and PyNADA into very large neural models deployed in real-time applications. We underline that this drawback is not particular to our activation function, e.g. sigmoid, tanh, Swish [11] and ELU [25] also imply using the exponential function.
In this paper, we proposed a biologically-inspired activation function and a new model of artificial neuron. The novel apical dendrite activation function (i) enables individual artificial neurons to solve nonlinearly separable problems such as the XOR logical function, and (ii) brings significant performance improvements for a broad range of neural architectures and tasks. Indeed, we observed consistent performance improvements over the most popular activation function, ReLU, across six benchmark data sets. The proposed ADA also outperformed other activations, RBF and Swish, that enable individual artificial neurons to solve XOR. Even though RBF and Swish share the capability of solving XOR with ADA, the accuracy rates of RBF and Swish across the six evaluation benchmarks are inconsistent, in some cases even failing to converge. The proposed pyramidal neural design represents another way to further boost the performance. Notably, we observed the largest performance improvements when we used ADA instead of ReLU, RBF or Swish for the apical tuft of the pyramidal neurons. In conclusion, we believe that the biologically-inspired ADA and PyNADA are useful additions to the set of deep learning building blocks.
Future work. Our research also opens a few directions of future research. Since the activation dampens along the positive side of the domain, we believe
40 Nonlinear Neurons with Human-like Apical Dendrite Activations
it is worth investigating if ADA is more robust to out-of-distribution or adversarial examples. As the gradient saturates on the positive side, other directions of study are to inject noise into ADA to avoid saturation [41] or to employ alternative optimization methods (that do not rely on gradients) in conjunction with ADA.
The authors thank reviewers for their valuable feedback, which led to significant improvements of the manuscript.
7.1 Compliance with Ethical Standards
This research does not involve human participants and/or animals.
7.2 Funding
This work was supported by a grant of the Romanian Ministry of Education and Research, CNCS - UEFISCDI, project number PN-III-P1-1.1-TE-2019-0235, within PNCDI III.
7.3 Conflicts of Interest
The authors have no competing interests to declare that are relevant to the content of this article.
7.4 Data Availability
The data sets used throughout the experiments are publicly available online.
[1] Khan, A., Sohail, A., Zahoora, U., Qureshi, A.S.: A survey of the recent architectures of deep convolutional neural networks. Artificial Intelligence Review 53(8), 5455–5516 (2020)
[2] Gidon, A., Zolnik, T.A., Fidzinski, P., Bolduan, F., Papoutsi, A., Poirazi, P., Holtkamp, M., Vida, I., Larkum, M.E.: Dendritic action potentials and computation in human layer 2/3 cortical neurons. Science 367(6473), 83–87 (2020)
[3] Knutson, B., Cooper, J.C.: The Lure of the Unknown. Neuron 51(3), 280–282 (2006)
[4] LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)
[5] Xu, D., Yan, Y., Ricci, E., Sebe, N.: Detecting anomalous events in videos by learning deep representations of appearance and motion. Computer Vision and Image Understanding 156, 117–127 (2017)
[6] Wang, X., Gao, L., Song, J., Zhen, X., Sebe, N., Shen, H.T.: Deep appearance and motion learning for egocentric activity recognition. Neurocomputing 275, 438–447 (2018)
[7] Rosenblatt, F.: The Perceptron: a probabilistic model for information storage and organization in the brain. Psychological review 65(6), 386 (1958)
[8] Minsky, M., Papert, S.A.: Perceptrons: An Introduction to Computational Geometry. MIT Press, Cambridge, Massachusetts (2017)
[9] Nair, V., Hinton, G.E.: Rectified Linear Units Improve Restricted Boltz- mann Machines. In: Proceedings of ICML, pp. 807–814 (2010)
[10] Maas, A.L., Hannun, A.Y., Ng, A.Y.: Rectifier nonlinearities improve neural network acoustic models. In: Proceedings of WDLASL (2013)
[11] Ramachandran, P., Zoph, B., Le, Q.V.: Searching for Activation Func- tions. In: Proceedings of ICLR Workshops (2018)
[12] Apicella, A., Donnarumma, F., Isgr`o, F., Prevete, R.: A survey on modern trainable activation functions. Neural Networks 138, 14–32 (2021)
[13] Dubey, S.R., Singh, S.K., Chaudhuri, B.B.: Activation functions in deep learning: A comprehensive survey and benchmark. Neurocomputing 503, 92–108 (2022)
[14] LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proceedings of the IEEE 86(11), 2278– 2324 (1998)
[15] Simonyan, K., Zisserman, A.: Very Deep Convolutional Networks for Large-Scale Image Recognition. In: Proceedings of ICLR (2015)
[16] He, K., Zhang, X., Ren, S., Sun, J.: Deep Residual Learning for Image Recognition. In: Proceedings of CVPR, pp. 770–778 (2016)
[17] Zhang, X., Zhao, J., LeCun, Y.: Character-level convolutional networks for text classification. In: Proceedings of NIPS, pp. 649–657 (2015)
[18] Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms. arXiv preprint arXiv:1708.07747 (2017)
42 Nonlinear Neurons with Human-like Apical Dendrite Activations
[19] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision 115(3), 211–252 (2015)
[20] Zhang, Z., Song, Y., Qi, H.: Age Progression/Regression by Condi- tional Adversarial Autoencoder. In: Proceedings of CVPR, pp. 5810–5818 (2017)
[21] Cao, H., Cooper, D.G., Keutmann, M.K., Gur, R.C., Nenkova, A., Verma, R.: CREMA-D: Crowd-sourced emotional multimodal actors dataset. IEEE Transactions on Affective Computing 5(4), 377–390 (2014)
[22] Butnaru, A.M., Ionescu, R.T.: MOROCO: The Moldavian and Romanian Dialectal Corpus. In: Proceedings of ACL, pp. 688–698 (2019)
[23] Hayou, S., Doucet, A., Rousseau, J.: On the Impact of the Activation function on Deep Neural Networks Training. In: Proceedings of ICML, pp. 2672–2680 (2019)
[24] He, K., Zhang, X., Ren, S., Sun, J.: Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. In: Proceedings of ICCV, pp. 1026–1034 (2015)
[25] Clevert, D.-A., Unterthiner, T., Hochreiter, S.: Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). In: Proceedings of ICLR (2016)
[26] Goodfellow, I.J., Warde-Farley, D., Mirza, M., Courville, A., Bengio, Y.: Maxout Networks. In: Proceedings of ICML, pp. 1319–1327 (2013)
[27] Widrow, B.: An Adaptive ‘Adaline’ Neuron Using Chemical ‘Memistors’. Technical Report 1553-2, Stanford Electronics Laboratories (1960)
[28] Anwani, N., Rajendran, B.: Training multi-layer spiking neural networks using NormAD based spatio-temporal error backpropagation. Neurocomputing 380, 67–77 (2020)
[29] Sarkar, T., Lieberth, K., Pavlou, A., Frank, T., Mailaender, V., McCulloch, I., Blom, P.W., Torriccelli, F., Gkoupidenis, P.: An organic artificial spiking neuron for in situ neuromorphic sensing and biointerfacing. Nature Electronics, 1–10 (2022)
[30] Tavanaei, A., Ghodrati, M., Kheradpisheh, S.R., Masquelier, T., Maida, A.: Deep learning in spiking neural networks. Neural Networks 111, 47–63 (2019)
[31] Guerguiev, J., Lillicrap, T.P., Richards, B.A.: Towards deep learning with segregated dendrites. eLife 6, 22901 (2017)
[32] Hawkins, J., Ahmad, S.: Why Neurons Have Thousands of Synapses, a Theory of Sequence Memory in Neocortex. Frontiers in Neural Circuits 10, 23 (2016)
[33] Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., Kudlur, M., Levenberg, J., Monga, R., Moore, S., Murray, D.G., Steiner, B., Tucker, P., Vasudevan, V., Warden, P., Wicke, M., Yu, Y., Zheng, X.: TensorFlow: A system for large-scale machine learning. In: Proceedings of OSDI, pp. 265–283 (2016)
[34] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: PyTorch: An Imperative Style, HighPerformance Deep Learning Library. In: Proceedings of NeurIPS, pp. 8024–8035 (2019)
[35] Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by back-propagating errors. Nature 323(6088), 533–536 (1986)
[36] Hu, J., Shen, L., Sun, G.: Squeeze-and-Excitation Networks. In: Proceed- ings of CVPR, pp. 7132–7141 (2018)
[37] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: Proceedings of ICLR (2015)
[38] Dietterich, T.G.: Approximate Statistical Tests for Comparing Super- vised Classification Learning Algorithms. Neural Computation 10(7), 1895–1923 (1998)
[39] Shukla, A., Vougioukas, K., Ma, P., Petridis, S., Pantic, M.: Visually guided self-supervised learning of speech representations. In: Proceedings of ICASSP, pp. 6299–6303 (2020)
[40] He, G., Liu, X., Fan, F., You, J.: Image2Audio: Facilitating Semi- supervised Audio Emotion Recognition with Facial Expression Image. In: Proceedings of CVPR Workshops, pp. 912–913 (2020)
[41] Gulcehre, C., Moczulski, M., Denil, M., Bengio, Y.: Noisy Activation Functions. In: Proceedings of ICML, pp. 3059–3068 (2016)