Dropout [1] was proposed as a means of regularization. The idea is to randomly drop units in a neural network during training time and then use the non-dropped trained neural network at test time. Monte Carlo dropout [2] uses dropout not only during training time but also at test time. For an input sample at test time, different instances of dropout produce different (deterministic) neural networks, which make different predictions, which in turn can be used to form measures on prediction uncertainty. However, to have a good measure on the prediction uncertainty, many instances of dropout are needed, and an input sample needs to pass through the neural network many times, leading to an increase in the computational complexity. To address this issue, at test time we enable one or more dropout layers only near the output of a neural network and reuse the outputs from earlier layers. This way, the computational complexity can be significantly reduced without affecting regularization.

As another contribution, we improve upon individual

predictions by leveraging the side information that we have on the predictions. For a sample that belongs to one of the classes in the training dataset, we define the ideal output from the softmax layer as a discrete probability distribution whose peak is equal to one and corresponds to the correct class and whose other entries are all zero. We utilize this side information to correct the prediction: enlarge the peak to one and reduce other entries to zero.

We propose an algorithm that uses the two techniques described above and apply it to the radio frequency (RF) transmitter classification problem [3][4], which is motivated for tasks such as network security and RF interference mitigation. The present work differs from prior RF fingerprint work in that a prediction uncertainty measure is produced and used to identify potential RF transmitters on which the classifier is trained. We show that the proposed algorithm can effectively identify transmitters that were not part of the training dataset, while correctly classifying transmitters on which it has been trained.

The remainder of this paper is organized as follows. We present our main contributions in sections 2 and 3. In section 4 we show the experimental results followed by the conclusion in section 5.

The direct approach to applying Monte Carlo dropout at test time in general is to pass the input sample through the entire neural network multiple times in order to generate multiple predictions and thereby incurring complexity proportional to the number of instances. To tackle this issue, at test time we enable one or more dropout layers near the output of the network while disabling, if any, other dropout layers, and reuse the computation up to the first enabled dropout layer. Specifically, we save the input to the first enabled dropout layer in the first pass, and in the remaining passes, we start from the saved input instead of recomputing it. This way, we can sig-nificantly reduce the computational complexity and memory usage (e.g., the dropout may be applied to a fully connected linear layer with smaller dimensions than the input sample, resulting in reduced storage for the second pass and so on), which may translate into savings in power consumption and time. The reduction depends on the network architecture. Generally, deeper networks, which tend to have dropout layers near the output, benefit more from this technique because there is greater reuse of the first inference. Since during training we use the same neural network (including dropout layers) as Monte Carlo dropout, our approach does not affect regularization. We look at the savings on time for a particular neural network that we will use for the RF signal classification problem in later sections. The network consists of 16 convolutional layers (each followed by batch normalization[5]) with ResNet structure[6], an average pooling layer, and a fully connected layer where dropout is applied and which feeds the softmax function. We measure the execution time for two parts of the computational graph: (i) the post-dropout portion (including dropout), and (ii) the pre-dropout portion. We run the experiment on a Nvidia GTX 1080 GPU for 1600 passes with batch size of 256 and plot the result in Fig. 1. On average, the time of (ii) is about 25 times as large as the time of (i), meaning a 24x speed up for the second pass and onward.

Fig. 1. The execution time for the post-dropout portion vs. the execution time for the pre-dropout portion for a 18-layer neural network on a GPU.

First, for brevity, we define a known transmitter to be a transmitter on which the classifier is trained, and a signal from which is called a known signal. We define an unknown transmitter to be one whose signals (called unknown signals) are not included in the training dataset.

We then present the simple ensemble average algorithm suggested in [2], and the error correction scheme which is incorporated into the simple ensemble average algorithm to form the proposed algorithm, followed by a theoretical analysis.

3.1. Simple Ensemble Average Algorithm

We view the softmax output as an imprecise estimate of the previously defined ideal distribution. To have a better estimate, we can average many softmax outputs. However, since the noise is not zero mean, the averaging has the limitation of only producing a biased estimate. We call this algorithm the simple ensemble average algorithm, with details in Algorithm 1. At Line 3, the peak of the ensemble averaged distribution t is compared with a threshold to decide whether the input is an unknown/random signal or a known signal. If , the signal is considered as an unknown signal or random signal and the category corresponding to the peak is the predicted category. Otherwise, the signal is considered as known.

3.2. Error Correction

The output of softmax is a vector, which we think of as a distribution giving the probability that the input sample belongs to each of the classes. For an input sample that belongs to one of the classes, if the prediction is correct, the probability mass of the correct class will be the peak of the probability vector, but may be less than 1. The ideal distribution is a vector with probability 1 for the correct class and 0 for the other classes. This distribution is ideal in the sense that if the input class were known, then the probability of all other classes would be zero. We leverage this side information to ‘correct’ the softmax output. Specifically, let the softmax output be , and and the corrected be , where k = 1, . . . , N and N is the number of instances of the neural network with dropout. We consider two thresholds and such that . If the peak of , i.e., is large enough (), we enlarge the peak to 1 and shrink the other probabilities to 0, as shown below

where is the probability mass for class i in vector , , and the 1 in the first row is at location . An example is illustrated in Fig. 2 (a), where category c is the correct class. For an input sample that does not belong to one of the classes, the ideal output will be a uniform distribution. Therefore, if is small enough (), which may indicate high uncertainty, we ‘correct’ the softmax output to a uniform distribution. If is between the two thresholds, we do not change . Clearly this procedure does not improve the correctness of predictions of individual neural networks. We investigate if the procedure applied to an ensemble of neural networks as detailed in Algorithm 2 could be used to improve the uncertainty measure of a prediction compared to the ensemble average algorithm.

If we set and , then it follows from (1) that always holds, the same as Algorithm 1. In other words, Algorithm 2 includes Algorithm 1 as a special case. If we properly choose the values for and , the performance of Algorithm 2 will be at least as good as that of Algorithm 1.

Fig. 2. Error correction on the softmax output, where c is the correct category and the peak is assumed to be at class c.

Note that if the correct class is c and the prediction is correct, i.e., , the correction procedure for the case may bring down the probability of class c, as shown in Fig. 2 (c). The remaining case does not affect the probability of class c, as shown in Fig. 2 (b). Also, the prediction from a neural network may be wrong. Thus, we need to consider all cases to evaluate the benefit of the propose procedure.

We now give the conditions under which the error correction procedure provides an overall gain in predicting an input sample that belongs to one of the categories for which an classifier is designed. Let the set of deterministic DNNs that make a right prediction be , and the complementary set be W. First consider a network . With Algorithm 2, the correct category c gains all the masses from the other categories equal to if , as shown in Fig. 2(a). Category c loses probability mass of if as shown in Fig. 2(c). Next consider a network . The correct category c loses all its mass if and gains mass if . Overall, the mass (normalized by K) for category c is

, whereis the indicator function, equal to 1 when the argument is true and 0 otherwise. As a comparison, the ensemble average algorithm (Algorithm 1) yields a mass for category c of

To simplify the result, we define conditional expectations for for the expectation conditioned on and

for and , and similar notations for . We also define conditional distribution functions: . Then, as

where is the probability that an instance of the neural network makes a correct prediction. If the above is greater than 0, the probability mass for the correct class will increase. To further simplify the analysis, suppose that the other classes have equal probability mass with Algorithm 2. Since the total probability mass is equal to 1, any of the incorrect class will have a reduced probability mass if . This is summarized below.

Theorem 3.1 When , if

, Algorithm 2 provides a more ac- curate probability distribution for the correct category than Algorithm 1.

The condition in Theorem 3.1 is easily met if is close to 1 and is relatively small.

3.3. Performance for unknown or random signals

For these two cases, the ideal distribution is , where , which represents maximum uncertainty in prediction. Write the softmax output for and instance k, where random variables satisfy and . The last constraint ensures to be non-negative. Assume that ’s are identically distributed. Then the ensemble averaging algorithm – Algorithm 1 – produces a uniform distribution as . From (1), there is no preference to any category under Algorithm 2. Thus, by symmetry, as , Algorithm 2 also produces a uniform distribution. Therefore, both algorithms have the same performance for unknown and random signals.

The LoRa radio [7] is a commercial long-range (e.g., 10km) and low-power radio characterized by chirp spread spectrum modulation and intended for the Internet of Things (IoT) applications. The signals are captured using USRP X310 with an oversampling factor of 4. The channel bandwidth is 1MHz at a carrier frequency of 902.3MHz. 1000 signals are captured for each of the 48 LoRa transmitters, 44 of which are the known categories and the remaining 4 the unknown. Each signal consists of 500 complex samples at the beginning of the packet including a ramp up portion and 500 complex samples at the end of the packet including a ramp down portion with 1000 IQ samples in total. Random shift in time is used for data augmentation.

Figure 3 shows the distributions resulting from Algorithm 1 (the left column) and Algorithm 2 (right column) for Gaussian random signals (top), unknown signals (middle), and known signals (bottom), where 0.92. The same input data and noise realization are used for each row for a fair comparison. The noise is generated with SNR= 20dB. We see that the resulting distributions from Algorithm 2 are closer to the respective ideal distributions. Further comparison is available in Fig. 4. Algorithm 2 provides the same accuracy or better accuracy than Algorithm 1 for all three types of signals, except for , where Algorithm 2 is slightly worse for the unknown signals.

We proposed a speed-up technique and an error correction technique for the Monte Carlo dropout algorithm and apply them to the RF transmitter classification problem with unknown transmitters. We gave a theoretical analysis. The effectiveness was shown through experimental results.

Acknowledgements

The authors would like to thank Dr. Philip Pietraski of InterDigital for insightful comments, and Sudhir Pattar and Ishan Sethi (as an intern at InterDigital) for collecting the radio signal data for the LoRa transmitters.

Fig. 3. Distributions resulting from Algorithm 1 (left column) and Algorithm 2 (right column) for (a) a random signal (Gaussian), (b) an unknown signal, and (c) a known signal.

Fig. 4. The prediction accuracy by Algorithm 1 (solid lines) and by the proposed Algorithm 2(dashed lines).

[1] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” J. Mach. Learn. Res., 2014.

[2] Y. Gal and Z. Ghahramani, “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in ICML, 2016.

[3] T. J. OShea, T. Roy, and T. C. Clancy, “Over-the-air deep learning based radio signal classification,” IEEE J. of Selected Topics in Signal Processing, 2018.

[4] K. Merchant, S. Revay, G. Stantchev, and B. Nousain, “Deep learning for rf device fingerprinting in cognitive communication networks,” IEEE J. of Selected Topics in Signal Processing, 2018.

[5] S. Ioffe and C. Szegedy, “Batch normalization: Acceler- ating deep network training by reducing internal covariate shift,” in ICML, 2015.

[6] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.

[7] “Semtech corporation,” Online, https://www.semtech.com/lora/lora-products, retrieved on Feb 19, 2019.