SAR image change detection technology is widely used inearth observation tasks such as environmental protection, urban research, and forest resource monitoring [1]. Compared with optical images, multi-temporal SAR images are usually contaminated by inherent speckle noise, which impacts much negative effects on change detection algorithms [2]-[5]. Due to the extreme shortage of multi-temporal SAR images with ground truth, supervised methods are limited in SAR image change detection. At present, unsupervised methods are widely studied and applied in this field. The main steps of unsupervised methods usually include: 1) Preprocessing; 2) difference image (DI) generation; 3) classification. A common
This work was supported in part by the National Science Foundation of China under Grant Number 61301224, in part by the Basic and Advanced Research Project in Chongqing under Grant Number cstc2017jcyjA1378 and cstc2016jcyjA0134. (Corresponding author: Xinzheng Zhang.)
X. Zhang, H. Su, X. Tanm, X.Zeng and X.Jian are with the College of Microelectronic and Communication Engineering, Chongqing University, Chongqing 400044, China (e-mail: zhangxinzheng@cqu.edu.cn).
X. Zhang,X. Tan, X.Zeng and X.Jian are with the Chongqing Key Laboratory of Space Information Network and Intelligent Information Fusion, Chongqing, 400044, China.
C. zhang and Peter M. Atkinson are with the Lancaster Environment Centre, Lancaster University, Lancaster, LA1 4YQ, United Kingdom; C. zhang, and Peter M. Atkinson are also with the UK Centre for Ecology & Hydrology, Library Avenue, Lancaster, LA1 4AP, United Kingdom
method of generating DI is to calculate the log-ratio of the two SAR images pixel by pixel, the obvious disadvantage of which is the poor noise robustness. In classification, unsupervised clustering algorithms are applied at early stages. For example, k-means clustering [6], multiple kernel k-means clustering [7], and fuzzy c-means clustering [8] have been extensively studied to classify pixels or produce pseudo-label training samples to subsequent classifiers from a DI. However, these existing clustering algorithms have significant disadvantages. On one hand, these distance-based clustering algorithms are extremely sensitive to speckle noise, which often leads to treat changes caused by speckles as real terrain objects changes. On the other hand, most of the existing clustering algorithms assume that the changed and unchanged classes are balanced. In many cases, the pixels of changed class are far less than the those of unchanged class, which is typical imbalanced. Traditional clustering methods will cause excessive false alarms when faced with imbalanced data.
In recent years, deep learning exhibits excellent performance in pattern recognition. Many researchers have introduced deep learning into SAR image change detection and achieved superior performance. Gao et al. use cluster-based PCANet [9] and CWNN [10] to exploit the key information, respectively. In [3], Geng et al. proposed a saliency-guided deep neural network for SAR image change detection. Li et al. proposed a SAR image change detection method based on convolutional neural network (CNN) [11]. Recently, researchers have noticed the imbalance in SAR image change detection. Wang et al. proposed an imbalanced learning method by morphologically supervised PCANet [12]. To our best knowledge, there are still few studies on the imbalanced SAR images change detection, it is a fact that deep learning methods have extremely high requirements on the quality of training samples. However, the problem is that, with strong speckle noise and imbalanced data, it is hard to obtain high reliable pseudo-label training samples for deep learning models. Apparently, high quality DI and effective clustering methods are benefit for this issue.
In this research, a novel approach based on deep learning is proposed for imbalanced SAR image change detection. Especially, a novel DI generation method to get deep difference image (DDI) and a parallel FCM clustering (PFCMC) method are developed in the proposed approach. Our approach integrates DDI, PFCMC and PCANet to implement imbalance multi-temporal SAR image with strong speckle noise change detection, which is abbreviated as DPPCANet. Our main
Fig. 1. The flowchart of the proposed change detection approach.
contributions are as follows: 1) Based on the ideas of convolution and pooling in deep learning, we developed weighted pooling and cumulative weighted pooling. This method of generating DDI can effectively suppress various speckle noise and enhance terrain changes than traditional method. 2) PFCMC method is developed for imbalanced SAR image based on combining nonlinear Sigmoid mapping, Gabor wavelets and basic FCM, it can provide more reliable pseudo-labels training samples in the case of significantly imbalanced SAR images. 3) In PCANet classification, over-sampling and under-sampling are employed to reduce the negative impact of imbalanced data. The remainder of this letter is organized as follows. Section II describes the proposed approach. Experimental results, comparison and analysis are in Section III. Finally, the conclusion is given in Section IV.
In this letter, the proposed approach mainly includes three parts: 1) generating a DDI; 2) parallel FCM clustering; 3) a PCANet with SVM are designed to classify pixels based on pseudo-label training samples. The flowchart of the proposed approach is illustrated in Fig. 1.
A. Deep Difference Image Gerneration
In deep CNN, average-pooling and maximum-pooling are important processes to extract deep features. Although average-pooling is effective in suppressing noise, it can also easily result in excessive loss of details information. As for maximum-pooling, it can effectively enhance image features, however, it is more sensitive to noise. Motivated by the above ideas, we have proposed a weighted-pooling method for generating a DDI based on the distance measure of the local window.
If the local window is size of , where k is odd , the weighted-pooling kernel
is a matrix with the same size as shown in Equation (1). Each element value is determined by the distance between the current position and the matrix center, which is calculated by Equation (2).
As a special case, the value of the element at the center of the matrix is defined as . After weighted-pooling convolution, each pixel in the weighedpooling image can be represented by Equation (3), where
is pixel in the original image
The log-ratio operator has weak anti-noise performance, there is still a certain degree of speckle noise in . Based on the sparse distribution of noise outliers and the aggregated distribution of real change in log-ratio image, we apply weighted-pooling operation with different local window size to
for T times, obtaining T pooled images. Then, all T pooled images is accumulated to generate the DDI denoted by
:
where and
denotes a pooled image of
after weighted-pooling with parameter
.The process of generating the DDI is exhibited in Fig. 2.
B. Parallel FCM Clustering
In SAR image change detection, the number of changed pixels is often much less than that of unchanged pixels, The traditional clustering methods have poor clustering effect in case of imbalanced data.We develop parallel FCM clustering based on Sigmoid mapping method to generate high quality
Fig. 2. The process of generating a deep difference image.
pseudo labels. In order to enhance the difference of pixel categories and effectively reduce the impact of sample imbalance characteristics, the PFCM method first employs two sigmoid functions to map DDI to obtain two mapped images. The Sigmoid mapping function is with two parameters and
, and a variable x is mapped by (6).
The Gabor wavelet transform is used to extract features from the two mapped images to obtain two sets of pixel-level feature vectors and
respectively, details of Gabor feature extraction can be referred in [9]. At last, FCM and simple coding method are used to get the final clustering results. The detailed descriptions of the parallel FCM algorithm are as follows: 1) Input: Deep difference image
. 2) Step1: Obtain
by normalizing and centralizing
, apply two Sigmoid mapping to
with two different parameter sets pixel-by-pixel to get
s
and
. 3) Step2: Perform Gabor feature extraction to
and
respectively. Two Gabor feature vector sets are obtained, which are denoted as
and
, where
represents a Gabor feature vector. 4) Step3: FCM is utilized to perform two class clustering on
and
to get two label sets
and
respectively, where
1, 2, ..., NM; p = 1, 2) represents a label corresponding to a Gabor feature vector. The value of
is 0 or 1. The simple averaging operation is used to encode the two label sets, obtaining the final label set Y =
, where
. 5) Step4: If
, assign the corresponding pixel to the changed class
, if
, assign the corresponding pixel to the unchanged class
, the others are assigned to the intermediate class
. 6) Output: The clustering result can be denoted by an image with labels
. The pixels belonging to
have the high probability to be changed, while the pixels belonging to
have the high probability to be unchanged. These two kinds of pixels can be chosen as samples for training PCANet model to classify those pixels belonging to
.
C. PCANet Classification Model PCANet is a kind of deep learning model with strong robustness to noise [13]. In this letter, a PCANet with the
similar two stages structure in [9] and linear SVM are adopted as the classification model.
Patches of size are generated pixel by pixel on two weighted-pooled SAR images
and
. The corresponding patches are connected to form a new patch set denoted as
, where
. Over-sampling the pixels of
and down-sampling the pixels of
to maintain a certain balance between the sample numbers. Randomly select S image-patch samples
and vectorize them to obtain feature vectors, then subtract vector mean from each a vector and combine them to a new feature matrix
. The expression for calculating the PCA filter is as:
(7) where mat(v) is a function that maps
to a matrix
, and
means the
-th principle eigenvector of
. Then the
-th filter output of the first stage in PCANet is:
where denotes 3-D convolution.
The first layer of PCANet calculation is completed to obtain . The process of the second stage is similar to the first stage, the same method is applied to each
to get the second stage output
where
.
We binarize outputs to obtain , where
is a Heaviside step function, whose value is one for positive and zero otherwise. Around each pixel of
, the vector of
binary bits can be converted into an integer value, the conversion formula is as follows:
The single integer-value image is obtained, and each pixel
is an integer in the range
. We further transform
into a histogram denoted by
. Then the PCANet feature of the image-patch
can be defined as
Every image-patch is processed by the above method to get PCANet feature, and the extracted features and clustering labels are fed into linear SVM to train model. Use the model to further classify the pixels belonging to
into
and
. Reconstruct the labels by position to get the final changed map.
A. Experimental Setup
In order to evaluate the performance of the proposed approach, we apply three real SAR datasets. The first is called the A dataset, which presents a section of two SAR images acquired by ERS-2 SAR sensor over the city of San Francisco.
Fig. 3. Visualize results.The first line is for dataset A, the second line is for dataset B, and the third line is for dataset C. (a) and (b) show two original SAR images, respectively; (c) The Ground truth; (d) The DI generated by the traditional log-ratio operator; (e) The DDI generated by our proposed method; (f) Parallel FCM clustering results; (g) Results of PCAkmeans; (h) Results of NR
Each of these images is size of . The second dataset and the third dataset are named the B and C dataset respectively, which presents two SAR images captured by the COSMO-Skymed SAR sensor with a size of
. The SAR images in B and C contain relatively strong speckle noise, and the areas of changed and unchanged in these SAR images are extremely imbalanced.
The following indicators are employed to evaluate the proposed approach: false positives (FP), false negatives (FN), percentage correct classification (PCC) and Kappa coefficient (KC). The FP denotes the number of unchanged pixels that are false detected, while FN refers the number of real changed pixels are detected as unchanged. True negatives (TN) indicates the unchanged pixels that are correctly detected, and true positives (TP) is the correct detection of changed pixels. Let Nc and Nu refer the number of truly changed and unchanged pixels of the ground truth respectively. We employ IR = Nc/Nu to measure the imbalance between the changed class and unchanged class. Overall errors (OE) OE = FN + FP is utilized to indicate the number of pixels for all detected errors. The PCC refers the ratio of correctly classified pixels to all pixels, which is expressed as PCC = (TP + TN)/(TP + TN + FP + FN). The KC is considered as an essential criterion, which is for consistency check. It is calculated as:
For comparison, four state-of-the-art approaches are utilized on the above three datasets. These approaches are PCA kmeans [14] , NR
ELM [15], GaborPCANet [9] and CWNN[10]. Moreover, the parameters of these methods are set with reference to the corresponding papers. In the proposed DP PCANet approach, we set k = 3 and , respectively. In order to reduce the number of hyperparameters, we define the relationship and constraint between the two sets of the center biases b in the sig function:
s.t.
. As mentioned in [9], the parameter of PCANet include image-patch size
, the filter numbers
and
, and the training samples S. We set
and
the
. At the stage of generating DDI and parallel clustering, we set T = 7, b = 0.1 for A; T = 11, b = 0.17 for B;
for C.
B. Analysis of Result
Comparing (d) and (e) in Fig. 3, it can be found that The proposed method of generating DDI can effectively suppress speckle noise and enhance the characteristics of the real changed area for all three datasets. This is mainly because the weighted pooling has noise reduction on the original image, in addition, the image features are effectively retained. Then, the cumulated weighted-pooling can further weaken the sparse high-value noise on the log-ratio image, and further strengthen real changes. The DDI is significantly better than traditional DI visually.
The imbalanced ratio was calculated: 0.0094 and
. As can be seen from (f) in Fig. 3, the PFCMC method can effectively adapt to images of different imbalanced ratio, and clustering results image is satisfied for all three datasets.
In Table I, we exhibit the values of the above-mentioned indicators to evaluate these change detection methods. The PCC and KC of DPPCANet are 99.08% and 0.9296, respectively, which are slightly higher than 98.90% and 0.9164 of GaborPCANet, 98.94% and 0.9231 of CWNN. It demonstrates that the change detection result of DP PCANet is almost identical to the Ground truth. For the dataset B with strong speckle noise, the PCC of DP PCANet reaches 99.52%, the KC reaches 0.7177. For C dataset, the PCC of DP PCANet reaches 98.65% and the KC is 0.6081. The indicators of our proposed algorithm are higher than other methods on three datasets. Comparing the change detection results of these five approaches in Fig. 3, we draw the conclusion that proposed approach is superior to other methods. Above results illustrates that our algorithm’s robustness and generalization are significantly better than other methods on the three sets of data, and have extremely strong detection performance.
The degree of imbalance in the three datasets is quite different, the imbalance of dataset B is very serious, especially. Various imbalance in these datasets has a strong negative effect on the clustering accuracy and the performance of the deep learning classification model. Due to imbalanced
TABLE I: CHANGE DETECTION RESULTS OF DIFFERENT METHODS ON THREE DATASETS.
characteristic, the other four methods identify large number of unchanged pixels as changed pixels, leading to great false alarms. Compared with other methods, the FP of our proposed approach are 221, 266, and 585 on the three sets of data. DP PCANet has the lowest FP and maintains the lowest OE, while keeping the highest PCC and KC. Experimental results demonstrate that DP PCANet has strong adaptive ability and generalization performance on imbalanced multi-temporal SAR image change detection tasks.
C. Analysis of Parameter b and T
We choose a set of T to explore the relationship between the cumulative times T and the performance of DP PCANet, KC is adopted as the evaluation indicator. The same way is also used to analyze the center bias b. The results are exhibited in Fig. 4.
From Fig. 4, it can be found that the parameter T and b have little effect on the dataset A, our algorithm can always maintain excellent performance for data A, the highest KC is 0.9296 when T = 7 and b = 0.1. For dataset B and C with strong speckle noise, the result becomes better as T increases. When T reaches 9 and more, the KC tends to be stable and maintain a highelevel. The above experimental results prove that more cumulative times has a stronger effect on reducing speckle noise and retaining the true change. In theory, the benchmark of center bias is 0. The clustering algorithm has no preference of any kind when b = 0. Experimental results show the PCC and KC are 98.76%, 0.5542 for data B and 98.60%, 0.5549 for data C when b = 0, it explains the benchmark parameters allow DP PCANet to show good performance in most data. there is a most suitable center biases b for SAR data with different characteristics, It is worth further research to automatically select appropriate b.
In this letter, we have proposed a robust imbalanced multi-temporal SAR change detection approach. The proposed DDI
Fig. 4. Relationship between KC and parameter
generation method can effectively reduce speckle noise and enhance features. A parallel FCM clustering method was developed to increase the gap between the change class and the unchanged class, which can obtain excellent clustering performance under imbalanced data, thereby providing highly reliable pseudo-label training samples. Over-sampling and under-sampling were employed to mitigate the imbalance effects on PCANet. Experiment results confirmed the effectiveness, generalization and robustness of the proposed approach.
[1] M. N. Sumaiya and R. Shantha Selva Kumari, “Logarithmic mean-based thresholding for SAR image change detection,” IEEE Geosci. Remote Sens. Lett., vol. 13, no. 11, pp. 1726–1728, Nov. 2016.
[2] M. Gong, Z. Zhou, and J. Ma, “Change detection in synthetic aperture radar images based on image fusion and fuzzy clustering,” IEEE Trans. Image Process., vol. 21, no. 4, pp. 2141–2151, Apr. 2012.
[3] J. Geng, X. Ma, X. Zhou and H. Wang, “Saliency-guided deep neural networks for SAR image change detection,” IEEE Trans. Geosci. Remote Sens., vol. 57, no. 10, pp. 7365–7377, Oct. 2019.
[4] M. Gong, J. Zhao, J. Liu and Q. Miao, “Change detection in synthetic aperture radar images based on deep neural networks,” IEEE Trans. Neural Netw., vol. 27, no. 1, pp. 125–138, Jan. 2016.
[5] L. Jia, M. Li, Y. Wu, P. Zhang, H. Chen, and L. An, “Semisupervised SAR image change detection using a cluster-neighborhood kernel,” IEEE Geosci. Remote Sens. Lett., vol. 11, no.8 pp. 1443–1447, Aug. 2014.
[6] Y. Zheng, X. Zhang, B. Hou, and G. Liu, “Using combined difference image and k-means clustering for SAR image change detection,” IEEE Geosci. Remote Sens. Lett., vol. 11, no. 3, pp. 691–695, Mar. 2014.
[7] L. Jia, M. Li, P. Zhang, Y. Wu, and H. Zhu, “SAR image change detection based on multiple kernel k-means clustering with local-neighborhood information,” IEEE Geosci. Remote Sens. Lett., vol. 13, no. 6, pp. 856-862, Jun. 2012.
[8] H. Li, T. Celik, and W. J. Emery, “Gabor feature based unsupervised change detection of multitemporal SAR images based on two-level clustering,” IEEE Geosci. Remote Sens. Lett., vol. 12, no. 12, pp. 2458– 2462, Dec. 2015.
[9] F. Gao, J. Dong, B. Li, and Q. Xu, “Automatic change detection in synthetic aperture radar images based on PCANet,” IEEE Geosci. Remote Sens. Lett., vol. 13, no. 12, pp. 1792–1796, Dec. 2016.
[10] F. Gao, X. Wang, Y. Gao, J. Dong , and S. Wang, “Sea ice change detection in SAR images based on convolutional-wavelet neural networks” IEEE Geosci. Remote Sens. Lett., vol. 16, no. 8, pp. 1240–1244, Aug. 2019.
[11] Y. Li, C. Peng, Y. Chen, and L. Jiao, “A deep learning method for change detection in synthetic aperture radar images,” IEEE Trans. Geosci. Remote Sens., vol. 57, no. 8, pp. 5751–5763, Aug. 2019.
[12] R. Wang, J. Zhang, J. Chen, and L. Jiao, “Imbalanced learning-based automatic SAR images change detection by morphologically supervised PCANet,” IEEE Geosci. Remote Sens. Lett., vol. 16, no. 4, pp. 554–558, Apr. 2019.
[13] T. H. Chan, K. Jia, S. Gao, J. Lu, Z. Zeng, and Y. Ma, “PCANet: A simple deep learning baseline for image classification?,” IEEE Trans. Image Process., vol. 24, no. 12, pp. 5.17–5.32, Dec. 2015.
[14] T. Celik, “Unsupervised change detection in satellite images using principal component analysis and k-means clustering,” IEEE Geosci. Remote Sens. Lett., vol. 6, no. 4, pp. 772–776, Oct. 2009.
[15] F. Gao, J. Dong, B. Li, Q. Xu, and C. Xie, “Change detection from synthetic aperture radar images based on neighborhood-based ratio and extreme learning machine,” J. Appl. Remote Sens., vol. 10, no. 4, Dec 2016, Art. no. 084684.