Multi-Resolution Multi-Modal Sensor Fusion For Remote Sensing Data With Label Uncertainty

2018·Arxiv

Abstract

Abstract

In remote sensing, each sensor can provide complementary or reinforcing information. It is valuable to fuse outputs from multiple sensors to boost overall performance. Previous supervised fusion methods often require accurate labels for each pixel in the training data. However, in many remote sensing applications, pixel-level labels are difficult or infeasible to obtain. In addition, outputs from multiple sensors often have different resolution or modalities. For example, rasterized hyperspectral imagery presents data in a pixel grid while airborne Light Detection and Ranging (LiDAR) generates dense threedimensional (3D) point clouds. It is often difficult to directly fuse such multi-modal, multi-resolution data. To address these challenges, we present a novel Multiple Instance Multi-Resolution Fusion (MIMRF) framework that can fuse multi-resolution and multi-modal sensor outputs while learning from automaticallygenerated, imprecisely-labeled data. Experiments were conducted on the MUUFL Gulfport hyperspectral and LiDAR data set and a remotely-sensed soybean and weed data set. Results show improved, consistent performance on scene understanding and agricultural applications when compared to traditional fusion methods.

Index Terms—sensor fusion, multi-resolution, multi-modal fusion, multiple instance learning, scene understanding, label uncertainty, hyperspectral, LiDAR, Choquet integral

I. INTRODUCTION AND MOTIVATION

In multi-sensor fusion, each sensor may provide complementary and reinforcing information that can be helpful in target detection, classification, and scene understanding [1]. It is useful to integrate and combine such outputs from multiple sensors to provide more accurate information over a scene. In this study, hyperspectral (HSI) and LiDAR data collected by us over the University of Southern Mississippi-Gulfpark campus were used [2, 3]. Figure 1 shows an RGB image over the scene. Figure 2 shows a scatter plot of the LiDAR point cloud data. As shown, there are a wide variety of materials in the scene, such as buildings, road, etc. The task is to use the HSI and LiDAR data we collected to classify materials in the scene.

There are several advantages to fuse both HSI and LiDAR data rather than using individual HSI and LiDAR information for this task. If a road and a building rooftop are built with the same material (say, asphalt), hyperspectral information alone may not be sufficient to tell the rooftop and the road apart.

X. Du was with the Department of Electrical and Computer Engineering, University of Missouri, Columbia, MO 65211 and is currently with the Department of Naval Architecture and Marine Engineering, University of Michigan, Ann Arbor, MI 48109 USA. E-mail: xiaodu@umich.edu.

A. Zare is with the Department of Electrical and Computer Engineering, University of Florida, Gainesville, FL 32611 USA. Email: azare@ece.ufl.edu.

This material is based upon work supported by the National Science Foundation under Grant IIS-1723891-CAREER: Supervised Learning for Incomplete and Uncertain Data.

However, the LiDAR data provides elevation information and can easily distinguish the two. On the other hand, a highway and a biking trail can be at the same elevation and using LiDAR data alone may not be sufficient to distinguish the two types of roads, but hyperspectral sensor can be very helpful in identifying the distinctive spectral characteristics between a highway, which is likely to be primarily asphalt, and a biking trail, which is likely to be covered in dirt. It would be, thus, valuable to fuse information from both HSI and LiDAR sensors to obtain a better classification result and a more comprehensive understanding of a scene [4–7].

Fig. 1: RGB images of the MUUFL Gulfport data. (a) Campus 1; (b) Campus 2. The black region in the top left corner of (b) is an invalid region from the data collection and was excluded from any training or testing process in the experiment. The “campus 1” and “campus 2” refer to data from two flights (more details please see Section IV-A).

There are four challenges associated with this fusion task. First, the HSI and LiDAR data we have are of different modalities. The HSI imagery is on a pixel grid and the raw LiDAR data is a point cloud. Many previous HSI and LiDAR fusion methods have been proposed in the literature [8–11]. However, nearly all of these fusion methods for HSI and LiDAR rely on having a rasterized LiDAR image with accurate image registration. In these methods, the LiDAR point cloud data was mapped in the same grid as the hyperspectral imagery and the fusion methods work with both HSI and LiDAR data in image form. However, the raw dense LiDAR point cloud can offer higher geographic accuracy [12] as the raw data does not depend on grid size. Besides, image registration and rasterization may introduce additional inaccuracy [13, 14]. It would be valuable to develop a multi-modal fusion algorithm that can handle directly the HSI imagery and raw LiDAR point cloud.

Fig. 2: An example of 3D scatterplot of LiDAR data over the University of Southern Mississippi - Gulfpark campus. The LiDAR points were colored by the RGB imagery provided by HSI sensors over the scene. “X” and “Y”-axes are Easting and Northing locations and “Z”-axis corresponds to the elevation information.

Second, the HSI and LiDAR data have different resolutions. The hyperspectral imagery we collected have a ground sample distance of 1m. That is to say, each pixel in the hyperspectral imagery covers area. The LiDAR data, on the other hand, have a higher resolution with 0.60m cross track and 0.78m along track spot spacing. There is more than one LiDAR data point inside a area. This means, for each pixel in the hyperspectral imagery, there are more than one LiDAR data point to which it corresponds. It would be, thus, important to develop a fusion method that can handle multi-resolution outputs from multi-modal sensors.

Third, there are inaccuracies of LiDAR measurements in the given data set after rasterization. For example, if we look at building edges on the grey roofs, some of the building points have similar elevation as the neighborhood grass pixels on the ground, which is obviously wrong. Some points on the sidewalks and dirt roads, on the other hand, have high elevation values similar to nearby trees and buildings, which is also wrong. Such incorrect information can be caused by a variety of factors such as poor alignment and rasterization to HSI grid, sensor inaccuracy, or missed edge points from the laser pulse discontinuousness in LiDAR [15]. Directly using such inaccurate measurements for fusion can cause further inaccuracy or error in classification, detection, or prediction. Therefore, we must develop a fusion algorithm that is able to handle such inaccurate/imprecise measurements.

Finally, to train a supervised fusion algorithm, training labels are needed. However, in this data set, accurate pixel-level labels for both HSI and LiDAR data are not available. There are more than 70,000 pixels in one HSI image and more than 170,000 data points in one LiDAR point cloud. It would be expensive and difficult for humans to manually label each pixel/data point. Even if we have experts manually label each pixel or data point in the scene, there are certain transition areas that are impossible to label. For example, as mentioned above, a pixel in the HSI image corresponds to . It is highly likely that a pixel at the edge of the building is mixed with both building and ground surface materials. In this case, it is impossible to provide accurate labels for each pixel/data point in the training data. However, in this example, one can easily circle a region in the scene that contains a building. That is to say, it is possible to identify a region or a set of pixels that contains the target material in the scene. We call such a region or a set of pixels a “bag”, based on the Multiple Instance Learning (MIL) framework [16]. Previous supervised fusion methods often require accurate pixel-level labels and cannot handle “bag-level” labels [17, 18]. We aim to develop a trained fusion algorithm that can handle such uncertainty in training labels.

This paper proposes a Multiple Instance Multi-Resolution Fusion (MIMRF) framework to address the above four challenges. The proposed MIMRF can fuse multi-resolution and multi-modal sensor outputs from multiple sensors while effectively learning from bag-level training labels.

II. RELATED WORK

The proposed MIMRF algorithm handles label uncertainty in training data by formulating the supervised fusion problem under the Multiple Instance Learning (MIL) framework. The MIMRF is based on the Choquet integral, an effective nonlinear fusion tool widely used in the literature. In this section, related work on multi-sensor fusion and the MIL framework is described. The basis of the proposed MIMRF fusion method, the Choquet integral, is also described in this section.

A. Multi-Resolution and Multi-Modal Sensor Fusion

Existing optical sensors produce data at varying spatial, temporal and spectral resolutions [19]. Those sensor outputs may also have different modalities, such as imagery and point clouds. Multi-resolution and multi-modal fusion methods have been studied and developed in the literature in order to better fuse such information from different sensors. Multi-sensor fusion has wide applications in remote sensing such as the extraction of urban road networks [20], building detection [21], precision agriculture [22], and anomaly detection in archaeology [23].

Nearly all previous multi-resolution fusion methods focus on fusing image data only. Pan-sharpening methods, for example, use a panchromatic image with higer spatial resolution to fuse with multi- or hyper-spectral images in order to obtain images with higher spectral and spatial resolution. However, methods like pan-sharpening only handles imagery and mostly focus on fusing only two images (a panchromatic and a multispectral or hyperspectral image) [24–31].

More specifically, regarding HSI and LiDAR fusion, previous fusion methods can only work with pre-rasterized LiDAR images [8–11]. In these methods, the LiDAR point cloud data was mapped in the same grid as the hyperspectral imagery and the fusion methods work with both HSI and LiDAR data in image form. However, as discussed in Introduction, the raw dense LiDAR point cloud can offer higher geographic accuracy and alleviate the necessity to geometrically align the HSI and LiDAR data [32, 33].

B. Multiple Instance Learning

As discussed in Introduction, the proposed MIMRF algorithm aims to handle bag-level training labels rather than pixel-level labels. Here, this label uncertainty problem is formulated under the Multiple Instance Learning (MIL) framework.

The MIL framework was first proposed in [16] to deal with uncertainties in training labels. In the MIL framework, training labels are associated with sets of data points (“bags”) instead of each data point (“instance”). In two-class classifi-cation/target detection applications, the standard MIL assumes that a bag is labeled positive if at least one instance in the bag is positive (target) and a bag is labeled negative if all the instances in the bag are negative (non-target). The MIL has wide applications in natural scene classification [34, 35], human action recognition in videos [36], object detection and tracking [37–39], context identification, and context-dependent learning in remote sensing data [40, 41].

The mi-SVM algorithm is a widely cited MIL method for classification [42–44] and will later be used as a comparison method in the experiments. The mi-SVM algorithm was proposed by Andrews et al. [45] as an MIL extension to support vector machine (SVM) learning approaches. The miSVM algorithm can work with bag-level labels and learns a linear discriminate function to separate the positive from the negative classes.

C. Fuzzy Measure and Choquet Integral

The proposed MIMRF algorithm uses the Choquet integral (CI) [46] to perform fusion. The CI is a powerful non-linear fusion and information aggregation framework and has wide applications in the literature [47–51]. In the field of remote sensing, the CI has been applied to multi-sensor fusion in landmine detection [1, 52], classifier fusion [53], and target detection using Landsat and hyperspectral imagery [53, 54] .

Compared with commonly used aggregation operators such as weighted arithmetic means [55], the CI is able to model complex, non-linear relationship amongst the combinations of the sources for fusion. The Choquet integral depends on “fuzzy measures”, whose values determine the fusion outcome. Depending on the set of real-valued fuzzy measure g it learns, the CI can flexibly represent a wide variety of aggregation operators [1, 49, 50, 53, 56, 57].

Suppose we are fusing m sources using the discrete Choquet integral. These fusion sources can come from sensory data (data-level fusion) or confidence values from detectors or classifiers on the raw sensor data (decision-level fusion). In the case where a sensor has multidimensional outputs, such as a 3D hyperspectral data cube, each band can be regarded as a “sensor”. In this paper, the term “sensor outputs” can refer to either sensor data or classifier outputs, and is equivalent with “fusion sources.” Let denote the m sources to be fused. The power set of all (crisp) subsets of S is denoted by . A monotonic and normalized fuzzy measure, g, is a real valued function that maps [46, 53, 58, 59]. The fuzzy measure used in this paper satisfies monotonicity and normalization properties, i.e. if and ; and g(S) = 1 [60]. Therefore, the power set has non-empty crisp subsets, and each element in the fuzzy measure corresponds to one of the subset. In this paper, the fuzzy measure g can be written as a vector of length , where the last measure element that correspond to the full set , and the remaining element values are unknown and to be learned. Let denote the (known) sensor output for the data point. If data-level fusion is performed, the function h is an identity function and simply means the data point from the

sensor. If decision-level fusion is performed, the function h can represent the detector or classifier used to produce a decision-level output, and the term represents the value from the classifier output or detector map to be fused. Note that each h here is assumed to have higher confidence values for target and lower confidence values for non-target/background data points, since we later assign label “1” to the target (positive) points and label “0” to non-target (negative) points. The discrete Choquet integral on instance given sensor outputs S is then computed as [53, 60, 61]

where S is sorted so that and . Note that the order of the sorting can be different for each . The term is the fuzzy measure element that corresponds to the subset . In this paper, we normalize all values to be between 0 and 1 for each sensor output, and the used in this paper is bounded between 0 and 1, thus the Choquet integral fusion values in this paper are bounded between 0 and 1 for all .

In the supervised fusion problem, we need to learn the unknown fuzzy measure element values in g, given known training data (the terms) and training labels. The measure elements represent the non-linear interactions and interdependencies between all subsets of the fusion sources in the aggregation process. When the number of measure elements reduces to m for m sources, the CI is essentially reduced to a simple linear weighted average. The learned fuzzy measure g can then be used to determine the fusion results for test data. In the literature, the fuzzy measure values can be learned using either quadratic programming or sampling techniques. The CI-QP (quadratic programming) approach [60] learns a fuzzy measure for Choquet integral by optimizing a least squares error objective using quadratic programming [62]. However, the CI-QP method requires pixel-level training labels and cannot work for MIL problems. We previously proposed a Multiple Instance Choquet Integral (MICI) classifier fusion framework that extends the standard CI fusion under the MIL framework [53, 63, 64]. However, the previous MICI models still assumes that each sensor sources must have the same number of data points and do not support multi-resolution fusion. The goal of this work is to develop a novel trained classifier fusion algorithm that can both handle multi-resolution data and learn from uncertain and imprecise training labels.

III. THE MULTIPLE INSTANCE MULTI-RESOLUTION FUSION (MIMRF) ALGORITHM

In this section, the proposed Multiple Instance MultiResolution Fusion (MIMRF) algorithm is described. The proposed MIMRF algorithm learns a monotonic, normalized fuzzy measure from training data and bag-level training labels. Then, the learned fuzzy measure is used with the Choquet integral to perform multi-resolution fusion.

A. Objective Function

We discussed in the Introduction that in the HSI/LiDAR fusion problem, due to differences in modality and resolution, there can be more than one LiDAR data point that correspond to each pixel in the HSI imagery. There may also be inaccuracies in the LiDAR data. Our proposed MIMRF algorithm aims to handle such challenges. We motivate our objective function with a simple example, as follows.

Suppose there are three LiDAR data points that correspond to one pixel in the HSI imagery, as shaded pink in Figure 3. Let denote the value of the pixel in the HSI imagery. Let denote the values of the three LiDAR points. Let denote the set of all possible matching combinations of the sensor outputs for pixel i, i.e.,

Fig. 3: Illustration for HSI and LiDAR fusion. All LiDAR data points in the pink shade fell in the same area covered by the pink pixel in the HSI image, and all LiDAR data points in the blue shade corresponds to the blue pixel in the HSI image. Note that there can be more than one LiDAR points in the area covered by one pixel in the HSI imagery.

Our objective function contains two levels of aggregation to solve the aforementioned challenges. We know from the above example that there can be more than one LiDAR data points that correspond to each pixel in the HSI imagery. Here we assume that at least one of the LiDAR points is accurate (has the correct height information), but we do not know which one. Other LiDAR points may carry erroneous or inaccurate information (for example, on the edge of a building). The first level of aggregation was designed to help automatically select the correct LiDAR point. For the negative data point in , we write the CI fusion as

(3) and for the positive data point in , we write the CI fusion as

where is the set of sensor outputs for the negative data point and is the set of sensor outputs for the positive data point as defined in (2); is the Choquet integral output for and is the Choquet integral output for . In this way, the min and max operators automatically identify and select one most favorable element in terms of CI from the set , which is assumed to be the data point with correct information corresponding to pixel i.

The second level of aggregation was designed to handle bag-level training labels. We set the desired label for a “negative” (non-target) data point to be “0” and the desired label for a “positive” (target) data point to be “1”, for two-class classification problems. Recall that the MIL assumes a bag is labeled positive if at least one instance in the bag is positive and a bag is labeled negative if all the instances in the bag are negative. That is to say, we wish to encourage all points in negative bags to have label “0” and at least one point in positive bags to have label “1”. We can write the objective function for negative bags as

Similarly, the objective function for positive bags can be written as

where is the total number of positive bags, is the total number of negative bags, is the set that corresponds to the instance in the negative bag, is the set that corresponds to the instance in positive bag, is the Choquet integral given fuzzy measure is the negative bag, and is the positive bag. By minimizing in (5), we encourage the fusion output of all the points in the the negative bag to the desired negative label of 0. By minimizing in (6), we encourage the fusion output of at least one of the points in the positive bag to the desired positive label of 1. This satisfies the MIL assumption and successfully handles label uncertainty. Although this paper focuses on binary classification, the MIL-based formulation can be extended to regression problems where the ground truth labels are of real values [64]. The least squares notation that we used above can be generalized to , where d is the desired binary or real label value. As a side note, when d is binary (0 or 1), both (5) and (6) could be simplified to .Thus, the objective function for the proposed Multiple Instance Multi-Resolution Fusion (MIMRF) algorithm is written as

where the boxed terms are from the first level of aggregation in (3) and (4), and the max and min operators outside the parentheses are from the second level of aggregation in (5) and (6).

B. Optimization

An evolutionary algorithm (EA) is used to optimize the objective function in (7) and learn the fuzzy measure g. The evolutionary algorithm used in this paper extends upon the method used in [53]. First, a population of fuzzy measures was generated randomly between [0, 1] that satisfies monotonicity. Let P denote the measure population size (i.e., how many measures are in the population in one generation). If P is small, there are fewer possibilities to update measure values and only a small part of search space is explored. If P is large, the optimization will slow down given the larger search space and more mutation possibilities. We used P = 30 in our experiments as it is a good balance between reasonable computation time and enough population to mutate and converge.

Then, in each iteration, new measure values were updated based on small-scale or large-scale mutations. To update the measure values, the valid interval of each fuzzy measure element in the population was computed. The “valid interval” of a fuzzy measure element is defined as how much the measure element value can change while still satisfying the monotonicity property [53]. In small-scale mutation, only one measure element value was updated according to their valid interval. In large-scale mutation, all the measure elements were updated. A Truncated Gaussian distribution [65] was used for sampling new measure values between the lower and upper bound of the valid interval since the full Gaussian may get invalid values for the measure elements. In our experiments, the variance of the Truncated Gaussian is set to 0.1 and the mean is equal to the previous measure element value before updating. The lower and upper bounds of the Truncated Gaussian are equal to the lower and upper bounds of the valid intervals of the measure element to be updated. This way, the updated value was bounded within the valid intervals and close to the previous measure element value. We conducted an experiment using our demo MIMRF code1 and compared using uniform distribution versus Truncated Gaussian, and using uniform distribution took much longer to converge (490 iterations) than the Truncated Gaussian distribution (80 iterations). This makes sense as the uniform distribution essentially conducts a random search at every mutation yet the Truncated Gaussian can “tweak” the updated measure element values in small scale and thus make the search more efficient. Denote as the rate of small-scale mutation (i.e., how often to sample a single measure element) and is the rate of large-scale mutation (i.e., how often to sample the entire measure). If is too small, the entire measure will be sampled most of the time and the optimization will be close to a random search. If is too large, it is likely that the EA will fall into a local minima. We use in our experiments so that a new measure element can be updated most of the time, but occasionally the entire

measure was updated to expand the search space and avoid local minima.

Once a new generation of measures with updated values were generated, the old and new measure element values were pooled together (size 2P) and their fitness values were computed using (7). Then, P/2 measures with the top 25% fitness values are kept and carried over to the next iteration (elitism), and the remaining P/2 measures to be carried over were sampled according to a multinomial distribution based on their fitness values from the remaining 75% of the parent and child population pool, following a similar approach in [53]. In the new measure population, the measure with the highest fitness value is kept as the current best measure . The process continues until a stopping criterion is reached, such as when the maximum number of iterations or the change in the objective function value from one iteration to the next is smaller than a fixed threshold, . In our experiments, we stop if the difference between the current optimal measure and the previous optimal measure is smaller than . We used a high maximum number of iteration I = 5000 and the algorithm will automatically stop if it reaches convergence before the maximum iteration. The number of iterations it takes to converge depends on the random initialization. In our hyperspectral and LiDAR fusion experiments below, the algorithm converged in 165 iterations on average.

At the end of the model learning process, the best measure with the optimum fitness value was returned as the learned fuzzy measure. We then use this learned to compute the pixel-level fusion results using the Choquet integral described in (1). More details about the evolutionary algorithm can be seen in [53] and the pseudocode of our learning scheme is shown in Algorithm 1.

Note that in our optimization and model learning process, we only use bag-level imprecise labels. This means, our method is not a standard supervised algorithm in which complete, pixel-wise training labels are required. Instead, we learn from imprecisely labeled data and produce precise classification maps through the fusion of multi-modal, multi-resolution data. In the fuzzy measure learning (“training”) stage, the algorithm learns the optimal fuzzy measure from imprecise bag-level labels. In the fusion stage, the algorithm produces complete, pixel-level fusion maps over the scene. The estimated represents the relationship among the fusion sources. Thus, the estimated can be used to fuse any scene as long as the sensor sources in the test scene have similar relationships as in training.

IV. EXPERIMENTS

This section presents experimental results of the proposed MIMRF algorithm on two real remote sensing data sets. First, the proposed MIMRF algorithm is used for hyperspectral and LiDAR fusion for scene understanding in the MUUFL Gulfport dataset. Then, the MIMRF algorithm is used for weed detection on a multi-resolution soybean and weed dataset. At the end of this section, discussions are provided on the assumptions, limitations, and applications of the proposed MIMRF framework.

A. MUUFL Gulfport HSI and LiDAR Fusion Data Set

This section desribes the MUUFL Gulfport hyperspectral and LiDAR fusion data set and presents experimental results. The proposed MIMRF was used to perform multi-resolution fusion on the hyperspectral imagery and raw LiDAR point cloud data.

1) Data Set Description: The MUUFL Gulfport hyperspectral imagery and LiDAR data set [2, 3, 67] was collected over the University of Southern Mississippi - Gulfpark campus in November 2011. The data set contains hyperspectral and LiDAR data from two flights. The RGB images from the HSI imagery of the two flights (named “campus 1” and “campus 2” data) are shown in Figure 1a and Figure 1b. Only the first 220 columns of the hyperspectral data were kept in order to exclude the bright beach sand regions in the lower right corner in the original images. The first four and last four bands of the data were removed due to noise. The size of the HSI image is for both flights in this experiment.

The LiDAR raw point cloud data was collected by the Gemini LiDAR sensor at 3500 ft over the scene [2]. The scatter plot of the raw LiDAR point cloud data over the entire Gulfpark campus is shown in Figure 2. Rasterized LiDAR imagery (pre-processed by 3001 Inc. and Optech Inc. using nearest neighbor methods) was used to generate results for comparison methods. The rasterized LiDAR imagery for campus 1 and campus 2 are shown in Figure 4a and Figure 4b.

Fig. 4: Raster image of the first return MUUFL Gulfport LiDAR data. The color represents the lidar height information (normalized). The rasterized image is in the same size as the hyperspectral imagery. (a) Campus 1 data; (b) Campus 2 data. (c) Color mapping.

2) Automated Training Label Generation: As discussed in the Introduction, hand-labeling is tedious, expensive, and prone to error, especially for large data sets. For transition regions such as edges of a building, it is impossible for humans to visually identify and correctly label each pixel by hand due to mixing. In this section, an automated label generation method is proposed to automatically generate bag-level training labels for this data set for regions where crowd-sourced map data is available.

The “bags” for this data set were constructed using the simple linear iterative clustering (SLIC) algorithm [68, 69]. The SLIC algorithm is a widely used, unsupervised, opensource superpixel segmentation algorithm that can produce spatially coherent regions [69]. The number of superpixels was set to 100 and the compactness factor was set to 20. We chose this parameter setting so that the resulting superpixels are large enough to be spatially coherent and computationally inexpensive, yet small enough to be able to shape the edges along the buildings and also produce entirely non-building (non-target) superpixels. Our parameter choice balances well between computation time and learning performance empirically. Figure 6b shows the SLIC segmentation result on the MUUFL Gulfport hyperspectral campus 1 data. The lines mark the boundaries for each superpixel. Each superpixel is treated as a “bag” and the pixels in each superpixel are the instances in the bag.

Training labels for each bag were automatically generated based on publicly available, crowdsourced map data from the Open Street Map (OSM) [70]. Figure 5 shows the map extracted from OSM over the campus area and the available tags, such as “highway”, “footway”, “building”, “parking”, etc. The blue lines correspond to asphalt, which includes road, highway and parking lot. The magenta lines correspond to sidewalk/footway. The green lines mark buildings. The black lines correspond to “other” tags (regarded as background in our experiments). Keypoints, such as corners of the buildings, were selected manually from both OSM map and HSI RGB imagery and an affine transformation [71] was used to map between the OSM Lat/Lon coordinates and the HSI pixel coordinates. Then, all the superpixels that contain at least one building pixel are labeled positive and all the superpixels that do not contain building pixels are labeled negative. Figure 6a shows the ground truth map for buildings with a grey roof and Figure 6b shows the bag-level labels for all superpixels.

Fig. 5: Open Street Map imagery over MUUFL Gulfport campus 1. The blue lines correspond to road and highway. The magenta lines correspond to sidewalk/footway. The green lines mark buildings. Here, the “building” tag is specific to the buildings with a grey asphalt roof. The black lines correspond to “other” tags.

Buildings are easy to identify in a scene due to their spatial coherence. However, producing an accurate pixel-level classification map for buildings is a challenging task due to many transition areas along the edges of the buildings. The transition areas can be only a few pixels or less from the top of a building to the neighboring ground surface. The edge areas are also very difficult or impossible to label manually. Similar observations can be made in a tree canopy, where pixels along the edge of tree leaves are also highly mixed and can be easily mis-identified as false positives. In the following, we first present the fusion source generation process in Section IV-A3 and the overall building detection results in Section IV-A4. Then, we further investigate the performance of the proposed algorithm specifically on those difficult transitional regions, such as building edges, in Section IV-A5.

3) Generation of Fusion Sources for Building Detection: In this experiment, three sources were generated for fusion (m = 3), one from HSI and two from LiDAR. First, building points were manually extracted from the hyperspectral imagery and the mean spectral signature of these point were computed. The adaptive coherence estimator (ACE) detector [72–74] was applied to the HSI imagery using spectral signature of the building points. The ACE detection map for buildings is shown in Figure 8a, where the ACE confidence map correctly highlights most buildings, but has some false positives on

asphalt roads, which have similar spectral signature as the roof materials. The ACE detector also has low confidence on the top right building, possibly due to the darkness of the roof.

Fig. 6: The Ground Truth map and the SLIC segmentation map of the MUUFL Gulfport HSI data for building detection. (a) The Ground Truth map of the buildings in MUUFL Gulfport HSI data. The yellow highlights the ground truth building locations [3]. (b) The SLIC segmentation result on MUUFL Gulfport HSI data. Red marks positive training bags and blue marks negative bags for building detection experiment.

Fig. 7: The histogram and peaks of the LiDAR values of building points. (a) The histogram of the LiDAR values of building points. (b) The peaks found based on the histogram in (a).

Then, two LiDAR confidence maps were generated to be fused with the ACE detection map. The LiDAR height information of extracted training building points were plotted in a histogram, as shown in Figure 7a. The peaks of the histogram were found by using the MATLAB findpeaks() function, as shown in Figure 7b. The Euclidean distances of all LiDAR points in the scene were computed against two peak height values of the building points, 16.8m and 13.9m, to produce two confidence maps, which matches the two distinct heights of grey-roof buildings in the scene. Then, we designed an exponential function to compute confidence values based on the distances, written as

where is the confidence value of the LiDAR point in the scene and is the Euclidean distance between the LiDAR point and the peak height value. In this way, the LiDAR maps have high confidence on points with similar

Fig. 8: The confidence maps from HSI and LiDAR data for building detection in the MUUFL Gulfport. These three sources are used for fusion. (a) ACE detection map from HSI data. (b)(c) The heightbased LiDAR confidence maps for building detection in the MUUFL Gulfport data. (b)(c) were distance maps computed against the two peaks found in Fig. 7. The colorbar is the same as in Figure 4c.

height to training building points and have low confidence on points with distinctly different heights than buildings. Figure 8b and Figure 8c show two confidence maps computed from a rasterized LiDAR map against the two peak height values, where Figure 8b highlights three buildings with height value around 16.8m and Figure 8c highlights the L-shaped building with height value around 13.9m. Both LiDAR con-fidence maps also highlight some trees of similar height. On the other hand, the ACE confidence map (Figure 8a) shows high values on some buildings and roads, and low values on trees. Thus, the purpose of our proposed MIMRF is to fuse all three confidence maps as shown in Figure 8 to the effect that it highlights all buildings only and suppress other noise objects (such as trees and roads). Note that the purpose of this experiment is not to persuade that this Euclidean-distance-based method is the best or the most general method to produce confidence maps, but rather given such ACE and LiDAR confidence maps that highlight different objects in the scene, our proposed MIMRF algorithm is able to perform fusion and produce a fused map that assigns high confidence on all the target buildings and low confidence on other non-target objects, which is the goal of an effective fusion method. Other feature extraction methods can be explored in future work for LiDAR data, such as texture or shape-based features. For the proposed MIMRF algorithm, the raw LiDAR points were directly used as input instead of the pre-rasterized LiDAR imagery. The rasterized LiDAR confidence maps as shown in Figure 8b and Figure 8c were used for non-multi-resolution comparison methods.

4) MIMRF Fusion Results on Building Detection: The proposed MIMRF algorithm was compared with a variety of methods to demonstrate its effectiveness in multi-resolution, multi-modal fusion with label uncertainty. First, the MIMRF was compared with results of individual sources, before fusion (i.e., the ACE confidence map and two LiDAR confidence maps). Then, the MIMRF was compared with widely adopted fusion and decision-making methods, including the Support Vector Machine (SVM) and the min/max/mean aggregation operators. Both SVM and the aggregation operators are applicable to images with pixel-to-pixel correspondence only

TABLE I: The characteristics of comparison methods. “Fusion” means fusion method. “CI” means the method uses the Choquet integral as a fusion tool. “MIL” means the method supports the MIL framework and can handle label uncertainty. “MR” means the method supports multi-resolution and multi-modal fusion. The table cell is marked “” if the method supports the heading conditions, and left blank if it does not work with the heading conditions. The top three rows indicate individual sources before fusion.

and cannot handle multi-resolution fusion. The CI-QP [47] approach, as mentioned in Section II-C, was also used as a comparison method. The CI-QP approach uses the Choquet integral to perform fusion, but CI-QP does not support MIL-type learning with bag-level labels. The mi-SVM [45] method, also mentioned in Section II-B, was used as another comparison method. The mi-SVM algorithm works under the MIL framework, but does not support multi-resolution fusion. Additionly, the previously proposed MICI classifier fusion algorithm [53] was used for comparison. The MICI method is an MIL extension on Choquet integral and can handle label uncertainty, but MICI is only applicable to data with pixel-to-pixel correspondence and does not support multi-resolution fusion either. The CI-QP, mi-SVM, and MICI methods only work with rasterized LiDAR imagery, while our proposed MIMRF can directly handle raw LiDAR point cloud data. Table I shows a comprehensive list of comparison methods and their characteristics.

Fig. 9: The fusion results for building detection in the MUUFL Gulfport data set. Train on campus 1 and test on campus 2. Fusion results by (a) SVM; (b) Min operator; (c) Max opeator; (d) Mean operator; (e) mi-SVM; (f) CI-QP; (g) MICI; (h) The proposed MIMRF algorithm.

Two-fold cross validation was performed on this data set,

i.e., training on one flight and testing on another flight. The fusion results when training on campus 1 and testing on campus 2 across all methods are shown in Figure 9. As shown, the MIMRF fusion results in Figure 9h accurately detects all four grey-roof buildings while having significantly less false positives on other non-building objects, such as trees, in the scene than comparison methods.

Figures 10a and Figures 10b show the overall ROC curve on building detection with cross validation. In addition, the Area Under Curve (AUC) results from the ROC curves were computed to provide a quantitative comparison. The first two columns of Table II shows the AUC results for building detection. The ACE, LiDAR1, and LiDAR2 rows are results from the three individual sources before fusion, and the methods below the dotted line are fusion results from all comparison methods. As shown, the proposed MIMRF algorithm produces the best or second best ROC curve and AUC results.

Fig. 10: The overall ROC curve for building detection for MUUFL Gulfport data. (a) Train on campus 1, test on campus 2; (b) Train on campus 2, test on campus 1.

Another evaluation metric, the Root Mean Square Error (RMSE), is used for comparison as well. The AUC evaluates how well the method detects the buildings (the higher AUC the better) and the RMSE shows how detection results on both the building and non-building points differ from the ground truth (the lower the RMSE the better). Table III shows the RMSE comparison results between MICI and MIMRF methods (the top methods with high AUC results). The proposed MIMRF has higher AUC performance and lower RMSE than comparison methods.

5) MIMRF Results on Edge Areas: As discussed in Section IV-A2, building detection is a challenging task along the edge areas due to spatial resolution. The edge areas refer to transition areas and boundaries regions where there is a sudden change in altitude, including but not limited to building edges and edges in a tree canopy. These edge areas are difficult to label and are prone to noise and misalignment in image registration and rasterization. One of the biggest difference between the proposed MIMRF algorithm and previous methods is that the proposed MIMRF can directly use raw LiDAR point cloud data while previous image fusion methods require rasterization. We conducted additional experiments as follows to investigate the performance of all fusion methods on such edge areas, where the rasterization may be noisy, inaccurate, or misaligned.

The rasterized LiDAR imagery in our MUUFL data set was pre-processed and provided by third party companies (3001 Inc. and Optech Inc.) with nearest-neighbor-based rasterization techniques. We call the provided LiDAR rasterization “Optech LiDAR map”. Recall that the proposed MIMRF algorithm can automatically select the “correct” LiDAR points for multi-resolution fusion with its objective function. Thus, we can plot the LiDAR points selected by the proposed MIMRF into a map. We call this map a “selected LiDAR map” by the proposed MIMRF algorithm. Figure 11a show the difference between the selected LiDAR map and the Optech rasterized LiDAR map. As shown, the differences are mainly along the edge areas, such as building edges or the edges of tree canopy. These pixels on the boundary are likely where the rasterization is misaligned, due to the drastic changes in elevation between an object and its surrounding pixels. This difference map is referred to as an “edge map” in the following discussions.

Fig. 11: The LiDAR edge map and the scoring results on the edge map in the MUUFL Gulfport data set. (a) The differences between the rasterized LiDAR imagery and the selected lidar points by the proposed MIMRF, i.e. the LiDAR edge map. (b) One example of the ROC curve results for building detection scoring on the LiDAR edge map.

We also generated three alternative rasterized LiDAR maps by taking the min, max, and mean of LiDAR points within neighborhood pixels. We computed the difference between the min/max/mean LiDAR map and the given Optech LiDAR map. Any pixel that has a difference value above a threshold is determined as “edge” pixels. We then look specifically at the detection results on those edge pixels. The purpose of this experiment is to show that the proposed MIMRF algorithm, which uses the raw LiDAR point cloud data, can select the correct LiDAR points from the point cloud and produce better performance especially in those edge areas when compared with other fusion methods that use the (inaccurate) rasterized imagery.

Figure 11b shows an example of the ROC curve results scored only on the edge pixels. Table IV and Table V present the AUC results with FAR up to and the RMSE of the fusion methods, scored only on the edges. In the tables, the “MIMRF diff map” refers to results scoring on the edges determined by the difference between the LiDAR points that were selected by the MIMRF method and the Optech rasterized LiDAR imagery. The “max/min/mean diff map” refers to results scoring on the edges between aggregating neighborhood LiDAR points using the max/min/mean operators and the Optech rasterized LiDAR imagery. As seen from the AUC and RMSE results, the proposed MIMRF algorithm has superior performance compared with other fusion methods, specifically on the edge areas.

Fig. 12: Results for the MUUFL Gulfport sidewalk and road detection experiments. (a)(d) The Ground Truth map of sidewalks and roads. (b)(e) The SLIC segmentation results on sidewalks and roads. Red marks positive training bags and blue marks negative bags for road detection experiment. (c)(f) One example of the MIMRF Fusion results for sidewalk and road detection, trained on campus 1 and tested on campus 2.

Fig. 13: The overall ROC curve for sidewalk detection for the MUUFL Gulfport data. Train on campus 1, test on campus 2.

Fig. 14: The overall ROC curve for road detection for the MUUFL Gulfport data. Train on campus 1, test on campus 2.

6) MUUFL Gulfport Sidewalk and Road Detection: We conducted similar experiments on other materials in the scene and present sidewalk and road detection results in this section. Sidewalks and roads are on ground surface level and do not have drastic altitude change as building edges. These additional experiments on sidewalk and road show the effectiveness of the proposed MIMRF algorithm in understanding other materials and objects in the scene.

Figure 12 shows the ground truth map and SLIC segmentation results for sidewalk and road detection experiments. Figure 12c and Figure 12f show the MIMRF fusion results for sidewalk and road detection. Figure 13 and Figure 14 show the cross-validated overall ROC curve results on sidewalk and road detection. The complete AUC results can be seen in Table II, which shows that the proposed MIMRF algorithm can successfully detect sidewalks and roads in the scene as well. Note that since sidewalks and roads have similar altitude with ground surface, the LiDAR sources do not have significant impact. That is why the ACE map from the HSI data has comparable AUC performance to the fusion results in sidewalk and road. Still, the MIMRF achieved best or second best in half of the experiments.

TABLE II: The AUC results of building, sidewalk, and road detection using MUUFL Gulfport HSI and LiDAR data. The best two results with the highest AUC were bolded and underlined, respectively. The standard deviation is in parentheses.

TABLE III: The RMSE results of MICI and MIMRF on building, sidewalk, and road detection. The best results with the lower RMSE are bolded. The standard deviation is in parentheses.

TABLE IV: The AUC and RMSE results of MICI and MIMRF on building detection, scored on the edge maps. Train on campus 1 and test on campus 2. The best two results with the highest AUC and lowest RMSE were bolded and underlined, respectively. The standard deviation is in parentheses.

TABLE V: The AUC and RMSE results of MICI and MIMRF on building detection, scored on the edge maps. Train on campus 2 and test on campus 1.

B. Soybean and Weed Data Set

The proposed MIMRF algorithm was originally motivated by the HSI/LiDAR fusion problem for scene understanding. However, the proposed MIMRF can also be used as a general multi-resolution fusion framework for many applications. This section provides additional experimental results of the proposed MIMRF on an agricultural data set.

The proposed MIMRF was used to detect weed in a remotely sensed, multi-resolution soybean and weed data set. In the data set, a height map and a RGB image are provided over a patch of soybean field and the goal is to detect weed amongst the soybean plants. The height map is in size and the RGB map is in size. Figure 15 shows the RGB map over the scene.

Three fusion sources were used for the proposed algorithm for weed detection. Figure 16 shows the height map over the soybean-weed field. Some weeds in the scene are slightly higher than soybean plants, indicating that height is an important feature for weed detection. To extract height features, four Gabor filters at angles , and were applied to the height map and the sum of the filtered images was used as one of the fusion sources, as shown in Figure 17c.

We also observed that, in this data set, the weed plants have lighter-colored pixels than the soybean plants. That is to say, lightness and color are useful sources of weed detection as well. We transformed the RGB values into LAB color space and the L- and B- band imagery were used as the other two sources for fusion. The L dimension provides information

Fig. 15: The RGB image of the soybean-weed data.

Fig. 16: The height map of the soybean-weed data.

about lightness and the B dimension is the color opponent for the blue-yellow space. Figure 17a and Figure 17b show the L and B band images, where weed pixels are highlighted.

Figure 18a shows the manual ground truth map for weed in this data set. Figure 18b shows the SLIC segmentation results. Similar to Section IV-A2, each superpixel in this segmentation was regarded as a “bag” and the colors mark the bag-level training labels. Figure 18c shows the fusion results of the proposed MIMRF fusion, where the bright yellow highlights the detected weed. Figure 19 shows the overall ROC curve results for all comparison methods. As shown, the proposed MIMRF method can effectively detect weed in the scene and produce an overall better ROC curve performance for weed detection.

Fig. 17: Three fusion sources of the soybean-weed data set. (a) The L-band image of the soybean-weed data. (b) The B-band image of the soybean-weed data. (c) The Gabor filtered image of the soybean-weed data height map.

C. Discussions

There are a few observations from the experiments regarding the assumptions, limitations, and applications of our proposed MIMRF algorithm. First, our proposed MIMRF algorithm

Fig. 18: (a) The Ground Truth map of weed in the soybean-weed data. The deep blue is the background (soybean plants) and the yellow marks the target (weed). (b) The SLIC segmentation map of the soybean-weed data. Red marks positive training bags and blue marks negative bags for weed detection experiment. (c) The confidence map obtained from the MIMRF fusion for the soybean-weed data.

Fig. 19: The overall ROC curve results for weed detection in the soybean-weed data across comparison methods.

operates under the assumption that there is at least one LiDAR point that corresponds to the hyperspectral pixel, since its purpose was to fuse LiDAR and HSI in the MUUFL Gulfport HSI/LiDAR fusion experiment (if there was zero LiDAR point, there would be nothing to fuse). In our experiments, this assumption is true, since the hyperspectral imagery in the MUUFL Gulfport dataset was collected natively on a pixel grid with an 1m ground sample distance, whereas the raw LiDAR data is a point cloud with a higher resolution of 0.60m cross track and 0.78m along track spot spacing. There certainly can exist (rarely, we argue) in other datasets where there is zero LiDAR point in a hyperspectral pixel. In this case, the Choquet integral, by its original definition, would not work since there would not be sufficient sources to “fuse”. However, we can account for this edge case by either manually defining that if there is zero LiDAR point in a hyperspectral pixel, the Choquet integral “fusion” result will be equal to the confidence value of the hyperspectral pixel, or assigning a LiDAR point value to the empty element such as using the average of neighboring LiDAR point values. This way, the math for the MIL bag-level labels in our proposed MIMRF algorithm in (5), (6), and (7) can still work in the same manner.

Second, the MIL framework is useful in modeling problems where the positive bags contain relatively few pixels of target, which is quite common in remote sensing applications. For example, for pixel-size and sub-pixel-size target detection applications, the target size can be only a few pixels or less. In our building detection experiments, some superpixels may contain only a few pixels of building, especially along the edge areas. With the MIL framework, annotators can easily circle a large region and say, “there is a target in here somewhere but we do not know the exact location.” In these cases, there will usually be more negative (background) points in a positive bag than the target, and MIL-based methods, such as the proposed MIMRF algorithm in this paper, can easily work with such imprecise labels while non-MIL methods will have difficulty. Quantitative results on the relationship between bag structure, such as noise, contamination, and percentage of primary instances, and our MIL-based fusion methods were presented in Section V-A and V-B of our previous work [64].

Finally, it is worth noting that our proposed MIMRF algorithm can act as a method for both tweaking up imprecisely labeled data and being applied out-of-sample. The MIMRF can be used as a method for generating pixel-level labels for imprecisely labeled data, as shown in the soybean-weed dataset experiments. Additionally, it can be applied out-of-sample (e.g., to another patch of weedy soybeans) as long as the data to be fused are from the same set of sensors/classifiers and that the sensors/classifiers have the same relationship as in training. This is because the learned represents the relationship between the sensors/classifiers used. Here is an easy example to illustrate this point. Let us assume for a certain problem the estimated measure [0, 0, 0, 1, 0, 0, 1] for three sources. This means the fusion result is the intersection of the source 1 and source 2, since . Therefore, this can be applied to any other problem as long as the desired fusion result is the intersection of source 1 and source 2. Our MUUFL Gulfport experiments illustrated this point by testing on a different flight – as long as the fusion sources (HSI and LiDAR confidence maps) have the same relationships as in training, the method can work well.

V. CONCLUSION

In this paper, we proposed a novel Multiple Instance MultiResolution Fusion (MIMRF) framework that can perform multi-resolution, multi-modal sensor fusion on remote sensing data with label uncertainty. The proposed MIMRF is novel and unique in that it directly fuses multi-resolution, multi-modal sensor outputs. Specifically, in the HSI/LiDAR fusion problem, the proposed MIMRF method can automatically select accurate LiDAR points from the raw, non-grid LiDAR point cloud using the Choquet integral and then use such selected points to perform fusion in an end-to-end manner. In a way, such selection of LiDAR points may be viewed as a special, autonomous rasterization technique without enforcing an overall assumption of nearest neighbor. Experimental results have shown superior performance especially at edge areas where tradition rasterization methods are likely to be wrong. Additionally, the proposed MIMRF does not require accurate pixel-wise training labels. Instead, the MIMRF method can learn directly from bag-level imprecise labels and produce pixel-level fusion results. Instead of the traditional manual labeling process, we show satisfactory results by simply using publicly available, crowd-sourced data such as the Open Street Map to automatically generate imprecise bag-level labels for the proposed algorithm before fusion. Thus, we envision others being able to use the proposed self-supervised learning approach to perform pixel-level classification and produce fusion maps with minimal human intervention for a variety of multi-resolution fusion applications.

Although the proposed MIMRF was originally motivated by the hyperspectral imagery and LiDAR point cloud fusion problem in remote sensing, the method is a general framework can be applied to many multi-resolution and multi-modal fusion applications with uncertian labels, such as precision agriculture. Future work will include investigating and incorporating alternative fusion sources and features, such as texture-based LiDAR features, and applying the proposed MIMRF algorithm to other fusion applications. Additionally, future investigation into other types of fuzzy measures, such as the binary fuzzy measure [75–77], can be conducted to compress the number of independent parameters in the fuzzy measure g for more efficient learning.

REFERENCES

[1] P. Gader, A. Mendez-Vasquez, K. Chamberlin, J. Bolton, and A. Zare, “Multi-sensor and algorithm fusion with the Choquet integral: applications to landmine detection,” in IEEE Int. Geosci. Remote Sens. Symp. (IGARSS), vol. 3, Sept. 2004, pp. 1605–1608.

[2] P. Gader, A. Zare, R. Close, J. Aitken, and G. Tuell, “MUUFL Gulfport Hyperspectral and LiDAR Airborne Data Set,” University of Florida, Gainesville, FL, Tech. Rep. Rep. REP-2013-570, Oct. 2013.

[3] X. Du and A. Zare, “Technical report: scene label ground truth map for MUUFL Gulfport Data Set,” University of Florida, Gainesville, FL., Tech. Rep. Tech. Rep. 20170417, 2017.

[4] C. Pohl and J. L. V. Genderen, “Multisensor image fusion in remote sensing: Concepts, methods and applications,” Int. J. Remote Sens., vol. 19, no. 5, pp. 823–854, 1998.

[5] M. Liggins II, D. Hall, and J. Llinas, Handbook of multisensor data fusion. CRC Press, 2008.

[6] J. Zhang, “Multi-source remote sensing data fusion: status and trends,” Int. J. Image and Data Fusion, vol. 1, no. 1, pp. 5–24, 2010.

[7] J. M. Bioucas-Dias, A. Plaza, G. Camps-Valls, P. Scheunders, N. Nasrabadi, and J. Chanussot, “Hyperspectral remote sensing data analysis and future challenges,” IEEE Geosci. Remote Sens. Mag., vol. 1, no. 2, pp. 6–36, 2013.

[8] Y. Li and E. B. Olson, “Extracting general-purpose features from LiDAR data,” in IEEE Int. Conf. Robotics and Automation (ICRA), May 2010, pp. 1388–1393.

[9] M. Khodadadzadeh, J. Li, S. Prasad, and A. Plaza, “Fusion of hyperspectral and LiDAR remote sensing data using multiple

feature learning,” IEEE J. Sel. Topics. Appl. Earth Observ., vol. 8, no. 6, pp. 2971–2983, 2015.

[10] B. Rasti, P. Ghamisi, J. Plaza, and A. Plaza, “Fusion of hyper-spectral and LiDAR data using sparse and low-rank component analysis,” IEEE Trans. Geosci. Remote Sens., vol. 55, no. 11, pp. 6354–6365, 2017.

[11] R. Luo, W. Liao, H. Zhang, L. Zhang, P. Scheunders, Y. Pi, and W. Philips, “Fusion of hyperspectral and LiDAR data for classification of cloud-shadow mixed remote sensed scene,” IEEE J. Sel. Topics. Appl. Earth Observ., vol. 10, no. 8, pp. 3768–3781, 2017.

[12] A. Sampath and J. Shan, “Segmentation and reconstruction of polyhedral building roofs from aerial LiDAR point clouds,” IEEE Trans. Geosci. Remote Sens., vol. 48, no. 3, pp. 1554– 1567, Mar. 2010.

[13] S. Cao, X. Zhu, Y. Pan, and Q. Yu, “A stable land cover patches method for automatic registration of multitemporal remote sensing images,” IEEE J. Sel. Topics. Appl. Earth Observ., vol. 7, no. 8, pp. 3502–3512, Aug 2014.

[14] G. Brigot, E. Colin-Koeniguer, A. Plyer, and F. Janez, “Adap- tation and evaluation of an optical flow method applied to coregistration of forest remote sensing images,” IEEE J. Sel. Topics. Appl. Earth Observ., vol. 9, no. 7, pp. 2923–2939, 2016.

[15] Y. Li and H. Wu, “Adaptive building edge detection by combin- ing LiDAR data and aerial images,” The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. 37, no. Part B1, pp. 197–202, 2008.

[16] T. G. Dietterich, R. H. Lathrop, and T. Lozano-Pérez, “Solving the multiple instance problem with axis-parallel rectangles,” Artif. Intell., vol. 89, no. 1-2, pp. 31–71, Jan. 1997.

[17] H. Yang, Q. Du, and B. Ma, “Decision fusion on supervised and unsupervised classifiers for hyperspectral imagery,” IEEE Trans. Geosci. Remote Sens. Lett., vol. 7, no. 4, pp. 875–879, Oct. 2010.

[18] M. Dalponte, L. Bruzzone, and D. Gianelle, “Fusion of hy- perspectral and LIDAR remote sensing data for classification of complex forest areas,” IEEE Trans. Geosci. Remote Sens., vol. 46, no. 5, pp. 1416–1427, 2008.

[19] H. Shen, X. Meng, and L. Zhang, “An integrated framework for the spatio-temporal-spectral fusion of remote sensing images,” IEEE Trans. Geosci. Remote Sens., vol. 54, no. 12, pp. 7135– 7148, 2016.

[20] S. Hinz and A. Baumgartner, “Automatic extraction of urban road networks from multi-view aerial imagery,” ISPRS J. Photogrammetry and Remote Sensing, vol. 58, no. 1, pp. 83–98, 2003.

[21] K. Stankov and D.-C. He, “Detection of buildings in multispec- tral very high spatial resolution images using the percentage occupancy hit-or-miss transform,” IEEE J. Sel. Topics. Appl. Earth Observ., vol. 7, no. 10, pp. 4069–4080, 2014.

[22] R. L. King, “A challenge for high spatial, spectral, and temporal resolution data fusion,” in IEEE Int. Geosci. Remote Sens. Symp. (IGARSS), vol. 6, 2000, pp. 2602–2604.

[23] A. Y.-M. Lin, A. Novo, S. Har-Noy, N. D. Ricklin, and K. Stamatiou, “Combining geoeye-1 satellite remote sensing, uav aerial imaging, and geophysical surveys in anomaly detection applied to archaeology,” IEEE J. Sel. Topics. Appl. Earth Observ., vol. 4, no. 4, pp. 870–876, 2011.

[24] M. Ehlers, S. Klonus, P. Johan Åstrand, and P. Rosso, “Multi- sensor image fusion for pansharpening in remote sensing,” Int. J. Image and Data Fusion, vol. 1, no. 1, pp. 25–45, 2010.

[25] C. Shi, F. Liu, L. Li, L. Jiao, Y. Duan, and S. Wang, “Learning interpolation via regional map for pan-sharpening,” IEEE Trans. Geosci. Remote Sens., vol. 53, no. 6, pp. 3417–3431, 2015.

[26] P. Liu, L. Xiao, J. Zhang, and B. Naz, “Spatial-hessian-feature-guided variational model for pan-sharpening,” IEEE Trans. Geosci. Remote Sens., vol. 54, no. 4, pp. 2235–2253, 2016.

[27] Q. Du, N. H. Younan, R. King, and V. P. Shah, “On the performance evaluation of pan-sharpening techniques,” IEEE

Geosci. Remote Sens. Lett., vol. 4, no. 4, pp. 518–522, 2007.

[28] D. M. McKeown, S. D. Cochran, S. J. Ford, J. C. McGlone, J. A. Shufelt, and D. A. Yocum, “Fusion of HYDICE hyperspectral data with panchromatic imagery for cartographic feature extraction,” IEEE Trans. Geosci. Remote Sens., vol. 37, no. 3, pp. 1261–1277, 1999.

[29] G. A. Licciardi, M. M. Khan, J. Chanussot, A. Montanvert, L. Condat, and C. Jutten, “Fusion of hyperspectral and panchromatic images using multiresolution analysis and nonlinear PCA band reduction,” EURASIP J. Advances in Signal processing, vol. 2012, no. 1, pp. 1–17, 2012.

[30] R. B. Gomez, A. Jazaeri, and M. Kafatos, “Waveletbased hyperspectral and multispectral image fusion,” in Aerospace/Defense Sensing, Simulation, and Controls. International Society for Optics and Photonics, 2001, pp. 36–42.

[31] Z. Chen, H. Pu, B. Wang, and G.-M. Jiang, “Fusion of hy- perspectral and multispectral images: A novel framework based on generalization of pan-sharpening methods,” IEEE Geosci. Remote Sens. Lett., vol. 11, no. 8, pp. 1418–1422, 2014.

[32] V. De Silva, J. Roche, and A. Kondoz, “Fusion of LiDAR and camera sensor data for environment sensing in driverless vehicles,” arXiv preprint arXiv:1710.06230, 2017.

[33] W. Maddern and P. Newman, “Real-time probabilistic fusion of sparse 3D LIDAR and dense stereo,” in IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), 2016, pp. 2181–2188.

[34] O. Maron and A. L. Ratan, “Multiple-instance learning for natural scene classification,” in Proc. 15th Int. Conf. Mach. Learn., 1998, pp. 341–349.

[35] Z.-H. Zhou and M.-L. Zhang, “Multi-instance multi-label learn- ing with application to scene classification,” in Proc. Adv. Neural Inf. Process. Syst. (NIPS), 2006, pp. 1609–1616.

[36] S. Ali and M. Shah, “Human action recognition in videos using kinematic features and multiple instance learning,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 2, pp. 288–303, 2010.

[37] C. Zhang, J. C. Platt, and P. A. Viola, “Multiple instance boosting for object detection,” in Proc. Adv. Neural Inf. Process. Syst. (NIPS), 2005, pp. 1417–1424.

[38] B. Babenko, M.-H. Yang, and S. Belongie, “Visual tracking with online multiple instance learning,” in IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2009, pp. 983–990.

[39] ——, “Robust object tracking with online multiple instance learning,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 8, pp. 1619–1632, 2011.

[40] X. Du, A. Zare, and J. T. Cobb, “Possibilistic context identi- fication for SAS imagery,” in Proc. SPIE 9454, Detection and Sensing of Mines, Explosive Objects, and Obscured Targets XX, no. 94541I, May 2015.

[41] P. Torrione, C. Ratto, and L. M. Collins, “Multiple instance and context dependent learning in hyperspectral data,” in 1st Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (WHISPERS). IEEE, 2009, pp. 1–4.

[42] L. Xu, D. A. Clausi, F. Li, and A. Wong, “Weakly supervised classification of remotely sensed imagery using label constraint and edge penalty,” IEEE Trans. Geosci. Remote Sens., vol. 55, no. 3, pp. 1424–1436, March 2017.

[43] A. Zare, C. Jiao, and T. Glenn, “Discriminative multiple instance hyperspectral target characterization,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 40, no. 10, pp. 2342–2354, Oct 2018.

[44] L. Cao, F. Luo, L. Chen, Y. Sheng, H. Wang, C. Wang, and R. Ji, “Weakly supervised vehicle detection in satellite images via multi-instance discriminative learning,” Pattern Recognition, vol. 64, pp. 417 – 424, 2017.

[45] S. Andrews, “Support vector machines for mulitple-instance learning,” in Ann. Conf. Neural Inf. Proc. Systems (NIPS), 2002.

[46] G. Choquet, “Theory of capacities,” in Annales de l’institut Fourier, vol. 5, 1954, pp. 131–295.

[47] M. Grabisch, “The application of fuzzy integrals in multicriteria decision making,” European J. Operational Research, vol. 89,

no. 3, pp. 445–456, Mar. 1996.

[48] ——, “A new algorithm for identifying fuzzy measures and its application to pattern recognition,” in Int. Joint Conf. 4th IEEE Int. Conf. Fuzzy Systems and 2nd Int. Fuzzy Eng. Symp., vol. 1, Mar. 1995, pp. 145–150.

[49] C. Labreuche and M. Grabisch, “The Choquet integral for the aggregation of interval scales in multicriteria decision making,” Fuzzy Sets and Systems, vol. 137, no. 1, pp. 11 – 26, 2003.

[50] A. Mendez-Vazquez, P. D. Gader, J. M. Keller, and K. Chamber- lin, “Minimum classification error training for Choquet integrals with applications to landmine detection,” IEEE Trans. Fuzzy Systems, vol. 16, no. 1, pp. 225–238, Feb 2008.

[51] A. Mendez-Vazquez and P. Gader, “Learning fuzzy measure parameters by logistic LASSO,” in Proc. Annual Conf. North American Fuzzy Information Processing Society (NAFIPS), May 2008, pp. 1–7.

[52] P. D. Gader, J. M. Keller, and B. N. Nelson, “Recognition technology for the detection of buried land mines,” IEEE Trans. Fuzzy Systems, vol. 9, no. 1, pp. 31–43, .Feb 2001.

[53] X. Du, A. Zare, J. Keller, and D. Anderson, “Multiple instance Choquet integral for classifier fusion,” in IEEE Congr. Evolutionary Computation (CEC), Vancouver, BC, July 2016, pp. 1054–1061.

[54] Q. Wang, C. Zheng, H. Yu, and D. Deng, “Integration of het- erogeneous classifiers based on Choquet fuzzy integral,” in 7th Int. Conf. Intelligent Human-Machine Systems and Cybernetics, vol. 1, Aug. 2015, pp. 543–547.

[55] J. Fodor, J.-L. Marichal, and M. Roubens, “Characterization of the ordered weighted averaging operators,” IEEE Trans. Fuzzy Systems, vol. 3, no. 2, pp. 236–240, May 1995.

[56] O. Maron, “Learning from ambiguity,” AI Technical Report 1639, Massachusetts Institute of Technology, 1998.

[57] J.-L. Marichal, “An axiomatic approach of the discrete Choquet integral as a tool to aggregate interacting criteria,” IEEE Trans. Fuzzy Systems, vol. 8, no. 6, pp. 800–807, Dec 2000.

[58] M. Sugeno, “Theory of fuzzy integrals and its applications,” Ph.D. dissertation, Tokyo Institute of Technology, 1974.

[59] M. Fitting and E. Orlowska, Eds., Beyond Two: Theory and Applications of Multiple-Valued Logic. Springer, 2003.

[60] J. M. Keller, D. Liu, and D. B. Fogel, Fundamentals of computational intelligence: Neural networks, fuzzy systems and evolutionary computation, 1st ed., ser. IEEE Press Series on Computational Intelligence. John Wiley & Sons, Inc., 2016.

[61] A. Mendez-Vazquez, “Information fusion and sparsity promo- tion using Choquet integrals,” Ph.D. dissertation, University of Florida, 2008.

[62] J. Nocedal and S. Wright, Numerical Optimization. SpringerVerlag New York, 2006.

[63] X. Du, “Multiple instance Choquet integral for multiresolution sensor fusion,” Ph.D. dissertation, University of Missouri, 2017.

[64] X. Du and A. Zare, “Multiple Instance Choquet Integral Classi- fier Fusion and Regression for Remote Sensing Applications,” IEEE Trans. Geosci. Remote Sens., vol. 57, no. 5, pp. 2741– 2753, May 2019.

[65] N. Johnson, S. Kotz, and N. Balakrishnan, Continuous Univariate Distributions, 2nd ed. Wiley-Interscience, Oct. 1994, vol. 1.

[66] X. Du and A. Zare, “GatorSense/MIMRF: Initial Release,” 2019. [Online]. Available: https://doi.org/10.5281/zenodo. 2638382

[67] A. Zare, P. Gader, J. Aitken, R. Close, G. Tuell, T. Glenn, D. Dranishnikov, and X. Du, “GatorSense/MUUFLGulfport: Release 01 (Version v0.1) [Data set],” https://github.com/ GatorSense/MUUFLGulfport/tree/v0.1, 2018, DOI: https://doi. org/10.5281/zenodo.1186326.

[68] R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Süsstrunk, “SLIC superpixels,” École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland, Tech. Rep., 2010.

[69] ——, “SLIC superpixels compared to state-of-the-art superpixel

methods,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 11, pp. 2274–2282, 2012.

[70] OSMcontributors, “Open Street Map,” 2018. [Online]. Available: https://www.openstreetmap.org

[71] R. J. Schalkoff, Digital image processing and computer vision. Wiley New York, 1989, vol. 286.

[72] L. L. Scharf and L. T. McWhorter, “Adaptive matched subspace detectors and adaptive coherence estimators,” in Proc. 30th Asilomar Conf. Signals Syst. IEEE, Nov. 1996, pp. 1114– 1117.

[73] S. Kraut, L. L. Scharf, and R. W. Butler, “The adaptive coherence estimator: a uniformly most-powerful-invariant adaptive detection statistic,” IEEE Trans. Signal Proc., vol. 53, no. 2, pp. 427–438, 2005.

[74] N. Pulsone and M. A. Zatman, “A computationally efficient two- step implementation of the GLRT,” IEEE Trans. Signal Proc., vol. 48, no. 3, pp. 609–616, Mar 2000.

[75] D. T. Anderson, M. Islam, R. King, N. Younan, J. Fairley, S. Howington, F. Petry, P. Elmore, and A. Zare, “Binary fuzzy measures and Choquet integration for multi-source fusion,” in Int. Conf. Military Technologies (ICMT). IEEE, 2017, pp. 676– 681.

[76] M. A. Islam, D. T. Anderson, X. Du, T. C. Havens, and C. Wagner, “Efficient binary fuzzy measure representation and Choquet integral learning,” in Int. Conf. Information Processing and Management of Uncertainty in Knowledge-Based Systems (IPMU). Cham, Switzerland: Springer, 2018, pp. 115–126.

[77] X. Du, A. Zare, and D. T. Anderson, “Multiple Instance Choquet Integral with Binary Fuzzy Measures for Remote Sensing Classifier Fusion with Imprecise Labels,” in Proc. IEEE Symp. Series on Computational Intelligence (SSCI), 2019, Accepted.

designed for accessibility and to further open science