Modern classifiers achieve human or super-human performance in a variety of tasks (Christopher, 2016), including speech (Graves et al., 2013) and image recognition (He et al., 2016), but they are typically not able to discriminate between ”known” and ”unknown” classes and may give high confidence predictions for unrecognizable objects (Nguyen et al., 2015). Here and throughout, we call a known class a class for which we have examples during the training phase, whereas an unknown class is a class for which we have no examples during that phase. The ability to distinguish between these two cases is important if there is the possibility that new classes arise in the future or if there have not been any examples of some classes in the training set due to their rarity. Recognizing unknown objects is the central goal of novelty and outlier detection. While detection algorithms can perform well in this task, they are usually not efficient to update when new training data arise and this is a serious limitation for an algorithm that is continuously used over a long time, also referred to as life-long learning. The best way to address the problem posed by life-long learning is to build algorithms that have efficient updating mechanisms and hence support incremental learning, i.e., the ability to learn from the arrival of new training data. Algorithms of this type can however be computationally expensive, or rely only on heuristic principles, and they usually perform only closed set classification and do not recognize unknown objects.
The task of distinguishing between known and unknown classes in an incremental way is called open set classification or open world recognition (Bendale and Boult, 2015) and it is the natural combination of novelty detection and life-long learning. We underline that in this context standard hyper parameter optimization procedures such as cross-validation are usually not available, since in the training set there are only known objects. For this reason, an algorithm designed for open set recognition should involve as few hyper parameters as possible.
The principle goals in the context of open set classification are to build an algorithm that
• is capable of distinguishing between known and unknown objects;
• supports incremental learning, i.e., an algorithm that is fast to update with the arrival of
• does not rely on too many hyper parameters whose optimization is difficult.
The problem of distinguishing known and unknown objects is strongly related to the problem of estimating the support of the distribution of the known examples (Sch¨olkopf et al., 2000). One direct way to perform open set recognition is to model the probability of an object of being from a known class (Scheirer et al., 2014) given its features. We follow the same route for the two algorithms that we propose.
The extreme value machine (EVM) introduced in Rudd et al. (2018) is an algorithm that uses extreme value theory to attack these problems. In the last few years, extreme value theory has become an important tool in multivariate statistics and machine learning. This is due to the fact that the extreme features, rather than the average ones, are the most important for discriminating between different objects (Scheirer, 2017). The EVM strongly relies on the geometry expressed by the known classes and hence can fail to distinguish between known and unknown objects when this geometry conveys misleading information about the unknown classes. For this reason, we propose two alternative approaches that use extreme value theory without using the geometry expressed by the known classes. We call them the GPD classifier (GPDC) and GEV classifier (GEVC), according to the respective approximations from extreme value theory that they rely on. We underline that, in the following, we consider only the Euclidean distance for simplicity, but the obtained results are more general and can be applied to any distance.
This paper is organized as follows. In Section 2 we summarize the previous work in this context. Section 3 states some fundamental results from extreme value theory that will be useful in the definition of the GEVC and GPDC. In Section 4 we describe the EVM and explain the main limitations of this approach. In Section 5 and 6 we describe the GPDC and GEVC and we theoretically justify their main properties. Finally, in Section 7, we evaluate and compare the performance of these open set classification algorithms on both simulated and real data.
Novelty detection (Pimentel et al., 2014) is a well established field that shares similar though not identical goals with outlier detection (Walfish, 2006). While the latter is focused on discovering which examples of the training data are not in agreement with the process that has generated the bulk of them, in novelty detection we suppose to have observations from a number of known classes and for a new sample we would like to determine whether it belongs to one of those existing classes or a new one. In this sense, novelty detection is more naturally related to classification. Machine learning techniques have been applied successfully to both tasks, improving considerably their performance (Abe et al., 2006; Shon and Moon, 2007; D´esir et al., 2013). Many novelty detection methods first estimate the distribution of the known data and then mark as unknown the new points associated with a low density regions (Bishop, 1994). We follow a similar approach for the two techniques that we propose. A relevant work in this direction is (Roberts, 1999) who fit a Gaussian mixture model to the training data and use extreme value theory to decide if a new point is known or unknown. Our methods will not rely on parametric assumptions for the multivariate density, which makes them more widely applicable.
Extreme value theory has been recognized in the last years as a powerful tool to increase the performance of open set classification (Geng et al., 2018) and, more generally, machine learning techniques (Jalalzai et al., 2018) with particular interest to computer vision (Scheirer, 2017). The main reason for this success is the fact that the tails of the distribution of distances between training observations can be modeled effectively using the asymptotic theory provided by extreme value theory (Scheirer et al., 2011). One example of this can be found in Fragoso et al. (2013) where the well-known RANSAC algorithm is improved using extreme value theory.
For incremental learning where the challenge is to build an algorithm that can easily be updated with the arrival of new training data, a simple approach is to use the k-nearest neighbor classifier (James et al., 2013), a route that is to some extent also followed by the GPDC and the GEVC. Another method that has the desired requirement to be naturally adapted to new data is the nearest class mean classifier (Mensink et al., 2012), which represents each class as a prototype vector that is the mean of all the examples for that class seen so far. More complex methods are based on machine learning approaches such as support vector machines (Ruping, 2001), neural networks (Rebuffi et al., 2017) or random forests (Saffari et al., 2009) and are in general computationally more demanding.
Finally, we note that the formulation of GPDC is partially related with the estimation of the right end point of a univariate distribution. As we will argue in the following, in the GPDC this problem takes a simpler form than in the general problem of upper end point estimation in
Theorem 1 (Fisher–Tippett–Gnedenko theorem (Coles et al., 2001))
sequence of i.i.d. samples from the distribution function
then if G is a non-degenerate distribution function it belongs to the class of generalized extreme
value (GEV) distributions with
where are the shape, location and scale parameters, respectively.
A similar theorem can be formulated for threshold exceedances.
Theorem 2 (Coles et al., 2001). Assume that Theorem 1 holds. Then, as u tends to the upper
endpoint of X, the distribution function of , conditional on X > u, is approximately
where ¯The distribution H is called the generalized Pareto distribution (GPD).
If we are interested in computing the distribution of the maximum of a large number of independent copies a given random variable, the first theorem suggests to fit a GEV distribution to various such maxima taken over blocks of the same lengths. On the other hand, if we are interested to model the right tail of a distribution, then the second theorem suggests to fit a GPD to all values above a high threshold of a sequence of independent replications of a random variable with this distribution. The two approaches are asymptotically equivalent and the parameters in the two distributions are deterministically linked. The following lemma will be useful to describe our algorithms.
Lemma 1 (Coles et al., 2001). Assume that Theorem 1 holds with 0 and let the upper
endpoint of F be denoted by
where
where 0. The distribution W belongs to the reversed Weibull family and it is a special case of the GEV distribution. Moreover, we have the representation
in terms of the parameters in Theorem 2.
In this section firstly we briefly describe the extreme value machine introduced in Rudd et al. (2018) and then we underline its possible limitations.
4.1 Algorithm description
The idea of the EVM is to approximate the distribution of the margin distance of each point in each class using extreme value theory. A new point is then classified as known if it is inside the margin of some point in the training set with high probability.
Denote the training data by , each of them labeled as a class
, i = 1, . . . , n. Here and throughout,
is the dimension of the predictor space and
number of different classes in the training set. Further, suppose that we have a new unlabeled point
that we would like to mark as known or unknown. In other words, our goal is to decide if
comes from the distribution of the training data or not. We introduce the concept of margin distance of a point
as half of the minimum distance between
and all the points belonging
to a different class in the training data set. More formally, the margin distance of
The idea is to model the lower tail of the distribution of by using the
as the input data. An equivalent problem, to which we can apply extreme value theory, is to retrieve the upper tail of the distribution of ¯
. To this end, the authors propose to use the Fisher–Tippett–Gnedenko theorem (cf., Theorem 1) to fit a GEV distribution to the k largest observed
for each point
. They claim that the asymptotic distribution of ¯
assuming is bounded above by zero as a negated distance. Estimating the parameters
and
for each training observation
separately yields estimates ˆ
of the distribution functions
. Those estimates are used to label a new point
as known if it is
likely enough to fall in at least one of the margins of the n training data, that is, if
and it is labeled as unknown otherwise. Here, is a probability threshold that is chosen by a heuristic formula in Rudd et al. (2018).
4.2 Limitation of the EVM
The EVM redefines the open set classification problem as a probabilistic framework within extreme value theory, but it has some drawbacks that we describe here and that will be the starting point to construct new methods that improve upon it.
First, assuming in Equation 1 that the upper endpoint is the training set share the same support and that it is impossible that two classes are perfectly separated. Indeed, if two classes are perfectly separated, then the distance between two points sampled from them is bounded from above by a strictly positive constant. This assumption is rather restrictive and it is hardly verifiable on observed data. Second, estimating the upper tail of ¯
based on the k largest observed
for each
should rely on Theorem 2 instead of Theorem 1 since this amounts to a threshold exceedance approach with a GPD as limiting distribution.
A further limitation of the EVM is related to the choice of the threshold in (2). This threshold controls the open set classification error and a bigger
implies a higher probability to label a new object as unknown. In a statistical setting this type of thresholds are chosen by fixing the type I error, in this case, the probability to classify a known object as unknown. Unfortunately, for the EVM algorithm the authors do not propose any procedure to choose
controlling the type I error and thus it is not clear how to properly tune it.
The most important drawback of the EVM is that it gives a non-justified premium to known classes far from all the others. In fact, if there are a lot of classes relatively close to each other and one other class far from them, then all the points of the latter class will have a large margin distance. For this reason, if we observe a new point from an unknown class located closer to this far away class than the other known classes, we always classify the new point as belonging to the far away class. The main issue here is that we cannot use the geometry given by the known classes to do open set classification because it does not convey any information regarding the unknown ones. Knowing that class A is very far from class B, does not necessarily imply that a new object is known only because the distance between it and the closest point in class A is much smaller than the distances between classes A and B. It might simply be that the new class is closer to class A than is class B. The EVM is implicitly assuming that the geometry of the known classes reflects also the behavior of the unknowns, a strong assumption that is again hard to verify. We will illustrate this problem in a simulation study in Section 7.
5.1 General setting
In this section we propose an algorithm for open set classification that uses the principles of extreme value theory and that does not rely on the geometry of the training set. This algorithm, which we call the GPDC, considers the distances between the new point that we want to mark as known or unknown, and the points composing the training data set. For simplicity, and since we are primarily interested in open set classification, all the known classes are seen as one big class. If there is high confidence that the lower endpoint of the distribution of these distances is zero, then the process that has generated the training data can also have generated the new point, and we thus mark the new point as known; otherwise we mark it as unknown.
To formally describe the algorithm, we first show some basic results that will help in understanding the GPDC. As before, we assume to have training data ,
with class labels . In total we have J known classes and we assume that each class is described by a continuous density function
defined on
, where the probability that a point in class
falls in the set
The process that has generated the training data set can be described as a mixture of these
density functions, with unconditional density
for weights [0, 1] with
is large evaluated at a point that has high chance to be known. Note that, in the following, we approximate directly f and then we do not have to estimate the weights
5.2 Extreme value theory and open set classification
In this setting the training data , can be thought as being sampled from only one class, the class of the knowns, with density f. Given a new point
without label, we are interested in deciding whether it is from a known class and has been generated by f, or whether it belongs to an unseen class and has been generated by an unknown density
We denote with , the distances
between
and the points in the training set. In the following, we are interested in the lower tail of the distribution D of the distance between
and a generic point sampled from f. Intuitively, if the lower tail of D behaves similarly as the corresponding quantity for training samples, then
has a high probability to be a known point. More mathematically, let
) denote a ball of radius
centered in
Moreover, the upper end point of is equal to zero if and only if there is no
0 such that
comes from the process that has generated the training data, i.e., that it is a known point.
Note that f is, in general, a density on a multivariate space, whereas ) is a simple univariate survival function. We can therefore model the right tail of
using extreme value theory. The following results make this more explicit.
Theorem 3. Denote the distances between and the points in the training set with
and assume that
is in the support of the known classes, that is, the upper end point of
is zero. Suppose the conditions of Theorem 2 hold, then the distribution of
, above a high threshold u < 0 (close to 0) can be approximated by a GPD distribution
with log-likelihood
where are the order statistics of the
and k is the number of exceedances above u.
Proof. Obviously, if comes from one of the classes composing the training set, the upper end point of
is zero since the minimum possible distance between
and a point generated from f is zero. We can use Theorem 2 to approximate the exceedances of
over a high
threshold u using a GPD. The log-likelihood of the exceedances is
Using Lemma 1 and the fact that the upper endpoint of is zero, we have that
and thus ¯
. Plugging this in, we obtain the simplified likelihood in 3 up to additive
constants, parametrized only by . The MLE of
The above result motivates to use the statistic (4) in an hypothesis test to decide whether the new point can come from a known class. The following results provides the theoretical background for this test.
where is the dimension of the predictor space.
Proof. We first note that the survival function of behaves at zero as
where is the volume of the ball
, and the approximation holds as
Consequently, the distribution of
is in the max-domain of a GEV distribution with shape
parameter . This implies that 1/D has a regularly varying survival function
for some slowly varying function at infinity. The upper tail of 1/D is thus in the max-domain
of attraction of a GEV distribution with shape parameter ˜0. Rewriting the statistic
using
and we recognize the classical Hill estimator (De Haan and Ferreira, 2007) applied to the sequence of independent copies of 1/D. The consistency of the Hill estimator (cf., Theorem 3.2.2 in De Haan and Ferreira (2007)) and the fact that 1/D has shape parameter 1/p yields the desired result.
So far we have supposed that the upper end point of is zero. Under this null hypothesis, i.e., that the new point is known, the above theorem shows that the statistic
is approximately the negative reciprocal of the predictor space dimension. Moreover, it is possible to show that the estimator (4) is a special case of the classical Hill estimator (De Haan and Ferreira, 2007) and hence it is asymptotically normal under a second order condition. Using this fact, we could even derive asymptotic confidence intervals for
based on it. The assumption on k(n) is common in extreme value statistics to ensure the right trade-off between bias and variance of a tail estimator.
Under the alternative, namely that the new point is unknown, we can encounter two situations. First, if the support of the new class is overlapping with the support of the known classes supp(f), then
might have upper endpoint zero if
falls into supp(f). Second, if the supports are not overlapping or
), then the upper endpoint of
is strictly smaller than zero. In this second case, the statistic
can be shown to converge to zero.
be the upper endpoint of
and suppose that
. Choose the threshold u to be the order statistic
, and assume that
Then the statistic in (4) converges almost surely to zero, that is,
Proof. We note that
Since almost surely as
, this proves the assertion.
We can use this fact to do a first test and mark a new point is significantly larger than
1 as an unknown point. Note that we use the quantity
instead of
to stabilize the asymptotic variance and to avoid numerical instabilities as
0 as
. In fact, under a second order condition and by asymptotic normality of the Hill estimator (cf., Theorem 3.2.5 in De Haan and Ferreira (2007)), the quantity
is approximately normal with mean
1 and variance 1. With this step we have excluded all the points that have no possibility to belong to one of the classes of the training set.
If, on the other hand, is close to
1, we cannot reject the null hypothesis that
In this case, Theorem 3 with threshold
yields the tail approximation for
where is the fitted GPD distribution. There is thus the possibility that
was generated from one of the known classes. Obviously, we have to exclude points that have positive, but very small probability of being knowns. We therefore compute the size of the ball around
that contains a fixed amount
0 of mass of f. More precisely, we let
0 be the (1
of for some small
, which we can compute from (5) as
Then , and the smaller
the higher the training density
) around
. Thus, if
obtained for the new point
is significantly larger than the corresponding quantity for a generic point in the training set, we mark
as unknown since the density
around
is too small. Otherwise we mark it as known. The probability
should be chosen small enough to have a good approximation of the magnitude of the density around
the same time large enough to obtain a reliable estimate of
. We choose
since this is sufficiently small and
+
) is usually a very good approximation of the true quantile.
5.3 The GPDC algorithm
Using the results on the asymptotic behavior of the statistic , we propose the following algorithm to classify a new point
1. Compute the negated distances and each point in the training
2. Estimate using only the biggest k negated distances
3. Perform the hypothesis test
4. Compute the (1)-quantile of
) using the estimated GPD
5. If the radius is bigger than a given threshold t > 0 mark
as unknown, otherwise
Some of the steps require further explanation. The number of upper order statistics k in the second step that is used to obtain corresponds to the classical bias-variance trade-off of the Hill estimator. There are widely used diagnostics plots that help to determine the best k. The outcome of our GPDC algorithm does not strongly depend on the choice of k, as soon as it is chosen small enough and of the order
), as suggested by Theorem 5.
Step 3. and step 5. require the choice of the thresholds s and t to perform the hypothesis tests. It is common to fix the type I error to some probability such as 5%, and then to compute the threshold that realizes this error. To this end we require the distribution of the test statistic under the null hypothesis. Instead of relying on asymptotic results, we propose to execute the algorithm using all the points in the training set as unknown one after the other in a jackknife fashion and to obtain in this way each time
. We then jointly set the thresholds
2)% quantiles of
, respectively, such that we mark at most
% of the training data as unknown following Bonferroni’s correction for multiple testing (Shaffer, 1995). Note that this requires to compute O(nk log n) distances, roughly the same as in the training phase of the EVM. We can do this procedure at the beginning of the life of the algorithm and repeat it once in a while if new training data arise.
We emphasize that, contrary to what it is stated in the EVM paper (Rudd et al., 2018), one should be careful to choose hyper parameters based on cross-validation by randomly splitting the classes in the training set in knowns and unknowns. This can be misleading since known classes convey no information on the unknowns.
It is important to underline that this algorithm relies on the asymptotic approximation by a GPD and hence it is asymptotically exact as the number of training samples tends to infinity. Moreover, it is completely kernel free and its implementation is very efficient using fast nearest-neighbor searching (Arya et al., 1998). In fact, during the evaluation of a new point we have to compute only O(k log n) distances, whereas with the EVM we have to compute O(n) distances. We stress also that the algorithm supports incremental learning since it does not require any training procedure once the thresholds for the hypothesis tests are fixed. If the interest is not only in open set classification our algorithm can be completed with an incremental classifier that performs standard classification only on the points marked as knowns.
Considering all training data as belonging to only one class is not a limitation of the algorithm. In fact, if one distribution dominates the others (especially regarding the tail behaviour) and thus causes a mixture f that is not informative for all training classes, a simple solution is to train separate GPDCs for each class and marking a new point as unknown if all the classifiers mark it as unknown. The same consideration holds for the GEVC.
The GPDC in the previous section was based on the approximation of threshold exceedances by the GPD distribution according to Theorem 2. In this section we propose another algorithm to do open set classification without relying on the geometry of the observed data. It is based on the approximation by the GEV distribution given in Theorem 1, and we thus refer to it as the GEVC.
For simplicity, as before, all known classes are collapsed into one big class and we denote the training data with the probability density describing the distribution of the training data. For each i = 1, . . . , n, we compute the distance between the training point
and the nearest training point to it, i.e.,
This phase, using fast neighbor search, requires to compute only O(n log n) distances. Note that
the quantities
are bounded above by zero. Even though these random variables are not exactly independent, their dependence seems weak enough to still apply methods from extreme value theory. Using this and Lemma 1 we can fit a reversed Weibull distribution to , and thus estimating the distribution of
, i.e., the distribution of the maximum of the negated distances between a known point and all the remaining points in the training set. Denote this estimated reversed Weibull distribution function by
When a new point
arises and we want to mark it as known or unknown we consider the
statistic
in order to perform the following hypothesis test:
Note that this operation requires to compute only O(log n) distances. Under the null hypothesis,
is approximately a sample of the distribution
. Thus we expect the quantity
to be not too small, whereas under the alternative hypothesis this is not guaranteed. We can then fix a level 0 for the type I error to determine if the new point
is known or unknown. Given the estimated distribution
that approximates
, we reject the null hypothesis if
. In this case, the smallest distance of the new point to a training sample is too large, and we therefore mark it as unknown. Otherwise it is marked as known. Note that, also with the GEVC, the threshold
permits to control directly the type I error. Moreover, the GEVC has no hyper parameters and it is fast to update when we add new data to the training set, since revising
requires only to compute the minimum distance between
and the added points.
Figure 1: Left panel: the simulated dataset for the toy example: training data (in black), the known (bright yellow) and unknown examples (dark yellow) from the test set. Right panel: the estimates for known (bright yellow) and unknown test data (dark yellow) for the simulated dataset.
In this section we compare the EVM introduced in Rudd et al. (2018) with our GPDC and the GEVC on simulated and real data. For completeness, when using real data, we compare the methods also with the One-Class SVM (Sch¨olkopf et al., 2000), a state of the art method for novelty detection. For this last technique, we use radial kernel and we select its hyper parameters
Figure 2: Left panel: decision boundaries for the GPDC (in red) and the GEVC (in magenta) with 01. Right panel: decision boundaries for the GPDC (in red) and the GEVC (in magenta) with
using part of the data as validation set, including both known and unknown objects, following what is proposed in the original paper. This approach is possible only in an experimental setting, since in real open world recognition we do not have any access to the unknown objects during the training phase.
7.1 Simulated data
We begin our experimental evaluation with a toy example that shows that the EVM may perform poorly when the geometry given by the training data set conveys misleading information about the unknown classes. The training set has n = 600 observations and it is composed of three classes, each one of them is sampled from a different bivariate normal distribution, i.e., the dimension of the predictor space is p = 2. The test data set has 800 observations and it is composed of examples from these three classes plus another unknown class, sampled from another bivariate normal distribution. Both the train and the test data are shown in the left panel of Figure 1. The unknown objects are well separated from the known ones.
We apply the EVM, the GPDC and the GEVC to this data set and, to solve the problem given by the fact that the probability threshold of the EVM is not interpretable as those of the GPDC and the GEVC, we choose to evaluate the performance of the different algorithms using the obtained area under the ROC Curve (AUC) (Bradley, 1997) that does not require to fix a threshold. For both the EVM and the GPDC we use k = 20, but the results are stable for different values of k. The EVM performs poorly in this rather simple scenario and has an AUC of 0.853. This is due to the fact that the unknown class is relatively close to the known
Figure 3: Results for the OLETTER protocol for the EVM (green line), the GPDC (red line), the GEVC (magenta line) and the One-Class SVM (blue line) with one standard deviation confidence intervals (dotted lines).
class at the bottom compared to the other known classes, even if it is perfectly separated from it. Conversely, the GPDC and the GEVC do not suffer of this type of issues since they do not rely on the distances between the known classes to infer about the unknown ones, and their results in this toy example are close to perfect for both algorithms, with AUC of 0.997 and 0.999, respectively. To show the effectiveness of the GPDC to determine whether new examples are in the support of the training distribution, we report the estimated of the test in point 3. of the algorithm in Section 5.3 for both known and unknown new data. The right panel of Figure 1 shows for this dataset that the
estimates for known data are, as expected by Theorem 4, close to
2. On the other hand, the estimates corresponding to unknown data are much closer to 0, as it is suggested by Theorem 5. The first test of the GPDC algorithm thus already has a good power to filter out the unknown classes for this data set. We note that, strictly speaking, the supports of all classes are overlapping, but for this finite number of data they are effectively separated so that the test works well.
Figure 2 shows the decision boundaries of the GEVC and the GPDC for 01 and
1. It can be seen that both algorithms produce highly flexible decision boundaries that are capable to follow the shape of the training data. In some sense, one can see these boundaries as level sets of the training density, and a new point is considered as unknown if it lies outside of these level sets. He and Einmahl (2017) also considered extrapolation of level sets into low density regions but using the more restrictive assumption of multivariate regular variation.
7.2 OLETTER protocol
In order to evaluate the open set classification performance of the GPDC and the GEVC for real data we compare the two techniques with the EVM and the One-Class SVM using the OLETTER protocol proposed in Bendale and Boult (2015). This protocol is based on the LETTER data set (Frey and Slate, 1991) that contains a total of 20000 observations of 26 different classes corresponding to handwritten letters. The predictor space is composed of p = 16 features that have been extracted from the handwritten letters. The training set has 15000 observations. The protocol consists in randomly selecting 15 classes that are considered as known during training and adding unknown classes by incrementally including subsets of the remaining 11 classes during testing, varying in this way the amount of openness. This process is then repeated 20 times, in a cross-validation fashion. We evaluate the performance of the different algorithms using the obtained F-measure (Huang and Ling, 2005) as done in the original paper (Bendale and Boult, 2015). For all the four algorithms we set dynamically the probability threshold using the heuristic rule proposed in Rudd et al. (2018) that accounts for the amount of openness at each step of the protocol. We set the hyper parameter k = 75 for the EVM as in original paper, and k = 22 for the GPDC, roughly corresponding to use the 0.25% of the biggest negated distances.
The results for the EVM, the GPDC and the GEVC are reported in Figure 3. It can be seen that all the three methods based on EVT perform better than the One-Class SVM, even if they have fewer hyper parameters than the latter method. Moreover, the hyper parameters of the One-Class SVM were tuned using also unknown examples and that is not realistic during a real use of an open set classifier. The GPDC is the best method in this case, while the GEVC is competitive with the other two even if it has no hyper parameters.
7.3 Diagnostics of thyroid disease
We consider an application of our algorithms for open set classification to diagnostics of thyroid disease. We analyse the thyroid dataset (Quinlan et al., 1986; Schiffmann et al., 1992), available at the UCI machine learning repository (Dua and Graff, 2017). The original dataset contains raw clinical measurements from healthy non-hypothyroid and sick patients. These raw measurements were pre-processed in order to easily apply directly neural networks and other common machine learning techniques, resulting in 21 features for each patient. In the final dataset there are 250 sick subjects and 6666 healthy subjects. We consider sick subjects as unknown objects and healthy subjects as known objects. We use all the 250 sick patients together with 250 randomly selected healthy patients to compose the test set. All the remaining healthy patients compose the training set.
Since in this case the known objects belong to only one class, the EVM cannot be used with this dataset. For this reason, we compare the GEVC and the GPDC only with the One-Class SVM. Also here, we use the AUC as evaluation measure. For the GPDC we train it with five different values of k, corresponding to using the most extreme 0.25%, 1%, 2.5%, 5% and 10%
Figure 4: Left panel: ROC curves obtained on the thyroid disease dataset for the One-Class SVM (blue line), the GPDC (red line) and the GEVC (magenta line). Right panel: AUC as a function of the number k of most extreme used distances for the GPDC.
distances, respectively. These are common choices in extreme value applications and finally we consider the value of k that gives the best ROC curve. For the One-Class SVM with radial kernel we consider different combinations of its hyper parameters and we retain the ones that gives the best results.
The ROC curves obtained for the GPDC, the GEVC and the One-Class SVM are shown in the left panel of Figure 4. The performance of the three methods are comparable, with an AUC of 0.931, 0.897, 0.912 respectively. We underline again this result since it is particularly favorable to both the GPDC and the GEVC that are reaching the same performance as a state of the art kernel based method like the One-Class SVM. These new methods are kernel free and the GPDC has only one hyper parameter that can be chosen based on EVT, whereas the GEVC has no hyper parameters at all.
The right panel of Figure 4 shows the performance of the GPDC as a function of the number k of most extreme distances used. It can be seen that the GPDC shows a good performance for a wide range of thresholds and the best results are achieved for fairyl small k as suggested wby EVT.
We present two new kernel free algorithms that perform open set classification using extreme value theory. These algorithms, called the GPDC and the GEVC, are fast to update with the arrival of new data and they are easy to adapt to an incremental framework. Moreover, they do not use the geometry of the known classes to infer about the unknowns and are thus able to overcome certain restrictions of previously proposed methods. Their performances are therefore good even when the unknown test data are close to a known class relatively to the other knowns. To show this fact and the effectiveness of the new GPDC and the GEVC, we compare them to the EVM (Rudd et al., 2018), another kernel free technique that uses EVT for open set classification, and the more classical kernel based One-Class SVM (Sch¨olkopf et al., 2000), a state of the art technique for novelty detection. The results of our methods on a simulated toy example and on real data sets are competitive. A major strength is that they perform well in very different situations. This good performance in general tasks is probably due to the fact that our methods rely on well-established statistical methods and asymptotically motivated approximations from univariate extreme value theory that apply under very mild conditions. We also underline that both the GPDC and the GEVC are computationally faster than the EVM during the evaluation phase.
The GPDC and the GEVC might be further improved by suitable representations of the data, for example using convolutional neural networks to extract features when working with images and we plan to perform more detailed evaluation of their performance in this direction on standard datasets such as ImageNet (Deng et al., 2009) or CIFAR-100 (Krizhevsky and Hinton, 2009). Furthermore, to be fully incremental, they need to be capable to store only a subset of the training data. This may be achieved developing a suitable sampling technique that reduces the dimensionality of the training dataset without affecting the asymptotic correctness of the algorithms.
The result in Theorem 4 on the shape parameter of sample distances can be of independent interest. In particular, it will be studied how this can be used for a general method for extrapolation of level sets into low density regions, similarly as in He and Einmahl (2017).
Sebastian Engelke was supported by the Swiss National Science Foundation; the paper was completed while he was a visitor at the Department of Statistical Sciences, University of Toronto.
N. Abe, B. Zadrozny, and J. Langford. Outlier detection by active learning. In International Conference on Knowledge Discovery and Data Mining. ACM, 2006.
S. Arya, D. M. Mount, N. S. Netanyahu, R. Silverman, and A. Y. Wu. An optimal algorithm for approximate nearest neighbor searching fixed dimensions. Journal of the ACM, 45(6), 1998.
A. Bendale and T. Boult. Towards open world recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2015.
C. M. Bishop. Novelty detection and neural network validation. IEE Proceedings-Vision, Image and Signal processing, 141(4), 1994.
A. P. Bradley. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition, 30(7), 1997.
M. B. Christopher. Pattern Recognition and Machine Learning. Springer-Verlag New York, 2016.
S. Coles, J. Bawa, L. Trenner, and P. Dorazio. An introduction to statistical modeling of extreme values. Springer, 2001.
L. De Haan and A. Ferreira. Extreme value theory: an introduction. Springer Science & Business Media, 2007.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In Conference on Computer Vision and Pattern Recognition, 2009.
C. D´esir, S. Bernard, C. Petitjean, and L. Heutte. One class random forests. Pattern Recognition, 46(12), 2013.
D. Dua and C. Graff. UCI machine learning repository, 2017. URL http://archive.ics.uci. edu/ml.
V. Fragoso, P. Sen, S. Rodriguez, and M. Turk. EVSAC: accelerating hypotheses generation by modeling matching scores with extreme value theory. In IEEE International Conference on Computer Vision, 2013.
P. W. Frey and D. J. Slate. Letter recognition using holland-style adaptive classifiers. Machine Learning, 6(2), 1991.
C. Geng, S. Huang, and S. Chen. Recent advances in open set recognition: A survey. Preprint arXiv:1811.08581, 2018.
A. Graves, A. Mohamed, and G. Hinton. Speech recognition with deep recurrent neural networks. In IEEE International Conference on Acoustics, Speech and Signal Processing, 2013.
P. Hall. On estimating the endpoint of a distribution. The Annals of Statistics, 10(2), 1982.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Yi He and John H. J. Einmahl. Estimation of extreme depth-based quantile regions. J. R. Stat. Soc. Ser. B. Stat. Methodol., 79:449–461, 2017.
J. Huang and C. X. Ling. Using AUC and accuracy in evaluating learning algorithms. IEEE Transactions on Knowledge and Data Engineering, 17(3), 2005.
H. Jalalzai, S. Cl´emen¸con, and A. Sabourin. On binary classification in extreme regions. In Advances in Neural Information Processing Systems, 2018.
G. James, D. Witten, T. Hastie, and R. Tibshirani. An introduction to statistical learning. Springer, 2013.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
T. Mensink, J. Verbeek, F. Perronnin, and G. Csurka. Metric learning for large scale image classification: Generalizing to new classes at near-zero cost. In European Conference on Computer Vision. Springer, 2012.
A. Nguyen, J. Yosinski, and J. Clune. Deep neural networks are easily fooled: high confidence predictions for unrecognizable images. In IEEE Conference on Computer Vision and Pattern Recognition, 2015.
M. A. F. Pimentel, D. A. Clifton, L. Clifton, and L. Tarassenko. A review of novelty detection. Signal Processing, 99, 2014.
J. R. Quinlan, P. J. Compton, K. A. Horn, and L. Lazarus. Inductive knowledge acquisition: a case study. In Proceedings of the second Australian Conference on the Applications of Expert Systems, 1986.
S. Rebuffi, A. Kolesnikov, and C. H. Lampert. iCaRL: Incremental classifier and representation learning. In Conference on Computer Vision and Pattern Recognition, 2017.
S. J. Roberts. Novelty detection using extreme value statistics. IEE Proceedings-Vision, Image and Signal Processing, 146(3), 1999.
E. M. Rudd, L. P. Jain, W. J. Scheirer, and T. E. Boult. The extreme value machine. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(3), 2018.
S. Ruping. Incremental learning with support vector machines. In IEEE International Conference on Data Mining, 2001.
A. Saffari, C. Leistner, J. Santner, M. Godec, and H. Bischof. On-line random forests. In IEEE International Conference on Computer Vision Workshops, 2009.
W. J. Scheirer. Extreme value theory-based methods for visual recognition. Synthesis Lectures on Computer Vision, 7(1), 2017.
W. J. Scheirer, A. Rocha, R. J. Micheals, and T. E. Boult. Meta-recognition: the theory and practice of recognition score analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(8), 2011.
W. J. Scheirer, L. P. Jain, and T. E. Boult. Probability models for open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(11), 2014.
W. Schiffmann, M. Joost, and R. Werner. Synthesis and performance analysis of multilayer neural network architectures. Technical report, University of Koblenz, 1992.
B. Sch¨olkopf, R. C. Williamson, A. J. Smola, J. Shawe-Taylor, and J. C. Platt. Support vector method for novelty detection. In Advances in Neural Information Processing Systems, 2000.
J. P. Shaffer. Multiple hypothesis testing. Annual review of psychology, 46(1), 1995.
T. Shon and J. Moon. A hybrid machine learning approach to network anomaly detection. Information Sciences, 177(18), 2007.
S. Walfish. A review of statistical outlier methods. Pharmaceutical Technology, 30(11), 2006.